Abandoning logging for structlog
The standard library’s logging
package is the default in the Python ecosystem. It works but it isn’t great. Setup is annoying:
import logging
logger = logging.getLogger(__name__)
# ...
logger.info("request method=%s", method)
The default formatting isn’t easy to parse and it doesn’t support structured logs.
structlog
is a better choice!
Some of the benefits
nice default formatting
And you can also use logfmt
in prod which most logging providers will parse automatically. Much better than having to look at json logs.
structured logs
You don’t have to munge a bunch of strings together yourself. No dealing with, “should I use %s
or %d
?”
So instead of:
log.info("processing item itemid=%s from queue request_id=%d", item_id, request_id)
we can:
log.info("processing item from queue", itemid=item_id, request_id=request_id)
or with bind
:
log = log.bind(item_id=item_id)
log.info("processing item from queue")
bind
You can build up context that gets shared with future log calls by calling .bind
with your log params.
log.info("starting up...")
user = get_user(request)
log = log.bind(user_id=user.id)
log.info("fetched user")
item = pop_from_queue(request)
log = log.bind(item_id=item.id)
log.info("processing item")
res = process_queue_item(item)
if not res.ok:
log.warning("failed to process item")
which results in the following when using the logfmt
renderer:
event="starting up..."
event="fetched user" user_id=usr_123
event="processing item" item_id=item_123 user_id=usr_123
event="failed to process item" item_id=item_123 user_id=usr_123
Structlog also integrates with ContextVars
and supports setting up processors
to munge your logs before they get written.
Conclusion
Use structlog
for logging in Python.