An effective logging strategy with Django

http://www.miximum.fr/wp-content/uploads/2012/09/mailbox-220x300.jpg

The more complex are my web projects, the more I realize the importance of a good logging strategy.

Working with Python and Django, I have access to excellent tools to log everything I want.

The python's standard logging module is really complete and functional, and gives you a bunch of loggers, handlers, filters, formatters and so on to log whatever you want wherever you want however you want.
The django's logging integration is very efficient and well documented.
Sentry is a log server that can catch and centralize your logs, errors and exceptions, and gives you great insights about your web app errors. Plus sentry's integration into django is excellent.

Problem is the documentation about logging is somewhat unclear. Well, rather than unclear, I should said incomplete. HOW to log is well documented, what's missing is info about a real logging strategy. WHAT, WHERE, WHEN should you log?

What's in a good logging strategy?

Logging information can serve different purposes. Here are a few use cases taken from Arrosware, my current side project :

When a user encounters a critical error, I want to be warned immediatly with an email. I should also be able to trace the source of the error, and retrieve all the needed data to reproduce it.
When a new account is created, I want to be warned immediatly, so I can check that everything is alright. I don't really need additional informations, though.
When a user tops up his sms account, I want that information to be stored, so I can retrieve it just in case something would go wrong. But I don't want to pollute my mailbox with that information.
Debug messages that I put into my code should appear in my development's environment, but must be discarded on my production server.

Now that I have my use cases, I can define specific actions depending on the log level :

DEBUG: Display the log on the console if in development environment, discard otherwise.
INFO: Send the log to syslog.
WARNING: Send the log to my mailbox.
ERROR: Send the log to sentry.

Obviously, actions must be cascading. An error message must be sent to sentry AND to my mailbox AND…

Now that we defined precisely our logging strategy, let's have a look at the technical implementation.

Logging in django

If you don't have a Sentry account yet, create one now. Believe me, you won't regret it. Then, you can install and configure raven, Sentry's Django client.

Now, open your django settings to configure logging:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        }
    },
    'formatters': {
        'verbose': {
            'format': '[contactor] %(levelname)s %(asctime)s %(message)s'
        },
    },
    'handlers': {
        # Send all messages to console
        'console': {
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
        },
        # Send info messages to syslog
        'syslog':{
            'level':'INFO',
            'class': 'logging.handlers.SysLogHandler',
            'facility': SysLogHandler.LOG_LOCAL2,
            'address': '/dev/log',
            'formatter': 'verbose',
        },
        # Warning messages are sent to admin emails
        'mail_admins': {
            'level': 'WARNING',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler',
        },
        # critical errors are logged to sentry
        'sentry': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'raven.contrib.django.handlers.SentryHandler',
        },
    },
    'loggers': {
        # This is the "catch all" logger
        '': {
            'handlers': ['console', 'syslog', 'mail_admins', 'sentry'],
            'level': 'DEBUG',
            'propagate': False,
        },
    }
}

And that's it! You can now use the standard python logging module to log anything you want.

# Any django views.py file
import logging


logger = logging.getLogger(__name__)

def some_view(request):
    logger.info('Log some action')

And you, what are your tips and tricks to handle your apps' logs?