Logging a Big Process

Lets say you have a web site. When the user clicks a link it runs a process that generates a huge report, lots of ins and outs. Lots of places where some of the data might be questionable, but not bad enough to give up. What you really want to do is warn the user. The problem is your code is pretty modular. You could pass around a variable to keep track of the issues, but wouldn’t it be better if there were a more unified approach? Some sort of error accumulator… maybe a logger. Wait that’s built in to Python. This works:

# other_module.py
import logging
logger = logging.getLogger('my logger')

def f3():
    logger.debug('test f3')

and

import logging
try:
    from cStringIO import StringIO      # Python 2
except ImportError:
    from io import StringIO
    
import other_module

logger = logging.getLogger('my logger')
logger.setLevel(logging.DEBUG)
logger.propagate = False
formatter = logging.Formatter('%(module)s.%(funcName)s:%(lineno)d - %(message)s')
log_stream = StringIO()
handler = logging.StreamHandler(log_stream)
handler.setFormatter(formatter)
logger.addHandler(handler)


def f1():
    logger.error('test f1')


def f2():
    logger.debug('test f2')


def complex_process():
    # Clear stream to limit errors to each call to main
    log_stream.seek(0)
    log_stream.truncate()

    f1()
    f2()
    other_module.f3()
    errors = log_stream.getvalue()
    print(errors)

complex_process()
complex_process()

Results in :

logging_example.f1:21 - test f1
logging_example.f2:25 - test f2
other_module.f3:8 - test f3

logging_example.f1:21 - test f1
logging_example.f2:25 - test f2
other_module.f3:8 - test f3

Warning: do not use logging.basicConfig() for this. As stated in the docs “it’s intended as a one-off simple configuration facility, only the first call will actually do anything: subsequent calls are effectively no-ops.” If you do use this function, it is likely it will not do anything and the logger you will end up using is the root logger. One big problem with that is you will receive logging messages from lots of other, unexpected modules, such as third party modules.

• logging