Creating and processing logs

Goals

importance of logging
configuring logger module
open log files

modules



In [ ]:

    
from __future__ import print_function

import logging
import logging.config
import logging.handlers
import mimetypes
import gzip, bz2
import yaml

Importance of Logging

All automation and scripting activity should be carefully logged.

The logging module:

can stream logs to files and network
is configurable via yml, dict, ini (deprecated)
optimizes output via log levels
takes care of rotation

Hint: manage logs with syslog.



In [ ]:

    
# Logs to stdout|err with a default loglevel.
# Logs are not printed on jupyter cells but you can  
#  check them in another terminal
logging.basicConfig(level=logging.DEBUG)

# Create a logger.
log = logging.getLogger()

# Logs supports a print-like syntax, and doesn't 
#  build unneeded message strings!
# still using old string interpolation in this casse is useful
log.info("Use %r instead of %s to avoid", [u"Serialization"], "issues")



In [ ]:

    
%cat logger.yml



In [ ]:

    
with open('logger.yml') as logger_config:
    logging.config.dictConfig(yaml.safe_load(logger_config))
    
# The ```os``` module goes to syslog
log.info("To syslog?")
log.error("To syslog")
log.debug("test debug")



In [ ]:

    
# To process compressed files, use an helper function. 
import mimetools
import gzip
import bz2

def log_open(path):
    """Open log files using its mimetype to choose the correct method"""
    l_type, l_encoding = mimetypes.guess_type(path)
    if l_encoding == 'gzip':
        return gzip.open(path, 'rb')
    elif l_encoding == 'bzip2':
        return bz2.BZ2File(path, 'rb')
    else:
        return open(path, 'rb')



In [ ]:

    
# Exercise:
#  log some messages modifying the default format string.
# use log_open to open gzip files or plain texts



In [ ]: