Creating and processing logs

Goals

importance of logging
configuring logger module
open log files

modules



In [1]:

    
from __future__ import print_function

import logging
import logging.config
import logging.handlers
import mimetypes
import gzip, bz2
import yaml

Importance of Logging

All automation and scripting activity should be carefully logged.

The logging module:

can stream logs to files and network
is configurable via yml
optimizes output via log levels
takes care of rotation

Hint: manage logs with syslog.



In [2]:

    
# Logs to stdout|err with a default loglevel.
# Logs are not printed on jupyter cells but you can  
#  check them in another terminal with
#  #docker-compose logs
logging.basicConfig(level=logging.DEBUG)

# Create a logger.
log = logging.getLogger()

# Logs supports a print-like syntax, and doesn't 
#  build unneeded message strings. 
log.info("Use %r instead of %s to avoid", [u"Serialization"], "issues")



In [3]:

    
%cat logger.yml









    



version: 1
formatters:
  detailed: 
    class: logging.Formatter
    format: '%(asctime)s %(name)-15s %(levelname)-8s %(processName)-10s %(message)s'

handlers:
  console: 
    class: logging.StreamHandler
    level: INFO
  syslog: 
    class: logging.handlers.SysLogHandler
    formatter: detailed
    level: DEBUG

loggers:
  os:
    handlers: [syslog]

root:
  handlers: [console, syslog]
  level: DEBUG



In [4]:

    
with open('logger.yml') as logger_config:
    logging.config.dictConfig(yaml.safe_load(logger_config))
    
# The ```os``` module goes to syslog
log.info("To syslog")









    



To syslog



In [5]:

    
# To process compressed files, use an helper function. 
import mimetools
import gzip
import bz2

def log_open(path):
    """Open log files using its mimetype to choose the correct method"""
    l_type, l_encoding = mimetypes.guess_type(path)
    if l_encoding == 'gzip':
        return gzip.open(path, 'rb')
    elif l_encoding == 'bzip2':
        return bz2.BZ2File(path, 'rb')
    else:
        return open(path, 'rb')



In [6]:

    
# Exercise:
#  log some messages modifying the default format string.
#  check the outcome in a separate terminal with
#  docker-compose logs