Notable Stuff from PyCon 2017

Portland, OR May 17 - 25, 2017

General PyCon 2017 Resources

  • List of all PyCon 2017 talks and descriptions is available here
  • Slides for many PyCon 2017 talks are on SpeakerDeck
  • Videos of all PyCon 2017 talks are on YouTube

Top Python Tools Enabling Python to Scale

These tools are being used by large enterprise companies to service hundreds of millions of monthly users with stacks based on Python and Django. Companies like Instagram, Dropbox, and Yelp are 100% Python and manage to scale to handle on the order of 1 billion users.

  1. Python 3.6 - First version of Python 3 that is better than 2.7
    • Lisa Guo and Hui Ding from Instagram gave a great keynote presentation on Python@Instagram
    • Results
      • Instagram ported their entire codebase from Python 2.7 to Python 3.6
      • Saved 12% CPU and 30% RAM globally
      • Took a team of developers 9 months
    • Reasons for porting
      • Optional type hinting
      • Better performance
      • Python 2.7 end-of-life in 2020
    • Biggest challenge
      • Unicode vs. byte string differences
  2. Cython - C-Extensions for Python
    • Excellent talk by Alex Orlov from Instagram on Cython as a Game Changer for Efficiency
    • Entire Instagram stack runs on Python 3 w/ small amounts of Cython
      • Services 700+ million users every month
    • Instagram sped up their overall entire codebase by 30%
      • Saving them millions of $ in server/computing costs
      • By Cythonizing < 1% of their code (10-15 files)
        • Critical files chosen after extensive profiling
      • Took only a few hours of developer time per file
      • Amazing return on investment (ROI) for the company!
        • Total time spent is 2 engineers, 2 months each!!
  3. SWIG - Simplified Wrapper and Interface Generator
    • Numerous talks on Tensorflow
      • Tensorflow is an open source library for machine intelligence developed by Google
        • One of the most popular Python libraries ever
      • It is fundamentally a C++ library, but has bindings for Python, Java, and Go
      • SWIG is what is used to generate all of these wrappers around the C++ code
    • SWIG auto-generates wrappers for existing C/C++ code
    • SWIG can target about 20 different programming languages
      • Python, Java, C#, Go, PHP, Javascript, Perl, Ruby, R, etc.
    • You create *.i interface files which tell it what and how to wrap, SWIG does the rest
  4. Mypy - Optional Static Typing for Python
    • Jukka Lehtosalo and David Fisher from Dropbox gave a good talk on Static Types for Python
    • mypy is an optional static type checker (static analysis tool) for Python
      • Relies on optional type hints available in Python 3.5 and 3.6
    • Optional type hints also make your code easier to read and maintain
    • Developed mainly by Dropbox to help with their millions of lines of Python
    • Can get benefits similar to a compiler finding stupid mistakes before runtime
    • This is what Guido van Rossum, Python BDFL has been working on

Cython

Cython is an optimising static compiler for both the Python programming language and the extended Cython programming language (based on Pyrex). It makes writing C extensions for Python as easy as Python itself. Cython gives you the combined power of Python and C to let you:

  • write Python code that calls back and forth from and to C or C++ code natively at any point
  • easily tune readable Python code into plain C performance by adding static type declarations

In [1]:
# Pure Python Fibonacci example
def compute_fibonacci(n):
    """Computes fibonacci sequence"""
    a = 1
    b = 1
    intermediate = 0
    for x in range(n):
        intermediate = a
        a += b
        b = intermediate
    return a

In [2]:
# To enable support for Cython compilation, install Cython and load the Cython extension from within Jupyter Notebook:
%load_ext Cython

In [3]:
%%cython
# Cython version of Fibonacci example
cpdef int compute_fibonacci_cython(int n):
    """ Compute the nth fibonacci number in a non-recursive fashion."""
    cdef int a, b, intermediate, x
    a = 1
    b = 1
    for x in range(n):
        intermediate = a
        a += b
        b = intermediate
    return a

In [4]:
N = 100
%timeit compute_fibonacci(N)


100000 loops, best of 3: 6.36 µs per loop

In [5]:
%timeit compute_fibonacci_cython(N)


The slowest run took 12.87 times longer than the fastest. This could mean that an intermediate result is being cached.
10000000 loops, best of 3: 96.7 ns per loop

In [6]:
6.36e-6/96.7e-9


Out[6]:
65.77042399172699

Python 3.5 and 3.6 Optional Type Hinting

The typing module supports type hints. Type hints for function arguments and return types was added in Python 3.5. Type hints for local variables was added in Python 3.6.

Using type hints improves the readability and maintainability of code. Additionally, IDEs and static analysis tools can verify that the type hints are being complied with.


In [7]:
# The function below takes and returns a string and is annotated as follows:
def greeting(name: str) -> str:
    return 'Hello ' + name

In [8]:
greeting('Todd')


Out[8]:
'Hello Todd'

In [9]:
greeting(1)


---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-f68ae3e5573d> in <module>()
----> 1 greeting(1)

<ipython-input-7-da17015b8057> in greeting(name)
      1 # The function below takes and returns a string and is annotated as follows:
      2 def greeting(name: str) -> str:
----> 3     return 'Hello ' + name

TypeError: must be str, not int

Mypy

Mypy is an optional static type checker for Python that aims to combine the benefits of dynamic (or "duck") typing and static typing. Mypy combines the expressive power and convenience of Python with a powerful type system and compile-time type checking

Other Interesting Tools Discussed at PyCon

  1. Micropython
    • Lean and efficient implementation of Python aimed at microcontrollers
    • Runs in 256 kB code space and 16 kB RAM
    • Includes a small subset of the Python 3 programming language and library
    • pyboard hardware from AdaFruit
    • CircuitPython sister project
    • Micropython documentation
    • Number of talks
      • IoT Lab with Micropython by Sev Leonard
      • The Memory Chronicles: A Tale of Two Pythons by Kavya Joshi
      • Look mum no hands! From blinking LEDs to a bike speedometer with MicroPython by Tim Head
  2. PyPy JIT is now mature and ready for commercial use (for Python 2.7 equivalent, Python 3.5 version is beta quality)
    • Fast alternative implementation of Python in Python
    • Typically much faster than CPython due to a just-in-time compiler (JIT)
      • On average, about 7.6 times faster
    • Typically uses less memory as well due to garbage collector improvements
    • No need to change your code at all - just run with pypy instead of python
  3. Python Vehicle Interface Toolkit (pyvit)
    • Hacking Cars with Python talk
    • Author is Eric Evenchick
      • Works at Atredis, Canadian security company
      • Owns small company LinkLayer which makes cheap open-source hardware for car hacking
        • CANtact CAN to USB converter for $60
    • pyvit deals with making the following things human-readable:
      • CAN Bus - protocol, frames, etc.
      • ISOTP
      • Unified Diagnostic Services (UDS)
  4. Conda Constructor
    • Tool for constructing a custom Anaconda installer
    • Free and open source
    • Build just what your customer needs, no need for 470 MB installer
    • Can have newer version of Python and packages than OS package manager
  5. Facebook has developed a Python bytecode to ebpf bytecode transpiler
    • Allows you to run Python bytecode inside the Linux kernel
    • Plans to open source it soon
    • Useful for writing advanced tracing and packet inspection/routing tools
    • Alex Gartrell from Facebook gave a presentation on Executing python functions in the Linux kernel by transpiling to ebpf
  6. Intel has the new Intel Distribution for Python
    • Free Python distribution
    • Focused on improved performance
      • Due to linking against Intel Math Kernel Library (MKL)
    • They also have tools (with I think they sell) which provide simultaneous C++ and Python debugging and profiling inside Visual Studio

Florida PyCon

Florida PyCon will be the first Saturday in October (2017-10-07) from 8AM to 5PM

The conference will be held in Orlando, FL at the Exchange downtown: Church Street Station Exchange 101 S Garland Ave Orlando, FL 32801

cmd2

Recently I have been maintaining the cmd2 open-source Python library for rapidly developing interactive command-line applications. It can be used to easily create a full-featured shell with capabilities similar to Bash out-of-the-box.

I met with the original author at PyCon and got joint ownership of the PyPI account so that I can publish releases and users can install the latest version via pip.

You can learn more about cmd2 by reading the documentation. Or if you are like me, just go look at some of the examples on GitHub.


In [ ]: