By Your Bootstraps: Porting Your Application to Python3

Tres Seaver

Objectives

• "Straddling" Python 2/3 in a single codebase
• choosing target python versions
• porting as an iterative process
• adding test coverage to reduce risk
• ...

Background

• ~180 kLOC Python

Porting strategies

• Port once, abandon Python2
  • Not The subject of this talk
  • Customers/users still need Python2
  • More feasible for applications than libraries
  • 2to3 may be useful starting point
• "Fix up" at installation using 2to3
  • Ship py2 code and run 2to3 live for py3 users
  • Python2 users unaffected
  • Python3 source "drifts" from canonical version (bug reports don't match)
  • Bug reports line numbers don't match
  • 2to3 painfully slow on large codebases
• "Straddling" in a single codebase (most-used strategy today)
  • Use "compatible subset" of Python syntax
  • Conditional imports mask stdlib changes
  • six module can help (but you might not need it)
  • Win!

Targeting Python Versions

• Need to do know your targeted version to "straddle" well
• Python2.6 is the bare-minimum reasonable version to support
• Python3.2 is the bare-minimum reasonable version to support
• Syntax changes make Python2 < 2.6 hard
  • No b'' literals
  • No except Exception as e:
• Much more cruft/pain
• Incompatibilities make Python3 < 3.2 hard
  • PEP 3333 fixes WSGI in Py3k
  • callable() restored in 3.2
• u'' restored in 3.3
• Python3.2 is the "system Python3" in some LTS Linuxes
• Summary: 2.6+, 3.2+; better: 2.7+, 3.3+

Managing Porting Risks

• Ports are great opportunities for bug injection
• Fear of breaking working software is the barrier
  • even more than the effort required
• Some mitigations also improve your software
  • improved testing
  • modernized idioms in python2
  • clarity in text vs. bytes

Bottom-up Porting

• Port packages with no dependencies first
• Then port packages with already-ported dependencies
  • Note python versions supported by dependencies

• Lather, rinse, repeat

• "Common subset" idioms

• Lennart Regebro's book: http://python3porting.com/noconv.html
• python2.7 -3 can point out problem areas
• "Modernize" idioms in Python2 code
• Distinguishing bytes vs. text
  • use b''/u'' for all literals; you want to always know what you're working with, and you should be explicit
• Adopt new syntax
  • except ... as ...
  • print()

• Use new stdlib facilities

  • StringIO.StringIO --> io.StringIO

• Testing Avoids Hair-Loss

• Make good test coverage the first step of the porting process!
• Untested code is where the bugs go to hide
• 100% coverage is ideal before porting
  • unit testing preferable for libraries
  • functional testing best for applications
• Measuring coverage
  • https://pypi.python.org/pypi/coverage
• Work to improve assertions as well as coverage
  • Assert contracts, not implementation details
• Doctests make straddling really hard
  • Think of doctests as documentation with verifiable examples
  • Replace doctests with unit/functional tests
  • If at all feasible, convert doctests to Sphinx examples
• Automate running tests
• tox helps ensure that tests pass under all supported versions
  • creates a virtualenv for each version and runs testing command
  • Testing with pypy and other versions
• Testing C extensions is harder; need a reference implementation

Hygeine Issues

• Use Trove classifiers in your setup.py file
• Consider bumping your major version number
• Apply continuous integration
  • Travis-CI, Jenkins, Shining Panda for Windows

Resources

• http://python3porting.com
• https://testrun.org/tox/latest/
• http://pypi.python.org/pypi/six