Packaging and distributing

In this lecture we will learn how to package a code and make it available through conda to easily distribute and install it.

Rules for good coding

First of all, you should as much as possible follow the style guide for coding in Python:

https://www.python.org/dev/peps/pep-0008/

This makes your code well organized and readable. It is better for you (if you are going to modify it in the future) and for other people who are trying to use or re-use your code.

Packages

A collection of modules in the same directory is called a package. For the package to be visible from Python, the directory must contain a file called __init__.py. The main purpose of this file is to signal to Python that the directory is a package and that all the other files in the directory ending in .py are importable.

The file __init__.py does not need to have any code in it. If it does, it is going to be executed before any other module in the package is imported.

The package takes the name of the directory and may have subdirectories called sub-packages. A package can have a structure like this:

packagename/
|---__init__.py
|---__main__.py
|---othermodule.py
|---more
|   |--__init.py__
|   |--moremodule.py
|   |--evenmoremodule.py
|---raw
|   |--data.txt
|   |--otherdata.txt
|   |--orphan.py

In this case the package packagename has three modules (__init__.py,__main__.py, othermodule.py) and one subpackage more. The raw directory does not count as subpackage because it does not contain the module __init__.py. This is true even if it contains the Python file orphan.py which is unreachable.

Inside a package you can import modules at the same level without giving the package name. This is called implicit relative importing:

import othermodule

To import a module from a subpackage:

import more.moremodule

Picking A Name

Python module/package names should generally follow the following constraints:

  • All lowercase
  • Unique on conda, even if you don’t want to make your package publicly available (you might want to specify it privately as a dependency later)
  • Underscore-separated or no word separators at all (don’t use hyphens)

Extra files

gitignore

Once completed a package, we have to add some extra files to upload them in remote repository. The first file will be a .gitignore. Typically, we should ignore any compiled file (.pyc) or extra files or directories that are created temporarily. For instance, if you create a project with spyder, a file called .spyderproject will exist in the directory.

So a minimum .gitignore will be:

# Compiled python modules.
*.pyc

# spyder project
.spyderproject

README

To make your project readable in github, you should add some explanations in a file called README.md. This can be simply a file with a title (# packagename), a description, some explanation and installation notes:

# packagename
This package does miracles

This package autoinstall itself and start regenerating your computer under your very eyes.

INSTALLATION NOTES
To install the file just call its name.

setup

To build the package in Python you will need a file called setup.py. An example is:

#!/usr/bin/env python

from distutils.core import setup

setup(name='packagename',
      version='0.1.alpha',
      description='The miracle package',
      author='Your name',
      author_email='yourname@gmail.com',
      url='https://github.com/yourname/packagename.git',
      scripts=['miracle']
     )

This will install your package and a script called miracle which resides in the package and allows one to start the package without calling it from Python.

Command line script

We can define a command line script, in the example miracle, simply calling the main program:

#!/usr/bin/env python
from sospex import __main__
__main__()

Conda files

Finally, you will need a few files to build a conda package. The first is called meta.yaml and contains a few fields describing the package, its source, the requirements, tests, and other info. Let's see an example:

package:
  name: packagename
  version: 0.1.alpha

source:
  git_rev: v0.1-alpha
  git_url: https://github.com/yourname/packagename.git

requirements:
  build:
    - python
    - setuptools
    - numpy

  run:
    - python
    - numpy
    - astropy >=1.2.1
    - scipy
    - wxpython


test:
  imports:
    - packagename

about:
  home: https://github.com/yourname/packagename
  license: GPL3
  license_file: LICENSE
  description: 'The package does miracles'

At last, we need files to install the package. One for UNIX and the other for WINDOWS. build.sh

python setup.py install     # Python command to install the script.

and build.bat

"%PYTHON%" setup.py install
if errorlevel 1 exit 1

We have finally to include the LICENSE file you decide to have.

Typically, you can choose your favorite license when you create a github repository. A choice of several licenses is available (including GNU public license 3.0). Otherwise, you can upload your own LICENSE.

Before proceeding any further, be sure to upload all these files on your git remote repository on github.

Developing the code

Once you created a package with a setup.py file it is possible to create a symlink and execute the code on your machine. This allows one to develop the code and immediately see if the modifications work. The command is:

pipy install . -e

Once you decide to create a conda package you will first unisntall this symlink and then create the conda package:

pipy uninstall package

where package is the name of your package.

Conda package

If you don't have it, at this point you need to install conda-build:

conda install conda-build

It is always better to check if the latest version is istalled:

conda upgrade conda
conda upgrade conda-build

At this point you can create your package:

conda build yourpackage

And you can install it locally and try it out:

conda install --use-local yourpackage

Upload new packages to Anaconda.org

After converting your files for use on other platforms, you may choose to upload your files to Anaconda.org. You will need an account on anaconda. If you haven’t already, open a free http://Anaconda.org account and record your new username and password.

Now, on your terminal, run:

conda install anaconda-client

and enter your Anaconda.org username and password. Again in your terminal window, log into your Anaconda.org account with the command:

anaconda login

Find out the name of your build with:

conda build yourpackage --output

If this name is: ~/anaconda/conda-bld/linux-64/yourpackage-0.12-py27_0.tar.bz, you can upload your package to Anaconda.org:

anaconda upload ~/anaconda/conda-bld/linux-64/yourpackage-0.12-py27_0.tar.bz

Once finished, logout:

anaconda logout

Now, you can search your package on Anaconda:

anaconda search -t conda yourpackage

obtaining something like this:

Packages:
     Name                      |  Version | Package Types   | Platforms      
     ------------------------- |   ------ | --------------- | ---------------
     darioflute/sospex         | 0.1.alpha | conda           | linux-64, osx-64
Found 1 packages

You can compile your program on another machine (for instance osx-64) and upload the other version. In this case, I compiled the same code on linux-64 and osx-64 and uploaded the two versions.

At this point, anybody can try your package by simply installing it from conda using your channel:

conda install -c yourchannel yourpackage

And, if everything works right, he can start directly using your script miracle.