| notebook.community

Today I watched a PyCon video on packaging to get a better feel of the tools avaialable and the problems faced by developers in packaging their softwares. (Ofcourse I watched a few other videos, read a few entries from the speakers blog etc.)

Then I began to fix the missing dependencies issue. The information to build a package are specified in the setup.py file of the package and because it is not a serialized data file like yaml or xml we cannot simply read the file and get that information. We also cannot simply regex setup.py because they python files are too flexible. To give an example, here is a the setup.py for package linecahe2:

import setuptools

setuptools.setup(
    setup_requires=['pbr'],
    pbr=True)

But the same requirements can also be specified as

import setuptools

requirements = ['pbr']
setuptools.setup(
    setup_requires=requirements,
    pbr=True)

Or the developer might decide to read the list from a file. And we cannot regex for such information, we need to execute setup.py. To solve the problem conda-skeleton monkeypatches the distutils module, to create a pkginfo.yaml file to contain all the setup information. It looks something like this:

patch
diff core.py core.py
--- core.py
+++ core.py
@@ -166,5 +166,40 @@ def setup (**attrs):
 \n
+# ====== BEGIN CONDA SKELETON PYPI PATCH ======
+
+import distutils.core
+import io
+import os.path
+import sys
+import yaml
+from yaml import Loader, SafeLoader
+
+# Override the default string handling function to always return unicode
+# objects (taken from StackOverflow)
+def construct_yaml_str(self, node):
+    return self.construct_scalar(node)
+Loader.add_constructor(u'tag:yaml.org,2002:str', construct_yaml_str)
+SafeLoader.add_constructor(u'tag:yaml.org,2002:str', construct_yaml_str)
+
+def setup(*args, **kwargs):
+    data = {{}}
+    data['install_requires'] = kwargs.get('install_requires', [])
+    data['extras_require'] = kwargs.get('extras_require', {{}})
+    data['entry_points'] = kwargs.get('entry_points', [])
+    data['packages'] = kwargs.get('packages', [])
+    data['setuptools'] = 'setuptools' in sys.modules
+    data['summary'] = kwargs.get('description', None)
+    data['homeurl'] = kwargs.get('url', None)
+    data['license'] = kwargs.get('license', None)
+    data['name'] = kwargs.get('name', '??PACKAGE-NAME-UNKNOWN??')
+    data['classifiers'] = kwargs.get('classifiers', None)
+    data['version'] = kwargs.get('version', '??PACKAGE-VERSION-UNKNOWN??')
+    with io.open(os.path.join("{}", "pkginfo.yaml"), 'w', encoding='utf-8') as fn:
+        fn.write(yaml.dump(data, encoding=None))
+
+
+# ======= END CONDA SKELETON PYPI PATCH ======

I noticed that it doesn't store the setup_requires information in pkginfo so I had to patch the patch, read the info and add it to the list of dependencies which is later added to meta.yaml. It was just a 5 line patch :)

patch
diff --git a/conda_build/pypi.py b/conda_build/pypi.py
index 5c1c631..3235d11 100644
--- a/conda_build/pypi.py
+++ b/conda_build/pypi.py
@@ -128,7 +128,7 @@ DISTUTILS_PATCH = '''\
 diff core.py core.py
 --- core.py
 +++ core.py
-@@ -166,5 +167,39 @@ def setup (**attrs):
+@@ -166,5 +167,40 @@ def setup (**attrs):
  \n
 +# ====== BEGIN CONDA SKELETON PYPI PATCH ======
 +
@@ -149,6 +149,7 @@ diff core.py core.py
 +def setup(*args, **kwargs):
 +    data = {{}}
 +    data['install_requires'] = kwargs.get('install_requires', [])
++    data['setup_requires'] = kwargs.get('setup_requires', {{}})
 +    data['extras_require'] = kwargs.get('extras_require', {{}})
 +    data['entry_points'] = kwargs.get('entry_points', [])
 +    data['packages'] = kwargs.get('packages', [])
@@ -516,7 +517,8 @@ def get_package_metadata(args, package, d, data):
                      % ','.join(extras))
         #... and collect all needed requirement specs in a single list:
         requires = []
-        for specs in [pkginfo['install_requires']] + extras_require:
+        for specs in [pkginfo['install_requires']] + \
+                [pkginfo['setup_requires']] + extras_require:
             if isinstance(specs, string_types):
                 requires.append(specs)
             else:

I did not know the specification of the diff format and I was making mistakes in the line @@ -166,5 +166,40 @@ def setup (**attrs):, the wikipedia page helped me out, I also found a blog post Guido on it https://en.wikipedia.org/wiki/Diff_utility#Unified_format

TODO