Handling version compatibility is one of the hardest challenges in building software. Up to now, conda-build provided helpful tools in terms of the ability to constrain or pin versions in recipes. The limiting thing about this capability was that it entailed editing a lot of recipes. Conda-build 3 introduces a new scheme for controlling version constraints, which enhances behavior in two ways. First, you can now set versions in an external file, and you can provide lists of versions for conda-build to loop over. Matrix builds are now much simpler and no longer require an external tool, such as conda-build-all. Second, there have been several new Jinja2 functions added, which allow recipe authors to express their constraints relative to the versions of packages installed at build time. This dynamic expression greatly cuts down on the need for editing recipes.
Each of these developments have enabled interesting new capabilities for cross-compiling, as well as improving package compatibility by adding more intelligent constraints.
This document is intended as a quick overview of new features in conda-build 3. For more information, see the docs PR at https://conda.io/docs/building/variants.html
These demos use conda-build's python API to render and build recipes. That API currently does not have a docs page, but is pretty self explanatory. See the source at https://github.com/conda/conda-build/blob/master/conda_build/api.py
This jupyter notebook itself is included in conda-build's tests folder. If you're interested in running this notebook yourself, see the tests/test-recipes/variants folder in a git checkout of the conda-build source. Tests are not included with conda packages of conda-build.
In [ ]:
from conda_build import api
import os
from pprint import pprint
First, set up some helper functions that will output recipe contents in a nice-to-read way:
In [ ]:
def print_yamls(recipe, **kwargs):
yamls = [api.output_yaml(m[0])
for m in api.render(recipe, verbose=False, permit_unsatisfiable_variants=True, **kwargs)]
for yaml in yamls:
print(yaml)
print('-' * 50)
In [ ]:
def print_outputs(recipe, **kwargs):
pprint(api.get_output_file_paths(recipe, verbose=False, **kwargs))
Most of the new functionality revolves around much more powerful use of jinja2 templates. The core idea is that there is now a separate configuration file that can be used to insert many different entries into your meta.yaml files.
In [ ]:
!cat 01_basic_templating/meta.yaml
In [ ]:
# The configuration is hierarchical - it can draw from many config files. One place they can live is alongside meta.yaml:
!cat 01_basic_templating/conda_build_config.yaml
Since we have one slot in meta.yaml, and two values for that one slot, we should end up with two output packages:
In [ ]:
print_outputs('01_basic_templating/')
In [ ]:
print_yamls('01_basic_templating/')
OK, that's fun already. But wait, there's more!
We saw a warning about "finalization." That's conda-build trying to figure out exactly what packages are going to be installed for the build process. This is all determined before the build. Doing so allows us to tell you the actual output filenames before you build anything. Conda-build will still render recipes if some dependencies are unavailable, but you obviously won't be able to actually build that recipe.
In [ ]:
!cat 02_python_version/meta.yaml
In [ ]:
!cat 02_python_version/conda_build_config.yaml
In [ ]:
print_yamls('02_python_version/')
Here you see that we have many more dependencies than we specified, and we have much more detailed pinning. This is a finalized recipe. It represents exactly the state that would be present for building (at least on the current platform).
So, this new way to pass versions is very fun, but there's a lot of code out there that uses the older way of doing things - environment variables and CLI arguments. Those still work. They override any conda_build_config.yaml settings.
In [ ]:
# Setting environment variables overrides the conda_build_config.yaml. This preserves older, well-established behavior.
os.environ["CONDA_PY"] = "3.4"
print_yamls('02_python_version/')
del os.environ['CONDA_PY']
In [ ]:
# passing python as an argument (CLI or to the API) also overrides conda_build_config.yaml
print_yamls('02_python_version/', python="3.6")
Wait a minute - what is that h7d013e7 gobbledygook in the build/string field?
Conda-build 3 aims to generalize pinning/constraints. Such constraints differentiate a package. For example, in the past, we have had things like py27np111 in filenames. This is the same idea, just generalized. Since we can't readily put every possible constraint into the filename, we have kept the old ones, but added the hash as a general solution.
There's more information about what goes into a hash at https://conda.io/docs/building/variants.html#differentiating-packages-built-with-different-variants
Let's take a look at how to inspect the hash contents of a built package.
In [ ]:
outputs = api.build('02_python_version/', python="3.6",
anaconda_upload=False)
pkg_file = outputs[0]
print(pkg_file)
In [ ]:
# using command line here just to show you that this command exists.
!conda inspect hash-inputs ~/miniconda3/conda-bld/osx-64/abc-1.0-py36hd0a5620_0.tar.bz2
pin_run_as_build is a special extra key in the config file. It is a generalization of the x.x concept that existed for numpy since 2015. There's more information at https://conda.io/docs/building/variants.html#customizing-compatibility
Each x indicates another level of pinning in the output recipe. Let's take a look at how we can control the relationship of these constraints. Before now you could certainly accomplish pinning, it just took more work. Now you can define your pinning expressions, and then change your target versions only in one config file.
In [ ]:
!cat 05_compatible/meta.yaml
This is effectively saying "add a runtime libpng constraint that follows conda-build's default behavior, relative to the version of libpng that was used at build time"
pin_compatible is a new helper function available to you in meta.yaml. The default behavior is: exact version match lower bound ("x.x.x.x.x.x.x"), next major version upper bound ("x")
In [ ]:
print_yamls('05_compatible/')
These constraints are completely customizable with pinning expressions:
In [ ]:
!cat 06_compatible_custom/meta.yaml
In [ ]:
print_yamls('06_compatible_custom/')
Finally, you can also manually specify version bounds. These supersede any relative constraints.
In [ ]:
!cat 07_compatible_custom_lower_upper/meta.yaml
In [ ]:
print_yamls('07_compatible_custom_lower_upper/')
Much of the development of conda-build 3 has been inspired by improving the compiler toolchain situation. Conda-build 3 adds special support for more dynamic specification of compilers.
In [ ]:
!cat 08_compiler/meta.yaml
By replacing any actual compiler with this jinja2 function, we're free to swap in different compilers based on the contents of the conda_build_config.yaml file (or other variant configuration). Rather than saying "I need gcc," we are saying "I need a C compiler."
By doing so, recipes are much more dynamic, and conda-build also helps to keep your recipes in line with respect to runtimes. We're also free to keep compilation and linking flags associated with specific "compiler" packages - allowing us to build against potentially multiple configurations (Release, Debug?). With cross compilers, we could also build for other platforms.
In [ ]:
!cat 09_cross/meta.yaml
In [ ]:
# but by adding in a base compiler name, and target platforms, we can make a build matrix
# This is not magic, the compiler packages must already exist. Conda-build is only following a naming scheme.
!cat 09_cross/conda_build_config.yaml
In [ ]:
print_yamls('09_cross/')
Finally, it is frequently a problem to remember to add runtime dependencies. Sometimes the recipe author is not entirely familiar with the lower level code, and has no idea about runtime dependencies. Other times, it's just a pain to keep versions of runtime dependencies in line. Conda-build 3 introduces a way of storing the required runtime dependencies on the package providing the dependency at build time.
For example, using g++ in a non-static configuration will require that the end-user have a sufficiently new libstdc++ runtime library available at runtime. Many people don't currently include this in their recipes. Sometimes the system libstdc++ is adequate, but often not. By imposing the downstream dependency, we can make sure that people don't forget the runtime dependency.
In [ ]:
# First, a package that provides some library.
# When anyone uses this library, they need to include the appropriate runtime.
!cat 10_runtimes/uses_run_exports/meta.yaml
In [ ]:
# This is the simple downstream package that uses the library provided in the previous recipe.
!cat 10_runtimes/consumes_exports/meta.yaml
In [ ]:
# let's build the former package first.
api.build('10_runtimes/uses_run_exports', anaconda_upload=False)
In [ ]:
print_yamls('10_runtimes/consumes_exports')
In the above recipe, note that bzip2 has been added as a runtime dependency, and is pinned according to conda-build's default pin_compatible scheme. This behavior can be overridden in recipes if necessary, but we hope it will prove useful.