Adapted from swcarpentry/make-novice repository.
GNU Make is a free, fast, well-documented, and very popular Make implementation. From now on, we will focus on it, and when we say Make, we mean GNU Make.
Cells that follow are the result of following this Makefiles tutorial.
NB: I have adapted the tutorial so that the steps take place in this Jupyter notebook so that the notebook can be transpiled into a Pelican blog post using a danielfrg/pelican-ipynb Pelican plugin. Some of the code is what is necessary to display output in the notebook and therefore the blog post.
Some Jupyter notebook housekeeping to set up some variables with path references.
In [1]:
import os
In [150]:
from IPython.core.display import Image, display
In [2]:
(
TAB_CHAR,
) = (
'\t',
)
In [3]:
home = os.path.expanduser('~')
repo_path is the path to a clone of swcarpentry/make-novice
In [4]:
repo_path = os.path.join(
home,
'Dropbox/spikes/make-novice',
)
In [5]:
assert os.path.exists(repo_path)
paths are the paths to child directories in a clone of swcarpentry/make-novice
In [6]:
paths = (
'code',
'data',
)
paths = (
code,
data,
) = [os.path.join(repo_path, path) for path in paths]
assert all(os.path.exists(path) for path in paths)
In [20]:
format_context = dict(zip(
('repo_path', 'data', 'code', 'tab',),
(repo_path, data, code, TAB_CHAR))
)
In [107]:
makefile_contents_0 = """# Count words.
{repo_path}/isles.dat : {data}/books/isles.txt
{tab}python {code}/wordcount.py {data}/books/isles.txt {repo_path}/isles.dat
""".format(**format_context)
Using the shell to create Makefile with contents the value of Python variable makefile_contents.
In [108]:
!printf "$makefile_contents_0" > Makefile
This is a build file, which for Make is called a Makefile - a file executed by Make. Note how it resembles one of the lines from our shell script.
# Count words.
/home/dmmmd/Dropbox/spikes/make-novice/isles.dat : /home/dmmmd/Dropbox/spikes/make-novice/data/books/isles.txt
python /home/dmmmd/Dropbox/spikes/make-novice/code/wordcount.py /home/dmmmd/Dropbox/spikes/make-novice/data/books/isles.txt /home/dmmmd/Dropbox/spikes/make-novice/isles.dat
Let us go through each line in turn:
- # denotes a comment. Any text from # to the end of the line is ignored by Make.
- isles.dat is a target, a file to be created, or built.
- books/isles.txt is a dependency, a file that is needed to build or update the target. Targets can have zero or more dependencies.
- A colon, :, separates targets from dependencies.
- python wordcount.py books/isles.txt isles.dat is an action, a command to run to build or update the target using the dependencies. Targets can have zero or more actions. These actions form a recipe to build the target from its dependencies and can be considered to be a shell script.
- Actions are indented using a single TAB character, not 8 spaces. This is a legacy of Make’s 1970’s origins. If the difference between spaces and a TAB character isn’t obvious in your editor, try moving your cursor from one side of the TAB to the other. It should jump four or more spaces.
- Together, the target, dependencies, and actions form a a rule.
Let’s first sure we start from scratch and delete the .dat and .png files we created earlier
In [38]:
!rm $repo_path/*.dat $repo_path/*.png
By default, Make looks for a Makefile, called Makefile, and we can run Make as follows
In [39]:
!make
# By default, Make prints out the actions it executes:
Let’s see if we got what we expected
In [40]:
!head -5 $repo_path/isles.dat
We don’t have to call our Makefile Makefile. However, if we call it something else we need to tell Make where to find it. This we can do using -f flag. For example, if our Makefile is named MyOtherMakefile:
In [109]:
!printf "$makefile_contents_0" > MyOtherMakeFile.mk
In [110]:
!make -f MyOtherMakeFile.mk
This is because our target, isles.dat, has now been created, and Make will not create it again. To see how this works, let’s pretend to update one of the text files. Rather than opening the file in an editor, we can use the shell touch command to update its timestamp (which would happen if we did edit the file)
In [57]:
!touch $data/books/isles.txt
If we compare the timestamps of books/isles.txt and isles.dat,
In [58]:
!ls -l $data/books/isles.txt $repo_path/isles.dat
# then we see that isles.dat, the target, is now older thanbooks/isles.txt, its dependency
If we run Make again,
In [59]:
!make
#then it recreates isles.dat
When it is asked to build a target, Make checks the ‘last modification time’ of both the target and its dependencies. If any dependency has been updated since the target, then the actions are re-run to update the target. Using this approach, Make knows to only rebuild the files that, either directly or indirectly, depend on the file that changed. This is called an incremental build.
up to date means that the Makefile has a rule for the file and the file is up to date whereas Nothing to be done means that the file exists but the Makefile has no rule for it.
In [61]:
!make $code/wordcount.py
By explicitly recording the inputs to and outputs from steps in our analysis and the dependencies between files, Makefiles act as a type of documentation, reducing the number of things we have to remember.
Let’s add another rule to the end of Makefile:
In [113]:
makefile_contents_1 = """
{repo_path}/abyss.dat : {data}/books/abyss.txt
{tab}python {code}/wordcount.py {data}/books/abyss.txt {repo_path}/abyss.dat
""".format(**format_context)
makefile_contents = ''.join((makefile_contents_0, makefile_contents_1))
In [114]:
# append makefile_contents to Makefile
!printf "$makefile_contents" > Makefile
In [76]:
!make
Nothing happens because Make attempts to build the first target it finds in the Makefile, the default target, which is isles.dat which is already up-to-date. We need to explicitly tell Make we want to build abyss.dat:
In [79]:
!make $repo_path/abyss.dat
We may want to remove all our data files so we can explicitly recreate them all. We can introduce a new target, and associated rule, to do this. We will call it clean, as this is a common name for rules that delete auto-generated files, like our .dat files:
In [121]:
makefile_contents_2 = """
{repo_path}/clean:
{tab}rm -f {repo_path}/*.dat
""".format(**format_context)
makefile_contents = ''.join((makefile_contents_0, makefile_contents_1, makefile_contents_2))
In [122]:
# add makefile_contents to Makefile
!printf "$makefile_contents" > Makefile
This is an example of a rule that has no dependencies. clean has no dependencies on any .dat file as it makes no sense to create these just to remove them. We just want to remove the data files whether or not they exist. If we run Make and specify this target,
In [123]:
!make clean
All .dat files are removed!
There is no actual thing built called clean. Rather, it is a short-hand that we can use to execute a useful sequence of actions. Such targets, though very useful, can lead to problems.
For example, let us recreate our data files, create a directory called clean, then run Make:
In [124]:
!make "$repo_path"/isles.dat "$repo_path"/abyss.dat
!mkdir "$repo_path"/clean
In [125]:
!make "$repo_path"/clean
Make finds a file (or directory) called clean and, as its clean rule has no dependencies, assumes that clean has been built and is up-to-date and so does not execute the rule’s actions. As we are using clean as a short-hand, we need to tell Make to always execute this rule if we run make clean, by telling Make that this is a phony target, that it does not build anything. This we do by marking the target as .PHONY:
In [142]:
makefile_contents_phony_clean = """
.PHONY : clean
clean:
{tab}rm -f {repo_path}/*.dat
""".format(**format_context)
makefile_contents = ''.join((makefile_contents_0, makefile_contents_1, makefile_contents_phony_clean))
In [143]:
# add makefile_contents to Makefile
!printf "$makefile_contents" > Makefile
Now get expected result.
In [145]:
!make clean
We can add a similar command to create all the data files. We can put this at the top of our Makefile so that it is the default target, which is executed by default if no target is given to the make command:
In [146]:
makefile_contents_create_all_data = """
.PHONY : dats
dats : {repo_path}/isles.dat {repo_path}/abyss.dat
""".format(**format_context)
makefile_contents = ''.join((
makefile_contents_0,
makefile_contents_1,
makefile_contents_phony_clean,
makefile_contents_create_all_data,
))
In [147]:
# add makefile_contents to Makefile
!printf "$makefile_contents" > Makefile
In [148]:
!make dats
In [149]:
!make dats
The following figure shows a graph of the dependencies embodied within our Makefile, involved in building the dats target
In [151]:
display(Image('http://swcarpentry.github.io/make-novice/fig/02-makefile.png', unconfined=True))