Learning how to make a Makefile

Adapted from swcarpentry/make-novice repository.

Make’s fundamental concepts are common across build tools.

GNU Make is a free, fast, well-documented, and very popular Make implementation. From now on, we will focus on it, and when we say Make, we mean GNU Make.

A tutorial named Makefiles—part 2 of the tutorial.

Cells that follow are the result of following this Makefiles tutorial.

Other blog posts

NB: I have adapted the tutorial so that the steps take place in this Jupyter notebook so that the notebook can be transpiled into a Pelican blog post using a danielfrg/pelican-ipynb Pelican plugin. Some of the code is what is necessary to display output in the notebook and therefore the blog post.

Some Jupyter notebook housekeeping to set up some variables with path references.

In [1]:
import os

In [150]:
from IPython.core.display import Image, display

In [2]:
) = (

In [3]:
home = os.path.expanduser('~')

repo_path is the path to a clone of swcarpentry/make-novice

In [4]:
repo_path = os.path.join(

In [5]:
assert os.path.exists(repo_path)

paths are the paths to child directories in a clone of swcarpentry/make-novice

In [6]:
paths = (
paths = (
) = [os.path.join(repo_path, path) for path in paths]
assert all(os.path.exists(path) for path in paths)

In [20]:
format_context = dict(zip(
    ('repo_path', 'data', 'code', 'tab',), 
    (repo_path, data, code, TAB_CHAR))

Begin tutorial.

Create a file, called Makefile, with the following content:

In [107]:
makefile_contents_0 = """# Count words.
{repo_path}/isles.dat : {data}/books/isles.txt
{tab}python {code}/wordcount.py {data}/books/isles.txt {repo_path}/isles.dat

Using the shell to create Makefile with contents the value of Python variable makefile_contents.

In [108]:
!printf "$makefile_contents_0" > Makefile

This is a build file, which for Make is called a Makefile - a file executed by Make. Note how it resembles one of the lines from our shell script.

# Count words.
/home/dmmmd/Dropbox/spikes/make-novice/isles.dat : /home/dmmmd/Dropbox/spikes/make-novice/data/books/isles.txt
    python /home/dmmmd/Dropbox/spikes/make-novice/code/wordcount.py /home/dmmmd/Dropbox/spikes/make-novice/data/books/isles.txt /home/dmmmd/Dropbox/spikes/make-novice/isles.dat

Let us go through each line in turn:

  • # denotes a comment. Any text from # to the end of the line is ignored by Make.
  • isles.dat is a target, a file to be created, or built.
  • books/isles.txt is a dependency, a file that is needed to build or update the target. Targets can have zero or more dependencies.
  • A colon, :, separates targets from dependencies.
  • python wordcount.py books/isles.txt isles.dat is an action, a command to run to build or update the target using the dependencies. Targets can have zero or more actions. These actions form a recipe to build the target from its dependencies and can be considered to be a shell script.
  • Actions are indented using a single TAB character, not 8 spaces. This is a legacy of Make’s 1970’s origins. If the difference between spaces and a TAB character isn’t obvious in your editor, try moving your cursor from one side of the TAB to the other. It should jump four or more spaces.
  • Together, the target, dependencies, and actions form a a rule.

Let’s first sure we start from scratch and delete the .dat and .png files we created earlier

In [38]:
!rm $repo_path/*.dat $repo_path/*.png

rm: cannot remove ‘/home/dmmmd/Dropbox/spikes/make-novice/*.png’: No such file or directory

By default, Make looks for a Makefile, called Makefile, and we can run Make as follows

In [39]:
# By default, Make prints out the actions it executes:

python /home/dmmmd/Dropbox/spikes/make-novice/code/wordcount.py /home/dmmmd/Dropbox/spikes/make-novice/data/books/isles.txt /home/dmmmd/Dropbox/spikes/make-novice/isles.dat

Let’s see if we got what we expected

In [40]:
!head -5 $repo_path/isles.dat

the 3822 6.737176097303014
of 2460 4.336329984135378
and 1723 3.0371937246606735
to 1479 2.607086197778953
a 1308 2.305658381808567

We don’t have to call our Makefile Makefile. However, if we call it something else we need to tell Make where to find it. This we can do using -f flag. For example, if our Makefile is named MyOtherMakefile:

In [109]:
!printf "$makefile_contents_0" > MyOtherMakeFile.mk

In [110]:
!make -f MyOtherMakeFile.mk

python /home/dmmmd/Dropbox/spikes/make-novice/code/wordcount.py /home/dmmmd/Dropbox/spikes/make-novice/data/books/isles.txt /home/dmmmd/Dropbox/spikes/make-novice/isles.dat

This is because our target, isles.dat, has now been created, and Make will not create it again. To see how this works, let’s pretend to update one of the text files. Rather than opening the file in an editor, we can use the shell touch command to update its timestamp (which would happen if we did edit the file)

In [57]:
!touch $data/books/isles.txt

If we compare the timestamps of books/isles.txt and isles.dat,

In [58]:
!ls -l $data/books/isles.txt $repo_path/isles.dat
# then we see that isles.dat, the target, is now older thanbooks/isles.txt, its dependency

-rw-rw-r-- 1 dmmmd dmmmd 323972 Sep  3 10:54 /home/dmmmd/Dropbox/spikes/make-novice/data/books/isles.txt
-rw-rw-r-- 1 dmmmd dmmmd 206660 Sep  3 10:53 /home/dmmmd/Dropbox/spikes/make-novice/isles.dat

If we run Make again,

In [59]:
#then it recreates isles.dat

python /home/dmmmd/Dropbox/spikes/make-novice/code/wordcount.py /home/dmmmd/Dropbox/spikes/make-novice/data/books/isles.txt /home/dmmmd/Dropbox/spikes/make-novice/isles.dat

When it is asked to build a target, Make checks the ‘last modification time’ of both the target and its dependencies. If any dependency has been updated since the target, then the actions are re-run to update the target. Using this approach, Make knows to only rebuild the files that, either directly or indirectly, depend on the file that changed. This is called an incremental build.

up to date means that the Makefile has a rule for the file and the file is up to date whereas Nothing to be done means that the file exists but the Makefile has no rule for it.

In [61]:
!make $code/wordcount.py

make: Nothing to be done for `/home/dmmmd/Dropbox/spikes/make-novice/code/wordcount.py'.

By explicitly recording the inputs to and outputs from steps in our analysis and the dependencies between files, Makefiles act as a type of documentation, reducing the number of things we have to remember.

Let’s add another rule to the end of Makefile:

In [113]:
makefile_contents_1 = """
{repo_path}/abyss.dat : {data}/books/abyss.txt
{tab}python {code}/wordcount.py {data}/books/abyss.txt {repo_path}/abyss.dat
makefile_contents = ''.join((makefile_contents_0, makefile_contents_1))

In [114]:
# append makefile_contents to Makefile
!printf "$makefile_contents" > Makefile

In [76]:

make: `/home/dmmmd/Dropbox/spikes/make-novice/isles.dat' is up to date.

Nothing happens because Make attempts to build the first target it finds in the Makefile, the default target, which is isles.dat which is already up-to-date. We need to explicitly tell Make we want to build abyss.dat:

In [79]:
!make $repo_path/abyss.dat

python /home/dmmmd/Dropbox/spikes/make-novice/code/wordcount.py /home/dmmmd/Dropbox/spikes/make-novice/data/books/abyss.txt /home/dmmmd/Dropbox/spikes/make-novice/abyss.dat

We may want to remove all our data files so we can explicitly recreate them all. We can introduce a new target, and associated rule, to do this. We will call it clean, as this is a common name for rules that delete auto-generated files, like our .dat files:

In [121]:
makefile_contents_2 = """
{tab}rm -f {repo_path}/*.dat
makefile_contents = ''.join((makefile_contents_0, makefile_contents_1, makefile_contents_2))

In [122]:
# add makefile_contents to Makefile
!printf "$makefile_contents" > Makefile

This is an example of a rule that has no dependencies. clean has no dependencies on any .dat file as it makes no sense to create these just to remove them. We just want to remove the data files whether or not they exist. If we run Make and specify this target,

In [123]:
!make clean

make: *** No rule to make target `clean'.  Stop.

All .dat files are removed!

There is no actual thing built called clean. Rather, it is a short-hand that we can use to execute a useful sequence of actions. Such targets, though very useful, can lead to problems.

For example, let us recreate our data files, create a directory called clean, then run Make:

In [124]:
!make "$repo_path"/isles.dat "$repo_path"/abyss.dat
!mkdir "$repo_path"/clean

make: `/home/dmmmd/Dropbox/spikes/make-novice/isles.dat' is up to date.
make: `/home/dmmmd/Dropbox/spikes/make-novice/abyss.dat' is up to date.
mkdir: cannot create directory ‘/home/dmmmd/Dropbox/spikes/make-novice/clean’: File exists

In [125]:
!make "$repo_path"/clean

make: `/home/dmmmd/Dropbox/spikes/make-novice/clean' is up to date.

Make finds a file (or directory) called clean and, as its clean rule has no dependencies, assumes that clean has been built and is up-to-date and so does not execute the rule’s actions. As we are using clean as a short-hand, we need to tell Make to always execute this rule if we run make clean, by telling Make that this is a phony target, that it does not build anything. This we do by marking the target as .PHONY:

In [142]:
makefile_contents_phony_clean = """
.PHONY : clean
{tab}rm -f {repo_path}/*.dat
makefile_contents = ''.join((makefile_contents_0, makefile_contents_1, makefile_contents_phony_clean))

In [143]:
# add makefile_contents to Makefile
!printf "$makefile_contents" > Makefile

Now get expected result.

In [145]:
!make clean

rm -f /home/dmmmd/Dropbox/spikes/make-novice/*.dat

We can add a similar command to create all the data files. We can put this at the top of our Makefile so that it is the default target, which is executed by default if no target is given to the make command:

In [146]:
makefile_contents_create_all_data = """
.PHONY : dats
dats : {repo_path}/isles.dat {repo_path}/abyss.dat
makefile_contents = ''.join((

In [147]:
# add makefile_contents to Makefile
!printf "$makefile_contents" > Makefile

In [148]:
!make dats

python /home/dmmmd/Dropbox/spikes/make-novice/code/wordcount.py /home/dmmmd/Dropbox/spikes/make-novice/data/books/isles.txt /home/dmmmd/Dropbox/spikes/make-novice/isles.dat
python /home/dmmmd/Dropbox/spikes/make-novice/code/wordcount.py /home/dmmmd/Dropbox/spikes/make-novice/data/books/abyss.txt /home/dmmmd/Dropbox/spikes/make-novice/abyss.dat

In [149]:
!make dats

make: Nothing to be done for `dats'.

The following figure shows a graph of the dependencies embodied within our Makefile, involved in building the dats target

In [151]:
display(Image('http://swcarpentry.github.io/make-novice/fig/02-makefile.png', unconfined=True))