DESI spectral extraction code benchmarks

Stephen Bailey
Lawrence Berkeley National Lab
Spring 2017

Update 2017-03-12

Intel engineers identified the cause of the previous lack of scaling with number of processes (an OpenMP bug on KNL). The work-around is:

• for multiprocessing, use $KMP_AFFINITY=disabled
• for MPI, use srun --cpu_bind=cores ...
• On Haswell, neither of these is necessary to achieve reasonable scaling

Introduction

These benchmarks test the DESI spectral extraction code, which is the most computationally intensive portion of our data processing pipeline. It performs a forward modeling analysis of astronomical spectra projected onto 2D CCD images. Early non-KNL benchmarking indicated that it was spending approximately 1/3 of its time in each of

• MKL via scipy.linalg.eigh
• Miscellaneous non-MKL compiled code in numpy and scipy
• Miscellaneous pure python

We have not recently re-benchmarked this code and these benchmarks use a smaller example problem that may have a different ratio.

See the end of this notebook for how to get the code and run the benchmarks.

Scaling with \$OMP_NUM_THREADS

Initial test: check if $OMP_NUM_THREADS matters for this code. Takeaway points:

• $OMP_NUM_THREADS doesn't matter for this code, presumably because it is spending most of its time in non-MKL / non-OpenMP code.
• single process Haswell ~6x faster than single-process KNL

example srun commands:

#- Haswell
srun -n 1 -c 64 --cpu_bind=cores python extract.py 1 4 16 32 64

#- KNL
srun -n 1 -c 256 --cpu_bind=cores python extract.py 1 4 16 64 256

On Haswell, no significant effect from setting or not

• OMP_PROC_BIND=spread
• OMP_PLACES=cores
• KMP_AFFINITY=disabled

The remaining tests use $OMP_NUM_THREADS=1

Scaling tests

Test the single node scaling performance for Haswell and KNL with two methods:

• python multiprocessing
• mpi4py

This is a data-parallel problem where different processes/ranks are working on different pieces of data without needing to communicate with each other. Under ideal scaling each iteration will take the same amount of well time such that more processes = more data processed = higher total rate. Takeaway points:

• Scaling is quite good up to the number of physical cores, with slight degredation after that on Haswell and slight gains on KNL.
• A Haswell nodes is 2-3x faster than a KNL node
• Not shown here:
  • On Haswell, not setting $OMP_NUM_THREADS=1 clobbers performance at larger concurrency, even though the previous test indicated that $OMP_NUM_THREADS > 1 doesn't help for a single process.

Startup time

Python startup time was significant for N>>1 MPI processes, especially on KNL. All code was installed to $SCRATCH or /global/common (fast metadata disks) and $PYTHONPATH did not have anything in /home or /project (slow metadata disks). "Wakeup" timed the following imports:

from mpi4py import MPI
comm = MPI.COMM_WORLD

import sys, os
import platform
import optparse
import multiprocessing
import numpy as np
from specter.extract import ex2d
import specter.psf
import knltest

Setup to reproduce these results

Create a basic python environment

Start with an IDP or Anaconda distribution with numpy, scipy, astropy (and possibly others...), e.g.

conda create -n knltest python=3.5 numpy scipy astropy
source activate knltest

Get the benchmark code

git clone https://github.com/sbailey/knltest
git clone https://github.com/desihub/specter

#- reference versions used for these tests
cd knltest; git checkout 065f961; cd ..
cd specter; git checkout 391ac23; cd ..

export PYTHONPATH=`pwd`/specter/py:$PYTHONPATH
cd knltest/code
export OMP_NUM_THREADS=1
export KMP_AFFINITY=disabled

Note: you can also run python setup.py install in the specter/ directory to do a normal python package installation. The above $PYTHONPATH trick is the minimally invasive option.

Run multiprocessing benchmark locally

This will run the extraction code with 1, 4, 16... parallel processes, where each process is given the same amount of work to do.

python extract-mp.py 1 4 16 32 64 --bundlesize 5 --numwave 50
python extract-mp.py 1 4 16 64 128 256 --bundlesize 5 --numwave 50
python extract-mp.py 4 16 64 128 256 --bundlesize 25 --numwave 50

There is also an mpi4py version of this test which uses MPI ranks instead of multiprocessing processes. In that case it will only use as many ranks as are specified in the options to extract-mpi.py.

MPI: run with SLURM to a compute node

Allocate just the number of MPI processes needed for the test

#- KNL
for n in 1 4 16 64 128 256; do
    let c=256/$n
    srun -n $n -c $c --cpu_bind=cores python extract-mpi.py $n --bundlesize 5 --numwave 50
done

#- Haswell
for n in 1 4 16 32 64; do
    let c=64/$n
    srun -n $n -c $c --cpu_bind=cores python extract-mpi.py $n --bundlesize 5 --numwave 50
done

#- For comparison: allocate 16 ranks but use fewer of them, leaving others idle
#- This hurts MPI scaling performance and is *not* what was shown above
srun -n 16 -c 1 --cpu_bind=cores python extract-mpi.py 1 4 16 --bundlesize 5 --numwave 50

Stephen's setup on Cori

module load python/3.5-anaconda
cd $SCRATCH/desi/knlcode/
export PYTHONPATH=`pwd`/specter/py:$PYTHONPATH
cd knltest/code