Introduction to reproducible research with Jupyter Notebook and Python

Learning objectives for this workshop:

1. Understand the concept, importance, and components of reproducible research.
2. Describe the architecture of a Jupyter notebook environment, including the notebook server, programming language kernels, interactive widgets, and situate Jupyter notebooks as part of the wider Jupyter ecosystem.
3. Use a Jupyter notebook to create, manage and report on the execution of a programmatic workflow using a structured document combining narrative text, executable code, executed code output and embedded media items.
4. Be able to access the wider Jupyter community, including but not limited to the Jupyter Google Group, Stack Overflow and communities around specific Python packages

• The sciences have with reproducibility problem. Nature article
• Many published studies cannot be reproduced:

Replication and reproduction:

• Peng (2009) Reproducible research and Biostatistics. Biostatistics 10: 405-408
  • Replication - when independent investigators use methods, protocols, data, and equipment to confirm scientific claims.
  • Reproduction - when data sets and computer code are made available for researchers to verify results.

Science retracts paper without agreement of lead author.

• Journal retracted a study of how canvassers can sway people's opinions about gay marriage.
• Editor-in-Chief: Original survey data:
  • not made available for independent reproduction of results
  • Survey incentives misrepresented
  • Sponsorship statement false
• Two grad students attempted to reproduce the study and could not.
• Concluded the data must have been fabricated. 538 story ## Methods we'll discuss today can't prevent fraud, but they can make it easier to discover such issues.

Retracted, but not fraud

• One researcher had seven papers retracted because of irreproducibility Retraction Watch
• Couldn't find the data Wiley
• "Extensive" errors force retraction of lymphoma paper JRO
• Many, many more Irreproducible examples

Seizure study retracted after authors realized data were "Terribly mixed"

• From the authors of Low Dose Lidocaine for Refractory Seizures in Preterm Neonates: "The article has been retracted at the request of the authors. After carefully re-examining the data presented in the article, they identified that data of two different hospitals got terribly mixed. The published results cannot be reproduced in accordance with scientific and clinical correctness." IJP

Bad spreadsheet merge kills depression paper, quick fix resurrects it

• The authors informed the journal that the merge of lab results and other survey data used in the paper resulted in an error regarding the identification codes.
• Original conclusion : Lower levels of CSF IL-6 were associated with current depression and with future depression.
• Revised conclusion: Higher levels of CSF IL-6 and IL-8 were associated with current depression.

LIGO (Laser Interferometer Gravitational-Wave Observatory)

• All data are publically available free of charge.
• Jupyter Notebooks running Python are produced for each publication. These notebooks allow full reproducibility: all analyses and figures can be recreated.
• Produce in-depth Tutorials using Jupyter Notebooks and Python
  • Signal processing tutorial

The Four Facets of reproducibility:

• Documentation: note the difference between binary files (e.g. docx) and .txt files and why text files are preferred for documentation.
• Organization: tools to organize your projects so that you don’t have a single folder with hundreds of files.
• Automation: the power of scripting to create automated data analyses.
• Dissemination: publishing is not the end of your analysis, rather it is a way station towards your future research and the future research of others.

Our reproducibility toolkit: Jupyter Notebook and the Python language

Why Python?

• Free!
• Open source
• Widely used and supported across many disciplines
• Can be used on Windows, Mac OS X, or Linux!