See Docs Run. Run, Docs, Run!

Catherine Devlin, PyTN 2014

What's a doc?

  • How you use my package

  • How you write a program

  • How you do some math, science, ...

  • How our business operates

Docs are a problem

You forget

  • While you write
  • As you update code

Are you writing this?

Boring

To write

To read

Solution

More code

  • forgetfulness
  • boredom

Plan

argumentclinic

  • Usage samples
  • Software requirements

Technologies

  • doctests
  • Sphinx
  • IPython Notebook
  • Dexy

Doctests


In [1]:
def argue(statement):
    """Provides an argument.
    
    >>> argue("It probably is.")
    "No it isn't."
    >>> argue("Fine, it isn't.")
    "Yes it is."
    """   
    return "No it isn't."

In [2]:
import doctest

if __name__ == '__main__':
    doctest.testmod()


**********************************************************************
File "__main__", line 6, in __main__.argue
Failed example:
    argue("Fine, it isn't.")
Expected:
    "Yes it is."
Got:
    "No it isn't."
**********************************************************************
1 items had failures:
   1 of   2 in __main__.argue
***Test Failed*** 1 failures.

New name: Executable docstrings

  • Accurate
  • Relevant
  • Readable
  • Fun

orthogonal to your unit testing

Lightweight markup

Plaintext
---------

- quick
- readable
- any editor
- *code*-friendly
- **VC**-friendly

Plaintext

  • quick
  • readable
  • any editor
  • code-friendly
  • VC-friendly

reStructuredText

  • Used by Sphinx
  • More sophisticated

Markdown

  • Used by IPython Notebook
  • Superset of HTML

Sphinx

Converts reStructuredText documents to attractive, interlinked, indexed, searchable tree of documents

While running sphinx-quickstart, answer

> autodoc: automatically insert docstrings from modules (y/N) [n]: y

In [3]:
cd ~/proj/argument-clinic/argumentclinic/docs/sphinx


/home/catherine/proj/argument-clinic/argumentclinic/docs/sphinx

In [4]:
ls


autodoc.rst   get_reqs.py   Makefile                  _static/
_build/       get_reqs.pyc  __pycache__/              _templates/
conf.fail.py  index.rst     requirements_autorun.rst  usage_autorun.rst
conf.py       make.bat      requirements_progout.rst  usage_doctest.rst

In [5]:
!make


Please use `make <target>' where <target> is one of
  html       to make standalone HTML files
  dirhtml    to make HTML files named index.html in directories
  singlehtml to make a single large HTML file
  pickle     to make pickle files
  json       to make JSON files
  htmlhelp   to make HTML files and a HTML help project
  qthelp     to make HTML files and a qthelp project
  devhelp    to make HTML files and a Devhelp project
  epub       to make an epub
  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter
  latexpdf   to make LaTeX files and run them through pdflatex
  latexpdfja to make LaTeX files and run them through platex/dvipdfmx
  text       to make text files
  man        to make manual pages
  texinfo    to make Texinfo files
  info       to make Texinfo files and run them through makeinfo
  gettext    to make PO message catalogs
  changes    to make an overview of all changed/added/deprecated items
  xml        to make Docutils-native XML files
  pseudoxml  to make pseudoxml-XML files for display purposes
  linkcheck  to check all external links for integrity
  doctest    to run all doctests embedded in the documentation (if enabled)

In [6]:
from argumentclinic.argumentclinic import argue
print(argue.__doc__)


---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-6-b03acca82d0c> in <module>()
----> 1 from argumentclinic.argumentclinic import argue
      2 print(argue.__doc__)

ImportError: No module named 'argumentclinic'

In [ ]:
cat autodoc.rst

In [ ]:
!make html

sphinx.ext.doctest

make doctest html complements sphinx.ext.autodoc

While running sphinx-quickstart, answer

> doctest: automatically test code snippets in doctest blocks (y/N) [n]: y

Then, build docs with

make doctest html

sphinxcontrib.autorun

Install normally

  hg clone http://bitbucket.org/birkenfeld/sphinx-contrib/
  cd sphinx-contrib/autorun
  python setup.py install

Enable within each Sphinx project

  echo "extensions.append('sphinxcontrib.autorun')" >> conf.py

Python 2 only (BOOOOO!)


In [ ]:
cd ~/proj/argument-clinic/argumentclinic/docs/sphinx/

In [ ]:
cat usage.rst

In [ ]:
cat get_reqs.py

In [ ]:
cat requirements_autorun.rst

sphinxcontrib.programoutput

Install

pip install sphinxcontrib-programoutput


  • Enable within each Sphinx project

    echo "extensions.append('sphinxcontrib.programoutput')" >> conf.py
  • Python 2 only (BOOOOOO!)


In [7]:
cat requirements_progout.rst


Software Requirements (with ``programoutput``)
==============================================

Python packages
---------------

.. program-output:: python get_reqs.py

PostgreSQL
----------

PostgreSQL 9.1 or better required.  Your system has:

.. command-output:: psql --version

Sphinx Advantages

  • Complex multi-document structure
  • Doc hosting at Read the DSphixocs
  • Many extensions and options
  • Generate variety of formats (PDF, ePub, LaTeX, ...)
  • Python standard

IPython Notebook

An interactive browser-based Python environment

Installation instructions

apt-get build-dep ipython-notebook
pip install ipython jinja2 pyzmq tornado

Start notebook server

ipython notebook   

Serving

  • Your own server
  • nbviewer + URL or gist
  • nbconvert to HTML or LaTeX
  • Live hosting at wakari.io

IPython Notebook Advantages

  • Ease of code / doc integration
  • User can experiment
  • Static, downloadable hosting at nbviewer
  • Live hosting at wakari.io

dexy

Framework for applying transformations to documents

Installing

pip install dexy

Steps

  • Source .rst or .md
  • dexy.txt or .yml recipe
  • dexy setup
  • dexy

In [8]:
cd ~/proj/argument-clinic/argumentclinic/docs/dexy/


/home/catherine/proj/argument-clinic/argumentclinic/docs/dexy

In [9]:
ls


arg_example.py  dexy.yaml  get_reqs.py  index.rst  output/  psql_version.bash

In [10]:
cat index.rst


argument-clinic
===============

`argument-clinic` supplies arguments as desired.

Usage
-----

::

    {{ d['arg_example.py'] | indent(4) }}

Result::

    {{ d['arg_example.py|py'] | indent(4) }}

Software requirements
---------------------

::

    {{ d['get_reqs.py|py'] | indent(4) }}

**Also requires** PostgreSQL_ v. 9.1 or above.  
You have: {{ d['psql_version.bash|bash'] }}

.. _PostgreSQL: http://postgresql.org


In [11]:
cat dexy.yaml


.rst|jinja|rstbody|easyhtml:
    - .py|py
    - .bash|bash
        

In [80]:
!dexy filters


Installed filters:
  abc : Runs `abcm2ps` on .abc music files.
  abcm : Runs `abcm2ps` on .abc music files, generating all output formats.
  apis : Base class for filters which post content to a remote API.
  applytemplate : Apply template to file. Template should specify %(content)s.
  archive : Creates a .tgz archive of all input documents.
  asciidoc : Runs asciidoc command.
  asciidoctor : Runs `asciidoctor`.
  asciisyn : Surrounds code with highlighting instructions for Asciidoctor
  bash : Runs bash scripts using 'bash' and returns stdout.
  bashint : Runs bash. use to run bash scripts.
  bn : Forces previous filter to output .bmp extension.
  botoup : Uses boto library to upload content to S3, returns the URL.
  bw : Converts color pdf to black and white.
  bwconv : Converts color pdf to black and white.
  c : Compile code using gcc and run.
  calibre : Runs `ebook-convert` command (part of calibre)
  casperjs : Runs scripts using casper js. Saves cookies.
  cb : Changes file extension to .sh
  cfussy : Compile code using gcc and run, raising an error if compiled code returns nonzero exit.
  ch : Changes file extension to .html
  chext : Dummy filter for allowing changing a file extension.
  cinput : Compile code using gcc and run with input.
  cj : Changes file extension to .json
  clang : Compile code using clang and run.
  clanginput : compile code using clang and run with input.
  clj : Runs clojure.
  cowsay : Runs input through 'cowsay'.
  cowthink : Runs input through 'cowthink'.
  cpickle : Forces previous filter to output .cpickle extension.
  cpp : Compile c++ code using cpp and run.
  cppinput : Compile c++ code using cpp and run with input.
  ct : Changes file extension to .txt
  customize : Add <script> tags or <link> tags to an HTML file's header.
  dexy : Filter which implements some default behaviors.
  dict : Returns an ordered dict with a single element.
  ditaa : Runs ditaa to generate images from ascii art.
  dot : Renders .dot files to either PNG or PDF images.
  dvilatex : Run Latex outputting a .dvi file.
  easyhtml : Wraps your text in HTML header/footer which includes Baseline CSS resets.
  easylatex : Wraps your text in LaTeX article header/footer.
  ebook : Runs `ebook-convert` command (part of calibre)
  embedfonts : Runs ghostscript ps2pdf with prepress settings.
  eps2pdf : Uses epstopdf to convert .eps files to .pdf
  epstopdf : Uses epstopdf to convert .eps files to .pdf
  escript : Runs Erlang scripts using the escript command.
  espeak : Runs espeak text to speech.
  f95 : Compiles and executes fortran code.
  figlet : Runs input through 'figlet'.
  filterargs : Prints out the args it receives.
  fn : Deprecated. No longer needed.
  fopdf : Uses asciidoctor-fopdf to generate PDF.
  forcebmp : Forces previous filter to output .bmp extension.
  forcegif : Forces previous filter to output .gif extension.
  forcehtml : Forces previous filter to output .html extension.
  forcejpg : Forces previous filter to output .jpg extension.
  forcejson : Forces previous filter to output .json extension.
  forcelatex : Forces previous filter to output .tex extension.
  forcepdf : Forces previous filter to output .pdf extension.
  forcepng : Forces previous filter to output .png extension.
  forcer : Forces previous filter to output .R extension.
  forcesvg : Forces previous filter to output .svg extension.
  forcetext : Forces previous filter to output .txt extension.
  forcexml : Forces previous filter to output .xml extension.
  fortran : Compiles and executes fortran code.
  ft : Apply another file to bottom of file.
  gcc : Compile code using gcc and run.
  gn : Forces previous filter to output .gif extension.
  go : Runs 'go run' command on an input .go file. http://golang.org/
  gotest : Runs 'go test' command on an input .go file. http://golang.org/
  graphviz : Renders .dot files to either PNG or PDF images.
  h : Forces previous filter to output .html extension.
  hd : Apply another file to top of file.
  head : Returns just the first 10 lines of input.
  htlatex : Generates HTML from Latex source using htlatex
  html2pdf : Deprecated, use casper.js instead.
  htmlsections : Splits files into sections based on comments like ### "foo"
  htmltidy : Uses tidy to clean and validate HTML.
  ipynb : Get data out of an IPython notebook.
  ipynbcasper : Launch IPython notebook and run a casperjs script against the server.
  ipynbx : Generates a static file based on an IPython notebook.
  ipython : Runs python code in the IPython console.
  irb : Runs ruby code in irb.
  irbout : Runs ruby scripts in irb.
  j : Forces previous filter to output .json extension.
  java : Compiles java code and runs main method.
  javac : Compiles java code and returns the .class object
  jinja : Runs the Jinja templating engine.
  jirb : Run jruby code in jirb.
  jlcon : Runs julia (.jl) files in the repl.
  jn : Forces previous filter to output .jpg extension.
  join : Takes sectioned code and joins it into a single section. Some filters which
  jruby : Run jruby code and return stdout.
  js : Runs code through rhino js interpreter.
  jsint : Runs rhino JavaScript interpeter.
  julia : Runs julia (.jl) files.
  jython : jython
  jythoni : jython in REPL
  keyvalueexample : Example of storing key value data.
  kshint : Runs ksh. Use to run bash scripts.
  kv : Creates a new key-value store.
  l : Forces previous filter to output .tex extension.
  latexdvi : Run Latex outputting a .dvi file.
  latextile : Converts textile to LaTeX using Redcloth.
  lines : Returns each line in its own section.
  lua : Runs code through lua interpreter.
  lynxdump : Converts HTML to plain text by using lynx -dump.
  lyx : Runs lyx to generate LaTeX output.
  lyxjinja : Converts dexy:foo.txt|bar into << d['foo.txt|bar'] >>
  make : Runs make tasks.
  man : Read command names from a file and fetch man pages for each.
  markdown : Runs a Markdown processor to convert markdown to HTML.
  matlabint : Runs matlab in REPL.
  newdoc : A filter which adds an extra document to the tree.
  node : Runs scripts using node js
  nodejs : Runs scripts using node js
  org : Convert .org files to other formats.
  others : Example of accessing other documents.
  outputabc : Only outputs extension .abc
  p : Forces previous filter to output .pdf extension.
  pandoc : Convert documents to various available output formats using pandoc.
  pdf2cairo : Runs `pdftocairo` from the poppler library.
  pdf2img : Runs ghostscript to convert PDF files to images.
  pdf2jpg : Converts a PDF file to a jpg image using ghostscript.
  pdf2text : Uses pdftotext from the poppler library to convert PDFs to text.
  pdfcrop : Runs the PDFcrop script http://pdfcrop.sourceforge.net/
  pdfinfo : Uses the pdfinfo script to retrieve metadata about a PDF.
  pdftotext : Uses pdftotext from the poppler library to convert PDFs to text.
  phantomjs : Runs scripts using phantom js.
  php : Runs php file.
  phpint : Runs PHP in interpeter mode.
  phrender : Renders HTML to PNG/PDF using phantom.js.
  pickle : Forces previous filter to output .pickle extension.
  pn : Forces previous filter to output .png extension.
  ppjson : Pretty prints JSON input.
  process : Calls `set_data` method to store output.
  processmanual : Writes output directly to output file.
  processtext : Uses process_text method
  processwithdict : Stores sectional data using `process` method.
  ps2pdf : Converts a postscript file to PDF format.
  pstopdf : Converts a postscript file to PDF format.
  py : Runs Python code and returns stdout.
  pycon : Runs python code in python's REPL.
  pydoc : Returns introspected python data in key-value storage format.
  pyg : Apply Pygments <http://pygments.org/> syntax highlighting.
  pyg4rst : Surrounds code with highlighting instructions for ReST
  pyin : Runs python code and passes input
  pyout : Runs Python code and returns stdout.
  pytest : Runs the tests in the specified Python modules.
  r : Runs R in REPL.
  ragel : Generates ruby source code from a ragel file.
  rageldot : Generates state chart in .dot format of ragel state machine.
  ragelruby : Generates ruby source code from a ragel file.
  ragelrubydot : Generates state chart in .dot format of ragel state machine for ruby.
  rb : Runs ruby scripts and return stdout.
  rbrepl : Runs ruby code in irb.
  rd2pdf : Generates a pdf from R documentation file.
  Rd2pdf : Generates a pdf from R documentation file.
  rdconv : Convert R documentation to other formats.
  redcloth : Converts textile to HTML using Redcloth.
  redclothl : Converts textile to LaTeX using Redcloth.
  regetron : Filter which loads .regex file into regetron and runs any input text against it.
  resub : Runs re.sub on each line of input.
  rhino : Runs code through rhino js interpreter.
  rhinoint : Runs rhino JavaScript interpeter.
  rint : Runs R in REPL.
  rintbatch : Runs R files in batch mode, returning an R console transcript.
  rintmock : Experimental filter to run R in sections without using pexpect.
  rlrb : Generates ruby source code from a ragel file.
  rlrbd : Generates state chart in .dot format of ragel state machine for ruby.
  rout : Runs R files in batch mode, returning just the output.
  routbatch : Runs R files in batch mode, returning just the output.
  rst : A 'native' ReST filter which uses the docutils library.
  rst2beamer : Runs rst2beamer command (docutils).
  rst2html : Convert rst to HTML
  rst2latex : Runs rst2latex command (docutils).
  rst2man : Runs rst2man command (docutils).
  rst2odt : Runs rst2odt command (docutils).
  rst2xml : Runs rst2xml command (docutils).
  rstbody : Returns just the body part of an ReST document.
  rstdocparts : Returns key-value storage of document parts.
  rstmeta : Extracts bibliographical metadata and makes this available to dexy.
  rust : Runs rust code.
  rustc : Runs rust code.
  rusti : Runs rust code in the rust repl (rusti). EXPERIMENTAL.
  scala : Compiles and runs .scala files.
  scalac : Compiles .scala code to .class files.
  scalai : Runs scala code in the REPL.
  sectionsbywhitespace : Del
  sed : Runs a sed script.
  sh : Runs bash scripts using 'sh' and returns stdout.
  shint : Runs bash. use to run bash scripts.
  slides : Converts paragraphs to HTML and wrap each slide in a header and footer.
  sloc : Runs code through sloccount.
  sloccount : Runs code through sloccount.
  soups : Split a HTML file into nested sections based on header tags.
  split : Generate index page linking to multiple pages from single source.
  ss : Add a blank space to the start of each line.
  stata : Runs stata files.
  statai : Runs stata files.
  strings : Clean non-printing characters from text using the 'strings' tool.
  svg : Forces previous filter to output .svg extension.
  svg2pdf : Converts an SVG file to PDF by running it through casper js.
  t : Forces previous filter to output .txt extension.
  tags : Wrap text in specified HTML tags.
  taverna : Runs workflows in Taverna via command line tool.
  template : Base class for templating system filters such as JinjaFilter. Templating
  textile : Converts textile to HTML using Redcloth.
  tgzdir : Create a .tgz archive containing the unprocessed files in a directory.
  tidy : Uses tidy to clean and validate HTML.
  tidycheck : Runs `tidy` to check for valid HTML.
  tidyerrors : Uses tidy to print HTML errors.
  tikz : Renders Tikz code to PDF.
  used : Runs `sed` on the input file.
  wc : Runs input through wc command line tool.
  wiki2beamer : Converts wiki content to beamer.
  wkhtmltopdf : Deprecated, use casper.js instead.
  wordpress : Posts to a WordPress blog.
  wrap : Wraps text after 79 characters (tries to preserve existing line breaks and
  x : Forces previous filter to output .xml extension.
  xelatex : Runs .tex files using xelatex.
  xetex : Runs .tex files using xelatex.
  xmlsec : Stores all elements in the input XML document which have any of the
  yamlargs : Specify attributes in YAML at top of file.
  zip : Creates a .zip archive of all input documents.

For more information about a particular filter,
use the -alias flag and specify the filter alias.

Dexy Advantages

  • Filters for many transformations
  • MD or reST
  • Most flexible

Possibilities

  • tox results
  • Living lecture notes
  • Tutorials that assist with configuration
  • Information-rich dashboards
  • Living organizational documentation

catherine.devlin@gmail.com

github.com/catherinedevlin/argument-clinic/

Notebook form