notebook.community

Edit and run

Protocol of experiments 08.05.2017

Goal: Provide proof of concept and working example for the idea of running ToolDog analyser in the Docker container

Requirements:

  1. Installed Docker
  2. Installed Python3, pip

First, we import a library for running bash commands from Python code.





In [1]:



    


import subprocess

Then we define two util functions.

First one if for running bash command.





In [2]:



    


def run_command(command):
    """Run bash command and return the result

    :param str command: String representation of bash command
    
    :return: Returns a generator of output of the result of running bash command in bytes
    :rtype: iter
    """
    command = command.split()
    p = subprocess.Popen(command,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT)
    return iter(p.stdout.readline, b'')

Second one is for writing string in the file.





In [87]:



    


def write_to_file(filename, string = ''):
    """Write string to file

    :param str filename: Filename
    :param str string: String to write
    """
    f = open(filename, 'w')
    f.write(string)
    f.close()

Than we define some constants for future use.





In [88]:



    


PYTHON_VERSION = 2 # version of python tool



    











In [89]:



    


TOOL_NAME = 'alignment' # name of the tool, assuming it is available via PyPi



    











In [90]:



    


OUTPUT_FORMAT = 'cwl' # output: cwl / xml



    











In [91]:



    


OUTPUT_FILENAME = TOOL_NAME + "." + OUTPUT_FORMAT # output filename



    











In [92]:



    


PYTHON_PATH = "/usr/local/lib/python3.5/dist-packages/" if PYTHON_VERSION == 3 else \
"/usr/local/lib/python2.7/dist-packages/" # PYTHONPATH, required for argparse2tool

Compose bash command for running detached docker container from tooldog-analyser-sandbox image, that will be destroyed after stopping.





In [93]:



    


cmd = 'docker run -d -e PYTHONPATH=' + PYTHON_PATH + ' --rm inkuzm/tooldog-analyser-sandbox tail -f /dev/null'

Save container ID for future use.





In [94]:



    


CONTAINER_ID = ""
for line in run_command(cmd):
    CONTAINER_ID = line.decode("utf-8")[:-1]
    print(CONTAINER_ID)



    


















    






bd47bbd3272a48739f8afadae8d24723a0508e330f41715cba60fe4dad953936

Yay! Now we can do stuff inside the container.

Let's define one more util function for running commands inside the created container.





In [95]:



    


def exec_command_in_container(command, cid):
    """Run bash command inside container cid

    :param str command: String representation of bash command
    
    :return: Returns a generator of output of the result of running bash command in bytes
    :rtype: iter
    """  
    cmd = 'docker exec ' + cid + ' ' + command
    return run_command(cmd)

We install argparse2tool:





In [96]:



    


for line in exec_command_in_container('pip3 install argparse2tool', CONTAINER_ID):
    print(line.decode("utf-8")[:-1])



    


















    






Collecting argparse2tool
  Downloading argparse2tool-0.4.3.post2.tar.gz
Collecting galaxyxml>=0.2.3 (from argparse2tool)
  Downloading galaxyxml-0.4.0.tar.gz
Collecting jinja2 (from argparse2tool)
  Downloading Jinja2-2.9.6-py2.py3-none-any.whl (340kB)
Collecting future (from argparse2tool)
  Downloading future-0.16.0.tar.gz (824kB)
Collecting lxml (from galaxyxml>=0.2.3->argparse2tool)
  Downloading lxml-3.7.3-cp35-cp35m-manylinux1_x86_64.whl (7.1MB)
Collecting MarkupSafe>=0.23 (from jinja2->argparse2tool)
  Downloading MarkupSafe-1.0.tar.gz
Building wheels for collected packages: argparse2tool, galaxyxml, future, MarkupSafe
  Running setup.py bdist_wheel for argparse2tool: started
  Running setup.py bdist_wheel for argparse2tool: finished with status 'error'
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-0do37j9w/argparse2tool/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmplqq66w3xpip-wheel- --python-tag cp35:
  /usr/lib/python3/dist-packages/setuptools/dist.py:285: UserWarning: Normalizing '0.4.3-2' to '0.4.3.post2'
    normalized_version,
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/argparse
  copying argparse/argparse_cwl_translation.py -> build/lib/argparse
  copying argparse/argparse_galaxy_translation.py -> build/lib/argparse
  copying argparse/__init__.py -> build/lib/argparse
  creating build/lib/argparse2tool
  copying argparse2tool/__init__.py -> build/lib/argparse2tool
  copying argparse2tool/check_path.py -> build/lib/argparse2tool
  creating build/lib/click
  copying click/click_cwl_translation.py -> build/lib/click
  copying click/__init__.py -> build/lib/click
  creating build/lib/argparse2tool/cmdline2gxml
  copying argparse2tool/cmdline2gxml/__init__.py -> build/lib/argparse2tool/cmdline2gxml
  creating build/lib/argparse2tool/cmdline2cwl
  copying argparse2tool/cmdline2cwl/cwl_tool.py -> build/lib/argparse2tool/cmdline2cwl
  copying argparse2tool/cmdline2cwl/__init__.py -> build/lib/argparse2tool/cmdline2cwl
  running egg_info
  writing entry points to argparse2tool.egg-info/entry_points.txt
  writing dependency_links to argparse2tool.egg-info/dependency_links.txt
  writing argparse2tool.egg-info/PKG-INFO
  writing requirements to argparse2tool.egg-info/requires.txt
  writing top-level names to argparse2tool.egg-info/top_level.txt
  warning: manifest_maker: standard file '-c' not found
  
  reading manifest file 'argparse2tool.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'argparse2tool.egg-info/SOURCES.txt'
  creating build/lib/argparse2tool/cmdline2cwl/templates
  copying argparse2tool/cmdline2cwl/templates/cwltool.j2 -> build/lib/argparse2tool/cmdline2cwl/templates
  copying argparse2tool/cmdline2cwl/templates/cwltool_inputs.j2 -> build/lib/argparse2tool/cmdline2cwl/templates
  copying argparse2tool/cmdline2cwl/templates/cwltool_outputs.j2 -> build/lib/argparse2tool/cmdline2cwl/templates
  installing to build/bdist.linux-x86_64/wheel
  running install
  running install_lib
  creating build/bdist.linux-x86_64
  creating build/bdist.linux-x86_64/wheel
  creating build/bdist.linux-x86_64/wheel/click
  copying build/lib/click/click_cwl_translation.py -> build/bdist.linux-x86_64/wheel/click
  copying build/lib/click/__init__.py -> build/bdist.linux-x86_64/wheel/click
  creating build/bdist.linux-x86_64/wheel/argparse
  copying build/lib/argparse/argparse_cwl_translation.py -> build/bdist.linux-x86_64/wheel/argparse
  copying build/lib/argparse/argparse_galaxy_translation.py -> build/bdist.linux-x86_64/wheel/argparse
  copying build/lib/argparse/__init__.py -> build/bdist.linux-x86_64/wheel/argparse
  creating build/bdist.linux-x86_64/wheel/argparse2tool
  creating build/bdist.linux-x86_64/wheel/argparse2tool/cmdline2cwl
  copying build/lib/argparse2tool/cmdline2cwl/cwl_tool.py -> build/bdist.linux-x86_64/wheel/argparse2tool/cmdline2cwl
  creating build/bdist.linux-x86_64/wheel/argparse2tool/cmdline2cwl/templates
  copying build/lib/argparse2tool/cmdline2cwl/templates/cwltool_inputs.j2 -> build/bdist.linux-x86_64/wheel/argparse2tool/cmdline2cwl/templates
  copying build/lib/argparse2tool/cmdline2cwl/templates/cwltool_outputs.j2 -> build/bdist.linux-x86_64/wheel/argparse2tool/cmdline2cwl/templates
  copying build/lib/argparse2tool/cmdline2cwl/templates/cwltool.j2 -> build/bdist.linux-x86_64/wheel/argparse2tool/cmdline2cwl/templates
  copying build/lib/argparse2tool/cmdline2cwl/__init__.py -> build/bdist.linux-x86_64/wheel/argparse2tool/cmdline2cwl
  creating build/bdist.linux-x86_64/wheel/argparse2tool/cmdline2gxml
  copying build/lib/argparse2tool/cmdline2gxml/__init__.py -> build/bdist.linux-x86_64/wheel/argparse2tool/cmdline2gxml
  copying build/lib/argparse2tool/__init__.py -> build/bdist.linux-x86_64/wheel/argparse2tool
  copying build/lib/argparse2tool/check_path.py -> build/bdist.linux-x86_64/wheel/argparse2tool
  running install_egg_info
  Copying argparse2tool.egg-info to build/bdist.linux-x86_64/wheel/argparse2tool-0.4.3.post2.egg-info
  running install_scripts
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-build-0do37j9w/argparse2tool/setup.py", line 25, in <module>
      include_package_data=True,
    File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/usr/lib/python3/dist-packages/wheel/bdist_wheel.py", line 215, in run
      self.run_command('install')
    File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/usr/lib/python3/dist-packages/setuptools/command/install.py", line 61, in run
      return orig.install.run(self)
    File "/usr/lib/python3.5/distutils/command/install.py", line 595, in run
      self.run_command(cmd_name)
    File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/usr/lib/python3/dist-packages/setuptools/command/install_scripts.py", line 16, in run
      import setuptools.command.easy_install as ei
    File "/usr/lib/python3/dist-packages/setuptools/command/easy_install.py", line 48, in <module>
      from setuptools.py31compat import get_path, get_config_vars
    File "/usr/lib/python3/dist-packages/setuptools/py31compat.py", line 2, in <module>
      import unittest
    File "/usr/lib/python3.5/unittest/__init__.py", line 64, in <module>
      from .main import TestProgram, main
    File "/usr/lib/python3.5/unittest/main.py", line 4, in <module>
      import argparse
    File "/tmp/pip-build-0do37j9w/argparse2tool/argparse/__init__.py", line 10, in <module>
      import galaxyxml.tool as gxt
  ImportError: No module named 'galaxyxml'
  
  ----------------------------------------
  Failed building wheel for argparse2tool
  Running setup.py clean for argparse2tool
  Running setup.py bdist_wheel for galaxyxml: started
  Running setup.py bdist_wheel for galaxyxml: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/d1/ea/77/e3b2e462a4608adfd57ccd3f08dc803be41631cd9a0abdea14
  Running setup.py bdist_wheel for future: started
  Running setup.py bdist_wheel for future: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/c2/50/7c/0d83b4baac4f63ff7a765bd16390d2ab43c93587fac9d6017a
  Running setup.py bdist_wheel for MarkupSafe: started
  Running setup.py bdist_wheel for MarkupSafe: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/88/a7/30/e39a54a87bcbe25308fa3ca64e8ddc75d9b3e5afa21ee32d57
Successfully built galaxyxml future MarkupSafe
Failed to build argparse2tool
Installing collected packages: lxml, galaxyxml, MarkupSafe, jinja2, future, argparse2tool
  Running setup.py install for argparse2tool: started
    Running setup.py install for argparse2tool: finished with status 'done'
Successfully installed MarkupSafe-1.0 argparse2tool-0.4.3.post2 future-0.16.0 galaxyxml-0.4.0 jinja2-2.9.6 lxml-3.7.3

As well as the tool for analysing:





In [97]:



    


for line in exec_command_in_container('pip3 install ' + TOOL_NAME, CONTAINER_ID):
    print(line.decode("utf-8")[:-1])



    


















    






Collecting tooldog
  Downloading tooldog-0.2.0.tar.gz
Collecting rdflib (from tooldog)
  Downloading rdflib-4.2.2-py3-none-any.whl (344kB)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from tooldog)
Requirement already satisfied: galaxyxml in /usr/local/lib/python3.5/dist-packages (from tooldog)
Collecting cwlgen (from tooldog)
  Downloading cwlgen-0.1.0.tar.gz
Collecting pyparsing (from rdflib->tooldog)
  Downloading pyparsing-2.2.0-py2.py3-none-any.whl (56kB)
Collecting isodate (from rdflib->tooldog)
  Downloading isodate-0.5.4.tar.gz
Requirement already satisfied: lxml in /usr/local/lib/python3.5/dist-packages (from galaxyxml->tooldog)
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from cwlgen->tooldog)
Collecting ruamel.yaml==0.13.13 (from cwlgen->tooldog)
  Downloading ruamel.yaml-0.13.13-cp35-cp35m-manylinux1_x86_64.whl (536kB)
Building wheels for collected packages: tooldog, cwlgen, isodate
  Running setup.py bdist_wheel for tooldog: started
  Running setup.py bdist_wheel for tooldog: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/69/61/26/59bf47f3f6cb6f86fe97126d5e9eb5f5edbf8a576be3c8d890
  Running setup.py bdist_wheel for cwlgen: started
  Running setup.py bdist_wheel for cwlgen: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/ef/59/eb/de39a2226114e01698e0b63c35a6d11d5b4bc20fc4c37b8c2d
  Running setup.py bdist_wheel for isodate: started
  Running setup.py bdist_wheel for isodate: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/61/c0/d2/6b4a10c222ba9261ab9872a8f05d471652962284e8c677e5e7
Successfully built tooldog cwlgen isodate
Installing collected packages: pyparsing, isodate, rdflib, ruamel.yaml, cwlgen, tooldog
Successfully installed cwlgen-0.1.0 isodate-0.5.4 pyparsing-2.2.0 rdflib-4.2.2 ruamel.yaml-0.13.13 tooldog-0.2.0

Now we are ready to generate CWL:





In [98]:



    


cwl_tool = '' # string representation of cwl tool
for line in exec_command_in_container('tooldog --generate_cwl_tool', CONTAINER_ID):
    cwl_tool += line.decode("utf-8")[:-1]

Write CWL to file:





In [99]:



    


write_to_file(OUTPUT_FILENAME, cwl_tool)

Finally, let's stop the container. It will be removed automatically.





In [100]:



    


run_command('docker stop ' + CONTAINER_ID)



    


















    
Out[100]:








<callable_iterator at 0x105212cf8>

















In [ ]:

Content source: khillion/ToolDog

Similar notebooks:

notebook.community | gallery | about