notebook.community

Edit and run 





In [1]:



    


%run -i nsls2_tag_build.py



    











In [2]:



    


pkg = 'nslsii'



    











In [3]:



    


run_container?



    


















    








Signature: run_container(*, pkg_name, pythons=('3.5', '3.6', '3.7'), numpy_versions=('1.14',))
Docstring:
Run a Docker container with supplied commands.

Parameters
----------
pkg_name : str
    the name of a package to build
pythons : tuple, optional
    versions of Python to build the package for
numpy_versions : tuple, optional
    versions of NumPy to build the package for
File:      ~/src/mrakitin/DAMA/lightsource2-recipes/scripts/nsls2_tag_build.py
Type:      function

Run directly synchronously





In [ ]:



    


out = run_container(pkg_name=pkg)



    











In [ ]:



    


print(out)

Run asynchronously with Dask!





In [2]:



    


from dask.distributed import Client, LocalCluster



    











In [3]:



    


cluster = LocalCluster(n_workers=8)



    











In [4]:



    


cluster



    


















    






 
 






















In [5]:



    


client = Client(cluster)



    











In [6]:



    


client



    


















    
Out[6]:












Client





Cluster


    
  
  • Workers: 8
    • 
  
  • Cores: 8
    • 
  
  • Memory: 34.36 GB
    • 























In [7]:



    


import datetime
def new_ts():
    start_time = datetime.datetime.now()
    return datetime.datetime.strftime(start_time, '%Y%m%d%H%M%S')



    











In [10]:



    


# packages = ['bluesky', 'databroker', 'ophyd']



    











In [11]:



    


packages = ['nslsii']



    











In [8]:



    


packages = ['analysis', 'collection']



    











In [12]:



    


# packages = []
# import glob
# for p in sorted(glob.glob(str(Path.cwd().parents[0] / Path('recipes-tag/*')))):
#     p = os.path.basename(p)
#     print(p)
#     packages.append(p)
# print(len(packages))



    











In [9]:



    


statuses = []
for p in packages:
    st = client.submit(run_container, pkg_name=p, pythons=('3.6', '3.7'),
                       numpy_versions=('1.14',),
                       key=f'{p}-{new_ts()}')
    statuses.append(st)



    











In [11]:



    


# for st in statuses:
#     print(st.result())



    











In [19]:



    


statuses



    


















    
Out[19]:








[<Future: status: finished, type: str, key: analysis-20180912171008>,
 <Future: status: finished, type: str, key: collection-20180912171008>]

















In [20]:



    


print(statuses[0].result())



    


















    






Logging summary to /root/auto-build-logs/09.12-21.10
Slack authentication successful.
{'allow_failures': True, 'numpy': ['1.14'], 'python': ['3.6', '3.7'], 'recipes_path': '/repo/recipes-tag/analysis', 'site': 'https://api.anaconda.org', 'token': 'blablabla', 'username': 'lightsource2-tag', 'upload': True}
Authenticating as the licode user
Authenticating to the NSLS-II team
Disabling binstar upload. If you want to turn it back on, execute: `conda config --set binstar_upload true`
recipes_path = /repo/recipes-tag/analysis

================================================================================
analysis
================================================================================

True     | 3.6   | 1.14  | linux-64/analysis-2018C3.1-0.tar.bz2
dependency_graph=<generator object resolve_dependencies at 0x7fe6abe67fc0>
Build Order.
linux-64/analysis-2018C3.1-0.tar.bz2
Building: linux-64/analysis-2018C3.1-0.tar.bz2
Build cmd: conda build /repo/recipes-tag/analysis --python 3.6 --numpy 1.14
Adding in variants from internal_defaults
Adding in variants from config.variant
Attempting to finalize metadata for analysis
BUILD START: ['analysis-2018C3.1-0.tar.bz2']
Solving environment: ...working... done
source tree in: /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/analysis_1536786634410/work
Packaging analysis
Packaging analysis-2018C3.1-0
number of files: 0
Fixing permissions
updating: analysis-2018C3.1-0.tar.bz2
TEST START: /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/linux-64/analysis-2018C3.1-0.tar.bz2
Updating index at /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/linux-64 to make package installable with dependencies
Updating index at /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/noarch to make package installable with dependencies
Adding in variants from /tmp/tmpe7l95_u3/info/recipe/conda_build_config.yaml
Nothing to test for: /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/linux-64/analysis-2018C3.1-0.tar.bz2
Renaming work directory,  /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/analysis_1536786634410/work  to  /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/analysis_1536786634410/work_moved_analysis-2018C3.1-0_linux-64_main_build_loop
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:

anaconda upload /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/linux-64/analysis-2018C3.1-0.tar.bz2

# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes

anaconda_upload is not set.  Not uploading wheels: []
####################################################################################
Resource usage summary:

Total time: 0:01:00.8
CPU usage: sys=0:00:00.0, user=0:00:00.0
Maximum memory usage observed: 0B
Total disk usage observed (not including envs): 0B


####################################################################################
Source and build intermediates have been left in /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld.
There are currently 1 accumulated.
To remove them, you can run the ```conda build purge``` command
UPLOAD START
Upload command=['anaconda', '-t', 'SCRUBBED', 'upload', '-u', 'lightsource2-tag', '/conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/linux-64/analysis-2018C3.1-0.tar.bz2']
Sending the following message to slack
Built and Uploaded linux-64/analysis-2018C3.1-0.tar.bz2 to lightsource2-tag on https://api.anaconda.org
Build summary
Expected 1 packages
Got 1 packages.
Breakdown is as follows
section: uploaded                 . number build: 1
section: no_token                 . number build: 0
section: upload_failed            . number build: 0
section: build_or_test_failed     . number build: 0
section: alreadybuilt             . number build: 0
Packages that were built and uploaded to the lightsource2-tag user on https://api.anaconda.org
['linux-64/analysis-2018C3.1-0.tar.bz2']


















In [21]:



    


print(statuses[1].result())



    


















    






Logging summary to /root/auto-build-logs/09.12-21.10
Slack authentication successful.
Authenticating as the licode user
Authenticating to the NSLS-II team
{'allow_failures': True, 'numpy': ['1.14'], 'python': ['3.6', '3.7'], 'recipes_path': '/repo/recipes-tag/collection', 'site': 'https://api.anaconda.org', 'token': 'blablabla', 'username': 'lightsource2-tag', 'upload': True}
Disabling binstar upload. If you want to turn it back on, execute: `conda config --set binstar_upload true`
recipes_path = /repo/recipes-tag/collection

================================================================================
collection
================================================================================

True     | 3.6   | 1.14  | linux-64/collection-2018C3.1-0.tar.bz2
dependency_graph=<generator object resolve_dependencies at 0x7f49da0d2fc0>
Build Order.
linux-64/collection-2018C3.1-0.tar.bz2
Building: linux-64/collection-2018C3.1-0.tar.bz2
Build cmd: conda build /repo/recipes-tag/collection --python 3.6 --numpy 1.14
Adding in variants from internal_defaults
Adding in variants from config.variant
Attempting to finalize metadata for collection
BUILD START: ['collection-2018C3.1-0.tar.bz2']
Solving environment: ...working... failed
Missing dependency analysis, but found recipe directory, so building analysis first
Attempting to finalize metadata for analysis
BUILD START: ['analysis-2018C3.1-0.tar.bz2']
Solving environment: ...working... done
source tree in: /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/analysis_1536786638041/work
Packaging analysis
Packaging analysis-2018C3.1-0
number of files: 0
Fixing permissions
updating: analysis-2018C3.1-0.tar.bz2
TEST START: /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/linux-64/analysis-2018C3.1-0.tar.bz2
Updating index at /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/linux-64 to make package installable with dependencies
Updating index at /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/noarch to make package installable with dependencies
Adding in variants from /tmp/tmp49awji3k/info/recipe/conda_build_config.yaml
Nothing to test for: /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/linux-64/analysis-2018C3.1-0.tar.bz2
Renaming work directory,  /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/analysis_1536786638041/work  to  /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/analysis_1536786638041/work_moved_analysis-2018C3.1-0_linux-64_main_build_loop
BUILD START: ['collection-2018C3.1-0.tar.bz2']
Solving environment: ...working... done
source tree in: /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/collection_1536786698043/work
Packaging collection
number of files: 0
Fixing permissions
updating: collection-2018C3.1-0.tar.bz2
TEST START: /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/linux-64/collection-2018C3.1-0.tar.bz2
Adding in variants from /tmp/tmpj_z8_0ax/info/recipe/conda_build_config.yaml
Nothing to test for: /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/linux-64/collection-2018C3.1-0.tar.bz2
Renaming work directory,  /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/collection_1536786698043/work  to  /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/collection_1536786698043/work_moved_collection-2018C3.1-0_linux-64_main_build_loop
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:

anaconda upload /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/linux-64/analysis-2018C3.1-0.tar.bz2
anaconda upload /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/linux-64/collection-2018C3.1-0.tar.bz2

# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes

anaconda_upload is not set.  Not uploading wheels: []
####################################################################################
Resource usage summary:

Total time: 0:02:17.3
CPU usage: sys=0:00:00.0, user=0:00:00.0
Maximum memory usage observed: 0B
Total disk usage observed (not including envs): 0B


####################################################################################
Source and build intermediates have been left in /conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld.
There are currently 3 accumulated.
To remove them, you can run the ```conda build purge``` command
UPLOAD START
Upload command=['anaconda', '-t', 'SCRUBBED', 'upload', '-u', 'lightsource2-tag', '/conda/tmp_envs/55fc582f7dd397d3a5f0/conda-bld/linux-64/collection-2018C3.1-0.tar.bz2']
Sending the following message to slack
Built and Uploaded linux-64/collection-2018C3.1-0.tar.bz2 to lightsource2-tag on https://api.anaconda.org
Build summary
Expected 1 packages
Got 1 packages.
Breakdown is as follows
section: uploaded                 . number build: 1
section: no_token                 . number build: 0
section: upload_failed            . number build: 0
section: build_or_test_failed     . number build: 0
section: alreadybuilt             . number build: 0
Packages that were built and uploaded to the lightsource2-tag user on https://api.anaconda.org
['linux-64/collection-2018C3.1-0.tar.bz2']











    






distributed.comm.tcp - WARNING - Closing dangling stream in <TCP local=tcp://127.0.0.1:62194 remote=tcp://127.0.0.1:58551>

















In [ ]:

Content source: NSLS-II/lightsource2-recipes

Similar notebooks:

notebook.community | gallery | about