Run this from the _talks/ directory, which contains .md files of all your talks. This scrapes the location YAML field from each .md file, geolocates it with geopy/Nominatim, and uses the getorg library to output data, HTML, and Javascript for a standalone cluster map.
In [1]:
!pip install getorg --upgrade
!pip install ipyleaflet --upgrade
import glob
import getorg
from geopy import Nominatim
Collecting getorg
Downloading https://files.pythonhosted.org/packages/d8/81/842a17df2892d8f357929a3a937d30f36bd11998b31b75e663515e24bf50/getorg-0.3.1-py2.py3-none-any.whl
Collecting retrying (from getorg)
Downloading https://files.pythonhosted.org/packages/44/ef/beae4b4ef80902f22e3af073397f079c96969c69b2c7d52a57ea9ae61c9d/retrying-1.3.3.tar.gz
Collecting geopy (from getorg)
Downloading https://files.pythonhosted.org/packages/9b/1b/f63484e2c257a79f9fed6243f638d0fc50566428b87994dbf2cdf681a05e/geopy-1.14.0-py2.py3-none-any.whl (77kB)
100% |████████████████████████████████| 81kB 1.3MB/s ta 0:00:01
Collecting pygithub (from getorg)
Downloading https://files.pythonhosted.org/packages/fd/28/c42bdd56111dae8c622c01af69e6d7f3d3ff0a96ad846f221749701f86b1/PyGithub-1.39.tar.gz (2.6MB)
100% |████████████████████████████████| 2.6MB 2.4MB/s ta 0:00:01
Requirement not upgraded as not directly required: six>=1.7.0 in /usr/local/lib/python3.5/dist-packages (from retrying->getorg) (1.11.0)
Collecting geographiclib<2,>=1.49 (from geopy->getorg)
Downloading https://files.pythonhosted.org/packages/5b/ac/4f348828091490d77899bc74e92238e2b55c59392f21948f296e94e50e2b/geographiclib-1.49.tar.gz
Collecting pyjwt (from pygithub->getorg)
Downloading https://files.pythonhosted.org/packages/93/d1/3378cc8184a6524dc92993090ee8b4c03847c567e298305d6cf86987e005/PyJWT-1.6.4-py2.py3-none-any.whl
Building wheels for collected packages: retrying, pygithub, geographiclib
Running setup.py bdist_wheel for retrying ... done
Stored in directory: /home/adriano/.cache/pip/wheels/d7/a9/33/acc7b709e2a35caa7d4cae442f6fe6fbf2c43f80823d46460c
Running setup.py bdist_wheel for pygithub ... done
Stored in directory: /home/adriano/.cache/pip/wheels/62/39/e7/d7a4b5d1fb37b89363a976553a0bea41f550ac54594f97a096
Running setup.py bdist_wheel for geographiclib ... done
Stored in directory: /home/adriano/.cache/pip/wheels/99/45/d1/14954797e2a976083182c2e7da9b4e924509e59b6e5c661061
Successfully built retrying pygithub geographiclib
Installing collected packages: retrying, geographiclib, geopy, pyjwt, pygithub, getorg
Could not install packages due to an EnvironmentError: [Errno 13] Permissão negada: '/usr/local/lib/python3.5/dist-packages/retrying.py'
Consider using the `--user` option or check the permissions.
Collecting ipyleaflet
Downloading https://files.pythonhosted.org/packages/38/b3/4b4833102a6d64b1bdd464b6e3cf9d8f9bf5471c1e7ca6c47a6b63597457/ipyleaflet-0.8.1-py2.py3-none-any.whl (778kB)
100% |████████████████████████████████| 788kB 2.1MB/s ta 0:00:01
Requirement not upgraded as not directly required: ipywidgets<8,>=7.0.0 in /usr/local/lib/python3.5/dist-packages (from ipyleaflet) (7.1.0)
Requirement not upgraded as not directly required: widgetsnbextension~=3.1.0 in /usr/local/lib/python3.5/dist-packages (from ipywidgets<8,>=7.0.0->ipyleaflet) (3.1.0)
Requirement not upgraded as not directly required: nbformat>=4.2.0 in /usr/local/lib/python3.5/dist-packages (from ipywidgets<8,>=7.0.0->ipyleaflet) (4.4.0)
Requirement not upgraded as not directly required: ipython>=4.0.0; python_version >= "3.3" in /usr/local/lib/python3.5/dist-packages (from ipywidgets<8,>=7.0.0->ipyleaflet) (6.3.1)
Requirement not upgraded as not directly required: ipykernel>=4.5.1 in /usr/local/lib/python3.5/dist-packages (from ipywidgets<8,>=7.0.0->ipyleaflet) (4.8.2)
Requirement not upgraded as not directly required: traitlets>=4.3.1 in /usr/local/lib/python3.5/dist-packages (from ipywidgets<8,>=7.0.0->ipyleaflet) (4.3.2)
Requirement not upgraded as not directly required: notebook>=4.4.1 in /usr/local/lib/python3.5/dist-packages (from widgetsnbextension~=3.1.0->ipywidgets<8,>=7.0.0->ipyleaflet) (5.3.1)
Requirement not upgraded as not directly required: jupyter-core in /usr/local/lib/python3.5/dist-packages (from nbformat>=4.2.0->ipywidgets<8,>=7.0.0->ipyleaflet) (4.4.0)
Requirement not upgraded as not directly required: jsonschema!=2.5.0,>=2.4 in /usr/local/lib/python3.5/dist-packages (from nbformat>=4.2.0->ipywidgets<8,>=7.0.0->ipyleaflet) (2.6.0)
Requirement not upgraded as not directly required: ipython-genutils in /usr/local/lib/python3.5/dist-packages (from nbformat>=4.2.0->ipywidgets<8,>=7.0.0->ipyleaflet) (0.2.0)
Requirement not upgraded as not directly required: pickleshare in /usr/local/lib/python3.5/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets<8,>=7.0.0->ipyleaflet) (0.7.4)
Requirement not upgraded as not directly required: backcall in /usr/local/lib/python3.5/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets<8,>=7.0.0->ipyleaflet) (0.1.0)
Requirement not upgraded as not directly required: simplegeneric>0.8 in /usr/lib/python3/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets<8,>=7.0.0->ipyleaflet) (0.8.1)
Requirement not upgraded as not directly required: pygments in /usr/local/lib/python3.5/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets<8,>=7.0.0->ipyleaflet) (2.2.0)
Requirement not upgraded as not directly required: prompt-toolkit<2.0.0,>=1.0.15 in /usr/local/lib/python3.5/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets<8,>=7.0.0->ipyleaflet) (1.0.15)
Requirement not upgraded as not directly required: setuptools>=18.5 in /usr/local/lib/python3.5/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets<8,>=7.0.0->ipyleaflet) (39.0.1)
Requirement not upgraded as not directly required: decorator in /usr/local/lib/python3.5/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets<8,>=7.0.0->ipyleaflet) (4.3.0)
Requirement not upgraded as not directly required: pexpect; sys_platform != "win32" in /usr/local/lib/python3.5/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets<8,>=7.0.0->ipyleaflet) (4.5.0)
Requirement not upgraded as not directly required: jedi>=0.10 in /usr/local/lib/python3.5/dist-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets<8,>=7.0.0->ipyleaflet) (0.12.0)
Requirement not upgraded as not directly required: tornado>=4.0 in /usr/local/lib/python3.5/dist-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7.0.0->ipyleaflet) (5.0.2)
Requirement not upgraded as not directly required: jupyter-client in /usr/local/lib/python3.5/dist-packages (from ipykernel>=4.5.1->ipywidgets<8,>=7.0.0->ipyleaflet) (5.2.3)
Requirement not upgraded as not directly required: six in /usr/local/lib/python3.5/dist-packages (from traitlets>=4.3.1->ipywidgets<8,>=7.0.0->ipyleaflet) (1.11.0)
Requirement not upgraded as not directly required: jinja2 in /usr/local/lib/python3.5/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.1.0->ipywidgets<8,>=7.0.0->ipyleaflet) (2.10)
Requirement not upgraded as not directly required: Send2Trash in /usr/local/lib/python3.5/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.1.0->ipywidgets<8,>=7.0.0->ipyleaflet) (1.4.2)
Requirement not upgraded as not directly required: terminado>=0.8.1 in /usr/local/lib/python3.5/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.1.0->ipywidgets<8,>=7.0.0->ipyleaflet) (0.8.1)
Requirement not upgraded as not directly required: nbconvert in /usr/local/lib/python3.5/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.1.0->ipywidgets<8,>=7.0.0->ipyleaflet) (5.3.1)
Requirement not upgraded as not directly required: wcwidth in /usr/local/lib/python3.5/dist-packages (from prompt-toolkit<2.0.0,>=1.0.15->ipython>=4.0.0; python_version >= "3.3"->ipywidgets<8,>=7.0.0->ipyleaflet) (0.1.7)
Requirement not upgraded as not directly required: ptyprocess>=0.5 in /usr/local/lib/python3.5/dist-packages (from pexpect; sys_platform != "win32"->ipython>=4.0.0; python_version >= "3.3"->ipywidgets<8,>=7.0.0->ipyleaflet) (0.5.2)
Requirement not upgraded as not directly required: parso>=0.2.0 in /usr/local/lib/python3.5/dist-packages (from jedi>=0.10->ipython>=4.0.0; python_version >= "3.3"->ipywidgets<8,>=7.0.0->ipyleaflet) (0.2.0)
Requirement not upgraded as not directly required: python-dateutil>=2.1 in /usr/local/lib/python3.5/dist-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets<8,>=7.0.0->ipyleaflet) (2.7.2)
Requirement not upgraded as not directly required: pyzmq>=13 in /usr/local/lib/python3.5/dist-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets<8,>=7.0.0->ipyleaflet) (17.0.0)
Requirement not upgraded as not directly required: MarkupSafe>=0.23 in /usr/local/lib/python3.5/dist-packages (from jinja2->notebook>=4.4.1->widgetsnbextension~=3.1.0->ipywidgets<8,>=7.0.0->ipyleaflet) (1.0)
Requirement not upgraded as not directly required: testpath in /usr/local/lib/python3.5/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.1.0->ipywidgets<8,>=7.0.0->ipyleaflet) (0.3.1)
Requirement not upgraded as not directly required: bleach in /usr/local/lib/python3.5/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.1.0->ipywidgets<8,>=7.0.0->ipyleaflet) (2.1.3)
Requirement not upgraded as not directly required: entrypoints>=0.2.2 in /usr/local/lib/python3.5/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.1.0->ipywidgets<8,>=7.0.0->ipyleaflet) (0.2.3)
Requirement not upgraded as not directly required: pandocfilters>=1.4.1 in /usr/local/lib/python3.5/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.1.0->ipywidgets<8,>=7.0.0->ipyleaflet) (1.4.2)
Requirement not upgraded as not directly required: mistune>=0.7.4 in /usr/local/lib/python3.5/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.1.0->ipywidgets<8,>=7.0.0->ipyleaflet) (0.8.3)
Requirement not upgraded as not directly required: html5lib!=1.0b1,!=1.0b2,!=1.0b3,!=1.0b4,!=1.0b5,!=1.0b6,!=1.0b7,!=1.0b8,>=0.99999999pre in /usr/local/lib/python3.5/dist-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.1.0->ipywidgets<8,>=7.0.0->ipyleaflet) (1.0.1)
Requirement not upgraded as not directly required: webencodings in /usr/local/lib/python3.5/dist-packages (from html5lib!=1.0b1,!=1.0b2,!=1.0b3,!=1.0b4,!=1.0b5,!=1.0b6,!=1.0b7,!=1.0b8,>=0.99999999pre->bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.1.0->ipywidgets<8,>=7.0.0->ipyleaflet) (0.5.1)
Installing collected packages: ipyleaflet
Could not install packages due to an EnvironmentError: [Errno 13] Permissão negada: '/usr/local/lib/python3.5/dist-packages/ipyleaflet-0.8.1.dist-info'
Consider using the `--user` option or check the permissions.
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-1-d50ee63e2040> in <module>()
2 get_ipython().system('pip install ipyleaflet --upgrade')
3 import glob
----> 4 import getorg
5 from geopy import Nominatim
ImportError: No module named 'getorg'
In [2]:
g = glob.glob('_talks/*.md')
In [3]:
geocoder = Nominatim()
location_dict = {}
location = ""
permalink = ""
title = ""
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-3-cd980ae579e7> in <module>()
----> 1 geocoder = Nominatim()
2 location_dict = {}
3 location = ""
4 permalink = ""
5 title = ""
NameError: name 'Nominatim' is not defined
In [4]:
for file in g:
with open(file, 'r') as f:
lines = f.read()
if lines.find('location: "') > 1:
loc_start = lines.find('location: "') + 11
lines_trim = lines[loc_start:]
loc_end = lines_trim.find('"')
location = lines_trim[:loc_end]
location_dict[location] = geocoder.geocode(location)
print(location, "\n", location_dict[location])
Mar Del Plata, Argentina
Mar del Plata, Partido de General Pueyrredón, Bs. As., Argentina
Bad Honnef, Germany
Bad Honnef, Rhein-Sieg-Kreis, Regierungsbezirk Köln, Nordrhein-Westfalen, 53604, Deutschland
São Paulo, Brazil
SP, Microrregião de São Paulo, RMSP, Mesorregião Metropolitana de São Paulo, SP, Região Sudeste, Brasil
Valparaíso, Chile
Valparaíso, Provincia de Valparaíso, V Región de Valparaíso, Chile
In [11]:
m = getorg.orgmap.create_map_obj()
getorg.orgmap.output_html_cluster_map(location_dict, folder_name="talkmap", hashed_usernames=False)
Out[11]:
'Written map to talkmap/'
In [ ]:
Content source: abbarreto/abbarreto.github.io
Similar notebooks: