This is by no means an exhaustive list, the point is just to give you a feeler for what's possible. If you have used Linux or Mac, or have written code in Ruby, chances are you have used Unix commands already. If you're a Windows user, here are two good resources:
Another great resource in general on the basics of unix commands:
In [7]:
!pwd
Some commonly used commands are below:
You can also combine the commands/flags. For example:
Read more on this topic here: https://www.mkssoftware.com/docs/man1/ls.1.asp
In [8]:
ls
In [9]:
!ls
In [10]:
ls -al
In [ ]:
ls -Al
In [11]:
!mkdir NewFolder
In [12]:
ls
In [13]:
cd NewFolder
In [14]:
!pwd
In [15]:
cd ..
In [16]:
!pwd
In [17]:
ls
If the folder is not empty, it need the "-r" flag.
Example: rmdir -r NewFolder
In [18]:
rmdir NewFolder
In [19]:
ls
Careful with the filenames! Will be overwritten without warning.
In [20]:
ls
In [21]:
# Copy in the same directory
!cp 01.Unix_and_Shell_Command_Basics.ipynb Notebook01.ipynb
In [22]:
ls
In [23]:
rm Notebook01.ipynb
In [24]:
ls
In [25]:
# Copy to another directory
!mkdir TempFolder
!cp 01.Unix_and_Shell_Command_Basics.ipynb TempFolder/File01.ipynb
!ls
In [26]:
cd TempFolder
In [27]:
ls
Note that this is different to rmdir, which exists to remove a directory
In [28]:
pwd
Out[28]:
In [29]:
!rm File01.ipynb
In [30]:
!ls
In [31]:
!pwd
In [32]:
!ls -al
In [33]:
!ls
In [34]:
cd ..
In [35]:
ls
This is close to the the 'cut' function available for files on Windows.
When you use the 'mv' command, a file is copied to a new location, and removed from it's original location.
In [36]:
pwd
Out[36]:
In [37]:
ls
In [38]:
rm -r TempFolder
In [39]:
ls
In [40]:
cp 01.Unix_and_Shell_Command_Basics.ipynb NewFile01.ipynb
In [41]:
ls
In [42]:
mkdir TempFolder02
In [43]:
ls
In [44]:
mv NewFile01.ipynb TempFolder02
In [45]:
ls
In [46]:
cd TempFolder02
In [47]:
ls
In [48]:
cd ..
In [49]:
rm -r TempFolder02
Let's begin by copying a simple tab-separated file. The format is as below:
In [50]:
!curl -L 'https://dl.dropboxusercontent.com/s/j2yh7nvlli1nsa5/gdp.txt'
In [51]:
!curl -L 'https://dl.dropboxusercontent.com/s/eqyhkf3tpgre0jb/foo.txt'
In [52]:
!curl -s "http://freegeoip.net/json/" | jq .
In case your system doesn't have jq, you can follow the instructions here.
In [53]:
!curl -s "http://api.open-notify.org/iss-now.json"
In [54]:
!curl -s "http://api.open-notify.org/astros.json"
Register for the Mashape API Market here: https://market.mashape.com
In [55]:
!curl -X POST --include 'https://community-sentiment.p.mashape.com/text/' \
-H 'X-Mashape-Key: YFWRiIyfNemshsFin8iTJy0XFUjNp1rXoY7jsnoPlVphvWnKY6' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'Accept: application/json' \
-d 'txt=My team lost badly! I am sad :('
Note: This is a free API, so I have exposed my API key in the code. In practice, if you are ever sharing code, please take adequate precautions, and never expose your private key.
In [56]:
pwd
Out[56]:
In [57]:
ls
In [58]:
cd Data
In [59]:
ls
In [60]:
!head -n 3 sample.txt
In [61]:
!tail -n 4 sample.txt
In [62]:
!cat sample.txt
In [63]:
# Selecting specific fields
!cut -f2,3 sample.txt
In [64]:
!sort sample.txt
In [65]:
!sort -k 2 sample.txt
In [67]:
!cat nyt.txt
In [66]:
!wc nyt.txt
# Where 21 is the number of lines, 245 is the number of words, and 1515 is the number of characters.
In [68]:
!wc -w nyt.txt
Grep is a pattern matching utility built into unix and it's flavors. The typical format is:
grep [option] [pattern] [file/s]
In [69]:
pwd
Out[69]:
In [70]:
ls
In [71]:
!cat nyt.txt
In [72]:
# Count the number of matches
!grep -c 'Kennedy' nyt.txt
In [73]:
!grep -o 'Kennedy' nyt.txt
More options for grep:
In [74]:
!curl -s 'http://freegeoip.net/json/' > location.json
In [75]:
!jq . location.json
In [ ]:
!curl -s 'http://freegeoip.net/json/' | jq .
This is really useful to quickly download a dataset using what is called an API Endpoint.
Let's download the 'Times Square Entertainment Venues' dataset from New York City's Open Data Portal to demonstrate this.
https://data.cityofnewyork.us/Business/Times-Square-Entertainment-Venues/jxdc-hnze
In [76]:
!curl "https://data.cityofnewyork.us/resource/2pc8-n4xe.json" > venues.json
In [77]:
!cat venues.json
In [78]:
!grep 'Ripley' venues.json
In [79]:
!grep -i 'Theater' venues.json
In [80]:
# Multiple flags, and multiple conditions
!grep -v -e 'Theater' -e 'Theatre' venues.json
In [ ]: