1.2 ls - List files and directory names, attributes

1.3 mkdir - Make a new directory

1.4 cd - Change to a particular directory

1.5 rmdir - Remove a directory

1.6 cp - Copy Files

1.7 rm - Remove files

1.8 mv : Move a file

1.9 CURL - Getting Data from the Command Line

1.10 head/tail

1.11 grep:

1.12 Redirection (or Downloading)

Overview

This is by no means an exhaustive list, the point is just to give you a feeler for what's possible. If you have used Linux or Mac, or have written code in Ruby, chances are you have used Unix commands already. If you're a Windows user, here's a good resource: https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/

Another great resource in general on the basics of unix commands:

http://matt.might.net/articles/basic-unix/

pwd - Print Working Directory



In [ ]:

    
!pwd

ls - List files and directory names, attributes

Some commonly used commands are below:

-A: list all of the contents of the queried directory, even hidden files.
-l: detailed format, display additional info for all files and directories.
-R: recursively list the contents of any subdirectories.
-t: sort files by the time of the last modification.
-S: sort files by size.
-r: reverse any sort order.
-h: when used in conjunction with -l, gives a more human-readable output.

You can also combine the commands/flags. For example:

Read more on this topic here: https://www.mkssoftware.com/docs/man1/ls.1.asp



In [ ]:

    
ls



In [ ]:

    
!ls



In [ ]:

    
ls -al



In [ ]:

    
ls -Al

mkdir - Make a new directory



In [ ]:

    
!mkdir NewFolder



In [ ]:

    
ls

cd - Change to a particular directory



In [ ]:

    
cd NewFolder



In [ ]:

    
cd ..



In [ ]:

    
ls

rmdir - Remove a directory

If the folder is not empty, it need the "-r" flag.

Example: rmdir -r NewFolder



In [ ]:

    
rmdir NewFolder



In [ ]:

    
ls

cp - Copy Files

Careful with the filenames! Will be overwritten without warning.



In [ ]:

    
ls



In [ ]:

    
# Copy in the same directory
!cp 01.Unix_and_Shell_Command_Basics.ipynb Notebook01.ipynb



In [ ]:

    
ls



In [ ]:

    
rm Notebook01.ipynb



In [ ]:

    
ls



In [ ]:

    
# Copy to another directory
!mkdir TempFolder
!cp 01.Unix_and_Shell_Command_Basics.ipynb TempFolder/File01.ipynb
!ls



In [ ]:

    
cd TempFolder



In [ ]:

    
ls

rm - Remove files

Note that this is different to rmdir, which exists to remove a directory



In [ ]:

    
ls



In [ ]:

    
pwd



In [ ]:

    
!rm File01.ipynb



In [ ]:

    
!ls



In [ ]:

    
!pwd



In [ ]:

    
!ls -al



In [ ]:

    
!cd ..



In [ ]:

    
!pwd



In [ ]:

    
!ls



In [ ]:

    
cd ..



In [ ]:

    
ls



In [ ]:

    
cp -i 01.Unix_and_Shell_Command_Basics.ipynb TempFolder/NewFile01.ipynb



In [ ]:

    
cd Tempfolder



In [ ]:

    
ls



In [ ]:

    
cp -i NewFile01.ipynb NewFile01.ipynb

mv : Move a file

This is close to the the 'cut' function available for files on Windows.

When you use the 'mv' command, a file is copied to a new location, and removed from it's original location.



In [ ]:

    
pwd



In [ ]:

    
ls



In [ ]:

    
rm -r TempFolder



In [ ]:

    
ls



In [ ]:

    
cp 01.Unix_and_Shell_Command_Basics.ipynb NewFile01.ipynb



In [ ]:

    
ls



In [ ]:

    
mkdir TempFolder02



In [ ]:

    
ls



In [ ]:

    
mv NewFile01.ipynb TempFolder02



In [ ]:

    
ls



In [ ]:

    
cd TempFolder02



In [ ]:

    
ls



In [ ]:

    
cd ..

CURL - Getting Data from the Command Line

Let's begin by copying a simple tab-separated file. The format is as below:

!curl -OptionalFlag 'http://url'



In [ ]:

    
!curl -L 'https://dl.dropboxusercontent.com/s/j2yh7nvlli1nsa5/gdp.txt'



In [ ]:

    
!curl -L 'https://dl.dropboxusercontent.com/s/eqyhkf3tpgre0jb/foo.txt'



In [ ]:

    
!curl -s "http://freegeoip.net/json/" | jq .



In [ ]:

    
!curl -s "http://api.open-notify.org/iss-now.json"



In [ ]:

    
!curl -s "http://api.open-notify.org/astros.json"



In [ ]:

    
!curl -X POST --include 'https://community-sentiment.p.mashape.com/text/' \
  -H 'X-Mashape-Key: YFWRiIyfNemshsFin8iTJy0XFUjNp1rXoY7jsnoPlVphvWnKY6' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -H 'Accept: application/json' \
  -d 'txt=My team lost! :('

Note: This is a free API, so I have exposed my API key in the code. In practice, if you are ever sharing code, please take adequate precautions, and never expose your private key.

head/tail



In [ ]:

    
pwd



In [ ]:

    
ls



In [ ]:

    
cd Data



In [ ]:

    
ls



In [ ]:

    
!head -n 3 sample.txt



In [ ]:

    
!tail -n 3 sample.txt



In [ ]:

    
!cat sample.txt



In [ ]:

    
# Selecting specific fields
!cut -f2,3 sample.txt



In [ ]:

    
!sort sample.txt



In [ ]:

    
!sort -k 2 sample.txt



In [ ]:

    
!wc sample.txt



In [ ]:

    
!wc -w sample.txt



In [ ]:

    
!find ~ -sample.txt 'sample.txt'

grep:

Grep is a pattern matching utility built into unix and it's flavors. The typical format is:

grep [option] [pattern] [file/s]



In [ ]:

    
pwd



In [ ]:

    
ls



In [ ]:

    
!cat nyt.txt



In [ ]:

    
# Count the number of matches
!grep -c 'Kennedy' nyt.txt



In [ ]:

    
!grep -o 'Kennedy' nyt.txt

More options for grep:

-c Print only a count of matched lines.
-l List only filenames
-i Ignore lowercase and uppercase distinctions
-o prints only the matching part of the line
-n Print matching line with its line number
-v Negate matches; print lines that do not match the regex
-r Recursively Search subdirectories listed



In [ ]:

    
!curl -s 'http://freegeoip.net/json/' > location.json



In [ ]:

    
!jq . location.json



In [ ]:

    
!curl -s 'http://freegeoip.net/json/' | jq .

Redirection (or Downloading)

This is really useful to quickly download a dataset using what is called an API Endpoint.

Let's download the 'Times Square Entertainment Venues' dataset from New York City's Open Data Portal to demonstrate this.

https://data.cityofnewyork.us/Business/Times-Square-Entertainment-Venues/jxdc-hnze



In [ ]:

    
!curl "https://data.cityofnewyork.us/resource/2pc8-n4xe.json" > venues.json



In [ ]:

    
!cat venues.json



In [ ]:

    
!grep 'Ripley' venues.json



In [ ]:

    
!grep -i 'Theater' venues.json



In [ ]:

    
# Multiple flags, and multiple conditions
!grep -v -e 'Theater' -e 'Theatre' venues.json



In [ ]:

Table of Contents