Overview

This is by no means an exhaustive list, the point is just to give you a feeler for what's possible. If you have used Linux or Mac, or have written code in Ruby, chances are you have used Unix commands already. If you're a Windows user, here are two good resources:

Another great resource in general on the basics of unix commands:

pwd - Print Working Directory


In [7]:
!pwd


/Volumes/HDD3TB/05. Work/03. Udemy/03. Final/pyds

ls - List files and directory names, attributes

Some commonly used commands are below:

  • -A: list all of the contents of the queried directory, even hidden files.
  • -l: detailed format, display additional info for all files and directories.
  • -R: recursively list the contents of any subdirectories.
  • -t: sort files by the time of the last modification.
  • -S: sort files by size.
  • -r: reverse any sort order.
  • -h: when used in conjunction with -l, gives a more human-readable output.

You can also combine the commands/flags. For example:

  • -al
  • -Al

Read more on this topic here: https://www.mkssoftware.com/docs/man1/ls.1.asp


In [8]:
ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data/
LICENSE
README.md
images/

In [9]:
!ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data
LICENSE
README.md
images

In [10]:
ls -al


total 2288
drwxr-xr-x  23 prasant  staff     782 Mar 14 21:14 ./
drwxr-xr-x   6 prasant  staff     204 Mar 11 21:04 ../
-rw-r--r--@  1 prasant  staff    6148 Mar  2 23:45 .DS_Store
drwxr-xr-x  15 prasant  staff     510 Mar  9 21:10 .git/
-rw-r--r--@  1 prasant  staff     329 Mar  2 14:53 .gitignore
drwxr-xr-x  13 prasant  staff     442 Mar  9 21:16 .ipynb_checkpoints/
-rw-r--r--@  1 prasant  staff   27863 Mar 14 21:14 01.Unix_and_Shell_Command_Basics.ipynb
-rw-r--r--@  1 prasant  staff   15343 Mar  5 04:13 02.Python_Basics.ipynb
-rw-r--r--@  1 prasant  staff   13087 Mar  2 18:53 03.All_about_Numbers.ipynb
-rw-r--r--@  1 prasant  staff   40825 Mar 11 01:04 04.String_me_along.ipynb
-rw-r--r--@  1 prasant  staff    8315 Mar  6 01:42 05.Booleans_True_or_False.ipynb
-rw-r--r--@  1 prasant  staff   34341 Mar  6 03:35 06.List_it_out.ipynb
-rw-r--r--@  1 prasant  staff   35965 Mar  8 00:04 07.Loop_it_up.ipynb
-rw-r--r--@  1 prasant  staff   24331 Mar  8 02:47 08.Let_it_Function.ipynb
-rw-r--r--@  1 prasant  staff   22151 Mar  8 22:56 09.Reading_writing_files.ipynb
-rw-r--r--@  1 prasant  staff  555392 Mar  9 21:10 10.Visualise_This.ipynb
-rw-r--r--@  1 prasant  staff   67424 Mar 10 03:25 11.Introduction_to_Numpy.ipynb
-rw-r--r--@  1 prasant  staff  229070 Mar 10 17:31 12.Introduction_to_Pandas.ipynb
-rw-r--r--   1 prasant  staff   22456 Mar  3 00:17 Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
drwxr-xr-x  20 prasant  staff     680 Mar  5 23:56 Data/
-rw-r--r--   1 prasant  staff    1075 Mar  2 13:12 LICENSE
-rw-r--r--   1 prasant  staff     834 Mar  3 00:26 README.md
drwxr-xr-x@ 25 prasant  staff     850 Mar  9 08:06 images/

In [ ]:
ls -Al

mkdir - Make a new directory


In [11]:
!mkdir NewFolder

In [12]:
ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data/
LICENSE
NewFolder/
README.md
images/

cd - Change to a particular directory


In [13]:
cd NewFolder


/Volumes/HDD3TB/05. Work/03. Udemy/03. Final/pyds/NewFolder

In [14]:
!pwd


/Volumes/HDD3TB/05. Work/03. Udemy/03. Final/pyds/NewFolder

In [15]:
cd ..


/Volumes/HDD3TB/05. Work/03. Udemy/03. Final/pyds

In [16]:
!pwd


/Volumes/HDD3TB/05. Work/03. Udemy/03. Final/pyds

In [17]:
ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data/
LICENSE
NewFolder/
README.md
images/

rmdir - Remove a directory

If the folder is not empty, it need the "-r" flag.

Example: rmdir -r NewFolder


In [18]:
rmdir NewFolder

In [19]:
ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data/
LICENSE
README.md
images/

cp - Copy Files

Careful with the filenames! Will be overwritten without warning.


In [20]:
ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data/
LICENSE
README.md
images/

In [21]:
# Copy in the same directory
!cp 01.Unix_and_Shell_Command_Basics.ipynb Notebook01.ipynb

In [22]:
ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data/
LICENSE
Notebook01.ipynb
README.md
images/

In [23]:
rm Notebook01.ipynb

In [24]:
ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data/
LICENSE
README.md
images/

In [25]:
# Copy to another directory
!mkdir TempFolder
!cp 01.Unix_and_Shell_Command_Basics.ipynb TempFolder/File01.ipynb
!ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data
LICENSE
README.md
TempFolder
images

In [26]:
cd TempFolder


/Volumes/HDD3TB/05. Work/03. Udemy/03. Final/pyds/TempFolder

In [27]:
ls


File01.ipynb

rm - Remove files

Note that this is different to rmdir, which exists to remove a directory


In [28]:
pwd


Out[28]:
'/Volumes/HDD3TB/05. Work/03. Udemy/03. Final/pyds/TempFolder'

In [29]:
!rm File01.ipynb

In [30]:
!ls

In [31]:
!pwd


/Volumes/HDD3TB/05. Work/03. Udemy/03. Final/pyds/TempFolder

In [32]:
!ls -al


total 0
drwxr-xr-x   2 prasant  staff   68 Mar 14 21:18 .
drwxr-xr-x  24 prasant  staff  816 Mar 14 21:18 ..

In [33]:
!ls

In [34]:
cd ..


/Volumes/HDD3TB/05. Work/03. Udemy/03. Final/pyds

In [35]:
ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data/
LICENSE
README.md
TempFolder/
images/

mv : Move a file

This is close to the the 'cut' function available for files on Windows.

When you use the 'mv' command, a file is copied to a new location, and removed from it's original location.


In [36]:
pwd


Out[36]:
'/Volumes/HDD3TB/05. Work/03. Udemy/03. Final/pyds'

In [37]:
ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data/
LICENSE
README.md
TempFolder/
images/

In [38]:
rm -r TempFolder

In [39]:
ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data/
LICENSE
README.md
images/

In [40]:
cp 01.Unix_and_Shell_Command_Basics.ipynb NewFile01.ipynb

In [41]:
ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data/
LICENSE
NewFile01.ipynb
README.md
images/

In [42]:
mkdir TempFolder02

In [43]:
ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data/
LICENSE
NewFile01.ipynb
README.md
TempFolder02/
images/

In [44]:
mv NewFile01.ipynb TempFolder02

In [45]:
ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data/
LICENSE
README.md
TempFolder02/
images/

In [46]:
cd TempFolder02


/Volumes/HDD3TB/05. Work/03. Udemy/03. Final/pyds/TempFolder02

In [47]:
ls


NewFile01.ipynb

In [48]:
cd ..


/Volumes/HDD3TB/05. Work/03. Udemy/03. Final/pyds

In [49]:
rm -r TempFolder02

CURL - Getting Data from the Command Line

Let's begin by copying a simple tab-separated file. The format is as below:


In [50]:
!curl -L 'https://dl.dropboxusercontent.com/s/j2yh7nvlli1nsa5/gdp.txt'


1	China	21,269,331.00
2	United States	$18,562,129.00
3	India	$8,720,758.00
4	Japan	$4,932,102.00
5	Germany	$3,979,664.00


In [51]:
!curl -L 'https://dl.dropboxusercontent.com/s/eqyhkf3tpgre0jb/foo.txt'


123	1346699925	11122	foo bar
222	1346699955	11145	bis baz
120	1346710100	11122	hei hew
234	1346700000	11135	cip bop
146	1346699999	11123	foo bar
99	1346750000	11135	bit bop
99	1346750000	11135	bop bip
99	1346750000	11135	bip bop
99	1346750000	11135	bip bop

In [52]:
!curl -s "http://freegeoip.net/json/" | jq .


{
  "ip": "176.227.198.122",
  "country_code": "GB",
  "country_name": "United Kingdom",
  "region_code": "",
  "region_name": "",
  "city": "",
  "zip_code": "",
  "time_zone": "Europe/London",
  "latitude": 51.4964,
  "longitude": -0.1224,
  "metro_code": 0
}

In case your system doesn't have jq, you can follow the instructions here.


In [53]:
!curl -s "http://api.open-notify.org/iss-now.json"


{"timestamp": 1489541023, "iss_position": {"longitude": "147.6925", "latitude": "-16.9322"}, "message": "success"}

In [54]:
!curl -s "http://api.open-notify.org/astros.json"


{"people": [{"craft": "ISS", "name": "Sergey Rizhikov"}, {"craft": "ISS", "name": "Andrey Borisenko"}, {"craft": "ISS", "name": "Shane Kimbrough"}, {"craft": "ISS", "name": "Oleg Novitskiy"}, {"craft": "ISS", "name": "Thomas Pesquet"}, {"craft": "ISS", "name": "Peggy Whitson"}], "message": "success", "number": 6}

Register for the Mashape API Market here: https://market.mashape.com


In [55]:
!curl -X POST --include 'https://community-sentiment.p.mashape.com/text/' \
  -H 'X-Mashape-Key: YFWRiIyfNemshsFin8iTJy0XFUjNp1rXoY7jsnoPlVphvWnKY6' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -H 'Accept: application/json' \
  -d 'txt=My team lost badly! I am sad :('











{
  "result": {
    "confidence": "84.2451", 
    "sentiment": "Negative"
  }
}

Note: This is a free API, so I have exposed my API key in the code. In practice, if you are ever sharing code, please take adequate precautions, and never expose your private key.

head/tail


In [56]:
pwd


Out[56]:
'/Volumes/HDD3TB/05. Work/03. Udemy/03. Final/pyds'

In [57]:
ls


01.Unix_and_Shell_Command_Basics.ipynb
02.Python_Basics.ipynb
03.All_about_Numbers.ipynb
04.String_me_along.ipynb
05.Booleans_True_or_False.ipynb
06.List_it_out.ipynb
07.Loop_it_up.ipynb
08.Let_it_Function.ipynb
09.Reading_writing_files.ipynb
10.Visualise_This.ipynb
11.Introduction_to_Numpy.ipynb
12.Introduction_to_Pandas.ipynb
Case+Study+1+-+Hunt+a+Serial+Killer.ipynb
Data/
LICENSE
README.md
images/

In [58]:
cd Data


/Volumes/HDD3TB/05. Work/03. Udemy/03. Final/pyds/Data

In [59]:
ls


Sopranos/                              gdp.txt
Times_Square_Entertainment_Venues.csv  location.json
bev-sales.xlsx                         nyc_rest.csv
cafe_sales.csv                         nyt.txt
cafe_sales2015.csv                     sample.txt
cafe_sales2015.txt                     tips.csv*
database.csv.zip                       venues.json
foo.txt                                wine/
gdp.csv

In [60]:
!head -n 3 sample.txt


1	China	21,269,331.00
2	United States	$18,562,129.00
3	India	$8,720,758.00

In [61]:
!tail -n 4 sample.txt


3	India	$8,720,758.00
4	Japan	$4,932,102.00
5	Germany	$3,979,664.00


In [62]:
!cat sample.txt


1	China	21,269,331.00
2	United States	$18,562,129.00
3	India	$8,720,758.00
4	Japan	$4,932,102.00
5	Germany	$3,979,664.00


In [63]:
# Selecting specific fields
!cut -f2,3 sample.txt


China	21,269,331.00
United States	$18,562,129.00
India	$8,720,758.00
Japan	$4,932,102.00
Germany	$3,979,664.00


In [64]:
!sort sample.txt


1	China	21,269,331.00
2	United States	$18,562,129.00
3	India	$8,720,758.00
4	Japan	$4,932,102.00
5	Germany	$3,979,664.00

In [65]:
!sort -k 2 sample.txt


1	China	21,269,331.00
5	Germany	$3,979,664.00
3	India	$8,720,758.00
4	Japan	$4,932,102.00
2	United States	$18,562,129.00

In [67]:
!cat nyt.txt


https://www.nytimes.com/2017/02/08/world/europe/letters-from-jacqueline-kennedy-to-the-man-she-didnt-marry.html

LONDON — In November 1967, four years after her husband’s assassination, Jacqueline Kennedy traveled to the temples of Angkor Wat in Cambodia on a much-publicized trip with David Ormsby Gore, a friend of her husband and himself a recent widower.

There was much speculation of a romantic attachment. A few months later, Mr. Ormsby Gore, a former British ambassador to Washington, proposed marriage. She turned him down.

In a handwritten letter, filled with anguish and a touch of cruelty, she explained her decision to marry Aristotle Onassis instead.

“If ever I can find some healing and some comfort — it has to be with somebody who is not part of all my world of past and pain,” she wrote. “I can find that now — if the world will let us.”

The letter was part of a set of papers found in locked red-leather cases discovered only last month in Wales at the family home of Mr. Ormsby Gore, who died in 1985. They are being auctioned in London next month by his grandson to help restore the house.

Continue reading the main story
RELATED COVERAGE


Annemarie Huste, a Kennedy Chef Who Said Too Much, Dies at 73 OCT. 22, 2016

‘Jacqueline Bouvier Kennedy Onassis,’ by Barbara Leaming DEC. 5, 2014

Jackie Kennedy’s Letters Taken Off the Auction Block MAY 23, 2014
The letters point to the depth of feeling behind the public mask of one of the most celebrated women of her time.

In [66]:
!wc nyt.txt

# Where 21 is the number of lines, 245 is the number of words, and 1515 is the number of characters.


      21     245    1515 nyt.txt

In [68]:
!wc -w nyt.txt


     245 nyt.txt

grep:

Grep is a pattern matching utility built into unix and it's flavors. The typical format is:

grep [option] [pattern] [file/s]


In [69]:
pwd


Out[69]:
'/Volumes/HDD3TB/05. Work/03. Udemy/03. Final/pyds/Data'

In [70]:
ls


Sopranos/                              gdp.txt
Times_Square_Entertainment_Venues.csv  location.json
bev-sales.xlsx                         nyc_rest.csv
cafe_sales.csv                         nyt.txt
cafe_sales2015.csv                     sample.txt
cafe_sales2015.txt                     tips.csv*
database.csv.zip                       venues.json
foo.txt                                wine/
gdp.csv

In [71]:
!cat nyt.txt


https://www.nytimes.com/2017/02/08/world/europe/letters-from-jacqueline-kennedy-to-the-man-she-didnt-marry.html

LONDON — In November 1967, four years after her husband’s assassination, Jacqueline Kennedy traveled to the temples of Angkor Wat in Cambodia on a much-publicized trip with David Ormsby Gore, a friend of her husband and himself a recent widower.

There was much speculation of a romantic attachment. A few months later, Mr. Ormsby Gore, a former British ambassador to Washington, proposed marriage. She turned him down.

In a handwritten letter, filled with anguish and a touch of cruelty, she explained her decision to marry Aristotle Onassis instead.

“If ever I can find some healing and some comfort — it has to be with somebody who is not part of all my world of past and pain,” she wrote. “I can find that now — if the world will let us.”

The letter was part of a set of papers found in locked red-leather cases discovered only last month in Wales at the family home of Mr. Ormsby Gore, who died in 1985. They are being auctioned in London next month by his grandson to help restore the house.

Continue reading the main story
RELATED COVERAGE


Annemarie Huste, a Kennedy Chef Who Said Too Much, Dies at 73 OCT. 22, 2016

‘Jacqueline Bouvier Kennedy Onassis,’ by Barbara Leaming DEC. 5, 2014

Jackie Kennedy’s Letters Taken Off the Auction Block MAY 23, 2014
The letters point to the depth of feeling behind the public mask of one of the most celebrated women of her time.

In [72]:
# Count the number of matches
!grep -c 'Kennedy' nyt.txt


4

In [73]:
!grep -o 'Kennedy' nyt.txt


Kennedy
Kennedy
Kennedy
Kennedy

More options for grep:

  • -c Print only a count of matched lines.
  • -l List only filenames
  • -i Ignore lowercase and uppercase distinctions
  • -o prints only the matching part of the line
  • -n Print matching line with its line number
  • -v Negate matches; print lines that do not match the regex
  • -r Recursively Search subdirectories listed

In [74]:
!curl -s 'http://freegeoip.net/json/' > location.json

In [75]:
!jq . location.json


{
  "ip": "176.227.198.122",
  "country_code": "GB",
  "country_name": "United Kingdom",
  "region_code": "",
  "region_name": "",
  "city": "",
  "zip_code": "",
  "time_zone": "Europe/London",
  "latitude": 51.4964,
  "longitude": -0.1224,
  "metro_code": 0
}

In [ ]:
!curl -s 'http://freegeoip.net/json/' | jq .

Redirection (or Downloading)

This is really useful to quickly download a dataset using what is called an API Endpoint.

Let's download the 'Times Square Entertainment Venues' dataset from New York City's Open Data Portal to demonstrate this.

https://data.cityofnewyork.us/Business/Times-Square-Entertainment-Venues/jxdc-hnze


In [76]:
!curl "https://data.cityofnewyork.us/resource/2pc8-n4xe.json" > venues.json


  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9794    0  9794    0     0   4282      0 --:--:--  0:00:02 --:--:--  4713

In [77]:
!cat venues.json


[{"company_name":"Cheetahs","phone":"(212) 819-9300","subindustry":"Nightclub","website":"www.cheetahnyc.com"}
,{"company_name":"Al Hirschfeld Theater","phone":"(212) 239-6200","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"The World Famous Laugh Factory","phone":"212-586-7829 x304","subindustry":"Comedy Club"}
,{"company_name":"Bernard B. Jacobs Theatre","phone":"(212) 239-2820","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Music Box Theater","phone":"212 239-6200","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Helen Hayes Theatre","phone":"(212) 944-9457","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Touch Nightclub","phone":"212 489-7656","subindustry":"Nightclub"}
,{"company_name":"Hawthorne Amusement","subindustry":"Attractions / Amusement"}
,{"company_name":"Regal Loews Cinemas 42nd Street E-Walk Theater","phone":"(212) 840-7761","subindustry":"Movies","website":"www.regmovies.com"}
,{"company_name":"Jujamcyn Theater","phone":"(212) 840-8181","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"St. James Theatre","phone":"(212) 239-6200","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Imperial Theatre","phone":"(815) 477-5076","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Brooks Atkinson Theatre","phone":"(212) 719-4099","sub_subindustry":"League","subindustry":"Theater","website":"http://www.brooksatkinsontheater.com/"}
,{"company_name":"Freidman Theatre","phone":"(212) 399-3000 x 5401","sub_subindustry":"Broadway - Non-League","subindustry":"Theater"}
,{"company_name":"Lyceum Theater","phone":"212 239-6200","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Ripley's Believe It Or Not!","phone":"212-398-3133","subindustry":"Attractions / Amusement","website":"www.ripleysnewyork.com"}
,{"company_name":"Shubert Theatre","phone":"2129443700","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"St Luke's Theater","phone":"(212) 246-8140","sub_subindustry":"Off-Broadway","subindustry":"Theater","website":"http://stlukestheatre.com/"}
,{"company_name":"Show World Center Inc.","phone":"(212) 489-7576","subindustry":"Nightclub"}
,{"company_name":"Lace 2","phone":"(212) 765-5047","subindustry":"Nightclub","website":"laceclubs.com"}
,{"company_name":"Flash Dancers","phone":"(212) 315-5107","subindustry":"Nightclub","website":"http://flashdancersnyc.com/"}
,{"company_name":"Snapple Theater (Jerry Orbach Theater)","phone":"(212) 246-3828","sub_subindustry":"Broadway - Non-League","subindustry":"Theater","website":"http://www.snappletheater.com"}
,{"company_name":"Don't Tell Mama","phone":"212-757-0788","subindustry":"Live Music"}
,{"company_name":"Private Storefront","subindustry":"Nightclub"}
,{"company_name":"Circle In The Square Theatre","phone":"2123070388","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Ha! Comedy Club","phone":"2129773884","subindustry":"Comedy Club","website":"www.hacomedynyc.com"}
,{"company_name":"American Theatre Wing","phone":"212-765-0606 x311","sub_subindustry":"Off-Broadway","subindustry":"Theater","website":"http://americantheatrewing.org/"}
,{"company_name":"Belasco","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Broadway Theatre","phone":"(212) 975-6644","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Nederlander Theatre","phone":"(212) 921-8000","sub_subindustry":"League","subindustry":"Theater","website":"http://www.nederlandertheatre.com/"}
,{"company_name":"New Amsterdam Theater","phone":"(212) 282-2900","sub_subindustry":"League","subindustry":"Theater","website":"http://www.newyorkcitytheatre.com/theaters/newamsterdamtheater/theater.php"}
,{"company_name":"Vacant (Bare Elegance Gentlemen's Club)","phone":"(212) 245-3494","subindustry":"Nightclub","website":"http://www.mentertainment.com/clubs/ny/bare/bare-00.htm"}
,{"company_name":"Mixed Emotions Video Rental","subindustry":"Movies"}
,{"company_name":"Walter Kerr Theatre","phone":"(212) 239-2974","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Longacre Theatre","phone":"(212) 239-6200","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Richard Rodgers Theatre","phone":"(212) 221-1211 \u000e","sub_subindustry":"League","subindustry":"Theater","website":"http://www.richardrodgerstheatre.com/"}
,{"company_name":"TSX: Times Square Expositions","phone":"646-619-8608","subindustry":"Museums / Galleries"}
,{"company_name":"Eugene O'Neill Theatre","phone":"(212) 840-8181","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"AMC Theatres","phone":"(212) 398-2597","subindustry":"Movies"}
,{"company_name":"Minskoff Theatre","phone":"(212) 869-0550","sub_subindustry":"League","subindustry":"Theater","website":"www.minskofftheatre.com"}
,{"company_name":"Liberty Theater","sub_subindustry":"Broadway - Non-League","subindustry":"Theater"}
,{"company_name":"Madame Tussauds New York","phone":"(212) 512-9600","subindustry":"Museums / Galleries","website":"http://www.nycwax.com"}
,{"company_name":"Nokia Theatre","phone":"(212) 930-1950","subindustry":"Live Music","website":"http://nokiatheatrenyc.com/"}
,{"company_name":"Ethel Barrymore","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Architectural History Foundation","phone":"(212) 286-5141","subindustry":"Museums / Galleries"}
,{"company_name":"Samuel J. Friedman Theatre","location_1_location":"261 47th St","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Henry Miller's Theatre","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Hollywood Theatre","phone":"(212) 262-3030","subindustry":"Movies"}
,{"company_name":"Town Hall","phone":"(212) 997-1003","sub_subindustry":"Nonprofit","subindustry":"Theater"}
,{"company_name":"The Cort Theatre","phone":"(212) 239-2820","sub_subindustry":"League","subindustry":"Theater","website":"http://www.newyorkcitytheatre.com/theaters/corttheater/theater.php"}
,{"company_name":"The Jam","phone":"2126268472","subindustry":"Live Music","website":"www.thejamnyc.com"}
,{"company_name":"Sage Theater","phone":"(212) 302-6665 \u000e","sub_subindustry":"Off-Broadway","subindustry":"Theater","website":"http://www.sagetheater.us/"}
,{"company_name":"Lace","phone":"(212) 764-6969","subindustry":"Nightclub"}
,{"company_name":"Times Square Art Center","phone":"(212) 586-7829 \u000e","subindustry":"Comedy Club","website":"http://timessquareartscenter.com/"}
,{"company_name":"China Club","phone":"(212) 575-5026","subindustry":"Nightclub","website":"http://www.chinaclubnyc.com/"}
,{"company_name":"Majestic Theatre","phone":"(212) 239-6200","sub_subindustry":"League","subindustry":"Theater","website":"http://www.newyorkcitytheatre.com/theaters/majestictheater/theater.php"}
,{"company_name":"The Times Center","phone":"(800) 272-9533","subindustry":"Lecture Hall","website":"http://thetimescenter.com/"}
,{"company_name":"Lunt-Fontanne Theatre","phone":"(212) 575-9200","sub_subindustry":"League","subindustry":"Theater","website":"http://luntfontannetheatre.com/"}
,{"company_name":"John Golden Theatre","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Gerald Schoenfeld Theatre","phone":"(212) 239-6200","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Neil Simon Theatre","phone":"(212) 757-8646","sub_subindustry":"League","subindustry":"Theater","website":"http://www.neilsimontheatre.com/"}
,{"company_name":"American Airlines Theatre","phone":"(212) 719-1300 \u000e","sub_subindustry":"League","subindustry":"Theater","website":"http://roundabouttheatre.org/"}
,{"company_name":"Roseland Ballroom","phone":"(212) 247-0200 \u000e","subindustry":"Live Music","website":"http://roselandballroom.com/"}
,{"company_name":"Winter Garden Theatre","phone":"(212) 239-6200","sub_subindustry":"League","subindustry":"Theater","website":"http://www.wintergarden-theater.com/"}
,{"company_name":"Gershwin Theatre","phone":"(212) 586-6510","sub_subindustry":"League","subindustry":"Theater","website":"http://www.gershwintheatre.com/"}
,{"company_name":"Booth Theatre","phone":"(212) 391-8886","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Palace Theatre","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Duke Theater","phone":"(646) 223-3000","sub_subindustry":"Off-Broadway","subindustry":"Theater","website":"http://www.new42.org/duke/duke_home.aspx"}
,{"company_name":"Marriott Marquis Theatre","phone":"(212) 382-0100","sub_subindustry":"League","subindustry":"Theater","website":"http://www.marquistheatre.com/"}
,{"company_name":"Ambassador Theatre","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"American Craftsman","phone":"(212) 399-2555","subindustry":"Museums / Galleries"}
,{"company_name":"August Wilson Theatre","phone":"(212) 239-6200","sub_subindustry":"League","subindustry":"Theater","website":"http://www.newyorkcitytheatre.com/theaters/augustwilsontheater/theater.php"}
,{"company_name":"Broadway Galleries","phone":"(212) 274-8993","subindustry":"Museums / Galleries","website":"http://www.broadwaygallerynyc.com/"}
,{"company_name":"Crown Art Gallery","phone":"(212) 757-8255","subindustry":"Museums / Galleries","website":"www.sohogallery.com"}
,{"company_name":"Foxwoods Theatre","phone":"(212) 556-4750","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"The Duke","sub_subindustry":"Off-Broadway","subindustry":"Theater","website":"http://www.new42.org/duke/duke_home.aspx"}
,{"company_name":"Carolines on Broadway","phone":"(212) 757-4100","subindustry":"Comedy Club","website":"https://www.carolines.com/"}]

In [78]:
!grep 'Ripley' venues.json


,{"company_name":"Ripley's Believe It Or Not!","phone":"212-398-3133","subindustry":"Attractions / Amusement","website":"www.ripleysnewyork.com"}

In [79]:
!grep -i 'Theater' venues.json


,{"company_name":"Al Hirschfeld Theater","phone":"(212) 239-6200","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Bernard B. Jacobs Theatre","phone":"(212) 239-2820","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Music Box Theater","phone":"212 239-6200","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Helen Hayes Theatre","phone":"(212) 944-9457","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Regal Loews Cinemas 42nd Street E-Walk Theater","phone":"(212) 840-7761","subindustry":"Movies","website":"www.regmovies.com"}
,{"company_name":"Jujamcyn Theater","phone":"(212) 840-8181","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"St. James Theatre","phone":"(212) 239-6200","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Imperial Theatre","phone":"(815) 477-5076","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Brooks Atkinson Theatre","phone":"(212) 719-4099","sub_subindustry":"League","subindustry":"Theater","website":"http://www.brooksatkinsontheater.com/"}
,{"company_name":"Freidman Theatre","phone":"(212) 399-3000 x 5401","sub_subindustry":"Broadway - Non-League","subindustry":"Theater"}
,{"company_name":"Lyceum Theater","phone":"212 239-6200","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Shubert Theatre","phone":"2129443700","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"St Luke's Theater","phone":"(212) 246-8140","sub_subindustry":"Off-Broadway","subindustry":"Theater","website":"http://stlukestheatre.com/"}
,{"company_name":"Snapple Theater (Jerry Orbach Theater)","phone":"(212) 246-3828","sub_subindustry":"Broadway - Non-League","subindustry":"Theater","website":"http://www.snappletheater.com"}
,{"company_name":"Circle In The Square Theatre","phone":"2123070388","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"American Theatre Wing","phone":"212-765-0606 x311","sub_subindustry":"Off-Broadway","subindustry":"Theater","website":"http://americantheatrewing.org/"}
,{"company_name":"Belasco","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Broadway Theatre","phone":"(212) 975-6644","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Nederlander Theatre","phone":"(212) 921-8000","sub_subindustry":"League","subindustry":"Theater","website":"http://www.nederlandertheatre.com/"}
,{"company_name":"New Amsterdam Theater","phone":"(212) 282-2900","sub_subindustry":"League","subindustry":"Theater","website":"http://www.newyorkcitytheatre.com/theaters/newamsterdamtheater/theater.php"}
,{"company_name":"Walter Kerr Theatre","phone":"(212) 239-2974","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Longacre Theatre","phone":"(212) 239-6200","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Richard Rodgers Theatre","phone":"(212) 221-1211 \u000e","sub_subindustry":"League","subindustry":"Theater","website":"http://www.richardrodgerstheatre.com/"}
,{"company_name":"Eugene O'Neill Theatre","phone":"(212) 840-8181","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Minskoff Theatre","phone":"(212) 869-0550","sub_subindustry":"League","subindustry":"Theater","website":"www.minskofftheatre.com"}
,{"company_name":"Liberty Theater","sub_subindustry":"Broadway - Non-League","subindustry":"Theater"}
,{"company_name":"Ethel Barrymore","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Samuel J. Friedman Theatre","location_1_location":"261 47th St","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Henry Miller's Theatre","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Town Hall","phone":"(212) 997-1003","sub_subindustry":"Nonprofit","subindustry":"Theater"}
,{"company_name":"The Cort Theatre","phone":"(212) 239-2820","sub_subindustry":"League","subindustry":"Theater","website":"http://www.newyorkcitytheatre.com/theaters/corttheater/theater.php"}
,{"company_name":"Sage Theater","phone":"(212) 302-6665 \u000e","sub_subindustry":"Off-Broadway","subindustry":"Theater","website":"http://www.sagetheater.us/"}
,{"company_name":"Majestic Theatre","phone":"(212) 239-6200","sub_subindustry":"League","subindustry":"Theater","website":"http://www.newyorkcitytheatre.com/theaters/majestictheater/theater.php"}
,{"company_name":"Lunt-Fontanne Theatre","phone":"(212) 575-9200","sub_subindustry":"League","subindustry":"Theater","website":"http://luntfontannetheatre.com/"}
,{"company_name":"John Golden Theatre","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Gerald Schoenfeld Theatre","phone":"(212) 239-6200","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Neil Simon Theatre","phone":"(212) 757-8646","sub_subindustry":"League","subindustry":"Theater","website":"http://www.neilsimontheatre.com/"}
,{"company_name":"American Airlines Theatre","phone":"(212) 719-1300 \u000e","sub_subindustry":"League","subindustry":"Theater","website":"http://roundabouttheatre.org/"}
,{"company_name":"Winter Garden Theatre","phone":"(212) 239-6200","sub_subindustry":"League","subindustry":"Theater","website":"http://www.wintergarden-theater.com/"}
,{"company_name":"Gershwin Theatre","phone":"(212) 586-6510","sub_subindustry":"League","subindustry":"Theater","website":"http://www.gershwintheatre.com/"}
,{"company_name":"Booth Theatre","phone":"(212) 391-8886","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Palace Theatre","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"Duke Theater","phone":"(646) 223-3000","sub_subindustry":"Off-Broadway","subindustry":"Theater","website":"http://www.new42.org/duke/duke_home.aspx"}
,{"company_name":"Marriott Marquis Theatre","phone":"(212) 382-0100","sub_subindustry":"League","subindustry":"Theater","website":"http://www.marquistheatre.com/"}
,{"company_name":"Ambassador Theatre","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"August Wilson Theatre","phone":"(212) 239-6200","sub_subindustry":"League","subindustry":"Theater","website":"http://www.newyorkcitytheatre.com/theaters/augustwilsontheater/theater.php"}
,{"company_name":"Foxwoods Theatre","phone":"(212) 556-4750","sub_subindustry":"League","subindustry":"Theater"}
,{"company_name":"The Duke","sub_subindustry":"Off-Broadway","subindustry":"Theater","website":"http://www.new42.org/duke/duke_home.aspx"}

In [80]:
# Multiple flags, and multiple conditions
!grep -v -e 'Theater' -e 'Theatre' venues.json


[{"company_name":"Cheetahs","phone":"(212) 819-9300","subindustry":"Nightclub","website":"www.cheetahnyc.com"}
,{"company_name":"The World Famous Laugh Factory","phone":"212-586-7829 x304","subindustry":"Comedy Club"}
,{"company_name":"Touch Nightclub","phone":"212 489-7656","subindustry":"Nightclub"}
,{"company_name":"Hawthorne Amusement","subindustry":"Attractions / Amusement"}
,{"company_name":"Ripley's Believe It Or Not!","phone":"212-398-3133","subindustry":"Attractions / Amusement","website":"www.ripleysnewyork.com"}
,{"company_name":"Show World Center Inc.","phone":"(212) 489-7576","subindustry":"Nightclub"}
,{"company_name":"Lace 2","phone":"(212) 765-5047","subindustry":"Nightclub","website":"laceclubs.com"}
,{"company_name":"Flash Dancers","phone":"(212) 315-5107","subindustry":"Nightclub","website":"http://flashdancersnyc.com/"}
,{"company_name":"Don't Tell Mama","phone":"212-757-0788","subindustry":"Live Music"}
,{"company_name":"Private Storefront","subindustry":"Nightclub"}
,{"company_name":"Ha! Comedy Club","phone":"2129773884","subindustry":"Comedy Club","website":"www.hacomedynyc.com"}
,{"company_name":"Vacant (Bare Elegance Gentlemen's Club)","phone":"(212) 245-3494","subindustry":"Nightclub","website":"http://www.mentertainment.com/clubs/ny/bare/bare-00.htm"}
,{"company_name":"Mixed Emotions Video Rental","subindustry":"Movies"}
,{"company_name":"TSX: Times Square Expositions","phone":"646-619-8608","subindustry":"Museums / Galleries"}
,{"company_name":"Madame Tussauds New York","phone":"(212) 512-9600","subindustry":"Museums / Galleries","website":"http://www.nycwax.com"}
,{"company_name":"Architectural History Foundation","phone":"(212) 286-5141","subindustry":"Museums / Galleries"}
,{"company_name":"The Jam","phone":"2126268472","subindustry":"Live Music","website":"www.thejamnyc.com"}
,{"company_name":"Lace","phone":"(212) 764-6969","subindustry":"Nightclub"}
,{"company_name":"Times Square Art Center","phone":"(212) 586-7829 \u000e","subindustry":"Comedy Club","website":"http://timessquareartscenter.com/"}
,{"company_name":"China Club","phone":"(212) 575-5026","subindustry":"Nightclub","website":"http://www.chinaclubnyc.com/"}
,{"company_name":"The Times Center","phone":"(800) 272-9533","subindustry":"Lecture Hall","website":"http://thetimescenter.com/"}
,{"company_name":"Roseland Ballroom","phone":"(212) 247-0200 \u000e","subindustry":"Live Music","website":"http://roselandballroom.com/"}
,{"company_name":"American Craftsman","phone":"(212) 399-2555","subindustry":"Museums / Galleries"}
,{"company_name":"Broadway Galleries","phone":"(212) 274-8993","subindustry":"Museums / Galleries","website":"http://www.broadwaygallerynyc.com/"}
,{"company_name":"Crown Art Gallery","phone":"(212) 757-8255","subindustry":"Museums / Galleries","website":"www.sohogallery.com"}
,{"company_name":"Carolines on Broadway","phone":"(212) 757-4100","subindustry":"Comedy Club","website":"https://www.carolines.com/"}]

In [ ]: