Goal 1

Injest file as pure text


In [279]:
MovieTextFile = open("tmdb_5000_movies.csv")
# for line in MovieTextFile:
#    print(line) # not quite right
# type(MovieTextFile)

Import data as a list of lines


In [180]:
import csv
with open("tmdb_5000_movies.csv",encoding="utf8") as f:
    reader = csv.reader(f)
    MovieList = list(reader)
MovieList[:5]


Out[180]:
[['budget',
  'genres',
  'homepage',
  'id',
  'keywords',
  'original_language',
  'original_title',
  'overview',
  'popularity',
  'production_companies',
  'production_countries',
  'release_date',
  'revenue',
  'runtime',
  'spoken_languages',
  'status',
  'tagline',
  'title',
  'vote_average',
  'vote_count'],
 ['237000000',
  '[{"id": 28, "name": "Action"}, {"id": 12, "name": "Adventure"}, {"id": 14, "name": "Fantasy"}, {"id": 878, "name": "Science Fiction"}]',
  'http://www.avatarmovie.com/',
  '19995',
  '[{"id": 1463, "name": "culture clash"}, {"id": 2964, "name": "future"}, {"id": 3386, "name": "space war"}, {"id": 3388, "name": "space colony"}, {"id": 3679, "name": "society"}, {"id": 3801, "name": "space travel"}, {"id": 9685, "name": "futuristic"}, {"id": 9840, "name": "romance"}, {"id": 9882, "name": "space"}, {"id": 9951, "name": "alien"}, {"id": 10148, "name": "tribe"}, {"id": 10158, "name": "alien planet"}, {"id": 10987, "name": "cgi"}, {"id": 11399, "name": "marine"}, {"id": 13065, "name": "soldier"}, {"id": 14643, "name": "battle"}, {"id": 14720, "name": "love affair"}, {"id": 165431, "name": "anti war"}, {"id": 193554, "name": "power relations"}, {"id": 206690, "name": "mind and soul"}, {"id": 209714, "name": "3d"}]',
  'en',
  'Avatar',
  'In the 22nd century, a paraplegic Marine is dispatched to the moon Pandora on a unique mission, but becomes torn between following orders and protecting an alien civilization.',
  '150.437577',
  '[{"name": "Ingenious Film Partners", "id": 289}, {"name": "Twentieth Century Fox Film Corporation", "id": 306}, {"name": "Dune Entertainment", "id": 444}, {"name": "Lightstorm Entertainment", "id": 574}]',
  '[{"iso_3166_1": "US", "name": "United States of America"}, {"iso_3166_1": "GB", "name": "United Kingdom"}]',
  '2009-12-10',
  '2787965087',
  '162',
  '[{"iso_639_1": "en", "name": "English"}, {"iso_639_1": "es", "name": "Espa\\u00f1ol"}]',
  'Released',
  'Enter the World of Pandora.',
  'Avatar',
  '7.2',
  '11800'],
 ['300000000',
  '[{"id": 12, "name": "Adventure"}, {"id": 14, "name": "Fantasy"}, {"id": 28, "name": "Action"}]',
  'http://disney.go.com/disneypictures/pirates/',
  '285',
  '[{"id": 270, "name": "ocean"}, {"id": 726, "name": "drug abuse"}, {"id": 911, "name": "exotic island"}, {"id": 1319, "name": "east india trading company"}, {"id": 2038, "name": "love of one\'s life"}, {"id": 2052, "name": "traitor"}, {"id": 2580, "name": "shipwreck"}, {"id": 2660, "name": "strong woman"}, {"id": 3799, "name": "ship"}, {"id": 5740, "name": "alliance"}, {"id": 5941, "name": "calypso"}, {"id": 6155, "name": "afterlife"}, {"id": 6211, "name": "fighter"}, {"id": 12988, "name": "pirate"}, {"id": 157186, "name": "swashbuckler"}, {"id": 179430, "name": "aftercreditsstinger"}]',
  'en',
  "Pirates of the Caribbean: At World's End",
  'Captain Barbossa, long believed to be dead, has come back to life and is headed to the edge of the Earth with Will Turner and Elizabeth Swann. But nothing is quite as it seems.',
  '139.082615',
  '[{"name": "Walt Disney Pictures", "id": 2}, {"name": "Jerry Bruckheimer Films", "id": 130}, {"name": "Second Mate Productions", "id": 19936}]',
  '[{"iso_3166_1": "US", "name": "United States of America"}]',
  '2007-05-19',
  '961000000',
  '169',
  '[{"iso_639_1": "en", "name": "English"}]',
  'Released',
  'At the end of the world, the adventure begins.',
  "Pirates of the Caribbean: At World's End",
  '6.9',
  '4500'],
 ['245000000',
  '[{"id": 28, "name": "Action"}, {"id": 12, "name": "Adventure"}, {"id": 80, "name": "Crime"}]',
  'http://www.sonypictures.com/movies/spectre/',
  '206647',
  '[{"id": 470, "name": "spy"}, {"id": 818, "name": "based on novel"}, {"id": 4289, "name": "secret agent"}, {"id": 9663, "name": "sequel"}, {"id": 14555, "name": "mi6"}, {"id": 156095, "name": "british secret service"}, {"id": 158431, "name": "united kingdom"}]',
  'en',
  'Spectre',
  'A cryptic message from Bond’s past sends him on a trail to uncover a sinister organization. While M battles political forces to keep the secret service alive, Bond peels back the layers of deceit to reveal the terrible truth behind SPECTRE.',
  '107.376788',
  '[{"name": "Columbia Pictures", "id": 5}, {"name": "Danjaq", "id": 10761}, {"name": "B24", "id": 69434}]',
  '[{"iso_3166_1": "GB", "name": "United Kingdom"}, {"iso_3166_1": "US", "name": "United States of America"}]',
  '2015-10-26',
  '880674609',
  '148',
  '[{"iso_639_1": "fr", "name": "Fran\\u00e7ais"}, {"iso_639_1": "en", "name": "English"}, {"iso_639_1": "es", "name": "Espa\\u00f1ol"}, {"iso_639_1": "it", "name": "Italiano"}, {"iso_639_1": "de", "name": "Deutsch"}]',
  'Released',
  'A Plan No One Escapes',
  'Spectre',
  '6.3',
  '4466'],
 ['250000000',
  '[{"id": 28, "name": "Action"}, {"id": 80, "name": "Crime"}, {"id": 18, "name": "Drama"}, {"id": 53, "name": "Thriller"}]',
  'http://www.thedarkknightrises.com/',
  '49026',
  '[{"id": 849, "name": "dc comics"}, {"id": 853, "name": "crime fighter"}, {"id": 949, "name": "terrorist"}, {"id": 1308, "name": "secret identity"}, {"id": 1437, "name": "burglar"}, {"id": 3051, "name": "hostage drama"}, {"id": 3562, "name": "time bomb"}, {"id": 6969, "name": "gotham city"}, {"id": 7002, "name": "vigilante"}, {"id": 9665, "name": "cover-up"}, {"id": 9715, "name": "superhero"}, {"id": 9990, "name": "villainess"}, {"id": 10044, "name": "tragic hero"}, {"id": 13015, "name": "terrorism"}, {"id": 14796, "name": "destruction"}, {"id": 18933, "name": "catwoman"}, {"id": 156082, "name": "cat burglar"}, {"id": 156395, "name": "imax"}, {"id": 173272, "name": "flood"}, {"id": 179093, "name": "criminal underworld"}, {"id": 230775, "name": "batman"}]',
  'en',
  'The Dark Knight Rises',
  "Following the death of District Attorney Harvey Dent, Batman assumes responsibility for Dent's crimes to protect the late attorney's reputation and is subsequently hunted by the Gotham City Police Department. Eight years later, Batman encounters the mysterious Selina Kyle and the villainous Bane, a new terrorist leader who overwhelms Gotham's finest. The Dark Knight resurfaces to protect a city that has branded him an enemy.",
  '112.31295',
  '[{"name": "Legendary Pictures", "id": 923}, {"name": "Warner Bros.", "id": 6194}, {"name": "DC Entertainment", "id": 9993}, {"name": "Syncopy", "id": 9996}]',
  '[{"iso_3166_1": "US", "name": "United States of America"}]',
  '2012-07-16',
  '1084939099',
  '165',
  '[{"iso_639_1": "en", "name": "English"}]',
  'Released',
  'The Legend Ends',
  'The Dark Knight Rises',
  '7.6',
  '9106']]

Import data as a data frame


In [173]:
import pandas as pd
movies = pd.read_csv("tmdb_5000_movies.csv")
movieFrame = pd.DataFrame(movies)
movieFrame[:5]


Out[173]:
budget genres homepage id keywords original_language original_title overview popularity production_companies production_countries release_date revenue runtime spoken_languages status tagline title vote_average vote_count
0 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... [{"iso_3166_1": "US", "name": "United States o... 2009-12-10 2787965087 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800
1 300000000 [{"id": 12, "name": "Adventure"}, {"id": 14, "... http://disney.go.com/disneypictures/pirates/ 285 [{"id": 270, "name": "ocean"}, {"id": 726, "na... en Pirates of the Caribbean: At World's End Captain Barbossa, long believed to be dead, ha... 139.082615 [{"name": "Walt Disney Pictures", "id": 2}, {"... [{"iso_3166_1": "US", "name": "United States o... 2007-05-19 961000000 169.0 [{"iso_639_1": "en", "name": "English"}] Released At the end of the world, the adventure begins. Pirates of the Caribbean: At World's End 6.9 4500
2 245000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.sonypictures.com/movies/spectre/ 206647 [{"id": 470, "name": "spy"}, {"id": 818, "name... en Spectre A cryptic message from Bond’s past sends him o... 107.376788 [{"name": "Columbia Pictures", "id": 5}, {"nam... [{"iso_3166_1": "GB", "name": "United Kingdom"... 2015-10-26 880674609 148.0 [{"iso_639_1": "fr", "name": "Fran\u00e7ais"},... Released A Plan No One Escapes Spectre 6.3 4466
3 250000000 [{"id": 28, "name": "Action"}, {"id": 80, "nam... http://www.thedarkknightrises.com/ 49026 [{"id": 849, "name": "dc comics"}, {"id": 853,... en The Dark Knight Rises Following the death of District Attorney Harve... 112.312950 [{"name": "Legendary Pictures", "id": 923}, {"... [{"iso_3166_1": "US", "name": "United States o... 2012-07-16 1084939099 165.0 [{"iso_639_1": "en", "name": "English"}] Released The Legend Ends The Dark Knight Rises 7.6 9106
4 260000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://movies.disney.com/john-carter 49529 [{"id": 818, "name": "based on novel"}, {"id":... en John Carter John Carter is a war-weary, former military ca... 43.926995 [{"name": "Walt Disney Pictures", "id": 2}] [{"iso_3166_1": "US", "name": "United States o... 2012-03-07 284139100 132.0 [{"iso_639_1": "en", "name": "English"}] Released Lost in our world, found in another. John Carter 6.1 2124

Goal 2


In [174]:
#pull out genres array of JSON strings from data frame

genres = movieFrame['genres']
# genresFrame = pd.DataFrame(genres)
genres[:5]


Out[174]:
0    [{"id": 28, "name": "Action"}, {"id": 12, "nam...
1    [{"id": 12, "name": "Adventure"}, {"id": 14, "...
2    [{"id": 28, "name": "Action"}, {"id": 12, "nam...
3    [{"id": 28, "name": "Action"}, {"id": 80, "nam...
4    [{"id": 28, "name": "Action"}, {"id": 12, "nam...
Name: genres, dtype: object

In [175]:
# Pull out list of names for each row of the data frame
# Start with testing first row and iterating through JSON string

import json
genreList = []
genre = json.loads(genres[0])
for i,val in enumerate(genre):
    genreList.append(genre[i]['name'])
genreList


Out[175]:
['Action', 'Adventure', 'Fantasy', 'Science Fiction']

In [197]:
# Iterate through indices of genre array to create a list of lists of genre names

import json
genresAll = []
for k,x in enumerate(genres):
    genreList = []
    genre = json.loads(genres[k])
    for i,val in enumerate(genre):
        genreList.append(genre[i]['name'])
    genresAll.append(genreList)
genresAll[:10]


Out[197]:
[['Action', 'Adventure', 'Fantasy', 'Science Fiction'],
 ['Adventure', 'Fantasy', 'Action'],
 ['Action', 'Adventure', 'Crime'],
 ['Action', 'Crime', 'Drama', 'Thriller'],
 ['Action', 'Adventure', 'Science Fiction'],
 ['Fantasy', 'Action', 'Adventure'],
 ['Animation', 'Family'],
 ['Action', 'Adventure', 'Science Fiction'],
 ['Adventure', 'Fantasy', 'Family'],
 ['Action', 'Adventure', 'Fantasy']]

In [241]:
genreSeries['W'] = pd.Series(genresAll,index=movieFrame.index)
genreFrame = pd.DataFrame(genreSeries['W'],columns=['GenreList'])
genreFrame[:5]


Out[241]:
GenreList
0 [Action, Adventure, Fantasy, Science Fiction]
1 [Adventure, Fantasy, Action]
2 [Action, Adventure, Crime]
3 [Action, Crime, Drama, Thriller]
4 [Action, Adventure, Science Fiction]

In [244]:
genreDummies = genreFrame.GenreList.astype(str).str.strip('[]').str.get_dummies(', ')
genreDummies[:10]


Out[244]:
'Action' 'Adventure' 'Animation' 'Comedy' 'Crime' 'Documentary' 'Drama' 'Family' 'Fantasy' 'Foreign' 'History' 'Horror' 'Music' 'Mystery' 'Romance' 'Science Fiction' 'TV Movie' 'Thriller' 'War' 'Western'
0 1 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0
1 1 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0
2 1 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
3 1 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0
4 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0
5 1 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0
6 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0
7 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0
8 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0
9 1 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0

In [248]:
# append lists as a column at end of dataframe
movieGenreFrame = pd.merge(movieFrame,genreFrame,how='inner',left_index=True, right_index=True)
movieGenreFrame[:5]


Out[248]:
budget genres homepage id keywords original_language original_title overview popularity production_companies ... release_date revenue runtime spoken_languages status tagline title vote_average vote_count GenreList
0 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 2009-12-10 2787965087 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction]
1 300000000 [{"id": 12, "name": "Adventure"}, {"id": 14, "... http://disney.go.com/disneypictures/pirates/ 285 [{"id": 270, "name": "ocean"}, {"id": 726, "na... en Pirates of the Caribbean: At World's End Captain Barbossa, long believed to be dead, ha... 139.082615 [{"name": "Walt Disney Pictures", "id": 2}, {"... ... 2007-05-19 961000000 169.0 [{"iso_639_1": "en", "name": "English"}] Released At the end of the world, the adventure begins. Pirates of the Caribbean: At World's End 6.9 4500 [Adventure, Fantasy, Action]
2 245000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.sonypictures.com/movies/spectre/ 206647 [{"id": 470, "name": "spy"}, {"id": 818, "name... en Spectre A cryptic message from Bond’s past sends him o... 107.376788 [{"name": "Columbia Pictures", "id": 5}, {"nam... ... 2015-10-26 880674609 148.0 [{"iso_639_1": "fr", "name": "Fran\u00e7ais"},... Released A Plan No One Escapes Spectre 6.3 4466 [Action, Adventure, Crime]
3 250000000 [{"id": 28, "name": "Action"}, {"id": 80, "nam... http://www.thedarkknightrises.com/ 49026 [{"id": 849, "name": "dc comics"}, {"id": 853,... en The Dark Knight Rises Following the death of District Attorney Harve... 112.312950 [{"name": "Legendary Pictures", "id": 923}, {"... ... 2012-07-16 1084939099 165.0 [{"iso_639_1": "en", "name": "English"}] Released The Legend Ends The Dark Knight Rises 7.6 9106 [Action, Crime, Drama, Thriller]
4 260000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://movies.disney.com/john-carter 49529 [{"id": 818, "name": "based on novel"}, {"id":... en John Carter John Carter is a war-weary, former military ca... 43.926995 [{"name": "Walt Disney Pictures", "id": 2}] ... 2012-03-07 284139100 132.0 [{"iso_639_1": "en", "name": "English"}] Released Lost in our world, found in another. John Carter 6.1 2124 [Action, Adventure, Science Fiction]

5 rows × 21 columns


In [249]:
wideMovieFrame = pd.merge(movieGenreFrame,genreDummies,how='inner',left_index=True,right_index=True)
wideMovieFrame[:5]


Out[249]:
budget genres homepage id keywords original_language original_title overview popularity production_companies ... 'History' 'Horror' 'Music' 'Mystery' 'Romance' 'Science Fiction' 'TV Movie' 'Thriller' 'War' 'Western'
0 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 0 0 0 0 0 1 0 0 0 0
1 300000000 [{"id": 12, "name": "Adventure"}, {"id": 14, "... http://disney.go.com/disneypictures/pirates/ 285 [{"id": 270, "name": "ocean"}, {"id": 726, "na... en Pirates of the Caribbean: At World's End Captain Barbossa, long believed to be dead, ha... 139.082615 [{"name": "Walt Disney Pictures", "id": 2}, {"... ... 0 0 0 0 0 0 0 0 0 0
2 245000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.sonypictures.com/movies/spectre/ 206647 [{"id": 470, "name": "spy"}, {"id": 818, "name... en Spectre A cryptic message from Bond’s past sends him o... 107.376788 [{"name": "Columbia Pictures", "id": 5}, {"nam... ... 0 0 0 0 0 0 0 0 0 0
3 250000000 [{"id": 28, "name": "Action"}, {"id": 80, "nam... http://www.thedarkknightrises.com/ 49026 [{"id": 849, "name": "dc comics"}, {"id": 853,... en The Dark Knight Rises Following the death of District Attorney Harve... 112.312950 [{"name": "Legendary Pictures", "id": 923}, {"... ... 0 0 0 0 0 0 0 1 0 0
4 260000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://movies.disney.com/john-carter 49529 [{"id": 818, "name": "based on novel"}, {"id":... en John Carter John Carter is a war-weary, former military ca... 43.926995 [{"name": "Walt Disney Pictures", "id": 2}] ... 0 0 0 0 0 1 0 0 0 0

5 rows × 41 columns

Goal 3


In [276]:
longMovieFrame = pd.melt(wideMovieFrame, id_vars=movieGenreFrame.columns, value_vars=genreDummies.columns, 
                         var_name='Genre',value_name="genre_present")
longMovieFrame[:10]


Out[276]:
budget genres homepage id keywords original_language original_title overview popularity production_companies ... runtime spoken_languages status tagline title vote_average vote_count GenreList Genre genre_present
0 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Action' 1
1 300000000 [{"id": 12, "name": "Adventure"}, {"id": 14, "... http://disney.go.com/disneypictures/pirates/ 285 [{"id": 270, "name": "ocean"}, {"id": 726, "na... en Pirates of the Caribbean: At World's End Captain Barbossa, long believed to be dead, ha... 139.082615 [{"name": "Walt Disney Pictures", "id": 2}, {"... ... 169.0 [{"iso_639_1": "en", "name": "English"}] Released At the end of the world, the adventure begins. Pirates of the Caribbean: At World's End 6.9 4500 [Adventure, Fantasy, Action] 'Action' 1
2 245000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.sonypictures.com/movies/spectre/ 206647 [{"id": 470, "name": "spy"}, {"id": 818, "name... en Spectre A cryptic message from Bond’s past sends him o... 107.376788 [{"name": "Columbia Pictures", "id": 5}, {"nam... ... 148.0 [{"iso_639_1": "fr", "name": "Fran\u00e7ais"},... Released A Plan No One Escapes Spectre 6.3 4466 [Action, Adventure, Crime] 'Action' 1
3 250000000 [{"id": 28, "name": "Action"}, {"id": 80, "nam... http://www.thedarkknightrises.com/ 49026 [{"id": 849, "name": "dc comics"}, {"id": 853,... en The Dark Knight Rises Following the death of District Attorney Harve... 112.312950 [{"name": "Legendary Pictures", "id": 923}, {"... ... 165.0 [{"iso_639_1": "en", "name": "English"}] Released The Legend Ends The Dark Knight Rises 7.6 9106 [Action, Crime, Drama, Thriller] 'Action' 1
4 260000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://movies.disney.com/john-carter 49529 [{"id": 818, "name": "based on novel"}, {"id":... en John Carter John Carter is a war-weary, former military ca... 43.926995 [{"name": "Walt Disney Pictures", "id": 2}] ... 132.0 [{"iso_639_1": "en", "name": "English"}] Released Lost in our world, found in another. John Carter 6.1 2124 [Action, Adventure, Science Fiction] 'Action' 1
5 258000000 [{"id": 14, "name": "Fantasy"}, {"id": 28, "na... http://www.sonypictures.com/movies/spider-man3/ 559 [{"id": 851, "name": "dual identity"}, {"id": ... en Spider-Man 3 The seemingly invincible Spider-Man goes up ag... 115.699814 [{"name": "Columbia Pictures", "id": 5}, {"nam... ... 139.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released The battle within. Spider-Man 3 5.9 3576 [Fantasy, Action, Adventure] 'Action' 1
6 260000000 [{"id": 16, "name": "Animation"}, {"id": 10751... http://disney.go.com/disneypictures/tangled/ 38757 [{"id": 1562, "name": "hostage"}, {"id": 2343,... en Tangled When the kingdom's most wanted-and most charmi... 48.681969 [{"name": "Walt Disney Pictures", "id": 2}, {"... ... 100.0 [{"iso_639_1": "en", "name": "English"}] Released They're taking adventure to new lengths. Tangled 7.4 3330 [Animation, Family] 'Action' 0
7 280000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://marvel.com/movies/movie/193/avengers_ag... 99861 [{"id": 8828, "name": "marvel comic"}, {"id": ... en Avengers: Age of Ultron When Tony Stark tries to jumpstart a dormant p... 134.279229 [{"name": "Marvel Studios", "id": 420}, {"name... ... 141.0 [{"iso_639_1": "en", "name": "English"}] Released A New Age Has Come. Avengers: Age of Ultron 7.3 6767 [Action, Adventure, Science Fiction] 'Action' 1
8 250000000 [{"id": 12, "name": "Adventure"}, {"id": 14, "... http://harrypotter.warnerbros.com/harrypottera... 767 [{"id": 616, "name": "witch"}, {"id": 2343, "n... en Harry Potter and the Half-Blood Prince As Harry begins his sixth year at Hogwarts, he... 98.885637 [{"name": "Warner Bros.", "id": 6194}, {"name"... ... 153.0 [{"iso_639_1": "en", "name": "English"}] Released Dark Secrets Revealed Harry Potter and the Half-Blood Prince 7.4 5293 [Adventure, Fantasy, Family] 'Action' 0
9 250000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.batmanvsupermandawnofjustice.com/ 209112 [{"id": 849, "name": "dc comics"}, {"id": 7002... en Batman v Superman: Dawn of Justice Fearing the actions of a god-like Super Hero l... 155.790452 [{"name": "DC Comics", "id": 429}, {"name": "A... ... 151.0 [{"iso_639_1": "en", "name": "English"}] Released Justice or revenge Batman v Superman: Dawn of Justice 5.7 7004 [Action, Adventure, Fantasy] 'Action' 1

10 rows × 23 columns


In [277]:
# test results with 'Avatar' example
longMovieFrame[longMovieFrame['title']=='Avatar']


Out[277]:
budget genres homepage id keywords original_language original_title overview popularity production_companies ... runtime spoken_languages status tagline title vote_average vote_count GenreList Genre genre_present
0 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Action' 1
4803 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Adventure' 1
9606 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Animation' 0
14409 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Comedy' 0
19212 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Crime' 0
24015 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Documentary' 0
28818 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Drama' 0
33621 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Family' 0
38424 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Fantasy' 1
43227 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Foreign' 0
48030 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'History' 0
52833 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Horror' 0
57636 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Music' 0
62439 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Mystery' 0
67242 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Romance' 0
72045 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Science Fiction' 1
76848 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'TV Movie' 0
81651 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Thriller' 0
86454 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'War' 0
91257 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Western' 0

20 rows × 23 columns


In [278]:
# If only retaining "true" genres
longMovieFrameTrimmed = longMovieFrame[longMovieFrame['genre_present']==1]
longMovieFrameTrimmed[:5]


Out[278]:
budget genres homepage id keywords original_language original_title overview popularity production_companies ... runtime spoken_languages status tagline title vote_average vote_count GenreList Genre genre_present
0 237000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.avatarmovie.com/ 19995 [{"id": 1463, "name": "culture clash"}, {"id":... en Avatar In the 22nd century, a paraplegic Marine is di... 150.437577 [{"name": "Ingenious Film Partners", "id": 289... ... 162.0 [{"iso_639_1": "en", "name": "English"}, {"iso... Released Enter the World of Pandora. Avatar 7.2 11800 [Action, Adventure, Fantasy, Science Fiction] 'Action' 1
1 300000000 [{"id": 12, "name": "Adventure"}, {"id": 14, "... http://disney.go.com/disneypictures/pirates/ 285 [{"id": 270, "name": "ocean"}, {"id": 726, "na... en Pirates of the Caribbean: At World's End Captain Barbossa, long believed to be dead, ha... 139.082615 [{"name": "Walt Disney Pictures", "id": 2}, {"... ... 169.0 [{"iso_639_1": "en", "name": "English"}] Released At the end of the world, the adventure begins. Pirates of the Caribbean: At World's End 6.9 4500 [Adventure, Fantasy, Action] 'Action' 1
2 245000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://www.sonypictures.com/movies/spectre/ 206647 [{"id": 470, "name": "spy"}, {"id": 818, "name... en Spectre A cryptic message from Bond’s past sends him o... 107.376788 [{"name": "Columbia Pictures", "id": 5}, {"nam... ... 148.0 [{"iso_639_1": "fr", "name": "Fran\u00e7ais"},... Released A Plan No One Escapes Spectre 6.3 4466 [Action, Adventure, Crime] 'Action' 1
3 250000000 [{"id": 28, "name": "Action"}, {"id": 80, "nam... http://www.thedarkknightrises.com/ 49026 [{"id": 849, "name": "dc comics"}, {"id": 853,... en The Dark Knight Rises Following the death of District Attorney Harve... 112.312950 [{"name": "Legendary Pictures", "id": 923}, {"... ... 165.0 [{"iso_639_1": "en", "name": "English"}] Released The Legend Ends The Dark Knight Rises 7.6 9106 [Action, Crime, Drama, Thriller] 'Action' 1
4 260000000 [{"id": 28, "name": "Action"}, {"id": 12, "nam... http://movies.disney.com/john-carter 49529 [{"id": 818, "name": "based on novel"}, {"id":... en John Carter John Carter is a war-weary, former military ca... 43.926995 [{"name": "Walt Disney Pictures", "id": 2}] ... 132.0 [{"iso_639_1": "en", "name": "English"}] Released Lost in our world, found in another. John Carter 6.1 2124 [Action, Adventure, Science Fiction] 'Action' 1

5 rows × 23 columns

I think the melted data is duplicating in the context of data storage, as much more storage is required to create a separate row for each possible genre of the movies data.

However, there is not data duplication in the context of the table design for analysis, because each row has a unique combination of attributes.