Movie Aggregator Ratings have No Relationship with Box Office Success

by Max Woolf

This notebook is licensed under the MIT License. If you use the code or data visualization designs contained within this notebook, it would be greatly appreciated if proper attribution is given back to this notebook and/or myself. Thanks! :)


In [1]:
options(warn = -1)

# IMPORTANT: This assumes that all packages in "Rstart.R" are installed,
# and the fonts "Source Sans Pro" and "Open Sans Condensed Bold" are installed
# via extrafont. If ggplot2 charts fail to render, you may need to change/remove the theme call.

source("Rstart.R")

sessionInfo()


Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

Registering fonts with R

Attaching package: ‘scales’

The following objects are masked from ‘package:readr’:

    col_factor, col_numeric

Out[1]:
R version 3.2.2 (2015-08-14)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.11.2 (El Capitan)

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] grid      stats     graphics  grDevices utils     datasets  methods  
[8] base     

other attached packages:
[1] stringr_1.0.0      digest_0.6.8       RColorBrewer_1.1-2 scales_0.3.0      
[5] extrafont_0.17     ggplot2_2.0.0      dplyr_0.4.3        readr_0.1.1       

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.1      Rttf2pt1_1.3.3   magrittr_1.5     munsell_0.4.2   
 [5] uuid_0.1-2       colorspace_1.2-6 R6_2.1.1         plyr_1.8.3      
 [9] tools_3.2.2      parallel_3.2.2   gtable_0.1.2     DBI_0.3.1       
[13] extrafontdb_1.0  assertthat_0.1   IRdisplay_0.3    repr_0.4        
[17] base64enc_0.1-3  IRkernel_0.5     evaluate_0.8     rzmq_0.7.7      
[21] stringi_0.5-5    jsonlite_0.9.17 

Process the Data

Take the movies data, load in R friendly format, and combine with Rotten Tomatoes data.


In [2]:
df <- read_delim("~/Downloads/omdb1215/omdbMovies.txt", "\t", col_types="iccicccccccidi_c_____")


|================================================================================| 100%  422 MB

In [3]:
print(df %>% head())
print(nrow(df))


Source: local data frame [6 x 15]

     ID    imdbID                  Title  Year    Rating Runtime
  (int)     (chr)                  (chr) (int)     (chr)   (chr)
1     1 tt0000001             Carmencita  1894 NOT RATED   1 min
2     2 tt0000002 Le clown et ses chiens  1892                  
3     3 tt0000003         Pauvre Pierrot  1892             4 min
4     4 tt0000004            Un bon bock  1892                  
5     5 tt0000005       Blacksmith Scene  1893   UNRATED   1 min
6     6 tt0000006      Chinese Opium Den  1894             1 min
Variables not shown: Genre (chr), Released (chr), Director (chr), Writer (chr),
  Cast (chr), Metacritic (int), imdbRating (dbl), imdbVotes (int), Plot (chr)
[1] 1109885

In [4]:
df_tomatoes <- read_delim("~/Downloads/omdb1215/tomatoes.txt",  "\t", col_types="i_diiiicidi_cc_c")

In [5]:
print(df_tomatoes %>% head())
print(nrow(df_tomatoes))


Source: local data frame [6 x 13]

     ID Rating Meter Reviews Fresh Rotten Consensus userMeter userRating
  (int)  (dbl) (int)   (int) (int)  (int)     (chr)     (int)      (dbl)
1     4     NA    NA      NA    NA     NA                 100        2.8
2     5     NA    NA      NA    NA     NA                  32        3.0
3    10     NA    NA      NA    NA     NA                  NA         NA
4    75     NA    NA      NA    NA     NA                  NA         NA
5    89     NA    NA      NA    NA     NA                  NA         NA
6   174     NA    NA      NA    NA     NA                  NA         NA
Variables not shown: userReviews (int), BoxOffice (chr), Production (chr),
  lastUpdated (chr)
[1] 98124

In [6]:
df <- df %>% left_join(df_tomatoes, by="ID")
rm(df_tomatoes)

In [7]:
print(df %>% head())


Source: local data frame [6 x 27]

     ID    imdbID                  Title  Year  Rating.x Runtime
  (int)     (chr)                  (chr) (int)     (chr)   (chr)
1     1 tt0000001             Carmencita  1894 NOT RATED   1 min
2     2 tt0000002 Le clown et ses chiens  1892                  
3     3 tt0000003         Pauvre Pierrot  1892             4 min
4     4 tt0000004            Un bon bock  1892                  
5     5 tt0000005       Blacksmith Scene  1893   UNRATED   1 min
6     6 tt0000006      Chinese Opium Den  1894             1 min
Variables not shown: Genre (chr), Released (chr), Director (chr), Writer (chr),
  Cast (chr), Metacritic (int), imdbRating (dbl), imdbVotes (int), Plot (chr),
  Rating.y (dbl), Meter (int), Reviews (int), Fresh (int), Rotten (int),
  Consensus (chr), userMeter (int), userRating (dbl), userReviews (int),
  BoxOffice (chr), Production (chr), lastUpdated (chr)

Tweak Columns to Numeric

Both Runtime and BoxOffice need to be converted to numeric values for aggregation.


In [8]:
parseRuntime <- function(x) {
    if (is.na(x) | x=="") {return (NA)}
    return(strsplit(x, " ")[[1]][1])
}

parseRuntime("4 min")


Out[8]:
'4'

In [9]:
df <- df %>% mutate(Runtime = as.numeric(sapply(Runtime, parseRuntime)))

In [10]:
print(df %>% select(Runtime) %>% head())


Source: local data frame [6 x 1]

  Runtime
    (dbl)
1       1
2      NA
3       4
4      NA
5       1
6       1

Parse BoxOffice the same way, but it is tricker and needs a look at the data.


In [11]:
print(df %>% select(BoxOffice) %>% filter(BoxOffice!='') %>% head())


Source: local data frame [6 x 1]

  BoxOffice
      (chr)
1     $0.3M
2    $51.0k
3     $0.5M
4    $35.3k
5     $0.2M
6     $1.1M

In [12]:
parseBoxOffice <- function(x) {
    unit <- 0
    if (is.na(x) | x=="") {return (NA)}
    if (substr(x, nchar(x), nchar(x)) == "k") {unit <- 10^3}
    else {unit <- 10^6}

    number <- as.numeric(substr(x,2,nchar(x)-1))

    return(number * unit)
}

parseBoxOffice("$0.3M")
parseBoxOffice("$51.0k")


Out[12]:
3e+05
Out[12]:
51000

In [13]:
df <- df %>% mutate(BoxOffice = as.numeric(sapply(BoxOffice, parseBoxOffice)))

Save the processed data for later so that it can be reimported easily!


In [ ]:
write.csv(df, "movies-processed.csv", row.names=F)

Rating Data Scatterplots

Now we can create scatterplots! First, we need to aggregate the data. We start with the Rotten Tomatoes Tomatometer review data:

Rotten Tomatoes Tomatometer


In [14]:
df_box <- df %>% select(Title, Meter, BoxOffice) %>% 
                filter(!is.na(Meter), !is.na(BoxOffice)) %>% 
                arrange(desc(BoxOffice))

print(df_box %>% head())
print(nrow(df_box))
print(cor(df_box$Meter, log10(df_box$BoxOffice)))


Source: local data frame [6 x 3]

                    Title Meter BoxOffice
                    (chr) (int)     (dbl)
1                  Avatar    83 760500000
2          Jurassic World    71 652200000
3            The Avengers    92 623300000
4         The Dark Knight    94 533300000
5 Avengers: Age of Ultron    74 459000000
6   The Dark Knight Rises    87 448100000
[1] 4863
[1] -0.18361

In [15]:
plot <- ggplot(df_box, aes(x=Meter, y=BoxOffice)) +
            annotate(geom="rect", xmin=60, xmax=100, ymin=min(df_box$Meter), ymax=Inf, fill="#1a1a1a", alpha=0.1) +
            geom_point(alpha=0.2, stroke=0, size=2, color="#e74c3c") +
            fte_theme() +
            scale_y_log10(breaks=10^(3:9), labels=dollar) +
            scale_x_continuous(breaks=seq(0,100, by=10), labels=paste0(seq(0,100, by=10), "%"), limits=c(0,100)) +
            labs(x="Rotten Tomatoes Tomatometer Score", y="Movie Box Office Gross",
                title="Scatterplot of Box Office Gross vs. RT Tomatometer") +
            geom_smooth(color="#1a1a1a", method="lm")

max_save(plot, "box-office-rating-1", "IMDb and Rotten Tomatoes")


In [16]:
plot <- ggplot(df_box, aes(x=Meter, y=BoxOffice)) +
            annotate(geom="rect", xmin=60, xmax=100, ymin=min(df_box$Meter), ymax=Inf, fill="#1a1a1a", alpha=0.1) +
            fte_theme() +
            scale_y_log10(breaks=10^(3:9), labels=dollar) +
            scale_x_continuous(breaks=seq(0,100, by=10), labels=paste0(seq(0,100, by=10), "%"), limits=c(0,100)) +
            stat_density2d(aes(fill= ..level..), geom="polygon", color="#e74c3c", size=0.2) +
            scale_fill_gradient(low="#eeeeee", high="#e74c3c") +
            labs(x="Rotten Tomatoes Tomatometer Score", y="Movie Box Office Gross",
                title="Contour Map of Box Office Gross vs. RT Tomatometer")

max_save(plot, "box-office-rating-2", "IMDb and Rotten Tomatoes")

Rotten Tomatoes Audience


In [17]:
df_box <- df %>% select(Title, userMeter, BoxOffice) %>% 
                filter(!is.na(userMeter), !is.na(BoxOffice)) %>% 
                arrange(desc(BoxOffice))

print(df_box %>% head())
print(nrow(df_box))
print(cor(df_box$userMeter, log10(df_box$BoxOffice)))


Source: local data frame [6 x 3]

                    Title userMeter BoxOffice
                    (chr)     (int)     (dbl)
1                  Avatar        82 760500000
2          Jurassic World        80 652200000
3            The Avengers        91 623300000
4         The Dark Knight        94 533300000
5 Avengers: Age of Ultron        85 459000000
6   The Dark Knight Rises        90 448100000
[1] 5163
[1] 0.04952105

In [18]:
plot <- ggplot(df_box, aes(x=userMeter, y=BoxOffice)) +
            geom_point(alpha=0.2, stroke=0, size=2, color="#27ae60") +
            fte_theme() +
            scale_y_log10(breaks=10^(3:9), labels=dollar) +
            scale_x_continuous(breaks=seq(0,100, by=10), labels=paste0(seq(0,100, by=10), "%"), limits=c(0,100)) +
            labs(x="Rotten Tomatoes Audience Score", y="Movie Box Office Gross",
                title="Scatterplot of Box Office Gross vs. RT Audience") +
            geom_smooth(color="#1a1a1a", method="lm")

max_save(plot, "box-office-rating-3", "IMDb and Rotten Tomatoes")


In [19]:
plot <- ggplot(df_box, aes(x=userMeter, y=BoxOffice)) +
            fte_theme() +
            scale_y_log10(breaks=10^(3:9), labels=dollar) +
            scale_x_continuous(breaks=seq(0,100, by=10), labels=paste0(seq(0,100, by=10), "%"), limits=c(0,100)) +
            stat_density2d(aes(fill= ..level..), geom="polygon", color="#27ae60", size=0.2) +
            scale_fill_gradient(low="#eeeeee", high="#27ae60") +
            labs(x="Rotten Tomatoes Audience Score", y="Movie Box Office Gross",
                title="Contour Map of Box Office Gross vs. RT Audience")

max_save(plot, "box-office-rating-4", "IMDb and Rotten Tomatoes")

IMDb Rating


In [20]:
df_box <- df %>% select(Title, imdbRating, BoxOffice) %>% 
                filter(!is.na(imdbRating), !is.na(BoxOffice)) %>% 
                arrange(desc(BoxOffice))

print(df_box %>% head())
print(nrow(df_box))
print(cor(df_box$imdbRating, log10(df_box$BoxOffice)))


Source: local data frame [6 x 3]

                    Title imdbRating BoxOffice
                    (chr)      (dbl)     (dbl)
1                  Avatar        7.9 760500000
2          Jurassic World        7.1 652200000
3            The Avengers        8.1 623300000
4         The Dark Knight        9.0 533300000
5 Avengers: Age of Ultron        7.6 459000000
6   The Dark Knight Rises        8.5 448100000
[1] 5167
[1] -0.001493861

In [21]:
plot <- ggplot(df_box, aes(x=imdbRating, y=BoxOffice)) +
            geom_point(alpha=0.2, stroke=0, size=2, color="#d35400") +
            fte_theme() +
            scale_y_log10(breaks=10^(3:9), labels=dollar) +
            scale_x_continuous(breaks=1:10, limits=c(0,10)) +
            labs(x="IMDb User Rating", y="Movie Box Office Gross",
                title="Scatterplot of Box Office Gross vs. IMDb User Rating") +
            geom_smooth(color="#1a1a1a", method="lm")

max_save(plot, "box-office-rating-5", "IMDb and Rotten Tomatoes")


In [22]:
plot <- ggplot(df_box, aes(x=imdbRating, y=BoxOffice)) +
            fte_theme() +
            scale_y_log10(breaks=10^(3:9), labels=dollar) +
            scale_x_continuous(breaks=1:10, limits=c(0,10)) +
            stat_density2d(aes(fill= ..level..), geom="polygon", color="#d35400", size=0.2) +
            scale_fill_gradient(low="#eeeeee", high="#d35400") +
            labs(x="IMDb User Rating", y="Movie Box Office Gross",
                title="Contour Map of Box Office Gross vs. IMDb User Rating")

max_save(plot, "box-office-rating-6", "IMDb and Rotten Tomatoes")

Metacritic


In [27]:
df_box <- df %>% select(Title, Metacritic, BoxOffice) %>% 
                filter(!is.na(Metacritic), !is.na(BoxOffice)) %>% 
                arrange(desc(BoxOffice))

print(df_box %>% head())
print(nrow(df_box))
print(cor(df_box$Metacritic, log10(df_box$BoxOffice)))


Source: local data frame [6 x 3]

                    Title Metacritic BoxOffice
                    (chr)      (int)     (dbl)
1                  Avatar         83 760500000
2          Jurassic World         59 652200000
3            The Avengers         69 623300000
4         The Dark Knight         82 533300000
5 Avengers: Age of Ultron         66 459000000
6   The Dark Knight Rises         78 448100000
[1] 4479
[1] -0.1312911

In [24]:
plot <- ggplot(df_box, aes(x=Metacritic, y=BoxOffice)) +
            geom_point(alpha=0.2, stroke=0, size=2, color="#8e44ad") +
            fte_theme() +
            scale_y_log10(breaks=10^(3:9), labels=dollar) +
            scale_x_continuous(breaks=seq(0,100, by=10)) +
            labs(x="Metacritic Score", y="Movie Box Office Gross",
                title="Scatterplot of Box Office Gross vs. Metacritic") +
            geom_smooth(color="#1a1a1a", method = "lm")

max_save(plot, "box-office-rating-7", "IMDb and Rotten Tomatoes")


In [30]:
plot <- ggplot(df_box, aes(x=Metacritic, y=BoxOffice)) +
            fte_theme() +
            scale_y_log10(breaks=10^(3:9), labels=dollar, limits=c(10^3,10^9)) +
            scale_x_continuous(breaks=seq(0,100, by=10), limits=c(0,100)) +
            stat_density2d(aes(fill= ..level..), geom="polygon", color="#8e44ad", size=0.2) +
            scale_fill_gradient(low="#eeeeee", high="#8e44ad") +
            labs(x="Metacritic Score", y="Movie Box Office Gross",
                title="Contour Map of Box Office Gross vs. Metacritic Score")

max_save(plot, "box-office-rating-8", "IMDb and Rotten Tomatoes")

The MIT License (MIT)

Copyright (c) 2016 Max Woolf

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.