English Wikipedia page views, 2008 - 2017

For this assignment, your job is to analyze traffic on English Wikipedia over time, and then document your process and the resulting dataset and visualization according to best practices for open research that were outlined for you in class.

Example API request

You can use this example API request as a starting point for building your API queries. Note that the Legacy Pagecounts API has slightly different schema than the pageview API shown here.

This sample API request would get you all pageviews by web crawlers on the mobile website for English Wikipedia during the month of September, 2017.

Scrape Pageview Mobile Site traffic (current api)


In [99]:
#current
import requests

endpoint = 'https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/{project}/{access}/{agent}/{granularity}/{start}/{end}'

headers={'User-Agent' : 'https://github.com/your_github_username', 'From' : 'abhiv@uw.edu'}

params = {'project' : 'en.wikipedia.org',
            'access' : 'mobile-web',
            'agent' : 'user',
            'granularity' : 'monthly',
            'start' : '2015070100',
            'end' : '2017091000'#use the first day of the following month to ensure a full month of data is collected
            }

api_call = requests.get(endpoint.format(**params))
pageview_mobile_site = api_call.json()
print(pageview_mobile_site)


{'items': [{'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015070100', 'views': 3179131148}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015080100', 'views': 3192663889}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015090100', 'views': 3073981649}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015100100', 'views': 3173975355}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015110100', 'views': 3142247145}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015120100', 'views': 3276836351}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016010100', 'views': 3611404079}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016020100', 'views': 3242448142}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016030100', 'views': 3288785117}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016040100', 'views': 3177044999}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016050100', 'views': 3296294723}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016060100', 'views': 3257882479}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016070100', 'views': 3395175122}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016080100', 'views': 3418646794}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016090100', 'views': 3310247842}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016100100', 'views': 3442109005}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016110100', 'views': 3507421156}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016120100', 'views': 3647567822}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017010100', 'views': 4020148351}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017020100', 'views': 3522702265}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017030100', 'views': 3719395296}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017040100', 'views': 3524571150}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017050100', 'views': 3567882051}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017060100', 'views': 3404097346}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017070100', 'views': 3600941034}, {'project': 'en.wikipedia', 'access': 'mobile-web', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017080100', 'views': 3502234506}]}

Scrape Pageview Mobile App traffic (current api)


In [98]:
#current
import requests

endpoint = 'https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/{project}/{access}/{agent}/{granularity}/{start}/{end}'

headers={'User-Agent' : 'https://github.com/your_github_username', 'From' : 'abhiv@uw.edu'}

params = {'project' : 'en.wikipedia.org',
            'access' : 'mobile-app',
            'agent' : 'user',
            'granularity' : 'monthly',
            'start' : '2015070100',
            'end' : '2017091000'#use the first day of the following month to ensure a full month of data is collected
            }

api_call = requests.get(endpoint.format(**params))
pageview_mobile_app = api_call.json()
print(pageview_mobile_app)


{'items': [{'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015070100', 'views': 109624146}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015080100', 'views': 109669149}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015090100', 'views': 96221684}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015100100', 'views': 94523777}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015110100', 'views': 94353925}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015120100', 'views': 99438956}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016010100', 'views': 106432767}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016020100', 'views': 92414130}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016030100', 'views': 97899074}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016040100', 'views': 81719003}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016050100', 'views': 98738513}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016060100', 'views': 96908466}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016070100', 'views': 101398640}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016080100', 'views': 97172509}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016090100', 'views': 83037939}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016100100', 'views': 67174886}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016110100', 'views': 83623769}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016120100', 'views': 128976033}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017010100', 'views': 211813191}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017020100', 'views': 189059134}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017030100', 'views': 184098693}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017040100', 'views': 115051969}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017050100', 'views': 118805669}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017060100', 'views': 115285847}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017070100', 'views': 124118219}, {'project': 'en.wikipedia', 'access': 'mobile-app', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017080100', 'views': 119171796}]}

Scrape Pageview Desktop Site traffic (current api)


In [82]:
#current
import requests

endpoint = 'https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/{project}/{access}/{agent}/{granularity}/{start}/{end}'

headers={'User-Agent' : 'https://github.com/your_github_username', 'From' : 'abhiv@uw.edu'}

params = {'project' : 'en.wikipedia.org',
            'access' : 'desktop',
            'agent' : 'user',
            'granularity' : 'monthly',
            'start' : '2015070100',
            'end' : '2017091000'#use the first day of the following month to ensure a full month of data is collected
            }

api_call = requests.get(endpoint.format(**params))
pageview_desktop_site = api_call.json()
print(pageview_desktop_site)


{'items': [{'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015070100', 'views': 4376666686}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015080100', 'views': 4332482183}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015090100', 'views': 4485491704}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015100100', 'views': 4477532755}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015110100', 'views': 4287720220}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015120100', 'views': 4100012037}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016010100', 'views': 4436179457}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016020100', 'views': 4250997185}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016030100', 'views': 4286590426}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016040100', 'views': 4149383857}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016050100', 'views': 4191778094}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016060100', 'views': 3888839711}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016070100', 'views': 4337865827}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016080100', 'views': 4695046216}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016090100', 'views': 4135006498}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016100100', 'views': 4361737690}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016110100', 'views': 4392068236}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016120100', 'views': 4209608578}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017010100', 'views': 4521980398}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017020100', 'views': 4026702163}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017030100', 'views': 4319971902}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017040100', 'views': 3951456992}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017050100', 'views': 4187870579}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017060100', 'views': 3604550997}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017070100', 'views': 3565444544}, {'project': 'en.wikipedia', 'access': 'desktop', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017080100', 'views': 3575572313}]}

Scrape Pageview All Site traffic (current api)


In [81]:
#current
import requests

endpoint = 'https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/{project}/{access}/{agent}/{granularity}/{start}/{end}'

headers={'User-Agent' : 'https://github.com/your_github_username', 'From' : 'abhiv@uw.edu'}

params = {'project' : 'en.wikipedia.org',
            'access' : 'all-access',
            'agent' : 'user',
            'granularity' : 'monthly',
            'start' : '2015070100',
            'end' : '2017091000'#use the first day of the following month to ensure a full month of data is collected
            }

api_call = requests.get(endpoint.format(**params))
pageview_all_site = api_call.json()
print(pageview_all_site)


{'items': [{'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015070100', 'views': 7665421980}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015080100', 'views': 7634815221}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015090100', 'views': 7655695037}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015100100', 'views': 7746031887}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015110100', 'views': 7524321290}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2015120100', 'views': 7476287344}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016010100', 'views': 8154016303}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016020100', 'views': 7585859457}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016030100', 'views': 7673274617}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016040100', 'views': 7408147859}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016050100', 'views': 7586811330}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016060100', 'views': 7243630656}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016070100', 'views': 7834439589}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016080100', 'views': 8210865519}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016090100', 'views': 7528292279}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016100100', 'views': 7871021581}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016110100', 'views': 7983113161}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2016120100', 'views': 7986152433}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017010100', 'views': 8753941940}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017020100', 'views': 7738463562}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017030100', 'views': 8223465891}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017040100', 'views': 7591080111}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017050100', 'views': 7874558299}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017060100', 'views': 7123934190}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017070100', 'views': 7290503797}, {'project': 'en.wikipedia', 'access': 'all-access', 'agent': 'user', 'granularity': 'monthly', 'timestamp': '2017080100', 'views': 7196978615}]}

Scrape Pagecount desktop Site traffic (legacy api)


In [62]:
#Legacy

import requests

endpoint = 'https://wikimedia.org/api/rest_v1/metrics/legacy/pagecounts/aggregate/{project}/{access}/{granularity}/{start}/{end}'
headers={'User-Agent' : 'https://github.com/your_github_username', 'From' : 'abhiv@uw.edu'}

params = {'project' : 'en.wikipedia.org',
            'access' : 'desktop-site',
            'granularity' : 'monthly',
            'start' : '2008010100',
            'end' : '2016071000'#use the first day of the following month to ensure a full month of data is collected
            }

api_call = requests.get(endpoint.format(**params))
pagecounts_desktop_site = api_call.json()
print(pagecounts_desktop_site)


{'items': [{'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2008010100', 'count': 4930902570}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2008020100', 'count': 4818393763}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2008030100', 'count': 4955405809}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2008040100', 'count': 5159162183}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2008050100', 'count': 5584691092}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2008060100', 'count': 5712104279}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2008070100', 'count': 5306302874}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2008080100', 'count': 5140155519}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2008090100', 'count': 5479533823}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2008100100', 'count': 5679440782}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2008110100', 'count': 5415832071}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2008120100', 'count': 5211708451}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2009010100', 'count': 5802681551}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2009020100', 'count': 5547320860}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2009030100', 'count': 6295159057}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2009040100', 'count': 5988817321}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2009050100', 'count': 6267516733}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2009060100', 'count': 5818924182}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2009070100', 'count': 5801646978}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2009080100', 'count': 5790850384}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2009090100', 'count': 4057515768}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2009100100', 'count': 6016107147}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2009110100', 'count': 5768486910}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2009120100', 'count': 5426505977}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2010010100', 'count': 5703465285}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2010020100', 'count': 5762451418}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2010030100', 'count': 6661347946}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2010040100', 'count': 6618552152}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2010050100', 'count': 6410578775}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2010060100', 'count': 4898035014}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2010070100', 'count': 5296177638}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2010080100', 'count': 7381346660}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2010090100', 'count': 7546488744}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2010100100', 'count': 10172844562}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2010110100', 'count': 6948678354}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2010120100', 'count': 7001952100}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2011010100', 'count': 7568511227}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2011020100', 'count': 6906248849}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2011030100', 'count': 7326545928}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2011040100', 'count': 6835492088}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2011050100', 'count': 7009799378}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2011060100', 'count': 6536675534}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2011070100', 'count': 6571730164}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2011080100', 'count': 6514500361}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2011090100', 'count': 5837738935}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2011100100', 'count': 6974424665}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2011110100', 'count': 7507641641}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2011120100', 'count': 6396357939}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2012010100', 'count': 7808477339}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2012020100', 'count': 7530127141}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2012030100', 'count': 7358543625}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2012040100', 'count': 7220029422}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2012050100', 'count': 7784479302}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2012060100', 'count': 7524265988}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2012070100', 'count': 7819376527}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2012080100', 'count': 7885641059}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2012090100', 'count': 7913689733}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2012100100', 'count': 8379960050}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2012110100', 'count': 8219987195}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2012120100', 'count': 8033826900}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2013010100', 'count': 9126210673}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2013020100', 'count': 8100597857}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2013030100', 'count': 8701419671}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2013040100', 'count': 8049296729}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2013050100', 'count': 8089099883}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2013060100', 'count': 7759959725}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2013070100', 'count': 8237301982}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2013080100', 'count': 9306122925}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2013090100', 'count': 9985326806}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2013100100', 'count': 10775158269}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2013110100', 'count': 10414877805}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2013120100', 'count': 7102459747}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2014010100', 'count': 7026531269}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2014020100', 'count': 6718243109}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2014030100', 'count': 7464128273}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2014040100', 'count': 7216245221}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2014050100', 'count': 7543277646}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2014060100', 'count': 6577352648}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2014070100', 'count': 6761059566}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2014080100', 'count': 6687313714}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2014090100', 'count': 7564600999}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2014100100', 'count': 6577533128}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2014110100', 'count': 6153537606}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2014120100', 'count': 5830332248}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2015010100', 'count': 6103767055}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2015020100', 'count': 5602710439}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2015030100', 'count': 6346602713}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2015040100', 'count': 6198945657}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2015050100', 'count': 6323801814}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2015060100', 'count': 5165413640}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2015070100', 'count': 5229226022}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2015080100', 'count': 5035534449}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2015090100', 'count': 5409631355}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2015100100', 'count': 5535704471}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2015110100', 'count': 5296956116}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2015120100', 'count': 5264446173}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2016010100', 'count': 5569632502}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2016020100', 'count': 5347709361}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2016030100', 'count': 5407676056}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2016040100', 'count': 5572235399}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2016050100', 'count': 5330532334}, {'project': 'en.wikipedia', 'access-site': 'desktop-site', 'granularity': 'monthly', 'timestamp': '2016060100', 'count': 4975092447}]}

Scrape Pagecount all Site traffic (legacy api)


In [89]:
#Legacy

import requests

endpoint = 'https://wikimedia.org/api/rest_v1/metrics/legacy/pagecounts/aggregate/{project}/{access}/{granularity}/{start}/{end}'
headers={'User-Agent' : 'https://github.com/your_github_username', 'From' : 'abhiv@uw.edu'}

params = {'project' : 'en.wikipedia.org',
            'access' : 'all-sites',
            'granularity' : 'monthly',
            'start' : '2008010100',
            'end' : '2016071000'#use the first day of the following month to ensure a full month of data is collected
            }

api_call = requests.get(endpoint.format(**params))
pagecounts_all_sites = api_call.json()
print(pagecounts_all_sites)


{'items': [{'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2008010100', 'count': 4930902570}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2008020100', 'count': 4818393763}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2008030100', 'count': 4955405809}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2008040100', 'count': 5159162183}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2008050100', 'count': 5584691092}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2008060100', 'count': 5712104279}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2008070100', 'count': 5306302874}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2008080100', 'count': 5140155519}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2008090100', 'count': 5479533823}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2008100100', 'count': 5679440782}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2008110100', 'count': 5415832071}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2008120100', 'count': 5211708451}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2009010100', 'count': 5802681551}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2009020100', 'count': 5547320860}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2009030100', 'count': 6295159057}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2009040100', 'count': 5988817321}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2009050100', 'count': 6267516733}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2009060100', 'count': 5818924182}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2009070100', 'count': 5801646978}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2009080100', 'count': 5790850384}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2009090100', 'count': 4057515768}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2009100100', 'count': 6016107147}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2009110100', 'count': 5768486910}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2009120100', 'count': 5426505977}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2010010100', 'count': 5703465285}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2010020100', 'count': 5762451418}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2010030100', 'count': 6661347946}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2010040100', 'count': 6618552152}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2010050100', 'count': 6410578775}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2010060100', 'count': 4898035014}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2010070100', 'count': 5296177638}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2010080100', 'count': 7381346660}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2010090100', 'count': 7546488744}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2010100100', 'count': 10172844562}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2010110100', 'count': 6948678354}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2010120100', 'count': 7001952100}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2011010100', 'count': 7568511227}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2011020100', 'count': 6906248849}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2011030100', 'count': 7326545928}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2011040100', 'count': 6835492088}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2011050100', 'count': 7009799378}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2011060100', 'count': 6536675534}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2011070100', 'count': 6571730164}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2011080100', 'count': 6514500361}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2011090100', 'count': 5837738935}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2011100100', 'count': 6974424665}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2011110100', 'count': 7507641641}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2011120100', 'count': 6396357939}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2012010100', 'count': 7808477339}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2012020100', 'count': 7530127141}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2012030100', 'count': 7358543625}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2012040100', 'count': 7220029422}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2012050100', 'count': 7784479302}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2012060100', 'count': 7524265988}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2012070100', 'count': 7819376527}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2012080100', 'count': 7885641059}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2012090100', 'count': 7913689733}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2012100100', 'count': 8379960050}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2012110100', 'count': 8219987195}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2012120100', 'count': 8033826900}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2013010100', 'count': 9126210673}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2013020100', 'count': 8100597857}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2013030100', 'count': 8701419671}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2013040100', 'count': 8049296729}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2013050100', 'count': 8089099883}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2013060100', 'count': 7759959725}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2013070100', 'count': 8237301982}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2013080100', 'count': 9306122925}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2013090100', 'count': 9985326806}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2013100100', 'count': 10775158269}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2013110100', 'count': 10414877805}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2013120100', 'count': 7102459747}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2014010100', 'count': 7026531269}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2014020100', 'count': 6718243109}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2014030100', 'count': 7464128273}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2014040100', 'count': 7216245221}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2014050100', 'count': 7543277646}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2014060100', 'count': 6577352648}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2014070100', 'count': 6761059566}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2014080100', 'count': 6687313714}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2014090100', 'count': 7564600999}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2014100100', 'count': 9669079813}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2014110100', 'count': 9181027274}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2014120100', 'count': 9109282269}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2015010100', 'count': 9589069146}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2015020100', 'count': 8694244918}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2015030100', 'count': 9677435301}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2015040100', 'count': 9421035574}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2015050100', 'count': 9657871297}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2015060100', 'count': 8203576103}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2015070100', 'count': 8483698717}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2015080100', 'count': 8304022031}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2015090100', 'count': 8582061182}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2015100100', 'count': 8781786976}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2015110100', 'count': 8515190628}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2015120100', 'count': 8651858036}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2016010100', 'count': 9309261244}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2016020100', 'count': 8680940753}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2016030100', 'count': 8827529692}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2016040100', 'count': 8873620523}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2016050100', 'count': 8748968139}, {'project': 'en.wikipedia', 'access-site': 'all-sites', 'granularity': 'monthly', 'timestamp': '2016060100', 'count': 8347710510}]}

Scrape Pagecount mobile Site traffic (legacy api)


In [63]:
#Legacy

import requests

endpoint = 'https://wikimedia.org/api/rest_v1/metrics/legacy/pagecounts/aggregate/{project}/{access}/{granularity}/{start}/{end}'
headers={'User-Agent' : 'https://github.com/your_github_username', 'From' : 'abhiv@uw.edu'}

params = {'project' : 'en.wikipedia.org',
            'access' : 'mobile-site',
            'granularity' : 'monthly',
            'start' : '2008010100',
            'end' : '2016071000'#use the first day of the following month to ensure a full month of data is collected
            }

api_call = requests.get(endpoint.format(**params))
pagecounts_mobile_site = api_call.json()
print(pagecounts_mobile_site)


{'items': [{'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2014100100', 'count': 3091546685}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2014110100', 'count': 3027489668}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2014120100', 'count': 3278950021}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2015010100', 'count': 3485302091}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2015020100', 'count': 3091534479}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2015030100', 'count': 3330832588}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2015040100', 'count': 3222089917}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2015050100', 'count': 3334069483}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2015060100', 'count': 3038162463}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2015070100', 'count': 3254472695}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2015080100', 'count': 3268487582}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2015090100', 'count': 3172429827}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2015100100', 'count': 3246082505}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2015110100', 'count': 3218234512}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2015120100', 'count': 3387411863}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2016010100', 'count': 3739628742}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2016020100', 'count': 3333231392}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2016030100', 'count': 3419853636}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2016040100', 'count': 3301385124}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2016050100', 'count': 3418435805}, {'project': 'en.wikipedia', 'access-site': 'mobile-site', 'granularity': 'monthly', 'timestamp': '2016060100', 'count': 3372618063}]}

In [29]:
countresponse['items']


Out[29]:
[{'access-site': 'all-sites',
  'count': 7546488744,
  'granularity': 'monthly',
  'project': 'en.wikipedia',
  'timestamp': '2010090100'}]

In [37]:
countresponse['items'][0]['count']


Out[37]:
7546488744

In [47]:
viewresponse['items'][3]['views']


Out[47]:
3416989181

In [70]:
pageview_mobile_site['items'][3]['timestamp']


Out[70]:
'2015100100'

Make a dictionary and add relevant info. timestamp = key, counts = value. Repeat for all


In [75]:
#pageview_mobile_site


i = 0
dict_pageview_mobile_site={}
for x in pageview_mobile_site['items']:
    dict_pageview_mobile_site[pageview_mobile_site['items'][i]['timestamp']] = pageview_mobile_site['items'][i]['views']
    
    i+=1

In [ ]:


In [76]:
dict_pageview_mobile_site


Out[76]:
{'2015070100': 3179131148,
 '2015080100': 3192663889,
 '2015090100': 3073981649,
 '2015100100': 3173975355,
 '2015110100': 3142247145,
 '2015120100': 3276836351,
 '2016010100': 3611404079,
 '2016020100': 3242448142,
 '2016030100': 3288785117,
 '2016040100': 3177044999,
 '2016050100': 3296294723,
 '2016060100': 3257882479,
 '2016070100': 3395175122,
 '2016080100': 3418646794,
 '2016090100': 3310247842,
 '2016100100': 3442109005,
 '2016110100': 3507421156,
 '2016120100': 3647567822,
 '2017010100': 4020148351,
 '2017020100': 3522702265,
 '2017030100': 3719395296,
 '2017040100': 3524571150,
 '2017050100': 3567882051,
 '2017060100': 3404097346,
 '2017070100': 3600941034,
 '2017080100': 3502234506}

In [77]:
#pageview_desktop_site


i = 0
dict_pageview_desktop_site={}
for x in pageview_desktop_site['items']:
    dict_pageview_desktop_site[pageview_desktop_site['items'][i]['timestamp']] = pageview_desktop_site['items'][i]['views']
    
    i+=1

In [78]:
dict_pageview_desktop_site


Out[78]:
{'2015070100': 4376666686,
 '2015080100': 4332482183,
 '2015090100': 4485491704,
 '2015100100': 4477532755,
 '2015110100': 4287720220,
 '2015120100': 4100012037,
 '2016010100': 4436179457,
 '2016020100': 4250997185,
 '2016030100': 4286590426,
 '2016040100': 4149383857,
 '2016050100': 4191778094,
 '2016060100': 3888839711,
 '2016070100': 4337865827,
 '2016080100': 4695046216,
 '2016090100': 4135006498,
 '2016100100': 4361737690,
 '2016110100': 4392068236,
 '2016120100': 4209608578,
 '2017010100': 4521980398,
 '2017020100': 4026702163,
 '2017030100': 4319971902,
 '2017040100': 3951456992,
 '2017050100': 4187870579,
 '2017060100': 3604550997,
 '2017070100': 3565444544,
 '2017080100': 3575572313}

In [83]:
#pageview_all_site


i = 0
dict_pageview_all_site={}
for x in pageview_all_site['items']:
    dict_pageview_all_site[pageview_all_site['items'][i]['timestamp']] = pageview_all_site['items'][i]['views']
    
    i+=1

In [85]:
dict_pageview_all_site


Out[85]:
{'2015070100': 7665421980,
 '2015080100': 7634815221,
 '2015090100': 7655695037,
 '2015100100': 7746031887,
 '2015110100': 7524321290,
 '2015120100': 7476287344,
 '2016010100': 8154016303,
 '2016020100': 7585859457,
 '2016030100': 7673274617,
 '2016040100': 7408147859,
 '2016050100': 7586811330,
 '2016060100': 7243630656,
 '2016070100': 7834439589,
 '2016080100': 8210865519,
 '2016090100': 7528292279,
 '2016100100': 7871021581,
 '2016110100': 7983113161,
 '2016120100': 7986152433,
 '2017010100': 8753941940,
 '2017020100': 7738463562,
 '2017030100': 8223465891,
 '2017040100': 7591080111,
 '2017050100': 7874558299,
 '2017060100': 7123934190,
 '2017070100': 7290503797,
 '2017080100': 7196978615}

In [92]:
#pagecount_all_site


i = 0
dict_pagecounts_all_sites={}
for x in pagecounts_all_sites['items']:
    dict_pagecounts_all_sites[pagecounts_all_sites['items'][i]['timestamp']] = pagecounts_all_sites['items'][i]['count']
    
    i+=1

In [ ]:


In [93]:
dict_pagecounts_all_sites


Out[93]:
{'2008010100': 4930902570,
 '2008020100': 4818393763,
 '2008030100': 4955405809,
 '2008040100': 5159162183,
 '2008050100': 5584691092,
 '2008060100': 5712104279,
 '2008070100': 5306302874,
 '2008080100': 5140155519,
 '2008090100': 5479533823,
 '2008100100': 5679440782,
 '2008110100': 5415832071,
 '2008120100': 5211708451,
 '2009010100': 5802681551,
 '2009020100': 5547320860,
 '2009030100': 6295159057,
 '2009040100': 5988817321,
 '2009050100': 6267516733,
 '2009060100': 5818924182,
 '2009070100': 5801646978,
 '2009080100': 5790850384,
 '2009090100': 4057515768,
 '2009100100': 6016107147,
 '2009110100': 5768486910,
 '2009120100': 5426505977,
 '2010010100': 5703465285,
 '2010020100': 5762451418,
 '2010030100': 6661347946,
 '2010040100': 6618552152,
 '2010050100': 6410578775,
 '2010060100': 4898035014,
 '2010070100': 5296177638,
 '2010080100': 7381346660,
 '2010090100': 7546488744,
 '2010100100': 10172844562,
 '2010110100': 6948678354,
 '2010120100': 7001952100,
 '2011010100': 7568511227,
 '2011020100': 6906248849,
 '2011030100': 7326545928,
 '2011040100': 6835492088,
 '2011050100': 7009799378,
 '2011060100': 6536675534,
 '2011070100': 6571730164,
 '2011080100': 6514500361,
 '2011090100': 5837738935,
 '2011100100': 6974424665,
 '2011110100': 7507641641,
 '2011120100': 6396357939,
 '2012010100': 7808477339,
 '2012020100': 7530127141,
 '2012030100': 7358543625,
 '2012040100': 7220029422,
 '2012050100': 7784479302,
 '2012060100': 7524265988,
 '2012070100': 7819376527,
 '2012080100': 7885641059,
 '2012090100': 7913689733,
 '2012100100': 8379960050,
 '2012110100': 8219987195,
 '2012120100': 8033826900,
 '2013010100': 9126210673,
 '2013020100': 8100597857,
 '2013030100': 8701419671,
 '2013040100': 8049296729,
 '2013050100': 8089099883,
 '2013060100': 7759959725,
 '2013070100': 8237301982,
 '2013080100': 9306122925,
 '2013090100': 9985326806,
 '2013100100': 10775158269,
 '2013110100': 10414877805,
 '2013120100': 7102459747,
 '2014010100': 7026531269,
 '2014020100': 6718243109,
 '2014030100': 7464128273,
 '2014040100': 7216245221,
 '2014050100': 7543277646,
 '2014060100': 6577352648,
 '2014070100': 6761059566,
 '2014080100': 6687313714,
 '2014090100': 7564600999,
 '2014100100': 9669079813,
 '2014110100': 9181027274,
 '2014120100': 9109282269,
 '2015010100': 9589069146,
 '2015020100': 8694244918,
 '2015030100': 9677435301,
 '2015040100': 9421035574,
 '2015050100': 9657871297,
 '2015060100': 8203576103,
 '2015070100': 8483698717,
 '2015080100': 8304022031,
 '2015090100': 8582061182,
 '2015100100': 8781786976,
 '2015110100': 8515190628,
 '2015120100': 8651858036,
 '2016010100': 9309261244,
 '2016020100': 8680940753,
 '2016030100': 8827529692,
 '2016040100': 8873620523,
 '2016050100': 8748968139,
 '2016060100': 8347710510}

In [95]:
i = 0
dict_pagecounts_desktop_site={}
for x in pagecounts_desktop_site['items']:
    dict_pagecounts_desktop_site[pagecounts_desktop_site['items'][i]['timestamp']] = pagecounts_desktop_site['items'][i]['count']
    
    i+=1

In [96]:
i = 0
dict_pagecounts_mobile_site={}
for x in pagecounts_mobile_site['items']:
    dict_pagecounts_mobile_site[pagecounts_mobile_site['items'][i]['timestamp']] = pagecounts_mobile_site['items'][i]['count']
    
    i+=1

In [101]:
i = 0
dict_pageview_mobile_app={}
for x in pageview_mobile_app['items']:
    dict_pageview_mobile_app[pageview_mobile_app['items'][i]['timestamp']] = pageview_mobile_app['items'][i]['views']
    
    i+=1

In [105]:
dict_pageview_mobile = {}
i=0
for x in pageview_mobile_app['items']:
    dict_pageview_mobile[pageview_mobile_app['items'][i]['timestamp']] = pageview_mobile_app['items'][i]['views']+pageview_mobile_site['items'][i]['views']
    
    i+=1

In [112]: