Batch Data Collection

This notebook will download all 120k+ articles with abstracts from the PLOS API.

Imports



In [1]:

    
import pandas as pd
import numpy as np 
import settings
import requests
import urllib
import time
from retrying import retry
import os

API Call Function



In [2]:

    
#adapted from Raymond's notebook

def plos_search(q,start=0,rows=100,fl=None, extras=None):

    BASE_URL = 'http://api.plos.org/search'
    DEFAULT_FL = ('abstract','author',
                  'id','journal','publication_date',
                  'score','title_display', 'subject','subject_level')
        #removed elements: eissn, article_type
    
    # fl indicates fields to return
    # http://wiki.apache.org/solr/CommonQueryParameters#fl
    
    if fl is None:
        fl_ = ",".join(DEFAULT_FL)
    else:
        fl_ = ",".join(fl)
        
    query = {'q':q,
             'start':start,
             'rows':rows,
             'api_key':settings.PLOS_KEY,
             'wt':'json',
             'fl':fl_,
             'fq': 'doc_type:full AND !article_type_facet:"Issue Image"'}
    
    if extras is not None:
        query.update(extras)
        
    query_url = BASE_URL + "?" +urllib.urlencode(query)
    
    r = requests.get(query_url)
    return r

Finding Parameters

Need to make sure the calls do not exceed the following:

7200 requests a day, 300 per hour, 10 per minute and allow 5 seconds for your search to return results.

To be safe there will be a 15 second wait between each call:

15 sec per call
4 calls per minute
240 calls per hour

Call for all articles



In [3]:

    
r = plos_search(q='*')

Check the total number of articles with abstracts.



In [4]:

    
tot_articles = r.json()['response']['numFound']
tot_articles









    Out[4]:





126718

With ~~118545~~ ~~119637~~ 126718 acticles total that means that we will have to perform ~1.2k API requests at 100 articles per request.

At 240 requests per hour it should take about 5.5 hours to get all the data needed.

Looping Function

This function will call the plos_search function every 15 seconds while incrementing the start number so that all of the articles can be pulled.



In [5]:

    
@retry(wait='exponential_sleep', wait_exponential_multiplier=1000, wait_exponential_max=10000, stop='stop_after_attempt', stop_max_attempt_number=200)
def data_request(end, start=0):
    current_end = end - start
    rows_per_query = 100
    loops = (current_end/rows_per_query) + 1
    for n in range(loops):
        current_pickle = '../data/pickles/plos_pickle_{0}.pkl'.format(n)
        
        #skips rows that have already been downloaded
        if os.path.exists(current_pickle):
            start += rows_per_query
            continue
            
        r = plos_search(q='subject:*', start=start, rows=rows_per_query)
        
        #increment the start for the next request
        start += rows_per_query
        
        #store data before next call
        data = r.json()['response']['docs']
        new_df = pd.DataFrame(data)
        
        #every request pickle the dataframe
        new_df.to_pickle(current_pickle)
        
        #prints status of job
        print start, " articles pickled"
        
        #wait a few seconds before the next loop
        time.sleep(7)
    print "DONE!"

Now we can run the function inputing the tot_articles as the end parameter.

Before proceeding with this notebook to obtain/update your dataset:

Make sure you have a data/pickles/ directory and it is empty.
Make sure that data/all_plos_df.pkl is suitably backed up (or does not exist).

Data collection log

First collection: crashed at 12100, 43800, 49500, 53500

Update 07-2014: What! No crashes!



In [6]:

    
all_plos_df = data_request(end=tot_articles)









    



100  articles pickled
200  articles pickled
300  articles pickled
400  articles pickled
500  articles pickled
600  articles pickled
700  articles pickled
800  articles pickled
900  articles pickled
1000  articles pickled
1100  articles pickled
1200  articles pickled
1300  articles pickled
1400  articles pickled
1500  articles pickled
1600  articles pickled
1700  articles pickled
1800  articles pickled
1900  articles pickled
2000  articles pickled
2100  articles pickled
2200  articles pickled
2300  articles pickled
2400  articles pickled
2500  articles pickled
2600  articles pickled
2700  articles pickled
2800  articles pickled
2900  articles pickled
3000  articles pickled
3100  articles pickled
3200  articles pickled
3300  articles pickled
3400  articles pickled
3500  articles pickled
3600  articles pickled
3700  articles pickled
3800  articles pickled
3900  articles pickled
4000  articles pickled
4100  articles pickled
4200  articles pickled
4300  articles pickled
4400  articles pickled
4500  articles pickled
4600  articles pickled
4700  articles pickled
4800  articles pickled
4900  articles pickled
5000  articles pickled
5100  articles pickled
5200  articles pickled
5300  articles pickled
5400  articles pickled
5500  articles pickled
5600  articles pickled
5700  articles pickled
5800  articles pickled
5900  articles pickled
6000  articles pickled
6100  articles pickled
6200  articles pickled
6300  articles pickled
6400  articles pickled
6500  articles pickled
6600  articles pickled
6700  articles pickled
6800  articles pickled
6900  articles pickled
7000  articles pickled
7100  articles pickled
7200  articles pickled
7300  articles pickled
7400  articles pickled
7500  articles pickled
7600  articles pickled
7700  articles pickled
7800  articles pickled
7900  articles pickled
8000  articles pickled
8100  articles pickled
8200  articles pickled
8300  articles pickled
8400  articles pickled
8500  articles pickled
8600  articles pickled
8700  articles pickled
8800  articles pickled
8900  articles pickled
9000  articles pickled
9100  articles pickled
9200  articles pickled
9300  articles pickled
9400  articles pickled
9500  articles pickled
9600  articles pickled
9700  articles pickled
9800  articles pickled
9900  articles pickled
10000  articles pickled
10100  articles pickled
10200  articles pickled
10300  articles pickled
10400  articles pickled
10500  articles pickled
10600  articles pickled
10700  articles pickled
10800  articles pickled
10900  articles pickled
11000  articles pickled
11100  articles pickled
11200  articles pickled
11300  articles pickled
11400  articles pickled
11500  articles pickled
11600  articles pickled
11700  articles pickled
11800  articles pickled
11900  articles pickled
12000  articles pickled
12100  articles pickled
12200  articles pickled
12300  articles pickled
12400  articles pickled
12500  articles pickled
12600  articles pickled
12700  articles pickled
12800  articles pickled
12900  articles pickled
13000  articles pickled
13100  articles pickled
13200  articles pickled
13300  articles pickled
13400  articles pickled
13500  articles pickled
13600  articles pickled
13700  articles pickled
13800  articles pickled
13900  articles pickled
14000  articles pickled
14100  articles pickled
14200  articles pickled
14300  articles pickled
14400  articles pickled
14500  articles pickled
14600  articles pickled
14700  articles pickled
14800  articles pickled
14900  articles pickled
15000  articles pickled
15100  articles pickled
15200  articles pickled
15300  articles pickled
15400  articles pickled
15500  articles pickled
15600  articles pickled
15700  articles pickled
15800  articles pickled
15900  articles pickled
16000  articles pickled
16100  articles pickled
16200  articles pickled
16300  articles pickled
16400  articles pickled
16500  articles pickled
16600  articles pickled
16700  articles pickled
16800  articles pickled
16900  articles pickled
17000  articles pickled
17100  articles pickled
17200  articles pickled
17300  articles pickled
17400  articles pickled
17500  articles pickled
17600  articles pickled
17700  articles pickled
17800  articles pickled
17900  articles pickled
18000  articles pickled
18100  articles pickled
18200  articles pickled
18300  articles pickled
18400  articles pickled
18500  articles pickled
18600  articles pickled
18700  articles pickled
18800  articles pickled
18900  articles pickled
19000  articles pickled
19100  articles pickled
19200  articles pickled
19300  articles pickled
19400  articles pickled
19500  articles pickled
19600  articles pickled
19700  articles pickled
19800  articles pickled
19900  articles pickled
20000  articles pickled
20100  articles pickled
20200  articles pickled
20300  articles pickled
20400  articles pickled
20500  articles pickled
20600  articles pickled
20700  articles pickled
20800  articles pickled
20900  articles pickled
21000  articles pickled
21100  articles pickled
21200  articles pickled
21300  articles pickled
21400  articles pickled
21500  articles pickled
21600  articles pickled
21700  articles pickled
21800  articles pickled
21900  articles pickled
22000  articles pickled
22100  articles pickled
22200  articles pickled
22300  articles pickled
22400  articles pickled
22500  articles pickled
22600  articles pickled
22700  articles pickled
22800  articles pickled
22900  articles pickled
23000  articles pickled
23100  articles pickled
23200  articles pickled
23300  articles pickled
23400  articles pickled
23500  articles pickled
23600  articles pickled
23700  articles pickled
23800  articles pickled
23900  articles pickled
24000  articles pickled
24100  articles pickled
24200  articles pickled
24300  articles pickled
24400  articles pickled
24500  articles pickled
24600  articles pickled
24700  articles pickled
24800  articles pickled
24900  articles pickled
25000  articles pickled
25100  articles pickled
25200  articles pickled
25300  articles pickled
25400  articles pickled
25500  articles pickled
25600  articles pickled
25700  articles pickled
25800  articles pickled
25900  articles pickled
26000  articles pickled
26100  articles pickled
26200  articles pickled
26300  articles pickled
26400  articles pickled
26500  articles pickled
26600  articles pickled
26700  articles pickled
26800  articles pickled
26900  articles pickled
27000  articles pickled
27100  articles pickled
27200  articles pickled
27300  articles pickled
27400  articles pickled
27500  articles pickled
27600  articles pickled
27700  articles pickled
27800  articles pickled
27900  articles pickled
28000  articles pickled
28100  articles pickled
28200  articles pickled
28300  articles pickled
28400  articles pickled
28500  articles pickled
28600  articles pickled
28700  articles pickled
28800  articles pickled
28900  articles pickled
29000  articles pickled
29100  articles pickled
29200  articles pickled
29300  articles pickled
29400  articles pickled
29500  articles pickled
29600  articles pickled
29700  articles pickled
29800  articles pickled
29900  articles pickled
30000  articles pickled
30100  articles pickled
30200  articles pickled
30300  articles pickled
30400  articles pickled
30500  articles pickled
30600  articles pickled
30700  articles pickled
30800  articles pickled
30900  articles pickled
31000  articles pickled
31100  articles pickled
31200  articles pickled
31300  articles pickled
31400  articles pickled
31500  articles pickled
31600  articles pickled
31700  articles pickled
31800  articles pickled
31900  articles pickled
32000  articles pickled
32100  articles pickled
32200  articles pickled
32300  articles pickled
32400  articles pickled
32500  articles pickled
32600  articles pickled
32700  articles pickled
32800  articles pickled
32900  articles pickled
33000  articles pickled
33100  articles pickled
33200  articles pickled
33300  articles pickled
33400  articles pickled
33500  articles pickled
33600  articles pickled
33700  articles pickled
33800  articles pickled
33900  articles pickled
34000  articles pickled
34100  articles pickled
34200  articles pickled
34300  articles pickled
34400  articles pickled
34500  articles pickled
34600  articles pickled
34700  articles pickled
34800  articles pickled
34900  articles pickled
35000  articles pickled
35100  articles pickled
35200  articles pickled
35300  articles pickled
35400  articles pickled
35500  articles pickled
35600  articles pickled
35700  articles pickled
35800  articles pickled
35900  articles pickled
36000  articles pickled
36100  articles pickled
36200  articles pickled
36300  articles pickled
36400  articles pickled
36500  articles pickled
36600  articles pickled
36700  articles pickled
36800  articles pickled
36900  articles pickled
37000  articles pickled
37100  articles pickled
37200  articles pickled
37300  articles pickled
37400  articles pickled
37500  articles pickled
37600  articles pickled
37700  articles pickled
37800  articles pickled
37900  articles pickled
38000  articles pickled
38100  articles pickled
38200  articles pickled
38300  articles pickled
38400  articles pickled
38500  articles pickled
38600  articles pickled
38700  articles pickled
38800  articles pickled
38900  articles pickled
39000  articles pickled
39100  articles pickled
39200  articles pickled
39300  articles pickled
39400  articles pickled
39500  articles pickled
39600  articles pickled
39700  articles pickled
39800  articles pickled
39900  articles pickled
40000  articles pickled
40100  articles pickled
40200  articles pickled
40300  articles pickled
40400  articles pickled
40500  articles pickled
40600  articles pickled
40700  articles pickled
40800  articles pickled
40900  articles pickled
41000  articles pickled
41100  articles pickled
41200  articles pickled
41300  articles pickled
41400  articles pickled
41500  articles pickled
41600  articles pickled
41700  articles pickled
41800  articles pickled
41900  articles pickled
42000  articles pickled
42100  articles pickled
42200  articles pickled
42300  articles pickled
42400  articles pickled
42500  articles pickled
42600  articles pickled
42700  articles pickled
42800  articles pickled
42900  articles pickled
43000  articles pickled
43100  articles pickled
43200  articles pickled
43300  articles pickled
43400  articles pickled
43500  articles pickled
43600  articles pickled
43700  articles pickled
43800  articles pickled
43900  articles pickled
44000  articles pickled
44100  articles pickled
44200  articles pickled
44300  articles pickled
44400  articles pickled
44500  articles pickled
44600  articles pickled
44700  articles pickled
44800  articles pickled
44900  articles pickled
45000  articles pickled
45100  articles pickled
45200  articles pickled
45300  articles pickled
45400  articles pickled
45500  articles pickled
45600  articles pickled
45700  articles pickled
45800  articles pickled
45900  articles pickled
46000  articles pickled
46100  articles pickled
46200  articles pickled
46300  articles pickled
46400  articles pickled
46500  articles pickled
46600  articles pickled
46700  articles pickled
46800  articles pickled
46900  articles pickled
47000  articles pickled
47100  articles pickled
47200  articles pickled
47300  articles pickled
47400  articles pickled
47500  articles pickled
47600  articles pickled
47700  articles pickled
47800  articles pickled
47900  articles pickled
48000  articles pickled
48100  articles pickled
48200  articles pickled
48300  articles pickled
48400  articles pickled
48500  articles pickled
48600  articles pickled
48700  articles pickled
48800  articles pickled
48900  articles pickled
49000  articles pickled
49100  articles pickled
49200  articles pickled
49300  articles pickled
49400  articles pickled
49500  articles pickled
49600  articles pickled
49700  articles pickled
49800  articles pickled
49900  articles pickled
50000  articles pickled
50100  articles pickled
50200  articles pickled
50300  articles pickled
50400  articles pickled
50500  articles pickled
50600  articles pickled
50700  articles pickled
50800  articles pickled
50900  articles pickled
51000  articles pickled
51100  articles pickled
51200  articles pickled
51300  articles pickled
51400  articles pickled
51500  articles pickled
51600  articles pickled
51700  articles pickled
51800  articles pickled
51900  articles pickled
52000  articles pickled
52100  articles pickled
52200  articles pickled
52300  articles pickled
52400  articles pickled
52500  articles pickled
52600  articles pickled
52700  articles pickled
52800  articles pickled
52900  articles pickled
53000  articles pickled
53100  articles pickled
53200  articles pickled
53300  articles pickled
53400  articles pickled
53500  articles pickled
53600  articles pickled
53700  articles pickled
53800  articles pickled
53900  articles pickled
54000  articles pickled
54100  articles pickled
54200  articles pickled
54300  articles pickled
54400  articles pickled
54500  articles pickled
54600  articles pickled
54700  articles pickled
54800  articles pickled
54900  articles pickled
55000  articles pickled
55100  articles pickled
55200  articles pickled
55300  articles pickled
55400  articles pickled
55500  articles pickled
55600  articles pickled
55700  articles pickled
55800  articles pickled
55900  articles pickled
56000  articles pickled
56100  articles pickled
56200  articles pickled
56300  articles pickled
56400  articles pickled
56500  articles pickled
56600  articles pickled
56700  articles pickled
56800  articles pickled
56900  articles pickled
57000  articles pickled
57100  articles pickled
57200  articles pickled
57300  articles pickled
57400  articles pickled
57500  articles pickled
57600  articles pickled
57700  articles pickled
57800  articles pickled
57900  articles pickled
58000  articles pickled
58100  articles pickled
58200  articles pickled
58300  articles pickled
58400  articles pickled
58500  articles pickled
58600  articles pickled
58700  articles pickled
58800  articles pickled
58900  articles pickled
59000  articles pickled
59100  articles pickled
59200  articles pickled
59300  articles pickled
59400  articles pickled
59500  articles pickled
59600  articles pickled
59700  articles pickled
59800  articles pickled
59900  articles pickled
60000  articles pickled
60100  articles pickled
60200  articles pickled
60300  articles pickled
60400  articles pickled
60500  articles pickled
60600  articles pickled
60700  articles pickled
60800  articles pickled
60900  articles pickled
61000  articles pickled
61100  articles pickled
61200  articles pickled
61300  articles pickled
61400  articles pickled
61500  articles pickled
61600  articles pickled
61700  articles pickled
61800  articles pickled
61900  articles pickled
62000  articles pickled
62100  articles pickled
62200  articles pickled
62300  articles pickled
62400  articles pickled
62500  articles pickled
62600  articles pickled
62700  articles pickled
62800  articles pickled
62900  articles pickled
63000  articles pickled
63100  articles pickled
63200  articles pickled
63300  articles pickled
63400  articles pickled
63500  articles pickled
63600  articles pickled
63700  articles pickled
63800  articles pickled
63900  articles pickled
64000  articles pickled
64100  articles pickled
64200  articles pickled
64300  articles pickled
64400  articles pickled
64500  articles pickled
64600  articles pickled
64700  articles pickled
64800  articles pickled
64900  articles pickled
65000  articles pickled
65100  articles pickled
65200  articles pickled
65300  articles pickled
65400  articles pickled
65500  articles pickled
65600  articles pickled
65700  articles pickled
65800  articles pickled
65900  articles pickled
66000  articles pickled
66100  articles pickled
66200  articles pickled
66300  articles pickled
66400  articles pickled
66500  articles pickled
66600  articles pickled
66700  articles pickled
66800  articles pickled
66900  articles pickled
67000  articles pickled
67100  articles pickled
67200  articles pickled
67300  articles pickled
67400  articles pickled
67500  articles pickled
67600  articles pickled
67700  articles pickled
67800  articles pickled
67900  articles pickled
68000  articles pickled
68100  articles pickled
68200  articles pickled
68300  articles pickled
68400  articles pickled
68500  articles pickled
68600  articles pickled
68700  articles pickled
68800  articles pickled
68900  articles pickled
69000  articles pickled
69100  articles pickled
69200  articles pickled
69300  articles pickled
69400  articles pickled
69500  articles pickled
69600  articles pickled
69700  articles pickled
69800  articles pickled
69900  articles pickled
70000  articles pickled
70100  articles pickled
70200  articles pickled
70300  articles pickled
70400  articles pickled
70500  articles pickled
70600  articles pickled
70700  articles pickled
70800  articles pickled
70900  articles pickled
71000  articles pickled
71100  articles pickled
71200  articles pickled
71300  articles pickled
71400  articles pickled
71500  articles pickled
71600  articles pickled
71700  articles pickled
71800  articles pickled
71900  articles pickled
72000  articles pickled
72100  articles pickled
72200  articles pickled
72300  articles pickled
72400  articles pickled
72500  articles pickled
72600  articles pickled
72700  articles pickled
72800  articles pickled
72900  articles pickled
73000  articles pickled
73100  articles pickled
73200  articles pickled
73300  articles pickled
73400  articles pickled
73500  articles pickled
73600  articles pickled
73700  articles pickled
73800  articles pickled
73900  articles pickled
74000  articles pickled
74100  articles pickled
74200  articles pickled
74300  articles pickled
74400  articles pickled
74500  articles pickled
74600  articles pickled
74700  articles pickled
74800  articles pickled
74900  articles pickled
75000  articles pickled
75100  articles pickled
75200  articles pickled
75300  articles pickled
75400  articles pickled
75500  articles pickled
75600  articles pickled
75700  articles pickled
75800  articles pickled
75900  articles pickled
76000  articles pickled
76100  articles pickled
76200  articles pickled
76300  articles pickled
76400  articles pickled
76500  articles pickled
76600  articles pickled
76700  articles pickled
76800  articles pickled
76900  articles pickled
77000  articles pickled
77100  articles pickled
77200  articles pickled
77300  articles pickled
77400  articles pickled
77500  articles pickled
77600  articles pickled
77700  articles pickled
77800  articles pickled
77900  articles pickled
78000  articles pickled
78100  articles pickled
78200  articles pickled
78300  articles pickled
78400  articles pickled
78500  articles pickled
78600  articles pickled
78700  articles pickled
78800  articles pickled
78900  articles pickled
79000  articles pickled
79100  articles pickled
79200  articles pickled
79300  articles pickled
79400  articles pickled
79500  articles pickled
79600  articles pickled
79700  articles pickled
79800  articles pickled
79900  articles pickled
80000  articles pickled
80100  articles pickled
80200  articles pickled
80300  articles pickled
80400  articles pickled
80500  articles pickled
80600  articles pickled
80700  articles pickled
80800  articles pickled
80900  articles pickled
81000  articles pickled
81100  articles pickled
81200  articles pickled
81300  articles pickled
81400  articles pickled
81500  articles pickled
81600  articles pickled
81700  articles pickled
81800  articles pickled
81900  articles pickled
82000  articles pickled
82100  articles pickled
82200  articles pickled
82300  articles pickled
82400  articles pickled
82500  articles pickled
82600  articles pickled
82700  articles pickled
82800  articles pickled
82900  articles pickled
83000  articles pickled
83100  articles pickled
83200  articles pickled
83300  articles pickled
83400  articles pickled
83500  articles pickled
83600  articles pickled
83700  articles pickled
83800  articles pickled
83900  articles pickled
84000  articles pickled
84100  articles pickled
84200  articles pickled
84300  articles pickled
84400  articles pickled
84500  articles pickled
84600  articles pickled
84700  articles pickled
84800  articles pickled
84900  articles pickled
85000  articles pickled
85100  articles pickled
85200  articles pickled
85300  articles pickled
85400  articles pickled
85500  articles pickled
85600  articles pickled
85700  articles pickled
85800  articles pickled
85900  articles pickled
86000  articles pickled
86100  articles pickled
86200  articles pickled
86300  articles pickled
86400  articles pickled
86500  articles pickled
86600  articles pickled
86700  articles pickled
86800  articles pickled
86900  articles pickled
87000  articles pickled
87100  articles pickled
87200  articles pickled
87300  articles pickled
87400  articles pickled
87500  articles pickled
87600  articles pickled
87700  articles pickled
87800  articles pickled
87900  articles pickled
88000  articles pickled
88100  articles pickled
88200  articles pickled
88300  articles pickled
88400  articles pickled
88500  articles pickled
88600  articles pickled
88700  articles pickled
88800  articles pickled
88900  articles pickled
89000  articles pickled
89100  articles pickled
89200  articles pickled
89300  articles pickled
89400  articles pickled
89500  articles pickled
89600  articles pickled
89700  articles pickled
89800  articles pickled
89900  articles pickled
90000  articles pickled
90100  articles pickled
90200  articles pickled
90300  articles pickled
90400  articles pickled
90500  articles pickled
90600  articles pickled
90700  articles pickled
90800  articles pickled
90900  articles pickled
91000  articles pickled
91100  articles pickled
91200  articles pickled
91300  articles pickled
91400  articles pickled
91500  articles pickled
91600  articles pickled
91700  articles pickled
91800  articles pickled
91900  articles pickled
92000  articles pickled
92100  articles pickled
92200  articles pickled
92300  articles pickled
92400  articles pickled
92500  articles pickled
92600  articles pickled
92700  articles pickled
92800  articles pickled
92900  articles pickled
93000  articles pickled
93100  articles pickled
93200  articles pickled
93300  articles pickled
93400  articles pickled
93500  articles pickled
93600  articles pickled
93700  articles pickled
93800  articles pickled
93900  articles pickled
94000  articles pickled
94100  articles pickled
94200  articles pickled
94300  articles pickled
94400  articles pickled
94500  articles pickled
94600  articles pickled
94700  articles pickled
94800  articles pickled
94900  articles pickled
95000  articles pickled
95100  articles pickled
95200  articles pickled
95300  articles pickled
95400  articles pickled
95500  articles pickled
95600  articles pickled
95700  articles pickled
95800  articles pickled
95900  articles pickled
96000  articles pickled
96100  articles pickled
96200  articles pickled
96300  articles pickled
96400  articles pickled
96500  articles pickled
96600  articles pickled
96700  articles pickled
96800  articles pickled
96900  articles pickled
97000  articles pickled
97100  articles pickled
97200  articles pickled
97300  articles pickled
97400  articles pickled
97500  articles pickled
97600  articles pickled
97700  articles pickled
97800  articles pickled
97900  articles pickled
98000  articles pickled
98100  articles pickled
98200  articles pickled
98300  articles pickled
98400  articles pickled
98500  articles pickled
98600  articles pickled
98700  articles pickled
98800  articles pickled
98900  articles pickled
99000  articles pickled
99100  articles pickled
99200  articles pickled
99300  articles pickled
99400  articles pickled
99500  articles pickled
99600  articles pickled
99700  articles pickled
99800  articles pickled
99900  articles pickled
100000  articles pickled
100100  articles pickled
100200  articles pickled
100300  articles pickled
100400  articles pickled
100500  articles pickled
100600  articles pickled
100700  articles pickled
100800  articles pickled
100900  articles pickled
101000  articles pickled
101100  articles pickled
101200  articles pickled
101300  articles pickled
101400  articles pickled
101500  articles pickled
101600  articles pickled
101700  articles pickled
101800  articles pickled
101900  articles pickled
102000  articles pickled
102100  articles pickled
102200  articles pickled
102300  articles pickled
102400  articles pickled
102500  articles pickled
102600  articles pickled
102700  articles pickled
102800  articles pickled
102900  articles pickled
103000  articles pickled
103100  articles pickled
103200  articles pickled
103300  articles pickled
103400  articles pickled
103500  articles pickled
103600  articles pickled
103700  articles pickled
103800  articles pickled
103900  articles pickled
104000  articles pickled
104100  articles pickled
104200  articles pickled
104300  articles pickled
104400  articles pickled
104500  articles pickled
104600  articles pickled
104700  articles pickled
104800  articles pickled
104900  articles pickled
105000  articles pickled
105100  articles pickled
105200  articles pickled
105300  articles pickled
105400  articles pickled
105500  articles pickled
105600  articles pickled
105700  articles pickled
105800  articles pickled
105900  articles pickled
106000  articles pickled
106100  articles pickled
106200  articles pickled
106300  articles pickled
106400  articles pickled
106500  articles pickled
106600  articles pickled
106700  articles pickled
106800  articles pickled
106900  articles pickled
107000  articles pickled
107100  articles pickled
107200  articles pickled
107300  articles pickled
107400  articles pickled
107500  articles pickled
107600  articles pickled
107700  articles pickled
107800  articles pickled
107900  articles pickled
108000  articles pickled
108100  articles pickled
108200  articles pickled
108300  articles pickled
108400  articles pickled
108500  articles pickled
108600  articles pickled
108700  articles pickled
108800  articles pickled
108900  articles pickled
109000  articles pickled
109100  articles pickled
109200  articles pickled
109300  articles pickled
109400  articles pickled
109500  articles pickled
109600  articles pickled
109700  articles pickled
109800  articles pickled
109900  articles pickled
110000  articles pickled
110100  articles pickled
110200  articles pickled
110300  articles pickled
110400  articles pickled
110500  articles pickled
110600  articles pickled
110700  articles pickled
110800  articles pickled
110900  articles pickled
111000  articles pickled
111100  articles pickled
111200  articles pickled
111300  articles pickled
111400  articles pickled
111500  articles pickled
111600  articles pickled
111700  articles pickled
111800  articles pickled
111900  articles pickled
112000  articles pickled
112100  articles pickled
112200  articles pickled
112300  articles pickled
112400  articles pickled
112500  articles pickled
112600  articles pickled
112700  articles pickled
112800  articles pickled
112900  articles pickled
113000  articles pickled
113100  articles pickled
113200  articles pickled
113300  articles pickled
113400  articles pickled
113500  articles pickled
113600  articles pickled
113700  articles pickled
113800  articles pickled
113900  articles pickled
114000  articles pickled
114100  articles pickled
114200  articles pickled
114300  articles pickled
114400  articles pickled
114500  articles pickled
114600  articles pickled
114700  articles pickled
114800  articles pickled
114900  articles pickled
115000  articles pickled
115100  articles pickled
115200  articles pickled
115300  articles pickled
115400  articles pickled
115500  articles pickled
115600  articles pickled
115700  articles pickled
115800  articles pickled
115900  articles pickled
116000  articles pickled
116100  articles pickled
116200  articles pickled
116300  articles pickled
116400  articles pickled
116500  articles pickled
116600  articles pickled
116700  articles pickled
116800  articles pickled
116900  articles pickled
117000  articles pickled
117100  articles pickled
117200  articles pickled
117300  articles pickled
117400  articles pickled
117500  articles pickled
117600  articles pickled
117700  articles pickled
117800  articles pickled
117900  articles pickled
118000  articles pickled
118100  articles pickled
118200  articles pickled
118300  articles pickled
118400  articles pickled
118500  articles pickled
118600  articles pickled
118700  articles pickled
118800  articles pickled
118900  articles pickled
119000  articles pickled
119100  articles pickled
119200  articles pickled
119300  articles pickled
119400  articles pickled
119500  articles pickled
119600  articles pickled
119700  articles pickled
119800  articles pickled
119900  articles pickled
120000  articles pickled
120100  articles pickled
120200  articles pickled
120300  articles pickled
120400  articles pickled
120500  articles pickled
120600  articles pickled
120700  articles pickled
120800  articles pickled
120900  articles pickled
121000  articles pickled
121100  articles pickled
121200  articles pickled
121300  articles pickled
121400  articles pickled
121500  articles pickled
121600  articles pickled
121700  articles pickled
121800  articles pickled
121900  articles pickled
122000  articles pickled
122100  articles pickled
122200  articles pickled
122300  articles pickled
122400  articles pickled
122500  articles pickled
122600  articles pickled
122700  articles pickled
122800  articles pickled
122900  articles pickled
123000  articles pickled
123100  articles pickled
123200  articles pickled
123300  articles pickled
123400  articles pickled
123500  articles pickled
123600  articles pickled
123700  articles pickled
123800  articles pickled
123900  articles pickled
124000  articles pickled
124100  articles pickled
124200  articles pickled
124300  articles pickled
124400  articles pickled
124500  articles pickled
124600  articles pickled
124700  articles pickled
124800  articles pickled
124900  articles pickled
125000  articles pickled
125100  articles pickled
125200  articles pickled
125300  articles pickled
125400  articles pickled
125500  articles pickled
125600  articles pickled
125700  articles pickled
125800  articles pickled
125900  articles pickled
126000  articles pickled
126100  articles pickled
126200  articles pickled
126300  articles pickled
126400  articles pickled
126500  articles pickled
126600  articles pickled
126700  articles pickled
126800  articles pickled
DONE!

Exploring Output



In [7]:

    
#concatenates pickled files

pieces = []

for i in range(0,1267): # replace end of range(0,x) with the number of the last file!
    path = '../data/pickles/plos_pickle_{0}.pkl'.format(i)
    frame = pd.read_pickle(path)
    
    pieces.append(frame)

all_plos_df_for_export = pd.concat(pieces)



In [8]:

    
all_plos_df_for_export.to_pickle('../data/all_plos_df.pkl')



In [9]:

    
all_plos_df = pd.read_pickle('../data/all_plos_df.pkl')



In [10]:

    
# total number of articles (originally 115489)
len(all_plos_df)









    Out[10]:





121450



In [11]:

    
all_plos_df.sort(columns = 'id').head()









    Out[11]:






  
    
      
      abstract
      author
      id
      journal
      publication_date
      score
      subject
      title_display
    
  
  
    
      27
       [\n        During lymphocyte development, V(D)...
       [Alfred Ian Lee, Sebastian D Fugmann, Lindsay ...
       10.1371/journal.pbio.0000001
       PLoS Biology
       2003-10-13T00:00:00Z
       1
       [/Biology and life sciences/Molecular biology/...
       A Functional Analysis of the Spacer of V(D)J R...
    
    
      59
       [\n        Because of the constant threat pose...
       [David Wang, Anatoly Urisman, Yu-Tsueng Liu, M...
       10.1371/journal.pbio.0000002
       PLoS Biology
       2003-11-17T00:00:00Z
       1
       [/Biology and life sciences/Molecular biology/...
       Viral Discovery and Sequence Recovery Using DN...
    
    
      65
                                                      []
                                                     NaN
       10.1371/journal.pbio.0000003
       PLoS Biology
       2003-11-17T00:00:00Z
       1
       [/Biology and life sciences/Organisms/Viruses/...
                           Gene Chip for Viral Discovery
    
    
      51
                                                      []
                                                     NaN
       10.1371/journal.pbio.0000004
       PLoS Biology
       2003-10-13T00:00:00Z
       1
       [/Medicine and health sciences/Immunology/Immu...
                      Functional Analysis of RSS Spacers
    
    
      69
       [\n        Plasmodium falciparum is the causat...
       [Zbynek Bozdech, Manuel Llinás, Brian Lee Pull...
       10.1371/journal.pbio.0000005
       PLoS Biology
       2003-08-18T00:00:00Z
       1
       [/Biology and life sciences/Biochemistry/Prote...
       The Transcriptome of the Intraerythrocytic Dev...
    
  

5 rows × 8 columns



In [12]:

    
len(list(all_plos_df.id))









    Out[12]:





121450



In [13]:

    
print list(all_plos_df.subject)[0]









    



[u'/Medicine and health sciences/Pathology and laboratory medicine/Signs and symptoms/Fatigue', u'/Biology and life sciences/Anatomy/Musculoskeletal system/Muscles/Muscle fibers', u'/Medicine and health sciences/Physiology/Muscle physiology/Muscle contraction', u'/Biology and life sciences/Cell biology/Cellular types/Animal cells/Muscle fibers', u'/Biology and life sciences/Physiology/Muscle physiology/Muscle functions', u'/Medicine and health sciences/Anatomy/Musculoskeletal system/Muscles/Skeletal muscles', u'/Biology and life sciences/Anatomy/Musculoskeletal system/Muscles/Skeletal muscles', u'/Medicine and health sciences/Physiology/Muscle physiology/Muscle functions', u'/Biology and life sciences/Cell biology/Cellular types/Animal cells/Muscle fibers/Skeletal muscle fibers', u'/Biology and life sciences/Anatomy/Musculoskeletal system/Muscles/Muscle fibers/Skeletal muscle fibers', u'/Biology and life sciences/Physiology/Physiological processes/Homeostasis', u'/Biology and life sciences/Physiology/Muscle physiology/Muscle contraction', u'/Medicine and health sciences/Physiology/Physiological processes/Homeostasis', u'/Research and analysis methods/Imaging techniques/Fluorescence imaging', u'/Medicine and health sciences/Anatomy/Musculoskeletal system/Muscles/Muscle fibers', u'/Medicine and health sciences/Anatomy/Musculoskeletal system/Muscles/Muscle fibers/Skeletal muscle fibers']



In [14]:

    
all_plos_df.tail()









    Out[14]:






  
    
      
      abstract
      author
      id
      journal
      publication_date
      score
      subject
      title_display
    
  
  
    
      45
       [Background: Besides having an impact on human...
       [Anna Puig-Oliveras, Yuliaxis Ramayo-Caldas, J...
       10.1371/journal.pone.0099720
       PLoS ONE
       2014-06-13T00:00:00Z
       1
       [/Biology and life sciences/Biochemistry/Lipid...
       Differences in Muscle Transcriptome among Pigs...
    
    
      46
       [\nThe nuclear receptors and xenosensors const...
       [Marianne Mathäs, Christian Nußhag , Oliver Bu...
       10.1371/journal.pone.0096263
       PLoS ONE
       2014-05-05T00:00:00Z
       1
       [/Biology and life sciences/Biochemistry/Prote...
       Structural and Functional Similarity of Amphib...
    
    
      47
       [Background: Body dissatisfaction is the most ...
       [Blanca Ortega-Roldán, Sonia Rodríguez-Ruiz, P...
       10.1371/journal.pone.0102595
       PLoS ONE
       2014-07-18T00:00:00Z
       1
       [/Medicine and health sciences/Anatomy/Integum...
       The Emotional and Attentional Impact of Exposu...
    
    
      48
       [Objective: Maternal mortality (MM) is a core ...
       [Juliana C Giordano, Mary A Parpinelli, Jose G...
       10.1371/journal.pone.0097401
       PLoS ONE
       2014-05-13T00:00:00Z
       1
       [/Medicine and health sciences/Women's health/...
       The Burden of Eclampsia: Results from a Multic...
    
    
      49
       [\nOur previous studies showed positive correl...
       [Mohammed M H Al-Gayyar, Barbara A Mysona, Sur...
       10.1371/journal.pone.0054692
       PLoS ONE
       2013-01-24T00:00:00Z
       1
       [/Biology and life sciences/Biochemistry/Enzym...
       Diabetes and Overexpression of proNGF Cause Re...
    
  

5 rows × 8 columns

Cleaning Output

Testing retry decorator

This is making sure that the retry decorator works



In [15]:

    
import random

@retry
def do_something_unreliable():
    if random.randint(0, 2) > 1:
        raise IOError("Broken sauce, everything is hosed!!!111one")
    else:
        return "Awesome sauce!"

print do_something_unreliable()









    



Awesome sauce!

	abstract	author	id	journal	publication_date	score	subject	title_display
27	[\n During lymphocyte development, V(D)...	[Alfred Ian Lee, Sebastian D Fugmann, Lindsay ...	10.1371/journal.pbio.0000001	PLoS Biology	2003-10-13T00:00:00Z	1	[/Biology and life sciences/Molecular biology/...	A Functional Analysis of the Spacer of V(D)J R...
59	[\n Because of the constant threat pose...	[David Wang, Anatoly Urisman, Yu-Tsueng Liu, M...	10.1371/journal.pbio.0000002	PLoS Biology	2003-11-17T00:00:00Z	1	[/Biology and life sciences/Molecular biology/...	Viral Discovery and Sequence Recovery Using DN...
65	[]	NaN	10.1371/journal.pbio.0000003	PLoS Biology	2003-11-17T00:00:00Z	1	[/Biology and life sciences/Organisms/Viruses/...	Gene Chip for Viral Discovery
51	[]	NaN	10.1371/journal.pbio.0000004	PLoS Biology	2003-10-13T00:00:00Z	1	[/Medicine and health sciences/Immunology/Immu...	Functional Analysis of RSS Spacers
69	[\n Plasmodium falciparum is the causat...	[Zbynek Bozdech, Manuel Llinás, Brian Lee Pull...	10.1371/journal.pbio.0000005	PLoS Biology	2003-08-18T00:00:00Z	1	[/Biology and life sciences/Biochemistry/Prote...	The Transcriptome of the Intraerythrocytic Dev...

	abstract	author	id	journal	publication_date	score	subject	title_display
45	[Background: Besides having an impact on human...	[Anna Puig-Oliveras, Yuliaxis Ramayo-Caldas, J...	10.1371/journal.pone.0099720	PLoS ONE	2014-06-13T00:00:00Z	1	[/Biology and life sciences/Biochemistry/Lipid...	Differences in Muscle Transcriptome among Pigs...
46	[\nThe nuclear receptors and xenosensors const...	[Marianne Mathäs, Christian Nußhag , Oliver Bu...	10.1371/journal.pone.0096263	PLoS ONE	2014-05-05T00:00:00Z	1	[/Biology and life sciences/Biochemistry/Prote...	Structural and Functional Similarity of Amphib...
47	[Background: Body dissatisfaction is the most ...	[Blanca Ortega-Roldán, Sonia Rodríguez-Ruiz, P...	10.1371/journal.pone.0102595	PLoS ONE	2014-07-18T00:00:00Z	1	[/Medicine and health sciences/Anatomy/Integum...	The Emotional and Attentional Impact of Exposu...
48	[Objective: Maternal mortality (MM) is a core ...	[Juliana C Giordano, Mary A Parpinelli, Jose G...	10.1371/journal.pone.0097401	PLoS ONE	2014-05-13T00:00:00Z	1	[/Medicine and health sciences/Women's health/...	The Burden of Eclampsia: Results from a Multic...
49	[\nOur previous studies showed positive correl...	[Mohammed M H Al-Gayyar, Barbara A Mysona, Sur...	10.1371/journal.pone.0054692	PLoS ONE	2013-01-24T00:00:00Z	1	[/Biology and life sciences/Biochemistry/Enzym...	Diabetes and Overexpression of proNGF Cause Re...