GitHub - Data Extraction

The file ../data/RPackage-Repositories-150101-150601.csv contains a list of GitHub repositories that are candidates to store a package related to R. Those candidates were collected from the activity on GitHub between 15-01 and 15-06. Those candidates all contain a DESCRIPTION file at the root of the repository.

We git clone-ed each of those repository. This notebook will parse those git repositories and extract the DESCRIPTION file of each commit.


In [9]:
import pandas
from datetime import date

We will make use of the following commands:

  • git clone <url> <path> where is the url of the repository and is the location to store the repository.
  • git log --follow --format="%H/%ci" <path> where will be DESCRIPTION. The output of this command is a list of / for this file.
  • git show <commit>:<path> where is the considered commit, and will be DESCRIPTION. This command outputs the content of the file at the given commit.

In [10]:
github = pandas.DataFrame.from_csv('../data/RPackage-Repositories-150101-150601.csv')
repositories = github[['owner.login', 'name']].rename(columns={'owner.login': 'owner', 'name': 'repositories'})

FILENAME = '../data/github-raw-150601.csv'

# Root of the directory where the repositories were collected
GIT_DIR = '/data/github/'

We will retrieve a lot of data, we can benefit from IPython's parallel computation tool.

To use this notebook, you need either to configure your IPController or to start a cluster of IPython nodes, using ipcluster start -n 4 for example. See https://ipython.org/ipython-doc/dev/parallel/parallel_process.html for more information.

It seems that most recent versions of IPython Notebook can directly start cluster from the web interface, under the Cluster tab.


In [11]:
from IPython import parallel
clients = parallel.Client()
clients.block = False # asynchronous computations
print 'Clients:', str(clients.ids)


Clients: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]

In [12]:
def get_data_from((owner, repository)):
    # Move to target directory
    try:
        os.chdir(os.path.join(GIT_DIR, owner, repository))
    except OSError as e: 
        # Should happen when directory does not exist
        return []
    
    data_list = []
    
    # Get commits for DESCRIPTION
    try:
        commits = subprocess.check_output(['git', 'log', '--format=%H/%ci', '--', 'DESCRIPTION'])
    except subprocess.CalledProcessError as e:
        # Should not happen!?
        raise Exception(owner + ' ' + repository + '/ log : ' + e.output)
        
    for commit in [x for x in commits.split('\n') if len(x.strip())!=0]:
        commit_sha, date = map(lambda x: x.strip(), commit.split('/'))
        
        # Get file content
        try:
            content = subprocess.check_output(['git', 'show', '{id}:{path}'.format(id=commit_sha, path='DESCRIPTION')])
        except subprocess.CalledProcessError as e:
            # Could happen when DESCRIPTION was added in this commit. Silently ignore
            continue
        
        try:
            metadata = deb822.Deb822(content.split('\n'))
        except Exception as e: 
            # I don't known which are the exceptions that Deb822 may throw!
            continue # Go further
            
        data = {}
        
        for md in ['Package', 'Version', 'License', 'Imports', 'Suggests', 'Depends', 'Author', 'Authors', 'Maintainer']:
            data[md] = metadata.get(md, '')
        
        data['CommitDate'] = date
        data['Owner'] = owner
        data['Repository'] = repository
        data_list.append(data)

    # Return to root directory
    os.chdir(GIT_DIR)
    return data_list

In [13]:
data = []

clients[:].execute('import subprocess, os')
clients[:].execute('from debian import deb822')
clients[:]['GIT_DIR'] = GIT_DIR

balanced = clients.load_balanced_view()

items = [(owner, repo) for idx, (owner, repo) in repositories.iterrows()]

print len(items), 'items'
    
res = balanced.map(get_data_from, items, ordered=False, timeout=15)

import time
while not res.ready():
    time.sleep(5)
    print res.progress, ' ', 
    
for result in res.result:
    data.extend(result)


6533 items
818   1229   1865   2408   2983   3536   4123   4685   5212   5825   6406   6533  

In [14]:
df = pandas.DataFrame.from_records(data)
df.to_csv(FILENAME, encoding='utf-8')
print len(df), 'items'
print len(df.drop_duplicates(['Package'])), 'packages'
print len(df.drop_duplicates(['Owner', 'Repository'])), 'repositories'
print len(df.drop_duplicates(['Package', 'Version'])), 'pairs (package, version)'


101208 items
6293 packages
6513 repositories
68833 pairs (package, version)

In [15]:
df


Out[15]:
Author Authors CommitDate Depends Imports License Maintainer Owner Package Repository Suggests Version
0 Yann Ruffieux, contributions from Debjani Bhow... 2015-04-16 20:02:44 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.35.0
1 Yann Ruffieux, contributions from Debjani Bhow... 2015-04-16 19:42:01 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.34.0
2 Yann Ruffieux, contributions from Debjani Bhow... 2014-10-13 21:47:41 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.33.0
3 Yann Ruffieux, contributions from Debjani Bhow... 2014-10-13 21:38:33 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.32.0
4 Yann Ruffieux, contributions from Debjani Bhow... 2014-04-11 21:21:21 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.31.0
5 Yann Ruffieux, contributions from Debjani Bhow... 2014-04-11 21:07:21 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.30.0
6 Yann Ruffieux, contributions from Debjani Bhow... 2014-03-04 22:12:21 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.29.2
7 Yann Ruffieux, contributions from Debjani Bhow... 2013-10-18 19:59:46 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.29.1
8 Yann Ruffieux, contributions from Debjani Bhow... 2013-10-14 21:47:19 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.29.0
9 Yann Ruffieux, contributions from Debjani Bhow... 2013-10-14 21:29:21 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.28.0
10 Yann Ruffieux, contributions from Debjani Bhow... 2013-04-03 22:06:11 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.27.0
11 Yann Ruffieux, contributions from Debjani Bhow... 2013-04-03 21:32:27 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.26.0
12 Yann Ruffieux, contributions from Debjani Bhow... 2012-10-01 22:43:56 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.25.0
13 Yann Ruffieux, contributions from Debjani Bhow... 2012-10-01 22:16:24 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.24.0
14 Yann Ruffieux, contributions from Debjani Bhow... 2012-03-30 22:15:56 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.23.0
15 Yann Ruffieux, contributions from Debjani Bhow... 2012-03-30 22:05:02 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.22.0
16 Yann Ruffieux, contributions from Debjani Bhow... 2011-10-31 23:12:28 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.21.0
17 Yann Ruffieux, contributions from Debjani Bhow... 2011-10-31 22:59:03 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.20.0
18 Yann Ruffieux, contributions from Debjani Bhow... 2011-04-13 22:36:43 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.19.0
19 Yann Ruffieux, contributions from Debjani Bhow... 2011-04-13 22:27:35 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.18.0
20 Yann Ruffieux, contributions from Debjani Bhow... 2010-10-18 05:57:44 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.17.0
21 Yann Ruffieux, contributions from Debjani Bhow... 2010-10-18 05:34:23 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.16.0
22 Yann Ruffieux, contributions from Debjani Bhow... 2010-04-22 18:45:15 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.15.0
23 Yann Ruffieux, contributions from Debjani Bhow... 2010-04-22 18:21:23 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.14.0
24 Yann Ruffieux, contributions from Debjani Bhow... 2009-10-27 23:45:24 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.13.0
25 Yann Ruffieux, contributions from Debjani Bhow... 2009-10-27 23:33:29 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.12.0
26 Yann Ruffieux, contributions from Debjani Bhow... 2009-09-06 03:06:55 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL (>= 2) Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.11.1
27 Yann Ruffieux, contributions from Debjani Bhow... 2009-04-20 23:44:07 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL version 2 or newer Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.11.0
28 Yann Ruffieux, contributions from Debjani Bhow... 2009-04-20 23:24:12 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL version 2 or newer Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.10.0
29 Yann Ruffieux, contributions from Debjani Bhow... 2009-03-24 23:31:29 +0000 R (>= 2.6.0),stats Biobase, graphics, grDevices, methods, stats, ... GPL version 2 or newer Yann Ruffieux <yann.ruffieux@epfl.ch> Bioconductor-mirror lapmix lapmix 1.9.2
... ... ... ... ... ... ... ... ... ... ... ... ...
101178 Kevin M. Middleton <middletonk@missouri.edu>, ... 2012-10-01 09:56:56 -0500 \n R (>= 2.10),\n nlme,\n lattice,\n ... GPL-2 Kevin M. Middleton <middletonk@missouri.edu> kmiddleton abd abd \n boot,\n car,\n ggplot2,\n plyr,... 0.2-5
101179 Kevin M. Middleton <middletonk@missouri.edu>, ... 2012-07-29 09:10:20 -0500 \n R (>= 2.10),\n nlme,\n lattice,\n ... GPL-2 Kevin M. Middleton <middletonk@missouri.edu> kmiddleton abd abd \n boot,\n car,\n ggplot2,\n plyr,... 0.2-4
101180 Kevin M. Middleton <middletonk@missouri.edu>, ... 2012-07-28 16:41:35 -0500 \n R (>= 2.10),\n nlme,\n lattice,\n ... GPL-2 Kevin M. Middleton <middletonk@missouri.edu> kmiddleton abd abd \n boot,\n car,\n ggplot2,\n plyr,... 0.2-3
101181 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2012-07-28 16:35:37 -0500 \n R (>= 2.10),\n nlme,\n lattice,\n ... GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd \n boot,\n car,\n ggplot2,\n plyr,... 0.2-3
101182 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2012-04-03 22:50:37 -0700 \n R (>= 2.10),\n nlme,\n lattice,\n ... GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd \n boot,\n car,\n ggplot2,\n plyr,... 0.2-2
101183 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2012-03-29 17:03:25 -0400 \n R (>= 2.10),\n nlme,\n lattice,\n ... GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd \n boot,\n car,\n ggplot2,\n plyr,... 0.2-1
101184 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2012-03-29 16:57:22 -0400 \n R (>= 2.10),\n nlme,\n lattice,\n ... GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd \n boot,\n car,\n ggplot2,\n plyr,... 0.2-1
101185 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2012-03-09 12:58:13 -0800 \n R (>= 2.10),\n nlme,\n lattice,\n ... GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd \n boot,\n car,\n ggplot2,\n plyr,... 0.1-24
101186 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2012-02-16 11:13:37 -0500 \n R (>= 2.10),\n nlme,\n lattice,\n ... GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd \n boot,\n car,\n ggplot2,\n plyr,... 0.1-24
101187 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2012-02-16 11:08:29 -0500 \n R (>= 2.10),\n nlme,\n lattice,\n ... GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd \n boot,\n car,\n ggplot2,\n plyr,... 0.1-24
101188 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2012-02-16 00:22:31 -0800 \n R (>= 2.10),\n nlme,\n lattice,\n ... GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd \n boot,\n car,\n ggplot2,\n plyr,... 0.1-24
101189 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2012-02-16 02:11:10 -0500 \n R (>= 2.10),\n nlme,\n lattice,\n ... GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd \n boot,\n car,\n ggplot2,\n plyr,... 0.1-24
101190 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2012-02-16 01:07:42 -0500 \n R (>= 2.10),\n nlme,\n lattice,\n ... GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd \n boot,\n car,\n ggplot2,\n plyr,... 0.1-24
101191 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2012-02-16 00:28:23 -0500 R (>= 2.10), nlme, lattice, grid, mosaic GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd boot, car, ggplot2, plyr, HH, vcd, Hmisc 0.1-24
101192 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2012-02-16 00:10:41 -0500 R (>= 2.10), nlme, lattice, grid, mosaic GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd boot, car, ggplot2, plyr, HH, vcd, Hmisc 0.1-24
101193 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2012-02-15 20:51:05 -0800 R (>= 2.10), nlme, lattice, grid GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd boot, car, ggplot2, plyr, HH, vcd, Hmisc 0.1-24
101194 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2011-10-21 11:33:38 -0700 R (>= 2.10), nlme, lattice, grid GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd boot, car, ggplot2, plyr, HH, vcd, Hmisc 0.1-23
101195 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2011-09-12 17:28:25 -0700 R (>= 2.10), nlme, lattice, grid GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd boot, car, ggplot2, plyr, HH, vcd, Hmisc 0.1-22
101196 Kevin M. Middleton <kmm@csusb.edu>, Randall Pr... 2011-08-07 17:52:52 -0700 R (>= 2.10), nlme, lattice, grid GPL-2 Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd boot, car, ggplot2, HH, plyr, vcd, Hmisc 0.1-18
101197 Kevin M. Middleton 2009-12-07 01:26:16 +0000 GPL Kevin M. Middleton <kmm@csusb.edu> kmiddleton abd abd 0.1
101198 Lars Kotthoff [aut,cre],\n Bernd Bischl [au... 2015-02-03 00:00:00 +0000 R (>= 2.10), mlr (>= 2.1) rJava, parallelMap, ggplot2, checkmate BSD_3_clause + file LICENSE Lars Kotthoff <lars.kotthoff@insight-centre.org> cran llama llama testthat, ParamHelpers 0.8.1
101199 Lars Kotthoff [aut,cre],\n Bernd Bischl [au... 2015-01-31 00:00:00 +0000 R (>= 2.10), mlr (>= 2.1) rJava, parallelMap, ggplot2, checkmate BSD_3_clause + file LICENSE Lars Kotthoff <lars.kotthoff@insight-centre.org> cran llama llama testthat, ParamHelpers 0.8
101200 Lars Kotthoff [aut,cre],\n Bernd Bischl [au... 2014-09-20 00:00:00 +0000 R (>= 2.10), mlr (>= 2.1) plyr, rJava, parallelMap BSD_3_clause + file LICENSE Lars Kotthoff <lars.kotthoff@insight-centre.org> cran llama llama FSelector, testthat, ParamHelpers 0.7.2
101201 Lars Kotthoff [aut,cre],\n Bernd Bischl [au... 2014-08-13 00:00:00 +0000 R (>= 2.10), mlr (>= 2.1) plyr, rJava, parallelMap BSD_3_clause + file LICENSE Lars Kotthoff <lars.kotthoff@insight-centre.org> cran llama llama FSelector, testthat, ParamHelpers 0.7.1
101202 Lars Kotthoff [aut,cre],\n Bernd Bischl [au... 2014-07-22 00:00:00 +0000 R (>= 2.10), mlr (>= 2.1) plyr, rJava, parallelMap BSD_3_clause + file LICENSE Lars Kotthoff <lars.kotthoff@insight-centre.org> cran llama llama FSelector, testthat, ParamHelpers 0.7
101203 Lars Kotthoff [aut, cre],\n Barry Hurley [c... 2014-04-29 00:00:00 +0000 R (>= 2.10) plyr, rJava, parallelMap BSD_3_clause + file LICENSE Lars Kotthoff <larsko@4c.ucc.ie> cran llama llama RWeka, FSelector, e1071, flexclust, testthat 0.6
101204 Lars Kotthoff [aut, cre],\n Barry Hurley [c... 2014-01-15 00:00:00 +0000 plyr, rJava, parallelMap BSD_3_clause + file LICENSE Lars Kotthoff <larsko@4c.ucc.ie> cran llama llama RWeka, FSelector, e1071, flexclust, testthat 0.5.1
101205 Lars Kotthoff, contributions by Barry Hurley 2013-07-05 00:00:00 +0000 plyr, foreach BSD Lars Kotthoff <larsko@4c.ucc.ie> cran llama llama RWeka, FSelector, e1071, flexclust, testthat 0.4
101206 Lars Kotthoff, contributions by Barry Hurley 2013-05-14 00:00:00 +0000 plyr, foreach BSD Lars Kotthoff <larsko@4c.ucc.ie> cran llama llama RWeka, FSelector, e1071, flexclust, testthat 0.2
101207 Lars Kotthoff, contributions by Barry Hurley 2013-04-02 00:00:00 +0000 plyr, foreach BSD Lars Kotthoff <larsko@4c.ucc.ie> cran llama llama RWeka, FSelector, e1071, flexclust, testthat 0.1

101208 rows × 12 columns