DATASCI W261: Machine Learning at Scale

Week 10, Homework 9

Group C: Katrina Adams, Eric Freeman, Doug Kelley

kradams@ischool.berkeley.edu, ericfreeman@ischool.berkeley.edu, kelleydac@ischool.berkeley.edu

3 November 2015



In [15]:

    
%cd ~/Documents/W261/hw9/









    



/Users/davidadams/Documents/W261/hw9

HW9.0:
What is PageRank and what is it used for in the context of web search?
What modifications have to be made to the webgraph in order to leverage the machinery of Markov Chains to compute the steady stade distibuton?
OPTIONAL: In topic-specific pagerank, how can we insure that the irreducible property is satified? (HINT: see HW9.4)

__HW 9.0 ANSWER from Doug's notebook__

HW7.5.1:
Can we utilize combiners in the HW 7 to perform the shortest path implementation?

Does order inversion help with the HW 7 shortest path implementation?

__HW 7.5.1 ANSWER from Doug's notebook__

HW 9.1: MRJob implementation of basic PageRank

Write a basic MRJob implementation of the iterative PageRank algorithm that takes sparse adjacency lists as input (as explored in HW 7).
Make sure that your implementation utilizes teleportation (1-damping/the number of nodes in the network), and further, distributes the mass of dangling nodes with each iteration so that the output of each iteration is correctly normalized (sums to 1).
[NOTE: The PageRank algorithm assumes that a random surfer (walker), starting from a random web page, chooses the next page to which it will move by clicking at random, with probability d, one of the hyperlinks in the current page. This probability is represented by a so-called ‘damping factor’ d, where d ∈ (0, 1). Otherwise, with probability (1 − d), the surfer jumps to any web page in the network. If a page is a dangling end, meaning it has no outgoing hyperlinks, the random surfer selects an arbitrary web page from a uniform distribution and “teleports” to that page]

As you build your code, use the test data

s3://ucb-mids-mls-networks/PageRank-test.txt

with teleportation parameter set to 0.15 (1-d, where d, the damping factor is set to 0.85), and crosscheck your work with the true result, displayed in the first image in the Wikipedia article:

https://en.wikipedia.org/wiki/PageRank

and here for reference are the corresponding PageRank probabilities:

A,0.033
B,0.384
C,0.343
D,0.039
E,0.081
F,0.039
G,0.016
H,0.016
I,0.016
J,0.016
K,0.016



In [ ]:

    
%%writefile MRJob_NumberOfNodes.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.step import MRStep   
from ast import literal_eval
from sys import maxint

class MRJobNumNodes(MRJob):
    

    def steps(self):
        mapper_only = 0
        if mapper_only:
            return [MRStep(mapper = self.mapper)]
        return [MRStep(
                mapper = self.mapper,
                reducer = self.reducer
                ),
               MRStep(
                reducer = self.reducer_count
                )]
    
    def mapper(self, _, line):
        line = line.strip().split('\t')
        key = line[0]
        value = literal_eval(line[1])
        destnodes = value.keys()
        
        yield key, None
        
        for node in destnodes:
            yield node, None
           
    def reducer(self,node,_):
        yield None, node
        
    def reducer_count(self,_,nodes):
        count = 0
        for node in nodes:
            count+=1
        yield None, count

        
if __name__ == '__main__':
    MRJobNumNodes.run()



In [ ]:

    
%%writefile MRJob_PreprocessGraph.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.step import MRStep   
from ast import literal_eval
from sys import maxint


class MRJobPreprocessGraph(MRJob):
    

    def steps(self):
        mapper_only = 0
        if mapper_only:
            return [MRStep(mapper_init = self.mapper_init,
                           mapper = self.mapper)]
        return [MRStep(
                mapper_init = self.mapper_init,
                mapper = self.mapper,
                reducer = self.reducer
                )]
    
    def mapper_init(self):
        N = 11
        self.PR = 1.0/N
        
    
    def mapper(self, _, line):
        line = line.strip().split('\t')
        key = line[0]
        edges = literal_eval(line[1])
        value = [self.PR, edges]
        yield key, value
        
        for node in edges.keys():
            yield node, [self.PR]

           
    def reducer(self,key,value):
        PR = 0
        edges = {}
        for v in value:
            if len(v)==1:
                PR = v[0]
            else:
                PR = v[0]
                edges = v[1]
        yield key, [PR, edges]

    
    '''
    def mapper(self, _, line):
        N = 11
        line = line.strip().split('\t')
        key = line[0]
        edges = literal_eval(line[1])
        PR = 1.0/N
        value = [PR, edges]
        yield None,{key:value}

           
    def reducer(self,_,data):
        for line in data:
            key = line.keys()[0]
            value = line[key]
            yield key, value
    '''
        
if __name__ == '__main__':
    MRJobPreprocessGraph.run()



In [101]:

    
!python MRJob_PreprocessGraph.py PageRank-test.txt > PageRank-test_input2.txt









    



using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233/step-0-mapper_part-00000
Counters from step 1:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233/step-0-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233/step-0-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233/step-0-reducer_part-00000
Counters from step 1:
  (no counters found)
Moving /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233/step-0-reducer_part-00000 -> /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233/output/part-00000
Streaming final output from /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233/output
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233



In [102]:

    
!cat PageRank-test_input2.txt









    



"A"	[0.09090909090909091, {}]
"B"	[0.09090909090909091, {"C": 1}]
"C"	[0.09090909090909091, {"B": 1}]
"D"	[0.09090909090909091, {"A": 1, "B": 1}]
"E"	[0.09090909090909091, {"B": 1, "D": 1, "F": 1}]
"F"	[0.09090909090909091, {"B": 1, "E": 1}]
"G"	[0.09090909090909091, {"B": 1, "E": 1}]
"H"	[0.09090909090909091, {"B": 1, "E": 1}]
"I"	[0.09090909090909091, {"B": 1, "E": 1}]
"J"	[0.09090909090909091, {"E": 1}]
"K"	[0.09090909090909091, {"E": 1}]



In [43]:

    
%%writefile MRJob_PageRank.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.step import MRStep   
from ast import literal_eval
from mrjob.compat import jobconf_from_env
import sys

class MRJobPageRank(MRJob):
    

    def steps(self):
        mapper_only = 0
        if mapper_only:
            return [MRStep(mapper = self.mapper)]
        
        
        return [MRStep(
                mapper = self.mapper_firstiter)]+\
                [MRStep(
                mapper = self.mapper,
                combiner = self.combiner,
                reducer = self.reducer)]*10
    
    def mapper_firstiter(self, _, line):
        line = line.strip().split('\t')
        key = line[0]
        value = literal_eval(line[1])

        PR = value[0]
        edges = value[1]
        
        yield key, [None,edges]
        
        destnodes = edges.keys()
        outdeg = len(destnodes)
        
        if outdeg==0:
            yield '*m',[PR]
        
        for node in destnodes:
            yield node, [1.0*PR/outdeg]
            

    def mapper(self, key, value):

        sys.stderr.write("MAPPER INPUT: ({0}, {1})\n".format(key,value))
        #sys.stderr.write("MAPPER")
        
        PR = value[0]
        edges = value[1]
        
        yield key, [None,edges]
        
        destnodes = edges.keys()
        outdeg = len(destnodes)
        
        if outdeg==0:
            yield '*m',[PR]
        
        for node in destnodes:
            yield node, [1.0*PR/outdeg]
            
            
    def combiner(self,node,data):
        PR = 0
        edges = {}
        for value in data:
            if value[0]==None:
                yield node, value
            else:
                PR+=value[0]
        yield node, [PR]
    
    
    def reducer(self,node,data):
        
        #sys.stderr.write("REDUCER NODE: {0}\n".format(node))
        
        G = int(jobconf_from_env('G'))
        alpha = float(jobconf_from_env('alpha'))
        
        PR = 0
        edges = {}
        
        for value in data:
            #yield node, value
            if value[0]==None:
                edges = value[1]
                
            else:
                PR+=value[0]
        
        if node=='*m':
            self.m = PR
        
        #sys.stderr.write("REDUCER PR: {0}\n".format(PR))
        
        else:
            PR_adj = alpha*(1.0/G)+(1-alpha)*(1.0*self.m/G+PR) 
            #yield node, PR
            #yield node, PR_adj
            
            #line = node+'\t'+str([PR_adj, edges])
            #yield line, ''
            
            yield node, [PR_adj, edges]

   
if __name__ == '__main__':
    MRJobPageRank.run()









    



Overwriting MRJob_PageRank.py



In [19]:

    
%load_ext autoreload
%autoreload 2

from MRJob_NumberOfNodes import MRJobNumNodes
from MRJob_PreprocessGraph import MRJobPreprocessGraph
from MRJob_PageRank import MRJobPageRank

def numNodes(graphfilename):
    #mr_job = MRJobNumNodes(args=[graphfilename, "--strict-protocols", "-r", "emr", "--num-ec2-instances", "2", "--ec2-task-instance-type", "m1.small", "--pool-emr-job-flows", "--max-hours-idle=1"])
    mr_job_numnodes = MRJobNumNodes(args=[graphfilename])

    with mr_job_numnodes.make_runner() as runner:
        runner.run()
        for line in runner.stream_output():
            null,count =  mr_job_numnodes.parse_output_line(line)
            print "There are "+str(count)+" nodes in the graph."
                    
    return None

def preprocessGraph(graphfilename):
    #mr_job = MRJobNumNodes(args=[graphfilename, "--strict-protocols", "-r", "emr", "--num-ec2-instances", "2", "--ec2-task-instance-type", "m1.small", "--pool-emr-job-flows", "--max-hours-idle=1"])
    mr_job_preprocess = MRJobPreprocessGraph(args=[graphfilename])
    outputfilename = graphfilename.split('.')[0]+'_input.txt'
    with mr_job_preprocess.make_runner() as runner:
        with open(outputfilename, 'w') as f:
            runner.run()
            for line in runner.stream_output():
                node,value =  mr_job_preprocess.parse_output_line(line)
                f.write(node+'\t'+str(value)+'\n')
                    
    return None


def pagerank(graphinputfilename):
    #mr_job = MRJobNumNodes(args=[graphfilename, "--strict-protocols", "-r", "emr", "--num-ec2-instances", "2", "--ec2-task-instance-type", "m1.small", "--pool-emr-job-flows", "--max-hours-idle=1"])
    mr_job_pr1 = MRJobPageRank(args=[graphinputfilename,'--jobconf','alpha=0.15','--jobconf','G=11'])
    danglingmass= 0
    with mr_job_pr1.make_runner() as runner:
        runner.run()
        with open(graphinputfilename, 'w+') as f:
            for line in runner.stream_output():
                node,value =  mr_job_pr1.parse_output_line(line)
                f.write(node+'\t'+str(value)+'\n')
                print line

    return None




def hw9_1():

    graphfilename = "PageRank-test.txt"
    #numNodes(graphfilename)
    
    preprocessGraph(graphfilename)

    graphinputfilename = graphfilename.split('.')[0]+'_input.txt'
    
    #pagerank_step1(graphinputfilename, alpha, G)
    
    #pagerank_step1(graphinputfilename, alpha, G)
    
    for i in range(1):
        print "Iteration",i
        pagerank(graphinputfilename)
    
    
    
    
    return None
        
  

hw9_1()









    



WARNING:mrjob.runner:
WARNING:mrjob.runner:PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols
WARNING:mrjob.runner:
WARNING:mrjob.runner:
WARNING:mrjob.runner:PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols
WARNING:mrjob.runner:






    



Iteration 0
"A"	[0.059297520661157024, {}]

"B"	[0.3168732782369146, {"C": 1}]

"C"	[0.09793388429752066, {"B": 1}]

"D"	[0.046418732782369146, {"A": 1, "B": 1}]

"E"	[0.32975206611570246, {"B": 1, "D": 1, "F": 1}]

"F"	[0.046418732782369146, {"B": 1, "E": 1}]

"G"	[0.02066115702479339, {"B": 1, "E": 1}]

"H"	[0.02066115702479339, {"B": 1, "E": 1}]

"I"	[0.02066115702479339, {"B": 1, "E": 1}]

"J"	[0.02066115702479339, {"E": 1}]

"K"	[0.02066115702479339, {"E": 1}]

HW 9.2: Exploring PageRank teleportation and network plots

In order to overcome problems such as disconnected components, the damping factor (a typical value for d is 0.85) can be varied.
Using the graph in HW1, plot the test graph (using networkx, https://networkx.github.io/) for several values of the damping parameter alpha, so that each nodes radius is proportional to its PageRank score. In particular you should do this for the following damping factors: [0,0.25,0.5,0.75, 0.85, 1].
Note your plots should look like the following:

https://en.wikipedia.org/wiki/PageRank#/media/File:PageRanks-Example.svg

__HW 9.2 from Doug's notebook__

HW 9.3: Applying PageRank to the Wikipedia hyperlinks network

Run your PageRank implementation on the Wikipedia dataset for 10 iterations, and display the top 100 ranked nodes (with alpha = 0.85).

Run your PageRank implementation on the Wikipedia dataset for 50 iterations, and display the top 100 ranked nodes (with teleportation factor of 0.15).
Have the top 100 ranked pages changed? Comment on your findings. Plot both 100 curves.



In [1]:

    
!python MRJob_NumberOfNodes.py all-pages-indexed-out.txt -r emr --num-ec2-instances 10 --ec2-task-instance-type m1.medium









    



using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_NumberOfNodes.davidadams.20151103.041200.745722
writing master bootstrap script to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_NumberOfNodes.davidadams.20151103.041200.745722/b.py

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

Copying non-input files into s3://ucb-mids-mls-katieadams/tmp/MRJob_NumberOfNodes.davidadams.20151103.041200.745722/files/
Waiting 5.0s for S3 eventual consistency
Creating Elastic MapReduce job flow
Job flow created with ID: j-DZOEYU00LV2O
Created new job flow j-DZOEYU00LV2O
Job launched 31.0s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 62.1s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 93.2s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 124.2s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 155.3s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 186.3s ago, status STARTING: Configuring cluster software
Job launched 217.3s ago, status STARTING: Configuring cluster software
Job launched 248.4s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 279.4s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 310.5s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 341.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 373.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 404.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 435.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 467.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 498.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 529.4s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 560.5s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 592.0s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 623.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 654.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 685.4s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 716.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 748.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 779.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 810.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 841.5s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 872.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 904.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 935.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 966.7s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 997.8s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1029.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1060.4s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1091.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1123.0s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1154.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1185.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1217.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1248.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1279.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1310.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1341.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1373.0s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1404.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1435.8s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1467.0s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1498.0s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1530.0s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1561.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1592.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1623.8s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1655.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1686.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1717.7s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1748.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1780.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1811.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1842.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1873.7s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1905.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1936.4s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1967.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1999.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 2030.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2061.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2093.0s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2124.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2155.8s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2186.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2218.5s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2249.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2281.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2312.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2343.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2375.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2406.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2437.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2468.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job completed.
Running time was 2178.0s (not counting time spent waiting for the EC2 instances)
ec2_key_pair_file not specified, going to S3
Fetching counters from S3...
Waiting 5.0s for S3 eventual consistency
Counters from step 1:
  File Input Format Counters :
    Bytes Read: 2090716941
  File Output Format Counters :
    Bytes Written: 231965329
  FileSystemCounters:
    FILE_BYTES_READ: 1339265877
    FILE_BYTES_WRITTEN: 1551090734
    HDFS_BYTES_READ: 6156
    HDFS_BYTES_WRITTEN: 231965329
    S3_BYTES_READ: 2090716941
  Job Counters :
    Launched map tasks: 36
    Launched reduce tasks: 18
    Rack-local map tasks: 36
    SLOTS_MILLIS_MAPS: 18794374
    SLOTS_MILLIS_REDUCES: 9141359
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 12628600
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2090459616
    Map input records: 5781290
    Map output bytes: 2266859162
    Map output materialized bytes: 502615651
    Map output records: 147895347
    Physical memory (bytes) snapshot: 13983682560
    Reduce input groups: 15192277
    Reduce input records: 147895347
    Reduce output records: 15192277
    Reduce shuffle bytes: 502615651
    SPLIT_RAW_BYTES: 6156
    Spilled Records: 443686041
    Total committed heap usage (bytes): 11517841408
    Virtual memory (bytes) snapshot: 33275449344
Counters from step 2:
  File Input Format Counters :
    Bytes Read: 233302834
  File Output Format Counters :
    Bytes Written: 14
  FileSystemCounters:
    FILE_BYTES_READ: 194894274
    FILE_BYTES_WRITTEN: 248314033
    HDFS_BYTES_READ: 233310349
    S3_BYTES_WRITTEN: 14
  Job Counters :
    Data-local map tasks: 42
    Launched map tasks: 50
    Launched reduce tasks: 22
    Rack-local map tasks: 8
    SLOTS_MILLIS_MAPS: 1219921
    SLOTS_MILLIS_REDUCES: 1225560
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 550640
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 231965329
    Map input records: 15192277
    Map output bytes: 231965329
    Map output materialized bytes: 87202083
    Map output records: 15192277
    Physical memory (bytes) snapshot: 11391102976
    Reduce input groups: 1
    Reduce input records: 15192277
    Reduce output records: 1
    Reduce shuffle bytes: 87202083
    SPLIT_RAW_BYTES: 7515
    Spilled Records: 42980204
    Total committed heap usage (bytes): 7509975040
    Virtual memory (bytes) snapshot: 38757552128
Streaming final output from s3://ucb-mids-mls-katieadams/tmp/MRJob_NumberOfNodes.davidadams.20151103.041200.745722/output/
null	15192277
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_NumberOfNodes.davidadams.20151103.041200.745722
Removing all files in s3://ucb-mids-mls-katieadams/tmp/MRJob_NumberOfNodes.davidadams.20151103.041200.745722/
Removing all files in s3://ucb-mids-mls-katieadams/tmp/logs/j-DZOEYU00LV2O/
Terminating job flow: j-DZOEYU00LV2O



In [155]:

    
%%writefile MRJob_PreprocessGraph.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.step import MRStep   
from ast import literal_eval
from mrjob.compat import jobconf_from_env


class MRJobPreprocessGraph(MRJob):
    

    def steps(self):
        mapper_only = 0
        if mapper_only:
            return [MRStep(mapper_init = self.mapper_init,
                           mapper = self.mapper)]
        return [MRStep(
                mapper_init = self.mapper_init,
                mapper = self.mapper,
                reducer = self.reducer
                )]
    
    def mapper_init(self):
        G = int(jobconf_from_env('G'))
        self.PR = 1.0/G
        
    
    def mapper(self, _, line):
        line = line.strip().split('\t')
        key = line[0]
        edges = literal_eval(line[1])
        value = [self.PR, edges]
        yield int(key), value
        
        for node in edges.keys():
            yield int(node), [self.PR]

           
    def reducer(self,key,value):
        PR = 0
        edges = {}
        for v in value:
            if len(v)==1:
                PR = v[0]
            else:
                PR = v[0]
                edges = v[1]
        yield int(key), [PR, edges]

    
        
if __name__ == '__main__':
    MRJobPreprocessGraph.run()









    



Overwriting MRJob_PreprocessGraph.py

Preprocess Indexed Graph



In [114]:

    
!python MRJob_PreprocessGraph.py PageRank-testIndexed.txt --jobconf 'G=11' > PageRank-testIndexed_input.txt









    



using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283/step-0-mapper_part-00000
Counters from step 1:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283/step-0-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283/step-0-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283/step-0-reducer_part-00000
Counters from step 1:
  (no counters found)
Moving /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283/step-0-reducer_part-00000 -> /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283/output/part-00000
Streaming final output from /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283/output
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283



In [160]:

    
%%writefile MRJob_PageRank.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.step import MRStep   
from ast import literal_eval
from mrjob.compat import jobconf_from_env


class MRJobPageRank(MRJob):

    
    
    def configure_options(self):
        super(MRJobPageRank, self).configure_options()
        
        self.add_passthrough_option(
                '--iterations',
                dest = 'iterations',
                default = 10,
                type='int',
                help='number of iterations to run')
    

    def steps(self):        
        
        mapper_only = 0
        if mapper_only:
            return [MRStep(mapper = self.mapper)]
        
        return [MRStep(
                mapper = self.mapper_firstiter,
                reducer = self.reducer
                ),
               MRStep(
                reducer = self.reducer_redistr)]+\
                [MRStep(
                mapper = self.mapper,
                reducer = self.reducer
                ),
               MRStep(
                reducer = self.reducer_redistr)]*self.options.iterations
        
    
    def mapper_firstiter(self, _, line):
        line = line.strip().split('\t')
        key = line[0]
        value = literal_eval(line[1])

        PR = value[0]
        edges = value[1]
        
        yield int(key), [None,edges]
        
        destnodes = edges.keys()
        outdeg = len(destnodes)
        
        if outdeg==0:
            yield '*m',[PR]
        
        for node in destnodes:
            yield int(node), [1.0*PR/outdeg]
            

    def mapper(self, key, value):

        PR = value[0]
        edges = value[1]
        
        yield int(key), [None,edges]
        
        destnodes = edges.keys()
        outdeg = len(destnodes)
        
        if outdeg==0:
            yield '*m',[PR]
        
        for node in destnodes:
            yield int(node), [1.0*PR/outdeg]
  
            
    def reducer(self,node,data):
        
        alpha = float(jobconf_from_env('alpha'))
        G = int(jobconf_from_env('G'))
        
        PR = 0
        edges = {}
        for value in data:
            if len(value)==1:
                PR+=value[0]
            else:
                edges = value[1]
        
        if node=='*m':
            PR_adj = (1.0/G)*(1.0-alpha)*PR
            for i in range(G):
                yield i+1, [PR_adj]
        else:
            PR_adj = alpha*(1.0/G)+(1.0-alpha)*PR
            yield int(node), [PR_adj, edges]
        
    
    def reducer_redistr(self, node, data):
        
        PR = 0
        edges = {}
        for value in data:
            PR+=value[0]
            if len(value)>1:
                edges = value[1]
        
        yield int(node), [PR, edges]
        
        
   
if __name__ == '__main__':
    MRJobPageRank.run()









    



Overwriting MRJob_PageRank.py

Test locally on small test graph



In [146]:

    
!python MRJob_PageRank.py PageRank-testIndexed_input.txt --jobconf alpha=0.15 --jobconf G=11 --iterations 50









    



using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-0-mapper_part-00000
Counters from step 1:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-0-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-0-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-0-reducer_part-00000
Counters from step 1:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-1-mapper_part-00000
Counters from step 2:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-1-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-1-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-1-reducer_part-00000
Counters from step 2:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-2-mapper_part-00000
Counters from step 3:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-2-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-2-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-2-reducer_part-00000
Counters from step 3:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-3-mapper_part-00000
Counters from step 4:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-3-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-3-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-3-reducer_part-00000
Counters from step 4:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-4-mapper_part-00000
Counters from step 5:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-4-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-4-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-4-reducer_part-00000
Counters from step 5:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-5-mapper_part-00000
Counters from step 6:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-5-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-5-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-5-reducer_part-00000
Counters from step 6:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-6-mapper_part-00000
Counters from step 7:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-6-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-6-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-6-reducer_part-00000
Counters from step 7:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-7-mapper_part-00000
Counters from step 8:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-7-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-7-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-7-reducer_part-00000
Counters from step 8:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-8-mapper_part-00000
Counters from step 9:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-8-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-8-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-8-reducer_part-00000
Counters from step 9:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-9-mapper_part-00000
Counters from step 10:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-9-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-9-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-9-reducer_part-00000
Counters from step 10:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-10-mapper_part-00000
Counters from step 11:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-10-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-10-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-10-reducer_part-00000
Counters from step 11:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-11-mapper_part-00000
Counters from step 12:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-11-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-11-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-11-reducer_part-00000
Counters from step 12:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-12-mapper_part-00000
Counters from step 13:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-12-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-12-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-12-reducer_part-00000
Counters from step 13:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-13-mapper_part-00000
Counters from step 14:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-13-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-13-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-13-reducer_part-00000
Counters from step 14:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-14-mapper_part-00000
Counters from step 15:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-14-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-14-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-14-reducer_part-00000
Counters from step 15:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-15-mapper_part-00000
Counters from step 16:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-15-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-15-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-15-reducer_part-00000
Counters from step 16:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-16-mapper_part-00000
Counters from step 17:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-16-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-16-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-16-reducer_part-00000
Counters from step 17:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-17-mapper_part-00000
Counters from step 18:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-17-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-17-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-17-reducer_part-00000
Counters from step 18:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-18-mapper_part-00000
Counters from step 19:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-18-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-18-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-18-reducer_part-00000
Counters from step 19:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-19-mapper_part-00000
Counters from step 20:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-19-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-19-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-19-reducer_part-00000
Counters from step 20:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-20-mapper_part-00000
Counters from step 21:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-20-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-20-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-20-reducer_part-00000
Counters from step 21:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-21-mapper_part-00000
Counters from step 22:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-21-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-21-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-21-reducer_part-00000
Counters from step 22:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-22-mapper_part-00000
Counters from step 23:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-22-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-22-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-22-reducer_part-00000
Counters from step 23:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-23-mapper_part-00000
Counters from step 24:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-23-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-23-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-23-reducer_part-00000
Counters from step 24:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-24-mapper_part-00000
Counters from step 25:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-24-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-24-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-24-reducer_part-00000
Counters from step 25:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-25-mapper_part-00000
Counters from step 26:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-25-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-25-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-25-reducer_part-00000
Counters from step 26:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-26-mapper_part-00000
Counters from step 27:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-26-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-26-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-26-reducer_part-00000
Counters from step 27:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-27-mapper_part-00000
Counters from step 28:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-27-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-27-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-27-reducer_part-00000
Counters from step 28:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-28-mapper_part-00000
Counters from step 29:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-28-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-28-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-28-reducer_part-00000
Counters from step 29:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-29-mapper_part-00000
Counters from step 30:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-29-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-29-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-29-reducer_part-00000
Counters from step 30:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-30-mapper_part-00000
Counters from step 31:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-30-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-30-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-30-reducer_part-00000
Counters from step 31:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-31-mapper_part-00000
Counters from step 32:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-31-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-31-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-31-reducer_part-00000
Counters from step 32:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-32-mapper_part-00000
Counters from step 33:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-32-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-32-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-32-reducer_part-00000
Counters from step 33:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-33-mapper_part-00000
Counters from step 34:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-33-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-33-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-33-reducer_part-00000
Counters from step 34:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-34-mapper_part-00000
Counters from step 35:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-34-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-34-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-34-reducer_part-00000
Counters from step 35:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-35-mapper_part-00000
Counters from step 36:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-35-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-35-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-35-reducer_part-00000
Counters from step 36:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-36-mapper_part-00000
Counters from step 37:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-36-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-36-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-36-reducer_part-00000
Counters from step 37:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-37-mapper_part-00000
Counters from step 38:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-37-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-37-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-37-reducer_part-00000
Counters from step 38:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-38-mapper_part-00000
Counters from step 39:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-38-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-38-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-38-reducer_part-00000
Counters from step 39:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-39-mapper_part-00000
Counters from step 40:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-39-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-39-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-39-reducer_part-00000
Counters from step 40:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-40-mapper_part-00000
Counters from step 41:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-40-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-40-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-40-reducer_part-00000
Counters from step 41:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-41-mapper_part-00000
Counters from step 42:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-41-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-41-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-41-reducer_part-00000
Counters from step 42:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-42-mapper_part-00000
Counters from step 43:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-42-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-42-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-42-reducer_part-00000
Counters from step 43:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-43-mapper_part-00000
Counters from step 44:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-43-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-43-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-43-reducer_part-00000
Counters from step 44:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-44-mapper_part-00000
Counters from step 45:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-44-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-44-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-44-reducer_part-00000
Counters from step 45:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-45-mapper_part-00000
Counters from step 46:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-45-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-45-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-45-reducer_part-00000
Counters from step 46:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-46-mapper_part-00000
Counters from step 47:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-46-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-46-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-46-reducer_part-00000
Counters from step 47:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-47-mapper_part-00000
Counters from step 48:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-47-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-47-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-47-reducer_part-00000
Counters from step 48:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-48-mapper_part-00000
Counters from step 49:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-48-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-48-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-48-reducer_part-00000
Counters from step 49:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-49-mapper_part-00000
Counters from step 50:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-49-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-49-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-49-reducer_part-00000
Counters from step 50:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-50-mapper_part-00000
Counters from step 51:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-50-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-50-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-50-reducer_part-00000
Counters from step 51:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-51-mapper_part-00000
Counters from step 52:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-51-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-51-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-51-reducer_part-00000
Counters from step 52:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-52-mapper_part-00000
Counters from step 53:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-52-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-52-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-52-reducer_part-00000
Counters from step 53:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-53-mapper_part-00000
Counters from step 54:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-53-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-53-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-53-reducer_part-00000
Counters from step 54:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-54-mapper_part-00000
Counters from step 55:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-54-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-54-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-54-reducer_part-00000
Counters from step 55:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-55-mapper_part-00000
Counters from step 56:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-55-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-55-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-55-reducer_part-00000
Counters from step 56:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-56-mapper_part-00000
Counters from step 57:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-56-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-56-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-56-reducer_part-00000
Counters from step 57:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-57-mapper_part-00000
Counters from step 58:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-57-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-57-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-57-reducer_part-00000
Counters from step 58:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-58-mapper_part-00000
Counters from step 59:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-58-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-58-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-58-reducer_part-00000
Counters from step 59:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-59-mapper_part-00000
Counters from step 60:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-59-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-59-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-59-reducer_part-00000
Counters from step 60:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-60-mapper_part-00000
Counters from step 61:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-60-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-60-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-60-reducer_part-00000
Counters from step 61:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-61-mapper_part-00000
Counters from step 62:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-61-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-61-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-61-reducer_part-00000
Counters from step 62:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-62-mapper_part-00000
Counters from step 63:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-62-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-62-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-62-reducer_part-00000
Counters from step 63:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-63-mapper_part-00000
Counters from step 64:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-63-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-63-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-63-reducer_part-00000
Counters from step 64:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-64-mapper_part-00000
Counters from step 65:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-64-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-64-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-64-reducer_part-00000
Counters from step 65:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-65-mapper_part-00000
Counters from step 66:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-65-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-65-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-65-reducer_part-00000
Counters from step 66:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-66-mapper_part-00000
Counters from step 67:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-66-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-66-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-66-reducer_part-00000
Counters from step 67:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-67-mapper_part-00000
Counters from step 68:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-67-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-67-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-67-reducer_part-00000
Counters from step 68:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-68-mapper_part-00000
Counters from step 69:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-68-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-68-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-68-reducer_part-00000
Counters from step 69:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-69-mapper_part-00000
Counters from step 70:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-69-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-69-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-69-reducer_part-00000
Counters from step 70:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-70-mapper_part-00000
Counters from step 71:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-70-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-70-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-70-reducer_part-00000
Counters from step 71:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-71-mapper_part-00000
Counters from step 72:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-71-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-71-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-71-reducer_part-00000
Counters from step 72:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-72-mapper_part-00000
Counters from step 73:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-72-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-72-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-72-reducer_part-00000
Counters from step 73:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-73-mapper_part-00000
Counters from step 74:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-73-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-73-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-73-reducer_part-00000
Counters from step 74:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-74-mapper_part-00000
Counters from step 75:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-74-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-74-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-74-reducer_part-00000
Counters from step 75:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-75-mapper_part-00000
Counters from step 76:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-75-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-75-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-75-reducer_part-00000
Counters from step 76:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-76-mapper_part-00000
Counters from step 77:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-76-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-76-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-76-reducer_part-00000
Counters from step 77:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-77-mapper_part-00000
Counters from step 78:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-77-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-77-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-77-reducer_part-00000
Counters from step 78:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-78-mapper_part-00000
Counters from step 79:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-78-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-78-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-78-reducer_part-00000
Counters from step 79:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-79-mapper_part-00000
Counters from step 80:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-79-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-79-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-79-reducer_part-00000
Counters from step 80:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-80-mapper_part-00000
Counters from step 81:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-80-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-80-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-80-reducer_part-00000
Counters from step 81:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-81-mapper_part-00000
Counters from step 82:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-81-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-81-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-81-reducer_part-00000
Counters from step 82:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-82-mapper_part-00000
Counters from step 83:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-82-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-82-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-82-reducer_part-00000
Counters from step 83:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-83-mapper_part-00000
Counters from step 84:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-83-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-83-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-83-reducer_part-00000
Counters from step 84:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-84-mapper_part-00000
Counters from step 85:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-84-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-84-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-84-reducer_part-00000
Counters from step 85:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-85-mapper_part-00000
Counters from step 86:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-85-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-85-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-85-reducer_part-00000
Counters from step 86:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-86-mapper_part-00000
Counters from step 87:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-86-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-86-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-86-reducer_part-00000
Counters from step 87:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-87-mapper_part-00000
Counters from step 88:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-87-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-87-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-87-reducer_part-00000
Counters from step 88:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-88-mapper_part-00000
Counters from step 89:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-88-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-88-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-88-reducer_part-00000
Counters from step 89:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-89-mapper_part-00000
Counters from step 90:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-89-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-89-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-89-reducer_part-00000
Counters from step 90:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-90-mapper_part-00000
Counters from step 91:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-90-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-90-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-90-reducer_part-00000
Counters from step 91:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-91-mapper_part-00000
Counters from step 92:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-91-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-91-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-91-reducer_part-00000
Counters from step 92:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-92-mapper_part-00000
Counters from step 93:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-92-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-92-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-92-reducer_part-00000
Counters from step 93:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-93-mapper_part-00000
Counters from step 94:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-93-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-93-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-93-reducer_part-00000
Counters from step 94:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-94-mapper_part-00000
Counters from step 95:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-94-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-94-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-94-reducer_part-00000
Counters from step 95:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-95-mapper_part-00000
Counters from step 96:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-95-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-95-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-95-reducer_part-00000
Counters from step 96:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-96-mapper_part-00000
Counters from step 97:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-96-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-96-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-96-reducer_part-00000
Counters from step 97:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-97-mapper_part-00000
Counters from step 98:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-97-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-97-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-97-reducer_part-00000
Counters from step 98:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-98-mapper_part-00000
Counters from step 99:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-98-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-98-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-98-reducer_part-00000
Counters from step 99:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-99-mapper_part-00000
Counters from step 100:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-99-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-99-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-99-reducer_part-00000
Counters from step 100:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-100-mapper_part-00000
Counters from step 101:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-100-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-100-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-100-reducer_part-00000
Counters from step 101:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-101-mapper_part-00000
Counters from step 102:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-101-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-101-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-101-reducer_part-00000
Counters from step 102:
  (no counters found)
Moving /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-101-reducer_part-00000 -> /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/output/part-00000
Streaming final output from /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/output
1	[0.03278149315934598, {}]
10	[0.01616947901685868, {"5": 1}]
11	[0.01616947901685868, {"5": 1}]
2	[0.3844274414951183, {"3": 1}]
3	[0.34288379282680403, {"2": 1}]
4	[0.03908709209996824, {"1": 1, "2": 1}]
5	[0.0808856932345013, {"2": 1, "4": 1, "6": 1}]
6	[0.03908709209996824, {"2": 1, "5": 1}]
7	[0.01616947901685868, {"2": 1, "5": 1}]
8	[0.01616947901685868, {"2": 1, "5": 1}]
9	[0.01616947901685868, {"2": 1, "5": 1}]
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498

Test on EMR with small test graph



In [153]:

    
!python MRJob_PageRank.py s3://ucb-mids-mls-katieadams/PageRank-testIndexed_input.txt --jobconf alpha=0.15 --jobconf G=11 --iterations 5 -r emr --num-ec2-instances 2 --ec2-task-instance-type m1.medium --output-dir s3://ucb-mids-mls-katieadams/outputtest --no-output









    



using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.184011.103955
writing master bootstrap script to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.184011.103955/b.py

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

Copying non-input files into s3://ucb-mids-mls-katieadams/tmp/MRJob_PageRank.davidadams.20151105.184011.103955/files/
Waiting 5.0s for S3 eventual consistency
Creating Elastic MapReduce job flow
Job flow created with ID: j-3GVS3SQZV97HL
Created new job flow j-3GVS3SQZV97HL
Job launched 31.3s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 62.6s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 93.9s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 125.3s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 156.5s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 187.9s ago, status STARTING: Configuring cluster software
Job launched 219.2s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 250.5s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 281.9s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 313.3s ago, status RUNNING: Running step
Job launched 344.6s ago, status RUNNING: Running step
Job launched 375.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 1 of 12)
Job launched 407.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 1 of 12)
Job launched 438.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 1 of 12)
Job launched 470.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 2 of 12)
Job launched 501.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 2 of 12)
Job launched 532.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 2 of 12)
Job launched 564.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 2 of 12)
Job launched 595.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 3 of 12)
Job launched 626.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 3 of 12)
Job launched 658.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 3 of 12)
Job launched 689.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 4 of 12)
Job launched 721.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 4 of 12)
Job launched 752.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 4 of 12)
Job launched 784.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 4 of 12)
Job launched 815.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 5 of 12)
Job launched 846.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 5 of 12)
Job launched 878.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 6 of 12)
Job launched 909.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 6 of 12)
Job launched 940.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 6 of 12)
Job launched 972.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 7 of 12)
Job launched 1003.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 7 of 12)
Job launched 1034.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 7 of 12)
Job launched 1066.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 8 of 12)
Job launched 1097.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 8 of 12)
Job launched 1129.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 8 of 12)
Job launched 1160.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 9 of 12)
Job launched 1191.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 9 of 12)
Job launched 1223.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 9 of 12)
Job launched 1254.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 9 of 12)
Job launched 1285.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 10 of 12)
Job launched 1317.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 10 of 12)
Job launched 1348.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 11 of 12)
Job launched 1380.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 11 of 12)
Job launched 1411.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 11 of 12)
Job launched 1442.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 11 of 12)
Job launched 1474.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 12 of 12)
Job launched 1505.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 12 of 12)
Job launched 1537.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 12 of 12)
Job completed.
Running time was 1181.0s (not counting time spent waiting for the EC2 instances)
ec2_key_pair_file not specified, going to S3
Fetching counters from S3...
Waiting 5.0s for S3 eventual consistency
Counters from step 1:
  File Input Format Counters :
    Bytes Read: 1071
  File Output Format Counters :
    Bytes Written: 723
  FileSystemCounters:
    FILE_BYTES_READ: 262
    FILE_BYTES_WRITTEN: 134357
    HDFS_BYTES_READ: 448
    HDFS_BYTES_WRITTEN: 723
    S3_BYTES_READ: 1071
  Job Counters :
    Launched map tasks: 4
    Launched reduce tasks: 1
    Rack-local map tasks: 4
    SLOTS_MILLIS_MAPS: 75404
    SLOTS_MILLIS_REDUCES: 28978
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 15570
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 426
    Map input records: 11
    Map output bytes: 714
    Map output materialized bytes: 406
    Map output records: 29
    Physical memory (bytes) snapshot: 863514624
    Reduce input groups: 12
    Reduce input records: 29
    Reduce output records: 22
    Reduce shuffle bytes: 406
    SPLIT_RAW_BYTES: 448
    Spilled Records: 58
    Total committed heap usage (bytes): 606822400
    Virtual memory (bytes) snapshot: 3141820416
Counters from step 2:
  File Input Format Counters :
    Bytes Read: 1815
  File Output Format Counters :
    Bytes Written: 434
  FileSystemCounters:
    FILE_BYTES_READ: 316
    FILE_BYTES_WRITTEN: 134217
    HDFS_BYTES_READ: 2463
    HDFS_BYTES_WRITTEN: 434
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 53696
    SLOTS_MILLIS_REDUCES: 30624
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 7480
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 723
    Map input records: 22
    Map output bytes: 723
    Map output materialized bytes: 407
    Map output records: 22
    Physical memory (bytes) snapshot: 813916160
    Reduce input groups: 11
    Reduce input records: 22
    Reduce output records: 11
    Reduce shuffle bytes: 407
    SPLIT_RAW_BYTES: 648
    Spilled Records: 44
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3186962432
Counters from step 3:
  File Input Format Counters :
    Bytes Read: 1091
  File Output Format Counters :
    Bytes Written: 721
  FileSystemCounters:
    FILE_BYTES_READ: 354
    FILE_BYTES_WRITTEN: 134664
    HDFS_BYTES_READ: 1739
    HDFS_BYTES_WRITTEN: 721
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 58610
    SLOTS_MILLIS_REDUCES: 31101
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 7780
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 434
    Map input records: 11
    Map output bytes: 709
    Map output materialized bytes: 521
    Map output records: 29
    Physical memory (bytes) snapshot: 848228352
    Reduce input groups: 12
    Reduce input records: 29
    Reduce output records: 22
    Reduce shuffle bytes: 521
    SPLIT_RAW_BYTES: 648
    Spilled Records: 58
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3195428864
Counters from step 4:
  File Input Format Counters :
    Bytes Read: 1807
  File Output Format Counters :
    Bytes Written: 432
  FileSystemCounters:
    FILE_BYTES_READ: 363
    FILE_BYTES_WRITTEN: 134320
    HDFS_BYTES_READ: 2455
    HDFS_BYTES_WRITTEN: 432
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 53470
    SLOTS_MILLIS_REDUCES: 30040
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 6310
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 721
    Map input records: 22
    Map output bytes: 721
    Map output materialized bytes: 468
    Map output records: 22
    Physical memory (bytes) snapshot: 821350400
    Reduce input groups: 11
    Reduce input records: 22
    Reduce output records: 11
    Reduce shuffle bytes: 468
    SPLIT_RAW_BYTES: 648
    Spilled Records: 44
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3194445824
Counters from step 5:
  File Input Format Counters :
    Bytes Read: 1083
  File Output Format Counters :
    Bytes Written: 721
  FileSystemCounters:
    FILE_BYTES_READ: 372
    FILE_BYTES_WRITTEN: 134696
    HDFS_BYTES_READ: 1731
    HDFS_BYTES_WRITTEN: 721
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 60919
    SLOTS_MILLIS_REDUCES: 30315
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 7470
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 432
    Map input records: 11
    Map output bytes: 708
    Map output materialized bytes: 525
    Map output records: 29
    Physical memory (bytes) snapshot: 808402944
    Reduce input groups: 12
    Reduce input records: 29
    Reduce output records: 22
    Reduce shuffle bytes: 525
    SPLIT_RAW_BYTES: 648
    Spilled Records: 58
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3052974080
Counters from step 6:
  File Input Format Counters :
    Bytes Read: 1807
  File Output Format Counters :
    Bytes Written: 432
  FileSystemCounters:
    FILE_BYTES_READ: 359
    FILE_BYTES_WRITTEN: 134307
    HDFS_BYTES_READ: 2455
    HDFS_BYTES_WRITTEN: 432
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 53663
    SLOTS_MILLIS_REDUCES: 33822
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 5930
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 721
    Map input records: 22
    Map output bytes: 721
    Map output materialized bytes: 464
    Map output records: 22
    Physical memory (bytes) snapshot: 822284288
    Reduce input groups: 11
    Reduce input records: 22
    Reduce output records: 11
    Reduce shuffle bytes: 464
    SPLIT_RAW_BYTES: 648
    Spilled Records: 44
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3193946112
Counters from step 7:
  File Input Format Counters :
    Bytes Read: 1083
  File Output Format Counters :
    Bytes Written: 723
  FileSystemCounters:
    FILE_BYTES_READ: 387
    FILE_BYTES_WRITTEN: 134721
    HDFS_BYTES_READ: 1731
    HDFS_BYTES_WRITTEN: 723
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 59800
    SLOTS_MILLIS_REDUCES: 27413
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 6700
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 432
    Map input records: 11
    Map output bytes: 716
    Map output materialized bytes: 545
    Map output records: 29
    Physical memory (bytes) snapshot: 832032768
    Reduce input groups: 12
    Reduce input records: 29
    Reduce output records: 22
    Reduce shuffle bytes: 545
    SPLIT_RAW_BYTES: 648
    Spilled Records: 58
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3218030592
Counters from step 8:
  File Input Format Counters :
    Bytes Read: 1815
  File Output Format Counters :
    Bytes Written: 435
  FileSystemCounters:
    FILE_BYTES_READ: 377
    FILE_BYTES_WRITTEN: 134334
    HDFS_BYTES_READ: 2463
    HDFS_BYTES_WRITTEN: 435
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 51581
    SLOTS_MILLIS_REDUCES: 30076
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 6270
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 723
    Map input records: 22
    Map output bytes: 723
    Map output materialized bytes: 468
    Map output records: 22
    Physical memory (bytes) snapshot: 814055424
    Reduce input groups: 11
    Reduce input records: 22
    Reduce output records: 11
    Reduce shuffle bytes: 468
    SPLIT_RAW_BYTES: 648
    Spilled Records: 44
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3212292096
Counters from step 9:
  File Input Format Counters :
    Bytes Read: 1095
  File Output Format Counters :
    Bytes Written: 723
  FileSystemCounters:
    FILE_BYTES_READ: 377
    FILE_BYTES_WRITTEN: 134705
    HDFS_BYTES_READ: 1743
    HDFS_BYTES_WRITTEN: 723
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 61722
    SLOTS_MILLIS_REDUCES: 35642
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 6860
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 435
    Map input records: 11
    Map output bytes: 718
    Map output materialized bytes: 539
    Map output records: 29
    Physical memory (bytes) snapshot: 808210432
    Reduce input groups: 12
    Reduce input records: 29
    Reduce output records: 22
    Reduce shuffle bytes: 539
    SPLIT_RAW_BYTES: 648
    Spilled Records: 58
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3142438912
Counters from step 10:
  File Input Format Counters :
    Bytes Read: 1815
  File Output Format Counters :
    Bytes Written: 435
  FileSystemCounters:
    FILE_BYTES_READ: 360
    FILE_BYTES_WRITTEN: 134330
    HDFS_BYTES_READ: 2463
    HDFS_BYTES_WRITTEN: 435
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 50209
    SLOTS_MILLIS_REDUCES: 33726
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 6510
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 723
    Map input records: 22
    Map output bytes: 723
    Map output materialized bytes: 471
    Map output records: 22
    Physical memory (bytes) snapshot: 816439296
    Reduce input groups: 11
    Reduce input records: 22
    Reduce output records: 11
    Reduce shuffle bytes: 471
    SPLIT_RAW_BYTES: 648
    Spilled Records: 44
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3261022208
Counters from step 11:
  File Input Format Counters :
    Bytes Read: 1095
  File Output Format Counters :
    Bytes Written: 722
  FileSystemCounters:
    FILE_BYTES_READ: 366
    FILE_BYTES_WRITTEN: 134704
    HDFS_BYTES_READ: 1747
    HDFS_BYTES_WRITTEN: 722
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 56179
    SLOTS_MILLIS_REDUCES: 29148
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 7230
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 435
    Map input records: 11
    Map output bytes: 709
    Map output materialized bytes: 529
    Map output records: 29
    Physical memory (bytes) snapshot: 804708352
    Reduce input groups: 12
    Reduce input records: 29
    Reduce output records: 22
    Reduce shuffle bytes: 529
    SPLIT_RAW_BYTES: 652
    Spilled Records: 58
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3168595968
Counters from step 12:
  File Input Format Counters :
    Bytes Read: 1811
  File Output Format Counters :
    Bytes Written: 433
  FileSystemCounters:
    FILE_BYTES_READ: 375
    FILE_BYTES_WRITTEN: 134144
    HDFS_BYTES_READ: 2463
    S3_BYTES_WRITTEN: 433
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 64230
    SLOTS_MILLIS_REDUCES: 37415
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 3840
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 722
    Map input records: 22
    Map output bytes: 722
    Map output materialized bytes: 465
    Map output records: 22
    Physical memory (bytes) snapshot: 850178048
    Reduce input groups: 11
    Reduce input records: 22
    Reduce output records: 11
    Reduce shuffle bytes: 465
    SPLIT_RAW_BYTES: 652
    Spilled Records: 44
    Total committed heap usage (bytes): 606781440
    Virtual memory (bytes) snapshot: 3205238784
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.184011.103955
Removing all files in s3://ucb-mids-mls-katieadams/tmp/MRJob_PageRank.davidadams.20151105.184011.103955/
Removing all files in s3://ucb-mids-mls-katieadams/tmp/logs/j-3GVS3SQZV97HL/
Terminating job flow: j-3GVS3SQZV97HL

Preprocess wikipedia graph



In [156]:

    
!python MRJob_PreprocessGraph.py s3://ucb-mids-mls-networks/wikipedia/all-pages-indexed-out.txt --jobconf G=15192277 -r emr --num-ec2-instances 15 --ec2-task-instance-type m1.medium --output-dir s3://ucb-mids-mls-katieadams/output-wiki-preprocess --no-output









    



using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.193712.552180
writing master bootstrap script to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.193712.552180/b.py

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

Copying non-input files into s3://ucb-mids-mls-katieadams/tmp/MRJob_PreprocessGraph.davidadams.20151105.193712.552180/files/
Waiting 5.0s for S3 eventual consistency
Creating Elastic MapReduce job flow
Job flow created with ID: j-1FW01DVX72RC0
Created new job flow j-1FW01DVX72RC0
Job launched 31.0s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 62.0s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 93.1s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 124.2s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 155.2s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 186.3s ago, status STARTING: Configuring cluster software
Job launched 217.3s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 248.4s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 279.4s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 310.6s ago, status RUNNING: Running step
Job launched 341.7s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 373.2s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 404.4s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 435.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 466.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 497.8s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 528.9s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 560.0s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 591.0s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 622.1s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 653.2s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 684.2s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 715.4s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 746.4s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 777.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 808.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 839.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 871.1s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 902.1s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 933.3s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 964.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 995.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1026.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1057.8s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1088.9s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1120.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1151.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1182.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1213.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1244.8s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1275.9s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1307.0s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1338.0s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1369.1s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1400.2s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1431.3s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1462.4s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1493.4s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1524.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1555.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1586.7s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1617.7s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1648.7s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1679.9s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1710.9s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1742.0s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1773.0s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1804.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1835.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1866.8s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1897.9s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1929.4s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1960.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1992.0s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 2023.2s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 2054.3s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job completed.
Running time was 1719.0s (not counting time spent waiting for the EC2 instances)
ec2_key_pair_file not specified, going to S3
Fetching counters from S3...
Waiting 5.0s for S3 eventual consistency
Counters from step 1:
  File Input Format Counters :
    Bytes Read: 2090954960
  File Output Format Counters :
    Bytes Written: 2591707083
  FileSystemCounters:
    FILE_BYTES_READ: 4353073459
    FILE_BYTES_WRITTEN: 5656577823
    HDFS_BYTES_READ: 6440
    S3_BYTES_READ: 2090954960
    S3_BYTES_WRITTEN: 2591707083
  Job Counters :
    Launched map tasks: 56
    Launched reduce tasks: 28
    Rack-local map tasks: 56
    SLOTS_MILLIS_MAPS: 29454941
    SLOTS_MILLIS_REDUCES: 14725336
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 19681830
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2090459616
    Map input records: 5781290
    Map output bytes: 6983848875
    Map output materialized bytes: 1913399068
    Map output records: 147895347
    Physical memory (bytes) snapshot: 22735384576
    Reduce input groups: 15192277
    Reduce input records: 147895347
    Reduce output records: 15192277
    Reduce shuffle bytes: 1913399068
    SPLIT_RAW_BYTES: 6440
    Spilled Records: 443686041
    Total committed heap usage (bytes): 18729271296
    Virtual memory (bytes) snapshot: 52397174784
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.193712.552180
Removing all files in s3://ucb-mids-mls-katieadams/tmp/MRJob_PreprocessGraph.davidadams.20151105.193712.552180/
Removing all files in s3://ucb-mids-mls-katieadams/tmp/logs/j-1FW01DVX72RC0/
Terminating job flow: j-1FW01DVX72RC0

PageRank on Wikipedia graph with alpha=0.85, 10 iterations



In [1]:

    
!python MRJob_PageRank.py s3://ucb-mids-mls-katieadams/output-wiki-preprocess/* --jobconf alpha=0.85 --jobconf G=15192277 --iterations 9 -r emr --num-ec2-instances 15 --ec2-instance-type m1.medium --output-dir s3://ucb-mids-mls-katieadams/output-wiki-10x-2 --no-output









    



using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151106.092145.089293
writing master bootstrap script to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151106.092145.089293/b.py

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

Copying non-input files into s3://ucb-mids-mls-katieadams/tmp/MRJob_PageRank.davidadams.20151106.092145.089293/files/
Waiting 5.0s for S3 eventual consistency
Creating Elastic MapReduce job flow
Job flow created with ID: j-89QKHDVN70CM
Created new job flow j-89QKHDVN70CM
Job launched 31.2s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 62.3s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 93.5s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 124.7s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 155.8s ago, status STARTING: Configuring cluster software
Job launched 187.0s ago, status STARTING: Configuring cluster software
Job launched 218.3s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 249.5s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 280.9s ago, status RUNNING: Running step
Job launched 312.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 343.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 374.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 405.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 436.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 468.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 499.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 531.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 563.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 594.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 625.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 656.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 688.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 719.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 750.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 781.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 812.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 844.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 876.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 907.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 939.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 970.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1001.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1033.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1064.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1095.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1127.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1158.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1190.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1221.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1253.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1284.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1315.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1346.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1378.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1409.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1440.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1472.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1504.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1535.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1566.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1597.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1628.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1660.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1691.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1722.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1753.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1784.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1815.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1847.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 1 of 20)
Job launched 1878.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 2 of 20)
Job launched 1909.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 2 of 20)
Job launched 1940.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 2 of 20)
Job launched 1971.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 2 of 20)
Job launched 2002.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 2 of 20)
Job launched 2034.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 2 of 20)
Job launched 2065.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 2 of 20)
Job launched 2096.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 2 of 20)
Job launched 2127.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2158.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2190.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2221.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2252.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2284.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2315.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2347.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2378.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2409.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2440.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2471.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2503.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2534.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2565.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2597.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2628.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2659.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2690.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2721.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2752.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2784.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2815.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2846.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2877.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2908.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2940.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 2971.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3003.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3034.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3065.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3096.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3128.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3159.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3190.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3222.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3253.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3285.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3316.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3348.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3380.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3411.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3443.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3474.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3506.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3538.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3569.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 3 of 20)
Job launched 3600.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 4 of 20)
Job launched 3632.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 4 of 20)
Job launched 3663.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 4 of 20)
Job launched 3694.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 4 of 20)
Job launched 3725.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 4 of 20)
Job launched 3757.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 4 of 20)
Job launched 3788.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 4 of 20)
Job launched 3820.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 4 of 20)
Job launched 3851.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 4 of 20)
Job launched 3882.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 3914.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 3945.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 3976.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4009.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4040.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4072.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4103.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4135.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4167.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4199.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4230.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4261.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4293.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4325.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4356.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4388.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4419.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4451.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4483.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4515.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4546.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4578.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4609.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4641.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4673.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4704.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4736.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4768.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4799.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4831.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4862.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4894.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4926.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4957.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 4989.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 5020.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 5051.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 5082.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 5114.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 5145.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 5177.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 5209.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 5241.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 5272.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 5304.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 5 of 20)
Job launched 5336.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 6 of 20)
Job launched 5367.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 6 of 20)
Job launched 5399.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 6 of 20)
Job launched 5430.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 6 of 20)
Job launched 5462.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 6 of 20)
Job launched 5494.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 6 of 20)
Job launched 5525.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 6 of 20)
Job launched 5557.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 6 of 20)
Job launched 5588.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 5620.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 5651.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 5683.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 5714.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 5746.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 5777.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 5809.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 5840.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 5873.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 5904.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 5936.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 5967.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 5998.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6029.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6061.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6092.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6124.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6156.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6188.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6219.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6251.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6282.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6314.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6345.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6377.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6408.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6440.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6471.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6503.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6534.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6566.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6597.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6629.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6660.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6692.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6723.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6755.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6787.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6818.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6850.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6881.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6913.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6944.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 6975.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 7 of 20)
Job launched 7007.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 8 of 20)
Job launched 7038.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 8 of 20)
Job launched 7070.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 8 of 20)
Job launched 7101.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 8 of 20)
Job launched 7133.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 8 of 20)
Job launched 7165.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 8 of 20)
Job launched 7196.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 8 of 20)
Job launched 7228.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 8 of 20)
Job launched 7259.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7291.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7322.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7354.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7385.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7416.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7448.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7479.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7510.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7542.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7573.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7605.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7636.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7668.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7699.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7731.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7762.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7794.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7826.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7857.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7888.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7920.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7951.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 7983.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8014.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8046.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8077.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8108.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8139.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8171.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8202.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8233.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8265.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8296.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8328.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8359.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8390.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8422.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8453.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8485.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8516.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8547.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8578.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8610.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8641.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8673.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 9 of 20)
Job launched 8704.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 10 of 20)
Job launched 8736.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 10 of 20)
Job launched 8767.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 10 of 20)
Job launched 8798.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 10 of 20)
Job launched 8829.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 10 of 20)
Job launched 8861.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 10 of 20)
Job launched 8892.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 10 of 20)
Job launched 8924.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 10 of 20)
Job launched 8955.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 8986.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9018.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9049.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9081.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9112.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9143.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9174.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9205.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9237.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9268.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9299.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9330.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9363.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9394.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9425.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9456.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9488.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9519.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9550.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9582.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9613.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9644.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9676.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9707.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9739.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9770.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9802.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9833.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9865.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9897.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9928.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9960.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 9991.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 10022.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 10054.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 10085.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 10117.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 10148.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 10180.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 10211.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 10242.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 10274.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 10305.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 10337.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 10368.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 11 of 20)
Job launched 10399.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 12 of 20)
Job launched 10431.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 12 of 20)
Job launched 10463.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 12 of 20)
Job launched 10495.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 12 of 20)
Job launched 10526.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 12 of 20)
Job launched 10557.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 12 of 20)
Job launched 10589.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 12 of 20)
Job launched 10620.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 12 of 20)
Job launched 10652.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 12 of 20)
Job launched 10683.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 10715.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 10746.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 10778.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 10810.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 10842.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 10873.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 10905.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 10937.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 10968.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11000.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11031.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11063.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11094.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11125.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11156.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11188.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11219.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11251.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11282.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11314.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11346.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11377.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11409.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11440.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11472.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11503.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11534.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11566.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11597.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11629.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11660.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11692.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11723.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11754.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11786.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11818.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11849.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11881.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11912.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11944.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 11975.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 12007.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 12039.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 13 of 20)
Job launched 12070.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 14 of 20)
Job launched 12101.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 14 of 20)
Job launched 12133.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 14 of 20)
Job launched 12164.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 14 of 20)
Job launched 12196.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 14 of 20)
Job launched 12227.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 14 of 20)
Job launched 12258.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 14 of 20)
Job launched 12290.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 14 of 20)
Job launched 12321.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 14 of 20)
Job launched 12353.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12385.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12416.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12447.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12478.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12510.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12541.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12573.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12604.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12636.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12667.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12698.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12730.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12761.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12792.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12824.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12855.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12887.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12918.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12949.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 12980.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13012.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13043.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13074.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13105.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13136.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13168.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13199.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13230.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13262.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13293.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13325.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13356.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13387.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13418.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13450.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13481.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13513.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13544.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13576.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13607.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13639.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13670.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13702.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13733.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13765.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13796.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 15 of 20)
Job launched 13828.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 16 of 20)
Job launched 13860.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 16 of 20)
Job launched 13891.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 16 of 20)
Job launched 13922.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 16 of 20)
Job launched 13954.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 16 of 20)
Job launched 13985.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 16 of 20)
Job launched 14017.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 16 of 20)
Job launched 14048.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 16 of 20)
Job launched 14080.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14112.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14143.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14174.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14206.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14237.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14269.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14300.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14332.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14363.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14394.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14426.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14457.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14489.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14520.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14551.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14582.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14614.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14645.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14676.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14708.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14739.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14770.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14801.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14833.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14864.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14895.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14926.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14958.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 14989.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15020.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15052.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15083.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15114.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15145.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15177.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15208.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15240.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15271.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15302.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15334.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15366.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15397.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15429.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15460.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15491.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15523.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 17 of 20)
Job launched 15554.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 18 of 20)
Job launched 15586.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 18 of 20)
Job launched 15617.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 18 of 20)
Job launched 15648.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 18 of 20)
Job launched 15680.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 18 of 20)
Job launched 15711.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 18 of 20)
Job launched 15742.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 18 of 20)
Job launched 15774.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 18 of 20)
Job launched 15805.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 18 of 20)
Job launched 15836.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 15868.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 15899.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 15931.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 15963.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 15995.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16026.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16057.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16088.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16120.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16151.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16183.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16214.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16246.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16277.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16309.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16341.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16372.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16404.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16435.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16467.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16499.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16530.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16561.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16593.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16625.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16656.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16687.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16718.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16750.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16782.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16813.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16845.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16876.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16907.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16938.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 16970.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 17001.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 17032.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 17063.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 17095.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 17126.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 17158.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 17190.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 17221.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 17252.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 17284.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 19 of 20)
Job launched 17315.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 20 of 20)
Job launched 17347.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 20 of 20)
Job launched 17378.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 20 of 20)
Job launched 17409.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 20 of 20)
Job launched 17441.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 20 of 20)
Job launched 17472.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 20 of 20)
Job launched 17503.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 20 of 20)
Job launched 17535.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.092145.089293: Step 20 of 20)
Job completed.
Running time was 17245.0s (not counting time spent waiting for the EC2 instances)
ec2_key_pair_file not specified, going to S3
Fetching counters from S3...
Waiting 5.0s for S3 eventual consistency
Counters from step 1:
  File Input Format Counters :
    Bytes Read: 2591995371
  File Output Format Counters :
    Bytes Written: 3095267180
  FileSystemCounters:
    FILE_BYTES_READ: 6239753113
    FILE_BYTES_WRITTEN: 9059365348
    HDFS_BYTES_READ: 8280
    HDFS_BYTES_WRITTEN: 3095267180
    S3_BYTES_READ: 2591995371
  Job Counters :
    Launched map tasks: 76
    Launched reduce tasks: 29
    Rack-local map tasks: 76
    SLOTS_MILLIS_MAPS: 16512939
    SLOTS_MILLIS_REDUCES: 8835031
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 11853270
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2591707083
    Map input records: 15192277
    Map output bytes: 7328997236
    Map output materialized bytes: 3044414470
    Map output records: 166717321
    Physical memory (bytes) snapshot: 39456071680
    Reduce input groups: 15192278
    Reduce input records: 166717321
    Reduce output records: 30384554
    Reduce shuffle bytes: 3044414470
    SPLIT_RAW_BYTES: 8280
    Spilled Records: 500151963
    Total committed heap usage (bytes): 36309200896
    Virtual memory (bytes) snapshot: 91756265472
Counters from step 2:
  File Input Format Counters :
    Bytes Read: 3096576040
  File Output Format Counters :
    Bytes Written: 2590079507
  FileSystemCounters:
    FILE_BYTES_READ: 1755514384
    FILE_BYTES_WRITTEN: 3262910027
    HDFS_BYTES_READ: 3096585436
    HDFS_BYTES_WRITTEN: 2590079507
  Job Counters :
    Data-local map tasks: 53
    Launched map tasks: 58
    Launched reduce tasks: 28
    Rack-local map tasks: 5
    SLOTS_MILLIS_MAPS: 1043162
    SLOTS_MILLIS_REDUCES: 2272034
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 1795880
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 3095267180
    Map input records: 30384554
    Map output bytes: 3101499421
    Map output materialized bytes: 1552954077
    Map output records: 30384554
    Physical memory (bytes) snapshot: 26368348160
    Reduce input groups: 15192277
    Reduce input records: 30384554
    Reduce output records: 15192277
    Reduce shuffle bytes: 1552954077
    SPLIT_RAW_BYTES: 9396
    Spilled Records: 76268128
    Total committed heap usage (bytes): 23373279232
    Virtual memory (bytes) snapshot: 77878226944
Counters from step 3:
  File Input Format Counters :
    Bytes Read: 2590895387
  File Output Format Counters :
    Bytes Written: 3095491631
  FileSystemCounters:
    FILE_BYTES_READ: 8946343673
    FILE_BYTES_WRITTEN: 13036998696
    HDFS_BYTES_READ: 2590907051
    HDFS_BYTES_WRITTEN: 3095491631
  Job Counters :
    Data-local map tasks: 67
    Launched map tasks: 77
    Launched reduce tasks: 29
    Rack-local map tasks: 10
    SLOTS_MILLIS_MAPS: 13245176
    SLOTS_MILLIS_REDUCES: 8574647
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 10334470
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2590079507
    Map input records: 15192277
    Map output bytes: 7326794788
    Map output materialized bytes: 4395225314
    Map output records: 166717321
    Physical memory (bytes) snapshot: 38445289472
    Reduce input groups: 15192278
    Reduce input records: 166717321
    Reduce output records: 30384554
    Reduce shuffle bytes: 4395225314
    SPLIT_RAW_BYTES: 11664
    Spilled Records: 500151963
    Total committed heap usage (bytes): 35096408064
    Virtual memory (bytes) snapshot: 90676666368
Counters from step 4:
  File Input Format Counters :
    Bytes Read: 3096707115
  File Output Format Counters :
    Bytes Written: 2590158730
  FileSystemCounters:
    FILE_BYTES_READ: 1767236266
    FILE_BYTES_WRITTEN: 3287470513
    HDFS_BYTES_READ: 3096716511
    HDFS_BYTES_WRITTEN: 2590158730
  Job Counters :
    Data-local map tasks: 53
    Launched map tasks: 58
    Launched reduce tasks: 28
    Rack-local map tasks: 5
    SLOTS_MILLIS_MAPS: 1061965
    SLOTS_MILLIS_REDUCES: 2273750
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 1805190
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 3095491631
    Map input records: 30384554
    Map output bytes: 3101723773
    Map output materialized bytes: 1565802911
    Map output records: 30384554
    Physical memory (bytes) snapshot: 26458726400
    Reduce input groups: 15192277
    Reduce input records: 30384554
    Reduce output records: 15192277
    Reduce shuffle bytes: 1565802911
    SPLIT_RAW_BYTES: 9396
    Spilled Records: 76268299
    Total committed heap usage (bytes): 23376138240
    Virtual memory (bytes) snapshot: 77670535168
Counters from step 5:
  File Input Format Counters :
    Bytes Read: 2590906690
  File Output Format Counters :
    Bytes Written: 3095569763
  FileSystemCounters:
    FILE_BYTES_READ: 9021427244
    FILE_BYTES_WRITTEN: 13151872447
    HDFS_BYTES_READ: 2590918354
    HDFS_BYTES_WRITTEN: 3095569763
  Job Counters :
    Data-local map tasks: 67
    Launched map tasks: 75
    Launched reduce tasks: 29
    Rack-local map tasks: 8
    SLOTS_MILLIS_MAPS: 13293993
    SLOTS_MILLIS_REDUCES: 8410132
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 10350950
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2590158730
    Map input records: 15192277
    Map output bytes: 7326726434
    Map output materialized bytes: 4434005399
    Map output records: 166717321
    Physical memory (bytes) snapshot: 39067983872
    Reduce input groups: 15192278
    Reduce input records: 166717321
    Reduce output records: 30384554
    Reduce shuffle bytes: 4434005399
    SPLIT_RAW_BYTES: 11664
    Spilled Records: 500151963
    Total committed heap usage (bytes): 35771572224
    Virtual memory (bytes) snapshot: 91091222528
Counters from step 6:
  File Input Format Counters :
    Bytes Read: 3096707127
  File Output Format Counters :
    Bytes Written: 2590153434
  FileSystemCounters:
    FILE_BYTES_READ: 1767239767
    FILE_BYTES_WRITTEN: 3287559128
    HDFS_BYTES_READ: 3096716523
    HDFS_BYTES_WRITTEN: 2590153434
  Job Counters :
    Data-local map tasks: 52
    Launched map tasks: 58
    Launched reduce tasks: 28
    Rack-local map tasks: 6
    SLOTS_MILLIS_MAPS: 1038790
    SLOTS_MILLIS_REDUCES: 2247759
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 1806020
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 3095569763
    Map input records: 30384554
    Map output bytes: 3101802083
    Map output materialized bytes: 1565863279
    Map output records: 30384554
    Physical memory (bytes) snapshot: 26060283904
    Reduce input groups: 15192277
    Reduce input records: 30384554
    Reduce output records: 15192277
    Reduce shuffle bytes: 1565863279
    SPLIT_RAW_BYTES: 9396
    Spilled Records: 76268378
    Total committed heap usage (bytes): 23131783168
    Virtual memory (bytes) snapshot: 77659213824
Counters from step 7:
  File Input Format Counters :
    Bytes Read: 2590905954
  File Output Format Counters :
    Bytes Written: 3095570581
  FileSystemCounters:
    FILE_BYTES_READ: 9022843625
    FILE_BYTES_WRITTEN: 13152314893
    HDFS_BYTES_READ: 2590917618
    HDFS_BYTES_WRITTEN: 3095570581
  Job Counters :
    Data-local map tasks: 68
    Launched map tasks: 76
    Launched reduce tasks: 29
    Rack-local map tasks: 8
    SLOTS_MILLIS_MAPS: 13259931
    SLOTS_MILLIS_REDUCES: 8026070
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 10352330
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2590153434
    Map input records: 15192277
    Map output bytes: 7326493297
    Map output materialized bytes: 4433776073
    Map output records: 166717321
    Physical memory (bytes) snapshot: 38420738048
    Reduce input groups: 15192278
    Reduce input records: 166717321
    Reduce output records: 30384554
    Reduce shuffle bytes: 4433776073
    SPLIT_RAW_BYTES: 11664
    Spilled Records: 500151963
    Total committed heap usage (bytes): 35133829120
    Virtual memory (bytes) snapshot: 90676359168
Counters from step 8:
  File Input Format Counters :
    Bytes Read: 3096707105
  File Output Format Counters :
    Bytes Written: 2590132773
  FileSystemCounters:
    FILE_BYTES_READ: 1767228859
    FILE_BYTES_WRITTEN: 3287454394
    HDFS_BYTES_READ: 3096716501
    HDFS_BYTES_WRITTEN: 2590132773
  Job Counters :
    Data-local map tasks: 55
    Launched map tasks: 58
    Launched reduce tasks: 28
    Rack-local map tasks: 3
    SLOTS_MILLIS_MAPS: 1073760
    SLOTS_MILLIS_REDUCES: 2321121
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 1790900
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 3095570581
    Map input records: 30384554
    Map output bytes: 3101803006
    Map output materialized bytes: 1565811242
    Map output records: 30384554
    Physical memory (bytes) snapshot: 26210385920
    Reduce input groups: 15192277
    Reduce input records: 30384554
    Reduce output records: 15192277
    Reduce shuffle bytes: 1565811242
    SPLIT_RAW_BYTES: 9396
    Spilled Records: 76268378
    Total committed heap usage (bytes): 23013568512
    Virtual memory (bytes) snapshot: 77263212544
Counters from step 9:
  File Input Format Counters :
    Bytes Read: 2590903005
  File Output Format Counters :
    Bytes Written: 3095545693
  FileSystemCounters:
    FILE_BYTES_READ: 9023508021
    FILE_BYTES_WRITTEN: 13156508881
    HDFS_BYTES_READ: 2590914669
    HDFS_BYTES_WRITTEN: 3095545693
  Job Counters :
    Data-local map tasks: 70
    Launched map tasks: 77
    Launched reduce tasks: 29
    Rack-local map tasks: 7
    SLOTS_MILLIS_MAPS: 13273014
    SLOTS_MILLIS_REDUCES: 8279337
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 10350170
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2590132773
    Map input records: 15192277
    Map output bytes: 7325507631
    Map output materialized bytes: 4435724010
    Map output records: 166717321
    Physical memory (bytes) snapshot: 38797692928
    Reduce input groups: 15192278
    Reduce input records: 166717321
    Reduce output records: 30384554
    Reduce shuffle bytes: 4435724010
    SPLIT_RAW_BYTES: 11664
    Spilled Records: 500151963
    Total committed heap usage (bytes): 35608485888
    Virtual memory (bytes) snapshot: 90986057728
Counters from step 10:
  File Input Format Counters :
    Bytes Read: 3096707137
  File Output Format Counters :
    Bytes Written: 2590126879
  FileSystemCounters:
    FILE_BYTES_READ: 1767387796
    FILE_BYTES_WRITTEN: 3287670120
    HDFS_BYTES_READ: 3096716533
    HDFS_BYTES_WRITTEN: 2590126879
  Job Counters :
    Data-local map tasks: 52
    Launched map tasks: 58
    Launched reduce tasks: 28
    Rack-local map tasks: 6
    SLOTS_MILLIS_MAPS: 1043786
    SLOTS_MILLIS_REDUCES: 2319225
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 1811640
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 3095545693
    Map input records: 30384554
    Map output bytes: 3101778221
    Map output materialized bytes: 1565883140
    Map output records: 30384554
    Physical memory (bytes) snapshot: 26086326272
    Reduce input groups: 15192277
    Reduce input records: 30384554
    Reduce output records: 15192277
    Reduce shuffle bytes: 1565883140
    SPLIT_RAW_BYTES: 9396
    Spilled Records: 76268355
    Total committed heap usage (bytes): 22944612352
    Virtual memory (bytes) snapshot: 77392400384
Counters from step 11:
  File Input Format Counters :
    Bytes Read: 2590902151
  File Output Format Counters :
    Bytes Written: 3095588727
  FileSystemCounters:
    FILE_BYTES_READ: 9021814767
    FILE_BYTES_WRITTEN: 13153351786
    HDFS_BYTES_READ: 2590913887
    HDFS_BYTES_WRITTEN: 3095588727
  Job Counters :
    Data-local map tasks: 65
    Launched map tasks: 75
    Launched reduce tasks: 29
    Rack-local map tasks: 10
    SLOTS_MILLIS_MAPS: 13143105
    SLOTS_MILLIS_REDUCES: 8019296
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 10336470
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2590126879
    Map input records: 15192277
    Map output bytes: 7326268561
    Map output materialized bytes: 4434064489
    Map output records: 166717321
    Physical memory (bytes) snapshot: 38523809792
    Reduce input groups: 15192278
    Reduce input records: 166717321
    Reduce output records: 30384554
    Reduce shuffle bytes: 4434064489
    SPLIT_RAW_BYTES: 11736
    Spilled Records: 500151963
    Total committed heap usage (bytes): 35258867712
    Virtual memory (bytes) snapshot: 91031216128
Counters from step 12:
  File Input Format Counters :
    Bytes Read: 3096707107
  File Output Format Counters :
    Bytes Written: 2590131435
  FileSystemCounters:
    FILE_BYTES_READ: 1767304883
    FILE_BYTES_WRITTEN: 3287450583
    HDFS_BYTES_READ: 3096716561
    HDFS_BYTES_WRITTEN: 2590131435
  Job Counters :
    Data-local map tasks: 58
    Launched map tasks: 66
    Launched reduce tasks: 28
    Rack-local map tasks: 8
    SLOTS_MILLIS_MAPS: 1161493
    SLOTS_MILLIS_REDUCES: 2288021
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 1804710
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 3095588727
    Map input records: 30384554
    Map output bytes: 3101821201
    Map output materialized bytes: 1565752623
    Map output records: 30384554
    Physical memory (bytes) snapshot: 26141241344
    Reduce input groups: 15192277
    Reduce input records: 30384554
    Reduce output records: 15192277
    Reduce shuffle bytes: 1565752623
    SPLIT_RAW_BYTES: 9454
    Spilled Records: 76268400
    Total committed heap usage (bytes): 23448043520
    Virtual memory (bytes) snapshot: 77699227648
Counters from step 13:
  File Input Format Counters :
    Bytes Read: 2590902771
  File Output Format Counters :
    Bytes Written: 3095543328
  FileSystemCounters:
    FILE_BYTES_READ: 9023212669
    FILE_BYTES_WRITTEN: 13154294139
    HDFS_BYTES_READ: 2590914507
    HDFS_BYTES_WRITTEN: 3095543328
  Job Counters :
    Data-local map tasks: 65
    Launched map tasks: 75
    Launched reduce tasks: 29
    Rack-local map tasks: 10
    SLOTS_MILLIS_MAPS: 13095880
    SLOTS_MILLIS_REDUCES: 8069082
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 10337930
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2590131435
    Map input records: 15192277
    Map output bytes: 7326561129
    Map output materialized bytes: 4434131611
    Map output records: 166717321
    Physical memory (bytes) snapshot: 38378049536
    Reduce input groups: 15192278
    Reduce input records: 166717321
    Reduce output records: 30384554
    Reduce shuffle bytes: 4434131611
    SPLIT_RAW_BYTES: 11736
    Spilled Records: 500151963
    Total committed heap usage (bytes): 35291758592
    Virtual memory (bytes) snapshot: 90960289792
Counters from step 14:
  File Input Format Counters :
    Bytes Read: 3096707124
  File Output Format Counters :
    Bytes Written: 2590119786
  FileSystemCounters:
    FILE_BYTES_READ: 1767283293
    FILE_BYTES_WRITTEN: 3287611196
    HDFS_BYTES_READ: 3096716578
    HDFS_BYTES_WRITTEN: 2590119786
  Job Counters :
    Data-local map tasks: 51
    Launched map tasks: 58
    Launched reduce tasks: 28
    Rack-local map tasks: 7
    SLOTS_MILLIS_MAPS: 1048507
    SLOTS_MILLIS_REDUCES: 2389842
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 1813980
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 3095543328
    Map input records: 30384554
    Map output bytes: 3101775805
    Map output materialized bytes: 1565895998
    Map output records: 30384554
    Physical memory (bytes) snapshot: 26187882496
    Reduce input groups: 15192277
    Reduce input records: 30384554
    Reduce output records: 15192277
    Reduce shuffle bytes: 1565895998
    SPLIT_RAW_BYTES: 9454
    Spilled Records: 76268350
    Total committed heap usage (bytes): 23070986240
    Virtual memory (bytes) snapshot: 77634584576
Counters from step 15:
  File Input Format Counters :
    Bytes Read: 2590901106
  File Output Format Counters :
    Bytes Written: 3080292734
  FileSystemCounters:
    FILE_BYTES_READ: 9021875108
    FILE_BYTES_WRITTEN: 13151042905
    HDFS_BYTES_READ: 2590912842
    HDFS_BYTES_WRITTEN: 3080292734
  Job Counters :
    Data-local map tasks: 67
    Launched map tasks: 77
    Launched reduce tasks: 29
    Rack-local map tasks: 10
    SLOTS_MILLIS_MAPS: 13189292
    SLOTS_MILLIS_REDUCES: 8547588
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 10354940
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2590119786
    Map input records: 15192277
    Map output bytes: 7326760349
    Map output materialized bytes: 4433269820
    Map output records: 166717321
    Physical memory (bytes) snapshot: 38720475136
    Reduce input groups: 15192278
    Reduce input records: 166717321
    Reduce output records: 30384554
    Reduce shuffle bytes: 4433269820
    SPLIT_RAW_BYTES: 11736
    Spilled Records: 500151963
    Total committed heap usage (bytes): 35357093888
    Virtual memory (bytes) snapshot: 90854281216
Counters from step 16:
  File Input Format Counters :
    Bytes Read: 3081719901
  File Output Format Counters :
    Bytes Written: 2590137246
  FileSystemCounters:
    FILE_BYTES_READ: 1735816328
    FILE_BYTES_WRITTEN: 3261498832
    HDFS_BYTES_READ: 3081729192
    HDFS_BYTES_WRITTEN: 2590137246
  Job Counters :
    Data-local map tasks: 53
    Launched map tasks: 57
    Launched reduce tasks: 28
    Rack-local map tasks: 4
    SLOTS_MILLIS_MAPS: 1050016
    SLOTS_MILLIS_REDUCES: 2330547
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 1796950
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 3080292734
    Map input records: 30384554
    Map output bytes: 3086524488
    Map output materialized bytes: 1565618021
    Map output records: 30384554
    Physical memory (bytes) snapshot: 25698140160
    Reduce input groups: 15192277
    Reduce input records: 30384554
    Reduce output records: 15192277
    Reduce shuffle bytes: 1565618021
    SPLIT_RAW_BYTES: 9291
    Spilled Records: 75990943
    Total committed heap usage (bytes): 22695849984
    Virtual memory (bytes) snapshot: 76832542720
Counters from step 17:
  File Input Format Counters :
    Bytes Read: 2590903638
  File Output Format Counters :
    Bytes Written: 3095573909
  FileSystemCounters:
    FILE_BYTES_READ: 9022358597
    FILE_BYTES_WRITTEN: 13153550960
    HDFS_BYTES_READ: 2590915374
    HDFS_BYTES_WRITTEN: 3095573909
  Job Counters :
    Data-local map tasks: 65
    Launched map tasks: 75
    Launched reduce tasks: 29
    Rack-local map tasks: 10
    SLOTS_MILLIS_MAPS: 13091055
    SLOTS_MILLIS_REDUCES: 8456997
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 10366760
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2590137246
    Map input records: 15192277
    Map output bytes: 7326360158
    Map output materialized bytes: 4434424962
    Map output records: 166717321
    Physical memory (bytes) snapshot: 38558990336
    Reduce input groups: 15192278
    Reduce input records: 166717321
    Reduce output records: 30384554
    Reduce shuffle bytes: 4434424962
    SPLIT_RAW_BYTES: 11736
    Spilled Records: 500151963
    Total committed heap usage (bytes): 35310469120
    Virtual memory (bytes) snapshot: 89994711040
Counters from step 18:
  File Input Format Counters :
    Bytes Read: 3096707129
  File Output Format Counters :
    Bytes Written: 2590124251
  FileSystemCounters:
    FILE_BYTES_READ: 1767300814
    FILE_BYTES_WRITTEN: 3287573707
    HDFS_BYTES_READ: 3096716583
    HDFS_BYTES_WRITTEN: 2590124251
  Job Counters :
    Data-local map tasks: 52
    Launched map tasks: 58
    Launched reduce tasks: 28
    Rack-local map tasks: 6
    SLOTS_MILLIS_MAPS: 1074438
    SLOTS_MILLIS_REDUCES: 2298780
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 1806340
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 3095573909
    Map input records: 30384554
    Map output bytes: 3101806399
    Map output materialized bytes: 1565858576
    Map output records: 30384554
    Physical memory (bytes) snapshot: 26284998656
    Reduce input groups: 15192277
    Reduce input records: 30384554
    Reduce output records: 15192277
    Reduce shuffle bytes: 1565858576
    SPLIT_RAW_BYTES: 9454
    Spilled Records: 76268381
    Total committed heap usage (bytes): 22958530560
    Virtual memory (bytes) snapshot: 77550149632
Counters from step 19:
  File Input Format Counters :
    Bytes Read: 2590901779
  File Output Format Counters :
    Bytes Written: 3095559074
  FileSystemCounters:
    FILE_BYTES_READ: 9020693730
    FILE_BYTES_WRITTEN: 13151740148
    HDFS_BYTES_READ: 2590913515
    HDFS_BYTES_WRITTEN: 3095559074
  Job Counters :
    Data-local map tasks: 67
    Launched map tasks: 77
    Launched reduce tasks: 29
    Rack-local map tasks: 10
    SLOTS_MILLIS_MAPS: 13194075
    SLOTS_MILLIS_REDUCES: 8266579
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 10335350
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2590124251
    Map input records: 15192277
    Map output bytes: 7326134127
    Map output materialized bytes: 4434266987
    Map output records: 166717321
    Physical memory (bytes) snapshot: 38232666112
    Reduce input groups: 15192278
    Reduce input records: 166717321
    Reduce output records: 30384554
    Reduce shuffle bytes: 4434266987
    SPLIT_RAW_BYTES: 11736
    Spilled Records: 500151963
    Total committed heap usage (bytes): 35009875968
    Virtual memory (bytes) snapshot: 90989158400
Counters from step 20:
  File Input Format Counters :
    Bytes Read: 3096707134
  File Output Format Counters :
    Bytes Written: 2590158072
  FileSystemCounters:
    FILE_BYTES_READ: 1767254322
    FILE_BYTES_WRITTEN: 3287450824
    HDFS_BYTES_READ: 3096716588
    S3_BYTES_WRITTEN: 2590158072
  Job Counters :
    Data-local map tasks: 53
    Launched map tasks: 58
    Launched reduce tasks: 28
    Rack-local map tasks: 5
    SLOTS_MILLIS_MAPS: 1171624
    SLOTS_MILLIS_REDUCES: 2568241
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 1855610
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 3095559074
    Map input records: 30384554
    Map output bytes: 3101791408
    Map output materialized bytes: 1565791837
    Map output records: 30384554
    Physical memory (bytes) snapshot: 27193098240
    Reduce input groups: 15192277
    Reduce input records: 30384554
    Reduce output records: 15192277
    Reduce shuffle bytes: 1565791837
    SPLIT_RAW_BYTES: 9454
    Spilled Records: 76268376
    Total committed heap usage (bytes): 23560667136
    Virtual memory (bytes) snapshot: 77967884288
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151106.092145.089293
Removing all files in s3://ucb-mids-mls-katieadams/tmp/MRJob_PageRank.davidadams.20151106.092145.089293/
Removing all files in s3://ucb-mids-mls-katieadams/tmp/logs/j-89QKHDVN70CM/
Terminating job flow: j-89QKHDVN70CM

PageRank on Wikipedia graph with alpha=0.15, 50 iterations

Still running, 80% done, should be done my morning (barring any disasters)



In [ ]:

    
!python MRJob_PageRank.py s3://ucb-mids-mls-katieadams/output-wiki-preprocess/* --jobconf alpha=0.15 --jobconf G=15192277 --iterations 49 -r emr --num-ec2-instances 15 --ec2-instance-type m1.medium --output-dir s3://ucb-mids-mls-katieadams/output-wiki-50x --no-output









    



using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151106.091946.408922
writing master bootstrap script to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151106.091946.408922/b.py

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

Copying non-input files into s3://ucb-mids-mls-katieadams/tmp/MRJob_PageRank.davidadams.20151106.091946.408922/files/
Waiting 5.0s for S3 eventual consistency
Creating Elastic MapReduce job flow
Job flow created with ID: j-23K5T9VKSZO64
Created new job flow j-23K5T9VKSZO64

HW 9.4: Topic-specific PageRank implementation using MRJob

Modify your PageRank implementation to produce a topic specific PageRank implementation, as described in:

http://www-cs-students.stanford.edu/~taherh/papers/topic-sensitive-pagerank.pdf

Note in this article that there is a special caveat to ensure that the transition matrix is irreducible.
This caveat lies in footnote 3 on page 3:

A minor caveat: to ensure that M is irreducible when p
contains any 0 entries, nodes not reachable from nonzero
nodes in p should be removed. In practice this is not problematic.

and must be adhered to for convergence to be guaranteed.

Run topic specific PageRank on the following randomly generated network of 100 nodes:
s3://ucb-mids-mls-networks/randNet.txt

which are organized into ten topics, as described in the file:
s3://ucb-mids-mls-networks/randNet_topics.txt

Since there are 10 topics, your result should be 11 PageRank vectors (one for the vanilla PageRank implementation in 9.1, and one for each topic with the topic specific implementation). Print out the top ten ranking nodes and their topics for each of the 11 versions, and comment on your result.
Assume a teleportation factor of 0.15 in all your analyses.



In [28]:

    
%%writefile MRJob_PreprocessGraph_topic.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.step import MRStep   
from ast import literal_eval
from mrjob.compat import jobconf_from_env


class MRJobPreprocessGraphTopic(MRJob):
    

    def steps(self):
        mapper_only = 0
        if mapper_only:
            return [MRStep(mapper_init = self.mapper_init,
                           mapper = self.mapper)]
        return [MRStep(
                mapper_init = self.mapper_init,
                mapper = self.mapper,
                reducer = self.reducer
                )]
    
    def mapper_init(self):
        self.topic_labels = {}
        topicIdxFilename = jobconf_from_env('topicIdxFilename')
        with open(topicIdxFilename, 'r') as f:
            for line in f.readlines():
                line = line.strip()
                p,t = line.split('\t')
                self.topic_labels[int(p)]=int(t)
                
        G = int(jobconf_from_env('G'))
        self.PR_init = 1.0/G
        
    
    def mapper(self, _, line):
        line = line.strip().split('\t')
        key = int(line[0])
        edges = literal_eval(line[1])
        value = {}
        value['PR']=[self.PR_init]*11
        value['edges']=edges.keys()
        value['topic']=self.topic_labels[key]
            
        yield key, value
        
        for node in edges.keys():
            key=int(node)
            value = {}
            value['PR']=[self.PR_init]*11
            value['edges']=None
            value['topic']=self.topic_labels[key]
            yield key, value

           
    def reducer(self,key,value):
        data = {}
        data['topic']=None
        data['PR']=None
        data['edges']=[]
        for v in value:
            if data['topic']==None:
                data['topic']=v['topic']
            if v['edges']==None:
                data['PR']=v['PR']
            else:
                data['PR']=v['PR']
                data['edges']=v['edges']
        yield int(key), data

    
        
if __name__ == '__main__':
    MRJobPreprocessGraphTopic.run()









    



Overwriting MRJob_PreprocessGraph_topic.py

Preprocess RandNet locally to include initialized page ranks and topics



In [29]:

    
!python MRJob_PreprocessGraph_topic.py randNet.txt --file 'randNet_topics.txt' --jobconf 'G=100' --jobconf 'topicIdxFilename=randNet_topics.txt'> randNet_input.txt









    



using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151106.210948.584583

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151106.210948.584583/step-0-mapper_part-00000
Counters from step 1:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151106.210948.584583/step-0-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151106.210948.584583/step-0-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151106.210948.584583/step-0-reducer_part-00000
Counters from step 1:
  (no counters found)
Moving /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151106.210948.584583/step-0-reducer_part-00000 -> /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151106.210948.584583/output/part-00000
Streaming final output from /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151106.210948.584583/output
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151106.210948.584583

Test preprocessing on emr



In [31]:

    
!python MRJob_PreprocessGraph_topic.py randNet.txt --file 'randNet_topics.txt' --jobconf 'G=100' --jobconf 'topicIdxFilename=randNet_topics.txt' -r emr --num-ec2-instances 2 --ec2-task-instance-type m1.medium --output-dir s3://ucb-mids-mls-katieadams/output-wiki-preprocess-topics --no-output









    



using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151106.211319.807689
writing master bootstrap script to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151106.211319.807689/b.py

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

Copying non-input files into s3://ucb-mids-mls-katieadams/tmp/MRJob_PreprocessGraph_topic.davidadams.20151106.211319.807689/files/
Waiting 5.0s for S3 eventual consistency
Creating Elastic MapReduce job flow
Job flow created with ID: j-VP5CFD55XZ98
Created new job flow j-VP5CFD55XZ98
Job launched 31.1s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 62.2s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 93.3s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 124.5s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 155.6s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 186.6s ago, status STARTING: Configuring cluster software
Job launched 217.8s ago, status STARTING: Configuring cluster software
Job launched 248.8s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 279.9s ago, status RUNNING: Running step
Job launched 311.1s ago, status RUNNING: Running step
Job launched 342.2s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151106.211319.807689: Step 1 of 1)
Job launched 373.2s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151106.211319.807689: Step 1 of 1)
Job launched 404.3s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151106.211319.807689: Step 1 of 1)
Job completed.
Running time was 113.0s (not counting time spent waiting for the EC2 instances)
ec2_key_pair_file not specified, going to S3
Fetching counters from S3...
Waiting 5.0s for S3 eventual consistency
Counters from step 1:
  File Input Format Counters :
    Bytes Read: 21735
  File Output Format Counters :
    Bytes Written: 15616
  FileSystemCounters:
    FILE_BYTES_READ: 10867
    FILE_BYTES_WRITTEN: 160006
    HDFS_BYTES_READ: 660
    S3_BYTES_READ: 21735
    S3_BYTES_WRITTEN: 15616
  Job Counters :
    Launched map tasks: 4
    Launched reduce tasks: 1
    Rack-local map tasks: 4
    SLOTS_MILLIS_MAPS: 76467
    SLOTS_MILLIS_REDUCES: 35966
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 4910
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 8691
    Map input records: 100
    Map output bytes: 113303
    Map output materialized bytes: 12125
    Map output records: 1029
    Physical memory (bytes) snapshot: 863367168
    Reduce input groups: 100
    Reduce input records: 1029
    Reduce output records: 100
    Reduce shuffle bytes: 12125
    SPLIT_RAW_BYTES: 660
    Spilled Records: 2058
    Total committed heap usage (bytes): 606838784
    Virtual memory (bytes) snapshot: 3231440896
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151106.211319.807689
Removing all files in s3://ucb-mids-mls-katieadams/tmp/MRJob_PreprocessGraph_topic.davidadams.20151106.211319.807689/
Removing all files in s3://ucb-mids-mls-katieadams/tmp/logs/j-VP5CFD55XZ98/
Terminating job flow: j-VP5CFD55XZ98



In [30]:

    
!head -1 randNet_input.txt









    



1	{"topic": 10, "PR": [0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01], "edges": ["11", "27", "63", "46", "47", "35", "89", "5"]}



In [18]:

    
from collections import defaultdict
def numNodesByTopic(topicsIdxFile, topicsCountFile):
    numNodesTopic = defaultdict(int)
    with open('randNet_topics.txt', 'r') as f:
        for line in f.readlines():
            line = line.strip()
            p,t = line.split('\t')
            numNodesTopic[int(t)]+=1
            numNodesTopic['*']+=1
    with open('randNet_topicCount.txt','w') as f:
        for k,v in numNodesTopic.iteritems():
            line = str(k)+'\t'+str(v)+'\n'
            f.write(line)

            
topicsIdxFile, topicsCountFile
numNodesByTopic(topicsIdxFile, topicsCountFile)



In [21]:

    
!cat randNet_topicCount.txt



In [62]:

    
%%writefile MRJob_PageRank_Topic.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.step import MRStep   
from ast import literal_eval
from mrjob.compat import jobconf_from_env
import sys


class MRJobPageRankTopic(MRJob):
    
    
    def configure_options(self):
        super(MRJobPageRankTopic, self).configure_options()
        
        self.add_passthrough_option(
                '--iterations',
                dest = 'iterations',
                default = 10,
                type='int',
                help='number of iterations to run')
    

    def steps(self):        
        
        mapper_only = 0
        if mapper_only:
            return [MRStep(mapper = self.mapper_firstiter)]
        
        return [MRStep(
                mapper = self.mapper_firstiter,
                reducer_init = self.reducer_init,
                reducer = self.reducer
                ),
               MRStep(
                reducer = self.reducer_redistr)]+\
                [MRStep(
                mapper = self.mapper,
                reducer_init = self.reducer_init,
                reducer = self.reducer
                ),
               MRStep(
                reducer = self.reducer_redistr)]*self.options.iterations
        
    
    def mapper_firstiter(self, _, line):
        
        line = line.strip().split('\t')
        key = line[0]
        value = literal_eval(line[1])
        
        graphvalue = {}
        graphvalue['PR']=None
        graphvalue['edges']=value['edges']
        graphvalue['topic']=value['topic']
        
        yield int(key), graphvalue
        
        destnodes = value['edges']
        outdeg = len(destnodes)
        
        sendvalue = {}
        sendvalue['PR']=[]
        sendvalue['topic']=None
        sendvalue['edges']=None
        
        if outdeg==0:
            sendvalue['PR']=value['PR']
            yield '*m',sendvalue
        else:
            
            for i in range(len(value['PR'])):
                PRi = value['PR'][i]
                sendvalue['PR'].append(1.0*PRi/outdeg)
            for node in destnodes:
                yield int(node), sendvalue
            

    def mapper(self, key, value):
       
        graphvalue = {}
        graphvalue['PR']=None
        graphvalue['edges']=value['edges']
        graphvalue['topic']=value['topic']
        
        yield int(key), graphvalue
        
        destnodes = value['edges']
        outdeg = len(destnodes)
        
        sendvalue = {}
        sendvalue['PR']=[]
        sendvalue['topic']=None
        sendvalue['edges']=None
        
        if outdeg==0:
            sendvalue['PR']=value['PR']
            yield '*m',sendvalue
        else:
            for i in range(len(value['PR'])):
                PRi = value['PR'][i]
                sendvalue['PR'].append(1.0*PRi/outdeg)
            for node in destnodes:
                yield int(node), sendvalue
  
    def reducer_init(self):
        
        self.alpha = float(jobconf_from_env('alpha'))
        self.G = int(jobconf_from_env('G'))
        self.n = int(jobconf_from_env('num_topics'))
        self.beta = float(jobconf_from_env('beta'))
        topicCountFilename = jobconf_from_env('topicCountFilename')
        
        self.T = {}
        with open(topicCountFilename, 'r') as f:
            for line in f.readlines():
                topic,count = line.strip().split('\t')
                if topic=='*':
                    topic = 0
                else: 
                    topic = int(topic)
                self.T[topic]=int(count)
            
    def reducer(self,node,data):
        
        sendvalue = {}
        sendvalue['PR']=[0]*(self.n+1)
        sendvalue['topic']=None
        sendvalue['edges']=None
       
        for value in data:
            if value['edges']==None:
                for i in range(len(value['PR'])):
                    sendvalue['PR'][i]+=value['PR'][i]
            else:
                sendvalue['topic']=int(value['topic'])
                sendvalue['edges']=value['edges']
        
        if node=='*m':
            PR_adj = []
            for i in range(self.n+1):
                PR_adj.append([(1.0/self.G)*(1.0-self.alpha)*sendvalue['PR'][i]])
            sendvalue['PR']=PR_adj
            for i in range(self.G):
                yield i+1, sendvalue
        else:
            PR_adj = []
            PR_adj.append(self.alpha*(1.0/self.G)+(1.0-self.alpha)*sendvalue['PR'][i])
            for i in range(1,self.n+1):
                if sendvalue['topic']==i:
                    PR_adj.append(self.alpha*self.beta*(1.0/self.T[i])+(1.0-self.alpha)*sendvalue['PR'][i])
                else:
                    PR_adj.append(self.alpha*(1-self.beta)*(1.0/(self.G-self.T[i]))+(1.0-self.alpha)*sendvalue['PR'][i])
                    
            sendvalue['PR']=PR_adj
            yield int(node), sendvalue
        
    
    def reducer_redistr(self, node, data):
        
        n = int(jobconf_from_env('num_topics'))
        
        finalvalue = {}
        finalvalue['PR']=[0]*(n+1)
        finalvalue['topic']=None
        finalvalue['edges']=None
        
        for value in data:
            for i in range(n+1):
                finalvalue['PR'][i]+=value['PR'][i]
            if value['edges']!=None:
                finalvalue['edges'] = value['edges']
                finalvalue['topic']=int(value['topic'])
        
        yield int(node), finalvalue
        
   
if __name__ == '__main__':
    MRJobPageRankTopic.run()









    



Overwriting MRJob_PageRank_Topic.py



In [65]:

    
!python MRJob_PageRank_Topic.py randNet_input.txt --file 'randNet_topicCount.txt' --jobconf 'G=100' --jobconf 'alpha=0.15'\
                                                    --jobconf 'beta=0.99' --jobconf 'topicCountFilename=randNet_topicCount.txt'\
                                                    --jobconf 'num_topics=10' --iterations 9 > randNet_Topic_output.txt









    



using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-0-mapper_part-00000
Counters from step 1:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-0-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-0-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-0-reducer_part-00000
Counters from step 1:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-1-mapper_part-00000
Counters from step 2:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-1-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-1-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-1-reducer_part-00000
Counters from step 2:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-2-mapper_part-00000
Counters from step 3:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-2-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-2-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-2-reducer_part-00000
Counters from step 3:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-3-mapper_part-00000
Counters from step 4:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-3-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-3-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-3-reducer_part-00000
Counters from step 4:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-4-mapper_part-00000
Counters from step 5:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-4-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-4-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-4-reducer_part-00000
Counters from step 5:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-5-mapper_part-00000
Counters from step 6:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-5-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-5-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-5-reducer_part-00000
Counters from step 6:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-6-mapper_part-00000
Counters from step 7:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-6-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-6-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-6-reducer_part-00000
Counters from step 7:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-7-mapper_part-00000
Counters from step 8:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-7-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-7-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-7-reducer_part-00000
Counters from step 8:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-8-mapper_part-00000
Counters from step 9:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-8-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-8-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-8-reducer_part-00000
Counters from step 9:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-9-mapper_part-00000
Counters from step 10:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-9-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-9-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-9-reducer_part-00000
Counters from step 10:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-10-mapper_part-00000
Counters from step 11:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-10-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-10-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-10-reducer_part-00000
Counters from step 11:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-11-mapper_part-00000
Counters from step 12:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-11-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-11-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-11-reducer_part-00000
Counters from step 12:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-12-mapper_part-00000
Counters from step 13:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-12-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-12-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-12-reducer_part-00000
Counters from step 13:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-13-mapper_part-00000
Counters from step 14:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-13-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-13-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-13-reducer_part-00000
Counters from step 14:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-14-mapper_part-00000
Counters from step 15:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-14-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-14-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-14-reducer_part-00000
Counters from step 15:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-15-mapper_part-00000
Counters from step 16:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-15-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-15-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-15-reducer_part-00000
Counters from step 16:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-16-mapper_part-00000
Counters from step 17:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-16-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-16-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-16-reducer_part-00000
Counters from step 17:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-17-mapper_part-00000
Counters from step 18:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-17-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-17-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-17-reducer_part-00000
Counters from step 18:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-18-mapper_part-00000
Counters from step 19:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-18-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-18-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-18-reducer_part-00000
Counters from step 19:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-19-mapper_part-00000
Counters from step 20:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-19-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-19-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-19-reducer_part-00000
Counters from step 20:
  (no counters found)
Moving /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/step-19-reducer_part-00000 -> /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/output/part-00000
Streaming final output from /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776/output
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank_Topic.davidadams.20151107.031945.328776



In [8]:

    
%%writefile MRtop10.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.protocol import JSONProtocol
from mrjob.step import MRStep
import re
from itertools import combinations
from ast import literal_eval
import sys
import string
from operator import itemgetter
from collections import defaultdict
from mrjob.compat import jobconf_from_env

'''
    Modified from Doug and Group E's code
'''


class MRtop10(MRJob):
    OUTPUT_PROTOCOL = JSONProtocol
    mapperListDict = defaultdict(list)
    reducerListDict = defaultdict(list)
    
    def jobconf(self):
        orig_jobconf = super(MRtop10, self).jobconf()        
        custom_jobconf = {
            'mapred.output.key.comparator.class': 'org.apache.hadoop.mapred.lib.KeyFieldBasedComparator',
            'mapred.text.key.comparator.options': '-k1rn',
            'mapred.reduce.tasks': '1',
        }
        combined_jobconf = orig_jobconf
        combined_jobconf.update(custom_jobconf)
        self.jobconf = combined_jobconf
        return combined_jobconf

    def steps(self):
        return [MRStep(
                mapper = self.mapper, 
                reducer_init = self.reducer_init,
                reducer = self.reducer,
                reducer_final = self.reducer_final
            )]
    def getFloatKey(item):
        return float(item[1])
    
    def mapper(self, key, line):

        fields = line.strip().split('\t')
        nodeid = fields[0]
        nodeDict = literal_eval(fields[1])
        for i in range(11):
            yield float(nodeDict['PR'][i]), (i, nodeid, nodeDict['topic'])

    def reducer_init(self):
        self.n = int(jobconf_from_env('num_topics'))
        self.top_pages = defaultdict(list)
            
    def reducer(self, score, nodetups):
        for node in nodetups:
            topic = node[0]
            if len(self.top_pages[topic])<10:
                self.top_pages[topic].append([score, node])

                
    def reducer_final(self):
        for page, scores in self.top_pages.iteritems():
            for score in scores:
                rank = score[0]
                node = score[1][1]
                topic = score[1][2]
                yield "Vector "+str(page), (rank, node, topic)

                
                
if __name__ == '__main__':
    MRtop10.run()









    



Overwriting MRtop10.py



In [10]:

    
!python MRtop10.py randNet_Topic_output.txt --jobconf 'num_topics=10' -r emr









    



using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRtop10.davidadams.20151107.045928.232288
writing master bootstrap script to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRtop10.davidadams.20151107.045928.232288/b.py

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

Copying non-input files into s3://ucb-mids-mls-katieadams/tmp/MRtop10.davidadams.20151107.045928.232288/files/
Waiting 5.0s for S3 eventual consistency
Creating Elastic MapReduce job flow
Job flow created with ID: j-25CYGX08O6UFX
Created new job flow j-25CYGX08O6UFX
Job launched 31.3s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 62.5s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 93.6s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 125.2s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 156.4s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 188.1s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 219.3s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 250.6s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 281.8s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 313.0s ago, status RUNNING: Running step
Job launched 344.2s ago, status RUNNING: Running step (MRtop10.davidadams.20151107.045928.232288: Step 1 of 1)
Job launched 375.4s ago, status RUNNING: Running step (MRtop10.davidadams.20151107.045928.232288: Step 1 of 1)
Job launched 406.5s ago, status RUNNING: Running step (MRtop10.davidadams.20151107.045928.232288: Step 1 of 1)
Job launched 437.6s ago, status RUNNING: Running step (MRtop10.davidadams.20151107.045928.232288: Step 1 of 1)
Job completed.
Running time was 120.0s (not counting time spent waiting for the EC2 instances)
ec2_key_pair_file not specified, going to S3
Fetching counters from S3...
Waiting 5.0s for S3 eventual consistency
Counters from step 1:
  File Input Format Counters :
    Bytes Read: 49631
  File Output Format Counters :
    Bytes Written: 4740
  FileSystemCounters:
    FILE_BYTES_READ: 23913
    FILE_BYTES_WRITTEN: 183059
    HDFS_BYTES_READ: 632
    S3_BYTES_READ: 49631
    S3_BYTES_WRITTEN: 4740
  Job Counters :
    Launched map tasks: 4
    Launched reduce tasks: 1
    Rack-local map tasks: 4
    SLOTS_MILLIS_MAPS: 79773
    SLOTS_MILLIS_REDUCES: 37341
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 6590
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 33090
    Map input records: 100
    Map output bytes: 38120
    Map output materialized bytes: 24782
    Map output records: 1100
    Physical memory (bytes) snapshot: 871424000
    Reduce input groups: 1100
    Reduce input records: 1100
    Reduce output records: 110
    Reduce shuffle bytes: 24782
    SPLIT_RAW_BYTES: 632
    Spilled Records: 2200
    Total committed heap usage (bytes): 606838784
    Virtual memory (bytes) snapshot: 3240636416
Streaming final output from s3://ucb-mids-mls-katieadams/tmp/MRtop10.davidadams.20151107.045928.232288/output/
"Vector 0"	[0.018450492994579813, "63", 4]
"Vector 0"	[0.017279654921740766, "61", 8]
"Vector 0"	[0.016480564476828941, "15", 3]
"Vector 0"	[0.015456832014712155, "74", 10]
"Vector 0"	[0.014915487546274911, "92", 1]
"Vector 0"	[0.014886676428498469, "100", 8]
"Vector 0"	[0.014724267309124666, "90", 5]
"Vector 0"	[0.014495723860992503, "13", 6]
"Vector 0"	[0.014248697591624835, "85", 7]
"Vector 0"	[0.014141694236526918, "71", 2]
"Vector 1"	[0.020645898325225003, "32", 1]
"Vector 1"	[0.02054756962678736, "77", 1]
"Vector 1"	[0.019754313100732439, "52", 1]
"Vector 1"	[0.019529238246263958, "92", 1]
"Vector 1"	[0.018565525448276273, "10", 1]
"Vector 1"	[0.018522539827068918, "27", 1]
"Vector 1"	[0.017840510571832245, "85", 7]
"Vector 1"	[0.017692389508390936, "98", 1]
"Vector 1"	[0.017514128674978226, "46", 1]
"Vector 1"	[0.016028121317860768, "74", 10]
"Vector 2"	[0.03084746002637323, "58", 2]
"Vector 2"	[0.029665243324997904, "71", 2]
"Vector 2"	[0.029296846892829638, "9", 2]
"Vector 2"	[0.028914805418738834, "73", 2]
"Vector 2"	[0.026888935387018635, "12", 2]
"Vector 2"	[0.025799681887661255, "59", 2]
"Vector 2"	[0.024849600531229588, "75", 2]
"Vector 2"	[0.022858211891270377, "82", 2]
"Vector 2"	[0.016322098538761654, "52", 1]
"Vector 2"	[0.01515867219849016, "17", 10]
"Vector 3"	[0.031529068397347246, "15", 3]
"Vector 3"	[0.027076597693858763, "70", 3]
"Vector 3"	[0.026527964044297941, "86", 3]
"Vector 3"	[0.024463316416756184, "91", 3]
"Vector 3"	[0.024148527884714856, "66", 3]
"Vector 3"	[0.023705087243707054, "2", 3]
"Vector 3"	[0.022767111411451758, "31", 3]
"Vector 3"	[0.022178501156581275, "40", 3]
"Vector 3"	[0.019745051424447244, "20", 3]
"Vector 3"	[0.01589997292892676, "74", 10]
"Vector 4"	[0.026202043207444136, "63", 4]
"Vector 4"	[0.021760079633412529, "83", 4]
"Vector 4"	[0.020623779777070743, "65", 4]
"Vector 4"	[0.020210101152229841, "78", 4]
"Vector 4"	[0.019908459750517585, "41", 4]
"Vector 4"	[0.0195198800713322, "84", 4]
"Vector 4"	[0.018428650565935505, "79", 4]
"Vector 4"	[0.017515429366835658, "38", 4]
"Vector 4"	[0.016752163539375291, "15", 3]
"Vector 4"	[0.016694729499179141, "72", 4]
"Vector 5"	[0.028963266500299595, "99", 5]
"Vector 5"	[0.028344984671495925, "90", 5]
"Vector 5"	[0.027168703255116916, "88", 5]
"Vector 5"	[0.026830795332821233, "51", 5]
"Vector 5"	[0.025553308477928548, "45", 5]
"Vector 5"	[0.023919955028234258, "5", 5]
"Vector 5"	[0.023909744584958762, "34", 5]
"Vector 5"	[0.023363274382490386, "4", 5]
"Vector 5"	[0.022839620864129398, "80", 5]
"Vector 5"	[0.016741485269746215, "100", 8]
"Vector 6"	[0.034571570877532272, "13", 6]
"Vector 6"	[0.032853906714508899, "56", 6]
"Vector 6"	[0.031777155876364656, "37", 6]
"Vector 6"	[0.031339648198519324, "11", 6]
"Vector 6"	[0.03012079634490477, "69", 6]
"Vector 6"	[0.02834890267401214, "23", 6]
"Vector 6"	[0.017242580581008006, "15", 3]
"Vector 6"	[0.016989438344123808, "85", 7]
"Vector 6"	[0.01660216963107063, "52", 1]
"Vector 6"	[0.015461058978408842, "74", 10]
"Vector 7"	[0.026794227495348936, "85", 7]
"Vector 7"	[0.026608778613501603, "25", 7]
"Vector 7"	[0.024827245457770573, "28", 7]
"Vector 7"	[0.024767654194382913, "53", 7]
"Vector 7"	[0.024200828678767147, "35", 7]
"Vector 7"	[0.023395013979336465, "97", 7]
"Vector 7"	[0.022865013665705175, "47", 7]
"Vector 7"	[0.022560854063987799, "55", 7]
"Vector 7"	[0.022134839131015584, "30", 7]
"Vector 7"	[0.020089080196239371, "50", 7]
"Vector 8"	[0.032866030460939763, "100", 8]
"Vector 8"	[0.027858570822448362, "61", 8]
"Vector 8"	[0.027195521505354078, "39", 8]
"Vector 8"	[0.027153099743645281, "8", 8]
"Vector 8"	[0.025346793563654338, "62", 8]
"Vector 8"	[0.025298416024840099, "87", 8]
"Vector 8"	[0.023506852654286991, "6", 8]
"Vector 8"	[0.022891738248983953, "54", 8]
"Vector 8"	[0.020622660245957835, "18", 8]
"Vector 8"	[0.015379799633741306, "9", 2]
"Vector 9"	[0.030198889934084209, "94", 9]
"Vector 9"	[0.029495197684241921, "14", 9]
"Vector 9"	[0.029197862681776883, "42", 9]
"Vector 9"	[0.028399569111395757, "21", 9]
"Vector 9"	[0.027461793981146124, "57", 9]
"Vector 9"	[0.026260949679310815, "96", 9]
"Vector 9"	[0.025773074359528945, "24", 9]
"Vector 9"	[0.017161492066928675, "63", 4]
"Vector 9"	[0.016377328880887894, "61", 8]
"Vector 9"	[0.014277712975620794, "74", 10]
"Vector 10"	[0.026331832014712155, "74", 10]
"Vector 10"	[0.023590536112239317, "17", 10]
"Vector 10"	[0.023574267728797862, "49", 10]
"Vector 10"	[0.020629340496081131, "95", 10]
"Vector 10"	[0.019913805171583917, "7", 10]
"Vector 10"	[0.019364174927853535, "43", 10]
"Vector 10"	[0.019038271865623197, "68", 10]
"Vector 10"	[0.019005045680706455, "48", 10]
"Vector 10"	[0.018998083854615253, "1", 10]
"Vector 10"	[0.018639556040770718, "3", 10]
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRtop10.davidadams.20151107.045928.232288
Removing all files in s3://ucb-mids-mls-katieadams/tmp/MRtop10.davidadams.20151107.045928.232288/
Removing all files in s3://ucb-mids-mls-katieadams/tmp/logs/j-25CYGX08O6UFX/
Terminating job flow: j-25CYGX08O6UFX

HW 9.5: Applying topic-specific PageRank to Wikipedia

Here you will apply your topic-specific PageRank implementation to Wikipedia, defining topics (very arbitrarily) for each page by the length (number of characters) of the name of the article mod 10, so that there are 10 topics. Once again, print out the top ten ranking nodes and their topics for each of the 11 versions, and comment on your result.
Assume a teleportation factor of 0.15 in all your analyses.



In [25]:

    
def assignWikiTopics():
    ftopics = open('wiki_topics.txt','w')
    i=0
    with open('indices.txt','r') as fidx:
        for line in fidx.readlines():
            i+=1
            if i%500000==0:
                print "progress: ",i*1.0/15192277
            line = line.strip().split('\t')
            topic = len(line[0])%10
            idx = int(line[1])
            ftopics.write(str(idx)+'\t'+str(topic)+'\n')
    ftopics.close()

assignWikiTopics()









    



progress:  0.00329114588945
progress:  0.00658229177891
progress:  0.00987343766836
progress:  0.0131645835578
progress:  0.0164557294473
progress:  0.0197468753367
progress:  0.0230380212262
progress:  0.0263291671156
progress:  0.0296203130051
progress:  0.0329114588945
progress:  0.036202604784
progress:  0.0394937506735
progress:  0.0427848965629
progress:  0.0460760424524
progress:  0.0493671883418
progress:  0.0526583342313
progress:  0.0559494801207
progress:  0.0592406260102
progress:  0.0625317718996
progress:  0.0658229177891
progress:  0.0691140636785
progress:  0.072405209568
progress:  0.0756963554574
progress:  0.0789875013469
progress:  0.0822786472364
progress:  0.0855697931258
progress:  0.0888609390153
progress:  0.0921520849047
progress:  0.0954432307942
progress:  0.0987343766836
progress:  0.102025522573
progress:  0.105316668463
progress:  0.108607814352
progress:  0.111898960241
progress:  0.115190106131
progress:  0.11848125202
progress:  0.12177239791
progress:  0.125063543799
progress:  0.128354689689
progress:  0.131645835578
progress:  0.134936981468
progress:  0.138228127357
progress:  0.141519273247
progress:  0.144810419136
progress:  0.148101565025
progress:  0.151392710915
progress:  0.154683856804
progress:  0.157975002694
progress:  0.161266148583
progress:  0.164557294473
progress:  0.167848440362
progress:  0.171139586252
progress:  0.174430732141
progress:  0.177721878031
progress:  0.18101302392
progress:  0.184304169809
progress:  0.187595315699
progress:  0.190886461588
progress:  0.194177607478
progress:  0.197468753367
progress:  0.200759899257
progress:  0.204051045146
progress:  0.207342191036
progress:  0.210633336925
progress:  0.213924482815
progress:  0.217215628704
progress:  0.220506774593
progress:  0.223797920483
progress:  0.227089066372
progress:  0.230380212262
progress:  0.233671358151
progress:  0.236962504041
progress:  0.24025364993
progress:  0.24354479582
progress:  0.246835941709
progress:  0.250127087599
progress:  0.253418233488
progress:  0.256709379377
progress:  0.260000525267
progress:  0.263291671156
progress:  0.266582817046
progress:  0.269873962935
progress:  0.273165108825
progress:  0.276456254714
progress:  0.279747400604
progress:  0.283038546493
progress:  0.286329692383
progress:  0.289620838272
progress:  0.292911984161
progress:  0.296203130051
progress:  0.29949427594
progress:  0.30278542183
progress:  0.306076567719
progress:  0.309367713609
progress:  0.312658859498
progress:  0.315950005388
progress:  0.319241151277
progress:  0.322532297167
progress:  0.325823443056
progress:  0.329114588945
progress:  0.332405734835
progress:  0.335696880724
progress:  0.338988026614
progress:  0.342279172503
progress:  0.345570318393
progress:  0.348861464282
progress:  0.352152610172
progress:  0.355443756061
progress:  0.358734901951
progress:  0.36202604784
progress:  0.365317193729
progress:  0.368608339619
progress:  0.371899485508
progress:  0.375190631398
progress:  0.378481777287
progress:  0.381772923177
progress:  0.385064069066
progress:  0.388355214956
progress:  0.391646360845
progress:  0.394937506735
progress:  0.398228652624
progress:  0.401519798513
progress:  0.404810944403
progress:  0.408102090292
progress:  0.411393236182
progress:  0.414684382071
progress:  0.417975527961
progress:  0.42126667385
progress:  0.42455781974
progress:  0.427848965629
progress:  0.431140111519
progress:  0.434431257408
progress:  0.437722403297
progress:  0.441013549187
progress:  0.444304695076
progress:  0.447595840966
progress:  0.450886986855
progress:  0.454178132745
progress:  0.457469278634
progress:  0.460760424524
progress:  0.464051570413
progress:  0.467342716303
progress:  0.470633862192
progress:  0.473925008081
progress:  0.477216153971
progress:  0.48050729986
progress:  0.48379844575
progress:  0.487089591639
progress:  0.490380737529
progress:  0.493671883418
progress:  0.496963029308
progress:  0.500254175197
progress:  0.503545321086
progress:  0.506836466976
progress:  0.510127612865
progress:  0.513418758755
progress:  0.516709904644
progress:  0.520001050534
progress:  0.523292196423
progress:  0.526583342313
progress:  0.529874488202
progress:  0.533165634092
progress:  0.536456779981
progress:  0.53974792587
progress:  0.54303907176
progress:  0.546330217649
progress:  0.549621363539
progress:  0.552912509428
progress:  0.556203655318
progress:  0.559494801207
progress:  0.562785947097
progress:  0.566077092986
progress:  0.569368238876
progress:  0.572659384765
progress:  0.575950530654
progress:  0.579241676544
progress:  0.582532822433
progress:  0.585823968323
progress:  0.589115114212
progress:  0.592406260102
progress:  0.595697405991
progress:  0.598988551881
progress:  0.60227969777
progress:  0.60557084366
progress:  0.608861989549
progress:  0.612153135438
progress:  0.615444281328
progress:  0.618735427217
progress:  0.622026573107
progress:  0.625317718996
progress:  0.628608864886
progress:  0.631900010775
progress:  0.635191156665
progress:  0.638482302554
progress:  0.641773448444
progress:  0.645064594333
progress:  0.648355740222
progress:  0.651646886112
progress:  0.654938032001
progress:  0.658229177891
progress:  0.66152032378
progress:  0.66481146967
progress:  0.668102615559
progress:  0.671393761449
progress:  0.674684907338
progress:  0.677976053228
progress:  0.681267199117
progress:  0.684558345006
progress:  0.687849490896
progress:  0.691140636785
progress:  0.694431782675
progress:  0.697722928564
progress:  0.701014074454
progress:  0.704305220343
progress:  0.707596366233
progress:  0.710887512122
progress:  0.714178658012
progress:  0.717469803901
progress:  0.72076094979
progress:  0.72405209568
progress:  0.727343241569
progress:  0.730634387459
progress:  0.733925533348
progress:  0.737216679238
progress:  0.740507825127
progress:  0.743798971017
progress:  0.747090116906
progress:  0.750381262796
progress:  0.753672408685
progress:  0.756963554574
progress:  0.760254700464
progress:  0.763545846353
progress:  0.766836992243
progress:  0.770128138132
progress:  0.773419284022
progress:  0.776710429911
progress:  0.780001575801
progress:  0.78329272169
progress:  0.78658386758
progress:  0.789875013469
progress:  0.793166159358
progress:  0.796457305248
progress:  0.799748451137
progress:  0.803039597027
progress:  0.806330742916
progress:  0.809621888806
progress:  0.812913034695
progress:  0.816204180585
progress:  0.819495326474
progress:  0.822786472364
progress:  0.826077618253
progress:  0.829368764142
progress:  0.832659910032
progress:  0.835951055921
progress:  0.839242201811
progress:  0.8425333477
progress:  0.84582449359
progress:  0.849115639479
progress:  0.852406785369
progress:  0.855697931258
progress:  0.858989077148
progress:  0.862280223037
progress:  0.865571368926
progress:  0.868862514816
progress:  0.872153660705
progress:  0.875444806595
progress:  0.878735952484
progress:  0.882027098374
progress:  0.885318244263
progress:  0.888609390153
progress:  0.891900536042
progress:  0.895191681932
progress:  0.898482827821
progress:  0.90177397371
progress:  0.9050651196
progress:  0.908356265489
progress:  0.911647411379
progress:  0.914938557268
progress:  0.918229703158
progress:  0.921520849047
progress:  0.924811994937
progress:  0.928103140826
progress:  0.931394286716
progress:  0.934685432605
progress:  0.937976578494
progress:  0.941267724384
progress:  0.944558870273
progress:  0.947850016163
progress:  0.951141162052
progress:  0.954432307942
progress:  0.957723453831
progress:  0.961014599721
progress:  0.96430574561
progress:  0.9675968915
progress:  0.970888037389
progress:  0.974179183278
progress:  0.977470329168
progress:  0.980761475057
progress:  0.984052620947
progress:  0.987343766836
progress:  0.990634912726
progress:  0.993926058615
progress:  0.997217204505



In [34]:

    
!python MRJob_PreprocessGraph_topic.py s3://ucb-mids-mls-networks/wikipedia/all-pages-indexed-out.txt --file 'wiki_topics.txt' --jobconf G=15192277 --jobconf 'topicIdxFilename=wiki_topics.txt' -r emr --num-ec2-instances 4 --ec2-instance-type m1.large --output-dir s3://ucb-mids-mls-katieadams/output-wiki-preprocess-topics --no-output









    



using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300
writing master bootstrap script to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300/b.py

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

Copying non-input files into s3://ucb-mids-mls-katieadams/tmp/MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300/files/
Waiting 5.0s for S3 eventual consistency
Creating Elastic MapReduce job flow
Job flow created with ID: j-273B8CMZ72WBW
Created new job flow j-273B8CMZ72WBW
Job launched 31.1s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 62.7s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 93.8s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 125.3s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 156.5s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 187.7s ago, status STARTING: Configuring cluster software
Job launched 218.8s ago, status STARTING: Configuring cluster software
Job launched 250.0s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 281.1s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 312.3s ago, status RUNNING: Running step
Job launched 343.4s ago, status RUNNING: Running step
Job launched 374.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300: Step 1 of 1)
Job launched 405.7s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300: Step 1 of 1)
Job launched 436.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300: Step 1 of 1)
Job launched 467.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300: Step 1 of 1)
Job launched 498.7s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300: Step 1 of 1)
Job launched 530.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300: Step 1 of 1)
Job launched 561.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300: Step 1 of 1)
Job launched 592.7s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300: Step 1 of 1)
Job launched 623.8s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300: Step 1 of 1)
Job launched 655.0s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300: Step 1 of 1)
Job launched 686.1s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300: Step 1 of 1)
Job launched 717.2s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300: Step 1 of 1)
Job launched 748.9s ago, status RUNNING: Running step (MRJob_PreprocessGraph_topic.davidadams.20151107.002601.081300: Step 1 of 1)
Job on job flow j-273B8CMZ72WBW failed with status TERMINATING: Shut down as step failed
Logs are in s3://ucb-mids-mls-katieadams/tmp/logs/j-273B8CMZ72WBW/
ec2_key_pair_file not specified, going to S3
Scanning S3 logs for probable cause of failure
Waiting 5.0s for S3 eventual consistency
Terminating job flow: j-273B8CMZ72WBW
Traceback (most recent call last):
  File "MRJob_PreprocessGraph_topic.py", line 73, in <module>
    MRJobPreprocessGraphTopic.run()
  File "/Library/Python/2.7/site-packages/mrjob/job.py", line 461, in run
    mr_job.execute()
  File "/Library/Python/2.7/site-packages/mrjob/job.py", line 479, in execute
    super(MRJob, self).execute()
  File "/Library/Python/2.7/site-packages/mrjob/launch.py", line 153, in execute
    self.run_job()
  File "/Library/Python/2.7/site-packages/mrjob/launch.py", line 216, in run_job
    runner.run()
  File "/Library/Python/2.7/site-packages/mrjob/runner.py", line 470, in run
    self._run()
  File "/Library/Python/2.7/site-packages/mrjob/emr.py", line 882, in _run
    self._wait_for_job_to_complete()
  File "/Library/Python/2.7/site-packages/mrjob/emr.py", line 1767, in _wait_for_job_to_complete
    raise Exception(msg)
Exception: Job on job flow j-273B8CMZ72WBW failed with status TERMINATING: Shut down as step failed

Memory Error on emr => running locally -- ran out of disk space in map step



In [35]:

    
!python MRJob_PreprocessGraph_topic.py all-pages-indexed-out.txt --file 'wiki_topics.txt' --jobconf 'G=15192277' --jobconf 'topicIdxFilename=wiki_topics.txt'









    



using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151107.010126.858967

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph_topic.davidadams.20151107.010126.858967/step-0-mapper_part-00000
^CTraceback (most recent call last):
  File "MRJob_PreprocessGraph_topic.py", line 73, in <module>
    MRJobPreprocessGraphTopic.run()
  File "/Library/Python/2.7/site-packages/mrjob/job.py", line 461, in run
    mr_job.execute()
  File "/Library/Python/2.7/site-packages/mrjob/job.py", line 479, in execute
    super(MRJob, self).execute()
  File "/Library/Python/2.7/site-packages/mrjob/launch.py", line 153, in execute
    self.run_job()
  File "/Library/Python/2.7/site-packages/mrjob/launch.py", line 216, in run_job
    runner.run()
  File "/Library/Python/2.7/site-packages/mrjob/runner.py", line 470, in run
    self._run()
  File "/Library/Python/2.7/site-packages/mrjob/sim.py", line 173, in _run
    self._invoke_step(step_num, 'mapper')
  File "/Library/Python/2.7/site-packages/mrjob/sim.py", line 260, in _invoke_step
    working_dir, env)
  File "/Library/Python/2.7/site-packages/mrjob/inline.py", line 160, in _run_step
    child_instance.execute()
  File "/Library/Python/2.7/site-packages/mrjob/job.py", line 470, in execute
    self.run_mapper(self.options.step_num)
  File "/Library/Python/2.7/site-packages/mrjob/job.py", line 535, in run_mapper
    for out_key, out_value in mapper(key, value) or ():
  File "MRJob_PreprocessGraph_topic.py", line 51, in mapper
    value['topic']=self.topic_labels[key]
KeyboardInterrupt



In [ ]:



In [ ]: