DATASCI W261: Machine Learning at Scale

Week 10, Homework 9

Group C: Katrina Adams, Eric Freeman, Doug Kelley

kradams@ischool.berkeley.edu, ericfreeman@ischool.berkeley.edu, kelleydac@ischool.berkeley.edu

3 November 2015


Some Imports


In [15]:
#%matplotlib inline

#import matplotlib.pyplot as plt
#import numpy as np

%cd ~/Documents/W261/hw9/


/Users/davidadams/Documents/W261/hw9

HW9.0:
What is PageRank and what is it used for in the context of web search?
What modifications have to be made to the webgraph in order to leverage the machinery of Markov Chains to compute the steady stade distibuton?
OPTIONAL: In topic-specific pagerank, how can we insure that the irreducible property is satified? (HINT: see HW9.4)

__HW 9.0 ANSWER__


HW 9.1: MRJob implementation of basic PageRank

Write a basic MRJob implementation of the iterative PageRank algorithm that takes sparse adjacency lists as input (as explored in HW 7).
Make sure that your implementation utilizes teleportation (1-damping/the number of nodes in the network), and further, distributes the mass of dangling nodes with each iteration so that the output of each iteration is correctly normalized (sums to 1).
[NOTE: The PageRank algorithm assumes that a random surfer (walker), starting from a random web page, chooses the next page to which it will move by clicking at random, with probability d, one of the hyperlinks in the current page. This probability is represented by a so-called ‘damping factor’ d, where d ∈ (0, 1). Otherwise, with probability (1 − d), the surfer jumps to any web page in the network. If a page is a dangling end, meaning it has no outgoing hyperlinks, the random surfer selects an arbitrary web page from a uniform distribution and “teleports” to that page]

As you build your code, use the test data

s3://ucb-mids-mls-networks/PageRank-test.txt

with teleportation parameter set to 0.15 (1-d, where d, the damping factor is set to 0.85), and crosscheck your work with the true result, displayed in the first image in the Wikipedia article:

https://en.wikipedia.org/wiki/PageRank

and here for reference are the corresponding PageRank probabilities:

A,0.033
B,0.384
C,0.343
D,0.039
E,0.081
F,0.039
G,0.016
H,0.016
I,0.016
J,0.016
K,0.016


In [ ]:
'''

Map: emit graph and PR/(num outgoing nodes)
Reduce: sum PR's and emit with graph

'''

In [ ]:
%%writefile MRJob_NumberOfNodes.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.step import MRStep   
from ast import literal_eval
from sys import maxint

class MRJobNumNodes(MRJob):
    

    def steps(self):
        mapper_only = 0
        if mapper_only:
            return [MRStep(mapper = self.mapper)]
        return [MRStep(
                mapper = self.mapper,
                reducer = self.reducer
                ),
               MRStep(
                reducer = self.reducer_count
                )]
    
    def mapper(self, _, line):
        line = line.strip().split('\t')
        key = line[0]
        value = literal_eval(line[1])
        destnodes = value.keys()
        
        yield key, None
        
        for node in destnodes:
            yield node, None
           
    def reducer(self,node,_):
        yield None, node
        
    def reducer_count(self,_,nodes):
        count = 0
        for node in nodes:
            count+=1
        yield None, count

        
if __name__ == '__main__':
    MRJobNumNodes.run()

In [ ]:
%%writefile MRJob_PreprocessGraph.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.step import MRStep   
from ast import literal_eval
from sys import maxint


class MRJobPreprocessGraph(MRJob):
    

    def steps(self):
        mapper_only = 0
        if mapper_only:
            return [MRStep(mapper_init = self.mapper_init,
                           mapper = self.mapper)]
        return [MRStep(
                mapper_init = self.mapper_init,
                mapper = self.mapper,
                reducer = self.reducer
                )]
    
    def mapper_init(self):
        N = 11
        self.PR = 1.0/N
        
    
    def mapper(self, _, line):
        line = line.strip().split('\t')
        key = line[0]
        edges = literal_eval(line[1])
        value = [self.PR, edges]
        yield key, value
        
        for node in edges.keys():
            yield node, [self.PR]

           
    def reducer(self,key,value):
        PR = 0
        edges = {}
        for v in value:
            if len(v)==1:
                PR = v[0]
            else:
                PR = v[0]
                edges = v[1]
        yield key, [PR, edges]

    
    '''
    def mapper(self, _, line):
        N = 11
        line = line.strip().split('\t')
        key = line[0]
        edges = literal_eval(line[1])
        PR = 1.0/N
        value = [PR, edges]
        yield None,{key:value}

           
    def reducer(self,_,data):
        for line in data:
            key = line.keys()[0]
            value = line[key]
            yield key, value
    '''
        
if __name__ == '__main__':
    MRJobPreprocessGraph.run()

In [101]:
!python MRJob_PreprocessGraph.py PageRank-test.txt > PageRank-test_input2.txt


using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233/step-0-mapper_part-00000
Counters from step 1:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233/step-0-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233/step-0-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233/step-0-reducer_part-00000
Counters from step 1:
  (no counters found)
Moving /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233/step-0-reducer_part-00000 -> /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233/output/part-00000
Streaming final output from /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233/output
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.171519.674233

In [102]:
!cat PageRank-test_input2.txt


"A"	[0.09090909090909091, {}]
"B"	[0.09090909090909091, {"C": 1}]
"C"	[0.09090909090909091, {"B": 1}]
"D"	[0.09090909090909091, {"A": 1, "B": 1}]
"E"	[0.09090909090909091, {"B": 1, "D": 1, "F": 1}]
"F"	[0.09090909090909091, {"B": 1, "E": 1}]
"G"	[0.09090909090909091, {"B": 1, "E": 1}]
"H"	[0.09090909090909091, {"B": 1, "E": 1}]
"I"	[0.09090909090909091, {"B": 1, "E": 1}]
"J"	[0.09090909090909091, {"E": 1}]
"K"	[0.09090909090909091, {"E": 1}]

In [ ]:
%%writefile MRJob_PageRank.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.step import MRStep   
from ast import literal_eval
from mrjob.compat import jobconf_from_env

class MRJobPageRank(MRJob):
    

    def steps(self):
        mapper_only = 0
        if mapper_only:
            return [MRStep(mapper = self.mapper)]
        
        #return [MRStep(
        #        mapper = self.mapper,
        #        reducer = self.reducer
        #        )]
        
        return [MRStep(
                mapper = self.mapper,
                reducer = self.reducer
                ),
               MRStep(
                mapper = self.mapper_redistr)]
    
    dangling_mass = 1000
    
    def mapper(self, _, line):
        line = line.strip().split('\t')
        key = line[0]
        value = literal_eval(line[1])

        PR = value[0]
        edges = value[1]
        
        yield key, [None,edges]
        
        destnodes = edges.keys()
        outdeg = len(destnodes)
        
        if outdeg==0:
            yield '*m',[PR]
        
        for node in destnodes:
            yield node, [1.0*PR/outdeg]
            
            
    def reducer(self,node,data):#, parameter=dangling_mass):
        #global dangling_mass
        PR = 0
        edges = {}
        for value in data:
            yield node, data
            if len(value)==1:
                PR+=value[0]
            else:
                edges = value[1]
        
        if node=='*m':
            self.dangling_mass = PR
        else:
            yield node, [PR, edges]
        
    
    def mapper_redistr(self, node, data):#, parameter=dangling_mass):
        G = int(jobconf_from_env('G'))
        alpha = float(jobconf_from_env('alpha'))
        m = self.dangling_mass
        
        PR = data[0]
        edges = data[1]
        
        PR_adj = alpha*(1.0/G)+(1-alpha)*(m/G+PR)
        
        yield node, [PR_adj, edges]

   
if __name__ == '__main__':
    MRJobPageRank.run()

In [43]:
%%writefile MRJob_PageRank.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.step import MRStep   
from ast import literal_eval
from mrjob.compat import jobconf_from_env
import sys

class MRJobPageRank(MRJob):
    

    def steps(self):
        mapper_only = 0
        if mapper_only:
            return [MRStep(mapper = self.mapper)]
        
        
        return [MRStep(
                mapper = self.mapper_firstiter)]+\
                [MRStep(
                mapper = self.mapper,
                combiner = self.combiner,
                reducer = self.reducer)]*10
    
    def mapper_firstiter(self, _, line):
        line = line.strip().split('\t')
        key = line[0]
        value = literal_eval(line[1])

        PR = value[0]
        edges = value[1]
        
        yield key, [None,edges]
        
        destnodes = edges.keys()
        outdeg = len(destnodes)
        
        if outdeg==0:
            yield '*m',[PR]
        
        for node in destnodes:
            yield node, [1.0*PR/outdeg]
            

    def mapper(self, key, value):

        sys.stderr.write("MAPPER INPUT: ({0}, {1})\n".format(key,value))
        #sys.stderr.write("MAPPER")
        
        PR = value[0]
        edges = value[1]
        
        yield key, [None,edges]
        
        destnodes = edges.keys()
        outdeg = len(destnodes)
        
        if outdeg==0:
            yield '*m',[PR]
        
        for node in destnodes:
            yield node, [1.0*PR/outdeg]
            
            
    def combiner(self,node,data):
        PR = 0
        edges = {}
        for value in data:
            if value[0]==None:
                yield node, value
            else:
                PR+=value[0]
        yield node, [PR]
    
    
    def reducer(self,node,data):
        
        #sys.stderr.write("REDUCER NODE: {0}\n".format(node))
        
        G = int(jobconf_from_env('G'))
        alpha = float(jobconf_from_env('alpha'))
        
        PR = 0
        edges = {}
        
        for value in data:
            #yield node, value
            if value[0]==None:
                edges = value[1]
                
            else:
                PR+=value[0]
        
        if node=='*m':
            self.m = PR
        
        #sys.stderr.write("REDUCER PR: {0}\n".format(PR))
        
        else:
            PR_adj = alpha*(1.0/G)+(1-alpha)*(1.0*self.m/G+PR) 
            #yield node, PR
            #yield node, PR_adj
            
            #line = node+'\t'+str([PR_adj, edges])
            #yield line, ''
            
            yield node, [PR_adj, edges]

   
if __name__ == '__main__':
    MRJobPageRank.run()


Overwriting MRJob_PageRank.py

In [19]:
%load_ext autoreload
%autoreload 2

from MRJob_NumberOfNodes import MRJobNumNodes
from MRJob_PreprocessGraph import MRJobPreprocessGraph
from MRJob_PageRank import MRJobPageRank

def numNodes(graphfilename):
    #mr_job = MRJobNumNodes(args=[graphfilename, "--strict-protocols", "-r", "emr", "--num-ec2-instances", "2", "--ec2-task-instance-type", "m1.small", "--pool-emr-job-flows", "--max-hours-idle=1"])
    mr_job_numnodes = MRJobNumNodes(args=[graphfilename])

    with mr_job_numnodes.make_runner() as runner:
        runner.run()
        for line in runner.stream_output():
            null,count =  mr_job_numnodes.parse_output_line(line)
            print "There are "+str(count)+" nodes in the graph."
                    
    return None

def preprocessGraph(graphfilename):
    #mr_job = MRJobNumNodes(args=[graphfilename, "--strict-protocols", "-r", "emr", "--num-ec2-instances", "2", "--ec2-task-instance-type", "m1.small", "--pool-emr-job-flows", "--max-hours-idle=1"])
    mr_job_preprocess = MRJobPreprocessGraph(args=[graphfilename])
    outputfilename = graphfilename.split('.')[0]+'_input.txt'
    with mr_job_preprocess.make_runner() as runner:
        with open(outputfilename, 'w') as f:
            runner.run()
            for line in runner.stream_output():
                node,value =  mr_job_preprocess.parse_output_line(line)
                f.write(node+'\t'+str(value)+'\n')
                    
    return None


def pagerank(graphinputfilename):
    #mr_job = MRJobNumNodes(args=[graphfilename, "--strict-protocols", "-r", "emr", "--num-ec2-instances", "2", "--ec2-task-instance-type", "m1.small", "--pool-emr-job-flows", "--max-hours-idle=1"])
    mr_job_pr1 = MRJobPageRank(args=[graphinputfilename,'--jobconf','alpha=0.15','--jobconf','G=11'])
    danglingmass= 0
    with mr_job_pr1.make_runner() as runner:
        runner.run()
        with open(graphinputfilename, 'w+') as f:
            for line in runner.stream_output():
                node,value =  mr_job_pr1.parse_output_line(line)
                f.write(node+'\t'+str(value)+'\n')
                print line

    return None




def hw9_1():

    graphfilename = "PageRank-test.txt"
    #numNodes(graphfilename)
    
    preprocessGraph(graphfilename)

    graphinputfilename = graphfilename.split('.')[0]+'_input.txt'
    
    #pagerank_step1(graphinputfilename, alpha, G)
    
    #pagerank_step1(graphinputfilename, alpha, G)
    
    for i in range(1):
        print "Iteration",i
        pagerank(graphinputfilename)
    
    
    
    
    return None
        
  

hw9_1()


WARNING:mrjob.runner:
WARNING:mrjob.runner:PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols
WARNING:mrjob.runner:
WARNING:mrjob.runner:
WARNING:mrjob.runner:PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols
WARNING:mrjob.runner:
Iteration 0
"A"	[0.059297520661157024, {}]

"B"	[0.3168732782369146, {"C": 1}]

"C"	[0.09793388429752066, {"B": 1}]

"D"	[0.046418732782369146, {"A": 1, "B": 1}]

"E"	[0.32975206611570246, {"B": 1, "D": 1, "F": 1}]

"F"	[0.046418732782369146, {"B": 1, "E": 1}]

"G"	[0.02066115702479339, {"B": 1, "E": 1}]

"H"	[0.02066115702479339, {"B": 1, "E": 1}]

"I"	[0.02066115702479339, {"B": 1, "E": 1}]

"J"	[0.02066115702479339, {"E": 1}]

"K"	[0.02066115702479339, {"E": 1}]

HW 9.2: Exploring PageRank teleportation and network plots

In order to overcome problems such as disconnected components, the damping factor (a typical value for d is 0.85) can be varied.
Using the graph in HW1, plot the test graph (using networkx, https://networkx.github.io/) for several values of the damping parameter alpha, so that each nodes radius is proportional to its PageRank score. In particular you should do this for the following damping factors: [0,0.25,0.5,0.75, 0.85, 1].
Note your plots should look like the following:

https://en.wikipedia.org/wiki/PageRank#/media/File:PageRanks-Example.svg

HW 9.3: Applying PageRank to the Wikipedia hyperlinks network

Run your PageRank implementation on the Wikipedia dataset for 10 iterations, and display the top 100 ranked nodes (with alpha = 0.85).

Run your PageRank implementation on the Wikipedia dataset for 50 iterations, and display the top 100 ranked nodes (with teleportation factor of 0.15).
Have the top 100 ranked pages changed? Comment on your findings. Plot both 100 curves.


In [61]:
'''
    OLD PAGE RANK WITH GLOBAL VAR FOR DANGLING MASS
'''


%%writefile MRJob_PageRank.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.step import MRStep   
from ast import literal_eval
from mrjob.compat import jobconf_from_env
import sys

class MRJobPageRank(MRJob):
    

    def steps(self):
        mapper_only = 0
        if mapper_only:
            return [MRStep(mapper = self.mapper)]
        
        return [MRStep(
                mapper = self.mapper_firstiter,
                reducer = self.reducer
                ),
               MRStep(
                mapper = self.mapper_redistr)]+\
                [MRStep(
                mapper = self.mapper,
                reducer = self.reducer
                ),
               MRStep(
                mapper = self.mapper_redistr)]*10
    
    dangling_mass = None
    
    
    
    def mapper_firstiter(self, _, line):
        line = line.strip().split('\t')
        key = line[0]
        value = literal_eval(line[1])

        PR = value[0]
        edges = value[1]
        
        yield key, [None,edges]
        
        destnodes = edges.keys()
        outdeg = len(destnodes)
        
        if outdeg==0:
            yield '*m',[PR]
        
        for node in destnodes:
            yield node, [1.0*PR/outdeg]
            

    def mapper(self, key, value):

        #sys.stderr.write("\nMAPPER INPUT: ({0}, {1})\n".format(key,value))
        #sys.stderr.write("MAPPER")
        
        PR = value[0]
        edges = value[1]
        
        yield key, [None,edges]
        
        destnodes = edges.keys()
        outdeg = len(destnodes)
        
        if outdeg==0:
            yield '*m',[PR]
        
        for node in destnodes:
            yield node, [1.0*PR/outdeg]
  
            
    def reducer(self,node,data):
        
        #sys.stderr.write("REDUCER INPUT: ({0}, {1})\n".format(node,data))
        #global dangling_mass
        PR = 0
        edges = {}
        for value in data:
            yield node, data
            if len(value)==1:
                PR+=value[0]
            else:
                edges = value[1]
        
        if node=='*m':
            #dangling_mass = PR
            with open('danglingmass.txt', 'w+') as mfile:
                mfile.write(str(PR))
        else:
            yield node, [PR, edges]
        
    
    def mapper_redistr(self, node, data, parameter=dangling_mass):
        
        #sys.stderr.write("MAPPER2 INPUT: ({0}, {1})\n".format(node,data))
        
        G = int(jobconf_from_env('G'))
        alpha = float(jobconf_from_env('alpha'))
        #global dangling_mass
        with open('danglingmass.txt', 'r') as mfile:
                mline = mfile.readline()
        m = float(mline.strip())
        #m = dangling_mass
        
        #yield node, G*alpha
        PR = data[0]
        edges = data[1]
        
        PR_adj = alpha*(1.0/G)+(1-alpha)*(m/G+PR)
        
        yield node, [PR_adj, edges]

   
if __name__ == '__main__':
    MRJobPageRank.run()


Overwriting MRJob_PageRank.py

Testing boto


In [84]:
import boto
from boto.s3.connection import S3Connection
from boto.s3.key import Key

conn = S3Connection('XXXXX', 'XXXXX')

b = conn.get_bucket('ucb-mids-mls-katieadams')

k = Key(b)
k.key = 'danglingmass.txt'
m = 0.000056789
k.set_contents_from_string(str(m))
#k.set_contents_from_filename('danglingmass.txt')
m = float(k.get_contents_as_string())*2
print m
k.get_contents_to_filename('danglingmass_fromS3.txt')
b.delete_key(k)


0.000113578
Out[84]:
<Key: ucb-mids-mls-katieadams,<Key: ucb-mids-mls-katieadams,danglingmass.txt>>

2nd MOST CURRENT PAGE RANK

write to S3 file for dangling mass, there might be a problem with deleting the dangling mass file if run on multiple instances on emr (haven't tested)


In [99]:
%%writefile MRJob_PageRank.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.step import MRStep   
from ast import literal_eval
from mrjob.compat import jobconf_from_env
import sys
import boto
from boto.s3.connection import S3Connection
from boto.s3.key import Key


class MRJobPageRank(MRJob):


    def steps(self):
        mapper_only = 0
        if mapper_only:
            return [MRStep(mapper = self.mapper)]
        
        return [MRStep(
                mapper = self.mapper_firstiter,
                reducer_init = self.reducer_init,
                reducer = self.reducer
                ),
               MRStep(
                mapper = self.mapper_redistr)]+\
                [MRStep(
                mapper = self.mapper,
                reducer_init = self.reducer_init,
                reducer = self.reducer
                ),
               MRStep(
                mapper = self.mapper_redistr)]*10
    
    dangling_mass = None
    
    
    
    def mapper_firstiter(self, _, line):
        line = line.strip().split('\t')
        key = line[0]
        value = literal_eval(line[1])

        PR = value[0]
        edges = value[1]
        
        yield key, [None,edges]
        
        destnodes = edges.keys()
        outdeg = len(destnodes)
        
        if outdeg==0:
            yield '*m',[PR]
        
        for node in destnodes:
            yield node, [1.0*PR/outdeg]
            

    def mapper(self, key, value):

        #sys.stderr.write("\nMAPPER INPUT: ({0}, {1})\n".format(key,value))
        #sys.stderr.write("MAPPER")
        
        PR = value[0]
        edges = value[1]
        
        yield key, [None,edges]
        
        destnodes = edges.keys()
        outdeg = len(destnodes)
        
        if outdeg==0:
            yield '*m',[PR]
        
        for node in destnodes:
            yield node, [1.0*PR/outdeg]
  
            
    def reducer_init(self):
        conn = S3Connection('XXXXX', 'XXXXX')
        b = conn.get_bucket('ucb-mids-mls-katieadams')
        self.k = Key(b)
        self.k.key = 'danglingmass.txt'
        b.delete_key(self.k)
        
    
    def reducer(self,node,data):
        
        
        #sys.stderr.write("REDUCER INPUT: ({0}, {1})\n".format(node,data))
        #global dangling_mass
        PR = 0
        edges = {}
        for value in data:
            yield node, data
            if len(value)==1:
                PR+=value[0]
            else:
                edges = value[1]
        
        if node=='*m':
            self.k.set_contents_from_string(str(PR))
            #with open('danglingmass.txt', 'w+') as mfile:
            #    mfile.write(str(PR))
        else:
            yield node, [PR, edges]
        
    
    def mapper_redistr(self, node, data):
        
        conn = S3Connection('XXXXXX', 'XXXXXX')
        b = conn.get_bucket('ucb-mids-mls-katieadams')
        k = Key(b)
        k.key = 'danglingmass.txt'
        
        #sys.stderr.write("MAPPER2 INPUT: ({0}, {1})\n".format(node,data))
        
        G = int(jobconf_from_env('G'))
        alpha = float(jobconf_from_env('alpha'))
        
        m = float(k.get_contents_as_string())

        
        #with open('danglingmass.txt', 'r') as mfile:
        #        mline = mfile.readline()
        #m = float(mline.strip())

        PR = data[0]
        edges = data[1]
        
        PR_adj = alpha*(1.0/G)+(1-alpha)*(m/G+PR)
        
        yield node, [PR_adj, edges]

   
if __name__ == '__main__':
    MRJobPageRank.run()


Overwriting MRJob_PageRank.py

In [100]:
!python MRJob_PageRank.py PageRank-test_input.txt --jobconf alpha=0.15 --jobconf G=11 --file danglingmass.txt


using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-0-mapper_part-00000
Counters from step 1:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-0-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-0-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-0-reducer_part-00000
Counters from step 1:
  Unencodable output:
    TypeError: 29
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-1-mapper_part-00000
Counters from step 2:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-2-mapper_part-00000
Counters from step 3:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-2-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-2-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-2-reducer_part-00000
Counters from step 3:
  Unencodable output:
    TypeError: 29
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-3-mapper_part-00000
Counters from step 4:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-4-mapper_part-00000
Counters from step 5:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-4-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-4-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-4-reducer_part-00000
Counters from step 5:
  Unencodable output:
    TypeError: 29
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-5-mapper_part-00000
Counters from step 6:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-6-mapper_part-00000
Counters from step 7:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-6-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-6-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-6-reducer_part-00000
Counters from step 7:
  Unencodable output:
    TypeError: 29
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-7-mapper_part-00000
Counters from step 8:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-8-mapper_part-00000
Counters from step 9:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-8-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-8-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-8-reducer_part-00000
Counters from step 9:
  Unencodable output:
    TypeError: 29
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-9-mapper_part-00000
Counters from step 10:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-10-mapper_part-00000
Counters from step 11:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-10-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-10-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-10-reducer_part-00000
Counters from step 11:
  Unencodable output:
    TypeError: 29
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-11-mapper_part-00000
Counters from step 12:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-12-mapper_part-00000
Counters from step 13:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-12-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-12-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-12-reducer_part-00000
Counters from step 13:
  Unencodable output:
    TypeError: 29
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-13-mapper_part-00000
Counters from step 14:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-14-mapper_part-00000
Counters from step 15:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-14-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-14-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-14-reducer_part-00000
Counters from step 15:
  Unencodable output:
    TypeError: 29
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-15-mapper_part-00000
Counters from step 16:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-16-mapper_part-00000
Counters from step 17:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-16-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-16-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-16-reducer_part-00000
Counters from step 17:
  Unencodable output:
    TypeError: 29
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-17-mapper_part-00000
Counters from step 18:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-18-mapper_part-00000
Counters from step 19:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-18-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-18-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-18-reducer_part-00000
Counters from step 19:
  Unencodable output:
    TypeError: 29
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-19-mapper_part-00000
Counters from step 20:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-20-mapper_part-00000
Counters from step 21:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-20-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-20-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-20-reducer_part-00000
Counters from step 21:
  Unencodable output:
    TypeError: 29
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-21-mapper_part-00000
Counters from step 22:
  (no counters found)
Moving /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/step-21-mapper_part-00000 -> /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/output/part-00000
Streaming final output from /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691/output
"A"	[0.03278149364164708, {}]
"B"	[0.38508446172839705, {"C": 1}]
"C"	[0.34222676993466455, {"B": 1}]
"D"	[0.03908709259192743, {"A": 1, "B": 1}]
"E"	[0.08088569410129864, {"B": 1, "D": 1, "F": 1}]
"F"	[0.03908709259192743, {"B": 1, "E": 1}]
"G"	[0.016169479082034546, {"B": 1, "E": 1}]
"H"	[0.016169479082034546, {"B": 1, "E": 1}]
"I"	[0.016169479082034546, {"B": 1, "E": 1}]
"J"	[0.016169479082034546, {"E": 1}]
"K"	[0.016169479082034546, {"E": 1}]
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.161807.922691

MOST CURRENT PAGE RANK

write mass to every node (loop over indices)

Preprocess Indexed Graph


In [155]:
%%writefile MRJob_PreprocessGraph.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.step import MRStep   
from ast import literal_eval
from mrjob.compat import jobconf_from_env


class MRJobPreprocessGraph(MRJob):
    

    def steps(self):
        mapper_only = 0
        if mapper_only:
            return [MRStep(mapper_init = self.mapper_init,
                           mapper = self.mapper)]
        return [MRStep(
                mapper_init = self.mapper_init,
                mapper = self.mapper,
                reducer = self.reducer
                )]
    
    def mapper_init(self):
        G = int(jobconf_from_env('G'))
        self.PR = 1.0/G
        
    
    def mapper(self, _, line):
        line = line.strip().split('\t')
        key = line[0]
        edges = literal_eval(line[1])
        value = [self.PR, edges]
        yield int(key), value
        
        for node in edges.keys():
            yield int(node), [self.PR]

           
    def reducer(self,key,value):
        PR = 0
        edges = {}
        for v in value:
            if len(v)==1:
                PR = v[0]
            else:
                PR = v[0]
                edges = v[1]
        yield int(key), [PR, edges]

    
        
if __name__ == '__main__':
    MRJobPreprocessGraph.run()


Overwriting MRJob_PreprocessGraph.py

In [114]:
!python MRJob_PreprocessGraph.py PageRank-testIndexed.txt --jobconf 'G=11' > PageRank-testIndexed_input.txt


using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283/step-0-mapper_part-00000
Counters from step 1:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283/step-0-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283/step-0-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283/step-0-reducer_part-00000
Counters from step 1:
  (no counters found)
Moving /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283/step-0-reducer_part-00000 -> /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283/output/part-00000
Streaming final output from /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283/output
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.173105.524283

In [160]:
%%writefile MRJob_PageRank.py
#!/usr/bin/env python
from mrjob.job import MRJob
from mrjob.step import MRStep   
from ast import literal_eval
from mrjob.compat import jobconf_from_env


class MRJobPageRank(MRJob):

    
    
    def configure_options(self):
        super(MRJobPageRank, self).configure_options()
        
        self.add_passthrough_option(
                '--iterations',
                dest = 'iterations',
                default = 10,
                type='int',
                help='number of iterations to run')
    

    def steps(self):        
        
        mapper_only = 0
        if mapper_only:
            return [MRStep(mapper = self.mapper)]
        
        return [MRStep(
                mapper = self.mapper_firstiter,
                reducer = self.reducer
                ),
               MRStep(
                reducer = self.reducer_redistr)]+\
                [MRStep(
                mapper = self.mapper,
                reducer = self.reducer
                ),
               MRStep(
                reducer = self.reducer_redistr)]*self.options.iterations
        
    
    def mapper_firstiter(self, _, line):
        line = line.strip().split('\t')
        key = line[0]
        value = literal_eval(line[1])

        PR = value[0]
        edges = value[1]
        
        yield int(key), [None,edges]
        
        destnodes = edges.keys()
        outdeg = len(destnodes)
        
        if outdeg==0:
            yield '*m',[PR]
        
        for node in destnodes:
            yield int(node), [1.0*PR/outdeg]
            

    def mapper(self, key, value):

        PR = value[0]
        edges = value[1]
        
        yield int(key), [None,edges]
        
        destnodes = edges.keys()
        outdeg = len(destnodes)
        
        if outdeg==0:
            yield '*m',[PR]
        
        for node in destnodes:
            yield int(node), [1.0*PR/outdeg]
  
            
    def reducer(self,node,data):
        
        alpha = float(jobconf_from_env('alpha'))
        G = int(jobconf_from_env('G'))
        
        PR = 0
        edges = {}
        for value in data:
            if len(value)==1:
                PR+=value[0]
            else:
                edges = value[1]
        
        if node=='*m':
            PR_adj = (1.0/G)*(1.0-alpha)*PR
            for i in range(G):
                yield i+1, [PR_adj]
        else:
            PR_adj = alpha*(1.0/G)+(1.0-alpha)*PR
            yield int(node), [PR_adj, edges]
        
    
    def reducer_redistr(self, node, data):
        
        PR = 0
        edges = {}
        for value in data:
            PR+=value[0]
            if len(value)>1:
                edges = value[1]
        
        yield int(node), [PR, edges]
        
        
   
if __name__ == '__main__':
    MRJobPageRank.run()


Overwriting MRJob_PageRank.py

In [146]:
!python MRJob_PageRank.py PageRank-testIndexed_input.txt --jobconf alpha=0.15 --jobconf G=11 --iterations 50


using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-0-mapper_part-00000
Counters from step 1:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-0-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-0-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-0-reducer_part-00000
Counters from step 1:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-1-mapper_part-00000
Counters from step 2:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-1-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-1-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-1-reducer_part-00000
Counters from step 2:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-2-mapper_part-00000
Counters from step 3:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-2-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-2-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-2-reducer_part-00000
Counters from step 3:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-3-mapper_part-00000
Counters from step 4:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-3-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-3-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-3-reducer_part-00000
Counters from step 4:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-4-mapper_part-00000
Counters from step 5:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-4-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-4-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-4-reducer_part-00000
Counters from step 5:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-5-mapper_part-00000
Counters from step 6:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-5-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-5-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-5-reducer_part-00000
Counters from step 6:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-6-mapper_part-00000
Counters from step 7:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-6-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-6-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-6-reducer_part-00000
Counters from step 7:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-7-mapper_part-00000
Counters from step 8:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-7-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-7-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-7-reducer_part-00000
Counters from step 8:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-8-mapper_part-00000
Counters from step 9:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-8-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-8-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-8-reducer_part-00000
Counters from step 9:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-9-mapper_part-00000
Counters from step 10:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-9-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-9-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-9-reducer_part-00000
Counters from step 10:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-10-mapper_part-00000
Counters from step 11:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-10-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-10-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-10-reducer_part-00000
Counters from step 11:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-11-mapper_part-00000
Counters from step 12:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-11-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-11-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-11-reducer_part-00000
Counters from step 12:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-12-mapper_part-00000
Counters from step 13:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-12-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-12-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-12-reducer_part-00000
Counters from step 13:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-13-mapper_part-00000
Counters from step 14:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-13-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-13-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-13-reducer_part-00000
Counters from step 14:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-14-mapper_part-00000
Counters from step 15:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-14-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-14-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-14-reducer_part-00000
Counters from step 15:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-15-mapper_part-00000
Counters from step 16:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-15-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-15-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-15-reducer_part-00000
Counters from step 16:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-16-mapper_part-00000
Counters from step 17:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-16-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-16-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-16-reducer_part-00000
Counters from step 17:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-17-mapper_part-00000
Counters from step 18:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-17-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-17-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-17-reducer_part-00000
Counters from step 18:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-18-mapper_part-00000
Counters from step 19:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-18-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-18-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-18-reducer_part-00000
Counters from step 19:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-19-mapper_part-00000
Counters from step 20:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-19-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-19-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-19-reducer_part-00000
Counters from step 20:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-20-mapper_part-00000
Counters from step 21:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-20-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-20-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-20-reducer_part-00000
Counters from step 21:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-21-mapper_part-00000
Counters from step 22:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-21-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-21-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-21-reducer_part-00000
Counters from step 22:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-22-mapper_part-00000
Counters from step 23:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-22-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-22-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-22-reducer_part-00000
Counters from step 23:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-23-mapper_part-00000
Counters from step 24:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-23-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-23-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-23-reducer_part-00000
Counters from step 24:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-24-mapper_part-00000
Counters from step 25:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-24-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-24-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-24-reducer_part-00000
Counters from step 25:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-25-mapper_part-00000
Counters from step 26:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-25-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-25-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-25-reducer_part-00000
Counters from step 26:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-26-mapper_part-00000
Counters from step 27:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-26-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-26-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-26-reducer_part-00000
Counters from step 27:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-27-mapper_part-00000
Counters from step 28:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-27-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-27-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-27-reducer_part-00000
Counters from step 28:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-28-mapper_part-00000
Counters from step 29:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-28-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-28-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-28-reducer_part-00000
Counters from step 29:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-29-mapper_part-00000
Counters from step 30:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-29-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-29-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-29-reducer_part-00000
Counters from step 30:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-30-mapper_part-00000
Counters from step 31:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-30-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-30-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-30-reducer_part-00000
Counters from step 31:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-31-mapper_part-00000
Counters from step 32:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-31-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-31-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-31-reducer_part-00000
Counters from step 32:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-32-mapper_part-00000
Counters from step 33:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-32-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-32-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-32-reducer_part-00000
Counters from step 33:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-33-mapper_part-00000
Counters from step 34:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-33-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-33-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-33-reducer_part-00000
Counters from step 34:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-34-mapper_part-00000
Counters from step 35:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-34-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-34-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-34-reducer_part-00000
Counters from step 35:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-35-mapper_part-00000
Counters from step 36:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-35-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-35-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-35-reducer_part-00000
Counters from step 36:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-36-mapper_part-00000
Counters from step 37:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-36-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-36-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-36-reducer_part-00000
Counters from step 37:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-37-mapper_part-00000
Counters from step 38:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-37-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-37-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-37-reducer_part-00000
Counters from step 38:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-38-mapper_part-00000
Counters from step 39:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-38-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-38-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-38-reducer_part-00000
Counters from step 39:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-39-mapper_part-00000
Counters from step 40:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-39-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-39-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-39-reducer_part-00000
Counters from step 40:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-40-mapper_part-00000
Counters from step 41:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-40-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-40-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-40-reducer_part-00000
Counters from step 41:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-41-mapper_part-00000
Counters from step 42:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-41-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-41-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-41-reducer_part-00000
Counters from step 42:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-42-mapper_part-00000
Counters from step 43:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-42-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-42-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-42-reducer_part-00000
Counters from step 43:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-43-mapper_part-00000
Counters from step 44:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-43-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-43-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-43-reducer_part-00000
Counters from step 44:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-44-mapper_part-00000
Counters from step 45:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-44-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-44-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-44-reducer_part-00000
Counters from step 45:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-45-mapper_part-00000
Counters from step 46:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-45-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-45-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-45-reducer_part-00000
Counters from step 46:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-46-mapper_part-00000
Counters from step 47:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-46-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-46-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-46-reducer_part-00000
Counters from step 47:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-47-mapper_part-00000
Counters from step 48:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-47-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-47-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-47-reducer_part-00000
Counters from step 48:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-48-mapper_part-00000
Counters from step 49:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-48-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-48-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-48-reducer_part-00000
Counters from step 49:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-49-mapper_part-00000
Counters from step 50:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-49-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-49-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-49-reducer_part-00000
Counters from step 50:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-50-mapper_part-00000
Counters from step 51:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-50-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-50-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-50-reducer_part-00000
Counters from step 51:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-51-mapper_part-00000
Counters from step 52:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-51-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-51-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-51-reducer_part-00000
Counters from step 52:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-52-mapper_part-00000
Counters from step 53:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-52-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-52-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-52-reducer_part-00000
Counters from step 53:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-53-mapper_part-00000
Counters from step 54:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-53-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-53-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-53-reducer_part-00000
Counters from step 54:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-54-mapper_part-00000
Counters from step 55:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-54-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-54-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-54-reducer_part-00000
Counters from step 55:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-55-mapper_part-00000
Counters from step 56:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-55-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-55-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-55-reducer_part-00000
Counters from step 56:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-56-mapper_part-00000
Counters from step 57:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-56-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-56-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-56-reducer_part-00000
Counters from step 57:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-57-mapper_part-00000
Counters from step 58:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-57-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-57-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-57-reducer_part-00000
Counters from step 58:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-58-mapper_part-00000
Counters from step 59:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-58-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-58-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-58-reducer_part-00000
Counters from step 59:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-59-mapper_part-00000
Counters from step 60:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-59-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-59-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-59-reducer_part-00000
Counters from step 60:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-60-mapper_part-00000
Counters from step 61:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-60-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-60-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-60-reducer_part-00000
Counters from step 61:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-61-mapper_part-00000
Counters from step 62:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-61-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-61-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-61-reducer_part-00000
Counters from step 62:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-62-mapper_part-00000
Counters from step 63:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-62-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-62-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-62-reducer_part-00000
Counters from step 63:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-63-mapper_part-00000
Counters from step 64:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-63-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-63-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-63-reducer_part-00000
Counters from step 64:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-64-mapper_part-00000
Counters from step 65:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-64-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-64-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-64-reducer_part-00000
Counters from step 65:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-65-mapper_part-00000
Counters from step 66:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-65-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-65-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-65-reducer_part-00000
Counters from step 66:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-66-mapper_part-00000
Counters from step 67:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-66-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-66-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-66-reducer_part-00000
Counters from step 67:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-67-mapper_part-00000
Counters from step 68:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-67-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-67-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-67-reducer_part-00000
Counters from step 68:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-68-mapper_part-00000
Counters from step 69:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-68-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-68-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-68-reducer_part-00000
Counters from step 69:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-69-mapper_part-00000
Counters from step 70:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-69-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-69-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-69-reducer_part-00000
Counters from step 70:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-70-mapper_part-00000
Counters from step 71:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-70-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-70-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-70-reducer_part-00000
Counters from step 71:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-71-mapper_part-00000
Counters from step 72:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-71-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-71-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-71-reducer_part-00000
Counters from step 72:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-72-mapper_part-00000
Counters from step 73:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-72-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-72-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-72-reducer_part-00000
Counters from step 73:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-73-mapper_part-00000
Counters from step 74:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-73-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-73-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-73-reducer_part-00000
Counters from step 74:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-74-mapper_part-00000
Counters from step 75:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-74-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-74-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-74-reducer_part-00000
Counters from step 75:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-75-mapper_part-00000
Counters from step 76:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-75-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-75-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-75-reducer_part-00000
Counters from step 76:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-76-mapper_part-00000
Counters from step 77:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-76-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-76-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-76-reducer_part-00000
Counters from step 77:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-77-mapper_part-00000
Counters from step 78:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-77-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-77-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-77-reducer_part-00000
Counters from step 78:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-78-mapper_part-00000
Counters from step 79:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-78-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-78-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-78-reducer_part-00000
Counters from step 79:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-79-mapper_part-00000
Counters from step 80:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-79-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-79-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-79-reducer_part-00000
Counters from step 80:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-80-mapper_part-00000
Counters from step 81:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-80-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-80-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-80-reducer_part-00000
Counters from step 81:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-81-mapper_part-00000
Counters from step 82:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-81-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-81-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-81-reducer_part-00000
Counters from step 82:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-82-mapper_part-00000
Counters from step 83:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-82-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-82-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-82-reducer_part-00000
Counters from step 83:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-83-mapper_part-00000
Counters from step 84:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-83-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-83-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-83-reducer_part-00000
Counters from step 84:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-84-mapper_part-00000
Counters from step 85:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-84-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-84-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-84-reducer_part-00000
Counters from step 85:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-85-mapper_part-00000
Counters from step 86:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-85-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-85-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-85-reducer_part-00000
Counters from step 86:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-86-mapper_part-00000
Counters from step 87:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-86-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-86-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-86-reducer_part-00000
Counters from step 87:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-87-mapper_part-00000
Counters from step 88:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-87-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-87-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-87-reducer_part-00000
Counters from step 88:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-88-mapper_part-00000
Counters from step 89:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-88-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-88-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-88-reducer_part-00000
Counters from step 89:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-89-mapper_part-00000
Counters from step 90:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-89-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-89-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-89-reducer_part-00000
Counters from step 90:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-90-mapper_part-00000
Counters from step 91:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-90-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-90-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-90-reducer_part-00000
Counters from step 91:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-91-mapper_part-00000
Counters from step 92:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-91-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-91-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-91-reducer_part-00000
Counters from step 92:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-92-mapper_part-00000
Counters from step 93:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-92-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-92-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-92-reducer_part-00000
Counters from step 93:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-93-mapper_part-00000
Counters from step 94:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-93-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-93-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-93-reducer_part-00000
Counters from step 94:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-94-mapper_part-00000
Counters from step 95:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-94-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-94-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-94-reducer_part-00000
Counters from step 95:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-95-mapper_part-00000
Counters from step 96:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-95-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-95-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-95-reducer_part-00000
Counters from step 96:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-96-mapper_part-00000
Counters from step 97:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-96-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-96-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-96-reducer_part-00000
Counters from step 97:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-97-mapper_part-00000
Counters from step 98:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-97-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-97-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-97-reducer_part-00000
Counters from step 98:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-98-mapper_part-00000
Counters from step 99:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-98-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-98-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-98-reducer_part-00000
Counters from step 99:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-99-mapper_part-00000
Counters from step 100:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-99-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-99-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-99-reducer_part-00000
Counters from step 100:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-100-mapper_part-00000
Counters from step 101:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-100-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-100-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-100-reducer_part-00000
Counters from step 101:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-101-mapper_part-00000
Counters from step 102:
  (no counters found)
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-101-mapper-sorted
> sort /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-101-mapper_part-00000
writing to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-101-reducer_part-00000
Counters from step 102:
  (no counters found)
Moving /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/step-101-reducer_part-00000 -> /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/output/part-00000
Streaming final output from /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498/output
1	[0.03278149315934598, {}]
10	[0.01616947901685868, {"5": 1}]
11	[0.01616947901685868, {"5": 1}]
2	[0.3844274414951183, {"3": 1}]
3	[0.34288379282680403, {"2": 1}]
4	[0.03908709209996824, {"1": 1, "2": 1}]
5	[0.0808856932345013, {"2": 1, "4": 1, "6": 1}]
6	[0.03908709209996824, {"2": 1, "5": 1}]
7	[0.01616947901685868, {"2": 1, "5": 1}]
8	[0.01616947901685868, {"2": 1, "5": 1}]
9	[0.01616947901685868, {"2": 1, "5": 1}]
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.180016.946498

In [150]:
!python MRJob_PageRank.py s3://ucb-mids-mls-katieadams/PageRank-testIndexed_input.txt --jobconf alpha=0.15 --jobconf G=11 --iterations 5


using configs in /Users/davidadams/.mrjob.conf
Traceback (most recent call last):
  File "MRJob_PageRank.py", line 123, in <module>
    MRJobPageRank.run()
  File "/Library/Python/2.7/site-packages/mrjob/job.py", line 461, in run
    mr_job.execute()
  File "/Library/Python/2.7/site-packages/mrjob/job.py", line 479, in execute
    super(MRJob, self).execute()
  File "/Library/Python/2.7/site-packages/mrjob/launch.py", line 153, in execute
    self.run_job()
  File "/Library/Python/2.7/site-packages/mrjob/launch.py", line 216, in run_job
    runner.run()
  File "/Library/Python/2.7/site-packages/mrjob/runner.py", line 470, in run
    self._run()
  File "/Library/Python/2.7/site-packages/mrjob/sim.py", line 164, in _run
    _error_on_bad_paths(self.fs, self._input_paths)
  File "/Library/Python/2.7/site-packages/mrjob/sim.py", line 549, in _error_on_bad_paths
    "None found in %s" % paths)
ValueError: At least one valid path is required. None found in ['s3://ucb-mids-mls-katieadams/PageRank-testIndexed_input.txt']

In [ ]:
!python MRJob_PageRank.py s3://ucb-mids-mls-katieadams/PageRank-testIndexed_input.txt --jobconf alpha=0.15 --jobconf G=11 --iterations 5 --output-dir s3://ucb-mids-mls-katieadams/outputtest --no-output

In [ ]:
!python MRJob_PageRank.py s3://ucb-mids-mls-katieadams/PageRank-testIndexed_input.txt --jobconf alpha=0.15 --jobconf G=11 --iterations 5 -r emr --num-ec2-instances 2 --ec2-task-instance-type m1.medium --output-dir s3://ucb-mids-mls-katieadams/outputtest --no-output --pool-emr-job-flows --max-hours-idle=1

In [153]:
!python MRJob_PageRank.py s3://ucb-mids-mls-katieadams/PageRank-testIndexed_input.txt --jobconf alpha=0.15 --jobconf G=11 --iterations 5 -r emr --num-ec2-instances 2 --ec2-task-instance-type m1.medium --output-dir s3://ucb-mids-mls-katieadams/outputtest --no-output


using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.184011.103955
writing master bootstrap script to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.184011.103955/b.py

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

Copying non-input files into s3://ucb-mids-mls-katieadams/tmp/MRJob_PageRank.davidadams.20151105.184011.103955/files/
Waiting 5.0s for S3 eventual consistency
Creating Elastic MapReduce job flow
Job flow created with ID: j-3GVS3SQZV97HL
Created new job flow j-3GVS3SQZV97HL
Job launched 31.3s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 62.6s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 93.9s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 125.3s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 156.5s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 187.9s ago, status STARTING: Configuring cluster software
Job launched 219.2s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 250.5s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 281.9s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 313.3s ago, status RUNNING: Running step
Job launched 344.6s ago, status RUNNING: Running step
Job launched 375.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 1 of 12)
Job launched 407.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 1 of 12)
Job launched 438.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 1 of 12)
Job launched 470.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 2 of 12)
Job launched 501.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 2 of 12)
Job launched 532.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 2 of 12)
Job launched 564.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 2 of 12)
Job launched 595.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 3 of 12)
Job launched 626.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 3 of 12)
Job launched 658.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 3 of 12)
Job launched 689.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 4 of 12)
Job launched 721.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 4 of 12)
Job launched 752.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 4 of 12)
Job launched 784.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 4 of 12)
Job launched 815.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 5 of 12)
Job launched 846.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 5 of 12)
Job launched 878.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 6 of 12)
Job launched 909.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 6 of 12)
Job launched 940.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 6 of 12)
Job launched 972.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 7 of 12)
Job launched 1003.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 7 of 12)
Job launched 1034.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 7 of 12)
Job launched 1066.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 8 of 12)
Job launched 1097.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 8 of 12)
Job launched 1129.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 8 of 12)
Job launched 1160.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 9 of 12)
Job launched 1191.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 9 of 12)
Job launched 1223.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 9 of 12)
Job launched 1254.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 9 of 12)
Job launched 1285.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 10 of 12)
Job launched 1317.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 10 of 12)
Job launched 1348.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 11 of 12)
Job launched 1380.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 11 of 12)
Job launched 1411.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 11 of 12)
Job launched 1442.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 11 of 12)
Job launched 1474.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 12 of 12)
Job launched 1505.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 12 of 12)
Job launched 1537.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151105.184011.103955: Step 12 of 12)
Job completed.
Running time was 1181.0s (not counting time spent waiting for the EC2 instances)
ec2_key_pair_file not specified, going to S3
Fetching counters from S3...
Waiting 5.0s for S3 eventual consistency
Counters from step 1:
  File Input Format Counters :
    Bytes Read: 1071
  File Output Format Counters :
    Bytes Written: 723
  FileSystemCounters:
    FILE_BYTES_READ: 262
    FILE_BYTES_WRITTEN: 134357
    HDFS_BYTES_READ: 448
    HDFS_BYTES_WRITTEN: 723
    S3_BYTES_READ: 1071
  Job Counters :
    Launched map tasks: 4
    Launched reduce tasks: 1
    Rack-local map tasks: 4
    SLOTS_MILLIS_MAPS: 75404
    SLOTS_MILLIS_REDUCES: 28978
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 15570
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 426
    Map input records: 11
    Map output bytes: 714
    Map output materialized bytes: 406
    Map output records: 29
    Physical memory (bytes) snapshot: 863514624
    Reduce input groups: 12
    Reduce input records: 29
    Reduce output records: 22
    Reduce shuffle bytes: 406
    SPLIT_RAW_BYTES: 448
    Spilled Records: 58
    Total committed heap usage (bytes): 606822400
    Virtual memory (bytes) snapshot: 3141820416
Counters from step 2:
  File Input Format Counters :
    Bytes Read: 1815
  File Output Format Counters :
    Bytes Written: 434
  FileSystemCounters:
    FILE_BYTES_READ: 316
    FILE_BYTES_WRITTEN: 134217
    HDFS_BYTES_READ: 2463
    HDFS_BYTES_WRITTEN: 434
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 53696
    SLOTS_MILLIS_REDUCES: 30624
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 7480
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 723
    Map input records: 22
    Map output bytes: 723
    Map output materialized bytes: 407
    Map output records: 22
    Physical memory (bytes) snapshot: 813916160
    Reduce input groups: 11
    Reduce input records: 22
    Reduce output records: 11
    Reduce shuffle bytes: 407
    SPLIT_RAW_BYTES: 648
    Spilled Records: 44
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3186962432
Counters from step 3:
  File Input Format Counters :
    Bytes Read: 1091
  File Output Format Counters :
    Bytes Written: 721
  FileSystemCounters:
    FILE_BYTES_READ: 354
    FILE_BYTES_WRITTEN: 134664
    HDFS_BYTES_READ: 1739
    HDFS_BYTES_WRITTEN: 721
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 58610
    SLOTS_MILLIS_REDUCES: 31101
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 7780
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 434
    Map input records: 11
    Map output bytes: 709
    Map output materialized bytes: 521
    Map output records: 29
    Physical memory (bytes) snapshot: 848228352
    Reduce input groups: 12
    Reduce input records: 29
    Reduce output records: 22
    Reduce shuffle bytes: 521
    SPLIT_RAW_BYTES: 648
    Spilled Records: 58
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3195428864
Counters from step 4:
  File Input Format Counters :
    Bytes Read: 1807
  File Output Format Counters :
    Bytes Written: 432
  FileSystemCounters:
    FILE_BYTES_READ: 363
    FILE_BYTES_WRITTEN: 134320
    HDFS_BYTES_READ: 2455
    HDFS_BYTES_WRITTEN: 432
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 53470
    SLOTS_MILLIS_REDUCES: 30040
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 6310
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 721
    Map input records: 22
    Map output bytes: 721
    Map output materialized bytes: 468
    Map output records: 22
    Physical memory (bytes) snapshot: 821350400
    Reduce input groups: 11
    Reduce input records: 22
    Reduce output records: 11
    Reduce shuffle bytes: 468
    SPLIT_RAW_BYTES: 648
    Spilled Records: 44
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3194445824
Counters from step 5:
  File Input Format Counters :
    Bytes Read: 1083
  File Output Format Counters :
    Bytes Written: 721
  FileSystemCounters:
    FILE_BYTES_READ: 372
    FILE_BYTES_WRITTEN: 134696
    HDFS_BYTES_READ: 1731
    HDFS_BYTES_WRITTEN: 721
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 60919
    SLOTS_MILLIS_REDUCES: 30315
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 7470
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 432
    Map input records: 11
    Map output bytes: 708
    Map output materialized bytes: 525
    Map output records: 29
    Physical memory (bytes) snapshot: 808402944
    Reduce input groups: 12
    Reduce input records: 29
    Reduce output records: 22
    Reduce shuffle bytes: 525
    SPLIT_RAW_BYTES: 648
    Spilled Records: 58
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3052974080
Counters from step 6:
  File Input Format Counters :
    Bytes Read: 1807
  File Output Format Counters :
    Bytes Written: 432
  FileSystemCounters:
    FILE_BYTES_READ: 359
    FILE_BYTES_WRITTEN: 134307
    HDFS_BYTES_READ: 2455
    HDFS_BYTES_WRITTEN: 432
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 53663
    SLOTS_MILLIS_REDUCES: 33822
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 5930
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 721
    Map input records: 22
    Map output bytes: 721
    Map output materialized bytes: 464
    Map output records: 22
    Physical memory (bytes) snapshot: 822284288
    Reduce input groups: 11
    Reduce input records: 22
    Reduce output records: 11
    Reduce shuffle bytes: 464
    SPLIT_RAW_BYTES: 648
    Spilled Records: 44
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3193946112
Counters from step 7:
  File Input Format Counters :
    Bytes Read: 1083
  File Output Format Counters :
    Bytes Written: 723
  FileSystemCounters:
    FILE_BYTES_READ: 387
    FILE_BYTES_WRITTEN: 134721
    HDFS_BYTES_READ: 1731
    HDFS_BYTES_WRITTEN: 723
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 59800
    SLOTS_MILLIS_REDUCES: 27413
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 6700
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 432
    Map input records: 11
    Map output bytes: 716
    Map output materialized bytes: 545
    Map output records: 29
    Physical memory (bytes) snapshot: 832032768
    Reduce input groups: 12
    Reduce input records: 29
    Reduce output records: 22
    Reduce shuffle bytes: 545
    SPLIT_RAW_BYTES: 648
    Spilled Records: 58
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3218030592
Counters from step 8:
  File Input Format Counters :
    Bytes Read: 1815
  File Output Format Counters :
    Bytes Written: 435
  FileSystemCounters:
    FILE_BYTES_READ: 377
    FILE_BYTES_WRITTEN: 134334
    HDFS_BYTES_READ: 2463
    HDFS_BYTES_WRITTEN: 435
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 51581
    SLOTS_MILLIS_REDUCES: 30076
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 6270
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 723
    Map input records: 22
    Map output bytes: 723
    Map output materialized bytes: 468
    Map output records: 22
    Physical memory (bytes) snapshot: 814055424
    Reduce input groups: 11
    Reduce input records: 22
    Reduce output records: 11
    Reduce shuffle bytes: 468
    SPLIT_RAW_BYTES: 648
    Spilled Records: 44
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3212292096
Counters from step 9:
  File Input Format Counters :
    Bytes Read: 1095
  File Output Format Counters :
    Bytes Written: 723
  FileSystemCounters:
    FILE_BYTES_READ: 377
    FILE_BYTES_WRITTEN: 134705
    HDFS_BYTES_READ: 1743
    HDFS_BYTES_WRITTEN: 723
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 61722
    SLOTS_MILLIS_REDUCES: 35642
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 6860
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 435
    Map input records: 11
    Map output bytes: 718
    Map output materialized bytes: 539
    Map output records: 29
    Physical memory (bytes) snapshot: 808210432
    Reduce input groups: 12
    Reduce input records: 29
    Reduce output records: 22
    Reduce shuffle bytes: 539
    SPLIT_RAW_BYTES: 648
    Spilled Records: 58
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3142438912
Counters from step 10:
  File Input Format Counters :
    Bytes Read: 1815
  File Output Format Counters :
    Bytes Written: 435
  FileSystemCounters:
    FILE_BYTES_READ: 360
    FILE_BYTES_WRITTEN: 134330
    HDFS_BYTES_READ: 2463
    HDFS_BYTES_WRITTEN: 435
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 50209
    SLOTS_MILLIS_REDUCES: 33726
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 6510
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 723
    Map input records: 22
    Map output bytes: 723
    Map output materialized bytes: 471
    Map output records: 22
    Physical memory (bytes) snapshot: 816439296
    Reduce input groups: 11
    Reduce input records: 22
    Reduce output records: 11
    Reduce shuffle bytes: 471
    SPLIT_RAW_BYTES: 648
    Spilled Records: 44
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3261022208
Counters from step 11:
  File Input Format Counters :
    Bytes Read: 1095
  File Output Format Counters :
    Bytes Written: 722
  FileSystemCounters:
    FILE_BYTES_READ: 366
    FILE_BYTES_WRITTEN: 134704
    HDFS_BYTES_READ: 1747
    HDFS_BYTES_WRITTEN: 722
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 56179
    SLOTS_MILLIS_REDUCES: 29148
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 7230
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 435
    Map input records: 11
    Map output bytes: 709
    Map output materialized bytes: 529
    Map output records: 29
    Physical memory (bytes) snapshot: 804708352
    Reduce input groups: 12
    Reduce input records: 29
    Reduce output records: 22
    Reduce shuffle bytes: 529
    SPLIT_RAW_BYTES: 652
    Spilled Records: 58
    Total committed heap usage (bytes): 605634560
    Virtual memory (bytes) snapshot: 3168595968
Counters from step 12:
  File Input Format Counters :
    Bytes Read: 1811
  File Output Format Counters :
    Bytes Written: 433
  FileSystemCounters:
    FILE_BYTES_READ: 375
    FILE_BYTES_WRITTEN: 134144
    HDFS_BYTES_READ: 2463
    S3_BYTES_WRITTEN: 433
  Job Counters :
    Data-local map tasks: 4
    Launched map tasks: 4
    Launched reduce tasks: 1
    SLOTS_MILLIS_MAPS: 64230
    SLOTS_MILLIS_REDUCES: 37415
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 3840
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 722
    Map input records: 22
    Map output bytes: 722
    Map output materialized bytes: 465
    Map output records: 22
    Physical memory (bytes) snapshot: 850178048
    Reduce input groups: 11
    Reduce input records: 22
    Reduce output records: 11
    Reduce shuffle bytes: 465
    SPLIT_RAW_BYTES: 652
    Spilled Records: 44
    Total committed heap usage (bytes): 606781440
    Virtual memory (bytes) snapshot: 3205238784
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151105.184011.103955
Removing all files in s3://ucb-mids-mls-katieadams/tmp/MRJob_PageRank.davidadams.20151105.184011.103955/
Removing all files in s3://ucb-mids-mls-katieadams/tmp/logs/j-3GVS3SQZV97HL/
Terminating job flow: j-3GVS3SQZV97HL

In [156]:
!python MRJob_PreprocessGraph.py s3://ucb-mids-mls-networks/wikipedia/all-pages-indexed-out.txt --jobconf G=15192277 -r emr --num-ec2-instances 15 --ec2-task-instance-type m1.medium --output-dir s3://ucb-mids-mls-katieadams/output-wiki-preprocess --no-output


using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.193712.552180
writing master bootstrap script to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.193712.552180/b.py

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

Copying non-input files into s3://ucb-mids-mls-katieadams/tmp/MRJob_PreprocessGraph.davidadams.20151105.193712.552180/files/
Waiting 5.0s for S3 eventual consistency
Creating Elastic MapReduce job flow
Job flow created with ID: j-1FW01DVX72RC0
Created new job flow j-1FW01DVX72RC0
Job launched 31.0s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 62.0s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 93.1s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 124.2s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 155.2s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 186.3s ago, status STARTING: Configuring cluster software
Job launched 217.3s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 248.4s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 279.4s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 310.6s ago, status RUNNING: Running step
Job launched 341.7s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 373.2s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 404.4s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 435.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 466.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 497.8s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 528.9s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 560.0s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 591.0s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 622.1s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 653.2s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 684.2s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 715.4s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 746.4s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 777.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 808.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 839.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 871.1s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 902.1s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 933.3s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 964.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 995.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1026.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1057.8s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1088.9s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1120.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1151.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1182.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1213.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1244.8s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1275.9s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1307.0s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1338.0s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1369.1s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1400.2s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1431.3s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1462.4s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1493.4s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1524.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1555.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1586.7s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1617.7s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1648.7s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1679.9s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1710.9s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1742.0s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1773.0s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1804.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1835.6s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1866.8s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1897.9s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1929.4s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1960.5s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 1992.0s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 2023.2s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job launched 2054.3s ago, status RUNNING: Running step (MRJob_PreprocessGraph.davidadams.20151105.193712.552180: Step 1 of 1)
Job completed.
Running time was 1719.0s (not counting time spent waiting for the EC2 instances)
ec2_key_pair_file not specified, going to S3
Fetching counters from S3...
Waiting 5.0s for S3 eventual consistency
Counters from step 1:
  File Input Format Counters :
    Bytes Read: 2090954960
  File Output Format Counters :
    Bytes Written: 2591707083
  FileSystemCounters:
    FILE_BYTES_READ: 4353073459
    FILE_BYTES_WRITTEN: 5656577823
    HDFS_BYTES_READ: 6440
    S3_BYTES_READ: 2090954960
    S3_BYTES_WRITTEN: 2591707083
  Job Counters :
    Launched map tasks: 56
    Launched reduce tasks: 28
    Rack-local map tasks: 56
    SLOTS_MILLIS_MAPS: 29454941
    SLOTS_MILLIS_REDUCES: 14725336
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 19681830
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2090459616
    Map input records: 5781290
    Map output bytes: 6983848875
    Map output materialized bytes: 1913399068
    Map output records: 147895347
    Physical memory (bytes) snapshot: 22735384576
    Reduce input groups: 15192277
    Reduce input records: 147895347
    Reduce output records: 15192277
    Reduce shuffle bytes: 1913399068
    SPLIT_RAW_BYTES: 6440
    Spilled Records: 443686041
    Total committed heap usage (bytes): 18729271296
    Virtual memory (bytes) snapshot: 52397174784
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PreprocessGraph.davidadams.20151105.193712.552180
Removing all files in s3://ucb-mids-mls-katieadams/tmp/MRJob_PreprocessGraph.davidadams.20151105.193712.552180/
Removing all files in s3://ucb-mids-mls-katieadams/tmp/logs/j-1FW01DVX72RC0/
Terminating job flow: j-1FW01DVX72RC0

In [158]:
!python MRJob_PageRank.py s3://ucb-mids-mls-katieadams/output-wiki-preprocess/* --jobconf alpha=0.15 --jobconf G=15192277 --iterations 1 -r emr --num-ec2-instances 15 --ec2-instance-type m1.medium --output-dir s3://ucb-mids-mls-katieadams/output-wiki-degub --no-output


using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151106.003911.848321
writing master bootstrap script to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151106.003911.848321/b.py

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

Copying non-input files into s3://ucb-mids-mls-katieadams/tmp/MRJob_PageRank.davidadams.20151106.003911.848321/files/
Waiting 5.0s for S3 eventual consistency
Creating Elastic MapReduce job flow
Job flow created with ID: j-2XFJ8UA6NC93N
Created new job flow j-2XFJ8UA6NC93N
Job launched 31.0s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 62.2s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 93.3s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 124.8s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 156.0s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 187.6s ago, status STARTING: Configuring cluster software
Job launched 218.7s ago, status STARTING: Configuring cluster software
Job launched 249.8s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 280.9s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 312.2s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 343.3s ago, status RUNNING: Running step
Job launched 374.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 405.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 436.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 467.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 499.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 530.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 561.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 592.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 623.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 654.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 686.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 717.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 749.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 780.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 811.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 842.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 873.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 904.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 935.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 966.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 998.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1029.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1060.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1091.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1122.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1154.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1185.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1216.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1247.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1278.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1310.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1341.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1372.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1403.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1434.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1466.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1497.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1528.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1559.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1590.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1622.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1653.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1685.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1716.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1747.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1778.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1810.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1841.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1872.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1903.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1935.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1966.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 1997.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2028.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2060.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2091.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2123.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2154.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2185.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2216.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2248.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2279.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2310.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2341.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2373.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2404.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2436.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2467.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2498.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2529.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2560.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2591.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2622.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2654.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2685.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2716.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2747.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2778.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2809.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2841.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2872.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2903.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2935.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2966.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 2997.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3028.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3059.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3091.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3122.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3153.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3184.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3215.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3246.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3278.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3309.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3340.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3371.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3402.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3433.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3464.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3495.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3526.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3557.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3589.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 1 of 4)
Job launched 3620.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 2 of 4)
Job launched 3651.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 2 of 4)
Job launched 3682.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 2 of 4)
Job launched 3713.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 2 of 4)
Job launched 3745.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 2 of 4)
Job launched 3776.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 2 of 4)
Job launched 3807.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 2 of 4)
Job launched 3838.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 2 of 4)
Job launched 3869.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 2 of 4)
Job launched 3900.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 2 of 4)
Job launched 3931.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 2 of 4)
Job launched 3962.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 2 of 4)
Job launched 3994.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 2 of 4)
Job launched 4025.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 2 of 4)
Job launched 4056.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 2 of 4)
Job launched 4087.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4118.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4149.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4180.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4211.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4243.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4274.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4305.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4336.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4367.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4398.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4429.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4460.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4491.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4523.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4554.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4585.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4616.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4647.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4678.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4709.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4741.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4772.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4804.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4835.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4866.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4897.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4928.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4959.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 4990.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5021.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5053.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5084.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5115.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5146.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5177.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5208.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5239.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5270.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5301.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5332.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5363.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5394.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5426.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5457.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5488.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5519.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5550.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5581.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5613.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5644.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5675.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5706.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5738.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5769.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5801.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5831.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5863.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5894.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5925.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5956.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 5987.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6018.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6049.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6080.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6111.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6142.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6173.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6204.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6235.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6266.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6298.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6329.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6360.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6391.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6422.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6453.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6484.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6515.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6547.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6578.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6609.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6641.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6672.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6703.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6734.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6765.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6796.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6827.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6858.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6889.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6921.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6952.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 6983.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7014.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7045.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7076.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7107.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7139.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7170.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7201.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7232.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7263.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7294.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7325.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7356.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7388.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7419.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7450.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7481.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7512.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7543.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7574.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7606.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 3 of 4)
Job launched 7637.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job launched 7668.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job launched 7699.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job launched 7730.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job launched 7761.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job launched 7792.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job launched 7824.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job launched 7855.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job launched 7886.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job launched 7917.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job launched 7948.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job launched 7979.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job launched 8010.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job launched 8042.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job launched 8073.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job launched 8104.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.003911.848321: Step 4 of 4)
Job completed.
Running time was 7718.0s (not counting time spent waiting for the EC2 instances)
ec2_key_pair_file not specified, going to S3
Fetching counters from S3...
Waiting 5.0s for S3 eventual consistency
Counters from step 1:
  File Input Format Counters :
    Bytes Read: 2592037330
  File Output Format Counters :
    Bytes Written: 3095361748
  FileSystemCounters:
    FILE_BYTES_READ: 6332206931
    FILE_BYTES_WRITTEN: 9061970381
    HDFS_BYTES_READ: 8280
    HDFS_BYTES_WRITTEN: 3095361748
    S3_BYTES_READ: 2592037330
  Job Counters :
    Launched map tasks: 77
    Launched reduce tasks: 29
    Rack-local map tasks: 77
    SLOTS_MILLIS_MAPS: 34643805
    SLOTS_MILLIS_REDUCES: 18648552
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 24455480
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2591707083
    Map input records: 15192277
    Map output bytes: 7328997236
    Map output materialized bytes: 3044346363
    Map output records: 166717321
    Physical memory (bytes) snapshot: 27373441024
    Reduce input groups: 15192278
    Reduce input records: 166717321
    Reduce output records: 30384554
    Reduce shuffle bytes: 3044346363
    SPLIT_RAW_BYTES: 8280
    Spilled Records: 500151963
    Total committed heap usage (bytes): 22779584512
    Virtual memory (bytes) snapshot: 62592917504
Counters from step 2:
  File Input Format Counters :
    Bytes Read: 3096707152
  File Output Format Counters :
    Bytes Written: 2590076875
  FileSystemCounters:
    FILE_BYTES_READ: 3282759355
    FILE_BYTES_WRITTEN: 4688510384
    HDFS_BYTES_READ: 3096716548
    HDFS_BYTES_WRITTEN: 2590076875
  Job Counters :
    Data-local map tasks: 58
    Launched map tasks: 66
    Launched reduce tasks: 28
    Rack-local map tasks: 8
    SLOTS_MILLIS_MAPS: 2656810
    SLOTS_MILLIS_REDUCES: 4569017
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 3749950
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 3095361748
    Map input records: 30384554
    Map output bytes: 3101593924
    Map output materialized bytes: 1563430435
    Map output records: 30384554
    Physical memory (bytes) snapshot: 15655055360
    Reduce input groups: 15192277
    Reduce input records: 30384554
    Reduce output records: 15192277
    Reduce shuffle bytes: 1563430435
    SPLIT_RAW_BYTES: 9396
    Spilled Records: 91117618
    Total committed heap usage (bytes): 11129204736
    Virtual memory (bytes) snapshot: 52572155904
Counters from step 3:
  File Input Format Counters :
    Bytes Read: 2590895011
  File Output Format Counters :
    Bytes Written: 3095554927
  FileSystemCounters:
    FILE_BYTES_READ: 9176271701
    FILE_BYTES_WRITTEN: 13007444081
    HDFS_BYTES_READ: 2590906675
    HDFS_BYTES_WRITTEN: 3095554927
  Job Counters :
    Data-local map tasks: 66
    Launched map tasks: 77
    Launched reduce tasks: 32
    Rack-local map tasks: 11
    SLOTS_MILLIS_MAPS: 27990714
    SLOTS_MILLIS_REDUCES: 17983502
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 21165060
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2590076875
    Map input records: 15192277
    Map output bytes: 7326349997
    Map output materialized bytes: 4401158326
    Map output records: 166717321
    Physical memory (bytes) snapshot: 27366088704
    Reduce input groups: 15192278
    Reduce input records: 166717321
    Reduce output records: 30384554
    Reduce shuffle bytes: 4401158326
    SPLIT_RAW_BYTES: 11664
    Spilled Records: 500151963
    Total committed heap usage (bytes): 23618211840
    Virtual memory (bytes) snapshot: 62096572416
Counters from step 4:
  File Input Format Counters :
    Bytes Read: 3096707131
  File Output Format Counters :
    Bytes Written: 2590125765
  FileSystemCounters:
    FILE_BYTES_READ: 3312921597
    FILE_BYTES_WRITTEN: 4733767141
    HDFS_BYTES_READ: 3096716527
    S3_BYTES_WRITTEN: 2590125765
  Job Counters :
    Data-local map tasks: 57
    Launched map tasks: 67
    Launched reduce tasks: 27
    Rack-local map tasks: 10
    SLOTS_MILLIS_MAPS: 2956052
    SLOTS_MILLIS_REDUCES: 4406377
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 3802720
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 3095554927
    Map input records: 30384554
    Map output bytes: 3101787319
    Map output materialized bytes: 1578527993
    Map output records: 30384554
    Physical memory (bytes) snapshot: 16381284352
    Reduce input groups: 15192277
    Reduce input records: 30384554
    Reduce output records: 15192277
    Reduce shuffle bytes: 1578527993
    SPLIT_RAW_BYTES: 9396
    Spilled Records: 91117844
    Total committed heap usage (bytes): 11255508992
    Virtual memory (bytes) snapshot: 52646264832
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151106.003911.848321
Removing all files in s3://ucb-mids-mls-katieadams/tmp/MRJob_PageRank.davidadams.20151106.003911.848321/
Removing all files in s3://ucb-mids-mls-katieadams/tmp/logs/j-2XFJ8UA6NC93N/
Terminating job flow: j-2XFJ8UA6NC93N

In [ ]:
!python MRJob_PageRank.py s3://ucb-mids-mls-katieadams/output-wiki-preprocess/* --jobconf alpha=0.85 --jobconf G=15192277 --iterations 9 -r emr --num-ec2-instances 15 --ec2-instance-type m1.medium --output-dir s3://ucb-mids-mls-katieadams/output-wiki-10x --no-output


using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151106.032358.020374
writing master bootstrap script to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_PageRank.davidadams.20151106.032358.020374/b.py

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

Copying non-input files into s3://ucb-mids-mls-katieadams/tmp/MRJob_PageRank.davidadams.20151106.032358.020374/files/
Waiting 5.0s for S3 eventual consistency
Creating Elastic MapReduce job flow
Job flow created with ID: j-28VJ80LKEW1U9
Created new job flow j-28VJ80LKEW1U9
Job launched 31.4s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 62.9s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 94.1s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 125.9s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 157.2s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 188.5s ago, status STARTING: Configuring cluster software
Job launched 219.7s ago, status STARTING: Configuring cluster software
Job launched 251.3s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 282.6s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 313.9s ago, status RUNNING: Running step
Job launched 345.2s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 376.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 408.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 440.1s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 471.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 503.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 534.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 566.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 597.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 628.7s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 659.9s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 691.5s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 722.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 754.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 786.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 817.8s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 849.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 880.4s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 911.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 943.0s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 974.3s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)
Job launched 1005.6s ago, status RUNNING: Running step (MRJob_PageRank.davidadams.20151106.032358.020374: Step 1 of 20)

In [ ]:
#%load_ext autoreload
%autoreload 2

from MRJob_NumberOfNodes import MRJobNumNodes
from MRJob_PreprocessGraph import MRJobPreprocessGraph
from MRJob_PageRank import MRJobPageRank

def numNodes(graphfilename):
    #mr_job = MRJobNumNodes(args=[graphfilename, "--strict-protocols", "-r", "emr", "--num-ec2-instances", "2", "--ec2-task-instance-type", "m1.small", "--pool-emr-job-flows", "--max-hours-idle=1"])
    mr_job_numnodes = MRJobNumNodes(args=[graphfilename])

    with mr_job_numnodes.make_runner() as runner:
        runner.run()
        for line in runner.stream_output():
            null,count =  mr_job_numnodes.parse_output_line(line)
            print "There are "+str(count)+" nodes in the graph."
                    
    return None

def preprocessGraph(graphfilename):
    #mr_job = MRJobNumNodes(args=[graphfilename, "--strict-protocols", "-r", "emr", "--num-ec2-instances", "2", "--ec2-task-instance-type", "m1.small", "--pool-emr-job-flows", "--max-hours-idle=1"])
    mr_job_preprocess = MRJobPreprocessGraph(args=[graphfilename])
    outputfilename = graphfilename.split('.')[0]+'_input.txt'
    with mr_job_preprocess.make_runner() as runner:
        with open(outputfilename, 'w') as f:
            runner.run()
            for line in runner.stream_output():
                node,value =  mr_job_preprocess.parse_output_line(line)
                f.write(node+'\t'+str(value)+'\n')
                    
    return None


def pagerank(graphinputfilename,alpha,G):
    #mr_job = MRJobNumNodes(args=[graphfilename, "--strict-protocols", "-r", "emr", "--num-ec2-instances", "2", "--ec2-task-instance-type", "m1.small", "--pool-emr-job-flows", "--max-hours-idle=1"])
    mr_job_pr1 = MRJobPageRank(args=[graphinputfilename,'--jobconf','alpha=0.15','--jobconf','G=11'])
    danglingmass= 0
    with mr_job_pr1.make_runner() as runner:
        runner.run()
        with open(graphinputfilename, 'w+') as f:
            for line in runner.stream_output():
                node,value =  mr_job_pr1.parse_output_line(line)
                f.write(node+'\t'+str(value)+'\n')
                print line

    return None




def hw9_1():

    graphfilename = "PageRank-test.txt"
    #numNodes(graphfilename)
    
    preprocessGraph(graphfilename)

    graphinputfilename = graphfilename.split('.')[0]+'_input.txt'
    
    alpha = 0.85
    G = 11
    
    #pagerank_step1(graphinputfilename, alpha, G)
    
    #pagerank_step1(graphinputfilename, alpha, G)
    
    for i in range(20):
        print "Iteration",i
        pagerank(graphinputfilename, alpha, G)
    
    
    
    
    return None
        
  

hw9_1()

In [1]:
!python MRJob_NumberOfNodes.py all-pages-indexed-out.txt -r emr --num-ec2-instances 10 --ec2-task-instance-type m1.medium


using configs in /Users/davidadams/.mrjob.conf
creating tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_NumberOfNodes.davidadams.20151103.041200.745722
writing master bootstrap script to /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_NumberOfNodes.davidadams.20151103.041200.745722/b.py

PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols

Copying non-input files into s3://ucb-mids-mls-katieadams/tmp/MRJob_NumberOfNodes.davidadams.20151103.041200.745722/files/
Waiting 5.0s for S3 eventual consistency
Creating Elastic MapReduce job flow
Job flow created with ID: j-DZOEYU00LV2O
Created new job flow j-DZOEYU00LV2O
Job launched 31.0s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 62.1s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 93.2s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 124.2s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 155.3s ago, status STARTING: Provisioning Amazon EC2 capacity
Job launched 186.3s ago, status STARTING: Configuring cluster software
Job launched 217.3s ago, status STARTING: Configuring cluster software
Job launched 248.4s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 279.4s ago, status BOOTSTRAPPING: Running bootstrap actions
Job launched 310.5s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 341.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 373.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 404.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 435.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 467.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 498.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 529.4s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 560.5s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 592.0s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 623.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 654.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 685.4s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 716.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 748.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 779.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 810.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 841.5s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 872.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 904.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 935.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 966.7s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 997.8s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1029.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1060.4s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1091.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1123.0s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1154.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1185.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1217.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1248.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1279.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1310.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1341.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1373.0s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1404.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1435.8s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1467.0s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1498.0s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1530.0s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1561.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1592.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1623.8s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1655.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1686.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1717.7s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1748.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1780.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1811.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1842.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1873.7s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1905.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1936.4s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1967.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 1999.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 1 of 2)
Job launched 2030.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2061.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2093.0s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2124.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2155.8s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2186.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2218.5s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2249.6s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2281.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2312.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2343.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2375.1s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2406.2s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2437.3s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job launched 2468.9s ago, status RUNNING: Running step (MRJob_NumberOfNodes.davidadams.20151103.041200.745722: Step 2 of 2)
Job completed.
Running time was 2178.0s (not counting time spent waiting for the EC2 instances)
ec2_key_pair_file not specified, going to S3
Fetching counters from S3...
Waiting 5.0s for S3 eventual consistency
Counters from step 1:
  File Input Format Counters :
    Bytes Read: 2090716941
  File Output Format Counters :
    Bytes Written: 231965329
  FileSystemCounters:
    FILE_BYTES_READ: 1339265877
    FILE_BYTES_WRITTEN: 1551090734
    HDFS_BYTES_READ: 6156
    HDFS_BYTES_WRITTEN: 231965329
    S3_BYTES_READ: 2090716941
  Job Counters :
    Launched map tasks: 36
    Launched reduce tasks: 18
    Rack-local map tasks: 36
    SLOTS_MILLIS_MAPS: 18794374
    SLOTS_MILLIS_REDUCES: 9141359
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 12628600
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 2090459616
    Map input records: 5781290
    Map output bytes: 2266859162
    Map output materialized bytes: 502615651
    Map output records: 147895347
    Physical memory (bytes) snapshot: 13983682560
    Reduce input groups: 15192277
    Reduce input records: 147895347
    Reduce output records: 15192277
    Reduce shuffle bytes: 502615651
    SPLIT_RAW_BYTES: 6156
    Spilled Records: 443686041
    Total committed heap usage (bytes): 11517841408
    Virtual memory (bytes) snapshot: 33275449344
Counters from step 2:
  File Input Format Counters :
    Bytes Read: 233302834
  File Output Format Counters :
    Bytes Written: 14
  FileSystemCounters:
    FILE_BYTES_READ: 194894274
    FILE_BYTES_WRITTEN: 248314033
    HDFS_BYTES_READ: 233310349
    S3_BYTES_WRITTEN: 14
  Job Counters :
    Data-local map tasks: 42
    Launched map tasks: 50
    Launched reduce tasks: 22
    Rack-local map tasks: 8
    SLOTS_MILLIS_MAPS: 1219921
    SLOTS_MILLIS_REDUCES: 1225560
    Total time spent by all maps waiting after reserving slots (ms): 0
    Total time spent by all reduces waiting after reserving slots (ms): 0
  Map-Reduce Framework:
    CPU time spent (ms): 550640
    Combine input records: 0
    Combine output records: 0
    Map input bytes: 231965329
    Map input records: 15192277
    Map output bytes: 231965329
    Map output materialized bytes: 87202083
    Map output records: 15192277
    Physical memory (bytes) snapshot: 11391102976
    Reduce input groups: 1
    Reduce input records: 15192277
    Reduce output records: 1
    Reduce shuffle bytes: 87202083
    SPLIT_RAW_BYTES: 7515
    Spilled Records: 42980204
    Total committed heap usage (bytes): 7509975040
    Virtual memory (bytes) snapshot: 38757552128
Streaming final output from s3://ucb-mids-mls-katieadams/tmp/MRJob_NumberOfNodes.davidadams.20151103.041200.745722/output/
null	15192277
removing tmp directory /var/folders/7t/6bhz6vw52k52g_3jqj57mz6r0000gn/T/MRJob_NumberOfNodes.davidadams.20151103.041200.745722
Removing all files in s3://ucb-mids-mls-katieadams/tmp/MRJob_NumberOfNodes.davidadams.20151103.041200.745722/
Removing all files in s3://ucb-mids-mls-katieadams/tmp/logs/j-DZOEYU00LV2O/
Terminating job flow: j-DZOEYU00LV2O

In [ ]:
#%load_ext autoreload
%autoreload 2

from MRJob_NumberOfNodes import MRJobNumNodes
from MRJob_PreprocessGraph import MRJobPreprocessGraph
from MRJob_PageRank import MRJobPageRank

def numNodes(graphfilename):
    #mr_job = MRJobNumNodes(args=[graphfilename, "--strict-protocols", "-r", "emr", "--num-ec2-instances", "2", "--ec2-task-instance-type", "m1.small", "--pool-emr-job-flows", "--max-hours-idle=1"])
    mr_job_numnodes = MRJobNumNodes(args=[graphfilename])

    with mr_job_numnodes.make_runner() as runner:
        runner.run()
        for line in runner.stream_output():
            null,count =  mr_job_numnodes.parse_output_line(line)
            print "There are "+str(count)+" nodes in the graph."
                    
    return None

def preprocessGraph(graphfilename):
    #mr_job = MRJobNumNodes(args=[graphfilename, "--strict-protocols", "-r", "emr", "--num-ec2-instances", "2", "--ec2-task-instance-type", "m1.small", "--pool-emr-job-flows", "--max-hours-idle=1"])
    mr_job_preprocess = MRJobPreprocessGraph(args=[graphfilename])
    outputfilename = 'PageRank-wiki_input.txt'
    with mr_job_preprocess.make_runner() as runner:
        with open(outputfilename, 'w') as f:
            runner.run()
            for line in runner.stream_output():
                node,value =  mr_job_preprocess.parse_output_line(line)
                f.write(node+'\t'+str(value)+'\n')
                    
    return None


def pagerank(graphinputfilename,alpha,G):
    #mr_job_pr1 = MRJobPageRank(args=[graphinputfilename,'--jobconf','alpha=0.15','--jobconf','G=11','-r',"emr", "--num-ec2-instances", "1", "--ec2-task-instance-type", "m1.small", "--pool-emr-job-flows", "--max-hours-idle=1"])
    
    mr_job_pr1 = MRJobPageRank(args=[graphinputfilename,'--jobconf','alpha=0.15','--jobconf','G=11','--jobconf', 'mapred.reduce.tasks=1','-r',"emr", "--num-ec2-instances", "10", "--ec2-task-instance-type", "m1.medium", "--pool-emr-job-flows", "--max-hours-idle=1"])
    
    
    #mr_job_pr1 = MRJobPageRank(args=[graphinputfilename,'--jobconf','alpha=0.15','--jobconf','G=11'])
    with mr_job_pr1.make_runner() as runner:
        runner.run()
        with open(graphinputfilename, 'w+') as f:
            for line in runner.stream_output():
                node,value =  mr_job_pr1.parse_output_line(line)
                f.write(node+'\t'+str(value)+'\n')
                print line

    return None




def hw9_3():

    #graphfilename = "s3://ucb-mids-mls-katieadams/PageRank-test.txt"
    graphfilename = "all-pages-indexed-out.txt"
    #graphfilename = "PageRank-test.txt"
    #numNodes(graphfilename)
    
    preprocessGraph(graphfilename)

    graphinputfilename = "PageRank-wiki_input.txt"
    
    alpha = 0.85
    G = 11
    
    #pagerank_step1(graphinputfilename, alpha, G)
    
    #pagerank(graphinputfilename, alpha, G)
    
    for i in range(10):
        print "Iteration",i
        pagerank(graphinputfilename, alpha, G)
    
    
    
    
    return None
        
  

hw9_3()


ERROR: Line magic function `%autoreload` not found.
WARNING:mrjob.runner:
WARNING:mrjob.runner:PLEASE NOTE: Starting in mrjob v0.5.0, protocols will be strict by default. It's recommended you run your job with --strict-protocols or set up mrjob.conf as described at https://pythonhosted.org/mrjob/whats-new.html#ready-for-strict-protocols
WARNING:mrjob.runner:

HW 9.4: Topic-specific PageRank implementation using MRJob

Modify your PageRank implementation to produce a topic specific PageRank implementation, as described in:

http://www-cs-students.stanford.edu/~taherh/papers/topic-sensitive-pagerank.pdf

Note in this article that there is a special caveat to ensure that the transition matrix is irreducible.
This caveat lies in footnote 3 on page 3:

A minor caveat: to ensure that M is irreducible when p
contains any 0 entries, nodes not reachable from nonzero
nodes in p should be removed. In practice this is not problematic.

and must be adhered to for convergence to be guaranteed.

Run topic specific PageRank on the following randomly generated network of 100 nodes:
s3://ucb-mids-mls-networks/randNet.txt

which are organized into ten topics, as described in the file:
s3://ucb-mids-mls-networks/randNet_topics.txt

Since there are 10 topics, your result should be 11 PageRank vectors (one for the vanilla PageRank implementation in 9.1, and one for each topic with the topic specific implementation). Print out the top ten ranking nodes and their topics for each of the 11 versions, and comment on your result.
Assume a teleportation factor of 0.15 in all your analyses.

HW 9.5: Applying topic-specific PageRank to Wikipedia

Here you will apply your topic-specific PageRank implementation to Wikipedia, defining topics (very arbitrarily) for each page by the length (number of characters) of the name of the article mod 10, so that there are 10 topics. Once again, print out the top ten ranking nodes and their topics for each of the 11 versions, and comment on your result.
Assume a teleportation factor of 0.15 in all your analyses.


In [ ]:


In [ ]: