This notebook divide a single mailing list corpus into threads.

What it does: -identifies the more participated threads -identifies the long lasting threads -export each thread's emails into seperate .csv files, setting thresholds of participation and duration

Parameters to set options: -set a single URL related to a mailing list, setting the 'url' variable -it exports files in the file path specified in the variable ‘path’ -you can set a threshold of participation and of duration for the threads to export, by setting 'min_participation' and 'min_duration' variables



In [1]:

    
%matplotlib inline
from bigbang.archive import Archive
from bigbang.archive import load as load_archive
from bigbang.thread import Thread
from bigbang.thread import Node
from bigbang.utils import remove_quoted
import matplotlib.pyplot as plt
import datetime
import csv
from collections import defaultdict

First, collect data from a public email archive.



In [6]:

    
#insert one URL related to the mailing list of interest
url = "http://mm.icann.org/pipermail/wp4/"

try:
    arch_path = '../archives/'+url[:-1].replace('://','_/')+'.csv'
    arx = load_archive(arch_path)
except:
    arch_path = '../archives/'+url[:-1].replace('//','/')+'.csv'
    print(url)
    arx = load_archive(arch_path)

Let's check the number of threads in this mailing list corpus



In [27]:

    
print(len(arx.get_threads()))









    Out[27]:





38

We can plot the number of people participating in each thread.



In [60]:

    
n = [t.get_num_people() for t in arx.get_threads()]

plt.hist(n, bins = 20)
plt.xlabel('number of email-address in a thread')
plt.show()

The duration of a thread is the amount of elapsed time between its first and last message.

Let's plot the number of threads per each number of days of duration



In [61]:

    
y = [t.get_duration().days for t in arx.get_threads()]

plt.hist(y, bins = (10))
plt.xlabel('duration of a thread(days)')
plt.show()

Export the content of each thread into a .csv file (named: thread_1.csv, thread2.csv, ...).

You can set a minimum level of participation and duration, based on the previous analyses



In [62]:

    
#Insert the participation threshold (number of people)
#(for no threeshold: 'min_participation = 0') 
min_participation = 0

#Insert the duration threshold (number of days)
#(for no threeshold: 'min_duration = 0')
min_duration = 0

#Insert the directory path where to save the files
path = 'c:/users/davide/bigbang/'



i = 0
for thread in arx.get_threads():
    if thread.get_num_people() >= min_participation and thread.get_duration().days >= min_duration:
            i += 1
            f = open(path+'thread_'+str(i)+'.csv', "wb")
            f_w = csv.writer(f)
            f_w.writerow(thread.get_content())
            f.close()