

In [1]:

    
using JSON, ProgressMeter, JLD, LightGraphs

Exploring the different communities

The goal here is to use a couple of standard algorithms and compare the result with the regularized NMF.



In [2]:

    
# serialization options for later
di = JLD.load("/media/henripal/hd1/data/temp.jld", "di")
names = JLD.load("/media/henripal/hd1/data/names.jld","names")
name_to_index = JLD.load("/media/henripal/hd1/data/name_to_index.jld", "name_to_index");
graph = JLD.load("/media/henripal/hd1/data/graph.jld", "graph");



In [4]:

    
user_cluster_NMF = JLD.load("/media/henripal/hd1/data/U_60.jld");

Alternative method 1: Louvain

Ran this - takes a really long time. Couple days.



In [4]:

    
using CommunityDetection









    



WARNING: Method definition (::Type{Graphs.KeyVertex})(Int64, #K<:Any) in module Graphs at /home/henripal/.julia/v0.5/Graphs/src/common.jl:12 overwritten at /home/henripal/.julia/v0.5/Graphs/src/common.jl:16.
WARNING: Method definition (::Type{Graphs.Edge})(Int64, #V<:Any, #V<:Any) in module Graphs at /home/henripal/.julia/v0.5/Graphs/src/common.jl:54 overwritten at /home/henripal/.julia/v0.5/Graphs/src/common.jl:60.
WARNING: Method definition (::Type{Graphs.ExEdge})(Int64, #V<:Any, #V<:Any, Base.Dict{String, Any}) in module Graphs at /home/henripal/.julia/v0.5/Graphs/src/common.jl:72 overwritten at /home/henripal/.julia/v0.5/Graphs/src/common.jl:83.
WARNING: Method definition (::Type{Graphs.TargetIterator})(#G<:Graphs.AbstractGraph, #EList<:Any) in module Graphs at /home/henripal/.julia/v0.5/Graphs/src/common.jl:123 overwritten at /home/henripal/.julia/v0.5/Graphs/src/common.jl:127.
WARNING: Method definition (::Type{Graphs.SourceIterator})(#G<:Graphs.AbstractGraph, #EList<:Any) in module Graphs at /home/henripal/.julia/v0.5/Graphs/src/common.jl:141 overwritten at /home/henripal/.julia/v0.5/Graphs/src/common.jl:145.
WARNING: Method definition edge_property_requirement(Graphs.AbstractEdgePropertyInspector{#T<:Any}, Graphs.AbstractGraph{#V<:Any, E<:Any}) in module Graphs at /home/henripal/.julia/v0.5/Graphs/src/common.jl:164 overwritten at /home/henripal/.julia/v0.5/Graphs/src/common.jl:179.
WARNING: Method definition vertex_index(#V<:Union{Graphs.KeyVertex, Graphs.ExVertex}, Graphs.GenericGraph{#V<:Union{Graphs.KeyVertex, Graphs.ExVertex}, E<:Any, VList<:Any, EList<:Any, IncList<:Any}) in module Graphs at /home/henripal/.julia/v0.5/Graphs/src/graph.jl:65 overwritten at /home/henripal/.julia/v0.5/Graphs/src/graph.jl:67.
WARNING: Method definition (::Type{Graphs.GDistanceVisitor})(#G<:Graphs.AbstractGraph, #DMap<:Any) in module Graphs at /home/henripal/.julia/v0.5/Graphs/src/breadth_first_visit.jl:107 overwritten at /home/henripal/.julia/v0.5/Graphs/src/breadth_first_visit.jl:111.
WARNING: Method definition update_partition!(CommunityDetection.FlowPartition{#V<:Any}) in module CommunityDetection at /home/henripal/.julia/v0.5/CommunityDetection/src/FlowPartition.jl:40 overwritten at /home/henripal/.julia/v0.5/CommunityDetection/src/FlowPartition.jl:46.
WARNING: Method definition plogp(Any) in module CommunityDetection at /home/henripal/.julia/v0.5/CommunityDetection/src/FlowPartition.jl:143 overwritten at /home/henripal/.julia/v0.5/CommunityDetection/src/DiFlowPartition.jl:214.
WARNING: Method definition plogp(Array{Float64, 1}) in module CommunityDetection at /home/henripal/.julia/v0.5/CommunityDetection/src/FlowPartition.jl:144 overwritten at /home/henripal/.julia/v0.5/CommunityDetection/src/DiFlowPartition.jl:215.
WARNING: Method definition conductance(Any, Any) in module CommunityDetection at /home/henripal/.julia/v0.5/CommunityDetection/src/metrics.jl:32 overwritten at /home/henripal/.julia/v0.5/CommunityDetection/src/metrics.jl:262.
WARNING: Method definition _diff_move(CommunityDetection.MPartition{#V<:Any}, #V<:Any, Int64) in module CommunityDetection at /home/henripal/.julia/v0.5/CommunityDetection/src/Optimiser.jl:264 overwritten at /home/henripal/.julia/v0.5/CommunityDetection/src/Optimiser.jl:276.
WARNING: Method definition _quality(CommunityDetection.MPartition{#V<:Any}) in module CommunityDetection at /home/henripal/.julia/v0.5/CommunityDetection/src/Optimiser.jl:265 overwritten at /home/henripal/.julia/v0.5/CommunityDetection/src/Optimiser.jl:277.
WARNING: Method definition _diff_move(CommunityDetection.MPartition{#V<:Any}, #V<:Any, Int64) in module CommunityDetection at /home/henripal/.julia/v0.5/CommunityDetection/src/Optimiser.jl:276 overwritten at /home/henripal/.julia/v0.5/CommunityDetection/src/Optimiser.jl:279.
WARNING: Method definition _quality(CommunityDetection.MPartition{#V<:Any}) in module CommunityDetection at /home/henripal/.julia/v0.5/CommunityDetection/src/Optimiser.jl:277 overwritten at /home/henripal/.julia/v0.5/CommunityDetection/src/Optimiser.jl:280.
WARNING: Method definition slpa(Graphs.AbstractGraph{#V<:Any, E<:Any}) in module CommunityDetection at /home/henripal/.julia/v0.5/CommunityDetection/src/slpa.jl:82 overwritten at /home/henripal/.julia/v0.5/CommunityDetection/src/slpa.jl:111.
WARNING: Method definition slpa(Graphs.AbstractGraph{#V<:Any, E<:Any}, Array{Int64, 1}) in module CommunityDetection at /home/henripal/.julia/v0.5/CommunityDetection/src/slpa.jl:96 overwritten at /home/henripal/.julia/v0.5/CommunityDetection/src/slpa.jl:117.



In [5]:

    
# argh Julia. convering lighgraph to Graph
graph_graph = Graphs.simple_graph(nv(graph), is_directed=LightGraphs.is_directed(graph))
for e in LightGraphs.edges(graph)
    Graphs.add_edge!(graph_graph,src(e), dst(e))
end



In [6]:

    
graph_graph









    Out[6]:





Undirected Graph (1205559 vertices, 3166659 edges)



In [7]:

    
mp = mpartition(graph_graph)









    Out[7]:





CommunityDetection.MPartition{Int64}(CommunityDetection.MGraph{Int64}(Undirected Graph (1205559 vertices, 3166659 edges),[1,1,1,1,1,1,1,1,1,1  …  1,1,1,1,1,1,1,1,1,1],[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0  …  1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0],[8.0,16.0,141.0,1.0,1.0,8509.0,2735.0,2100.0,10636.0,1.0  …  1.0,1.0,0.0,1.0,1.0,2.0,2.0,1.0,2.0,1.0],[8.0,16.0,141.0,1.0,1.0,8509.0,2735.0,2100.0,10636.0,1.0  …  1.0,1.0,0.0,1.0,1.0,2.0,2.0,1.0,2.0,1.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0,0.0,0.0  …  0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],3.166659e6,1205559,false,4.35767378467768e-6),[1,2,3,4,5,6,7,8,9,10  …  1205550,1205551,1205552,1205553,1205554,1205555,1205556,1205557,1205558,1205559],Dict(247825=>CommunityDetection.MGroup{Int64}(Set([247825]),1,0.0,1.0,1.0),572246=>CommunityDetection.MGroup{Int64}(Set([572246]),1,0.0,3.0,3.0),787911=>CommunityDetection.MGroup{Int64}(Set([787911]),1,0.0,0.0,0.0),1133174=>CommunityDetection.MGroup{Int64}(Set([1133174]),1,0.0,1.0,1.0),349542=>CommunityDetection.MGroup{Int64}(Set([349542]),1,0.0,1.0,1.0),689101=>CommunityDetection.MGroup{Int64}(Set([689101]),1,0.0,6.0,6.0),323003=>CommunityDetection.MGroup{Int64}(Set([323003]),1,0.0,1.0,1.0),875606=>CommunityDetection.MGroup{Int64}(Set([875606]),1,0.0,1.0,1.0),893890=>CommunityDetection.MGroup{Int64}(Set([893890]),1,0.0,6.0,6.0),355530=>CommunityDetection.MGroup{Int64}(Set([355530]),1,0.0,1.0,1.0)…),335.0,0)



In [8]:

    
optimize_partition!(mp)









    Out[8]:





0.5950719028331881



In [14]:

    
JLD.save("/media/henripal/hd1/data/louvain_membership.jld", "membership", mp.membership, "community", mp.community)