notebook.community

Edit and run

Load and Print all the accuracies of the 10 runs 10-fold cross-validation for nbc versus aode on each dataset





In [1]:



    


ClassID = readdlm("Data/ClassifierID.dat", ',')
ClassNames = readdlm("Data/ClassifierNames.dat", ',')
DatasetID = readdlm("Data/DatasetID.dat", ',');
DatasetNames = readdlm("Data/DatasetNames.dat", ',');
Percent_correct = readdlm("Data/Percent_correct.dat", ',');

i=1 #nbc
j=2 #aode
println("Accuracies for ", ClassNames[i,1], "(first row) and ", ClassNames[j,1], "(second row) on 10 runs of 10-fold cross-validation")
println()
println()

indi=find(x->x==i,ClassID)
indj=find(x->x==j,ClassID)
for d=1:Int32(maximum(DatasetID))
    indd=find(x->x==d,DatasetID)
    indid=intersect(indi,indd)
    indjd=intersect(indj,indd)
    acci=Percent_correct[indid]
    acci=round(acci,2)
    accj=Percent_correct[indjd]
    accj=round(accj,2)
    #Latex Formatted
    println("$(DatasetNames[d]) & $(acci[1])  & $(acci[2])  & $(acci[3])  & $(acci[4]) & \\dots & $(acci[end-1]) & $(acci[end])\\\\")
    println("$(DatasetNames[d]) & $(accj[1])  & $(accj[2])  & $(accj[3])  & $(accj[4]) & \\dots & $(accj[end-1]) & $(accj[end])\\\\")
    #if d==2
    #df= convert(DataFrame, acci')
    #    writetable("nbc_audiology.csv", df )
    #df=convert(DataFrame, accj')
    #writetable("aode_audiology.csv", df )
    #break
    #end
end



    


















    






Accuracies for nbc(first row) and aode(second row) on 10 runs of 10-fold cross-validation


anneal & 94.44  & 98.89  & 94.44  & 98.89 & \dots & 94.38 & 97.75\\
anneal & 96.67  & 100.0  & 96.67  & 100.0 & \dots & 96.63 & 97.75\\
audiology & 73.91  & 69.56  & 73.91  & 60.87 & \dots & 72.73 & 59.09\\
audiology & 73.91  & 69.56  & 78.26  & 60.87 & \dots & 72.73 & 59.09\\
wisconsin-breast-cancer & 90.32  & 90.32  & 87.1  & 86.67 & \dots & 86.67 & 90.0\\
wisconsin-breast-cancer & 87.1  & 87.1  & 87.1  & 86.67 & \dots & 83.33 & 86.67\\
cmc & 51.35  & 50.68  & 54.73  & 59.18 & \dots & 50.34 & 48.3\\
cmc & 52.7  & 50.68  & 52.7  & 55.1 & \dots & 52.38 & 48.98\\
contact-lenses & 66.67  & 66.67  & 66.67  & 66.67 & \dots & 100.0 & 100.0\\
contact-lenses & 66.67  & 66.67  & 66.67  & 66.67 & \dots & 100.0 & 100.0\\
credit & 84.06  & 82.61  & 84.06  & 88.41 & \dots & 78.26 & 84.06\\
credit & 84.06  & 82.61  & 84.06  & 89.86 & \dots & 79.71 & 85.51\\
german-credit & 85.51  & 79.71  & 82.61  & 84.06 & \dots & 76.81 & 85.51\\
german-credit & 85.51  & 79.71  & 84.06  & 86.96 & \dots & 78.26 & 86.96\\
pima-diabetes & 85.29  & 79.41  & 85.29  & 82.35 & \dots & 81.82 & 75.76\\
pima-diabetes & 85.29  & 73.53  & 85.29  & 82.35 & \dots & 84.85 & 75.76\\
ecoli & 48.65  & 51.35  & 63.51  & 40.54 & \dots & 50.68 & 47.94\\
ecoli & 54.05  & 66.22  & 58.11  & 45.95 & \dots & 60.27 & 53.42\\
eucalyptus & 70.0  & 70.0  & 77.0  & 79.0 & \dots & 75.0 & 72.0\\
eucalyptus & 74.0  & 68.0  & 77.0  & 77.0 & \dots & 77.0 & 74.0\\
glass & 63.64  & 77.27  & 59.09  & 68.18 & \dots & 76.19 & 66.67\\
glass & 63.64  & 81.82  & 72.73  & 77.27 & \dots & 71.43 & 66.67\\
grub-damage & 56.25  & 50.0  & 43.75  & 43.75 & \dots & 40.0 & 60.0\\
grub-damage & 43.75  & 56.25  & 37.5  & 50.0 & \dots & 33.33 & 53.33\\
haberman & 67.74  & 77.42  & 77.42  & 70.97 & \dots & 73.33 & 73.33\\
haberman & 74.19  & 74.19  & 77.42  & 74.19 & \dots & 76.67 & 66.67\\
hayes-roth & 62.5  & 62.5  & 62.5  & 62.5 & \dots & 56.25 & 56.25\\
hayes-roth & 62.5  & 62.5  & 62.5  & 62.5 & \dots & 56.25 & 56.25\\
cleeland-14 & 100.0  & 93.75  & 75.0  & 75.0 & \dots & 73.33 & 93.33\\
cleeland-14 & 100.0  & 93.75  & 75.0  & 75.0 & \dots & 73.33 & 93.33\\
hungarian-14 & 73.33  & 76.67  & 90.0  & 86.67 & \dots & 75.86 & 89.66\\
hungarian-14 & 73.33  & 76.67  & 90.0  & 86.67 & \dots & 75.86 & 89.66\\
hepatitis & 97.88  & 98.15  & 97.88  & 99.2 & \dots & 97.88 & 98.14\\
hepatitis & 98.15  & 98.68  & 98.41  & 98.94 & \dots & 98.14 & 98.41\\
hypothyroid & 88.89  & 82.86  & 88.57  & 88.57 & \dots & 88.57 & 88.57\\
hypothyroid & 91.67  & 85.71  & 88.57  & 91.43 & \dots & 85.71 & 94.29\\
ionosphere & 93.33  & 100.0  & 100.0  & 100.0 & \dots & 93.33 & 93.33\\
ionosphere & 93.33  & 100.0  & 100.0  & 100.0 & \dots & 93.33 & 93.33\\
iris & 90.31  & 84.38  & 88.12  & 87.19 & \dots & 87.46 & 88.72\\
iris & 91.25  & 90.31  & 91.88  & 90.62 & \dots & 91.54 & 91.54\\
kr-s-kp & 66.67  & 66.67  & 83.33  & 66.67 & \dots & 100.0 & 100.0\\
kr-s-kp & 66.67  & 66.67  & 83.33  & 66.67 & \dots & 100.0 & 100.0\\
labor & 48.57  & 57.14  & 54.29  & 57.14 & \dots & 61.76 & 58.82\\
labor & 48.57  & 57.14  & 54.29  & 57.14 & \dots & 61.76 & 58.82\\
lier-disorders & 86.67  & 93.33  & 93.33  & 73.33 & \dots & 100.0 & 85.71\\
lier-disorders & 93.33  & 93.33  & 93.33  & 80.0 & \dots & 100.0 & 85.71\\
lymphography & 55.74  & 61.67  & 63.33  & 61.67 & \dots & 63.33 & 61.67\\
lymphography & 60.66  & 63.33  & 63.33  & 61.67 & \dots & 66.67 & 63.33\\
monks1 & 75.0  & 71.43  & 75.0  & 73.21 & \dots & 74.54 & 65.46\\
monks1 & 80.36  & 83.93  & 91.07  & 83.93 & \dots & 85.46 & 80.0\\
monks3 & 96.43  & 98.21  & 92.86  & 96.43 & \dots & 98.18 & 92.73\\
monks3 & 96.43  & 98.21  & 92.86  & 98.21 & \dots & 98.18 & 92.73\\
monks & 95.82  & 95.57  & 96.31  & 95.33 & \dots & 96.55 & 96.92\\
monks & 99.88  & 99.88  & 99.88  & 100.0 & \dots & 100.0 & 100.0\\
mushroom & 90.43  & 90.05  & 91.51  & 89.43 & \dots & 90.97 & 89.89\\
mushroom & 92.75  & 91.9  & 93.75  & 92.13 & \dots & 93.6 & 92.36\\
nursery & 91.82  & 91.99  & 91.46  & 92.88 & \dots & 91.64 & 91.28\\
nursery & 96.62  & 97.51  & 96.8  & 96.8 & \dots & 97.69 & 96.44\\
optdigits & 93.98  & 92.34  & 93.43  & 94.15 & \dots & 95.25 & 93.6\\
optdigits & 97.26  & 95.8  & 96.35  & 96.89 & \dots & 97.62 & 96.16\\
page-blocks & 75.0  & 75.0  & 75.0  & 100.0 & \dots & 100.0 & 100.0\\
page-blocks & 75.0  & 75.0  & 75.0  & 100.0 & \dots & 100.0 & 100.0\\
pasture-production & 88.46  & 89.09  & 89.08  & 85.53 & \dots & 85.8 & 87.53\\
pasture-production & 96.54  & 97.91  & 98.73  & 97.45 & \dots & 97.54 & 98.0\\
pendigits & 74.03  & 71.43  & 81.82  & 76.62 & \dots & 80.26 & 65.79\\
pendigits & 72.73  & 71.43  & 81.82  & 80.52 & \dots & 78.95 & 64.47\\
postoperatie & 66.67  & 66.67  & 66.67  & 66.67 & \dots & 66.67 & 55.56\\
postoperatie & 66.67  & 66.67  & 77.78  & 66.67 & \dots & 66.67 & 66.67\\
primary-tumor & 55.88  & 52.94  & 47.06  & 44.12 & \dots & 47.06 & 42.42\\
primary-tumor & 47.06  & 52.94  & 50.0  & 44.12 & \dots & 47.06 & 39.39\\
segment & 90.04  & 91.78  & 89.61  & 90.91 & \dots & 92.21 & 92.64\\
segment & 93.07  & 95.67  & 95.67  & 94.37 & \dots & 95.67 & 96.97\\
solar-flare-C & 81.82  & 87.88  & 84.85  & 84.38 & \dots & 90.62 & 84.38\\
solar-flare-C & 90.91  & 87.88  & 87.88  & 90.62 & \dots & 90.62 & 90.62\\
solar-flare-m & 90.91  & 78.79  & 90.91  & 78.12 & \dots & 78.12 & 87.5\\
solar-flare-m & 93.94  & 84.85  & 87.88  & 84.38 & \dots & 78.12 & 90.62\\
solar-flare-X & 100.0  & 93.94  & 90.91  & 93.75 & \dots & 96.88 & 90.62\\
solar-flare-X & 96.97  & 96.97  & 96.97  & 100.0 & \dots & 96.88 & 96.88\\
sonar & 85.71  & 76.19  & 80.95  & 85.71 & \dots & 60.0 & 70.0\\
sonar & 85.71  & 76.19  & 80.95  & 85.71 & \dots & 60.0 & 70.0\\
soybean & 94.2  & 92.75  & 95.65  & 91.18 & \dots & 88.24 & 91.18\\
soybean & 95.65  & 94.2  & 95.65  & 91.18 & \dots & 88.24 & 92.65\\
spambase & 90.67  & 89.78  & 89.56  & 90.22 & \dots & 90.0 & 89.35\\
spambase & 94.79  & 93.91  & 91.3  & 93.04 & \dots & 93.26 & 92.61\\
spect-reordered & 74.07  & 70.37  & 77.78  & 77.78 & \dots & 88.46 & 88.46\\
spect-reordered & 74.07  & 66.67  & 81.48  & 81.48 & \dots & 88.46 & 88.46\\
splice & 95.61  & 94.98  & 96.24  & 94.67 & \dots & 95.3 & 95.3\\
splice & 95.61  & 96.55  & 96.86  & 96.55 & \dots & 95.92 & 94.98\\
squash-stored & 66.67  & 50.0  & 60.0  & 60.0 & \dots & 60.0 & 40.0\\
squash-stored & 66.67  & 33.33  & 60.0  & 60.0 & \dots & 80.0 & 40.0\\
squash-unstored & 50.0  & 66.67  & 40.0  & 80.0 & \dots & 80.0 & 80.0\\
squash-unstored & 66.67  & 100.0  & 40.0  & 80.0 & \dots & 80.0 & 80.0\\
tae & 50.0  & 40.0  & 60.0  & 46.67 & \dots & 46.67 & 53.33\\
tae & 50.0  & 40.0  & 60.0  & 46.67 & \dots & 46.67 & 53.33\\
credit & 56.57  & 64.65  & 52.52  & 59.6 & \dots & 49.5 & 57.58\\
credit & 76.77  & 83.84  & 76.77  & 76.77 & \dots & 70.71 & 74.75\\
owel & 80.8  & 80.4  & 80.0  & 80.0 & \dots & 80.0 & 80.0\\
owel & 87.4  & 84.6  & 84.4  & 85.4 & \dots & 86.6 & 86.4\\
waveform & 85.71  & 42.86  & 57.14  & 83.33 & \dots & 83.33 & 16.67\\
waveform & 85.71  & 42.86  & 57.14  & 83.33 & \dots & 83.33 & 33.33\\
white-clover & 94.44  & 100.0  & 94.44  & 100.0 & \dots & 94.12 & 100.0\\
white-clover & 94.44  & 100.0  & 94.44  & 94.44 & \dots & 94.12 & 100.0\\
wine & 100.0  & 95.71  & 97.14  & 94.29 & \dots & 97.14 & 97.1\\
wine & 100.0  & 95.71  & 97.14  & 92.86 & \dots & 97.14 & 97.1\\
yeast & 57.72  & 55.03  & 59.06  & 58.39 & \dots & 55.4 & 55.4\\
yeast & 57.05  & 55.03  & 59.06  & 58.39 & \dots & 54.05 & 55.4\\
zoo & 81.82  & 100.0  & 100.0  & 90.0 & \dots & 90.0 & 100.0\\
zoo & 90.91  & 100.0  & 100.0  & 90.0 & \dots & 90.0 & 100.0\\

Hereafter we compute the $p$-values dataset-by-dataset by applying the frequentist correlated t-test





In [2]:



    


using Distributions
using DataFrames
include("Tests/ttest_correlated.jl")

#correlation for 10 runs 10 folds cross-validation
rho=1/10

#Classifiers
i=1 #nbc
j=2 #aode
println("two-sided p-values for ", ClassNames[i,1], " vs. ", ClassNames[j,1])
println()

indi=find(x->x==i,ClassID)
indj=find(x->x==j,ClassID)
pvalue=Float64[]
for d=1:Int32(maximum(DatasetID))
    indd=find(x->x==d,DatasetID)
    indid=intersect(indi,indd)
    indjd=intersect(indj,indd)
    acci=Percent_correct[indid]/100
    accj=Percent_correct[indjd]/100
    #return p-value and confidence interval
    p,ci=ttest_correlated(acci-accj,0,rho,0,0.05)
    push!(pvalue,p)   
    if mod(d,3)==0
        # Latex Formatted
        @printf "%s & %1.3f & %s & %1.3f & %s & %1.3f \\\\\n" DatasetNames[d-2] pvalue[d-2] DatasetNames[d-1] pvalue[d-1] DatasetNames[d] pvalue[d]   
    end
end



    


















    






two-sided p-values for nbc vs. aode

anneal & 0.001 & audiology & 0.622 & wisconsin-breast-cancer & 0.598 \\
cmc & 0.338 & contact-lenses & 0.643 & credit & 0.479 \\
german-credit & 0.171 & pima-diabetes & 0.781 & ecoli & 0.001 \\
eucalyptus & 0.258 & glass & 0.162 & grub-damage & 0.090 \\
haberman & 0.671 & hayes-roth & 1.000 & cleeland-14 & 0.525 \\
hungarian-14 & 0.878 & hepatitis & 0.048 & hypothyroid & 0.287 \\
ionosphere & 0.684 & iris & 0.000 & kr-s-kp & 0.646 \\
labor & 1.000 & lier-disorders & 0.270 & lymphography & 0.018 \\
monks1 & 0.000 & monks3 & 0.220 & monks & 0.000 \\
mushroom & 0.000 & nursery & 0.000 & optdigits & 0.000 \\
page-blocks & 0.687 & pasture-production & 0.000 & pendigits & 0.452 \\
postoperatie & 0.582 & primary-tumor & 0.492 & segment & 0.000 \\
solar-flare-C & 0.035 & solar-flare-m & 0.596 & solar-flare-X & 0.004 \\
sonar & 0.777 & soybean & 0.049 & spambase & 0.000 \\
spect-reordered & 0.198 & splice & 0.004 & squash-stored & 0.940 \\
squash-unstored & 0.304 & tae & 0.684 & credit & 0.000 \\
owel & 0.000 & waveform & 0.417 & white-clover & 0.463 \\
wine & 0.671 & yeast & 0.576 & zoo & 0.435 \\

Piechart of the percentage of comparisons that are significant and non-significant





In [3]:



    


using PyPlot

res=find(x->x<0.05,pvalue)
labels = ["significant", "non-significant"]


colors = ["salmon","lightsalmon"]
sizes = NaN*zeros(2)
explode = zeros(length(sizes))
explode[1] = 0.1
sizes= [length(res)/length(pvalue)*100, (1-length(res)/length(pvalue))*100 ]

fig = figure("pyplot_piechart",figsize=(3,3))
    ax = fig[:add_subplot](1,1,1)
p,text,autotext=ax[:pie](sizes,
        labels=labels,
        shadow=false,
        startangle=90,
         colors=colors,
        explode=explode,
autopct="%1.1f%%")


io=open("Plots/piechart.pdf","w") 
writemime(io, "application/pdf", fig)



    


















    

















    






WARNING: Base.writemime is deprecated.
  likely near In[3]:25
WARNING: Base.writemime is deprecated.
  likely near In[3]:25

















In [ ]:

Content source: BayesianTestsML/tutorial

Similar notebooks:

notebook.community | gallery | about