In [ ]:
import utils
%matplotlib inline
%load_ext autoreload
%autoreload 2
user_pairs = [
['annotator1', 'annotator2'],
]
In [10]:
fine_results = utils.do_iaa_pairs(user_pairs)
utils.print_iaa_pairs(fine_results, user_pairs)
annotator1, annotator2
Q1: 0.600
Q2: 0.730
Q3: 0.551
In [22]:
course_results = utils.do_iaa_pairs(user_pairs, collection='round1', level='coarse')
utils.print_iaa_pairs(course_results, user_pairs)
annotator1, annotator2
Q1: 0.602
Q2: 0.906
Q3: 0.665
In [17]:
print(utils.get_user_results('annotator1', collection='round1'))
print('\n')
print(utils.get_user_results('annotator2', collection='round1'))
*** Annotator: annotator1 ***
===================================
17 Skipped Queries:
"lowongan kerja bumn"
--- "Non-English; couldn't interpret."
"oper"
--- "Either non-English or incomplete. Perhaps most likely an unfinished request for Opera House; but could also be opera in general, Opera web browser, a misspelling, etc."
"ww"
--- "Incomplete/typo/unintelligible"
"cr"
--- "incomplete"
"muah"
--- "Indeterminate - returns results for acronyms, onomatopoeia, non-English words and one obscure shop in a Sydney suburb, but who knows if that existed in 2008."
"to"
--- "incomplete?"
"ilatino ilatino"
--- "returned no results unless corrected to 'latino'; reduplication makes it seem like a specific usage, can't interpret."
"colour"
--- "too generic"
"Short Term"
--- "Underspecified"
"Mother mary be intec"
--- "incomplete; can't interpret."
"zhi"
--- "Non-English"
"face"
--- "Indeterminate: incomplete request for Facebook? Looking for face products? An acronym? Top three search results show all these"
"123"
--- "indeterminate: multiple websites begin with 123"
"darli"
--- "incomplete or non-English"
"whispet"
--- "could be typo for 'whisper', or one of a number of niche meanings. Indeterminate"
"was ist los mit zar rand"
--- "non-English"
"lo"
--- "incomplete"
483 Annotations:
1. Is this query best answered with a pin on a map?
True: 176
False: 307
2. Is a location explicit in the query?
YY: 159
YN: 7
NY: 138
NN: 179
3. What type of query is this?
IAD: 21
IDC: 21
IDO: 10
ILI: 95
ILO: 122
IUN: 52
NAV: 108
RDE: 17
RIN: 24
RMA: 11
ROB: 2
*** Annotator: annotator2 ***
===================================
19 Skipped Queries:
"kenh14"
--- "likely vietnamese. drama show? had to translate"
"info:mybill.vodafone.com.au/Download/doc.pdf?id=Lpxubgg9AxGjzwpyT89hJG22yBOUoNko4 oIDVN7Zqz3gOT/gALaTCEhx4NaG9d1PZ9y3YvDPbUYKp5AV8NaDg=="
--- "url"
"site:topics.nytimes.com"
--- "cut and paste"
"keep cal"
--- "incomplete?"
"sparkasse.it"
--- "foreign language"
"-33.870601151.2087237"
--- "numbers only"
"tjjtds"
--- "foreign language"
"cr"
--- "incomplete"
"http:/www.lwchc.org.au/"
--- "url"
"123"
--- "incomplete"
"other education jobs in"
--- "incomplete"
"was ist los mit zar rand"
--- "foreign language"
"dantri"
--- "foreign language?"
"ww"
--- "incomplete"
"come trovare fb da n cell"
--- "foreign language"
"lowongan kerja bumn"
--- "LOTE"
"fghjj"
--- "unintelligible"
"neuseelanf"
--- "LOTE"
"ilatino ilatino"
--- "LOTE"
481 Annotations:
1. Is this query best answered with a pin on a map?
True: 111
False: 370
2. Is a location explicit in the query?
YY: 159
YN: 5
NY: 107
NN: 210
3. What type of query is this?
IAD: 16
IDC: 19
IDO: 5
ILI: 85
ILO: 88
IUN: 135
NAV: 92
RDE: 14
RDO: 3
RIN: 15
RMA: 4
ROB: 5
In [21]:
for question in (1,2,3):
print(utils.show_agreement(question, ['annotator1', 'annotator2'], collection='round1'))
print('\n')
Question 1:
Number all agree: 391
Number with some disagreement: 79
annotator1 annotator2
1 0 australia
1 0 Retail Sales in Sydney Plaza
1 0 westfield sydney
1 0 Sydney Australia
1 0 tom ford sandrine sunglasses
1 0 kogan
1 0 Sydney NSW
1 0 event cinema george street
1 0 black and white roshe
1 0 accommodation Cairns
1 0 Sydney
1 0 Cars Vans
1 0 garden wall art
1 0 amaysim
0 1 Northern Beaches Sydney NSW accommodation
0 1 Korean
0 1 manly council
1 0 monster charlize theron
1 0 herschel
1 0 salvatore ferragamo
1 0 gumtree sydney
1 0 computer
1 0 acquescutum jacket
1 0 clothes rack
1 0 bath
1 0 agent musical theatre melbourne
1 0 solitaire engagement rings
1 0 last minute hotel deals
1 0 hebrew or hungarian jobs
1 0 All Categories Gym
1 0 cleaner/dishwasher
1 0 All Categories Rooms to rent in bondi beach
1 0 hdmi media hub
1 0 Adidas originals
1 0 deck oven repair sydney
0 1 microsoft
1 0 siemens
1 0 opening kmart sydney broadway road
1 0 nightbus sydney to airport
1 0 sandboarding
1 0 haviland gold on ivory
1 0 Vision street wear longboard longboard
1 0 buying a longboard surfboard
1 0 stila seoul palette
1 0 night bus sydney
1 0 blue mountain tours
1 0 Electric guitar
0 1 All Categories scooter
1 0 apartments brooklyn ny
1 0 route 1 san francisco los angeles
1 0 Sydney City Shared Room Darling Harbour
1 0 buses sydney
1 0 cream and gold engagement cake
1 0 COMMERCIAL CLOTH STAND CLOTHING RACK DISPLAY FOR MARKET
1 0 TOYOTA HI ACE CAMPER ..HI ROOF Campervan
1 0 westfield stdney
1 0 ara
1 0 face atlier ulltra foundation cclarebear
1 0 shopping The Village de Balgowlah Sydney
1 0 used subaru outback for sale
1 0 EUIV
1 0 Valco Seville Glider Chair Black
1 0 adhesive velcro tape
1 0 bom sydney
1 0 Restaurants Sydney City of Sydney CBD Cheap eats
1 0 All New and Used Cars
1 0 Real estate agents Elizabeth BayKings CrossPotts PointRushcutters BayWoolloomooloo NSW
1 0 travel bug sydney
1 0 josh goot to 57 william st
1 0 mobile hog roast
1 0 tow home
0 1 matthew lee worldcitimedical
0 1 show pony
1 0 manfrotto tripod for 650d
1 0 rentawreck alexandria
0 1 tree of life opening hours
1 0 Carols on the beach Sydney
1 0 new zealand skin sheep gloves
1 0 sydney rent a car brisbane
Question 2:
Number all agree: 388
Number with some disagreement: 82
annotator1 annotator2
NY NN All Categories Washing machine
NY NN rock bolting
NY NN translate.google.com.vn
NY NN tom ford sandrine sunglasses
NY NN kogan
NY NN OVERNIGHT/SUPERVISOR NEEDED
NY NN manager
NY NN black and white roshe
YY NN sydney herald student discount subscribe
YY NY myer
NY NN tablet
NN NY one legal
NN NY Sports
NY NN amaysim
NY NN Area manager
NY NN clothes hang rails
NN NY Korean
NY NN All Categories Mirror
NN YY gumtree australia
NY NN herschel
NY NN Sewell magazine
NY NN salvatore ferragamo
NY NN Recruitment
NY NN computer
YY NY australia post
NY YY fc bayern
NN NY paper
NY NN bath
YY NN Charity Advocate for CARE Australia
NY YY pie by micks
YY YN telstra stoes cbd
NN YY LOVELY CLEAN SHAREHOUSE IN NEWTOWN FOR 1 MALE TWINSHARE
NN NY microsoft
NN NY bn2411 prada
NY NN lovisa world manager
NY NN WANT TO JOIN OUR MUFFIN BREAK TEAM?
YY YN topgear festival australia fee
YY NN australian tax calculator
NY NN outbound sales
NN NY abc1 careers
NY NN siemens
NN NY T321543 Snow Leopard Havana Carnac
NY YY paddys markets
NY NN mayan
NY NN sandboarding
NN NY Apple iPhone 5 (16GB White) Unlocked iphone 5
NY NN haviland gold on ivory
NN NY kindle voucher
NY NN Vision street wear longboard longboard
NY NN stila seoul palette
NN NY rails update multiple records at once
YN YY http://www.weather.com.au/nsw/sydney
NY YY AFP raid the australian copies newspaper street
NY NN Tent 3 persons wok cooker tent
YY NY chinasmack japanese
NY NN cream and gold engagement cake
NY NN COMMERCIAL CLOTH STAND CLOTHING RACK DISPLAY FOR MARKET
NY NN Locked 32G iPhone 4S - used but good condition iphone
NY YY nine west au
NN NY Models required $500 for 5 hours
YY NY the parlour westfield
NY NN ara
YN NY fernwood
NN NY october 12 artist
NN NY Clean Quiet and Fully Furnished
NY NN http:/game.com.au/
NY NN EUIV
NY NN Valco Seville Glider Chair Black
NY NN http:/www.youtube.com/watch?v=0krWlZUqWqI
NY YY marea dress code
NY NN adhesive velcro tape
NY NN can i get bill
YY NN vodafone australia prepaid data packs
YN YY josh goot to 57 william st
NN NY Now Hiring KITCHEN HANDS!!!
NY YY LEADING NSW ENERGY RETAILER. TOP $$$. 10 SPOTS TO FILL. FULLTIME.
NN NY show pony
NY NN manfrotto tripod for 650d
YY NN chennai supe
NY NN new zealand skin sheep gloves
NY NN PERFECT FOR OZZIES AND BACKPACKERS ALIKE Backpacker
NY YY the book kitchen cafe
Question 3:
Number all agree: 303
Number with some disagreement: 167
annotator1 annotator2
RMA IUN australia
ILO ILI coach jobs in Brisbane QLD
ILI ROB doctor fanfic reinette
RMA IUN Sydney Australia
ILI IUN chinese new year 2013 sydney
ILO IUN tom ford sandrine sunglasses
ILO NAV kogan
IUN RDO Office
RMA IUN Sydney NSW
ILI IUN marketing assistant
IUN IDC agile
IDC IUN good weekend fashion editor
IDC IUN sydney sunset
ILI IUN Housekeeping
ILI NAV OVERNIGHT/SUPERVISOR NEEDED
ILI IUN manager
ILO IUN black and white roshe
RIN NAV sydney herald student discount subscribe
RMA IUN Sydney
IUN ILO Sports
ILO ILI uniting churches in sydney
IDO IUN roxy jecenko husband arrested
ILI RIN Property for Rent
ILO IUN garden wall art
ILO NAV amaysim
IUN ILO alexandria
ILO ILI Restaurants
ILI IUN Area manager
IUN RDE Sport
ILI IUN poison ring
NAV ILI used BMW 320i SEDAN
IUN ILI Korean
RIN RMA manly council
RDE IUN monster charlize theron
IUN RDE bizarre news
ILO IUN herschel
ILI ROB Sewell magazine
ILO IUN salvatore ferragamo
ILO IUN computer
NAV IUN free wifi westfield sydney
ILO ILI Pizza
ILO IUN acquescutum jacket
ILI IAD missha black ghassoul peel off nose pack review
NAV IUN qdfur
NAV RIN translate
ILI IUN clothes rack
ILI IUN Dawes Point Haymarket Millers Point Sydney Circular Quay Cockle Bay Walsh Bay Darling Harbour Darlinghurst Surry Hills East Sydney Strawberry Hills Elizabeth Bay Potts Point Rushcutters Bay Woolloomooloo Kings Cross Bondi Bondi Beach North Bondi Tamarama Ben Buckler Clovelly Randwick Randwick North Coogee South Coogee Lavender Bay Mcmahons Point North Sydney Waverton The Rocks
ILO IUN crowne plaza gold tower gold coast
RDE IUN graffiti by dma crew melbourne
ILO IUN bath
RIN IUN cute teddybear
RDE IUN turning Japanese
ILI IUN solitaire engagement rings
ILI IUN Charity Advocate for CARE Australia
RIN IUN messina triple chocolate cake
RIN IUN colour chart
IAD IDO waiting for hr to respond interview
ILO IUN hdmi media hub
ILI ILO piasau miri luxury hause
ILO IUN Adidas originals
ILO IUN Canberra ACT
IDO IUN effective teaching by john hattie
NAV ILO microsoft
ILI IUN bn2411 prada
NAV ILI Wait and Bar Staff Full and Part Time
IDO IAD calf exercises
NAV IUN lovisa world manager
ILO IUN Paris
ILO ILI Luggage Stores Bondi Junction
ILI ILO moskva kyrka
IAD IUN salted shredded pork
ILI IUN outbound sales
NAV ILI abc1 careers
ILI ROB cockney slang dictionary
ILO NAV siemens
RIN ILI ebay aztec rug
IDO IUN postcard peter skrzynecki techniques
RIN IUN T321543 Snow Leopard Havana Carnac
NAV ILO paddys markets
IDC IUN opening kmart sydney broadway road
ROB RMA north sydney travel access guide
ILO ILI active labour hire sydney
NAV IUN hao123
ILI IUN sandboarding
NAV ILI Apple iPhone 5 (16GB White) Unlocked iphone 5
ILO IUN haviland gold on ivory
NAV IUN kindle voucher
ILO IUN denmark
IDO IAD commers in between numbers
IDO IUN first choice white card
ILO IAD buying a longboard surfboard
ILO IUN stila seoul palette
RIN NAV night bus sydney
IDC IDO C.A.T manga author
ILO IUN Electric guitar
IAD IUN ruby blank 2 values if
ILI ILO All Categories scooter
ILI NAV sydney sell your van for us
IDC ILI omd price in australia
RDE IUN Endless Love
NAV RDO Feast magazine
NAV ILI nme new releases
ILI IUN AFP raid the australian copies newspaper street
RIN ILI pyrmont gym sales
IAD IUN ruby crash caching on
ILI NAV Tent 3 persons wok cooker tent
NAV RDE FOXTEL LIFESTYLE CHANNEL
RIN IUN g star raw clemence poesy
NAV IUN altex romania
RMA IUN route 1 san francisco los angeles
RDE IUN izombie
NAV IUN abc
ILI NAV buses sydney
ILO ILI gay clubs sydney
ILO ILI libraries in waverton sydney
ILI IUN cream and gold engagement cake
ILI IUN COMMERCIAL CLOTH STAND CLOTHING RACK DISPLAY FOR MARKET
NAV ILI Yamaha T MAX 500 in White WANTED
ILI IUN hills quick kerb driveways
IUN IDC Extract
ILO IUN ara
ILO IUN face atlier ulltra foundation cclarebear
NAV IUN nepean mums
ILO IUN shopping The Village de Balgowlah Sydney
IDO NAV october 12 artist
ILO ILI rivers store australia
NAV ILI Clean Quiet and Fully Furnished
IAD IDO mac air vs mac pr
RDE IUN hentai woody
ILO IUN jillys education
RIN RDE beatiful savior
ILO IUN EUIV
ILO IUN Valco Seville Glider Chair Black
NAV RDE http:/www.youtube.com/watch?v=0krWlZUqWqI
IDO NAV THE FINAL OFFER SASS AND BIDE
ILO IUN adhesive velcro tape
NAV ILI Extras Required for Telemovie
RIN IUN LG 50LA6230 50 3D LED LCD TV
IUN ILI latin american republics
IAD ILI best place to buy bose theater system
ILO IUN Kangaroo Island Australie
IAD IUN can i get bill
IUN IDC fit
NAV RIN bom sydney
RIN NAV vodafone australia prepaid data packs
RMA ILO maps westfield sydney
IAD IUN holiday in sydney
ILO ILI Real estate agents Elizabeth BayKings CrossPotts PointRushcutters BayWoolloomooloo NSW
NAV RDO kakao talk
ILI IUN travel bug sydney
RMA NAV josh goot to 57 william st
RIN IUN sydney single rail
ILI NAV ultimatedb
IDC ILO matthew lee worldcitimedical
NAV IUN Shop Assistant in City Martin Place
ILO IUN Darwin Australia
ILI IAD best blu ray player
ILI NAV Now Hiring KITCHEN HANDS!!!
ILI RIN Book titles by keywords
RIN ILO show pony
ILO IUN manfrotto tripod for 650d
IAD IUN approved fallback mean
ILO RIN rentawreck alexandria
NAV RIN Couchsurfing Sydney city
RMA ROB Frenchs Forest map sydney
ILO ILI new zealand skin sheep gloves
RMA ILO westfield CBF
Content source: ned2/tator
Similar notebooks: