Round 1 Annotation Results

500 queries annotated by Amy and Sarah.


In [ ]:
import utils

%matplotlib inline
%load_ext autoreload
%autoreload 2

user_pairs = [
    ['annotator1', 'annotator2'],
]

Inter-annotator Agreement Scores for Round 1

Fine Grained Agreement


In [10]:
fine_results = utils.do_iaa_pairs(user_pairs)
utils.print_iaa_pairs(fine_results, user_pairs)


    annotator1, annotator2
Q1: 0.600                    
Q2: 0.730                    
Q3: 0.551                    

Course grained agreement


In [22]:
course_results = utils.do_iaa_pairs(user_pairs, collection='round1', level='coarse')
utils.print_iaa_pairs(course_results, user_pairs)


    annotator1, annotator2
Q1: 0.602                    
Q2: 0.906                    
Q3: 0.665                    

Overview of responses


In [17]:
print(utils.get_user_results('annotator1', collection='round1'))
print('\n')
print(utils.get_user_results('annotator2', collection='round1'))


*** Annotator: annotator1 ***
===================================

17 Skipped Queries:

    "lowongan kerja bumn"
    --- "Non-English; couldn't interpret."
    "oper"
    --- "Either non-English or incomplete. Perhaps most likely an unfinished request for Opera House; but could also be opera in general, Opera web browser, a misspelling, etc."
    "ww"
    --- "Incomplete/typo/unintelligible"
    "cr"
    --- "incomplete"
    "muah"
    --- "Indeterminate - returns results for acronyms, onomatopoeia, non-English words and one obscure shop in a Sydney suburb, but who knows if that existed in 2008."
    "to"
    --- "incomplete?"
    "ilatino ilatino"
    --- "returned no results unless corrected to 'latino'; reduplication makes it seem like a specific usage, can't interpret."
    "colour"
    --- "too generic"
    "Short Term"
    --- "Underspecified"
    "Mother mary be intec"
    --- "incomplete; can't interpret."
    "zhi"
    --- "Non-English"
    "face"
    --- "Indeterminate: incomplete request for Facebook? Looking for face products? An acronym? Top three search results show all these"
    "123"
    --- "indeterminate: multiple websites begin with 123"
    "darli"
    --- "incomplete or non-English"
    "whispet"
    --- "could be typo for 'whisper', or one of a number of niche meanings. Indeterminate"
    "was ist los mit zar rand"
    --- "non-English"
    "lo"
    --- "incomplete"

483 Annotations:

1. Is this query best answered with a pin on a map?
True: 176
False: 307

2. Is a location explicit in the query?
YY: 159
YN: 7
NY: 138
NN: 179

3. What type of query is this?
IAD: 21
IDC: 21
IDO: 10
ILI: 95
ILO: 122
IUN: 52
NAV: 108
RDE: 17
RIN: 24
RMA: 11
ROB: 2


*** Annotator: annotator2 ***
===================================

19 Skipped Queries:

    "kenh14"
    --- "likely vietnamese. drama show? had to translate"
    "info:mybill.vodafone.com.au/Download/doc.pdf?id=Lpxubgg9AxGjzwpyT89hJG22yBOUoNko4 oIDVN7Zqz3gOT/gALaTCEhx4NaG9d1PZ9y3YvDPbUYKp5AV8NaDg=="
    --- "url"
    "site:topics.nytimes.com"
    --- "cut and paste"
    "keep cal"
    --- "incomplete?"
    "sparkasse.it"
    --- "foreign language"
    "-33.870601151.2087237"
    --- "numbers only"
    "tjjtds"
    --- "foreign language"
    "cr"
    --- "incomplete"
    "http:/www.lwchc.org.au/"
    --- "url"
    "123"
    --- "incomplete"
    "other education jobs in"
    --- "incomplete"
    "was ist los mit zar rand"
    --- "foreign language"
    "dantri"
    --- "foreign language?"
    "ww"
    --- "incomplete"
    "come trovare fb da n cell"
    --- "foreign language"
    "lowongan kerja bumn"
    --- "LOTE"
    "fghjj"
    --- "unintelligible"
    "neuseelanf"
    --- "LOTE"
    "ilatino ilatino"
    --- "LOTE"

481 Annotations:

1. Is this query best answered with a pin on a map?
True: 111
False: 370

2. Is a location explicit in the query?
YY: 159
YN: 5
NY: 107
NN: 210

3. What type of query is this?
IAD: 16
IDC: 19
IDO: 5
ILI: 85
ILO: 88
IUN: 135
NAV: 92
RDE: 14
RDO: 3
RIN: 15
RMA: 4
ROB: 5

Disagreements


In [21]:
for question in (1,2,3):
    print(utils.show_agreement(question, ['annotator1', 'annotator2'], collection='round1'))
    print('\n')


Question 1:
Number all agree: 391
Number with some disagreement: 79

annotator1  annotator2  
1           0           australia
1           0           Retail Sales in Sydney Plaza
1           0           westfield sydney
1           0           Sydney Australia
1           0           tom ford sandrine sunglasses
1           0           kogan
1           0           Sydney NSW
1           0           event cinema george street
1           0           black and white roshe
1           0           accommodation Cairns
1           0           Sydney
1           0           Cars Vans
1           0           garden wall art
1           0           amaysim
0           1           Northern Beaches Sydney NSW accommodation
0           1           Korean
0           1           manly council
1           0           monster charlize theron
1           0           herschel
1           0           salvatore ferragamo
1           0           gumtree sydney
1           0           computer
1           0           acquescutum jacket
1           0           clothes rack
1           0           bath
1           0           agent musical theatre melbourne
1           0           solitaire engagement rings
1           0           last minute hotel deals
1           0           hebrew or hungarian jobs
1           0           All Categories Gym
1           0           cleaner/dishwasher
1           0           All Categories Rooms to rent in bondi beach
1           0           hdmi media hub
1           0           Adidas originals
1           0           deck oven repair sydney
0           1           microsoft
1           0           siemens
1           0           opening kmart sydney broadway road
1           0           nightbus sydney to airport
1           0           sandboarding
1           0           haviland gold on ivory
1           0           Vision street wear longboard longboard
1           0           buying a longboard surfboard
1           0           stila seoul palette
1           0           night bus sydney
1           0           blue mountain tours
1           0           Electric guitar
0           1           All Categories scooter
1           0           apartments brooklyn ny
1           0           route 1 san francisco los angeles
1           0           Sydney City Shared Room Darling Harbour
1           0           buses sydney
1           0           cream and gold engagement cake
1           0           COMMERCIAL CLOTH STAND CLOTHING RACK DISPLAY FOR MARKET
1           0           TOYOTA HI ACE CAMPER ..HI ROOF Campervan
1           0           westfield stdney
1           0           ara
1           0           face atlier ulltra foundation cclarebear
1           0           shopping The Village de Balgowlah Sydney
1           0           used subaru outback for sale
1           0           EUIV
1           0           Valco Seville Glider Chair Black
1           0           adhesive velcro tape
1           0           bom sydney
1           0           Restaurants Sydney City of Sydney CBD Cheap eats
1           0           All New and Used Cars
1           0           Real estate agents Elizabeth BayKings CrossPotts PointRushcutters BayWoolloomooloo NSW
1           0           travel bug sydney
1           0           josh goot to 57 william st
1           0           mobile hog roast
1           0           tow home
0           1           matthew lee worldcitimedical
0           1           show pony
1           0           manfrotto tripod for 650d
1           0           rentawreck alexandria
0           1           tree of life opening hours
1           0           Carols on the beach Sydney
1           0           new zealand skin sheep gloves
1           0           sydney rent a car brisbane


Question 2:
Number all agree: 388
Number with some disagreement: 82

annotator1  annotator2  
NY          NN          All Categories Washing machine
NY          NN          rock bolting
NY          NN          translate.google.com.vn
NY          NN          tom ford sandrine sunglasses
NY          NN          kogan
NY          NN          OVERNIGHT/SUPERVISOR NEEDED
NY          NN          manager
NY          NN          black and white roshe
YY          NN          sydney herald student discount subscribe
YY          NY          myer
NY          NN          tablet
NN          NY          one legal
NN          NY          Sports
NY          NN          amaysim
NY          NN          Area manager
NY          NN          clothes hang rails
NN          NY          Korean
NY          NN          All Categories Mirror
NN          YY          gumtree australia
NY          NN          herschel
NY          NN          Sewell magazine
NY          NN          salvatore ferragamo
NY          NN          Recruitment
NY          NN          computer
YY          NY          australia post
NY          YY          fc bayern
NN          NY          paper
NY          NN          bath
YY          NN          Charity Advocate for CARE Australia
NY          YY          pie by micks
YY          YN          telstra stoes cbd
NN          YY          LOVELY CLEAN SHAREHOUSE IN NEWTOWN FOR 1 MALE TWINSHARE
NN          NY          microsoft
NN          NY          bn2411 prada
NY          NN          lovisa world manager
NY          NN          WANT TO JOIN OUR MUFFIN BREAK TEAM?
YY          YN          topgear festival  australia fee
YY          NN          australian tax calculator
NY          NN          outbound sales
NN          NY          abc1 careers
NY          NN          siemens
NN          NY          T321543 Snow Leopard Havana Carnac
NY          YY          paddys markets
NY          NN          mayan
NY          NN          sandboarding
NN          NY          Apple iPhone 5 (16GB White) Unlocked iphone 5
NY          NN          haviland gold on ivory
NN          NY          kindle voucher
NY          NN          Vision street wear longboard longboard
NY          NN          stila seoul palette
NN          NY          rails update multiple records at once
YN          YY          http://www.weather.com.au/nsw/sydney
NY          YY          AFP raid the australian copies newspaper street
NY          NN          Tent 3 persons   wok cooker tent
YY          NY          chinasmack japanese
NY          NN          cream and gold engagement cake
NY          NN          COMMERCIAL CLOTH STAND CLOTHING RACK DISPLAY FOR MARKET
NY          NN          Locked 32G iPhone 4S - used but good condition iphone
NY          YY          nine west au
NN          NY          Models required $500 for 5 hours
YY          NY          the parlour westfield
NY          NN          ara
YN          NY          fernwood
NN          NY          october 12 artist
NN          NY          Clean Quiet and Fully Furnished
NY          NN          http:/game.com.au/
NY          NN          EUIV
NY          NN          Valco Seville Glider Chair Black
NY          NN          http:/www.youtube.com/watch?v=0krWlZUqWqI
NY          YY          marea dress code
NY          NN          adhesive velcro tape
NY          NN          can i get bill
YY          NN          vodafone australia prepaid data packs
YN          YY          josh goot to 57 william st
NN          NY          Now Hiring KITCHEN HANDS!!!
NY          YY          LEADING NSW ENERGY RETAILER. TOP $$$. 10 SPOTS TO FILL. FULLTIME.
NN          NY          show pony
NY          NN          manfrotto tripod for 650d
YY          NN          chennai supe
NY          NN          new zealand skin sheep gloves
NY          NN          PERFECT FOR OZZIES AND BACKPACKERS ALIKE Backpacker
NY          YY          the book kitchen cafe


Question 3:
Number all agree: 303
Number with some disagreement: 167

annotator1  annotator2  
RMA         IUN         australia
ILO         ILI         coach jobs in Brisbane QLD
ILI         ROB         doctor fanfic reinette
RMA         IUN         Sydney Australia
ILI         IUN         chinese new year 2013 sydney
ILO         IUN         tom ford sandrine sunglasses
ILO         NAV         kogan
IUN         RDO         Office
RMA         IUN         Sydney NSW
ILI         IUN         marketing assistant
IUN         IDC         agile
IDC         IUN         good weekend fashion editor
IDC         IUN         sydney sunset
ILI         IUN         Housekeeping
ILI         NAV         OVERNIGHT/SUPERVISOR NEEDED
ILI         IUN         manager
ILO         IUN         black and white roshe
RIN         NAV         sydney herald student discount subscribe
RMA         IUN         Sydney
IUN         ILO         Sports
ILO         ILI         uniting churches in sydney
IDO         IUN         roxy jecenko husband arrested
ILI         RIN         Property for Rent
ILO         IUN         garden wall art
ILO         NAV         amaysim
IUN         ILO         alexandria
ILO         ILI         Restaurants
ILI         IUN         Area manager
IUN         RDE         Sport
ILI         IUN         poison ring
NAV         ILI         used BMW 320i SEDAN
IUN         ILI         Korean
RIN         RMA         manly council
RDE         IUN         monster charlize theron
IUN         RDE         bizarre news
ILO         IUN         herschel
ILI         ROB         Sewell magazine
ILO         IUN         salvatore ferragamo
ILO         IUN         computer
NAV         IUN         free wifi westfield sydney
ILO         ILI         Pizza
ILO         IUN         acquescutum jacket
ILI         IAD         missha black ghassoul peel off nose pack review
NAV         IUN         qdfur
NAV         RIN         translate
ILI         IUN         clothes rack
ILI         IUN         Dawes Point Haymarket Millers Point Sydney Circular Quay Cockle Bay Walsh Bay Darling Harbour Darlinghurst Surry Hills East Sydney Strawberry Hills Elizabeth Bay Potts Point Rushcutters Bay Woolloomooloo Kings Cross Bondi Bondi Beach North Bondi Tamarama Ben Buckler Clovelly Randwick Randwick North Coogee South Coogee Lavender Bay Mcmahons Point North Sydney Waverton The Rocks
ILO         IUN         crowne plaza gold tower gold coast
RDE         IUN         graffiti by dma crew melbourne
ILO         IUN         bath
RIN         IUN         cute teddybear
RDE         IUN         turning Japanese
ILI         IUN         solitaire engagement rings
ILI         IUN         Charity Advocate for CARE Australia
RIN         IUN         messina triple chocolate cake
RIN         IUN         colour chart
IAD         IDO         waiting for hr to respond interview
ILO         IUN         hdmi media hub
ILI         ILO         piasau miri luxury hause
ILO         IUN         Adidas originals
ILO         IUN         Canberra ACT
IDO         IUN         effective teaching by john hattie
NAV         ILO         microsoft
ILI         IUN         bn2411 prada
NAV         ILI         Wait and Bar Staff  Full and Part Time
IDO         IAD         calf exercises
NAV         IUN         lovisa world manager
ILO         IUN         Paris
ILO         ILI         Luggage Stores Bondi Junction
ILI         ILO         moskva kyrka
IAD         IUN         salted shredded pork
ILI         IUN         outbound sales
NAV         ILI         abc1 careers
ILI         ROB         cockney slang dictionary
ILO         NAV         siemens
RIN         ILI         ebay aztec rug
IDO         IUN         postcard peter skrzynecki techniques
RIN         IUN         T321543 Snow Leopard Havana Carnac
NAV         ILO         paddys markets
IDC         IUN         opening kmart sydney broadway road
ROB         RMA         north sydney travel access guide
ILO         ILI         active labour hire sydney
NAV         IUN         hao123
ILI         IUN         sandboarding
NAV         ILI         Apple iPhone 5 (16GB White) Unlocked iphone 5
ILO         IUN         haviland gold on ivory
NAV         IUN         kindle voucher
ILO         IUN         denmark
IDO         IAD         commers in between numbers
IDO         IUN         first choice white card
ILO         IAD         buying a longboard surfboard
ILO         IUN         stila seoul palette
RIN         NAV         night bus sydney
IDC         IDO         C.A.T manga author
ILO         IUN         Electric guitar
IAD         IUN         ruby blank 2 values if
ILI         ILO         All Categories scooter
ILI         NAV         sydney sell your van for us
IDC         ILI         omd price in australia
RDE         IUN         Endless Love
NAV         RDO         Feast magazine
NAV         ILI         nme new releases
ILI         IUN         AFP raid the australian copies newspaper street
RIN         ILI         pyrmont gym sales
IAD         IUN         ruby crash caching on
ILI         NAV         Tent 3 persons   wok cooker tent
NAV         RDE         FOXTEL LIFESTYLE CHANNEL
RIN         IUN         g star raw clemence poesy
NAV         IUN         altex romania
RMA         IUN         route 1 san francisco los angeles
RDE         IUN         izombie
NAV         IUN         abc
ILI         NAV         buses sydney
ILO         ILI         gay clubs sydney
ILO         ILI         libraries in waverton sydney
ILI         IUN         cream and gold engagement cake
ILI         IUN         COMMERCIAL CLOTH STAND CLOTHING RACK DISPLAY FOR MARKET
NAV         ILI         Yamaha   T MAX  500   in White   WANTED
ILI         IUN         hills quick kerb driveways
IUN         IDC         Extract
ILO         IUN         ara
ILO         IUN         face atlier ulltra foundation cclarebear
NAV         IUN         nepean mums
ILO         IUN         shopping The Village de Balgowlah Sydney
IDO         NAV         october 12 artist
ILO         ILI         rivers store australia
NAV         ILI         Clean Quiet and Fully Furnished
IAD         IDO         mac air vs mac pr
RDE         IUN         hentai woody
ILO         IUN         jillys education
RIN         RDE         beatiful savior
ILO         IUN         EUIV
ILO         IUN         Valco Seville Glider Chair Black
NAV         RDE         http:/www.youtube.com/watch?v=0krWlZUqWqI
IDO         NAV         THE FINAL OFFER SASS AND BIDE
ILO         IUN         adhesive velcro tape
NAV         ILI         Extras Required for Telemovie
RIN         IUN         LG 50LA6230 50 3D LED LCD TV
IUN         ILI         latin american republics
IAD         ILI         best place to buy bose theater system
ILO         IUN         Kangaroo Island Australie
IAD         IUN         can i get bill
IUN         IDC         fit
NAV         RIN         bom sydney
RIN         NAV         vodafone australia prepaid data packs
RMA         ILO         maps westfield sydney
IAD         IUN         holiday in sydney
ILO         ILI         Real estate agents Elizabeth BayKings CrossPotts PointRushcutters BayWoolloomooloo NSW
NAV         RDO         kakao talk
ILI         IUN         travel bug sydney
RMA         NAV         josh goot to 57 william st
RIN         IUN         sydney single rail
ILI         NAV         ultimatedb
IDC         ILO         matthew lee worldcitimedical
NAV         IUN         Shop Assistant in City Martin Place
ILO         IUN         Darwin Australia
ILI         IAD         best blu ray player
ILI         NAV         Now Hiring KITCHEN HANDS!!!
ILI         RIN         Book titles by keywords
RIN         ILO         show pony
ILO         IUN         manfrotto tripod for 650d
IAD         IUN         approved fallback mean
ILO         RIN         rentawreck alexandria
NAV         RIN         Couchsurfing Sydney city
RMA         ROB         Frenchs Forest map sydney
ILO         ILI         new zealand skin sheep gloves
RMA         ILO         westfield CBF