Filter evaluations by precision-recall analysis

1 Setup

Flags and settings.


In [1]:
SAMPLE_SIZE = 100

Imports and database setup.


In [2]:
from random import sample
from textwrap import indent, fill

import numpy as np

%cd -q ..
from brainscopypaste.conf import settings
%cd -q notebooks
from brainscopypaste.db import Cluster, Quote, Substitution
from brainscopypaste.utils import init_db, session_scope, langdetect
from brainscopypaste.filter import filter_quote_offset
from brainscopypaste.mine import Model, Time, Source, Past, Durl
engine = init_db()

2 Evaluate language filtering

Either run this cell to generate a new selection...


In [3]:
#with session_scope() as session:
#    quote_ids = sample([id for (id,)
#                        in session.query(Quote.id).filter(Quote.filtered == False)],
#                       SAMPLE_SIZE)

... or this one to use the previous selection (which has already been coded).


In [4]:
quote_ids = [1956761, 1987670, 2013139, 1762813, 514301, 203909, 2369165, 1912805, 1534840,
             464066, 152061, 720930, 2331572, 268789, 457062, 1219889, 790024, 2309594,
             2712468, 1282691, 730591, 1840441, 382029, 376767, 1058965, 2423390, 1013911,
             1558617, 2491471, 777666, 648372, 2123449, 630199, 1479403, 488899, 2501461,
             1453754, 1190828, 239246, 1981112, 491273, 2554944, 80036, 1433989, 1067430,
             67526, 937337, 952234, 110443, 123775, 1454732, 1600502, 1877933, 1746630,
             1387646, 654399, 1896017, 254697, 72175, 657151, 2296113, 2692900, 2340689,
             1253031, 2671619, 1299967, 2323288, 321528, 2391467, 2704508, 2596163, 2044260,
             420936, 448593, 966688, 252997, 1573373, 328472, 1176294, 494275, 1110521, 74025,
             647887, 2469274, 1748974, 209377, 116114, 2657880, 923507, 2530004, 200625, 671881,
             209809, 2651830, 1105095, 120217, 16153, 95646, 1162318, 1267527]

In [5]:
with session_scope() as session:
    strings = [session.query(Quote).get(id).string for id in quote_ids]
strings_langs = [(string, langdetect(string)) for string in strings]

In [6]:
print("Over a sample of {}, {} quotes are rejected because their detected "
      "language is not English"
      .format(SAMPLE_SIZE, np.sum([lang != 'en' for _, lang in strings_langs])))


Over a sample of 100, 17 quotes are rejected because their detected language is not English

Here are the individual strings and their detected languages.


In [7]:
for i, (string, lang) in enumerate(strings_langs):
    title = ' {} / {}'.format(i + 1, SAMPLE_SIZE)
    print('-' * (80 - len(title)) + title)
    print('Language:', lang)
    print()
    print(indent(fill(string), ' ' * 5))
    print()


------------------------------------------------------------------------ 1 / 100
Language: en

     some states swear in their new governors in early december a month
     after the election and parliamentary democracies produce new prime
     ministers within a day or two of the election

------------------------------------------------------------------------ 2 / 100
Language: en

     they want to set an example for harlow nicole and joel are doing this
     to seal their love legally they want the ceremony to coincide with
     harlow's first birthday and be one big joyous occasion for everybody

------------------------------------------------------------------------ 3 / 100
Language: en

     i cannot believe what i have seen in the last 36 hours i have seen
     dead bodies blood everywhere and only heard gunshots

------------------------------------------------------------------------ 4 / 100
Language: en

     but i have to reiterate once again that we only have one president at
     a time and i want to be very careful that we are sending the right
     signals to the world as a whole that i am not the president and i
     won't be until january 20th

------------------------------------------------------------------------ 5 / 100
Language: en

     i extracted some hair from the alleged corpse and examined it and had
     some concerns

------------------------------------------------------------------------ 6 / 100
Language: en

     it ain't what you don't know that gets you into trouble it's what you
     know for sure that just ain't so

------------------------------------------------------------------------ 7 / 100
Language: en

     we're absolutely sure we want to keep him ric's a good player and you
     don't sell your good players

------------------------------------------------------------------------ 8 / 100
Language: en

     there are some games that are hardcore experiences that the kudos lies
     in getting it first

------------------------------------------------------------------------ 9 / 100
Language: en

     christian belief is not about worrying or not enjoying life

----------------------------------------------------------------------- 10 / 100
Language: en

     oh i think so

----------------------------------------------------------------------- 11 / 100
Language: en

     why did god do this to me

----------------------------------------------------------------------- 12 / 100
Language: en

     it's my party and i'll die if i want to

----------------------------------------------------------------------- 13 / 100
Language: es

     jugamos con gran intensidad durante la mayor parte de los 48 minutos y
     c mo acabamos fue la diferencia fue un partido f sico pudimos
     contenerlos y mantener nuestra voz

----------------------------------------------------------------------- 14 / 100
Language: en

     a red herring

----------------------------------------------------------------------- 15 / 100
Language: de

     ihre hausaufgaben gemacht

----------------------------------------------------------------------- 16 / 100
Language: en

     an invisible product

----------------------------------------------------------------------- 17 / 100
Language: en

     tap our natural gas reserves

----------------------------------------------------------------------- 18 / 100
Language: en

     the last kicks of a dying horse

----------------------------------------------------------------------- 19 / 100
Language: en

     for those who wish to obstruct get ready to get knocked over

----------------------------------------------------------------------- 20 / 100
Language: en

     this is someone who sees american as imperfect enough to pal around
     with terrorists who targeted their own country

----------------------------------------------------------------------- 21 / 100
Language: en

     god will take care of us

----------------------------------------------------------------------- 22 / 100
Language: en

     governs with the skill and the grace and the greatness of which he is
     capable

----------------------------------------------------------------------- 23 / 100
Language: ca

     en la buena direcci n

----------------------------------------------------------------------- 24 / 100
Language: en

     i'm a dude playing a dude disguised as another dude

----------------------------------------------------------------------- 25 / 100
Language: en

     the pagans and the abortionists and the feminists and the gays and the
     lesbians who are actively trying to make that an alternative lifestyle
     the aclu people for the american way all of them who have tried to
     secularize america

----------------------------------------------------------------------- 26 / 100
Language: en

     after 62 days after the careful and painstaking hand inspection of
     nearly 3 million ballots after hours and hours of hard work by
     elections officials and volunteers across the state i am proud and
     humbled to stand before you as the next senator from minnesota

----------------------------------------------------------------------- 27 / 100
Language: en

     the art of participation 1950 to now

----------------------------------------------------------------------- 28 / 100
Language: en

     at a regulatory commission hearing a claim for wrongful dismissal from
     newcastle united defender habib beye was upheld

----------------------------------------------------------------------- 29 / 100
Language: en

     it is our position that we have done everything that's required and
     that yes i should be seated forthwith and i should be seated prior to
     the inauguration of our 44th president a seat for which i now hold

----------------------------------------------------------------------- 30 / 100
Language: en

     my father saved my life

----------------------------------------------------------------------- 31 / 100
Language: sv

     grand d fi

----------------------------------------------------------------------- 32 / 100
Language: en

     if people are saving for a rainy day i can tell them that it's pouring
     outside

----------------------------------------------------------------------- 33 / 100
Language: en

     unfortunately to date our demands have not been addressed rather more
     civilians including women and children are losing their lives as a
     result of air raids

----------------------------------------------------------------------- 34 / 100
Language: en

     it was tough for me to even get out of bed and i had days when i
     couldn't walk let alone write a rhyme

----------------------------------------------------------------------- 35 / 100
Language: en

     let there be no doubt these are dangerous individuals these men were
     not simply in possession of material which expressed extremist views
     they were also in possession of material that was operationally useful
     to anyone wishing to carry out an act of violence or terrorism

----------------------------------------------------------------------- 36 / 100
Language: en

     according to reuters potsdam university in germany is now teaching
     social skills as part of their it courses this is intended to ease
     entry into

----------------------------------------------------------------------- 37 / 100
Language: en

     totally fabricated 99 per cent of it is totally untrue

----------------------------------------------------------------------- 38 / 100
Language: en

     twas the night before

----------------------------------------------------------------------- 39 / 100
Language: en

     repent or burn

----------------------------------------------------------------------- 40 / 100
Language: en

     blessed is the fruit of your womb

----------------------------------------------------------------------- 41 / 100
Language: fr

     spit out dunces

----------------------------------------------------------------------- 42 / 100
Language: en

     but always within the confines of a budget which is controlled by the
     owners and the club

----------------------------------------------------------------------- 43 / 100
Language: es

     cuando hay un problema o dificultad

----------------------------------------------------------------------- 44 / 100
Language: en

     they tried hard over the last few months but their fighting was
     getting out of control

----------------------------------------------------------------------- 45 / 100
Language: en

     the more pressure there is on these companies the less we will see in
     terms of affordable housing

----------------------------------------------------------------------- 46 / 100
Language: en

     how it's made

----------------------------------------------------------------------- 47 / 100
Language: en

     he will be expelled

----------------------------------------------------------------------- 48 / 100
Language: it

     4 or more illegal aliens

----------------------------------------------------------------------- 49 / 100
Language: en

     quit playing games with my heart

----------------------------------------------------------------------- 50 / 100
Language: sq

     a short time

----------------------------------------------------------------------- 51 / 100
Language: en

     if it is a strain then he could be out for about a week

----------------------------------------------------------------------- 52 / 100
Language: en

     fallen off the edge of the cliff

----------------------------------------------------------------------- 53 / 100
Language: en

     this will be good for him and for the nation

----------------------------------------------------------------------- 54 / 100
Language: de

     darf ich mal mit deinem hund reden

----------------------------------------------------------------------- 55 / 100
Language: en

     will urge north korea to take immediate steps to implement the
     commitments it made this summer as part the agreement reached with
     japan

----------------------------------------------------------------------- 56 / 100
Language: en

     we didn't know what was going on

----------------------------------------------------------------------- 57 / 100
Language: en

     no automatic right of access

----------------------------------------------------------------------- 58 / 100
Language: en

     make no law respecting an establishment of religion

----------------------------------------------------------------------- 59 / 100
Language: en

     it's going to be special

----------------------------------------------------------------------- 60 / 100
Language: en

     nicollette sheridan and michael bolton have amicably ended their
     engagement they appreciate your respect for their privacy in this
     matter

----------------------------------------------------------------------- 61 / 100
Language: en

     this is very very very bad

----------------------------------------------------------------------- 62 / 100
Language: en

     but it doesn't make it right because we've had other things we're
     ashamed of like slavery

----------------------------------------------------------------------- 63 / 100
Language: es

     no se trata de ning n truco visual como pasaba con las antiguas
     producciones en este formato ser una aut ntica revoluci n en los a os
     20 fue el paso del mudo al sonoro en los 30 del blanco y negro al
     color y ahora cambiaremos del 2d al 3d

----------------------------------------------------------------------- 64 / 100
Language: en

     car of the year 2009

----------------------------------------------------------------------- 65 / 100
Language: en

     we are targeting a milestone release for the first week of february
     targeting the htc touch pro

----------------------------------------------------------------------- 66 / 100
Language: en

     nothing is ever done

----------------------------------------------------------------------- 67 / 100
Language: es

     s lo para cumplir

----------------------------------------------------------------------- 68 / 100
Language: en

     detention of political dissidents human-rights advocates and religious
     activists

----------------------------------------------------------------------- 69 / 100
Language: en

     pornographic grossly offensive disgusting or otherwise of an obscene
     character

----------------------------------------------------------------------- 70 / 100
Language: en

     if you knew me it's tough for me to concentrate out here but on the
     flip side i'm on the pga tour and i'm not going to take it for granted
     so i know when to put my partying on hold and i know when to put in
     hard work yeah there's some good sights out there

----------------------------------------------------------------------- 71 / 100
Language: en

     the national transportation safety board has now begun their
     investigation of this accident and we are offering our full
     participation and cooperation

----------------------------------------------------------------------- 72 / 100
Language: en

     maybe i had traumatic stress i just remember i did not want to be at
     home when i was in my car i was driving and i was going somewhere

----------------------------------------------------------------------- 73 / 100
Language: en

     the white cliffs of dover

----------------------------------------------------------------------- 74 / 100
Language: en

     behind the story

----------------------------------------------------------------------- 75 / 100
Language: en

     illegal street racing

----------------------------------------------------------------------- 76 / 100
Language: cy

     i'm a huge fan

----------------------------------------------------------------------- 77 / 100
Language: en

     after my last two years i've learned just not to judge anyone and go
     into our friendship with an open mind and not really worry about the
     age or anything

----------------------------------------------------------------------- 78 / 100
Language: en

     build web site

----------------------------------------------------------------------- 79 / 100
Language: en

     criminal act of terrorism

----------------------------------------------------------------------- 80 / 100
Language: en

     catch and shoot

----------------------------------------------------------------------- 81 / 100
Language: fr

     le mot juste

----------------------------------------------------------------------- 82 / 100
Language: en

     dead rising chop till you drop

----------------------------------------------------------------------- 83 / 100
Language: en

     he don't belong in political office

----------------------------------------------------------------------- 84 / 100
Language: en

     they vastly overestimate how upset they would feel in bad situations
     such as hearing a racial slur

----------------------------------------------------------------------- 85 / 100
Language: en

     bliss it was to be alive but to be young was very heaven

----------------------------------------------------------------------- 86 / 100
Language: en

     that's to be determined the human rights abuses inside the country
     still exist and persist

----------------------------------------------------------------------- 87 / 100
Language: en

     web page complete

----------------------------------------------------------------------- 88 / 100
Language: sv

     i'm having a rihanna fixation myself

----------------------------------------------------------------------- 89 / 100
Language: en

     i want my daughters to have exactly the same opportunities as your
     sons

----------------------------------------------------------------------- 90 / 100
Language: en

     i am alive and plan on continuing to stay that way

----------------------------------------------------------------------- 91 / 100
Language: en

     culturally historically or aesthetically

----------------------------------------------------------------------- 92 / 100
Language: en

     georgia's aggression against south ossetia

----------------------------------------------------------------------- 93 / 100
Language: en

     the harvest is plentiful but the workers are few 38 ask the lord of
     the harvest therefore to send out workers into his harvest field

----------------------------------------------------------------------- 94 / 100
Language: de

     wir europ er sind aus humanit ren gr nden gefragt

----------------------------------------------------------------------- 95 / 100
Language: en

     stunning and unprecedented in its scope and lack of detail

----------------------------------------------------------------------- 96 / 100
Language: en

     more of this

----------------------------------------------------------------------- 97 / 100
Language: en

     because it works

----------------------------------------------------------------------- 98 / 100
Language: cy

     i am afraid

----------------------------------------------------------------------- 99 / 100
Language: en

     he's supposed to send a message out to everybody in a couple of days
     to let everybody know that he's feeling better

---------------------------------------------------------------------- 100 / 100
Language: fr

     les mauvaises pratiques en mati re de pr ts la mauvaise gestion des
     entreprises financi res et le manque de surveillance du gouvernement f
     d ral depuis des d cennies ne sont pas r solus aujourd'hui

The question, here, is whether language is properly detected or not (both precision and recall are important to us here). Hand-coding these sentences gives a direct precision-recall answer to this. See the paper for details.

3 Evaluate full cluster filtering

Either run this cell to generate a new selection...


In [8]:
#with session_scope() as session:
#    cluster_ids = sample([id for (id,)
#                          in session.query(Cluster.id).filter(Cluster.filtered == False)],
#                         SAMPLE_SIZE)

... or this one to use the previous selection (which has already been coded).


In [9]:
cluster_ids = [678375, 1030236, 200131, 227946, 182411, 1782509, 2571784, 2385863, 548464,
               1417058, 442671, 417270, 206587, 2119286, 2509349, 1363677, 2384075, 98089,
               2472784, 2291625, 1252250, 766764, 2005041, 1786498, 2532146, 547696, 119025,
               99671, 2293637, 37913, 198342, 1824441, 393254, 1438581, 726763, 1162402, 136659,
               700003, 1530925, 868029, 1988243, 1528193, 453911, 1118515, 942719, 163403,
               37335, 2531211, 1207286, 139905, 2098233, 985830, 1309232, 1949178, 1208225,
               163161, 2390394, 1000682, 33330, 1438761, 143158, 914871, 1012869, 1515844,
               2155238, 318470, 1705922, 1998918, 1559891, 1567974, 1072475, 1377888, 2659498,
               214657, 2307338, 2702297, 1718647, 1830269, 1075089, 563301, 2675991, 1936421,
               650209, 1000108, 270310, 224933, 1078355, 2665218, 988251, 921771, 193937,
               1005385, 922307, 966693, 1468503, 1112509, 476413, 1537564, 799385, 1344350]

In [10]:
with session_scope() as session:
    clusters = [session.query(Cluster).get(id) for id in cluster_ids]
    strings_kepts = []
    for c in clusters:
        fcluster = c.filter()
        if fcluster is not None:
            kept_quote_ids = set([q.id - filter_quote_offset() for q in fcluster.quotes])
        else:
            kept_quote_ids = set([])
        strings_kepts.append(([(q.string, q.id in kept_quote_ids) for q in c.quotes],
                              c.filter() is not None))

In [11]:
print("Over a sample of {}, {} clusters are rejected by the cluster filter"
      .format(SAMPLE_SIZE, SAMPLE_SIZE - np.sum([kept for _, kept in strings_kepts])))


Over a sample of 100, 34 clusters are rejected by the cluster filter

Here are the individual cluster strings and their respective rejected/kept status.


In [12]:
for i, (strings, ckept) in enumerate(strings_kepts):
    title = ' {} / {}'.format(i + 1, SAMPLE_SIZE)
    print('-' * (80 - len(title)) + title)
    print('Kept:',  'yes' if ckept else 'no')
    print()
    for string, skept in strings:
        fstring = indent(fill(string), ' ' * 5)
        if not skept:
            fstring = fstring[0] + 'x' + fstring[2:]
        print(fstring)
        print()


------------------------------------------------------------------------ 1 / 100
Kept: no

 x   it was actually the first time that i cried since the whole incident
     started

 x   the enormity of the situation

 x   i thought i was in the end zone

 x   i thought i was in the clear

 x   and it was the first time that i thought i was in the mafia

 x   just last night actually for the first time i saw some of the
     reenactments about how the airplane actually had to land on the water
     with the nose high and i think for the first time kind of the enormity
     of the situation really hit me and it was actually the first time that
     i cried since the whole incident started

------------------------------------------------------------------------ 2 / 100
Kept: yes

 x   everyone is entitled to an informed opinion

     everyone is entitled to an opinion and so is obama and his staff then
     again you know what they say about opinions

     you know what they say about it

     everyone is entitled to an opinion and so is obama and his staff then
     again you know what they say about opinions unfortunately for them to
     make such a comment about my daughter was a big mistake look at
     angelina jolie and the wonderful things she has done in her life and
     now watch how lindsay does the same

     unfortunately for them to make such a comment about my daughter was a
     big mistake look at angelina jolie and the wonderful things she has
     done in her life and now watch how lindsay does the same

------------------------------------------------------------------------ 3 / 100
Kept: no

 x   p2p bill of rights

 x   p2p bill of rights and responsibilities

 x   bill of rights and responsibilities

------------------------------------------------------------------------ 4 / 100
Kept: no

 x   os melhores do mundo

 x   melhores do mundo

 x   o que a gente fez hoje foi unir uma torcida maravilhosa um espet culo
     incr vel com os melhores pilotos do mundo e uma causa que um momento
     triste para santa catarina mas que a gente conseguiu fazer muita coisa
     para essas pessoas que perderam muito

 x   um momento triste

------------------------------------------------------------------------ 5 / 100
Kept: yes

 x   hillary for president

 x   hillary clinton for president

 x   clinton for president

     support of hillary clinton for president debt relief

     a final evening in support of hillary clinton for president debt
     relief

------------------------------------------------------------------------ 6 / 100
Kept: yes

     i don't think that president bush will be presumptuous in tying to
     talk to barack obama about how he makes his decisions or how barack
     obama should make decisions

     i don't think that president bush will be presumptuous in trying to
     talk to barack obama about how he makes decisions or how barack obama
     should make decisions

     i don't think that president bush will be presumptuous in trying to
     talk to barack obama about how he makes decisions or how barack obama
     should make decisions i think that the president probably will share
     how he's made decisions and some of the things that he feels are
     important

------------------------------------------------------------------------ 7 / 100
Kept: no

 x   when i read

 x   i like to be in the environment

 x   as a child studying history and looking at documents i wondered why
     didn't someone stand up and try to stop it when i read the script it
     was entertaining and informative to know what the challenges were and
     what it was like to be in the environment

------------------------------------------------------------------------ 8 / 100
Kept: no

 x   a o nuevo

 x   feliz a o

 x   feliz a o nuevo

 x   la filarm nica de viena y yo les deseamos feliz a o nuevo

------------------------------------------------------------------------ 9 / 100
Kept: yes

     i think we're sort of blessed in that we are all pretty good friends

 x   we are all friends

 x   we are all good friends

----------------------------------------------------------------------- 10 / 100
Kept: yes

 x   a lifelong republican

     john mccain in 2000 as a lifelong republican i worked to get you
     elected instead of george w bush in return you wrote an endorsement of
     one of my books about military service you seemed to be a man who put
     principle ahead of mere political gain

     lifelong republican who worked to get mccain elected instead of george
     w bush

----------------------------------------------------------------------- 11 / 100
Kept: no

 x   from hell's heart i stab at thee for hate's sake i spit my last breath
     at thee

 x   from hell's heart i stab at thee

 x   to the last i will grapple with thee from hell's heart i stab at thee
     for hate's sake i spit my last breath at thee

 x   for hate's sake i spit my last breath at thee

----------------------------------------------------------------------- 12 / 100
Kept: yes

     this is proof of the microscope's potential we are now ready to start
     doing scientific experiments that will add a new dimension to
     measurements being made by other phoenix lander instruments

     this is proof of the microscope's potential

     we are now ready to start doing scientific experiments that will add a
     new dimension to measurements being made by other phoenix lander
     instruments

----------------------------------------------------------------------- 13 / 100
Kept: no

 x   and the two shall become one flesh

 x   the two become one

 x   so then they are no longer two but one flesh therefore what god has
     joined together let not man separate

 x   for this reason a man will leave his father and mother and be united
     to his wife and they will become one flesh

 x   become one flesh

 x   that at the beginning the creator made them male and female

 x   the two shall become one flesh

 x   that at the beginning the creator made them male and female and said
     for this reason a man will leave his father and mother and be united
     to his wife and the two will become one flesh

 x   let no man put asunder

 x   that at the beginning the creator made them male and female and said
     for this reason a man will leave his father and mother and be united
     to his wife and the two will become one flesh so they are no longer
     two but one therefore what god has joined together let man not
     separate

 x   two become one

 x   for this reason a man will leave his father and mother and be united
     to his wife and the two will become one flesh

 x   and the two shall become one

 x   for this reason a man shall leave his father and mother and be joined
     to his wife and the two shall become one flesh

 x   therefore a man shall leave his father and mother and be joined to his
     wife and they shall become one flesh

 x   the two will become one flesh

 x   so they are no longer two but one flesh

 x   made them male and female

 x   therefore what god has joined together let man not separate

 x   what god has joined together let man not separate

 x   what therefore god has joined together let no man separate

 x   what therefore god has joined together let not man separate

 x   so they are no longer two but one therefore what god has joined
     together let man not separate

 x   god made them male and female

 x   that at the beginning the creator made them male and female 5 and said
     for this reason a man will leave his father and mother and be united
     to his wife and the two will become one flesh 6 so they are no longer
     two but one therefore what god has joined together let man not
     separate

 x   what god has joined together let no man put asunder

 x   what god has joined together let not man put asunder

 x   but at the beginning of creation god made them male and female for
     this reason a man will leave his father and mother and be united to
     his wife and the two will become one flesh so they are no longer two
     but one therefore what god has joined together let man not separate

----------------------------------------------------------------------- 14 / 100
Kept: yes

 x   king and queen

 x   queen of england

 x   but not too much

     americans are people who love to indulge and deep in our hearts want
     our leaders to be like the king and queen of england but not too much

     americans are people who love to indulge and deep in our hearts want
     our leaders to be like the king and queen of england-but not too much

----------------------------------------------------------------------- 15 / 100
Kept: yes

     however lots of mucus and inflammation can cause congestion especially
     in little noses cough and cold medicines and decongestants are
     dangerous and neither effective nor safe for young children
     medications to dry up nasal passages also have problems

     cough and cold medicines and decongestants are dangerous and neither
     effective nor safe for young children medications to dry up nasal
     passages also have problems

     medications to dry up nasal passages also have problems

----------------------------------------------------------------------- 16 / 100
Kept: yes

     whereas wall street by various calculations has to date lost within
     the financial sector 1 1 5 trillion the reality is that at today's
     rate we are losing natural capital at least between 2 5 trillion every
     year

 x   1 1 5

 x   1 5 trillion

     so whereas wall street by various calculations has to date lost within
     the financial sector 1 1 5 trillion the reality is that at today's
     rate we are losing natural capital at least between 2 5 trillion every
     year

----------------------------------------------------------------------- 17 / 100
Kept: yes

 x   as i have always said

     it does give us an opportunity with our odi side as i have always said
     our one-day side is still a work in progress it is still a young side
     and we are giving an opportunity to some younger guys to stake a claim

     it does give us an opportunity with our odi side

----------------------------------------------------------------------- 18 / 100
Kept: yes

     for someone to discuss retirement and then to come back and change his
     mind a day later i'm worried a lot about it i'm worried about where
     he's at mentally because these things happen sometimes and they come
     back up again because maybe they come back and maybe it isn't
     necessarily for the right reasons so were going to continue to
     communicate with him and push him out there on the field and get him
     better

     for someone to discuss retirement and then to come back and change his
     mind a day later i'm worried a lot about it

 x   sometimes they come back

----------------------------------------------------------------------- 19 / 100
Kept: no

 x   does not torture

 x   the united states congress does not

 x   i was clear throughout this campaign and was clear throughout this
     transition that under my administration the united states does not
     torture we will abide by the geneva conventions we will uphold our
     highest ideals

 x   under my administration the united states does not torture

 x   i was clear throughout this campaign and have been clear throughout
     this transition that under my administration the united states does
     not torture

 x   we will abide by the geneva conventions and we will uphold our highest
     values and ideals

 x   i was clear throughout this campaign and have been clear throughout
     this transition that under my administration the united states does
     not torture we will abide by the geneva conventions that we will
     uphold our highest values and ideals

 x   under my administration the united states does not torture we will
     abide by the geneva conventions we will uphold our highest ideals

 x   i was clear throughout this campaign and was clear throughout this
     transition that under my administration the united states does not
     torture

 x   we will abide by the geneva conventions we will uphold our highest
     ideals

 x   under my administration the united states does not torture we will
     abide by the geneva conventions we will uphold our highest values and
     ideals

----------------------------------------------------------------------- 20 / 100
Kept: no

 x   and that would be a good thing right

 x   if he was inviting the rev warren to participate in a forum and to
     make a speech that would be a good thing

 x   if he was inviting the reverend warren to participate in a forum and
     to make a speech that would be a good thing

----------------------------------------------------------------------- 21 / 100
Kept: no

 x   de tensi n

 x   algunos aspectos de la actual situaci n en estados unidos se parecen a
     los anteriores episodios de tensi n financiera relacionados con el
     sector financiero que estuvieron seguidos por recesiones

 x   algunos aspectos de la actual situaci n en estados unidos se parecen a
     los anteriores episodios de tensi n en el sector financiero que
     estuvieron seguidos por recesiones

----------------------------------------------------------------------- 22 / 100
Kept: yes

     ready to tax ready to spend not ready to lead

     ready to tax ready to spend but not ready to lead

     they're ready to tax ready to spend but not ready to lead

----------------------------------------------------------------------- 23 / 100
Kept: yes

     the number must be significantly more in total

     i have seen myself three or four injured britons and most of those
     have with them a partner of some sort who they are with that's in one
     hospital i think the number must be significantly more in total than
     that but i don't have an absolute figure at the moment

 x   must be significantly more

----------------------------------------------------------------------- 24 / 100
Kept: yes

 x   like a boy

 x   it's never come up

     it's never really come up as an issue

     we're close we've been through a lot they're supportive of me whether
     i'm with a guy or a girl they're just supportive of me as a person

     ali's very mature i've told her that it's okay to like a boy or a girl
     if you're comfortable with it and it's something you believe you want
     to do and i told her not to be afraid of who she wants to be

     ali's known samantha for a really long time and she's like whatever it
     is i support you i probably won't ever do what it is you're doing but
     i'm happy for you ali's very mature i've told her that it's okay to
     like a boy or a girl if you're comfortable with it and it's something
     you believe you want to do and i told her not to be afraid of who she
     wants to be

     ali's known samantha for a really long time

     ali's very mature i've told her that it's ok to like a boy or a girl
     if you're comfortable with it and it's something you believe you want
     to do and i told her not to be afraid of who she wants to be

     it's never really come up as an issue we're close we've been through a
     lot they're supportive of me whether i'm with a guy or a girl they're
     just supportive of me as a person ali's known samantha for a really
     long time and she's like whatever it is i support you i probably won't
     ever do what it is you're doing but i'm happy for you ali's very
     mature i've told her that it's okay to like a boy or a girl if you're
     comfortable with it and it's something you believe you want to do and
     i told her not to be afraid of who she wants to be

     ali has known samantha for a really long time

     it's never really come up as an issue we're close we've been through a
     lot they're supportive of me whether i'm with a guy or a girl they're
     just supportive of me as a person

----------------------------------------------------------------------- 25 / 100
Kept: yes

     witnessed the most horrific war injuries in men women and children of
     all ages in numbers almost too large to comprehend

     witnessed the most horrific war injuries in men women and children of
     all ages in numbers almost too large to comprehend the wounded dying
     and dead have streamed into the overcrowded hospital in endless
     convoys of ambulances and private cars and wrapped in blankets in the
     caring arms of others the endless and intense bombardments from
     israeli air ground and naval forces have missed no targets not even
     the hospital

     the most horrific war injuries in men women and children of all ages
     in numbers too large to comprehend the wounded dying and dead have
     streamed into the overcrowded hospital in endless convoys of
     ambulances and private cars and wrapped in blankets in the caring arms
     of others the endless and intense bombardment from israeli air ground
     and naval forces have missed no targets not even hospitals

----------------------------------------------------------------------- 26 / 100
Kept: yes

     it's not for everyone it's not all doctors who will go to the private
     side that's for sure and it's not feasible everywhere

 x   it's not for everyone

 x   not for everyone

----------------------------------------------------------------------- 27 / 100
Kept: yes

     yeah thanks wow you really know your stuff

 x   know your stuff

 x   you know your stuff

----------------------------------------------------------------------- 28 / 100
Kept: no

 x   a confesi n de parte relevo de prueba

 x   a confesi n de parte relevo de pruebas

 x   a confesi n de partes relevo de pruebas

----------------------------------------------------------------------- 29 / 100
Kept: yes

 x   why would you do that

 x   why would you

 x   now why would you do that

 x   would you do that

     people say why would you do that

----------------------------------------------------------------------- 30 / 100
Kept: yes

 x   most innovative company

 x   creative and innovative

     we're delighted with the extension and expansion of take-two's
     relationship with the rockstar team the talent and creativity at
     rockstar are unparalleled in our industry their dedication to making
     extraordinary games has captured the enthusiasm of millions of fans
     around the world and has contributed enormously to establishing take-
     two as a force in the global video game marketplace and advancing our
     goal to become the most creative and most innovative company in the
     industry in addition this new chapter in our successful relationship
     further aligns the interests of our creative teams with those of our
     shareholders

     we're delighted with the extension and expansion of take-two's
     relationship with the rockstar team

     the talent and creativity at rockstar are unparalleled in our industry

----------------------------------------------------------------------- 31 / 100
Kept: no

 x   generally recognized as safe

 x   culinary herbs and spices are all generally recognized as safe and
     have been time-tested in the diet

 x   generally recognized as safe and effective

----------------------------------------------------------------------- 32 / 100
Kept: yes

 x   for no particular reason

 x   no particular reason

     for no particular reason activated the submarine's anti-fire system

----------------------------------------------------------------------- 33 / 100
Kept: yes

 x   ma ma where's my pa

 x   ma ma where's my pa gone to the white house ha ha ha

     gone to the white house ha ha ha

----------------------------------------------------------------------- 34 / 100
Kept: yes

     most of these taliban killed are foreign fighters who entered
     afghanistan to destabilize the country

     most of these taliban killed are foreign fighters who entered
     afghanistan to destabilise the country

 x   destabilize the country

----------------------------------------------------------------------- 35 / 100
Kept: yes

 x   storm of the century

 x   of the century

     you need to be scared and you need to get your butts out of new
     orleans right now

     get your butts out of new orleans now

 x   the storm of the century

 x   you need to be scared

     you need to be concerned and you need to get your butts moving out of
     new orleans right now this is the storm of the century

     you need to be concerned and you need to get your butts moving and out
     of new orleans right now

     you need to be scared you need to be concerned and you need to get
     your butts moving out of new orleans right now this is the storm of
     the century

     this is the storm of the century

 x   need to be scared

----------------------------------------------------------------------- 36 / 100
Kept: yes

 x   you look younger

 x   it doesn't do anything

     i really think we should stop asking women their ages it doesn't do
     anything for us we live in a society that judges age if you can look
     younger and get away with it do

----------------------------------------------------------------------- 37 / 100
Kept: no

 x   i am the resurrection and the life he who believes in me will live
     even though he dies and whoever lives and believes in me will never
     die

 x   do you believe this

 x   i am the resurrection and the life he who believes in me will live
     even though he dies and whoever lives and believes in me will never
     die do you believe this

 x   i am the resurrection and the life he who believes in me though he may
     die he shall live and whoever lives and believes in me shall never die

 x   i am the resurrection and the life those who believe in me even though
     they die will live and everyone who lives and believes in me will
     never die

 x   i am the resurrection and the life he who believes in me though he may
     die he shall live

 x   whoever lives and believes in me will never die

 x   i am the resurrection and the life he who believes in me will live
     even though he dies 26 and whoever lives and believes in me will never
     die do you believe this

 x   i am the resurrection and the life he who believes in me will live
     even though he dies

 x   he who believes in me

 x   i am the resurrection and the life he who believes in me though he die
     yet shall he live and whoever lives and believes in me shall never die

 x   i am the resurrection and the life whoever believes in me even if he
     dies will live and everyone who lives and believes in me will never
     die do you believe this

 x   i am the resurrection and the life he who believes in me will live
     even if he dies

----------------------------------------------------------------------- 38 / 100
Kept: yes

     america we are better than these eight years we are a better country
     than this

 x   america we are a better country than this

     but the failure to respond is a direct result of a broken politics in
     washington and the failed policies of george w bush america we are
     better than these last eight years we are a better country than this

 x   better than this

 x   america we are better than these last eight years we are a better
     country than this

     better than these last eight years

 x   we are a better country than this

     we are better than these last eight years we are a better country than
     this

 x   america we are better than these last eight years

     america we are better than the last eight years we are a better
     country than this

     america we are better than these eight years

 x   we are better than these last eight years

     these challenges are not all of government's making but the failure to
     respond is a direct result of a broken politics in washington and the
     failed policies of george w bush america we are better than these last
     eight years we are a better country than this

     we are a better country

     we are better than these last 8 years

     these challenges are not all of government's making but the failure to
     respond is a direct result of a broken politics in washington and the
     failed presidency of george w bush america we are better than these
     last eight years we are a better country than this

 x   a better country

 x   we are better

----------------------------------------------------------------------- 39 / 100
Kept: no

 x   he was not above the law

 x   believed he was above the law

 x   i think he thought he was above the law for the longest time

----------------------------------------------------------------------- 40 / 100
Kept: yes

     it's a very competitive tri-nations series this year

     it's a very competitive tri-nations series this year every match has
     been more important than the one before it and this test decides the
     championship

     every match has been more important than the one before it and this
     test decides the championship

----------------------------------------------------------------------- 41 / 100
Kept: yes

 x   recharge your batteries

     at pixel qi we have a new series of inventions that go well beyond the
     olpc screen that we are developing right now

     we can enable an increase of 5-10x battery life between charges
     compared with a standard notebook this means that rather than needing
     to recharge your batteries every few hours you could run 20-40 hours
     of use on a one charge

     at pixel qi we have a new series of inventions that go well beyond the
     olpc screen that we are developing right now we can enable an increase
     of 5-10x battery life between charges compared with a standard
     notebook this means that rather than needing to recharge your
     batteries every few hours you could run 20-40 hours of use on a one
     charge

----------------------------------------------------------------------- 42 / 100
Kept: no

 x   von den aussagen von michael ballack bin ich total berrascht dass er
     nun ber die medien solch kritischen t ne u ert verwundert und entt
     uscht mich

 x   verwundert und entt uscht mich

 x   verwundert und entt uscht

----------------------------------------------------------------------- 43 / 100
Kept: no

 x   je lui ai exprim mon souhait de me rendre au tibet pour voir moi-m me
     ce qui se passe il a trouv que c tait une bonne id e

 x   elle-m me ce qui se passe

 x   pour voir moi-m me ce qui s'y passe

----------------------------------------------------------------------- 44 / 100
Kept: no

 x   it's a safe banking system a sound banking system our regulators are
     on top of it this is a very manageable situation

 x   i think it's going to be months that we're working our way through
     this period clearly months but again it's a safe banking system a
     sound banking system our regulators are on top of it this is a very
     manageable situation

 x   is a very manageable situation

 x   sound banking system

 x   this is a very serious situation

 x   this is a very challenging situation

 x   this is a very tragic situation

 x   they are on top of it

 x   this is a very bad situation

----------------------------------------------------------------------- 45 / 100
Kept: yes

 x   when and how

     when and how are you going to start fighting back

     when and how are you going to start fighting back against attack ads
     and the

     when and how are you going to start fighting back against attack ads
     and the smear campaigns

----------------------------------------------------------------------- 46 / 100
Kept: yes

 x   beware of the dog

 x   beware of dog

 x   beware the dog

     beware of the dog who thinks

----------------------------------------------------------------------- 47 / 100
Kept: yes

 x   you can't have it both ways

 x   you can't have it

 x   they can't have it both ways

 x   can't have it both ways

     either the children are out of bounds and you don't put them in the
     photo ops or you complain when somebody wants to talk about them you
     can't have it both ways

     either the children are out of bounds and you don't put them in the
     photo ops or you don't complain when somebody wants to talk about them
     you can't have it both ways

 x   wants to talk

----------------------------------------------------------------------- 48 / 100
Kept: yes

     we've fired four workers for showing up drunk and two of them were
     drivers

     how can we allow a drunk to work who might run somebody over

     we've fired four workers for showing up drunk and two of them were
     drivers how can we allow a drunk to work who might run somebody over

----------------------------------------------------------------------- 49 / 100
Kept: yes

     remember canada is not the united states the fundamentals of the
     canadian economy are sound

     canada is not the united states

     remember canada is not the united states

----------------------------------------------------------------------- 50 / 100
Kept: no

 x   how many divisions has the pope

 x   how many divisions does the pope have

 x   how many divisions

----------------------------------------------------------------------- 51 / 100
Kept: yes

     the silly songs the spectacle the grandiose foolishness of it all

     let's hope that between norton and the musical brilliance of lloyd
     webber royaume uni will gather its proper share of douze points and
     bring the grand prix home to where it belongs

     the silly songs the spectacle the grandiose foolishness of it all
     let's hope that between norton and the musical brilliance of lloyd
     webber royaume uni will gather its proper share of douze points and
     bring the grand prix home to where it belongs

----------------------------------------------------------------------- 52 / 100
Kept: no

 x   jesus is the lord

 x   wherefore i give you to understand that no man speaking by the spirit
     of god calleth jesus accursed and that no man can say that jesus is
     the lord but by the holy ghost

 x   no man can say that jesus is the lord but by the holy ghost

----------------------------------------------------------------------- 53 / 100
Kept: yes

 x   you can't say that

 x   now i don't know

 x   i don't what did you say

     if you can't say that we're oversold now i don't know what you say
     you're at least due for a bounce if nothing else

----------------------------------------------------------------------- 54 / 100
Kept: yes

 x   from the wilderness

     a renewed sense of optimism

     i think state dinners are going to be big national social events with
     the obamas there's clearly a renewed sense of optimism and style just
     a sense of joy that we're back from the wilderness

----------------------------------------------------------------------- 55 / 100
Kept: yes

     this is a huge cow patty with a piece of marshmallow stuck in the
     middle and i'm not going to eat that cow patty

     i'm not going to eat that cow patty

     this is a huge cow patty with a piece of marshmallow stuck in the
     middle of it and i am not going to eat that cow patty

     madam speaker this is a huge cow patty with a piece of marshmallow
     stuck in the middle of it

     a huge cow patty with a piece of marshmallow stuck in the middle i'm
     not going to eat that cow patty

     madam speaker this is a huge cow patty with a piece of marshmallow
     stuck in the middle of it and i'm not going to eat that cow patty

----------------------------------------------------------------------- 56 / 100
Kept: no

 x   al qaeda warrior

 x   he is an al qaeda warrior

 x   he is an al qaeda man but was not among the top hierarchy he was
     involved in carrying out ied blasts in afghanistan

----------------------------------------------------------------------- 57 / 100
Kept: yes

 x   it didn't occur to me

 x   it didn't even occur to me

 x   oh you know

 x   a big issue

     i kind of thought to myself oh you know maybe they're going to say
     something it didn't occur to me that they were going to make it such a
     big issue

----------------------------------------------------------------------- 58 / 100
Kept: no

 x   heads i win tails i don't lose much

 x   heads i win tails i win

 x   heads i win tails i don't lose too much

----------------------------------------------------------------------- 59 / 100
Kept: no

 x   duped by the us on a colossal scale

 x   today's time story further demonstrates that the british government
     has allowed itself to be duped by the us on a colossal scale

 x   has allowed itself to be influenced by ngo's

----------------------------------------------------------------------- 60 / 100
Kept: yes

     there have been a number of stories in the newspapers saying he has
     said this but every time we speak to him he says there is no truth in
     him looking for a move

     there have been a number of casualties

 x   a number of stories

----------------------------------------------------------------------- 61 / 100
Kept: yes

     opportunity to consolidate behind a single government

 x   journey to the east

     illinois senator barack obama's journey to the middle east took him to
     the west bank thursday for a meeting with the man elected to replace
     yasser arafat for a time thursday in the west bank there was only the
     clatter of cameras as the newly elected president of the palestinian
     authority mahmoud abbas met with illinois senator barack obama at a
     meeting with palestinian students thursday obama said the u s will
     never recognize winning hamas candidates unless the group renounces
     its fundamental mission to eliminate israel and obama told abc7 he
     delivered that message to the palestinian president part of the
     opportunity here with this upcoming election is to consolidate behind
     a single government with a single authority that can then negotiate as
     a reliable partner with israel said obama

----------------------------------------------------------------------- 62 / 100
Kept: yes

 x   flawed moral reasoning

     meet the press has become a national window on the flawed moral
     reasoning of some catholic public servants

     national window on the flawed moral reasoning of some catholic public
     servants

----------------------------------------------------------------------- 63 / 100
Kept: yes

     has a toned-down version of the porn actress look favored by this
     decade's woman the over-treated hair puffy lips and permanently
     alarmed expression

     palin has a toned-down version of the porn actress look favoured by
     this decade's woman the overtreated hair puffy lips and permanently
     alarmed expression

     toned-down version of the porn actress look

     has a toned-down version of the porn actress look favored by this
     decade's woman the overtreated hair puffy lips and permanently alarmed
     expression

     a toned-down version of the porn actress look

     palin has a toned-down version of the porn actress look favored by
     this decade's woman the overtreated hair puffy lips and permanently
     alarmed expression

----------------------------------------------------------------------- 64 / 100
Kept: yes

     i'm supporting her in all the ways that i can

 x   a good friend

 x   a very good

 x   very good friend

 x   just for her

 x   she's very good

     she's a very good friend i'm supporting her in all the ways that i can
     i'm just there for her i speak to her a lot

     she's a very good friend i'm supporting her in all the ways that i can
     i'm just there for her

     you know she's a dear friend and i'm supporting her in all the ways
     that i can

 x   i'm just here for her

     i'm just here on the other end of the phone really

     i speak to her a lot

     she's a very good friend

     i'm supporting her in all the ways that i can i'm just there for her i
     speak to her a lot

     i'm just here for her i'm just here on the other end of the phone
     really i speak to her a lot

     you know she's a dear friend and i'm supporting her in all the ways
     that i can i'm just here for her i'm just here on the other end of the
     phone really i speak to her a lot

     she's a dear friend and i'm supporting her in all the ways that i can

     she's a very good friend i'm supporting her in all the ways that i can
     i'm just there

     she's a very good friend i'm supporting her in all the ways that i can

     i'm just there for her i speak to her a lot

 x   a very good friend

 x   there for her

----------------------------------------------------------------------- 65 / 100
Kept: yes

 x   i miss her so much

     she's just such a lovely wife and mother who always loves me and our
     babies i miss her so much

     she's just such a lovely wife and mother who always loves me and the
     babies i just miss her so much

     she loves me and babies i just miss her so much

----------------------------------------------------------------------- 66 / 100
Kept: no

 x   back at ya

 x   right back at ya

 x   kirby right back at ya

----------------------------------------------------------------------- 67 / 100
Kept: no

 x   it's been a long journey

 x   it's been a long long journey so long that a whole book was written
     about a part of it it's obvious to anyone who is familiar with the
     story and the history that it is one of the project that to make an
     understatement did not turn out the way it was originally planned

 x   long long journey

 x   my friends it's been a long long journey

----------------------------------------------------------------------- 68 / 100
Kept: no

 x   is not going to change

 x   balance of power

 x   is not going to change the dynamic quickly

 x   a few russian ships is not going to change the balance of power

 x   a few russian ships are not going to change the balance of power

----------------------------------------------------------------------- 69 / 100
Kept: yes

 x   closely and creatively

     the fdic is working closely and creatively with treasury to realize
     the potential benefits of this authority

     realize the potential benefits of this authority

----------------------------------------------------------------------- 70 / 100
Kept: no

 x   to make it in this industry you gotta have f ability

 x   to make it in this industry you gotta have f ability to make it in
     this game women have to be f able

 x   to make it in this industry you gotta have f kability to make it in
     this game women have to be f kable

----------------------------------------------------------------------- 71 / 100
Kept: yes

     we are there for the long-term

     but i am also a long-term investor and that is probably more important
     to the club and to you because it means we are here for the long haul
     and that we will act always in the best interests of the club and all
     of its stakeholders but especially you the fans

     we are there for the long haul

 x   all of its stakeholders

----------------------------------------------------------------------- 72 / 100
Kept: no

 x   it is pretty cool

 x   it is a pretty cool thing

 x   to have the first fan be a white sox fan is a pretty cool thing

----------------------------------------------------------------------- 73 / 100
Kept: no

 x   we were expecting these conditions to help us so i am surprised that
     we have ended up so far back in qualifying today the biggest problem
     seems to be that we can't get the car and tyres to work in the wet
     conditions with very low downforce the first set of tyres that we ran
     seemed to be working okay and i was running just outside the top ten
     however on the new set that we used for my last three quick laps the
     car was just undriveable it had no balance and i couldn't find any
     grip so every time i hit the brakes the front and rear wheels would
     lock we'll have to have a look at the data and find out why the car
     was so difficult to drive but it's obviously very disappointing to be
     starting the race from so far back on the grid

 x   it's obviously very disappointing

 x   then we'll have to have a look at that individually

----------------------------------------------------------------------- 74 / 100
Kept: yes

 x   and human rights

     i have spoken clearly candidly and consistently with china's leaders
     about our deep concerns over religious freedom and human rights

     deep concerns over religious freedom and human rights

----------------------------------------------------------------------- 75 / 100
Kept: yes

     this division does not help our effort to get an agreement from the
     alliance of motion picture amp television producers that our members
     will ratify allen said

     this division does not help our effort to get an agreement that our
     members will ratify

     this division does not help our effort to get an agreement from the
     amptp that our members will ratify

----------------------------------------------------------------------- 76 / 100
Kept: yes

 x   not getting enough

     the ordinary people here in gaza are not getting enough help and are
     not getting it quickly enough

 x   not getting it

----------------------------------------------------------------------- 77 / 100
Kept: no

 x   la mejor campa a

 x   todos los que hicieron posible la mejor campa a de la historia de
     estados unidos muy especialmente dedic este triunfo a su mujer
     michelle y a sus dos hijas quienes le acompa aron en este momento tan
     especial

 x   todos los que hicieron posible la mejor campa a de la historia de
     estados unidos

----------------------------------------------------------------------- 78 / 100
Kept: yes

 x   unique value proposition

     we have seen an average of 4 5 million unique users per month visiting
     cashback who have conducted more than 68 million commercial queries we
     believe this early traction speaks to the differentiated and unique
     value proposition of microsoft live search cashback for both consumers
     and advertisers especially in these tough economic times

     we believe this early traction speaks to the differentiated and unique
     value proposition of microsoft live search cashback for both consumers
     and advertisers especially in these tough economic times

     we have seen an average of 4 5 million unique users per month visiting
     cashback who have conducted more than 68 million commercial queries

----------------------------------------------------------------------- 79 / 100
Kept: yes

     it's kind of hard to tell the average american that we are going to
     continue to have foreclosures that destabilize neighborhoods and
     deprive cities of revenue they need but we're going to buy up the bad
     paper

     it's kind of hard to tell the average american that we're going to
     continue to have foreclosures that destabilize neighborhoods and
     deprive cities of revenues they need but we're going to buy up the bad
     paper

     it's hard to tell the average american that we're going to continue to
     have foreclosures that destabilize neighborhoods and deprive cities of
     revenues they need but we're going to buy up the bad paper

----------------------------------------------------------------------- 80 / 100
Kept: yes

     the baader-meinhof complex der baader-meinhof komplex

 x   der baader-meinhof komplex

 x   the baader-meinhof complex

----------------------------------------------------------------------- 81 / 100
Kept: no

 x   there are 25 charred bodies they got caught in the fire

 x   other side of the street

 x   there are 25 charred bodies they got caught in the fire and tried to
     escape to the other side of the street but there was no way out

----------------------------------------------------------------------- 82 / 100
Kept: yes

 x   read them all

 x   i read them all

     of course i read them all

----------------------------------------------------------------------- 83 / 100
Kept: no

 x   mountains of the moon

 x   over the mountains

 x   over the mountains of the moon down the valley of the shadow ride
     boldly ride

----------------------------------------------------------------------- 84 / 100
Kept: yes

     well understood and beyond dispute

     the physics and chemistry of the greenhouse-gas effect are well
     understood and beyond dispute science has also achieved an
     overwhelming consensus that the increase in greenhouse gases is
     largely of human origin tracing back to the industrial revolution and
     accelerating in recent years as carbon dioxide and methane the
     products of fossil fuel use have entered the atmosphere in increasing
     quantities modeling the climate has proven to be a complex scientific
     task but although the models are far from perfect many of their
     predictions are so alarming that conservative risk-averse policymaking
     requires that they be considered with extraordinary gravity

     science has also achieved an overwhelming consensus that the increase
     in greenhouse gases is largely of human origin tracing back to the
     industrial revolution and accelerating in recent years as carbon
     dioxide and methane the products of fossil fuel use have entered the
     atmosphere in increasing quantities modeling the climate has proven to
     be a complex scientific task but although the models are far from
     perfect many of their predictions are so alarming that conservative
     risk-averse policymaking requires that they be considered with
     extraordinary gravity

----------------------------------------------------------------------- 85 / 100
Kept: yes

 x   i think that's wonderful

     i think that's a wonderful thing and i think the american uniform is
     going to play a very important role in helping this country choose the
     right path at a very important crossroads in our history

 x   choose the right path

----------------------------------------------------------------------- 86 / 100
Kept: yes

 x   oh you're one of those people

 x   oh you're one of those

     i've kept out of doing this kind of interview you have to be quite
     careful unless you're one of those people who's happy to give
     everything of themselves and i for one am not

 x   you're one of those

----------------------------------------------------------------------- 87 / 100
Kept: yes

 x   war on nature

     bush's holy war on nature

 x   holy war on nature

----------------------------------------------------------------------- 88 / 100
Kept: yes

 x   louis vuitton don

 x   the louis vuitton don

     and until then i will be in the building swagger on a hundred-thousand
     trillion

     i know i've been called the louis vuitton don i know i've been called
     a lot of names uh mostly the don the louis vuitton don due to what
     happened when the red shoes hit the runway i was forced to change my
     name to martin louis the king jr address me as such and until then i
     will be in the building swagger on a hundred thousand trillion

     i know i've been called the louis vuitton don i've been called a lot
     of names due to what happened when the red shoes hit the runway i was
     forced to change my name to martin louis the king jr address me as
     such

----------------------------------------------------------------------- 89 / 100
Kept: no

 x   institucionalidad democr tica

 x   a todos los actores pol ticos y sociales involucrados a que tomen las
     medidas para cesen inmediatamente las acciones de violencia intimidaci
     n y de desacato a la institucionalidad democr tica y al orden jur dico
     establecido

 x   a todos los actores pol ticos y sociales involucrados a que tomen las
     medidas necesarias para que cesen inmediatamente las acciones de
     violencia intimidaci n y de desacato a la institucionalidad democr
     tica y al orden jur dico establecido

 x   a todos los actores pol ticos y sociales involucrados que tomen las
     medidas necesarias para que cesen inmediatamente las acciones de
     violencia

----------------------------------------------------------------------- 90 / 100
Kept: no

 x   vencer serenamente al dolor

 x   dif ciles retos personales

 x   vencer serenamente al dolor a la sensaci n de soledad y a la tentaci n
     de no querer afrontar los nuevos m ltiples y dif ciles retos
     personales y familiares

----------------------------------------------------------------------- 91 / 100
Kept: no

 x   if you have to ask then you can't afford it

 x   if you have to ask you can't afford it

 x   if you have to ask

 x   you have to ask

 x   you can't afford it

----------------------------------------------------------------------- 92 / 100
Kept: yes

     people living with mental illnesses often are among the most
     vulnerable in our society

     unfortunately they also are often overlooked during disasters

     people living with mental illnesses often are among the most
     vulnerable in our society unfortunately they also are often overlooked
     during disasters

----------------------------------------------------------------------- 93 / 100
Kept: yes

     close genuine female relationships are not what generally gets
     depicted in movies and tv shows

     like if you're watching the hills or 90210 all the backstabbing shapes
     the way we act you go to school and you think your job is to find a
     sworn enemy and be jealous of each other

     close genuine 90210 all of the backstabbing shapes the way we act you
     go to school and you think that your job is to find a sworn enemy and
     be jealous of each other

     like if you're watching the hills or 90210 all the backstabbing shapes
     the way we act

     close genuine female relationships are not what generally gets
     depicted in movies and tv shows like if you're watching the hills or
     90210 all the backstabbing shapes the way we act you go to school and
     you think your job is to find a sworn enemy and be jealous of each
     other

----------------------------------------------------------------------- 94 / 100
Kept: yes

     the steps we are announcing today along with significant commitments
     from the private sector are intended to mitigate the potential risks
     and disruptions to markets

     along with significant commitments from the private sector are
     intended to mitigate the potential risks and disruptions to the
     markets

     the steps we are announcing today along with significant commitments
     from the private sector are intended to mitigate the potential risks
     and disruptions to markets he added

     are intended to mitigate the potential risks and disruptions to
     markets

     to mitigate the potential risks and disruptions to markets

 x   mitigate the risks

----------------------------------------------------------------------- 95 / 100
Kept: yes

     it's not as bad as we anticipated

     it's not as bad as was first expected

     it's not as bad as we first thought he went into the board awkward

----------------------------------------------------------------------- 96 / 100
Kept: yes

     he brought a slightly more jazzy and ethereal element to it all

     he had some elusive quality let's call it soul that glued the whole
     thing together you notice it when it's missing

     he brought a slightly more jazzy and ethereal element to it all he had
     some elusive quality let's call it soul that glued the whole thing
     together you notice it when it's missing

----------------------------------------------------------------------- 97 / 100
Kept: yes

 x   you're doing what

 x   i am impressed

     you'll be in our thoughts and prayers i am very impressed by what
     you're doing and what you're prepared to do coming from a state that
     is not often hit by a hurricane i'm incredibly impressed

 x   i am very impressed

----------------------------------------------------------------------- 98 / 100
Kept: yes

     it gives me great pleasure to secure the services of this extremely
     talented batsman for next season

     he will bring star quality to our batting line-up for both
     championship and one-day cricket and his wealth of experience is
     perfect for our younger batsman to learn from and develop around

     it gives me great pleasure to secure the services of this extremely
     talented batsman for next season he will bring star quality to our
     batting line-up for both championship and one-day cricket and his
     wealth of experience is perfect for our younger batsman to learn from
     and develop around

----------------------------------------------------------------------- 99 / 100
Kept: yes

     feed a cold and starve a fever

 x   feed a cold

 x   feed a cold starve a fever

 x   starve a fever

---------------------------------------------------------------------- 100 / 100
Kept: yes

     we will miss his play on both ends of the floor as well as his
     leadership

     we will miss his play on both ends of the floor as well as his
     leadership with the young players i am hopeful others will step
     forward to fill those roles marcus has been a joy to coach and i look
     forward to his return

     we will miss his play on both ends of the floor as well as his
     leadership with the young players

The question, here, is whether the cluster filter lets us keep only real clusters (i.e. not clusters of causally unrelated quotes; in other words, we'd like high precision, and we're not really interested in recall). The codings for these clusters are in the codings/filter_evaluations_clusters-precision-recall file, and precision-recall follows from them.

4 Evaluate the substitution filtering

4.0 Sample clusters and substitutions

Sample clusters

Either run this cell to generate a new selection...


In [13]:
#with session_scope() as session:
#    # Sample 5 times more to make sure we get at least 100 substitutions
#    cluster_ids = sample([id for (id,)
#                          in session.query(Cluster.id).filter(Cluster.filtered == True)],
#                         15 * SAMPLE_SIZE)

... or this one to use the previous selection (which has already been coded).


In [14]:
cluster_ids = [
    1001058714, 1000234230, 1001661015, 1000793763, 1000992927, 1001447160, 1000928909,
    1000652632, 1002429300, 1000424511, 1000028725, 1001804178, 1001962169, 1002010696,
    1002620164, 1000541603, 1002038276, 1002178216, 1001264080, 1000658653, 1002656290,
    1000602035, 1002444841, 1001270720, 1000980555, 1000049133, 1001594109, 1000874654,
    1000270878, 1001267640, 1000587081, 1001502597, 1001290073, 1000742456, 1000803305,
    1001768144, 1001790796, 1000420570, 1002288719, 1002195335, 1000630276, 1001310108,
    1002544437, 1001355443, 1001215220, 1002184159, 1001954566, 1000417270, 1001062143,
    1000193302, 1001073091, 1001318792, 1001333529, 1002230713, 1000253352, 1002165662,
    1001000108, 1001492202, 1002697831, 1002554478, 1002203884, 1000368041, 1000743708,
    1001803629, 1001123734, 1001892371, 1001817722, 1001887535, 1001566025, 1001800790,
    1000778799, 1000917456, 1001772704, 1001500452, 1001176167, 1001580298, 1000786039,
    1000276995, 1001329982, 1002416136, 1001861324, 1000314572, 1000873226, 1002526111,
    1001214205, 1002498111, 1000073947, 1001636514, 1000381995, 1001214611, 1001605320,
    1000100464, 1000853926, 1002047137, 1000532977, 1002113182, 1002222906, 1001773241,
    1000492857, 1000949715, 1000164833, 1000263282, 1000669491, 1000537798, 1000493265,
    1001739413, 1002188621, 1002425058, 1000997844, 1001798690, 1000158229, 1000131343,
    1001936787, 1001056713, 1000069848, 1001032119, 1000492141, 1000001841, 1000062295,
    1001365462, 1001259190, 1002360197, 1000417145, 1001266464, 1000395595, 1002109225,
    1000519609, 1002341113, 1002627477, 1001447816, 1002598858, 1000739902, 1001334186,
    1000421527, 1001369574, 1001468521, 1001221778, 1002397089, 1000194804, 1001834376,
    1000847659, 1000857363, 1001389106, 1000404597, 1002673066, 1002010483, 1002482970,
    1001309374, 1002135445, 1000676392, 1001402554, 1000023387, 1000861742, 1000739666,
    1002208103, 1001610066, 1002091675, 1000055741, 1000098549, 1002606093, 1001019247,
    1002081961, 1000046067, 1002202634, 1002023513, 1001803811, 1002270829, 1001090860,
    1000942827, 1002488258, 1000073692, 1002394081, 1002642836, 1000964777, 1001327690,
    1000005769, 1001231453, 1001843109, 1002095337, 1000144749, 1000856624, 1001408503,
    1002077967, 1000689166, 1002604060, 1002619245, 1000260034, 1002549738, 1000886518,
    1001146812, 1001499168, 1000981840, 1000510458, 1001824441, 1000023650, 1002645020,
    1002476030, 1001445978, 1001729766, 1002396899, 1002704984, 1000456379, 1000164580,
    1001594835, 1001879217, 1001658148, 1000222104, 1000175721, 1001934132, 1000458812,
    1000597134, 1000983803, 1001045067, 1001934934, 1001101372, 1002064679, 1001354812,
    1001387573, 1001552266, 1001049178, 1000211591, 1001535131, 1000068077, 1002667160,
    1001958712, 1002049050, 1001354277, 1001406656, 1001333150, 1001800505, 1001948524,
    1000398683, 1002031029, 1001171493, 1001750650, 1000486399, 1002344374, 1000798536,
    1002483840, 1001327852, 1000574008, 1002405878, 1001603998, 1001658395, 1000311644,
    1001804403, 1001249334, 1001265825, 1001434048, 1001393656, 1001794262, 1001120310,
    1001337829, 1001450332, 1002610877, 1001479081, 1000806304, 1000310768, 1000399139,
    1001749658, 1000062651, 1001672488, 1000168730, 1000287594, 1000031766, 1002453238,
    1000907880, 1000172027, 1002431192, 1001999079, 1000072697, 1001986304, 1002115343,
    1001729577, 1001750064, 1000630283, 1000459150, 1002202737, 1002558133, 1001727072,
    1000130129, 1000657054, 1000701933, 1000953247, 1002349120, 1001881970, 1002607340,
    1002248182, 1001559921, 1001252335, 1000301764, 1001307914, 1001284847, 1001340444,
    1001785513, 1000158707, 1000072111, 1002577972, 1001846670, 1002551562, 1000423962,
    1001392074, 1002514611, 1001067331, 1002031448, 1000179811, 1001451249, 1002350833,
    1000722484, 1000541045, 1001513981, 1002327622, 1001060749, 1002530663, 1002644774,
    1001007144, 1000257361, 1002563963, 1002394972, 1001058481, 1000854528, 1002613959,
    1001676866, 1000073613, 1000209836, 1001243402, 1002440440, 1002364622, 1000090769,
    1002031916, 1001978714, 1002287608, 1001866823, 1001583138, 1002663786, 1002190005,
    1001188041, 1000497933, 1000061665, 1000964817, 1001367239, 1001935938, 1002551108,
    1002637119, 1000525740, 1002690274, 1000762606, 1002218984, 1002540254, 1002012695,
    1002535392, 1002045997, 1001761229, 1001345695, 1001540708, 1000409133, 1002281542,
    1002031988, 1001905217, 1001927225, 1000456501, 1000103149, 1001595739, 1002629715,
    1000633871, 1002582219, 1000125203, 1002622788, 1000152868, 1001691221, 1002550150,
    1001367563, 1001517118, 1000871577, 1002129646, 1001997590, 1000404780, 1002438973,
    1002241374, 1001038946, 1000088364, 1001781958, 1001978086, 1001035980, 1001110133,
    1001604717, 1002078831, 1002488583, 1000792847, 1000641673, 1001165015, 1001283536,
    1001500299, 1001336583, 1001464976, 1001775715, 1001334053, 1001752220, 1001059238,
    1000061280, 1000343167, 1002262659, 1001017521, 1000926454, 1001549906, 1001399657,
    1001596953, 1001511989, 1002134226, 1000997323, 1002423339, 1002561081, 1002351089,
    1000541923, 1001581475, 1001867280, 1001543175, 1001503038, 1000093464, 1002711441,
    1000730805, 1001702743, 1001948884, 1000065378, 1000215299, 1001326290, 1001638327,
    1000007295, 1001515868, 1002212135, 1001575847, 1002161948, 1001357387, 1001766184,
    1000727306, 1000699503, 1000075976, 1002128547, 1002602954, 1001760223, 1000659375,
    1002062608, 1000750867, 1001376415, 1000364749, 1001058467, 1000618691, 1001144316,
    1001536519, 1002587087, 1001101833, 1000247382, 1000609876, 1001645973, 1001840226,
    1001602395, 1002684664, 1000776962, 1001157755, 1002342517, 1000022381, 1001920189,
    1001801346, 1001162402, 1002689307, 1002156227, 1001687826, 1002596959, 1002349648,
    1001894040, 1001450690, 1001735336, 1001498688, 1001180316, 1000810720, 1001220544,
    1001529035, 1001497563, 1000466115, 1000218057, 1002531776, 1001810079, 1000174553,
    1000145964, 1001540017, 1002531576, 1001728260, 1002033880, 1001150137, 1001953094,
    1002077979, 1001228678, 1002168298, 1000151281, 1002335600, 1002526758, 1001714987,
    1001291988, 1000419221, 1002166820, 1000925310, 1000982141, 1001751176, 1001191639,
    1000428712, 1000885331, 1000130902, 1001035008, 1000538396, 1002436753, 1002272435,
    1001444011, 1000873940, 1000706171, 1000465018, 1000006623, 1000275277, 1002396180,
    1002582721, 1000118329, 1000037089, 1000453971, 1000963958, 1000640803, 1001129926,
    1001224067, 1000047577, 1000393604, 1001300568, 1000147695, 1000808608, 1001958358,
    1001269596, 1002713792, 1002275177, 1002410739, 1002315383, 1000223504, 1001345610,
    1000036857, 1002471258, 1001375224, 1002695360, 1001511054, 1002619075, 1002701703,
    1000428130, 1000207474, 1000367210, 1002676594, 1002290678, 1000780766, 1002248783,
    1000331455, 1000869013, 1002351131, 1001827981, 1001624275, 1001359969, 1000562736,
    1000393157, 1001947238, 1002170490, 1002493182, 1000951354, 1000734774, 1000272483,
    1001067850, 1002234686, 1002259959, 1001223336, 1002086622, 1002102465, 1002404394,
    1000833030, 1001881525, 1002214009, 1001142763, 1001334771, 1001399809, 1000971111,
    1000841798, 1000351662, 1000918770, 1000051328, 1001085842, 1001426393, 1001499469,
    1002673391, 1002439989, 1002656579, 1000503223, 1000199277, 1001102502, 1000804320,
    1001829920, 1002377023, 1000669157, 1000789097, 1002582408, 1000755806, 1000826738,
    1001776772, 1001368136, 1000804767, 1002148559, 1001719604, 1002662183, 1002320519,
    1000047835, 1000240968, 1000980627, 1000889403, 1002350939, 1001131879, 1001271289,
    1002320021, 1000812131, 1001424421, 1002461839, 1001799302, 1000362981, 1001116881,
    1001201610, 1001904014, 1002449788, 1000361053, 1000143359, 1000922307, 1001675730,
    1000773716, 1000964829, 1000830471, 1000431368, 1002320455, 1002506830, 1001079629,
    1002573767, 1000514517, 1002143145, 1001158803, 1001644001, 1002618006, 1002492025,
    1000676599, 1001041617, 1002145451, 1000017230, 1001866916, 1001265926, 1002274458,
    1001815958, 1002702195, 1002236588, 1002247341, 1000118770, 1002178890, 1001472895,
    1000063547, 1000762613, 1002297834, 1000807747, 1001970314, 1002276553, 1000562856,
    1001140852, 1001601748, 1002639679, 1001182746, 1002650044, 1001227746, 1000170311,
    1000564560, 1001740055, 1001904646, 1000539800, 1001501829, 1000531802, 1000128379,
    1000790358, 1001940684, 1000158028, 1001881112, 1002492922, 1000803880, 1000643541,
    1000052799, 1000297615, 1001726109, 1002543957, 1000054716, 1000155125, 1002652618,
    1000769467, 1002042242, 1000368676, 1000858673, 1001822210, 1000350231, 1001442450,
    1000050760, 1002041174, 1000233535, 1001572148, 1001432199, 1001742887, 1001172196,
    1000364258, 1001970826, 1002105767, 1002037960, 1000451410, 1002099403, 1000967350,
    1000147974, 1000341770, 1001680878, 1001613527, 1002528697, 1002715601, 1002474922,
    1000818246, 1001278242, 1000681152, 1000997312, 1000010867, 1000952787, 1000993230,
    1001786224, 1000203905, 1000911729, 1000885883, 1002075501, 1001614854, 1001924725,
    1000385909, 1001433831, 1000201148, 1001606931, 1001941294, 1000962064, 1001860524,
    1000774230, 1002033685, 1001261980, 1000288611, 1001334187, 1001921383, 1002166359,
    1001938346, 1001262841, 1000993575, 1001270506, 1002702506, 1001842295, 1000382304,
    1000042449, 1000021771, 1000481204, 1002121723, 1001652776, 1001226182, 1002225233,
    1001253286, 1001562116, 1000154064, 1000094446, 1001633759, 1001546738, 1001004738,
    1002556326, 1001205897, 1000902561, 1002001937, 1001700843, 1002435437, 1000989228,
    1001653126, 1001334470, 1002071621, 1001522103, 1001548001, 1001285963, 1000090895,
    1000746072, 1001574873, 1002175814, 1002531988, 1002144195, 1001770967, 1001451200,
    1001856086, 1000559388, 1001297817, 1000837494, 1001534871, 1000220313, 1001259385,
    1002100069, 1001332921, 1000544786, 1001841512, 1001383653, 1000684470, 1002161309,
    1002231396, 1002348115, 1001758109, 1001747036, 1000284497, 1000832130, 1001282170,
    1002010186, 1000427241, 1001581775, 1000578192, 1000078262, 1000016694, 1002705118,
    1001733624, 1001363335, 1002436545, 1001328603, 1002180807, 1000767722, 1002485195,
    1002272469, 1001271235, 1001481701, 1001348533, 1000388469, 1001219997, 1001460577,
    1001590875, 1001240142, 1000238824, 1000121322, 1002004474, 1001592727, 1001385776,
    1002470366, 1002664998, 1001997277, 1001120647, 1002575276, 1001722300, 1002216129,
    1000067607, 1000994775, 1001498352, 1002699359, 1000587363, 1000217706, 1002109991,
    1000288982, 1002699713, 1002233823, 1001153232, 1000914136, 1002548433, 1001734510,
    1002342520, 1002452277, 1001100860, 1000950342, 1000388189, 1000127650, 1002315582,
    1001566979, 1000957774, 1000052007, 1000364394, 1001996793, 1001908176, 1002105936,
    1002086992, 1002016069, 1001183918, 1000800618, 1001022239, 1001270359, 1002242646,
    1002505634, 1000716919, 1000901112, 1002481410, 1000298970, 1000516124, 1000057052,
    1002361761, 1002647536, 1000845811, 1002171985, 1002050592, 1000006586, 1001255552,
    1001837963, 1001256691, 1001761220, 1000938672, 1000530532, 1000311534, 1001776809,
    1000318357, 1002053600, 1001444195, 1002365910, 1001746878, 1000228196, 1002060204,
    1001739958, 1001886840, 1000698210, 1002509868, 1001502009, 1001893527, 1002638619,
    1002066935, 1001550280, 1002294683, 1000107898, 1000010223, 1001508748, 1001689201,
    1001003737, 1001014351, 1002082703, 1002224120, 1002157272, 1000882553, 1000053316,
    1001414012, 1001342905, 1002156509, 1000251513, 1001232667, 1001399253, 1001855493,
    1001013265, 1001688212, 1002583768, 1001529428, 1001127460, 1001698379, 1002315633,
    1002318183, 1001620522, 1001378536, 1001574209, 1000569652, 1001604632, 1002697581,
    1001575675, 1001117505, 1002051194, 1000183177, 1000855203, 1001423033, 1001777736,
    1002318334, 1001056375, 1002426820, 1002236516, 1000150483, 1001578508, 1001358370,
    1000011455, 1002362070, 1000806847, 1001801348, 1000085743, 1001413671, 1001540172,
    1001829995, 1001851813, 1001597485, 1000078746, 1001840646, 1001995820, 1000160512,
    1002451933, 1001191721, 1002203693, 1000238465, 1002033572, 1001320063, 1000872380,
    1000623197, 1002147129, 1000011914, 1001530603, 1002446364, 1000079514, 1002448631,
    1001076382, 1002223593, 1000490815, 1000630667, 1000376213, 1002462823, 1000941817,
    1002357231, 1001797587, 1000926673, 1000594216, 1001861749, 1001684910, 1002501511,
    1001579538, 1002606713, 1000695748, 1000903985, 1001546192, 1001996840, 1001472327,
    1000032826, 1001545056, 1002014395, 1002095028, 1001913231, 1000075824, 1000952746,
    1001853596, 1002182047, 1002628022, 1002004078, 1001566113, 1002163349, 1000790111,
    1002285133, 1002494233, 1000397761, 1001439740, 1001101891, 1001618046, 1001291605,
    1002314992, 1001234028, 1001559891, 1001477621, 1000780535, 1000875694, 1000645845,
    1000484118, 1002111321, 1001878295, 1002646120, 1002039447, 1002029116, 1000156966,
    1000010498, 1001769120, 1000175849, 1000418713, 1001915737, 1000709712, 1001922556,
    1001283094, 1002208729, 1000333316, 1000012505, 1001047126, 1001172398, 1000177517,
    1001487440, 1000001465, 1000882506, 1002659495, 1002052955, 1001398557, 1001303140,
    1000860772, 1002525449, 1002527325, 1002183331, 1001776468, 1000205762, 1000597585,
    1000113771, 1001870016, 1000025477, 1002086198, 1000502730, 1001981118, 1002473830,
    1001480043, 1002528637, 1000838413, 1001653286, 1001330880, 1002679849, 1002506095,
    1000486043, 1001982507, 1002103300, 1000257805, 1001964267, 1000874462, 1001326641,
    1001782479, 1001021296, 1001867545, 1000694265, 1002033815, 1000304505, 1000985515,
    1000135881, 1002495192, 1000821217, 1001810184, 1002310190, 1002635800, 1000729831,
    1000784775, 1001668437, 1002274466, 1000797484, 1002537649, 1000475111, 1000437411,
    1001657664, 1001278789, 1001836754, 1001983301, 1001494451, 1002357233, 1002212481,
    1001782077, 1001629749, 1000824935, 1002484633, 1000050369, 1000716672, 1001310861,
    1001954605, 1002473461, 1000000607, 1000825586, 1001437178, 1000204392, 1002359775,
    1001499856, 1001810430, 1002093548, 1001448073, 1001173925, 1000746976, 1002502228,
    1000177952, 1001304657, 1001653859, 1000214289, 1000157134, 1001597852, 1002060579,
    1000735315, 1002615587, 1001055343, 1002059341, 1000356177, 1000447259, 1000699286,
    1002457427, 1001989841, 1001801036, 1001434979, 1001957769, 1000007005, 1000916205,
    1001169392, 1001889767, 1000075558, 1002025030, 1001976875, 1001560208, 1001115658,
    1001598286, 1001205754, 1002231878, 1000813552, 1001981838, 1000677677, 1001053575,
    1002255917, 1002025140, 1001734307, 1001057270, 1001087649, 1002574757, 1001915861,
    1001674897, 1001364136, 1002154777, 1001019768, 1001614035, 1000379490, 1001908572,
    1002591014, 1000164476, 1000021830, 1002468679, 1001717996, 1001998254, 1000470822,
    1001431751, 1001319414, 1002050667, 1000560326, 1001259195, 1000866918, 1001379510,
    1002113186, 1001517995, 1001646691, 1000465784, 1001374778, 1001896574, 1002089665,
    1000872939, 1001914402, 1001909903, 1002351476, 1002080607, 1001717244, 1000018324,
    1001994530, 1001422230, 1001353347, 1000339608, 1000475920, 1001694035, 1000982021,
    1001590385, 1001535698, 1000544821, 1001163684, 1000525655, 1002208497, 1001140991,
    1000766128, 1000607192, 1000151074, 1000723980, 1002133819, 1000500270, 1000834415,
    1002225356, 1000373694, 1001824087, 1000067090, 1000297958, 1002317409, 1001770960,
    1001508749, 1000167477, 1001947065, 1001940057, 1000781919, 1000988086, 1002518061,
    1001558269, 1001815508, 1001091260, 1000263913, 1000405911, 1001511990, 1001723810,
    1001074987, 1002376815, 1001680122, 1001506913, 1001594016, 1001519696, 1001789977,
    1000692462, 1000029089, 1000338311, 1001956638, 1002575343, 1001274784, 1001851680,
    1002390771, 1001326868, 1001743949, 1002550572, 1002559841, 1002530236, 1001459660,
    1000449148, 1002536266, 1002701796, 1002253595, 1001124589, 1001133908, 1002463064,
    1000392223, 1001121726, 1001289467, 1000289591, 1002105764, 1002381694, 1002341368,
    1000827595, 1002084983, 1001682153, 1000633018, 1001113583, 1000928142, 1000033000,
    1000621627, 1001084527, 1001200554, 1001458154, 1002412589, 1001972202, 1002691528,
    1000265992, 1002659026, 1001366843, 1000467244, 1001664670, 1002489296, 1002483595,
    1001900662, 1002686469, 1001330713, 1000830278, 1000795336, 1002364454, 1000811242,
    1001678392, 1000033963, 1002634804, 1001804844, 1000527231, 1002344474, 1002369472,
    1001578312, 1001845592, 1001540167, 1002337924, 1000539731, 1002311229, 1000201698,
    1000953073, 1000629221, 1002663893, 1001791750, 1001914401, 1002604562, 1000828450,
    1002613802, 1001268711, 1000927905, 1001578575, 1001005287, 1002131006, 1002396856,
    1000010276, 1001202203, 1000137591, 1001716758, 1001404162, 1002497367, 1000410225,
    1000002836, 1000456777, 1000005943, 1002065249, 1002074323, 1000814733, 1002442276,
    1002663916, 1001447228, 1000979497, 1000956742, 1001038308, 1002638096, 1000415025,
    1001868899, 1001017794, 1000056720, 1000058183, 1000821880, 1001228149, 1000480755,
    1002499177, 1001537485, 1000339767, 1002044113, 1002573766, 1001269992, 1001354843,
    1000118888, 1000149428, 1001803975, 1000222326, 1001257849, 1002187778, 1002149432,
    1001445651, 1001971190, 1001377902, 1000168509, 1001351656, 1001713927, 1001104442,
    1001890822, 1001869495, 1002575689, 1000762777, 1000150393, 1001259333, 1000565784,
    1002312763, 1001455163, 1002431805, 1002170320, 1001727497, 1002120786, 1001211845,
    1002009293, 1002002759, 1002256034, 1000960075, 1000905489, 1002410523, 1000381073,
    1000173214, 1002395671, 1002256806, 1001768142, 1000177679, 1002692425, 1002088771,
    1000856361, 1001423847, 1001575759, 1000449711, 1001550004, 1001554359, 1001906605,
    1001710361, 1001862213, 1002048548, 1000734007, 1001059720, 1002352183, 1001121521,
    1000213563, 1001664822, 1002450758, 1001128119, 1001734650, 1000402851, 1000517539,
    1001565687, 1001288446, 1000169483, 1000780042, 1000862817, 1000904640, 1001952654,
    1000734391, 1001156795, 1001074923, 1001855482, 1001336701, 1001371862, 1002545044,
    1001454360, 1001719858, 1001598709, 1001450807, 1000713806, 1000792912, 1002677052,
    1001580304, 1002257558, 1001500784, 1002261599, 1000810175, 1001677954, 1001202743,
    1002314510, 1001777562, 1000822655, 1001002115, 1001217819, 1002520043, 1000716537,
    1001958057, 1002669138, 1002021941, 1000828887, 1000177607, 1001993644, 1002485054,
    1001084532, 1000857473,
]

Sample substitutions

Either run this cellto generate a new selection...


In [15]:
#substitution_ids = {}
#with session_scope() as session:
#    for max_distance in range(1, 3):
#        model = Model(Time.discrete, Source.majority, Past.last_bin, Durl.all, max_distance)
#        # Sample 5 times more to make sure we get at least 100 substitutions
#        # even with different quotes
#        substitution_ids[model] = sample(
#            [id for (id,)
#             in session.query(Substitution.id).filter(Substitution.model == model)],
#            5 * SAMPLE_SIZE
#        )

... or this one to use the previous selection (which has already been coded).


In [16]:
substitution_ids = {
    Model(Time.discrete, Source.majority, Past.last_bin, Durl.all, 1): [
        5472, 10466, 8963, 6380, 778, 9747, 474, 830, 8524, 8271,
        674, 6907, 5478, 9421, 8203, 7668, 6772, 2201, 6117, 3916,
        8525, 10101, 1863, 1846, 2741, 8648, 9870, 4681, 1871, 7960,
        6830, 7839, 4664, 2974, 5660, 7994, 4239, 5473, 8170, 10334,
        4306, 9317, 1887, 9638, 6799, 6115, 849, 3303, 4259, 6105,
        7455, 10475, 2176, 5343, 913, 8808, 420, 489, 5431, 872,
        6754, 389, 9343, 8201, 6234, 498, 3411, 9359, 8402, 7506,
        7084, 6809, 7417, 9337, 1924, 5685, 648, 415, 5702, 4612,
        10714, 8804, 4269, 6758, 10980, 7140, 1967, 2968, 5443, 2661,
        9635, 3911, 4346, 3496, 946, 879, 4845, 7148, 3445, 6574,
        5650, 8046, 8943, 7115, 9914, 5688, 54, 1425, 932, 1498,
        3318, 7357, 10723, 571, 6440, 10721, 5723, 4271, 943, 854,
        4067, 2694, 6135, 57, 2434, 8190, 417, 5571, 2213, 6612,
        2674, 7063, 4353, 6653, 800, 6138, 2688, 459, 859, 2430,
        5269, 9871, 10381, 8986, 2072, 9381, 200, 354, 4194, 7394,
        9877, 7033, 723, 9883, 8626, 4206, 486, 3617, 243, 5339,
        5355, 9338, 1724, 811, 6130, 2224, 3311, 4616, 4521, 9410,
        5482, 6369, 7963, 10768, 4256, 4061, 565, 3623, 6657, 141,
        4226, 8642, 11052, 4052, 786, 40, 2025, 632, 541, 5755,
        7486, 2706, 9560, 213, 10879, 5441, 5800, 8684, 2953, 6822,
        411, 9342, 6847, 9494, 392, 6820, 1841, 6945, 5656, 467,
        6433, 9407, 442, 6663, 1996, 5639, 7359, 2162, 8546, 4488,
        7548, 6777, 3649, 2435, 4313, 7125, 348, 10991, 7082, 6795,
        7142, 9535, 5464, 4620, 867, 9760, 8974, 5705, 4223, 7458,
        8382, 8076, 2199, 6183, 10719, 10356, 988, 4330, 10971, 6871,
        4374, 8899, 42, 2157, 8092, 5514, 744, 9361, 1961, 5555,
        8163, 4463, 8887, 4134, 865, 432, 1413, 5754, 5469, 9414,
        3305, 5669, 6181, 570, 7551, 8759, 695, 2419, 6903, 2202,
        969, 7570, 2438, 5429, 4465, 179, 5239, 10438, 7501, 573,
        2559, 9486, 9457, 6146, 949, 1377, 8926, 4390, 9745, 5780,
        3404, 279, 9490, 3765, 4197, 2164, 4691, 5753, 10775, 277,
        11048, 8414, 2014, 7678, 10338, 10636, 6367, 140, 4348, 4429,
        3910, 4220, 410, 2095, 3294, 2230, 4833, 2045, 3626, 5466,
        3306, 3622, 2243, 5322, 8510, 2192, 6235, 4231, 564, 9423,
        9333, 5492, 1725, 10378, 8270, 10173, 2154, 201, 2151, 8562,
        3928, 7006, 5647, 9228, 2721, 8970, 3909, 2686, 9061, 5765,
        7027, 824, 5436, 4860, 7534, 2090, 8192, 4126, 9466, 3446,
        1009, 3450, 5248, 6182, 9888, 22, 513, 7389, 8819, 992,
        11047, 5549, 1519, 7352, 10615, 616, 3624, 7112, 3290, 2181,
        8941, 10877, 977, 2719, 4221, 10399, 2200, 5504, 6238, 9562,
        10862, 9418, 10709, 3618, 9063, 9388, 7137, 897, 3407, 6362,
        10468, 2047, 5673, 10618, 9552, 851, 3416, 4058, 8947, 6814,
        5581, 701, 972, 4340, 4215, 689, 8713, 3956, 8707, 6176,
        3413, 10021, 4516, 4780, 9422, 3347, 7032, 3340, 6431, 5724,
        4288, 4496, 9417, 6119, 4057, 325, 4413, 9869, 8500, 220,
        6152, 9385, 3293, 7996, 793, 219, 4456, 939, 8086, 613,
        6948, 212, 1965, 4417, 10623, 804, 4449, 5424, 9766, 7542,
        9782, 4361, 6356, 1964, 8061, 9393, 8566, 4240, 9401, 4323,
        5484, 10423, 501, 8033, 3414, 8542, 4257, 6462, 4048, 2034,
        7669, 3277, 4295, 7113, 973, 9588, 4480, 2463, 4451, 11113
    ],
    Model(Time.discrete, Source.majority, Past.last_bin, Durl.all, 2): [
        7242, 6639, 9155, 6065, 8704, 3122, 9273, 3556, 7633, 1514,
        2335, 7235, 1309, 4084, 3690, 6671, 4171, 3205, 1207, 2595,
        2587, 4151, 2790, 9504, 3008, 9022, 1610, 3169, 9163, 10168,
        2783, 9271, 8480, 2497, 3782, 9805, 3991, 9513, 10277, 2374,
        5875, 5527, 6079, 8333, 11029, 3028, 3366, 7255, 6691, 3160,
        2108, 10525, 10165, 1810, 6285, 2738, 3897, 2917, 10509, 8718,
        10011, 2801, 1703, 8783, 4901, 6560, 5945, 669, 9647, 3802,
        604, 8024, 7037, 7303, 4930, 2729, 6622, 2346, 4744, 3386,
        10184, 7689, 1708, 3536, 10207, 3973, 7517, 5877, 6349, 2534,
        8851, 1481, 7818, 2912, 5895, 9050, 2483, 1910, 8360, 3992,
        4899, 1147, 5040, 1222, 4766, 3190, 2305, 8869, 10458, 773,
        2730, 4994, 10863, 7299, 7566, 3534, 5334, 6016, 1490, 7904,
        3686, 8794, 7664, 8698, 1239, 1809, 4185, 7523, 1915, 1894,
        6569, 10556, 4018, 7376, 7699, 10256, 9717, 8717, 2765, 5843,
        2266, 9266, 9921, 7896, 7400, 7091, 1800, 4887, 7860, 1454,
        812, 3831, 68, 9168, 9933, 4541, 3696, 9804, 2288, 3329,
        7612, 2500, 1494, 8122, 7167, 9614, 9172, 3486, 6000, 8482,
        8479, 3660, 10679, 10053, 8923, 1312, 10216, 9127, 2516, 9024,
        8322, 3794, 9817, 10508, 7323, 5372, 9778, 1628, 1359, 5092,
        6670, 8738, 3899, 7284, 9911, 4722, 1674, 3067, 10746, 2284,
        4019, 10223, 8139, 9034, 6887, 7629, 3700, 1899, 2736, 5335,
        8690, 2980, 10641, 9726, 7431, 2793, 4716, 3082, 5532, 3006,
        2498, 1397, 7041, 1311, 1493, 6507, 10681, 257, 9895, 10098,
        7653, 6561, 10542, 600, 5220, 1149, 10573, 5311, 2622, 9117,
        5373, 5064, 3687, 8865, 7354, 9822, 11044, 5292, 8267, 8030,
        1775, 8313, 9595, 8247, 578, 4509, 6320, 3674, 3115, 8456,
        9001, 3482, 5396, 11096, 7934, 2678, 8108, 10962, 5079, 6995,
        9603, 5048, 7695, 3161, 6517, 5022, 5059, 10265, 10130, 5121,
        3584, 7716, 6197, 10282, 4628, 11060, 5003, 3509, 9068, 644,
        2537, 4761, 3374, 1877, 1353, 10655, 11076, 9553, 5119, 107,
        9610, 3801, 7880, 10871, 3144, 4578, 4767, 7774, 653, 3753,
        251, 1067, 10076, 1460, 9184, 10580, 8921, 8604, 6709, 2914,
        5143, 145, 10453, 9861, 8003, 4167, 9819, 5007, 10941, 5290,
        7891, 8228, 2833, 4163, 1757, 1826, 3177, 3581, 4965, 7078,
        6219, 3381, 8848, 1412, 7381, 9837, 1209, 2352, 6516, 314,
        10728, 8349, 1423, 9057, 8356, 91, 6198, 159, 9939, 8769,
        2878, 10560, 10729, 9287, 9735, 4591, 9907, 211, 1472, 5595,
        6740, 9069, 1768, 5392, 4092, 3002, 6201, 6419, 9053, 2481,
        1408, 1621, 4525, 6955, 7759, 8745, 7575, 2649, 658, 5938,
        4139, 1602, 3085, 1895, 5106, 1650, 9721, 4954, 9521, 3763,
        3221, 4972, 6093, 5826, 6340, 2623, 7742, 11065, 113, 5280,
        6668, 7035, 3050, 4915, 7265, 8733, 7271, 1157, 9976, 10112,
        3713, 10088, 7942, 9083, 2540, 4963, 2527, 5512, 6651, 9722,
        5905, 1597, 6606, 5948, 172, 4991, 4756, 1111, 9708, 1231,
        8615, 1560, 3134, 3708, 7892, 4593, 3850, 1566, 7438, 8113,
        4181, 1182, 1685, 5405, 5913, 7779, 8107, 1177, 1734, 5950,
        2600, 4913, 2261, 6039, 6297, 10818, 1154, 8788, 5289, 118,
        1645, 5197, 3118, 2792, 1945, 10583, 1049, 3324, 8121, 2533,
        10425, 7713, 9774, 10939, 6957, 6558, 6705, 10066, 2116, 10930,
        4633, 1194, 1692, 5828, 9840, 3780, 4508, 4995, 1624, 10176
    ]
}

Our substitution printing function:


In [17]:
def print_substitution(number, substitution):
    title = ' {} / {}'.format(number, SAMPLE_SIZE)
    print('-' * (80 - len(title)) + title)
    if substitution.validate():
        print('Kept: yes')
    else:
        print('Kept: no')
    print()
    print('     Tokens: {tokens[0]} -> {tokens[1]}'
          .format(tokens=substitution.tokens))
    print('     Lemmas: {lemmas[0]} -> {lemmas[1]}'
          .format(lemmas=substitution.lemmas))
    print()
    print(indent(fill(substitution.source.string), ' ' * 5))
    print()
    print(indent(fill(substitution.destination.string), ' ' * 5))
    print()

def mine_print_substitutions(model):
    seen = 0
    seen_substitutions = set()
    for cluster_id in cluster_ids:
        if seen >= SAMPLE_SIZE:
            break

        model.drop_caches()
        with session_scope() as session:
            cluster = session.query(Cluster).get(cluster_id)

            for substitution in cluster.substitutions(model):
                session.rollback()

                if seen >= SAMPLE_SIZE:
                    break
                if (substitution.destination.id, substitution.position) in seen_substitutions:
                    break

                seen += 1
                seen_substitutions.add((substitution.destination.id, substitution.position))
                print_substitution(seen, substitution)

    if seen < SAMPLE_SIZE:
        print("Didn't find {} substitutions, you should sample more clusters"
              .format(SAMPLE_SIZE))


def db_print_substitutions(model):
    seen = 0
    seen_substitutions = set()
    for substitution_id in substitution_ids[model]:
        if seen >= SAMPLE_SIZE:
            break

        with session_scope() as session:
            substitution = session.query(Substitution).get(substitution_id)

            if (substitution.destination.id, substitution.position) in seen_substitutions:
                continue

            seen += 1
            seen_substitutions.add((substitution.destination.id, substitution.position))
            print_substitution(seen, substitution)

4.1 Single-substitution mining

Here are the individual substitutions and their respective rejected/kept status, with single substitution.


In [18]:
mine_print_substitutions(Model(Time.discrete, Source.majority, Past.last_bin, Durl.all, 1))


------------------------------------------------------------------------ 1 / 100
Kept: no

     Tokens: that -> it
     Lemmas: that -> it

     i hope laura and i did the same thing but i believe he will and i know
     his girls are on his mind and he wants to make sure that first and
     foremost he is a good dad and i think that's going to be an important
     part of his presidency

     but i believe he will and i know his girls are on his mind and he
     wants to make sure that first and foremost he's a good dad and i think
     it's going to be an important part of his presidency

------------------------------------------------------------------------ 2 / 100
Kept: no

     Tokens: higher-level -> higher
     Lemmas: higher-level -> high

     going forward we'll use jquery as one of the libraries used to
     implement higher-level controls in the asp net ajax control toolkit as
     well as to implement new ajax server-side helper methods for asp net
     mvc

     going forward we'll use jquery as one of the libraries used to
     implement higher

------------------------------------------------------------------------ 3 / 100
Kept: no

     Tokens: 's -> 's
     Lemmas: be -> 's

     this is proof of the microscope's potential we are now ready to start
     doing scientific experiments that will add a new dimension to
     measurements being made by other phoenix lander instruments

     this is proof of the microscope's potential

------------------------------------------------------------------------ 4 / 100
Kept: no

     Tokens: superconducting -> discovery
     Lemmas: superconducting -> discovery

     because the lhc is a superconducting machine that works at very low
     temperatures in order to get in and fix it we've got to warm it up
     then we go and fix it and then we cool it down again and that's a
     process that's likely to take two months

     the lhc is a discovery machine

------------------------------------------------------------------------ 5 / 100
Kept: no

     Tokens: beginning -> beginning
     Lemmas: beginning -> begin

     a momentous economic slowdown is now under way we believe the
     deterioration in spain is just in the beginning stages the bulk of the
     pain will be suffered in 2009

     a momentous economic slowdown is now under way we believe the
     deterioration in spain is just in the beginning stages

------------------------------------------------------------------------ 6 / 100
Kept: no

     Tokens: could -> can
     Lemmas: could -> can

     i am so damn glad that you could never be my wife cause i surely
     wouldn't have to listen to that prattle from you every day

     i am so damn glad that you can never be my wife cause i surely
     wouldn't have to listen to that prattle from you every day

------------------------------------------------------------------------ 7 / 100
Kept: no

     Tokens: theaters -> theatres
     Lemmas: theater -> theatre

     the theaters are closed there is just no business at all down there

     the theatres are closed there is just no business at all down there

------------------------------------------------------------------------ 8 / 100
Kept: no

     Tokens: for -> of
     Lemmas: for -> of

     from the very beginning i have said i am going to support the
     candidate that has the best chance for changing the way washington
     works and getting things done and i will be voting for barack obama
     and clapping

     the best chance of changing the way washington works

------------------------------------------------------------------------ 9 / 100
Kept: no

     Tokens: sovereignty -> sovereign
     Lemmas: sovereignty -> sovereign

     what is clear is that russia has invaded georgia's sovereignty has
     encroached on georgia's sovereignty and it is very important for us to
     resolve this issue as quickly as possible

     what is clear is that russia has invaded georgia's sovereign has
     encroached on georgia's sovereignty

----------------------------------------------------------------------- 10 / 100
Kept: no

     Tokens: are -> have
     Lemmas: be -> have

     cleary there's been a number of incidents recently and it's important
     that we go in and double check and make sure that all the standards
     are being maintained

     there have been a number of incidents recently and it's important that
     we go in and double check and make sure that all the standards have
     been maintained

----------------------------------------------------------------------- 11 / 100
Kept: no

     Tokens: angelina -> she
     Lemmas: angelina -> she

     what angelina did was very uncool

     what she did was very uncool

----------------------------------------------------------------------- 12 / 100
Kept: no

     Tokens: very -> uncool
     Lemmas: very -> uncool

     what angelina did was very uncool

     what angelina did was uncool

----------------------------------------------------------------------- 13 / 100
Kept: no

     Tokens: what -> inexplicable
     Lemmas: what -> inexplicable

     your relationship with george bush seems what's the word i'm looking
     for inexplicable

     your relationship with george bush seems inexplicable

----------------------------------------------------------------------- 14 / 100
Kept: no

     Tokens: phosphorus -> phosphorous
     Lemmas: phosphorus -> phosphorous

     white phosphorus can burn down houses and cause horrific burns when it
     touches the skin

     white phosphorous can burn down houses and cause horrific burns when
     it touches the skin

----------------------------------------------------------------------- 15 / 100
Kept: no

     Tokens: phosphorous -> phosphorus
     Lemmas: phosphorous -> phosphorus

     white phosphorous can burn down houses and cause horrific burns when
     it touches the skin

     white phosphorus can burn down houses and cause horrific burns when it
     touches the skin

----------------------------------------------------------------------- 16 / 100
Kept: no

     Tokens: it -> that
     Lemmas: it -> that

     growing up i always wanted to be an olympian now to be the most
     decorated olympian of all time it just sounds weird saying it started
     setting in a little bit after the butterfly i was just trying to focus
     on my next race but i just kept thinking wow greatest olympian of all
     time it's a pretty cool title i'm definitely honored

     that's a pretty cool title

----------------------------------------------------------------------- 17 / 100
Kept: yes

     Tokens: responsibility -> response
     Lemmas: responsibility -> response

     the responsibility of the pakistani government should be one of
     cooperation and of action

     the response of the pakistani government should be one of cooperation
     and of action

----------------------------------------------------------------------- 18 / 100
Kept: no

     Tokens: path-breaking -> pathbreaking
     Lemmas: path-breaking -> pathbreaking

     how is this happening with a path-breaking public-private coalition
     solid science better statistics and precise financing with the
     coordination of the right countries and partners and above all with
     the leadership

     with a pathbreaking public-private coalition solid science better
     statistics and precise financing with the coordination of the right
     countries and partners and above all with the leadership

----------------------------------------------------------------------- 19 / 100
Kept: no

     Tokens: sen -> senator
     Lemmas: sen -> senator

     was the last best chance to bring greater oversight and tighter
     regulation to freddie and fannie and they used every means they could
     to defeat sen hagel's legislation every step of the way

     was the last best chance to bring greater oversight and tighter
     regulation to freddie and fannie and they used every means they could
     to defeat senator hagel's legislation every step of the way

----------------------------------------------------------------------- 20 / 100
Kept: no

     Tokens: re-looking -> relooking
     Lemmas: re-looking -> relooking

     we are not interested in going back and re-looking at an acquisition

     we are not interested in going back and relooking at an acquisition

----------------------------------------------------------------------- 21 / 100
Kept: no

     Tokens: he -> jett
     Lemmas: he -> jett

     jett may have still been alive when john administered cpr and then the
     emt took over jett was pronounced dead at the hospital i like to
     believe john had a chance to say goodbye he may have died in his dad's
     arms i am not certain

     i like to believe john had a chance to say goodbye jett may have died
     in his dad's arms

----------------------------------------------------------------------- 22 / 100
Kept: no

     Tokens: cease-fire -> ceasefire
     Lemmas: cease-fire -> ceasefire

     georgia expresses its readiness to immediately start negotiations with
     the russian federation on cease-fire and termination of hostilities

     georgia expresses its readiness to immediately start negotiations with
     the russian federation on ceasefire and termination of hostilities

----------------------------------------------------------------------- 23 / 100
Kept: no

     Tokens: democrat -> democratic
     Lemmas: democrat -> democratic

     in that era almost all black americans were republicans why from its
     founding in 1854 as the anti-slavery party until today the republican
     party has championed freedom and civil rights for blacks and as one
     pundit so succinctly stated the democrat party is as it always has
     been the party of the four s's slavery secession segregation and now
     socialism

     and as one pundit so succinctly stated the democratic party is as it
     always has been the party of the four s's slavery secession
     segregation and now socialism

----------------------------------------------------------------------- 24 / 100
Kept: no

     Tokens: ministerial -> that
     Lemmas: ministerial -> that

     that ministerial list is what they wish to happen it was not agreed on
     as far as we know there was no agreement on the allocation of cabinet
     posts and we are waiting for the mediator mr mbeki to come and resolve
     the impasse

     that list is what they wish to happen it was not agreed on

----------------------------------------------------------------------- 25 / 100
Kept: no

     Tokens: out-scaled -> outscaled
     Lemmas: out-scaled -> outscaled

     michael's talent out-scaled even his own dinosaurs of jurassic park

     michael's talent outscaled even his own dinosaurs of jurassic park

----------------------------------------------------------------------- 26 / 100
Kept: no

     Tokens: could -> can
     Lemmas: could -> can

     and so our troops are a front line against the taliban there is a
     chain of terror that comes from the pakistani and afghan mountains
     right across europe and could end up very easily on the streets of
     britain

     there is a chain of terror that comes from the pakistani and afghan
     mountains right across europe and can end up very easily on the
     streets of britain

----------------------------------------------------------------------- 27 / 100
Kept: no

     Tokens: recognizing -> recognising
     Lemmas: recognize -> recognise

     i am utterly deeply humbled and chuffed by the fact that i'm a stamp
     i'm going to be licked by millions of australians and i can't wait it
     is also recognizing the value of the arts in australia it brings it
     into general consumption and if i'm part of that i feel intensely
     proud

     it is also recognising the value of the arts in australia it brings it
     into general consumption and if i'm part of that i feel intensely
     proud

----------------------------------------------------------------------- 28 / 100
Kept: no

     Tokens: over -> here
     Lemmas: over -> here

     whether it's joe the plumber in ohio or joe over here

     whether it's joe the plumber in ohio or joe here

----------------------------------------------------------------------- 29 / 100
Kept: no

     Tokens: this -> the
     Lemmas: this -> the

     this law needs to be changed to focus on infants

     the law needs to be changed

----------------------------------------------------------------------- 30 / 100
Kept: no

     Tokens: it -> drugs
     Lemmas: it -> drug

     i dragged amy into it and without me there is no doubt that she would
     never have gone down that road i ruined something beautiful now i have
     to let her go to save her life i am not abandoning her i am doing this
     out of love

     i dragged amy into drugs and without me there is no doubt that she
     would never have gone down that road

----------------------------------------------------------------------- 31 / 100
Kept: no

     Tokens: and -> on
     Lemmas: and -> on

     we are aware of a global virus for which there are some public alerts
     and we've seen some of these on our networks and we are taking steps
     to identify and mitigate the virus

     we are aware of a global virus for which there are some public alerts
     on

----------------------------------------------------------------------- 32 / 100
Kept: yes

     Tokens: material -> physical
     Lemmas: material -> physical

     he hates the american material occupation as much as he hates the
     iranian moral occupation

     he hates the american physical occupation as much as he hates the
     iranian moral occupation

----------------------------------------------------------------------- 33 / 100
Kept: no

     Tokens: against -> aginst
     Lemmas: against -> aginst

     sharon is delighted to have won her case she would prefer not to have
     to take legal action against the media but had no alternative in this
     case

     sharon is delighted to have won her case she would prefer not to have
     to take legal action aginst the media but had no alternative in this
     case

----------------------------------------------------------------------- 34 / 100
Kept: no

     Tokens: organised -> organized
     Lemmas: organise -> organize

     legally relations between two countries cannot be organised by a law
     they should be arranged according to international law through
     treaties or agreements

     legally relations between two countries cannot be organized by a law
     they should be arranged according to international law through
     treaties or agreements

----------------------------------------------------------------------- 35 / 100
Kept: no

     Tokens: does -> he
     Lemmas: do -> he

     he does say he wishes the outfits were for his eyes only

     he says he wishes the outfits were for his eyes only

----------------------------------------------------------------------- 36 / 100
Kept: no

     Tokens: raiding -> raiding
     Lemmas: raiding -> raid

     for apple mac and unix lovers these major spam attacks just mean a
     clogged-up inbox not an infected operating system but organised
     criminals are causing havoc for windows users in the hunt for cold
     hard cash too many people are clicking without thinking exposing
     themselves to hackers who are hell-bent on gaining access to
     confidential information and raiding bank accounts the advice is
     simple you should never open unsolicited attachments however tempting
     they may appear

     too many people are clicking without thinking exposing themselves to
     hackers who are hell-bent on gaining access to confidential
     information and raiding bank accounts

----------------------------------------------------------------------- 37 / 100
Kept: no

     Tokens: one -> 1
     Lemmas: one -> 1

     assuming the economy bottoms out in the second half of 2009 job losses
     are likely to continue into 2010 in all probability taking the final
     toll of lost jobs to around one million

     assuming the economy bottoms out in the second half of 2009 job losses
     are likely to continue into 2010 in all probability taking the final
     toll of lost jobs to around 1 million

----------------------------------------------------------------------- 38 / 100
Kept: no

     Tokens: 1 -> one
     Lemmas: 1 -> one

     assuming the economy bottoms out in the second half of 2009 job losses
     are likely to continue into 2010 in all probability taking the final
     toll of lost jobs to around 1 million

     assuming the economy bottoms out in the second half of 2009 job losses
     are likely to continue into 2010 in all probability taking the final
     toll of lost jobs to around one million

----------------------------------------------------------------------- 39 / 100
Kept: no

     Tokens: if -> of
     Lemmas: if -> of

     if it did then i could walk away it does seem to fly in the face if
     circumstantial evidence

     if it did then i could walk away it does seem to fly in the face of
     circumstantial evidence

----------------------------------------------------------------------- 40 / 100
Kept: no

     Tokens: 18 -> eighteen
     Lemmas: @card@ -> eighteen

     we are convinced that we can overcome this crisis in a period of 18
     months

     we are convinced that we can overcome this crisis in a period of
     eighteen months

----------------------------------------------------------------------- 41 / 100
Kept: no

     Tokens: axe -> ax
     Lemmas: axe -> ax

     make sure you have an axe because you will be busting your way out to
     get on your roof with waters surrounding you

     make sure you have an ax

----------------------------------------------------------------------- 42 / 100
Kept: no

     Tokens: on-line -> online
     Lemmas: on-line -> online

     it may have started on-line but it existed entirely in the real world
     and it hurts just as much

     it may have started online but it existed entirely in the real world
     and it hurts just as much

----------------------------------------------------------------------- 43 / 100
Kept: no

     Tokens: scrutinize -> scrutinise
     Lemmas: scrutinize -> scrutinise

     we will wait patiently listen to their words carefully scrutinize
     their actions under a magnifier and if change happens truly and
     fundamentally we will welcome that

     we will wait patiently listen to their words carefully scrutinise
     their actions under a magnifier and if change happens truly and
     fundamentally we will welcome that

----------------------------------------------------------------------- 44 / 100
Kept: no

     Tokens: as -> condemns
     Lemmas: as -> condemn

     strongly condemns the irresponsible provocations which led to this
     situation as well as the disproportionate use of force

     condemns the disproportionate use of force

----------------------------------------------------------------------- 45 / 100
Kept: no

     Tokens: a -> the
     Lemmas: a -> the

     what i essentially said to him was five years ago when you were in a
     position to buy your business you needed a tax cut then and what i
     want to do is to make sure that the plumber the nurse the firefighter
     the teacher the young entrepreneur who doesn't yet have money i want
     to give them a tax break now

     what i essentially said to him was five years ago when you were in the
     position to buy your business you needed a tax cut then

----------------------------------------------------------------------- 46 / 100
Kept: no

     Tokens: an -> steadiness
     Lemmas: an -> steadiness

     displayed a steadiness an intellectual curiosity a depth of knowledge
     and an approach to looking at problems like this

     steadiness intellectual curiosity a depth of knowledge

----------------------------------------------------------------------- 47 / 100
Kept: no

     Tokens: recapitalise -> recapitalize
     Lemmas: recapitalise -> recapitalize

     we are far from the end of the road back to stability but the plan to
     recapitalise our banking system both here and abroad will i believe
     come to be seen as the moment in the banking crisis of the past year
     when we turned the corner

     we are far from the end of the road back to stability but the plan to
     recapitalize our banking system both here and abroad will i believe
     come to be seen as the moment in the banking crisis of the past year
     when we turned the corner

----------------------------------------------------------------------- 48 / 100
Kept: no

     Tokens: steadiness -> he
     Lemmas: steadiness -> he

     obama displayed a steadiness showed intellectual vigor he has a
     definitive way of doing business that will do us well

     he showed intellectual vigor he has a definitive way of doing business
     that will do us well

----------------------------------------------------------------------- 49 / 100
Kept: no

     Tokens: table -> we
     Lemmas: table -> we

     all of us around the table know we've got to get something done as
     quickly as possible

     we know we've got to get something done as quickly as possible

----------------------------------------------------------------------- 50 / 100
Kept: no

     Tokens: eight -> 8
     Lemmas: eight -> 8

     if we are going to go down this road you know barack obama was eight
     years old somehow responsible for bill ayers

     if we are going to go down this road you know barack obama was 8 years
     old somehow responsible for bill ayers

----------------------------------------------------------------------- 51 / 100
Kept: no

     Tokens: that -> the
     Lemmas: that -> the

     come to ces to see all that palm new-ness you've been waiting for

     come to ces to see all the palm new-ness you've been waiting for

----------------------------------------------------------------------- 52 / 100
Kept: no

     Tokens: the -> that
     Lemmas: the -> that

     come to ces to see all the palm new-ness you've been waiting for

     all that palm new-ness you've been waiting for

----------------------------------------------------------------------- 53 / 100
Kept: no

     Tokens: general -> gen
     Lemmas: general -> gen

     george bush has given a mission to general petraeus and he has done
     his best to try to figure out how to put lipstick on a pig

     george bush has given a mission to gen petraeus and he has done his
     best to try to figure out how to put lipstick on a pig

----------------------------------------------------------------------- 54 / 100
Kept: no

     Tokens: hundred -> 200
     Lemmas: hundred -> @card@

     john mccain voted to cut education funding against accountability
     standards he even proposed abolishing the department of education and
     john mccain's economic plan gives two hundred billion more to special
     interests while taking money away from public schools

     200 billion more to special interests while taking money away from
     public schools

----------------------------------------------------------------------- 55 / 100
Kept: no

     Tokens: recognized -> recognised
     Lemmas: recognize -> recognise

     in 2006 i made a serious error in judgment and conducted myself in a
     way that was disloyal to my family and to my core beliefs i recognized
     my mistake and i told my wife that i had a liaison with another woman
     and i asked for her forgiveness although i was honest in every painful
     detail with my family i did not tell the public

     in 2006 i made a serious error in judgment and conducted myself in a
     way that was disloyal to my family and to my core beliefs i recognised
     my mistake and i told my wife that i had a liaison with another woman
     and i asked for her forgiveness

----------------------------------------------------------------------- 56 / 100
Kept: no

     Tokens: high -> school
     Lemmas: high -> school

     really really good i went through a stage when i was in high school
     when it wasn't so good but now i'm at a stage where i really
     appreciate her a lot more

     when i was in school

----------------------------------------------------------------------- 57 / 100
Kept: no

     Tokens: brought -> been
     Lemmas: bring -> be

     we've had a steady stream of patients every day but the last 24 hours
     has brought about triple the number of cases

     we've had a steady stream of patients every day but the last 24 hours
     has been about triple the number of cases

----------------------------------------------------------------------- 58 / 100
Kept: no

     Tokens: it -> them
     Lemmas: it -> them

     taxes are always a redistribution of money most of the taxes that are
     redistributed go back to those who pay it in roads and airports and
     hospitals and schools and taxes are necessary for the common good and
     there's nothing wrong with examining what our tax structure is and who
     should be paying more who should be paying less and for us to say that
     makes you a socialist i think is an unfortunate characterization and
     is inaccurate

     taxes are always a redistribution of money most of the taxes that are
     redistributed go back to those who pay them in roads and airports and
     hospitals and schools

----------------------------------------------------------------------- 59 / 100
Kept: no

     Tokens: and -> or
     Lemmas: and -> or

     taxes are always a redistribution of money most of the taxes that are
     redistributed go back to those who pay it in roads and airports and
     hospitals and schools and taxes are necessary for the common good and
     there's nothing wrong with examining what our tax structure is and who
     should be paying more who should be paying less and for us to say that
     makes you a socialist i think is an unfortunate characterization and
     is inaccurate

     and taxes are necessary for the common good and there's nothing wrong
     with examining what our tax structure is or who should be paying more
     who should be paying less

----------------------------------------------------------------------- 60 / 100
Kept: no

     Tokens: living -> living
     Lemmas: live -> living

     my father saved my life i probably wouldn't be here if it wasn't for
     him i've not always been a good daughter but he basically gave up his
     job and his life to look after me at a time when i wasn't even sure i
     wanted to go on living i owe him my life

     basically gave up his job and his life to look after me at a time when
     i wasn't even sure i wanted to go on living

----------------------------------------------------------------------- 61 / 100
Kept: no

     Tokens: living -> living
     Lemmas: live -> living

     my father saved my life i probably wouldn't be here if it wasn't for
     him i've not always been a good daughter but he basically gave up his
     job and his life to look after me at a time when i wasn't even sure i
     wanted to go on living i owe him my life

     my father saved my life i probably wouldn't be here if it wasn't for
     him i've not always been a good daughter but he basically gave up his
     job and his life to look after me at a time when i wasn't even sure i
     wanted to go on living

----------------------------------------------------------------------- 62 / 100
Kept: no

     Tokens: mccain -> he
     Lemmas: mccain -> he

     then again that same day when asked how many houses he owns mccain
     lost track he couldn't remember well it's seven seven houses

     he lost track he couldn't remember

----------------------------------------------------------------------- 63 / 100
Kept: no

     Tokens: n't -> not
     Lemmas: n't -> not

     i don't see there is anything wrong with it if everybody concerned
     agrees

     i do not see there is anything wrong with it

----------------------------------------------------------------------- 64 / 100
Kept: no

     Tokens: everyone -> evryone
     Lemmas: everyone -> evryone

     i never planned otherwise everyone's making a big deal that i don't
     have a contract signed i've never even asked to do that

     i never planned otherwise evryone's making a big deal that i don't
     have a contract signed i've never even asked to do that

----------------------------------------------------------------------- 65 / 100
Kept: no

     Tokens: must -> to
     Lemmas: must -> to

     but we must insulate main street from wall street and keep people in
     their homes by reducing mortgage foreclosures

     to insulate main street from wall street

----------------------------------------------------------------------- 66 / 100
Kept: no

     Tokens: current -> all
     Lemmas: current -> all

     our goal when the t-mobile g1 becomes available in october is to
     provide affordable high-speed data service allowing customers to
     experience the full data capabilities of the device and our 3g network
     at the same time we have a responsibility to provide the best network
     experience for all of our customers so we reserve the right to
     temporarily reduce data throughput for a small fraction of our
     customers who have excessive or disproportionate usage that interferes
     with our network performance or our ability to provide quality service
     to all of our customers we removed the 1gb soft limit from our policy
     statement and we are confident that t-mobile g1 customers will enjoy
     the high speed of data access over our 3g network the specific terms
     for our new data plans are still being reviewed and once they are
     final we will be certain to share this broadly with current customers
     and potential new customers

     the specific terms for our new data plans are still being reviewed and
     once they are final we will be certain to share this broadly with all
     customers

----------------------------------------------------------------------- 67 / 100
Kept: no

     Tokens: in -> into
     Lemmas: in -> into

     not that i don't feel like i'm part of the team but when you get that
     nice celebration coming in the dugout and you're getting your ass
     hammered by guys there's no better feeling than to have that done

     when you get that nice celebration coming into the dugout and you're
     getting your ass hammered by guys there's no better feeling than to
     have that done

----------------------------------------------------------------------- 68 / 100
Kept: no

     Tokens: terrifying -> terrifying
     Lemmas: terrify -> terrifying

     i'm in a state of shock and the thought of looking like this at 57 is
     terrifying having seen these images i can honestly say i am never
     going to use a sunbed again

     i'm in a state of shock and the thought of looking like this at 57 is
     terrifying

----------------------------------------------------------------------- 69 / 100
Kept: no

     Tokens: awfully -> awful
     Lemmas: awfully -> awful

     you're hearing an awfully lot about me most of which is not true but
     you're not hearing a lot about you

     you're hearing an awful lot about me most of which is not true

----------------------------------------------------------------------- 70 / 100
Kept: no

     Tokens: patrolman -> ptlm
     Lemmas: patrolman -> ptlm

     the gas was very odorous and created contact of an insulting or
     provoking nature with patrolman parsons

     was very odorous and created contact of an insulting or provoking
     nature with ptlm parsons

----------------------------------------------------------------------- 71 / 100
Kept: no

     Tokens: prevailing -> prevailing
     Lemmas: prevailing -> prevail

     i wish that your voice will become the prevailing voice of the whole
     region of all people

     prevailing voice of the whole region

----------------------------------------------------------------------- 72 / 100
Kept: no

     Tokens: fourteen -> 14
     Lemmas: fourteen -> @card@

     it's amazing how much women's tennis has grown since i joined the tour
     fourteen years ago i am very proud to have reached this milestone for
     me my family and all women athletes out there this achievement really
     shows that if you put your mind to it you can achieve anything

     it's amazing how much women's tennis has grown since i joined the tour
     14 years ago

----------------------------------------------------------------------- 73 / 100
Kept: yes

     Tokens: reckless -> callous
     Lemmas: reckless -> callous

     negligence so gross wanton and culpable as to show a reckless
     disregard for human life

     negligence so gross wanton and culpable as to show a callous disregard
     for human life

----------------------------------------------------------------------- 74 / 100
Kept: yes

     Tokens: election -> campaign
     Lemmas: election -> campaign

     this election is not about issues

     this campaign is not about issues

----------------------------------------------------------------------- 75 / 100
Kept: yes

     Tokens: top -> toughest
     Lemmas: top -> tough

     google doesn't exactly bubble to the top of the list of the top
     competitors we've got going in mobile they might someday but right now

     google doesn't exactly bubble to the top of the list of the toughest
     competitors we've got going in mobile

----------------------------------------------------------------------- 76 / 100
Kept: yes

     Tokens: contract -> patience
     Lemmas: contract -> patience

     i am at the end of my contract in may

     at the end of my patience

----------------------------------------------------------------------- 77 / 100
Kept: yes

     Tokens: unmentionable -> unimaginable
     Lemmas: unmentionable -> unimaginable

     if they think this is the way to go we will end up with millions of
     small farmers all over the world being driven off their land into
     unsustainable unmanageable degraded and dysfunctional conurbations of
     unmentionable awfulness

     unsustainable unmanageable degraded and dysfunctional conurbations of
     unimaginable awfulness

----------------------------------------------------------------------- 78 / 100
Kept: no

     Tokens: curios -> curious
     Lemmas: curio -> curious

     the curios case of benjamin button

     the curious case of benjamin

----------------------------------------------------------------------- 79 / 100
Kept: yes

     Tokens: hope -> partner
     Lemmas: hope -> partner

     i strongly condemn the people who've been stabbed and until they
     abandon their practice of wandering around clutching their sides and
     bleeding there is no hope for peace

     there is no partner for peace

----------------------------------------------------------------------- 80 / 100
Kept: no

     Tokens: honor -> honour
     Lemmas: honor -> honour

     his long and quiet devotion to advancing literacy the arts and
     humanities on a national and local scale deserves our appreciation it
     is our honor to bestow the guild's highest tribute on this
     extraordinary actor

     it is our honour to bestow the guild's highest tribute on this
     extraordinary actor

----------------------------------------------------------------------- 81 / 100
Kept: no

     Tokens: mid-air -> midair
     Lemmas: mid-air -> midair

     usually with a mid-air collision you have very serious damage and very
     serious injuries if you have survivors at all

     usually with a midair collision you have very serious damage and very
     serious injuries if you have survivors at all

----------------------------------------------------------------------- 82 / 100
Kept: no

     Tokens: which -> who
     Lemmas: which -> who

     those who come to britain must play by the rules and benefit the
     country this new route for students will ensure we know exactly who is
     coming here to study and stamp out bogus colleges which facilitate the
     lawbreakers

     those who come to britain must play by the rules and benefit the
     country this new route for students will ensure we know exactly who is
     coming here to study and stamp out bogus colleges who facilitate the
     lawbreakers

----------------------------------------------------------------------- 83 / 100
Kept: yes

     Tokens: clashes -> conflict
     Lemmas: clash -> conflict

     the perverse logic of clashes and violence

     perverse logic of conflict and violence

----------------------------------------------------------------------- 84 / 100
Kept: yes

     Tokens: clashes -> confrontation
     Lemmas: clash -> confrontation

     the perverse logic of clashes and violence

     perverse logic of confrontation and violence

----------------------------------------------------------------------- 85 / 100
Kept: no

     Tokens: de-mineralised -> demineralised
     Lemmas: de-mineralised -> demineralise

     both technologies work extremely well in hot dry desert locations csp
     produces a lot of waste heat and we'd be able to use that to evaporate
     more seawater from the greenhouse and csp needs a supply of clean de-
     mineralised water in order for the electricity generating turbines to
     function and to keep the mirrors at peak output it just so happens the
     seawater greenhouse produces large quantities of this

     and csp needs a supply of clean demineralised water in order for the
     electricity generating turbines to function and to keep the mirrors at
     peak output it just so happens the seawater greenhouse produces large
     quantities of this

----------------------------------------------------------------------- 86 / 100
Kept: no

     Tokens: yall -> y'all
     Lemmas: yall -> y'all

     it baffles me how yall let these blog and magazines control your lifes

     it baffles me how y'all let these blog and magazines control your
     lifes

----------------------------------------------------------------------- 87 / 100
Kept: yes

     Tokens: commercial -> financial
     Lemmas: commercial -> financial

     it is evident that the group which carried out these attacks based
     outside the country had come with single-minded determination to
     create havoc in the commercial capital of the country

     it is evident that the group which carried out these attacks based
     outside the country had come with single-minded determination to
     create havoc in the financial capital of the country

----------------------------------------------------------------------- 88 / 100
Kept: no

     Tokens: the -> this
     Lemmas: the -> this

     subject to regulation under the act

     subject to regulation under this act

----------------------------------------------------------------------- 89 / 100
Kept: no

     Tokens: program -> programme
     Lemmas: program -> programme

     we will not permit any funds to be borrowed from the advanced
     technology program unless there is a guarantee that those funds will
     be replenished in a matter of weeks so as not to delay that crucial
     initiative

     we will not permit any funds to be borrowed from the advanced
     technology programme unless there is a guarantee that those funds will
     be replenished in a matter of weeks so as not to delay that crucial
     initiative

----------------------------------------------------------------------- 90 / 100
Kept: no

     Tokens: time -> anytime
     Lemmas: time -> anytime

     any time you can come back from a 24-6 deficit it's a building block
     and something every team should be proud of

     anytime you can come back from a 24-6 deficit it's a building block

----------------------------------------------------------------------- 91 / 100
Kept: no

     Tokens: labour -> labor
     Lemmas: labour -> labor

     if barack obama had lived here i would be very surprised if even
     somebody as brilliant as him would have been able to break through the
     institutional stranglehold on power within the labour party

     break through the institutional stranglehold on power within the labor
     party

----------------------------------------------------------------------- 92 / 100
Kept: no

     Tokens: decarbonisation -> decarburization
     Lemmas: decarbonisation -> decarburization

     preventing catastrophic and irreversible damage to the global climate
     ultimately requires a major decarbonisation of the world energy
     sources

     preventing catastrophic and irreversible damage to the global climate
     ultimately requires a major decarburization of the world energy
     sources

----------------------------------------------------------------------- 93 / 100
Kept: no

     Tokens: is -> has
     Lemmas: be -> have

     people will be appalled that the government is resorted to raiding
     pension funds in order to plug the black hole in the public finances

     people will be appalled that the government has resorted to raiding
     pension funds in order to plug the black hole in the public finances

----------------------------------------------------------------------- 94 / 100
Kept: no

     Tokens: trail-blazed -> trailblazed
     Lemmas: trail-blazed -> trailblazed

     my career in public service was made possible by the path heroes like
     the tuskegee airmen trail-blazed

     my career in public service was made possible by the path heroes like
     the tuskegee airmen trailblazed

----------------------------------------------------------------------- 95 / 100
Kept: no

     Tokens: assured -> assured
     Lemmas: assure -> assured

     nvidia assured apple that mac computers with these graphics processors
     were not affected

     assured apple that mac computers with these graphics processors were
     not affected

----------------------------------------------------------------------- 96 / 100
Kept: no

     Tokens: sitcoms -> sit-coms
     Lemmas: sitcom -> sit-coms

     for a kid who no one's talking to about sex and then he watches
     sitcoms on tv where sex is presented as this is what the cool people
     do

     for a kid who no one's talking to about sex and then he watches sit-
     coms on tv where sex is presented as this is what the cool people do

----------------------------------------------------------------------- 97 / 100
Kept: no

     Tokens: makers -> policymakers
     Lemmas: maker -> policymaker

     policy makers may soon be forced to close financial markets as the
     panic selling accelerates indeed we have now reached a point where
     fundamentals and long term valuation considerations do not matter any
     more for financial markets there is a free fall as most investors are
     rapidly deleveraging and we are on the verge of a a capitulation
     collapse what matters now is only flows rather than stocks and
     fundamentals and flows are unidirectional as everyone is selling and
     no one is buying as trying to buy equities is like catching a falling
     knife there are no buyers in these dysfunctional markets only sellers
     and panic is the ugly state of this destabilizing game we have reached
     the scary point where the dysfunctional behavior of financial markets
     has destructive effects on the financial system and much worse on the
     real economies so it is time to think about more radical policy
     actions and government interventions

     policymakers may soon be forced to close financial markets as the
     panic selling accelerates

----------------------------------------------------------------------- 98 / 100
Kept: no

     Tokens: obama -> personally
     Lemmas: obama -> personally

     i've never made a racist comment and i never attacked him obama
     personally

     i've never made a racist comment and i never attacked him personally

----------------------------------------------------------------------- 99 / 100
Kept: no

     Tokens: sen -> senator
     Lemmas: sen -> senator

     these are decisions that sen mccain has made i have great confidence
     in him

     these are decisions that senator mccain has made i have great
     confidence in him

---------------------------------------------------------------------- 100 / 100
Kept: no

     Tokens: christ -> jesus
     Lemmas: christ -> jesus

     mrs palin needs to be reminded that jesus christ was a community
     organizer and pontius pilate was a governor

     jesus was a community organizer and pontius pilate was a governor

Now with only valid substitutions, to make the precision analysis more accurate.


In [19]:
db_print_substitutions(Model(Time.discrete, Source.majority, Past.last_bin, Durl.all, 1))


------------------------------------------------------------------------ 1 / 100
Kept: yes

     Tokens: required -> necessary
     Lemmas: require -> necessary

     the auto companies must not squander this chance to reform bad
     management practices and begin the long-term restructuring that is
     absolutely required to save this critical industry and the millions of
     american jobs that depend on it

     the auto companies must not squander this chance to reform bad
     management practices and begin the long-term restructuring that is
     absolutely necessary to save this critical industry and the millions
     of american jobs that depend on it

------------------------------------------------------------------------ 2 / 100
Kept: yes

     Tokens: info -> information
     Lemmas: info -> information

     i'm not kidding around can you put that aside and understand that the
     case we are trying here and the info you're going to hear about here
     is totally separate from that case

     can you put that aside and understand that the case we are trying here
     and the information you're going to hear about here is totally
     separate from that case

------------------------------------------------------------------------ 3 / 100
Kept: yes

     Tokens: think -> believe
     Lemmas: think -> believe

     we think a decision could be taken but i think it will happen in
     algeria

     we believe a decision could be taken but i think it will happen in
     algeria

------------------------------------------------------------------------ 4 / 100
Kept: yes

     Tokens: automakers -> autos
     Lemmas: automaker -> auto

     the automakers will get the money as quickly as we can prudently do it

     the autos will get the money as quickly as we can prudently do it

------------------------------------------------------------------------ 5 / 100
Kept: yes

     Tokens: prepared -> ready
     Lemmas: prepare -> ready

     are you prepared to take the oath senator

     are you ready to take the oath

------------------------------------------------------------------------ 6 / 100
Kept: yes

     Tokens: material -> physical
     Lemmas: material -> physical

     he hates the american material occupation as much as he hates the
     iranian moral occupation

     he hates the american physical occupation as much as he hates the
     iranian moral occupation

------------------------------------------------------------------------ 7 / 100
Kept: yes

     Tokens: willing -> going
     Lemmas: willing -> go

     they're not willing to give me anything except appreciation expletive
     them

     not going to give me anything except appreciation

------------------------------------------------------------------------ 8 / 100
Kept: yes

     Tokens: manufacturing -> producing
     Lemmas: manufacturing -> produce

     according to the criminal law there is no capital punishment for the
     crime of manufacturing and selling fake or substandard products
     therefore the most severe penalty for her would be life imprisonment

     producing and selling fake or substandard products

------------------------------------------------------------------------ 9 / 100
Kept: yes

     Tokens: truly -> really
     Lemmas: truly -> really

     we had a relationship that went far beyond friendship j rg and i were
     connected by something truly special he was the man of my life

     we had a relationship that went far beyond friendship j rg and i were
     connected by something really special he was the man of my life

----------------------------------------------------------------------- 10 / 100
Kept: yes

     Tokens: determination -> reflections
     Lemmas: determination -> reflection

     they misunderestimated the compassion of our country i think they
     misunderestimated the will and determination of the commander in chief
     too

     reflections of the commander in chief

----------------------------------------------------------------------- 11 / 100
Kept: yes

     Tokens: catch -> get
     Lemmas: catch -> get

     the republicans can't seem to catch a break when it comes to august
     and when it comes to the weather

     the republicans can't seem to get a break when it comes to august and
     when it comes to the weather

----------------------------------------------------------------------- 12 / 100
Kept: yes

     Tokens: situation -> crisis
     Lemmas: situation -> crisis

     the most serious financial situation in generations

     the most serious financial crisis in generations

----------------------------------------------------------------------- 13 / 100
Kept: yes

     Tokens: unmentionable -> unimaginable
     Lemmas: unmentionable -> unimaginable

     if they think this is the way to go we will end up with millions of
     small farmers all over the world being driven off their land into
     unsustainable unmanageable degraded and dysfunctional conurbations of
     unmentionable awfulness

     unsustainable unmanageable degraded and dysfunctional conurbations of
     unimaginable awfulness

----------------------------------------------------------------------- 14 / 100
Kept: yes

     Tokens: witnessed -> seen
     Lemmas: witness -> see

     palestine has never witnessed an uglier massacre

     palestine has never seen an uglier massacre

----------------------------------------------------------------------- 15 / 100
Kept: yes

     Tokens: socialism -> welfare
     Lemmas: socialism -> welfare

     this is socialism for the rich

     this is welfare for the rich

----------------------------------------------------------------------- 16 / 100
Kept: yes

     Tokens: scared -> afraid
     Lemmas: scare -> afraid

     this is a menace a cancer in pakistan which we will eliminate we will
     not be scared of these cowards

     we will not be afraid of these cowards

----------------------------------------------------------------------- 17 / 100
Kept: yes

     Tokens: blind -> blank
     Lemmas: blind -> blank

     this is no blank check or blind hope

     this is no blank check or blank hope

----------------------------------------------------------------------- 18 / 100
Kept: yes

     Tokens: greatest -> biggest
     Lemmas: great -> big

     mexican dtos represent the greatest organized crime threat to the
     united states

     biggest organized crime threat to the united states

----------------------------------------------------------------------- 19 / 100
Kept: yes

     Tokens: admire -> respect
     Lemmas: admire -> respect

     but we will be respectful i admire sen obama and his accomplishments

     i respect sen obama and his accomplishments

----------------------------------------------------------------------- 20 / 100
Kept: yes

     Tokens: agency -> cia
     Lemmas: agency -> cia

     my position has consistently been that i believe the agency is best-
     served by having an intelligence professional in charge at this time

     my position has consistently been that i believe the cia is best-
     served by having an intelligence professional in charge at this time

----------------------------------------------------------------------- 21 / 100
Kept: yes

     Tokens: things -> faith
     Lemmas: thing -> faith

     this new energy plan is straightforward highly cost-effective and
     creates a clean energy pathway for the rest of the country to follow
     it is the shape of things to come

     the shape of faith to come

----------------------------------------------------------------------- 22 / 100
Kept: yes

     Tokens: crisis -> problem
     Lemmas: crisis -> problem

     this crisis did not develop overnight and it will not be solved
     overnight

     this problem did not develop overnight and it will not be solved
     overnight

----------------------------------------------------------------------- 23 / 100
Kept: yes

     Tokens: clashes -> confrontation
     Lemmas: clash -> confrontation

     the perverse logic of clashes and violence

     perverse logic of confrontation and violence

----------------------------------------------------------------------- 24 / 100
Kept: yes

     Tokens: cool -> steady
     Lemmas: cool -> steady

     a cool hand at the tiller

     steady hand at the tiller

----------------------------------------------------------------------- 25 / 100
Kept: yes

     Tokens: study -> post
     Lemmas: study -> post

     this is an extremely important study

     this is an extremely important post

----------------------------------------------------------------------- 26 / 100
Kept: yes

     Tokens: patients -> parents
     Lemmas: patient -> parent

     this is good news for patients

     this is good news for parents

----------------------------------------------------------------------- 27 / 100
Kept: yes

     Tokens: selling -> changing
     Lemmas: sell -> change

     when a man loves a woman he can't conceivably sell that love i'm in
     love with arsenal i have no intention of selling it my stake

     i have no intention of changing it

----------------------------------------------------------------------- 28 / 100
Kept: yes

     Tokens: believe -> feel
     Lemmas: believe -> feel

     we believe it is inappropriate to use young private citizens for
     marketing purposes

     we feel it is inappropriate to use young private citizens for
     marketing purposes

----------------------------------------------------------------------- 29 / 100
Kept: yes

     Tokens: portray -> reflect
     Lemmas: portray -> reflect

     the intent was to portray a renaissance-like mood on the cover

     the intent was to reflect a renaissance-like mood on the cover

----------------------------------------------------------------------- 30 / 100
Kept: yes

     Tokens: sec -> commission
     Lemmas: sec -> commission

     i am disappointed that the sec chose to bring this case based upon its
     enforcement staff's win-at-any-cost ambitions the staff's process was
     result-oriented facts be damned the government's claims are false and
     they will be proven to be so

     i am disappointed that the commission chose to bring this case based
     upon its enforcement staff's win-at-any-cost ambitions

----------------------------------------------------------------------- 31 / 100
Kept: yes

     Tokens: check -> cheque
     Lemmas: check -> cheque

     is not a blank check

     is not a blank cheque

----------------------------------------------------------------------- 32 / 100
Kept: yes

     Tokens: instability -> volatility
     Lemmas: instability -> volatility

     while market imbalances will feed instability the era of cheap oil is
     over

     while market imbalances will feed volatility the era of cheap oil is
     over

----------------------------------------------------------------------- 33 / 100
Kept: yes

     Tokens: friday -> monday
     Lemmas: friday -> monday

     america feels differently today than it did on friday about this bill
     and he believes the house vote tomorrow will reflect that shift

     america feels differently today than it did on monday about this bill

----------------------------------------------------------------------- 34 / 100
Kept: yes

     Tokens: solution -> body
     Lemmas: solution -> body

     blaming cities for greenhouse gas emissions misses the point that
     cities are a large part of the solution

     a large part of the body

----------------------------------------------------------------------- 35 / 100
Kept: yes

     Tokens: fine -> dedicated
     Lemmas: fine -> dedicated

     roland burris is a good man and a fine public servant but the senate
     democrats made it clear weeks ago that they cannot accept an
     appointment made by a governor who is accused of selling this very
     senate seat i agree with their decision and it is extremely
     disappointing that governor blagojevich has chosen to ignore it while
     governor blagojevich is entitled to his day in court the people of
     illinois are entitled to a functioning government and major decisions
     free of taint and controversy

     roland burris is a good man and a dedicated public servant

----------------------------------------------------------------------- 36 / 100
Kept: yes

     Tokens: manufacture -> design
     Lemmas: manufacture -> design

     custom manufacture of computer hardware for use in data centers and
     mega-scale computing environments for others

     design of computer hardware for use in data centers and mega-scale
     computing environments for others

----------------------------------------------------------------------- 37 / 100
Kept: yes

     Tokens: frail -> fragile
     Lemmas: frail -> fragile

     it is clear that the financial system although improved from where it
     was in september is still frail

     it is clear that the financial system although improved from where it
     was in september is still fragile

----------------------------------------------------------------------- 38 / 100
Kept: yes

     Tokens: sec -> commission
     Lemmas: sec -> commission

     i am disappointed that the sec chose to bring this case based upon its
     enforcement staff's win-at-any-cost ambitions the staff's process was
     result-oriented facts be damned the government's claims are false and
     they will be proven to be so

     i am disappointed that the commission chose to bring this case based
     upon its enforcement staff's win-at-any-cost ambitions the staff's
     process was result-oriented facts be damned the government's claims
     are false and they will be proven to be so

----------------------------------------------------------------------- 39 / 100
Kept: yes

     Tokens: know -> boast
     Lemmas: know -> boast

     you can't change america when you know your first four years as
     president will look exactly like the last eight years of george bush's
     presidency

     you can't change america when you boast

----------------------------------------------------------------------- 40 / 100
Kept: yes

     Tokens: opportunity -> deal
     Lemmas: opportunity -> deal

     strategic opportunity of a lifetime

     strategic deal of a lifetime

----------------------------------------------------------------------- 41 / 100
Kept: yes

     Tokens: paid -> trained
     Lemmas: pay -> train

     we were simply doing the jobs we were paid to do

     we were simply doing the jobs we were trained to do

----------------------------------------------------------------------- 42 / 100
Kept: yes

     Tokens: borders -> limit
     Lemmas: border -> limit

     from tomorrow representatives of the european union will begin
     conducting monitoring up to the southern borders of the security zone

     up to the southern limit of the security zone

----------------------------------------------------------------------- 43 / 100
Kept: yes

     Tokens: required -> necessary
     Lemmas: require -> necessary

     the auto companies must not squander this chance to reform bad
     management practices and begin the long-term restructuring that is
     absolutely required to save this critical industry and the millions of
     american jobs that depend on it

     squander this chance to reform bad management practices and begin the
     long-term restructuring that is absolutely necessary to save this
     critical industry and the millions of american jobs that depend on it

----------------------------------------------------------------------- 44 / 100
Kept: yes

     Tokens: massive -> grave
     Lemmas: massive -> grave

     a flagrant and massive violation of international humanitarian law as
     laid down in article 33 of the fourth geneva convention

     grave violation of international humanitarian law

----------------------------------------------------------------------- 45 / 100
Kept: yes

     Tokens: chatty -> friendly
     Lemmas: chatty -> friendly

     they were very chatty pamela was being her usual flirty self and
     michael seemed to be responding they are such a strange couple but
     they seemed to really hit it off

     they were very friendly pamela was being her usual flirty self and
     michael seemed to be responding they are such a strange couple but
     they seemed to really hit it off

----------------------------------------------------------------------- 46 / 100
Kept: yes

     Tokens: world -> global
     Lemmas: world -> global

     the united states will lose its superpower status in the world
     financial system the world financial system will become more multi-
     polar

     the united states will lose its superpower status in the global
     financial system

----------------------------------------------------------------------- 47 / 100
Kept: yes

     Tokens: dolls -> girls
     Lemmas: doll -> girl

     there's nothing on the dolls that refers to the obama girls

     there's nothing on the girls that refers to the obama girls

----------------------------------------------------------------------- 48 / 100
Kept: yes

     Tokens: candidates -> politicians
     Lemmas: candidate -> politician

     with the economy in turmoil and the american dream at risk the
     american people don't want to hear candidates attack each other you
     want to hear how we're going to attack the challenges facing middle
     class families each and every day

     with the economy in turmoil and the american dream at risk the
     american people don't want to hear politicians attack each other

----------------------------------------------------------------------- 49 / 100
Kept: yes

     Tokens: presidency -> president
     Lemmas: presidency -> president

     securing cyberspace for the 44th presidency

     securing cyberspace for the 44th president

----------------------------------------------------------------------- 50 / 100
Kept: yes

     Tokens: i -> bye
     Lemmas: i -> bye

     i'm thinking rihanna i'll see you at the white house oh and i might
     paint it pink i hope that's cool with you guys

     oh and i might paint it pink bye

----------------------------------------------------------------------- 51 / 100
Kept: yes

     Tokens: diligently -> vigilantly
     Lemmas: diligently -> vigilantly

     we must adhere to our values as diligently as we protect our safety
     with no exceptions

     adhere to our values as vigilantly as we protect our safety with no
     exceptions

----------------------------------------------------------------------- 52 / 100
Kept: yes

     Tokens: f -> fuck
     Lemmas: f -> fuck

     f off elton i'm 40 years younger than you i have my whole life ahead
     of me

     fuck off elton i'm 40 years younger than you i have my whole life
     ahead of me

----------------------------------------------------------------------- 53 / 100
Kept: yes

     Tokens: lunchtime -> luncheon
     Lemmas: lunchtime -> luncheon

     at a lunchtime meeting about school reform in a chicago skyscraper

     at a luncheon meeting about school reform

----------------------------------------------------------------------- 54 / 100
Kept: yes

     Tokens: countries -> folks
     Lemmas: country -> folk

     we have to stop sending 700 billion a year to countries that don't
     like us very much

     folks that don't like us very much

----------------------------------------------------------------------- 55 / 100
Kept: yes

     Tokens: feel -> believe
     Lemmas: feel -> believe

     we feel it is inappropriate to use young private citizens for
     marketing purposes

     we believe it is inappropriate to use young private citizens for
     marketing purposes

----------------------------------------------------------------------- 56 / 100
Kept: yes

     Tokens: palestine -> palestinian
     Lemmas: palestine -> palestinian

     the israeli military must have been aware of the situation but did not
     assist the wounded neither did they make it possible for us or the
     palestine red crescent to assist the wounded

     the israeli military must have been aware of the situation but did not
     assist the wounded neither did they make it possible for us or the
     palestinian red crescent to assist the wounded

----------------------------------------------------------------------- 57 / 100
Kept: yes

     Tokens: expletive -> fuck
     Lemmas: expletive -> fuck

     they're not willing to give me anything except appreciation expletive
     them

     they're not willing to give me anything except appreciation fuck them

----------------------------------------------------------------------- 58 / 100
Kept: yes

     Tokens: i -> love
     Lemmas: i -> love

     i'm bringing a message of peace and i think that's what the region
     needs

     a message of peace and love

----------------------------------------------------------------------- 59 / 100
Kept: yes

     Tokens: men -> britons
     Lemmas: man -> briton

     we call on those holding the men to release them and the rest of the
     crew immediately

     we call on those holding the britons to release them and the rest of
     the crew immediately

----------------------------------------------------------------------- 60 / 100
Kept: yes

     Tokens: food -> agricultural
     Lemmas: food -> agricultural

     we should go back to a policy of maximum food self-sufficiency it is
     crazy for us to think we can develop countries around the world
     without increasing their ability to feed themselves

     we should go back to a policy of maximum agricultural self-sufficiency

----------------------------------------------------------------------- 61 / 100
Kept: yes

     Tokens: see -> watch
     Lemmas: see -> watch

     it will be a stirring sight to see president obama his wife michelle
     and their beautiful girls step through the doors of the white house

     it will be a stirring sight to watch president obama his wife michelle
     and their beautiful girls step through the doors of the white house

----------------------------------------------------------------------- 62 / 100
Kept: yes

     Tokens: pipeline -> line
     Lemmas: pipeline -> line

     i think god's will has to be done in unifying people and companies to
     get that gas pipeline built so pray for that i can do my job there in
     developing my natural resources but all of that doesn't do any good if
     the people of alaska's heart is not good with god

     i think god's will has to be done in unifying people and companies to
     get that gas line built so pray for that

----------------------------------------------------------------------- 63 / 100
Kept: yes

     Tokens: disorganized -> scattered
     Lemmas: disorganize -> scatter

     the aggressor has been punished and suffered very significant losses
     its military has been disorganized

     the aggressor has been punished and suffered very significant losses
     its military has been scattered

----------------------------------------------------------------------- 64 / 100
Kept: yes

     Tokens: solution -> position
     Lemmas: solution -> position

     like all compromises this one has its drawbacks it includes a limited
     amount of new offshore drilling and while i still don't believe that's
     a particularly meaningful short-term or long-term solution i am
     willing to consider it if it's necessary to actually pass a
     comprehensive plan

     i still don't believe that's a particularly meaningful short-term or
     long-term position

----------------------------------------------------------------------- 65 / 100
Kept: yes

     Tokens: concept -> conception
     Lemmas: concept -> conception

     the invasion of iraq was a bandit act an act of blatant state
     terrorism demonstrating absolute contempt for the concept of
     international law the invasion was an act intended to consolidate
     american military and economic control of the middle east masquerading
     as a last resort all other justifications having failed to justify
     themselves as liberation

     a bandit act an act of blatant state terrorism demonstrating absolute
     contempt for the conception of international law

----------------------------------------------------------------------- 66 / 100
Kept: yes

     Tokens: athletes -> world
     Lemmas: athlete -> world

     take significant action to prevent the spread and use of performance-
     enhancing substances i think it's a game we're going to be in for a
     long time what i mean by that is there is somebody in a laboratory
     right now trying to develop some type of substance that can't be
     detected and we've got to stay ahead of it it's not good for the
     athletes it's not good for the sports it's very bad for those who
     don't do it and i think it can attack the very integrity of all sports
     going all the way down to high school

     it's not good for the world

----------------------------------------------------------------------- 67 / 100
Kept: yes

     Tokens: fuck -> f
     Lemmas: fuck -> f

     it's dan the man fuck me you look like a raghead

     f me you look like a raghead

----------------------------------------------------------------------- 68 / 100
Kept: yes

     Tokens: testing -> tests
     Lemmas: testing -> test

     at this point field tests on the powder have been negative additional
     testing will be completed even sending a hoax letter is a serious
     crime

     at this point field tests on the powder have been negative additional
     tests will be completed

----------------------------------------------------------------------- 69 / 100
Kept: yes

     Tokens: meet -> honor
     Lemmas: meet -> honor

     we expect russia to meet its commitment to cease all military
     activities in georgia and we expect all russian forces that entered
     georgia in recent days to withdraw from that country

     we expect russia to honor its commitment

----------------------------------------------------------------------- 70 / 100
Kept: yes

     Tokens: african -> african-american
     Lemmas: african -> african-american

     in the face of all these attacks that are clearly being made on the
     african community why is it that you have not had the ability to not
     one time speak to the interests and even speak on behalf of the
     oppressed and exploited african community or black community in this
     country

     why is it that you have not had the ability to not one time speak to
     the interests and even speak on behalf of the oppressed and exploited
     african-american community or black community in this country

----------------------------------------------------------------------- 71 / 100
Kept: yes

     Tokens: inaugural -> inauguration
     Lemmas: inaugural -> inauguration

     we are one the obama inaugural celebration at the lincoln memorial

     we are one the obama inauguration celebration at the lincoln memorial

----------------------------------------------------------------------- 72 / 100
Kept: yes

     Tokens: story -> history
     Lemmas: story -> history

     standing on the steps of the capitol will be a man whose story
     reflects the enduring promise of our land

     standing on the steps of the capitol will be a man whose history
     reflects the enduring promise of our land

----------------------------------------------------------------------- 73 / 100
Kept: yes

     Tokens: story -> history
     Lemmas: story -> history

     standing on the steps of the capitol will be a man whose story
     reflects the enduring promise of our land

     a man whose history reflects the enduring promise of our land

----------------------------------------------------------------------- 74 / 100
Kept: yes

     Tokens: fragile -> frail
     Lemmas: fragile -> frail

     it is clear that the financial system although improved from where it
     was in september is still fragile and i felt that it would be
     irresponsible for me with the first 350 billion already spent to enter
     into the administration without any potential ammunition should there
     be some sort of emergency or weakening of the financial system

     it is clear that the financial system although improved from where it
     was in september is still frail

----------------------------------------------------------------------- 75 / 100
Kept: yes

     Tokens: careful -> danger
     Lemmas: careful -> danger

     sterling has devalued rapidly against the euro and the dollar we are
     in danger if the government is not careful of having a proper sterling
     collapse a run on the pound

     danger of having a proper sterling collapse a run on the pound

----------------------------------------------------------------------- 76 / 100
Kept: yes

     Tokens: fuck -> expletive
     Lemmas: fuck -> expletive

     they're not willing to give me anything except appreciation fuck them

     willing to give me anything except appreciation expletive them

----------------------------------------------------------------------- 77 / 100
Kept: yes

     Tokens: government -> washington
     Lemmas: government -> washington

     the american people would be troubled if i selected a treasury
     secretary or a chairman of the national economic council at one of the
     most critical economic times in our history who had no experience in
     government whatsoever

     had no experience in washington whatsoever

----------------------------------------------------------------------- 78 / 100
Kept: yes

     Tokens: agreement -> text
     Lemmas: agreement -> text

     establishing checkpoints buffer zones are definitely not part of the
     agreement

     establishing checkpoints buffer zones are definitely not part of the
     text

----------------------------------------------------------------------- 79 / 100
Kept: yes

     Tokens: markets -> economy
     Lemmas: market -> economy

     our economy and our markets will not recover until the bulk of this
     housing correction is behind us

     our economy will not recover until the bulk of this housing correction
     is behind us

----------------------------------------------------------------------- 80 / 100
Kept: yes

     Tokens: crisis -> challenge
     Lemmas: crisis -> challenge

     immediately after i become president i will confront this economic
     crisis head-on by taking all necessary steps to ease the credit crisis
     help hardworking families and restore growth and prosperity

     immediately after i become president i will confront this economic
     challenge head-on by taking all necessary steps to ease the credit
     crisis help hardworking families and restore growth and prosperity

----------------------------------------------------------------------- 81 / 100
Kept: yes

     Tokens: inaugural -> inauguration
     Lemmas: inaugural -> inauguration

     we are one the obama inaugural celebration at the lincoln memorial

     we are one the obama inauguration celebration

----------------------------------------------------------------------- 82 / 100
Kept: yes

     Tokens: platform -> plat
     Lemmas: platform -> plat

     there has been absolutely no discussion about having sarah palin on my
     show at the beginning of this presidential campaign when i decided
     that i was going to take my first public stance in support of a
     candidate i made the decision not to use my show as a platform for any
     of the candidates i agree that sarah palin would be a fantastic
     interview and i would love to have her on after the campaign is over

     at the beginning of this presidential campaign when i decided that i
     was going to take my first public stance in support of a candidate i
     made the decision not to use my show as a plat for any of the
     candidates

----------------------------------------------------------------------- 83 / 100
Kept: yes

     Tokens: fossil -> arabs
     Lemmas: fossil -> arab

     are bought and paid for by the fossil fuel industry usually

     bought and paid for by the arabs

----------------------------------------------------------------------- 84 / 100
Kept: yes

     Tokens: rightfully -> rightly
     Lemmas: rightfully -> rightly

     the american people are angry about executive compensation and
     rightfully so

     the american people are angry about executive compensation and rightly
     so

----------------------------------------------------------------------- 85 / 100
Kept: yes

     Tokens: fast -> quickly
     Lemmas: fast -> quickly

     we didn't set out to conquer gaza we didn't set out to control gaza we
     don't want to remain in gaza and we intend on leaving gaza as fast as
     possible

     we didn't set out to control gaza we don't want to remain in gaza and
     we intend on leaving gaza as quickly as possible

----------------------------------------------------------------------- 86 / 100
Kept: yes

     Tokens: get -> buy
     Lemmas: get -> buy

     it's just a dog you can get another one

     you can buy another one

----------------------------------------------------------------------- 87 / 100
Kept: yes

     Tokens: grandpa -> bye
     Lemmas: grandpa -> bye

     granny i've got to go send my love to the corgis and grandpa god save
     you yeah that's great see you bye

     bye god save you yeah that's great

----------------------------------------------------------------------- 88 / 100
Kept: yes

     Tokens: system -> situation
     Lemmas: system -> situation

     whether it's government or the financial system we're not going to be
     able to do what is needed to be done to stabilize our financial system
     if the american people read about this type of outrageous behavior

     we're not going to be able to do what is needed to be done to
     stabilize our financial situation if the american people read about
     this type of outrageous behavior

----------------------------------------------------------------------- 89 / 100
Kept: yes

     Tokens: investigating -> recovering
     Lemmas: investigate -> recover

     we are in the process of investigating this

     are in the process of recovering

----------------------------------------------------------------------- 90 / 100
Kept: yes

     Tokens: confrontation -> conflict
     Lemmas: confrontation -> conflict

     all it can to help the israelis and palestinians on this dead-end road
     and not to give in to the perverse logic of confrontation and violence

     perverse logic of conflict and violence

----------------------------------------------------------------------- 91 / 100
Kept: yes

     Tokens: expletive -> f
     Lemmas: expletive -> f

     don't let nobody outta here stand the expletive up before it gets ugly
     in here

     stand the f up before it gets ugly in here

----------------------------------------------------------------------- 92 / 100
Kept: yes

     Tokens: cause -> create
     Lemmas: cause -> create

     a failure of either of them would cause great turmoil in our financial
     markets here at home and around the globe

     create great turmoil in our financial markets here at home and around
     the globe

----------------------------------------------------------------------- 93 / 100
Kept: yes

     Tokens: libra -> lyrics
     Lemmas: libra -> lyric

     dancing with the stars toni braxton libra

     dancing with the stars toni braxton lyrics

----------------------------------------------------------------------- 94 / 100
Kept: yes

     Tokens: administration -> administrative
     Lemmas: administration -> administrative

     decisions by the secretary pursuant to the authority of this act are
     non-reviewable and committed to agency discretion and may not be
     reviewed by any court of law or any administration agency

     non-reviewable and committed to agency discretion and may not be
     reviewed by any court of law or any administrative agency

----------------------------------------------------------------------- 95 / 100
Kept: yes

     Tokens: press -> pressure
     Lemmas: press -> pressure

     the mexican possibility may seem less likely but the government its
     politicians police and judicial infrastructure are all under sustained
     assault and press by criminal gangs and drug cartels how that internal
     conflict turns out over the next several years will have a major
     impact on the stability of the mexican state any descent by mexico
     into chaos would demand an american response based on the serious
     implications for homeland security alone

     the mexican possibility may seem less likely but the government its
     politicians police and judicial infrastructure are all under sustained
     assault and pressure by criminal gangs and drug cartels how that
     internal conflict turns out over the next several years will have a
     major impact on the stability of the mexican state any descent by
     mexico into chaos would demand an american response based on the
     serious implications for homeland security alone

----------------------------------------------------------------------- 96 / 100
Kept: yes

     Tokens: laptop -> notebook
     Lemmas: laptop -> notebook

     the laptop pc is no longer a tool only for the business market or a
     computer for the well-off consumer it's now a computer for everyman

     the notebook pc is no longer a tool only for the business market or a
     computer for the well-off consumer it's now a computer for everyman

----------------------------------------------------------------------- 97 / 100
Kept: yes

     Tokens: bill -> act
     Lemmas: bill -> act

     american recovery and reinvestment bill

     american recovery and reinvestment act

----------------------------------------------------------------------- 98 / 100
Kept: yes

     Tokens: tv -> television
     Lemmas: tv -> television

     the barack obama campaign is asking missouri law enforcement to target
     anyone who lies or runs a misleading tv ad during the presidential
     campaign

     anyone who lies or runs a misleading television ad

----------------------------------------------------------------------- 99 / 100
Kept: yes

     Tokens: press -> pressure
     Lemmas: press -> pressure

     the mexican possibility may seem less likely but the government its
     politicians police and judicial infrastructure are all under sustained
     assault and press by criminal gangs and drug cartels how that internal
     conflict turns out over the next several years will have a major
     impact on the stability of the mexican state any descent by mexico
     into chaos would demand an american response based on the serious
     implications for homeland security alone

     the mexican possibility may seem less likely but the government its
     politicians police and judicial infrastructure are all under sustained
     assault and pressure by criminal gangs and drug cartels

---------------------------------------------------------------------- 100 / 100
Kept: yes

     Tokens: case -> need
     Lemmas: case -> need

     now that there is no cholera there is no case for war

     now that there is no cholera there is no need for war

Here, the only possible source of bad filtering is stopwords and rejection because of levenshtein distance <= 1 (the other checks are very specific).

We already know we don't want substitutions on stopwords because we want to focus on meaningful changes, so the question is:

  1. precision of the filter (i.e. are there any obvious missing checks?)
  2. whether the levenshtein distance <= 1 rejection causes many meaningful substitutions to be lost or not: we want to know if there is high recall for this, so as to know whether using orthographic neighborhood density is useful or not.

The codings the substitutions are in codings/filter_evaluations_substitutions-precision-recall; precision answers the first question, recall the second one.

4.2 Double-substitution mining

Now the same with double-substitutions.


In [20]:
mine_print_substitutions(Model(Time.discrete, Source.majority, Past.last_bin, Durl.all, 2))


------------------------------------------------------------------------ 1 / 100
Kept: no

     Tokens: that -> it
     Lemmas: that -> it

     i hope laura and i did the same thing but i believe he will and i know
     his girls are on his mind and he wants to make sure that first and
     foremost he is a good dad and i think that's going to be an important
     part of his presidency

     but i believe he will and i know his girls are on his mind and he
     wants to make sure that first and foremost he's a good dad and i think
     it's going to be an important part of his presidency

------------------------------------------------------------------------ 2 / 100
Kept: no

     Tokens: unmistakeably -> unmistakably
     Lemmas: unmistakeably -> unmistakably

     the finger of suspicion unmistakeably points to the territory of our
     neighbour pakistan

     the finger of suspicion unmistakably points to the territory of our
     neighbor pakistan

------------------------------------------------------------------------ 3 / 100
Kept: no

     Tokens: neighbour -> neighbor
     Lemmas: neighbour -> neighbor

     the finger of suspicion unmistakeably points to the territory of our
     neighbour pakistan

     the finger of suspicion unmistakably points to the territory of our
     neighbor pakistan

------------------------------------------------------------------------ 4 / 100
Kept: no

     Tokens: plans -> actions
     Lemmas: plan -> action

     client has a history dating to his graduate days of homicidal threats
     plans and actions towards therapists

     client has a history dating to his graduate days of homicidal threats
     actions plans

------------------------------------------------------------------------ 5 / 100
Kept: no

     Tokens: and -> plans
     Lemmas: and -> plan

     client has a history dating to his graduate days of homicidal threats
     plans and actions towards therapists

     client has a history dating to his graduate days of homicidal threats
     actions plans

------------------------------------------------------------------------ 6 / 100
Kept: no

     Tokens: higher-level -> higher
     Lemmas: higher-level -> high

     going forward we'll use jquery as one of the libraries used to
     implement higher-level controls in the asp net ajax control toolkit as
     well as to implement new ajax server-side helper methods for asp net
     mvc

     going forward we'll use jquery as one of the libraries used to
     implement higher

------------------------------------------------------------------------ 7 / 100
Kept: no

     Tokens: 's -> 's
     Lemmas: be -> 's

     this is proof of the microscope's potential we are now ready to start
     doing scientific experiments that will add a new dimension to
     measurements being made by other phoenix lander instruments

     this is proof of the microscope's potential

------------------------------------------------------------------------ 8 / 100
Kept: no

     Tokens: superconducting -> discovery
     Lemmas: superconducting -> discovery

     because the lhc is a superconducting machine that works at very low
     temperatures in order to get in and fix it we've got to warm it up
     then we go and fix it and then we cool it down again and that's a
     process that's likely to take two months

     the lhc is a discovery machine

------------------------------------------------------------------------ 9 / 100
Kept: no

     Tokens: beginning -> beginning
     Lemmas: beginning -> begin

     a momentous economic slowdown is now under way we believe the
     deterioration in spain is just in the beginning stages the bulk of the
     pain will be suffered in 2009

     a momentous economic slowdown is now under way we believe the
     deterioration in spain is just in the beginning stages

----------------------------------------------------------------------- 10 / 100
Kept: no

     Tokens: expect -> i
     Lemmas: expect -> i

     i expect us to make the playoffs

     i expect to make the playoffs

----------------------------------------------------------------------- 11 / 100
Kept: no

     Tokens: us -> expect
     Lemmas: us -> expect

     i expect us to make the playoffs

     i expect to make the playoffs

----------------------------------------------------------------------- 12 / 100
Kept: no

     Tokens: could -> can
     Lemmas: could -> can

     i am so damn glad that you could never be my wife cause i surely
     wouldn't have to listen to that prattle from you every day

     i am so damn glad that you can never be my wife cause i surely
     wouldn't have to listen to that prattle from you every day

----------------------------------------------------------------------- 13 / 100
Kept: no

     Tokens: theaters -> theatres
     Lemmas: theater -> theatre

     the theaters are closed there is just no business at all down there

     the theatres are closed there is just no business at all down there

----------------------------------------------------------------------- 14 / 100
Kept: no

     Tokens: we -> this
     Lemmas: we -> this

     we may be seeing the tip of the iceberg of a trend towards less
     comprehensive skimpier health insurance coverage for many working
     people

     this may be only the tip of the iceberg

----------------------------------------------------------------------- 15 / 100
Kept: no

     Tokens: seeing -> only
     Lemmas: see -> only

     we may be seeing the tip of the iceberg of a trend towards less
     comprehensive skimpier health insurance coverage for many working
     people

     this may be only the tip of the iceberg

----------------------------------------------------------------------- 16 / 100
Kept: no

     Tokens: that -> who
     Lemmas: that -> who

     from the very beginning i have said i am going to support the
     candidate that has the best chance for changing the way washington
     works and getting things done and i will be voting for barack obama
     and clapping

     from the very beginning i've said i'm going to support the candidate
     who has the best chance of changing the way washington works and
     getting things done

----------------------------------------------------------------------- 17 / 100
Kept: no

     Tokens: for -> of
     Lemmas: for -> of

     from the very beginning i have said i am going to support the
     candidate that has the best chance for changing the way washington
     works and getting things done and i will be voting for barack obama
     and clapping

     from the very beginning i've said i'm going to support the candidate
     who has the best chance of changing the way washington works and
     getting things done

----------------------------------------------------------------------- 18 / 100
Kept: no

     Tokens: will -> not
     Lemmas: will -> not

     by one complicated measure the number of votes obama may lose due to
     his race is roughly equal to the number who will vote for him because
     he is black and both those numbers appear to be small possibly just 1
     percentage point in each direction

     not vote for obama because he is black

----------------------------------------------------------------------- 19 / 100
Kept: no

     Tokens: him -> obama
     Lemmas: him -> obama

     by one complicated measure the number of votes obama may lose due to
     his race is roughly equal to the number who will vote for him because
     he is black and both those numbers appear to be small possibly just 1
     percentage point in each direction

     not vote for obama because he is black

----------------------------------------------------------------------- 20 / 100
Kept: no

     Tokens: sovereignty -> sovereign
     Lemmas: sovereignty -> sovereign

     what is clear is that russia has invaded georgia's sovereignty has
     encroached on georgia's sovereignty and it is very important for us to
     resolve this issue as quickly as possible

     what is clear is that russia has invaded georgia's sovereign has
     encroached on georgia's sovereignty

----------------------------------------------------------------------- 21 / 100
Kept: no

     Tokens: are -> have
     Lemmas: be -> have

     cleary there's been a number of incidents recently and it's important
     that we go in and double check and make sure that all the standards
     are being maintained

     there have been a number of incidents recently and it's important that
     we go in and double check and make sure that all the standards have
     been maintained

----------------------------------------------------------------------- 22 / 100
Kept: no

     Tokens: nato -> if
     Lemmas: nato -> if

     if nato suddenly takes military actions against abkhazia and south
     ossetia acting solely in support of tbilisi this will mean a
     declaration of war on russia

     if nato takes military actions against abkhazia and south ossetia
     acting solely in support of tbilisi this will mean a declaration of
     war on russia

----------------------------------------------------------------------- 23 / 100
Kept: no

     Tokens: suddenly -> nato
     Lemmas: suddenly -> nato

     if nato suddenly takes military actions against abkhazia and south
     ossetia acting solely in support of tbilisi this will mean a
     declaration of war on russia

     if nato takes military actions against abkhazia and south ossetia
     acting solely in support of tbilisi this will mean a declaration of
     war on russia

----------------------------------------------------------------------- 24 / 100
Kept: no

     Tokens: the -> american
     Lemmas: the -> american

     written in the dark of night that won't do a damn thing about the
     supply of energy it's a hoax on the american people

     won't do a damn thing about american energy

----------------------------------------------------------------------- 25 / 100
Kept: no

     Tokens: supply -> energy
     Lemmas: supply -> energy

     written in the dark of night that won't do a damn thing about the
     supply of energy it's a hoax on the american people

     won't do a damn thing about american energy

----------------------------------------------------------------------- 26 / 100
Kept: no

     Tokens: angelina -> she
     Lemmas: angelina -> she

     what angelina did was very uncool

     what she did was very uncool

----------------------------------------------------------------------- 27 / 100
Kept: no

     Tokens: very -> uncool
     Lemmas: very -> uncool

     what angelina did was very uncool

     what angelina did was uncool

----------------------------------------------------------------------- 28 / 100
Kept: no

     Tokens: what -> inexplicable
     Lemmas: what -> inexplicable

     your relationship with george bush seems what's the word i'm looking
     for inexplicable

     your relationship with george bush seems inexplicable

----------------------------------------------------------------------- 29 / 100
Kept: yes

     Tokens: concerned -> closely
     Lemmas: concern -> closely

     the american people are concerned about the situation in our financial
     markets and our economy and i share their concerns

     closely monitor the situation in our financial markets

----------------------------------------------------------------------- 30 / 100
Kept: yes

     Tokens: about -> monitor
     Lemmas: about -> monitor

     the american people are concerned about the situation in our financial
     markets and our economy and i share their concerns

     closely monitor the situation in our financial markets

----------------------------------------------------------------------- 31 / 100
Kept: no

     Tokens: phosphorus -> phosphorous
     Lemmas: phosphorus -> phosphorous

     white phosphorus can burn down houses and cause horrific burns when it
     touches the skin

     white phosphorous can burn down houses and cause horrific burns when
     it touches the skin

----------------------------------------------------------------------- 32 / 100
Kept: no

     Tokens: phosphorous -> phosphorus
     Lemmas: phosphorous -> phosphorus

     white phosphorous can burn down houses and cause horrific burns when
     it touches the skin

     white phosphorus can burn down houses and cause horrific burns when it
     touches the skin

----------------------------------------------------------------------- 33 / 100
Kept: no

     Tokens: it -> that
     Lemmas: it -> that

     growing up i always wanted to be an olympian now to be the most
     decorated olympian of all time it just sounds weird saying it started
     setting in a little bit after the butterfly i was just trying to focus
     on my next race but i just kept thinking wow greatest olympian of all
     time it's a pretty cool title i'm definitely honored

     that's a pretty cool title

----------------------------------------------------------------------- 34 / 100
Kept: no

     Tokens: what -> do
     Lemmas: what -> do

     i think the convention's gone pretty well so far what do you think

     i think the convention's gone pretty well so far don't you think

----------------------------------------------------------------------- 35 / 100
Kept: no

     Tokens: do -> n't
     Lemmas: do -> n't

     i think the convention's gone pretty well so far what do you think

     i think the convention's gone pretty well so far don't you think

----------------------------------------------------------------------- 36 / 100
Kept: no

     Tokens: know -> they
     Lemmas: know -> they

     we know that they can't take 20 wickets and they are on the back foot
     they couldn't get me and bhaji out we are in with a big chance they
     are under pressure we know that

     they know they can't take 20 wickets and they are on the back foot

----------------------------------------------------------------------- 37 / 100
Kept: no

     Tokens: that -> know
     Lemmas: that -> know

     we know that they can't take 20 wickets and they are on the back foot
     they couldn't get me and bhaji out we are in with a big chance they
     are under pressure we know that

     they know they can't take 20 wickets and they are on the back foot

----------------------------------------------------------------------- 38 / 100
Kept: yes

     Tokens: responsibility -> response
     Lemmas: responsibility -> response

     the responsibility of the pakistani government should be one of
     cooperation and of action

     the response of the pakistani government should be one of cooperation
     and of action

----------------------------------------------------------------------- 39 / 100
Kept: no

     Tokens: path-breaking -> pathbreaking
     Lemmas: path-breaking -> pathbreaking

     how is this happening with a path-breaking public-private coalition
     solid science better statistics and precise financing with the
     coordination of the right countries and partners and above all with
     the leadership

     with a pathbreaking public-private coalition solid science better
     statistics and precise financing with the coordination of the right
     countries and partners and above all with the leadership

----------------------------------------------------------------------- 40 / 100
Kept: no

     Tokens: sen -> senator
     Lemmas: sen -> senator

     was the last best chance to bring greater oversight and tighter
     regulation to freddie and fannie and they used every means they could
     to defeat sen hagel's legislation every step of the way

     was the last best chance to bring greater oversight and tighter
     regulation to freddie and fannie and they used every means they could
     to defeat senator hagel's legislation every step of the way

----------------------------------------------------------------------- 41 / 100
Kept: no

     Tokens: re-looking -> relooking
     Lemmas: re-looking -> relooking

     we are not interested in going back and re-looking at an acquisition

     we are not interested in going back and relooking at an acquisition

----------------------------------------------------------------------- 42 / 100
Kept: no

     Tokens: you -> do
     Lemmas: you -> do

     miss jolie says she likes john mccain too and hasn't decided who to
     endorse huh aren't you supposed to be somewhat enlightened or do you
     not know that the african daughter you hold in every picture had
     parents who suffered and died because of the republican party's
     worldwide economic assault on africa over the last few decades since
     reagan

     do you know that the african daughter you hold in every picture had
     parents who suffered and died because of the republican party's
     worldwide economic assault on africa over the last few decades since
     reagan

----------------------------------------------------------------------- 43 / 100
Kept: no

     Tokens: not -> you
     Lemmas: not -> you

     miss jolie says she likes john mccain too and hasn't decided who to
     endorse huh aren't you supposed to be somewhat enlightened or do you
     not know that the african daughter you hold in every picture had
     parents who suffered and died because of the republican party's
     worldwide economic assault on africa over the last few decades since
     reagan

     do you know that the african daughter you hold in every picture had
     parents who suffered and died because of the republican party's
     worldwide economic assault on africa over the last few decades since
     reagan

----------------------------------------------------------------------- 44 / 100
Kept: no

     Tokens: he -> jett
     Lemmas: he -> jett

     jett may have still been alive when john administered cpr and then the
     emt took over jett was pronounced dead at the hospital i like to
     believe john had a chance to say goodbye he may have died in his dad's
     arms i am not certain

     i like to believe john had a chance to say goodbye jett may have died
     in his dad's arms

----------------------------------------------------------------------- 45 / 100
Kept: no

     Tokens: cease-fire -> ceasefire
     Lemmas: cease-fire -> ceasefire

     georgia expresses its readiness to immediately start negotiations with
     the russian federation on cease-fire and termination of hostilities

     georgia expresses its readiness to immediately start negotiations with
     the russian federation on ceasefire and termination of hostilities

----------------------------------------------------------------------- 46 / 100
Kept: no

     Tokens: democrat -> democratic
     Lemmas: democrat -> democratic

     in that era almost all black americans were republicans why from its
     founding in 1854 as the anti-slavery party until today the republican
     party has championed freedom and civil rights for blacks and as one
     pundit so succinctly stated the democrat party is as it always has
     been the party of the four s's slavery secession segregation and now
     socialism

     and as one pundit so succinctly stated the democratic party is as it
     always has been the party of the four s's slavery secession
     segregation and now socialism

----------------------------------------------------------------------- 47 / 100
Kept: no

     Tokens: ministerial -> that
     Lemmas: ministerial -> that

     that ministerial list is what they wish to happen it was not agreed on
     as far as we know there was no agreement on the allocation of cabinet
     posts and we are waiting for the mediator mr mbeki to come and resolve
     the impasse

     that list is what they wish to happen it was not agreed on

----------------------------------------------------------------------- 48 / 100
Kept: no

     Tokens: out-scaled -> outscaled
     Lemmas: out-scaled -> outscaled

     michael's talent out-scaled even his own dinosaurs of jurassic park

     michael's talent outscaled even his own dinosaurs of jurassic park

----------------------------------------------------------------------- 49 / 100
Kept: no

     Tokens: violations -> this
     Lemmas: violation -> this

     we therefore feel that these repeated defaults and violations have
     forced us to withdraw our support from the ruling coalition and sit on
     the opposition benches

     this situation forced us to withdraw our support

----------------------------------------------------------------------- 50 / 100
Kept: no

     Tokens: have -> situation
     Lemmas: have -> situation

     we therefore feel that these repeated defaults and violations have
     forced us to withdraw our support from the ruling coalition and sit on
     the opposition benches

     this situation forced us to withdraw our support

----------------------------------------------------------------------- 51 / 100
Kept: no

     Tokens: could -> can
     Lemmas: could -> can

     and so our troops are a front line against the taliban there is a
     chain of terror that comes from the pakistani and afghan mountains
     right across europe and could end up very easily on the streets of
     britain

     there is a chain of terror that comes from the pakistani and afghan
     mountains right across europe and can end up very easily on the
     streets of britain

----------------------------------------------------------------------- 52 / 100
Kept: no

     Tokens: recognizing -> recognising
     Lemmas: recognize -> recognise

     i am utterly deeply humbled and chuffed by the fact that i'm a stamp
     i'm going to be licked by millions of australians and i can't wait it
     is also recognizing the value of the arts in australia it brings it
     into general consumption and if i'm part of that i feel intensely
     proud

     it is also recognising the value of the arts in australia it brings it
     into general consumption and if i'm part of that i feel intensely
     proud

----------------------------------------------------------------------- 53 / 100
Kept: no

     Tokens: over -> here
     Lemmas: over -> here

     whether it's joe the plumber in ohio or joe over here

     whether it's joe the plumber in ohio or joe here

----------------------------------------------------------------------- 54 / 100
Kept: no

     Tokens: 's -> theres
     Lemmas: be -> theres

     o'leary o'reilly o'hare and o'hara there's no one as irish as barack
     o'bama

     theres no one as irish as barack obama

----------------------------------------------------------------------- 55 / 100
Kept: no

     Tokens: o'bama -> obama
     Lemmas: o'bama -> obama

     o'leary o'reilly o'hare and o'hara there's no one as irish as barack
     o'bama

     theres no one as irish as barack obama

----------------------------------------------------------------------- 56 / 100
Kept: no

     Tokens: this -> the
     Lemmas: this -> the

     this law needs to be changed to focus on infants

     the law needs to be changed

----------------------------------------------------------------------- 57 / 100
Kept: no

     Tokens: the -> consequences
     Lemmas: the -> consequence

     i am now prepared to subject myself to the laws of the land and face
     the consequences thereof

     i am now prepared to subject myself to the laws of the land and face
     consequences thereof

----------------------------------------------------------------------- 58 / 100
Kept: no

     Tokens: consequences -> thereof
     Lemmas: consequence -> thereof

     i am now prepared to subject myself to the laws of the land and face
     the consequences thereof

     i am now prepared to subject myself to the laws of the land and face
     consequences thereof

----------------------------------------------------------------------- 59 / 100
Kept: no

     Tokens: it -> drugs
     Lemmas: it -> drug

     i dragged amy into it and without me there is no doubt that she would
     never have gone down that road i ruined something beautiful now i have
     to let her go to save her life i am not abandoning her i am doing this
     out of love

     i dragged amy into drugs and without me there is no doubt that she
     would never have gone down that road

----------------------------------------------------------------------- 60 / 100
Kept: no

     Tokens: and -> on
     Lemmas: and -> on

     we are aware of a global virus for which there are some public alerts
     and we've seen some of these on our networks and we are taking steps
     to identify and mitigate the virus

     we are aware of a global virus for which there are some public alerts
     on

----------------------------------------------------------------------- 61 / 100
Kept: yes

     Tokens: material -> physical
     Lemmas: material -> physical

     he hates the american material occupation as much as he hates the
     iranian moral occupation

     he hates the american physical occupation as much as he hates the
     iranian moral occupation

----------------------------------------------------------------------- 62 / 100
Kept: no

     Tokens: against -> aginst
     Lemmas: against -> aginst

     sharon is delighted to have won her case she would prefer not to have
     to take legal action against the media but had no alternative in this
     case

     sharon is delighted to have won her case she would prefer not to have
     to take legal action aginst the media but had no alternative in this
     case

----------------------------------------------------------------------- 63 / 100
Kept: no

     Tokens: organised -> organized
     Lemmas: organise -> organize

     legally relations between two countries cannot be organised by a law
     they should be arranged according to international law through
     treaties or agreements

     legally relations between two countries cannot be organized by a law
     they should be arranged according to international law through
     treaties or agreements

----------------------------------------------------------------------- 64 / 100
Kept: no

     Tokens: does -> he
     Lemmas: do -> he

     he does say he wishes the outfits were for his eyes only

     he says he wishes the outfits were for his eyes only

----------------------------------------------------------------------- 65 / 100
Kept: no

     Tokens: raiding -> raiding
     Lemmas: raiding -> raid

     for apple mac and unix lovers these major spam attacks just mean a
     clogged-up inbox not an infected operating system but organised
     criminals are causing havoc for windows users in the hunt for cold
     hard cash too many people are clicking without thinking exposing
     themselves to hackers who are hell-bent on gaining access to
     confidential information and raiding bank accounts the advice is
     simple you should never open unsolicited attachments however tempting
     they may appear

     too many people are clicking without thinking exposing themselves to
     hackers who are hell-bent on gaining access to confidential
     information and raiding bank accounts

----------------------------------------------------------------------- 66 / 100
Kept: yes

     Tokens: police -> car
     Lemmas: police -> car

     david and victoria beckham were driving a car that had been scratched
     the day before when it had been left parked at the airport they were
     not involved in any accident at all there was no police no crash no
     paparazzi chase etc just very incorrect reports via internet gossip
     sites

     there was no car crash

----------------------------------------------------------------------- 67 / 100
Kept: no

     Tokens: no -> crash
     Lemmas: no -> crash

     david and victoria beckham were driving a car that had been scratched
     the day before when it had been left parked at the airport they were
     not involved in any accident at all there was no police no crash no
     paparazzi chase etc just very incorrect reports via internet gossip
     sites

     there was no car crash

----------------------------------------------------------------------- 68 / 100
Kept: no

     Tokens: you -> we
     Lemmas: you -> we

     we're making common mistakes things you're stressing in practice and
     we need to clean it up our house is messy right now and nobody is
     happy about it

     we're making common mistakes things we're stressing in practice and we
     need to clean them up

----------------------------------------------------------------------- 69 / 100
Kept: no

     Tokens: it -> them
     Lemmas: it -> them

     we're making common mistakes things you're stressing in practice and
     we need to clean it up our house is messy right now and nobody is
     happy about it

     we're making common mistakes things we're stressing in practice and we
     need to clean them up

----------------------------------------------------------------------- 70 / 100
Kept: no

     Tokens: 'm -> i
     Lemmas: be -> i

     i'm more upset at myself with what i said i'm really bad at interviews
     and anyone who knows me knew i meant no harm i'm a goofball mostly i'm
     very sorry to have hurt hef playboy made me who i am and hef has done
     more for me than anyone else in my heart i will always love hef and
     playboy

     i'm sorry to have hurt hef

----------------------------------------------------------------------- 71 / 100
Kept: no

     Tokens: very -> 'm
     Lemmas: very -> be

     i'm more upset at myself with what i said i'm really bad at interviews
     and anyone who knows me knew i meant no harm i'm a goofball mostly i'm
     very sorry to have hurt hef playboy made me who i am and hef has done
     more for me than anyone else in my heart i will always love hef and
     playboy

     i'm sorry to have hurt hef

----------------------------------------------------------------------- 72 / 100
Kept: no

     Tokens: one -> 1
     Lemmas: one -> 1

     assuming the economy bottoms out in the second half of 2009 job losses
     are likely to continue into 2010 in all probability taking the final
     toll of lost jobs to around one million

     assuming the economy bottoms out in the second half of 2009 job losses
     are likely to continue into 2010 in all probability taking the final
     toll of lost jobs to around 1 million

----------------------------------------------------------------------- 73 / 100
Kept: no

     Tokens: 1 -> one
     Lemmas: 1 -> one

     assuming the economy bottoms out in the second half of 2009 job losses
     are likely to continue into 2010 in all probability taking the final
     toll of lost jobs to around 1 million

     assuming the economy bottoms out in the second half of 2009 job losses
     are likely to continue into 2010 in all probability taking the final
     toll of lost jobs to around one million

----------------------------------------------------------------------- 74 / 100
Kept: no

     Tokens: if -> of
     Lemmas: if -> of

     if it did then i could walk away it does seem to fly in the face if
     circumstantial evidence

     if it did then i could walk away it does seem to fly in the face of
     circumstantial evidence

----------------------------------------------------------------------- 75 / 100
Kept: no

     Tokens: 18 -> eighteen
     Lemmas: @card@ -> eighteen

     we are convinced that we can overcome this crisis in a period of 18
     months

     we are convinced that we can overcome this crisis in a period of
     eighteen months

----------------------------------------------------------------------- 76 / 100
Kept: no

     Tokens: dni -> continuity
     Lemmas: dni -> continuity

     there's got to be some continuity and the leadership of both the cia
     and the dni is going to be pivotal to keeping us safe and secure

     continuity is going to be pivotal in keeping us safe and secure

----------------------------------------------------------------------- 77 / 100
Kept: no

     Tokens: to -> in
     Lemmas: to -> in

     there's got to be some continuity and the leadership of both the cia
     and the dni is going to be pivotal to keeping us safe and secure

     continuity is going to be pivotal in keeping us safe and secure

----------------------------------------------------------------------- 78 / 100
Kept: no

     Tokens: axe -> ax
     Lemmas: axe -> ax

     make sure you have an axe because you will be busting your way out to
     get on your roof with waters surrounding you

     make sure you have an ax

----------------------------------------------------------------------- 79 / 100
Kept: no

     Tokens: on-line -> online
     Lemmas: on-line -> online

     it may have started on-line but it existed entirely in the real world
     and it hurts just as much

     it may have started online but it existed entirely in the real world
     and it hurts just as much

----------------------------------------------------------------------- 80 / 100
Kept: no

     Tokens: scrutinize -> scrutinise
     Lemmas: scrutinize -> scrutinise

     we will wait patiently listen to their words carefully scrutinize
     their actions under a magnifier and if change happens truly and
     fundamentally we will welcome that

     we will wait patiently listen to their words carefully scrutinise
     their actions under a magnifier and if change happens truly and
     fundamentally we will welcome that

----------------------------------------------------------------------- 81 / 100
Kept: no

     Tokens: 15 -> years
     Lemmas: @card@ -> year

     in the bible it happened but girls today i don't marry em if they want
     to at 14 15 years old because we won't do it even though i believe
     it's ok

     in the bible it happened but girls today i don't marry em if they want
     to at 14-15 years old

----------------------------------------------------------------------- 82 / 100
Kept: no

     Tokens: years -> old
     Lemmas: year -> old

     in the bible it happened but girls today i don't marry em if they want
     to at 14 15 years old because we won't do it even though i believe
     it's ok

     in the bible it happened but girls today i don't marry em if they want
     to at 14-15 years old

----------------------------------------------------------------------- 83 / 100
Kept: no

     Tokens: as -> condemns
     Lemmas: as -> condemn

     strongly condemns the irresponsible provocations which led to this
     situation as well as the disproportionate use of force

     condemns the disproportionate use of force

----------------------------------------------------------------------- 84 / 100
Kept: no

     Tokens: a -> the
     Lemmas: a -> the

     what i essentially said to him was five years ago when you were in a
     position to buy your business you needed a tax cut then and what i
     want to do is to make sure that the plumber the nurse the firefighter
     the teacher the young entrepreneur who doesn't yet have money i want
     to give them a tax break now

     what i essentially said to him was five years ago when you were in the
     position to buy your business you needed a tax cut then

----------------------------------------------------------------------- 85 / 100
Kept: no

     Tokens: an -> steadiness
     Lemmas: an -> steadiness

     displayed a steadiness an intellectual curiosity a depth of knowledge
     and an approach to looking at problems like this

     steadiness intellectual curiosity and depth of knowledge

----------------------------------------------------------------------- 86 / 100
Kept: no

     Tokens: a -> and
     Lemmas: a -> and

     displayed a steadiness an intellectual curiosity a depth of knowledge
     and an approach to looking at problems like this

     steadiness intellectual curiosity and depth of knowledge

----------------------------------------------------------------------- 87 / 100
Kept: no

     Tokens: recapitalise -> recapitalize
     Lemmas: recapitalise -> recapitalize

     we are far from the end of the road back to stability but the plan to
     recapitalise our banking system both here and abroad will i believe
     come to be seen as the moment in the banking crisis of the past year
     when we turned the corner

     we are far from the end of the road back to stability but the plan to
     recapitalize our banking system both here and abroad will i believe
     come to be seen as the moment in the banking crisis of the past year
     when we turned the corner

----------------------------------------------------------------------- 88 / 100
Kept: no

     Tokens: steadiness -> he
     Lemmas: steadiness -> he

     obama displayed a steadiness showed intellectual vigor he has a
     definitive way of doing business that will do us well

     he showed intellectual vigor he has a definitive way of doing business
     that will do us well

----------------------------------------------------------------------- 89 / 100
Kept: no

     Tokens: table -> we
     Lemmas: table -> we

     all of us around the table know we've got to get something done as
     quickly as possible

     we know we've got to get something done as quickly as possible

----------------------------------------------------------------------- 90 / 100
Kept: no

     Tokens: eight -> 8
     Lemmas: eight -> 8

     if we are going to go down this road you know barack obama was eight
     years old somehow responsible for bill ayers

     if we are going to go down this road you know barack obama was 8 years
     old somehow responsible for bill ayers

----------------------------------------------------------------------- 91 / 100
Kept: no

     Tokens: that -> the
     Lemmas: that -> the

     come to ces to see all that palm new-ness you've been waiting for

     come to ces to see all the palm new-ness you've been waiting for

----------------------------------------------------------------------- 92 / 100
Kept: no

     Tokens: the -> that
     Lemmas: the -> that

     come to ces to see all the palm new-ness you've been waiting for

     all that palm new-ness you've been waiting for

----------------------------------------------------------------------- 93 / 100
Kept: no

     Tokens: general -> gen
     Lemmas: general -> gen

     george bush has given a mission to general petraeus and he has done
     his best to try to figure out how to put lipstick on a pig

     george bush has given a mission to gen petraeus and he has done his
     best to try to figure out how to put lipstick on a pig

----------------------------------------------------------------------- 94 / 100
Kept: no

     Tokens: trahearne -> traherne
     Lemmas: trahearne -> traherne

     when i finally caught up with abraham trahearne he was drinking beer
     with an alcoholic bulldog named fireball roberts in a ramshackle joint
     just outside of sonora california drinking the heart right out of a
     fine spring afternoon

     when i finally caught up with abraham traherne he was drinking beer
     with an alcoholic bulldog named fireball roberts in a ramshackle joint
     just outside of sonoma california drinking the heart right out of a
     fine spring afternoon

----------------------------------------------------------------------- 95 / 100
Kept: no

     Tokens: sonora -> sonoma
     Lemmas: sonora -> sonoma

     when i finally caught up with abraham trahearne he was drinking beer
     with an alcoholic bulldog named fireball roberts in a ramshackle joint
     just outside of sonora california drinking the heart right out of a
     fine spring afternoon

     when i finally caught up with abraham traherne he was drinking beer
     with an alcoholic bulldog named fireball roberts in a ramshackle joint
     just outside of sonoma california drinking the heart right out of a
     fine spring afternoon

----------------------------------------------------------------------- 96 / 100
Kept: no

     Tokens: a -> trillion
     Lemmas: a -> trillion

     peter orszag now forecasts that at the current course and speed a
     trillion-dollar deficit will be here before we even start the next
     budget

     trillion dollar deficit will be here before we even start the next
     budget

----------------------------------------------------------------------- 97 / 100
Kept: no

     Tokens: trillion-dollar -> dollar
     Lemmas: trillion-dollar -> dollar

     peter orszag now forecasts that at the current course and speed a
     trillion-dollar deficit will be here before we even start the next
     budget

     trillion dollar deficit will be here before we even start the next
     budget

----------------------------------------------------------------------- 98 / 100
Kept: no

     Tokens: hundred -> 200
     Lemmas: hundred -> @card@

     john mccain voted to cut education funding against accountability
     standards he even proposed abolishing the department of education and
     john mccain's economic plan gives two hundred billion more to special
     interests while taking money away from public schools

     200 billion more to special interests while taking money away from
     public schools

----------------------------------------------------------------------- 99 / 100
Kept: no

     Tokens: recognized -> recognised
     Lemmas: recognize -> recognise

     in 2006 i made a serious error in judgment and conducted myself in a
     way that was disloyal to my family and to my core beliefs i recognized
     my mistake and i told my wife that i had a liaison with another woman
     and i asked for her forgiveness although i was honest in every painful
     detail with my family i did not tell the public

     in 2006 i made a serious error in judgment and conducted myself in a
     way that was disloyal to my family and to my core beliefs i recognised
     my mistake and i told my wife that i had a liaison with another woman
     and i asked for her forgiveness

---------------------------------------------------------------------- 100 / 100
Kept: no

     Tokens: to -> and
     Lemmas: to -> and

     their leader is a danger to his own people to the region to the world
     he's a disgrace to this house

     a danger to his own people and the world

Now with only valid substitutions, to make the precision analysis more accurate.


In [21]:
db_print_substitutions(Model(Time.discrete, Source.majority, Past.last_bin, Durl.all, 2))


------------------------------------------------------------------------ 1 / 100
Kept: yes

     Tokens: witnessed -> seen
     Lemmas: witness -> see

     palestine has never witnessed an uglier massacre

     palestine has never seen an uglier massacre

------------------------------------------------------------------------ 2 / 100
Kept: yes

     Tokens: world -> global
     Lemmas: world -> global

     the united states will lose its superpower status in the world
     financial system the world financial system will become more multi-
     polar

     the united states will lose its superpower status in the global
     financial system

------------------------------------------------------------------------ 3 / 100
Kept: yes

     Tokens: borders -> limit
     Lemmas: border -> limit

     from tomorrow representatives of the european union will begin
     conducting monitoring up to the southern borders of the security zone

     up to the southern limit of the security zone

------------------------------------------------------------------------ 4 / 100
Kept: yes

     Tokens: believe -> feel
     Lemmas: believe -> feel

     we believe it is inappropriate to use young private citizens for
     marketing purposes

     we feel it is inappropriate to use young private citizens for
     marketing purposes

------------------------------------------------------------------------ 5 / 100
Kept: yes

     Tokens: spend -> talk
     Lemmas: spend -> talk

     see it would be funny but the news media decided that would be the
     lead story yesterday this happens every election cycle every four
     years this is what we do this is what they want to spend two of the
     last 55 days talking about enough

     this is what they want to talk about

------------------------------------------------------------------------ 6 / 100
Kept: yes

     Tokens: time -> assessment
     Lemmas: time -> assessment

     we have continued to examine the relative benefits of purchasing
     illiquid mortgage-related assets our assessment at this time is that
     this is not the most effective way to use bailout funds

     our assessment is that this is not the most effective way

------------------------------------------------------------------------ 7 / 100
Kept: yes

     Tokens: think -> believe
     Lemmas: think -> believe

     we think a decision could be taken but i think it will happen in
     algeria

     we believe a decision could be taken but i think it will happen in
     algeria

------------------------------------------------------------------------ 8 / 100
Kept: yes

     Tokens: frail -> fragile
     Lemmas: frail -> fragile

     it is clear that the financial system although improved from where it
     was in september is still frail

     it is clear that the financial system although improved from where it
     was in september is still fragile

------------------------------------------------------------------------ 9 / 100
Kept: yes

     Tokens: ill -> know
     Lemmas: ill -> know

     without going through the details too much she's gravely ill we
     weren't sure and i'm still not sure whether she makes it to election
     day

     you know we weren't sure and i'm still not sure whether she makes it
     to election day

----------------------------------------------------------------------- 10 / 100
Kept: yes

     Tokens: prepared -> ready
     Lemmas: prepare -> ready

     are you prepared to take the oath senator

     are you ready to take the oath

----------------------------------------------------------------------- 11 / 100
Kept: yes

     Tokens: cause -> create
     Lemmas: cause -> create

     would cause great turmoil in our financial markets here at home and
     around the globe

     create great turmoil in our financial markets here at home and around
     the globe

----------------------------------------------------------------------- 12 / 100
Kept: yes

     Tokens: yesterday -> friday
     Lemmas: yesterday -> friday

     when they were saying they had to leave that an employee got killed
     people were yelling i've been on line since yesterday morning

     when they were saying they had to leave that an employee got killed
     people were yelling i've been on line since friday morning

----------------------------------------------------------------------- 13 / 100
Kept: yes

     Tokens: absence -> lack
     Lemmas: absence -> lack

     i think many of us have been disappointed with the lack of clarity the
     absence of transparency

     i think many of us have been disappointed with the absence of clarity
     the lack of transparency

----------------------------------------------------------------------- 14 / 100
Kept: yes

     Tokens: appointed -> junior
     Lemmas: appoint -> junior

     i'm presenting myself as the legally appointed senator from the state
     of illinois it is my hope and prayer that they recognize that the
     appointment is legal

     the junior senator from the state of illinois

----------------------------------------------------------------------- 15 / 100
Kept: yes

     Tokens: play -> separate
     Lemmas: play -> separate

     i don't really mix them up i play my music and i go with my husband
     when he needs me

     i don't really mix them up i separate them

----------------------------------------------------------------------- 16 / 100
Kept: yes

     Tokens: movie -> video
     Lemmas: movie -> video

     you look just awesome in this new movie

     you look just awesome in this new video

----------------------------------------------------------------------- 17 / 100
Kept: yes

     Tokens: punish -> hurt
     Lemmas: punish -> hurt

     because somebody hurt their feelings they decide to punish the country

     because somebody hurt their feelings they decided to hurt the country

----------------------------------------------------------------------- 18 / 100
Kept: yes

     Tokens: going -> willing
     Lemmas: go -> willing

     not going to give me anything except appreciation

     not willing to give blagojevich anything except appreciation

----------------------------------------------------------------------- 19 / 100
Kept: yes

     Tokens: socialism -> welfare
     Lemmas: socialism -> welfare

     this is socialism for the rich

     this is welfare for the rich

----------------------------------------------------------------------- 20 / 100
Kept: yes

     Tokens: expletive -> fuck
     Lemmas: expletive -> fuck

     our recommendation is fire all those expletive people get em the
     expletive out of there and get us some editorial support

     our recommendation is fire all those fucking people get em the fuck
     out of there

----------------------------------------------------------------------- 21 / 100
Kept: yes

     Tokens: fuck -> f
     Lemmas: fuck -> f

     ya fuck with me i'll kick your ass

     ya f with me i'll kick ass

----------------------------------------------------------------------- 22 / 100
Kept: yes

     Tokens: solution -> body
     Lemmas: solution -> body

     blaming cities for greenhouse gas emissions misses the point that
     cities are a large part of the solution

     a large part of the body

----------------------------------------------------------------------- 23 / 100
Kept: yes

     Tokens: branches -> proxies
     Lemmas: branch -> proxy

     but this is an all-out war against hamas and its branches the
     restraint that we have demonstrated is the source of our strength when
     it is time to fight

     an all-out war with hamas and its proxies

----------------------------------------------------------------------- 24 / 100
Kept: yes

     Tokens: decline -> emergency
     Lemmas: decline -> emergency

     if we are not able to do that then social unrest may happen in many
     countries including advanced economies we are facing an unprecedented
     decline in output all around the planet the people have reacted with
     feelings going from surprise to anger and from anger to fear

     we are facing an unprecedented emergency

----------------------------------------------------------------------- 25 / 100
Kept: yes

     Tokens: responsibly -> possibly
     Lemmas: responsibly -> possibly

     management continues to insist on terms we cannot responsibly accept
     on behalf of our members

     insist on terms we cannot possibly accept

----------------------------------------------------------------------- 26 / 100
Kept: yes

     Tokens: f -> fuck
     Lemmas: f -> fuck

     ya f with me i'll kick your ass

     ya fuck with me i'll kick your ass

----------------------------------------------------------------------- 27 / 100
Kept: yes

     Tokens: situation -> crisis
     Lemmas: situation -> crisis

     the most serious financial situation in generations

     the most serious financial crisis in generations

----------------------------------------------------------------------- 28 / 100
Kept: yes

     Tokens: reserve -> fed
     Lemmas: reserve -> fed

     the federal reserve will employ all available tools to promote the
     resumption of sustainable economic growth and to preserve price
     stability

     the fed will employ all available tools to promote the resumption of
     sustainable economic growth

----------------------------------------------------------------------- 29 / 100
Kept: yes

     Tokens: fuck -> f
     Lemmas: fuck -> f

     it's dan the man fuck me you look like a raghead

     f me you look like a raghead

----------------------------------------------------------------------- 30 / 100
Kept: yes

     Tokens: public -> people
     Lemmas: public -> people

     until we are positive the technology will protect the american public

     until we are positive the technology will protect the american people

----------------------------------------------------------------------- 31 / 100
Kept: yes

     Tokens: case -> need
     Lemmas: case -> need

     now that there is no cholera there is no case for war

     now there is no cholera there is no need for war

----------------------------------------------------------------------- 32 / 100
Kept: yes

     Tokens: holiday -> year
     Lemmas: holiday -> year

     what are you thankful for this holiday season

     what are you thankful for this year

----------------------------------------------------------------------- 33 / 100
Kept: yes

     Tokens: required -> necessary
     Lemmas: require -> necessary

     the auto companies must not squander this chance to reform bad
     management practices and begin the long-term restructuring that is
     absolutely required to save this critical industry and the millions of
     american jobs that depend on it

     the auto companies must not squander this chance to reform bad
     management practices and begin the long-term restructuring that is
     absolutely necessary to save this critical industry and the millions
     of american jobs that depend on it

----------------------------------------------------------------------- 34 / 100
Kept: yes

     Tokens: thing -> person
     Lemmas: thing -> person

     the most normal thing in the world

     most important person in the world

----------------------------------------------------------------------- 35 / 100
Kept: yes

     Tokens: think -> know
     Lemmas: think -> know

     i think that she will bring enormous energy and intellect and skill to
     the position

     i know that she will bring enormous energy and intellect and skill to
     the position

----------------------------------------------------------------------- 36 / 100
Kept: yes

     Tokens: bad -> crimes
     Lemmas: bad -> crime

     those who speak of change must apologise to the iranian people and try
     to repair their past bad acts and the crimes they committed against
     iran

     those who speak of change must apologise to the iranian people and try
     to repair their past crimes

----------------------------------------------------------------------- 37 / 100
Kept: yes

     Tokens: scent -> spray
     Lemmas: scent -> spray

     the scent of seduction with a hint of flame-broiled meat

     body spray of seduction with a hint of flame-broiled meat

----------------------------------------------------------------------- 38 / 100
Kept: yes

     Tokens: love -> salute
     Lemmas: love -> salute

     i love each and every one of you

     i salute each and every one of you

----------------------------------------------------------------------- 39 / 100
Kept: yes

     Tokens: spouse -> husband
     Lemmas: spouse -> husband

     where did you meet your spouse

     where did you meet your husband

----------------------------------------------------------------------- 40 / 100
Kept: yes

     Tokens: admire -> respect
     Lemmas: admire -> respect

     but we will be respectful i admire sen obama and his accomplishments

     i respect sen obama and his accomplishments

----------------------------------------------------------------------- 41 / 100
Kept: yes

     Tokens: bureau -> crime
     Lemmas: bureau -> crime

     several miles away from the first bureau of investigative services
     said the search for the boy had been centered farther east based on
     information they had about where balfour's current girlfriend lives

     several miles away from the first crime scene

----------------------------------------------------------------------- 42 / 100
Kept: yes

     Tokens: story -> history
     Lemmas: story -> history

     standing on the steps of the capitol will be a man whose story
     reflects the enduring promise of our land

     standing on the steps of the capitol will be a man whose history
     reflects the enduring promise of our land

----------------------------------------------------------------------- 43 / 100
Kept: yes

     Tokens: small -> reported
     Lemmas: small -> report

     the jobs we create will be in businesses large and small across a wide
     range of industries

     declines reported across a wide range of industries

----------------------------------------------------------------------- 44 / 100
Kept: yes

     Tokens: give -> get
     Lemmas: give -> get

     we ask you to help us work for that day when black will not be asked
     to give back when brown can stick around when yellow will be mellow
     when the red man can get ahead man and when white will embrace what is
     right

     when black will not be asked to get back when brown can stick around
     when yellow will be mellow when the red man can get ahead man and when
     white will embrace what is right

----------------------------------------------------------------------- 45 / 100
Kept: yes

     Tokens: concept -> conception
     Lemmas: concept -> conception

     the invasion of iraq was a bandit act an act of blatant state
     terrorism demonstrating absolute contempt for the concept of
     international law the invasion was an act intended to consolidate
     american military and economic control of the middle east masquerading
     as a last resort all other justifications having failed to justify
     themselves as liberation

     a bandit act an act of blatant state terrorism demonstrating absolute
     contempt for the conception of international law

----------------------------------------------------------------------- 46 / 100
Kept: yes

     Tokens: crisis -> challenge
     Lemmas: crisis -> challenge

     immediately after i become president i will confront this economic
     crisis head-on by taking all necessary steps to ease the credit crisis
     help hardworking families and restore growth and prosperity

     immediately after i become president i will confront this economic
     challenge head-on by taking all necessary steps to ease the credit
     crisis help hardworking families and restore growth and prosperity

----------------------------------------------------------------------- 47 / 100
Kept: yes

     Tokens: mom -> mama
     Lemmas: mom -> mama

     i just want to say mom it's real hard to sing when you're in the
     second row crying

     mama it's real hard to sing when you're in the second row crying

----------------------------------------------------------------------- 48 / 100
Kept: yes

     Tokens: expletive -> f
     Lemmas: expletive -> f

     if they don't perform expletive em

     if they don't perform f em

----------------------------------------------------------------------- 49 / 100
Kept: yes

     Tokens: demonstrates -> cent
     Lemmas: demonstrate -> cent

     today's results actually show a more complex story than a straight
     pass or fail by one machine where the machines were identified
     correctly by the human interrogators as machines the conversational
     abilities of each machine was scored at 80 and 90 this demonstrates
     how close machines are getting to reaching the milestone of
     communicating with us in a way in which we are comfortable that
     eventual day will herald a new phase in our relationship with machines
     bringing closer the time in which robots start to play an active role
     in our daily lives

     today's results actually show a more complex story than a straight
     pass or fail by one machine where the machines were identified
     correctly by the human interrogators as machines the conversational
     abilities of each machine was scored at 80 and 90 per cent

----------------------------------------------------------------------- 50 / 100
Kept: yes

     Tokens: visiting -> foreign
     Lemmas: visit -> foreign

     insulting a visiting head of state

     insulting a foreign head of state

----------------------------------------------------------------------- 51 / 100
Kept: yes

     Tokens: fundamentals -> foundations
     Lemmas: fundamental -> foundation

     i believe the fundamentals of this economy are strong and i believe
     they will remain strong

     i believe the foundations of this economy are strong

----------------------------------------------------------------------- 52 / 100
Kept: yes

     Tokens: violence -> hostilities
     Lemmas: violence -> hostility

     condemns all acts of violence and terror directed against civilians

     condemns all acts of hostilities and terror directed against civilians

----------------------------------------------------------------------- 53 / 100
Kept: yes

     Tokens: scared -> afraid
     Lemmas: scare -> afraid

     this is a menace a cancer in pakistan which we will eliminate we will
     not be scared of these cowards

     we will not be afraid of these cowards

----------------------------------------------------------------------- 54 / 100
Kept: yes

     Tokens: rightfully -> rightly
     Lemmas: rightfully -> rightly

     the american people are angry about executive compensation and
     rightfully so

     the american people are angry about executive compensation and rightly
     so

----------------------------------------------------------------------- 55 / 100
Kept: yes

     Tokens: expletive -> f
     Lemmas: expletive -> f

     i've got this thing and it's expletive golden and uh uh i'm just not
     giving it up for expletive nothing i'm not gonna do it and and i can
     always use it i can parachute me there

     i've got this thing and it's f golden and uh uh i'm just not giving it
     up for f nothing

----------------------------------------------------------------------- 56 / 100
Kept: yes

     Tokens: half -> part
     Lemmas: half -> part

     we built significant momentum following the completion of our
     corporate transformation and our business results were on track
     through most of 2008 with digital revenue up 10 percent in the first
     half of the year however during the last three months of the year we
     experienced dramatic declines in several of our key businesses due to
     the slowdown in consumer spending and significantly reduced demand for
     capital equipment

     in the first part of the year

----------------------------------------------------------------------- 57 / 100
Kept: yes

     Tokens: expletive -> f
     Lemmas: expletive -> f

     i've got this thing and it's expletive golden and uh uh i'm just not
     giving it up for expletive nothing i'm not gonna do it and and i can
     always use it i can parachute me there

     i've got this thing and it's f golden and uh uh i'm just not giving it
     up for f nothing i'm not gonna do it

----------------------------------------------------------------------- 58 / 100
Kept: yes

     Tokens: view -> word
     Lemmas: view -> word

     i'm looking for a church that preaches my view of the word i think the
     blank translation should be used i think blank should be talked about
     a lot while blank should be ignored and if you ever stop preaching my
     view of the word i will leave and tell others that you don't preach
     the word

     i'm looking for a church that preaches the word

----------------------------------------------------------------------- 59 / 100
Kept: yes

     Tokens: patience -> equality
     Lemmas: patience -> equality

     what do we want patience when do we want it now

     what do we want equality when do we want it now

----------------------------------------------------------------------- 60 / 100
Kept: yes

     Tokens: gravest -> greatest
     Lemmas: grave -> great

     will you appoint a special prosecutor ideally patrick fitzgerald to
     independently investigate the gravest crimes of the bush
     administration including torture and warrantless wiretapping

     will you appoint a special prosecutor ideally patrick fitzgerald to
     independently investigate the greatest crimes of the bush
     administration including torture and warrantless wiretapping

----------------------------------------------------------------------- 61 / 100
Kept: yes

     Tokens: sec -> commission
     Lemmas: sec -> commission

     i am disappointed that the sec chose to bring this case based upon its
     enforcement staff's win-at-any-cost ambitions the staff's process was
     result-oriented facts be damned the government's claims are false and
     they will be proven to be so

     i am disappointed that the commission chose to bring this case based
     upon its enforcement staff's win-at-any-cost ambitions

----------------------------------------------------------------------- 62 / 100
Kept: yes

     Tokens: give -> get
     Lemmas: give -> get

     we ask you to help us work for that day when black will not be asked
     to give back when brown can stick around when yellow will be mellow
     when the red man can get ahead man and when white will embrace what is
     right

     help us work for that day when black will not be asked to get back
     when brown can stick around when yellow will be mellow when the red
     man can get ahead man and when white will embrace what is right

----------------------------------------------------------------------- 63 / 100
Kept: yes

     Tokens: honorable -> graceful
     Lemmas: honorable -> graceful

     if people want to throw me under the bus my personal belief is that
     the most honorable thing to do is to lie there

     if someone wants to throw me under the bus my personal belief is that
     the most graceful thing to do is to lie there

----------------------------------------------------------------------- 64 / 100
Kept: yes

     Tokens: markets -> economy
     Lemmas: market -> economy

     our economy and our markets will not recover until the bulk of this
     housing correction is behind us

     our economy will not recover until the bulk of this housing correction
     is behind us

----------------------------------------------------------------------- 65 / 100
Kept: yes

     Tokens: machine -> computer
     Lemmas: machine -> computer

     know how to make a 500 machine that's not a piece of junk

     know how to make a 500 computer that's not a piece of junk

----------------------------------------------------------------------- 66 / 100
Kept: yes

     Tokens: al-qaida -> qaeda
     Lemmas: al-qaida -> qaeda

     this war is not quite over but it's being won and primarily by the
     people of anbar al-qaida has not been entirely defeated in anbar but
     their end is near and they know it

     al qaeda has not been entirely defeated in anbar but their end is near
     and they know it

----------------------------------------------------------------------- 67 / 100
Kept: yes

     Tokens: appeal -> wait
     Lemmas: appeal -> wait

     we are not going to appeal

     we are not going to wait

----------------------------------------------------------------------- 68 / 100
Kept: yes

     Tokens: find -> say
     Lemmas: find -> say

     it would be hard to find anybody i think at the u n who would not
     believe that obama would be a considerable improvement over any other
     alternative

     it would be hard to say

----------------------------------------------------------------------- 69 / 100
Kept: yes

     Tokens: about -> monitor
     Lemmas: about -> monitor

     the american people are concerned about the situation in our financial
     markets and our economy and i share their concerns

     closely monitor the situation in our financial markets

----------------------------------------------------------------------- 70 / 100
Kept: yes

     Tokens: grandpa -> bye
     Lemmas: grandpa -> bye

     granny i've got to go send my love to the corgis and grandpa god save
     you yeah that's great see you bye

     bye god save you yeah that's great

----------------------------------------------------------------------- 71 / 100
Kept: yes

     Tokens: rights -> freedom
     Lemmas: right -> freedom

     that comment is part of a much larger nuanced and yes provocative
     that's what i do piece from my show about racism freedom women's
     rights and the extreme views of governor sarah palin

     women racism freedom and the extreme views of governor sarah palin

----------------------------------------------------------------------- 72 / 100
Kept: yes

     Tokens: achieve -> accomplish
     Lemmas: achieve -> accomplish

     liberals hate real americans that work and achieve and believe in god

     liberals hate real americans that work and accomplish and achieve

----------------------------------------------------------------------- 73 / 100
Kept: yes

     Tokens: means -> protests
     Lemmas: mean -> protest

     will be forced to react and not only through diplomatic means

     react and not only through diplomatic protests

----------------------------------------------------------------------- 74 / 100
Kept: yes

     Tokens: launch -> use
     Lemmas: launch -> use

     instead of caring about the people of gaza hamas decided to use gaza
     to launch rockets to kill innocent israelis

     instead of caring about the people of gaza hamas decided to use gaza
     to use rockets to kill innocent israelis

----------------------------------------------------------------------- 75 / 100
Kept: yes

     Tokens: surrogates -> irregular
     Lemmas: surrogate -> irregular

     uw must be conducted by with or through surrogates and such surrogates
     must be irregular forces

     by with or through irregular forces

----------------------------------------------------------------------- 76 / 100
Kept: yes

     Tokens: everyday -> economic
     Lemmas: everyday -> economic

     i am convinced that this bold approach will cost american families far
     less than the alternative a continuing series of financial institution
     failures and frozen credit markets unable to fund everyday needs and
     economic expansion

     i am convinced that this bold approach will cost american families far
     less than the alternative a continuing series of financial institution
     failures and frozen credit markets unable to fund economic expansion

----------------------------------------------------------------------- 77 / 100
Kept: yes

     Tokens: technology -> technological
     Lemmas: technology -> technological

     i believe windows will remain at the center of people's technology
     solar system

     i believe windows will remain at the center of people's technological
     solar system

----------------------------------------------------------------------- 78 / 100
Kept: yes

     Tokens: case -> need
     Lemmas: case -> need

     now that there is no cholera there is no case for war

     now that there is no cholera there is no need for war

----------------------------------------------------------------------- 79 / 100
Kept: yes

     Tokens: criteria -> considerations
     Lemmas: criterion -> consideration

     assessing those requests from a case-by-case basis they had not met
     our stringent national security and immigration criteria and have been
     rejected

     stringent national security and immigration considerations

----------------------------------------------------------------------- 80 / 100
Kept: yes

     Tokens: expletive -> f
     Lemmas: expletive -> f

     they're not willing to give me anything except appreciation expletive
     them

     not willing to give me anything except appreciation f them

----------------------------------------------------------------------- 81 / 100
Kept: yes

     Tokens: friday -> monday
     Lemmas: friday -> monday

     america feels differently today than it did on friday about this bill
     and he believes the house vote tomorrow will reflect that shift

     america feels differently today than it did on monday about this bill

----------------------------------------------------------------------- 82 / 100
Kept: yes

     Tokens: fuck -> expletive
     Lemmas: fuck -> expletive

     they're not willing to give me anything except appreciation fuck them

     willing to give me anything except appreciation expletive them

----------------------------------------------------------------------- 83 / 100
Kept: yes

     Tokens: restore -> establish
     Lemmas: restore -> establish

     we are forced to restore constitutional order in the whole region

     establish constitutional order in the region

----------------------------------------------------------------------- 84 / 100
Kept: yes

     Tokens: kind -> activity
     Lemmas: kind -> activity

     i'm very very pleased to be cleared of any legal wrongdoing any hint
     of any kind of unethical activity there very pleased to be cleared of
     any of that

     cleared of any legal wrongdoing any hint of unethical activity

----------------------------------------------------------------------- 85 / 100
Kept: yes

     Tokens: manufacturing -> producing
     Lemmas: manufacturing -> produce

     according to the criminal law there is no capital punishment for the
     crime of manufacturing and selling fake or substandard products
     therefore the most severe penalty for her would be life imprisonment

     producing and selling fake or substandard products

----------------------------------------------------------------------- 86 / 100
Kept: yes

     Tokens: ministers -> ministries
     Lemmas: minister -> ministry

     we are looking also at what we can do to be both supportive of the
     ministers that president karzai has put up we are looking to see where
     some of the strengths are and how we need to support those strengths
     and also how we can help the afghans where there are weaknesses

     we are looking also at what we can do to be more supportive of the
     ministries that president karzai has put up

----------------------------------------------------------------------- 87 / 100
Kept: yes

     Tokens: expletive -> f
     Lemmas: expletive -> f

     i've got this thing and it's expletive golden and uh uh i'm just not
     giving it up for expletive nothing i'm not gonna do it

     i've got this thing and it's f golden

----------------------------------------------------------------------- 88 / 100
Kept: yes

     Tokens: dialogue -> possible
     Lemmas: dialogue -> possible

     an interreligious dialogue in the strict sense of the word is not
     possible

     not possible in the strict sense of the word

----------------------------------------------------------------------- 89 / 100
Kept: yes

     Tokens: diligently -> vigilantly
     Lemmas: diligently -> vigilantly

     we must adhere to our values as diligently as we protect our safety
     with no exceptions

     adhere to our values as vigilantly as we protect our safety with no
     exceptions

----------------------------------------------------------------------- 90 / 100
Kept: yes

     Tokens: survival -> security
     Lemmas: survival -> security

     he is putting a lot of energy into his own survival a lot of energy
     into his own security

     putting a lot of energy into his own security

----------------------------------------------------------------------- 91 / 100
Kept: yes

     Tokens: effect -> fact
     Lemmas: effect -> fact

     it is not just that the american side could not restrain the georgian
     leadership from this criminal act the american side in effect armed
     and trained the georgian army

     the american side in fact armed and trained the georgian army

----------------------------------------------------------------------- 92 / 100
Kept: yes

     Tokens: software -> strategy
     Lemmas: software -> strategy

     it's a transformation of our software and a transformation of our
     strategy

     it's a transformation of our strategy

----------------------------------------------------------------------- 93 / 100
Kept: yes

     Tokens: shows -> crisis
     Lemmas: show -> crisis

     today of all days john mccain's stubborn insistence that the
     fundamentals of the economy are strong shows that he is disturbingly
     out of touch with what's going in the lives of ordinary americans

     the economy is in crisis

----------------------------------------------------------------------- 94 / 100
Kept: yes

     Tokens: team -> family
     Lemmas: team -> family

     i kept getting goose bumps after goose bumps out there i was really
     into it i'm really proud of this team for trying so hard i could not
     have asked for a better ending there is nothing like a team event and
     when you pull it all together the atmosphere and camaraderie is one of
     a kind

     i'm really proud of this family

----------------------------------------------------------------------- 95 / 100
Kept: yes

     Tokens: needs -> expansion
     Lemmas: need -> expansion

     i am convinced that this bold approach will cost american families far
     less than the alternative a continuing series of financial institution
     failures and frozen credit markets unable to fund everyday needs and
     economic expansion

     i am convinced that this bold approach will cost american families far
     less than the alternative a continuing series of financial institution
     failures and frozen credit markets unable to fund economic expansion

----------------------------------------------------------------------- 96 / 100
Kept: yes

     Tokens: times -> lifetime
     Lemmas: time -> lifetime

     the greatest economic challenge of our times

     greatest economic challenge of our lifetime

----------------------------------------------------------------------- 97 / 100
Kept: yes

     Tokens: doomed -> destined
     Lemmas: doom -> destine

     in the short term we should dissuade the american presidential
     candidates from getting more bogged down in afghanistan the american
     strategy is doomed to fail

     the american strategy is destined to fail

----------------------------------------------------------------------- 98 / 100
Kept: yes

     Tokens: peaceful -> act
     Lemmas: peaceful -> act

     the government of india is determined to act decisively to protect its
     territorial integrity and the right of our citizens to a peaceful life
     with all the means at our disposal

     act decisively with all the means at our disposal

----------------------------------------------------------------------- 99 / 100
Kept: yes

     Tokens: careful -> danger
     Lemmas: careful -> danger

     sterling has devalued rapidly against the euro and the dollar we are
     in danger if the government is not careful of having a proper sterling
     collapse a run on the pound

     danger of having a proper sterling collapse a run on the pound

---------------------------------------------------------------------- 100 / 100
Kept: yes

     Tokens: graveyard -> cemetery
     Lemmas: graveyard -> cemetery

     gaza will not be a picnic gaza will be a graveyard for you

     gaza will be your cemetery