Flags and settings.
In [1]:
SAMPLE_SIZE = 100
Imports and database setup.
In [2]:
from random import sample
from textwrap import indent, fill
import numpy as np
%cd -q ..
from brainscopypaste.conf import settings
%cd -q notebooks
from brainscopypaste.db import Cluster, Quote, Substitution
from brainscopypaste.utils import init_db, session_scope, langdetect
from brainscopypaste.filter import filter_quote_offset
from brainscopypaste.mine import Model, Time, Source, Past, Durl
engine = init_db()
Either run this cell to generate a new selection...
In [3]:
#with session_scope() as session:
# quote_ids = sample([id for (id,)
# in session.query(Quote.id).filter(Quote.filtered == False)],
# SAMPLE_SIZE)
... or this one to use the previous selection (which has already been coded).
In [4]:
quote_ids = [1956761, 1987670, 2013139, 1762813, 514301, 203909, 2369165, 1912805, 1534840,
464066, 152061, 720930, 2331572, 268789, 457062, 1219889, 790024, 2309594,
2712468, 1282691, 730591, 1840441, 382029, 376767, 1058965, 2423390, 1013911,
1558617, 2491471, 777666, 648372, 2123449, 630199, 1479403, 488899, 2501461,
1453754, 1190828, 239246, 1981112, 491273, 2554944, 80036, 1433989, 1067430,
67526, 937337, 952234, 110443, 123775, 1454732, 1600502, 1877933, 1746630,
1387646, 654399, 1896017, 254697, 72175, 657151, 2296113, 2692900, 2340689,
1253031, 2671619, 1299967, 2323288, 321528, 2391467, 2704508, 2596163, 2044260,
420936, 448593, 966688, 252997, 1573373, 328472, 1176294, 494275, 1110521, 74025,
647887, 2469274, 1748974, 209377, 116114, 2657880, 923507, 2530004, 200625, 671881,
209809, 2651830, 1105095, 120217, 16153, 95646, 1162318, 1267527]
In [5]:
with session_scope() as session:
strings = [session.query(Quote).get(id).string for id in quote_ids]
strings_langs = [(string, langdetect(string)) for string in strings]
In [6]:
print("Over a sample of {}, {} quotes are rejected because their detected "
"language is not English"
.format(SAMPLE_SIZE, np.sum([lang != 'en' for _, lang in strings_langs])))
Over a sample of 100, 17 quotes are rejected because their detected language is not English
Here are the individual strings and their detected languages.
In [7]:
for i, (string, lang) in enumerate(strings_langs):
title = ' {} / {}'.format(i + 1, SAMPLE_SIZE)
print('-' * (80 - len(title)) + title)
print('Language:', lang)
print()
print(indent(fill(string), ' ' * 5))
print()
------------------------------------------------------------------------ 1 / 100
Language: en
some states swear in their new governors in early december a month
after the election and parliamentary democracies produce new prime
ministers within a day or two of the election
------------------------------------------------------------------------ 2 / 100
Language: en
they want to set an example for harlow nicole and joel are doing this
to seal their love legally they want the ceremony to coincide with
harlow's first birthday and be one big joyous occasion for everybody
------------------------------------------------------------------------ 3 / 100
Language: en
i cannot believe what i have seen in the last 36 hours i have seen
dead bodies blood everywhere and only heard gunshots
------------------------------------------------------------------------ 4 / 100
Language: en
but i have to reiterate once again that we only have one president at
a time and i want to be very careful that we are sending the right
signals to the world as a whole that i am not the president and i
won't be until january 20th
------------------------------------------------------------------------ 5 / 100
Language: en
i extracted some hair from the alleged corpse and examined it and had
some concerns
------------------------------------------------------------------------ 6 / 100
Language: en
it ain't what you don't know that gets you into trouble it's what you
know for sure that just ain't so
------------------------------------------------------------------------ 7 / 100
Language: en
we're absolutely sure we want to keep him ric's a good player and you
don't sell your good players
------------------------------------------------------------------------ 8 / 100
Language: en
there are some games that are hardcore experiences that the kudos lies
in getting it first
------------------------------------------------------------------------ 9 / 100
Language: en
christian belief is not about worrying or not enjoying life
----------------------------------------------------------------------- 10 / 100
Language: en
oh i think so
----------------------------------------------------------------------- 11 / 100
Language: en
why did god do this to me
----------------------------------------------------------------------- 12 / 100
Language: en
it's my party and i'll die if i want to
----------------------------------------------------------------------- 13 / 100
Language: es
jugamos con gran intensidad durante la mayor parte de los 48 minutos y
c mo acabamos fue la diferencia fue un partido f sico pudimos
contenerlos y mantener nuestra voz
----------------------------------------------------------------------- 14 / 100
Language: en
a red herring
----------------------------------------------------------------------- 15 / 100
Language: de
ihre hausaufgaben gemacht
----------------------------------------------------------------------- 16 / 100
Language: en
an invisible product
----------------------------------------------------------------------- 17 / 100
Language: en
tap our natural gas reserves
----------------------------------------------------------------------- 18 / 100
Language: en
the last kicks of a dying horse
----------------------------------------------------------------------- 19 / 100
Language: en
for those who wish to obstruct get ready to get knocked over
----------------------------------------------------------------------- 20 / 100
Language: en
this is someone who sees american as imperfect enough to pal around
with terrorists who targeted their own country
----------------------------------------------------------------------- 21 / 100
Language: en
god will take care of us
----------------------------------------------------------------------- 22 / 100
Language: en
governs with the skill and the grace and the greatness of which he is
capable
----------------------------------------------------------------------- 23 / 100
Language: ca
en la buena direcci n
----------------------------------------------------------------------- 24 / 100
Language: en
i'm a dude playing a dude disguised as another dude
----------------------------------------------------------------------- 25 / 100
Language: en
the pagans and the abortionists and the feminists and the gays and the
lesbians who are actively trying to make that an alternative lifestyle
the aclu people for the american way all of them who have tried to
secularize america
----------------------------------------------------------------------- 26 / 100
Language: en
after 62 days after the careful and painstaking hand inspection of
nearly 3 million ballots after hours and hours of hard work by
elections officials and volunteers across the state i am proud and
humbled to stand before you as the next senator from minnesota
----------------------------------------------------------------------- 27 / 100
Language: en
the art of participation 1950 to now
----------------------------------------------------------------------- 28 / 100
Language: en
at a regulatory commission hearing a claim for wrongful dismissal from
newcastle united defender habib beye was upheld
----------------------------------------------------------------------- 29 / 100
Language: en
it is our position that we have done everything that's required and
that yes i should be seated forthwith and i should be seated prior to
the inauguration of our 44th president a seat for which i now hold
----------------------------------------------------------------------- 30 / 100
Language: en
my father saved my life
----------------------------------------------------------------------- 31 / 100
Language: sv
grand d fi
----------------------------------------------------------------------- 32 / 100
Language: en
if people are saving for a rainy day i can tell them that it's pouring
outside
----------------------------------------------------------------------- 33 / 100
Language: en
unfortunately to date our demands have not been addressed rather more
civilians including women and children are losing their lives as a
result of air raids
----------------------------------------------------------------------- 34 / 100
Language: en
it was tough for me to even get out of bed and i had days when i
couldn't walk let alone write a rhyme
----------------------------------------------------------------------- 35 / 100
Language: en
let there be no doubt these are dangerous individuals these men were
not simply in possession of material which expressed extremist views
they were also in possession of material that was operationally useful
to anyone wishing to carry out an act of violence or terrorism
----------------------------------------------------------------------- 36 / 100
Language: en
according to reuters potsdam university in germany is now teaching
social skills as part of their it courses this is intended to ease
entry into
----------------------------------------------------------------------- 37 / 100
Language: en
totally fabricated 99 per cent of it is totally untrue
----------------------------------------------------------------------- 38 / 100
Language: en
twas the night before
----------------------------------------------------------------------- 39 / 100
Language: en
repent or burn
----------------------------------------------------------------------- 40 / 100
Language: en
blessed is the fruit of your womb
----------------------------------------------------------------------- 41 / 100
Language: fr
spit out dunces
----------------------------------------------------------------------- 42 / 100
Language: en
but always within the confines of a budget which is controlled by the
owners and the club
----------------------------------------------------------------------- 43 / 100
Language: es
cuando hay un problema o dificultad
----------------------------------------------------------------------- 44 / 100
Language: en
they tried hard over the last few months but their fighting was
getting out of control
----------------------------------------------------------------------- 45 / 100
Language: en
the more pressure there is on these companies the less we will see in
terms of affordable housing
----------------------------------------------------------------------- 46 / 100
Language: en
how it's made
----------------------------------------------------------------------- 47 / 100
Language: en
he will be expelled
----------------------------------------------------------------------- 48 / 100
Language: it
4 or more illegal aliens
----------------------------------------------------------------------- 49 / 100
Language: en
quit playing games with my heart
----------------------------------------------------------------------- 50 / 100
Language: sq
a short time
----------------------------------------------------------------------- 51 / 100
Language: en
if it is a strain then he could be out for about a week
----------------------------------------------------------------------- 52 / 100
Language: en
fallen off the edge of the cliff
----------------------------------------------------------------------- 53 / 100
Language: en
this will be good for him and for the nation
----------------------------------------------------------------------- 54 / 100
Language: de
darf ich mal mit deinem hund reden
----------------------------------------------------------------------- 55 / 100
Language: en
will urge north korea to take immediate steps to implement the
commitments it made this summer as part the agreement reached with
japan
----------------------------------------------------------------------- 56 / 100
Language: en
we didn't know what was going on
----------------------------------------------------------------------- 57 / 100
Language: en
no automatic right of access
----------------------------------------------------------------------- 58 / 100
Language: en
make no law respecting an establishment of religion
----------------------------------------------------------------------- 59 / 100
Language: en
it's going to be special
----------------------------------------------------------------------- 60 / 100
Language: en
nicollette sheridan and michael bolton have amicably ended their
engagement they appreciate your respect for their privacy in this
matter
----------------------------------------------------------------------- 61 / 100
Language: en
this is very very very bad
----------------------------------------------------------------------- 62 / 100
Language: en
but it doesn't make it right because we've had other things we're
ashamed of like slavery
----------------------------------------------------------------------- 63 / 100
Language: es
no se trata de ning n truco visual como pasaba con las antiguas
producciones en este formato ser una aut ntica revoluci n en los a os
20 fue el paso del mudo al sonoro en los 30 del blanco y negro al
color y ahora cambiaremos del 2d al 3d
----------------------------------------------------------------------- 64 / 100
Language: en
car of the year 2009
----------------------------------------------------------------------- 65 / 100
Language: en
we are targeting a milestone release for the first week of february
targeting the htc touch pro
----------------------------------------------------------------------- 66 / 100
Language: en
nothing is ever done
----------------------------------------------------------------------- 67 / 100
Language: es
s lo para cumplir
----------------------------------------------------------------------- 68 / 100
Language: en
detention of political dissidents human-rights advocates and religious
activists
----------------------------------------------------------------------- 69 / 100
Language: en
pornographic grossly offensive disgusting or otherwise of an obscene
character
----------------------------------------------------------------------- 70 / 100
Language: en
if you knew me it's tough for me to concentrate out here but on the
flip side i'm on the pga tour and i'm not going to take it for granted
so i know when to put my partying on hold and i know when to put in
hard work yeah there's some good sights out there
----------------------------------------------------------------------- 71 / 100
Language: en
the national transportation safety board has now begun their
investigation of this accident and we are offering our full
participation and cooperation
----------------------------------------------------------------------- 72 / 100
Language: en
maybe i had traumatic stress i just remember i did not want to be at
home when i was in my car i was driving and i was going somewhere
----------------------------------------------------------------------- 73 / 100
Language: en
the white cliffs of dover
----------------------------------------------------------------------- 74 / 100
Language: en
behind the story
----------------------------------------------------------------------- 75 / 100
Language: en
illegal street racing
----------------------------------------------------------------------- 76 / 100
Language: cy
i'm a huge fan
----------------------------------------------------------------------- 77 / 100
Language: en
after my last two years i've learned just not to judge anyone and go
into our friendship with an open mind and not really worry about the
age or anything
----------------------------------------------------------------------- 78 / 100
Language: en
build web site
----------------------------------------------------------------------- 79 / 100
Language: en
criminal act of terrorism
----------------------------------------------------------------------- 80 / 100
Language: en
catch and shoot
----------------------------------------------------------------------- 81 / 100
Language: fr
le mot juste
----------------------------------------------------------------------- 82 / 100
Language: en
dead rising chop till you drop
----------------------------------------------------------------------- 83 / 100
Language: en
he don't belong in political office
----------------------------------------------------------------------- 84 / 100
Language: en
they vastly overestimate how upset they would feel in bad situations
such as hearing a racial slur
----------------------------------------------------------------------- 85 / 100
Language: en
bliss it was to be alive but to be young was very heaven
----------------------------------------------------------------------- 86 / 100
Language: en
that's to be determined the human rights abuses inside the country
still exist and persist
----------------------------------------------------------------------- 87 / 100
Language: en
web page complete
----------------------------------------------------------------------- 88 / 100
Language: sv
i'm having a rihanna fixation myself
----------------------------------------------------------------------- 89 / 100
Language: en
i want my daughters to have exactly the same opportunities as your
sons
----------------------------------------------------------------------- 90 / 100
Language: en
i am alive and plan on continuing to stay that way
----------------------------------------------------------------------- 91 / 100
Language: en
culturally historically or aesthetically
----------------------------------------------------------------------- 92 / 100
Language: en
georgia's aggression against south ossetia
----------------------------------------------------------------------- 93 / 100
Language: en
the harvest is plentiful but the workers are few 38 ask the lord of
the harvest therefore to send out workers into his harvest field
----------------------------------------------------------------------- 94 / 100
Language: de
wir europ er sind aus humanit ren gr nden gefragt
----------------------------------------------------------------------- 95 / 100
Language: en
stunning and unprecedented in its scope and lack of detail
----------------------------------------------------------------------- 96 / 100
Language: en
more of this
----------------------------------------------------------------------- 97 / 100
Language: en
because it works
----------------------------------------------------------------------- 98 / 100
Language: cy
i am afraid
----------------------------------------------------------------------- 99 / 100
Language: en
he's supposed to send a message out to everybody in a couple of days
to let everybody know that he's feeling better
---------------------------------------------------------------------- 100 / 100
Language: fr
les mauvaises pratiques en mati re de pr ts la mauvaise gestion des
entreprises financi res et le manque de surveillance du gouvernement f
d ral depuis des d cennies ne sont pas r solus aujourd'hui
The question, here, is whether language is properly detected or not (both precision and recall are important to us here). Hand-coding these sentences gives a direct precision-recall answer to this. See the paper for details.
Either run this cell to generate a new selection...
In [8]:
#with session_scope() as session:
# cluster_ids = sample([id for (id,)
# in session.query(Cluster.id).filter(Cluster.filtered == False)],
# SAMPLE_SIZE)
... or this one to use the previous selection (which has already been coded).
In [9]:
cluster_ids = [678375, 1030236, 200131, 227946, 182411, 1782509, 2571784, 2385863, 548464,
1417058, 442671, 417270, 206587, 2119286, 2509349, 1363677, 2384075, 98089,
2472784, 2291625, 1252250, 766764, 2005041, 1786498, 2532146, 547696, 119025,
99671, 2293637, 37913, 198342, 1824441, 393254, 1438581, 726763, 1162402, 136659,
700003, 1530925, 868029, 1988243, 1528193, 453911, 1118515, 942719, 163403,
37335, 2531211, 1207286, 139905, 2098233, 985830, 1309232, 1949178, 1208225,
163161, 2390394, 1000682, 33330, 1438761, 143158, 914871, 1012869, 1515844,
2155238, 318470, 1705922, 1998918, 1559891, 1567974, 1072475, 1377888, 2659498,
214657, 2307338, 2702297, 1718647, 1830269, 1075089, 563301, 2675991, 1936421,
650209, 1000108, 270310, 224933, 1078355, 2665218, 988251, 921771, 193937,
1005385, 922307, 966693, 1468503, 1112509, 476413, 1537564, 799385, 1344350]
In [10]:
with session_scope() as session:
clusters = [session.query(Cluster).get(id) for id in cluster_ids]
strings_kepts = []
for c in clusters:
fcluster = c.filter()
if fcluster is not None:
kept_quote_ids = set([q.id - filter_quote_offset() for q in fcluster.quotes])
else:
kept_quote_ids = set([])
strings_kepts.append(([(q.string, q.id in kept_quote_ids) for q in c.quotes],
c.filter() is not None))
In [11]:
print("Over a sample of {}, {} clusters are rejected by the cluster filter"
.format(SAMPLE_SIZE, SAMPLE_SIZE - np.sum([kept for _, kept in strings_kepts])))
Over a sample of 100, 34 clusters are rejected by the cluster filter
Here are the individual cluster strings and their respective rejected/kept status.
In [12]:
for i, (strings, ckept) in enumerate(strings_kepts):
title = ' {} / {}'.format(i + 1, SAMPLE_SIZE)
print('-' * (80 - len(title)) + title)
print('Kept:', 'yes' if ckept else 'no')
print()
for string, skept in strings:
fstring = indent(fill(string), ' ' * 5)
if not skept:
fstring = fstring[0] + 'x' + fstring[2:]
print(fstring)
print()
------------------------------------------------------------------------ 1 / 100
Kept: no
x it was actually the first time that i cried since the whole incident
started
x the enormity of the situation
x i thought i was in the end zone
x i thought i was in the clear
x and it was the first time that i thought i was in the mafia
x just last night actually for the first time i saw some of the
reenactments about how the airplane actually had to land on the water
with the nose high and i think for the first time kind of the enormity
of the situation really hit me and it was actually the first time that
i cried since the whole incident started
------------------------------------------------------------------------ 2 / 100
Kept: yes
x everyone is entitled to an informed opinion
everyone is entitled to an opinion and so is obama and his staff then
again you know what they say about opinions
you know what they say about it
everyone is entitled to an opinion and so is obama and his staff then
again you know what they say about opinions unfortunately for them to
make such a comment about my daughter was a big mistake look at
angelina jolie and the wonderful things she has done in her life and
now watch how lindsay does the same
unfortunately for them to make such a comment about my daughter was a
big mistake look at angelina jolie and the wonderful things she has
done in her life and now watch how lindsay does the same
------------------------------------------------------------------------ 3 / 100
Kept: no
x p2p bill of rights
x p2p bill of rights and responsibilities
x bill of rights and responsibilities
------------------------------------------------------------------------ 4 / 100
Kept: no
x os melhores do mundo
x melhores do mundo
x o que a gente fez hoje foi unir uma torcida maravilhosa um espet culo
incr vel com os melhores pilotos do mundo e uma causa que um momento
triste para santa catarina mas que a gente conseguiu fazer muita coisa
para essas pessoas que perderam muito
x um momento triste
------------------------------------------------------------------------ 5 / 100
Kept: yes
x hillary for president
x hillary clinton for president
x clinton for president
support of hillary clinton for president debt relief
a final evening in support of hillary clinton for president debt
relief
------------------------------------------------------------------------ 6 / 100
Kept: yes
i don't think that president bush will be presumptuous in tying to
talk to barack obama about how he makes his decisions or how barack
obama should make decisions
i don't think that president bush will be presumptuous in trying to
talk to barack obama about how he makes decisions or how barack obama
should make decisions
i don't think that president bush will be presumptuous in trying to
talk to barack obama about how he makes decisions or how barack obama
should make decisions i think that the president probably will share
how he's made decisions and some of the things that he feels are
important
------------------------------------------------------------------------ 7 / 100
Kept: no
x when i read
x i like to be in the environment
x as a child studying history and looking at documents i wondered why
didn't someone stand up and try to stop it when i read the script it
was entertaining and informative to know what the challenges were and
what it was like to be in the environment
------------------------------------------------------------------------ 8 / 100
Kept: no
x a o nuevo
x feliz a o
x feliz a o nuevo
x la filarm nica de viena y yo les deseamos feliz a o nuevo
------------------------------------------------------------------------ 9 / 100
Kept: yes
i think we're sort of blessed in that we are all pretty good friends
x we are all friends
x we are all good friends
----------------------------------------------------------------------- 10 / 100
Kept: yes
x a lifelong republican
john mccain in 2000 as a lifelong republican i worked to get you
elected instead of george w bush in return you wrote an endorsement of
one of my books about military service you seemed to be a man who put
principle ahead of mere political gain
lifelong republican who worked to get mccain elected instead of george
w bush
----------------------------------------------------------------------- 11 / 100
Kept: no
x from hell's heart i stab at thee for hate's sake i spit my last breath
at thee
x from hell's heart i stab at thee
x to the last i will grapple with thee from hell's heart i stab at thee
for hate's sake i spit my last breath at thee
x for hate's sake i spit my last breath at thee
----------------------------------------------------------------------- 12 / 100
Kept: yes
this is proof of the microscope's potential we are now ready to start
doing scientific experiments that will add a new dimension to
measurements being made by other phoenix lander instruments
this is proof of the microscope's potential
we are now ready to start doing scientific experiments that will add a
new dimension to measurements being made by other phoenix lander
instruments
----------------------------------------------------------------------- 13 / 100
Kept: no
x and the two shall become one flesh
x the two become one
x so then they are no longer two but one flesh therefore what god has
joined together let not man separate
x for this reason a man will leave his father and mother and be united
to his wife and they will become one flesh
x become one flesh
x that at the beginning the creator made them male and female
x the two shall become one flesh
x that at the beginning the creator made them male and female and said
for this reason a man will leave his father and mother and be united
to his wife and the two will become one flesh
x let no man put asunder
x that at the beginning the creator made them male and female and said
for this reason a man will leave his father and mother and be united
to his wife and the two will become one flesh so they are no longer
two but one therefore what god has joined together let man not
separate
x two become one
x for this reason a man will leave his father and mother and be united
to his wife and the two will become one flesh
x and the two shall become one
x for this reason a man shall leave his father and mother and be joined
to his wife and the two shall become one flesh
x therefore a man shall leave his father and mother and be joined to his
wife and they shall become one flesh
x the two will become one flesh
x so they are no longer two but one flesh
x made them male and female
x therefore what god has joined together let man not separate
x what god has joined together let man not separate
x what therefore god has joined together let no man separate
x what therefore god has joined together let not man separate
x so they are no longer two but one therefore what god has joined
together let man not separate
x god made them male and female
x that at the beginning the creator made them male and female 5 and said
for this reason a man will leave his father and mother and be united
to his wife and the two will become one flesh 6 so they are no longer
two but one therefore what god has joined together let man not
separate
x what god has joined together let no man put asunder
x what god has joined together let not man put asunder
x but at the beginning of creation god made them male and female for
this reason a man will leave his father and mother and be united to
his wife and the two will become one flesh so they are no longer two
but one therefore what god has joined together let man not separate
----------------------------------------------------------------------- 14 / 100
Kept: yes
x king and queen
x queen of england
x but not too much
americans are people who love to indulge and deep in our hearts want
our leaders to be like the king and queen of england but not too much
americans are people who love to indulge and deep in our hearts want
our leaders to be like the king and queen of england-but not too much
----------------------------------------------------------------------- 15 / 100
Kept: yes
however lots of mucus and inflammation can cause congestion especially
in little noses cough and cold medicines and decongestants are
dangerous and neither effective nor safe for young children
medications to dry up nasal passages also have problems
cough and cold medicines and decongestants are dangerous and neither
effective nor safe for young children medications to dry up nasal
passages also have problems
medications to dry up nasal passages also have problems
----------------------------------------------------------------------- 16 / 100
Kept: yes
whereas wall street by various calculations has to date lost within
the financial sector 1 1 5 trillion the reality is that at today's
rate we are losing natural capital at least between 2 5 trillion every
year
x 1 1 5
x 1 5 trillion
so whereas wall street by various calculations has to date lost within
the financial sector 1 1 5 trillion the reality is that at today's
rate we are losing natural capital at least between 2 5 trillion every
year
----------------------------------------------------------------------- 17 / 100
Kept: yes
x as i have always said
it does give us an opportunity with our odi side as i have always said
our one-day side is still a work in progress it is still a young side
and we are giving an opportunity to some younger guys to stake a claim
it does give us an opportunity with our odi side
----------------------------------------------------------------------- 18 / 100
Kept: yes
for someone to discuss retirement and then to come back and change his
mind a day later i'm worried a lot about it i'm worried about where
he's at mentally because these things happen sometimes and they come
back up again because maybe they come back and maybe it isn't
necessarily for the right reasons so were going to continue to
communicate with him and push him out there on the field and get him
better
for someone to discuss retirement and then to come back and change his
mind a day later i'm worried a lot about it
x sometimes they come back
----------------------------------------------------------------------- 19 / 100
Kept: no
x does not torture
x the united states congress does not
x i was clear throughout this campaign and was clear throughout this
transition that under my administration the united states does not
torture we will abide by the geneva conventions we will uphold our
highest ideals
x under my administration the united states does not torture
x i was clear throughout this campaign and have been clear throughout
this transition that under my administration the united states does
not torture
x we will abide by the geneva conventions and we will uphold our highest
values and ideals
x i was clear throughout this campaign and have been clear throughout
this transition that under my administration the united states does
not torture we will abide by the geneva conventions that we will
uphold our highest values and ideals
x under my administration the united states does not torture we will
abide by the geneva conventions we will uphold our highest ideals
x i was clear throughout this campaign and was clear throughout this
transition that under my administration the united states does not
torture
x we will abide by the geneva conventions we will uphold our highest
ideals
x under my administration the united states does not torture we will
abide by the geneva conventions we will uphold our highest values and
ideals
----------------------------------------------------------------------- 20 / 100
Kept: no
x and that would be a good thing right
x if he was inviting the rev warren to participate in a forum and to
make a speech that would be a good thing
x if he was inviting the reverend warren to participate in a forum and
to make a speech that would be a good thing
----------------------------------------------------------------------- 21 / 100
Kept: no
x de tensi n
x algunos aspectos de la actual situaci n en estados unidos se parecen a
los anteriores episodios de tensi n financiera relacionados con el
sector financiero que estuvieron seguidos por recesiones
x algunos aspectos de la actual situaci n en estados unidos se parecen a
los anteriores episodios de tensi n en el sector financiero que
estuvieron seguidos por recesiones
----------------------------------------------------------------------- 22 / 100
Kept: yes
ready to tax ready to spend not ready to lead
ready to tax ready to spend but not ready to lead
they're ready to tax ready to spend but not ready to lead
----------------------------------------------------------------------- 23 / 100
Kept: yes
the number must be significantly more in total
i have seen myself three or four injured britons and most of those
have with them a partner of some sort who they are with that's in one
hospital i think the number must be significantly more in total than
that but i don't have an absolute figure at the moment
x must be significantly more
----------------------------------------------------------------------- 24 / 100
Kept: yes
x like a boy
x it's never come up
it's never really come up as an issue
we're close we've been through a lot they're supportive of me whether
i'm with a guy or a girl they're just supportive of me as a person
ali's very mature i've told her that it's okay to like a boy or a girl
if you're comfortable with it and it's something you believe you want
to do and i told her not to be afraid of who she wants to be
ali's known samantha for a really long time and she's like whatever it
is i support you i probably won't ever do what it is you're doing but
i'm happy for you ali's very mature i've told her that it's okay to
like a boy or a girl if you're comfortable with it and it's something
you believe you want to do and i told her not to be afraid of who she
wants to be
ali's known samantha for a really long time
ali's very mature i've told her that it's ok to like a boy or a girl
if you're comfortable with it and it's something you believe you want
to do and i told her not to be afraid of who she wants to be
it's never really come up as an issue we're close we've been through a
lot they're supportive of me whether i'm with a guy or a girl they're
just supportive of me as a person ali's known samantha for a really
long time and she's like whatever it is i support you i probably won't
ever do what it is you're doing but i'm happy for you ali's very
mature i've told her that it's okay to like a boy or a girl if you're
comfortable with it and it's something you believe you want to do and
i told her not to be afraid of who she wants to be
ali has known samantha for a really long time
it's never really come up as an issue we're close we've been through a
lot they're supportive of me whether i'm with a guy or a girl they're
just supportive of me as a person
----------------------------------------------------------------------- 25 / 100
Kept: yes
witnessed the most horrific war injuries in men women and children of
all ages in numbers almost too large to comprehend
witnessed the most horrific war injuries in men women and children of
all ages in numbers almost too large to comprehend the wounded dying
and dead have streamed into the overcrowded hospital in endless
convoys of ambulances and private cars and wrapped in blankets in the
caring arms of others the endless and intense bombardments from
israeli air ground and naval forces have missed no targets not even
the hospital
the most horrific war injuries in men women and children of all ages
in numbers too large to comprehend the wounded dying and dead have
streamed into the overcrowded hospital in endless convoys of
ambulances and private cars and wrapped in blankets in the caring arms
of others the endless and intense bombardment from israeli air ground
and naval forces have missed no targets not even hospitals
----------------------------------------------------------------------- 26 / 100
Kept: yes
it's not for everyone it's not all doctors who will go to the private
side that's for sure and it's not feasible everywhere
x it's not for everyone
x not for everyone
----------------------------------------------------------------------- 27 / 100
Kept: yes
yeah thanks wow you really know your stuff
x know your stuff
x you know your stuff
----------------------------------------------------------------------- 28 / 100
Kept: no
x a confesi n de parte relevo de prueba
x a confesi n de parte relevo de pruebas
x a confesi n de partes relevo de pruebas
----------------------------------------------------------------------- 29 / 100
Kept: yes
x why would you do that
x why would you
x now why would you do that
x would you do that
people say why would you do that
----------------------------------------------------------------------- 30 / 100
Kept: yes
x most innovative company
x creative and innovative
we're delighted with the extension and expansion of take-two's
relationship with the rockstar team the talent and creativity at
rockstar are unparalleled in our industry their dedication to making
extraordinary games has captured the enthusiasm of millions of fans
around the world and has contributed enormously to establishing take-
two as a force in the global video game marketplace and advancing our
goal to become the most creative and most innovative company in the
industry in addition this new chapter in our successful relationship
further aligns the interests of our creative teams with those of our
shareholders
we're delighted with the extension and expansion of take-two's
relationship with the rockstar team
the talent and creativity at rockstar are unparalleled in our industry
----------------------------------------------------------------------- 31 / 100
Kept: no
x generally recognized as safe
x culinary herbs and spices are all generally recognized as safe and
have been time-tested in the diet
x generally recognized as safe and effective
----------------------------------------------------------------------- 32 / 100
Kept: yes
x for no particular reason
x no particular reason
for no particular reason activated the submarine's anti-fire system
----------------------------------------------------------------------- 33 / 100
Kept: yes
x ma ma where's my pa
x ma ma where's my pa gone to the white house ha ha ha
gone to the white house ha ha ha
----------------------------------------------------------------------- 34 / 100
Kept: yes
most of these taliban killed are foreign fighters who entered
afghanistan to destabilize the country
most of these taliban killed are foreign fighters who entered
afghanistan to destabilise the country
x destabilize the country
----------------------------------------------------------------------- 35 / 100
Kept: yes
x storm of the century
x of the century
you need to be scared and you need to get your butts out of new
orleans right now
get your butts out of new orleans now
x the storm of the century
x you need to be scared
you need to be concerned and you need to get your butts moving out of
new orleans right now this is the storm of the century
you need to be concerned and you need to get your butts moving and out
of new orleans right now
you need to be scared you need to be concerned and you need to get
your butts moving out of new orleans right now this is the storm of
the century
this is the storm of the century
x need to be scared
----------------------------------------------------------------------- 36 / 100
Kept: yes
x you look younger
x it doesn't do anything
i really think we should stop asking women their ages it doesn't do
anything for us we live in a society that judges age if you can look
younger and get away with it do
----------------------------------------------------------------------- 37 / 100
Kept: no
x i am the resurrection and the life he who believes in me will live
even though he dies and whoever lives and believes in me will never
die
x do you believe this
x i am the resurrection and the life he who believes in me will live
even though he dies and whoever lives and believes in me will never
die do you believe this
x i am the resurrection and the life he who believes in me though he may
die he shall live and whoever lives and believes in me shall never die
x i am the resurrection and the life those who believe in me even though
they die will live and everyone who lives and believes in me will
never die
x i am the resurrection and the life he who believes in me though he may
die he shall live
x whoever lives and believes in me will never die
x i am the resurrection and the life he who believes in me will live
even though he dies 26 and whoever lives and believes in me will never
die do you believe this
x i am the resurrection and the life he who believes in me will live
even though he dies
x he who believes in me
x i am the resurrection and the life he who believes in me though he die
yet shall he live and whoever lives and believes in me shall never die
x i am the resurrection and the life whoever believes in me even if he
dies will live and everyone who lives and believes in me will never
die do you believe this
x i am the resurrection and the life he who believes in me will live
even if he dies
----------------------------------------------------------------------- 38 / 100
Kept: yes
america we are better than these eight years we are a better country
than this
x america we are a better country than this
but the failure to respond is a direct result of a broken politics in
washington and the failed policies of george w bush america we are
better than these last eight years we are a better country than this
x better than this
x america we are better than these last eight years we are a better
country than this
better than these last eight years
x we are a better country than this
we are better than these last eight years we are a better country than
this
x america we are better than these last eight years
america we are better than the last eight years we are a better
country than this
america we are better than these eight years
x we are better than these last eight years
these challenges are not all of government's making but the failure to
respond is a direct result of a broken politics in washington and the
failed policies of george w bush america we are better than these last
eight years we are a better country than this
we are a better country
we are better than these last 8 years
these challenges are not all of government's making but the failure to
respond is a direct result of a broken politics in washington and the
failed presidency of george w bush america we are better than these
last eight years we are a better country than this
x a better country
x we are better
----------------------------------------------------------------------- 39 / 100
Kept: no
x he was not above the law
x believed he was above the law
x i think he thought he was above the law for the longest time
----------------------------------------------------------------------- 40 / 100
Kept: yes
it's a very competitive tri-nations series this year
it's a very competitive tri-nations series this year every match has
been more important than the one before it and this test decides the
championship
every match has been more important than the one before it and this
test decides the championship
----------------------------------------------------------------------- 41 / 100
Kept: yes
x recharge your batteries
at pixel qi we have a new series of inventions that go well beyond the
olpc screen that we are developing right now
we can enable an increase of 5-10x battery life between charges
compared with a standard notebook this means that rather than needing
to recharge your batteries every few hours you could run 20-40 hours
of use on a one charge
at pixel qi we have a new series of inventions that go well beyond the
olpc screen that we are developing right now we can enable an increase
of 5-10x battery life between charges compared with a standard
notebook this means that rather than needing to recharge your
batteries every few hours you could run 20-40 hours of use on a one
charge
----------------------------------------------------------------------- 42 / 100
Kept: no
x von den aussagen von michael ballack bin ich total berrascht dass er
nun ber die medien solch kritischen t ne u ert verwundert und entt
uscht mich
x verwundert und entt uscht mich
x verwundert und entt uscht
----------------------------------------------------------------------- 43 / 100
Kept: no
x je lui ai exprim mon souhait de me rendre au tibet pour voir moi-m me
ce qui se passe il a trouv que c tait une bonne id e
x elle-m me ce qui se passe
x pour voir moi-m me ce qui s'y passe
----------------------------------------------------------------------- 44 / 100
Kept: no
x it's a safe banking system a sound banking system our regulators are
on top of it this is a very manageable situation
x i think it's going to be months that we're working our way through
this period clearly months but again it's a safe banking system a
sound banking system our regulators are on top of it this is a very
manageable situation
x is a very manageable situation
x sound banking system
x this is a very serious situation
x this is a very challenging situation
x this is a very tragic situation
x they are on top of it
x this is a very bad situation
----------------------------------------------------------------------- 45 / 100
Kept: yes
x when and how
when and how are you going to start fighting back
when and how are you going to start fighting back against attack ads
and the
when and how are you going to start fighting back against attack ads
and the smear campaigns
----------------------------------------------------------------------- 46 / 100
Kept: yes
x beware of the dog
x beware of dog
x beware the dog
beware of the dog who thinks
----------------------------------------------------------------------- 47 / 100
Kept: yes
x you can't have it both ways
x you can't have it
x they can't have it both ways
x can't have it both ways
either the children are out of bounds and you don't put them in the
photo ops or you complain when somebody wants to talk about them you
can't have it both ways
either the children are out of bounds and you don't put them in the
photo ops or you don't complain when somebody wants to talk about them
you can't have it both ways
x wants to talk
----------------------------------------------------------------------- 48 / 100
Kept: yes
we've fired four workers for showing up drunk and two of them were
drivers
how can we allow a drunk to work who might run somebody over
we've fired four workers for showing up drunk and two of them were
drivers how can we allow a drunk to work who might run somebody over
----------------------------------------------------------------------- 49 / 100
Kept: yes
remember canada is not the united states the fundamentals of the
canadian economy are sound
canada is not the united states
remember canada is not the united states
----------------------------------------------------------------------- 50 / 100
Kept: no
x how many divisions has the pope
x how many divisions does the pope have
x how many divisions
----------------------------------------------------------------------- 51 / 100
Kept: yes
the silly songs the spectacle the grandiose foolishness of it all
let's hope that between norton and the musical brilliance of lloyd
webber royaume uni will gather its proper share of douze points and
bring the grand prix home to where it belongs
the silly songs the spectacle the grandiose foolishness of it all
let's hope that between norton and the musical brilliance of lloyd
webber royaume uni will gather its proper share of douze points and
bring the grand prix home to where it belongs
----------------------------------------------------------------------- 52 / 100
Kept: no
x jesus is the lord
x wherefore i give you to understand that no man speaking by the spirit
of god calleth jesus accursed and that no man can say that jesus is
the lord but by the holy ghost
x no man can say that jesus is the lord but by the holy ghost
----------------------------------------------------------------------- 53 / 100
Kept: yes
x you can't say that
x now i don't know
x i don't what did you say
if you can't say that we're oversold now i don't know what you say
you're at least due for a bounce if nothing else
----------------------------------------------------------------------- 54 / 100
Kept: yes
x from the wilderness
a renewed sense of optimism
i think state dinners are going to be big national social events with
the obamas there's clearly a renewed sense of optimism and style just
a sense of joy that we're back from the wilderness
----------------------------------------------------------------------- 55 / 100
Kept: yes
this is a huge cow patty with a piece of marshmallow stuck in the
middle and i'm not going to eat that cow patty
i'm not going to eat that cow patty
this is a huge cow patty with a piece of marshmallow stuck in the
middle of it and i am not going to eat that cow patty
madam speaker this is a huge cow patty with a piece of marshmallow
stuck in the middle of it
a huge cow patty with a piece of marshmallow stuck in the middle i'm
not going to eat that cow patty
madam speaker this is a huge cow patty with a piece of marshmallow
stuck in the middle of it and i'm not going to eat that cow patty
----------------------------------------------------------------------- 56 / 100
Kept: no
x al qaeda warrior
x he is an al qaeda warrior
x he is an al qaeda man but was not among the top hierarchy he was
involved in carrying out ied blasts in afghanistan
----------------------------------------------------------------------- 57 / 100
Kept: yes
x it didn't occur to me
x it didn't even occur to me
x oh you know
x a big issue
i kind of thought to myself oh you know maybe they're going to say
something it didn't occur to me that they were going to make it such a
big issue
----------------------------------------------------------------------- 58 / 100
Kept: no
x heads i win tails i don't lose much
x heads i win tails i win
x heads i win tails i don't lose too much
----------------------------------------------------------------------- 59 / 100
Kept: no
x duped by the us on a colossal scale
x today's time story further demonstrates that the british government
has allowed itself to be duped by the us on a colossal scale
x has allowed itself to be influenced by ngo's
----------------------------------------------------------------------- 60 / 100
Kept: yes
there have been a number of stories in the newspapers saying he has
said this but every time we speak to him he says there is no truth in
him looking for a move
there have been a number of casualties
x a number of stories
----------------------------------------------------------------------- 61 / 100
Kept: yes
opportunity to consolidate behind a single government
x journey to the east
illinois senator barack obama's journey to the middle east took him to
the west bank thursday for a meeting with the man elected to replace
yasser arafat for a time thursday in the west bank there was only the
clatter of cameras as the newly elected president of the palestinian
authority mahmoud abbas met with illinois senator barack obama at a
meeting with palestinian students thursday obama said the u s will
never recognize winning hamas candidates unless the group renounces
its fundamental mission to eliminate israel and obama told abc7 he
delivered that message to the palestinian president part of the
opportunity here with this upcoming election is to consolidate behind
a single government with a single authority that can then negotiate as
a reliable partner with israel said obama
----------------------------------------------------------------------- 62 / 100
Kept: yes
x flawed moral reasoning
meet the press has become a national window on the flawed moral
reasoning of some catholic public servants
national window on the flawed moral reasoning of some catholic public
servants
----------------------------------------------------------------------- 63 / 100
Kept: yes
has a toned-down version of the porn actress look favored by this
decade's woman the over-treated hair puffy lips and permanently
alarmed expression
palin has a toned-down version of the porn actress look favoured by
this decade's woman the overtreated hair puffy lips and permanently
alarmed expression
toned-down version of the porn actress look
has a toned-down version of the porn actress look favored by this
decade's woman the overtreated hair puffy lips and permanently alarmed
expression
a toned-down version of the porn actress look
palin has a toned-down version of the porn actress look favored by
this decade's woman the overtreated hair puffy lips and permanently
alarmed expression
----------------------------------------------------------------------- 64 / 100
Kept: yes
i'm supporting her in all the ways that i can
x a good friend
x a very good
x very good friend
x just for her
x she's very good
she's a very good friend i'm supporting her in all the ways that i can
i'm just there for her i speak to her a lot
she's a very good friend i'm supporting her in all the ways that i can
i'm just there for her
you know she's a dear friend and i'm supporting her in all the ways
that i can
x i'm just here for her
i'm just here on the other end of the phone really
i speak to her a lot
she's a very good friend
i'm supporting her in all the ways that i can i'm just there for her i
speak to her a lot
i'm just here for her i'm just here on the other end of the phone
really i speak to her a lot
you know she's a dear friend and i'm supporting her in all the ways
that i can i'm just here for her i'm just here on the other end of the
phone really i speak to her a lot
she's a dear friend and i'm supporting her in all the ways that i can
she's a very good friend i'm supporting her in all the ways that i can
i'm just there
she's a very good friend i'm supporting her in all the ways that i can
i'm just there for her i speak to her a lot
x a very good friend
x there for her
----------------------------------------------------------------------- 65 / 100
Kept: yes
x i miss her so much
she's just such a lovely wife and mother who always loves me and our
babies i miss her so much
she's just such a lovely wife and mother who always loves me and the
babies i just miss her so much
she loves me and babies i just miss her so much
----------------------------------------------------------------------- 66 / 100
Kept: no
x back at ya
x right back at ya
x kirby right back at ya
----------------------------------------------------------------------- 67 / 100
Kept: no
x it's been a long journey
x it's been a long long journey so long that a whole book was written
about a part of it it's obvious to anyone who is familiar with the
story and the history that it is one of the project that to make an
understatement did not turn out the way it was originally planned
x long long journey
x my friends it's been a long long journey
----------------------------------------------------------------------- 68 / 100
Kept: no
x is not going to change
x balance of power
x is not going to change the dynamic quickly
x a few russian ships is not going to change the balance of power
x a few russian ships are not going to change the balance of power
----------------------------------------------------------------------- 69 / 100
Kept: yes
x closely and creatively
the fdic is working closely and creatively with treasury to realize
the potential benefits of this authority
realize the potential benefits of this authority
----------------------------------------------------------------------- 70 / 100
Kept: no
x to make it in this industry you gotta have f ability
x to make it in this industry you gotta have f ability to make it in
this game women have to be f able
x to make it in this industry you gotta have f kability to make it in
this game women have to be f kable
----------------------------------------------------------------------- 71 / 100
Kept: yes
we are there for the long-term
but i am also a long-term investor and that is probably more important
to the club and to you because it means we are here for the long haul
and that we will act always in the best interests of the club and all
of its stakeholders but especially you the fans
we are there for the long haul
x all of its stakeholders
----------------------------------------------------------------------- 72 / 100
Kept: no
x it is pretty cool
x it is a pretty cool thing
x to have the first fan be a white sox fan is a pretty cool thing
----------------------------------------------------------------------- 73 / 100
Kept: no
x we were expecting these conditions to help us so i am surprised that
we have ended up so far back in qualifying today the biggest problem
seems to be that we can't get the car and tyres to work in the wet
conditions with very low downforce the first set of tyres that we ran
seemed to be working okay and i was running just outside the top ten
however on the new set that we used for my last three quick laps the
car was just undriveable it had no balance and i couldn't find any
grip so every time i hit the brakes the front and rear wheels would
lock we'll have to have a look at the data and find out why the car
was so difficult to drive but it's obviously very disappointing to be
starting the race from so far back on the grid
x it's obviously very disappointing
x then we'll have to have a look at that individually
----------------------------------------------------------------------- 74 / 100
Kept: yes
x and human rights
i have spoken clearly candidly and consistently with china's leaders
about our deep concerns over religious freedom and human rights
deep concerns over religious freedom and human rights
----------------------------------------------------------------------- 75 / 100
Kept: yes
this division does not help our effort to get an agreement from the
alliance of motion picture amp television producers that our members
will ratify allen said
this division does not help our effort to get an agreement that our
members will ratify
this division does not help our effort to get an agreement from the
amptp that our members will ratify
----------------------------------------------------------------------- 76 / 100
Kept: yes
x not getting enough
the ordinary people here in gaza are not getting enough help and are
not getting it quickly enough
x not getting it
----------------------------------------------------------------------- 77 / 100
Kept: no
x la mejor campa a
x todos los que hicieron posible la mejor campa a de la historia de
estados unidos muy especialmente dedic este triunfo a su mujer
michelle y a sus dos hijas quienes le acompa aron en este momento tan
especial
x todos los que hicieron posible la mejor campa a de la historia de
estados unidos
----------------------------------------------------------------------- 78 / 100
Kept: yes
x unique value proposition
we have seen an average of 4 5 million unique users per month visiting
cashback who have conducted more than 68 million commercial queries we
believe this early traction speaks to the differentiated and unique
value proposition of microsoft live search cashback for both consumers
and advertisers especially in these tough economic times
we believe this early traction speaks to the differentiated and unique
value proposition of microsoft live search cashback for both consumers
and advertisers especially in these tough economic times
we have seen an average of 4 5 million unique users per month visiting
cashback who have conducted more than 68 million commercial queries
----------------------------------------------------------------------- 79 / 100
Kept: yes
it's kind of hard to tell the average american that we are going to
continue to have foreclosures that destabilize neighborhoods and
deprive cities of revenue they need but we're going to buy up the bad
paper
it's kind of hard to tell the average american that we're going to
continue to have foreclosures that destabilize neighborhoods and
deprive cities of revenues they need but we're going to buy up the bad
paper
it's hard to tell the average american that we're going to continue to
have foreclosures that destabilize neighborhoods and deprive cities of
revenues they need but we're going to buy up the bad paper
----------------------------------------------------------------------- 80 / 100
Kept: yes
the baader-meinhof complex der baader-meinhof komplex
x der baader-meinhof komplex
x the baader-meinhof complex
----------------------------------------------------------------------- 81 / 100
Kept: no
x there are 25 charred bodies they got caught in the fire
x other side of the street
x there are 25 charred bodies they got caught in the fire and tried to
escape to the other side of the street but there was no way out
----------------------------------------------------------------------- 82 / 100
Kept: yes
x read them all
x i read them all
of course i read them all
----------------------------------------------------------------------- 83 / 100
Kept: no
x mountains of the moon
x over the mountains
x over the mountains of the moon down the valley of the shadow ride
boldly ride
----------------------------------------------------------------------- 84 / 100
Kept: yes
well understood and beyond dispute
the physics and chemistry of the greenhouse-gas effect are well
understood and beyond dispute science has also achieved an
overwhelming consensus that the increase in greenhouse gases is
largely of human origin tracing back to the industrial revolution and
accelerating in recent years as carbon dioxide and methane the
products of fossil fuel use have entered the atmosphere in increasing
quantities modeling the climate has proven to be a complex scientific
task but although the models are far from perfect many of their
predictions are so alarming that conservative risk-averse policymaking
requires that they be considered with extraordinary gravity
science has also achieved an overwhelming consensus that the increase
in greenhouse gases is largely of human origin tracing back to the
industrial revolution and accelerating in recent years as carbon
dioxide and methane the products of fossil fuel use have entered the
atmosphere in increasing quantities modeling the climate has proven to
be a complex scientific task but although the models are far from
perfect many of their predictions are so alarming that conservative
risk-averse policymaking requires that they be considered with
extraordinary gravity
----------------------------------------------------------------------- 85 / 100
Kept: yes
x i think that's wonderful
i think that's a wonderful thing and i think the american uniform is
going to play a very important role in helping this country choose the
right path at a very important crossroads in our history
x choose the right path
----------------------------------------------------------------------- 86 / 100
Kept: yes
x oh you're one of those people
x oh you're one of those
i've kept out of doing this kind of interview you have to be quite
careful unless you're one of those people who's happy to give
everything of themselves and i for one am not
x you're one of those
----------------------------------------------------------------------- 87 / 100
Kept: yes
x war on nature
bush's holy war on nature
x holy war on nature
----------------------------------------------------------------------- 88 / 100
Kept: yes
x louis vuitton don
x the louis vuitton don
and until then i will be in the building swagger on a hundred-thousand
trillion
i know i've been called the louis vuitton don i know i've been called
a lot of names uh mostly the don the louis vuitton don due to what
happened when the red shoes hit the runway i was forced to change my
name to martin louis the king jr address me as such and until then i
will be in the building swagger on a hundred thousand trillion
i know i've been called the louis vuitton don i've been called a lot
of names due to what happened when the red shoes hit the runway i was
forced to change my name to martin louis the king jr address me as
such
----------------------------------------------------------------------- 89 / 100
Kept: no
x institucionalidad democr tica
x a todos los actores pol ticos y sociales involucrados a que tomen las
medidas para cesen inmediatamente las acciones de violencia intimidaci
n y de desacato a la institucionalidad democr tica y al orden jur dico
establecido
x a todos los actores pol ticos y sociales involucrados a que tomen las
medidas necesarias para que cesen inmediatamente las acciones de
violencia intimidaci n y de desacato a la institucionalidad democr
tica y al orden jur dico establecido
x a todos los actores pol ticos y sociales involucrados que tomen las
medidas necesarias para que cesen inmediatamente las acciones de
violencia
----------------------------------------------------------------------- 90 / 100
Kept: no
x vencer serenamente al dolor
x dif ciles retos personales
x vencer serenamente al dolor a la sensaci n de soledad y a la tentaci n
de no querer afrontar los nuevos m ltiples y dif ciles retos
personales y familiares
----------------------------------------------------------------------- 91 / 100
Kept: no
x if you have to ask then you can't afford it
x if you have to ask you can't afford it
x if you have to ask
x you have to ask
x you can't afford it
----------------------------------------------------------------------- 92 / 100
Kept: yes
people living with mental illnesses often are among the most
vulnerable in our society
unfortunately they also are often overlooked during disasters
people living with mental illnesses often are among the most
vulnerable in our society unfortunately they also are often overlooked
during disasters
----------------------------------------------------------------------- 93 / 100
Kept: yes
close genuine female relationships are not what generally gets
depicted in movies and tv shows
like if you're watching the hills or 90210 all the backstabbing shapes
the way we act you go to school and you think your job is to find a
sworn enemy and be jealous of each other
close genuine 90210 all of the backstabbing shapes the way we act you
go to school and you think that your job is to find a sworn enemy and
be jealous of each other
like if you're watching the hills or 90210 all the backstabbing shapes
the way we act
close genuine female relationships are not what generally gets
depicted in movies and tv shows like if you're watching the hills or
90210 all the backstabbing shapes the way we act you go to school and
you think your job is to find a sworn enemy and be jealous of each
other
----------------------------------------------------------------------- 94 / 100
Kept: yes
the steps we are announcing today along with significant commitments
from the private sector are intended to mitigate the potential risks
and disruptions to markets
along with significant commitments from the private sector are
intended to mitigate the potential risks and disruptions to the
markets
the steps we are announcing today along with significant commitments
from the private sector are intended to mitigate the potential risks
and disruptions to markets he added
are intended to mitigate the potential risks and disruptions to
markets
to mitigate the potential risks and disruptions to markets
x mitigate the risks
----------------------------------------------------------------------- 95 / 100
Kept: yes
it's not as bad as we anticipated
it's not as bad as was first expected
it's not as bad as we first thought he went into the board awkward
----------------------------------------------------------------------- 96 / 100
Kept: yes
he brought a slightly more jazzy and ethereal element to it all
he had some elusive quality let's call it soul that glued the whole
thing together you notice it when it's missing
he brought a slightly more jazzy and ethereal element to it all he had
some elusive quality let's call it soul that glued the whole thing
together you notice it when it's missing
----------------------------------------------------------------------- 97 / 100
Kept: yes
x you're doing what
x i am impressed
you'll be in our thoughts and prayers i am very impressed by what
you're doing and what you're prepared to do coming from a state that
is not often hit by a hurricane i'm incredibly impressed
x i am very impressed
----------------------------------------------------------------------- 98 / 100
Kept: yes
it gives me great pleasure to secure the services of this extremely
talented batsman for next season
he will bring star quality to our batting line-up for both
championship and one-day cricket and his wealth of experience is
perfect for our younger batsman to learn from and develop around
it gives me great pleasure to secure the services of this extremely
talented batsman for next season he will bring star quality to our
batting line-up for both championship and one-day cricket and his
wealth of experience is perfect for our younger batsman to learn from
and develop around
----------------------------------------------------------------------- 99 / 100
Kept: yes
feed a cold and starve a fever
x feed a cold
x feed a cold starve a fever
x starve a fever
---------------------------------------------------------------------- 100 / 100
Kept: yes
we will miss his play on both ends of the floor as well as his
leadership
we will miss his play on both ends of the floor as well as his
leadership with the young players i am hopeful others will step
forward to fill those roles marcus has been a joy to coach and i look
forward to his return
we will miss his play on both ends of the floor as well as his
leadership with the young players
The question, here, is whether the cluster filter lets us keep only real clusters (i.e. not clusters of causally unrelated quotes; in other words, we'd like high precision, and we're not really interested in recall). The codings for these clusters are in the codings/filter_evaluations_clusters-precision-recall file, and precision-recall follows from them.
Sample clusters
Either run this cell to generate a new selection...
In [13]:
#with session_scope() as session:
# # Sample 5 times more to make sure we get at least 100 substitutions
# cluster_ids = sample([id for (id,)
# in session.query(Cluster.id).filter(Cluster.filtered == True)],
# 15 * SAMPLE_SIZE)
... or this one to use the previous selection (which has already been coded).
In [14]:
cluster_ids = [
1001058714, 1000234230, 1001661015, 1000793763, 1000992927, 1001447160, 1000928909,
1000652632, 1002429300, 1000424511, 1000028725, 1001804178, 1001962169, 1002010696,
1002620164, 1000541603, 1002038276, 1002178216, 1001264080, 1000658653, 1002656290,
1000602035, 1002444841, 1001270720, 1000980555, 1000049133, 1001594109, 1000874654,
1000270878, 1001267640, 1000587081, 1001502597, 1001290073, 1000742456, 1000803305,
1001768144, 1001790796, 1000420570, 1002288719, 1002195335, 1000630276, 1001310108,
1002544437, 1001355443, 1001215220, 1002184159, 1001954566, 1000417270, 1001062143,
1000193302, 1001073091, 1001318792, 1001333529, 1002230713, 1000253352, 1002165662,
1001000108, 1001492202, 1002697831, 1002554478, 1002203884, 1000368041, 1000743708,
1001803629, 1001123734, 1001892371, 1001817722, 1001887535, 1001566025, 1001800790,
1000778799, 1000917456, 1001772704, 1001500452, 1001176167, 1001580298, 1000786039,
1000276995, 1001329982, 1002416136, 1001861324, 1000314572, 1000873226, 1002526111,
1001214205, 1002498111, 1000073947, 1001636514, 1000381995, 1001214611, 1001605320,
1000100464, 1000853926, 1002047137, 1000532977, 1002113182, 1002222906, 1001773241,
1000492857, 1000949715, 1000164833, 1000263282, 1000669491, 1000537798, 1000493265,
1001739413, 1002188621, 1002425058, 1000997844, 1001798690, 1000158229, 1000131343,
1001936787, 1001056713, 1000069848, 1001032119, 1000492141, 1000001841, 1000062295,
1001365462, 1001259190, 1002360197, 1000417145, 1001266464, 1000395595, 1002109225,
1000519609, 1002341113, 1002627477, 1001447816, 1002598858, 1000739902, 1001334186,
1000421527, 1001369574, 1001468521, 1001221778, 1002397089, 1000194804, 1001834376,
1000847659, 1000857363, 1001389106, 1000404597, 1002673066, 1002010483, 1002482970,
1001309374, 1002135445, 1000676392, 1001402554, 1000023387, 1000861742, 1000739666,
1002208103, 1001610066, 1002091675, 1000055741, 1000098549, 1002606093, 1001019247,
1002081961, 1000046067, 1002202634, 1002023513, 1001803811, 1002270829, 1001090860,
1000942827, 1002488258, 1000073692, 1002394081, 1002642836, 1000964777, 1001327690,
1000005769, 1001231453, 1001843109, 1002095337, 1000144749, 1000856624, 1001408503,
1002077967, 1000689166, 1002604060, 1002619245, 1000260034, 1002549738, 1000886518,
1001146812, 1001499168, 1000981840, 1000510458, 1001824441, 1000023650, 1002645020,
1002476030, 1001445978, 1001729766, 1002396899, 1002704984, 1000456379, 1000164580,
1001594835, 1001879217, 1001658148, 1000222104, 1000175721, 1001934132, 1000458812,
1000597134, 1000983803, 1001045067, 1001934934, 1001101372, 1002064679, 1001354812,
1001387573, 1001552266, 1001049178, 1000211591, 1001535131, 1000068077, 1002667160,
1001958712, 1002049050, 1001354277, 1001406656, 1001333150, 1001800505, 1001948524,
1000398683, 1002031029, 1001171493, 1001750650, 1000486399, 1002344374, 1000798536,
1002483840, 1001327852, 1000574008, 1002405878, 1001603998, 1001658395, 1000311644,
1001804403, 1001249334, 1001265825, 1001434048, 1001393656, 1001794262, 1001120310,
1001337829, 1001450332, 1002610877, 1001479081, 1000806304, 1000310768, 1000399139,
1001749658, 1000062651, 1001672488, 1000168730, 1000287594, 1000031766, 1002453238,
1000907880, 1000172027, 1002431192, 1001999079, 1000072697, 1001986304, 1002115343,
1001729577, 1001750064, 1000630283, 1000459150, 1002202737, 1002558133, 1001727072,
1000130129, 1000657054, 1000701933, 1000953247, 1002349120, 1001881970, 1002607340,
1002248182, 1001559921, 1001252335, 1000301764, 1001307914, 1001284847, 1001340444,
1001785513, 1000158707, 1000072111, 1002577972, 1001846670, 1002551562, 1000423962,
1001392074, 1002514611, 1001067331, 1002031448, 1000179811, 1001451249, 1002350833,
1000722484, 1000541045, 1001513981, 1002327622, 1001060749, 1002530663, 1002644774,
1001007144, 1000257361, 1002563963, 1002394972, 1001058481, 1000854528, 1002613959,
1001676866, 1000073613, 1000209836, 1001243402, 1002440440, 1002364622, 1000090769,
1002031916, 1001978714, 1002287608, 1001866823, 1001583138, 1002663786, 1002190005,
1001188041, 1000497933, 1000061665, 1000964817, 1001367239, 1001935938, 1002551108,
1002637119, 1000525740, 1002690274, 1000762606, 1002218984, 1002540254, 1002012695,
1002535392, 1002045997, 1001761229, 1001345695, 1001540708, 1000409133, 1002281542,
1002031988, 1001905217, 1001927225, 1000456501, 1000103149, 1001595739, 1002629715,
1000633871, 1002582219, 1000125203, 1002622788, 1000152868, 1001691221, 1002550150,
1001367563, 1001517118, 1000871577, 1002129646, 1001997590, 1000404780, 1002438973,
1002241374, 1001038946, 1000088364, 1001781958, 1001978086, 1001035980, 1001110133,
1001604717, 1002078831, 1002488583, 1000792847, 1000641673, 1001165015, 1001283536,
1001500299, 1001336583, 1001464976, 1001775715, 1001334053, 1001752220, 1001059238,
1000061280, 1000343167, 1002262659, 1001017521, 1000926454, 1001549906, 1001399657,
1001596953, 1001511989, 1002134226, 1000997323, 1002423339, 1002561081, 1002351089,
1000541923, 1001581475, 1001867280, 1001543175, 1001503038, 1000093464, 1002711441,
1000730805, 1001702743, 1001948884, 1000065378, 1000215299, 1001326290, 1001638327,
1000007295, 1001515868, 1002212135, 1001575847, 1002161948, 1001357387, 1001766184,
1000727306, 1000699503, 1000075976, 1002128547, 1002602954, 1001760223, 1000659375,
1002062608, 1000750867, 1001376415, 1000364749, 1001058467, 1000618691, 1001144316,
1001536519, 1002587087, 1001101833, 1000247382, 1000609876, 1001645973, 1001840226,
1001602395, 1002684664, 1000776962, 1001157755, 1002342517, 1000022381, 1001920189,
1001801346, 1001162402, 1002689307, 1002156227, 1001687826, 1002596959, 1002349648,
1001894040, 1001450690, 1001735336, 1001498688, 1001180316, 1000810720, 1001220544,
1001529035, 1001497563, 1000466115, 1000218057, 1002531776, 1001810079, 1000174553,
1000145964, 1001540017, 1002531576, 1001728260, 1002033880, 1001150137, 1001953094,
1002077979, 1001228678, 1002168298, 1000151281, 1002335600, 1002526758, 1001714987,
1001291988, 1000419221, 1002166820, 1000925310, 1000982141, 1001751176, 1001191639,
1000428712, 1000885331, 1000130902, 1001035008, 1000538396, 1002436753, 1002272435,
1001444011, 1000873940, 1000706171, 1000465018, 1000006623, 1000275277, 1002396180,
1002582721, 1000118329, 1000037089, 1000453971, 1000963958, 1000640803, 1001129926,
1001224067, 1000047577, 1000393604, 1001300568, 1000147695, 1000808608, 1001958358,
1001269596, 1002713792, 1002275177, 1002410739, 1002315383, 1000223504, 1001345610,
1000036857, 1002471258, 1001375224, 1002695360, 1001511054, 1002619075, 1002701703,
1000428130, 1000207474, 1000367210, 1002676594, 1002290678, 1000780766, 1002248783,
1000331455, 1000869013, 1002351131, 1001827981, 1001624275, 1001359969, 1000562736,
1000393157, 1001947238, 1002170490, 1002493182, 1000951354, 1000734774, 1000272483,
1001067850, 1002234686, 1002259959, 1001223336, 1002086622, 1002102465, 1002404394,
1000833030, 1001881525, 1002214009, 1001142763, 1001334771, 1001399809, 1000971111,
1000841798, 1000351662, 1000918770, 1000051328, 1001085842, 1001426393, 1001499469,
1002673391, 1002439989, 1002656579, 1000503223, 1000199277, 1001102502, 1000804320,
1001829920, 1002377023, 1000669157, 1000789097, 1002582408, 1000755806, 1000826738,
1001776772, 1001368136, 1000804767, 1002148559, 1001719604, 1002662183, 1002320519,
1000047835, 1000240968, 1000980627, 1000889403, 1002350939, 1001131879, 1001271289,
1002320021, 1000812131, 1001424421, 1002461839, 1001799302, 1000362981, 1001116881,
1001201610, 1001904014, 1002449788, 1000361053, 1000143359, 1000922307, 1001675730,
1000773716, 1000964829, 1000830471, 1000431368, 1002320455, 1002506830, 1001079629,
1002573767, 1000514517, 1002143145, 1001158803, 1001644001, 1002618006, 1002492025,
1000676599, 1001041617, 1002145451, 1000017230, 1001866916, 1001265926, 1002274458,
1001815958, 1002702195, 1002236588, 1002247341, 1000118770, 1002178890, 1001472895,
1000063547, 1000762613, 1002297834, 1000807747, 1001970314, 1002276553, 1000562856,
1001140852, 1001601748, 1002639679, 1001182746, 1002650044, 1001227746, 1000170311,
1000564560, 1001740055, 1001904646, 1000539800, 1001501829, 1000531802, 1000128379,
1000790358, 1001940684, 1000158028, 1001881112, 1002492922, 1000803880, 1000643541,
1000052799, 1000297615, 1001726109, 1002543957, 1000054716, 1000155125, 1002652618,
1000769467, 1002042242, 1000368676, 1000858673, 1001822210, 1000350231, 1001442450,
1000050760, 1002041174, 1000233535, 1001572148, 1001432199, 1001742887, 1001172196,
1000364258, 1001970826, 1002105767, 1002037960, 1000451410, 1002099403, 1000967350,
1000147974, 1000341770, 1001680878, 1001613527, 1002528697, 1002715601, 1002474922,
1000818246, 1001278242, 1000681152, 1000997312, 1000010867, 1000952787, 1000993230,
1001786224, 1000203905, 1000911729, 1000885883, 1002075501, 1001614854, 1001924725,
1000385909, 1001433831, 1000201148, 1001606931, 1001941294, 1000962064, 1001860524,
1000774230, 1002033685, 1001261980, 1000288611, 1001334187, 1001921383, 1002166359,
1001938346, 1001262841, 1000993575, 1001270506, 1002702506, 1001842295, 1000382304,
1000042449, 1000021771, 1000481204, 1002121723, 1001652776, 1001226182, 1002225233,
1001253286, 1001562116, 1000154064, 1000094446, 1001633759, 1001546738, 1001004738,
1002556326, 1001205897, 1000902561, 1002001937, 1001700843, 1002435437, 1000989228,
1001653126, 1001334470, 1002071621, 1001522103, 1001548001, 1001285963, 1000090895,
1000746072, 1001574873, 1002175814, 1002531988, 1002144195, 1001770967, 1001451200,
1001856086, 1000559388, 1001297817, 1000837494, 1001534871, 1000220313, 1001259385,
1002100069, 1001332921, 1000544786, 1001841512, 1001383653, 1000684470, 1002161309,
1002231396, 1002348115, 1001758109, 1001747036, 1000284497, 1000832130, 1001282170,
1002010186, 1000427241, 1001581775, 1000578192, 1000078262, 1000016694, 1002705118,
1001733624, 1001363335, 1002436545, 1001328603, 1002180807, 1000767722, 1002485195,
1002272469, 1001271235, 1001481701, 1001348533, 1000388469, 1001219997, 1001460577,
1001590875, 1001240142, 1000238824, 1000121322, 1002004474, 1001592727, 1001385776,
1002470366, 1002664998, 1001997277, 1001120647, 1002575276, 1001722300, 1002216129,
1000067607, 1000994775, 1001498352, 1002699359, 1000587363, 1000217706, 1002109991,
1000288982, 1002699713, 1002233823, 1001153232, 1000914136, 1002548433, 1001734510,
1002342520, 1002452277, 1001100860, 1000950342, 1000388189, 1000127650, 1002315582,
1001566979, 1000957774, 1000052007, 1000364394, 1001996793, 1001908176, 1002105936,
1002086992, 1002016069, 1001183918, 1000800618, 1001022239, 1001270359, 1002242646,
1002505634, 1000716919, 1000901112, 1002481410, 1000298970, 1000516124, 1000057052,
1002361761, 1002647536, 1000845811, 1002171985, 1002050592, 1000006586, 1001255552,
1001837963, 1001256691, 1001761220, 1000938672, 1000530532, 1000311534, 1001776809,
1000318357, 1002053600, 1001444195, 1002365910, 1001746878, 1000228196, 1002060204,
1001739958, 1001886840, 1000698210, 1002509868, 1001502009, 1001893527, 1002638619,
1002066935, 1001550280, 1002294683, 1000107898, 1000010223, 1001508748, 1001689201,
1001003737, 1001014351, 1002082703, 1002224120, 1002157272, 1000882553, 1000053316,
1001414012, 1001342905, 1002156509, 1000251513, 1001232667, 1001399253, 1001855493,
1001013265, 1001688212, 1002583768, 1001529428, 1001127460, 1001698379, 1002315633,
1002318183, 1001620522, 1001378536, 1001574209, 1000569652, 1001604632, 1002697581,
1001575675, 1001117505, 1002051194, 1000183177, 1000855203, 1001423033, 1001777736,
1002318334, 1001056375, 1002426820, 1002236516, 1000150483, 1001578508, 1001358370,
1000011455, 1002362070, 1000806847, 1001801348, 1000085743, 1001413671, 1001540172,
1001829995, 1001851813, 1001597485, 1000078746, 1001840646, 1001995820, 1000160512,
1002451933, 1001191721, 1002203693, 1000238465, 1002033572, 1001320063, 1000872380,
1000623197, 1002147129, 1000011914, 1001530603, 1002446364, 1000079514, 1002448631,
1001076382, 1002223593, 1000490815, 1000630667, 1000376213, 1002462823, 1000941817,
1002357231, 1001797587, 1000926673, 1000594216, 1001861749, 1001684910, 1002501511,
1001579538, 1002606713, 1000695748, 1000903985, 1001546192, 1001996840, 1001472327,
1000032826, 1001545056, 1002014395, 1002095028, 1001913231, 1000075824, 1000952746,
1001853596, 1002182047, 1002628022, 1002004078, 1001566113, 1002163349, 1000790111,
1002285133, 1002494233, 1000397761, 1001439740, 1001101891, 1001618046, 1001291605,
1002314992, 1001234028, 1001559891, 1001477621, 1000780535, 1000875694, 1000645845,
1000484118, 1002111321, 1001878295, 1002646120, 1002039447, 1002029116, 1000156966,
1000010498, 1001769120, 1000175849, 1000418713, 1001915737, 1000709712, 1001922556,
1001283094, 1002208729, 1000333316, 1000012505, 1001047126, 1001172398, 1000177517,
1001487440, 1000001465, 1000882506, 1002659495, 1002052955, 1001398557, 1001303140,
1000860772, 1002525449, 1002527325, 1002183331, 1001776468, 1000205762, 1000597585,
1000113771, 1001870016, 1000025477, 1002086198, 1000502730, 1001981118, 1002473830,
1001480043, 1002528637, 1000838413, 1001653286, 1001330880, 1002679849, 1002506095,
1000486043, 1001982507, 1002103300, 1000257805, 1001964267, 1000874462, 1001326641,
1001782479, 1001021296, 1001867545, 1000694265, 1002033815, 1000304505, 1000985515,
1000135881, 1002495192, 1000821217, 1001810184, 1002310190, 1002635800, 1000729831,
1000784775, 1001668437, 1002274466, 1000797484, 1002537649, 1000475111, 1000437411,
1001657664, 1001278789, 1001836754, 1001983301, 1001494451, 1002357233, 1002212481,
1001782077, 1001629749, 1000824935, 1002484633, 1000050369, 1000716672, 1001310861,
1001954605, 1002473461, 1000000607, 1000825586, 1001437178, 1000204392, 1002359775,
1001499856, 1001810430, 1002093548, 1001448073, 1001173925, 1000746976, 1002502228,
1000177952, 1001304657, 1001653859, 1000214289, 1000157134, 1001597852, 1002060579,
1000735315, 1002615587, 1001055343, 1002059341, 1000356177, 1000447259, 1000699286,
1002457427, 1001989841, 1001801036, 1001434979, 1001957769, 1000007005, 1000916205,
1001169392, 1001889767, 1000075558, 1002025030, 1001976875, 1001560208, 1001115658,
1001598286, 1001205754, 1002231878, 1000813552, 1001981838, 1000677677, 1001053575,
1002255917, 1002025140, 1001734307, 1001057270, 1001087649, 1002574757, 1001915861,
1001674897, 1001364136, 1002154777, 1001019768, 1001614035, 1000379490, 1001908572,
1002591014, 1000164476, 1000021830, 1002468679, 1001717996, 1001998254, 1000470822,
1001431751, 1001319414, 1002050667, 1000560326, 1001259195, 1000866918, 1001379510,
1002113186, 1001517995, 1001646691, 1000465784, 1001374778, 1001896574, 1002089665,
1000872939, 1001914402, 1001909903, 1002351476, 1002080607, 1001717244, 1000018324,
1001994530, 1001422230, 1001353347, 1000339608, 1000475920, 1001694035, 1000982021,
1001590385, 1001535698, 1000544821, 1001163684, 1000525655, 1002208497, 1001140991,
1000766128, 1000607192, 1000151074, 1000723980, 1002133819, 1000500270, 1000834415,
1002225356, 1000373694, 1001824087, 1000067090, 1000297958, 1002317409, 1001770960,
1001508749, 1000167477, 1001947065, 1001940057, 1000781919, 1000988086, 1002518061,
1001558269, 1001815508, 1001091260, 1000263913, 1000405911, 1001511990, 1001723810,
1001074987, 1002376815, 1001680122, 1001506913, 1001594016, 1001519696, 1001789977,
1000692462, 1000029089, 1000338311, 1001956638, 1002575343, 1001274784, 1001851680,
1002390771, 1001326868, 1001743949, 1002550572, 1002559841, 1002530236, 1001459660,
1000449148, 1002536266, 1002701796, 1002253595, 1001124589, 1001133908, 1002463064,
1000392223, 1001121726, 1001289467, 1000289591, 1002105764, 1002381694, 1002341368,
1000827595, 1002084983, 1001682153, 1000633018, 1001113583, 1000928142, 1000033000,
1000621627, 1001084527, 1001200554, 1001458154, 1002412589, 1001972202, 1002691528,
1000265992, 1002659026, 1001366843, 1000467244, 1001664670, 1002489296, 1002483595,
1001900662, 1002686469, 1001330713, 1000830278, 1000795336, 1002364454, 1000811242,
1001678392, 1000033963, 1002634804, 1001804844, 1000527231, 1002344474, 1002369472,
1001578312, 1001845592, 1001540167, 1002337924, 1000539731, 1002311229, 1000201698,
1000953073, 1000629221, 1002663893, 1001791750, 1001914401, 1002604562, 1000828450,
1002613802, 1001268711, 1000927905, 1001578575, 1001005287, 1002131006, 1002396856,
1000010276, 1001202203, 1000137591, 1001716758, 1001404162, 1002497367, 1000410225,
1000002836, 1000456777, 1000005943, 1002065249, 1002074323, 1000814733, 1002442276,
1002663916, 1001447228, 1000979497, 1000956742, 1001038308, 1002638096, 1000415025,
1001868899, 1001017794, 1000056720, 1000058183, 1000821880, 1001228149, 1000480755,
1002499177, 1001537485, 1000339767, 1002044113, 1002573766, 1001269992, 1001354843,
1000118888, 1000149428, 1001803975, 1000222326, 1001257849, 1002187778, 1002149432,
1001445651, 1001971190, 1001377902, 1000168509, 1001351656, 1001713927, 1001104442,
1001890822, 1001869495, 1002575689, 1000762777, 1000150393, 1001259333, 1000565784,
1002312763, 1001455163, 1002431805, 1002170320, 1001727497, 1002120786, 1001211845,
1002009293, 1002002759, 1002256034, 1000960075, 1000905489, 1002410523, 1000381073,
1000173214, 1002395671, 1002256806, 1001768142, 1000177679, 1002692425, 1002088771,
1000856361, 1001423847, 1001575759, 1000449711, 1001550004, 1001554359, 1001906605,
1001710361, 1001862213, 1002048548, 1000734007, 1001059720, 1002352183, 1001121521,
1000213563, 1001664822, 1002450758, 1001128119, 1001734650, 1000402851, 1000517539,
1001565687, 1001288446, 1000169483, 1000780042, 1000862817, 1000904640, 1001952654,
1000734391, 1001156795, 1001074923, 1001855482, 1001336701, 1001371862, 1002545044,
1001454360, 1001719858, 1001598709, 1001450807, 1000713806, 1000792912, 1002677052,
1001580304, 1002257558, 1001500784, 1002261599, 1000810175, 1001677954, 1001202743,
1002314510, 1001777562, 1000822655, 1001002115, 1001217819, 1002520043, 1000716537,
1001958057, 1002669138, 1002021941, 1000828887, 1000177607, 1001993644, 1002485054,
1001084532, 1000857473,
]
Sample substitutions
Either run this cellto generate a new selection...
In [15]:
#substitution_ids = {}
#with session_scope() as session:
# for max_distance in range(1, 3):
# model = Model(Time.discrete, Source.majority, Past.last_bin, Durl.all, max_distance)
# # Sample 5 times more to make sure we get at least 100 substitutions
# # even with different quotes
# substitution_ids[model] = sample(
# [id for (id,)
# in session.query(Substitution.id).filter(Substitution.model == model)],
# 5 * SAMPLE_SIZE
# )
... or this one to use the previous selection (which has already been coded).
In [16]:
substitution_ids = {
Model(Time.discrete, Source.majority, Past.last_bin, Durl.all, 1): [
5472, 10466, 8963, 6380, 778, 9747, 474, 830, 8524, 8271,
674, 6907, 5478, 9421, 8203, 7668, 6772, 2201, 6117, 3916,
8525, 10101, 1863, 1846, 2741, 8648, 9870, 4681, 1871, 7960,
6830, 7839, 4664, 2974, 5660, 7994, 4239, 5473, 8170, 10334,
4306, 9317, 1887, 9638, 6799, 6115, 849, 3303, 4259, 6105,
7455, 10475, 2176, 5343, 913, 8808, 420, 489, 5431, 872,
6754, 389, 9343, 8201, 6234, 498, 3411, 9359, 8402, 7506,
7084, 6809, 7417, 9337, 1924, 5685, 648, 415, 5702, 4612,
10714, 8804, 4269, 6758, 10980, 7140, 1967, 2968, 5443, 2661,
9635, 3911, 4346, 3496, 946, 879, 4845, 7148, 3445, 6574,
5650, 8046, 8943, 7115, 9914, 5688, 54, 1425, 932, 1498,
3318, 7357, 10723, 571, 6440, 10721, 5723, 4271, 943, 854,
4067, 2694, 6135, 57, 2434, 8190, 417, 5571, 2213, 6612,
2674, 7063, 4353, 6653, 800, 6138, 2688, 459, 859, 2430,
5269, 9871, 10381, 8986, 2072, 9381, 200, 354, 4194, 7394,
9877, 7033, 723, 9883, 8626, 4206, 486, 3617, 243, 5339,
5355, 9338, 1724, 811, 6130, 2224, 3311, 4616, 4521, 9410,
5482, 6369, 7963, 10768, 4256, 4061, 565, 3623, 6657, 141,
4226, 8642, 11052, 4052, 786, 40, 2025, 632, 541, 5755,
7486, 2706, 9560, 213, 10879, 5441, 5800, 8684, 2953, 6822,
411, 9342, 6847, 9494, 392, 6820, 1841, 6945, 5656, 467,
6433, 9407, 442, 6663, 1996, 5639, 7359, 2162, 8546, 4488,
7548, 6777, 3649, 2435, 4313, 7125, 348, 10991, 7082, 6795,
7142, 9535, 5464, 4620, 867, 9760, 8974, 5705, 4223, 7458,
8382, 8076, 2199, 6183, 10719, 10356, 988, 4330, 10971, 6871,
4374, 8899, 42, 2157, 8092, 5514, 744, 9361, 1961, 5555,
8163, 4463, 8887, 4134, 865, 432, 1413, 5754, 5469, 9414,
3305, 5669, 6181, 570, 7551, 8759, 695, 2419, 6903, 2202,
969, 7570, 2438, 5429, 4465, 179, 5239, 10438, 7501, 573,
2559, 9486, 9457, 6146, 949, 1377, 8926, 4390, 9745, 5780,
3404, 279, 9490, 3765, 4197, 2164, 4691, 5753, 10775, 277,
11048, 8414, 2014, 7678, 10338, 10636, 6367, 140, 4348, 4429,
3910, 4220, 410, 2095, 3294, 2230, 4833, 2045, 3626, 5466,
3306, 3622, 2243, 5322, 8510, 2192, 6235, 4231, 564, 9423,
9333, 5492, 1725, 10378, 8270, 10173, 2154, 201, 2151, 8562,
3928, 7006, 5647, 9228, 2721, 8970, 3909, 2686, 9061, 5765,
7027, 824, 5436, 4860, 7534, 2090, 8192, 4126, 9466, 3446,
1009, 3450, 5248, 6182, 9888, 22, 513, 7389, 8819, 992,
11047, 5549, 1519, 7352, 10615, 616, 3624, 7112, 3290, 2181,
8941, 10877, 977, 2719, 4221, 10399, 2200, 5504, 6238, 9562,
10862, 9418, 10709, 3618, 9063, 9388, 7137, 897, 3407, 6362,
10468, 2047, 5673, 10618, 9552, 851, 3416, 4058, 8947, 6814,
5581, 701, 972, 4340, 4215, 689, 8713, 3956, 8707, 6176,
3413, 10021, 4516, 4780, 9422, 3347, 7032, 3340, 6431, 5724,
4288, 4496, 9417, 6119, 4057, 325, 4413, 9869, 8500, 220,
6152, 9385, 3293, 7996, 793, 219, 4456, 939, 8086, 613,
6948, 212, 1965, 4417, 10623, 804, 4449, 5424, 9766, 7542,
9782, 4361, 6356, 1964, 8061, 9393, 8566, 4240, 9401, 4323,
5484, 10423, 501, 8033, 3414, 8542, 4257, 6462, 4048, 2034,
7669, 3277, 4295, 7113, 973, 9588, 4480, 2463, 4451, 11113
],
Model(Time.discrete, Source.majority, Past.last_bin, Durl.all, 2): [
7242, 6639, 9155, 6065, 8704, 3122, 9273, 3556, 7633, 1514,
2335, 7235, 1309, 4084, 3690, 6671, 4171, 3205, 1207, 2595,
2587, 4151, 2790, 9504, 3008, 9022, 1610, 3169, 9163, 10168,
2783, 9271, 8480, 2497, 3782, 9805, 3991, 9513, 10277, 2374,
5875, 5527, 6079, 8333, 11029, 3028, 3366, 7255, 6691, 3160,
2108, 10525, 10165, 1810, 6285, 2738, 3897, 2917, 10509, 8718,
10011, 2801, 1703, 8783, 4901, 6560, 5945, 669, 9647, 3802,
604, 8024, 7037, 7303, 4930, 2729, 6622, 2346, 4744, 3386,
10184, 7689, 1708, 3536, 10207, 3973, 7517, 5877, 6349, 2534,
8851, 1481, 7818, 2912, 5895, 9050, 2483, 1910, 8360, 3992,
4899, 1147, 5040, 1222, 4766, 3190, 2305, 8869, 10458, 773,
2730, 4994, 10863, 7299, 7566, 3534, 5334, 6016, 1490, 7904,
3686, 8794, 7664, 8698, 1239, 1809, 4185, 7523, 1915, 1894,
6569, 10556, 4018, 7376, 7699, 10256, 9717, 8717, 2765, 5843,
2266, 9266, 9921, 7896, 7400, 7091, 1800, 4887, 7860, 1454,
812, 3831, 68, 9168, 9933, 4541, 3696, 9804, 2288, 3329,
7612, 2500, 1494, 8122, 7167, 9614, 9172, 3486, 6000, 8482,
8479, 3660, 10679, 10053, 8923, 1312, 10216, 9127, 2516, 9024,
8322, 3794, 9817, 10508, 7323, 5372, 9778, 1628, 1359, 5092,
6670, 8738, 3899, 7284, 9911, 4722, 1674, 3067, 10746, 2284,
4019, 10223, 8139, 9034, 6887, 7629, 3700, 1899, 2736, 5335,
8690, 2980, 10641, 9726, 7431, 2793, 4716, 3082, 5532, 3006,
2498, 1397, 7041, 1311, 1493, 6507, 10681, 257, 9895, 10098,
7653, 6561, 10542, 600, 5220, 1149, 10573, 5311, 2622, 9117,
5373, 5064, 3687, 8865, 7354, 9822, 11044, 5292, 8267, 8030,
1775, 8313, 9595, 8247, 578, 4509, 6320, 3674, 3115, 8456,
9001, 3482, 5396, 11096, 7934, 2678, 8108, 10962, 5079, 6995,
9603, 5048, 7695, 3161, 6517, 5022, 5059, 10265, 10130, 5121,
3584, 7716, 6197, 10282, 4628, 11060, 5003, 3509, 9068, 644,
2537, 4761, 3374, 1877, 1353, 10655, 11076, 9553, 5119, 107,
9610, 3801, 7880, 10871, 3144, 4578, 4767, 7774, 653, 3753,
251, 1067, 10076, 1460, 9184, 10580, 8921, 8604, 6709, 2914,
5143, 145, 10453, 9861, 8003, 4167, 9819, 5007, 10941, 5290,
7891, 8228, 2833, 4163, 1757, 1826, 3177, 3581, 4965, 7078,
6219, 3381, 8848, 1412, 7381, 9837, 1209, 2352, 6516, 314,
10728, 8349, 1423, 9057, 8356, 91, 6198, 159, 9939, 8769,
2878, 10560, 10729, 9287, 9735, 4591, 9907, 211, 1472, 5595,
6740, 9069, 1768, 5392, 4092, 3002, 6201, 6419, 9053, 2481,
1408, 1621, 4525, 6955, 7759, 8745, 7575, 2649, 658, 5938,
4139, 1602, 3085, 1895, 5106, 1650, 9721, 4954, 9521, 3763,
3221, 4972, 6093, 5826, 6340, 2623, 7742, 11065, 113, 5280,
6668, 7035, 3050, 4915, 7265, 8733, 7271, 1157, 9976, 10112,
3713, 10088, 7942, 9083, 2540, 4963, 2527, 5512, 6651, 9722,
5905, 1597, 6606, 5948, 172, 4991, 4756, 1111, 9708, 1231,
8615, 1560, 3134, 3708, 7892, 4593, 3850, 1566, 7438, 8113,
4181, 1182, 1685, 5405, 5913, 7779, 8107, 1177, 1734, 5950,
2600, 4913, 2261, 6039, 6297, 10818, 1154, 8788, 5289, 118,
1645, 5197, 3118, 2792, 1945, 10583, 1049, 3324, 8121, 2533,
10425, 7713, 9774, 10939, 6957, 6558, 6705, 10066, 2116, 10930,
4633, 1194, 1692, 5828, 9840, 3780, 4508, 4995, 1624, 10176
]
}
Our substitution printing function:
In [17]:
def print_substitution(number, substitution):
title = ' {} / {}'.format(number, SAMPLE_SIZE)
print('-' * (80 - len(title)) + title)
if substitution.validate():
print('Kept: yes')
else:
print('Kept: no')
print()
print(' Tokens: {tokens[0]} -> {tokens[1]}'
.format(tokens=substitution.tokens))
print(' Lemmas: {lemmas[0]} -> {lemmas[1]}'
.format(lemmas=substitution.lemmas))
print()
print(indent(fill(substitution.source.string), ' ' * 5))
print()
print(indent(fill(substitution.destination.string), ' ' * 5))
print()
def mine_print_substitutions(model):
seen = 0
seen_substitutions = set()
for cluster_id in cluster_ids:
if seen >= SAMPLE_SIZE:
break
model.drop_caches()
with session_scope() as session:
cluster = session.query(Cluster).get(cluster_id)
for substitution in cluster.substitutions(model):
session.rollback()
if seen >= SAMPLE_SIZE:
break
if (substitution.destination.id, substitution.position) in seen_substitutions:
break
seen += 1
seen_substitutions.add((substitution.destination.id, substitution.position))
print_substitution(seen, substitution)
if seen < SAMPLE_SIZE:
print("Didn't find {} substitutions, you should sample more clusters"
.format(SAMPLE_SIZE))
def db_print_substitutions(model):
seen = 0
seen_substitutions = set()
for substitution_id in substitution_ids[model]:
if seen >= SAMPLE_SIZE:
break
with session_scope() as session:
substitution = session.query(Substitution).get(substitution_id)
if (substitution.destination.id, substitution.position) in seen_substitutions:
continue
seen += 1
seen_substitutions.add((substitution.destination.id, substitution.position))
print_substitution(seen, substitution)
Here are the individual substitutions and their respective rejected/kept status, with single substitution.
In [18]:
mine_print_substitutions(Model(Time.discrete, Source.majority, Past.last_bin, Durl.all, 1))
------------------------------------------------------------------------ 1 / 100
Kept: no
Tokens: that -> it
Lemmas: that -> it
i hope laura and i did the same thing but i believe he will and i know
his girls are on his mind and he wants to make sure that first and
foremost he is a good dad and i think that's going to be an important
part of his presidency
but i believe he will and i know his girls are on his mind and he
wants to make sure that first and foremost he's a good dad and i think
it's going to be an important part of his presidency
------------------------------------------------------------------------ 2 / 100
Kept: no
Tokens: higher-level -> higher
Lemmas: higher-level -> high
going forward we'll use jquery as one of the libraries used to
implement higher-level controls in the asp net ajax control toolkit as
well as to implement new ajax server-side helper methods for asp net
mvc
going forward we'll use jquery as one of the libraries used to
implement higher
------------------------------------------------------------------------ 3 / 100
Kept: no
Tokens: 's -> 's
Lemmas: be -> 's
this is proof of the microscope's potential we are now ready to start
doing scientific experiments that will add a new dimension to
measurements being made by other phoenix lander instruments
this is proof of the microscope's potential
------------------------------------------------------------------------ 4 / 100
Kept: no
Tokens: superconducting -> discovery
Lemmas: superconducting -> discovery
because the lhc is a superconducting machine that works at very low
temperatures in order to get in and fix it we've got to warm it up
then we go and fix it and then we cool it down again and that's a
process that's likely to take two months
the lhc is a discovery machine
------------------------------------------------------------------------ 5 / 100
Kept: no
Tokens: beginning -> beginning
Lemmas: beginning -> begin
a momentous economic slowdown is now under way we believe the
deterioration in spain is just in the beginning stages the bulk of the
pain will be suffered in 2009
a momentous economic slowdown is now under way we believe the
deterioration in spain is just in the beginning stages
------------------------------------------------------------------------ 6 / 100
Kept: no
Tokens: could -> can
Lemmas: could -> can
i am so damn glad that you could never be my wife cause i surely
wouldn't have to listen to that prattle from you every day
i am so damn glad that you can never be my wife cause i surely
wouldn't have to listen to that prattle from you every day
------------------------------------------------------------------------ 7 / 100
Kept: no
Tokens: theaters -> theatres
Lemmas: theater -> theatre
the theaters are closed there is just no business at all down there
the theatres are closed there is just no business at all down there
------------------------------------------------------------------------ 8 / 100
Kept: no
Tokens: for -> of
Lemmas: for -> of
from the very beginning i have said i am going to support the
candidate that has the best chance for changing the way washington
works and getting things done and i will be voting for barack obama
and clapping
the best chance of changing the way washington works
------------------------------------------------------------------------ 9 / 100
Kept: no
Tokens: sovereignty -> sovereign
Lemmas: sovereignty -> sovereign
what is clear is that russia has invaded georgia's sovereignty has
encroached on georgia's sovereignty and it is very important for us to
resolve this issue as quickly as possible
what is clear is that russia has invaded georgia's sovereign has
encroached on georgia's sovereignty
----------------------------------------------------------------------- 10 / 100
Kept: no
Tokens: are -> have
Lemmas: be -> have
cleary there's been a number of incidents recently and it's important
that we go in and double check and make sure that all the standards
are being maintained
there have been a number of incidents recently and it's important that
we go in and double check and make sure that all the standards have
been maintained
----------------------------------------------------------------------- 11 / 100
Kept: no
Tokens: angelina -> she
Lemmas: angelina -> she
what angelina did was very uncool
what she did was very uncool
----------------------------------------------------------------------- 12 / 100
Kept: no
Tokens: very -> uncool
Lemmas: very -> uncool
what angelina did was very uncool
what angelina did was uncool
----------------------------------------------------------------------- 13 / 100
Kept: no
Tokens: what -> inexplicable
Lemmas: what -> inexplicable
your relationship with george bush seems what's the word i'm looking
for inexplicable
your relationship with george bush seems inexplicable
----------------------------------------------------------------------- 14 / 100
Kept: no
Tokens: phosphorus -> phosphorous
Lemmas: phosphorus -> phosphorous
white phosphorus can burn down houses and cause horrific burns when it
touches the skin
white phosphorous can burn down houses and cause horrific burns when
it touches the skin
----------------------------------------------------------------------- 15 / 100
Kept: no
Tokens: phosphorous -> phosphorus
Lemmas: phosphorous -> phosphorus
white phosphorous can burn down houses and cause horrific burns when
it touches the skin
white phosphorus can burn down houses and cause horrific burns when it
touches the skin
----------------------------------------------------------------------- 16 / 100
Kept: no
Tokens: it -> that
Lemmas: it -> that
growing up i always wanted to be an olympian now to be the most
decorated olympian of all time it just sounds weird saying it started
setting in a little bit after the butterfly i was just trying to focus
on my next race but i just kept thinking wow greatest olympian of all
time it's a pretty cool title i'm definitely honored
that's a pretty cool title
----------------------------------------------------------------------- 17 / 100
Kept: yes
Tokens: responsibility -> response
Lemmas: responsibility -> response
the responsibility of the pakistani government should be one of
cooperation and of action
the response of the pakistani government should be one of cooperation
and of action
----------------------------------------------------------------------- 18 / 100
Kept: no
Tokens: path-breaking -> pathbreaking
Lemmas: path-breaking -> pathbreaking
how is this happening with a path-breaking public-private coalition
solid science better statistics and precise financing with the
coordination of the right countries and partners and above all with
the leadership
with a pathbreaking public-private coalition solid science better
statistics and precise financing with the coordination of the right
countries and partners and above all with the leadership
----------------------------------------------------------------------- 19 / 100
Kept: no
Tokens: sen -> senator
Lemmas: sen -> senator
was the last best chance to bring greater oversight and tighter
regulation to freddie and fannie and they used every means they could
to defeat sen hagel's legislation every step of the way
was the last best chance to bring greater oversight and tighter
regulation to freddie and fannie and they used every means they could
to defeat senator hagel's legislation every step of the way
----------------------------------------------------------------------- 20 / 100
Kept: no
Tokens: re-looking -> relooking
Lemmas: re-looking -> relooking
we are not interested in going back and re-looking at an acquisition
we are not interested in going back and relooking at an acquisition
----------------------------------------------------------------------- 21 / 100
Kept: no
Tokens: he -> jett
Lemmas: he -> jett
jett may have still been alive when john administered cpr and then the
emt took over jett was pronounced dead at the hospital i like to
believe john had a chance to say goodbye he may have died in his dad's
arms i am not certain
i like to believe john had a chance to say goodbye jett may have died
in his dad's arms
----------------------------------------------------------------------- 22 / 100
Kept: no
Tokens: cease-fire -> ceasefire
Lemmas: cease-fire -> ceasefire
georgia expresses its readiness to immediately start negotiations with
the russian federation on cease-fire and termination of hostilities
georgia expresses its readiness to immediately start negotiations with
the russian federation on ceasefire and termination of hostilities
----------------------------------------------------------------------- 23 / 100
Kept: no
Tokens: democrat -> democratic
Lemmas: democrat -> democratic
in that era almost all black americans were republicans why from its
founding in 1854 as the anti-slavery party until today the republican
party has championed freedom and civil rights for blacks and as one
pundit so succinctly stated the democrat party is as it always has
been the party of the four s's slavery secession segregation and now
socialism
and as one pundit so succinctly stated the democratic party is as it
always has been the party of the four s's slavery secession
segregation and now socialism
----------------------------------------------------------------------- 24 / 100
Kept: no
Tokens: ministerial -> that
Lemmas: ministerial -> that
that ministerial list is what they wish to happen it was not agreed on
as far as we know there was no agreement on the allocation of cabinet
posts and we are waiting for the mediator mr mbeki to come and resolve
the impasse
that list is what they wish to happen it was not agreed on
----------------------------------------------------------------------- 25 / 100
Kept: no
Tokens: out-scaled -> outscaled
Lemmas: out-scaled -> outscaled
michael's talent out-scaled even his own dinosaurs of jurassic park
michael's talent outscaled even his own dinosaurs of jurassic park
----------------------------------------------------------------------- 26 / 100
Kept: no
Tokens: could -> can
Lemmas: could -> can
and so our troops are a front line against the taliban there is a
chain of terror that comes from the pakistani and afghan mountains
right across europe and could end up very easily on the streets of
britain
there is a chain of terror that comes from the pakistani and afghan
mountains right across europe and can end up very easily on the
streets of britain
----------------------------------------------------------------------- 27 / 100
Kept: no
Tokens: recognizing -> recognising
Lemmas: recognize -> recognise
i am utterly deeply humbled and chuffed by the fact that i'm a stamp
i'm going to be licked by millions of australians and i can't wait it
is also recognizing the value of the arts in australia it brings it
into general consumption and if i'm part of that i feel intensely
proud
it is also recognising the value of the arts in australia it brings it
into general consumption and if i'm part of that i feel intensely
proud
----------------------------------------------------------------------- 28 / 100
Kept: no
Tokens: over -> here
Lemmas: over -> here
whether it's joe the plumber in ohio or joe over here
whether it's joe the plumber in ohio or joe here
----------------------------------------------------------------------- 29 / 100
Kept: no
Tokens: this -> the
Lemmas: this -> the
this law needs to be changed to focus on infants
the law needs to be changed
----------------------------------------------------------------------- 30 / 100
Kept: no
Tokens: it -> drugs
Lemmas: it -> drug
i dragged amy into it and without me there is no doubt that she would
never have gone down that road i ruined something beautiful now i have
to let her go to save her life i am not abandoning her i am doing this
out of love
i dragged amy into drugs and without me there is no doubt that she
would never have gone down that road
----------------------------------------------------------------------- 31 / 100
Kept: no
Tokens: and -> on
Lemmas: and -> on
we are aware of a global virus for which there are some public alerts
and we've seen some of these on our networks and we are taking steps
to identify and mitigate the virus
we are aware of a global virus for which there are some public alerts
on
----------------------------------------------------------------------- 32 / 100
Kept: yes
Tokens: material -> physical
Lemmas: material -> physical
he hates the american material occupation as much as he hates the
iranian moral occupation
he hates the american physical occupation as much as he hates the
iranian moral occupation
----------------------------------------------------------------------- 33 / 100
Kept: no
Tokens: against -> aginst
Lemmas: against -> aginst
sharon is delighted to have won her case she would prefer not to have
to take legal action against the media but had no alternative in this
case
sharon is delighted to have won her case she would prefer not to have
to take legal action aginst the media but had no alternative in this
case
----------------------------------------------------------------------- 34 / 100
Kept: no
Tokens: organised -> organized
Lemmas: organise -> organize
legally relations between two countries cannot be organised by a law
they should be arranged according to international law through
treaties or agreements
legally relations between two countries cannot be organized by a law
they should be arranged according to international law through
treaties or agreements
----------------------------------------------------------------------- 35 / 100
Kept: no
Tokens: does -> he
Lemmas: do -> he
he does say he wishes the outfits were for his eyes only
he says he wishes the outfits were for his eyes only
----------------------------------------------------------------------- 36 / 100
Kept: no
Tokens: raiding -> raiding
Lemmas: raiding -> raid
for apple mac and unix lovers these major spam attacks just mean a
clogged-up inbox not an infected operating system but organised
criminals are causing havoc for windows users in the hunt for cold
hard cash too many people are clicking without thinking exposing
themselves to hackers who are hell-bent on gaining access to
confidential information and raiding bank accounts the advice is
simple you should never open unsolicited attachments however tempting
they may appear
too many people are clicking without thinking exposing themselves to
hackers who are hell-bent on gaining access to confidential
information and raiding bank accounts
----------------------------------------------------------------------- 37 / 100
Kept: no
Tokens: one -> 1
Lemmas: one -> 1
assuming the economy bottoms out in the second half of 2009 job losses
are likely to continue into 2010 in all probability taking the final
toll of lost jobs to around one million
assuming the economy bottoms out in the second half of 2009 job losses
are likely to continue into 2010 in all probability taking the final
toll of lost jobs to around 1 million
----------------------------------------------------------------------- 38 / 100
Kept: no
Tokens: 1 -> one
Lemmas: 1 -> one
assuming the economy bottoms out in the second half of 2009 job losses
are likely to continue into 2010 in all probability taking the final
toll of lost jobs to around 1 million
assuming the economy bottoms out in the second half of 2009 job losses
are likely to continue into 2010 in all probability taking the final
toll of lost jobs to around one million
----------------------------------------------------------------------- 39 / 100
Kept: no
Tokens: if -> of
Lemmas: if -> of
if it did then i could walk away it does seem to fly in the face if
circumstantial evidence
if it did then i could walk away it does seem to fly in the face of
circumstantial evidence
----------------------------------------------------------------------- 40 / 100
Kept: no
Tokens: 18 -> eighteen
Lemmas: @card@ -> eighteen
we are convinced that we can overcome this crisis in a period of 18
months
we are convinced that we can overcome this crisis in a period of
eighteen months
----------------------------------------------------------------------- 41 / 100
Kept: no
Tokens: axe -> ax
Lemmas: axe -> ax
make sure you have an axe because you will be busting your way out to
get on your roof with waters surrounding you
make sure you have an ax
----------------------------------------------------------------------- 42 / 100
Kept: no
Tokens: on-line -> online
Lemmas: on-line -> online
it may have started on-line but it existed entirely in the real world
and it hurts just as much
it may have started online but it existed entirely in the real world
and it hurts just as much
----------------------------------------------------------------------- 43 / 100
Kept: no
Tokens: scrutinize -> scrutinise
Lemmas: scrutinize -> scrutinise
we will wait patiently listen to their words carefully scrutinize
their actions under a magnifier and if change happens truly and
fundamentally we will welcome that
we will wait patiently listen to their words carefully scrutinise
their actions under a magnifier and if change happens truly and
fundamentally we will welcome that
----------------------------------------------------------------------- 44 / 100
Kept: no
Tokens: as -> condemns
Lemmas: as -> condemn
strongly condemns the irresponsible provocations which led to this
situation as well as the disproportionate use of force
condemns the disproportionate use of force
----------------------------------------------------------------------- 45 / 100
Kept: no
Tokens: a -> the
Lemmas: a -> the
what i essentially said to him was five years ago when you were in a
position to buy your business you needed a tax cut then and what i
want to do is to make sure that the plumber the nurse the firefighter
the teacher the young entrepreneur who doesn't yet have money i want
to give them a tax break now
what i essentially said to him was five years ago when you were in the
position to buy your business you needed a tax cut then
----------------------------------------------------------------------- 46 / 100
Kept: no
Tokens: an -> steadiness
Lemmas: an -> steadiness
displayed a steadiness an intellectual curiosity a depth of knowledge
and an approach to looking at problems like this
steadiness intellectual curiosity a depth of knowledge
----------------------------------------------------------------------- 47 / 100
Kept: no
Tokens: recapitalise -> recapitalize
Lemmas: recapitalise -> recapitalize
we are far from the end of the road back to stability but the plan to
recapitalise our banking system both here and abroad will i believe
come to be seen as the moment in the banking crisis of the past year
when we turned the corner
we are far from the end of the road back to stability but the plan to
recapitalize our banking system both here and abroad will i believe
come to be seen as the moment in the banking crisis of the past year
when we turned the corner
----------------------------------------------------------------------- 48 / 100
Kept: no
Tokens: steadiness -> he
Lemmas: steadiness -> he
obama displayed a steadiness showed intellectual vigor he has a
definitive way of doing business that will do us well
he showed intellectual vigor he has a definitive way of doing business
that will do us well
----------------------------------------------------------------------- 49 / 100
Kept: no
Tokens: table -> we
Lemmas: table -> we
all of us around the table know we've got to get something done as
quickly as possible
we know we've got to get something done as quickly as possible
----------------------------------------------------------------------- 50 / 100
Kept: no
Tokens: eight -> 8
Lemmas: eight -> 8
if we are going to go down this road you know barack obama was eight
years old somehow responsible for bill ayers
if we are going to go down this road you know barack obama was 8 years
old somehow responsible for bill ayers
----------------------------------------------------------------------- 51 / 100
Kept: no
Tokens: that -> the
Lemmas: that -> the
come to ces to see all that palm new-ness you've been waiting for
come to ces to see all the palm new-ness you've been waiting for
----------------------------------------------------------------------- 52 / 100
Kept: no
Tokens: the -> that
Lemmas: the -> that
come to ces to see all the palm new-ness you've been waiting for
all that palm new-ness you've been waiting for
----------------------------------------------------------------------- 53 / 100
Kept: no
Tokens: general -> gen
Lemmas: general -> gen
george bush has given a mission to general petraeus and he has done
his best to try to figure out how to put lipstick on a pig
george bush has given a mission to gen petraeus and he has done his
best to try to figure out how to put lipstick on a pig
----------------------------------------------------------------------- 54 / 100
Kept: no
Tokens: hundred -> 200
Lemmas: hundred -> @card@
john mccain voted to cut education funding against accountability
standards he even proposed abolishing the department of education and
john mccain's economic plan gives two hundred billion more to special
interests while taking money away from public schools
200 billion more to special interests while taking money away from
public schools
----------------------------------------------------------------------- 55 / 100
Kept: no
Tokens: recognized -> recognised
Lemmas: recognize -> recognise
in 2006 i made a serious error in judgment and conducted myself in a
way that was disloyal to my family and to my core beliefs i recognized
my mistake and i told my wife that i had a liaison with another woman
and i asked for her forgiveness although i was honest in every painful
detail with my family i did not tell the public
in 2006 i made a serious error in judgment and conducted myself in a
way that was disloyal to my family and to my core beliefs i recognised
my mistake and i told my wife that i had a liaison with another woman
and i asked for her forgiveness
----------------------------------------------------------------------- 56 / 100
Kept: no
Tokens: high -> school
Lemmas: high -> school
really really good i went through a stage when i was in high school
when it wasn't so good but now i'm at a stage where i really
appreciate her a lot more
when i was in school
----------------------------------------------------------------------- 57 / 100
Kept: no
Tokens: brought -> been
Lemmas: bring -> be
we've had a steady stream of patients every day but the last 24 hours
has brought about triple the number of cases
we've had a steady stream of patients every day but the last 24 hours
has been about triple the number of cases
----------------------------------------------------------------------- 58 / 100
Kept: no
Tokens: it -> them
Lemmas: it -> them
taxes are always a redistribution of money most of the taxes that are
redistributed go back to those who pay it in roads and airports and
hospitals and schools and taxes are necessary for the common good and
there's nothing wrong with examining what our tax structure is and who
should be paying more who should be paying less and for us to say that
makes you a socialist i think is an unfortunate characterization and
is inaccurate
taxes are always a redistribution of money most of the taxes that are
redistributed go back to those who pay them in roads and airports and
hospitals and schools
----------------------------------------------------------------------- 59 / 100
Kept: no
Tokens: and -> or
Lemmas: and -> or
taxes are always a redistribution of money most of the taxes that are
redistributed go back to those who pay it in roads and airports and
hospitals and schools and taxes are necessary for the common good and
there's nothing wrong with examining what our tax structure is and who
should be paying more who should be paying less and for us to say that
makes you a socialist i think is an unfortunate characterization and
is inaccurate
and taxes are necessary for the common good and there's nothing wrong
with examining what our tax structure is or who should be paying more
who should be paying less
----------------------------------------------------------------------- 60 / 100
Kept: no
Tokens: living -> living
Lemmas: live -> living
my father saved my life i probably wouldn't be here if it wasn't for
him i've not always been a good daughter but he basically gave up his
job and his life to look after me at a time when i wasn't even sure i
wanted to go on living i owe him my life
basically gave up his job and his life to look after me at a time when
i wasn't even sure i wanted to go on living
----------------------------------------------------------------------- 61 / 100
Kept: no
Tokens: living -> living
Lemmas: live -> living
my father saved my life i probably wouldn't be here if it wasn't for
him i've not always been a good daughter but he basically gave up his
job and his life to look after me at a time when i wasn't even sure i
wanted to go on living i owe him my life
my father saved my life i probably wouldn't be here if it wasn't for
him i've not always been a good daughter but he basically gave up his
job and his life to look after me at a time when i wasn't even sure i
wanted to go on living
----------------------------------------------------------------------- 62 / 100
Kept: no
Tokens: mccain -> he
Lemmas: mccain -> he
then again that same day when asked how many houses he owns mccain
lost track he couldn't remember well it's seven seven houses
he lost track he couldn't remember
----------------------------------------------------------------------- 63 / 100
Kept: no
Tokens: n't -> not
Lemmas: n't -> not
i don't see there is anything wrong with it if everybody concerned
agrees
i do not see there is anything wrong with it
----------------------------------------------------------------------- 64 / 100
Kept: no
Tokens: everyone -> evryone
Lemmas: everyone -> evryone
i never planned otherwise everyone's making a big deal that i don't
have a contract signed i've never even asked to do that
i never planned otherwise evryone's making a big deal that i don't
have a contract signed i've never even asked to do that
----------------------------------------------------------------------- 65 / 100
Kept: no
Tokens: must -> to
Lemmas: must -> to
but we must insulate main street from wall street and keep people in
their homes by reducing mortgage foreclosures
to insulate main street from wall street
----------------------------------------------------------------------- 66 / 100
Kept: no
Tokens: current -> all
Lemmas: current -> all
our goal when the t-mobile g1 becomes available in october is to
provide affordable high-speed data service allowing customers to
experience the full data capabilities of the device and our 3g network
at the same time we have a responsibility to provide the best network
experience for all of our customers so we reserve the right to
temporarily reduce data throughput for a small fraction of our
customers who have excessive or disproportionate usage that interferes
with our network performance or our ability to provide quality service
to all of our customers we removed the 1gb soft limit from our policy
statement and we are confident that t-mobile g1 customers will enjoy
the high speed of data access over our 3g network the specific terms
for our new data plans are still being reviewed and once they are
final we will be certain to share this broadly with current customers
and potential new customers
the specific terms for our new data plans are still being reviewed and
once they are final we will be certain to share this broadly with all
customers
----------------------------------------------------------------------- 67 / 100
Kept: no
Tokens: in -> into
Lemmas: in -> into
not that i don't feel like i'm part of the team but when you get that
nice celebration coming in the dugout and you're getting your ass
hammered by guys there's no better feeling than to have that done
when you get that nice celebration coming into the dugout and you're
getting your ass hammered by guys there's no better feeling than to
have that done
----------------------------------------------------------------------- 68 / 100
Kept: no
Tokens: terrifying -> terrifying
Lemmas: terrify -> terrifying
i'm in a state of shock and the thought of looking like this at 57 is
terrifying having seen these images i can honestly say i am never
going to use a sunbed again
i'm in a state of shock and the thought of looking like this at 57 is
terrifying
----------------------------------------------------------------------- 69 / 100
Kept: no
Tokens: awfully -> awful
Lemmas: awfully -> awful
you're hearing an awfully lot about me most of which is not true but
you're not hearing a lot about you
you're hearing an awful lot about me most of which is not true
----------------------------------------------------------------------- 70 / 100
Kept: no
Tokens: patrolman -> ptlm
Lemmas: patrolman -> ptlm
the gas was very odorous and created contact of an insulting or
provoking nature with patrolman parsons
was very odorous and created contact of an insulting or provoking
nature with ptlm parsons
----------------------------------------------------------------------- 71 / 100
Kept: no
Tokens: prevailing -> prevailing
Lemmas: prevailing -> prevail
i wish that your voice will become the prevailing voice of the whole
region of all people
prevailing voice of the whole region
----------------------------------------------------------------------- 72 / 100
Kept: no
Tokens: fourteen -> 14
Lemmas: fourteen -> @card@
it's amazing how much women's tennis has grown since i joined the tour
fourteen years ago i am very proud to have reached this milestone for
me my family and all women athletes out there this achievement really
shows that if you put your mind to it you can achieve anything
it's amazing how much women's tennis has grown since i joined the tour
14 years ago
----------------------------------------------------------------------- 73 / 100
Kept: yes
Tokens: reckless -> callous
Lemmas: reckless -> callous
negligence so gross wanton and culpable as to show a reckless
disregard for human life
negligence so gross wanton and culpable as to show a callous disregard
for human life
----------------------------------------------------------------------- 74 / 100
Kept: yes
Tokens: election -> campaign
Lemmas: election -> campaign
this election is not about issues
this campaign is not about issues
----------------------------------------------------------------------- 75 / 100
Kept: yes
Tokens: top -> toughest
Lemmas: top -> tough
google doesn't exactly bubble to the top of the list of the top
competitors we've got going in mobile they might someday but right now
google doesn't exactly bubble to the top of the list of the toughest
competitors we've got going in mobile
----------------------------------------------------------------------- 76 / 100
Kept: yes
Tokens: contract -> patience
Lemmas: contract -> patience
i am at the end of my contract in may
at the end of my patience
----------------------------------------------------------------------- 77 / 100
Kept: yes
Tokens: unmentionable -> unimaginable
Lemmas: unmentionable -> unimaginable
if they think this is the way to go we will end up with millions of
small farmers all over the world being driven off their land into
unsustainable unmanageable degraded and dysfunctional conurbations of
unmentionable awfulness
unsustainable unmanageable degraded and dysfunctional conurbations of
unimaginable awfulness
----------------------------------------------------------------------- 78 / 100
Kept: no
Tokens: curios -> curious
Lemmas: curio -> curious
the curios case of benjamin button
the curious case of benjamin
----------------------------------------------------------------------- 79 / 100
Kept: yes
Tokens: hope -> partner
Lemmas: hope -> partner
i strongly condemn the people who've been stabbed and until they
abandon their practice of wandering around clutching their sides and
bleeding there is no hope for peace
there is no partner for peace
----------------------------------------------------------------------- 80 / 100
Kept: no
Tokens: honor -> honour
Lemmas: honor -> honour
his long and quiet devotion to advancing literacy the arts and
humanities on a national and local scale deserves our appreciation it
is our honor to bestow the guild's highest tribute on this
extraordinary actor
it is our honour to bestow the guild's highest tribute on this
extraordinary actor
----------------------------------------------------------------------- 81 / 100
Kept: no
Tokens: mid-air -> midair
Lemmas: mid-air -> midair
usually with a mid-air collision you have very serious damage and very
serious injuries if you have survivors at all
usually with a midair collision you have very serious damage and very
serious injuries if you have survivors at all
----------------------------------------------------------------------- 82 / 100
Kept: no
Tokens: which -> who
Lemmas: which -> who
those who come to britain must play by the rules and benefit the
country this new route for students will ensure we know exactly who is
coming here to study and stamp out bogus colleges which facilitate the
lawbreakers
those who come to britain must play by the rules and benefit the
country this new route for students will ensure we know exactly who is
coming here to study and stamp out bogus colleges who facilitate the
lawbreakers
----------------------------------------------------------------------- 83 / 100
Kept: yes
Tokens: clashes -> conflict
Lemmas: clash -> conflict
the perverse logic of clashes and violence
perverse logic of conflict and violence
----------------------------------------------------------------------- 84 / 100
Kept: yes
Tokens: clashes -> confrontation
Lemmas: clash -> confrontation
the perverse logic of clashes and violence
perverse logic of confrontation and violence
----------------------------------------------------------------------- 85 / 100
Kept: no
Tokens: de-mineralised -> demineralised
Lemmas: de-mineralised -> demineralise
both technologies work extremely well in hot dry desert locations csp
produces a lot of waste heat and we'd be able to use that to evaporate
more seawater from the greenhouse and csp needs a supply of clean de-
mineralised water in order for the electricity generating turbines to
function and to keep the mirrors at peak output it just so happens the
seawater greenhouse produces large quantities of this
and csp needs a supply of clean demineralised water in order for the
electricity generating turbines to function and to keep the mirrors at
peak output it just so happens the seawater greenhouse produces large
quantities of this
----------------------------------------------------------------------- 86 / 100
Kept: no
Tokens: yall -> y'all
Lemmas: yall -> y'all
it baffles me how yall let these blog and magazines control your lifes
it baffles me how y'all let these blog and magazines control your
lifes
----------------------------------------------------------------------- 87 / 100
Kept: yes
Tokens: commercial -> financial
Lemmas: commercial -> financial
it is evident that the group which carried out these attacks based
outside the country had come with single-minded determination to
create havoc in the commercial capital of the country
it is evident that the group which carried out these attacks based
outside the country had come with single-minded determination to
create havoc in the financial capital of the country
----------------------------------------------------------------------- 88 / 100
Kept: no
Tokens: the -> this
Lemmas: the -> this
subject to regulation under the act
subject to regulation under this act
----------------------------------------------------------------------- 89 / 100
Kept: no
Tokens: program -> programme
Lemmas: program -> programme
we will not permit any funds to be borrowed from the advanced
technology program unless there is a guarantee that those funds will
be replenished in a matter of weeks so as not to delay that crucial
initiative
we will not permit any funds to be borrowed from the advanced
technology programme unless there is a guarantee that those funds will
be replenished in a matter of weeks so as not to delay that crucial
initiative
----------------------------------------------------------------------- 90 / 100
Kept: no
Tokens: time -> anytime
Lemmas: time -> anytime
any time you can come back from a 24-6 deficit it's a building block
and something every team should be proud of
anytime you can come back from a 24-6 deficit it's a building block
----------------------------------------------------------------------- 91 / 100
Kept: no
Tokens: labour -> labor
Lemmas: labour -> labor
if barack obama had lived here i would be very surprised if even
somebody as brilliant as him would have been able to break through the
institutional stranglehold on power within the labour party
break through the institutional stranglehold on power within the labor
party
----------------------------------------------------------------------- 92 / 100
Kept: no
Tokens: decarbonisation -> decarburization
Lemmas: decarbonisation -> decarburization
preventing catastrophic and irreversible damage to the global climate
ultimately requires a major decarbonisation of the world energy
sources
preventing catastrophic and irreversible damage to the global climate
ultimately requires a major decarburization of the world energy
sources
----------------------------------------------------------------------- 93 / 100
Kept: no
Tokens: is -> has
Lemmas: be -> have
people will be appalled that the government is resorted to raiding
pension funds in order to plug the black hole in the public finances
people will be appalled that the government has resorted to raiding
pension funds in order to plug the black hole in the public finances
----------------------------------------------------------------------- 94 / 100
Kept: no
Tokens: trail-blazed -> trailblazed
Lemmas: trail-blazed -> trailblazed
my career in public service was made possible by the path heroes like
the tuskegee airmen trail-blazed
my career in public service was made possible by the path heroes like
the tuskegee airmen trailblazed
----------------------------------------------------------------------- 95 / 100
Kept: no
Tokens: assured -> assured
Lemmas: assure -> assured
nvidia assured apple that mac computers with these graphics processors
were not affected
assured apple that mac computers with these graphics processors were
not affected
----------------------------------------------------------------------- 96 / 100
Kept: no
Tokens: sitcoms -> sit-coms
Lemmas: sitcom -> sit-coms
for a kid who no one's talking to about sex and then he watches
sitcoms on tv where sex is presented as this is what the cool people
do
for a kid who no one's talking to about sex and then he watches sit-
coms on tv where sex is presented as this is what the cool people do
----------------------------------------------------------------------- 97 / 100
Kept: no
Tokens: makers -> policymakers
Lemmas: maker -> policymaker
policy makers may soon be forced to close financial markets as the
panic selling accelerates indeed we have now reached a point where
fundamentals and long term valuation considerations do not matter any
more for financial markets there is a free fall as most investors are
rapidly deleveraging and we are on the verge of a a capitulation
collapse what matters now is only flows rather than stocks and
fundamentals and flows are unidirectional as everyone is selling and
no one is buying as trying to buy equities is like catching a falling
knife there are no buyers in these dysfunctional markets only sellers
and panic is the ugly state of this destabilizing game we have reached
the scary point where the dysfunctional behavior of financial markets
has destructive effects on the financial system and much worse on the
real economies so it is time to think about more radical policy
actions and government interventions
policymakers may soon be forced to close financial markets as the
panic selling accelerates
----------------------------------------------------------------------- 98 / 100
Kept: no
Tokens: obama -> personally
Lemmas: obama -> personally
i've never made a racist comment and i never attacked him obama
personally
i've never made a racist comment and i never attacked him personally
----------------------------------------------------------------------- 99 / 100
Kept: no
Tokens: sen -> senator
Lemmas: sen -> senator
these are decisions that sen mccain has made i have great confidence
in him
these are decisions that senator mccain has made i have great
confidence in him
---------------------------------------------------------------------- 100 / 100
Kept: no
Tokens: christ -> jesus
Lemmas: christ -> jesus
mrs palin needs to be reminded that jesus christ was a community
organizer and pontius pilate was a governor
jesus was a community organizer and pontius pilate was a governor
Now with only valid substitutions, to make the precision analysis more accurate.
In [19]:
db_print_substitutions(Model(Time.discrete, Source.majority, Past.last_bin, Durl.all, 1))
------------------------------------------------------------------------ 1 / 100
Kept: yes
Tokens: required -> necessary
Lemmas: require -> necessary
the auto companies must not squander this chance to reform bad
management practices and begin the long-term restructuring that is
absolutely required to save this critical industry and the millions of
american jobs that depend on it
the auto companies must not squander this chance to reform bad
management practices and begin the long-term restructuring that is
absolutely necessary to save this critical industry and the millions
of american jobs that depend on it
------------------------------------------------------------------------ 2 / 100
Kept: yes
Tokens: info -> information
Lemmas: info -> information
i'm not kidding around can you put that aside and understand that the
case we are trying here and the info you're going to hear about here
is totally separate from that case
can you put that aside and understand that the case we are trying here
and the information you're going to hear about here is totally
separate from that case
------------------------------------------------------------------------ 3 / 100
Kept: yes
Tokens: think -> believe
Lemmas: think -> believe
we think a decision could be taken but i think it will happen in
algeria
we believe a decision could be taken but i think it will happen in
algeria
------------------------------------------------------------------------ 4 / 100
Kept: yes
Tokens: automakers -> autos
Lemmas: automaker -> auto
the automakers will get the money as quickly as we can prudently do it
the autos will get the money as quickly as we can prudently do it
------------------------------------------------------------------------ 5 / 100
Kept: yes
Tokens: prepared -> ready
Lemmas: prepare -> ready
are you prepared to take the oath senator
are you ready to take the oath
------------------------------------------------------------------------ 6 / 100
Kept: yes
Tokens: material -> physical
Lemmas: material -> physical
he hates the american material occupation as much as he hates the
iranian moral occupation
he hates the american physical occupation as much as he hates the
iranian moral occupation
------------------------------------------------------------------------ 7 / 100
Kept: yes
Tokens: willing -> going
Lemmas: willing -> go
they're not willing to give me anything except appreciation expletive
them
not going to give me anything except appreciation
------------------------------------------------------------------------ 8 / 100
Kept: yes
Tokens: manufacturing -> producing
Lemmas: manufacturing -> produce
according to the criminal law there is no capital punishment for the
crime of manufacturing and selling fake or substandard products
therefore the most severe penalty for her would be life imprisonment
producing and selling fake or substandard products
------------------------------------------------------------------------ 9 / 100
Kept: yes
Tokens: truly -> really
Lemmas: truly -> really
we had a relationship that went far beyond friendship j rg and i were
connected by something truly special he was the man of my life
we had a relationship that went far beyond friendship j rg and i were
connected by something really special he was the man of my life
----------------------------------------------------------------------- 10 / 100
Kept: yes
Tokens: determination -> reflections
Lemmas: determination -> reflection
they misunderestimated the compassion of our country i think they
misunderestimated the will and determination of the commander in chief
too
reflections of the commander in chief
----------------------------------------------------------------------- 11 / 100
Kept: yes
Tokens: catch -> get
Lemmas: catch -> get
the republicans can't seem to catch a break when it comes to august
and when it comes to the weather
the republicans can't seem to get a break when it comes to august and
when it comes to the weather
----------------------------------------------------------------------- 12 / 100
Kept: yes
Tokens: situation -> crisis
Lemmas: situation -> crisis
the most serious financial situation in generations
the most serious financial crisis in generations
----------------------------------------------------------------------- 13 / 100
Kept: yes
Tokens: unmentionable -> unimaginable
Lemmas: unmentionable -> unimaginable
if they think this is the way to go we will end up with millions of
small farmers all over the world being driven off their land into
unsustainable unmanageable degraded and dysfunctional conurbations of
unmentionable awfulness
unsustainable unmanageable degraded and dysfunctional conurbations of
unimaginable awfulness
----------------------------------------------------------------------- 14 / 100
Kept: yes
Tokens: witnessed -> seen
Lemmas: witness -> see
palestine has never witnessed an uglier massacre
palestine has never seen an uglier massacre
----------------------------------------------------------------------- 15 / 100
Kept: yes
Tokens: socialism -> welfare
Lemmas: socialism -> welfare
this is socialism for the rich
this is welfare for the rich
----------------------------------------------------------------------- 16 / 100
Kept: yes
Tokens: scared -> afraid
Lemmas: scare -> afraid
this is a menace a cancer in pakistan which we will eliminate we will
not be scared of these cowards
we will not be afraid of these cowards
----------------------------------------------------------------------- 17 / 100
Kept: yes
Tokens: blind -> blank
Lemmas: blind -> blank
this is no blank check or blind hope
this is no blank check or blank hope
----------------------------------------------------------------------- 18 / 100
Kept: yes
Tokens: greatest -> biggest
Lemmas: great -> big
mexican dtos represent the greatest organized crime threat to the
united states
biggest organized crime threat to the united states
----------------------------------------------------------------------- 19 / 100
Kept: yes
Tokens: admire -> respect
Lemmas: admire -> respect
but we will be respectful i admire sen obama and his accomplishments
i respect sen obama and his accomplishments
----------------------------------------------------------------------- 20 / 100
Kept: yes
Tokens: agency -> cia
Lemmas: agency -> cia
my position has consistently been that i believe the agency is best-
served by having an intelligence professional in charge at this time
my position has consistently been that i believe the cia is best-
served by having an intelligence professional in charge at this time
----------------------------------------------------------------------- 21 / 100
Kept: yes
Tokens: things -> faith
Lemmas: thing -> faith
this new energy plan is straightforward highly cost-effective and
creates a clean energy pathway for the rest of the country to follow
it is the shape of things to come
the shape of faith to come
----------------------------------------------------------------------- 22 / 100
Kept: yes
Tokens: crisis -> problem
Lemmas: crisis -> problem
this crisis did not develop overnight and it will not be solved
overnight
this problem did not develop overnight and it will not be solved
overnight
----------------------------------------------------------------------- 23 / 100
Kept: yes
Tokens: clashes -> confrontation
Lemmas: clash -> confrontation
the perverse logic of clashes and violence
perverse logic of confrontation and violence
----------------------------------------------------------------------- 24 / 100
Kept: yes
Tokens: cool -> steady
Lemmas: cool -> steady
a cool hand at the tiller
steady hand at the tiller
----------------------------------------------------------------------- 25 / 100
Kept: yes
Tokens: study -> post
Lemmas: study -> post
this is an extremely important study
this is an extremely important post
----------------------------------------------------------------------- 26 / 100
Kept: yes
Tokens: patients -> parents
Lemmas: patient -> parent
this is good news for patients
this is good news for parents
----------------------------------------------------------------------- 27 / 100
Kept: yes
Tokens: selling -> changing
Lemmas: sell -> change
when a man loves a woman he can't conceivably sell that love i'm in
love with arsenal i have no intention of selling it my stake
i have no intention of changing it
----------------------------------------------------------------------- 28 / 100
Kept: yes
Tokens: believe -> feel
Lemmas: believe -> feel
we believe it is inappropriate to use young private citizens for
marketing purposes
we feel it is inappropriate to use young private citizens for
marketing purposes
----------------------------------------------------------------------- 29 / 100
Kept: yes
Tokens: portray -> reflect
Lemmas: portray -> reflect
the intent was to portray a renaissance-like mood on the cover
the intent was to reflect a renaissance-like mood on the cover
----------------------------------------------------------------------- 30 / 100
Kept: yes
Tokens: sec -> commission
Lemmas: sec -> commission
i am disappointed that the sec chose to bring this case based upon its
enforcement staff's win-at-any-cost ambitions the staff's process was
result-oriented facts be damned the government's claims are false and
they will be proven to be so
i am disappointed that the commission chose to bring this case based
upon its enforcement staff's win-at-any-cost ambitions
----------------------------------------------------------------------- 31 / 100
Kept: yes
Tokens: check -> cheque
Lemmas: check -> cheque
is not a blank check
is not a blank cheque
----------------------------------------------------------------------- 32 / 100
Kept: yes
Tokens: instability -> volatility
Lemmas: instability -> volatility
while market imbalances will feed instability the era of cheap oil is
over
while market imbalances will feed volatility the era of cheap oil is
over
----------------------------------------------------------------------- 33 / 100
Kept: yes
Tokens: friday -> monday
Lemmas: friday -> monday
america feels differently today than it did on friday about this bill
and he believes the house vote tomorrow will reflect that shift
america feels differently today than it did on monday about this bill
----------------------------------------------------------------------- 34 / 100
Kept: yes
Tokens: solution -> body
Lemmas: solution -> body
blaming cities for greenhouse gas emissions misses the point that
cities are a large part of the solution
a large part of the body
----------------------------------------------------------------------- 35 / 100
Kept: yes
Tokens: fine -> dedicated
Lemmas: fine -> dedicated
roland burris is a good man and a fine public servant but the senate
democrats made it clear weeks ago that they cannot accept an
appointment made by a governor who is accused of selling this very
senate seat i agree with their decision and it is extremely
disappointing that governor blagojevich has chosen to ignore it while
governor blagojevich is entitled to his day in court the people of
illinois are entitled to a functioning government and major decisions
free of taint and controversy
roland burris is a good man and a dedicated public servant
----------------------------------------------------------------------- 36 / 100
Kept: yes
Tokens: manufacture -> design
Lemmas: manufacture -> design
custom manufacture of computer hardware for use in data centers and
mega-scale computing environments for others
design of computer hardware for use in data centers and mega-scale
computing environments for others
----------------------------------------------------------------------- 37 / 100
Kept: yes
Tokens: frail -> fragile
Lemmas: frail -> fragile
it is clear that the financial system although improved from where it
was in september is still frail
it is clear that the financial system although improved from where it
was in september is still fragile
----------------------------------------------------------------------- 38 / 100
Kept: yes
Tokens: sec -> commission
Lemmas: sec -> commission
i am disappointed that the sec chose to bring this case based upon its
enforcement staff's win-at-any-cost ambitions the staff's process was
result-oriented facts be damned the government's claims are false and
they will be proven to be so
i am disappointed that the commission chose to bring this case based
upon its enforcement staff's win-at-any-cost ambitions the staff's
process was result-oriented facts be damned the government's claims
are false and they will be proven to be so
----------------------------------------------------------------------- 39 / 100
Kept: yes
Tokens: know -> boast
Lemmas: know -> boast
you can't change america when you know your first four years as
president will look exactly like the last eight years of george bush's
presidency
you can't change america when you boast
----------------------------------------------------------------------- 40 / 100
Kept: yes
Tokens: opportunity -> deal
Lemmas: opportunity -> deal
strategic opportunity of a lifetime
strategic deal of a lifetime
----------------------------------------------------------------------- 41 / 100
Kept: yes
Tokens: paid -> trained
Lemmas: pay -> train
we were simply doing the jobs we were paid to do
we were simply doing the jobs we were trained to do
----------------------------------------------------------------------- 42 / 100
Kept: yes
Tokens: borders -> limit
Lemmas: border -> limit
from tomorrow representatives of the european union will begin
conducting monitoring up to the southern borders of the security zone
up to the southern limit of the security zone
----------------------------------------------------------------------- 43 / 100
Kept: yes
Tokens: required -> necessary
Lemmas: require -> necessary
the auto companies must not squander this chance to reform bad
management practices and begin the long-term restructuring that is
absolutely required to save this critical industry and the millions of
american jobs that depend on it
squander this chance to reform bad management practices and begin the
long-term restructuring that is absolutely necessary to save this
critical industry and the millions of american jobs that depend on it
----------------------------------------------------------------------- 44 / 100
Kept: yes
Tokens: massive -> grave
Lemmas: massive -> grave
a flagrant and massive violation of international humanitarian law as
laid down in article 33 of the fourth geneva convention
grave violation of international humanitarian law
----------------------------------------------------------------------- 45 / 100
Kept: yes
Tokens: chatty -> friendly
Lemmas: chatty -> friendly
they were very chatty pamela was being her usual flirty self and
michael seemed to be responding they are such a strange couple but
they seemed to really hit it off
they were very friendly pamela was being her usual flirty self and
michael seemed to be responding they are such a strange couple but
they seemed to really hit it off
----------------------------------------------------------------------- 46 / 100
Kept: yes
Tokens: world -> global
Lemmas: world -> global
the united states will lose its superpower status in the world
financial system the world financial system will become more multi-
polar
the united states will lose its superpower status in the global
financial system
----------------------------------------------------------------------- 47 / 100
Kept: yes
Tokens: dolls -> girls
Lemmas: doll -> girl
there's nothing on the dolls that refers to the obama girls
there's nothing on the girls that refers to the obama girls
----------------------------------------------------------------------- 48 / 100
Kept: yes
Tokens: candidates -> politicians
Lemmas: candidate -> politician
with the economy in turmoil and the american dream at risk the
american people don't want to hear candidates attack each other you
want to hear how we're going to attack the challenges facing middle
class families each and every day
with the economy in turmoil and the american dream at risk the
american people don't want to hear politicians attack each other
----------------------------------------------------------------------- 49 / 100
Kept: yes
Tokens: presidency -> president
Lemmas: presidency -> president
securing cyberspace for the 44th presidency
securing cyberspace for the 44th president
----------------------------------------------------------------------- 50 / 100
Kept: yes
Tokens: i -> bye
Lemmas: i -> bye
i'm thinking rihanna i'll see you at the white house oh and i might
paint it pink i hope that's cool with you guys
oh and i might paint it pink bye
----------------------------------------------------------------------- 51 / 100
Kept: yes
Tokens: diligently -> vigilantly
Lemmas: diligently -> vigilantly
we must adhere to our values as diligently as we protect our safety
with no exceptions
adhere to our values as vigilantly as we protect our safety with no
exceptions
----------------------------------------------------------------------- 52 / 100
Kept: yes
Tokens: f -> fuck
Lemmas: f -> fuck
f off elton i'm 40 years younger than you i have my whole life ahead
of me
fuck off elton i'm 40 years younger than you i have my whole life
ahead of me
----------------------------------------------------------------------- 53 / 100
Kept: yes
Tokens: lunchtime -> luncheon
Lemmas: lunchtime -> luncheon
at a lunchtime meeting about school reform in a chicago skyscraper
at a luncheon meeting about school reform
----------------------------------------------------------------------- 54 / 100
Kept: yes
Tokens: countries -> folks
Lemmas: country -> folk
we have to stop sending 700 billion a year to countries that don't
like us very much
folks that don't like us very much
----------------------------------------------------------------------- 55 / 100
Kept: yes
Tokens: feel -> believe
Lemmas: feel -> believe
we feel it is inappropriate to use young private citizens for
marketing purposes
we believe it is inappropriate to use young private citizens for
marketing purposes
----------------------------------------------------------------------- 56 / 100
Kept: yes
Tokens: palestine -> palestinian
Lemmas: palestine -> palestinian
the israeli military must have been aware of the situation but did not
assist the wounded neither did they make it possible for us or the
palestine red crescent to assist the wounded
the israeli military must have been aware of the situation but did not
assist the wounded neither did they make it possible for us or the
palestinian red crescent to assist the wounded
----------------------------------------------------------------------- 57 / 100
Kept: yes
Tokens: expletive -> fuck
Lemmas: expletive -> fuck
they're not willing to give me anything except appreciation expletive
them
they're not willing to give me anything except appreciation fuck them
----------------------------------------------------------------------- 58 / 100
Kept: yes
Tokens: i -> love
Lemmas: i -> love
i'm bringing a message of peace and i think that's what the region
needs
a message of peace and love
----------------------------------------------------------------------- 59 / 100
Kept: yes
Tokens: men -> britons
Lemmas: man -> briton
we call on those holding the men to release them and the rest of the
crew immediately
we call on those holding the britons to release them and the rest of
the crew immediately
----------------------------------------------------------------------- 60 / 100
Kept: yes
Tokens: food -> agricultural
Lemmas: food -> agricultural
we should go back to a policy of maximum food self-sufficiency it is
crazy for us to think we can develop countries around the world
without increasing their ability to feed themselves
we should go back to a policy of maximum agricultural self-sufficiency
----------------------------------------------------------------------- 61 / 100
Kept: yes
Tokens: see -> watch
Lemmas: see -> watch
it will be a stirring sight to see president obama his wife michelle
and their beautiful girls step through the doors of the white house
it will be a stirring sight to watch president obama his wife michelle
and their beautiful girls step through the doors of the white house
----------------------------------------------------------------------- 62 / 100
Kept: yes
Tokens: pipeline -> line
Lemmas: pipeline -> line
i think god's will has to be done in unifying people and companies to
get that gas pipeline built so pray for that i can do my job there in
developing my natural resources but all of that doesn't do any good if
the people of alaska's heart is not good with god
i think god's will has to be done in unifying people and companies to
get that gas line built so pray for that
----------------------------------------------------------------------- 63 / 100
Kept: yes
Tokens: disorganized -> scattered
Lemmas: disorganize -> scatter
the aggressor has been punished and suffered very significant losses
its military has been disorganized
the aggressor has been punished and suffered very significant losses
its military has been scattered
----------------------------------------------------------------------- 64 / 100
Kept: yes
Tokens: solution -> position
Lemmas: solution -> position
like all compromises this one has its drawbacks it includes a limited
amount of new offshore drilling and while i still don't believe that's
a particularly meaningful short-term or long-term solution i am
willing to consider it if it's necessary to actually pass a
comprehensive plan
i still don't believe that's a particularly meaningful short-term or
long-term position
----------------------------------------------------------------------- 65 / 100
Kept: yes
Tokens: concept -> conception
Lemmas: concept -> conception
the invasion of iraq was a bandit act an act of blatant state
terrorism demonstrating absolute contempt for the concept of
international law the invasion was an act intended to consolidate
american military and economic control of the middle east masquerading
as a last resort all other justifications having failed to justify
themselves as liberation
a bandit act an act of blatant state terrorism demonstrating absolute
contempt for the conception of international law
----------------------------------------------------------------------- 66 / 100
Kept: yes
Tokens: athletes -> world
Lemmas: athlete -> world
take significant action to prevent the spread and use of performance-
enhancing substances i think it's a game we're going to be in for a
long time what i mean by that is there is somebody in a laboratory
right now trying to develop some type of substance that can't be
detected and we've got to stay ahead of it it's not good for the
athletes it's not good for the sports it's very bad for those who
don't do it and i think it can attack the very integrity of all sports
going all the way down to high school
it's not good for the world
----------------------------------------------------------------------- 67 / 100
Kept: yes
Tokens: fuck -> f
Lemmas: fuck -> f
it's dan the man fuck me you look like a raghead
f me you look like a raghead
----------------------------------------------------------------------- 68 / 100
Kept: yes
Tokens: testing -> tests
Lemmas: testing -> test
at this point field tests on the powder have been negative additional
testing will be completed even sending a hoax letter is a serious
crime
at this point field tests on the powder have been negative additional
tests will be completed
----------------------------------------------------------------------- 69 / 100
Kept: yes
Tokens: meet -> honor
Lemmas: meet -> honor
we expect russia to meet its commitment to cease all military
activities in georgia and we expect all russian forces that entered
georgia in recent days to withdraw from that country
we expect russia to honor its commitment
----------------------------------------------------------------------- 70 / 100
Kept: yes
Tokens: african -> african-american
Lemmas: african -> african-american
in the face of all these attacks that are clearly being made on the
african community why is it that you have not had the ability to not
one time speak to the interests and even speak on behalf of the
oppressed and exploited african community or black community in this
country
why is it that you have not had the ability to not one time speak to
the interests and even speak on behalf of the oppressed and exploited
african-american community or black community in this country
----------------------------------------------------------------------- 71 / 100
Kept: yes
Tokens: inaugural -> inauguration
Lemmas: inaugural -> inauguration
we are one the obama inaugural celebration at the lincoln memorial
we are one the obama inauguration celebration at the lincoln memorial
----------------------------------------------------------------------- 72 / 100
Kept: yes
Tokens: story -> history
Lemmas: story -> history
standing on the steps of the capitol will be a man whose story
reflects the enduring promise of our land
standing on the steps of the capitol will be a man whose history
reflects the enduring promise of our land
----------------------------------------------------------------------- 73 / 100
Kept: yes
Tokens: story -> history
Lemmas: story -> history
standing on the steps of the capitol will be a man whose story
reflects the enduring promise of our land
a man whose history reflects the enduring promise of our land
----------------------------------------------------------------------- 74 / 100
Kept: yes
Tokens: fragile -> frail
Lemmas: fragile -> frail
it is clear that the financial system although improved from where it
was in september is still fragile and i felt that it would be
irresponsible for me with the first 350 billion already spent to enter
into the administration without any potential ammunition should there
be some sort of emergency or weakening of the financial system
it is clear that the financial system although improved from where it
was in september is still frail
----------------------------------------------------------------------- 75 / 100
Kept: yes
Tokens: careful -> danger
Lemmas: careful -> danger
sterling has devalued rapidly against the euro and the dollar we are
in danger if the government is not careful of having a proper sterling
collapse a run on the pound
danger of having a proper sterling collapse a run on the pound
----------------------------------------------------------------------- 76 / 100
Kept: yes
Tokens: fuck -> expletive
Lemmas: fuck -> expletive
they're not willing to give me anything except appreciation fuck them
willing to give me anything except appreciation expletive them
----------------------------------------------------------------------- 77 / 100
Kept: yes
Tokens: government -> washington
Lemmas: government -> washington
the american people would be troubled if i selected a treasury
secretary or a chairman of the national economic council at one of the
most critical economic times in our history who had no experience in
government whatsoever
had no experience in washington whatsoever
----------------------------------------------------------------------- 78 / 100
Kept: yes
Tokens: agreement -> text
Lemmas: agreement -> text
establishing checkpoints buffer zones are definitely not part of the
agreement
establishing checkpoints buffer zones are definitely not part of the
text
----------------------------------------------------------------------- 79 / 100
Kept: yes
Tokens: markets -> economy
Lemmas: market -> economy
our economy and our markets will not recover until the bulk of this
housing correction is behind us
our economy will not recover until the bulk of this housing correction
is behind us
----------------------------------------------------------------------- 80 / 100
Kept: yes
Tokens: crisis -> challenge
Lemmas: crisis -> challenge
immediately after i become president i will confront this economic
crisis head-on by taking all necessary steps to ease the credit crisis
help hardworking families and restore growth and prosperity
immediately after i become president i will confront this economic
challenge head-on by taking all necessary steps to ease the credit
crisis help hardworking families and restore growth and prosperity
----------------------------------------------------------------------- 81 / 100
Kept: yes
Tokens: inaugural -> inauguration
Lemmas: inaugural -> inauguration
we are one the obama inaugural celebration at the lincoln memorial
we are one the obama inauguration celebration
----------------------------------------------------------------------- 82 / 100
Kept: yes
Tokens: platform -> plat
Lemmas: platform -> plat
there has been absolutely no discussion about having sarah palin on my
show at the beginning of this presidential campaign when i decided
that i was going to take my first public stance in support of a
candidate i made the decision not to use my show as a platform for any
of the candidates i agree that sarah palin would be a fantastic
interview and i would love to have her on after the campaign is over
at the beginning of this presidential campaign when i decided that i
was going to take my first public stance in support of a candidate i
made the decision not to use my show as a plat for any of the
candidates
----------------------------------------------------------------------- 83 / 100
Kept: yes
Tokens: fossil -> arabs
Lemmas: fossil -> arab
are bought and paid for by the fossil fuel industry usually
bought and paid for by the arabs
----------------------------------------------------------------------- 84 / 100
Kept: yes
Tokens: rightfully -> rightly
Lemmas: rightfully -> rightly
the american people are angry about executive compensation and
rightfully so
the american people are angry about executive compensation and rightly
so
----------------------------------------------------------------------- 85 / 100
Kept: yes
Tokens: fast -> quickly
Lemmas: fast -> quickly
we didn't set out to conquer gaza we didn't set out to control gaza we
don't want to remain in gaza and we intend on leaving gaza as fast as
possible
we didn't set out to control gaza we don't want to remain in gaza and
we intend on leaving gaza as quickly as possible
----------------------------------------------------------------------- 86 / 100
Kept: yes
Tokens: get -> buy
Lemmas: get -> buy
it's just a dog you can get another one
you can buy another one
----------------------------------------------------------------------- 87 / 100
Kept: yes
Tokens: grandpa -> bye
Lemmas: grandpa -> bye
granny i've got to go send my love to the corgis and grandpa god save
you yeah that's great see you bye
bye god save you yeah that's great
----------------------------------------------------------------------- 88 / 100
Kept: yes
Tokens: system -> situation
Lemmas: system -> situation
whether it's government or the financial system we're not going to be
able to do what is needed to be done to stabilize our financial system
if the american people read about this type of outrageous behavior
we're not going to be able to do what is needed to be done to
stabilize our financial situation if the american people read about
this type of outrageous behavior
----------------------------------------------------------------------- 89 / 100
Kept: yes
Tokens: investigating -> recovering
Lemmas: investigate -> recover
we are in the process of investigating this
are in the process of recovering
----------------------------------------------------------------------- 90 / 100
Kept: yes
Tokens: confrontation -> conflict
Lemmas: confrontation -> conflict
all it can to help the israelis and palestinians on this dead-end road
and not to give in to the perverse logic of confrontation and violence
perverse logic of conflict and violence
----------------------------------------------------------------------- 91 / 100
Kept: yes
Tokens: expletive -> f
Lemmas: expletive -> f
don't let nobody outta here stand the expletive up before it gets ugly
in here
stand the f up before it gets ugly in here
----------------------------------------------------------------------- 92 / 100
Kept: yes
Tokens: cause -> create
Lemmas: cause -> create
a failure of either of them would cause great turmoil in our financial
markets here at home and around the globe
create great turmoil in our financial markets here at home and around
the globe
----------------------------------------------------------------------- 93 / 100
Kept: yes
Tokens: libra -> lyrics
Lemmas: libra -> lyric
dancing with the stars toni braxton libra
dancing with the stars toni braxton lyrics
----------------------------------------------------------------------- 94 / 100
Kept: yes
Tokens: administration -> administrative
Lemmas: administration -> administrative
decisions by the secretary pursuant to the authority of this act are
non-reviewable and committed to agency discretion and may not be
reviewed by any court of law or any administration agency
non-reviewable and committed to agency discretion and may not be
reviewed by any court of law or any administrative agency
----------------------------------------------------------------------- 95 / 100
Kept: yes
Tokens: press -> pressure
Lemmas: press -> pressure
the mexican possibility may seem less likely but the government its
politicians police and judicial infrastructure are all under sustained
assault and press by criminal gangs and drug cartels how that internal
conflict turns out over the next several years will have a major
impact on the stability of the mexican state any descent by mexico
into chaos would demand an american response based on the serious
implications for homeland security alone
the mexican possibility may seem less likely but the government its
politicians police and judicial infrastructure are all under sustained
assault and pressure by criminal gangs and drug cartels how that
internal conflict turns out over the next several years will have a
major impact on the stability of the mexican state any descent by
mexico into chaos would demand an american response based on the
serious implications for homeland security alone
----------------------------------------------------------------------- 96 / 100
Kept: yes
Tokens: laptop -> notebook
Lemmas: laptop -> notebook
the laptop pc is no longer a tool only for the business market or a
computer for the well-off consumer it's now a computer for everyman
the notebook pc is no longer a tool only for the business market or a
computer for the well-off consumer it's now a computer for everyman
----------------------------------------------------------------------- 97 / 100
Kept: yes
Tokens: bill -> act
Lemmas: bill -> act
american recovery and reinvestment bill
american recovery and reinvestment act
----------------------------------------------------------------------- 98 / 100
Kept: yes
Tokens: tv -> television
Lemmas: tv -> television
the barack obama campaign is asking missouri law enforcement to target
anyone who lies or runs a misleading tv ad during the presidential
campaign
anyone who lies or runs a misleading television ad
----------------------------------------------------------------------- 99 / 100
Kept: yes
Tokens: press -> pressure
Lemmas: press -> pressure
the mexican possibility may seem less likely but the government its
politicians police and judicial infrastructure are all under sustained
assault and press by criminal gangs and drug cartels how that internal
conflict turns out over the next several years will have a major
impact on the stability of the mexican state any descent by mexico
into chaos would demand an american response based on the serious
implications for homeland security alone
the mexican possibility may seem less likely but the government its
politicians police and judicial infrastructure are all under sustained
assault and pressure by criminal gangs and drug cartels
---------------------------------------------------------------------- 100 / 100
Kept: yes
Tokens: case -> need
Lemmas: case -> need
now that there is no cholera there is no case for war
now that there is no cholera there is no need for war
Here, the only possible source of bad filtering is stopwords and rejection because of levenshtein distance <= 1 (the other checks are very specific).
We already know we don't want substitutions on stopwords because we want to focus on meaningful changes, so the question is:
The codings the substitutions are in codings/filter_evaluations_substitutions-precision-recall; precision answers the first question, recall the second one.
Now the same with double-substitutions.
In [20]:
mine_print_substitutions(Model(Time.discrete, Source.majority, Past.last_bin, Durl.all, 2))
------------------------------------------------------------------------ 1 / 100
Kept: no
Tokens: that -> it
Lemmas: that -> it
i hope laura and i did the same thing but i believe he will and i know
his girls are on his mind and he wants to make sure that first and
foremost he is a good dad and i think that's going to be an important
part of his presidency
but i believe he will and i know his girls are on his mind and he
wants to make sure that first and foremost he's a good dad and i think
it's going to be an important part of his presidency
------------------------------------------------------------------------ 2 / 100
Kept: no
Tokens: unmistakeably -> unmistakably
Lemmas: unmistakeably -> unmistakably
the finger of suspicion unmistakeably points to the territory of our
neighbour pakistan
the finger of suspicion unmistakably points to the territory of our
neighbor pakistan
------------------------------------------------------------------------ 3 / 100
Kept: no
Tokens: neighbour -> neighbor
Lemmas: neighbour -> neighbor
the finger of suspicion unmistakeably points to the territory of our
neighbour pakistan
the finger of suspicion unmistakably points to the territory of our
neighbor pakistan
------------------------------------------------------------------------ 4 / 100
Kept: no
Tokens: plans -> actions
Lemmas: plan -> action
client has a history dating to his graduate days of homicidal threats
plans and actions towards therapists
client has a history dating to his graduate days of homicidal threats
actions plans
------------------------------------------------------------------------ 5 / 100
Kept: no
Tokens: and -> plans
Lemmas: and -> plan
client has a history dating to his graduate days of homicidal threats
plans and actions towards therapists
client has a history dating to his graduate days of homicidal threats
actions plans
------------------------------------------------------------------------ 6 / 100
Kept: no
Tokens: higher-level -> higher
Lemmas: higher-level -> high
going forward we'll use jquery as one of the libraries used to
implement higher-level controls in the asp net ajax control toolkit as
well as to implement new ajax server-side helper methods for asp net
mvc
going forward we'll use jquery as one of the libraries used to
implement higher
------------------------------------------------------------------------ 7 / 100
Kept: no
Tokens: 's -> 's
Lemmas: be -> 's
this is proof of the microscope's potential we are now ready to start
doing scientific experiments that will add a new dimension to
measurements being made by other phoenix lander instruments
this is proof of the microscope's potential
------------------------------------------------------------------------ 8 / 100
Kept: no
Tokens: superconducting -> discovery
Lemmas: superconducting -> discovery
because the lhc is a superconducting machine that works at very low
temperatures in order to get in and fix it we've got to warm it up
then we go and fix it and then we cool it down again and that's a
process that's likely to take two months
the lhc is a discovery machine
------------------------------------------------------------------------ 9 / 100
Kept: no
Tokens: beginning -> beginning
Lemmas: beginning -> begin
a momentous economic slowdown is now under way we believe the
deterioration in spain is just in the beginning stages the bulk of the
pain will be suffered in 2009
a momentous economic slowdown is now under way we believe the
deterioration in spain is just in the beginning stages
----------------------------------------------------------------------- 10 / 100
Kept: no
Tokens: expect -> i
Lemmas: expect -> i
i expect us to make the playoffs
i expect to make the playoffs
----------------------------------------------------------------------- 11 / 100
Kept: no
Tokens: us -> expect
Lemmas: us -> expect
i expect us to make the playoffs
i expect to make the playoffs
----------------------------------------------------------------------- 12 / 100
Kept: no
Tokens: could -> can
Lemmas: could -> can
i am so damn glad that you could never be my wife cause i surely
wouldn't have to listen to that prattle from you every day
i am so damn glad that you can never be my wife cause i surely
wouldn't have to listen to that prattle from you every day
----------------------------------------------------------------------- 13 / 100
Kept: no
Tokens: theaters -> theatres
Lemmas: theater -> theatre
the theaters are closed there is just no business at all down there
the theatres are closed there is just no business at all down there
----------------------------------------------------------------------- 14 / 100
Kept: no
Tokens: we -> this
Lemmas: we -> this
we may be seeing the tip of the iceberg of a trend towards less
comprehensive skimpier health insurance coverage for many working
people
this may be only the tip of the iceberg
----------------------------------------------------------------------- 15 / 100
Kept: no
Tokens: seeing -> only
Lemmas: see -> only
we may be seeing the tip of the iceberg of a trend towards less
comprehensive skimpier health insurance coverage for many working
people
this may be only the tip of the iceberg
----------------------------------------------------------------------- 16 / 100
Kept: no
Tokens: that -> who
Lemmas: that -> who
from the very beginning i have said i am going to support the
candidate that has the best chance for changing the way washington
works and getting things done and i will be voting for barack obama
and clapping
from the very beginning i've said i'm going to support the candidate
who has the best chance of changing the way washington works and
getting things done
----------------------------------------------------------------------- 17 / 100
Kept: no
Tokens: for -> of
Lemmas: for -> of
from the very beginning i have said i am going to support the
candidate that has the best chance for changing the way washington
works and getting things done and i will be voting for barack obama
and clapping
from the very beginning i've said i'm going to support the candidate
who has the best chance of changing the way washington works and
getting things done
----------------------------------------------------------------------- 18 / 100
Kept: no
Tokens: will -> not
Lemmas: will -> not
by one complicated measure the number of votes obama may lose due to
his race is roughly equal to the number who will vote for him because
he is black and both those numbers appear to be small possibly just 1
percentage point in each direction
not vote for obama because he is black
----------------------------------------------------------------------- 19 / 100
Kept: no
Tokens: him -> obama
Lemmas: him -> obama
by one complicated measure the number of votes obama may lose due to
his race is roughly equal to the number who will vote for him because
he is black and both those numbers appear to be small possibly just 1
percentage point in each direction
not vote for obama because he is black
----------------------------------------------------------------------- 20 / 100
Kept: no
Tokens: sovereignty -> sovereign
Lemmas: sovereignty -> sovereign
what is clear is that russia has invaded georgia's sovereignty has
encroached on georgia's sovereignty and it is very important for us to
resolve this issue as quickly as possible
what is clear is that russia has invaded georgia's sovereign has
encroached on georgia's sovereignty
----------------------------------------------------------------------- 21 / 100
Kept: no
Tokens: are -> have
Lemmas: be -> have
cleary there's been a number of incidents recently and it's important
that we go in and double check and make sure that all the standards
are being maintained
there have been a number of incidents recently and it's important that
we go in and double check and make sure that all the standards have
been maintained
----------------------------------------------------------------------- 22 / 100
Kept: no
Tokens: nato -> if
Lemmas: nato -> if
if nato suddenly takes military actions against abkhazia and south
ossetia acting solely in support of tbilisi this will mean a
declaration of war on russia
if nato takes military actions against abkhazia and south ossetia
acting solely in support of tbilisi this will mean a declaration of
war on russia
----------------------------------------------------------------------- 23 / 100
Kept: no
Tokens: suddenly -> nato
Lemmas: suddenly -> nato
if nato suddenly takes military actions against abkhazia and south
ossetia acting solely in support of tbilisi this will mean a
declaration of war on russia
if nato takes military actions against abkhazia and south ossetia
acting solely in support of tbilisi this will mean a declaration of
war on russia
----------------------------------------------------------------------- 24 / 100
Kept: no
Tokens: the -> american
Lemmas: the -> american
written in the dark of night that won't do a damn thing about the
supply of energy it's a hoax on the american people
won't do a damn thing about american energy
----------------------------------------------------------------------- 25 / 100
Kept: no
Tokens: supply -> energy
Lemmas: supply -> energy
written in the dark of night that won't do a damn thing about the
supply of energy it's a hoax on the american people
won't do a damn thing about american energy
----------------------------------------------------------------------- 26 / 100
Kept: no
Tokens: angelina -> she
Lemmas: angelina -> she
what angelina did was very uncool
what she did was very uncool
----------------------------------------------------------------------- 27 / 100
Kept: no
Tokens: very -> uncool
Lemmas: very -> uncool
what angelina did was very uncool
what angelina did was uncool
----------------------------------------------------------------------- 28 / 100
Kept: no
Tokens: what -> inexplicable
Lemmas: what -> inexplicable
your relationship with george bush seems what's the word i'm looking
for inexplicable
your relationship with george bush seems inexplicable
----------------------------------------------------------------------- 29 / 100
Kept: yes
Tokens: concerned -> closely
Lemmas: concern -> closely
the american people are concerned about the situation in our financial
markets and our economy and i share their concerns
closely monitor the situation in our financial markets
----------------------------------------------------------------------- 30 / 100
Kept: yes
Tokens: about -> monitor
Lemmas: about -> monitor
the american people are concerned about the situation in our financial
markets and our economy and i share their concerns
closely monitor the situation in our financial markets
----------------------------------------------------------------------- 31 / 100
Kept: no
Tokens: phosphorus -> phosphorous
Lemmas: phosphorus -> phosphorous
white phosphorus can burn down houses and cause horrific burns when it
touches the skin
white phosphorous can burn down houses and cause horrific burns when
it touches the skin
----------------------------------------------------------------------- 32 / 100
Kept: no
Tokens: phosphorous -> phosphorus
Lemmas: phosphorous -> phosphorus
white phosphorous can burn down houses and cause horrific burns when
it touches the skin
white phosphorus can burn down houses and cause horrific burns when it
touches the skin
----------------------------------------------------------------------- 33 / 100
Kept: no
Tokens: it -> that
Lemmas: it -> that
growing up i always wanted to be an olympian now to be the most
decorated olympian of all time it just sounds weird saying it started
setting in a little bit after the butterfly i was just trying to focus
on my next race but i just kept thinking wow greatest olympian of all
time it's a pretty cool title i'm definitely honored
that's a pretty cool title
----------------------------------------------------------------------- 34 / 100
Kept: no
Tokens: what -> do
Lemmas: what -> do
i think the convention's gone pretty well so far what do you think
i think the convention's gone pretty well so far don't you think
----------------------------------------------------------------------- 35 / 100
Kept: no
Tokens: do -> n't
Lemmas: do -> n't
i think the convention's gone pretty well so far what do you think
i think the convention's gone pretty well so far don't you think
----------------------------------------------------------------------- 36 / 100
Kept: no
Tokens: know -> they
Lemmas: know -> they
we know that they can't take 20 wickets and they are on the back foot
they couldn't get me and bhaji out we are in with a big chance they
are under pressure we know that
they know they can't take 20 wickets and they are on the back foot
----------------------------------------------------------------------- 37 / 100
Kept: no
Tokens: that -> know
Lemmas: that -> know
we know that they can't take 20 wickets and they are on the back foot
they couldn't get me and bhaji out we are in with a big chance they
are under pressure we know that
they know they can't take 20 wickets and they are on the back foot
----------------------------------------------------------------------- 38 / 100
Kept: yes
Tokens: responsibility -> response
Lemmas: responsibility -> response
the responsibility of the pakistani government should be one of
cooperation and of action
the response of the pakistani government should be one of cooperation
and of action
----------------------------------------------------------------------- 39 / 100
Kept: no
Tokens: path-breaking -> pathbreaking
Lemmas: path-breaking -> pathbreaking
how is this happening with a path-breaking public-private coalition
solid science better statistics and precise financing with the
coordination of the right countries and partners and above all with
the leadership
with a pathbreaking public-private coalition solid science better
statistics and precise financing with the coordination of the right
countries and partners and above all with the leadership
----------------------------------------------------------------------- 40 / 100
Kept: no
Tokens: sen -> senator
Lemmas: sen -> senator
was the last best chance to bring greater oversight and tighter
regulation to freddie and fannie and they used every means they could
to defeat sen hagel's legislation every step of the way
was the last best chance to bring greater oversight and tighter
regulation to freddie and fannie and they used every means they could
to defeat senator hagel's legislation every step of the way
----------------------------------------------------------------------- 41 / 100
Kept: no
Tokens: re-looking -> relooking
Lemmas: re-looking -> relooking
we are not interested in going back and re-looking at an acquisition
we are not interested in going back and relooking at an acquisition
----------------------------------------------------------------------- 42 / 100
Kept: no
Tokens: you -> do
Lemmas: you -> do
miss jolie says she likes john mccain too and hasn't decided who to
endorse huh aren't you supposed to be somewhat enlightened or do you
not know that the african daughter you hold in every picture had
parents who suffered and died because of the republican party's
worldwide economic assault on africa over the last few decades since
reagan
do you know that the african daughter you hold in every picture had
parents who suffered and died because of the republican party's
worldwide economic assault on africa over the last few decades since
reagan
----------------------------------------------------------------------- 43 / 100
Kept: no
Tokens: not -> you
Lemmas: not -> you
miss jolie says she likes john mccain too and hasn't decided who to
endorse huh aren't you supposed to be somewhat enlightened or do you
not know that the african daughter you hold in every picture had
parents who suffered and died because of the republican party's
worldwide economic assault on africa over the last few decades since
reagan
do you know that the african daughter you hold in every picture had
parents who suffered and died because of the republican party's
worldwide economic assault on africa over the last few decades since
reagan
----------------------------------------------------------------------- 44 / 100
Kept: no
Tokens: he -> jett
Lemmas: he -> jett
jett may have still been alive when john administered cpr and then the
emt took over jett was pronounced dead at the hospital i like to
believe john had a chance to say goodbye he may have died in his dad's
arms i am not certain
i like to believe john had a chance to say goodbye jett may have died
in his dad's arms
----------------------------------------------------------------------- 45 / 100
Kept: no
Tokens: cease-fire -> ceasefire
Lemmas: cease-fire -> ceasefire
georgia expresses its readiness to immediately start negotiations with
the russian federation on cease-fire and termination of hostilities
georgia expresses its readiness to immediately start negotiations with
the russian federation on ceasefire and termination of hostilities
----------------------------------------------------------------------- 46 / 100
Kept: no
Tokens: democrat -> democratic
Lemmas: democrat -> democratic
in that era almost all black americans were republicans why from its
founding in 1854 as the anti-slavery party until today the republican
party has championed freedom and civil rights for blacks and as one
pundit so succinctly stated the democrat party is as it always has
been the party of the four s's slavery secession segregation and now
socialism
and as one pundit so succinctly stated the democratic party is as it
always has been the party of the four s's slavery secession
segregation and now socialism
----------------------------------------------------------------------- 47 / 100
Kept: no
Tokens: ministerial -> that
Lemmas: ministerial -> that
that ministerial list is what they wish to happen it was not agreed on
as far as we know there was no agreement on the allocation of cabinet
posts and we are waiting for the mediator mr mbeki to come and resolve
the impasse
that list is what they wish to happen it was not agreed on
----------------------------------------------------------------------- 48 / 100
Kept: no
Tokens: out-scaled -> outscaled
Lemmas: out-scaled -> outscaled
michael's talent out-scaled even his own dinosaurs of jurassic park
michael's talent outscaled even his own dinosaurs of jurassic park
----------------------------------------------------------------------- 49 / 100
Kept: no
Tokens: violations -> this
Lemmas: violation -> this
we therefore feel that these repeated defaults and violations have
forced us to withdraw our support from the ruling coalition and sit on
the opposition benches
this situation forced us to withdraw our support
----------------------------------------------------------------------- 50 / 100
Kept: no
Tokens: have -> situation
Lemmas: have -> situation
we therefore feel that these repeated defaults and violations have
forced us to withdraw our support from the ruling coalition and sit on
the opposition benches
this situation forced us to withdraw our support
----------------------------------------------------------------------- 51 / 100
Kept: no
Tokens: could -> can
Lemmas: could -> can
and so our troops are a front line against the taliban there is a
chain of terror that comes from the pakistani and afghan mountains
right across europe and could end up very easily on the streets of
britain
there is a chain of terror that comes from the pakistani and afghan
mountains right across europe and can end up very easily on the
streets of britain
----------------------------------------------------------------------- 52 / 100
Kept: no
Tokens: recognizing -> recognising
Lemmas: recognize -> recognise
i am utterly deeply humbled and chuffed by the fact that i'm a stamp
i'm going to be licked by millions of australians and i can't wait it
is also recognizing the value of the arts in australia it brings it
into general consumption and if i'm part of that i feel intensely
proud
it is also recognising the value of the arts in australia it brings it
into general consumption and if i'm part of that i feel intensely
proud
----------------------------------------------------------------------- 53 / 100
Kept: no
Tokens: over -> here
Lemmas: over -> here
whether it's joe the plumber in ohio or joe over here
whether it's joe the plumber in ohio or joe here
----------------------------------------------------------------------- 54 / 100
Kept: no
Tokens: 's -> theres
Lemmas: be -> theres
o'leary o'reilly o'hare and o'hara there's no one as irish as barack
o'bama
theres no one as irish as barack obama
----------------------------------------------------------------------- 55 / 100
Kept: no
Tokens: o'bama -> obama
Lemmas: o'bama -> obama
o'leary o'reilly o'hare and o'hara there's no one as irish as barack
o'bama
theres no one as irish as barack obama
----------------------------------------------------------------------- 56 / 100
Kept: no
Tokens: this -> the
Lemmas: this -> the
this law needs to be changed to focus on infants
the law needs to be changed
----------------------------------------------------------------------- 57 / 100
Kept: no
Tokens: the -> consequences
Lemmas: the -> consequence
i am now prepared to subject myself to the laws of the land and face
the consequences thereof
i am now prepared to subject myself to the laws of the land and face
consequences thereof
----------------------------------------------------------------------- 58 / 100
Kept: no
Tokens: consequences -> thereof
Lemmas: consequence -> thereof
i am now prepared to subject myself to the laws of the land and face
the consequences thereof
i am now prepared to subject myself to the laws of the land and face
consequences thereof
----------------------------------------------------------------------- 59 / 100
Kept: no
Tokens: it -> drugs
Lemmas: it -> drug
i dragged amy into it and without me there is no doubt that she would
never have gone down that road i ruined something beautiful now i have
to let her go to save her life i am not abandoning her i am doing this
out of love
i dragged amy into drugs and without me there is no doubt that she
would never have gone down that road
----------------------------------------------------------------------- 60 / 100
Kept: no
Tokens: and -> on
Lemmas: and -> on
we are aware of a global virus for which there are some public alerts
and we've seen some of these on our networks and we are taking steps
to identify and mitigate the virus
we are aware of a global virus for which there are some public alerts
on
----------------------------------------------------------------------- 61 / 100
Kept: yes
Tokens: material -> physical
Lemmas: material -> physical
he hates the american material occupation as much as he hates the
iranian moral occupation
he hates the american physical occupation as much as he hates the
iranian moral occupation
----------------------------------------------------------------------- 62 / 100
Kept: no
Tokens: against -> aginst
Lemmas: against -> aginst
sharon is delighted to have won her case she would prefer not to have
to take legal action against the media but had no alternative in this
case
sharon is delighted to have won her case she would prefer not to have
to take legal action aginst the media but had no alternative in this
case
----------------------------------------------------------------------- 63 / 100
Kept: no
Tokens: organised -> organized
Lemmas: organise -> organize
legally relations between two countries cannot be organised by a law
they should be arranged according to international law through
treaties or agreements
legally relations between two countries cannot be organized by a law
they should be arranged according to international law through
treaties or agreements
----------------------------------------------------------------------- 64 / 100
Kept: no
Tokens: does -> he
Lemmas: do -> he
he does say he wishes the outfits were for his eyes only
he says he wishes the outfits were for his eyes only
----------------------------------------------------------------------- 65 / 100
Kept: no
Tokens: raiding -> raiding
Lemmas: raiding -> raid
for apple mac and unix lovers these major spam attacks just mean a
clogged-up inbox not an infected operating system but organised
criminals are causing havoc for windows users in the hunt for cold
hard cash too many people are clicking without thinking exposing
themselves to hackers who are hell-bent on gaining access to
confidential information and raiding bank accounts the advice is
simple you should never open unsolicited attachments however tempting
they may appear
too many people are clicking without thinking exposing themselves to
hackers who are hell-bent on gaining access to confidential
information and raiding bank accounts
----------------------------------------------------------------------- 66 / 100
Kept: yes
Tokens: police -> car
Lemmas: police -> car
david and victoria beckham were driving a car that had been scratched
the day before when it had been left parked at the airport they were
not involved in any accident at all there was no police no crash no
paparazzi chase etc just very incorrect reports via internet gossip
sites
there was no car crash
----------------------------------------------------------------------- 67 / 100
Kept: no
Tokens: no -> crash
Lemmas: no -> crash
david and victoria beckham were driving a car that had been scratched
the day before when it had been left parked at the airport they were
not involved in any accident at all there was no police no crash no
paparazzi chase etc just very incorrect reports via internet gossip
sites
there was no car crash
----------------------------------------------------------------------- 68 / 100
Kept: no
Tokens: you -> we
Lemmas: you -> we
we're making common mistakes things you're stressing in practice and
we need to clean it up our house is messy right now and nobody is
happy about it
we're making common mistakes things we're stressing in practice and we
need to clean them up
----------------------------------------------------------------------- 69 / 100
Kept: no
Tokens: it -> them
Lemmas: it -> them
we're making common mistakes things you're stressing in practice and
we need to clean it up our house is messy right now and nobody is
happy about it
we're making common mistakes things we're stressing in practice and we
need to clean them up
----------------------------------------------------------------------- 70 / 100
Kept: no
Tokens: 'm -> i
Lemmas: be -> i
i'm more upset at myself with what i said i'm really bad at interviews
and anyone who knows me knew i meant no harm i'm a goofball mostly i'm
very sorry to have hurt hef playboy made me who i am and hef has done
more for me than anyone else in my heart i will always love hef and
playboy
i'm sorry to have hurt hef
----------------------------------------------------------------------- 71 / 100
Kept: no
Tokens: very -> 'm
Lemmas: very -> be
i'm more upset at myself with what i said i'm really bad at interviews
and anyone who knows me knew i meant no harm i'm a goofball mostly i'm
very sorry to have hurt hef playboy made me who i am and hef has done
more for me than anyone else in my heart i will always love hef and
playboy
i'm sorry to have hurt hef
----------------------------------------------------------------------- 72 / 100
Kept: no
Tokens: one -> 1
Lemmas: one -> 1
assuming the economy bottoms out in the second half of 2009 job losses
are likely to continue into 2010 in all probability taking the final
toll of lost jobs to around one million
assuming the economy bottoms out in the second half of 2009 job losses
are likely to continue into 2010 in all probability taking the final
toll of lost jobs to around 1 million
----------------------------------------------------------------------- 73 / 100
Kept: no
Tokens: 1 -> one
Lemmas: 1 -> one
assuming the economy bottoms out in the second half of 2009 job losses
are likely to continue into 2010 in all probability taking the final
toll of lost jobs to around 1 million
assuming the economy bottoms out in the second half of 2009 job losses
are likely to continue into 2010 in all probability taking the final
toll of lost jobs to around one million
----------------------------------------------------------------------- 74 / 100
Kept: no
Tokens: if -> of
Lemmas: if -> of
if it did then i could walk away it does seem to fly in the face if
circumstantial evidence
if it did then i could walk away it does seem to fly in the face of
circumstantial evidence
----------------------------------------------------------------------- 75 / 100
Kept: no
Tokens: 18 -> eighteen
Lemmas: @card@ -> eighteen
we are convinced that we can overcome this crisis in a period of 18
months
we are convinced that we can overcome this crisis in a period of
eighteen months
----------------------------------------------------------------------- 76 / 100
Kept: no
Tokens: dni -> continuity
Lemmas: dni -> continuity
there's got to be some continuity and the leadership of both the cia
and the dni is going to be pivotal to keeping us safe and secure
continuity is going to be pivotal in keeping us safe and secure
----------------------------------------------------------------------- 77 / 100
Kept: no
Tokens: to -> in
Lemmas: to -> in
there's got to be some continuity and the leadership of both the cia
and the dni is going to be pivotal to keeping us safe and secure
continuity is going to be pivotal in keeping us safe and secure
----------------------------------------------------------------------- 78 / 100
Kept: no
Tokens: axe -> ax
Lemmas: axe -> ax
make sure you have an axe because you will be busting your way out to
get on your roof with waters surrounding you
make sure you have an ax
----------------------------------------------------------------------- 79 / 100
Kept: no
Tokens: on-line -> online
Lemmas: on-line -> online
it may have started on-line but it existed entirely in the real world
and it hurts just as much
it may have started online but it existed entirely in the real world
and it hurts just as much
----------------------------------------------------------------------- 80 / 100
Kept: no
Tokens: scrutinize -> scrutinise
Lemmas: scrutinize -> scrutinise
we will wait patiently listen to their words carefully scrutinize
their actions under a magnifier and if change happens truly and
fundamentally we will welcome that
we will wait patiently listen to their words carefully scrutinise
their actions under a magnifier and if change happens truly and
fundamentally we will welcome that
----------------------------------------------------------------------- 81 / 100
Kept: no
Tokens: 15 -> years
Lemmas: @card@ -> year
in the bible it happened but girls today i don't marry em if they want
to at 14 15 years old because we won't do it even though i believe
it's ok
in the bible it happened but girls today i don't marry em if they want
to at 14-15 years old
----------------------------------------------------------------------- 82 / 100
Kept: no
Tokens: years -> old
Lemmas: year -> old
in the bible it happened but girls today i don't marry em if they want
to at 14 15 years old because we won't do it even though i believe
it's ok
in the bible it happened but girls today i don't marry em if they want
to at 14-15 years old
----------------------------------------------------------------------- 83 / 100
Kept: no
Tokens: as -> condemns
Lemmas: as -> condemn
strongly condemns the irresponsible provocations which led to this
situation as well as the disproportionate use of force
condemns the disproportionate use of force
----------------------------------------------------------------------- 84 / 100
Kept: no
Tokens: a -> the
Lemmas: a -> the
what i essentially said to him was five years ago when you were in a
position to buy your business you needed a tax cut then and what i
want to do is to make sure that the plumber the nurse the firefighter
the teacher the young entrepreneur who doesn't yet have money i want
to give them a tax break now
what i essentially said to him was five years ago when you were in the
position to buy your business you needed a tax cut then
----------------------------------------------------------------------- 85 / 100
Kept: no
Tokens: an -> steadiness
Lemmas: an -> steadiness
displayed a steadiness an intellectual curiosity a depth of knowledge
and an approach to looking at problems like this
steadiness intellectual curiosity and depth of knowledge
----------------------------------------------------------------------- 86 / 100
Kept: no
Tokens: a -> and
Lemmas: a -> and
displayed a steadiness an intellectual curiosity a depth of knowledge
and an approach to looking at problems like this
steadiness intellectual curiosity and depth of knowledge
----------------------------------------------------------------------- 87 / 100
Kept: no
Tokens: recapitalise -> recapitalize
Lemmas: recapitalise -> recapitalize
we are far from the end of the road back to stability but the plan to
recapitalise our banking system both here and abroad will i believe
come to be seen as the moment in the banking crisis of the past year
when we turned the corner
we are far from the end of the road back to stability but the plan to
recapitalize our banking system both here and abroad will i believe
come to be seen as the moment in the banking crisis of the past year
when we turned the corner
----------------------------------------------------------------------- 88 / 100
Kept: no
Tokens: steadiness -> he
Lemmas: steadiness -> he
obama displayed a steadiness showed intellectual vigor he has a
definitive way of doing business that will do us well
he showed intellectual vigor he has a definitive way of doing business
that will do us well
----------------------------------------------------------------------- 89 / 100
Kept: no
Tokens: table -> we
Lemmas: table -> we
all of us around the table know we've got to get something done as
quickly as possible
we know we've got to get something done as quickly as possible
----------------------------------------------------------------------- 90 / 100
Kept: no
Tokens: eight -> 8
Lemmas: eight -> 8
if we are going to go down this road you know barack obama was eight
years old somehow responsible for bill ayers
if we are going to go down this road you know barack obama was 8 years
old somehow responsible for bill ayers
----------------------------------------------------------------------- 91 / 100
Kept: no
Tokens: that -> the
Lemmas: that -> the
come to ces to see all that palm new-ness you've been waiting for
come to ces to see all the palm new-ness you've been waiting for
----------------------------------------------------------------------- 92 / 100
Kept: no
Tokens: the -> that
Lemmas: the -> that
come to ces to see all the palm new-ness you've been waiting for
all that palm new-ness you've been waiting for
----------------------------------------------------------------------- 93 / 100
Kept: no
Tokens: general -> gen
Lemmas: general -> gen
george bush has given a mission to general petraeus and he has done
his best to try to figure out how to put lipstick on a pig
george bush has given a mission to gen petraeus and he has done his
best to try to figure out how to put lipstick on a pig
----------------------------------------------------------------------- 94 / 100
Kept: no
Tokens: trahearne -> traherne
Lemmas: trahearne -> traherne
when i finally caught up with abraham trahearne he was drinking beer
with an alcoholic bulldog named fireball roberts in a ramshackle joint
just outside of sonora california drinking the heart right out of a
fine spring afternoon
when i finally caught up with abraham traherne he was drinking beer
with an alcoholic bulldog named fireball roberts in a ramshackle joint
just outside of sonoma california drinking the heart right out of a
fine spring afternoon
----------------------------------------------------------------------- 95 / 100
Kept: no
Tokens: sonora -> sonoma
Lemmas: sonora -> sonoma
when i finally caught up with abraham trahearne he was drinking beer
with an alcoholic bulldog named fireball roberts in a ramshackle joint
just outside of sonora california drinking the heart right out of a
fine spring afternoon
when i finally caught up with abraham traherne he was drinking beer
with an alcoholic bulldog named fireball roberts in a ramshackle joint
just outside of sonoma california drinking the heart right out of a
fine spring afternoon
----------------------------------------------------------------------- 96 / 100
Kept: no
Tokens: a -> trillion
Lemmas: a -> trillion
peter orszag now forecasts that at the current course and speed a
trillion-dollar deficit will be here before we even start the next
budget
trillion dollar deficit will be here before we even start the next
budget
----------------------------------------------------------------------- 97 / 100
Kept: no
Tokens: trillion-dollar -> dollar
Lemmas: trillion-dollar -> dollar
peter orszag now forecasts that at the current course and speed a
trillion-dollar deficit will be here before we even start the next
budget
trillion dollar deficit will be here before we even start the next
budget
----------------------------------------------------------------------- 98 / 100
Kept: no
Tokens: hundred -> 200
Lemmas: hundred -> @card@
john mccain voted to cut education funding against accountability
standards he even proposed abolishing the department of education and
john mccain's economic plan gives two hundred billion more to special
interests while taking money away from public schools
200 billion more to special interests while taking money away from
public schools
----------------------------------------------------------------------- 99 / 100
Kept: no
Tokens: recognized -> recognised
Lemmas: recognize -> recognise
in 2006 i made a serious error in judgment and conducted myself in a
way that was disloyal to my family and to my core beliefs i recognized
my mistake and i told my wife that i had a liaison with another woman
and i asked for her forgiveness although i was honest in every painful
detail with my family i did not tell the public
in 2006 i made a serious error in judgment and conducted myself in a
way that was disloyal to my family and to my core beliefs i recognised
my mistake and i told my wife that i had a liaison with another woman
and i asked for her forgiveness
---------------------------------------------------------------------- 100 / 100
Kept: no
Tokens: to -> and
Lemmas: to -> and
their leader is a danger to his own people to the region to the world
he's a disgrace to this house
a danger to his own people and the world
Now with only valid substitutions, to make the precision analysis more accurate.
In [21]:
db_print_substitutions(Model(Time.discrete, Source.majority, Past.last_bin, Durl.all, 2))
------------------------------------------------------------------------ 1 / 100
Kept: yes
Tokens: witnessed -> seen
Lemmas: witness -> see
palestine has never witnessed an uglier massacre
palestine has never seen an uglier massacre
------------------------------------------------------------------------ 2 / 100
Kept: yes
Tokens: world -> global
Lemmas: world -> global
the united states will lose its superpower status in the world
financial system the world financial system will become more multi-
polar
the united states will lose its superpower status in the global
financial system
------------------------------------------------------------------------ 3 / 100
Kept: yes
Tokens: borders -> limit
Lemmas: border -> limit
from tomorrow representatives of the european union will begin
conducting monitoring up to the southern borders of the security zone
up to the southern limit of the security zone
------------------------------------------------------------------------ 4 / 100
Kept: yes
Tokens: believe -> feel
Lemmas: believe -> feel
we believe it is inappropriate to use young private citizens for
marketing purposes
we feel it is inappropriate to use young private citizens for
marketing purposes
------------------------------------------------------------------------ 5 / 100
Kept: yes
Tokens: spend -> talk
Lemmas: spend -> talk
see it would be funny but the news media decided that would be the
lead story yesterday this happens every election cycle every four
years this is what we do this is what they want to spend two of the
last 55 days talking about enough
this is what they want to talk about
------------------------------------------------------------------------ 6 / 100
Kept: yes
Tokens: time -> assessment
Lemmas: time -> assessment
we have continued to examine the relative benefits of purchasing
illiquid mortgage-related assets our assessment at this time is that
this is not the most effective way to use bailout funds
our assessment is that this is not the most effective way
------------------------------------------------------------------------ 7 / 100
Kept: yes
Tokens: think -> believe
Lemmas: think -> believe
we think a decision could be taken but i think it will happen in
algeria
we believe a decision could be taken but i think it will happen in
algeria
------------------------------------------------------------------------ 8 / 100
Kept: yes
Tokens: frail -> fragile
Lemmas: frail -> fragile
it is clear that the financial system although improved from where it
was in september is still frail
it is clear that the financial system although improved from where it
was in september is still fragile
------------------------------------------------------------------------ 9 / 100
Kept: yes
Tokens: ill -> know
Lemmas: ill -> know
without going through the details too much she's gravely ill we
weren't sure and i'm still not sure whether she makes it to election
day
you know we weren't sure and i'm still not sure whether she makes it
to election day
----------------------------------------------------------------------- 10 / 100
Kept: yes
Tokens: prepared -> ready
Lemmas: prepare -> ready
are you prepared to take the oath senator
are you ready to take the oath
----------------------------------------------------------------------- 11 / 100
Kept: yes
Tokens: cause -> create
Lemmas: cause -> create
would cause great turmoil in our financial markets here at home and
around the globe
create great turmoil in our financial markets here at home and around
the globe
----------------------------------------------------------------------- 12 / 100
Kept: yes
Tokens: yesterday -> friday
Lemmas: yesterday -> friday
when they were saying they had to leave that an employee got killed
people were yelling i've been on line since yesterday morning
when they were saying they had to leave that an employee got killed
people were yelling i've been on line since friday morning
----------------------------------------------------------------------- 13 / 100
Kept: yes
Tokens: absence -> lack
Lemmas: absence -> lack
i think many of us have been disappointed with the lack of clarity the
absence of transparency
i think many of us have been disappointed with the absence of clarity
the lack of transparency
----------------------------------------------------------------------- 14 / 100
Kept: yes
Tokens: appointed -> junior
Lemmas: appoint -> junior
i'm presenting myself as the legally appointed senator from the state
of illinois it is my hope and prayer that they recognize that the
appointment is legal
the junior senator from the state of illinois
----------------------------------------------------------------------- 15 / 100
Kept: yes
Tokens: play -> separate
Lemmas: play -> separate
i don't really mix them up i play my music and i go with my husband
when he needs me
i don't really mix them up i separate them
----------------------------------------------------------------------- 16 / 100
Kept: yes
Tokens: movie -> video
Lemmas: movie -> video
you look just awesome in this new movie
you look just awesome in this new video
----------------------------------------------------------------------- 17 / 100
Kept: yes
Tokens: punish -> hurt
Lemmas: punish -> hurt
because somebody hurt their feelings they decide to punish the country
because somebody hurt their feelings they decided to hurt the country
----------------------------------------------------------------------- 18 / 100
Kept: yes
Tokens: going -> willing
Lemmas: go -> willing
not going to give me anything except appreciation
not willing to give blagojevich anything except appreciation
----------------------------------------------------------------------- 19 / 100
Kept: yes
Tokens: socialism -> welfare
Lemmas: socialism -> welfare
this is socialism for the rich
this is welfare for the rich
----------------------------------------------------------------------- 20 / 100
Kept: yes
Tokens: expletive -> fuck
Lemmas: expletive -> fuck
our recommendation is fire all those expletive people get em the
expletive out of there and get us some editorial support
our recommendation is fire all those fucking people get em the fuck
out of there
----------------------------------------------------------------------- 21 / 100
Kept: yes
Tokens: fuck -> f
Lemmas: fuck -> f
ya fuck with me i'll kick your ass
ya f with me i'll kick ass
----------------------------------------------------------------------- 22 / 100
Kept: yes
Tokens: solution -> body
Lemmas: solution -> body
blaming cities for greenhouse gas emissions misses the point that
cities are a large part of the solution
a large part of the body
----------------------------------------------------------------------- 23 / 100
Kept: yes
Tokens: branches -> proxies
Lemmas: branch -> proxy
but this is an all-out war against hamas and its branches the
restraint that we have demonstrated is the source of our strength when
it is time to fight
an all-out war with hamas and its proxies
----------------------------------------------------------------------- 24 / 100
Kept: yes
Tokens: decline -> emergency
Lemmas: decline -> emergency
if we are not able to do that then social unrest may happen in many
countries including advanced economies we are facing an unprecedented
decline in output all around the planet the people have reacted with
feelings going from surprise to anger and from anger to fear
we are facing an unprecedented emergency
----------------------------------------------------------------------- 25 / 100
Kept: yes
Tokens: responsibly -> possibly
Lemmas: responsibly -> possibly
management continues to insist on terms we cannot responsibly accept
on behalf of our members
insist on terms we cannot possibly accept
----------------------------------------------------------------------- 26 / 100
Kept: yes
Tokens: f -> fuck
Lemmas: f -> fuck
ya f with me i'll kick your ass
ya fuck with me i'll kick your ass
----------------------------------------------------------------------- 27 / 100
Kept: yes
Tokens: situation -> crisis
Lemmas: situation -> crisis
the most serious financial situation in generations
the most serious financial crisis in generations
----------------------------------------------------------------------- 28 / 100
Kept: yes
Tokens: reserve -> fed
Lemmas: reserve -> fed
the federal reserve will employ all available tools to promote the
resumption of sustainable economic growth and to preserve price
stability
the fed will employ all available tools to promote the resumption of
sustainable economic growth
----------------------------------------------------------------------- 29 / 100
Kept: yes
Tokens: fuck -> f
Lemmas: fuck -> f
it's dan the man fuck me you look like a raghead
f me you look like a raghead
----------------------------------------------------------------------- 30 / 100
Kept: yes
Tokens: public -> people
Lemmas: public -> people
until we are positive the technology will protect the american public
until we are positive the technology will protect the american people
----------------------------------------------------------------------- 31 / 100
Kept: yes
Tokens: case -> need
Lemmas: case -> need
now that there is no cholera there is no case for war
now there is no cholera there is no need for war
----------------------------------------------------------------------- 32 / 100
Kept: yes
Tokens: holiday -> year
Lemmas: holiday -> year
what are you thankful for this holiday season
what are you thankful for this year
----------------------------------------------------------------------- 33 / 100
Kept: yes
Tokens: required -> necessary
Lemmas: require -> necessary
the auto companies must not squander this chance to reform bad
management practices and begin the long-term restructuring that is
absolutely required to save this critical industry and the millions of
american jobs that depend on it
the auto companies must not squander this chance to reform bad
management practices and begin the long-term restructuring that is
absolutely necessary to save this critical industry and the millions
of american jobs that depend on it
----------------------------------------------------------------------- 34 / 100
Kept: yes
Tokens: thing -> person
Lemmas: thing -> person
the most normal thing in the world
most important person in the world
----------------------------------------------------------------------- 35 / 100
Kept: yes
Tokens: think -> know
Lemmas: think -> know
i think that she will bring enormous energy and intellect and skill to
the position
i know that she will bring enormous energy and intellect and skill to
the position
----------------------------------------------------------------------- 36 / 100
Kept: yes
Tokens: bad -> crimes
Lemmas: bad -> crime
those who speak of change must apologise to the iranian people and try
to repair their past bad acts and the crimes they committed against
iran
those who speak of change must apologise to the iranian people and try
to repair their past crimes
----------------------------------------------------------------------- 37 / 100
Kept: yes
Tokens: scent -> spray
Lemmas: scent -> spray
the scent of seduction with a hint of flame-broiled meat
body spray of seduction with a hint of flame-broiled meat
----------------------------------------------------------------------- 38 / 100
Kept: yes
Tokens: love -> salute
Lemmas: love -> salute
i love each and every one of you
i salute each and every one of you
----------------------------------------------------------------------- 39 / 100
Kept: yes
Tokens: spouse -> husband
Lemmas: spouse -> husband
where did you meet your spouse
where did you meet your husband
----------------------------------------------------------------------- 40 / 100
Kept: yes
Tokens: admire -> respect
Lemmas: admire -> respect
but we will be respectful i admire sen obama and his accomplishments
i respect sen obama and his accomplishments
----------------------------------------------------------------------- 41 / 100
Kept: yes
Tokens: bureau -> crime
Lemmas: bureau -> crime
several miles away from the first bureau of investigative services
said the search for the boy had been centered farther east based on
information they had about where balfour's current girlfriend lives
several miles away from the first crime scene
----------------------------------------------------------------------- 42 / 100
Kept: yes
Tokens: story -> history
Lemmas: story -> history
standing on the steps of the capitol will be a man whose story
reflects the enduring promise of our land
standing on the steps of the capitol will be a man whose history
reflects the enduring promise of our land
----------------------------------------------------------------------- 43 / 100
Kept: yes
Tokens: small -> reported
Lemmas: small -> report
the jobs we create will be in businesses large and small across a wide
range of industries
declines reported across a wide range of industries
----------------------------------------------------------------------- 44 / 100
Kept: yes
Tokens: give -> get
Lemmas: give -> get
we ask you to help us work for that day when black will not be asked
to give back when brown can stick around when yellow will be mellow
when the red man can get ahead man and when white will embrace what is
right
when black will not be asked to get back when brown can stick around
when yellow will be mellow when the red man can get ahead man and when
white will embrace what is right
----------------------------------------------------------------------- 45 / 100
Kept: yes
Tokens: concept -> conception
Lemmas: concept -> conception
the invasion of iraq was a bandit act an act of blatant state
terrorism demonstrating absolute contempt for the concept of
international law the invasion was an act intended to consolidate
american military and economic control of the middle east masquerading
as a last resort all other justifications having failed to justify
themselves as liberation
a bandit act an act of blatant state terrorism demonstrating absolute
contempt for the conception of international law
----------------------------------------------------------------------- 46 / 100
Kept: yes
Tokens: crisis -> challenge
Lemmas: crisis -> challenge
immediately after i become president i will confront this economic
crisis head-on by taking all necessary steps to ease the credit crisis
help hardworking families and restore growth and prosperity
immediately after i become president i will confront this economic
challenge head-on by taking all necessary steps to ease the credit
crisis help hardworking families and restore growth and prosperity
----------------------------------------------------------------------- 47 / 100
Kept: yes
Tokens: mom -> mama
Lemmas: mom -> mama
i just want to say mom it's real hard to sing when you're in the
second row crying
mama it's real hard to sing when you're in the second row crying
----------------------------------------------------------------------- 48 / 100
Kept: yes
Tokens: expletive -> f
Lemmas: expletive -> f
if they don't perform expletive em
if they don't perform f em
----------------------------------------------------------------------- 49 / 100
Kept: yes
Tokens: demonstrates -> cent
Lemmas: demonstrate -> cent
today's results actually show a more complex story than a straight
pass or fail by one machine where the machines were identified
correctly by the human interrogators as machines the conversational
abilities of each machine was scored at 80 and 90 this demonstrates
how close machines are getting to reaching the milestone of
communicating with us in a way in which we are comfortable that
eventual day will herald a new phase in our relationship with machines
bringing closer the time in which robots start to play an active role
in our daily lives
today's results actually show a more complex story than a straight
pass or fail by one machine where the machines were identified
correctly by the human interrogators as machines the conversational
abilities of each machine was scored at 80 and 90 per cent
----------------------------------------------------------------------- 50 / 100
Kept: yes
Tokens: visiting -> foreign
Lemmas: visit -> foreign
insulting a visiting head of state
insulting a foreign head of state
----------------------------------------------------------------------- 51 / 100
Kept: yes
Tokens: fundamentals -> foundations
Lemmas: fundamental -> foundation
i believe the fundamentals of this economy are strong and i believe
they will remain strong
i believe the foundations of this economy are strong
----------------------------------------------------------------------- 52 / 100
Kept: yes
Tokens: violence -> hostilities
Lemmas: violence -> hostility
condemns all acts of violence and terror directed against civilians
condemns all acts of hostilities and terror directed against civilians
----------------------------------------------------------------------- 53 / 100
Kept: yes
Tokens: scared -> afraid
Lemmas: scare -> afraid
this is a menace a cancer in pakistan which we will eliminate we will
not be scared of these cowards
we will not be afraid of these cowards
----------------------------------------------------------------------- 54 / 100
Kept: yes
Tokens: rightfully -> rightly
Lemmas: rightfully -> rightly
the american people are angry about executive compensation and
rightfully so
the american people are angry about executive compensation and rightly
so
----------------------------------------------------------------------- 55 / 100
Kept: yes
Tokens: expletive -> f
Lemmas: expletive -> f
i've got this thing and it's expletive golden and uh uh i'm just not
giving it up for expletive nothing i'm not gonna do it and and i can
always use it i can parachute me there
i've got this thing and it's f golden and uh uh i'm just not giving it
up for f nothing
----------------------------------------------------------------------- 56 / 100
Kept: yes
Tokens: half -> part
Lemmas: half -> part
we built significant momentum following the completion of our
corporate transformation and our business results were on track
through most of 2008 with digital revenue up 10 percent in the first
half of the year however during the last three months of the year we
experienced dramatic declines in several of our key businesses due to
the slowdown in consumer spending and significantly reduced demand for
capital equipment
in the first part of the year
----------------------------------------------------------------------- 57 / 100
Kept: yes
Tokens: expletive -> f
Lemmas: expletive -> f
i've got this thing and it's expletive golden and uh uh i'm just not
giving it up for expletive nothing i'm not gonna do it and and i can
always use it i can parachute me there
i've got this thing and it's f golden and uh uh i'm just not giving it
up for f nothing i'm not gonna do it
----------------------------------------------------------------------- 58 / 100
Kept: yes
Tokens: view -> word
Lemmas: view -> word
i'm looking for a church that preaches my view of the word i think the
blank translation should be used i think blank should be talked about
a lot while blank should be ignored and if you ever stop preaching my
view of the word i will leave and tell others that you don't preach
the word
i'm looking for a church that preaches the word
----------------------------------------------------------------------- 59 / 100
Kept: yes
Tokens: patience -> equality
Lemmas: patience -> equality
what do we want patience when do we want it now
what do we want equality when do we want it now
----------------------------------------------------------------------- 60 / 100
Kept: yes
Tokens: gravest -> greatest
Lemmas: grave -> great
will you appoint a special prosecutor ideally patrick fitzgerald to
independently investigate the gravest crimes of the bush
administration including torture and warrantless wiretapping
will you appoint a special prosecutor ideally patrick fitzgerald to
independently investigate the greatest crimes of the bush
administration including torture and warrantless wiretapping
----------------------------------------------------------------------- 61 / 100
Kept: yes
Tokens: sec -> commission
Lemmas: sec -> commission
i am disappointed that the sec chose to bring this case based upon its
enforcement staff's win-at-any-cost ambitions the staff's process was
result-oriented facts be damned the government's claims are false and
they will be proven to be so
i am disappointed that the commission chose to bring this case based
upon its enforcement staff's win-at-any-cost ambitions
----------------------------------------------------------------------- 62 / 100
Kept: yes
Tokens: give -> get
Lemmas: give -> get
we ask you to help us work for that day when black will not be asked
to give back when brown can stick around when yellow will be mellow
when the red man can get ahead man and when white will embrace what is
right
help us work for that day when black will not be asked to get back
when brown can stick around when yellow will be mellow when the red
man can get ahead man and when white will embrace what is right
----------------------------------------------------------------------- 63 / 100
Kept: yes
Tokens: honorable -> graceful
Lemmas: honorable -> graceful
if people want to throw me under the bus my personal belief is that
the most honorable thing to do is to lie there
if someone wants to throw me under the bus my personal belief is that
the most graceful thing to do is to lie there
----------------------------------------------------------------------- 64 / 100
Kept: yes
Tokens: markets -> economy
Lemmas: market -> economy
our economy and our markets will not recover until the bulk of this
housing correction is behind us
our economy will not recover until the bulk of this housing correction
is behind us
----------------------------------------------------------------------- 65 / 100
Kept: yes
Tokens: machine -> computer
Lemmas: machine -> computer
know how to make a 500 machine that's not a piece of junk
know how to make a 500 computer that's not a piece of junk
----------------------------------------------------------------------- 66 / 100
Kept: yes
Tokens: al-qaida -> qaeda
Lemmas: al-qaida -> qaeda
this war is not quite over but it's being won and primarily by the
people of anbar al-qaida has not been entirely defeated in anbar but
their end is near and they know it
al qaeda has not been entirely defeated in anbar but their end is near
and they know it
----------------------------------------------------------------------- 67 / 100
Kept: yes
Tokens: appeal -> wait
Lemmas: appeal -> wait
we are not going to appeal
we are not going to wait
----------------------------------------------------------------------- 68 / 100
Kept: yes
Tokens: find -> say
Lemmas: find -> say
it would be hard to find anybody i think at the u n who would not
believe that obama would be a considerable improvement over any other
alternative
it would be hard to say
----------------------------------------------------------------------- 69 / 100
Kept: yes
Tokens: about -> monitor
Lemmas: about -> monitor
the american people are concerned about the situation in our financial
markets and our economy and i share their concerns
closely monitor the situation in our financial markets
----------------------------------------------------------------------- 70 / 100
Kept: yes
Tokens: grandpa -> bye
Lemmas: grandpa -> bye
granny i've got to go send my love to the corgis and grandpa god save
you yeah that's great see you bye
bye god save you yeah that's great
----------------------------------------------------------------------- 71 / 100
Kept: yes
Tokens: rights -> freedom
Lemmas: right -> freedom
that comment is part of a much larger nuanced and yes provocative
that's what i do piece from my show about racism freedom women's
rights and the extreme views of governor sarah palin
women racism freedom and the extreme views of governor sarah palin
----------------------------------------------------------------------- 72 / 100
Kept: yes
Tokens: achieve -> accomplish
Lemmas: achieve -> accomplish
liberals hate real americans that work and achieve and believe in god
liberals hate real americans that work and accomplish and achieve
----------------------------------------------------------------------- 73 / 100
Kept: yes
Tokens: means -> protests
Lemmas: mean -> protest
will be forced to react and not only through diplomatic means
react and not only through diplomatic protests
----------------------------------------------------------------------- 74 / 100
Kept: yes
Tokens: launch -> use
Lemmas: launch -> use
instead of caring about the people of gaza hamas decided to use gaza
to launch rockets to kill innocent israelis
instead of caring about the people of gaza hamas decided to use gaza
to use rockets to kill innocent israelis
----------------------------------------------------------------------- 75 / 100
Kept: yes
Tokens: surrogates -> irregular
Lemmas: surrogate -> irregular
uw must be conducted by with or through surrogates and such surrogates
must be irregular forces
by with or through irregular forces
----------------------------------------------------------------------- 76 / 100
Kept: yes
Tokens: everyday -> economic
Lemmas: everyday -> economic
i am convinced that this bold approach will cost american families far
less than the alternative a continuing series of financial institution
failures and frozen credit markets unable to fund everyday needs and
economic expansion
i am convinced that this bold approach will cost american families far
less than the alternative a continuing series of financial institution
failures and frozen credit markets unable to fund economic expansion
----------------------------------------------------------------------- 77 / 100
Kept: yes
Tokens: technology -> technological
Lemmas: technology -> technological
i believe windows will remain at the center of people's technology
solar system
i believe windows will remain at the center of people's technological
solar system
----------------------------------------------------------------------- 78 / 100
Kept: yes
Tokens: case -> need
Lemmas: case -> need
now that there is no cholera there is no case for war
now that there is no cholera there is no need for war
----------------------------------------------------------------------- 79 / 100
Kept: yes
Tokens: criteria -> considerations
Lemmas: criterion -> consideration
assessing those requests from a case-by-case basis they had not met
our stringent national security and immigration criteria and have been
rejected
stringent national security and immigration considerations
----------------------------------------------------------------------- 80 / 100
Kept: yes
Tokens: expletive -> f
Lemmas: expletive -> f
they're not willing to give me anything except appreciation expletive
them
not willing to give me anything except appreciation f them
----------------------------------------------------------------------- 81 / 100
Kept: yes
Tokens: friday -> monday
Lemmas: friday -> monday
america feels differently today than it did on friday about this bill
and he believes the house vote tomorrow will reflect that shift
america feels differently today than it did on monday about this bill
----------------------------------------------------------------------- 82 / 100
Kept: yes
Tokens: fuck -> expletive
Lemmas: fuck -> expletive
they're not willing to give me anything except appreciation fuck them
willing to give me anything except appreciation expletive them
----------------------------------------------------------------------- 83 / 100
Kept: yes
Tokens: restore -> establish
Lemmas: restore -> establish
we are forced to restore constitutional order in the whole region
establish constitutional order in the region
----------------------------------------------------------------------- 84 / 100
Kept: yes
Tokens: kind -> activity
Lemmas: kind -> activity
i'm very very pleased to be cleared of any legal wrongdoing any hint
of any kind of unethical activity there very pleased to be cleared of
any of that
cleared of any legal wrongdoing any hint of unethical activity
----------------------------------------------------------------------- 85 / 100
Kept: yes
Tokens: manufacturing -> producing
Lemmas: manufacturing -> produce
according to the criminal law there is no capital punishment for the
crime of manufacturing and selling fake or substandard products
therefore the most severe penalty for her would be life imprisonment
producing and selling fake or substandard products
----------------------------------------------------------------------- 86 / 100
Kept: yes
Tokens: ministers -> ministries
Lemmas: minister -> ministry
we are looking also at what we can do to be both supportive of the
ministers that president karzai has put up we are looking to see where
some of the strengths are and how we need to support those strengths
and also how we can help the afghans where there are weaknesses
we are looking also at what we can do to be more supportive of the
ministries that president karzai has put up
----------------------------------------------------------------------- 87 / 100
Kept: yes
Tokens: expletive -> f
Lemmas: expletive -> f
i've got this thing and it's expletive golden and uh uh i'm just not
giving it up for expletive nothing i'm not gonna do it
i've got this thing and it's f golden
----------------------------------------------------------------------- 88 / 100
Kept: yes
Tokens: dialogue -> possible
Lemmas: dialogue -> possible
an interreligious dialogue in the strict sense of the word is not
possible
not possible in the strict sense of the word
----------------------------------------------------------------------- 89 / 100
Kept: yes
Tokens: diligently -> vigilantly
Lemmas: diligently -> vigilantly
we must adhere to our values as diligently as we protect our safety
with no exceptions
adhere to our values as vigilantly as we protect our safety with no
exceptions
----------------------------------------------------------------------- 90 / 100
Kept: yes
Tokens: survival -> security
Lemmas: survival -> security
he is putting a lot of energy into his own survival a lot of energy
into his own security
putting a lot of energy into his own security
----------------------------------------------------------------------- 91 / 100
Kept: yes
Tokens: effect -> fact
Lemmas: effect -> fact
it is not just that the american side could not restrain the georgian
leadership from this criminal act the american side in effect armed
and trained the georgian army
the american side in fact armed and trained the georgian army
----------------------------------------------------------------------- 92 / 100
Kept: yes
Tokens: software -> strategy
Lemmas: software -> strategy
it's a transformation of our software and a transformation of our
strategy
it's a transformation of our strategy
----------------------------------------------------------------------- 93 / 100
Kept: yes
Tokens: shows -> crisis
Lemmas: show -> crisis
today of all days john mccain's stubborn insistence that the
fundamentals of the economy are strong shows that he is disturbingly
out of touch with what's going in the lives of ordinary americans
the economy is in crisis
----------------------------------------------------------------------- 94 / 100
Kept: yes
Tokens: team -> family
Lemmas: team -> family
i kept getting goose bumps after goose bumps out there i was really
into it i'm really proud of this team for trying so hard i could not
have asked for a better ending there is nothing like a team event and
when you pull it all together the atmosphere and camaraderie is one of
a kind
i'm really proud of this family
----------------------------------------------------------------------- 95 / 100
Kept: yes
Tokens: needs -> expansion
Lemmas: need -> expansion
i am convinced that this bold approach will cost american families far
less than the alternative a continuing series of financial institution
failures and frozen credit markets unable to fund everyday needs and
economic expansion
i am convinced that this bold approach will cost american families far
less than the alternative a continuing series of financial institution
failures and frozen credit markets unable to fund economic expansion
----------------------------------------------------------------------- 96 / 100
Kept: yes
Tokens: times -> lifetime
Lemmas: time -> lifetime
the greatest economic challenge of our times
greatest economic challenge of our lifetime
----------------------------------------------------------------------- 97 / 100
Kept: yes
Tokens: doomed -> destined
Lemmas: doom -> destine
in the short term we should dissuade the american presidential
candidates from getting more bogged down in afghanistan the american
strategy is doomed to fail
the american strategy is destined to fail
----------------------------------------------------------------------- 98 / 100
Kept: yes
Tokens: peaceful -> act
Lemmas: peaceful -> act
the government of india is determined to act decisively to protect its
territorial integrity and the right of our citizens to a peaceful life
with all the means at our disposal
act decisively with all the means at our disposal
----------------------------------------------------------------------- 99 / 100
Kept: yes
Tokens: careful -> danger
Lemmas: careful -> danger
sterling has devalued rapidly against the euro and the dollar we are
in danger if the government is not careful of having a proper sterling
collapse a run on the pound
danger of having a proper sterling collapse a run on the pound
---------------------------------------------------------------------- 100 / 100
Kept: yes
Tokens: graveyard -> cemetery
Lemmas: graveyard -> cemetery
gaza will not be a picnic gaza will be a graveyard for you
gaza will be your cemetery
Content source: wehlutyk/brainscopypaste
Similar notebooks: