Text Preprocessing

Author: Christin Seifert, licensed under the Creative Commons Attribution 3.0 Unported License https://creativecommons.org/licenses/by/3.0/

This is a tutorial for text preprocessing. In this tutorial you will see examples of

  • stemming
  • lemmatization and
  • stopword-removal

It is assumed that you have some general knowledge on

  • .. no particular knowledge required. You should be able to read texts, though ;-)

Let's work with texts from the web. Let's first have a look what nltk resources are readily available, so that we do not have to download them ourselves (Thanks, nltk folks!).


In [1]:
import nltk
from nltk.corpus import webtext
# this has to be run once to get the corpus
#nltk.download('webtext')

for fileid in webtext.fileids():
    print(fileid, webtext.raw(fileid)[:100], '...')


firefox.txt Cookie Manager: "Don't allow sites that set removed cookies to set future cookies" should stay check ...
grail.txt SCENE 1: [wind] [clop clop clop] 
KING ARTHUR: Whoa there!  [clop clop clop] 
SOLDIER #1: Halt!  Who ...
overheard.txt White guy: So, do you have any plans for this evening?
Asian girl: Yeah, being angry!
White guy: Oh, ...
pirates.txt PIRATES OF THE CARRIBEAN: DEAD MAN'S CHEST, by Ted Elliott & Terry Rossio
[view looking straight dow ...
singles.txt 25 SEXY MALE, seeks attrac older single lady, for discreet encounters.
35YO Security Guard, seeking  ...
wine.txt Lovely delicate, fragrant Rhone wine. Polished leather and strawberries. Perhaps a bit dilute, but g ...

The corpus about singles seems the most interesting. So let's load it. And since we want to learn text pre-processing we load the raw version (meaning no preprocessing is implictly done during loading). After loading let's inspect what is in there.


In [2]:
singles = nltk.corpus.webtext.raw('singles.txt')
singles


Out[2]:
'25 SEXY MALE, seeks attrac older single lady, for discreet encounters.\n35YO Security Guard, seeking lady in uniform for fun times.\n40 yo SINGLE DAD, sincere friendly DTE seeks r/ship with fem age open S/E\n44yo tall seeks working single mum or lady below 45 fship rship. Nat Open\n6.2 35 yr old OUTGOING M seeks fem 28-35 for o/door sports - w/e away\nA professional business male, late 40s, 6 feet tall, slim build, well groomed, great personality, home owner, interests include the arts travel and all things good, Ringwood area, is seeking a genuine female of similar age or older, in same area or surrounds, for a meaningful long term rship. Looking forward to hearing from you all.\nABLE young man seeks, sexy older women. Phone for fun ready to play\nAFFECTIONATE LADY Sought by generous guy, 40s, mutual fulfillment\nARE YOU ALONE or lost in a r/ship too, with no hope in sight? Maybe we could explore new beginnings together? Im 45 Slim/Med build, GSOH, high needs and looking for someone similar. You WONT be disappointed.\nAMIABLE 43 y.o. gentleman with European background, 170 cm, medium build, employed, never married, no children. Enjoys sports, music, cafes, beach &c. Seeks an honest, attractive lady with a European background, without children, who would like to get married and have chil dren in the future. 29-39 y.o. Prefer non-smoker and living in Adelaide.\nARE YOU A COPPER REDHEAD? I am 36 y.o. and looking for companionship/friendship. I enjoy the AFL, animals and dining out.\nATTRACTIVE 43 Y.O. 61, medium build, financially secure, no children, seeks attractive lady up to 40 y.o. with no children. I enjoy the beach, sports and music. View to permanent relationship.\nATTRACTIVE GUY Late 30s, 57, Taurus. Into fitness, cafes, movies, dinner parties. Seeks out there female for friendship and fun.\nAUSSIE 39 Solid build, likes music, movies, walks on the beach, single, would like to meet lady for friendship view relationship.\nAUSTRALIAN FATHER 58 With 11 y.o. son, living with me. Seeks nice, caring lady who likes children for permanent re lationship in coastal town on Eyre Peninsula. Age open to 55. \nAUSTRALIAN, SLIM, FUN 5 ft. 8. Enjoy BBQs, sport, fishing, going out for meal &c. Fun to be with. Looking for a com panion, aged between 35-50. Must be genuine and good fun. Possible relationship. \nBEAUTIFUL, INTELLIGENT CARING, MUSICAL Shy, late 20s, size 12, long blonde hair and likes to drink Guiness. If this is you then I want you.\nBLONDE HAIR, BLUE EYES Medium build, Im in my early 30s. Am honest, caring, likes fishing, animals, golf, bike riding, TV and DVDs, quiet nights at home, kissing and cuddling up to a special person in my life. Looking for a caring, honest lady for friendship to relationship.\nBUSINESSMAN 60\'\'ish Looking for lady, non-smoker, 56 or taller with business outlook, no ties, be prepared to travel and enjoy outdoor sports (cars, boats etc.), wining/ dining. Must have s.o.h. and have modern outlook. \nCUDDLY FULL FIGURED LADY 50 plus sought by Australian gent, early 60s, financially secure, non-drinker, non smoker, for permanent relationship.\nFIT ACTIVE EASYGOING 50 y.o. male, 6 ft 2, med. build, fit and healthy, fully employed no ties, non smoker, light drinker. Seeking female friend, 40s, attractive 5 ft 5+. Med. to slim build. View towards a permanent relationship. Give me a call lets have a chat.\nGENUINE AND HONEST Hi Im 44 with a good sense of humour, am romantic and love drives, fishing, camping and music. Love my 2 kids. Am looking for a lady with similar interests, aged between 38-45 for friendship/ possible relationship.\nGENUINE ATTRACTIVE MAN 40 y.o., no ties, secure, 5 ft. 9, slim build. I enjoy outdoors, gym, beach, country drives and quiet nights at home. Seeking an honest, caring woman, slim or med. build, who enjoys the simple things in life. Age open. \nGOOD LISTENER Well educated, tall, blue eyes, brown hair, Mid 20s, good looking, honest n/smoker. Likes movies, drives, nights in, looks not important, nor distance/ speed. Looking for someone to complete my social circles. Age open, friendship/relationship.\nGREEK/AUST. MALE Early 30s, 57, slim build, non-smoker, enjoys dining out, movies, bbqs., football and quiet nights at home. Seeking slim, attractive female, 20-30 with similar interests for friend ship/relationship.\nHANDSOME BUSINESSMAN Tall, slim, independant, under standing, mid 50s, looking for classy lady who wants to retain her privacy, still retain her independance and is looking for a special private relationship. \nHERE I AM PICK ME! PICK ME! Now that I have your attention Im a 35 y.o. 59 attractive guy, non-smoker/social drinker, fit and active, is seeking a slim- medium lady for friendship/ relationship. \nLIMESTONE COAST REGION Attractive, 6 fit and healthy 46 y.o. country guy. Interests incl. self sufficiencies, permaculture animals and travel would like to meet an attractive, non- smoking fun loving lady 35-45 for friendship view to permanent relationship\nMALE LATE 50\'\'s AUST Single, tall, prof. Interests: Music, theatre, dining, art, the beach and the environment. Seeking female with similar interests to share concerts, dining etc.\nOUTGOING GUY Late 30s, seeks lady, 25-50, size unimportant for fun and good times. \nRETIRED GENT Like to meet slim lady for long term caravan travel, heading Darwin in May, share driving. Must have sense of humour, non-smoker. Age open.\nROMANTIC 35 y.o. with one child. Likes dining, movies, beach, country drives, quiet nights, seeks employed 28 - 40 year old lady for relationship. Clare or Barossa region or northern suburbs.\nTALL, ATTRACTIVE MALE 40s no ties, down to earth, easy going with a sense of humour, understanding, with varied interests seeks genuine female for friendship, relationship, and quality times.\nVIETNAMESE MAN Single, never married, financially secure. Seeks Australian lady for genuine relationship, single mum welcome, aged up to 40.\nWIDOW EARLY 60\'\'s Self employed professional, well established, 55, slim build, UK born, many interests incl. Sports cars, sexy, young for age, looks and outlook seeks short or long term relationship with slim, petite lady, any age up to 60 years.\n31 YO GUY Slim, Seeks 25-35 yo female for friendship/relationship, kids ok. Eastern Suburbs\n43 yo Male, blond, outgoing, genuine, friendly seeking female bet/w 30-35 yo, easygoing, friendly.\n50 YEAR OLD Working tall guy, Would like to meet attractive lady for long term relationship.\nCENTRAL VICTORIAN single Dad with 2 teen daughters, 46 y.o. Looking for a special lady\nFIT & HEALTHY 60 y.o. self emp. tradesman, NS, seeks Indian/Sri Lankan lady, NS, Olive to Dark Complexion, 48-60 y.o. for F/dship Poss. rel.\nMATURE GENT. N/S. Non drinker. Looking for fship / rship with Asian or Indian lady. 35 - 50 years.\nROMANTIC Sexy country guy 58 NS seeks similar NS lady for fun and friendship.\nWELL DRESSED emotionally healthy man 37 like to meet full figured woman for relationship.\n48 slim, shy, S/S, S/D, likes NRL, movies, outdoors, nature walks and quiet time. DtoE, SOH, seeks same 38-50, children ok. Fship, poss Rship, genuine replies only. ALA. Lots of friends, but lonely.\n57YRS N/S, S/D. WLTM attractive classy Lady, slim to med build for outings, weekends away and poss perm rship, if suited. Nationality and age open.\nATTRACTIVE 34yo, GSOH, S/D. WLTM at tractive prof Ladies 40-50 who are time poor from work / career and like fun times and nights in or out. Have always connected more with older Women.\nE\'\'TAINING & interesting 46yo Prof. Kind, generous and a great listener who enjoys having fun. Witty & humorous comb with caring and responsible. Enjoys movies, music, reading & being social. WLTM Lady to enjoy company with view to beginning an exciting new rship.\nGENT 57yrs of age, GSOH, N/S, S/D, 177cm. Likes movies and theatre. Looking forward to meeting Lady with sim int. Please send photo. Fship, possible rship.\nGENTLEMAN is lonely, likes cooking, house work, gardening, wild life caring, and fishing. Would like to meet a lady to share this with me.\nHI, SWM, med build, 54y, likes quiet nts at home, self supporting, happy nat, fun loving, likes cooking. GSOH, S/S, S/D, seeking F 45-55 for fship, poss rship, likes to pamper, c/music, r/roll, loves life, not in a hurry.\nMALE 58 years old. Is there a Woman who would like to spend 1 weekend a month in Charters Towers especially country music festival weekend. If so please write. Ref 8826 \nPROFESSIONAL Male, tall, fit, 40s, new to area who enjoys walking, travel, outdoors, walks on bch, amongst many other interests. Seeking Christian Woman for fship, view to rship.\nSWM 45 DtoE, honest, S/D, 178cm, 79kg. Clean cut, intelligent, natur ist seeks petite open minded red hd or brunette. Asian, eurasian welcome. Home bch drives. Fship, poss rship.\nWLTM sincere, caring Lady to share life with. My age is 70 yrs. GSOH, easy going, honest, reliable, DtoE person. I like walks, the beach, fishing, the outdoors, movies, music, dining out, am a N/S, S/D. Looking for fship and a poss rship.\nASIAN LADY 25+ SOUGHT BY Aussie gent 40 for F/ship possible R/ship\nASIAN LADY sought. Kids OK as I love family. Nice & honest guy.\nASIAN LADY with GSOH sought by Businessman late 30s for f/ship r/ship\nATHLETIC tall slim gen prof 40s seeks friendly lady age open Morn Pen\nAre you out there & working 35-50yo NS, SD, GSOH, DTE, Caring thats me 2.\nAttractive young male seek ing casual meetings with mature ladies 30-50\nBRIGHT Boy with good brain, great cook & Physique. 42, 5/11 fair.\nBUSINESSMAN 60+ n/s s/d seeks lady age open for f/ship r/ship Im a good listener so lets have lunch and converse. Essendon area and inner suburbs\nBright happy positive prof male 48 fit/healthy n/s seeks lady same f/r/ship\nCARING LOVING & generous, seeks petite Female up to 38.\nCARING well dressed rom gent, seeks classy lady 35+ of elegance for r/ship\nCASUAL discreet weekday meetings no obligation East Suburbs.\nEUROPEAN Male 45, seeks married European lady for R/ship.\nFIT 36yo MARRIED male seeks attr married female for discreet encounter.\nGENT, 30\'\'s, seeks attr lady, easygoing, GSOH, for long term fship/rship.\nGREEK 41yo seeks Greek lady 30 - 37 for fship & rship\nGREEK GUY 46, seeking Greek lady up to 50yo\nGREEK Male Late 30\'\'s seeks Lady up to 37yrs for long term relationship\nGood Looking Male seeks larger lady for f/ship. Preferably nth subs\nLIKE TO BE MISTRESS of YOUR MAN like to be treated well. Bold DTE not precious, available during the day, appreciate a generous guy? Lets Talk! (50yo)\nLJ seeks JS of Bayswater Be mine.\nMALE 26 tall n/s seeks female early 30s for fun times South East Suburbs.\nMALE 38 seeking genuine lady s/s age open for f/ship possible r/ship\nMALE 60 - burly beared seeks intimate woman for outings n/s s/d F/ston/Pen\nMALE MID 30\'\'S Looking for no strings attached fun times.\nMARRIED HANDYMAN seeks lady for casual fun any age\nMARRIED MALE mid 40\'\'s attr missing TLC, seeks lady in similar position\nMARRIED MAN 50, attrac. fit, seeks lady 40-50 to share special times.\nMT ELIZA Business guy seeks gorg slim lady loves life with passion\nMale 44 fit running/gym seeks slim lady n/s 40-45 f/ship LTR\nMale Seeks nice girl 25 - 30 serious rship.\nMan 46 attractive fit, assertive, and kind. Would like to meet attractive fit stylish female. For dinner and wine. Northcote\nMarried Asian likes to meet lady for good times, Call me!\nPROF guy 68 GSOH casual seeks sexy lady n/s age open Mornington.\nSCORPIO 47 seeks passionate woman for discreet intimate encounters\nSEXY MARRIED MALE seeks married lady for discreet affair.\nSINGLE DAD 44yo, 6ft, 95 kg, male, romantic, caring, sincere & honest seeks medium/slim attrac single mum 35-45 def non smoker. Knox area\nSINGLE MALE 45 seeks female 40-50 for 4WD weekends away\nSINGLE MALE 49 seeks Fe male for f/ship possible r/ship kids OK\nSINGLE good looking 45 yo seeks 40+ lady casual f/ship & fun Married Ok\nSLIM attractive 40 looking for fun pref Asian.\nSTERN HEADMASTER With a firm hand, seeks naughty lady\nSgl 49 yo Male 6\'\' ave build seeks bi curious fem for fun fship poss rship.\nSingle dad. 42, East sub. 5"9 seeks woman 30+ for f/ship relationship\nTALL FAIR HAIR, mid 40\'\'s gent, affectionate/loyal, seeks lady 35-45 - R/ship\nWANT TO BE SPOILT have fun, be sexy\nWarm, natural sgl DTE 38yo, Prof. seeks similar lady, genuine fship/rship\nYOUNG attractive girls under 23 sought by good looking 30s guy.\n39 YR OLD male DTE GSOH looking for lady fun times & outings\nA BUSINESSMAN 47 seeks slim attract uni student for daytime meetings\nA FRIEND, A LOVER Self employed single 47 yo romantic easy going GSOH honest reliable looking to find a friend.\nASIAN GUY, 47 prof, well presented athletic no ties, N/S seeks lady to 47 for r/ship, cuddly lady welcome.\nATTRACT 32 yo male seeks a younger 18-27 lady for f/ship relationship\nCARING FIT SINGLE 52 yo Aust male. N/S GOSH financially secure seeks busty / curvaceous lady 42-48 for f/ship poss perm r/ship East suburbs.\nDISCIPLINARIAN Seeks lady for fun times\nEuro Guy Nth Subs seeks lady into cars, friendship, good times & family\nGOOD Guy seeking Asian lady, 20-30 N/S for f/ship r/ship.\nGUY SLIM FIT seeks lady, any age for intimate encounters\nGenuine, caring 52yo Male seeking DTE, female 50+ poss. R/ship.\nHARLEY RIDER 50 seeks casual affair with married lady.\nHERPES male seeks fem under 36 for r/ship, enjoys, outdoors, movies, dinners.\nJAPANESE/ASIAN 40-50 sought by Aussie mid 40s b/man f/ship r/ship\nLOVE to meet widowed lady over 50, no children in North West Suburbs.\nMALE 34 seeking DTE lady GSOH for casual fun, age & size open.\nMARRIED 32 fit attractive travels a lot looking for women for fun\nMARRIED 32 personal trainer looking for married woman age open for fun\nMARRIED Dark guy 37 seeks married lady 30-48 for discreet times. Sth E Subs.\nMARRIED MAN 42yo 6ft, fit, seeks Lady for discreet fun meetings.\nMid 30\'\'s Aussie guy, 6ft 2\'\' olive skin, never married no kids, but I have a dog who thinks he is human. I would like to meet a sexy, honest, reliable partner who enjoys being spoilt and is ready for commitment\nOLDER LADY sought by 32 male for friendship relation ship\nTALL MALE 35 European seeking fun, adventure & other things.\n2 NICE GUYS Aged 39 & 36 would like to meet two females for quality times, night clubbing, movies, dining out and friendship. TO\nA COUNTRY BOY 50+, good soh, non smoker, social drinker, Harley rider, semi retired, secure. Looking for nice attractive lady, slim to medium build, under 55 yrs. Must be non smoker, good soh to share life and fun times with. Possible relationship.\nA LITTLE MAGIC Looking for a lady, who is a non smoker, 45 - 52, slim to medium build, who likes long walks, articulate conversation, music, and a little fun with a 6 ft. 2 medium build blue/green eyed. gentleman.\nABOUND AND CAREFREE Vibrant personality, n-s, 5 ft. 11, 49 yrs. young, creative & adventurous, executive, articulate, intelligent & very flexible. Love a good laugh, love life & enjoy contrasts and the finer things in life. Seeks lady with same outlooks to 55 y.o.\nABOUT ME: 36 Easygoing, self employed, 511, medium build, smoker/social drinker, seeking slim-medium build woman who is happy in life, age open. \nACTIVE 57 YEARS 5 ft. 8, slim build, non-smoker, social drinker, living Southern suburbs. I enjoy a healthy, active lifestyle. Interests are: Sports, travel, dining out, movies and quiet nights at home. Seeking an affectionate, slim, easy going lady to 57 years for relationship.\nADELAIDE HILLS Good looking, single, Aussie country guy, 43 y.o. med. build, 5 ft. 8., honest and sincere. Likes BBQs, fishing, camping, dining out. Seeking genuine 30-45 y.o. lady who enjoys the outdoors and country life.\nADVENTUROUS 42 y.o. sensitive, non smoker, vegetarian, slim, 511", well travelled, beach loving. Seeks down to earth 25-43 y.o., slim, non smoker, into outdoor life, camping, walks and music. TO\nAGED 36, single, outgoing, good looking, short dark hair. Interested in AFL, basketball, cricket, 10 pin bowling and enjoy SANFL, travelling, going out for dinner, movies and nightclubs.\n52 Y.O. TERTIARY Educated professional woman, seeks professional, employed man, with interests in theatre, dining, music, good conversation etc. \nABOUT ME 53 y.o. lady, 5 ft 5, non smoker, social drinker. I enjoy gardening, music, movies, walking, dining out and quiet nights at home, V8 motor racing. I am an easygoing, honest and caring person, seeking gentleman 50 - 65 for friendship with view to relationship.\nATTRACTIVE EMPLOYED LADY Late 30s, seeking tall 5 ft. 10+ attractive to very attractive dark haired gentleman, strictly 37 - 41. No kids, working normal hours, not self employed, living near City.\nATTRACTIVE 50\'\'ish LADY Easygoing, down to earth. I enjoy travelling, movies, dining, wining, weekends away. Seeking gent 58-65 with similar interests. \nATTRACTIVE EASY GOING Intelligent, slim-medium build, 5 ft. 5, healthy living - exercise daily, employed and responsible. Seeking a trustworthy male 42-47 with integrity and good morals, non-smoker, social drinker. Children OK.\nBALLROOM DANCING SEEKING DANCING PARTNER Me: Mid 60s, 55", slimmish and after many years would like to enjoy dancing again. You must be taller than I and to avoid any complications, like me, unattached.\nCITY GIRL SEEKS COUNTRY ROMANCE Attractive 38 y.o. brunette living in Adelaide seeks country farmer/land owner (36-45), pref. clean shaven, non smoker, honest with strong family values for social outings and companionship. Other callers welcome to reply.\nDO YOU LOVE TO DANCE? Fun loving and employed, I enjoy gardening, BBQs, dining out, house parties &c. Looking for fit Rock N Roll dance partner. Friendship maybe leading to relationship.\nEARLY 60\'\'S Vibrant personality, good s.o.h. non-smoker/social drinker. Interests include beach walks, movies, fishing, dancing, looking for gent 510 plus. Prefer living western/southern suburbs. Friendship view to relationship.\nEASYGOING 53 Y.O. 5ft. 6 in. tall and of large build seeks a good man. I am a nonsmoker, social drinker, single mother of a 15 year old boy. Lives NE subs.\nFEISTY FOXY & A YUMMY GRANDMUMMY Late 40s, working full time. Looking for love & laughter. Are you at least 59, non-smoker, adventurous enough for exciting travel & comfortable enough to just watch TV, secure, fit but not fanatical? A bonus would be a love of dancing.\nGARDEN LOVER Hi! I am an Aust. lady, 62 yo, 5 ft. 1, slim-med., build, n-s, s-d. Interests: dining out, movies, entertain., quiet nights at home, walks on beach & gardening Seeks gentleman who is honest & looking for friendship, maybe lead to relationship.\nSEEKING HONEST MAN I am 41 y.o., 5 ft.4, med. build, employed, fun loving lady with a good SOH. Enjoy movies, dining out, country drives & quiet times. Seeks 35-45, honest man with good SOH & similar interests, friendship poss. relation ship. Southern area.\nVERY ROMANTIC LADY Attractive 54 y.o., medium build, who loves hugs and kisses and those special tingles when we click. No children but a home lover. Amongst other things I enjoy being by the river. Would like to meet a real romantic, age 50-65 y.o.\nWOMAN OF SUBSTANCE 56, 59 kg., 50, fit, n-s, elegant, articulate, seeks honest, sincere, social, n-s, well preserved 6. prof, sgle. Prince Charming. Living city/inner subs to share a future filled with fun, travel, adventure, glorious sunsets, candlelight dinners, fine wine, romance & true love.\nYORKE PENINSULA LADY Late 70s, 53, pleasingly plump, missing male conversation and company. My interests are: luncheons, cards, travel and indoor bowls. \nAFFECTIONATE loyal female, seeks tall male. 35-43 for genuine relationship/marriage. Melb. subs.\n43YO WLTM Male who enjoys life, dining in/ out, movies, long drives, car racing, gardening, travel. Must be employed, honest, and loving.\n48YO SWF WLTM a 42-54yo genuine, caring, honest and normal man for fship, poss rship. S/S, S/D, GSOH. Photo pls.\nA tall attractive Lady, 40 yrs young looking for a Male companion to go dancing with. Could be light hearted dance classes or out and about. I have some experience in Ballroom and Jive.\nLADY 51 WLTM SWM 50-57 for company and fship. S/S, S/D, enjoys movies, dining out and sedate outdoor walks. Photo appreciated.\nLADY 63. WLTM a genu ine person DtoE, possibly religious, loves horses, dogs, natural food (vegetarian), beaches, reading, fish ing. Friendship please.\nLADY, 43 well presented. WLTM Gent for dating, rship. Likes fishing, dining out, movies, life and wine. N/S, must be fin. secure. If you would like to meet a lovely, sexy lady please reply with a photo if poss. Genuine replies.\nSLIM well presented Fem. Early 50s, outgoing, intelligent, independant, into fitness, outdoor activites, sport, beach, music, theatre, movies, read ing. WLTM N/S Male with sim. int to share outings. Poss rship.\nSWF 40yo, S/D, S/S, sgl Mum. Looking to meet gen, honest, caring, non judgemental Guy up to 45yo. Must enjoy danc ing and life in general. Must be comfortable with themselves. Photo pls. Only gen enquiries.\nABBREVIATIONS GSOH Good sense of humour SOH Sense of humour N/D Non drinker S/D Social drinker S/S Social smoker N/S Non smoker ALA All letters answered WLTM Would like to meet DtoE Down to Earth DONT FORGET.. Its FREE to advertise in Perfect Match ! \n'

This is one big bunch of words. The reader also read the line endings \n, but did not split the text accordingly. We want to have one list entry per line. So let's split the raw text at the line endings. And inspect some random lines then.


In [5]:
import random 
slines = singles.splitlines()
for i in range(0,5):
    print(slines[random.randrange(len(slines))])


CUDDLY FULL FIGURED LADY 50 plus sought by Australian gent, early 60s, financially secure, non-drinker, non smoker, for permanent relationship.
GREEK Male Late 30''s seeks Lady up to 37yrs for long term relationship
MARRIED 32 personal trainer looking for married woman age open for fun
SWM 45 DtoE, honest, S/D, 178cm, 79kg. Clean cut, intelligent, natur ist seeks petite open minded red hd or brunette. Asian, eurasian welcome. Home bch drives. Fship, poss rship.
48YO SWF WLTM a 42-54yo genuine, caring, honest and normal man for fship, poss rship. S/S, S/D, GSOH. Photo pls.

Did you observe that some people use capitalized words, others don't? If not, run the above lines of code again to get some other examples. For most applications LADY, Lady and lady can and should be handled the same. So let's unify this. Note s.lower()works on a single string only. So in this tutorial we do it again on the raw text and also split the lines again.


In [7]:
slines2 = singles.lower().splitlines()
for i in range(0,5):
    print(slines2[random.randrange(len(slines2))])


about me 53 y.o. lady, 5 ft 5, non smoker, social drinker. i enjoy gardening, music, movies, walking, dining out and quiet nights at home, v8 motor racing. i am an easygoing, honest and caring person, seeking gentleman 50 - 65 for friendship with view to relationship.
a friend, a lover self employed single 47 yo romantic easy going gsoh honest reliable looking to find a friend.
6.2 35 yr old outgoing m seeks fem 28-35 for o/door sports - w/e away
single good looking 45 yo seeks 40+ lady casual f/ship & fun married ok
beautiful, intelligent caring, musical shy, late 20s, size 12, long blonde hair and likes to drink guiness. if this is you then i want you.

Tokenization

Next we figure out what our individual tokens are. Tokens might be words, but also numbers. That might seem easy. Just use the whitespace character for separation. But actually its not so easy. For instance, should "40-45" be one or two tokens? Here we will use a simple tokenizer, but generally it is worth investing some time in figuring out what individual tokens should be. We will only do this for a single example line from our corpus.


In [8]:
line = slines2[41]
print(line)
tokens = nltk.word_tokenize(line)
print(tokens)


romantic sexy country guy 58 ns seeks similar ns lady for fun and friendship.
['romantic', 'sexy', 'country', 'guy', '58', 'ns', 'seeks', 'similar', 'ns', 'lady', 'for', 'fun', 'and', 'friendship', '.']

Stopword removal

For some applications words like "not" and "we" do not carry any information but lead to a bigger vocabulary. So we might decide to remove them from the text. There are lists of stopwords available for many languages. Let's have a look what happens with our text. First let's have a look at the stopwords.


In [9]:
from nltk.corpus import stopwords
# has to be run only once to get the data
#nltk.download('stopwords')
print(stopwords.words('english'))


['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't"]

And remove them from the text.


In [10]:
filtered_words = [word for word in tokens if word not in stopwords.words('english')]

print(tokens)
print(filtered_words)


['romantic', 'sexy', 'country', 'guy', '58', 'ns', 'seeks', 'similar', 'ns', 'lady', 'for', 'fun', 'and', 'friendship', '.']
['romantic', 'sexy', 'country', 'guy', '58', 'ns', 'seeks', 'similar', 'ns', 'lady', 'fun', 'friendship', '.']

Stemming and Lemmatization

For some applications words with different inflections should be treated equally. So for instance, the occurrence of the word car has the same importance as the word cars. Stemmers and Lemmatizers adress this by conflating the words into their stems or lemmas. Let's see how they differ. First, we apply a very well known stemmer, the Porter stemmer. You also can try this interactively in this demo.


In [11]:
from nltk.stem.porter import *
stemmer = PorterStemmer()
porter_tokens = [stemmer.stem(word) for word in tokens]
print(porter_tokens)


['romant', 'sexi', 'countri', 'guy', '58', 'ns', 'seek', 'similar', 'ns', 'ladi', 'for', 'fun', 'and', 'friendship', '.']

In [12]:
# we only need to do this once
#nltk.download('wordnet')
from nltk.stem.wordnet import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
lemmas = [lemmatizer.lemmatize(word) for word in tokens]
print(lemmas)


['romantic', 'sexy', 'country', 'guy', '58', 'n', 'seek', 'similar', 'n', 'lady', 'for', 'fun', 'and', 'friendship', '.']

So, while a stemmer bluntly removes suffices a lemmatizer attempts to map a word to a canonical form (a form you find in a dictionary and is actually a word in the language. Both can not handly tokens that are not words (e.g. "42").

So far for this notebook. That's it. The end. Go!