NLTK 자연어 처리 패키지 소개

In [173]:

    

nltk.download('averaged_perceptron_tagger')
nltk.download("gutenberg")
nltk.download('punkt')
nltk.download('reuters')
nltk.download("stopwords")
nltk.download("taggers")
nltk.download("webtext")
nltk.download("wordnet")


    

    




[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /home/joel/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!
[nltk_data] Downloading package gutenberg to /home/joel/nltk_data...
[nltk_data]   Package gutenberg is already up-to-date!
[nltk_data] Downloading package punkt to /home/joel/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package reuters to /home/joel/nltk_data...
[nltk_data]   Package reuters is already up-to-date!
[nltk_data] Downloading package stopwords to /home/joel/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.
[nltk_data] Error loading taggers: Package 'taggers' not found in
[nltk_data]     index
[nltk_data] Downloading package webtext to /home/joel/nltk_data...
[nltk_data]   Package webtext is already up-to-date!
[nltk_data] Downloading package wordnet to /home/joel/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!






    
Out[173]:





True

In [9]:

    

nltk.corpus.gutenberg.fileids()


    

    
Out[9]:





[u'austen-emma.txt',
 u'austen-persuasion.txt',
 u'austen-sense.txt',
 u'bible-kjv.txt',
 u'blake-poems.txt',
 u'bryant-stories.txt',
 u'burgess-busterbrown.txt',
 u'carroll-alice.txt',
 u'chesterton-ball.txt',
 u'chesterton-brown.txt',
 u'chesterton-thursday.txt',
 u'edgeworth-parents.txt',
 u'melville-moby_dick.txt',
 u'milton-paradise.txt',
 u'shakespeare-caesar.txt',
 u'shakespeare-hamlet.txt',
 u'shakespeare-macbeth.txt',
 u'whitman-leaves.txt']

In [142]:

    

emma_raw = nltk.corpus.gutenberg.raw("austen-emma.txt")
print(emma_raw[:1302])


    

    




[Emma by Jane Austen 1816]

VOLUME I

CHAPTER I


Emma Woodhouse, handsome, clever, and rich, with a comfortable home
and happy disposition, seemed to unite some of the best blessings
of existence; and had lived nearly twenty-one years in the world
with very little to distress or vex her.

She was the youngest of the two daughters of a most affectionate,
indulgent father; and had, in consequence of her sister's marriage,
been mistress of his house from a very early period.  Her mother
had died too long ago for her to have more than an indistinct
remembrance of her caresses; and her place had been supplied
by an excellent woman as governess, who had fallen little short
of a mother in affection.

Sixteen years had Miss Taylor been in Mr. Woodhouse's family,
less as a governess than a friend, very fond of both daughters,
but particularly of Emma.  Between _them_ it was more the intimacy
of sisters.  Even before Miss Taylor had ceased to hold the nominal
office of governess, the mildness of her temper had hardly allowed
her to impose any restraint; and the shadow of authority being
now long passed away, they had been living together as friend and
friend very mutually attached, and Emma doing just what she liked;
highly esteeming Miss Taylor's judgment, but directed chiefly by
her own.

In [73]:

    

from nltk.tokenize import word_tokenize
word_tokenize(emma_raw[50:100])


    

    
Out[73]:





[u'Emma',
 u'Woodhouse',
 u',',
 u'handsome',
 u',',
 u'clever',
 u',',
 u'and',
 u'rich',
 u',',
 u'with',
 u'a']

In [81]:

    

from nltk.tokenize import RegexpTokenizer
t = RegexpTokenizer("[\w]+")
t.tokenize(emma_raw[50:100])


    

    
Out[81]:





[u'Emma', u'Woodhouse', u'handsome', u'clever', u'and', u'rich', u'with', u'a']

In [154]:

    

from nltk.tokenize import sent_tokenize
print(sent_tokenize(emma_raw[:1000])[3])


    

    




Sixteen years had Miss Taylor been in Mr. Woodhouse's family,
less as a governess than a friend, very fond of both daughters,
but particularly of Emma.

In [82]:

    

from nltk.stem import PorterStemmer
st = PorterStemmer()
st.stem("eating")


    

    
Out[82]:





u'eat'

In [89]:

    

from nltk.stem import LancasterStemmer
st = LancasterStemmer()
st.stem("shopping")


    

    
Out[89]:





'shop'

In [90]:

    

from nltk.stem import RegexpStemmer
st = RegexpStemmer("ing")
st.stem("cooking")


    

    
Out[90]:





u'cook'

In [95]:

    

from nltk.stem import WordNetLemmatizer
lm = WordNetLemmatizer()
print(lm.lemmatize("cooking"))
print(lm.lemmatize("cooking", pos="v"))
print(lm.lemmatize("cookbooks"))


    

    




cooking
cook
cookbook

In [99]:

    

print(WordNetLemmatizer().lemmatize("believes"))
print(LancasterStemmer().stem("believes"))


    

    




belief
believ

In [171]:

    

from nltk.tag import pos_tag
tagged_list = pos_tag(word_tokenize(emma_raw[:100]))
tagged_list


    

    
Out[171]:





[(u'[', 'NNS'),
 (u'Emma', 'NNP'),
 (u'by', 'IN'),
 (u'Jane', 'NNP'),
 (u'Austen', 'NNP'),
 (u'1816', 'CD'),
 (u']', 'NNP'),
 (u'VOLUME', 'NNP'),
 (u'I', 'PRP'),
 (u'CHAPTER', 'VBP'),
 (u'I', 'PRP'),
 (u'Emma', 'NNP'),
 (u'Woodhouse', 'NNP'),
 (u',', ','),
 (u'handsome', 'NN'),
 (u',', ','),
 (u'clever', 'NN'),
 (u',', ','),
 (u'and', 'CC'),
 (u'rich', 'JJ'),
 (u',', ','),
 (u'with', 'IN'),
 (u'a', 'DT')]

In [172]:

    

from nltk.tag import untag
untag(tagged_list)


    

    
Out[172]:





[u'[',
 u'Emma',
 u'by',
 u'Jane',
 u'Austen',
 u'1816',
 u']',
 u'VOLUME',
 u'I',
 u'CHAPTER',
 u'I',
 u'Emma',
 u'Woodhouse',
 u',',
 u'handsome',
 u',',
 u'clever',
 u',',
 u'and',
 u'rich',
 u',',
 u'with',
 u'a']