setting up nltk: installing, downloading, importing¶
# first try importing NLTK. If this works, skip down to the cekk that says "nltk.download()"
import nltk## if the above line give you an error, then install NLTK by running the conda install command below
## uncomment the line in order to run it!
# %conda install anaconda::nltk## then, download all the necessary text files
## uncomment the line below to run it
# nltk.download('all')# now, we can import nltk
import nltksentence = "Hello my name is Filipa and I teach python at Pratt Institute, School of Information."type(sentence)str# we can tokenize our string
tokens = nltk.word_tokenize(sentence)# tokens are the basic building block for all text analysis tasks
tokens['Hello',
'my',
'name',
'is',
'Filipa',
'and',
'I',
'teach',
'python',
'at',
'Pratt',
'Institute',
',',
'School',
'of',
'Information',
'.']# NLTK comes with a part of speech tagger
tagged = nltk.pos_tag(tokens)tagged[('Hello', 'NNP'),
('my', 'PRP$'),
('name', 'NN'),
('is', 'VBZ'),
('Filipa', 'NNP'),
('and', 'CC'),
('I', 'PRP'),
('teach', 'VBP'),
('python', 'NN'),
('at', 'IN'),
('Pratt', 'NNP'),
('Institute', 'NNP'),
(',', ','),
('School', 'NNP'),
('of', 'IN'),
('Information', 'NNP'),
('.', '.')]nltk.book¶
from nltk.book import **** Introductory Examples for the NLTK Book ***
Loading text1, ..., text9 and sent1, ..., sent9
Type the name of the text or sentence to view it.
Type: 'texts()' or 'sents()' to list the materials.
text1: Moby Dick by Herman Melville 1851
text2: Sense and Sensibility by Jane Austen 1811
text3: The Book of Genesis
text4: Inaugural Address Corpus
text5: Chat Corpus
text6: Monty Python and the Holy Grail
text7: Wall Street Journal
text8: Personals Corpus
text9: The Man Who Was Thursday by G . K . Chesterton 1908
text4len(text4)156288for i in text4[:20]:
print(i)Fellow
-
Citizens
of
the
Senate
and
of
the
House
of
Representatives
:
Among
the
vicissitudes
incident
to
life
no
for i in text2[:20]:
print(i)[
Sense
and
Sensibility
by
Jane
Austen
1811
]
CHAPTER
1
The
family
of
Dashwood
had
long
been
settled
in
type(text1)nltk.text.Text# turn the corpus into an nltk.text.Text object
moby = nltk.Text(text1)# use dir() to see all the functions associated with this new object
dir(moby)['_CONTEXT_RE',
'_COPY_TOKENS',
'__class__',
'__delattr__',
'__dict__',
'__dir__',
'__doc__',
'__eq__',
'__firstlineno__',
'__format__',
'__ge__',
'__getattribute__',
'__getitem__',
'__getstate__',
'__gt__',
'__hash__',
'__init__',
'__init_subclass__',
'__le__',
'__len__',
'__lt__',
'__module__',
'__ne__',
'__new__',
'__reduce__',
'__reduce_ex__',
'__repr__',
'__setattr__',
'__sizeof__',
'__static_attributes__',
'__str__',
'__subclasshook__',
'__weakref__',
'_context',
'_train_default_ngram_lm',
'collocation_list',
'collocations',
'common_contexts',
'concordance',
'concordance_list',
'count',
'dispersion_plot',
'findall',
'generate',
'index',
'name',
'plot',
'readability',
'similar',
'tokens',
'vocab']moby.similar('man')whale ship one it thing boat time all that ahab him sea captain whales
way matter moment and world as
moby.similar('woman')man king hand whale devil as bird mast job lord day serpent boat stone
ship harris captain mariner fellow hands
moby.dispersion_plot(['man', 'woman'])
text3.similar('man')land lord men place woman earth waters well city lad day cattle field
wife way flood servant people famine pillar
text3.similar('woman')lord land man earth men morning tree well people lad day place field
ground city canaanites country egyptian pit darkness
text5.similar('woman')ugly teacher tongue mistake virgin
text5.similar('man')girl gay part hey ah nick do ok yes male no how oh k jerk was sex yea
leave life
text3.dispersion_plot(['woman', 'man'])
text3.concordance('woman')Displaying 21 of 21 matches:
D God had taken from man , made he a woman , and brought her unto the man . And
flesh of my fle she shall be called Woman , because she was taken out of Man .
God had made . And he said unto the woman , Yea , hath God said , Ye shall not
f every tree of the garden ? And the woman said unto the serpent , We may eat o
die . And the serpent said unto the woman , Ye shall not surely die : For God
knowing good and evil . And when the woman saw that the tree was good for food
est not eat ? And the man said , The woman whom thou gavest to be with me , she
eat . And the LORD God said unto the woman , What is this that thou hast done ?
s this that thou hast done ? And the woman said , The serpent beguiled me , and
will put enmity between thee and the woman , and between thy seed and her seed
hou shalt bruise his heel . Unto the woman he said , I will greatly multiply th
ld now , I know that thou art a fair woman to look up Therefore it shall come t
nto Egypt , the Egyptians beheld the woman that she was very fair . The princes
d commended her before Phara and the woman was taken into Pharaoh ' s house . A
, thou art but a dead man , for the woman which thou hast taken ; for she is a
ant said unto him , Peradventure the woman will not be willing to follow me unt
unto my son from thence . And if the woman will not be willing to follow thee ,
id unto my master , Peradventure the woman will not follow me . And he said unt
raw for thy came let the same be the woman whom the LORD hath appointed out for
ite , to receive his pledge from the woman ' s ha but he found her not . Then h
, and Shaul the son of a Canaanitish woman . And the sons of Levi ; Gershon , K
# see more functions at
# https://www.nltk.org/api/nltk.text.Text.html # import them as corpus
from nltk.corpus import genesis, nps_chat, inauguralinaugural.fileids()['1789-Washington.txt',
'1793-Washington.txt',
'1797-Adams.txt',
'1801-Jefferson.txt',
'1805-Jefferson.txt',
'1809-Madison.txt',
'1813-Madison.txt',
'1817-Monroe.txt',
'1821-Monroe.txt',
'1825-Adams.txt',
'1829-Jackson.txt',
'1833-Jackson.txt',
'1837-VanBuren.txt',
'1841-Harrison.txt',
'1845-Polk.txt',
'1849-Taylor.txt',
'1853-Pierce.txt',
'1857-Buchanan.txt',
'1861-Lincoln.txt',
'1865-Lincoln.txt',
'1869-Grant.txt',
'1873-Grant.txt',
'1877-Hayes.txt',
'1881-Garfield.txt',
'1885-Cleveland.txt',
'1889-Harrison.txt',
'1893-Cleveland.txt',
'1897-McKinley.txt',
'1901-McKinley.txt',
'1905-Roosevelt.txt',
'1909-Taft.txt',
'1913-Wilson.txt',
'1917-Wilson.txt',
'1921-Harding.txt',
'1925-Coolidge.txt',
'1929-Hoover.txt',
'1933-Roosevelt.txt',
'1937-Roosevelt.txt',
'1941-Roosevelt.txt',
'1945-Roosevelt.txt',
'1949-Truman.txt',
'1953-Eisenhower.txt',
'1957-Eisenhower.txt',
'1961-Kennedy.txt',
'1965-Johnson.txt',
'1969-Nixon.txt',
'1973-Nixon.txt',
'1977-Carter.txt',
'1981-Reagan.txt',
'1985-Reagan.txt',
'1989-Bush.txt',
'1993-Clinton.txt',
'1997-Clinton.txt',
'2001-Bush.txt',
'2005-Bush.txt',
'2009-Obama.txt',
'2013-Obama.txt',
'2017-Trump.txt',
'2021-Biden.txt',
'2025-Trump.txt']trump_2 = inaugural.words('2025-Trump.txt')trump_2['Thank', 'you', '.', 'Thank', 'you', 'very', 'much', ...]# turn the corpus into an nltk.text.Text object
t_2 = nltk.Text(trump_2)t_2.similar('nation')citizens country world government society hurricane love republic
level rate military wars treaty
genesis.fileids()['english-kjv.txt',
'english-web.txt',
'finnish.txt',
'french.txt',
'german.txt',
'lolcat.txt',
'portuguese.txt',
'swedish.txt']nps_chat.fileids()['10-19-20s_706posts.xml',
'10-19-30s_705posts.xml',
'10-19-40s_686posts.xml',
'10-19-adults_706posts.xml',
'10-24-40s_706posts.xml',
'10-26-teens_706posts.xml',
'11-06-adults_706posts.xml',
'11-08-20s_705posts.xml',
'11-08-40s_706posts.xml',
'11-08-adults_705posts.xml',
'11-08-teens_706posts.xml',
'11-09-20s_706posts.xml',
'11-09-40s_706posts.xml',
'11-09-adults_706posts.xml',
'11-09-teens_706posts.xml']