Python nltk tokenize

>>> import nltk
>>> sentence = """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['At', 'eight', "o'clock", 'on', 'Thursday', 'morning',
'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']

Source: stackoverflow.com

Import word_tokenize

import nltk
from nltk import word_tokenize

Similar codes

5 votes131 views3 answers
How to add button in tkinter
Python
0 vote182 views1 answer
Keyword python
Python
0 vote128 views1 answer
Get token from request django
Python
1 vote128 views1 answer
How to find and replace all the punctuation in python strings
Python
1 vote81 views1 answer
Numpy get count of list
Python
0 vote125 views1 answer
Python enumerate list with comprehension
Python
0 vote153 views3 answers
How to install pandas in python by git
Python
1 vote151 views1 answer
Pandas new column from others
Python
8 votes170 views1 answer
Roc curve python
Python
0 vote98 views1 answer
Max path limit python
Python