Tokenization is the process of dividing text into smaller units called tokens. In NLTK, you can split text into word-level tokens with nltk.word_tokenize() or sentence-level tokens with nltk.sent_tokenize().

Word tokenization is useful when you need to analyze individual words and punctuation marks. Sentence tokenization is useful when you need to process each sentence separately before applying classification, tagging, summarization, or other natural language processing steps.

NLTK Tokenization

NLTK provides two methods: nltk.word_tokenize() to divide given text at word level and nltk.sent_tokenize() to divide given text at sentence level.

Install NLTK and Download Tokenizer Data

Install NLTK with pip before running the tokenization examples.

</>
Copy
python -m pip install nltk

NLTK keeps some tokenizer models in separate data packages. Recent NLTK versions may require the punkt_tab resource for word_tokenize() and sent_tokenize(). Download it once before tokenizing text.

</>
Copy
import nltk

nltk.download("punkt_tab")

Some older NLTK installations use the punkt package instead. When a LookupError occurs, read the resource name shown in the error and download that exact resource.

NLTK Word Tokenizer: nltk.word_tokenize()

The usage of these methods is provided below.

 tokens = nltk.word_tokenize(text)

where

  • text is the string provided as input.
  • word_tokenize() returns a list of strings (words) which can be stored as tokens.

The returned list can include punctuation marks as separate tokens. Therefore, the result is not always the same as splitting a string with text.split().

Example – Word Tokenizer

In the following example, we will learn how to divide given text into tokens at word level.

example.py – Python Program

</>
Copy
import nltk

# download nltk packages
# for tokenization
nltk.download('punkt')

# input string
text = """Sun rises in the east."""

# tokenize text to words
tokens = nltk.word_tokenize(text)

# print tokens
print(tokens)

Output

[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\TutorialKart\AppData\Roaming\nltk_data...
[nltk_data]   Package punkt is already up-to-date!
['Sun', 'rises', 'in', 'the', 'east', '.']

punkt is the required package for tokenization. Hence you may download it using nltk download manager or download it programmatically using nltk.download('punkt').

Notice that the period is returned as a separate token. This behavior is useful when punctuation needs to be analyzed or filtered independently.

How NLTK word_tokenize() Handles Punctuation and Contractions

word_tokenize() does more than split text at spaces. It uses NLTK tokenization rules to separate punctuation and handle common English contractions.

</>
Copy
from nltk.tokenize import word_tokenize

text = "I can't attend today's meeting, but I'll reply later."
tokens = word_tokenize(text)

print(tokens)
['I', 'ca', "n't", 'attend', 'today', "'s", 'meeting', ',', 'but', 'I', "'ll", 'reply', 'later', '.']

The exact token boundaries depend on the tokenizer and language rules. Review the output before building logic that assumes a contraction or punctuation mark will remain attached to a word.

Filter Punctuation from NLTK Word Tokens

When only alphabetic word tokens are required, use str.isalpha() to remove punctuation and numeric tokens.

</>
Copy
from nltk.tokenize import word_tokenize

text = "NLTK tokenized 3 sentences correctly."
tokens = word_tokenize(text)
word_tokens = [token for token in tokens if token.isalpha()]

print(word_tokens)
['NLTK', 'tokenized', 'sentences', 'correctly']

This filtering step also removes numbers. Use a different condition when numbers, hyphenated terms, email addresses, or other special tokens must be preserved.

NLTK Sentence Tokenizer: nltk.sent_tokenize()

</>
Copy
 tokens = nltk.sent_tokenize(text)

where

  • text is the string provided as input.
  • sent_tokenize() returns a list of strings (sentences) which can be stored as tokens.

The tokenizer uses punctuation and learned language patterns to identify sentence boundaries. A newline does not always indicate a new sentence, and a period does not always end one.

Example – Sentence Tokenizer

In this example, we will learn how to divide given text into tokens at sentence level.

example.py – Python Program

</>
Copy
import nltk

# download nltk packages
# for tokenization
nltk.download('punkt')

# input string
text = """Sun rises in the east.

Sun sets in the west."""

# tokenize at sentence level
tokens = nltk.sent_tokenize(text)

# print tokens
print(tokens)

Output

[nltk_data] Downloading package punkt to
[nltk_data]     C:\Users\TutorialKart\AppData\Roaming\nltk_data...
[nltk_data]   Package punkt is already up-to-date!
['Sun rises in the east.', 'Sun sets in the west.']

punkt is the required package for tokenization.

Tokenize a Paragraph into Sentences and Then Words

For paragraph processing, first divide the paragraph into sentences and then tokenize each sentence into words. This preserves the relationship between each word token and its sentence.

</>
Copy
from nltk.tokenize import sent_tokenize, word_tokenize

paragraph = "NLTK processes text. It can split text into sentences and words."

sentences = sent_tokenize(paragraph)

for sentence in sentences:
    words = word_tokenize(sentence)
    print(words)
['NLTK', 'processes', 'text', '.']
['It', 'can', 'split', 'text', 'into', 'sentences', 'and', 'words', '.']

Choose a Language for NLTK Sentence Tokenization

sent_tokenize() accepts a language argument. Use a supported tokenizer model that matches the language of the input text.

</>
Copy
from nltk.tokenize import sent_tokenize

text = "Das ist der erste Satz. Das ist der zweite Satz."
sentences = sent_tokenize(text, language="german")

print(sentences)
['Das ist der erste Satz.', 'Das ist der zweite Satz.']

Language support depends on the tokenizer resources installed with NLTK. Do not assume that the default English model will identify sentence boundaries correctly in every language.

Use TreebankWordTokenizer Without Downloading Punkt Data

When only English-style word tokenization is needed, TreebankWordTokenizer can tokenize a string directly without loading the sentence tokenizer used by word_tokenize().

</>
Copy
from nltk.tokenize import TreebankWordTokenizer

tokenizer = TreebankWordTokenizer()
text = "NLTK's tokenizer separates punctuation."

tokens = tokenizer.tokenize(text)
print(tokens)
['NLTK', "'s", 'tokenizer', 'separates', 'punctuation', '.']

This tokenizer does not perform sentence segmentation. Use sent_tokenize() first when the text must be divided into sentences.

Use wordpunct_tokenize() for Rule-Based Token Splitting

wordpunct_tokenize() separates text into alphabetic sequences, numeric sequences, and punctuation. It does not require the Punkt sentence model.

</>
Copy
from nltk.tokenize import wordpunct_tokenize

text = "The price changed from 10.5 to 12.75."
tokens = wordpunct_tokenize(text)

print(tokens)
['The', 'price', 'changed', 'from', '10', '.', '5', 'to', '12', '.', '75', '.']

This example shows why tokenizer choice matters: decimal numbers are divided around the period. Select a tokenizer according to the structure that must be preserved in the application.

NLTK word_tokenize() Compared with Python split()

Python’s split() method separates text primarily at whitespace. NLTK’s word_tokenize() applies language-aware tokenization rules and generally separates punctuation and contractions more carefully.

</>
Copy
from nltk.tokenize import word_tokenize

text = "Hello, world!"

print(text.split())
print(word_tokenize(text))
['Hello,', 'world!']
['Hello', ',', 'world', '!']

Use split() when whitespace separation is sufficient. Use an NLTK tokenizer when punctuation, contractions, sentence boundaries, or linguistic preprocessing matter.

Fix NLTK Tokenizer Resource Errors

LookupError for punkt_tab or punkt

A LookupError means that the tokenizer data cannot be found in the active NLTK data directories. Download the resource named in the error.

</>
Copy
import nltk

nltk.download("punkt_tab")

For an older installation that explicitly asks for punkt, use nltk.download("punkt").

NLTK data is installed for a different Python environment

Confirm that the same Python environment is used to install NLTK, download resources, and run the script. Virtual environments, notebooks, IDEs, and system terminals may use different Python interpreters.

Sentence boundaries are incorrect for abbreviations

Periods in abbreviations such as “Dr.”, “Mr.”, or initials may be mistaken for sentence endings in unfamiliar text domains. Test the sentence tokenizer with representative input and consider training or configuring a Punkt tokenizer when the default behavior is unsuitable.

NLTK Tokenization Frequently Asked Questions

What is tokenization in NLTK?

Tokenization in NLTK is the process of dividing text into units such as sentences, words, punctuation marks, or other token types. NLTK provides multiple tokenizer classes and convenience functions for different text-processing needs.

How do I tokenize text into words with NLTK?

Import word_tokenize from nltk.tokenize, download the required tokenizer data, and call word_tokenize(text). The function returns a list containing words and punctuation tokens.

How do I split a paragraph into sentences with NLTK?

Use sent_tokenize(paragraph). It returns a list in which each item is a detected sentence. You can then pass each sentence to word_tokenize() when both sentence-level and word-level tokens are required.

Why does NLTK word_tokenize() separate punctuation?

Punctuation carries grammatical and semantic information, so NLTK generally returns punctuation marks as independent tokens. Filter them after tokenization only when the application does not need them.

What is the difference between word_tokenize() and wordpunct_tokenize()?

word_tokenize() uses NLTK’s recommended word-tokenization workflow and language resources. wordpunct_tokenize() applies a simpler regular-expression-based split that separates alphabetic text, numbers, and punctuation and does not require Punkt data.

NLTK Tokenization Editorial QA Checklist

  • Run every word_tokenize() and sent_tokenize() example with the current NLTK release.
  • Confirm whether the current environment requires punkt_tab, punkt, or another resource named by NLTK.
  • Verify that punctuation and contraction outputs match the tokenizer used in each example.
  • Check that sentence-tokenization examples preserve abbreviations and sentence boundaries as described.
  • Ensure that new Python examples use language-python, commands use language-bash, and result blocks use output.
  • Confirm that examples do not treat tokenization as universally language-independent.