Tokenization (NLP)

TLDR: Tokenization splits text into smaller units called tokens. It is the first step in nearly every NLP pipeline.

Tokenization is the process of breaking text into tokens. A token can be a word, a subword, or a single character. Models cannot read raw text directly. They read sequences of numeric token IDs. Tokenization converts text into those IDs. It is the first step in natural language processing.

Types of Tokenization

1. Word Tokenization: Split text on spaces and punctuation.
2. Subword Tokenization: Split rare words into smaller known pieces.
3. Character Tokenization: Treat every character as one token.
4. Sentence Tokenization: Split a document into individual sentences.

How Subword Tokenization Works

Modern large language models use subword tokenization. Common words stay whole. Rare words split into smaller known pieces. This keeps the vocabulary small. It also handles unseen words gracefully. Byte-Pair Encoding (BPE) is the most common method.

Why Tokenization Matters

1. Vocabulary Size: It controls model size and processing speed.
2. Context Windows: Limits are measured in tokens, not words.
3. Cost: LLM API pricing is charged per token.
4. Language Coverage: Poor tokenization hurts non-English text quality.

Tokenization and Embeddings

After tokenization, each token maps to an embedding. An embedding is a numeric vector the model can process. See embeddings in machine learning. Token embeddings feed into the model’s attention layers. This pipeline turns raw text into model-ready numbers.

Building Tokenizers with the Right Data

Tokenizers are trained on large, diverse text corpora. The data shapes which subwords the tokenizer learns. Broad training data improves coverage across languages. Bright Data’s Web Scraper collects real-world multilingual text. Its datasets provide clean corpora for tokenizer and model training.