Tokenization is the process of splitting text into smaller parts called tokens. In many NLP workflows, tokens are words, numbers, punctuation marks, or other meaningful units extracted from a sentence.
Usually, these tokens are words, numbers, or punctuation marks.
Apache OpenNLP provides different tokenization strategies. Three common options are:
TokenizerME: A maximum entropy tokenizer that detects token boundaries based on a probability model.WhitespaceTokenizer: A whitespace tokenizer where non-whitespace sequences are identified as tokens.SimpleTokenizer: A character class tokenizer where sequences of the same character class are treated as tokens.
TokenizerME
In this case, we first need to load the pre-trained model into a stream.
We can download the model file from here, put it in the /resources folder and load it from there. This tokenizer can be used with a custom-trained model as well.
InputStream inputStream = getClass()
.getResourceAsStream("/models/en-token.bin");
Read the stream to a Tokenizer model.
TokenizerModel model = new TokenizerModel(inputStream);
Initialize the tokenizer with the model.
TokenizerME tokenizer = new TokenizerME(model);
Use TokenizerME.tokenize() method to extract the tokens to a String Array.
String[] tokens = tokenizer.tokenize("John has a sister named Penny.");
WhitespaceTokenizer
As the name suggests, this tokenizer simply splits the sentence into tokens using whitespace characters as delimiters:
WhitespaceTokenizer tokenizer = WhitespaceTokenizer.INSTANCE;
String[] tokens = tokenizer.tokenize("John has a sister named Penny.");
We’ll see that white spaces will split the sentence, so we get “Penny.” with the period character at the end instead of two different tokens for the word “Penny” and the period character.
SimpleTokenizer
This tokenizer is a little more sophisticated than WhitespaceTokenizer and splits the sentence into words, numbers, and punctuation marks. However, it’s the default behavior and doesn’t require any model:
SimpleTokenizer tokenizer = SimpleTokenizer.INSTANCE;
String[] tokens = tokenizer.tokenize("John has a sister named Penny.");
Why Tokenization Matters
Tokenization is usually the first step before frequency counting, part-of-speech tagging, lemmatization, keyword-in-context analysis, or more advanced NLP tasks. If token boundaries are poor, downstream results can also become less reliable.
For a practical example, the Word Frequency Counter starts by breaking text into analyzable units before showing word counts, percentages, lemmas, POS tags, and context examples.
Conclusion
In this Apache OpenNLP article, we have seen different tokenization approaches provided by the OpenNLP Tokenizer API. For most applications, the right tokenizer depends on the quality of the input text and the level of linguistic precision the task requires.