Tokenization

Tokenization is the process of breaking down a large paragraph of text into smaller parts, known as tokens. These tokens can be morphemes, word pieces, words, phrases, or sentences.

Papers published in this field over the years: