NLP models cannot process words directly, which is why tokenization[1] exists. Tokenization ends with a list of integers — token IDs like [30642, 1634, 318, ... ] .

Source: [Dev.to](https://dev.to/rishikadhar/what-a-neural-network-actually-computes-from-token-ids-to-matrix-multiplication-2emc)

Sponsored