A few weeks ago my friend @u84u and I ( @ronak-create ) had a simple, slightly annoying question: if LLMs get 30-45% smaller representations of text by tokenizing it into subwords instead of raw bytes, why don't we tokenize text before handing it to a byte-level compressor like LZMA or zstd? It ...
Source: [Dev.to](https://dev.to/ronak_parmar_033c50d168b5/we-tested-tokenize-before-you-compress-against-452-configurations-and-it-mostly-held-up-4m6p)