Byte-pair encoding was a data compression algorithm published by Philip Gage in 1994 and repurposed for neural machine translation by Sennrich, Haddow and Birch at ACL 2016. It is simple enough to implement over a coffee, and implementing it is the fastest way to stop finding tokenizer behaviour...
Source: [Dev.to](https://dev.to/multigrid/byte-pair-encoding-explained-by-building-one-24pk)