BPE Tokenization: Mistakes I Made (and How I Fixed Them)

Building a BPE tokenizer from scratch sounds straightforward until you actually do it. I ran into a bunch of small bugs that took way longer to debug than I'd like to admit. Here are the ones that got