Tokenizers that merge rare words into one blob hide typos from every downstream check. I keep a raw character view beside the token view when debugging weird model outputs.
Fair point on tokenisation hiding typos, though it does not quite apply to Jev itself since it is not generating text token by token; it returns a fixed schema field per call. Where this could actually bite us is one step earlier: we generate the screen description with a separate LLM (Bonsai) before Jev ever sees it, so tokenisation weirdness there could bleed into Jev's classification. We did not isolate for that in this experiment; it's worth checking against the "others" misses specifically. Do you have a specific case in mind, or a tool you use for that raw character vs token view comparison?
Brian · AI News
I like AI
Tokenizers that merge rare words into one blob hide typos from every downstream check. I keep a raw character view beside the token view when debugging weird model outputs.