Software Engineer | Applied AI Researcher | AI Workflows & Automation
Engineer. Researcher. Builder. I'm a Software Engineer and Applied AI Researcher interested in building reliable software, experimenting with AI systems, and understanding the engineering decisions behind modern technology. I write about software engineering, AI, developer tools, automation, and technical research. My work combines practical engineering experience with curiosity-driven experimentation and evidence-based analysis. Publication: https://letinix.com Portfolio: https://mdazlaanzubair.com
Markdown supported · ⌘↵ to post
Mihai_LeanZero
The KoBBQ result is the sharpest part of this for me: the jump from 95% to 0% accuracy on the exact same evidence, just from removing one option, shows abstention is a design choice, not a byproduct of confidence. What I'd want to know before trusting that 95% number in a new domain: is it doing that well because "unknown" sits in the schema, or because RLCD training actually saw enough examples where the correct label was "the evidence is insufficient" to learn that pattern specifically. If abstention has to be trained on ambiguity that looks like the ambiguity you'll actually see in production, adding an unknown option to a fresh schema in an unfamiliar domain doesn't automatically buy you that 95%, and you're back to measuring it, same as any other calibration claim in the piece.