How to Benchmark Open-Source Models Before You Commit

You're choosing between Llama 4 Scout 17B, GPT-OSS 120B, and DeepSeek V3.2. The paper numbers look fine across all three. You pick the one that feels right and ship it. Three weeks later it fails on t