Comment by OnlineProxy on "We benchmarked Yad against browser-use, then checked the answers"

I noticed the whole trace on the grading blind spot, this is basically what separates useful evals from vibes based ones. splitting like that is the right move because a single pass fail bit hides exactly the failure mode you care about most. on task 7 the model literally couldn't see the ratings in its input so calling that abstention correct isn't a stretch, it's just accurate scoring given what it had access to. one thing I'd push back on tho, don't treat the self recovery as proof of anything robust, run it a bunch of times with induced failures before you draw conclusions from that. also worth double checking which free tier models actually ran each side, that timing and step count gap you're seeing could just be model differences dressed up as an architecture story

Thanks for this, genuinely, this is the kind of comment that makes a post better after the fact.

Agreed on task 7. The model had no way to read those star ratings from the text it was given, so the abstention wasn't caution for its own sake, it was the only answer the input actually supported.

On the two pushbacks, we went and checked both instead of just taking the point. Model matching: browser-use's side is confirmed, minimax-m3 (free) via OpenRouter. Yad's side we can't verify for that specific run. The per-task model logging that would prove it turned out to be broken, an unrelated bugfix reverted the file that had it before the comparison ran, so the exact run has zero logged model data. Worse, we found that the same evening, a separate run answered every task with a paid fallback model instead of free tier due to an unrelated stale-config bug, so this exact failure mode is not hypothetical, it already happened once. We've now rebuilt the logging properly, tests green, and it turned out to be half broken on our main branch too, an actual bug your question surfaced. It's sitting as a pull request right now, not merged yet, so it's not live for runs today, but once it lands, future comparisons will have this nailed down automatically. This run doesn't get a retroactive fix. We'll say so plainly if we ever cite it again.

Self-recovery: you're right that one instance proves nothing. We built a harness that repeats the same task N times and forces an unrelated navigation right after the first real step, outside the agent's own loop, then checks if it still lands the answer. Tried it live three separate times tonight, 11 attempts total counting retries, and every single one hit a free-tier rate limit before the agent got past its first step. So there's no number yet, not a low one, none. The harness works, the electricity bill doesn't cooperate. Will post the actual recovery rate once we have one, good or bad.

Search Hashnode