---
title: "Comment by Ahmet Özel on \"I Ran 5 LLMs Through 10 Real Agent Coding Tasks. The Free One Won.\" | Hashnode"
description: "The shared 500-token cap makes output-budget behavior part of the comparison, which is useful for an agent that needs concise code. I would record each response"
url: "https://hashnode.com/posts/llm-agent-coding-benchmark-gemini-beat-gpt/69fec23d444f429cef788133/comment/6ae431a74ad0f273f3e70812"
---

[Discussion](https://hashnode.com/posts/llm-agent-coding-benchmark-gemini-beat-gpt/69fec23d444f429cef788133)

Comment [I Ran 5 LLMs Through 10 Real Agent Coding Tasks. The Free One Won.](https://hashnode.com/posts/llm-agent-coding-benchmark-gemini-beat-gpt/69fec23d444f429cef788133)

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[Oct 30](https://hashnode.com/posts/llm-agent-coding-benchmark-gemini-beat-gpt/69fec23d444f429cef788133/comment/6ae431a74ad0f273f3e70812)

The shared 500-token cap makes output-budget behavior part of the comparison, which is useful for an agent that needs concise code. I would record each response’s stop reason and whether executable code was complete before scoring it, though. A truncated explanation, a complete wrong program and a correct alternate shell syntax are different failure modes.

A second arm requesting code only under the same cap would help isolate instruction-following from implementation quality. Keeping both arms would preserve the practical observation without turning a budget-specific result into a general ranking of coding ability.
