---
title: "Discussion on \"I Ran 5 LLMs Through 10 Real Agent Coding Tasks. The Free One Won.\" | Hashnode"
description: "Discussion on \"I Ran 5 LLMs Through 10 Real Agent Coding Tasks. The Free One Won.\". What I Tested\nI gave 5 models the same 10 coding tasks — not LeetCode, not trivia. Tasks an autonomous agent actually does: parse a JSON config, find large files with a shell one-liner, fix a buggy merge function, write a concurrent HTTP fetcher. The..."
url: "https://hashnode.com/posts/llm-agent-coding-benchmark-gemini-beat-gpt/69fec23d444f429cef788133"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [I Ran 5 LLMs Through 10 Real Agent Coding Tasks. The Free One Won.](https://blog.workswithagents.dev/llm-agent-coding-benchmark-gemini-beat-gpt?utm_source=hashnode&utm_medium=feed)

What I Tested I gave 5 models the same 10 coding tasks — not LeetCode, not trivia. Tasks an autonomous agent actually does: parse a JSON config, find large files with a shell one-liner, fix a buggy merge function, write a concurrent HTTP fetcher. The...

May 9 · 3 min read · blog.workswithagents.dev  
Read the article

-   [#agents](https://hashnode.com/tag/agents)
-   [#ai](https://hashnode.com/tag/ai)
-   [#benchmark](https://hashnode.com/tag/benchmark)
-   [#llm](https://hashnode.com/tag/llm)

## Written by

[Vilius Vystartas](https://hashnode.com/@vilius)

@vilius · Cardiff, Uk

  
[View profile](https://hashnode.com/@vilius)

## Responses 1

Markdown supported · ⌘↵ to post

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[Oct 30](https://hashnode.com/posts/llm-agent-coding-benchmark-gemini-beat-gpt/69fec23d444f429cef788133/comment/6ae431a74ad0f273f3e70812)

The shared 500-token cap makes output-budget behavior part of the comparison, which is useful for an agent that needs concise code. I would record each response’s stop reason and whether executable code was complete before scoring it, though. A truncated explanation, a complete wrong program and a correct alternate shell syntax are different failure modes.

A second arm requesting code only under the same cap would help isolate instruction-following from implementation quality. Keeping both arms would preserve the practical observation without turning a budget-specific result into a general ranking of coding ability.
