---
title: "Discussion on \"5 Models, 467 Actions, 1 Winner — What We Learned Comparing LLMs on Real Code Generation\" | Hashnode"
description: "Discussion on \"5 Models, 467 Actions, 1 Winner — What We Learned Comparing LLMs on Real Code Generation\". Testing Claude, Kimi, DeepSeek on 467 real code generation tasks. The results changed how we think about model selection."
url: "https://hashnode.com/posts/5-models-467-actions-1-winner-what-we-learned-comparing-llms-on-real-code-generation/69cac9405e4a2652e7fde674"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [5 Models, 467 Actions, 1 Winner — What We Learned Comparing LLMs on Real Code Generation](https://tebza.hashnode.dev/5-models-467-actions-1-winner-what-we-learned-comparing-llms-on-real-code-generation?utm_source=hashnode&utm_medium=feed)

Testing Claude, Kimi, DeepSeek on 467 real code generation tasks. The results changed how we think about model selection.

Mar 30 · 10 min read · tebza.hashnode.dev  
Read the article

-   [#ai](https://hashnode.com/tag/ai)
-   [#llm](https://hashnode.com/tag/llm)
-   [#testing](https://hashnode.com/tag/testing)
-   [#web-development](https://hashnode.com/tag/web-development)

## Written by

[Tebogo Tseka](https://hashnode.com/@Tebza)

@Tebza

  
[View profile](https://hashnode.com/@Tebza)

Building with AWSomeness

## Responses 2

Markdown supported · ⌘↵ to post

[Aamer Mehaisi](https://hashnode.com/@mehaisi)

[Mar 31](https://hashnode.com/posts/5-models-467-actions-1-winner-what-we-learned-comparing-llms-on-real-code-generation/69cac9405e4a2652e7fde674/comment/69cbcd2521888d13e21a96b9)

This is exactly the kind of empirical work the AI engineering community needs more of — real evaluation frameworks with statistical rigor.

The **action-based pipeline** approach is particularly valuable because it mirrors how agents actually fail in production: errors compound across sequential operations. A model that's "good enough" on isolated tasks can become catastrophic when chained.

Your **Reasoning Model Trap** finding resonates strongly. I've observed similar behavior — models optimized for chain-of-thought reasoning often produce worse code outputs than simpler pattern-matching models. The architecture-task mismatch is real: reasoning isn't generation.

One insight that deserves more attention: **"Task type matters more than model choice."** The 10-point spread between easiest and hardest action dwarfed model differences. This suggests a practical strategy:

-   Route structured, well-defined tasks (SEO, accessibility, metadata) to cheaper models
-   Reserve premium models for spatial/visual tasks (layout, composition)

This isn't just cost optimization — it's reliability optimization. The variance problem you identified with DeepSeek V3.2 (95-point swing) is exactly why consistency premiums exist in production systems.

Question: Did you observe any correlation between token usage and quality? Curious if Sonnet's edge comes from context utilization or actual reasoning superiority.

[Ali Muwwakkil](https://hashnode.com/@enterpriseaileadersh)

[Mar 30](https://hashnode.com/posts/5-models-467-actions-1-winner-what-we-learned-comparing-llms-on-real-code-generation/69cac9405e4a2652e7fde674/comment/69cad2d1b3a51b36f5b43e5e)

In our experience with enterprise teams, the real challenge isn't just about choosing the right model, but integrating it effectively into existing workflows. Models like Claude can offer cost benefits, but their success often hinges on how well they're aligned with your team's processes. We've found that starting with a clear framework for prompt engineering can significantly enhance any model's performance, regardless of its initial capabilities. - Ali Muwwakkil (ali-muwwakkil on LinkedIn)