---
title: "Comment by Aamer Mehaisi on \"5 Models, 467 Actions, 1 Winner — What We Learned Comparing LLMs on Real Code Generation\" | Hashnode"
description: "This is exactly the kind of empirical work the AI engineering community needs more of — real evaluation frameworks with statistical rigor. The action-based pipe"
url: "https://hashnode.com/posts/5-models-467-actions-1-winner-what-we-learned-comparing-llms-on-real-code-generation/69cac9405e4a2652e7fde674/comment/69cbcd2521888d13e21a96b9"
---

[Discussion](https://hashnode.com/posts/5-models-467-actions-1-winner-what-we-learned-comparing-llms-on-real-code-generation/69cac9405e4a2652e7fde674)

Comment [5 Models, 467 Actions, 1 Winner — What We Learned Comparing LLMs on Real Code Generation](https://hashnode.com/posts/5-models-467-actions-1-winner-what-we-learned-comparing-llms-on-real-code-generation/69cac9405e4a2652e7fde674)

[Aamer Mehaisi](https://hashnode.com/@mehaisi)

[Mar 31](https://hashnode.com/posts/5-models-467-actions-1-winner-what-we-learned-comparing-llms-on-real-code-generation/69cac9405e4a2652e7fde674/comment/69cbcd2521888d13e21a96b9)

This is exactly the kind of empirical work the AI engineering community needs more of — real evaluation frameworks with statistical rigor.

The **action-based pipeline** approach is particularly valuable because it mirrors how agents actually fail in production: errors compound across sequential operations. A model that's "good enough" on isolated tasks can become catastrophic when chained.

Your **Reasoning Model Trap** finding resonates strongly. I've observed similar behavior — models optimized for chain-of-thought reasoning often produce worse code outputs than simpler pattern-matching models. The architecture-task mismatch is real: reasoning isn't generation.

One insight that deserves more attention: **"Task type matters more than model choice."** The 10-point spread between easiest and hardest action dwarfed model differences. This suggests a practical strategy:

-   Route structured, well-defined tasks (SEO, accessibility, metadata) to cheaper models
-   Reserve premium models for spatial/visual tasks (layout, composition)

This isn't just cost optimization — it's reliability optimization. The variance problem you identified with DeepSeek V3.2 (95-point swing) is exactly why consistency premiums exist in production systems.

Question: Did you observe any correlation between token usage and quality? Curious if Sonnet's edge comes from context utilization or actual reasoning superiority.