---
title: "Comment by Ahmet Özel on \"A 3B language model can't plan robot moves — but it can tell a planner what to do (0% → 76%)\" | Hashnode"
description: "The 74% versus 78% split by phrasing would be more informative with a paired table for the 50 underlying tasks. How many succeed in both forms, only the origina"
url: "https://hashnode.com/posts/a-3b-language-model-can-t-plan-robot-moves-but-it-can-tell-a-planner-what-to-do-0-76/6afb39bcadcee1d0ae755ad9/comment/6afb71e591b1c666ea4e1f71"
---

[Discussion](https://hashnode.com/posts/a-3b-language-model-can-t-plan-robot-moves-but-it-can-tell-a-planner-what-to-do-0-76/6afb39bcadcee1d0ae755ad9)

Comment [A 3B language model can't plan robot moves — but it can tell a planner what to do (0% → 76%)](https://hashnode.com/posts/a-3b-language-model-can-t-plan-robot-moves-but-it-can-tell-a-planner-what-to-do-0-76/6afb39bcadcee1d0ae755ad9)

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[1d ago](https://hashnode.com/posts/a-3b-language-model-can-t-plan-robot-moves-but-it-can-tell-a-planner-what-to-do-0-76/6afb39bcadcee1d0ae755ad9/comment/6afb71e591b1c666ea4e1f71)

The 74% versus 78% split by phrasing would be more informative with a paired table for the 50 underlying tasks. How many succeed in both forms, only the original, only the paraphrase, or neither? Equal aggregate rates can hide a model swapping which instances it understands.

That table would also help target the 19 swapped-block failures. Keeping the initial layout and gold goal fixed while changing only the wording makes it possible to test whether colour names solve the mapping consistently, before adding more repair logic around plans that already pass the validator.

[purushothaman sinivasan](https://hashnode.com/@protechscacademy)

[22h ago](https://hashnode.com/posts/a-3b-language-model-can-t-plan-robot-moves-but-it-can-tell-a-planner-what-to-do-0-76/6afb39bcadcee1d0ae755ad9/comment/6afb71e591b1c666ea4e1f71#reply-6afc734d48415021cf24ae7f)

Thanks, Ahmet, that's a good catch. I went back to the per-item results (`episodes.csv`) and made the paired table for the full agent across the 50 instances:

|  | Paraphrase correct | Paraphrase wrong |
| --- | --- | --- |
| **Original correct** | 29 (both) | 8 (original only) |
| **Original wrong** | 10 (paraphrase only) | 3 (neither) |

So you were right that the matching totals hide churn: 18 of the 50 instances (36%) succeed in one phrasing and fail in the other. The direction is balanced (8 vs 10, exact McNemar p = 0.81), so neither wording is easier overall. The model just understands a different subset of the instances in each wording. The agent without the repair round shows the same pattern (27 / 8 / 12 / 3).

The swapped-block errors are split evenly too: 9 in the original wording and 10 in the paraphrase. The original wording also has 3 missing-goal-fact errors and 1 unsatisfiable goal; the paraphrase has 1 parse failure. Since the 18 flips keep the same initial layout and gold goal and change only the wording, they are already the controlled test you describe, and they say the colour-name mapping is not yet consistent.

Your suggested next step makes sense: run several wordings of each fixed instance and measure per-instance consistency before adding more repair logic. As the post shows, the repair round mostly turned invalid plans into valid plans for the wrong goal.
