Comment by Mihai_LeanZero on "Local LLM Inference: What to Know Before You Start"

Mihai_LeanZero

Atlassian consultant and Forge developer at LeanZero. We do Data Center to Cloud migrations and build Forge apps, and I write up what I learn at leanzero.net.

3d ago

The part that stands out most is the tool-calling behavior switch, one model printing tool-call-like text instead of actually invoking a tool, then starting to call correctly right after it had seen the coding-focused model make real calls earlier in the same conversation. That reads more like an in-context effect than a capability gap, the model probably wasn't missing the ability to call tools, it was missing a concrete example of what a successful call looks like inside that client's specific prompt format. If that's right, the failure isn't really about the model at all, it's about whether the client's system prompt or tool schema gets surfaced to that model the same way it does to the coding-focused one, since some servers translate the tools field into quite different text depending on which chat template the runtime picks for a given model file. Did you get a chance to check whether the two models were actually served with the same chat template, or could the first one have been getting one that renders tool definitions poorly?

Ikraj Singh

Building Reliable AI Analytics Systems

2d ago

Thanks for the read. In this example, I initialized both models through Lemonade and then started OpenCode, so my assumption is that the same skill context was available to both model instances. One model had no trouble working with tools and skills, while the other printed JSON text instead of making the tool call. I reproduced that behavior multiple times. My hunch is that the latter model was more strongly biased toward an API format that was not fully compatible with the tools available in this setup. Once it saw successful examples of how the other model called tools, it appeared to adapt and make successful calls. However, I have not specifically checked if both models were served same template, something i will keep in mind if get a chance to re-check the setup again.

That's a reasonable hunch, and it's checkable without much extra work if Lemonade logs the fully rendered prompt at some verbosity level. Diffing the two rendered prompts around the tools block would show directly whether the same JSON-schema-in-system-message format was used for both models, rather than inferring it from the failure mode alone. Worth checking too whether the model that struggled ships multiple chat template variants in its GGUF metadata versus a single one, since some conversions carry a template that doesn't actually match what the model was fine-tuned on.

Search Hashnode