--- title: "Discussion on \"Local LLM Inference: What to Know Before You Start\" | Hashnode" description: "Discussion on \"Local LLM Inference: What to Know Before You Start\". My experience navigating models, servers, runtimes, and hardware compatibility." url: "https://hashnode.com/posts/local-llm-inference-what-to-know-before-you-start/6abb362687ee11aa2e1512ed" --- [Feed](https://hashnode.com/feed) / Discussion # [Local LLM Inference: What to Know Before You Start](https://ikrajsingh.hashnode.dev/local-llm-inference-what-to-know-before-you-start?utm_source=hashnode&utm_medium=feed) My experience navigating models, servers, runtimes, and hardware compatibility. Sep 29 · 10 min read · ikrajsingh.hashnode.dev Read the article - [#llm](https://hashnode.com/tag/llm) - [#llms](https://hashnode.com/tag/llms) - [#local-llm](https://hashnode.com/tag/local-llm) - [#local-inference](https://hashnode.com/tag/local-inference) - [#generative-ai](https://hashnode.com/tag/generative-ai) ## Written by [Ikraj Singh](https://hashnode.com/@ikraj-singh) @ikraj-singh · Sunnyvale, USA [View profile](https://hashnode.com/@ikraj-singh) Building Reliable AI Analytics Systems Senior Data Engineer passionate about building AI systems that are reliable, maintainable, and useful in production. ## Responses 7 Markdown supported · ⌘↵ to post [Kartik N V J K](https://hashnode.com/@kartik-nvj) [Sep 29](https://hashnode.com/posts/local-llm-inference-what-to-know-before-you-start/6abb362687ee11aa2e1512ed/comment/6abc154cf115f8caf396b76a) The detail that matches my own experience is Claude Code hitting API-compatibility problems against a local server while plain chat worked. I ran into the same wall: the multi-turn tool-call loop expects OpenAI-exact response shapes, and Lemonade's llama.cpp bridge drops fields the agent needs mid-task. Did switching to OpenCode fix the tool-call handshake for you, or did you still patch responses? [Ikraj Singh](https://hashnode.com/@ikraj-singh) [Sep 30](https://hashnode.com/posts/local-llm-inference-what-to-know-before-you-start/6abb362687ee11aa2e1512ed/comment/6abc154cf115f8caf396b76a#reply-6abc8d165b882b5f65c8872a) Thanks for the read! I switched to OpenCode completely, it worked with all the models I tried. It is not extensive like other CLI tools I have used like claude code or codex, but its close. [BotSailor](https://hashnode.com/@botsailor) [Sep 29](https://hashnode.com/posts/local-llm-inference-what-to-know-before-you-start/6abb362687ee11aa2e1512ed/comment/6abba1b8b41d85c26dbdf5b4) Really enjoyed this practical walkthrough. What stood out to me is how you explained that running an LLM locally is much more than just downloading a model—it’s really about getting the client, server, runtime, model format, hardware, and accelerator to work together. I especially liked the point about memory and model size not being the whole story. A model can technically fit in memory and still be too slow for a useful interactive experience, which is something that’s easy to overlook when getting started. The section on coding agents was also interesting. Multiple sequential model calls can make latency much more noticeable compared with simple chat. Overall, this feels like a very honest look at the amount of experimentation involved in local inference. Great read for anyone thinking about going beyond cloud-hosted models! [Ikraj Singh](https://hashnode.com/@ikraj-singh) [Sep 30](https://hashnode.com/posts/local-llm-inference-what-to-know-before-you-start/6abb362687ee11aa2e1512ed/comment/6abba1b8b41d85c26dbdf5b4#reply-6abc8c1f22b49008ceea2ad8) Thanks for your thoughtful read! [Mihai\_LeanZero](https://hashnode.com/@mihaileanzero) [Sep 29](https://hashnode.com/posts/local-llm-inference-what-to-know-before-you-start/6abb362687ee11aa2e1512ed/comment/6abb8b10126a47d11ee0a2da) The part that stands out most is the tool-calling behavior switch, one model printing tool-call-like text instead of actually invoking a tool, then starting to call correctly right after it had seen the coding-focused model make real calls earlier in the same conversation. That reads more like an in-context effect than a capability gap, the model probably wasn't missing the ability to call tools, it was missing a concrete example of what a successful call looks like inside that client's specific prompt format. If that's right, the failure isn't really about the model at all, it's about whether the client's system prompt or tool schema gets surfaced to that model the same way it does to the coding-focused one, since some servers translate the tools field into quite different text depending on which chat template the runtime picks for a given model file. Did you get a chance to check whether the two models were actually served with the same chat template, or could the first one have been getting one that renders tool definitions poorly? [Ikraj Singh](https://hashnode.com/@ikraj-singh) [Sep 30](https://hashnode.com/posts/local-llm-inference-what-to-know-before-you-start/6abb362687ee11aa2e1512ed/comment/6abb8b10126a47d11ee0a2da#reply-6abc8be5c3cb725eacfc6746) Thanks for the read. In this example, I initialized both models through Lemonade and then started OpenCode, so my assumption is that the same skill context was available to both model instances. One model had no trouble working with tools and skills, while the other printed JSON text instead of making the tool call. I reproduced that behavior multiple times. My hunch is that the latter model was more strongly biased toward an API format that was not fully compatible with the tools available in this setup. Once it saw successful examples of how the other model called tools, it appeared to adapt and make successful calls. However, I have not specifically checked if both models were served same template, something i will keep in mind if get a chance to re-check the setup again. [Mihai\_LeanZero](https://hashnode.com/@mihaileanzero) [Sep 30](https://hashnode.com/posts/local-llm-inference-what-to-know-before-you-start/6abb362687ee11aa2e1512ed/comment/6abb8b10126a47d11ee0a2da#reply-6abcd5ad074b1d04328f16a7) That's a reasonable hunch, and it's checkable without much extra work if Lemonade logs the fully rendered prompt at some verbosity level. Diffing the two rendered prompts around the tools block would show directly whether the same JSON-schema-in-system-message format was used for both models, rather than inferring it from the failure mode alone. Worth checking too whether the model that struggled ships multiple chat template variants in its GGUF metadata versus a single one, since some conversions carry a template that doesn't actually match what the model was fine-tuned on.