---
title: "Discussion on \"Using a mac mini (2012) for LLM inference with KoboldCpp \" | Hashnode"
description: "Discussion on \"Using a mac mini (2012) for LLM inference with KoboldCpp \". or... \"necessity is the mother of invention\""
url: "https://hashnode.com/posts/using-a-mac-mini-2012-for-llm-inference-with-koboldcpp/6aa184fbdb8661a7a6a47a26"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Using a mac mini (2012) for LLM inference with KoboldCpp](https://mamonu.hashnode.dev/using-a-mac-mini-2012-for-llm-inference-with-koboldcpp?utm_source=hashnode&utm_medium=feed)

or... "necessity is the mother of invention"

Sep 9 · 24 min read · mamonu.hashnode.dev  
Updated Oct 3  
Read the article

-   [#macmini2012](https://hashnode.com/tag/macmini2012)
-   [#local-llm](https://hashnode.com/tag/local-llm)
-   [#vector-database](https://hashnode.com/tag/vector-database)
-   [#koboldcpp](https://hashnode.com/tag/koboldcpp)
-   [#pgvector](https://hashnode.com/tag/pgvector)
-   [#optimisation](https://hashnode.com/tag/optimisation)
-   [#local-ai](https://hashnode.com/tag/local-ai)
-   [#local-ai-models](https://hashnode.com/tag/local-ai-models)

## Written by

[mamonu](https://hashnode.com/@mamonu)

@mamonu · London, UK

  
[View profile](https://hashnode.com/@mamonu)

quantised braiiins

## Responses 1

Markdown supported · ⌘↵ to post

[Yunus Emre Vurgun](https://hashnode.com/@yunusemrevurgun)

[Oct 20](https://hashnode.com/posts/using-a-mac-mini-2012-for-llm-inference-with-koboldcpp/6aa184fbdb8661a7a6a47a26/comment/6ad7280556de4f21ca8a7ad4)

The finding that architecture beat tuning is the useful one. A reason Ling-style MoE models fit this box so well is that decoding on an old CPU is limited by memory bandwidth, and a MoE only reads the active experts per token, so it moves far less data than a dense 7B. For the growing-context problem, one thing to try is a quantized KV cache only if your build supports it without the slowdown you saw, otherwise keeping the system prompt fixed lets prompt caching skip re-processing it each request. Also worth checking that the 2012 Mini isn't thermal throttling during long runs, since a sustained tokens/sec drop is easy to mistake for a context effect.