The finding that architecture beat tuning is the useful one. A reason Ling-style MoE models fit this box so well is that decoding on an old CPU is limited by memory bandwidth, and a MoE only reads the active experts per token, so it moves far less data than a dense 7B. For the growing-context problem, one thing to try is a quantized KV cache only if your build supports it without the slowdown you saw, otherwise keeping the system prompt fixed lets prompt caching skip re-processing it each request. Also worth checking that the 2012 Mini isn't thermal throttling during long runs, since a sustained tokens/sec drop is easy to mistake for a context effect.
Yunus Emre Vurgun
Writing about AI agents, hardware, and practical software.
The finding that architecture beat tuning is the useful one. A reason Ling-style MoE models fit this box so well is that decoding on an old CPU is limited by memory bandwidth, and a MoE only reads the active experts per token, so it moves far less data than a dense 7B. For the growing-context problem, one thing to try is a quantized KV cache only if your build supports it without the slowdown you saw, otherwise keeping the system prompt fixed lets prompt caching skip re-processing it each request. Also worth checking that the 2012 Mini isn't thermal throttling during long runs, since a sustained tokens/sec drop is easy to mistake for a context effect.