Passing down what we know. Discovering what's next.
Father of two, tech lover. Building systems by day, raising curious minds by night. I write hands-on guides and reproducible experiments on running, fine-tuning and evaluating AI locally on Apple Silicon, on AI agents, and on how AI changes software and finance. The code behind every experiment is on GitHub: github.com/hslimi/night-thoughts
Markdown supported · ⌘↵ to post
Ahmet Özel
The 35 GB INT4 example is a useful starting point, but the usable context length deserves its own memory budget. The compressed weights can fit while a longer conversation still runs out of memory because the attention cache and runtime buffers also occupy the same pool. That makes 'the model loads' a weaker test than 'the intended conversation runs.'
For the laptop reality check, I would record memory and response speed at several prompt lengths with the same offload settings. Leaving headroom for the operating system also matters on unified-memory machines. It would make the hardware examples easier to translate into an actual local workload.