Passing down what we know. Discovering what's next.
Father of two, tech lover. Building systems by day, raising curious minds by night. I write hands-on guides and reproducible experiments on running, fine-tuning and evaluating AI locally on Apple Silicon, on AI agents, and on how AI changes software and finance. The code behind every experiment is on GitHub: github.com/hslimi/night-thoughts
Markdown supported · ⌘↵ to post
The break-even table is the part I'd push on. $18K/month only makes sense if that node sits at roughly full utilization the whole month, and most agent workloads aren't anywhere near that -- they spike during business hours and sit mostly idle overnight. Once you factor in real utilization, or swap a reserved node for a per-second serverless GPU provider instead, the token volume where self-hosting wins drops well below the 100-500M/day line here, sometimes by an order of magnitude. Utilization rate deserves its own column next to token volume, not just token volume on its own.
Ahmet Özel
"Open is not the same as free" is the right thesis, and starting from the token is the correct move since most cost confusion is really unit confusion.
The number that decides it in practice is utilisation. A rented GPU costs the same at 5% as at 95%, so self-hosting only wins above a fairly high floor of steady traffic, and bursty workloads pay for idle silicon overnight.
Two costs that rarely make these comparisons: the evaluation and rollback machinery you now own, since a serving change that regresses quality becomes your problem rather than a provider's, and the engineer-hours spent keeping it running, which at small scale are usually larger than the GPU bill.