
An inference server stopped generating tokens while more than 200 GiB of host memory remained available. Its health endpoint kept returning HTTP 200. The first visible exception was queue.Full inside
Lead developer at Neuroprem | LLM systems & inference
Software engineer and lead developer at Neuroprem, working on AI systems and LLM infrastructure. I write about quantization, distillation, efficient inference, retrieval and memory for agentic systems: technical notes, benchmarks, architecture decisions and lessons from failed experiments. Currently preparing a custom quantized GLiNER model for release and experimenting with Qwen inference and distillation. Less hype. More engineering. Engineering log: https://t.me/badbat4560 Models: https://huggingface.co/badbat4560 Code: https://github.com/badbat4560 DEV: https://dev.to/badbat4560 Neuroprem: https://neuroprem.com Instagram: https://www.instagram.com/badbat4560/
Markdown supported · ⌘↵ to post
No responses yet.