Tag feed

#inference

100 posts0 followers

Explore Hashnode

Alternatives

Trending tags this week

WSwesley schollkonjo.hashnode.dev9h ago · 21 min read

I Couldn't Find a Local LLM Tool Fast Enough, So I Built My Own Called Squish

This requires Apple Silicon (M-series) and macOS 13 (Ventura) or later. If you're on Intel, Linux, or Windows, the numbers in this article will not apply to your hardware. TL;DR: Squish is a local LLM

0

YKYash Karechayash-karecha.hashnode.devJul 5 · 21 min read

I Built My Own ChatGPT Backend to Understand How AI Streaming Actually Works

Every time I use Claude or ChatGPT I get stuck on the same question: how does the text just show up, word by word, like the model is thinking out loud in real time? Why does it feel instant when it's

0

RCRobert Collinsselfenrichment.hashnode.devJul 1 · 8 min read

dgx-spark-inference - Keeping good habits around local inference

Local inference has a habit of becoming folklore. You launch a model from a shell history fragment you can never quite find again. A context setting lives in a note somewhere. A faster quantization ge

0

RRunwarerunware.hashnode.devJun 26 · 15 min read

The closed-source LLM premium has collapsed

Originally published on the Runware blog. Written by Ioana Hreninciuc, Co-Founder at Runware. Open-source models now match closed-model benchmarks at 87% lower cost. Here's what that means for develop

1

M

KNKunal Nayyarkunal221.hashnode.devJun 20 · 14 min read

The Great AI Brick Wall: How We Are Shattering the Memory Barrier

In the digital world, we often take the speed of our tools for granted. We trust our computers to compute instantly, and we expect immediate gratification. But if you have ever watched a Large Languag

0

RVRitikesh Valiritikeshvali.hashnode.devMay 30 · 2 min read

ml systems explained by someone who came from backend engineering, not ml

i've spent the last few years building distributed systems on azure, using services like key vault, blob storage, service bus, fabric, data factory, functions. outside of work i've been reading up on

0

KEKarim El Jamalijamali.hashnode.devMay 17 · 10 min read

Gateway API Inference Extensions: An AI Aware Load Balancer

Introduction If you’ve been anywhere near the Kubernetes ecosystem lately, you’ve probably heard the buzz around Gateway API — and the ongoing “Ingress is going away” narrative. I’ve covered Gateway A

0

RBRahul Boneyrahulboney.hashnode.devMay 14 · 4 min read

The Hidden Cost of "Free" GPUs: Upgrading a Duct-Taped AI Architecture

When you are bootstrapping an AI-heavy microservice architecture from scratch, you quickly learn one undeniable truth,Cloud GPUs are expensive. While building AerialVision,a real-time smart city traff

0

JKJangwook Kimeffloow.hashnode.devMay 11 · 5 min read

ReFlect: Training-Free Error Recovery for Long-Horizon LLM Reasoning

Long-horizon reasoning is where production LLM agents tend to quietly break. A model can produce a plausible-looking chain of thought, accept a wrong intermediate answer, and continue building on that error for every step that follows. By the time th...

0

JKJangwook Kimeffloow.hashnode.devMay 11 · 5 min read

PARSE: Faster LLM Inference via Parallel Prefix Speculative Decoding

Speculative decoding became the standard inference speedup technique through 2024 and 2025. The idea: a small draft model generates a sequence of candidate tokens, and a larger target model verifies them in parallel — accepting the longest valid pref...

0

#inference

Search Hashnode

#inference

Explore Hashnode

Trending tags this week

I Couldn't Find a Local LLM Tool Fast Enough, So I Built My Own Called Squish

I Built My Own ChatGPT Backend to Understand How AI Streaming Actually Works

dgx-spark-inference - Keeping good habits around local inference

The closed-source LLM premium has collapsed

The Great AI Brick Wall: How We Are Shattering the Memory Barrier

ml systems explained by someone who came from backend engineering, not ml

Gateway API Inference Extensions: An AI Aware Load Balancer

The Hidden Cost of "Free" GPUs: Upgrading a Duct-Taped AI Architecture

ReFlect: Training-Free Error Recovery for Long-Horizon LLM Reasoning

PARSE: Faster LLM Inference via Parallel Prefix Speculative Decoding