LLM inference runs in two phases: prefill reads the prompt, decode writes the answer. What to measure, cache and change for faster, cheaper inference.
Founder of BigCircle (https://bigcircle.ai). We build production AI systems with governance built in.
Founder of BigCircle (https://bigcircle.ai). We build production AI systems with governance built in.
Markdown supported · ⌘↵ to post
No responses yet.