Prompt Caching: Your Prompt's Prefix Is a Cache Key

Look at the input token count on a typical LLM feature and most of it is the same on every request. The system prompt, the tool definitions, the twelve few-shot examples, the product docs you stuffed