Apr 13 · 17 min read · This is the twentieth post of a course about AI product work, and it is the last. If you have read the previous nineteen in order, you have spent roughly 70,000 words of reading building a set of frameworks, decision rules, and skeptic kits for shipp...
Join discussion
Apr 13 · 18 min read · An engineering team I worked with shipped an AI feature that looked great on every dashboard they cared about. Eval pass rate: 88%. Latency P99: 2.3 seconds. Cost per call: well under budget. Deployment was clean. The team felt good and moved on. Twe...
Join discussion
Apr 13 · 19 min read · You have shipped the AI feature. Stage 4 is complete. Usage is growing. Then the finance email lands on a Tuesday: the feature is costing you real money — call it $22,000 a month at current volume, or roughly 14% of the revenue from the customers usi...
Join discussion
Apr 13 · 21 min read · At some point, an AI feature you ship will do something in public that you wish it hadn't. It will hallucinate a customer's name into an apology email. It will recommend a competitor in a chatbot exchange that ends up on social media. It will misread...
Join discussion
Apr 13 · 20 min read · You ran the evals. You built the trust UX. You passed the compliance checklist. The feature works. Now comes the week every AI PM dreads: the actual rollout. There are two ways to ship an AI feature to 100% of users. One is "big bang" — flip a flag, ...
Join discussion
Apr 13 · 19 min read · You were winning the enterprise deal. The buyer loved the demo, the ROI math worked, and the contract was in review. Then their security team sent over a questionnaire with 120 items, most of them about AI specifically. Your team answered maybe 30 of...
Join discussion
Apr 13 · 19 min read · You have done the work. Your eval set hits 88%. Your P3 stack grades honestly. Your failure modes are under their ceilings. The feature is objectively good. You ship it. A week later, the PM dashboard shows a 47% adoption rate — half your users tried...
Join discussion
Apr 13 · 16 min read · Your engineer walks into the ship review with a slide. The slide has one big number on it: "87%." Below the number, in smaller text: "eval set pass rate, up from 82% last week." The room nods. Someone says "great work." The PM is about to say "OK shi...
Join discussion
Apr 13 · 18 min read · You have an eval set. You know your rubric. You want a pass rate. Now a new question appears: who decides whether each individual output is a pass? You, reading them one by one? A rule engine that checks for specific strings? Another model that grade...
Join discussion
Apr 13 · 17 min read · Here is a pattern I have watched break half a dozen AI projects. The PM writes the PRD. It includes a line that says "quality bar: 85% pass rate on eval set." Engineering builds the feature. Engineering builds the eval set — because the PM is busy an...
Join discussion
