10+ years in software engineering, with deep expertise in frontend. Now going deep on LLMs — streaming, RAG, tool use, and everything in between.
Senior engineer with deep roots in React, TypeScript, and production-scale UI — including 4 years at Apple. Now focused on applied AI engineering: working hands-on with LLM APIs, RAG pipelines, agents, and the full stack of tooling that modern AI products are built on. Writing here to document the build, share what's non-obvious, and connect with teams working on hard AI problems.
Markdown supported · ⌘↵ to post
Ken
Good distinction between system failure and eval failure. The next thing I’d want is to make the eval itself a versioned artifact.
Each run should carry the fixture version, case type, scorer version, scoring dimensions, judge configuration if an LLM judge is used, and a small human-audited slice. Otherwise the score can look precise while the thing being measured keeps changing.
That also makes bad evals easier to debug. You can ask whether the system regressed, the case set is unrepresentative, the scorer is applying the wrong criterion, or the production workload drifted away from the fixture set.