The point that lands for me is that an AI request can return a clean 200 and still fail the product, which is exactly why HTTP-level monitoring misses the failures that matter. Invalid JSON or a RAG answer that ignores the retrieved context are workflow-level failures, so the signal has to come from evaluating the output, not the status code. Across that many models, are you scoring output quality inline per request, or sampling and judging asynchronously so the monitoring itself does not add latency to the hot path?
The point that lands for me is that an AI request can return a clean 200 and still fail the product, which is exactly why HTTP-level monitoring misses the failures that matter. Invalid JSON or a RAG answer that ignores the retrieved context are workflow-level failures, so the signal has to come from evaluating the output, not the status code. Across that many models, are you scoring output quality inline per request, or sampling and judging asynchronously so the monitoring itself does not add latency to the hot path?