Deepak Goyal

Great question — and honestly, I've stopped trying to draw the line too cleanly up front. The heuristic I use: if I can write down what "correct" means as a rule — an exact arXiv ID, an author, a date range, a section reference — it's structured, and it belongs in a metadata filter or BM25, not an embedding. The moment "correct" becomes a judgment ("is this paper about the same idea?"), it's genuinely semantic and the model earns its place. The test I apply: can I write the assertion that would fail if the retrieval is wrong? If yes, it's deterministic residue. But in practice I don't pre-classify every token — that's exactly why hybrid (BM25 + dense, fused with RRF) works. The obvious anchors I filter deterministically; for everything else I run both and let fusion decide. The clean split is the exception, not the rule. I went deeper on the retrieval architecture over an ArXiv corpus here, if useful: https://blog.deepakgoyal.ai/building-production-rag-arxiv-part-1 And +1 on the silent-failure point — a wrong answer gets caught downstream; wrong context quietly becomes everyone's ground truth.

Thanks! Though calibration sits a step before the metrics — precision/F1 tell you whether a finding is correct, not whether "[blocker]" reads the same to a human as to the tool emitting it. A vocabulary problem, not a model one, and no score catches it.

Hi Adam, Honestly, it came from observation more than theory. After building SMRITI, skills were changing almost daily — each iteration adding a few lines here, a new step there. Token usage kept climbing because every invocation loads the full skill file. At the same time, skills that grew too fast became harder to reason about. We looked at where all the existing skills actually landed and 100–110 was the natural sweet spot — tight enough to keep costs predictable, generous enough that nothing needed an emergency rewrite. So we just drew the line there. Thanks Deepak