Discussion on "RAG vs Long Context in .NET" | Hashnode

Discussion on "RAG vs Long Context in .NET" | Hashnode

Responses5

Markdown supported · ⌘↵ to post

Ahmet Özel

EvidencePacker re-counting the assembled messages on each trial is a defensible correctness baseline because metadata and framing consume tokens too. With many candidate sections, however, it repeatedly tokenizes the same growing prefix, so packing itself can become measurable work before the model call.

I would record packing latency alongside retrieval and generation, particularly for parent-expanded evidence. An incremental counter could reduce repeated work, but only after testing it against the complete serialized request; summing isolated text counts can miss boundary and framing effects. Keeping the existing full-count path as an oracle would make that optimization safer.

Patrick Kearns

Yeah, I’d expect packing latency to be measured alongside retrieval and generation when taking this into production. The full count approach gives the example a straightforward correctness baseline, if profiling shows it's becoming expensive, an incremental counter is a sensible optimisation, validated against that existing path.

Keeping the example's full-count path as the baseline makes sense. For an incremental implementation, I would build the parity fixtures around boundary cases: an empty section, a source label containing non-ASCII text, a section admitted exactly at the budget, and one rejected just above it.

Assert the selected section IDs as well as the final count. Two counters can disagree only slightly yet choose different evidence at the boundary, which changes the model input even when both totals look reasonable. Profiling can then decide whether the extra implementation is justified.

Evelyn Wyatt

The distinction between finding relevant evidence and preserving sufficient evidence for reasoning is an important one here. A retrieval system can return highly relevant chunks and still fail if the answer depends on a definition, exception, or relationship that wasn't included in the final evidence bundle. I’d be interested in evaluating “evidence completeness” alongside retrieval precision and recall: not just whether the expected passage was retrieved, but whether all dependencies required to justify the answer survived selection and packing. That metric could also make the RAG-versus-long-context decision more concrete. If a task repeatedly requires broad evidence relationships, long context or structured parent expansion may be the better baseline even when top-k retrieval looks strong.