@ajitabh
About
Real-world engineering insights from 20+ years building scalable systems.
Focused on AI, RAG architectures, and production-ready system design.
Available for
Nothing here yet.
Ajitabh Singh's blogs
Comments
AS
Absolutely agree. I don't think chunk size should be tuned by eye. My recommendation is to align chunks with the document's natural structure first, then compare strategies using retrieval metrics like Recall@K, Precision@K, and MRR on a representative query set. The key point for me is that chunk size is a workload-dependent decision, not a universal number.
AS
Andrew Fair points, but I think these examples actually reinforce the main point of the article. I’m not arguing that hierarchical or overlap-based chunking is universally better. The article explicitly says each strategy has trade-offs and that the right choice depends on the document and retrieval workload. The hierarchical example is a valid implementation failure, but not necessarily a failure of hierarchical chunking itself. Likewise, I explicitly call out that overlap reduces boundary impact but doesn't eliminate semantic splits. I agree that retrieval should be evaluated independently before generation. Ultimately, the goal is simple: test different strategies on representative queries and choose the one that gives the best retrieval quality for the document type, while considering cost and complexity.