Splitting the benchmark by query type is the right move, because the aggregate comparison is what makes this question unanswerable - a corpus where five percent of traffic is multi-hop will show GraphRAG barely winning overall while being dramatically better on the queries people actually complain about. The decision rule should probably run on your query log rather than on the corpus. One cost that tends to be underweighted next to indexing: entity resolution never finishes. Company X, Company X Ltd and the subsidiary filing under another name have to collapse into one node or the graph answers confidently about the wrong entity, and every new batch of documents reopens that work. Indexing cost is a number you can estimate; resolution quality is an ongoing maintenance commitment. Incremental updates deserve a line in the analysis too - a graph over a corpus that changes daily either gets rebuilt or drifts, and a stale graph is more dangerous than stale chunks because it looks authoritative.
Splitting the benchmark by query type is the right move, because the aggregate comparison is what makes this question unanswerable - a corpus where five percent of traffic is multi-hop will show GraphRAG barely winning overall while being dramatically better on the queries people actually complain about. The decision rule should probably run on your query log rather than on the corpus. One cost that tends to be underweighted next to indexing: entity resolution never finishes. Company X, Company X Ltd and the subsidiary filing under another name have to collapse into one node or the graph answers confidently about the wrong entity, and every new batch of documents reopens that work. Indexing cost is a number you can estimate; resolution quality is an ongoing maintenance commitment. Incremental updates deserve a line in the analysis too - a graph over a corpus that changes daily either gets rebuilt or drifts, and a stale graph is more dangerous than stale chunks because it looks authoritative.