Really clean writeup of the bi-encoder/cross-encoder split -- the "cost is linear in corpus size per query, forever" framing is the right way to explain why nobody just runs a cross-encoder over everything.

One thing I didn't see covered: what happens when the reranker call itself is slow or unavailable in production. Per your own diagram it's a synchronous stage sitting between retrieval and the LLM call, with its own latency and failure modes (loaded GPU, cold model server, network blip). Falling back to raw cosine order on a timeout keeps the request alive but silently degrades quality for however long the reranker is down; failing the request instead keeps quality guarantees but turns a reranker hiccup into a user-facing outage. Worth picking one of those on purpose before the first timeout shows up in prod rather than after.