Good catch, that's usually where naive quantization breaks down while perplexity stays flat.
Yes, recall held at 100% Top-1 across all 21 depth positions (0% to 100%) up to 1M tokens on NVIDIA L4 GPUs in GCP. The trick is computing INT8 scales per 512-token page rather than globally, so token 0 attention sinks don't blow out the dynamic range of middle tokens, combined with an orthonormal SVD basis that keeps >99.8% of key variance.
You can run the sweep yourself with python scripts/stress_test_niah_breaking_point.py cuda in the repo or check the full depth tables in our benchmarks.
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
NIAH holding at 100% Top-1 after INT8 SVD KV paging is the number I would have doubted first, since quantizing the cache usually costs long-context recall. I have watched perplexity stay flat while retrieval accuracy quietly dropped, so gating on a needle test is right. Did recall hold across depth, or mostly near the prompt edges?