IG
Good catch, that's usually where naive quantization breaks down while perplexity stays flat. Yes, recall held at 100% Top-1 across all 21 depth positions (0% to 100%) up to 1M tokens on NVIDIA L4 GPUs in GCP. The trick is computing INT8 scales per 512-token page rather than globally, so token 0 attention sinks don't blow out the dynamic range of middle tokens, combined with an orthonormal SVD basis that keeps >99.8% of key variance. You can run the sweep yourself with python scripts/stress_test_niah_breaking_point.py cuda in the repo or check the full depth tables in our benchmarks.