Putting the request after 7,000 tokens is a useful stress test, especially after training on 384-token inputs. I would vary its position next: beginning, middle, and end of the same filler, while keeping the request and allowed options unchanged.
The current setup always makes the final support request the relevant one. A second test could include a plausible but irrelevant ticket near the end while the real request sits in the middle. That would separate long-input tolerance from a preference for the last task-like passage. Reporting the twenty requests as paired outcomes would make those position effects visible without treating small percentage changes as stable rankings.
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
Putting the request after 7,000 tokens is a useful stress test, especially after training on 384-token inputs. I would vary its position next: beginning, middle, and end of the same filler, while keeping the request and allowed options unchanged.
The current setup always makes the final support request the relevant one. A second test could include a plausible but irrelevant ticket near the end while the real request sits in the middle. That would separate long-input tolerance from a preference for the last task-like passage. Reporting the twenty requests as paired outcomes would make those position effects visible without treating small percentage changes as stable rankings.