impl Iterator for Lexer<'_> {
type Item = (Token, Span);
fn next(&mut self) -> Option<Self::Item> {
- self.next_token();
+ self.emit_next_tokens();
+ self.token_queue.pop_front()
}
}
One question here
emit_next_tokens(); can push one or more tokens to the queue.
pop_front(); only consumes one. Does this not mean the queue will race ahead of the iteration?
I tried setting type Item = (Token, Span, u32 /Current Queue Size/) with a longer bit of code with a lot of indentation and it did race ahead. It's possible I missed something else elsewhere.