Going from 8128 to 775 output tokens by having the model emit stable block IDs instead of re-serializing the whole document is a genuinely clever reframing of the cost problem. Borrowing the dirty-path tracking idea from Slate's WeakMap is the part I didn't expect. How stable do the block IDs stay when the document is restructured heavily, like sections being merged or split?
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
Going from 8128 to 775 output tokens by having the model emit stable block IDs instead of re-serializing the whole document is a genuinely clever reframing of the cost problem. Borrowing the dirty-path tracking idea from Slate's WeakMap is the part I didn't expect. How stable do the block IDs stay when the document is restructured heavily, like sections being merged or split?