Splitting it into an ingestion workflow and a question-answering workflow is a good way to explain it, since most bugs end up living in the first one. A practical tip for the chunking step: store the source page number and section heading as metadata with each chunk, so the answer can cite where it came from and you can tell at a glance whether a bad answer came from retrieval or from the LLM. Which chunk size and overlap did you end up using for PDFs with tables?
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
The annual-leave example has a useful qualifier to carry into your evaluation set: the passage says eligible employees receive 20 days, while the question asks about employees generally. A fluent answer can quote the number correctly and still broaden who it applies to.
I would include a case where eligibility is unspecified and require the answer to retain that condition or ask for the missing detail. Pair it with a passage that explicitly defines the eligible group. That tests whether the generation step preserves the policy's scope, beyond finding a semantically similar paragraph and repeating its main number.