Discussion on "Deleting Personal Data from a RAG System in .NET"

Patrick Kearns · 2026-09-30T19:14:41.893Z

A customer asks you to remove their personal data. You delete their database record, remove the uploaded PDF and close the request. Later that afternoon, your AI assistant answers a question using a p

The distinction between an obsolete writer being cancelled and its external write actually settling is the important race here. A timed-out upsert can still complete at the provider after a verification lookup has reported absence.

I would add a test that delays the provider commit until after the first verification attempt, rather than only pausing the worker before the write. Completion evidence should identify the writer generation and settled pending-write set it was checked against. A timestamped negative lookup alone is a snapshot, not proof that no already-issued request can recreate the artefact.

You are right: settling outstanding writes and reconciling uncertain outcomes is already required in that section. My comment should have framed the delayed commit as a concrete test of your existing invariant, rather than suggesting the invariant was missing.

I would have that test assert the request remains pending after the first negative lookup, then completes only after the delayed provider result is reconciled and verification runs again. A separate case where the provider never settles would confirm that the deadline produces an unresolved outcome rather than false completion.

Yeah thats it, both are useful test cases. The delayed commit case should prove that the first negative lookup cant complete the deletion while a write remains outstanding. The 'never settles' case should prove that reaching the deadline leaves the outcome unresolved.

Those two outcomes should remain distinguishable in the completion evidence as well as the test assertions. The delayed-commit case can record the reconciled write result and the subsequent verification; the never-settles case should preserve the unresolved write identifier and generation.

A later reconciliation must then resume the pending request without treating the earlier deadline as evidence of absence. That gives the tests an observable state transition to assert, while keeping the existing settle-before-verification invariant explicit.

I like the distinction between “delete succeeded” and “data is actually gone.” Those are two different checks, especially once caches and old index versions get involved.

Yeah, people often forget about the Cache!

The resurrection tests are the most useful part of this. One addition worth running in production, not only in tests: a synthetic canary person. Every week, push a fake subject's documents through the normal ingestion path, file an erasure request for them, then check every store and the assistant's answers for their unique tokens a day later. It catches the new stores and ingestion paths nobody added to the inventory, which is the gap you flag as hardest to close.

The strongest point here is that erasure has to follow the data lineage, not just the original record. In a RAG system, the database row is only one representation; chunks, embeddings, caches, generated answers, logs, staged work, and even older index generations can all become independent persistence points.

One production detail I’d emphasize is treating the source revision as a fencing mechanism, not just metadata. In production RAG work at IT Path Solutions, this is the kind of boundary that prevents an already-running ingestion job from republishing an obsolete revision after deletion has started. Provenance tells you what to remove; controlled publication tells you it cannot quietly come back.

That distinction also makes verification much stronger: “the delete call succeeded” is an execution result, while “no covered revision can still be served or republished” is the actual system invariant.

The verification section is the strongest part of this for me -- treating "document not found" as the only thing that establishes logical absence, rather than trusting the delete call's own success response. One thing worth adding for Azure AI Search specifically: deletes propagate to replicas asynchronously, so a verification read issued right after the delete request can still land on a replica that hasn't caught up yet and come back with the document still present. If the verification step runs immediately after the delete call in the same task execution, that's a false "still there" -- harmless if it just triggers a retry, but it could get misread as "deletion failed, escalate" when the delete actually succeeded. Feels like it wants the same settle/retry treatment you already give the delete itself, applied to the verification read too.

Yeah good point, the verification read needs its own bounded retry window. I'd treat "still present" immediately after deletion as pending verification, then retry with backoff before escalating. If it remains present beyond that window, the erasure request stays open for investigation.

Latency maps close the UX. Audit still wants a tip that survives the vendor UI flip.

Curious which India DPI stack keeps that tip queryable without a second shadow ledger.

marker1002h0028

Not too sure for Indian DPI. Do you have access to Deleteceipt or similar?

The overnight-retry rewriting the deleted record is the part I see most teams miss: deletion gets treated as a single event, not something that has to hold against every async job that can re-ingest. Building lineage at ingest instead of similarity-searching the vector store for a name later is the right call, since that search never catches all the derived chunks. How do you handle a record that is mid-ingestion when the erasure request lands?

I think the ingestion would need to be Atomic really. I'd mark the subject as blocked before starting the purge, then have ingestion workers check that marker before processing and again when committing each derived write. Then the final check needs to be atomic with the write where possible otherwise, deletion can land between the check and the insert.

Search Hashnode

Deleting Personal Data from a RAG System in .NET

Responses15