--- title: "Discussion on \"Deleting Personal Data from a RAG System in .NET\" | Hashnode" description: "Discussion on \"Deleting Personal Data from a RAG System in .NET\". Following an erasure request through documents, chunks, vectors, caches and background jobs" url: "https://hashnode.com/posts/deleting-personal-data-from-a-rag-system-in-net/6abd5fa1c02e2338b2100943" --- [Feed](https://hashnode.com/feed) / Discussion # [Deleting Personal Data from a RAG System in .NET](https://dotnetdigest.com/deleting-personal-data-from-a-rag-system-in-net?utm_source=hashnode&utm_medium=feed) Following an erasure request through documents, chunks, vectors, caches and background jobs Sep 30 · 21 min read · dotnetdigest.com Read the article - [#ai](https://hashnode.com/tag/ai) - [#rag](https://hashnode.com/tag/rag) - [#artificial-intelligence](https://hashnode.com/tag/artificial-intelligence) - [#llm](https://hashnode.com/tag/llm) - [#vector-database](https://hashnode.com/tag/vector-database) - [#data-science](https://hashnode.com/tag/data-science) - [#microsoft](https://hashnode.com/tag/microsoft) - [#dotnet](https://hashnode.com/tag/dotnet) - [#csharp](https://hashnode.com/tag/csharp) - [#programming-blogs](https://hashnode.com/tag/programming-blogs) - [#software-developer](https://hashnode.com/tag/software-developer) - [#web-development](https://hashnode.com/tag/web-development) - [#software-engineering](https://hashnode.com/tag/software-engineering) - [#gdpr](https://hashnode.com/tag/gdpr) ## Written by [Patrick Kearns](https://hashnode.com/@fullstackcity) @fullstackcity · Dublin [View profile](https://hashnode.com/@fullstackcity) Senior .Net Engineer Senior Software Engineer specialising in cloud architecture, distributed systems, and modern .NET development, with over two decades of experience designing and delivering enterprise platforms in financial, insurance, and high-scale commercial environments. My focus is on building systems that are reliable, scalable, and maintainable over the long term. I’ve led modernisation initiatives moving legacy platforms to cloud-native Azure architectures, designed high-throughput streaming solutions to eliminate performance bottlenecks, and implemented secure microservices environments using container-based deployment models and event-driven integration patterns. From an architecture perspective, I have strong practical experience applying approaches such as Vertical Slice Architecture, Domain-Driven Design, Clean Architecture, and Hexagonal Architecture. I’m particularly interested in modular system design that balances delivery speed with long-term sustainability, and I enjoy solving complex problems involving distributed workflows, performance optimisation, and system reliability. I enjoy mentoring engineers, contributing to architectural decisions, and helping teams simplify complex systems into clear, maintainable designs. I’m always open to connecting with other engineers, architects, and technology leaders working on modern cloud and distributed system challenges. ## Responses 15 Markdown supported · ⌘↵ to post [Ahmet Özel](https://hashnode.com/@ahmetozel) [Oct 4](https://hashnode.com/posts/deleting-personal-data-from-a-rag-system-in-net/6abd5fa1c02e2338b2100943/comment/6ac2925502c54c62417e061f) The distinction between an obsolete writer being cancelled and its external write actually settling is the important race here. A timed-out upsert can still complete at the provider after a verification lookup has reported absence. I would add a test that delays the provider commit until after the first verification attempt, rather than only pausing the worker before the write. Completion evidence should identify the writer generation and settled pending-write set it was checked against. A timestamped negative lookup alone is a snapshot, not proof that no already-issued request can recreate the artefact. [Ahmet Özel](https://hashnode.com/@ahmetozel) [Oct 4](https://hashnode.com/posts/deleting-personal-data-from-a-rag-system-in-net/6abd5fa1c02e2338b2100943/comment/6ac2925502c54c62417e061f#reply-6ac29a2d7a597812ba13a4bf) You are right: settling outstanding writes and reconciling uncertain outcomes is already required in that section. My comment should have framed the delayed commit as a concrete test of your existing invariant, rather than suggesting the invariant was missing. I would have that test assert the request remains pending after the first negative lookup, then completes only after the delayed provider result is reconciled and verification runs again. A separate case where the provider never settles would confirm that the deadline produces an unresolved outcome rather than false completion. [Patrick Kearns](https://hashnode.com/@fullstackcity) [Oct 6](https://hashnode.com/posts/deleting-personal-data-from-a-rag-system-in-net/6abd5fa1c02e2338b2100943/comment/6ac2925502c54c62417e061f#reply-6ac53e950b369cf642d7463c) Yeah thats it, both are useful test cases. The delayed commit case should prove that the first negative lookup cant complete the deletion while a write remains outstanding. The 'never settles' case should prove that reaching the deadline leaves the outcome unresolved. [Ahmet Özel](https://hashnode.com/@ahmetozel) [Oct 7](https://hashnode.com/posts/deleting-personal-data-from-a-rag-system-in-net/6abd5fa1c02e2338b2100943/comment/6ac2925502c54c62417e061f#reply-6ac6558b6e54bc893ba94e28) Those two outcomes should remain distinguishable in the completion evidence as well as the test assertions. The delayed-commit case can record the reconciled write result and the subsequent verification; the never-settles case should preserve the unresolved write identifier and generation. A later reconciliation must then resume the pending request without treating the earlier deadline as evidence of absence. That gives the tests an observable state transition to assert, while keeping the existing settle-before-verification invariant explicit. [Julian Neagu](https://hashnode.com/@julianneagulab) [Oct 3](https://hashnode.com/posts/deleting-personal-data-from-a-rag-system-in-net/6abd5fa1c02e2338b2100943/comment/6ac0d79cade4fd09074f671b) I like the distinction between “delete succeeded” and “data is actually gone.” Those are two different checks, especially once caches and old index versions get involved. [Patrick Kearns](https://hashnode.com/@fullstackcity) [Oct 4](https://hashnode.com/posts/deleting-personal-data-from-a-rag-system-in-net/6abd5fa1c02e2338b2100943/comment/6ac0d79cade4fd09074f671b#reply-6ac296ea1e2f238c6fb4006d) Yeah, people often forget about the Cache! [X1Y1](https://hashnode.com/@x1y1) [Oct 3](https://hashnode.com/posts/deleting-personal-data-from-a-rag-system-in-net/6abd5fa1c02e2338b2100943/comment/6ac087ce524476e267523f50) The resurrection tests are the most useful part of this. One addition worth running in production, not only in tests: a synthetic canary person. Every week, push a fake subject's documents through the normal ingestion path, file an erasure request for them, then check every store and the assistant's answers for their unique tokens a day later. It catches the new stores and ingestion paths nobody added to the inventory, which is the gap you flag as hardest to close. [Mateo Ruiz](https://hashnode.com/@mateoruiz) [Oct 2](https://hashnode.com/posts/deleting-personal-data-from-a-rag-system-in-net/6abd5fa1c02e2338b2100943/comment/6abf7842ba66d9a05f68d801) The strongest point here is that erasure has to follow the data lineage, not just the original record. In a RAG system, the database row is only one representation; chunks, embeddings, caches, generated answers, logs, staged work, and even older index generations can all become independent persistence points. One production detail I’d emphasize is treating the source revision as a fencing mechanism, not just metadata. In production RAG work at IT Path Solutions, this is the kind of boundary that prevents an already-running ingestion job from republishing an obsolete revision after deletion has started. Provenance tells you what to remove; controlled publication tells you it cannot quietly come back. That distinction also makes verification much stronger: “the delete call succeeded” is an execution result, while “no covered revision can still be served or republished” is the actual system invariant. [Mihai\_LeanZero](https://hashnode.com/@mihaileanzero) [Oct 2](https://hashnode.com/posts/deleting-personal-data-from-a-rag-system-in-net/6abd5fa1c02e2338b2100943/comment/6abf507dc548f1f021941369) The verification section is the strongest part of this for me -- treating "document not found" as the only thing that establishes logical absence, rather than trusting the delete call's own success response. One thing worth adding for Azure AI Search specifically: deletes propagate to replicas asynchronously, so a verification read issued right after the delete request can still land on a replica that hasn't caught up yet and come back with the document still present. If the verification step runs immediately after the delete call in the same task execution, that's a false "still there" -- harmless if it just triggers a retry, but it could get misread as "deletion failed, escalate" when the delete actually succeeded. Feels like it wants the same settle/retry treatment you already give the delete itself, applied to the verification read too. [Patrick Kearns](https://hashnode.com/@fullstackcity) [Oct 2](https://hashnode.com/posts/deleting-personal-data-from-a-rag-system-in-net/6abd5fa1c02e2338b2100943/comment/6abf507dc548f1f021941369#reply-6abf5239632f5c0e51e90138) Yeah good point, the verification read needs its own bounded retry window. I'd treat "still present" immediately after deletion as pending verification, then retry with backoff before escalating. If it remains present beyond that window, the erasure request stays open for investigation. [indiainfranotes](https://hashnode.com/@indiainfranotes) [Oct 1](https://hashnode.com/posts/deleting-personal-data-from-a-rag-system-in-net/6abd5fa1c02e2338b2100943/comment/6abeb00ec729cf5044e3195c) Latency maps close the UX. Audit still wants a tip that survives the vendor UI flip. Curious which India DPI stack keeps that tip queryable without a second shadow ledger. marker1002h0028 [Patrick Kearns](https://hashnode.com/@fullstackcity) [Oct 2](https://hashnode.com/posts/deleting-personal-data-from-a-rag-system-in-net/6abd5fa1c02e2338b2100943/comment/6abeb00ec729cf5044e3195c#reply-6abf52df6d92aaf518540ddc) Not too sure for Indian DPI. Do you have access to Deleteceipt or similar? [Kartik N V J K](https://hashnode.com/@kartik-nvj) [Oct 1](https://hashnode.com/posts/deleting-personal-data-from-a-rag-system-in-net/6abd5fa1c02e2338b2100943/comment/6abead08656b10339d06ec14) The overnight-retry rewriting the deleted record is the part I see most teams miss: deletion gets treated as a single event, not something that has to hold against every async job that can re-ingest. Building lineage at ingest instead of similarity-searching the vector store for a name later is the right call, since that search never catches all the derived chunks. How do you handle a record that is mid-ingestion when the erasure request lands? [Patrick Kearns](https://hashnode.com/@fullstackcity) [Oct 2](https://hashnode.com/posts/deleting-personal-data-from-a-rag-system-in-net/6abd5fa1c02e2338b2100943/comment/6abead08656b10339d06ec14#reply-6abf51b499ee0b1509bbfc38) I think the ingestion would need to be Atomic really. I'd mark the subject as blocked before starting the purge, then have ingestion workers check that marker before processing and again when committing each derived write. Then the final check needs to be atomic with the write where possible otherwise, deletion can land between the check and the insert.