---
title: "Discussion on \"Designing the Ingestion Layer of a Production RAG System\" | Hashnode"
description: "Discussion on \"Designing the Ingestion Layer of a Production RAG System\". How a modular parsing, chunking, and embedding pipeline turns PDFs, HTML, DOCX, PPTX, and TXT files into queryable vectors in Qdrant  — and where it still breaks."
url: "https://hashnode.com/posts/designing-the-ingestion-layer-of-a-production-rag-system/6a6281ce6d3f3aafddb25818"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Designing the Ingestion Layer of a Production RAG System](https://adityabiranje210.hashnode.dev/designing-the-ingestion-layer-of-a-production-rag-system?utm_source=hashnode&utm_medium=feed)

How a modular parsing, chunking, and embedding pipeline turns PDFs, HTML, DOCX, PPTX, and TXT files into queryable vectors in Qdrant — and where it still breaks.

Jul 23 · 27 min read · adityabiranje210.hashnode.dev  
Read the article

-   [#rag](https://hashnode.com/tag/rag)
-   [#vector-databases](https://hashnode.com/tag/vector-databases)
-   [#qdrant](https://hashnode.com/tag/qdrant)
-   [#llm-engineering](https://hashnode.com/tag/llm-engineering)
-   [#llm](https://hashnode.com/tag/llm)
-   [#python](https://hashnode.com/tag/python)
-   [#vector-embeddings](https://hashnode.com/tag/vector-embeddings)
-   [#data-engineering](https://hashnode.com/tag/data-engineering)
-   [#observability](https://hashnode.com/tag/observability)

## Written by

[Aditya Biranje](https://hashnode.com/@AdityaBiranje)

@AdityaBiranje

  
[View profile](https://hashnode.com/@AdityaBiranje)

AI Researcher

## Responses 3

Markdown supported · ⌘↵ to post

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[Jul 25](https://hashnode.com/posts/designing-the-ingestion-layer-of-a-production-rag-system/6a6281ce6d3f3aafddb25818/comment/6a6538c21a8924530adff861)

The Gemini probe followed by an automatic all-mpnet-base-v2 fallback is resilient at the request level, but risky at the index level. If both providers can write into one collection, a dimension check catches only some mismatches; two models with equal dimensions would still create semantically incompatible vectors. I would pin one embedding fingerprint per ingestion run and collection, then queue failed batches for retry rather than silently changing the vector space mid-index.

[Aditya Biranje](https://hashnode.com/@AdityaBiranje)

[Jul 26](https://hashnode.com/posts/designing-the-ingestion-layer-of-a-production-rag-system/6a6281ce6d3f3aafddb25818/comment/6a6538c21a8924530adff861#reply-6a661be3c1680d1dbe8b73c2)

Good point! The fallback is decided once during initialization, so it doesn't switch providers mid-run. That said, pinning an embedding fingerprint to each collection and retrying failed batches is definitely a more production-ready approach. Thanks for the suggestion!

[Ahmet Özel](https://hashnode.com/@ahmetozel)

[Aug 5](https://hashnode.com/posts/designing-the-ingestion-layer-of-a-production-rag-system/6a6281ce6d3f3aafddb25818/comment/6a6538c21a8924530adff861#reply-6a7371f2821d07c352064c61)

Choosing the provider once at initialization removes the worst mid-run mixing risk. I would still record model ID, revision, dimension, normalization setting, and preprocessing version in the collection metadata, then reject later writers whose fingerprint differs. That makes a rebuild or migration intentional and gives retrieval traces enough information to explain why two index generations behave differently.