Thanks for your message. You are right that off-hour ingestion is more of a blanket statement, as it really depends on the usage pattern, the type of data, and the complexity of the ingestion pipeline. It's more of an art than a rule, and I've not seen AWS reference that provides substantial guidance yet.
So far I've worked on a few RAG system (not exclusively on AWS) with relatively static data which can afford scheduled ingestion. But I've not had the opportunity to work on one with real-time ingestion requirements just yet.
I know of the Bedrock KB streaming data ingestion feature that was announced in re:Invent 2024, but have not had a chance to experiment with it. Would be interesting to see how this changes things.
Sterling Blue
Thanks for the article - great solution! Appreciate the caution, but holding off ingestion 'during high usage periods' might not always be needed. Scaling can handle real-time updates, so it’s not a one-size-fits-all rule. Any AWS reference backing that recommendation?