Yes, we did use load balancing, in our case the biggest problems we faced were the context length of LLM and Rate Limits. We solved the problem by having multiple keys and equally distributing the queries between them using multithreading, as for the context length we just shifted to the K-means summarization technique for long documents.
Enrique Cardoza Thanks for the compliment , if you like our work feel free sharing . If you want to read the nitty-gritty details all the code is open source and you can visit and read it .