Great point! This was strictly for demonstration purposes, but I did collaborate with a real, qualified lawyer to develop this use case. Of course, the data used was public, and we went through several iterations on the prompts before settling on one that was both simple and effective for demonstration. As next steps, I would run some automated evaluations on my query tool using a set of prompts to ensure that relevant documents are being retrieved. I would likely manually generate about 100 ground truth question and answer pairs and aim for an accuracy of over 80% (a personal, somewhat arbitrary goal). Additionally, I would test the efficacy of the Agent framework in CrewAI to ensure that the prompts and multi-step reasoning do not have any failure points. If issues are found, I would iterate on the prompts until I am highly satisfied with the performance on my test cases.
@Farzzy528
About
Principal Product Manager at Microsoft CoreAI leading Retrieval-Augmented Generation (RAG) and knowledge retrieval capabilities, powering enterprise AI applications with grounded, production-scale search and retrieval.
Available for
Nothing here yet.