54 likes
·
5.8K reads
10 comments
Will this work on a mac, given Office365's mac shortcomings (in spite of the fact that mac O365 users pay the same price... yeah, I'm bitter.)?
Yep it will work on any machine as it's all cloud services!
what version of llamaindex are you using? I installed the latest version and it complains about defining the engine parameter missing.
I'm using Version: 0.11.2
I want to know what version of llama index are you using?
I'm using Version: 0.11.2
Did you benchmark this? I understand it gives in-depth analysis, but how much of it is factually correct and how much is not? Do you suggest any steps? (other than asking people to review it?)
Great point! This was strictly for demonstration purposes, but I did collaborate with a real, qualified lawyer to develop this use case. Of course, the data used was public, and we went through several iterations on the prompts before settling on one that was both simple and effective for demonstration.
As next steps, I would run some automated evaluations on my query tool using a set of prompts to ensure that relevant documents are being retrieved. I would likely manually generate about 100 ground truth question and answer pairs and aim for an accuracy of over 80% (a personal, somewhat arbitrary goal).
Additionally, I would test the efficacy of the Agent framework in CrewAI to ensure that the prompts and multi-step reasoning do not have any failure points. If issues are found, I would iterate on the prompts until I am highly satisfied with the performance on my test cases.