Kevin Naidoo

Thank you! Sounds like a good idea. I will definitely think about incorporating something like this.

Great, thank you for the suggestion; I will definitely look into this. Currently, there's also a console error log that will print errors if there's a syntax problem or if something is not implemented yet.

Hi John, Happy 2025, hope you have an awesome year ahead :-) Thanks for reading, and glad this was useful to you. Twilio is not a SIP client, they provide a managed VOIP solution, so basically, you can buy telephone numbers from them and programmatically connect code to a real-time call. With the media streams, they fork the audio from the real-time call and stream it to your WebSocket server. The WebSocket server then receives the audio and you can transcribe or pass that on to a real-time model and then send the audio back to the client. You can achieve the same by setting up Asterisk, but that is a bit more complicated and a pain to manage. One problem with Twilio is that the audio is streamed at 8Khz and is base64 encoded so there is a conversion step that might slow down your call. Here's some documentation on the Twilio approach: https://www.twilio.com/docs/voice/tutorials/consume-real-time-media-stream-using-websockets-python-and-flask#create-a-socket-decorator A better approach would be WebRTC(I built a simple client here: https://github.com/kevincoder-co-za/zazu-voiceai) or just build a SIP WebSocket server and use a provider like 3cx. I did play with a few libraries, before going with Twilio (it was an MVP so shipping fast was essential). Maybe they'll be of use: https://github.com/sipsorcery-org/sipsorcery https://www.mizu-voip.com/Software/SIPSDK/JavaSIPSDK.aspx https://www.pjsip.org/ https://github.com/emiago/sipgo https://www.linphone.org/en/voip-unified-communications-software Mizu worked nicely, I just didn't have much time to fully implement the SDK and it's a commercial product so there are licensing costs. Hope this helps.

I see. Yeah PyVOIP is more for educational purposes, its not efficient for production. You probably should use pjsua/pjsip or twilio media streams for your use case.

Hi Zaid, thanks for the question. Yes, you can but you need to format the audio into a WAV and ensure that the sample rate is 8000 Hz(mono channel). Then you can send that WAV file back to the user: audio = AudioSegment.from_mp3("your_file.mp3") audio = audio.set_frame_rate(8000).set_channels(1) audio.export("output.wav", format="wav") f = wave.open('output.wav', 'rb') frames = f.getnframes() data = f.readframes(frames) f.close() call.write_audio(data) I suggest looking into my web socket tutorial over at: https://kevincoder.co.za/how-i-used-voice-ai-to-bring-imaginary-characters-to-life You don't need to use WebRTC if you need the phone system, you can use Twilio media streams or forward the call from a PBX server like Asterisk. Hope this helps.

Awesome :-) PHP is amazing and efficient so it is always a good language to learn. All the best!

Hi Elon Musk, Faiss is in memory, it's efficient but for large applications, you would want to use replication to split up read/writes and distribute traffic better. This is why Qdrant is a better fit, furthermore, when you restart your application instance FAISS loses its memory, you can write the buffer to disk and reindex on the next startup but as you can imagine this is not very efficient. You can learn more about how to distribute Qdrant over many servers here: https://qdrant.tech/documentation/guides/distributed_deployment/ Qdrant can handle multiple concurrent writes just fine like any other DB, so updating your RAG data in real-time should not be a problem, as mentioned above you can always split the traffic with distributed deployments. As for multiple calls to GPT-3.5-Turbo, since the heavy lifting is done on the OpenAI side, you should not be using that much of GPU resources if any, so the only limitation would be your web server or application server and the number of requests it can handle per second. Try: https://github.com/tsenart/vegeta to simulate high traffic and see how your machines respond. All the best, hope this helps.

Elon Musk For chat, generally, web sockets make the most sense. PHP does have web socket support via: https://openswoole.com/ Open Swoole might introduce more complexity than you need, depending on how many connections you want to handle per second/minute, might just be better to use regular AJAX and HTTP requests. PHP 8 with PHP-FPM can handle these fairly efficiently, or in the Python world, FastAPI is a good choice. FYI: I do have a few tutorials on this blog relating to FastAPI and also using Qdrant as a vector store instead of FAISS. FAISS may not work so well for more advanced realtime chatbots.

Hi Elon Musk . RAG is essentially taking text data and vectorizing the data so that it can be searched. In the case of a website, you need to scrape the text of the website and then follow the RAG process as described in this article. You don't need to always use RAG, if you use Langchain tools for example, you can easily just query an API or scrape a page and then provide that data to the model in real-time. RAG helps to narrow down large chunks of text. Remember you pay for each token and models usually have a limitation on the maximum number of tokens you can send in a prompt, therefore using a RAG system - you minimize the amount of tokens sent. OpenAI uses a token-based costing system. A token is roughly 4 characters (in English). You get billed a different rate for text fed into the model and text generated from the model. GPT3.5-Turbo is generally the most cost-effective model, it's not the best model but for most use cases it's usually fine. With GPT3.5 Turbo expect to pay $0.50 / 1M tokens for text you pass into the model, and $1.50 / 1M tokens the model generates.

Ashish Agarwal Cool yeah Flask is very lean for an API, I stopped using it at some point and moved to FastAPI though for machine learning stuff.