Comment by Ravi Kasibhatla on "APEX in the AI era"

Hi Plamen,

Good Day and congrats for such an enriching blog.

Everything worked as your blog says (at least the very first time) and then onwards ngrok started giving troubles; may be because of the free account they might limit the traffic etc., but no issues.

I have created an Oracle Linux instance in the OCI Tenancy (because my APEX app is there) and completed the OLLAMA setup successfully. Checked that the ollama service is getting accessed through internet; checked that I could access the OLLMA service over internet using postman as well as curl ang got results correctly.

However, when tried to use the same URL (with the bare public IP that worked with cURL and Postman) APEX errored out saying taht the requested URL is prohibited.

I created an API Gateway in OCI and trying to pass through it to reach the OLLAM service, however the AP Gateway is throwing error.

If you are interested in probing and tweaking in to this architecture (accessing OLLAMA securely via OCI API Gateway) I am open to mutual colaboration.

Regards

Hi There. Don't bother...I got it working..!! Thanks for your post though, I got the idea of using open source local LLMs with APEX using my own OCI Instance and OCI API Gateways.

Regards

Hi Ravi, I'm glad it worked for you. As you have discovered, Ngrok's free accounts have some limitations - they use the default URLs which are too long. It also generates a new URL every time you re-start the service, so you need to re-configure the services using it.

But since you have Ollama on OCI and Ngrok is out of the equation, it's all good. Hope it performs well on the OCI and the cost is not too high.

Hi there

Thanks for your reply. For just a lab environment its doing "ok-ishh".

The VM instance I used is the smallest one (1 CPU + 16GB RAM) in OCI. After trying to pull more than one model with ollama, I found out that even the default disk size is also not enough and I had to increase that. :-)

During my testing phase there were some hiccups because I monitored that the ollama was thriving for both CPU and RAM and meanwhile the OCI API Gateway gets timed out.

So, for a real world scenario, one must go with a GPU based instance, a baremetal instance or at least an instance with considerable amout of resources so that the things work consistently.

My next adventure would be adding the RAG for this architecture...I might post the whole thing once I get things working.

Regards.

Ravi

Search Hashnode