Robert Collinsselfenrichment.hashnode.dev·Oct 28, 2024On KL-Divergence and Context Size Optimization in GGUF Quantizationnote: This paper was written with AI, and the exploration it describes was done collaboratively with AI. The "we" described here is us; me and a few models. With llama.cpp model quantization, properly adjusting models to keep their performance after ...46 readsgguf
Amos Gyamfiamosgyamfi.hashnode.dev·Aug 29, 2024The 6 Best LLM Tools To Run Models LocallyRunning large language models (LLMs) like ChatGPT and Claude usually involves sending data to servers managed by OpenAI and other AI model providers. While these services are secure, some businesses prefer to keep their data entirely offline for grea...jan
Shivang Agarwalshivangagarwal.hashnode.dev·Feb 29, 2024How to Run LLMs LocallyIntroduction As the Open Source LLMs evolve, the power of Generative AI is within reach of all. Against the common belief that you need mighty machines or graphic cards to use these models, many techniques have evolved that allow you to run these mod...65 readsLLM's
Akriti Upadhyayakritiu.hashnode.dev·Feb 2, 2024Integrating LlamaIndex and Qdrant Similarity Search for Patient Record RetrievalIntroduction The medical field is currently experiencing a remarkable surge in data, a result of the progress in medical technologies, digital health records (EHR), and wearable health devices. The ability to effectively manage and analyze this intri...RAG
Lars Grammelp42.hashnode.dev·Jan 13, 2024Create Your Own Local Chatbot with Next.js, Llama.cpp, and ModelFusionIn this blog post, we'll build a Next.js chatbot that runs on your computer. We'll use Llama.cpp to serve the OpenHermes 2.5 Mistral LLM (large language model) locally, the Vercel AI SDK to handle stream forwarding and rendering, and ModelFusion to i...6 likes·971 readsJavaScript
Kaushal Powarwrittenbykaushal.hashnode.dev·Jan 4, 2024How to convert HF (safetensors) 🤗 model to ggufYou want to convert Huggingface model to gguf format?I was struggling to tackle the same problem a few days ago. I finetuned a Llama 7B model and the model was saved in safetensor format. I wanted to use gguf model so I searched a lot and found a sol...1 like·4.5K readsLLMllamacpp
