Jan 26 路 7 min read 路 A comprehensive guide to running Large Language Models (LLMs) locally on your machine using various tools and platforms. 馃幀 Video Demonstration 1. 馃 Ollama - The Dominant Local LLM Ecosystem Ollama is the dominant ecosystem for running LLMs such a...
Join discussion
Jan 4 路 2 min read 路 No sooner had I build a GGUF model registry than llama.cpp released functionality to dynamically load and unload models from their new llama-server web UI! I had a play with this and realised that it doesn鈥檛 exactly work for my setup, mainly because ...
Join discussion
Jan 3 路 6 min read 路 I鈥檝e had my Spark for a couple of months now. Since it is my only personal computer I鈥檝e gone through the process of working out how to use local LLMs for general tasks like taking & editing notes, browsing the web and coding. This is what I鈥檝e learn...
Join discussion
Dec 27, 2025 路 5 min read 路 Prior to acquiring a DGX Spark, my experience running local LLMs was limited to basic experimentation with Ollama. So when I got my new toy I went a bit crazy downloading lots of different models. And then ended up with a page of notes containing a l...
Join discussion
Nov 30, 2025 路 7 min read 路 TL;DR The DGX Spark has enough unified RAM to load large LLMs, but using dense models makes everything slow. Before I realised the real bottleneck (MoE vs dense, covered in Part 2), I went deep into inference engines. Here鈥檚 how they compare on DGX S...
Join discussion
Jun 25, 2025 路 13 min read 路 The landscape of artificial intelligence has been transformed by large language models (LLMs), with tools like ChatGPT and Claude demonstrating unprecedented capabilities in natural language understanding and generation. However, relying solely on cl...
Join discussionMar 23, 2025 路 5 min read 路 For this guide we should be using an Ubuntu server with atleast 2 GiB of GPU VRAM, 2 VCPU Cores, 8 GiB RAM, and 50 GiB of storage. Let鈥檚 start by updating and installing necessary libraries. sudo apt update && sudo apt upgrade -y sudo apt install -y ...
Join discussion
Feb 11, 2025 路 8 min read 路 Originally published on https://www.ankitbabber.com I have a Mac with Intel silicon. I also have an eGPU with an AMD 6900XT (...allright!). BUT I COULDN'T HARNESS THAT POWER AND RUN A LLM LOCALLY WITH OLLAMA!!! If you have a Mac with Intel silicon, ...
Join discussionOct 28, 2024 路 13 min read 路 note: This paper was written with AI, and the exploration it describes was done collaboratively with AI. The "we" described here is us; me and a few models. With llama.cpp model quantization, properly adjusting models to keep their performance after ...
Join discussionAug 29, 2024 路 13 min read 路 Running large language models (LLMs) like ChatGPT and Claude usually involves sending data to servers managed by OpenAI and other AI model providers. While these services are secure, some businesses prefer to keep their data entirely offline for grea...
Join discussion