3d ago · 9 min read · Retrieval-Augmented Generation (RAG) solved the initial problem of LLM hallucinations by grounding models in factual data. But traditional RAG architectures share a fundamental flaw: they rely on static data. If you are building an AI agent for finan...
Join discussion
4d ago · 6 min read · Feeding raw HTML into a Retrieval-Augmented Generation (RAG) pipeline is computationally expensive and highly inefficient. Large Language Models (LLMs) operate on tokens, and HTML DOM structures are notoriously token-heavy. When you pipe raw HTML int...
Join discussion
6d ago · 5 min read · Disclaimer: This guide covers accessing publicly available data. Always review a site's robots.txt and Terms of Service before automated access. AI agents require robust, real-time data to execute complex tasks. Connecting an agent to public discussi...
Join discussion
May 7 · 4 min read · Disclaimer: This guide covers accessing publicly available data. Always review a site's robots.txt and Terms of Service before automated access. Building AI agents that interact with real-world e-commerce requires live data. Stale training data doesn...
Join discussion
May 6 · 6 min read · Building a basic web scraper is a ten-minute exercise. Scaling it to extract a million pages a day is a complex infrastructure engineering problem. When developers initially scope a data extraction project, the default choice is often open-source to...
Join discussion
May 5 · 8 min read · Building a Retrieval-Augmented Generation (RAG) pipeline requires feeding raw web data into a vector database. But web data is messy, HTML is bloated, and public endpoints aggressively rate-limit incoming traffic. Selecting the right web scraping API...
Join discussion
May 5 · 4 min read · Orchestration Allows Microservices to Be Unreliable (And That's a Good Thing) One of the first features I wanted to build for Kubernetes was service workflows: Service A starts, then B, then C. If B fails, A should know, and C shouldn't panic. Servic...
Join discussionMay 5 · 4 min read · Unlocking Reliability: Why Data Pipelines Need Declarative Deployment & GitOps You know the feeling: your data pipeline worked perfectly last week, and now it's throwing cryptic errors. The logs don't help. The documentation is outdated. Nobody's sur...
Join discussionMay 5 · 5 min read · The Myth of Portability: Helm and Kubernetes and the Data Pipeline Problem I spent years helping companies migrate from bash scripts to Chef, and later to containers. The conversation always started the same way: "We want modern infrastructure, but i...
Join discussionMay 5 · 4 min read · From Kubeflow to Real-World ML: Why Data Locality Matters More Than Compute When my co-founders, Jeremy Lewi, Vishnu Kannan, and I started Kubeflow back in 2017, we were trying to solve what felt like the biggest problem in machine learning. Brillian...
Join discussion
