1d ago · 6 min read · Feeding raw HTML into a Retrieval-Augmented Generation (RAG) pipeline is a fast way to burn through your LLM token budget. When building data pipelines that rely on publicly accessible web data, the difference between a cost-effective architecture an...
Join discussion
5d ago · 5 min read · Vision models like GPT-4o and Claude 3.5 Sonnet changed how we extract data from the web. Instead of maintaining fragile CSS selectors, engineers started sending screenshots or raw HTML to multimodal models to "see" the data. In 2026, this approach i...
Join discussion
Apr 14 · 7 min read · Build a Production Web Scraping Pipeline for RAG Applications in 2026 RAG applications live or die on data quality. Your embedding model can only be as good as the documents you feed it. This guide covers how to build a scraping pipeline that deliver...
Join discussion
Apr 4 · 8 min read · How to Feed Clean Web Data to RAG Pipelines Without Wasting 90% of Your LLM Tokens Raw HTML is the worst possible input for a RAG pipeline. A single product page carries 15,000 to 25,000 tokens of navigation chrome, analytics scripts, CSS classes, an...
Join discussion
Mar 28 · 6 min read · Originally published at adiyogiarts.com TECHNICAL FOUNDATIONS Synthetic Data & LLMs: Preventing Model Collapse in Pre-Training Fig. 1 — Synthetic Data & LLMs: Preventing Model Collapse in Pre-Training Synthetic data is crucial for pre-training Larg...
Join discussionMar 28 · 5 min read · Originally published at adiyogiarts.com RISK ALERT Preventing Model Collapse in LLM Synthetic Data Pipelines Fig. 1 — Preventing Model Collapse in LLM Synthetic Data Pipelines Large Language Models increasingly depend on synthetic data for pre-trai...
Join discussion
Mar 26 · 8 min read · Build a Real-Time Price Monitor with Python Price monitoring is straightforward in concept and a maintenance nightmare in practice. Anti-bot measures rotate. CSS selectors drift. IP bans accumulate silently. This guide builds a complete system that h...
Join discussion
Mar 25 · 8 min read · Build the Pipeline First, Optimize the Prompt Second Fetch rendered HTML, strip the noise, convert to markdown or typed JSON, then pass clean content to your agent. Done in that order, this pipeline cuts per-page token costs by 10–50x and eliminates ...
Join discussion
Mar 25 · 8 min read · Build a Cost-Effective Web Scraping Pipeline for LLM and RAG Applications The biggest quality problem in RAG pipelines isn't the embedding model or the vector store — it's the input data. Raw HTML fed into a chunker produces token-heavy garbage: navi...
Join discussion
Mar 25 · 9 min read · Web Scraping Pipeline for RAG: Feed Clean Data into Your LLM Without Token Waste Raw HTML is poison for RAG. A typical news article page is 45,000 characters—roughly 11,000 tokens. The actual article is 800 words, or about 1,100 tokens. You are payin...
Join discussion
