#data-pipelines articles

WKWesley Kambalekambale.devJul 29 · 13 min read

Feeding the Beast: Data Pipelines with Grain

Over the last seven weeks we have built models, written training loops, composed optimizers, bulletproofed our code with Chex, and learned to debug inside JIT-compiled functions. Every one of those ar

1

N

ASAnirudh Singh Bhadauriaanirudhbhadauria.hashnode.devJul 21 · 6 min read

Stop Building ETL Pipelines. Start Declaring Them ✨

For years, building data pipelines has followed almost the same pattern. Create a job. Schedule it. Read data. Transform it. Write it somewhere. Repeat. Whether you're using Azure Data Factory, AWS Gl

0

AAlterLabalterlab.hashnode.devMay 11 · 9 min read

Agentic RAG vs Traditional RAG: Architecting Real-Time AI Data Pipelines

Retrieval-Augmented Generation (RAG) solved the initial problem of LLM hallucinations by grounding models in factual data. But traditional RAG architectures share a fundamental flaw: they rely on static data. If you are building an AI agent for finan...

0

AAlterLabalterlab.hashnode.devMay 10 · 6 min read

RAG Pipelines: Why Markdown Extraction Beats HTML for Token Efficiency

Feeding raw HTML into a Retrieval-Augmented Generation (RAG) pipeline is computationally expensive and highly inefficient. Large Language Models (LLMs) operate on tokens, and HTML DOM structures are notoriously token-heavy. When you pipe raw HTML int...

1

O

AAlterLabalterlab.hashnode.devMay 8 · 5 min read

How to Give Your AI Agent Access to Reddit Data

Disclaimer: This guide covers accessing publicly available data. Always review a site's robots.txt and Terms of Service before automated access. AI agents require robust, real-time data to execute complex tasks. Connecting an agent to public discussi...

0

AAlterLabalterlab.hashnode.devMay 7 · 4 min read

How to Give Your AI Agent Access to Amazon Data

Disclaimer: This guide covers accessing publicly available data. Always review a site's robots.txt and Terms of Service before automated access. Building AI agents that interact with real-world e-commerce requires live data. Stale training data doesn...

0

AAlterLabalterlab.hashnode.devMay 6 · 6 min read

True Cost of Web Scraping: Open Source vs Managed APIs

Building a basic web scraper is a ten-minute exercise. Scaling it to extract a million pages a day is a complex infrastructure engineering problem. When developers initially scope a data extraction project, the default choice is often open-source to...

0

AAlterLabalterlab.hashnode.devMay 5 · 8 min read

Evaluating Web Scraping APIs for RAG Pipelines

Building a Retrieval-Augmented Generation (RAG) pipeline requires feeding raw web data into a vector database. But web data is messy, HTML is bloated, and public endpoints aggressively rate-limit incoming traffic. Selecting the right web scraping API...

0

DADavid Aronchickdistributedthoughts.orgMay 5 · 4 min read

Orchestration Allows Microservices to Be Unreliable (That's a Good Thing)

Orchestration Allows Microservices to Be Unreliable (And That's a Good Thing) One of the first features I wanted to build for Kubernetes was service workflows: Service A starts, then B, then C. If B fails, A should know, and C shouldn't panic. Servic...

0

DADavid Aronchickdistributedthoughts.orgMay 5 · 4 min read

Unlocking Reliability: Why Data Pipelines Need Declarative Deployment & GitOps

Unlocking Reliability: Why Data Pipelines Need Declarative Deployment & GitOps You know the feeling: your data pipeline worked perfectly last week, and now it's throwing cryptic errors. The logs don't help. The documentation is outdated. Nobody's sur...

0

#data-pipelines

#data-pipelines

Explore Hashnode

Trending tags this week

Feeding the Beast: Data Pipelines with Grain

Stop Building ETL Pipelines. Start Declaring Them ✨

Agentic RAG vs Traditional RAG: Architecting Real-Time AI Data Pipelines

RAG Pipelines: Why Markdown Extraction Beats HTML for Token Efficiency

How to Give Your AI Agent Access to Reddit Data

How to Give Your AI Agent Access to Amazon Data

True Cost of Web Scraping: Open Source vs Managed APIs

Evaluating Web Scraping APIs for RAG Pipelines

Orchestration Allows Microservices to Be Unreliable (That's a Good Thing)

Unlocking Reliability: Why Data Pipelines Need Declarative Deployment & GitOps

#data-pipelines

Search Hashnode

#data-pipelines

Explore Hashnode

Trending tags this week

Feeding the Beast: Data Pipelines with Grain

Stop Building ETL Pipelines. Start Declaring Them ✨

Agentic RAG vs Traditional RAG: Architecting Real-Time AI Data Pipelines

RAG Pipelines: Why Markdown Extraction Beats HTML for Token Efficiency

How to Give Your AI Agent Access to Reddit Data

How to Give Your AI Agent Access to Amazon Data

True Cost of Web Scraping: Open Source vs Managed APIs

Evaluating Web Scraping APIs for RAG Pipelines

Orchestration Allows Microservices to Be Unreliable (That's a Good Thing)

Unlocking Reliability: Why Data Pipelines Need Declarative Deployment & GitOps