@aawiegel

Tests Pass, Data Lies: Observability for Data Pipelines

5d ago · 9 min read · A vendor submitted a file with duplicate sample barcodes. Not a misconfiguration, not a code bug. The vendor sent the same barcode twice, for different samples, across two files. Every test passed. Th

Synthetic Data and the Privacy Problem: Beyond Alice and Bob

Mar 4 · 13 min read · The fixtures in this series have always been honest about what they were optimizing for. Posts 1 through 3 generated vendor CSV files designed to capture structural chaos: typos in column headers, shi

From Bronze to Silver: Staging, Intermediate, and the Art of the Trustworthy Join

Feb 25 · 18 min read · Part 3 solved vendor schema chaos by treating column names as data. The bronze layer now holds one row per attribute per measurement: lab_provided_attribute carries whatever the vendor typed as a colu

The Zen of the Bronze Layer: Embracing Schema Chaos

Feb 6 · 20 min read · In Part 1, we introduced the Medallion Architecture with clean, well-behaved vendor data. In Part 2, we watched the bronze layer transform from "just land the data" into an eight-step ingestion pipeli

When Bronze Goes Rogue: Schema Chaos in the Wild

Jan 28 · 12 min read · In Part 1, we explored the Medallion Architecture with clean, well-behaved vendor data. The bronze layer simply landed the raw CSV files. The silver layer standardized measurement names. The gold laye