the raw_title matching company check is the best bit here, catches selector drift a normal schema check would just wave through. my issue is the pooled 90 % gate tho, batches are mixed by domain so any single domain under 10% of rows can fail completely and the batch still passes, same regional variant problem the article itself talks about, per domain thresholds would fix that
also a None in those string fields throws attributeerror in strip and except valueerror won't catch it, row.get with a fallback empty string is a cheap fix. bigger issue is quarantine, invalid rows never reach public_market_observations so acceptance_rate there is always 1.0 and that 0.95 alert can never fire, and validation_status plus source_domain arent even columns the python writes
the raw_title matching company check is the best bit here, catches selector drift a normal schema check would just wave through. my issue is the pooled 90 % gate tho, batches are mixed by domain so any single domain under 10% of rows can fail completely and the batch still passes, same regional variant problem the article itself talks about, per domain thresholds would fix that
also a None in those string fields throws attributeerror in strip and except valueerror won't catch it, row.get with a fallback empty string is a cheap fix. bigger issue is quarantine, invalid rows never reach public_market_observations so acceptance_rate there is always 1.0 and that 0.95 alert can never fire, and validation_status plus source_domain arent even columns the python writes