The batch processing optimization is where the real wins are hiding. Seriously, the small-file GCS overhead usually ends up being a way bigger cost driver than throwing more compute at it. And are you already doing differential re-scraping to grab updated captions when popular videos get fresh ones, or are you just treating the corpus as append-only and calling it a day? On the timedtext endpoint, that 4% failure rate sounds pretty solid on paper, but here's the catch - datacenter IPs get hammered and start getting blocked after like 100-250 requests, which could tank your success rate if you're scaling up. If you end up needing residential proxies to keep throughput going, yeah, the costs add up like $3-8 per GB from proxy services, but for transcript data , usually sitting at 100kb-1mb per video, it's honestly not that painful, way less brutal than if you were downloading full videos. Also worth digging into: did you actually profile all the weird failure modes that yt-dlp handles - region locks, age gates, that kind of thing -or could you predict and filter those upfront before they waste your time? Last thing on my mind: have you played around with subtitle density as a CEFR signal? Like, educational channels tend to have way denser, slower captions that might actually signal vocabulary level better than just lemma frequency alone
The batch processing optimization is where the real wins are hiding. Seriously, the small-file GCS overhead usually ends up being a way bigger cost driver than throwing more compute at it. And are you already doing differential re-scraping to grab updated captions when popular videos get fresh ones, or are you just treating the corpus as append-only and calling it a day? On the timedtext endpoint, that 4% failure rate sounds pretty solid on paper, but here's the catch - datacenter IPs get hammered and start getting blocked after like 100-250 requests, which could tank your success rate if you're scaling up. If you end up needing residential proxies to keep throughput going, yeah, the costs add up like $3-8 per GB from proxy services, but for transcript data , usually sitting at 100kb-1mb per video, it's honestly not that painful, way less brutal than if you were downloading full videos. Also worth digging into: did you actually profile all the weird failure modes that yt-dlp handles - region locks, age gates, that kind of thing -or could you predict and filter those upfront before they waste your time? Last thing on my mind: have you played around with subtitle density as a CEFR signal? Like, educational channels tend to have way denser, slower captions that might actually signal vocabulary level better than just lemma frequency alone