the 500tb a month number always bugs me in these writeups because nobody asks what happens after you dedup. if youre storing first copy only and hashing content youre not looking at raw page count times average size, youre looking at unique bodies times size times whatever replication factor you picked, and those two diverge fast once overlapping crawls hit the same articles or product pages from different urls. i've seen real pipelines where content hash dedup alone cuts raw ingest by 30 to 40 percent before compression even touches it. same deal with the qps number, 400 aggregate doesnt tell a worker how many requests it can throw at one host, you need a separate token bucket or delay per domain or you get banned by the first site with any rate limiting at all. treating those two numbers as clean inputs instead of things that need their own logic is where people get tripped up in interviews