Everybody struggles: A caching tale

I am currently working on a tool that that automates finding external links on Wikipedia that may not serve as valid references to an article because the link could be broken. So the tool basically makes requests to all the links on a Wikipedia page ...