NT
Thanks for the suggestions! I'll keep them in mind for future projects. For this particular project it was a one-off scrape, so I didn't really care much about timestamps, caching, or incremental updates since I had to go through all the pages anyway. If it were a recurring crawl, those optimizations would definitely be worth implementing.