Getting a 389,000-page catalog crawled: the bot rule and the YandexBot 403 we almost shipped

We run OPTTOYSCHINA, a B2B toy marketplace with a catalog of around 389,000 products. Building the site is one problem. Getting a search engine to actually crawl and index that many pages — while the