A Disallow: /preview/ line went into robots.txt, and preview pages kept showing up in the access log with a crawler user agent on them. The rule was syntactically fine. The URLs it needed to cover wer
Markdown supported · ⌘↵ to post
No responses yet.