Great question. I’d use a two-stage approach: canonicalize the URL first—following redirects, removing tracking parameters, and checking rel="canonical"—then fall back to a normalized content hash when different URLs still serve the same page.

I’d also keep the original URLs as aliases and preserve retrievedAt for each fetch, so deduplication doesn’t remove provenance or freshness information. The canonical URL is the cheaper first check, while the content hash catches mirrors and unreliable canonical tags.