Scalability Challenge : How to remove duplicates in a large data set (~100M) ?

Dealing with large datasets is often daunting. With limited computing resources, particularly memory, it can be challenging to perform even basic tasks like counting distinct elements, membership check, filtering duplicate elements, finding minimum, ...