My current problem has approx 120 million objects in total. Every operation will use around 30-40 million objects from this superset based on some filters. We use Java, MySQL, Hibernate stack. While processing, we need all these objects in memory.
Please share the way you would handle such a problem. Thanks in Advance for the help.
Edit: Our solution is time critical. This is why we handle all operations in-memory.
j
stuff ;)
Could you explain in a little more detail why you need those objects in memory? and why some of the filtering cannot be pushed to SQL?
What is the problem? memory consumption? GC? performance?
Do you need throughput? does it happen once or 200 times. What are the constraints? couldn't you just serialize it on apache spark for a distributed map / reduce ?