Popular posts

Also may consider a in memory database store if you have to store the data in RAM for quick access. Chances are it handles the amount of data with less overhead/smaller footprint than big amount of Java objects and you just querying what you need at a point.

Sebastian very good point indeed. Still we need further information about why this solution has to be/was built in this certain way. Otherwise we may just add to the next bigger problem. Without purpose and constraints we're possibly just patch symptoms, maybe this whole thing can be a streamed process with aggregation pipelines maybe the global state is needed, maybe not :)

j : why you need those objects in memory? because the solution we need is time critical.

and why some of the filtering cannot be pushed to SQL? filtering can be pushed to SQL. But loading the filtered data back to memory would take time as the size of objects is of the order of 10 million.

does it happen once or 200 times. this happens in bursts. ie, in a day, the requests will come back to back for atleast 10 times per user.

couldn't you just serialize it on apache spark for a distributed map / reduce ? we have not tried this. this could be a solution.

Problems which we face now

• Loading of objects to memory on restart of the software is more than cceptable.
• Our current implementation was part of the Prototype, now we are facing scaling issues.

Sebastian Which in-memory DB would you recommend. We have filters based on date as well.

Well you have pretty defined needs so I guess you have to try out what fits best.

en.m.wikipedia.org/wiki/List_of_in-memory_databas…

I would start with Apache Ignite and HSQLDB

Roopak A N did the memory database work?

about the loading time did you try to use multithreading for loading ? I mainly used it for a kafka consumer but in theory you can separate the chunks and at least speed up the creation processes on multiple cores.

That would be my first thought about the loading times. Besides that java is in my mind not really the language I would use for big datasets, but that's taste.

yes prototypes are usually the first step to the downfall ;D ....