My node.js server is really becoming laggy or can say slow cause of huge db(mongodb) operations. I know there are some way to make it smooth and fast like clustering the server or server side caching.
It's there anything else including addition and tips for these two to make it faster. I would really appreciate the community guidelines in this. Also Would really help if its in node.js environment. And any better tips for server-side caching. Also is aggregates function for mongodb will help to make it fast or the default ones like collection.findById()?Thanks
Since you are running your server in Node, make sure you are not making synchronous calls - directly, or indirectly (through your dependencies). Even a small "wait" will choke your server if there's substantial amount of usage.
I have written an article around backend optimization. You can read it here:
The first thing I'd usually do in such a situation is (considering I have enough logging), run a stress test to see what all routes are giving me a problem. Once that's done, I see what do the fast ones have which the slow ones don't and vice-versa.
What exactly do you mean by "huge" MongoDB operations? How is your database connected to your server instance? Is it in the same subnet? Or it's a hosted service?
For client-side caching, I'd suggest you use ETags and then use the If-None-Match header when in the request.
You need to give us a little more information on your setup for us to help you. I am more than willing to update this information if you give me some specifics about the infrastructure.
The best way is to make multiple threads of your Nodejs process.
Well, if your server is getting slow, the first thing you should do is find out why it is slow exactly. So, you should profile the server application. Which call is slow? which function is slow? Which lines are slow? And only if you know that you can start with countermeasures. Anything else might yield a better performance, but maybe not. You don't want to gamble!
As for the two techniques you mentioned:
There are two ways to do that. The first one is making use of a process manager, like pm2 (which is based on the NodeJS Cluster module). It allows you to write a state-less server application and run multiple instances of it, making better use of a multi-core CPU. It will not fix MongoDB bottlenecks, though.
The second way is to run a server-cluster, meaning you scale to more than one physical server instance and let your application run on all of them. Clustering servers is hard, because of the IP address, and it'll be quite costly. Again, you can spread the load this way, but it will not fix any non-computing bottlenecks or stuff which is not part of your application.
If you think that db operations are what make your service slow, then caching queries might be a good way to speed up the system. However, there are only two hard things in computer science: Naming things, and caching. What should be cached? How long? How do you know when the data should be invalidated or updated? You will ideally have to write your own caching mechanism to meet your needs. You can either store your cache in-memory, or, if you use clustering, use a fast key-value database, like Redis or Memcached.
You could for example decide, that a user has to call a certain operation quite often while navigating your site, while the result usually doesn't change, so you hash the parameters which lead to the db operation and use the hash as key and the result of the db operation as value for the db. Before going to Mongo, you would first ask Redis if it has a value for the hash. Then, you will have to decide on invalidation. What if the data in the db changes? Is there any way to catch changes? If you can, that would be ideal, and you could add a mechanism which lets you easily invalidate all cached entries for such a queue. If you don't have an easy way, you might have to fall back to a timeout. However, you might still improve performance by refreshing the cache in the background and users will only be able to read the cache, never execute the db transaction themselves (except for the first time).
Unfortunately, I still didn't have the chance to play around a lot with MongoDB myself, so maybe someone else can answer your Mongo perf questions 🙃