I want to design an activity feed for our news website. People subscribe to news topics & other people. Based on this their feed is updated. What's the best architecture to implement this? My Tech Stack is :
Thanks.
This largely depends on how you are storing data in MongoDB. As you are allowing people to follow others and news items, I assume you can quickly create a query to retrieve these data from MongoDB. So, overall you need to think about 3 things :
How to aggregate data stored in MongoDB based on your app logic. This can be written in Ruby/Rails. You need to query MongoDB to retrieve different kind of data for a particular user. For example, if user A follows User B, User C and News N then you need to make appropriate queries to retrieve things like
Are there any recent developments in News N section that User A should know about?
Once you have the relevant data you need to store these in Redis for faster access. You can also sort these items according to the priority and store as SortedSets. This way your feed will be faster. Otherwise, querying and crunching data from MongoDB while rendering the feed will be way slower.
The last thing is that you need to run a cron job which runs in specific intervals (may be 1 hour, but depends on how frequently you receive new items), queries MongoDB and stores feed data for each user in Redis.
I think this will help you get started. This is no advanced stuff - but let us know if you find any difficulty while implementing this. I'll invite @lisol, @sportebois and @Imanov to join the discussion as they might have something to add (and correct me in case I missed something :) ).
Hope this helps!
This repo shows you how to build a feed with Node, MongoDB & Redis. github.com/GetStream/mongodb-activity-feed
Nick Parsons
I would strongly encourage you to not build an activity feed in-house. It's tedious and error prone, and heavily involved. With that said, I would look into a service such as Stream (https://getstream.io). Disclaimer: I currently work for the company.