So, I have a Rails based Newsfeed that's put together by picking artifacts from various things the user follows. I could get all of that in one shot in the right order and all works fine.

But wondering how to implement a "fetch more" concept in that so only a small subset is fetched at the beginning, and subsequent sets are served when asked for.

When accumulating the newsfeed items there's a lot more logic that goes in to pull the relevant ones, but not sure how to do that in batch. I know I can pass in an offset and size and get the next batch but doing the same logic all over again for every request is quite expensive, and at the end of each request most of the resultset would be ignored and only a subset is returned back.

Also, another issue is, between these batched requests if there's a new activity that's posted that could offset these batch calculations and end up returning duplicates as well.

Wondering how it's been done on many of the social media feeds. Any thoughts/suggestions would be greatly appreciated.