Comment by j on "What would be the tech stack of a high data volume system like Google analytics?"

if you go for google scale, Hbase, Apache Spark, Redshift are more along the line what you gonna need. Even with services like docs.treasuredata.com/articles/quickstart it's still just for long term storage.

You need a high write throughput and if you think of analytic system I would go for a document based storage system since every request can be different and you probably need to scale on geo locations. So no pgsql it's possible yes ... but it's not it's purpose you can use it for the administration parts and things that require consistency.

Cassandra is a column based system it's nice and probably good for this usecase but I still would put a processing pipeline in front of it. something like

but that's just a basic idea, you need to think of fallbacks based on CAP / PACELC and you need a queue for buffering so you have realtime updates that can be delivered to the end user as well as sending it to the API.

We're talking eventbased / stream based design that has to be designed async with eventual consistency.

those are my initial thoughts.

Ankit Singhaniya I think since treasure is a service it provides several engines in the backgound, I know off cassandra but I only had a glimpse on it.

about the consumer -> you need a consumer for kafka, and the consumer is basically just pulling the stream and distributing it to different points of the application, it allows you to control the stream. And the consumers scale different than the logic needed. You don't need it it's just an architecture decision.

the document db is just to keep a query-able document storage so you could easily store the request and since they are usually JSON they are storable without transformations and it's eventual consistency... :D if you don't like the idea take it out :) as I mentioned it was just a top of my head idea.

Search Hashnode