Comment by Jan Vladimir Mostert on "[SOLVED] Which stack would worth better regarding cost and scalability ? PHP, Node, ASP.NET"

SQL Server is not the right tool for doing petabytes of data unless you plan to do some serious complex sharding. Terabytes it can still do, but even then you need some serious tech / skills and infrastructure behind it.

With regards to programming stack ...

PHP, you'll need to seriously consider what you use to run the PHP scripts, Apache usually won't cut it, PHP FPM performs a lot better, but overall it's not that performant at scale, unless you go HHVM. Compiled languages will run circles around PHP, so PHP might push up your hardware cost significantly under heavy load unless you know what you're doing.

Depending on what you plan to do, NodeJS should do fine depending on your architecture; you'll eventually hit bottlenecks and probably won't utilize the hardware to its fullest, but by that time you can throw in more hardware.

ASP.NET I have limited experience with, so can't comment on that.

With regards to your startup, start small and work with a few clients before opening the floodgates; many startups I've worked with claimed they will do petabytes of data in their first few years, the biggest I've seen from those startups that claimed they were going to go into the petabyte scale very quickly hardly made it past the 200GB mark and by that time the business requirements changed which meant re-architecting a lot of things in any case.

Thanks for the answer. I totally agree that It might won't go to that scale and honestly I hope it never will. This project is in health care so less data means healthier people. But at the moment it looks like it is a good candidate for this amount of data, especially because of growing potential users.

Currently I am learning Nodejs (with a php and asp.net, javascript backrgound) because by my little research I found Node would be the best to use regarding cost and scalability. But because that is just my opinion I need some external opinion.

Quote from somebody at Microsoft: SQL Server, in this case the Parallel Data Warehouse appliance version is tested and scales up to hundreds of Terabytes. For anything that goes in the Petabytes, the solution is not only technical but also architectural. You must plan for backup, disaster recovery, maintenance etc. Most likely the result will be that you need to distribute your data across multiple systems to go into the petabyte range. But it depends a lot on the specific workload.

It certainly can be done, good luck!

Do the math, if you store a kilobyte of data per patient and you have a 100 Million patients, you would have a 100GB of data. Even at a 100kB per patient, you'll just scratch the Terabyte scale. Store images outside your DB and you should be fine.

Well we are talking about ~500 million potential users worldwide and it is growing. They would send data 3-10 times on average / day with comments. Also it will have a forum like or Hashnode like app, where users can share knowledge and experience. And that is the main stream. The app also could fit for healthy people who are not sick, but want to prevent it. Of course it is a different audience, but the big part of the app could be reused here too. So that is true I do not know how much data could this generate, but I do not want to fail because I underestimate the project. Also you are right it is not good to overestimate it either. The users won't store too much pictures, at least that is not in the original plan. What do you think, am I safe to say it won't generate that much data ? Also what would you say, what could be the max database size / year ?

If you're going to do a lot of writes at that scale, I'd rather go for something like Cassandra and figure out how to make it work from the start rather than tinkering with MSSQL knowing that your ceiling is way below what you intend to do with it. I know ING is using it and praises it for how well it scales, Netflix was using it at a point in time (they probably still do), see techblog.netflix.com/2011/11/benchmarking-cassand…

Going Cassandra will also allow you to cut back on the licensing fees for MSSQL dbs of which you'll need a lot.

I've spoken to several architects who've used MongoDB and said it starts hitting its ceiling much quicker than anticipated whereas Cassandra they could just keep on adding hardware and it kept on scaling, although they still had a DB team managing the Cassandra instances.

Here are some Cassandra benchmarks from 2012: planetcassandra.org/nosql-performance-benchmarks

Wow, thanks. You convinced me about apache cassandra. Honestly I never heard about this before so thank you very much about this information. Do you know, how should I manage a db host by myself or by an affordable offer at a hosting company ? I am talking about the start up period. Thank you again.

Anything managed will cost a lot more money and if something goes seriously wrong and you lose data, the provider will often not be liable for any losses. Personally I think you should invest some time into learning how to manage it yourself, it'll save a lot of cost in the beginning and put you in full control of the data;

Also have a look at which languages are supported when using Cassandra, not all languages have drivers for Cassandra: https://en.wikipedia.org/wiki/Apache_Cassandra

Last time I did a medical related project in ZA, none of the medical data was allowed to sit in a public cloud and had to be hosted locally as well as complied to certain standards of encryption (that was a few years ago, not sure if it still applies) - so that's also something to take into account - what the laws of respective countries say about storing medical data - in some countries you'll need to setup a DB on that country's infrastructure and use something like passing messages via RabbitMQ to access the data outside that country.

I'd start with 3 servers (that means you can mess around and see what happens when you remove one server and play around to recover that node again while you still have 2 servers as backup), hardware wise I would use SSDs that have any of the RAID levels that allows for redundancy, see https://en.wikipedia.org/wiki/Standard_RAID_levels Cassandra will also give you redundancy, but having the extra redundancy at a hardware level means if a harddrive fails, you can pop in another harddrive and Cassandra won't even know that anything went wrong.

Rackspace, Hetzner or any of those guys doing physical ROOT servers will do. If you want to keep cost low in the beginning, use virtual dedicated servers until you can afford the more beefy root servers. Compared to cloud, physical servers are still much cheaper. See the entry level pricing: hetzner.de/de/hosting/produkte_rootserver/ex41ssd

Search Hashnode