Is developing a large scale system really one of critical factors leading to becoming a good developer?

I have been keep thinking about this question for a long time. Let me give you some background to help you understand the question better.

Here is my assumption:

Large scale system has much more complexity than a small scale system
Many solutions that work for small scale system will no longer work in large scale system
When working with large scale system, you have to think more considerately and more about system quality: distributed architecture, reliability, performance, resilient, high concurrency, sharding, etc

Based upon my assumptions, I think that work on large scale system definitely help you improve and gain a lot of experience. It is just like solving a puzzle, the more complicate the puzzle is, the more you gonna learn, the more it is gonna take to tackle it.

But is it really true? I would like to hear your opinions.

I agree with your analysis. Developing large systems is qualitatively different from small projects.

In addition to what you mention, an important factor is that you won't know and understand all the code, and you will have to solve issues and build additions without knowing all the surrounding code.

So the code needs to have some qualities which are much more important for large systems:

Good encapsulation, because if a change in one place can have an unexpected effect in a far away place, that means developers will either break stuff, or have to read all of that code.
Predictable behavior, because people won't be reading the implementation.
Clear documentation, especially in the form of types and assertions (because that's what people actually follow). If a frontend function assumes input has been sanitized, take a EscapedString instead of String.
Bugs get more likely the larger the system grows, but they also get more expensive as 80% of the time will be analyzing context and only 20% fixing. So code should be more resilient (which is why static typing is more popular for larger systems).

These things are always good, but if it's your own project and you know all the code, it's not as important.

You can be a good developer without doing large scale development. You just probably won't be good at developing large systems.

Does driving a car make you a better cyclist? I don't know :) ....

I help build really large applications. I mean petabyte of data in a week and real-time communications with millions of users and clients.

I still suck at wordpress or writing binary communication protocols. I can use kafka and build clustered docker-boxes as worker systems with apache spark and write asynchronous worker from scratch in 8 different languages.

It makes you a better developer for bigger distributed systems.

The skillset is just different, it's more complex. Did you write a modern react app with redux maybe with worker systems and real time sockets, this is also very complex it is just deemed less special.

Being curious makes you a better developer, challenge yourself on a regular basis. The rest is often just problem domain switching ...

I do recommend building big things, actually I recommend switching through different industries and domains. It helps you gain perspective

Those are my thoughts.

Þorvaldur Rúnarsson that's a good article thx :) I will checkout some of the books ^^

Large scale problems are really optimization problems combined with distributed systems and large app (multiple team member) complexity. I'd argue that until your userbase is large enough to need a distributed system you probably don't need to change much between large scale and small scale. Good optimized code is good optimized code.

That said, it definitely helps to read up on large scale distributed systems to see how they do things. That will help you understand what complexities might occur if your system eventually scales.

The largest scale is global. Chances are if you have that kind of scale then you have bucketloads of money and manpower to throw at the problem. At that scale timing of multiple systems becomes an issue with server performance. There's a big difference between distributed systems in a local market vs one that needs to synchronize performance with multiple localization issues like language, legalities, sociological concepts, interface changes, etc.

My 2 Cents: developing large scale systems today is easier than ever or even possible, which is absolutely great. However a single developer will never good enough to handle the whole stack it should. That's teams are for :)

Sebastian I totally agree, even if one developer is good enough there is the factor of time. Very often in meetings with talented and smart people they tend to ignore that they are just 1 person and it's not only about ability it's about the amount of time needed to do things. So yes you need a good team :) it's actually paramount.

Thread

Is developing a large scale system really one of critical factors leading to becoming a good developer?

Responses(3)

Recent threads

Search Hashnode

Is developing a large scale system really one of critical factors leading to becoming a good developer?

Responses(3)

Recent threads