Comment by Siddarthan Sarumathi Pandian on "Ask anything to GitLab"

Siddarthan Sarumathi Pandian

Full Stack Dev at Agentdesks | Ex Hashnode | Ex Shippable | Ex Altair Engineering

Sep 22, 2017

I have to say, the way you fought your downtime with full transparency in February this year was amazing.

So, this is my question for you. Just like how buildings have mock fire drills, do you think programmers should simulate failures?

Let's face it, downtime is bad and no one wants it. That being said, one should be prepared to handle it. A team of engineers can take the test environment down and another team will try to bring it back up. So, when production actually goes down (god forbid), the engineers will be acclimatized to handling it in a nice way.

Job van der Voort

VP of Product at GitLab

Sep 27, 2017

Interesting question! I think any piece of critical infrastructure should be either resilient against failures or easily brought up again.

In practice, I wonder if anyone does this against their test instance. In my experience, test instances are brought down involuntary frequently enough that people are comfortable bringing it up again 😅

I do think it's important that every environment is well-instrumented and alerts go out if anything is off.

Dmitriy Zaporozhets

CTO at GitLab

Sep 27, 2017

That's interesting idea around having 2 teams fighting for test env. I think if your test environment is quite stable you should definitely try that. Otherwise, you probably have it failing on regular basis.

I think we (not only GitLab but any infrastructure team) should definitely practice recovery on regular basis. Somehow we manage to forget about it until someone accidentally does rm -rf in the wrong terminal :)

Search Hashnode