Always, have a beta environment and make sure your beta is exactly like your prod. It's very important to have an environment where you can test your product e2e before you take it to prod.
On a lighter note, be wary (very wary) of the rm -rf that caused the GitLab chaos.