GitLab.com in trouble

DavidZemon

Anyone using GitLab.com's issue tracker or merge requests? GitLab.com accidentally wiped production data... ouch!

Twitter account announcing the problem: https://twitter.com/gitlabstatus
Tweet announcing the problem:

https://twitter.com/gitlabstatus/status/826591961444384768

Google Doc explaining the problem in detail: https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub

Cluso99

Holy Smile! When the s**t hits the fan, it sure does a good job!

It's like plane crashes... It is never just one problem, but a series of problems all leading to disaster.

Heater.

Never heard of gitlab before. How is that possible. What do the offer that github does not?

KeithE

"out of 5 backup/replication techniques deployed none are working reliably or set up in the first place"

This is classic. Unless you verify that backups are working, then you need to assume that they are broken. e.g. don't just make a tape and take it off-site. Try to restore a system with that tape, and verify the completeness. Being in charge of this sort of thing can be tedious and boring, but that can quickly change as seen here. When I was at a startup we made sure to get the sysadmin to do some tests to verify the process, and the first round failed. And I think that what we were doing was simple compared to what GitLab is dealing with.