nav search
Data Center Software Security Transformation DevOps Business Personal Tech Science Emergent Tech Bootnotes BOFH

GitLab invokes the startup defence to explain data loss woes

'We're learning every day'. But does 'test your backups' really need to be learned?

By Simon Sharwood, 16 Feb 2017

When GitLab suffered its database deletion, outage and related failure of five backup tools, the company quickly offered The Register an interview. Which sounded like a good opportunity to learn just how a startup aiming for serious developers, and with US$25m of serious investors' cash in its keeping, could have failed to operate a proper data protection regime.

So we said "yes!" to the offer, because we also wanted to know the extent of the company's response to the incident. If GitLab could botch backup so badly, what else could it have found? We already knew it suffered a security panic in late 2016. What of governance? Financial control? And who knows what else?

But then GitLab then couldn't find time for the interview for a day … or two. And stopped answering emails. All while winning praise for its open response to the outage.

Once The Reg started chatting about the situation on Twitter, in response to data protection veteran Preston de Guise's scathing assessment of GitLab's actions, we suddenly received apologies about the slow response – tough and busy week, sorry – and an offer to chat.

Which we did, yesterday, with marketing veep Tim Anglade. We asked for someone beyond marketing as an alternative, and were told someone closer to the business' operations would join the call if available. They weren't.

Anglade articulated what I'd like to call “the startup defence”, which holds that upstarts are allowed to make mistakes while they learn about what it will take to scale. The defence also permits startups to take their eye off the ball a bit as they pour scarce resources into urgent priorities.

“Life in a startup is always about something you don't know knocking you out of the way,” Anglade told us.

But he also told us GitLab did know about data protection and how to do it right. “It is not like we knew about these things and didn't do them,” he said. “We were not willfully looking the other way.”

But we can plainly see that the company was not looking at indicators that would have told it that its data protection regime was failing. At least now the company realises that startups have to grow up.

“We have to think of GitLab not as this thing that grew out of our garage but a critical place for peoples' projects and businesses,” he said. “We want to do a full job of meeting the needs that people expect.”

The main theme to which Anglade kept returning was that GitLab has been very open about the incident and, indeed, that openness is at the core of the company's success. It published a warts-and-all Google Doc outlining the full extent of its upfuckery and a live-streamed video of its developers working to set things to rights. It's also published a full and frank account of the outage and a to-do list explaining its plans to harden its infrastructure and build proper disaster recovery tools to be overseen by “a data durability owner that will be responsible for making sure that things work.”

Anglade added that his biz is “reviewing the platform across all possible vectors from security, reliability, data integrity, existing people and external resources.”

Just how systematic that effort will be was hard to divine. Anglade said: “We are definitely reaching out to a lot of people now and people are offering help to assess our platform.” Those voices are coming from inside and outside the company.

He added that “the CEO is taking a personal interest in the review.”

A “personal interest”? To your humble hack's mind the chief exec should be all over this, unsentimentally exercising authority on behalf of investors and customers alike to ensure the business survives and then thrives.

If that's happening, Anglade couldn't or wouldn't explain the process that's being used. He did say: “We are trying to make it clear we are taking this seriously. It is a big learning experience. We hope never to have to apologise again. We need to learn a little bit more every day and demonstrate we are worthy of users' trust.”

The Register has no animus towards GitLab, or startups in general. It's clear that Silicon Valley's preferred way of doing business creates wealth and extraordinarily useful products. But when that way of doing business leads to organisations that move too fast to use well-known best practices, it makes us wonder if the model always works in users' interests.

But perhaps those worries are irrelevant: Anglade told us GitLab's investors have not offered “anything beyond normal feedback” and “will support the effort to make the changes we need to make.” ®

The Register - Independent news and views for the tech community. Part of Situation Publishing