How an engineer brought down Twitter

Twitter’s website has been broken in new ways — and while the company was able to recover from its latest debacle within hours, the story of how it broke suggests similar problems lie ahead.

On Monday morning, Twitter users logged in to find several connected issues. Clicking on links no longer opens them; Instead, users will see a cryptic error message reporting that “your current API plan does not include access to this endpoint.” The images also stopped loading. Some users have reported that they cannot access TweetDeck, the Twitter-owned client for professional users.

Chaos took over the timeline, as users tweeted loudly about the loss – often illustrating their points with images that nobody could see because they wouldn’t load.

“If you change now, everything will fall apart”

In a tweet, the company offered a more detailed explanation of what happened.

“Some features of Twitter may not be working as expected right now,” the company’s support account tweeted. “We made an internal change that had some unintended consequences.”

Changing the question about a project to close the free access to the Twitter API, Platformer can confirm. On February 1, the company announced that it would no longer support free access to its API, effectively ending the existence of third-party clients and limiting the ability of outside researchers to study the network. The company is building a new paid API for developers to use.

But in a sign of how deep Elon Musk’s cuts are at the company, only one reliable engineer is on site staffing the project, we’re told. On Monday, the engineer made a “bad configuration change” that “basically broke the Twitter API,” according to a current employee.

The change had ramifications within the company, bringing down much of Twitter’s internal functionality along with its public-facing APIs. In Slack, engineers responded with variations of “crap” and “Twitter is down – the entire thing” as they scrambled to fix the problem.

Musk is furious, we’re told.

“A small change in the API has big consequences,” Musk tweeted later in the day, after Twitter investor Marc Andreessen posted a screenshot showing that the company’s API failures were trending on the site. “The code stack is too brittle for no good reason. In the end a complete rewrite is needed.”

The relentless layoffs leave the company with less than 550 full-time engineers

Some current employees sympathized with that view, placing at least part of the blame for Twitter’s problems on technical failures that preceded Musk’s ownership of the company. The fail whale became an icon of old Twitter for a reason.

“There is so much technology debt from Twitter 1.0 that if you make a change now, it will break everything,” says one current employee.

However, when Musk took over the company, he promised to improve the site’s speed and stability. His colleagues screened existing staff for their technical skills, ultimately cutting thousands of workers deemed not “technical” enough to succeed under Musk’s leadership.

But the relentless layoffs have left the company with fewer than 550 full-time engineers, we’re told. And as former employees predicted from the start, the losses make Twitter more vulnerable to catastrophic losses.

Monday’s misconfiguration is at least the sixth high-profile Twitter outage this year:

“This type of outage has become so frequent that I think we’re all numb to it,” said one current employee.

And those are just the service outages. Other issues, such as one that led to Musk’s tweets becoming more visible on the timeline than any other user, also plagued the user base.

In many ways, Monday’s loss represents the end of Musk’s leadership of the company thus far. In a single-minded effort to cut costs on his $44 billion purchase, he cut staff and reduced Twitter’s free offerings.

This paves the way for an engineer to staff a major project – one that involves many critical interconnected systems that users and employees rely on.

And with some skilled workers ready to restore service, it took Twitter all morning to fix the problem. “This is what happens when you fire 90 percent of the company,” another employee said today.

Inside Twitter’s HQ, however, the mood is almost lighthearted. “We laughed,” said another employee today.

Leave a Comment