Some developers using the Twitter API may notice stale data returned when requesting JSON formatted responses. This is an issue with older cached data being served and we are actively working on a fix for this. Thanks for your patience!
We have to take the service into an unscheduled maintenance because of some issues we saw with one of our databases. Thank you for your patience as we work to bring it back up.
Update (9:16a): We’re back after an unplanned 30 minute maintenance. We’re watching the service closely as things return.
A number of folks are unable to view their replies tab. We fixed some of these cases yesterday and Thursday, but there are more who are still without. As a potential workaround, some have said that refreshing a few times will bring the replies back.
We’ll get this fixed up as soon as we can.
We have restored 99.6% of the following/followers that temporarily disappeared as a result of a database error. The remaining 0.4% are on their way back.
NOTE: Although we have fixed the problem, the numbers and profile pictures in your sidebar will probably not be accurate until tomorrow afternoon because they are cached.
We’re still putting follower/following relationships back into place. There will be inconsistencies over the next several hours as this restoration takes place. We will provide more updates as they become available.
Update (2:14p): we’re going into maintenance mode to speed up some of the recovery work we’re doing. We’re think the downtime will be brief.
Update (2:34p): Maintenance was only about 5 minutes. We’re still restoring data.
Update (3:36p): We believe that by around 7p Pacific around 95% of the data will be recovered.
We’re still in the process of recovering from the missing follower/following problem that occurred earlier today. Over the next several hours, you may see inaccurate counts or timeline inconsistencies as the correct data is propagated to all parts of the system.
One thing to note: Even after this recovery is complete, your counts may appear lower than previously. In almost all cases, this is not due to missing data. The counts we display on your profile page are not always up-to-date. For example, when we remove spammers from the system (which we’ve been doing a lot lately), the follower counts are not updated in real-time.
As we push out the changes to fix this afternoon’s problem, the counts will be updated to reflect the latest numbers.
Folks are reporting problems with missing followers - that is, seeing the follower counts drop on their profile pages.
We’re looking into this now.
Update: We’re going to go into maintenance mode to correct this problem.
Update (5:08p): We are out of maintenance mode. The missing followers and relationships are being restored. It may take several hours for your relationships to be back to the way they were before this outage occurred. We sincerely apologize for the disruption in service.
We’re investigating a service outage that began a few moments ago.
Update: Apologies for the late update, this outage was brief in duration. But we’re still investigating site issues this morning.
A number of users are seeing whales (over capacity messages) this morning. We believe this is because our master database is still warming after a RAM upgrade over the weekend. This may also explain some of the inconsistencies people are seeing on their timelines.
We’re investigating various inconsistencies with people’s timelines including significantly stale content. Some people are seeing older updates from folks. A work around is to go to a person’s profile page directly. We’ll keep you updated here.
We’re experiencing an issue loading profile images and other static content from Amazon S3. We’re working with Amazon to resolve this.
A monitoring process on one of our database machines spiraled out of control, causing the machine to become unresponsive. We’ve killed the process and are working on restoring services. Thank you for your patience.
Update: and we’re back. Load times may be a bit slow until all services have restarted.
We’re working to recover from an unplanned maintenance event this evening. This was caused by a problem with one of our database servers. We apologize for the downtime.
Although this outage was unplanned, we’re going to use this time to upgrade the affected database. We’re hopeful this will help with some of the whale blips we’ve seen a couple times this week.
Update (11:53p): We’re still working through this issue and dealing with some timeline inconsistencies (seeing updates from folks you don’t follow).
Update (2:10a): We’re testing to make sure everything is in a good state before bringing the web servers out of maintenance.
Update (2:20a): We’re out of maintenance. You may see updates from users you do not follow. This problem should not recur.
We had whale sightings for about 10 minutes this morning due to an issue with our master database. This affected all services. Apologies for the inconvenience.
As an update to my post earlier this week, just wanted to note that the last few days have continued to be good:
Whale sightings have continued to go down and pageviews up. Also, we’ve raised the API limit considerably (on an experimental basis).
We’re not out of the woods — but it’s been a good week. Thanks to our awesome team for their hard work.
-@ev
Last night’s maintenance was a success — and, in fact, it only took 30 minutes instead of the scheduled two hours. It was another small step, but definitely in the right direction.
As we mentioned yesterday, page response times have significantly improved in the last week, as you can see from this chart:
(Here’s a live version from Pingom, so you can check on us any time.)
What’s more, sightings of the famous “Fail Whale” have gone down precipitously.
As you can see, while you had approximately a 1-in-29 chance of hitting a whale nine days ago, you only had a 1-in-2000 chance yesterday. And this is with Twitter handling significantly more traffic and updates.
This isn’t good enough, of course. There’s still tons to do. But we have a plan and are encouraged by this improvement. As always, we appreciate your patience as we work through these kinks in our system.
-@ev
Just a reminder that we’ll be having a two hour maintenance window tonight starting at 10p Pacific.
While any downtime is unfortunate, we have been making noticeable improvements to site reliability and performance during these windows. A lot of the most recent work has focused on upgrading the caching layer we use.
As an example of the gains realized through this changes, we had one of our best Mondays in recent weeks today. Monday mornings are one of our busier times and we’ve had latency problems during the past several weeks. However, our external monitoring confirms that latency improved by more than 50% today compared to last Monday.
Still lots more to do to make Twitter more reliable and scalable. But these improvements are getting us the headroom we need to make changes like bringing back IM functionality and restoring the API rate limit.
Our planned maintenance window was a success and we’ve added some capacity to our cluster. Still more upgrades and expansions to do.
We’re hopeful that once we’ve improved the stability of the service we can bring back IM. It remains the highest priority feature we’re working to restore.
People are reporting errors when they update. Updates appear to vanish on the web (requires a page reload) and clients like Twitterrific & Twirl aren’t able to update at all. We’re working on a fix and intend to deploy it today.
Update (9:42a): One of our slave databases went down 10 minutes ago. We have to put the site into maintenance mode while we bring it back up. Thanks for your patience.
Update (10:09a): We’re serving pages again (& a few whales). We’re testing API updates.
Update (10:16a): Clients like Twitterific are reporting success again. We’re watching the situation, but consider this issue resolved.
