Unfortuantely, users are encountering some odd display issues on twitter.com as new code rolls out. This shouldn’t be the case and we’ll work to fix that; the display problem should resolve shortly.
Both user icon and user background uploading is currently disabled but we’re working on the fix.
Users in the percentage test of integrated search have had the trends in the sidebar turned off. We’ve also temporarily disabled background image uploading for all users.
We’re working on fixes to both of these issues.
We’ve seen an elevated number of whales and increased slowness of the site. We’ve identified the underlying problem and are working to deploy a fix.
Update: We fixed this.
Some users are not able to delete direct messages. We’ve found the problem and will ship the fix shortly.
Update: We fixed this.
We still have some problems with profile images, follower requests for protected users and device settings. These remaining issues are due to the data inconsistencies we posted about last week. The good news is that we expect these remaining issues to be fully resolved within the next 24-48 hours.
We’ve finished cleaning up the problems that caused users to see the wrong profile page when logging in as well as seeing misdelivered direct messages.
We’re now working through inconsistent follow requests for protected users, errors in phone settings and missing profile pictures.
Having isolated and removed the cause of the data inconsistency problem from yesterday, we continue to clean up the affected accounts.
Thus far we have corrected the problem with misdelivered direct messages. We’re now cleaning up the problems with registered phones, profile images and users getting the wrong homepage when logging in.
We will post more updates as we continue to resolve the problems.
A number of users are experiencing problems due to an underlying data inconsistency in our service. This problem manifests itself in a couple ways:
While these problems are significant, the good news is that data has not been lost. And we are working right now to put everything right.
Update (5:55a): The underlying bug has been identified and corrected. We are running clean-up routines to put everything back in its right place. We’ll provide an update on how much longer that will take as we learn more.
More information on OAuth has been posted to our main blog.
Many people are seeing missing user images in their timelines, tweets and other places throughout the site. We’re aware of this critical problem and will be working to fix it.
For some time, we’ve limited the number of accounts that a single person can follow in a day to 1000. A few days ago, we found that this limit was not being consistently applied and we started enforcing it for all users.
While there are technical reasons behind having some limit on following activites, this per-day limit exists to discourage spamminess. Also, it is unlikely that anyone can actually read tweets from thousands of accounts which makes the mass following activity disingenuous.
We understand that there may be exceptions where mass following is needed in excess of the daily limit: for example, some API applications or for the ability to exchange direct messages. There are also some who think it’s simply polite to follow back other accounts.
We’d like to find ways to directly support the other goals folks are trying to achieve. If you have feedback, please send a reply to @twitter and let us know. We’re listening and will be looking to do more to address the underlying needs.
The site is currently slow and you may experience an odd whale. We are working to correct this problem.
Update: the site is back to its normal speed. Some follow actions may be delayed.
We’ve addressed the most recent variant of the Mikeyy worm. Still, we suggest that you avoid viewing the profiles of users posting uncharacteristic or otherwise suspicious tweets.
We’re aware of the ongoing spam attack happening on Twitter and we’re working to bring it under control.
Due to the incredible rate of follows & unfollows of @aplusk, some users may experience a delay before the follow or unfollow takes effect. We’re working to speed this up; sorry for the inconvenience.
We are currently addressing a new variant of the Mikeyy worm. Please avoid viewing the profiles of users posting uncharacteristic or otherwise suspicious tweets.
Some users are currently receiving duplicate tweets over SMS. You might experience an interruption in your SMS delivery while we solve this problem.
Update: We have fixed the problem.
Updates aren’t showing up in Twitter search in real-time. Changes we’re making today will allow search to have more timely information.
Delivery delay, site latency and errors are much improved this morning due to changes made over the last several days.
We are still tracking a missing update problem (not receiving all tweets) for some users. We’ll be cleaning this up as quickly as we can.
More changes we made this afternoon have improved the latency and error rate on the site, but we have more to do. Many of these improvements will take a few days to roll out. We will continue to update here on our progress.
Two SMS bugs have been fixed that affected users on specific carriers (including Vodafone in the UK.) First, messages sent to Twitter were being posted multiple times. Second, Twitter did not respond appropriately to commands like OFF and ON.
Both of these problems are now resolved. Apologies for the confusion.
We’re working through some problems with high site latency and elevated error rates. The problem is more constrained that those we were dealing with last week. We will update as we gather more information and work toward a resolution.
Update (11:12a): We’re pushing changes now to address this morning’s problems. This may introduce some inconsistencies when you view your follower/following counts or when you attempt to follow another user.
We are currently addressing a new manifestation of the worm attack.
No passwords, phone numbers, or other sensitive information were compromised as part of this renewed attack.
Please see the additional information we posted regarding the incidents from this weekend.
Earlier today we were informed of a malicious site that was spreading links to StalkDaily.com on Twitter without user consent via a cross-site scripting vulnerability. We’ve taken steps to remove the offending updates, and to close the holes that allowed this “worm” to spread.
No passwords, phone numbers, or other sensitive information were compromised as part of this attack.
Some users will be missing tweets from the past couple hours; they’re not lost, just not visible in your timeline. We’ll be putting things back in order shortly.
This morning has been our best peak load period of the week with snappy page load times and low timeline delivery delays. Also, the missing or deleted timeilne issue should be resolved for most users. We know there is more to do - we need to take additional steps to keep these problems in check.
We still have some major bugs that need to be fixed. These include:
More can be seen on our known issues page.
We’ve made good progress on delivering tweets in a more timely fashion; timelines have been up-to-date all day. We are also cleaning up the issue with missing tweets / deleted timeilnes. This should be fully resolved in the next 24 hours (and is fixed for most users currently).
Site latency has been the biggest problem today. The site is much slower than is acceptable and we are still working on this problem.
Additionally, for the past several hours, Facebook has not been updated with new tweets and we will push a fix for this as soon as possible.
The site is quite slow this morning and we’re continuing to add capacity to address this problem.
Steps we took late last night have significantly improved tweet delivery time (new updates appear in timeilnes faster). However, some users will still be seeing the missing tweets or blank timeilne issue. We are deploying more code today that will isolate this issue.
We’ve been actively combatting the delays in posting tweets to timelines. In some cases, this will mean missing tweets so that we can deliver fresher content more quickly. This is only a temporary solution and we are bringing online more capacity so that we can delivery tweets in timely manner.
We’ve also restored status deletion which was temporarily disabled as a preventative measure.
Due to improvements made yesterday, site latency is much improved and the number of errors being thrown (whales, robots) is significantly down.
We still have changes to make and we are currently contending with delays in tweets appearing in timeilnes. Also: a number of users are seeing missing tweets (these tweets are not permanently lost). We are actively working on both of these problems.
We’re actively making improvements to deal with site-wide issues. The number of errors on the site is still quite high which is causing slowness. And we are dealing with delivery delays of outbound tweets to timelines.
The missing user icon problem of yesterday should now be resolved.
Update: Although previously uploaded images are re-appearing, we do still have problems with errors on image upload.
Update: Changes we’ve made has improved the delivery delay issue and we are continuing to work on the other problems.
We apologize for the lack in advance notice, but we are taking Twitter offline for one hour starting at 5:45p Pacific. This is a reschedule of the work we had planned for yesterday morning and is critical to address the site performance issues affecting the site.
Update (6:24p): And we’re back.
We’re seeing improvement in the rate of errors that’s been a problem for us for the past 12 hours. However, there’s still more work to do. Specifically, we’ve got more to do to improve delivery delays on tweets.
A number of users are seeing the default user icon in place of an uploaded user image. Also, some are missing direct messages from their sent tab.
We’re working on both problems now.
Update: We’re in the process of fixing these problems and expect the cleanup to be complete in the next 24 hours.
We’re contending with an elevated number of error pages this morning. Site latency has also been a problem. We’re working through both issues now.
Update: Some users may notice tweets missing in their timelines. These tweets have NOT been lost. We are working to correct the problem.
Twitter.com will be down for an hour beginning around 10AM PST on Sunday, April 5, 2009. We will use this maintenance window to upgrade systems and hardware.
Update: We are rescheduling this maintenance window; Twitter will be up all day today.
The last several hours have featured a significantly elevated number of errors on the Twitter website. We’ve just completed a deploy that has addressed a number of the underlying issues. There’s more to do but we’re pleased that the error rate (whales, robot pages) has returned to a nominal level.
We are still experiencing a problem sending messages reliably to T-Mobile. Due to a problem with our SMS provider, a significant number of outbound messages are not being delivered to T-Mobile subscribers.
We’re hoping to gain some improvement in the short-term and are working on a more robust solution.
Some users (especially those who have joined Twitter recently) won’t be able to find themselves in People Search. We’re working through the underlying bug and expect to see Search up-to-date quite soon.
Many users are running into errors when trying to upload either background or profile images.
We’re working on this problem and will update here with more information.
