The general public is fascinated with every bug that pops up on board the Mars Phoenix Lander because no matter how small, they always seem mission critical. It's exciting stuff and we hang on every bit of news. Will NASA scientists find a workaround for the short circuited mass spectrometer? Will the important science get done? Will we find water on Mars!?

Twitter's system architecture shortcomings are nowhere near as interesting and we're certainly not as exciting as rocket science or exploring Mars to most people. However, folks who use Twitter get frustrated when the service is slow or down which is why we have been trying lately to be more communicative about engineering and operations details.

The TechCrunch blog is particularly interested in behind-the-scenes details about Twitter engineering and systems because the folks who work there use Twitter and also have a very large audience of technology fans. Earlier this evening, TechCrunch posted some specific technology questions for us on their blog so we thought it appropriate to answer them here on our blog.

Before we share our answers, it's important to note one very big piece of information: We are currently taking a new approach to the way Twitter functions technically with the help of a recently enhanced staff of amazing systems engineers formerly of Google, IBM, and other high-profile technology companies added to our core team. Our answers below refer to how Twitter has worked historically—we know it is not correct and we're changing that.

Q: Is it true that you only have a single master MySQL server running replication to two slaves, and the architecture doesn’t auto-switch to a hot backup when the master goes down?
A: We currently use one database for writes with multiple slaves for read queries. As many know, replication of MySQL is no easy task, so we've brought in MySQL experts to help us with that immediately. We've also ordered new machines and failover infrastructure to handle emergencies.

There's a great piece in the Science section of Tomorrow's New York Times about NASA's artful use of Twitter to relay information to the people of Earth regarding the Mars Phoenix Lander.

The tweets were written in the first person, as if Phoenix the friendly spacecraft were sending out text messages to friends and fans. In part, that was to be more entertaining, Ms. McGregor said, but a larger reason was the austere limit of 140 characters per message imposed by Twitter, which turns tweets into a literary form akin to haiku.

JPL based Veronica McGregor has been playing the part of @MarsPhoenix on Twitter and she's mastered the technique. Gathering almost ten thousand followers in a few weeks is only a small feat when compared to something like, oh, SENDING A ROBOT TO MARS. We're very much enjoying the updates and looking forward to more.

We've created a new blog dedicated to status updates regarding Twitter performance and reliability. If something is going on technically, operationally, or otherwise we will put a link in your Twitter home page to a description on this new blog. This includes good news, bad news, warnings, and miscalleneous heads-up notices.

The sidebar of this new blog has a link to our current uptime statistics, our official help docments, our user discussion forum, our developer discussion forum, and our developer blog where Twitter system engineers publish technical articles and information.

Simply getting the word out from the Savannah is part of the challenge of protecting Kenya's wildlife. Masai tribesman Joseph Kimojino (@maratriangle on Twitter) is using Twitter as one way of raising awareness for the Mara Triangle wildlife park. For more information, read Life, Death and Twitter on the African Savannah.

This site features a rolling update of Tennis related Twitter updates. Plus, these folks are live Tweeting the French Open.

For those that read this blog, would you mind if I close commenting on each update after 2-3 days? I'm using moderation because we were having a problem with spam comments. My thinking is that 2-3 days gives folks time to make their voice heard and then I don't have to dedicate as much time to moderating the spam that ensues. Let me know what you think. Also, any other suggestions for how to manage comments are welcome.

TechCrunch?

A discussion fueled by hearsay heated up this week when a popular blogger accused us of refusing to enforce our Terms of Service. The accusation is damaging to our reputation and the additional insinuation that we don't care about our users is offensive to us personally.

Twitter is 16 employees made up of systems engineers and operators, product designers, and support specialists. We do not employ public relations professionals. This accusation caught us by surprise, putting us on the defensive in the middle of what continues to be a very busy work week.

What Really Happened?

A while back, this person submitted 13 individual tweets to our support staff that she considered to be harassment and wanted us to delete the owner's account. It should be noted that the woman who wrote these tweets deleted the account on her own in March.

Upon receiving the complaint, Twitter employees carefully reviewed the 13 alleged offending tweets in order to determine if they violated our Terms of Service. Of the 13 tweets, only one mentioned the blogger by name and it called her this word: "experienced."

There was one post that contained a particularly unpleasant word but it did not identify a specific name. None of the tweets contained threats, physical or otherwise. Some of them may have been insults about personality or physical attributes but again, they did not identify anyone specifically.

During the course of this public discussion it has come to light that these two women have a history of unpleasant interaction over the Internet. While the complainant may indeed feel harassed she is wrong to accuse Twitter of negligence. We review every complaint and we have on occasion deleted accounts for code of conduct violations. In this case, we saw no violation.

Note that this same popular blogger issued another complaint more recently about a different account which was some sort of "confession" mashup that allowed multiple people to post to one Twitter account. Our support staff investigated as soon as we received the complaint but the account had already been deleted by it's creator. This complaint was separate from the original complaint and added confusion. This was not the account that inspired the accusation.

Keeping our Terms of Service Current

The Twitter Terms of Service document is two years old and in need of updating. We were engaged in a review prior to this incident and there is no connection. There are issues of liability, indemnity, and various legalese that need to be brought up to speed with the current state of Twitter, Inc. Additionally, we plan to look very closely at the code of conduct section which refers broadly to "harassment." We think we can make this more specific to avoid future confusion.

In the meantime, we stand behind our current Terms, we care about users, and we take every complaint seriously.

Around 11 am in San Francisco, our main database db006, crashed because of too many connections. We have to put the service into an unscheduled maintenance mode to recover. Folks will see degraded service for the next few hours.

We'll keep you updated here and in Satisfaction.

We found an errant API project eating way too much of our Jabber (a flavor of instant messenger) resources. This activity (which we've corrected) had an affect of overloading our main database, resulting in the error pages and slowness most people are now encountering.

There's some discussion in our forum right now about content disputes and the specifics of our Terms of Service. It seems there is room in our Terms for folks to debate the difference between an update and an insult. There is some confusion surrounding our official policy with regard to taking action.

Some people think we should ban one person if another person is unhappy with the content—or more specifically, if they personally define that content as "harrassment." In the case being discussed, we didn't perceive a violation of our Terms after a careful review.

This speaks to our larger stance that Twitter is a communication utility, not a mediator of content. For those who are interested in this debate, please note that we are engaged in editing our ToS so it more clearly states the scenarios in which we will take action. The revised ToS will reflect this earlier statement of our policy on abuse.

As we wrote yesterday, we're working to bring more visibility into our road ahead.  Along those lines, Alex Payne wrote a great post about our past, present, and future architecture challenges.

I have this graph up on my screen all the time. It should be flat. This week has been rough.

We've gone through our various databases, caches, web servers, daemons, and despite some increased traffic activity across the board, all systems are running nominally. The truth is we're not sure what's happening. It seems to be occurring in-between these parts.

We're busy working on instrumenting and adding meters to provide visibility into what's slowing Twitter down. We'll use this data both to alleviate the current woes and to help inform our long-term architecture work to make Twitter a utility service people can count on. We've definitely failed that aim this week.

Thanks for your patience during these current frustrations (and those to come) as we figure out how to work the kinks out. Thanks also for speaking up: we're listening. In addition to providing visibility into our systems, we're working to give everyone greater visibility into our roadmap to solve these ongoing problems. More to come.

Wow, we had a surprise delivery of Extreme Pizza for everyone at Twitter HQ at lunchtime today thanks to a friendly group of Twitterers. Thanks for doing that, Shannon Whitley, Michael Doeff, MarhabaLife, LLC, Heike McDoniel, Michael Keliher, Ben Jackson, Micah Apparel, William Knoll, Jennifer Leggio, and Suzanne Ally. You guys are awesome!

Downtime is not good. We caused a database to fail during a routine update early this afternoon. We switched to a replica and expect this recovery to take place quickly. We're all working on it and watching right now as Twitter gets back up to speed. We have a thread open on our support forum which we'll update when we have more details to share. Getting our act together is something we continue to work on as we grow our company and our service.

Control Lights with Twitter from Justin on Vimeo.

BusinessWeek has an interesting (to us) article on Twitter this week: "The key question today isn't what's dumb on Twitter, but instead how a service with bite-size messages topping out at 140 characters can be smart, useful, maybe even necessary."

My favorite part, though, is on page 11 of the slide show where they couldn't help asking: "About That Biz Plan?"

It is not entirely to do with the Democrats, Space Aliens, Mysterious Men in Black, or Arugula. In fact, this afternoon's service interruption does not have a very exciting explanation.

Part of our caching service required an unscheduled restart. That means a slow rebuilding of data. You may notice some of the normal browsing related features (such as pagination) are missing while we repopulate the caching service. This is so we can get it done quicker.

Update: It's not Groundhog Day but it sure feels like it. We are recovering today from the same problem as yesterday. This service interruption is our own fault. If you have questions that are only satisfied by technical answers please join our technical discussion forum on Google groups.

I see a lot of confusion around how "at replies"—tweets that start with an @ and then a username—work on Twitter. E.g.:

@Veronica I vote they should be $2.00 each. Cut down on junk mail. Isn't that the only use of postal mail these days, anyway?
10:59 AM May 09, 2008 from web in reply to Veronica

Starting my update with "@veronica" designates that I'm addressing her, specifically, much as you would in a group conversation. Also like a group conversation, she's not the only one who can see what I'm saying. If I wanted to "whisper" something instead, I'd start my message with "d veronica"—what we call a "direct message." (A person must be following you in order for you to direct message them, but not to @reply them.)

As you may know, @replies were not originally part of Twitter. They were embraced by the community first, and then we built them into the system. First, we linked the username (when using the web interface), then we added a replies tab, so you could see replies to you from people you weren't following. We also added a setting to determine what @replies you see (more about that in a minute). And most recently, we added the "swoosh" to the web interface, so you could more easily reply. Other Twitter clients have also embraced @replies in cool ways.

Today, @replies are a critical part of how Twitter works. However, they're not perfect. In fact, some people are either loath to use them or annoyed by them (or both). I believe a big reason for this is that's it's not obvious exactly how they work. In fact, I talk to even advanced Twitter users on a regular basis whom I end up explaining what the @replies setting does and who see what replies.

@Replies To You
If someone @replies you—note, this means they start a tweet with @yourusername—you will see that in your main timeline if you follow the person. But you'll see it in your replies tab, whether you follow them or not (unless you've blocked them):



In some desktop clients, like Twhirl, you can see @replies to you, along with all the updates from people follow, which is neat.

The Setting
There is an @ Replies setting you can find under Settings / Notices:



This has nothing to do with @replies directed to you. This is about what @replies you see from people you follow. The default—@ replies to the people I'm following—is probably what you have it set on (98% of people do). That means, if you're following me, but not following @veronica, you wouldn't see the tweet above (unless you went to my profile).

The beauty of this is that I can feel free to @reply Veronica without worrying about the fact that only a subset of my followers also follow Veronica, so they won't know what I'm talking about. My followers will only see my update if they follow both of us (if they have their setting on the default).

We're trying to avoid the situation of you hearing someone answer a question when you didn't hear the question (for instance). Also, you don't have to hear answers to the question from people you don't want to hear from. (If you're not following them, you won't see their answer.)

This is the main thing that people are confused about, I've found. There are good reasons for this. For one, it didn't use to work like this. (Since @replies were just normal tweets at one point, all your followers would see all of your, no matter who you were replying to.) Secondly, we don't explain it very well (thus, the need for this post). And third, some people do have their setting at "all @ replies"—so they see all the replies people they're following make, even if they're not following the person being replied to. Many people I've talked to have this setting on and don't realize what it actually does. (Usually, they just want to see @replies directed to them).

Conclusion
1) You should feel free to @reply people and not worry about it being out of context to some of your followers. In general, they won't see it.

2) If you're seeing @replies directed to people you don't follow and don't want to, check your setting.

3) This is obviously too confusing. We want to make some changes to make it more clear. We could clarify the setting. But my preference is to take out the setting altogether and just make it work like the default. That way, it works the same for everyone. (If you have strong opinions on this, leave a comment.)

Cool Tools
There are a lot of other ways we'd like to enhance this functionality. But for now, Quotably has worked around the opaqueness of conversations on Twitter and aggregates @replies in a useful way. Also, Summize let's you see all replies to someone.

Tracking You
Another question I get is how to get the equivalent of the Replies tab (@replies to you, especially from people you don't follow) over SMS. The work-around for this (for some people) is just to Track your name. E.g., I send 'track ev' and get a message every time someone mentions @ev. This is not ideal, because Track currently ignores the "@" symbol, so if you have username that is common, you'll get lots of irrelevant stuff. We plan to fix that. (Also, yes, we will get the replies tab on m.twitter.com. Soon.)

The folks over at MySpace chose Twitter along with eBay, Yahoo!, and Photobucket as launch partners for a new Data Availability initiative announced today. We're excited because it adds value to Twitter—after all, finding friends to follow on Twitter makes it much better.

We're still exploring all the cool stuff we'll be able to do with this Data Availability. We'll keep you posted as we take advantage to make improvements to Twitter features and functionality.

Breaking news, Twitter style: "The 'Falls Church Incident' was earthshaking only in the most literal sense, but it is an interesting proof of concept that news can be broken on Twitter."

This is another Twitter card by the someecards guys. Funny.

The database is back up. We're conducting a series of health and sanity checks to make sure everything is working the way it should. Our first quick assessment is that, yes, this problem is over.

For those of you who have spotted intermittent errors tonight, we're working on it. We restarted a database that normally restarts very quickly without resulting in any site degradation. Tonight, that routine changed—it's taking a long time to restart which is causing intermittent 500 errors for some web visitors. We're bringing it up to speed now.

Update: The database is back up.