DNS Outage

Early in the morning there was a DNS error which caused many clients to be unable to resolve our addresses. I was asleep at the time but the Akismet team fixed the error shortly after it was reported, but unfortunately due to the nature of DNS some clients appear to still be caching the “not found” errors, even 6-7 hours after it’s been fixed.

This sucks, and any time there is a problem like this we have a thousand ideas about how to prevent anything like this from happening again. Over the next day or two we’ll be deciding Because of the nature of Akismet it hurts so much when there are any system problems, which is why we’ve been so fanatical about performance and reliabilit through several doublings of the volume of spam.

Now might be a good time to point out that we have a new version of the Akismet plugin for WordPress that adds a “Recheck queue” button to the moderation page. That way if something happens and you have a lot of comments stuck in moderation, with a single click Akismet can recheck them and clear out any known spam.

New Ruby on Rails code

Josh French just wrote in that there is a new Ruby on Rails Akismet project on Ruby Forge. He says, “It’s a tiny bit smarter about grabbing some request info from the environment and more flexible when taking arguments to the public functions. Installs into the plugins dir rather than lib/, for maximum Railsyness.”

Anything with maximum Railsyness is A+ in my book.

Testimonial: Kristie

Just got a great testimonial from Kristie MacLaughlin:

I started getting hit with spam about two weeks ago, and was horrifed to watch it grow exponentially by the hour. I have my comments forwarded to my Blackberry, and it almost shut down my service! I found out about Akismet from a friend, and installed it immediately.

I cannot believe the difference! In two weeks, it was caught over 4000 spam comments! It is absolutely amazing. I cannot thank you enough for making this service available. Spam almost caused me to shut down my blog, but now, I can express myself again without having to spend most of my blogging time deleting spam.

I love Akismet! I will recommend it to EVERYONE I know.

Kristie, it’s our pleasure. Thanks for your kind words.

Blank Response Bug

Some people had been reporting Akismet sending a blank response instead of the expected “false.” We tracked down this bug earlier today and squashed it, and in our monitoring since then haven’t seen it pop back up. Special thanks to Rom Walton, Aaron Swartz, Barnaby Scott for helping track down the issue.

Tres Million

There have been some big spikes in spam lately, and yesterday was the first day we processed more than 3 million comments. The spam has been coming in hard and fast, according to Six Apart’s status blog it was even the cause of Typepad downtime this morning.

Load-wise we’ve been fine, but there has been some reduced effectiveness the past week, especially with the “nonsense spam” of random characters and domains. We’ve not sure what the point of that comment spam is, but we’ve modified how Akismet learns to better deal with random nonsense content, so over the remainder of the week you should these drop off and eventually disappear, most likely around the beginning of next week.

When we do have blips, thank you being so diligent in marking things as spam, not only are you nuking things off your site you’re also protecting the entire Akismet community and helping the system learn from its mistakes. Working together is the only way we’re going to beat the spammers in the long term!