Dries Buytaert

The future is a RESTful Drupal

Last weekend, we held a sprint at the Acquia offices for the Web Services and Context Core (WSCCI) Initiative for Drupal 8. This was an important sprint for the future of Drupal. This blog post provides a high-level overview of what was discussed and agreed upon; the different sprint participants will be laying out more technical details in follow-up blog posts.

Overall, a wide range of experience levels, skill sets, and perspectives were brought to the table, with the goal of the sprint being to clearly define the initiative’s scope, get agreement on what we wanted to accomplish and why, and lay out a clear plan for how to accomplish this.

In attendance were:

• Larry "Crell" Garfield, lead architect of the WSCCI initiative
• Daniel "sun" Kudwien, top Drupal core contributor
• Fabien "fabpot" Potencier, lead developer of the Symfony project
• Kris "EclipseGc" Vanderwater and James "neclimdul" Gilliland, who have helped develop and implement the plugin system as part of WSCCI
• Greg "heyrocker" Dunlap, initiative lead of the Configuration Management Initiative for Drupal 8
• Moshe Weitzman, long-time Drupal core contributor
• Angela "webchick" Byron, community cat herder and Drupal 7 core co-maintainer
• Randy "rfay" Fay, Justin "beejeebus" Randall and Alex "effulgentsia" Bronstein, core developers and sane voices of reason with an outside perspective
• Dries Buytaert (me), project lead

Scope

The WSCCI initiative, as envisioned by Larry Garfield, was originally set to address Drupal's web services and flexible page layout capabilities. We discovered that both would require significant changes to Drupal core, and it was difficult to build consensus online, so we decided to get together for 3 days and to flesh out what we actually wanted to accomplish, and how.

At the sprint, we first attempted to articulate all of the problems that WSCCI was trying to solve, which included: multiple page layouts, better UI/UX to manage blocks, partial page rendering (ESI, AHAH), contextual blocks, different response types per delivery callback/URL, plugin system / swappable subsystems, lazy loading bootstrap (convert subsystems to PSR-0 classes), infrastructure for building web services, dependency injection, and so on.

We then did a round of voting where we could each choose 3 of those things in order to try to determine which of those were the most important.

Two things became instantly clear during this exercise:

1. The items encompassed under WSCCI really spanned at least 3 separate major areas: Web Services, more robust ESI-based layouts (think Panels only more powerful), and cleaning up our underlying toolset to be a more loosely-coupled framework.
2. The underlying architecture to support RESTful calls to Drupal that makes all of the other things possible was deemed the most important thing to focus on.

Scope resolution

After a good chunk of discussions, all were in agreement to scale back the scope of the initiative to just the "Web Services" piece, and spin off the Layout/blocks/related-UI parts to a separate effort.

Furthermore, some efforts, such as PSR-0 and Unified Plugin system, were only semi-related to the WSCCI initiative in the first place, and just happened to become relevant for it. Work on those efforts will continue as part of the general Framework community efforts.

Architecture

Fabien was able to offer a tremendous number of insights as to how various Symfony2 components could help provide underlying structure for Drupal core to support Web Services out of the box. Essentially, most of what the WSCCI team had been trying to work toward, in the abstract, was already implemented within Symfony2. While some implementation details were different than what we had in mind, the end result is almost exactly what we have been trying to accomplish. We therefore agreed that the best way forward was to leverage several Symfony2 components within Drupal as a way to speed progress toward that end.

Benefits

Some of the concrete benefits that would come out of these changes are

• An underlying framework that is a first-class REST citizen. That means developing web services becomes easier and more efficient, because the plumbing is already in place. An HTML page is a particular case of a REST service.
• Because the core system will be fully REST-ified, we'll be able to improve existing APIs and expose Drupal content in a natively RESTful way. For example, our current AJAX system doesn't comply with REST standards, but with these changes, can be cleaned up to do so.
• That in turn makes Drupal-to-Drupal communication, content staging, content sharing, and a host of other related tasks easier.
• The use of widely used libraries and techniques makes Drupal more approachable to new developers.

Why does this matter?

As it has evolved into an increasingly powerful system, Drupal has gotten increasingly complex and is not as easy to start developing with as it once was. Many developers are nervous about continuing that trend. Managing complexity is a challenge faced by many software projects, and one approach is to use standardized patterns and components.

Due to its long support for PHP 4, as well as some unique needs, Drupal does not take full advantage of standardized patterns and components. The complexity of the custom code that’s used and the non-standard architecture combines to create a barrier to entry for developers new to Drupal (both experienced and novice developers alike).

Meanwhile, the web is constantly changing around us. Web services and mobile are more important than ever, and with that comes the need to have more flexible page and layout capabilities. Now feels like the right time to modernize Drupal’s capabilities to bring it to the forefront of modern PHP systems and web systems in general.

While changing Drupal's core plumbing to address these needs, it's also a good opportunity to do so using more widely understood and modern techniques and libraries. Leveraging the work of a fellow open source community lets us get a jump on these changes to build a more powerful, more flexible, and more easily learnable system in less time than it would take to go it our own.

While these changes may seem large, and some of them are, we believe that they will achieve the right balance between empowering Drupal's design and architecture, and moving toward more modern, standard, well-tested code and techniques to empower a new generation of developers. I hope you are as excited as we are!

Drupal 8 feature freeze: December 1st, 2012

Last summer, I blogged about how I think about Drupal release date planning, tying it to the Gartner hype cycle and the corollary Drupal mood cycle.

The release timeline I laid out in my previous blog post was a Drupal 8 release 18 months after Drupal 7 had achieved the “Plateau of Productivity", the point in time where developing in Drupal 6 seems mostly pointless due to the maturity of Drupal 7.

At that time, I said that I felt that Drupal 7’s “plateau of productivity" was about 6-9 months away. Today, almost 9 months later, I think that by any reasonable measure we are currently there. There are over 300,000 live Drupal 7 installations, which represents nearly 50% of all reported Drupal sites. The top Drupal modules all have Drupal 7 releases, the vast majority of which are either stable releases or release candidates.

Having reached the “plateau of productivity" also means that I feel comfortable announcing the Drupal 8 release timeline (after catch and I talked about it). Without further ado, here is how the rest of the Drupal release cycle breaks down:

Timeline:

• December 1, 2012: Feature freeze. No new features are allowed (unless specifically exempted), focus turns instead to API and UI clean-ups and polishing of existing features.
• February 1, 2013: Code freeze: focus on bug fixes, stabilization. No API changes, instead focusing on bug fixing, preparing for release, and getting the count of critical bugs down to 0.
• August, 2013 (DrupalCon Europe 2013): Drupal 8 released, to wild, international fanfare. :-)

This means that Drupal 8 is 18 months away. Time to shift Drupal 8 core development into higher gear!

The ~6-month window for bug fixes laid out here is obviously much shorter than the 18-month window for bug fixes we ended up having with Drupal 7, but the hope is that the issue count thresholds that we’ve introduced this release will ensure this process is much shorter than in Drupal 7, since we’ll be going from approximately 15 down to 0, rather than approximately 300 to 0.

This timeline also means that if there are Drupal 8 initiatives you’d like to see happen, or other specific features or things you want to see fixed in Drupal core, now is the time to make those things happen. If you’ve never helped with Drupal core development before and would like to, stop by IRC during Core office hours, or join us at DrupalCon Denver. There will also be plenty of other sprints at DrupalCon around various Drupal core initiatives, and you can always start your own!

See you in Denver and in the issue queue! :-)

Mollom.com website redesign (Woot!)

We're proud to present a new design for the Mollom.com website.

We first launched the Mollom.com site in 2007. For more than four years, Mollom.com was using the same design. As we grew Mollom, we wanted to address some of the issues that we've been stewing over since our original design. We have been planning to redesign the site for over a year now but work on the Mollom web service and developing new Mollom products have always had a higher priority so we haven't found the time to complete the new design until now.

The old Mollom.com design that we used from 2007 to early 2012.

The new design is the first step in our plans to reorganize the website. We still have updates to make to the content of some pages, for example. Already, we think the new design is a fresh new change that improves usability.

Take a look at the new mollom.com, we hope you like it!

The new Mollom.com website design.

Core Conversations at DrupalCon Denver

Like at previous DrupalCon's, I'm co-organizing a Core Conversations track at DrupalCon Denver.

The Core Conversations track is a place for people actively working on Drupal or Drupal.org to meet and plan the future of Drupal. Each session is either two 15 minute or one 30 minute presentation, followed by 30 minutes of discussion.

I know a lot of you contribute to Drupal or want to start contributing. If so, Core Conversations are a unique opportunity to present in front of key Drupal contributors, and to make the case for why we need to do more of A or B (e.g. authoring experience improvements, API overhauls, etc.). We need UX conversations, performance conversations, feature conversations, etc. Please share your ideas with the world through Drupal core.

If you have ideas for Drupal core, and you are attending DrupalCon, I suggest that you submit a proposal as soon as possible. The deadline is February 1st so don't wait too long. To get your ideas flowing, here are our conversations from Drupalcon London and Drupal Chicago.

Drupal Association community elections

When we designed a new governance structure for the Drupal Association last year, we decided that most of the board is selected through a nominating committee with the goal to carefully balance many factors like needed skills and geographical and sector representation. However, it was also deemed important that we have directors chosen directly by the Drupal community to make sure that the community is always well-represented.

I'm excited that we're holding our first open community elections. Two community "at large" directors will be elected to the Drupal Association Board of Directors. If you'd like to consider running, please have a look at the "At-large" nominations page. And if you're a Drupal community member, please make time to participate in discussions with candidates and of course to vote, starting January 26. (This process was vetted openly in the community by the Elections Committee and numerous community volunteers at http://groups.drupal.org/drupal-association.)

Your participation will help us take this next important step in implementing a new improved governance structure to strengthen the Drupal Association. Thanks!

Mollom 2011 retrospective

2011 was another excellent year for Mollom. We ended the year having blocked 630 million spam messages, up from 352 million spam messages blocked in 2010 -- and that doesn't even count some of our largest customers like Netlog and other large social networks. And, as in 2010, we ended 2011 with a spam classification efficiency of 99.95%, meaning that only 5 in 10,000 spam messages were not caught by Mollom.

The number of active sites protected by Mollom grew from 28,000 at the end of 2010 to almost 45,000 at the end of 2011. Revenues grew by more than 50% with virtually no sales or marketing efforts.

Almost the entire Mollom team in the Mollom office in Ghent: sun, Ben, Cedric, Thomas, Johan and Vicky. Missing in the picture are Keith and Dries.

All our revenue is invested back into the company. In 2011, we used those funds to grow our team and to fund development on an entirely new product, which may end up rebooting or repositioning Mollom altogether.

Specifically, we have been worked hard on what will be a "hosted comment moderation interface". That interface will provide an optimized moderation environment that will make it easier to moderate multiple websites, either as an individual or as part of a team of moderators. To do so we introduced a new backend with a REST-based API to replace our original XML-RPC API, we rewrote the Mollom module for Drupal, and started to change our website.

We also faced some new challenges in 2011 -- our support requests increased substantially, mostly due to the variety of sites that are now using Mollom. Based on many of these user requests, we tweaked our classifier performance, which resulted in a dramatic decrease in how often Mollom presents a CAPTCHA challenge, and in doing so, solved a number of real-world issues our clients were having with Mollom performance. Rolling out changes without impacting our up-time statistics was no small challenge -- every change we made on the backend has to be weighed against the impact it has on the effectiveness and responsiveness of Mollom on the client side.

2012 may also bring us some additional competition -- some of the world's best venture capitalists invested $8 million in a company called Impermium. Investments like this validate our belief that the social web needs good anti-spam filtering solutions. Impermium is still building its first product but will definitely be a company to watch.

Regardless of what happens in the social web spam market, we'll be busy in 2012. The first half of 2012, you'll notice some new things popping up on Mollom. Our primary goal for 2012 will be to make the "hosted comment moderation interface" available commercially and to refresh our website. Along with launching a new product, we plan to ramp up our sales and marketing efforts. It is time to do so now the Mollom technology has matured after years of intensive investment. We've also got additional work to do to continue to improve accuracy, maintain our high uptime statistics, and work with other open source developers on improvements to Mollom clients for non-Drupal systems.

In short, 2011 was a great year for Mollom. We're happy doing what we do, and we feel that we're helping to make the web a slightly better place. We wouldn't have made it this far without you -- our customers, users and friends. Without you, we wouldn't be a company at all. Thank you for 2011! We're looking forward to sharing a great 2012 with you.

Acquia retrospective 2011

It's that time of year again! In good tradition, here is my retrospective on Acquia's accomplishments for 2011. (You can also read my 2009 and 2010 retrospectives.)

While 2011 was only Acquia's third full year in business (i.e. revenue-bearing year), 2011 was absolutely jam-packed. Starting with executing on our product strategy and vision, to a trip to the Caribbean for the entire company, to being selected by Forbes magazine as one of America's 100 most promising companies, 2011 was full of amazing successes, both for Drupal and for Acquia.

In this post, I'll provide some more detail on what Acquia accomplished in 2011; I'll discuss our business as a whole, our products, our relation with the Drupal community and my role within the company. I have a separate blog post to reflect on how Drupal fared in 2011.

Acquia business retrospective

In 2011 we saw record bookings and continued momentum. We finished the year with 11 consecutive quarters of revenue growth and beating our plan.

Acquia, along with our partners, had more and more engagements with big and well-known organizations, like Paypal, Twitter, Al Jazeera, World Economic Forum, the U.S. House of Representatives, and many more.

Most importantly, customer satisfaction and renewals continued to climb, and are best in class compared to other companies in our industry. Rapid customer growth has resulted in surging ticket counts, now numbering in thousands each month. Sustaining high levels of satisfaction and servicing these tickets has proven to be challenging at times. As a result, we significantly evolved our customer on-boarding process, customer communication, and account management, and we've continued to invest in hiring many great people.

Because things went so well, we decided to accelerate sales and marketing and raised more money mid-2011. We raised $15 million in a fourth round of funding. Our previous investors affirmed their confidence by participating in this round, and they were joined by Tenaya Capital.

In January 2011, we also launched Acquia Europe and overachieved our goals there. We now have about 20 people in Europe.

We ended up growing the company from 80 full-time employees to 175, and growing our bookings by 230%. Mid-way through 2011, our existing office space simply couldn't contain us any longer, so we burst out at the end of August and moved to a bigger 35,000 square feet (3,250 square meter) office where we have had a lot of fun.

Despite our success in growing our staff, the availability of quality candidates continues to be the number one challenge for our continued growth. We're trying to help change that. Together with our partners, we delivered 200 training classes worldwide and we've launched an internal training program called Acquia U, to provide immersive training to a select group of new entry level employees (recent college graduates and career changers).

We've also grown Acquia through the acquisition of companies started by talented people within the Drupal community. This year, Acquia acquired two Drupal companies: security specialist Growing Venture Solutions and migration expert Cyrve. We wanted to do these acquisitions because they create a win-win-win situation for the Drupal community, our partners, and our customers.

Acquia product retrospective

On the product side, Acquia achieved everything in line with the product strategy and vision that I outlined in early 2011. If you're not already familiar with Acquia's products, it's worth reading that post first for context.

We rebooted the Acquia Network. We added two of our own services to the Acquia Network with the new Insight and SEO Grader tools, which provides active site testing for security, performance, and search engine optimization best practices for all of your sites.

In addition to adding our own services, we also added complimentary services and tools from our partners, including New Relic (performance monitoring), Drupalize.me (over 200 hours of Drupal video training from Lullabot), Blitz.io (load testing), Utest (crowd sourced manual testing), and Mobify (mobile delivery of Drupal sites). Lastly we re-built the Acquia Library, our knowledge base on everything Drupal and Acquia. Everything combined, we made massive improvements to the Acquia Network.

We also launched Dev Cloud, a single-server version of Managed Cloud. We now deliver over 4 billion page views a month and 70 terrabytes of data from our Drupal-tuned cloud platform. Our operations team now manages over 2,500 servers through Amazon EC2, up from 500 servers in 2011 and 100 at the end of 2010.

A major low-light was the famous Amazon outage in April 2011. Even though only two enterprise customers were affected, out of a couple hundred at that time, we made fairly significant changes to our roadmap to limit future outages. We've since added features to Acquia Cloud like multi-datacenter failover (both multi-region and multi-availability zone across continents) to increase the service level agreement (SLA) we provide to levels beyond what Amazon provides directly.

2011 was also the year that we commercially launched Drupal Gardens at DrupalCon Chicago after spending considerable design and engineering time on the new Views 3 user interface. Since then, Drupal Gardens has added many requested features and now is hosting over 75,000 Drupal 7 sites including some really large enterprise customers, though we can't talk about them quite yet.

We also did a lot of other things; from relaunching Acquia.com on Drupal 7, to adding support for Drupal 7 and Drupal 8 to Acquia Dev Desktop, to improving both Acquia Commons and COD.

All in all, 2011 was a very productive year for our engineers and product managers.

Community and Acquia

In everything we do, we try to raise the tide for the Drupal community at large. In 2011, we continued our long track record of giving back to the larger Drupal community.

Roughly 30% of our engineering time flows back to the Drupal community and resulted in numerous improvements, including core bug fixes, contributed module porting, and usability improvements to modules such as Date, Media, and Views. We participated in the University of Minnesota usability testing, in addition to performing more than 20 internal usability tests on Drupal and Drupal Gardens whose results have been fed into the community.

We participated in and organized many sprints, including the Drupal 7 media sprint, the Drush Code Sprint, and Multilingual Drupal Code Code Sprint.

In total, Acquia sponsored over 58 community events in the last 3 months of 2011 alone, and covered travel and accommodation costs for dozens of Acquians to contribute in person to the success of these events around the world. We also took the lead in organizing and running several of them.

Our marketing team contributed great sales and marketing collateral to the Drupal Association (creative commons-licensed), to help others in the community to promote and grow Drupal.

In addition, we also had some struggles …

Acquia is obviously interested in helping to make Drupal the best it can possibly be and we're proud of major contributions we make to the Drupal project. For example, due to concerns about the lack of Drupal marketing, we launched the Drupal Showcase site as a resource to enable the community to help market Drupal. And since the adoption and growth of Drupal is vitally important, I, supported by the rest of the Acquia leadership team, made a decision to fund a major usability initiative during Drupal 7's development.

However, some of these community investment decisions have backfired on us, and caused community backlash and criticism. Sometimes over smaller things that are easily corrected, as in the case of the Drupal Showcase (moving it from an acquia.com sub-domain and adding a field for attribution), and other times because of questions and concerns about Acquia's influence, as in the case of Drupal 7 usability.

Acquia is in a position where not only can we give back, we want to give back. And furthermore, I feel that corporate sponsorship (not just from Acquia) is important to Drupal's continued growth and success. But when major investments into Drupal like these backfire, it definitely gives us pause in continuing to make these kinds of large investments. Nevertheless, I'd love to contribute more and bigger changes to Drupal, particularly Drupal core, in a constructive and healthy way. As Acquia, we'll continue to refine how we work with the community to find the right balance. As a community, we need to figure out how to better embrace corporate sponsorship. Something to brainstorm about together in this new year.