A little over a year and a half ago, we had a dramatic rethink of the technologies and development workflows for building with WordPress.
Our existing codebase and workflows had served us well, but ten years of legacy was beginning to seriously hinder us from building the modern, fast, and mobile-friendly experiences that our users expect. It seemed like collaboration between developers and designers was not firing on all cylinders. So we asked ourselves the question:
“What would WordPress.com look like if we were to start building it today?”
A New Beginning: Prototyping and Iterating
We’d asked ourselves this question before, and had our fair share of initiatives that didn’t result in useful change. Looking back, we were able to pinpoint our biggest mistakes: we’d been starting with a muddy vision, and were trying to solve an ill-defined problem. These insights really helped us change our approach.
One of the original Calypso prototype screens, listing all of your WordPress sites.
Calypso, the codename for this new WordPress admin interface project, started differently. To present a clear vision, we built an aspirational HTML/CSS design prototype — based on clearly defined product goals — that allowed us to imagine what a new WordPress.com could look like when complete. We knew it would change over time as we launched parts to our users, but the vision provided all of Automattic with something to aim for and get excited about.
Once the Calypso prototype was in a good place, the early days of development were all about making tough decisions such as which language to use, whether to use a framework, and how we would extend our API. Automattic had just acquired Cloudup, an API-powered file-sharing tool built with JavaScript. The Cloudup team showed us a solid, maintainable, and scalable path towards making WordPress.com completely JavaScript-based and API-powered.
Since WordPress is a PHP-powered application, our company-wide development skill-set has historically been PHP-heavy with a sprinkling of advanced JavaScript. This made Calypso intimidating to other engineers and designers at the company for much of the first six months of its development — we were building something that few people could jump in on.
Even core Calypso project team members had to get over our intimidation. None of us were strong JavaScript developers. But as each day passed our experience built, we made mistakes, we reviewed them, we fixed them, and we learned. Once we had the project moving, we set better examples for other engineers, and shared our knowledge across the company.
One great change came out of building an early design prototype: improved collaboration using GitHub. Calypso prototyping was done collaboratively between a handful of designers in GitHub; although many of us had long used GitHub for personal projects it was relatively new for internal projects, which historically used Trac for most project management and bug tracking. Using GitHub helped us see how much easier internal collaboration could be, and how to allow for much greater feedback on individual work being done.
Peer code reviews show no sign of slowing up and are now widely accepted.
As GitHub had worked so well for the prototyping stage we switched for all Calypso development, allowing us to harness the pull request (PR) system for peer code reviews, and build our own custom GitHub-based workflow. Code reviews were new for many developers — traditionally at Automattic, we have had no systematic peer code review system outside of the VIP team’s daily code review of client sites. Code review, though it initially added to the intimidation of starting to work with Calypso, greatly increased the quality of our codebase and helped everyone level up their JavaScript skills.
What started as a team of seven people working on Calypso quickly spread to a cross-section of teams with ten, then 14, then 20 Automatticians actively working in the Calypso codebase. Two months after the launch of the first Calypso-powered feature on WordPress.com, we had 40 contributors working on Calypso across five different teams. We iterated over the next year with the “release early, release often” Automattic mindset, launching 40 distinct Calypso-powered features on WordPress.com with over 100 individual contributors.
By the middle of 2015 the Calypso codebase was in good enough shape to be used outside of the web browser. Since Calypso is entirely JavaScript, HTML, and CSS, it can run locally on a device with a lightweight node.js server setup. Using a technology called Electron, we built native desktop clients running the same code bundled inside the applications. We started work first on a native Mac desktop app, which is now available, and continued that work on soon-to-be-launched Windows and Linux apps. Seeing these apps come together and using them internally really started to justify all the hard work we’d spent building the Calypso codebase.
Open Sourcing Calypso, the Power Behind WordPress.com
One of our Calypso developer hangouts in progress, and Team IO, who built the Calypso editor, at our all-company Grand Meetup in October.
Over the past year and a half, Calypso has gone from an idea to an aspirational prototype to a fully functioning product built, launched iteratively, and used by millions of WordPress.com users. Internally, it’s been a period of great change and growth. We’ve embraced cross-team collaboration through GitHub and peer code reviews through the PR review system, gone from just a couple of great JavaScript developers to a company full of them, and seen incredible collaboration between designers and developers on a daily basis.
We’re proud to be able to open source all of the hard work we’ve put in, and to continue to build on the product in an open way. You can read more about opening up Calypso development on our CEO Matt Mullenweg’s site.
Over the next few months, we’ll publish more in-depth posts exploring the technicals and workflows behind Calypso: how we manage our own unique GitHub flows, how we’ve used other popular open source libraries like React and concepts like Flux, and our experiences bundling and launching native app clients. Keep an eye out for those by following this blog (in the bottom right), and in the meantime, check out the active Calypso codebase as we continue to iterate on it.
Andy Peatling
Calypso Project Lead
Great work! Really excited to see WordPress getting such a huge update after all these years. BTW, you’ve probably spotted this already, but there’s a typo in the PDF – “Remotely hosted virual machines”
Thanks! Updated this.
Will you consider redoing the WordPress core using a similar technology stack?
Calypso is an Automattic project while WordPress itself is created by the community at large. So that’s not up to us.
However I think it’s extremely unlikely that it’ll happen. Plugins are one of WordPress’s strongest strengths as well as the fact that a plugin written 5 years ago can usually still work on WordPress today. Switching to something else would break all of that. Additionally PHP/MySQL are available on nearly every single webserver at nearly every hosting company while Node.js is not.
WordPress is however in the process of introducing a REST API directly into the core product (no plugin required) and that’s where the really interesting things will come from.
You could for example write a Calypso-like front end for your website (Calypso is an admin interface) and that’d mean that a user visiting your website would never see a PHP-powered page. WordPress would just be silently chugging along in the background handling the REST API requests and storing all of your data.
Looking forward to this.
This is exactly what we’ve done with our new website at ustwo: it’s a React SPA just like Calypso, on top of a headless WP setup. We’ve also open sourced it so people can have a look: https://github.com/ustwo/ustwo.com-frontend
daain: That’s awesome!
I had moved away from WP to Jekyll, mainly because content pages could be hosted directly on github, without the need of a “server”. Now that it’s pure JS, any pointers on how to host page content using a Github/AWS set-up?
If by content pages you mean the front end of the WordPress site, then no. Calypso is a WordPress administration interface and doesn’t cover the display side.
First off, great work. This is really amazing and I can’t wait to start using it. Although, just out of professional curiosity I do have a few questions.
1) Did you do your mobile applications in native Objective C or did you use something like ReactNative?
2) Do you guys plan on releasing the source code to the Electron app on GitHub so people can contribute and see how things were done?
Great questions. Our native mobile apps are not written using React native. We do plan on open sourcing the desktop application code, we’re just not quite ready yet.
The mobile apps are in Objective-C/Swift and Java. We do have some hybrid pieces (like the visual editor) but no plans at the time for React Native. http://github.com/wordpress-mobile 🙂
“You could for example write a Calypso-like front end for your website (Calypso is an admin interface) and that’d mean that a user visiting your website would never see a PHP-powered page. WordPress would just be silently chugging along in the background handling the REST API requests and storing all of your data.”
Can someone point to any link that explains this “a user visiting your website would never see a PHP-powered page”?
PHP’s charm and the charm of WordPress ( and other such PHP scripts) was that it was easy and intuitive to learn, hack, customize and self-host our sites. With advent of FB and lot of complexities in web dev ( no more just PHP, MySQL and html) the number of hobby users or independent users of the web are falling drastically. Many youths of today have no email and will just use FB and some Apps. The meaning of internet also loses its meaning if it is just dominated by FB, Google or even WP.com.
While it is good for Devs, good for “Apps” (company owned stuffs unlike free web) and good for “fast”ness (modern browsers, to me, are fairly fast with WP3+) it is really sad that the web has become complex so much that no longer any kid or youth will create or be able to create anything like WP, Drupal etc.
The clarification in the comments that “Calypso is a WordPress administration interface” is really helpful. I didn’t get that from the original piece. Does this only affect wordpress.com sites? What about self-hosted that use Jetpack?
It also works for WordPress.org sites running the Jetpack plugin. I’ve updated the post to clarify the administration interface part.
Great news. Gives hope to php developers like myself who also feel intimidated by all these new JS technologies. I’m also very much looking forward to ES2015 (the future javascript) that gives us a more robust language to develop the next generation apps. It’s a Brave New World!
Great work guys! I am very curious to read more in-depth articles regarding the creation process behind Calypso. It is really inspiring to see a huge team as your to move from one codebase to another and I am sure it will be helpful for many companies to discover how you applyed this change 😀
Anyway, congrats again guys. I am already using it and I couldn’t be more happy to see your use of the REST API!
Thanks to all for the hard work!
This is a great story. I’m very impressive with what you and the team have done so far. It’s not just the final app for Mac, but the great journey with lots of tries with Javascript and that shows us how much effort Automattic team put on this project to lead the technology.
Thanks for the hard work. I love it.
Can’t wait for Calypso! :3
Does anyone know if Calypso has been used to interface to other blogging or CMS systems? I just released ConnextCMS on GitHub, an open source JavaScript based front end extension for KeystoneJS CMS, which runs in node. It sounds very similar to Calypso in scope.
If someone who is more familiar with Calypso would like to contact me, I’d love to discuss the similarities and differences between the two software systems. For those interested, you can learn more about ConnextCMS and KeystoneJS at ConnextCMS.com
Calypso can only be used to interface with WordPress via the WordPress.com REST API right now.
Feel free to look at the code and chime in on Issues and Pull Requests on our GitHub repository. That’s probably the best way to find out more about the project, and how we work today!
Will it work with uMatrix enabled?
I don’t see why not. 🙂 Have you noticed anything broken with that browser extension/add-on active?