scribu

3D maps using D3 and three.js

2015-02-23T00:00:00+02:00

Last weekend I participated in the Open Data Day Hackathon in Cluj-Napoca. I was glad to see that there were other people nearby interested in open data.

My idea was to make a 3D map of Romania, with each county elevated according to its population. It was really satisfying to go from this:

to this:

Here’s a live demo (requires a browser with WebGL support).

The main advantage of this visualization mode is that you can see all the counties at once, so you get a holistic sense of population distribution, as well as population density.

Features

You can select the census year to view data from. If you hover over a county with the mouse (or tap on it on a touchscreen), you can see the county name and population count in the upper-right corner.

The map can be tilted by dragging and zoomed by scrolling, so you can get the best view of a particular area.

Implementation

From a topojson file, the geographic data for each county is projected to a 2D path using D3. This path is then converted to 3D geometry using d3-threeD and extruded based on the census data. Finally, the geometry is rendered using three.js. The approach was inspired by Jos Dirksen’s post.

One thing I learned is that d3-threeD chokes on shapes with holes in them, so I had to remove them.

To detect hovering over a particular county, a raycaster is used to find if there’s an interesection between the cursor position and any of the county meshes.

For UI rendering I used React, mostly because I wanted to play with it.

For more details about the implementation, check out the source code.

Properly forwarding email to Gmail

2014-11-18T00:00:00+02:00

So I have a custom email address, mail@example.com, which I want to forward to example@gmail.com.

Since I use NameCheap as my registrar, I had the option of setting up email forwarding via their UI. Very easy, but with drawbacks: sometimes email just wouldn’t arrive and I had no way of figuring out what the problem was.

Then, while moving my hosting to DigitalOcean, I decided to do the forwarding on my own server. So I configured the MX records and added virtual aliases in Postfix and pretty soon got a strange error from Google’s SMTP servers:

Our system has detected an unusual rate of 421-4.7.0 unsolicited mail originating from your IP address. To protect our 421-4.7.0 users from spam, mail sent from your IP address has been temporarily 421-4.7.0 rate limited. Please visit 421-4.7.0 http://www.google.com/mail/help/bulk_mail.html to review our Bulk 421 4.7.0 Email Senders Guidelines. f14si1032016icj.42 - gsmtp (in reply to end of DATA command))

After a bit of Googling, it seems somebody had the same problem and one suggested fix went like this: instead of pushing email to Gmail, make Gmail fetch it via POP3, so that’s what I did:

Enabled POP3 access by installing Courier.
Created a separate postmaster user (which doesn’t have sudo).
Routed all incoming mail to the postmater user.
Added the credentials in Gmail (Settings → Accounts and Import).

and it seems to work great. Gmail fetches mail from my server at its own pace, while leaving malicious email in place. I should probably set up a cron job to delete messages left in Maildir/cur/.

Basically, everything you need to know is in this excellent tutorial on the Ubuntu Wiki.

For a long time, for me Postfix was that thing that never works right, but after reading a few clear tutorials, it actually sort of makes sense. So, drop me a line anytime. :)

Left WordPress

2014-11-18T00:00:00+02:00

It seems like most people that follow me on Twitter still think that I’m involved with WordPress. I haven’t done any WordPress-related work in over 6 months and I don’t intend to do any WordPress-related work in the future.

I stopped using WordPress at the end of 2012. I stopped contributing to WordPress completely more than a year ago. All my plugins have been up for adoption for quite a while. Six months ago, I gave the reign of WP-CLI to Daniel Bachhuber and it seems to be going well. I don’t blog about WordPress anymore and I don’t go to WordCamps either.

Ok, but why? A couple of reasons:

Firstly, technical aspects: crappy language (PHP, an ancient version to boot) and crappy architecture (everything touches WP_Query, a god object if I ever saw one).

Secondly, slow development process: I think one of the required ingredients for the success of the WordPress platform is its commitment to backwards compatibility. The downside is that WordPress is stuck with its crappy architecture forever and improvements take a lot more effort to land.

Lastly, I realized that there’s a whole world out there beyond web content management systems that I wanted to explore.

Having said all that, I will miss the WordPress community. It’s hard to leave behind all the friendly faces I’ve spent so much time with online (and sometimes even offline), but it’s something I have to do.

Onward!

High Tech's Folly

2014-06-28T00:00:00+03:00

I am afflicted with this disease:

In large measure the high casualty rate of knowledge-based industry is the fault of the knowledge-based, and especially the high-tech, entrepreneurs themselves. They tend to be contemptuous of anything that is not “advanced knowledge”, and particularly of anyone who is not a specialist in their own area. They tend to be infatuated by their own technology, often believing that “quality” means what is technically sophisticated rather than what gives value to the user.

Peter F. Drucker - Innovation and Entrepreneurship

Travis CI Build Statistics

2014-04-10T00:00:00+03:00

I don’t know about you, but I think Travis CI is the best thing that happened to open-source development since Github.

I noticed that my builds seemed to be getting slower lately. Looking at the build history in the regular Travis CI interface wasn’t very conclusive, because you can only see 10 or 20 builds at a time:

Even after pressing the “Show more” button repeadetly, I still wasn’t sure that it wasn’t just a random fluctuation. What I needed was to see all the builds at the same time.

After a few hours of reading the Travis API docs and fiddling with D3, this is what I came up with:

It incrementally loads the most recent ~500 builds and plots them individually and again grouped by day. Clicking on one of the thin bars sends you to that build’s page on travis-ci.org where you can see all the available information about it.

Looking at the build durations bar chart on the left, you can see that builds have indeed gotten slower, from 15-20 minutes per build, to about 20-25 minutes.

You can inspect the build times for your own projects as well: http://scribu.github.io/travis-stats/

If you have other neat visualization ideas, open an issue on Github: https://github.com/scribu/travis-stats

Cross-browser AJAX uploads with Ember.js and mOxie

2014-02-24T00:00:00+02:00

Implementing single-page web applications that work on all browsers remains a challenge. For the basic task of uploading files, you still need some sort of polyfill or library that adds support for older browsers (read IE 8 and 9, which are still in wide use).

In this tutorial I’m going to describe how to integrate one such library, called mOxie, with one client-side MVC framework, called Ember.js.

0. Getting the mOxie library

I’m going to assume you already have an Ember app going, so the first step is acquiring the mOxie files. You can either use the pre-built files or compile your own. For example, we won’t need XHR2 support in this tutorial, so we can leave it out.

1. Defining the template

The next thing we have to do is write the Handlebars template that will contain all the UI elements we need:

The UI has several components:

error and progress notifications
the list of selected files
the button for selecting more files
the button for initiating the upload

2. Initializing the file picker

In the template above we placed the button inside a view. We can use that view to convert the <button> into a file picker:

Here we create a mOxie.FileInput instance once the template containing the button is rendered.

3. Adding/removing files

The view we defined in the previous step will send events up to the controller, which has to respond to them:

The neat thing about Ember.js is that it will automatically re-render the template whenever the attachments property is modified.

4. Uploading the files

Finally, when the user wants to submit the form, we have to actually send the files to the server:

We start uploading all the files concurrently. When one is done, we increment a counter. When all of them are done, we clear the queue. Did I mention promises are great?

And here are the helper functions used in the controller above:

I wrapped both the mOxie.FileReader process and the AJAX request in RSVP promises so that chaining and utility methods such as .catch() always work as expected.

Demo

I’ve set up a quick demo so that you can see it in action.

This is just a starting point, of course. You can add all sorts of usability enhancements, such as progress bars, image previews etc. Happy hacking!

New Maintainer for Plugin Dependencies: X-Team

2014-01-27T00:00:00+02:00

I handed off development of the Plugin Dependencies plugin to X-Team. They have released all sorts of interesting tools; check them out.

The official Github repository is now https://github.com/x-team/wp-plugin-dependencies.

Quote From "The Little Prince"

2013-11-24T00:00:00+02:00

“Where are the people?” the little prince asked, politely.

The flower had once seen a caravan passing.

“People?” she echoed. “I think there are six or seven of them in existence. I saw them, several years ago. But one never knows where to find them. The wind blows them away. They have no roots, and that makes their life very difficult.”

Creating test doubles in pure PHP

2013-09-20T00:00:00+03:00

The PHP world is not known for good unit test coverage. It’s mostly a cultural issue, but there is a technical aspect to it as well.

PHPUnit allows you to create mock objects, but that assumes your codebase uses the Depedency Injection pattern. If not, it’s very hard to add unit tests without doing major refactorings, because the language doesn’t support monkey patching (i.e. redefining functions and methods at runtime).

You could install the runkit extension, which allows you to replace everything, including constants. However, that means that anyone who wants to run the tests needs to re-compile their PHP.

Patchwork is a PHP library that makes it possible to redefine user-defined functions and methods at runtime, replicating the functionality of runkit_function_redefine in pure PHP 5.3 code.

I started using it to create mocks and it’s great, but I had this nagging thought: how the hell does it work?

I set out to figure out what black magic the author uses, only to find that he already wrote a very easy to understand description of the implementation, like any responsible developer would. Neat!

Gems like these are few and far between in the PHP ecosystem, but they do exist.

The Martin Luther Of Science

2013-09-13T00:00:00+03:00

Science (from Latin scientia, meaning “knowledge”) is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the universe.

Most people, whether religious or not, agree that the scientific method, as described above, is a Good Thing.

Unfortunately, since the nineteenth century, the scientific method has been tightly coupled with a belief system called materialism:

In philosophy, the theory of materialism holds that the only thing that exists is matter or energy; that all things are composed of material and all phenomena (including consciousness) are the result of material interactions.

The main thesis of Rupert Sheldrake’s book, Science Set Free, is that science is being held back by this dogmatic belief in materialism. Below is a TED talk in which he summarizes the book, much better than I could:

In an ironic twist, the above video was banned from the official TED channel, proving that some ideas are taboo and will be deemed “unscientific” by the establishment, evidence be damned.