Present Perfect

We had a problem with one of the encoders producing artifacts under certain conditions. It was hard to reproduce, but it usually happened on cartoons, so some of the web developers helped the core team out to see where they were triggered and spent half an hour watching anime cartoons looking for artifacts.

The boss walked past when one of them was watching the cartoon. A week later, he informed the development manager that his guys were watching cartoons on the job. It wasn’t his business, of course, but the boss thought he should know.

So the development manager, in his next sitdown with the developer, said: “Don’t get upset, but I wanted to let you know that the boss has caught you watching anime at work…”

Needless to say the developer was rightfully upset, wondering how the boss could possibly think he was stupid enough to be watching cartoons for fun in plain sight at work…

Our company has a history of working with interns, thanks to our marketing manager. One day, our Operational Manager got an intern. He’s easy-going and gets along with everyone in the company. The intern came for her first day, and joined him in a bunch of meetings as he took the time to explain what sort of things the Operations department actually does.

At the end of the day, he spoke the now-famous words “Espero verte maƱana” – I hope to see you tomorrow!

He didn’t. She never came back!

Well, the cat has been out of the bag for a few days and I have been too busy to blog about it.

But today as I wait for my team to do a final deploy fixing a bug with too-long URL names for Flash Media Encoder, I have some spare time to mention what’s going on and make some people an offer they cannot refuse.

So, for the past half year of so we’ve been hacking away at a new service to solve a very specific problem in streaming. From 2005-2010 the streaming world mostly settled on Flash as a common platform, which was an unstable equilibrium for everyone involved, but it seemed to work. However, with the amount of codecs, devices and platforms there are today, this equilibrium has been falling. The introduction of iPhone, Microsoft’s heavy pushing of Silverlight (paying companies to stream in it – and funnily enough those companies usually stop using Silverlight when the money faucet closes), GoogleTV, the introduction of WebM, the arrival of HTML5 (ironically pushed by Apple – yay – even though their HTML5 sites usually only work in Safari – boo)… all these movements served to upset the status quo once again.

To the eye of the casual observer, it would seem that all streaming has standardized on H264, and so transmuxing technologies are popping up – taking the same video encoding and just remux it for different technologies. However, in practice, H264 is a collection of many techniques and profiles, different levels of complexity, and not all devices support the same profiles and techniques. If you want to stream to all H264 devices with just one encoding, you’ll have to settle for the least common denominator in terms of quality, and you’ll have to pick a resolution that works subpar for all of them.

Now, content producers hate this sort of situation. They just want to get the signal out there, because that’s what matters. The codec and the streaming is just the technological means to get it across the internet. And now the market is asking them to put a bunch of machines in their facilities, learn a lot of technologies they’d rather not worry about, consume heaps of bandwidth to send each version online, and then have to do it all over again each time something changes out there – a new codec, a new device, a new favorite resolution, …

Our answer to this problem is simple: send us one encoding, we will do the rest. Our service will take your live stream, transcode it to as many different encodings as you want, and hand them off to a CDN. That’s basically it. Want full HTML5 coverage ? We’ll do it for you – H264 single and multibitrate, Theora, WebM, and a Flash fallback. Want Silverlight, Flash RTMP, Windows Media MMS ? All there.

Services like this already exist for ondemand – see zencoder and encoding.com and Panda. Live is just inherently more difficult – you don’t get to work with nice single finished files, and it has to happen right now. But this is exactly the sort of thing a framework like GStreamer is good for.

In reality we aren’t doing anything new here – Flumotion runs a CDN that already provides this service to customers. The difference is that this time, you will be able to set it up yourself online. A standard integration time with any CDN is around two weeks. This service will cut that time down to five minutes. We’re not quite there yet, but we’re close.

What’s that you say ? Something about an offer ? Oh, right. It’s always pained me to see that, when we wanted to stream a conference for free, it was still quite a bit of work in the setup stage for our support team, and hence we didn’t stream as many conferences as I would have liked to. Similarly, it pains me to see a lot of customers not even considering free formats.

So the offer is simple. If you are running an event or a conference that flies under a Free/Open banner, and you’re willing to stream only in free formats (meaning, Theora and WebM), and you’re willing to ride the rough wave of innovation as we shake out our last bugs, we want to help you out. Send us the signal, we’ll do the rest. Drop me a line and let’s see how we can set it up. Offer limited, standard handwavy disclaimers apply, you’ll have to take my word for it, etc…

If you’re in the streaming industry, I will be demoing this new service next week on Wednesday around 2.00 pm local time in New York City, at Streaming Media East. And after that our Beta program starts.

Feel free to follow our twitter feed and find us on Facebook somewhere, as the kids these days say…

Happy streaming!

I have been hacking on Paisley more recently. I actually got to hack during work time on Paisley, because the guys needed a feature I had developed – change notification. But more on that later.

I eked out a four day hacking session over this easter weekend, and my primary goal is to make some good advances on my music database system using CouchDB. The code is still in prototype stage, and I wanted to start removing hacks, tightening things down, and adding tests. But suddenly I found myself having to add a bunch of unicode() calls on data coming back from paisley just because I was being stricter on the input to paisley functions.

I didn’t want to have to deal with unicode, again, as it detracted me of the core of my application. But I didn’t like paying the technical debt either of not understanding what was going wrong under the hood.

From my limited understanding, JSON is an object notation format used for exchange of information between processes and applications. A JSON string is in unicode, which is great. It would be pretty useless otherwise in today’s world. So I should be able to send in unicode to JSON libraries and get unicode back out.

A recent change in Paisley by one of the maintainers prefers simplejson over the stdlib-json. I first thought this change was to blame for my problems.

And yes, when decoding a JSON object, text was returned as str instead of unicode objects. Now, this is only when the text is in fact ASCII and hence works fine both as str and as unicode. And I’m sure opinions will differ here – but I think that a JSON library should *always* deserialize text to the same type of object by default – ie, unicode.

Clearly, simplejson disagrees with me. But I didn’t have this problem a few weeks ago, so something changed! What gives? And changing back to json over simplejson didn’t fix it either!

After some googling, I stumbled upon this bug report. Apparently, in 2.7, the C-based implementation deserializes ASCII text as str instead of unicode. The Python-based one always returns unicode for text. And in previous Pythons, both always returned unicode for text.

In essence, my problem boiled down to this:

[thomas@ana ~]$ ipython
Python 2.7 (r27:82500, Sep 16 2010, 18:02:00)
Type "copyright", "credits" or "license" for more information.
IPython 0.10.2 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.
In [1]: from json import decoder
In [2]: decoder.py_scanstring('"str"', 1)
Out[2]: (u'str', 5)
In [3]: decoder.c_scanstring('"str"', 1)
Out[3]: ('str', 5)

versus

[py-2.6] [thomas@ana ~]$ python
Python 2.6.2 (r262:71600, Sep 29 2009, 21:49:07)
[GCC 4.4.1 20090725 (Red Hat 4.4.1-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from json import decoder
>>> decoder.py_scanstring('"str"', 1)
(u'str', 5)
>>> decoder.c_scanstring('"str"', 1)
(u'str', 5)

Note how the last command returned a normal str object.

And yes, in the past few weeks since I last tested this, I did indeed upgrade my machine to Fedora 14, pulling in Python 2.7

simplejson seems to always deserialize to str when it can. I would consider that a bug – ie, ‘be strict in what you produce’.

As for Paisley, I made a feature-unicode branch on github, and this commit introduces a compatibility pjson module. By default it is STRICT, ie it wants unicode back always, and tests for the buggy behaviour, and does an alternative loads implementation that falls back to the python one. I’m sure some Paisley devs will prefer simplejson still, so you can change STRICTness and prefer simplejson.

Now, back to the hack.

I never really got why the Mac is thought to be so userfriendly. This weekend we planned to backup and upgrade an older 2007 iMac running 10.4 Tiger to 10.5 Leopard.

I first wanted to make sure we could make a bootable copy of the hard drive. We got a WD MyBook Studio which is supposedly what you’d get for a Mac, with a fancy e-ink display for name and space left.

When attached to Firewire it first of all was not recognized at all. Over USB it worked, and recommended we do a firmware upgrade. After doing the firmware upgrade and rebooting, the drive wouldn’t light up anymore. A heavy paperweight, essentially. Trickery with dmesg showed that something does get recognized on the USB port, but that’s it. After an hour and a half of trying out various firmware uploading tools, we gave up and sent it back to the store, and settled for a standard no-additional-firmware USB drive.

Let Superduper run for a night backing up 120 GB of drive over 4 hours, and we were good to go (incidentally, I created two boot partitions, so after naming the first one ‘Bootie’, the second one named itself. A for the system drive, B for the first boot drive, and C for the second.)

Then comes the reboot. You’re supposed to hold the Option key during boot. With Macs, this always gives me anxiety – do you start holding down a key before or after you turn it on ? Can you hold it down while you are rebooting ? When can you let go ? There are simply no clues. Between the sound, the grey screen, and the apple, you have no idea what is going on. At least, with the usual PC boot screens, you can check for common problems like ‘is the keyboard even working’. I get Apple Anxiety all the time.

In this particular case, apparently I misremembered what the Option key was in the first place. I was holding down that four-lobed rotated clover key. But apparently Option is the Alt/railroad join key.

How is not labeling a key with the same name your software uses considered userfriendly by anyone ?

After holding down that railroad join key before rebooting until a boot menu pops up, we could choose the Bootie drive and boot from it. At least that bit was easy to use, and worked.

Make another backup just in case, then reboot with an official install CD of 10.5 from work.

This time you have to hold down the C key. I still don’t understand why having to search the net for random information JUST so you can boot from a CD is so much better than a simple boot menu and a prompt to get into it.

And after a lot of whirring and booting into the installer, it simply pops up a message saying ‘Mac OS X cannot be installed on the computer.’ This software cannot be installed on this computer.

No further explanation. How hard could it be to tell me ?

Googling, it turns out that grey install discs are tied to a specific model. The disc came from a MacMini.

And again, after much Googling, it looks like the 30 euro retail version of Snow Leopard can be installed over Tiger on intel Macs, so maybe we should just wait until Monday to upgrade.

Now, if only we could actually get the CD out of the drive when rebooting, as the installer runs from CD it doesn’t let you eject, and when you reboot I don’t know the magic key combination to eject… and I just *know* I’ve had to do this before on a MacMini and all I remember is that it was some stupid combination of tricks…