Note: This is a blog post draft that was originally written in January 2013, left unattended for 1689 days, and rediscovered and published unedited in August 2017.

During the five years since Mopidy entered the browser with its HTTP JSON-RPC API and the Mopidy.js JavaScript library, a number of successful Mopidy web clients have been built on top of this foundation. The APIs themselves have survived the test of time and have required minimal maintenance, just as I hoped when implementing the APIs back in November 2012.

Mopidy is a music server written in Python. It plays music from various sources, including local disk and Spotify. Mopidy can be remote controlled by, among others, MPD clients.

In Mopidy 0.10, released in the middle of December, we added an HTTP frontend. The HTTP frontend takes Mopidy’s full core API and makes it available from JavaScript in the browser. This means that you now can make your own web clients for Mopidy in JavaScript, and Wouter van Wijk have already started on his client (updated link).

I’d like to write a bit about how we made the HTTP client.

From the start, I was quite clear on having enough work to do on the server side of Mopidy. Thus, making my own client to test out REST APIs and to make sure the APIs we exposed were usable was mostly out of the question. Also, making a REST API would require us to spend huge amounts of time on figuring out how to best remap our procedural “core API” to REST resources. When the remapping would be complete, if ever, we would be stuck with the maintenance of yet another API on top of the core API, with all the associated mismatches and hacks required to match them up. We got enough of them in the MPD frontend.

Wouter had experimented a bit himself with making an HTTP frontend and suggested that we should go for an RPC model. I was reluctant at first. RPC reminds me of SOAP, the nineties, and other similar ideas which I don’t exactly regard as the state of the art.

After some thought, I figured that with an RPC API, I could probably make the entire API on the server side dynamically. In other words, I could get it done in a lot less time than a REST API, and it would maintain itself. At least, it would maintain itself to any degree such a thing is possible. If we added a new method to the core API, it would immediately be available through the HTTP RPC API. This would of course also mean that if we changed anything in the existing core API, we’d break any web client that use that part of the API. After some discussion we decided that we were OK with this drawback. After all, we intend the core API to become quite stable with time, where time approximately equals the release of v1.0. Also, the initial development work and future maintenance work associated with making an RPC API was within our reach, without distracting us for too long from work on the core code, the MPD and MPRIS frontends, and the local storage and Spotify backends. You see, we already got a healthy list of moving parts to keep oiled and working.

The part that really got me excited on an RPC API (of all things to get excited about) was the insight that if I added some introspection support to the API, I could make a JavaScript library to rebuild the entire API in the browser. So, instead of just providing a web service endpoint URL to potential client providers, we could offer them a complete API in the browser. Ready for development. Queue your first music track in a few minutes of development. That’s a good value proposition for aspiring client developers, if you ask me.

So, I made a new HTTP frontend. It started a CherryPy web server. Then it plugged ws4py into CherryPy, and we got a working WebSocket. Then I made my custom RPC API, using JSON as the transport format. I sent messages from Chrome’s console, and music started playing. It worked. It wasn’t tested, but I was happy so far.

Then Thomas, my main co-developer, wise as always, pasted the URL to the JSON-RPC 2.0 specification. Humpfh, I thought, not leaving it much chance. I read through the rather short spec, and concluded that it was really close to what I’d reinvented, minus support for calls with both positional and named arguments at the same time.

Cutting it short, I spent the next day or so implementing JSON-RPC 2.0, this time with tests. I plugged it into the HTTP frontend, and it worked.

Now you might say: Why didn’t you use one of the 25 or so existing JSON-RPC implementations on PyPI. Because there are 25 or so. How am I to review them in less time than it takes to implement the perfect one for my needs? Many of the alternatives provide examples of how to execute a Python file, and then magically a web server will be running. That’s not a selling point to me, since JSON-RPC got nothing to do with webservers, or the message transport for that mather. JSON-RPC is simply a mapping between a JSON format and method calls. It should be implemented by some function/object that accepts JSON, makes the required Python calls, and then returns the return values as JSON again. That’s it. No web server. Maybe some API introspection.

Digression aside, I’m considering extracting and releasing our JSON-RPC adapter. Then it’ll be N+1 competing standards, eh, I mean, implementations.

Next up was the JavaScript library. The main discussion here was actually where to place the code in our pure-Python repo. Bikeshedding of easily reversible decisions continues to be the easiest discussions to have. We ended on js/ in the root of the repo. How imaginative of us.

First thing in any new JavaScript project is of course to set up Buster.JS for testing. I also tried out the Grunt build tool for the first time. (My friend Pål recently wrote an introduction to Grunt featured at HN and in JavaScript Weekly.) Buster.JS in combination with Grunt and PhantomJS was a delight. If you simply run grunt watch and then modify a source file, your code is linted and tested in a headless browser in second or so. This makes JavaScript development for the browser feel like server side development. If this sounds interesting, check out our JavaScript project setup (updated link).

TODO

• Mopidy.js usage examples
• Invitation to develop clients

Note: To quickly address the above TODOs from January 2013: Usage examples can be found in the Mopidy docs, and an invitation to develop clients obviously wasn’t needed, as a dozen clients was made without it.

Today I released the first stable release of netsgiro to PyPI. netsgiro is a Python 3.4+ library for parsing and building Nets “OCR” files.

AvtaleGiro is a direct debit solution that is in widespread use in Norway, with more than 15 000 companies offering it to their customers. OCR Giro is used by Nets and Norwegian banks to update payees on recent deposits to their bank accounts. In combination, AvtaleGiro and OCR Giro allows for a high level of automation of invoicing and payment processing.

The “OCR” file format and file format name originates in days when giro payments were delivered on paper to your bank and then processed either manually or using optical character recognition, OCR. I’m not sure how old the format is, but some of the examples in the OCR Giro specification use dates in 1993, and the specification changelog starts in 1999 and ends in 2003. A couple of decades later, the file format is still in daily use by Nets’ AvtaleGiro and OCR Giro services. In other words, I have high hopes that this will be a very stable open source project requiring minimal maintenance efforts.

Here’s an example OCR file, to be used in later examples:

>>> data = '''
... NY000010555555551000081000080800000000000000000000000000000000000000000000000000
... NY210020000000000400008688888888888000000000000000000000000000000000000000000000
... NY2121300000001170604           00000000000000100          008000011688373000000
... NY2121310000001NAVN                                                        00000
... NY212149000000140011 Gjelder Faktura: 168837  Dato: 19/03/0400000000000000000000
... NY212149000000140012                  ForfallsDato: 17/06/0400000000000000000000
... NY2121300000002170604           00000000000000100          008000021688389000000
... NY2121310000002NAVN                                                        00000
... NY212149000000240011 Gjelder Faktura: 168838  Dato: 19/03/0400000000000000000000
... NY212149000000240012                  ForfallsDato: 17/06/0400000000000000000000
... NY2121300000003170604           00000000000000100          008000031688395000000
... NY2121310000003NAVN                                                        00000
... NY2121300000004170604           00000000000000100          008000041688401000000
... NY2121310000004NAVN                                                        00000
... NY2121300000005170604           00000000000000100          008000051688416000000
... NY2121310000005NAVN                                                        00000
... NY212149000000540011 Gjelder Faktura: 168841  Dato: 19/03/0400000000000000000000
... NY212149000000540012                  ForfallsDato: 17/06/0400000000000000000000
... NY2102300000006170604           00000000000000100          008000061688422000000
... NY2102310000006NAVN                                                        00000
... NY210088000000060000002000000000000000600170604170604000000000000000000000000000
... NY000089000000060000002200000000000000600170604000000000000000000000000000000000
... '''.strip()

netsgiro is obviously not a child of recent changes in my spare time interests, but was conceived as a part of our ongoing efforts in automating all aspects of invoicing and payment processing for Otovo’s residential solar and electricity customers. As such, netsgiro is Otovo’s first open source project. Hopefully, it will soon get some company on our public GitHub profile.

As of netsgiro 1.0.0, sloccount(1) counts 1160 lines of code and 948 lines of tests, providing 97% statement coverage. In other words, netsgiro is a small library trying to do one thing well. Meanwhile, enough effort has been poured into it that I’m happy I was immediately allowed to open source the library, hopefully saving others and my future self from doing the same work again.

The library is cleanly split in two layers. The lower level is called the “records API” and is imported from netsgiro.records. The records API parses OCR files line by line and returns one record object for each line it has parsed. This is done with good help from Python’s multiline regexps and enum.IntEnum, and Hynek Schlawack’s excellent attrs. Conversely, you can also create a bunch of record objects and convert them to an OCR file.

>>> import netsgiro.records
>>> records = netsgiro.records.parse(data)
>>> len(records)
22
>>> from pprint import pprint
>>> pprint(records)
[TransmissionStart(service_code=<ServiceCode.NONE: 0>, transmission_number='1000081', data_transmitter='55555555', data_recipient='00008080'),
 AssignmentStart(service_code=<ServiceCode.AVTALEGIRO: 21>, assignment_type=<AssignmentType.TRANSACTIONS: 0>, assignment_number='4000086', assignment_account='88888888888', agreement_id='000000000'),
 TransactionAmountItem1(service_code=<ServiceCode.AVTALEGIRO: 21>, transaction_type=<TransactionType.PURCHASE_WITH_TEXT: 21>, transaction_number=1, nets_date=datetime.date(2004, 6, 17), amount=100, kid='008000011688373', centre_id=None, day_code=None, partial_settlement_number=None, partial_settlement_serial_number=None, sign=None),
 TransactionAmountItem2(service_code=<ServiceCode.AVTALEGIRO: 21>, transaction_type=<TransactionType.PURCHASE_WITH_TEXT: 21>, transaction_number=1, reference=None, form_number=None, bank_date=None, debit_account=None, _filler=None, payer_name='NAVN'),
 TransactionSpecification(service_code=<ServiceCode.AVTALEGIRO: 21>, transaction_type=<TransactionType.PURCHASE_WITH_TEXT: 21>, transaction_number=1, line_number=1, column_number=1, text=' Gjelder Faktura: 168837  Dato: 19/03/04'),
 TransactionSpecification(service_code=<ServiceCode.AVTALEGIRO: 21>, transaction_type=<TransactionType.PURCHASE_WITH_TEXT: 21>, transaction_number=1, line_number=1, column_number=2, text='                  ForfallsDato: 17/06/04'),
 ...
 AssignmentEnd(service_code=<ServiceCode.AVTALEGIRO: 21>, assignment_type=<AssignmentType.TRANSACTIONS: 0>, num_transactions=6, num_records=20, total_amount=600, nets_date_1=datetime.date(2004, 6, 17), nets_date_2=datetime.date(2004, 6, 17), nets_date_3=None),
 TransmissionEnd(service_code=<ServiceCode.NONE: 0>, num_transactions=6, num_records=22, total_amount=600, nets_date=datetime.date(2004, 6, 17))]

The higher level “objects API” is imported from netsgiro. It combines multiple records into higher level objects. For example, an AvtaleGiro payment request can consist of up to 86 records, which in the higher level API is represented by a single PaymentRequest object.

>>> import netsgiro
>>> transmission = netsgiro.parse(data)
>>> transmission
Transmission(number='1000081', data_transmitter='55555555', data_recipient='00008080', date=datetime.date(2004, 6, 17))
>>> len(transmission.assignments)
1
>>> transmission.assignments[0]
Assignment(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<AssignmentType.TRANSACTIONS: 0>, number='4000086', account='88888888888', agreement_id='000000000', date=None)
>>> len(transmission.assignments[0].transactions)
6
>>> from pprint import pprint
>>> pprint(transmission.assignments[0].transactions)
[PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.PURCHASE_WITH_TEXT: 21>, number=1, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000011688373', reference=None, text=' Gjelder Faktura: 168837  Dato: 19/03/04                  ForfallsDato: 17/06/04\n', payer_name='NAVN'),
 PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.PURCHASE_WITH_TEXT: 21>, number=2, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000021688389', reference=None, text=' Gjelder Faktura: 168838  Dato: 19/03/04                  ForfallsDato: 17/06/04\n', payer_name='NAVN'),
 PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.PURCHASE_WITH_TEXT: 21>, number=3, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000031688395', reference=None, text='', payer_name='NAVN'),
 PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.PURCHASE_WITH_TEXT: 21>, number=4, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000041688401', reference=None, text='', payer_name='NAVN'),
 PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.PURCHASE_WITH_TEXT: 21>, number=5, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000051688416', reference=None, text=' Gjelder Faktura: 168841  Dato: 19/03/04                  ForfallsDato: 17/06/04\n', payer_name='NAVN'),
 PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.AVTALEGIRO_WITH_PAYEE_NOTIFICATION: 2>, number=6, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000061688422', reference=None, text='', payer_name='NAVN')]

The higher level API also features some helper methods to quickly build payment requests, the only file variant typically created by anyone else than Nets.

>>> from datetime import date
>>> from decimal import Decimal
>>> import netsgiro
>>> transmission = netsgiro.Transmission(
... 	number='1703231',
...	data_transmitter='01234567',
...	data_recipient=netsgiro.NETS_ID)
>>> assignment = transmission.add_assignment(
... 	service_code=netsgiro.ServiceCode.AVTALEGIRO,
...	assignment_type=netsgiro.AssignmentType.TRANSACTIONS,
...	number='0323001',
...	account='99998877777')
>>> payment_request = assignment.add_payment_request(
...     kid='000133700501645',
...     due_date=date(2017, 4, 6),
...     amount=Decimal('5244.63'),
...     reference='ACME invoice #50164',
...     payer_name='Wonderland',
...     bank_notification=None)
>>> transmission.get_num_transactions()
1
>>> transmission.get_total_amount()
Decimal('5244.63')
>>> data = transmission.to_ocr()
>>> print(data)
NY000010012345671703231000080800000000000000000000000000000000000000000000000000
NY210020000000000032300199998877777000000000000000000000000000000000000000000000
NY2102300000001060417           00000000000524463          000133700501645000000
NY2102310000001Wonderland                         ACME invoice #50164      00000
NY210088000000010000000400000000000524463060417060417000000000000000000000000000
NY000089000000010000000600000000000524463060417000000000000000000000000000000000

Otovo has been using netsgiro in production for about a month now, and so far so good. We’re surely not the only shop in Norway doing invoicing with Python, so I hope netsgiro will be a useful building block for others as well. If you’re interested in learning more, start with the quickstart guide and then continue with the API reference.

Happy invoicing!

The following is a short summary of my open source work in March and April, almost like in previous months, except that I haven’t spent as much time as previously on Open Source the last two months.

Debian

• Attended one day of the Debian Bug Squashing Party in Oslo.

• Uploaded mopidy 2.0.2-2: Fixed RC bug #819174.

• Uploaded mopidy-dleyna 1.0.5-2: Added missing dependency.

Mopidy

• Bugfixes for the upcoming Mopidy 2.0.1 (which should have been released a long time ago): merged PR #1455, created PR #1493.

• Started on, but didn’t finish, fixing the Travis CI setup for Mopidy-GMusic.

• Upgraded the Mopidy project server from Ubuntu 14.04 LTS to 16.04 LTS. Rebuilt the discuss.mopidy.com Discourse/Docker instance.

• Accepted Lars Kruse as the new maintainer of Mopidy-Beets. Thanks!

• The extensions still in need of a new maintainer are:

  • Mopidy-GMusic
  • Mopidy-NAD
  • Mopidy-SoundCloud
  • Mopidy-YouTube

  If you’re a user of any of these and want to contribute, please step up. Instructions can be found in the README of any of these projects.