Last.fm – the Blog

In our ongoing effort to make Last.fm the best place for music data, we’ve been using audio fingerprint tech since 2007 to match metadata information (i.e. artist and track names) by analyzing the audio.

This helped us automate the process of spelling correction, and as by-product we released an open source library and a client which can be used to identify the correct metadata for mp3s.

We have recently reached the milestone of 90 million fingerprint entries and all this information is now fully accessible via a new track.getFingerprintMetadata call in our official API, with musicbrainz ids and full UTF-8 support.

If you are a developer you might want to look into the library module (which generates the fingerprint) available on github (and it’s LGPL!). If you just want to play with it, you can use our sample client (available for Windows, OS X and Linux), which query the fingerprint service for an id, then pokes the API for the metadata.

Running the fingerprint client is a piece of cake. Just type

lastfmfpclient mysterious.mp3

and it will generate the following xml output (which is taken from the API):

<?xml version=“1.0” encoding=“utf-8”?>
<lfm status=“ok“>
  <tracks>
    <track rank=“1“>
      <name>One Summer’s Day</name>
      <mbid></mbid>
      <url>http://www.last.fm/music/%E4%B9%85%E7%9F%B3%E8%AD%B2/_/One+Summer%27s+Day</url>
      <streamable fulltrack=“0“>0</streamable>
      <artist>
        <name>久石譲</name>
        <mbid>44c64a30-1d58-49c5-b314-6e02fba49526</mbid>
        <url>http://www.last.fm/music/%E4%B9%85%E7%9F%B3%E8%AD%B2</url>
      </artist>
      <image size=“small“>http://userserve-ak.last.fm/serve/34s/3737553.jpg</image>
      <image size=“medium“>http://userserve-ak.last.fm/serve/64s/3737553.jpg</image>
      <image size=“large“>http://userserve-ak.last.fm/serve/126/3737553.jpg</image>
      <image size=“extralarge“>http://userserve-ak.last.fm/serve/300×300/3737553.jpg</image>
    </track>
    <track rank=“0.404785“>
      <name>Ano Natsu he</name>
      <mbid></mbid>
      <url>http://www.last.fm/music/%E4%B9%85%E7%9F%B3%E8%AD%B2/_/Ano+Natsu+he</url>
      <streamable fulltrack=“0“>0</streamable>
      <artist>
        <name>久石譲</name>
        <mbid>44c64a30-1d58-49c5-b314-6e02fba49526</mbid>
        <url>http://www.last.fm/music/%E4%B9%85%E7%9F%B3%E8%AD%B2</url>
      </artist>
    </track>
    <track rank=“0.144543“>
      <name>あの夏へ</name>
      <mbid></mbid>
      <url>http://www.last.fm/music/%E4%B9%85%E7%9F%B3%E8%AD%B2/_/%E3%81%82%E3%81%AE%E5%A4%8F%E3%81%B8</url>
      <streamable fulltrack=“0“>0</streamable>
      <artist>
        <name>久石譲</name>
        <mbid>44c64a30-1d58-49c5-b314-6e02fba49526</mbid>
        <url>http://www.last.fm/music/%E4%B9%85%E7%9F%B3%E8%AD%B2</url>
      </artist>
      <image size=“small“>http://userserve-ak.last.fm/serve/34s/6624395.jpg</image>
      <image size=“medium“>http://userserve-ak.last.fm/serve/64s/6624395.jpg</image>
      <image size=“large“>http://userserve-ak.last.fm/serve/126/6624395.jpg</image>
      <image size=“extralarge“>http://userserve-ak.last.fm/serve/300×300/6624395.jpg</image>
    </track>
    …
  </tracks>
</lfm>

Now that we’ve seen the return of full length previews for independent artists, we wanted to go a step further and build some new discovery tools to help introduce users to emerging artists offering direct-from-artist tracks, and vice versa.

After some late nights in the Last.fm MIR laboratory, we’ve come up with a dedicated direct-from-artist recommendations page on our playground, where you can play a personalised selection of full length tracks.

Check out your own recommendations:
http://playground.last.fm/demo/directrecs

Please try it out and let us know what you discover! We’re going to keep working on this functionality and building it into the rest of Last.fm, so your feedback can help us improve.

Along with our support for Music Hack Days, the Last.fm team often gets involved with the MIR (Music Information Retrieval) community, a group at the forefront of recommendation technology.

This year we are helping organizing a music recommendation workshop within RecSys entitled WOMRAD 2010. The topics of interests vary from music similarity measures, recommendation, music cognition, user modeling and many more.

If you are a researcher in the field or you are just interested in the topic, join us in Barcelona this September the 26th to meet and exchange ideas!

Being in charge of the database that stores everyone’s scrobbles is a great responsibility and one I don’t take lightly. One thing that I get asked now and again is, “I’ve accidentally deleted Artist X from my profile, can you restore it please?”. Sheesh, the work of a Scrobble Overlord is never done! :)

Well, the simple answer is, “yes I can” but up until now I’ve not had anything that allows it to be done with the click of a button or two. I decided enough was enough and the result is this Playground app.

Ladies and gentlemen, I bring you the self-service Artist & Track Restore Tool.

Okay, so just a very brief explanation of how it works (skip this if you don’t care). When you delete anything from your library we don’t actually delete it, we move it to a different table in your database. We need to do this so we can keep track of what you’ve deleted to make sure we don’t keep offering it to you as a recommendation.

All this Playground app does is allow you to review what’s in your deleted tracks table and if you choose to restore them it just puts them back into your main scrobble table.

One thing to be mindful of is that your playcount may not update immediately since it is cached. This will auto-correct within approximately 24 hours.

Please note that this tool cannot be used to restore a deleted or wiped profile as can be found in the account settings… where erase really means just that!

I think this is a useful tool and hopefully you will too, but please let us know why if you don’t!

My first festival was Lollapalooza 1995. I had to go because my favorite band, Sonic Youth, was headlining (they’re still my favorite band). I lived in Houston and the festival was in Austin but I was too young to drive so my mom made the two hour trip so that I could attend with two friends. A transcendental 20+ minute performance of “The Diamond Sea” was the highlight of the whole festival. Also playing that year were Moby (way before he hit it big with his album Play), Cypress Hill, Blonde Redhead, Built to Spill, Mike Watt, The Roots, and Yo La Tengo.

Photo: Russ Garrett

At Last.fm we know from experience that many of the best music memories happen when you’re rocking out in a field with thousands of other sunburnt/soaking wet compatriots. So today we are excited to launch Last.fm Festivals, your personalized guide to festivals around the world. For the first time you can easily browse upcoming festivals in your town, your country or beyond, based on your personal music profile. Choose your location and sort the results either by date or by compatibility.

Once you’ve found your perfect festival, check out the festival page where Last.fm looks at the line-up for the festival and compares it to your Last.fm profile to generate a personal line-up recommendation list so you can be sure you don’t miss a thing. You can also browse the line-up by genre to see the artists that you might like but aren’t part of your Last.fm profile yet. Plus you can start a festival radio station to listen to all the artists playing.

Check it out: http://www.last.fm/festivals

This is just the beginning. Over the next month we’ll be rolling out additional features to help you find up and coming artists you should check out (so you can say you saw them before they were big), find festival buddies and easily share your festival picks.

Read more about all the new features on the forum post and let us know what you think.

Since last week’s announcement of our track page beta and streaming changes we’ve been receiving tons of feedback. Responses have come via this blog, our discussion forums, and elsewhere, and we’ve been paying attention to both the positive and the negative. The fact that so many of you are taking the time to offer thoughts and constructive criticism means a lot to us.


The team spent the better part of last week looking at a few different improvements we might be able to introduce in response to some of this feedback. In particular, we first wanted to address the requests from the many independent artists who use our Music Manager to upload their tracks directly onto Last.fm.



Our renewed focus on the features that make Last.fm unique — scrobbling, personalised radio, and being the online home for your music taste –– means we won’t be returning our own on-demand track offering. Instead, based on your suggestions, we’ll be connecting people to more streaming services in more countries by the end of the year, partners that focus squarely on providing the best on-demand experience possible and scrobbling it. We also have plenty of new music discovery and exploration improvements coming to the Last.fm website and beyond, helping make your scrobbles even more useful to you.


However, today we’re happy to announce that unsigned artists who hold global rights to their music will soon be able to again offer tracks in full to their fans via their track pages. For the first time, this option will be clearly labeled as “Play direct from artist”, and the plays will scrobble via an improved player. As with the previous “full-length preview” option this replaces, these plays are considered promotional and don’t accrue royalties via the Artist Royalty Programme. In fact there are no further changes to the ARP, so as before, plays on Last.fm radio will continue to earn royalties. We hope this helps close the scrobbling loop for artists who can’t easily get exposure for their music on other services.
 

We’ll also be looking at other improvements for music manager artists, including better ways of showcasing promoted tracks on your artist profile and the ability to link your account to content you may host elsewhere. However our top priority is to get the work done on the new “Play direct from artist” feature and release it as soon as possible. The next available slot to go live with this functionality is in the third week of May. Pushing new updates live on a site with more than 40 million users a month takes time — not just for development but also QA, testing, and translation into 12 languages!


We’ll follow up this post in a couple of days with another one where Matt Knapman will explain a little more about the development process at Last.fm, how we gathered your feedback, and a little about plans for community support as a whole. In the meantime, keep it coming.

April 21 update:
We’ve posted an official announcement in response to feedback, check it out here!

—

A few weeks ago, Adrian used this space to introduce music application developers to the next generation of our Scrobbling APIs and ask the question “But does it scrobble?”. Meanwhile, Stefan has been spotted at SXSW and around Last.HQ wearing this t-shirt:

Why the recent fuss over our favourite neologism? Well, scrobbling forms the basis of one of our core missions: helping users make the most of their music listening across a variety of services and platforms by saving it all in one place, their Last.fm profile. We’ve been at this a while and have spent the last seven years working hard on making it more useful to wield a Last.fm profile through features like personalized music recommendations and concert listings, radio streams, a rich API, and different ways to share, reflect, and show off your own music taste as well as that of your friends.

Now, on the eve of receiving our 40 billionth scrobble, we’re ready to start rolling out some changes that focus us even more firmly on this mission. We think it’s more relevant in 2010 than ever before.

The first of these goes live today! But before we fully integrate these new ideas, we’re trying a few out in beta form on our track pages. Here’s a quick look at what’s new — for more detail and to leave feedback, check out this post on our forums.

Closing the loop with partner playlinks

The list of music players and devices that scrobble has been growing steadily since 2003, and we now have over 600 scrobbling partners. One of the coolest things to happen in online music over the last couple years is the emergence of “cloud” services, applications that make it easier than ever before to play whatever you can think of, whenever you want.

As passionate, tech-savvy music fans, Last.fm users are among the first to adopt new music technologies. Accordingly, many of these new services are now adding — or have already added — scrobbling support. Our scrobbling data shows that, for some time now, people have been using multiple music services and devices, then coming back to their Last.fm profiles to answer the question “what should I hear next?” and to see / show off all their listening united in one place.

We think this is great. Today we’re taking the first steps in “closing the loop” and nurturing this usage even further by adding links on our track pages to some of our favourite music services that scrobble.

Example partner links on the new Last.fm track page beta, click for full screenshot

Our beta launch partners for this integration are Spotify (in the UK, France, Spain, and most of Scandinavia), MOG (in the US), and The Hype Machine (worldwide). We’re also working on connecting our users to more great services soon, including We7 (UK), VEVO (US/Canada, and our first 3rd-party video scrobbling partner!), and others.

No two streaming services are alike; they all have different user experiences, geographic restrictions, catalogues and specialisations. We’re starting small today, but our vision is for Last.fm to efficiently connect any user to ALL of the relevant streaming options in their country for every track we know about, as well as being able to personalise listening preferences Last.fm-wide.

Some of our partners

These changes also mean that we are retiring our own on-demand track streaming, which we’ve run for the last two years in the US, UK, and Germany. Consistent with how people have used Last.fm since the early days, engagement with features like user profiles and personalised radio stations has remained much higher than usage of on-demand playback. We feel strongly that we can better fulfill our core mission by instead connecting our users to services in the ecosystem that, unlike us, focus primarily on a jukebox-in-the-sky streaming experience. Similarly, we’ll no longer be hosting music videos ourselves, instead integrating video from Youtube (now in higher quality!) and soon, VEVO.

We believe that this renewed focus on Last.fm as the definitive online home of your music taste and your base for music discovery – regardless of where you listen – will help improve not just our users’ musical lives but the overall online ecosystem as well.

Making scrobbles work harder

Unlike almost any other music service out there, we’re built and curated entirely by you, the music listener. Because of scrobbling, Last.fm knows 40 billion different things about music and that knowledge is growing every day. So as part of the track page beta we’re trying out some new ways of representing that information, on both the global and personal levels.

We’re just getting started with this – now we need your feedback so that we can roll this and related functionality out across the rest of the Last.fm website and beyond.

We want to hear more from you – who you’d like us to partner with, cool new services, data visualisation and feature ideas, and any other thoughts you have about the changes. See you on the forums!

—

Update: To clarify, today’s changes don’t affect Last.fm radio stations at all, just single-track playback on the website. Radio is one of our flagship music discovery features and has long been the most popular way to stream music on Last.fm; we’ll keep improving it in future. Play on!

It feels like just the other week that I posted this on the Last.fm developer forum to get feedback and ideas on a new version of our scrobbling API that we were mulling over.

For those less technically-inclined, the scrobbling API defines how data gets transmitted to Last.fm every time you listen to a song. Scrobbles are incredibly important to us. They’re the building blocks of your music profile, and put together, they power basically everything that Last.fm knows about music.

The current API does a decent enough job, but we’ve had many developers complain about it being inconsistent with the other ways of accessing Last.fm data (via our web services) and its rather shoddy feedback on errors.

We also wanted to make the API more extensible so that we could define certain information which must always be submitted (like track and artist name) while allowing us to provide extra functionality in future via optional fields that wouldn’t break existing scrobblers.

600 ways to scrobble

Our scrobbling servers get a lot of traffic – at certain times of the day we have nearly 800 people telling us what they are listening to every second, and we are nearing our 40 billionth scrobble! There are also many different ways to scrobble the music you’re hearing, some developed by us (such as our official Last.fm, Android, and iPhone apps) as well as applications developed by third parties and music-loving geeks from all over the world.

Scrobbles-per-second monitor in the Last.fm operations room, powered by CactiView.

All told we have more than 600 scrobblers created by people other than us, covering popular online services like Spotify and The Hype Machine, hardware devices like the Onkyo TX-NR807 and the Logitech Squeezebox, as well as online storage services like Bitspace, extensions for browsers like Chrome (via Chrome Scrobbler), Opera (via Seesu) and Firefox (via FoxyTunes), and finally, for the real geeks, plugins for Gnome’s totem player and a promising-looking fork of Amarok called Clementine, to name just a few. A fair share of all existing scrobblers is listed on build.last.fm – browse around if you’re curious!

Preparing a new version of the scrobbling API

Given the heavy use of the current scrobbling API, releasing a new version of it is not something we take lightly – which is why it’s taken more than a year to get to where we are today. My post back in January 2009 generated pages of suggestions, plenty of e-mail conversations with developers and led to many hours of internal discussions and arguments involving nearly everyone in the company in some way.

We are finally able to unveil our first draft of what the new API might look like. Please bear in mind that this is not complete or final; we’re releasing it as a “request for comment” from the developer and user community. All the technical details can be read on our forum here and we’d like to keep detailed discussion there. We’ll be monitoring the post and taking feedback onboard.

Here’s a summary of just some of the highlights planned for the new API:

• The scrobbling API will become a fully-fledged member of the Last.fm Web Services under a new “Scrobble” package joining its friends Track.love and Track.ban instead of being all sad and lonely on the sidelines. This should simplify things for developers by having one unified authentication, request and response mechanism. We also hope that this will lead to applications which currently just scrobble to use the rest of our API and vice-versa, with the end result being cooler apps with more features for everyone.
• Migrating to the web services will improve our ability to track the use of scrobble applications, so we can do groovy things like charts of the most popular scrobblers, and analyses of musical tastes across different scrobblers. Yes, we will finally be able to answer the burning question – “Do Amarok users have better taste than XBox Live users?” We hope that our scrobbling partners and their users will be able to do cool things with this data.
• Corrections information will be returned where relevant so users can be prompted to fix any incorrect metadata they may have.
• Changes to Last.fm radio scrobbling will allow us to improve our recommendations. We’ll get more specific listener feedback because loves, bans and skips can be tied to a specific radio stream, not just to a particular track.
• We’ll return more detailed error messages which should simplify the process of developing a scrobbler.
• Third party developers will be able to upload their own icons which will show up on a Last.fm user’s profile when they are listening with a particular scrobbler. We currently provide this as a service for our most popular scrobblers but will extend this to all third party apps (this was our most requested feature after improved error logging!).

There were a lot of great ideas which didn’t make the cut, but the new API should allow us to add new features more easily and we plan to expand on this release in the future. After a round of feedback from the community we hope to put a beta version of the API out for testing and will then work towards finalising it a month or two after that. Third party developers will then be able to start updating their existing applications (or writing new ones) and passing the benefits of the new features on to you, our faithful users.

We’re hoping that by making scrobbling development easier we will be taking more steps towards getting every musical device on the planet scrobbling. Let us know what you think.

Being a sysadmin at Last.fm can be a rather tough and at times unrewarding job. It involves maintaining hundreds of servers that get hammered by millions of users every single day, and to complicate things even further they’re spread across three different datacentres. Since Laurie is currently our longest-serving sysadmin, he’s already shed substantial amounts of blood, sweat and tears keeping Last.fm running smoothly. So of course I made him wet his pants the other day by firing up a Hadoop job while he was sitting next to the nodes in the datacentre. Having started to feel a bit of remorse, I thought it’d be only right to finally give him back some love.

Luckily, making Laurie happy isn’t that hard — all it really takes is showing him some graphs. It’s not an accident that Google returns him as top result for the “if it moves, graph it” saying. He even showcased his obsession with graphs over the Xmas holidays, using his Cacti skills to graph parking availability at large UK shopping centres.

Graphs are clearly Laurie’s raison d‘être, so it didn’t take me long to figure out that a great way of thanking him would be to write some code that does something we’ve been working towards for some time at Last.fm: generating personalized, real-time scrobbling history graphs. And while I was at it, I turned the code into a Playground demo that we’ve made available not only to subscribers, but to anyone who has ever scrobbled a track.

Here’s the first graph Laurie saw when checking out the Scrobbling Timeline demo:

Apparently, he recently reached his 20,000th scrobble milestone by listening to a Geek Rock track. By clicking on the cumulative check box, he was able to transform this graph into the following one:

Looks like he’s been scrobbling pretty steadily, with a minor drop in spring/summer for the last 3 years. (At this point he said “wow, that’s cool!”, which made me feel all warm and fuzzy inside.)

I didn’t really keep track of what else Laurie did with the demo, but presumably he discovered that he could restrict the graphs to a specific artist and that it was also possible to compare his scrobbling timelines with those of his friends. In any case, he seemed to like it. Hopefully you’ll like it too, and please let us know why if you don’t!

Last weekend I was lucky enough to voyage to Stockholm with Jonty and Michael to represent Team Last.fm at Music Hack Day.

Music Hack Day’s premise is simple – find the best and brightest tech and music geeks, get them all together for a weekend, mix in APIs and workshops from every online music service worth its salt, and then spend 24 hours making… well, anything!

Started by Soundcloud’s Dave Haynes here in London last July, subsequent Music Hack Days in Berlin and Boston have cemented their reputation as the best tech events going, music or no. (Anthony from the Hype Machine did a nice write-up on some of the ingredients that make them great.)

So it was with some excitement that we boarded our plane on Friday and headed north. In addition to our standard hack day paraphernalia — laptops, check; giant headphones, check; world’s tiniest Guitar Hero, check — we also carted along some limited edition stickers, newspapers, and a short presentation on the venerable Last.fm API. (You can grab those slides here as a PDF download.)

Photo by Brian Whitman.

Stockholm certainly didn’t disappoint — the weekend was awesome! We came, we hacked, we even conquered.

We also learnt a lot. Some highlights included…

Swedish hospitality

“Hospitality” isn’t generally up there on the list of familiar Swedish traits (unlike, say, tasteful flat-pack furniture, or expensive booze). But our hosts — Henrik and Mattias — made everyone feel welcome and created an environment that let everyone just get on with creating cool stuff.

There were some uniquely Swedish touches too, like the delicious bread and cheese breakfasts and the snow-based beer fridge. Oh, and the Batmobile showed up. No, really.

APIs in the mirror

Though we’ve offered public APIs to developers since 2003, nothing makes you see them in a new light like face-to-face interaction with people trying to make clever and unusual things with them. We’ve come back to London with a long list of suggested improvements, things that could be clearer in the docs, and even a couple of bug fixes that were reported by intrepid Stockholm hackers. Thanks to everyone who spoke to us!

We also handed out free subscriptions to everyone who demo’d a hack that used the Last.fm API.

A few of our favourites:

• My City vs. Your City Uses our new geo.getMetro* city charts API to compare top artists across hundreds of cities worldwide. Neat!
• SimilarArtists A simple way to generate Spotify playlists of recommended music based on Last.fm similar artists.
• Holodeck An attractive way to create an artist website that based on content from SoundCloud, Last.fm, Songkick, and Tumblr.
• Mashboard A dashboard for your Soundcloud tracks that pulls in rich audio metadata from Echo Nest. And it scrobbles!

We also managed to sneak in a few hacks of our own:

• HacKey Ever wondered what your favourite key is? Thanks to the Last.fm and Echo Nest APIs, now you can find out.
• ProximRadio and Blobble Jonty and Michael came up with a deadly trio of new tech that enables a long-standing dream: proximity-based multi-profile radio stations, complete with group scrobbling. Whoa.

A complete list of hacks is available here.

The online music ecosystem = crazy delicious

It was humbling to be in the presence of so many talented companies and developers, from the music mad scientists of The Echo Nest to the streaming wizards of Spotify, not to mention entire teams who travelled to Stockholm from Songkick, Soundcloud, and many others.

It’s pretty clear that 2010 is going to be an exciting year in music and tech. (And not just because people are building Playdar-enabled beatmatched collaborative Spotify playlist generators that scrobble via robot arm attachments…although that helps.) Team Last.fm will be in attendance the next Music Hack Day and also at some events of our own, so stay tuned.

Until then, happy hacking!