Last.fm – the Blog

About 6 weeks ago I started a short internship at Last.fm. For my project I wanted to explore Last.fm’s data to learn how listening preferences vary with the listener’s age and gender. Apart from the science, the most important thing I found is that you can make awesome plots with this information.

I started by making a chart to show what kind of music you “should” be listening to if you really want to fit in with the most common artists in your age range and gender:

The sizes of the artists’ names indicate how popular they are, while their position shows the gender mix and average age of their listeners. Based on the positions of the larger names, it’s already obvious which age category is most common amongst Last.fm users.

So, you can now use this plot to decide which music you might want to listen to. For example, if you are a healthy young male in your early twenties, you probably should listen to bands such as Iron Maiden and Metallica. Gorillaz and Radiohead might just be acceptable. If you get older you can then switch to artists like Neil Young and Genesis. It’s all quite obvious really.

Of course, when I realized what nice plots I could make, I tried it on several other types of data as well. Tags for example:

You can use it in the same way as the previous plot. Apparently females like using band names as tags (Super junior, McFly), while males prefer finding lots of ways to say the same thing (metal, jazz). Most importantly we have just used science to prove that men don’t listen to much k-pop.

Obviously music is the most important data that’s available at Last.fm, but there are some other profile items that can be interesting too. The words used in the ‘About Me’ section on users’ profile pages might even lead to the most interesting plot of them all:

There are actually so many fun facts about this plot that it’s just best to check it out yourself. The most obvious one is which hobbies you “should” have depending on your gender. Or you can find out at what age you should retire.

I used all of this to create a fun new playground demo that enables all Last.fm users to compare themselves with their friends. This is the plot for the data and recommendations team for example:

We’ve even thought of those of you who like to print their visualisations as a poster by providing a bigger PDF version that has more artist names on it.

Hopefully you’ll enjoy this demo as much as we did. In any case, we’d love you all to let us know what you think.

A bit less than a year ago we launched the VIP zone on our Playground, with the promise that we would keep adding fancy visualizations to it as a special treat for our loyal subscribers. We already delivered on this promise with the personalised Listening Trends and Music Universe visualisations, and today we’re delivering some more.

This time around we got inspired by the WOMRAD 2010 paper Rocking around the clock eight days a week: An exploration of temporal patterns of music listening. By applying some nifty circular statistics formulas, we managed to create an interesting new visualisation that shows at what times of the day a given Last.fm subscriber has been listening to music over a certain time period. Here’s an example:

In this case we’re looking at Norman‘s listening behaviour for the past 90 days. Red and green represent weekdays and weekends, respectively, and the longer the hand the more the listening was focused around the time to which it points. Generally speaking, Norman seems to listen to music at later times of the day in weekends than on weekdays, and his listening seems to be less restricted to certain hours in the weekend. It’s also quite clear that he tends to listen to music from 10AM to 7PM on weekdays, which isn’t that much of a surprise since those are our working hours here at Last.fm. He accidentally left his radio playing overnight a few times though, as indicated by the smaller red bars from 8PM until 9AM.

Our beloved LAST.HQ‘s listening clock for the same time period is a more extreme example:

Since we use this account for the reception radio in our offices — which plays pretty much 24/7 — the listening is spread out across all times of the day, leading to two hands that are extremely tiny and cute.

We very much hope you’ll enjoy playing around with this new visualisation, and that some of you will point to particularly interesting listening clocks or discuss potential improvements in our Playground forums. Meanwhile, we’ll start working on the next one!

In our ongoing effort to make Last.fm the best place for music data, we’ve been using audio fingerprint tech since 2007 to match metadata information (i.e. artist and track names) by analyzing the audio.

This helped us automate the process of spelling correction, and as by-product we released an open source library and a client which can be used to identify the correct metadata for mp3s.

We have recently reached the milestone of 90 million fingerprint entries and all this information is now fully accessible via a new track.getFingerprintMetadata call in our official API, with musicbrainz ids and full UTF-8 support.

If you are a developer you might want to look into the library module (which generates the fingerprint) available on github (and it’s LGPL!). If you just want to play with it, you can use our sample client (available for Windows, OS X and Linux), which query the fingerprint service for an id, then pokes the API for the metadata.

Running the fingerprint client is a piece of cake. Just type

lastfmfpclient mysterious.mp3

and it will generate the following xml output (which is taken from the API):

<?xml version=“1.0” encoding=“utf-8”?>
<lfm status=“ok“>
  <tracks>
    <track rank=“1“>
      <name>One Summer’s Day</name>
      <mbid></mbid>
      <url>http://www.last.fm/music/%E4%B9%85%E7%9F%B3%E8%AD%B2/_/One+Summer%27s+Day</url>
      <streamable fulltrack=“0“>0</streamable>
      <artist>
        <name>久石譲</name>
        <mbid>44c64a30-1d58-49c5-b314-6e02fba49526</mbid>
        <url>http://www.last.fm/music/%E4%B9%85%E7%9F%B3%E8%AD%B2</url>
      </artist>
      <image size=“small“>http://userserve-ak.last.fm/serve/34s/3737553.jpg</image>
      <image size=“medium“>http://userserve-ak.last.fm/serve/64s/3737553.jpg</image>
      <image size=“large“>http://userserve-ak.last.fm/serve/126/3737553.jpg</image>
      <image size=“extralarge“>http://userserve-ak.last.fm/serve/300×300/3737553.jpg</image>
    </track>
    <track rank=“0.404785“>
      <name>Ano Natsu he</name>
      <mbid></mbid>
      <url>http://www.last.fm/music/%E4%B9%85%E7%9F%B3%E8%AD%B2/_/Ano+Natsu+he</url>
      <streamable fulltrack=“0“>0</streamable>
      <artist>
        <name>久石譲</name>
        <mbid>44c64a30-1d58-49c5-b314-6e02fba49526</mbid>
        <url>http://www.last.fm/music/%E4%B9%85%E7%9F%B3%E8%AD%B2</url>
      </artist>
    </track>
    <track rank=“0.144543“>
      <name>あの夏へ</name>
      <mbid></mbid>
      <url>http://www.last.fm/music/%E4%B9%85%E7%9F%B3%E8%AD%B2/_/%E3%81%82%E3%81%AE%E5%A4%8F%E3%81%B8</url>
      <streamable fulltrack=“0“>0</streamable>
      <artist>
        <name>久石譲</name>
        <mbid>44c64a30-1d58-49c5-b314-6e02fba49526</mbid>
        <url>http://www.last.fm/music/%E4%B9%85%E7%9F%B3%E8%AD%B2</url>
      </artist>
      <image size=“small“>http://userserve-ak.last.fm/serve/34s/6624395.jpg</image>
      <image size=“medium“>http://userserve-ak.last.fm/serve/64s/6624395.jpg</image>
      <image size=“large“>http://userserve-ak.last.fm/serve/126/6624395.jpg</image>
      <image size=“extralarge“>http://userserve-ak.last.fm/serve/300×300/6624395.jpg</image>
    </track>
    …
  </tracks>
</lfm>

Now that we’ve seen the return of full length previews for independent artists, we wanted to go a step further and build some new discovery tools to help introduce users to emerging artists offering direct-from-artist tracks, and vice versa.

After some late nights in the Last.fm MIR laboratory, we’ve come up with a dedicated direct-from-artist recommendations page on our playground, where you can play a personalised selection of full length tracks.

Check out your own recommendations:
http://playground.last.fm/demo/directrecs

Please try it out and let us know what you discover! We’re going to keep working on this functionality and building it into the rest of Last.fm, so your feedback can help us improve.

Along with our support for Music Hack Days, the Last.fm team often gets involved with the MIR (Music Information Retrieval) community, a group at the forefront of recommendation technology.

This year we are helping organizing a music recommendation workshop within RecSys entitled WOMRAD 2010. The topics of interests vary from music similarity measures, recommendation, music cognition, user modeling and many more.

If you are a researcher in the field or you are just interested in the topic, join us in Barcelona this September the 26th to meet and exchange ideas!

Being in charge of the database that stores everyone’s scrobbles is a great responsibility and one I don’t take lightly. One thing that I get asked now and again is, “I’ve accidentally deleted Artist X from my profile, can you restore it please?”. Sheesh, the work of a Scrobble Overlord is never done! :)

Well, the simple answer is, “yes I can” but up until now I’ve not had anything that allows it to be done with the click of a button or two. I decided enough was enough and the result is this Playground app.

Ladies and gentlemen, I bring you the self-service Artist & Track Restore Tool.

Okay, so just a very brief explanation of how it works (skip this if you don’t care). When you delete anything from your library we don’t actually delete it, we move it to a different table in your database. We need to do this so we can keep track of what you’ve deleted to make sure we don’t keep offering it to you as a recommendation.

All this Playground app does is allow you to review what’s in your deleted tracks table and if you choose to restore them it just puts them back into your main scrobble table.

One thing to be mindful of is that your playcount may not update immediately since it is cached. This will auto-correct within approximately 24 hours.

Please note that this tool cannot be used to restore a deleted or wiped profile as can be found in the account settings… where erase really means just that!

I think this is a useful tool and hopefully you will too, but please let us know why if you don’t!

My first festival was Lollapalooza 1995. I had to go because my favorite band, Sonic Youth, was headlining (they’re still my favorite band). I lived in Houston and the festival was in Austin but I was too young to drive so my mom made the two hour trip so that I could attend with two friends. A transcendental 20+ minute performance of “The Diamond Sea” was the highlight of the whole festival. Also playing that year were Moby (way before he hit it big with his album Play), Cypress Hill, Blonde Redhead, Built to Spill, Mike Watt, The Roots, and Yo La Tengo.

Photo: Russ Garrett

At Last.fm we know from experience that many of the best music memories happen when you’re rocking out in a field with thousands of other sunburnt/soaking wet compatriots. So today we are excited to launch Last.fm Festivals, your personalized guide to festivals around the world. For the first time you can easily browse upcoming festivals in your town, your country or beyond, based on your personal music profile. Choose your location and sort the results either by date or by compatibility.

Once you’ve found your perfect festival, check out the festival page where Last.fm looks at the line-up for the festival and compares it to your Last.fm profile to generate a personal line-up recommendation list so you can be sure you don’t miss a thing. You can also browse the line-up by genre to see the artists that you might like but aren’t part of your Last.fm profile yet. Plus you can start a festival radio station to listen to all the artists playing.

Check it out: http://www.last.fm/festivals

This is just the beginning. Over the next month we’ll be rolling out additional features to help you find up and coming artists you should check out (so you can say you saw them before they were big), find festival buddies and easily share your festival picks.

Read more about all the new features on the forum post and let us know what you think.

Since last week’s announcement of our track page beta and streaming changes we’ve been receiving tons of feedback. Responses have come via this blog, our discussion forums, and elsewhere, and we’ve been paying attention to both the positive and the negative. The fact that so many of you are taking the time to offer thoughts and constructive criticism means a lot to us.


The team spent the better part of last week looking at a few different improvements we might be able to introduce in response to some of this feedback. In particular, we first wanted to address the requests from the many independent artists who use our Music Manager to upload their tracks directly onto Last.fm.



Our renewed focus on the features that make Last.fm unique — scrobbling, personalised radio, and being the online home for your music taste –– means we won’t be returning our own on-demand track offering. Instead, based on your suggestions, we’ll be connecting people to more streaming services in more countries by the end of the year, partners that focus squarely on providing the best on-demand experience possible and scrobbling it. We also have plenty of new music discovery and exploration improvements coming to the Last.fm website and beyond, helping make your scrobbles even more useful to you.


However, today we’re happy to announce that unsigned artists who hold global rights to their music will soon be able to again offer tracks in full to their fans via their track pages. For the first time, this option will be clearly labeled as “Play direct from artist”, and the plays will scrobble via an improved player. As with the previous “full-length preview” option this replaces, these plays are considered promotional and don’t accrue royalties via the Artist Royalty Programme. In fact there are no further changes to the ARP, so as before, plays on Last.fm radio will continue to earn royalties. We hope this helps close the scrobbling loop for artists who can’t easily get exposure for their music on other services.
 

We’ll also be looking at other improvements for music manager artists, including better ways of showcasing promoted tracks on your artist profile and the ability to link your account to content you may host elsewhere. However our top priority is to get the work done on the new “Play direct from artist” feature and release it as soon as possible. The next available slot to go live with this functionality is in the third week of May. Pushing new updates live on a site with more than 40 million users a month takes time — not just for development but also QA, testing, and translation into 12 languages!


We’ll follow up this post in a couple of days with another one where Matt Knapman will explain a little more about the development process at Last.fm, how we gathered your feedback, and a little about plans for community support as a whole. In the meantime, keep it coming.

April 21 update:
We’ve posted an official announcement in response to feedback, check it out here!

—

A few weeks ago, Adrian used this space to introduce music application developers to the next generation of our Scrobbling APIs and ask the question “But does it scrobble?”. Meanwhile, Stefan has been spotted at SXSW and around Last.HQ wearing this t-shirt:

Why the recent fuss over our favourite neologism? Well, scrobbling forms the basis of one of our core missions: helping users make the most of their music listening across a variety of services and platforms by saving it all in one place, their Last.fm profile. We’ve been at this a while and have spent the last seven years working hard on making it more useful to wield a Last.fm profile through features like personalized music recommendations and concert listings, radio streams, a rich API, and different ways to share, reflect, and show off your own music taste as well as that of your friends.

Now, on the eve of receiving our 40 billionth scrobble, we’re ready to start rolling out some changes that focus us even more firmly on this mission. We think it’s more relevant in 2010 than ever before.

The first of these goes live today! But before we fully integrate these new ideas, we’re trying a few out in beta form on our track pages. Here’s a quick look at what’s new — for more detail and to leave feedback, check out this post on our forums.

Closing the loop with partner playlinks

The list of music players and devices that scrobble has been growing steadily since 2003, and we now have over 600 scrobbling partners. One of the coolest things to happen in online music over the last couple years is the emergence of “cloud” services, applications that make it easier than ever before to play whatever you can think of, whenever you want.

As passionate, tech-savvy music fans, Last.fm users are among the first to adopt new music technologies. Accordingly, many of these new services are now adding — or have already added — scrobbling support. Our scrobbling data shows that, for some time now, people have been using multiple music services and devices, then coming back to their Last.fm profiles to answer the question “what should I hear next?” and to see / show off all their listening united in one place.

We think this is great. Today we’re taking the first steps in “closing the loop” and nurturing this usage even further by adding links on our track pages to some of our favourite music services that scrobble.

Example partner links on the new Last.fm track page beta, click for full screenshot

Our beta launch partners for this integration are Spotify (in the UK, France, Spain, and most of Scandinavia), MOG (in the US), and The Hype Machine (worldwide). We’re also working on connecting our users to more great services soon, including We7 (UK), VEVO (US/Canada, and our first 3rd-party video scrobbling partner!), and others.

No two streaming services are alike; they all have different user experiences, geographic restrictions, catalogues and specialisations. We’re starting small today, but our vision is for Last.fm to efficiently connect any user to ALL of the relevant streaming options in their country for every track we know about, as well as being able to personalise listening preferences Last.fm-wide.

Some of our partners

These changes also mean that we are retiring our own on-demand track streaming, which we’ve run for the last two years in the US, UK, and Germany. Consistent with how people have used Last.fm since the early days, engagement with features like user profiles and personalised radio stations has remained much higher than usage of on-demand playback. We feel strongly that we can better fulfill our core mission by instead connecting our users to services in the ecosystem that, unlike us, focus primarily on a jukebox-in-the-sky streaming experience. Similarly, we’ll no longer be hosting music videos ourselves, instead integrating video from Youtube (now in higher quality!) and soon, VEVO.

We believe that this renewed focus on Last.fm as the definitive online home of your music taste and your base for music discovery – regardless of where you listen – will help improve not just our users’ musical lives but the overall online ecosystem as well.

Making scrobbles work harder

Unlike almost any other music service out there, we’re built and curated entirely by you, the music listener. Because of scrobbling, Last.fm knows 40 billion different things about music and that knowledge is growing every day. So as part of the track page beta we’re trying out some new ways of representing that information, on both the global and personal levels.

We’re just getting started with this – now we need your feedback so that we can roll this and related functionality out across the rest of the Last.fm website and beyond.

We want to hear more from you – who you’d like us to partner with, cool new services, data visualisation and feature ideas, and any other thoughts you have about the changes. See you on the forums!

—

Update: To clarify, today’s changes don’t affect Last.fm radio stations at all, just single-track playback on the website. Radio is one of our flagship music discovery features and has long been the most popular way to stream music on Last.fm; we’ll keep improving it in future. Play on!

It feels like just the other week that I posted this on the Last.fm developer forum to get feedback and ideas on a new version of our scrobbling API that we were mulling over.

For those less technically-inclined, the scrobbling API defines how data gets transmitted to Last.fm every time you listen to a song. Scrobbles are incredibly important to us. They’re the building blocks of your music profile, and put together, they power basically everything that Last.fm knows about music.

The current API does a decent enough job, but we’ve had many developers complain about it being inconsistent with the other ways of accessing Last.fm data (via our web services) and its rather shoddy feedback on errors.

We also wanted to make the API more extensible so that we could define certain information which must always be submitted (like track and artist name) while allowing us to provide extra functionality in future via optional fields that wouldn’t break existing scrobblers.

600 ways to scrobble

Our scrobbling servers get a lot of traffic – at certain times of the day we have nearly 800 people telling us what they are listening to every second, and we are nearing our 40 billionth scrobble! There are also many different ways to scrobble the music you’re hearing, some developed by us (such as our official Last.fm, Android, and iPhone apps) as well as applications developed by third parties and music-loving geeks from all over the world.

Scrobbles-per-second monitor in the Last.fm operations room, powered by CactiView.

All told we have more than 600 scrobblers created by people other than us, covering popular online services like Spotify and The Hype Machine, hardware devices like the Onkyo TX-NR807 and the Logitech Squeezebox, as well as online storage services like Bitspace, extensions for browsers like Chrome (via Chrome Scrobbler), Opera (via Seesu) and Firefox (via FoxyTunes), and finally, for the real geeks, plugins for Gnome’s totem player and a promising-looking fork of Amarok called Clementine, to name just a few. A fair share of all existing scrobblers is listed on build.last.fm – browse around if you’re curious!

Preparing a new version of the scrobbling API

Given the heavy use of the current scrobbling API, releasing a new version of it is not something we take lightly – which is why it’s taken more than a year to get to where we are today. My post back in January 2009 generated pages of suggestions, plenty of e-mail conversations with developers and led to many hours of internal discussions and arguments involving nearly everyone in the company in some way.

We are finally able to unveil our first draft of what the new API might look like. Please bear in mind that this is not complete or final; we’re releasing it as a “request for comment” from the developer and user community. All the technical details can be read on our forum here and we’d like to keep detailed discussion there. We’ll be monitoring the post and taking feedback onboard.

Here’s a summary of just some of the highlights planned for the new API:

• The scrobbling API will become a fully-fledged member of the Last.fm Web Services under a new “Scrobble” package joining its friends Track.love and Track.ban instead of being all sad and lonely on the sidelines. This should simplify things for developers by having one unified authentication, request and response mechanism. We also hope that this will lead to applications which currently just scrobble to use the rest of our API and vice-versa, with the end result being cooler apps with more features for everyone.
• Migrating to the web services will improve our ability to track the use of scrobble applications, so we can do groovy things like charts of the most popular scrobblers, and analyses of musical tastes across different scrobblers. Yes, we will finally be able to answer the burning question – “Do Amarok users have better taste than XBox Live users?” We hope that our scrobbling partners and their users will be able to do cool things with this data.
• Corrections information will be returned where relevant so users can be prompted to fix any incorrect metadata they may have.
• Changes to Last.fm radio scrobbling will allow us to improve our recommendations. We’ll get more specific listener feedback because loves, bans and skips can be tied to a specific radio stream, not just to a particular track.
• We’ll return more detailed error messages which should simplify the process of developing a scrobbler.
• Third party developers will be able to upload their own icons which will show up on a Last.fm user’s profile when they are listening with a particular scrobbler. We currently provide this as a service for our most popular scrobblers but will extend this to all third party apps (this was our most requested feature after improved error logging!).

There were a lot of great ideas which didn’t make the cut, but the new API should allow us to add new features more easily and we plan to expand on this release in the future. After a round of feedback from the community we hope to put a beta version of the API out for testing and will then work towards finalising it a month or two after that. Third party developers will then be able to start updating their existing applications (or writing new ones) and passing the benefits of the new features on to you, our faithful users.

We’re hoping that by making scrobbling development easier we will be taking more steps towards getting every musical device on the planet scrobbling. Let us know what you think.