Braindump

Music discovery tools

Much to my surprise there has been an upswing in the number of small and extremely useful music discovery sites or apps that I use.

Peel
There are obviously tons of great music blogs out there, ranging from the big established sites to the smaller more niche ones. I personally love the hand curated music selection but I don’t care so much for spending all that time reading the blogs themselves. Peel nicely solves this problem by parsing the blogs and then just provides a list of the tracks so you can listen to them directly. It’s also easy to download them, add to iTunes and sync them to my iPhone. Perfect for the way I want to interact with music blogs, easily worth the $25.



This is my jam
I think of This is my jam as Twitter but for music. It has a lot of the same components, you follow people, others follow you, but instead of tweets there are songs. The songs stay up for a week or shorter. As your friends add new songs they populate your timeline with deliciousness, similar to how the Twitter timeline works. Extremely handy for just playing through everything your friends are into right now.







8tracks
If This is my Jam is Twitter for music, 8tracks would be Tumblr. The long form of curated music. At its core 8tracks is a pretty straight forward playlist site. You upload songs, give the playlist a snazzy title and a picture. You follow people and all that jazz. It’s not as direct as just uploading a track to This is my Jam, which is both good and bad. Usually people put some more effort into their playlist, in particular so that the songs work together.




Later.fm
For some reason the following scenario happens to me a lot. I’m at work, I get a link from somewhere that leads to a couple of songs. I don’t have time to listen to them straight away or even download them. Instead I click on my “Listen later” bookmarklet in my browser. That quickly adds all of the songs to Later.fm so I can listen to it later. A smaller use case perhaps, but I have found it saves a lot of time.




Disclaimer: I know some of the authors of these tools.

Local pay as you go SIM cards with data

As previously mentioned I am thoroughly addicted to my iPhone. This manifests itself especially when abroad. During trips the need for information about transport, places and activities is even greater then usual. Unfortunately this greater need is efficiently suppressed by extreme roaming costs, that is, unless you get a local pay as you go SIM card with decent data rates.

Finding good rates abroad is usually tricky if you don’t know the local market and/or language. Scanning through various mobile phone forums can sometimes help but it is time consuming and stale information is commonplace.

To help resolve this issue I have put together a small wiki with information about sensible pay as you go data plans divided up by country. So far there’s only information about UK, US and Sweden in there but I welcome everyone to add more.

Check it out

Going cold turkey on physical media

With all them services and hardwares out there, is it possible to go cold turkey on physical media? Why make the digital switch? And if you do, is the selection good enough?

It’s heavy, I’m lazy
My generation is the first to have grown up with the internet and dirt cheap flights. I’m glad, since these two pillars of modern life has given me and others greater flexibility when deciding where to live. Moving to a new place, sometimes in new countries, has almost exclusively been a positive experience. However, it is not without drawbacks. Don’t worry, I’ll save my rant about British banks for another blog post.

One downside is the practical limit to the amount of stuff that can be hauled between flats, especially when moving internationally. While it might appear to be an insignificant detail it is having a big impact on how I think. All that stuff gathered through the years has started to become more of a burden than an asset. Every time the moving van is loaded there are more books, DVDs, games and CDs to fill it with, many of which have already been read, seen, finished or ripped to mp3. What previously would be a common impulse buy is now a major decision on par with “should I get that vasectomy from the village butcher or not?”

With yet another international move coming up the idea to go cold turkey on physical media has popped up once again, sandwiched in-between thoughts such as “What did I have for lunch today?” and “How you do spell the back-to-the-future-esque surname of that dark haired woman in Dirty projectors?”.



Digitally yours
Enough waffling. This is a quick experiment to see if going completely digital would be possible for me, your mileage may vary. I have compiled a list of books, films, games and albums that I will try to download, stream or by other means legally consume without buying the physical product.

The upsides to going digital are obvious, besides the mentioned reduction of shelf space required it promises instantish gratification. I want to listen to that song now, not next week when I have time to go to a record shop or when the delivery will arrive. Then there’s the ease of use aspect, mp3 files for example can be played on almost any device anywhere these days, not just the livingroom stereo. Same for many video formats, I was surprised to see that my dad’s new kitchen TV could play Xvid files from USB sticks.
There are unfortunately downsides too, such as DRM and other lock-in techniques.


Music
Music is the easiest of the four. Since it compresses well it has been a solved problem for quite some time, although mainly illegally so. 10 years later the legal options are catching up.
I picked out the top 15 tracks from different artists in my 3 month chart at Last.fm. I then simply searched for all of them and counted the results. If I only found the unremixed or original version of a track I marked that as half a point.

The tracks
The Radio Dept. – Heaven’s on Fire
Jonathan Johansson – Aldrig ensam
LCD Soundsystem – Dance Yrself Clean
Noir Désir – Le Vent Nous Portera
Phosphorescent – The Mermaid Parade
Foals – Spanish Sahara
Surfer Blood – Take It Easy
Familjen – När planeterna stannat
Stina Nordenstam – I Dream of Jeannie
Real Estate – Fake Blues
The National – Bloodbuzz Ohio
Teenage Fanclub – The World’ll Be OK (Alternative version)
She & Him – In the Sun
Teddybears ft. Eve – Rocket Scientist
Radiohead – Nude (Holy Fuck Remix)


Results
Spotify - 70%
iTunes - 86%
Amazon MP3 - 76%
Hype Machine - 80%
Combined result - 97%

Altogether a decent result, what some services lacked the others made up for. This might just work.





Games - Xbox 360
My platforms of choice are the Xbox 360 and PC. I picked out five fairly big and recent titles on each platform that have turned up on my gaming radar (also known as Gadar™).

Battlefield: Bad company 2
Call of duty: Modern warfare 2
Just cause 2
Alan Wake
Red dead redemption



Results
There is no need for a graph since the one place to buy these games online, Xbox Live Marketplace, did not have a single one of them when I checked. Disastrous! It mostly contains older titles. Does anyone know the reason why? If there is one thing iPhone apps have taught me it’s that I buy lots more when it’s just a click and a download away.


Games - PC
Battlefield: Bad company 2
Call of duty: Modern warfare 2
Just cause 2
Napoleon: Total War
Alien Breed: Impact

 
Results
Steam - 100%
Direct2drive - 20%
Combined result - 100%

Not much to say, Steam is where it’s at, not much point in looking elsewhere. The newly added Mac support is also neat for those of us that have tasted the forbidden fruit.


Films
Below are 10 recent films I like. Half mainstream-ish and half foreign productions.

Youth in revolt
An Education
Zombieland
The Road
District 9
Män som hatar kvinnor / The girl with the dragon tattoo (se)
O cheiro do ralo / Drained (br)
El secreto de sus ojos / The secret in their eyes (ar)
Død snø / Dead snow (no)
Le scaphandre et le papillon / The diving bell and the butterfly (fr)

Results
iTunes - 50%
Zune - 20%
Netflix - 30%
Hulu - 0%
Voddler - 20%
CDON - 60%
Random torrent site - 100%
Combined result (legal) - 70%






Better then expected, but not great. The movies are spread out over the services making them hard to find. Is there a search engine for this stuff yet? If not, someone should build it and tell me about it. In this test I also counted those from those services not strictly available in the EU.
Sadly it is still a lot less hassle to pirate a film than to watch it legally.


Books
In my mind the big contenders are Amazon’s Kindle and Apple’s iPad and their respective book stores, but I will admit I have not done much if any research. I picked a selection of mostly work related books, since that seems to be what I make time to read.

Programming Scala (Dean Wampler, Alex Payne)
The Ruby Programming Language (David Flanagan, Yukihiro Matsumoto)
Hadoop: The Definitive Guide (Tom White)
Confessions of a public speaker (Scott Berkun)
Atlas Shrugged (Ayn Rand)

Results
Amazon Kindle - 60%
iBooks - 0%
O’Reilly - 80%
Combined result - 80%

As it turns out most of what I read are O’Reilly books, so this test is quite skewed. Surprised by the lack of books in the iBooks store.




Summary
Doable but plenty of room for improvement.

The selection is lacking, especially when it comes to Xbox 360 games. Equally annoying as a customer is the need for looking through a handful of different services in order to find that one movie you want to see. Other then that both books and movies do alright. Music and PC games are the only two solid results with excellent coverage.

One can naively hope that as the content industry further feels the squeeze from piracy that they would feel inclined to actually give the customers what they want, the way they want it.

Contributing to Open street map with an iPhone

Open street map turned out to be a great help on the trip last year. So when I found out the area around where I live right now has quite poor coverage I decided to help out by adding a few roads. Here’s how you can do it with the help of an iPhone.

What is Open street map?

OpenStreetMap creates and provides free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive, or unexpected ways.

Gathering traces
There are many ways of contributing to OSM. Since I wanted to gather information on road using the iPhone GPS I started looking for a suitable app. There are a few but the best one for this purpose seems to be the OSMTrack app.

To use it simply download the app and create an OSM account. Before you set off just click the start button and start driving around. Once you are finished click stop and then the upload button to send the collected traces to your OSM account.



Finding the trace
Once safely back near a computer you log in to Open street map and click your username in the top right corner and then “my traces”. This gives you a list of all the collected tracks/traces. Click edit on one of them, now the real map editing begins. There are a few different ways to do it but I did the following.

Editing the map
Click “Convert GPS track to ways” and “Edit with save”. This ought to convert your collected points into a red way. I have noticed that it sometimes refuses to do so, don’t know why.

Next I removed the parts of the way that is already in OSM. To do so click on one of the unwanted points and press delete until you are happy. Pressing enter will stop the editing and escape will remove the way completely. Shift+click on another way will connect the two, used to create intersections for example. Pressing X will split the way in two. Perhaps you want to split a way into two different roads. For more information see the Potlatch/Primer.

Now you ought to have what looks like one or more roads added to the map. Before they will show up you have to mark them with the specific type of road it is. There are a bunch of great examples here. To change the type click on the way and then on the drop down at the bottom of the page that says “(no preset)”.

Once you are done don’t forget to click the save button at the bottom right of the page.

That’s it! You are now a contributing member of society, congrats!

Oh, one more thing. I found OpenMaps to be a great OSM viewing app with offline capabilities for the iPhone.

Travel in the 21st century

I am currently on a trip through parts of south east asia and the US with Steve Gravell and Tobias Köppen. Naturally I couldn’t leave my Eee PC or iPhone at home, so during the trip I have gathered a few tips for the geeky traveler. Nothing revolutionary, but I found these apps and websites to be quite helpful. In my usual manner I also complain a bit and suggest improvements. If you have any additional recommendations please add a comment.

Maps
Unfortunately the holy trinity of Google maps, GPS and an unlimited data plan is one short when abroad. Without a data plan you are plunged back into the dark ages again and roaming is not an option. Luckily there is an iPhone app called oMaps that allows you to mark certain areas on the map for offline use. Perfect for getting the city map while you are in a hotel room near wifi, mark a few places you want to visit and off you go.
oMaps uses OpenStreetMap as the data source, the license used enables the offline mode. OpenStreetMap builds its map from user contributions so some areas have excellent coverage whilst others are lacking. You can help out by contributing straight from your iPhone, although I had limited success with that so far.

Music
Music is essential on a trip, perfect for all those long bus, plane and boat rides. I love the Last.fm iPhone app but unfortunately we are once again back to the problem of not having a data plan and no offline mode. I decided to give the Spotify app a go. It allows you to select playlists that will be available in offline mode. I can easily search for and add new albums as I go on the trip and have them synced while I am near wifi. The price is quite hefty at €9.99 a month but worth it when on the move.
As many others I would kill for being able to run the app in the background. When me and Steve drove to Napa valley I was using the phone to navigate, which meant we couldn’t listen to my music at the same time. It was also the most ass-clinching car ride in a long long time, but that’s another story. While I am at it, how about adding some sort of playlist rss subscription feature to the app? For example I read the music blog Let’s pretend we’re bunny rabbits. They regularly post playlists with a spotify link and I’d like to see that automatically pop up in the Spotify app.

Steve and I were having a few beers in a horrible bar in Kuala Lumpur when the DJ announced that the next song was part of a competition. Figure out what song it is and you win two pitchers of beer. Steve had the great idea try out Shazam. Shazam is a nifty service and iPhone app that can figure out the artist and title of a track from about 10 seconds of audio. After a couple of tries Steve got a result. We won the two pitchers and subsequently had the worst hangover of the trip the next day.

Pictures
Unlike Tobias who brought three cameras on the trip I travel light. I decided my iPhone camera is good enough for the few pics I bother taking. I use Pano to take panorama pictures. It seems AutoStitch is also good or even better, but I haven’t tried it. Then I use the official Flickr app to upload the pics as soon as I get near wifi. That way friends and family can see what you are up to as you go along.

Communications
Besides e-mail I post various updates to Twitter using Tweetie. Of course the Skype app is also useful for keeping in touch with the old world. The only problem is that access to wifi and the overlap of waking hours in the different timezones rarely happen at the same time.

What to do?
Steve had the great idea of creating a custom Google map that we can all add places to visit to. We started looking at various sources for ideas on what to add to the map. I found wikitravel to be a great source of general location information. We picked the first couple of accommodations from tripadvisor reviews, it worked surprisingly well. We also managed to get friends and friends of friends to add their own tips to the map. Thanks everyone!
The only problem is that the maps app on the iPhone doesn’t seem to be able to read custom maps. Google Earth was supposed to do it but we can’t get it to work either.

I started using Foursquare when it launched in London just before I left on this trip. You simply “check in” using the app when you get to a restaurant, bar or another venue of some type. Basically scrobbling for places instead of music. Unfortunately there is no way to get recommendations from the app yet. It now roughly knows what kind of places me and my friends like in London. From that data I want it to tell me where I should go when I get to a new city. I can’t even figure out how to get it to give me a list of places my friends have been to the most number of times.

Aardvark is a service that can help you find the answer to questions that normal search engines struggle with. Instead of looking up the results in an index it forwards the question to real people via various IM services. Aardvark keeps track of what kind of topics their users know enough about to answer and routes questions accordingly. It has worked quite well so far, it is especially worthwhile when your travel question is sent to a local that hopefully knows a lot more about an area then fellow travelers.

Games
Rock band and Civilization for the iPhone are quite fun and suitably time consuming. Unfortunately they also eat tons of battery, so make sure you don’t draing the phone completely. Rock band has a multiplayer mode that works over bluetooth, perfect for playing on planes to discover that they don’t crash when you do so.

Wifi
I have been pleasantly surprised by the number of hostels and restaurants who provide free wifi. Often they are password protected though, so it takes a while to grab someone that knows the password and understands why you are waving an iPhone around. All phrasebooks should contain the important phrase “Excuse me, what is the wifi password?” from now on.
On some networks just trying the classic passwords such as “1234567890” often work. At other times I have considered trying out wpacracker, but that is probably both illegal and not very economical, but looks like fun. It basically lets you use a couple of hundred machines to run a dictionary attack against a network password for a small fee.

DATA!
As you may have noticed throughout this post, I feel castrated when without internet access. It is amazing how quickly one gets used to having it everywhere. I will try almost anything to get my fix, except paying the roaming charges. Having the cost tick away makes me stressed out and constantly thinking about it.
Since the stay in the US would be roughly two months in total I thought it would be worth getting an AT&T subscription with unlimited data. It cost about $70 per month, I earned that money back the first day compared to paying the UK roaming charges. Thanks to Karl for helping me out, thanks to the AT&T staff for eventually figuring out how get it set up for a dirty foreigner like me and finally thanks to the guys who released the software that I unlocked the iPhone with.

I am longing for the day when one can travel around the world without the fear of getting a million dollar phone bill. Luckily the EU seems to be working towards lower roaming costs, but only within the EU of course. It’s a great first step.

I have been looking for a pay as you go sim card on some of the places we have been to, but there doesn’t seem to be any that I have seen that provide unlimited data plans for a fixed cost per week or month. I welcome any suggestions for how to get around this issue when abroad.

Leaving Last.fm

After four amazing years at Last.fm in London I have decided it’s time to move on.

I’m honored to have been a part of the company since it was just four guys in a dodgy Whitechapel office (and Russ working from uni) to where we are today, via vc funding and the sale of the company to CBS.

The greatest part of the job has been the people, clever guys and girls who are great at what they are doing (and nice too, bonus!). Keep an eye on them, I’m sure we’ll see more interesting stuff created at the hands of Last.fm staff, past and present.

Thanks to the various dev team members who have encouraged me to work on open source projects such as Hadoop, it’s been very educational.

I’m going spend the next few months travelling and then we’ll see what happens, if anyone wants to get in touch, send me a tweet or a message.

A warning goes out to the Last.fm alumni around the world, keep your sofas at the ready, I might be knocking on your door any day now.

Zohmg

Interns are great
This summer we’ve had two great interns in the Last.fm data team, they have been working on a project named Zohmg.


From the announcement

I’m happy to announce Zohmg, a data store for aggregation of multi-dimensional time series data built on top of Hadoop, Dumbo and HBase. Data is imported with a mapreduce job and is exported through an HTTP API.

A typical use-case for Zohmg is the analysis of Apache log files. The analyst would be interested in breaking down pageviews by path, user agent, country of origin, etc. In-house at Last.fm, we have successfully demo’d an installation that served access data in realtime for millions of paths broken down by several dimension.

Zohmg 0.2.0
Congrats to both Fredrik Möllerstrand and Per Andersson on their first public release that just went out.

For more information check out the readme.

NOSQL debrief

The relatively young but rapidly growing “nosql” community met last Thursday in San Francisco. The idea was to give attendees a solid introduction to how distributed, non relational databases work as well as an overview of the various projects out there. If I may say so myself we succeeded in doing both. Thanks to all the presenters for very interesting talks and everyone for great hallway discussions.



Presentation slides and videos
Intro session - Todd Lipcon, Cloudera (slides, video1, video2)
Voldemort - Jay Kreps, Linkedin (slides pdf ppt, video1, video2)
Cassandra - Avinash Lakshman, Facebook (slides pdf ppt, video)
Dynomite - Cliff Moon, Powerset (slides, video)
HBase - Ryan Rawson, Stumbleupon (slides, video)
Hypertable - Doug Judd, Zvents (slides pdf ppt, video1, video2)
CouchDB - Chris Anderson, couch.io (slides, video1, video2)

VPork - Jon Travis, Springsource (slides, video)
MongoDb - Dwight Merriman, 10gen (slides, video)
Infinite Scalability - Jonas S Karlsson, Google (slides, video)

Some videos by Digg’s John Quinn, the rest by Martin Dittus from Last.fm. Pictures by Russ Garrett from Last.fm.

NOSQL mailing list
At the event I got requests to set up a NOSQL mailing list as a cross project discussion forum.
Hopefully it will encourage collaboration and exchange of ideas. If that sounds interesting subscribe here.

Sponsors
Thanks again to presenters and the sponsors (last.fm, cbsi, digg and github).

NOSQL meetup

Hadoop summit
I’m going to attend the Hadoop summit in San Francisco in June, had a great time last year, learned a bunch of stuff and got to meet a lot of people I previously only knew by name.

NOSQL
To make the most of the flight money I’m putting together a free meetup about “open source, distributed, non relational databases” or NOSQL for short.

It’s taking place on the 11th of June, the day after the Hadoop summit in San Francisco. CBS interactive have been kind enough to provide us with both a venue and free lunch!

If you wish to attend, please register.

Preliminary schedule
09.45: Doors open
10.00: Intro session (Todd Lipcon, Cloudera)
10.40: Voldemort (Jay Kreps, Linkedin)
11.20: Short break
11.30: Cassandra (Avinash Lakshman, Facebook)
12.10: Free lunch (sponsored by CBSi)
13.10: Dynomite (Cliff Moon, Powerset)
13.50: HBase (Ryan Rawson, Stumbleupon)
14.30: Short break
14.40: Hypertable (Doug Judd, Zvents)
15.20: Panel discussion
16.00: End of meetup, relocate to a pub called Kate O’Brien’s nearby

Location
Magma room, CBS interactive
235 Second Street
San Francisco, CA 94105

VPork

VPork background
With the wide range of distributed, non relational databases out there it is hard to know which one to choose. One part of the puzzle is of course performance. Personally I’m interested in low response times.

A couple of weeks ago Jon Travis put up a useful program called VPork on his github repository. It’s a fairly straight forward performance testing tool for Voldemort, written in Groovy. You can find the announcement on the Voldemort mailing list.

Short description of how it works from the wiki:
* A single JVM is started, with any number of client threads
* Each thread executes for a given number of iterations
* For each iteration, the thread can read an existing record, and/or create a new one
* The probability of each read/write is configurable
* The location of where reads happen is configurable (by default, it reads the most recently written records, trailing off to less frequent reads of writes which occurred long ago)

At the end of the run it gives you info such as average, 99th percentile and standard deviation of read and write latency.

VPork + Cassandra
I’m interested in both Voldemort and the Cassandra project. Thus, it seemed like a good idea to add the ability to benchmark Cassandra to VPork. The result can be found my branch here.

How to run a one node Cassandra test (on Ubuntu)
First let’s fetch Cassandra, there’s no official release yet so we’ll use a nightly build.
sudo apt-get install sun-java6-jdk
wget http://hudson.zones.apache.org/hudson/job/Cassandra/lastSuccessfulBuild/artifact/cassandra/build/cassandra-0.3.0-dev.tgz
tar -zxvf cassandra-0.3.0-dev.tgz
sudo mkdir -p /var/cassandra/logs
sudo chown -R `whoami` /var/cassandra/
cd cassandra-0.3.0-dev
bin/cassandra -f

Now Cassandra should be up and running, let’s start VPork.
sudo apt-get install git-core groovy
git clone git://github.com/johanoskarsson/vpork.git
cd vpork
./vpork.sh configs/cassandra/30-thread-pork.groovy configs/cassandra/nodes.conf

After a while hopefully you’ll get some meaningful results. This is of course a very basic test, you probably want to add more Cassandra nodes, run the client on another node etc.

Where do we go from here?
VPork has some drawbacks such as suffering from the power-of-ten syndrome and it doesn’t warm up the databases before it starts measuring, but it’s a good start!

I’d love to see other interesting storage engine tests added to VPork to give users a simple way of comparing them with the load pattern they expect. There’s an issue open for it over in HBase land.
It’s fairly easy to do, create a class that implements a createClient() method returning a client that implements the basic get(String key) and put(String key, byte[] value) methods.

Another possible use is to alert developers of performance regressions. There’s been discussions about setting up nightly benchmarks on multiple servers to do just that.

Disclaimer: I’m no performance testing expert or a statistician (or a magician for that matter, but it’s not important right now).