SVÄRJÄ

Whenever I get asked for Swedish travel tips I completely blank. I usually blurt out something like: “If it’s summer visit the Stockholm archipelago and during winter time check out the Ice hotel”. I have not been to the latter and only once or twice to the former.

However, I was talking to @jco recently about his move from San Francisco to Stockholm and in a moment of weakness I offered to send him some cultural tidbits to get him up to speed. This is an area I feel somewhat more comfortable in. Below is my introduction to the country I grew up in. I’ve tried to include both items I think are genuinely good but also the cringe worthy moments, songs, films and tv shows that most Swedes know about but wish they didn’t.

Note that I take an almost Techcrunchy approach to journalistic freedom in this post. I will pass opinions off as fact, the research will be minimal and I’ll do anything to get my point across. It’s also worth adding the disclaimer that I haven’t lived in Sweden (voluntarily) for the last seven years, so I’m somewhat out of date.

Music

If there’s one thing Sweden has done right it’s music. For such a small and sparsely populated nation we’ve produced a vast catalogue of great tracks. I have assembled a playlist I feel accurately represents what we have to offer, at least if you share my taste in music. Amongst them are some songs in Swedish, some in English, a few would be recognized by most of my countrymen and for others you might have to dig a bit deeper to find a fellow fan.

Then there’s the bland but chart topping nonsense they play on the radio. For some reason Swedish writers and producers have managed to be involved in the creation of quite a few Billboard hits. A bit of Wikipedia research suggests it took off in the 90ies and they’re still going strong with songs on Taylor Swift and Katy Perry albums.

Some say our success is due to the ubiquitous and affordable music schools, others attribute it to the fairly generous unemployment benefits. Supposedly the latter is great if you’d rather rehearse a few songs instead of going to work.

While there’s a lot to be proud of, let’s dig into the slightly more embarrassing parts.
There’s a genre of music in Sweden called “dansband”. If you boil it down it’s just all around shitty music but it’s perfect for old people to dance to. For those of us that can’t dance there’s still an upside: the band outfits, in particular those from the 70ies. Just marvel at these album covers.

Eurovision song contest has always been a source of great bewilderment for me. If you have been lucky enough to be spared exposure to this event, let’s just say it’s a music competition amongst Euro-ish nations.
Eurovision is a big deal in Sweden. So big in fact that they have somehow managed to turn the process of selecting our contribution into a month long televised event at six massive concert venues. It’s all quite dreadful.

Enjoy this example from ‘91. Snappy dressers, those dancers.

As a bonus, here’s Sweden hosting it in ‘85.

Movies

Tomas Alfredson recently directed Tinker Tailor Soldier Spy, based on the novel by John Le Carré. That movie is decidedly not very Swedish, so I’d like to direct your attention to two of his previous films.

Let the right one in

This one seems to have made it outside the country, at least to the homes of film buffs.
From the Wikipedia page: “the film tells the story of a bullied 12-year-old boy who develops a friendship with a vampire child in Blackeberg, a suburb of Stockholm, in the early 1980s. Alfredson, unconcerned with the horror and vampire conventions, decided to tone down many elements of the novel and focus primarily on the relationship between the two main characters”.

And if you don’t like the subtlety of the Swedish version there’s an American shot for shot remake. You can watch that one without subtitles, safe in the knowledge that they probably spent more on the unnecessary CG effects than the total budget of the original film.

Fyra nyanser av brunt

The title refers to the four stories this film is made up of, all depicting kind of shitty situations. The film was also released as four separate tv episodes, so together they make for quite a long movie. If I recall correctly there were even an intermission half way through when I saw it at the cinema.

The writers and cast are made up of a group of comedians, actors and general media elite types often referred to as “Killinggänget”. They’ve also made a number of great tv shows, for example “Torsk på Tallin”.

Below is a scene from Fyra nyanser av brunt, all in Swedish, of a couple reading the same book together. That’s it. It triggers a mild depression whenever I watch it. The inane discussion, how neat everything is. My personal hell.

Tillsammans

Another filmmaker that has had some impact abroad is Lukas Moodysson. His eccentric streak aside he’s made two films that do a great job of capturing two very different times.

Set in Sweden in 1975 Tillsammans is about life in a socialist commune. Lots of free love experiments with questionable results, people working as welders and unusual upbringings.

Fucking Åmål

I’m not sure I can watch this movie again, the cringe levels are off the charts. It’s Moodysson’s debut film about teenage life in a small Swedish town. Pretty much a spot on portrayal as I recall it.

TV

Let’s see what’s on TV.

Musikbyrån

One of my favourite tv shows growing up was Musikbyrån, literally translated: the Music bureau. It was a straight and to the point show about music with interviews, shows, video clips and whatnot. It was on one of the state tv channels so no ads or pressure to include what is popular on the charts.

Below is short clip from the show. It’s an interview with Trent Reznor of Nine inch nails and The Social network fame, he’s telling people to get off his lawn.

På spåret

I don’t get the appeal of “På spåret”. It’s some sort of quiz show with celebrities. It’s been running since ‘87. I wouldn’t be surprised if my parents have watched it since then.

Recently they revamped it a bit with new hosts, contemporary music guests and it seems people my age are now watching it unironically. Have they learned nothing?!

Below is a clip from the first episode. Awesomely awkward.

Filip och Fredrik

These two fast talkers have been creating some of the weirder tv shows in the last 10 years. One highlight would be High Chaparall, where they travel to the US and hang out with C-level celebrities. Gary Busey for example.
Here’s a short clip of Filip doing a biopsy on Fredrik to extract a small bit of meat that Fredrik then eats. As you do.

Ett år i Sverige

What would a year look like in Sweden? Starting off in January; it’s cold. No surprises there. Around easter you get to eat Semla. It’s basically a bun filled with whipped cream. Delicious. We keep eating these even though they supposedly killed our king back in 1771. Health recommendation: don’t eat 14 of them in one go.

On the 30th of April we light massive bonfires. It’s an excellent excuse to get drunk. In fact, that’s the telltale sign of any proper Swedish holiday. Speaking of, next up would be midsummer. Here’s an excellent German IKEA commercial that outlines the important parts.

There’s a national day in there somewhere, but that is not really celebrated last I checked.

For the most part we skip Halloween, bonfire night and all that rubbish. Instead the next most important holiday involves watching Disney cartoons at 3pm. Some of you might call this “Christmas eve”. This Slate article describes the phenomenon the best.

Swedish history

Below are a couple of interesting episodes in Swedish history, according to me.

1785–1878
When most of Europe was out colonizing the world the Swedes must have been busy elsewhere. Probably for the best. We did however have one colony for almost 100 years, the Caribbean island of Saint Barthélemy. We gave that back to the French though, huge mistake if you ask me. We could have had the first “high tax haven”. Don’t think you pay enough tax, Mr Buffet? Transfer your money to Saint Barthélemy.

1814
According to Wikipedia we haven’t been to war since 1814. That wasn’t so hard, was it? Just let the Nazis use your railroad and sell weapons to the other side, all at the same time. Easy.

1970ies
As depicted in “Tillsammans” mentioned above the 70ies in Sweden had a somewhat red filter over it. Lots of anti Vietnam war demonstrations, the US withdrew their ambassador and the Swedish left wing party even collected money for the Viet Cong. I found this Swedish banner when I went to Vietnam, it was in the “Museum of American war crimes”.

1981
In the latter part of the cold war a Soviet submarine runs aground near a Swedish naval base. This was quite a big deal as you can imagine. The incident lead to most the eighties being spent chasing submarines, often later found out to be marine animals of some sort.

1986
In ‘86 our prime minister, Olof Palme, was murdered on his way home from the cinema. The perpetrator has not been caught.
While the US has a long tradition of killing or otherwise injuring presidents this was the first time a Swedish leader had been killed in hundreds of years.
Largely shielded from troubles the country was, as seen from our time, somewhat naive. In a recent documentary about Palme there’s a scene where he asks the journalists to turn off their cameras so he can fetch his house key from the hiding place.

1993
The first proper commercial radio station in Sweden didn’t appear until 1993. This might baffle some americans. I remember the first time I heard a radio jingle, this might have been the mid 90ies. I didn’t understand why they would play such an annoying fucking music snippet and all they did was sing the radio station’s name over and over. At first I thought it was a joke, but soon realized that it wasn’t. Brave new world.

The good bits

I’ve probably pissed off one or two Brits or Americans by moving to their country and having the audacity to complain about something. Often just because it’s different from what I’m used to. I’m even harsher in my judgement of Sweden though, as @jco would tell you. But there are of course tons of good stuff too. So here goes, these are the things that I’m proud of after having been away for a while.

According to this Wikipedia page Sweden has some of the best income equality in the world. I assume high taxes (spent fairly wisely) will do that to a country. For example university studies are free, ensuring that you can go regardless of your parents economy. In addition to there not being a tuition fee you get a, by international standards, generous sum of money each month from a government run organization called CSN. It’s enough to live on without having to work while studying. Part of that is a loan and part is a government subsidy.

Gay marriage is allowed. Abortion is legal and pretty much a non issue. Probably because “only 1 in 10 Swedes thinks religion is important in daily life”.

Dealings with the government seems to be a lot smoother than compared to the US. The taxes may be high but at least they are easy to file. A pre filled form is sent out with your salary and most other forms of income on it. If you think it looks correct you just text a code to a phone number. If you need to change it you do it online on the tax authority’s website. In fact most government websites accept something called Bank Id, where they trust your banks to identify you. I sorely miss this when dealing with the DMV.
Speaking of banking, checks have been out of fashion for a long time. Electronic transfers between accounts are easy and free.

Transparency and openness as the default. Remember the US election last year when Romney refused to release his tax returns? In Sweden tax returns and much more are public. Want to find out who owns a car? Text the plate number to 72503 and you’ll get the name and city of the owner.
Another aspect of this openness is Allemansrätten. From Wikipedia: “Allemansrätten gives a person the right to access, walk, cycle, ride, ski, and camp on any land—with the exception of private gardens, the immediate vicinity of a dwelling house and land under cultivation”. Better than risk getting shot for trespassing.

San Francisco-Stockholm
These two cities have a few things in common. They are roughly the same size at 800-900k, both are quite beautiful in their own ways and they seem to have a constant lack of available apartments. There’s plenty that tell them apart though. For example Stockholm has a pretty decent subway system. Muni/BART leaves something to be desired when it comes to coverage, reliability and speed in SF.
Then there’s the vast number of homeless people in SF. This article estimates it to 13500 (or 6500 depending on how you count) and this report states the same number in Stockholm is about 400. I don’t think those numbers are directly comparable and there’s of course the difference in climate, but it gives an indication.

All that said I have no plans to move back anytime soon. I hope @jco has a great time and I’m eagerly awaiting his report.

The Twitter stack

For various reasons, including performance and cost, Twitter has poured significant engineering effort into breaking down the site backend into smaller JVM based services. As a nice side effect we’ve been able to open source several of the libraries and other useful tools that came out of this effort.

While there is a fair amount of information about these projects available as docs or slides I found no simple, high level introduction to what we can unofficially call the Twitter stack. So here it is. It’s worth noting that all this information is about open source projects, that it is public already and that I am not writing this as part of my job at Twitter or on their behalf.

Now, granted these were not all conceived at Twitter and plenty of other companies have similar solutions. However I think the software mentioned below is quite powerful and with most of it released as open source it is a fairly compelling platform to base new services off of.

I will describe the projects from a Scala perspective, but quite a few are useful in Java programs as well. See the Twitter Scala school for an intro to the language, although that is not required to understand this post.

Finagle

At the heart of a service lies the Finagle library. By abstracting away the fundamental underpinnings of an RPC system, Finagle greatly reduces the complexity that service developers have to deal with. It allows us to focus on writing application-specific business logic instead of dwelling on lower level details of distributed systems. Ultimately the website itself uses these services to perform operations or fetch data needed to render the HTML. At Twitter the internal services use the Thrift protocol, but Finagle supports other protocols too such as Protocol buffers and HTTP.

Setting up a service using Finagle
A quick dive into how you would set up a Thrift service using Finagle.

1. Write a Thrift file defining your API. It should contain the structs, exceptions and methods needed to describe the service functionality. See Thrift Interface Description Language (IDL) docs, in particular the examples at the end for more info.
2. Use the Thrift file as input for a code generator that spits out code in your language. For Scala and Finagle based projects I would recommend Scrooge.
3. Implement the Scala trait generated from your Thrift IDL. This is where the actual functionality of your service goes.
4. Provide the Finagle server builder an instance of the implementation above, a port to bind to and any other settings you might need and start it up.

That looks pretty similar to just using plain Thrift without Finagle. However, there are quite a few improvements such as excellent monitoring support, tracing and Finagle makes it easy to write your service in an asynchronous fashion. More about these features later.

You can also use Finagle as a client. It takes care of all the boring stuff such as timeouts, retries and load balancing for you.

Ostrich

So let’s say we have a Finagle Thrift service running. It’s doing very important work. Obviously you want to make sure it keeps doing that work and that it performs well. This is where Ostrich comes in.

Metrics
Ostrich makes it easy to expose various metrics from your service. Let’s say you want to count how many times a particular piece of code is run. In your service you’d write a line of code that looks something like this:

Stats.incr(“some_important_counter”)

As simple as that. The counter named some_important_counter will be incremented by 1.

In addition to just straight up counters you can get gauges that report on the value of a variable:

Stats.addGauge("current_temperature") { myThermometer.temperature }

or you can time a snippet of code to track the performance

Stats.time("translation") {
 document.translate("de", "en")
}

Those and other examples can be found in the Ostrich readme.

Export metrics
Ostrich runs a small http admin interface to expose these metrics and other functionality. To fetch them you would simply hit http://hostname:port/stats.json to get the current snapshot of the metrics as JSON. At Twitter the stats from each service will be ingested from Ostrich by our internal observability stack, providing us with fancy graphs, alerting and so on.

To tie this back to our previous section: If you provide a Finagle client or server builder with an Ostrich backed StatsReceiver it’ll happily splurt out tons of metrics about how the service is performing, the latencies for the RPC calls and the number of calls to each method to name a few.

Ostrich can also deal with configuring your service, shutting down all the components gracefully and more.

This is an example of what a dashboard could look like with stats gathered from Ostrich by our observability stack. Screenshot from @raffi’s presentation deck.

Zipkin

Ostrich and Finagle combined gives us good service level metrics. However, one downside of a more service oriented architecture is that it’s hard to get a high level performance overview of a single request throughout the stack. 
Perhaps you are a developer tasked with improving performance of a particular external api endpoint. With Zipkin you can get a visual representation of where most of the time to fulfill the request was spent. Think Firebug or Chrome developer tools for the back end. Zipkin is a implementation of a tracing system based off of the Google Dapper paper.

Finagle-Zipkin
So how does it work? There’s a finagle-zipkin module that will hook into the transmission logic of Finagle and time each operation performed by the service. It also passes request identifiers down to any services it relies on, this is how we can tie all the tracing data together. The tracing data is logged to the Zipkin backend and finally we can display and visualize that data in the Zipkin UI.

Let’s say we use Zipkin to inspect a request and we see that it spent most of it’s time waiting for a query to a MySQL database. We could then also see the actual SQL query sent and draw some conclusions from it. Other times perhaps a GC in a Scala service was a fault. Either way, the hope is that a glance at the trace view will reveal where the developer should spend effort improving performance.

Enabling tracing for Finagle services is often as simple as adding

.tracerFactory(ZipkinTracer())

to your ClientBuilder or ServerBuilder. Setting up the whole Zipkin stack is a bit more work though, check out the docs for further assistance.

Trace view, taken from my Strange loop talk about Zipkin.

Mesos

Mesos describes itself as “a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks”. I’ll try to go through this section without using buzzwords such as “private cloud”, although technically I just did.

The core Mesos project is an open source Apache incubator project. On top of it you can run schedulers that deal with more specific technologies, for example Storm and Hadoop. The idea being that the same hardware can be used for multiple purposes, reducing wasted resources.

In addition to using Storm on top of Mesos we deploy some of our JVM-based services to internal Mesos clusters. With the proper configuration it takes care of concerns such as rack diversity, rescheduling if a machine goes down and so on.

The constraints imposed by Mesos have the positive side effect of enforcing adherence to various good distributed systems practices. For example:

• Service owners shouldn’t make any assumptions about jobs’ lifetimes, as the Mesos scheduler can move jobs to new hosts at any time.
• Jobs shouldn’t write to local disk, since persistence is not guaranteed.
• Deploy tooling and configs shouldn’t use static server lists, since Mesos implies deployment to a dynamic environment.

Iago

Before putting your new service into production you might want to check how it performs under load. That’s where Iago (formerly Parrot) comes in handy. It’s a load testing framework that is pretty easy to use.

The process might look something like this:

1. Collect relevant traffic logs that you want to use as the basis for your load test.
2. Write a configuration file for the test. It contains the hostnames to send load to, the number of requests per second, the load pattern and so on.
3. Write the actual load test. It receives a log line, you transform that into a request to a client.
4. Run the load test. At Twitter this will start up a few tasks in a Mesos cluster, send the traffic and log metrics.

Example
A load test class could be as simple as this:

class LoadTest(parrotService: ParrotService[ParrotRequest, Array[Byte]]) extends     
 ThriftRecordProcessor(parrotService) {

 val client = new YourService.FinagledClient(service, new TBinaryProtocol.Factory())

 def processLines(job: ParrotJob, lines: Seq[String]) {
   lines foreach {line =>client.doSomething(line) }
 }
}

This class will feed each log line to your service’s doSomething method, according to the parameters defined in the configuration of parrotService.

ZooKeeper

ZooKeeper is an Apache project that is handy for all kinds of distributed systems coordination.

One use case for ZooKeeper within Twitter is service discovery. Finagle services register themselves in ZooKeeper using our ServerSet library, see finagle-serversets. This allows clients to simply say they’d like to communicate with “the production cluster for service a in data centre b” and the ServerSet implementation will ensure an up-to-date host list is available. Whenever new capacity is added the client will automatically be aware and will start load balancing across all servers.

Scalding

From the Scalding github page: “Scalding is a Scala library that makes it easy to write MapReduce jobs in Hadoop. Instead of forcing you to write raw map and reduce functions, Scalding allows you to write code that looks like natural Scala”.

As it turns out services that receive a lot of traffic generate tons of log entries. These can provide useful insights into user behavior or perhaps you need to transform them to be suitable as Iago load test input.

I have to admit I was a bit sceptical about Scalding at first. It seemed there were already plenty of ways to write Hadoop jobs. Pig, Hive, plain MapReduce, Cascading and so on. However, when the rest of your project is in Scala it is very handy to be able to write Hadoop jobs in the same language. The syntax is often very close to the one used by Scala’s collection library, so you feel right at home, the difference being that with Scalding you might process terabytes of data with the same lines of code.

A simple word count example from their tutorial:

  TextLine(args("input"))
   .read
   .flatMap('line -> 'word){ line : String => line.split("\\s")}
   .groupBy('word){group => group.size}
   .write(Tsv(args("output")))

jvmgcprof

One of the well known downsides of relying on the JVM for time sensitive requests is that garbage collection pauses could ruin your day. If you’re unlucky a GC pause might hit at the wrong time, causing some requests to perform poorly or even timeout. Worst case that might have knock on effects that leads to downtime.

As a first line of defence against GC issues you should of course tweak your JVM startup parameters to suit the kind of work the service is undertaking. I’ve found these slides from Twitter alumni Attila Szegedi extremely helpful.

Of course, you could minimize GC issues by reducing the amount of garbage your service generates. Start your service with jvmgcprof and it’ll help you reach that goal. If you already use Ostrich to track metrics in your service you can tell jvmgcprof which metric represents the work completed. For example you might want to know how many kilobytes of garbage is generated per incoming Thrift request. The jvmgcprof output for that could look something like this.

2797MB w=101223 (231MB/s 28kB/w)
50.00%  8   297
90.00%  14  542
95.00%  15  572
99.00%  61  2237
99.90%  2620    94821
99.99%  2652    95974

On the first line you can see that the number requests or work were 101223 for the period monitored, with 231MB/s of garbage or 28kB per request. The garbage per request can easily be compared after changes has been made to see if they had a positive or negative impact on garbage generation. See the jvmgcprof readme for more information.

Summary

It’s no surprise, but it turns out that having a common stack is very beneficial. Improvements and bug fixes made by one team will benefit others. There is of course another side to that coin, sometimes bugs are introduced that might just be triggered in your service. However, as an example, when developing Zipkin it was immensely helpful to be able to assume that everyone used Finagle. That way they would get tracing for free once we were done.

I have left out some of the benefits of the Twitter stack and how we use Scala, such as the very convenient way Futures allow you to deal with results from asynchronous requests. I hope to write a more in depth post on how to set up a Twitter style service that would deal with the details omitted in this article. In the meantime you can check out the Scala school for more information.

Thanks to everyone who worked on the projects mentioned in this article, too many to name but you know who you are.

Zipkin - a distributed tracing framework

I’ve been working on a Twitter project called Zipkin for quite a while now. It’s a distributed tracing framework similar to Dapper. Thankfully Twitter has a very liberal open sourcing policy so we were able to put the whole beast on Github today.

From the announcement:

Zipkin is a distributed tracing system that we created to help us gather timing data for all the disparate services involved in managing a request to the Twitter API. As an analogy, think of it as a performance profiler, like Firebug, but tailored for a website backend instead of a browser. In short, it makes Twitter faster.

I won’t duplicate too much of the information already out there in this post. Check out the rest of the blog post for further information. I also recommend the README and the Dapper paper for some more details.

Music discovery tools

Much to my surprise there has been an upswing in the number of small and extremely useful music discovery sites or apps that I use.

Peel
There are obviously tons of great music blogs out there, ranging from the big established sites to the smaller more niche ones. I personally love the hand curated music selection but I don’t care so much for spending all that time reading the blogs themselves. Peel nicely solves this problem by parsing the blogs and then just provides a list of the tracks so you can listen to them directly. It’s also easy to download them, add to iTunes and sync them to my iPhone. Perfect for the way I want to interact with music blogs, easily worth the $25.



This is my jam
I think of This is my jam as Twitter but for music. It has a lot of the same components, you follow people, others follow you, but instead of tweets there are songs. The songs stay up for a week or shorter. As your friends add new songs they populate your timeline with deliciousness, similar to how the Twitter timeline works. Extremely handy for just playing through everything your friends are into right now.







8tracks
If This is my Jam is Twitter for music, 8tracks would be Tumblr. The long form of curated music. At its core 8tracks is a pretty straight forward playlist site. You upload songs, give the playlist a snazzy title and a picture. You follow people and all that jazz. It’s not as direct as just uploading a track to This is my Jam, which is both good and bad. Usually people put some more effort into their playlist, in particular so that the songs work together.




Later.fm
For some reason the following scenario happens to me a lot. I’m at work, I get a link from somewhere that leads to a couple of songs. I don’t have time to listen to them straight away or even download them. Instead I click on my “Listen later” bookmarklet in my browser. That quickly adds all of the songs to Later.fm so I can listen to it later. A smaller use case perhaps, but I have found it saves a lot of time.




Disclaimer: I know some of the authors of these tools.

Local pay as you go SIM cards with data

As previously mentioned I am thoroughly addicted to my iPhone. This manifests itself especially when abroad. During trips the need for information about transport, places and activities is even greater then usual. Unfortunately this greater need is efficiently suppressed by extreme roaming costs, that is, unless you get a local pay as you go SIM card with decent data rates.

Finding good rates abroad is usually tricky if you don’t know the local market and/or language. Scanning through various mobile phone forums can sometimes help but it is time consuming and stale information is commonplace.

To help resolve this issue I have put together a small wiki with information about sensible pay as you go data plans divided up by country. So far there’s only information about UK, US and Sweden in there but I welcome everyone to add more.

Check it out

Going cold turkey on physical media

With all them services and hardwares out there, is it possible to go cold turkey on physical media? Why make the digital switch? And if you do, is the selection good enough?

It’s heavy, I’m lazy
My generation is the first to have grown up with the internet and dirt cheap flights. I’m glad, since these two pillars of modern life has given me and others greater flexibility when deciding where to live. Moving to a new place, sometimes in new countries, has almost exclusively been a positive experience. However, it is not without drawbacks. Don’t worry, I’ll save my rant about British banks for another blog post.

One downside is the practical limit to the amount of stuff that can be hauled between flats, especially when moving internationally. While it might appear to be an insignificant detail it is having a big impact on how I think. All that stuff gathered through the years has started to become more of a burden than an asset. Every time the moving van is loaded there are more books, DVDs, games and CDs to fill it with, many of which have already been read, seen, finished or ripped to mp3. What previously would be a common impulse buy is now a major decision on par with “should I get that vasectomy from the village butcher or not?”

With yet another international move coming up the idea to go cold turkey on physical media has popped up once again, sandwiched in-between thoughts such as “What did I have for lunch today?” and “How you do spell the back-to-the-future-esque surname of that dark haired woman in Dirty projectors?”.



Digitally yours
Enough waffling. This is a quick experiment to see if going completely digital would be possible for me, your mileage may vary. I have compiled a list of books, films, games and albums that I will try to download, stream or by other means legally consume without buying the physical product.

The upsides to going digital are obvious, besides the mentioned reduction of shelf space required it promises instantish gratification. I want to listen to that song now, not next week when I have time to go to a record shop or when the delivery will arrive. Then there’s the ease of use aspect, mp3 files for example can be played on almost any device anywhere these days, not just the livingroom stereo. Same for many video formats, I was surprised to see that my dad’s new kitchen TV could play Xvid files from USB sticks.
There are unfortunately downsides too, such as DRM and other lock-in techniques.


Music
Music is the easiest of the four. Since it compresses well it has been a solved problem for quite some time, although mainly illegally so. 10 years later the legal options are catching up.
I picked out the top 15 tracks from different artists in my 3 month chart at Last.fm. I then simply searched for all of them and counted the results. If I only found the unremixed or original version of a track I marked that as half a point.

The tracks
The Radio Dept. – Heaven’s on Fire
Jonathan Johansson – Aldrig ensam
LCD Soundsystem – Dance Yrself Clean
Noir Désir – Le Vent Nous Portera
Phosphorescent – The Mermaid Parade
Foals – Spanish Sahara
Surfer Blood – Take It Easy
Familjen – När planeterna stannat
Stina Nordenstam – I Dream of Jeannie
Real Estate – Fake Blues
The National – Bloodbuzz Ohio
Teenage Fanclub – The World’ll Be OK (Alternative version)
She & Him – In the Sun
Teddybears ft. Eve – Rocket Scientist
Radiohead – Nude (Holy Fuck Remix)


Results
Spotify - 70%
iTunes - 86%
Amazon MP3 - 76%
Hype Machine - 80%
Combined result - 97%

Altogether a decent result, what some services lacked the others made up for. This might just work.





Games - Xbox 360
My platforms of choice are the Xbox 360 and PC. I picked out five fairly big and recent titles on each platform that have turned up on my gaming radar (also known as Gadar™).

Battlefield: Bad company 2
Call of duty: Modern warfare 2
Just cause 2
Alan Wake
Red dead redemption



Results
There is no need for a graph since the one place to buy these games online, Xbox Live Marketplace, did not have a single one of them when I checked. Disastrous! It mostly contains older titles. Does anyone know the reason why? If there is one thing iPhone apps have taught me it’s that I buy lots more when it’s just a click and a download away.


Games - PC
Battlefield: Bad company 2
Call of duty: Modern warfare 2
Just cause 2
Napoleon: Total War
Alien Breed: Impact

 
Results
Steam - 100%
Direct2drive - 20%
Combined result - 100%

Not much to say, Steam is where it’s at, not much point in looking elsewhere. The newly added Mac support is also neat for those of us that have tasted the forbidden fruit.


Films
Below are 10 recent films I like. Half mainstream-ish and half foreign productions.

Youth in revolt
An Education
Zombieland
The Road
District 9
Män som hatar kvinnor / The girl with the dragon tattoo (se)
O cheiro do ralo / Drained (br)
El secreto de sus ojos / The secret in their eyes (ar)
Død snø / Dead snow (no)
Le scaphandre et le papillon / The diving bell and the butterfly (fr)

Results
iTunes - 50%
Zune - 20%
Netflix - 30%
Hulu - 0%
Voddler - 20%
CDON - 60%
Random torrent site - 100%
Combined result (legal) - 70%






Better then expected, but not great. The movies are spread out over the services making them hard to find. Is there a search engine for this stuff yet? If not, someone should build it and tell me about it. In this test I also counted those from those services not strictly available in the EU.
Sadly it is still a lot less hassle to pirate a film than to watch it legally.


Books
In my mind the big contenders are Amazon’s Kindle and Apple’s iPad and their respective book stores, but I will admit I have not done much if any research. I picked a selection of mostly work related books, since that seems to be what I make time to read.

Programming Scala (Dean Wampler, Alex Payne)
The Ruby Programming Language (David Flanagan, Yukihiro Matsumoto)
Hadoop: The Definitive Guide (Tom White)
Confessions of a public speaker (Scott Berkun)
Atlas Shrugged (Ayn Rand)

Results
Amazon Kindle - 60%
iBooks - 0%
O’Reilly - 80%
Combined result - 80%

As it turns out most of what I read are O’Reilly books, so this test is quite skewed. Surprised by the lack of books in the iBooks store.




Summary
Doable but plenty of room for improvement.

The selection is lacking, especially when it comes to Xbox 360 games. Equally annoying as a customer is the need for looking through a handful of different services in order to find that one movie you want to see. Other then that both books and movies do alright. Music and PC games are the only two solid results with excellent coverage.

One can naively hope that as the content industry further feels the squeeze from piracy that they would feel inclined to actually give the customers what they want, the way they want it.

Contributing to Open street map with an iPhone

Open street map turned out to be a great help on the trip last year. So when I found out the area around where I live right now has quite poor coverage I decided to help out by adding a few roads. Here’s how you can do it with the help of an iPhone.

What is Open street map?

OpenStreetMap creates and provides free geographic data such as street maps to anyone who wants them. The project was started because most maps you think of as free actually have legal or technical restrictions on their use, holding back people from using them in creative, productive, or unexpected ways.

Gathering traces
There are many ways of contributing to OSM. Since I wanted to gather information on road using the iPhone GPS I started looking for a suitable app. There are a few but the best one for this purpose seems to be the OSMTrack app.

To use it simply download the app and create an OSM account. Before you set off just click the start button and start driving around. Once you are finished click stop and then the upload button to send the collected traces to your OSM account.



Finding the trace
Once safely back near a computer you log in to Open street map and click your username in the top right corner and then “my traces”. This gives you a list of all the collected tracks/traces. Click edit on one of them, now the real map editing begins. There are a few different ways to do it but I did the following.

Editing the map
Click “Convert GPS track to ways” and “Edit with save”. This ought to convert your collected points into a red way. I have noticed that it sometimes refuses to do so, don’t know why.

Next I removed the parts of the way that is already in OSM. To do so click on one of the unwanted points and press delete until you are happy. Pressing enter will stop the editing and escape will remove the way completely. Shift+click on another way will connect the two, used to create intersections for example. Pressing X will split the way in two. Perhaps you want to split a way into two different roads. For more information see the Potlatch/Primer.

Now you ought to have what looks like one or more roads added to the map. Before they will show up you have to mark them with the specific type of road it is. There are a bunch of great examples here. To change the type click on the way and then on the drop down at the bottom of the page that says “(no preset)”.

Once you are done don’t forget to click the save button at the bottom right of the page.

That’s it! You are now a contributing member of society, congrats!

Oh, one more thing. I found OpenMaps to be a great OSM viewing app with offline capabilities for the iPhone.

Travel in the 21st century

I am currently on a trip through parts of south east asia and the US with Steve Gravell and Tobias Köppen. Naturally I couldn’t leave my Eee PC or iPhone at home, so during the trip I have gathered a few tips for the geeky traveler. Nothing revolutionary, but I found these apps and websites to be quite helpful. In my usual manner I also complain a bit and suggest improvements. If you have any additional recommendations please add a comment.

Maps
Unfortunately the holy trinity of Google maps, GPS and an unlimited data plan is one short when abroad. Without a data plan you are plunged back into the dark ages again and roaming is not an option. Luckily there is an iPhone app called oMaps that allows you to mark certain areas on the map for offline use. Perfect for getting the city map while you are in a hotel room near wifi, mark a few places you want to visit and off you go.
oMaps uses OpenStreetMap as the data source, the license used enables the offline mode. OpenStreetMap builds its map from user contributions so some areas have excellent coverage whilst others are lacking. You can help out by contributing straight from your iPhone, although I had limited success with that so far.

Music
Music is essential on a trip, perfect for all those long bus, plane and boat rides. I love the Last.fm iPhone app but unfortunately we are once again back to the problem of not having a data plan and no offline mode. I decided to give the Spotify app a go. It allows you to select playlists that will be available in offline mode. I can easily search for and add new albums as I go on the trip and have them synced while I am near wifi. The price is quite hefty at €9.99 a month but worth it when on the move.
As many others I would kill for being able to run the app in the background. When me and Steve drove to Napa valley I was using the phone to navigate, which meant we couldn’t listen to my music at the same time. It was also the most ass-clinching car ride in a long long time, but that’s another story. While I am at it, how about adding some sort of playlist rss subscription feature to the app? For example I read the music blog Let’s pretend we’re bunny rabbits. They regularly post playlists with a spotify link and I’d like to see that automatically pop up in the Spotify app.

Steve and I were having a few beers in a horrible bar in Kuala Lumpur when the DJ announced that the next song was part of a competition. Figure out what song it is and you win two pitchers of beer. Steve had the great idea try out Shazam. Shazam is a nifty service and iPhone app that can figure out the artist and title of a track from about 10 seconds of audio. After a couple of tries Steve got a result. We won the two pitchers and subsequently had the worst hangover of the trip the next day.

Pictures
Unlike Tobias who brought three cameras on the trip I travel light. I decided my iPhone camera is good enough for the few pics I bother taking. I use Pano to take panorama pictures. It seems AutoStitch is also good or even better, but I haven’t tried it. Then I use the official Flickr app to upload the pics as soon as I get near wifi. That way friends and family can see what you are up to as you go along.

Communications
Besides e-mail I post various updates to Twitter using Tweetie. Of course the Skype app is also useful for keeping in touch with the old world. The only problem is that access to wifi and the overlap of waking hours in the different timezones rarely happen at the same time.

What to do?
Steve had the great idea of creating a custom Google map that we can all add places to visit to. We started looking at various sources for ideas on what to add to the map. I found wikitravel to be a great source of general location information. We picked the first couple of accommodations from tripadvisor reviews, it worked surprisingly well. We also managed to get friends and friends of friends to add their own tips to the map. Thanks everyone!
The only problem is that the maps app on the iPhone doesn’t seem to be able to read custom maps. Google Earth was supposed to do it but we can’t get it to work either.

I started using Foursquare when it launched in London just before I left on this trip. You simply “check in” using the app when you get to a restaurant, bar or another venue of some type. Basically scrobbling for places instead of music. Unfortunately there is no way to get recommendations from the app yet. It now roughly knows what kind of places me and my friends like in London. From that data I want it to tell me where I should go when I get to a new city. I can’t even figure out how to get it to give me a list of places my friends have been to the most number of times.

Aardvark is a service that can help you find the answer to questions that normal search engines struggle with. Instead of looking up the results in an index it forwards the question to real people via various IM services. Aardvark keeps track of what kind of topics their users know enough about to answer and routes questions accordingly. It has worked quite well so far, it is especially worthwhile when your travel question is sent to a local that hopefully knows a lot more about an area then fellow travelers.

Games
Rock band and Civilization for the iPhone are quite fun and suitably time consuming. Unfortunately they also eat tons of battery, so make sure you don’t draing the phone completely. Rock band has a multiplayer mode that works over bluetooth, perfect for playing on planes to discover that they don’t crash when you do so.

Wifi
I have been pleasantly surprised by the number of hostels and restaurants who provide free wifi. Often they are password protected though, so it takes a while to grab someone that knows the password and understands why you are waving an iPhone around. All phrasebooks should contain the important phrase “Excuse me, what is the wifi password?” from now on.
On some networks just trying the classic passwords such as “1234567890” often work. At other times I have considered trying out wpacracker, but that is probably both illegal and not very economical, but looks like fun. It basically lets you use a couple of hundred machines to run a dictionary attack against a network password for a small fee.

DATA!
As you may have noticed throughout this post, I feel castrated when without internet access. It is amazing how quickly one gets used to having it everywhere. I will try almost anything to get my fix, except paying the roaming charges. Having the cost tick away makes me stressed out and constantly thinking about it.
Since the stay in the US would be roughly two months in total I thought it would be worth getting an AT&T subscription with unlimited data. It cost about $70 per month, I earned that money back the first day compared to paying the UK roaming charges. Thanks to Karl for helping me out, thanks to the AT&T staff for eventually figuring out how get it set up for a dirty foreigner like me and finally thanks to the guys who released the software that I unlocked the iPhone with.

I am longing for the day when one can travel around the world without the fear of getting a million dollar phone bill. Luckily the EU seems to be working towards lower roaming costs, but only within the EU of course. It’s a great first step.

I have been looking for a pay as you go sim card on some of the places we have been to, but there doesn’t seem to be any that I have seen that provide unlimited data plans for a fixed cost per week or month. I welcome any suggestions for how to get around this issue when abroad.

Leaving Last.fm

After four amazing years at Last.fm in London I have decided it’s time to move on.

I’m honored to have been a part of the company since it was just four guys in a dodgy Whitechapel office (and Russ working from uni) to where we are today, via vc funding and the sale of the company to CBS.

The greatest part of the job has been the people, clever guys and girls who are great at what they are doing (and nice too, bonus!). Keep an eye on them, I’m sure we’ll see more interesting stuff created at the hands of Last.fm staff, past and present.

Thanks to the various dev team members who have encouraged me to work on open source projects such as Hadoop, it’s been very educational.

I’m going spend the next few months travelling and then we’ll see what happens, if anyone wants to get in touch, send me a tweet or a message.

A warning goes out to the Last.fm alumni around the world, keep your sofas at the ready, I might be knocking on your door any day now.

Zohmg

Interns are great
This summer we’ve had two great interns in the Last.fm data team, they have been working on a project named Zohmg.


From the announcement

I’m happy to announce Zohmg, a data store for aggregation of multi-dimensional time series data built on top of Hadoop, Dumbo and HBase. Data is imported with a mapreduce job and is exported through an HTTP API.

A typical use-case for Zohmg is the analysis of Apache log files. The analyst would be interested in breaking down pageviews by path, user agent, country of origin, etc. In-house at Last.fm, we have successfully demo’d an installation that served access data in realtime for millions of paths broken down by several dimension.

Zohmg 0.2.0
Congrats to both Fredrik Möllerstrand and Per Andersson on their first public release that just went out.

For more information check out the readme.