Easy data visualization with WebKit

At Spotify, we recently put up two large TV screens on the walls of our Stockholm office (most R&D is done there). The idea is to visualize & communicate that “stuff is happening” without actually revealing any critical data (since a lot of external people are visiting the office).

Today me, Andreas Öman and Emil Hesslow – fellow Spotifiers – kicked off a cozy little Hack Night at the office, trying to create something simple yet impressive to have running on one of the TV screens.

We ended up writing a real-time search query visualization in just about a few hours. It looks like this and is smoothly animated:

Try a demo version here… (Tested in Safari, iPhone, Firefox and Chrome).

How did we manage to build a real-time scalable system and high-performance viz in such an awfully short time?!

Well, for starters we used WebKit through Cocui which instantly gave us full screen high-performance hardware-accelerated drawing (yes, it’s a long sentence with cool words but those things shouldn’t be taken for granted).

But… where does the data come from? From the internets? – Not really, but it sure travels in internets-style. We use a dumb pub/sub message queue. In one end a client (the WebKit/Cocui app in HTML/JavaScript) is listening (subscribing). In the other end one of our search servers are pushing messages into the queue in batches.

          	[batch of search queries during last minute]
          	                   ||
          	                   ||
          	                   \/
          	             [message queue]
          	              ||        ||
          	              ||        ||
          	              \/        \/
          	           [client]  [client]

The client simply enqueues these search queries when they are delivered while at the same time dequeueing search queries. We do it this way (batches and a queue) because we simply have too high rate of searches. It would be almost impossible to read anything if we actually sent every single message. To give the feeling of real time we use a random delay when dequeueing queries.

When a query is dequeued it’s formatted into a chunk of HTML (a div and an a tag) and then prepended to the body using jQuery with animated effects.

Don’t forget to try the demo version (Does not work in Internet Explorer) which is simply the client without any real-time data. Note that this demo uses static data for demonstration purposes, not an actual real-time stream. The real-time data stream is only available within our office and thus not available for public use.

Update: Successfully tested with iPhone, Firefox and Chrome – thank you readers!

Here’s the full Cocui demo application: sptv1-demo-cocui-app.zip (for Mac OS X 10.5 and newer).