Importance of transitions in interactive visualisations.

I am a very big fan of transitions when presenting complex data with interactive visualisations. I believe that transitions play an important role in building a continuous narrative in audience’s mind when they are trying to understand connected series of information. To be clear, I am talking specifically about transition of data between the stages of visualisation and not the transition of the canvas of the visualisation. Today lets look at one of my favourite visualisations of all time – this brilliant interactive one by NYtimes, showcasing the budget proposal made by US government in 2012

The story here starts with showing whole budget along with the scale of 900bn budget deficit proposed, then we move to see that the most of the spending is mandatory over which the govt. has no control and the smaller discretionary spending is the one which we are having the discussion on, then we move to see these discretionary spending categories compared with each other to the gainers and losers from last year and finally we see them put together along with the mandatory spending to complete the picture.

The amazing thing about this graphic is the way the circles (departments / budget categories), which are the basic unit of the analysis, move between the stages of the story – specially from step 2 to step 3. Not only it reinforces the concept of mandatory spending it also prepares us to what we are going to be see in the next stage. This half second transition fundamentally changes the clarity and legibility of the whole story for a casual audience.

I am currently working on a medium sized dataset trying to filter it with various rules to produce a final number and this kind of visualisation seems perfect to show the changes in the results as I move to different stages of filtering. I will be posting on the progress and stay tuned for the updates.

 

Advertisements

Mapping distribution of National Institutions in Higher Education in India [R + tidyverse + tmap]

Since I started learning R and moved away from proprietary data analysis and GIS packages, I have been amazed by the availability of free data-sets and tools enabling people to make awesome analysis and visualisations. Today we’ll look into a simple exercise of taking open data from different sources and combining them using opensource and free tools to produce maps which help us understand the data better.

We’ll use data from two sources, a tabular file on distribution of national institutes of higher education in India from here and shape file containing geographic information on boundaries of Indian states from here. I have cleaned and prepared both data which can be downloaded here. First we need to download the archive and extract/ unzip it to our working directory. Once that is done we can start combining and plotting the data.

# Loading the data
data <- read.csv("institutions.csv")
library(rgdal) # library for reading shape files
states <- readOGR(".", "india_state")

Note that, in readOGR, the first parameter is the folder at which the shape file is kept (it should be “.” if we directly unzipped the files to working directory) and second one is the name of the shape file.

# calculating the total number institutions and 
# the number of institutions per 10 million people
data <-  data %>%
mutate( Total = rowSums(.[,3:10]) )%>%
mutate( Totalppm = Total/Population)

# merging the data into the shapefile
states <- states %>%
merge(data,by="state")

Now we have succesfully merged the data into the shape file we can see this by asking states@data at the console. Now we need to map the data using tmap. First of all we load the library by running library(tmap). This is complex package and has a lot of dependencies so it might take a while to install and download. As a first step we plot just the borders of the state by running,

tm_shape(states) +
    tm_borders(col = "grey")

1

We can add labels to the map by adding a tm_text layer to it. Notice that the order of the layer is important since things overlap on each other.

tm_shape(states) +
    tm_borders(col = "grey") +
    tm_text("state", root = 10, size = "AREA")

2

Now we can plot a single variable on the map as the fill colour for the states. For example If we want to highlight all the states with an IIT, we do,

tm_shape(states) +
    tm_fill("IIT") +
    tm_borders(col = "grey") +
    tm_text("state", root = 10, size = "AREA")

3

We can plot multiple variables on the same plot side by side by just passing a vector of variables to compare. Notice that we switched of legends on one of the layer here by setting legend.size.show=FALSE.

tm_shape(states) +
    tm_fill(c("IIT", "IIM")) +
    tm_borders(col = "grey") +
    tm_text("state", root = 10, size = "AREA",
            legend.size.show = FALSE)

4

Finally we can plot the number of institutions per population by,

tm_shape(states) +
    tm_fill("Totalppm",
            title = "Institutions per 10m people") +
    tm_borders(col = 'grey') +
    tm_text("state", root = 10, size = "AREA",
            legend.size.show = FALSE)

5

This map is very uninformative because of the small Himalayan states with low populations skewing the whole distribution. So we  have to classify this data ourselves by giving a custom breaks parameter,

tm_shape(states) +
    tm_fill("Totalppm", 
            breaks = (1:10) / 2,
            title = "Institutions per 10m people") +
    tm_borders(col = 'grey') +
    tm_text("state", root = 10, size = "AREA",
            legend.size.show = FALSE)

6

Now this shows a possible north south divide in the distribution of institutions per person. This may be because of most of the national institutions in North being located in Delhi, while in the South Bangalore, Mumbai and Chennai compete for them.

That completes the post for today. To summarise we took a tabular data, joined and plotted it with geographic data and uncovered new information which are not present in them individually!

The Idea

The idea is simple and have been done by numerous people before. It is a map based directory for a group of small people. The idea is an extension of the work I did years ago (post link). I built this site waiting for an entire day in the university campus for a (very annoying) friend who kept postponing the meeting. We finally met at the end of the day and by that time I had finished the web app. I had recently learned to use server side coding (using PHP), databases (MySQL) and had figured out that hosting is not rocket science. Coupled with some previous experience with Google maps API and JavaScript. I quickly built this crowd sourced project where everyone enters their Name and City and the map shows all the people, aggregated based on the zoom level. Simple, Clear and nothing fancy.

crowdsourced

Once this was posted to the common forums and people started using it. It became clear that I can do more with it. Suggestions kept coming in and after a point all the suggestions amounted to creating a whole new social network. Though I agreed there were loads of things which could have been improved with the app, I felt the usefulness of this map/visualisation was its simplicity and clarity. The things I felt could be improved were

  • data collection from users. Though it is simple as it is, I feel it can be improved by utilising availability of location data and connecting to other platforms (fb obviously). Also possible account management for users, edit and delete their own data.
  • Search and find people. This was never implemented. It could have been very useful for people to search and find others.
  • Some form of possible communication with other users. When you have a map like this and if you are able to find someone in the map, naturally you try to click on the person to see more details and communicate with them (even if it is a link to their email).

Things I specifically don’t want to do

  • Build a suite of information on the users and nag them to fill the information up. No body cares about someone’s primary school or their fourth work fax number. This will just make the whole thing a directory created with a aim to increase the amount of information rather than usefulness of the information.
  • Build a suite of interactions within the users and app while creating user interface elements for those interactions. This will lead into user being presented with loads of buttons& forms and when these UI elements starts covering the entire screen they will be folded into scores of menus. This just defeats the purpose of the entire thing. If people wanted millions of options to interact with the people they know, they can use Facebook or any social network for that.
  • Anything which is not location based or adds value based on location. Since location and mapping is the central theme of this project.

With all these things in consideration, I spent some time sketching out some preliminary ideas shown below,

I’ll explain them in detail in the following post.

P.S: I still think the idea shown in the sketch is too cluttered and needs significant simplification to be anywhere near neat and useful.

Augmented Reality + Heritage Conservation

We (Kostas, Daniel Lam and I ) have finally finished an augmented reality based app for an exhibit at the “Almost Lost: London’s Buildings Loved and Loathed” exhibition organised by English heritage at Quadriga Gallery, Wellington Arch. The event ( http://goo.gl/Qq6twI ) is going to be open until 2nd Feb – if anyone is interested in the architectural conservation history of london.

Our work in the exhibit was to build an AR app which adds animations (smoke,people, carriages) to a physical model and also augments it with the current day 3D buildings in the same area and RAF imagery. Screen Shots of the app (in the making) are below. The press coverage of the event can be seen at http://goo.gl/DtPG60 andhttp://goo.gl/OvQvMT .

Credits: 3D model of Bloomsbury by Blom

New Project – RefNet

After trying to organise the reading for my research last week, I realised that research process in my mind is not organised as a list or a checklist but as a network of interconnected ideas from various sources. This is where I felt the reference managers which I was using were failing miserably. Though they did a good job in organising the meta data on the papers, books and articles which I was reading and including them as references in my write-up, they did not help me in the research process. My research still remained as an exercise where I go through search engines and list of references in other papers manually and trying to put together all the stuff in my mind by myself. This is where I decided that If I cannot find a tool which I want I would rather build one myself and also that all the things I learned about networks and web development in the past year has to be put in use somewhere.

So here it is, RefNet – A reference manager which organises the references/bibliography as network of objects rather than a list. The idea is to build a tool where you can drag and drop papers and books as objects, and based on the citations in them they are organised as network of interconnected ideas. I started a github repository and using vivagraph library (inspired from here), put together a very preliminary working concept and added some data on the things I have been reading the past week. The result is as below, (click the image for interactive version)

refnet

The plan forward is to make the tool more dynamic with drag and drop option, automatic citation importing from a database such as web of knowledge, possibly a suggestion tool to say which papers to read further based on the network properties and finally a plugin to integrate this with google docs/ ms word. As mentioned earlier the project and code as of now is up on github (here) and would be really happy to collaborate with interested people on building this.

First Crowdsourced Project.

crowdsourced

The Image above is a static screenshot of dynamic, interactive and crowdsourced map I created to map people from School of Planning and Architecture, Delhi and see how they are distributed all over the world. I initially circulated within my batch and later in a broader group and the response has been really good so far with the counter crossing 100 as of yesterday.Though it is not some thing really advanced or jaw dropping, I am really excited to see how easy it is to collect and visualize data (especially geographic) if one knows the right tools. The tools used are MySQL server, Apache (PHP) server, JavaScript (with jQuery), Google Maps API v3, Chrome and Sublimetext.

The visualisation is similar to what I did for the IRIS competition earlier but the difference is in the backend. Instead of reading a preset datafile and displaying it, this map here has a MySQL database in backend and queries it through PHP and visualises the result. It also has PHP based POST mechanism to send data to the database from the user. The best part is that none of the data in the image above is collected or entered by me (except for my two data points). It is rather generated by the people who individually entered their own locations.

Webkit Speech API and Google Maps

For the past couple of weeks in addition to working for the dissertation project, I was working sporadically on a demo of a speech integrated map using x-webkit-speech API provided by Google Chrome and Google Maps JavaScript API.Now that the dissertation is over, here is the final polished version of the demo, (http://goo.gl/pqYUsN)

Demo for mixing -x-webkit-speech and google maps API

The current functions available with this map are,

1) Speak to navigate – Click on the microphone button (or  click “ctrl+alt+.” ) on the text box and start speaking. The field will recognise when you stop speaking, analyse and interpret the sound as text and if it is a place,  takes you right to place you just spoke.

2) Zoom and Pan – you can use the same trick with some preliminary commands as well, the system as of now understands ,

east direction“, “west direction“, “south direction” and “north direction” will pan the map in the corresponding direction.
zoom in/ zoom out ” zooms the map.

3) Other commands,

satellite – switches the map to a satellite map
simple – switched the map to, above shown simplified default look and feel.

As usual, I request the readers to give it a try and share the results & problems in the comments section below. Also feel free to put in your suggestions and point out any mistakes.