A Bloomberg like ‘Terminal’ for open data?

bloomberg_terminal

Recently I had a tour of to Bloomberg HQ in London from my friend who started a job there. It was the first time I actually saw Bloomberg terminal and had a go at it. There are three things I found very fascinating and which Bloomberg terminal has done right.

  1. The User Interface.
  2. Data completeness.
  3. Community.

User Interface (UI)

Bloomberg terminal interface is neither command line nor GUI, it is a hybrid. If I have seen any other software which masters this kind of hybrid interface that would be AutoCAD. I have worked with AutoCAD exclusively for almost 4 years from 2006 to 2010. In that time almost everything I did was inAutoCAD – presentation, maps, drawing, simple mathematical calculations, geometric problems. I used it for anything that required visual thinking or communication. Since then I have used other suite of software aimed at other things – Adobe Suite for graphic design, Max for 3D modelling, Unity for game development etc etc. but never have met an interface that match up to even half of what AutoCAD has (or it used to have). The key thing at which AutoCAD excels is taking the best of a two interface types, combining them in all possible ways and let the users figure out how to use this blend effectively for their needs.

There are two major types of interfaces when it comes to productivity software – Graphical User Interface (GUI) and Command Line Interface (CLI). In a GUI every action you do has a on screen visual element which you can interact with. The best example for this is Microsoft Office. Literally every thing you do in office has a dialog, menu or button somewhere. The other extreme is commandline interface, where everything you do is or part of a command/function. Even thought there is a lot of debate about which one is better and a trememdous amount of personal preference (I am a commandline guy through and through), I have to admit that both have their advantages and disadvantages.

GUI – is easy to learn, incredibly hard to master, simple things can be done quickly but repetitive complex tasks cause problems and finally handles spatial or graphical information well but struggles with factual information. For example, Microsoft Word is easy to learn, anybody can pick up within an hour but word “experts” are hard to come by. Typing a simple letter in Word is terribly easy but complex such as making customised labels for 200 items are truly horrifying. Drawing things, moving features which have 2/3 dimensions to them is very clear but logic such as if-then-else, concepts such as routines, functions, recursion are alomst impossible. I still haven’t seen any visual/GUI programming thing that is worth using, I think after certain level of abstract concepts cannot be visualised with GUI. Commandline is complete oppsosite of this, it can do complex, abstract, repetitive tasks very easily but struggles with anything that has spatial element to it. eg. graphic design, CAD etc are extremely hard without GUI. It has a huge value for a poweruser but has no ease of use.

Except for very few old ones, I dont think any software stick with just one or the other philosophy anymore. With web and mobile apps, things have gotten more and more ambigous. GUI applications tend to provide some commanline like functionality with shortcuts (eg. Excel, Adobe) and some form of programming or scripting interface for complex tasks (eg. VB scripting). CLI applications tend to solve their spatial information problem by adopting layouts (vim, mutt) and also tend to be modular so that GUI can be built on top of them if necessary. Usually well thought out software borrow from both and try to strike a balance but very few are successful in my opinion. AutoCAD and Bloomberg terminal are those who got this so perfect that once you get used to them it is hard to move to anything else.

From what I have seen, Bloomberg terminal seems to be inherently command-line based while having a powerful GUI alongside. The GUI seems to be strictly utilitarian (maps , charts etc.) and all the GUI elements are mapped to some key or other. It seems to be easy to use for new users with the GUI and at the same time rewards them immensely if they learn to use CLI.

Data Completeness

If you are in the realm of financial markets, Bloomberg terminal provides a perfect bottom line for all your data and visualisation needs. There are no extra plugins to buy, no external tools to use with it and no major data cleaning or transformation needed to use it. One can get the data, analyse, plot and infer from it all in the terminal without using anything else. The data is ready to use, vis tools are ready to use and all you concentrate about is extracting the meaning out of it.

Bloomberg does have the advantage of working with data from the highly regulated financial markets but also has an immense challange in other sources of data, for example news and social media which can be quite varied and diverse. Aggregating all these sources and providing them comprehensively in a single place creates amazing value. 80% of all data analysis is formatting the data and the terminal takes that out of it. Also it seems that they go for the comprehensiveness. In their field, they pretty much have eveything covered and constantly have people covering new things that are popping up. The general attitude is – If it is not on terminal either it is irrelevant or it doesn’t exists.

The open-data stores miss this regularly by being in multitude of places, embracing multitude of formats using a directory based storage which is specific towards a certain set of activities. For example, London and UK has their own data stores which are all full of static datasets which needs to be accessed using own authentication methods in at least 5 different formats. Simple things like plotting the boundary of a borough of london will take all your knowledge of GIS and spatial data structures. Bloomberg terminal just takes all this away from the user. You don’t even have to know what a csv is to see how the markets are doing. When we take away this task off the users shoulders we can empower the user in ways we might have never thought about.

For example, if I ask the question of how many people are there in Camden Borough of London right now. There is no one place to quickly get the answer. Yes, of course I can Google it but it is equivalent to searching a library rather than a database. The data is open, available and free. The only thing that is preventing that to be used by anyone is the way systems are set up to access the data. This is where the terminal shines. If you want to know something, ask the terminal, if cannot find it, call the 24×7 support and ask for it. It is not there, it most probably doesn’t exists.

Social Network

This is a big one. It is absolutely amazing how the terminal users is connected not only to the data and customer service but also to each other. They can mail each other, they can chat with each other, they have forums, groups etc. etc. With Bloomberg tv they can even communicate with people who are being interviewed on TV. Social media streams into the terminal. If you are a trader working with other traders, terminal is one stop solution. No need for 15 other services to do the basic job (which is just data analysis and communication). I use at 10 different communication tools to collaborate with people I work with and each tool has it own work flow. Though I tend to get everything down to a unix shell interface for my own sanity, I can imagine how cool it must be to have a consistent work flow with everyone within one single application. Just by connecting the users to each other, the terminal adds exponential value to every user!

Open data terminal?

I personally want these advantages to be available for open data. Imagine a open data terminal built on a hybrid user interface which has a comprehensive set of data available to public along with tools to interpret them and communicate with other users simultaneously. It will be such a game changer for so many people. For example, public institutions who make policies based on the data, research institutions who use the data to advance our understanding of the world and volunteers/ journalists who report on the data can do their jobs quick and easy without spending large amounts of time and resources in the data cleaning and verification process. I would like to see all the open data stores work together to make a common platform for making all their products available to the users in a simple ubiqutous way. Just imagine doing,

data [dataname]@ visualisation [vis-type] parameters

to just about any data source available. Imagine adding the vast spatial data from openstreet map to this mix! It will be phenomenal. May be as a hobby project, in the coming months I should pick up a small unit (a borough or small town) and try to gather all open data possible and build something similar to what I have in mind. Lets see how it goes…

Advertisements

New Project – RefNet

After trying to organise the reading for my research last week, I realised that research process in my mind is not organised as a list or a checklist but as a network of interconnected ideas from various sources. This is where I felt the reference managers which I was using were failing miserably. Though they did a good job in organising the meta data on the papers, books and articles which I was reading and including them as references in my write-up, they did not help me in the research process. My research still remained as an exercise where I go through search engines and list of references in other papers manually and trying to put together all the stuff in my mind by myself. This is where I decided that If I cannot find a tool which I want I would rather build one myself and also that all the things I learned about networks and web development in the past year has to be put in use somewhere.

So here it is, RefNet – A reference manager which organises the references/bibliography as network of objects rather than a list. The idea is to build a tool where you can drag and drop papers and books as objects, and based on the citations in them they are organised as network of interconnected ideas. I started a github repository and using vivagraph library (inspired from here), put together a very preliminary working concept and added some data on the things I have been reading the past week. The result is as below, (click the image for interactive version)

refnet

The plan forward is to make the tool more dynamic with drag and drop option, automatic citation importing from a database such as web of knowledge, possibly a suggestion tool to say which papers to read further based on the network properties and finally a plugin to integrate this with google docs/ ms word. As mentioned earlier the project and code as of now is up on github (here) and would be really happy to collaborate with interested people on building this.