Posts Tagged ‘software

The Idea

leave a comment »

The idea is simple and have been done by numerous people before. It is a map based directory for a group of small people. The idea is an extension of the work I did years ago (post link). I built this site waiting for an entire day in the university campus for a (very annoying) friend who kept postponing the meeting. We finally met at the end of the day and by that time I had finished the web app. I had recently learned to use server side coding (using PHP), databases (MySQL) and had figured out that hosting is not rocket science. Coupled with some previous experience with Google maps API and JavaScript. I quickly built this crowd sourced project where everyone enters their Name and City and the map shows all the people, aggregated based on the zoom level. Simple, Clear and nothing fancy.

crowdsourced

Once this was posted to the common forums and people started using it. It became clear that I can do more with it. Suggestions kept coming in and after a point all the suggestions amounted to creating a whole new social network. Though I agreed there were loads of things which could have been improved with the app, I felt the usefulness of this map/visualisation was its simplicity and clarity. The things I felt could be improved were

  • data collection from users. Though it is simple as it is, I feel it can be improved by utilising availability of location data and connecting to other platforms (fb obviously). Also possible account management for users, edit and delete their own data.
  • Search and find people. This was never implemented. It could have been very useful for people to search and find others.
  • Some form of possible communication with other users. When you have a map like this and if you are able to find someone in the map, naturally you try to click on the person to see more details and communicate with them (even if it is a link to their email).

Things I specifically don’t want to do

  • Build a suite of information on the users and nag them to fill the information up. No body cares about someone’s primary school or their fourth work fax number. This will just make the whole thing a directory created with a aim to increase the amount of information rather than usefulness of the information.
  • Build a suite of interactions within the users and app while creating user interface elements for those interactions. This will lead into user being presented with loads of buttons& forms and when these UI elements starts covering the entire screen they will be folded into scores of menus. This just defeats the purpose of the entire thing. If people wanted millions of options to interact with the people they know, they can use Facebook or any social network for that.
  • Anything which is not location based or adds value based on location. Since location and mapping is the central theme of this project.

With all these things in consideration, I spent some time sketching out some preliminary ideas shown below,

I’ll explain them in detail in the following post.

P.S: I still think the idea shown in the sketch is too cluttered and needs significant simplification to be anywhere near neat and useful.

Written by sbmkvp

May 9, 2016 at 5:57 pm

aosm() – Acquiring, Filtering and Analysing OpenStreetMap Data

with 4 comments

Following up the last post, where I have outlined my overall understanding and plan for the intended function, today I have finished the first working code of the aosm( ) function and as promised in the last post, I am posting the code along with the explanations and instructions. Before going into the function I would like to stress the importance of getting the system setup  for running the function. I think getting all the software installations right, with the system environment variable updated with their executable files, is the biggest problem I faced while trying to get the function running on other systems. So please read the system requirements section carefully before trying to run the script.

aosm.r (alternate link)

instructions (alternate link)

System Requirements

1. Windows Operating system (windows 7 preferably) with administrator rights since installation of the software programs listed below is required.

2. R 2.15.2 (http://www.r-project.org/) installed and if possible R studio (http://www.rstudio.com/) which provides a better user interface (R studio needs an R installation to work.). Since there is nothing else in the script file, the function can be loaded to R workspace directly by running source(“c:\\%location%\\aosm.r”) command. [update(27 Mar ’13): the R session must be run in administrator mode for the system() commands to work in windows 8.]

3. Osmosis (http://wiki.openstreetmap.org/wiki/Osmosis) installed in the system, with ‘osmosis.bat’ (which is inside the bin folder) file location added to the system path variable. This is really important, the function relies heavily on osmosis without which it’ll never work. Osmosis is based on java, so make sure you have java installed and system path variable is updated with java location as well. To check if everything is OK, open command prompt and type run the commands “java” and “osmosis” to see if they are recognized.

4. 7zip (http://www.7-zip.org/) installed in the system with ‘7z.exe’ file location added to the system path variable. This is also equally important if you don’t have the OSM data locally in .osm format. After using the installer to install the program manually add the 7-zip folder to the system path. Again to check, run “7z” in the command prompt and see if it is recognized.

5. Since there is a lot of data which needs to be downloaded and extracted it is recommended to have atleast 2 GB of free space in the hard drive. The OSM data for London is around 150MB when zipped and it is almost 1.5 GB when extracted, so make sure you don’t run out of space. If you have multiple drives, please set your working directory to a drive with minimum 2GB space before running the function (the function does not changes the working directory at all, so all the downloads/ temporary files are kept in the current working directory)

6. Finally most important one is internet connection. The function was envisioned as a way to get data directly from the internet and the option to read local data is built to optimize the running time in consecutive runs. So the function makes a lot of references to internet and strictly requires internet connectivity to work. (I know this is absurd and am trying to build a work around. I realized this being a real problem when internet outage lasting for half a day in the UCL Halls left me paralyzed when developing the function)

The Function

Syntax:

name.object <- aosm (“world”, ”geo-filter”, ”tag-filter”, ”analysis”, ”type “)

Inputs:

“world” – A string, which is the name of the city for which the data to be developed. The string has to be in lowercases and if the city name has spaces in it, then it has to be replaced with “-“. (E.g. “london”, “san-francisco”). The script checks for local file availability in the current working directory and if it cannot find one, then it downloads it from the OSM extracts. There are three formats to supply this data locally: ‘.osm’ file, ‘.osm.pbf’ file or ‘osm.bz’ archive.

“geo-filter” – A string, which is the name of the area within the city for which the data has to be extracted.The string has to be in lowercases and if the name has spaces in it, then it has to be replaced with “-“. (E.g. “Islington”, “city-of-london”). It denotes the name of a .poly file which can be supplied locally or can be downloaded by the function from the internet. as of now, I have boundaries of all the boroughs in London hosted in the server (http://balaspa.50webs.com/poly/) and would be updating it as I get more time. The other way to supply the boundary file is to have a shape file in the current working directory named “boundary.shp” with the polygon you want to use with the name of the polygon in the attribute table under the header “NAME”. By default any shape file in the current directory with the name – “boundary.shp” will be converted into individual polygon files with the string in the “NAME” column as file names. for example, if you keep a london borough boundary shape file (with the name “boundary.shp”) in the working directory, the function will extract all the boroughs as .poly files.

“tag-filter” – A string, denoting filter definition. The syntax is “switch_name”. Where, “switch” is either “d” or “t” denoting if the name is a definition file or the tag filter in itself. OSM has a really straight forward way of tagging its features which is every tag has Key and Value. For example a way can have a “building” tag and a value of “yes” which marks it as a building and a way can have a key of “highway” and the value of “residential” which makes it a residential street. So there is two ways of building a tag based filter one is by just writing  the key and values directly in the function (t_highway,residential or t_building,yes) or for more complex filters, by making a definition file and keep it a text file in the working directory and pointing it in the function (for example d_landuse, where the function will search for a file named “landuse.txt” for the definitions). There are two sample definition files I have hosted in the server which the function can download (http://balaspa.50webs.com/def/). If there is only one value in the tag-filter definition, then all the features with the corresponding key are extracted regard less of the values. One can see what are all the keys and values used in OSM by the volunteers in the wiki page or taginfo, which will give an idea of how things are organised in OSM.

“analysis” – A string, which defines the type of analysis to be done on the data extracted and the type of result expected from the function. Currently supports the following the values of  “default” (for an sp object), “utm” (for an sp object with CRS), “cn” (for count of features), “ar” (for sum of all the areas of the features) and “len” (for sum of all lengths/perimeters of the features). A detailed explanation can be found in the outputs section.

“type” – A string with one of the these three values – “points”, “lines” or “polygons”. This is determine the type of sp object which is returned by the function.

Outputs:

The output from the function differs significantly based on the “analysis” string in the input. The possible strings and corresponding analysis are given below.

“default” – returns an sp class object (SpatialPointsDataFrame, SpatialLinesDataFrame, SpatialPolygonsDataFrame) with the added attributes showing the key-value tags and the name tags without any CRS information

“utm” – returns a similar object to above but with the CRS information using Universal Transverse Mercator and WGS84

“cn” – returns the count of features

“area” – returns the sum of areas of all the features in the resulted data in square meters

“len” – returns the length/perimeter of all the features in the resulted data in meters.

Methodology

The function aosm( ) takes 5 inputs and applies 16 sub functions on the inputs to generate the results. Since I have attached the source code of the script and this blog is getting really long, I would like keep the explanation brief.

The function first sets the environment by installing all the required packages. It then checks the WD for boundary shape file and if found, converts it to .poly files. It then evaluates the inputs to see where and in what formats do the required data exists and creates a data frame explaining the situation. This involves checking for locally available data and data available in the internet sources for all compatible formats. The next step checks the situation and evaluates if the function can continue. If it finds any errors or missing information, it reports the error and shuts down the function before any intensive task is started. Once the validity of the inputs are confirmed, the function then arranges the data from the available formats, downloads it and converts it to the desired format. Here local data is given preference over data on the internet. Once the data is arranged, the function invokes osmosis for the filtering process and makes a system() call based on the inputs. Once the filtering process is complete, osmar is used to import the filtered data file to a sp object and the extra attribute information is attached to it. The resulted object is then projected using UTM projection and WGS84 datum. As the final step, based on the inputs, the function applies appropriate analysis on the sp object and returns the results.

If you are really interested in how the script works, then the following chart explains how all the sub functions are tied together and process the inputs. I would also recommend consulting the instructions file referred in the start which explains all the functions as well.

Complete Flow Diagram of the script

Demo

map<-aosm(“london”,”city-of-london”,”t_building”,”default”,”polygon”) +plot(map,col=”#B43104″,bor=“#B43104”)

or

map<-aosm(“london”,”city-of-london”,”t_building”,”utm”,”polygon”) +plot(map,col=”#B43104″,bor=“#B43104”)

will return the following plot,

count<-aosm(“london”,”city-of-london”,”t_building”,”cn”,”polygons”)

will return the value 1364

area<-aosm(“london”,”city-of-london”,”t_building”,”ar”,”polygons”)

will return the value  1312264.90810641

length<-aosm(“london”,”city-of-london”,”t_building”,”len”,”polygons”)

will return the value 337414.610756376

So concluding this extremely long and drab post, I would request the readers to give it a try and share the results & problems in the comments section below. Also feel free to put in your suggestions, point out any mistakes and tell me about any other existing solutions which may serve same purpose.

Written by sbmkvp

January 8, 2013 at 10:04 pm

Experiments with Google Maps API

leave a comment »

After spending lot of time with the KML documentation and C, It was time for me to go beyond the associated platform restrictions (C programs are compiled for a single platform) and start doing the computing and displaying maps on the web. This sparked my venture into the Google maps API and  JavaScript.

The starting point of this entire exercise was my short internship with Report Bee (A tech start-up with the vision to revolutionize the education industry).  My job there was to inquire into the ways in which geographic visualization can enrich the analysis of the data present in the results of public examinations in Tamil Nadu and device a frame work for its implementation. It was an amazing ride to for me to start from absolutely zero, with an oversimplified view of things and vague ideas and go on to find out that the whole thing is neither simple nor easy as I thought. Though the exercise proved to have an extremely large scope to be covered in such a short internship, I picked up a lot of knowledge while figuring out my way.

My first challenge, after getting lost in the quagmire of tools and platforms, was to finalize a platform to  learn and work on. The main dilemma was between Google Maps and Open Street Maps. While I liked the freedom and openness of OSM with the base data for the maps, I also found Google Maps really compelling, with its relatively simple JavaScript API and extensive documentation. Though there was a lot of restrictions introduced in Google Maps recently, I chose it over OSM considering its simplicity and my previous experience with KML. Moreover almost all the maps API (Google maps, Map-Box, Bing Maps etc.) had JavaScript in the core so I thought may be it won’t be hard to learn to use others platforms if I get used to one of them.

So after almost three weeks of wrestling with Google Maps API and JavaScript. The end result was that I was able to produce a website with a custom styled map highlighting some of the features in the base map with some kind of user input to point location and has an implementation of Geo-coding API for finding and navigating to addresses. The whole process was extremely interesting and especially JavaScript surprised me with its simplicity and flexibility. I have attached a screenshot of the result below and the website can be viewedhere. (http://balaspa.50webs.com/mapv2/)

Experiments with Google Maps API.

Results

The map highlights all the parks in green colour. The instruction are as below

  1. Zoom and Pan can be done through the mouse as it is done in Google Maps.
  2. Clicking on the map creates a Placemark and adds it to the map.
  3. The ‘Eye’ button on the top right corner toggles the visibility of the Placemarks.
  4. Double clicking the eye clears all the Placemarks.
  5.  The search box below navigates to the location typed on pressing the ‘return’ key.

Since I have covered a lot of basics in this exercise (Customized Base map, simple ways to input and output data, rudimentary ways to build a UI over the map), my next step would be learning to organize a data sets in a server side database and plotting it on the map and also to look into the Open Street Map API parallelly.

Update (21 August 2013): Was randomly reading about the Google speech recognition API for chrome and stumbled upon a simple way to integrate speech based input fields in websites. Updated the search box with speech recognition. In my trials it looks clean enough and works well enough. Wish there were more options in controlling the behaviour of the box but I think its just a matter of figuring out.

Update (21 August 2013): Updated the speech recognition again. Now press “ctrl+shift+.” and speak the place name the page directly takes you there. No more clicking and pressing enter.

Written by sbmkvp

October 29, 2012 at 4:21 am