Sending mail from command line using mutt

Sometimes you just don’t have the patience to open a GUI. Imagine you are working on a terminal remotely through a very feeble internet connection and after hours of data wrangling you got your results in one small package. Now all you want is to email this 200kb document (average size of a 20k word .txt document). You can either load a GUI, open a browser, open gmail (the login page itself is 2MB), attach the file and send the email or just execute a one line command which does everything for you. With some minimal setup you can do the latter – sending email via CLI just like any other shell command. You can even include this in your scripts (send mail when the script finishes running etc).

We will do this using a terminal program called “mutt” which also has a brilliant CLI interface and will configure it to use gmail via imap. First step is to install mutt using a package manager, (apt/ yum/ pacman for linux and brew for macosx). I am doing this in Arch with pacman. I am installing mutt and smtp-forwarder and then create necessary folders and files for mutt.

sudo paman -S mutt smtp-forwarder
mkdir -p ~/.mutt/cache/headers
mkdir ~/.mutt/cache/bodies
touch ~/.mutt/certificates
touch ~/.mutt/muttrc

Edit the muttrc file with your favourite text editor and add these configurations, (make sure to change the username to your username and if your are using two factor authentication with gmail the password has to be generated from App passwords.

set ssl_starttls=yes
set ssl_force_tls=yes
set imap_user = 'username@gmail.com'
set imap_pass = 'yourpassword'
set from= 'username@gmail.com'
set realname='yourname'
set folder = imaps://imap.gmail.com/
set spoolfile = imaps://imap.gmail.com/INBOX
set postponed="imaps://imap.gmail.com/[Gmail]/Drafts"
set header_cache = "~/.mutt/cache/headers"
set message_cachedir = "~/.mutt/cache/bodies"
set certificate_file = "~/.mutt/certificates"
set smtp_url = 'smtps://username@smtp.gmail.com:465/'
set imap_pass = 'yourpassword'
set move = no
set imap_keepalive = 900
set editor = vim
bind pager j next-line
bind pager k previous-line
set sort = threads
set sort_aux = reverse-date-sent
unset imap_passive
set imap_check_subscribed
set mail_check=60
set timeout=10

That is it! Now we can send mail from terminal by just passing some text or a file with the text to the mutt command,

echo "email body"  | mutt -s "email-subject" -- recipient@gmail.com
mutt -s "email-subject" -- recipient@gmail.com <  file_with_body_text.txt

we can even attach files like this,

echo "please find attached"  | mutt -s "email-subject" -a "attachment.pdf" -- recipient@gmail.com

 

Advertisements

Installing Arch Linux (and R) on an Android device

This is a really recent development and I am very excited about this. I finally found a way to have a somewhat proper Linux installation on my phone. Though it might not be the best place to have a CLI, it is really promising and I can rely on this to do some small stuff on the go. As the tools I use are getting simpler (Photoshop vs Imagemagick) and the hardware of the phones I own are getting better, it is should possible for my phone to do the things my 5 year old laptop could handle provided the right environment.

This is done by installing a full Arch installation on an Android phone under the termux environment using the installer from TermuxArch. The installation here is actually way easier than installing Arch on a normal desktop. We start by installing termux android app. When we open termux we get a bash shell. From here we install wget by running, pkg install wget . When this is complete we download and run the Arch installation script by,

# Download the script
wget https://raw.githubusercontent.com/sdrausty/TermuxArch/master/setupTermuxArch.sh 
# Adding execute permissions
chmod a+x setupTermuxArch.sh
# Run the script
./setupTermuxArch.sh

Now we can just follow the instructions in the script which will download and unpack a base Arch Linux system and ask you to edit the mirror list. At this point, just  un-comment (remove the #) of the closest mirrors and save and exit the file. When the installation is complete you have a vanilla arch system on your mobile! Now we can theoretically install and use any program I have on my work desktop on my phone which including to ssh, vim, git, latex, R, node, postgres, mongodb, etc etc. I can even ssh into my work desktop straight from here. Below are some screenshots of the system (the chart is done entirely on phone!).

 

Mapping distribution of National Institutions in Higher Education in India [R + tidyverse + tmap]

Since I started learning R and moved away from proprietary data analysis and GIS packages, I have been amazed by the availability of free data-sets and tools enabling people to make awesome analysis and visualisations. Today we’ll look into a simple exercise of taking open data from different sources and combining them using opensource and free tools to produce maps which help us understand the data better.

We’ll use data from two sources, a tabular file on distribution of national institutes of higher education in India from here and shape file containing geographic information on boundaries of Indian states from here. I have cleaned and prepared both data which can be downloaded here. First we need to download the archive and extract/ unzip it to our working directory. Once that is done we can start combining and plotting the data.

# Loading the data
data <- read.csv("institutions.csv")
library(rgdal) # library for reading shape files
states <- readOGR(".", "india_state")

Note that, in readOGR, the first parameter is the folder at which the shape file is kept (it should be “.” if we directly unzipped the files to working directory) and second one is the name of the shape file.

# calculating the total number institutions and 
# the number of institutions per 10 million people
data <-  data %>%
mutate( Total = rowSums(.[,3:10]) )%>%
mutate( Totalppm = Total/Population)

# merging the data into the shapefile
states <- states %>%
merge(data,by="state")

Now we have succesfully merged the data into the shape file we can see this by asking states@data at the console. Now we need to map the data using tmap. First of all we load the library by running library(tmap). This is complex package and has a lot of dependencies so it might take a while to install and download. As a first step we plot just the borders of the state by running,

tm_shape(states) +
    tm_borders(col = "grey")

1

We can add labels to the map by adding a tm_text layer to it. Notice that the order of the layer is important since things overlap on each other.

tm_shape(states) +
    tm_borders(col = "grey") +
    tm_text("state", root = 10, size = "AREA")

2

Now we can plot a single variable on the map as the fill colour for the states. For example If we want to highlight all the states with an IIT, we do,

tm_shape(states) +
    tm_fill("IIT") +
    tm_borders(col = "grey") +
    tm_text("state", root = 10, size = "AREA")

3

We can plot multiple variables on the same plot side by side by just passing a vector of variables to compare. Notice that we switched of legends on one of the layer here by setting legend.size.show=FALSE.

tm_shape(states) +
    tm_fill(c("IIT", "IIM")) +
    tm_borders(col = "grey") +
    tm_text("state", root = 10, size = "AREA",
            legend.size.show = FALSE)

4

Finally we can plot the number of institutions per population by,

tm_shape(states) +
    tm_fill("Totalppm",
            title = "Institutions per 10m people") +
    tm_borders(col = 'grey') +
    tm_text("state", root = 10, size = "AREA",
            legend.size.show = FALSE)

5

This map is very uninformative because of the small Himalayan states with low populations skewing the whole distribution. So we  have to classify this data ourselves by giving a custom breaks parameter,

tm_shape(states) +
    tm_fill("Totalppm", 
            breaks = (1:10) / 2,
            title = "Institutions per 10m people") +
    tm_borders(col = 'grey') +
    tm_text("state", root = 10, size = "AREA",
            legend.size.show = FALSE)

6

Now this shows a possible north south divide in the distribution of institutions per person. This may be because of most of the national institutions in North being located in Delhi, while in the South Bangalore, Mumbai and Chennai compete for them.

That completes the post for today. To summarise we took a tabular data, joined and plotted it with geographic data and uncovered new information which are not present in them individually!

Data manipulation basics with tidyverse – Part 2 – Basic functions

In part 1 we saw how to use pipes to pass data between functions so that we can write R code like a sentence. The second impressive thing with tidyverse is the grammar for manipulating data. The way the functions are structured and named in tidyverse gives us a consistent way of writing R code which is clear, concise and readable. Except for very few cases, I almost always find myself using just 5 basic function with tidyverse ,

  • select – select columns
  • filter – select records
  • mutate – modify columns
  • summarise – combine records
  • arrange – arrange records

consider the sample table below,

| name  |  year | sex | town   |
 ----------------------------
|  A    |  1998 |  M  | London |
|  B    |  1995 |  M  | Berlin |
|  C    |  1994 |  F  | London |
|  D    |  2000 |  F  | Madrid |
|  E    |  1995 |  M  | Berlin |

1) Select function is to select vertical columns from the table. for example, select(year,sex,town) will return table with just the the three columns selected. We can even rename the columns as we select them (or use rename() as well), select( year, sex, city = town)

|  year | sex | city   |
 ---------------------
|  1998 |  M  | London |
|  1995 |  M  | Berlin |
|  1994 |  F  | London |
|  2000 |  F  | Madrid |
|  1995 |  M  | Berlin |

2) Filter function is to select records based on a criteria. for example, filter(year < 2000) will select only records where year is less than 2000. we can even combine multiple criteria with logical operators & (and) and | (or),

|  year | sex | city   |
 ---------------------
|  1998 |  M  | London |
|  1995 |  M  | Berlin |
|  1994 |  F  | London |
|  1995 |  M  | Berlin |

3) Mutate function modifies columns. for example, mutate(age = 2018 - year) will create a new column with name age and calculate it based on year,

|  year | sex | city   |  age |
 -------------------------------
|  1998 |  M  | London |  20  |
|  1995 |  M  | Berlin |  22  |
|  1994 |  F  | London |  23  |
|  1995 |  M  | Berlin |  22  |

4) Summarise is a two-part function which combines records based on one or more columns based on a formula (function). for example, if we need average age of people in cities according to gender, we can do – group_by(city,sex) %>% summarise(average.age=mean(age)) gives us,

|  city  | sex |  average.age |
-------------------------------
| Berlin |  M  |       23     |
| London |  F  |       24     |
| London |  M  |       20     |

5) Arrage function arranges records based on the value in the columns specified. for example, arrange(average.age) gives us,

|  city  | sex |  average.age |
-------------------------------
| London |  M  |      20      |
| Berlin |  M  |      23      |
| London |  F  |      24      |

I have found that most of the data manipulation can be done combining these 5 functions in tidyverse and the best part is that the resulting code translates really well to english. All the stuff we did earlier can be written down in a single line, clearly without any intermediate objects or referring to the data we are working on repeatedly. For example,

people %>% 
    select(year, sex, city) %>%
    filter(year < 2000) %>%
    mutate(age = 2018 - year) %>%
    group_by(city, sex) %>% summarise(average.age = mean(age))
    arrange(average.age)

Translates to,
Take the people table, select year, sex and city columns, filter for records where year is less than 2000, calculate age column from year, group the table by city and age and find out average age for the groups and arrange the records by age.

Minimal Latex environment for Linux (vim + entr + mupdf)

In the past year I have completely moved to the linux environment and it has been great for everything I do except for document writing. I have read about Latex and played with it for sometime but the whole process of write file > build pdf > view pdf looked to be tedious and using a IDE was not my preference (I could just use office on wine if thats the case). All I wanted was a way to edit text file using vim and see the resulting pdf in a window realtime.

After playing with few pdf viewers and utilities. I think I have the perfect solution. It has three components

  • vim (text editor)
  • entr (simple linux program to monitor files)
  • mupdf (lightweight pdf viewer)

Open the .tex file in vim, open the final pdf file in mupdf. The run the following command in the background. Thats all.

ls your.tex | entr /bin/sh -c “pdflatex your.tex && pkill -HUP mupdf”

To explain, ‘ls your.tex’ lists the file to be monitored, entr runs the command between the double quotes using the program /bin/sh when the monitored file is changed. The commands inside the quotes are to build the pdf using pdflatex and update mupdf. So now every time you write your .tex file in vim, mupdf will show the final output. The entire environment is shown in the video below.

ps. I am using i3 for the split between terminal and pdf viewer and screen for split inside terminal.

Understanding Javascript Objects and JSON Data.

The first time I heard of JSON (JavaScript Object Notation) when was trying to get data out of twitter. At that time I was new to javascript and was figuring out a lot of stuff at once so the whole thing was very confusing and incredible hard to grasp. Now when I think back, It would have saved me a lot of time if someone just gave a clear overview of how objects worked in javascript and how the same pattern (jargon alert!) when used to transfer data becomes JSON. This is the reason why I am writing this post. As a disclaimer, I am not a programmer but an urban planner and my understanding of this subject is purely based on my practical experience trying to build things relevant in my field so bear with me if any of this is inaccurate or wrong. Please feel free to point out the mistakes.

To start, let us set up the environment for learning. Since objects in JavaScript and JSON data are abstract concepts there is no way for a normal person to understand these without actually seeing them in action. So it is time to open the console in your browser (preferably chrome) and start typing. The console in chrome can be started by pressing (cmd+alt+j or ctrl+shift+j) and you can just type in the commands one by one. It is as simple as that.

Chrome JS Console

By default the console opens in the global namespace of the tab which is open. In plain english the console is like a virtual space where you create, destroy, modify objects and these objects gets displayed in the browser window (rendering) based on their properties. For example the tab has an object associated with it, the html document, the body and every element (image, text etc) has an object associated with it. This is called DOM – Document Object Model which is used to manipulate HTML elements with JavaScript. The point is this is a virtual space to contain objects and this is where we would be working. This space already has a lot of objects and one shouldn’t confuse them with our own. To check our understanding lets type ‘window’ in the console and press return. It returns some text, this is the object which denotes the chrome window. When you click the triangle in the left side (expand it) you can see all the contents of the object.

Screen Shot 2014-12-03 at 16.01.29

Now typing window.innerHeight or window.innerWidth will give you the height and width of the window. You can resize the window and type it again to see it changing. Here the important things are, 1) We can create objects in this virtual space with properties to represent stuff we see (a window in this case) 2) the “.” denotes the property of an object (i’ll elaborate this later).

Now it is time for us to create our own object. Lets say this object is a digital representation of yourself. Lets start by creating the object by typing me = {}. This literally translates to english as ‘me’ is an empty object. The curly brackets denote that ‘me’ is an object and it is empty since it has nothing in it. Now typing ‘me’ again and pressing enter will return an empty object. Now lets go one step ahead and ask my name. Since it is a property, you use the “.”. So type “me.name” and see the result. It says it is “undefined” because we have created the object but never defined any properties.  Lets set one by typing me.name = ‘bala’ (note the single quotes. i’ll explain later) . Now asking me.name will return the name “bala” and asking for the whole object ‘me’ will now give the properties as well.

Screen Shot 2014-12-03 at 16.16.35

If you have come this far then you understand object, its properties, how to create and return them. Now it is time to understand some data structures (this sounds way too complex than necessary). For practical purposes we just have to know 4 major data structures – Number, String (of characters), Array (of elements) and Boolean. Most of the data we use will fall under these data structure. Numbers are stored as is without any special way to mark them. for example, Typing in me.age = 25 will set the property ‘age’ in me as a number which is 25. Strings are denoted by double/single quotes around them if the quotes are not present then it is considered as a name of an object (variable) and will return an error if it cannot find one. for example, me.city = london will return an error while me.city = ‘london’ will be OK. Arrays are collection of data in a specific order and is denoted by square brackets. for example, me.address = [221,’b’,’Baker Street’] will set the address as an array of 3 elements. typing me.address will return and array, while typing me.address[0] will return 221. the square brackets with a number is the way of accessing an element within an array when you know its place (the counting starts at 0). Try to get the street name out of the ‘me’ object.

Screen Shot 2014-12-03 at 16.48.55

A string is like a special case of array (with only single characters) so you can do the array type of queries to strings as well. for example, me.city[0] will return ‘l’. Boolean is either ‘true’ or ‘false’ (without quotes). for example, you can do me.graduate=true. This helps where you can directly use this instead checking for conditions (like if else statement). With this 4 basic data type you can create a model for almost every kind of objects we usually encounter. To sum up, we create objects with curly brackets, object is something which has properties, properties can be of various types of data (number,string, array, boolean), we set and access properties of objects by using the ‘.’, arrays are created by using square brackets and we set and access contents of array by using square brackets.

There is one more thing we need to know which is that there is a simpler way to create objects than setting properties one by one. which brings us closer to the JSON. instead of using the ‘.’ we can directly write the contents of the object using ‘,’ and ‘:’. So combining all the steps above in creating the me object, we just do,

Screen Shot 2014-12-04 at 18.23.37

With these basics we can now move to interesting stuff – Nesting and References.

Nesting:

When we talked about data structures though we talked about 4, we actually learned 5. The fifth one is objects. This means that a property of an object can be an object. This introduces amazing capabilities to javascript objects. for example, me.education = {} will create an empty object for education. and me.education.school = ‘kvp’ will set the property for me.education. This process can be theoretically repeated forever (if the memory permits).

Screen Shot 2014-12-04 at 17.56.38

This has two major significance. 1) While modelling sparse data, this makes our object memory efficient (this is the reason it is used in data transfer). for example, a table which shows scores of 5 students in 10 different courses and every student attends 3 courses can be modelled as a javascript object as shown below,

Screen Shot 2014-12-04 at 18.13.21

Screen Shot 2014-12-04 at 18.12.52

You essentially don’t need to have null/empty spaces. If something is null then it is just not there. 2) you can model infinitely rich objects by nesting different types of objects together. e.g. in the above example if some course has two markers and two scores you just introduce and object instead of the number, no need to change your structure of the data ( schema ).

Screen Shot 2014-12-04 at 18.28.44

References:

This is another way javascript optimises memory usage. Every object’s association with its name is just a reference. for example create an variable a = 20, create another variable using this variable b = a , check both variables. now do b = 40 and check the value of a. The results are shown below. This is logical and is what you expect to happen.

Screen Shot 2014-12-04 at 18.34.36

Now try and do the same with objects, a = { name:’bala’, age:25 } ; b = a ; b.age = 27. Now check the contents of a!

Screen Shot 2014-12-04 at 18.42.44

This behaviour is because JavaScript does not duplicates objects in memory when assigned to variables but just references the object to the variable name. This allows us to create infinitely rich data structures with finite memory. Funny example is below,

Screen Shot 2014-12-04 at 18.43.53

That concludes our overall introduction to understanding objects in javascript. With this foundation, let us move to JSON.

JSON

To be brief, JSON is just a notation to create javascript objects. It is exactly the same as what we discussed above. The only thing which is new is the property names should be enclosed in quotes (since JSON has no concept of variables). You evaluate a JSON string and it gives you a JavaScript object thats it. Nothing more. While thinking about JSON, as an alternative you can think of it as a collection of key value pairs (this kind of thinking helps while working with php). Keys are always strings and values can be anything (even another collection of key value pairs).

Screen Shot 2014-12-04 at 18.51.11

The difference between JavaScript objects and JSON is that JSON is a notation i.e. it is a text file written outside javascript and cannot be executed. It is a normal data / text file like csv. You can essentially read a csv as a string, split it into parts using the newline and comma characters and create an object/array in javascript but with JSON it is much simpler (just do eval) and more powerful (nesting).

Thats it. Hope this helps absolute beginners as myself in understanding these concepts faster.

P.S. If you have come this far, congratulations! You now know how data is modelled and stored in mongodb! All you have to do to use mongodb is learn commands.

Setting up Asset Server with Unity3D over local-network

There are two problems faced by everyone who is starting with Unity3D. First is version control – There is virtually no rollback mechanism and any changes done at the project level is absolutely irreversible which causes most of the data loss. Second is collaboration – if you are working as a part of a team, it is absolutely painful to transfer and sync files across workstations. Unity has an elegant built in solution for both these problems with its asset server and team license. But setting up an simple local asset server with version control is tough if you are an absolute beginner let alone making it remotely accessible. So here is a walk through for setting up an asset server over a local network in windows environment which I figured out after spending a some time.

The Basics:

Before we go into the entire process, it is better to understand how a local network works and how does a server – client relationship works in it. When I say a local network, I mean a set of computers connected through wifi/lan with or without a router. Here the router is also a computer in itself (usually running an embedded Linux) which is capable of only one specialised function – ‘routing’ traffic between computers inside and outside of the network.

I think it is better to understand the setup using a real world analogy. Consider every computer as a house/property in real-world and communication between them as sending mail between these properties. Here the local network would be a neighbourhood and the router would be a neighbourhood post office. Post office is a property in itself but its only purpose is handling mail.

Now imagine that you are in property A and a letter has to be sent to Property B within the neighbourhood. So naturally you will write the address of property B on the envelope and send it to the post office and the post office delivers it to property B. Here the address of a property has a digital counterpart – IP address. IP address is unique to every computer within a local network and even the router (post office) has one. Since post office has a fixed address, all you mail by default goes to the post office. This is a simple process and it is exactly how a local network of computers works, except for a small complication.

Properties usually don’t have just one occupant; they have multiple people living in them. So what if you want to send a letter to an occupant living on property B? You write the number of his mailbox along with the address. Isn’t it? This is where the concept of ‘ports’ come in. so imagine a computer in a local network with multiple programs running on it. Every program which wants to connect to another specifically listens to a ‘port’ allocated to it, so that any packet received at the port directly goes to the program. The diagram below illustrates the concept.

loc

Translating this concept to our unity setup, we have two types of programs, unity asset server and the Unity3D software itself. unity asset server will reside in one computer in the network which will be our server and listen to a specific port and Unity3D software (client) will be residing in all the computers in the network. One interesting case in this setup is that it is possible to have a Unity3D (client) running on the server computer itself. The diagram below illustrates the setup,

loc2

With this background on the overall setup, now we can move forward to implement this in the network.

Requirements:

  • Unity3D installed with Pro licence – Specifically, Team license [link] (in all machines)
  • TKdiff installer [download]
  • Unity Asset Server installer [download] (not cache server)

Install and Start the Asset Server:

This is a straightforward step except for the admin password, which is really important. Make sure that you remember this for later. This process has to be done only on server computer. Once installed and started, the asset server program in the server machine starts listening to the port 10733.

Find out your server Address and Add a Firewall exception:

With the asset server running in the server computer, we need to know the address of the server in the network to communicate with it. It can be found out as shown in screenshots below. (Remember that if you constantly shuffle your computers in the network, the server’s address might change as well. To overcome this you can either set the ip of the server manually in the TCP/IPv4 properties or can access the settings in your router to make it assign constant ip to certain computers). Note down your IP address and keep it aside for later use.

Since the windows firewall usually blocks all connections to programs in the computer unless it is explicitly allowed, any incoming and outgoing connections through this port has to be set to be allowed in all types of networks (private to be specific) as shown below.

Now out server is ready and accepting connections. now we have to connect to this server via unity3D client.

Install and Setup tkDiff:

Before connecting to the asset server we need to some preparation at client side as well. First is to install a diff tool – tkDiff. Installing tkDiff is straightforward as well. Since this is an external software which unity uses to display and resolve the difference between two files clashing during the sync (commit), you have to tell unity to use this through the preferences dialog after installation. This process has to be done on all the client computers.

Set Google DNS: This is not that important/ necessary at local network but do set your dns servers to google DNS whenever you run into strange bugs (sometimes your public IP and local ip can be the same creating problems)

Using the Asset Server:

Its done! The server is set up and the clients are ready to connect to the server. To test the availability of the server from the client, you can use this app (https://code.google.com/p/paping/) with your server address and 10733 port. Now we have to set up projects and users in the server for the client to connect to the network – this is the administration and in unity asset server this has to carried out from the client end for decent GUI.

Administration: The administration is done through the asset server window (ctrl+0) in unity 3D, as shown below.

Now there is a project folder (actually a database) created in the server machine to which we’ll connect our local project folders in Unity3D running in client machines.

Client-side: Now from same asset server (ctrl+0) window, we can connect to the server from the overview tab. Here the server IP is the same but the username and password would be the one we created using administration.

Thats it… done. Now you have a proper asset server managing your project files which can accessed and edited collaboratively by the users (as allowed in the administration panel), with version control.

For details on using the version control and asset server (pulling updates, committing changes, resolving conflicts etc.) please check this documentation. If you need an introduction to version control in general consult here  (particularly the ‘Centralised VC’ section). Hope this is useful.