I have recently been playing around with ways of visualising my sent emails for an assignment at university. I have over 6 years worth of sent mail sitting in Gmail, collecting electronic dust (and occasionally being dusted off and looked at).
I downloaded all of my sent mail from Gmail (over 3000 conversations), making use of my university’s internet connection. I made use of thunderbird for this, as it stores emails in flat files on disk. I made a simple python script that takes this raw file and converts it into JSON.
import mailbox
import json
# path to you sent mail mbox
mb = mailbox.mbox('Sent Mail')
fout = file('sent.json', 'w')
items = []
fields = ['date', 'subject', 'to']
for i in range(len(mb)):
obj = {}
for item in mb.get_message(i).items():
if item[0].lower() in fields:
obj[item[0]] = item[1]
mb.remove(i)
items.append(obj)
json.dump(items, fout)
fout.close()
A nice javascript vis library called arbor makes it easy to take a graph and apply a force directed algorithm to it. It allows you to only worry about the visualisation side by separating the layout computation from the graphical display. Using this library and some basic javascript processing I was able to produce the following display.
I have obscured the recipient names for the sake of their privacy only (my own privacy has already been destroyed by the likes of such sites as Facebook – see: Facebook is an appalling spying machine).
The line widths between time periods and recipients relates to the number of emails sent to that person. Nodes can be dragged about and added in by dragging the sliders to alter the time range.
The most interesting part of this process was probably making a wordle based on all of the recipient names for all of the sent emails. The wordles changing from year to year showed the people that I was in the most communication with and perhaps most important in a given time in my life.
To leave you with an idea – someone needs to make a web app to anonymize wordles (ie replace words with random, but unique common words of the same length). A google search did not find anything meaningful.
** EDIT **
Lots of people have been asking for the code. I will provide the link to my github project but it will not include the data file. https://github.com/oughton/email-wordle/
Hi oughton,
A very good post about email visualization. I liked how you used a slider
below your graphical visulalization to show your visulaization dynamically. I was wondering how did you do it with arbor.js. I mean could you please let me know what tools or libraries did you exactly use for that. I would want to make that kind of stuff too.
Cheers
Thanks for the comment.
arbor.js takes a json object as input to draw the graph.
All you have to do is modify the object and then tell arbor to redraw graph. I am not using any special tools. Just arbor.js and javascript.
I can email you the code if you want to check it out.
Hmm..thank you for letting me know. And the process would be similar if we want to show instead of a graph, some details instead of a graph that changes dynamically while we move the slider..right? Also it would be great if you can email me the code. As I am new to these things, an example code would be of great help to me. Thanks in advance.
Cheers
I would really like to see how you used images instead of circles and basic shapes. I would really appreciate it if you were to send me the code. Thanks.
Have a look at one of the arbor.js examples and you will see a redraw function.
This redraw function is called every time the force directed layout changed to update the display.
So to draw an image, in your redraw function you would put for example:
var image = new Image();
image.src = "path/yourimage.jpg";
particleSystem.eachNode(function(node, pt) {
// call the canvas image drawing function
ctx.drawImage(image, pt.x-w/2, pt.y-w/2); // w = width of the image
}
Could you please send me the code for the email visualization code above. It would prove to be a really good reference for my work.
Hi,
Is it possible that you share the code for the redrawing part when you move the slider? I would be grateful for this.
Thank you.
I have now put this up on github: https://github.com/oughton/email-wordle/
Great work. I really enjoy the concept. Just found arbor today, and I am enjoying it very much. Though I am still early in my JavaScript development.
Nice work Joel,
working on a 50000px x 50000px site using xml and arbor to connect all my friends and there friends from face book. Really need parallel computing to take full advantage of this but will give it a go.
Check out our small NetMap http://marketstone.com/#netmap or it’s located at the lower right hand corner of the site.
Again nice work!