Flot Plugins – Easy Time Series Navigation

Over the last two summer breaks I have been working on projects that make use of the flot javascript plotting library.

Part of these projects required a bunch of extra functionality to be added to the standard flot release. I modulated some of the functionality in to flot plugins. By the end of my time playing around with flot I had a bunch of different plugins and have just made all of those public on github to stop them collecting dust!

Plugins developed (in order of interest):

Over a series of blog posts I am going to describe what each plugin is and how it works and hopefully it comes in use for people at some stage.

Easy Time Series Navigation

Github Code
Live Example

The first plugin I am going to explain is the easy time series navigation plugin. The code can be found on github here.

It is developed for use with time series navigation graphs. By that, I mean graphs that can be dragged forward and back in time by clicking and dragging the mouse. This dragging capability is added using the navigation plugin that is included with the standard flot package.

The plugin makes it to trivial to snap to a current range in time. A time range can be either: hour, day, week, month or year. Moving forward and back in time for the given time range is also possible. This is useful if say, we wanted to view the next or previous week from the current graph position.

Here is a screenshot of a flot graph showing latency over time between Maxnet and Auckland University. A single week is shown because I clicked the ‘now’ button. Clicking ‘Week>>’ or ‘<<week’ would move forward and back in time respectively.

Flot Time Snapping Functionality

The plugin just adds the functionality for flot to alter the xaxis position based on the given range and axis number. The actual UI controls are up to you to implement 🙂

There are four public functions added to the flot API to bind your buttons to. I am just going to copy and paste the documentation from the script itself here to explain those.

* This plugin adds four public functions to the flot API
* > nextRange, prevRange, now and snapTo
*
* Each of these function take two of the same parameters: range and xNum.
*
* xNum = the xaxis number (0, 1, 2, …)
* in most cases this will be 0 unless you have multiple xaxis’
*
* range = the time range for snapping.
* possible values are: hour, day, week, month or year
*
* eg.
* nextRange(“week”, 0) – will move the xaxis time range to
* one week in the future.
*
* now(“day”, 0) – will snap the time range to the start
* of the current day (midnight).
*/

The plugin does a bunch of funky time handling stuff that you won’t want to do yourself. The open source javascript date library, datejs, comes with some handy methods that make the code very clean.

Lets take snapping to a given year as an example. If it is not January then we need to move the time to the previous years January.  The Datejs ‘last’ method makes this trivial.

Then we need to move to the first day of January. There is conveniently a ‘moveToFirstDayOfMonth’ method. Finally, the end of the range needs to be one year after January 1st.

case "year":
  if (!startDate.is().january()) startDate.last().january();

  startDate.moveToFirstDayOfMonth().clearTime();
  end = 365 * MS_1DAY;
  break;

There are similar case blocks for the other range types. Then, the xaxis minimum value is set to the start date timestamp and the maximum is set to that value plus the end.

Note that it is possible to specify a time zone offset in the xaxis options.

// don't allow navigating beyond the current time
if (startDate.compareTo(new Date()) > 0) return { min: axis.min, max: axis.max  };

axis.min = startDate.getTime() + plot.getAxes().xaxis.options.tzOffset;
axis.max = axis.min + end;

plot.setupGrid();
plot.draw();

return { min: axis.min, max: axis.max };

Edit: I have added a basic example to show how the API could be included in your code. See the github examples folder.

MetService Hacked – How it Happened

By now, most of you should be aware of the recent attack on the MetService website that hit in a very busy period and infected many visitors computers. It was noticeable in the form of fake virus scanner alerts on infected computers.

Everything that I have read so far explains what the trojan virus does (see this blog post) and the type of vulnerability:

We now know that the ad server was compromised by a malicious attack, through a vulnerability which allowed someone to upload a binary file into the database. This file contained JavaScript code which redirects the browser to a website which downloads malware files to the client machine.

What these sites don’t tell us is what applications on MetService’s ad server resulted in them being compromised.

I decided to do a little poking around while sitting outside in the sun on a quiet Friday.

Firstly, I used the chrome debugger to find out details about their ad server.

Turns out that the address is simply: ads.metservice.com.  With all the magical (php) scripts living in ads.metservice.com/openx-2.6.3/www/delivery. Please turn off directory indexing MetService developers (we really don’t need to see this information).

From this we can see that they are running ‘openx-2.6.3’

So a quick search at on the web for exploits on this version of the ‘OpenX’ ad tool comes up with exploits-7883. This exploit works simply by using the following url to return passwords:

http://ads.metservice.com/openx/www/delivery/fc.php?MAX_type=%20../../../../../../../../../../../../../../../etc/passwd%00

This exploit is a mix of ‘Poison Null Byte‘ and ‘Local File Inclusion‘.

This is most likely how the hacker forced their way through MetService’s security although there are more exploits found in this list which are also possible. Or perhaps I am being too harsh and this is a zero day exploit.

If we assume that it is the security vulnerability mentioned above, then this could have been avoided by keeping OpenX up to date with patches.

MetService does not appear to be keeping their PHP scripts up to date if we go by the folder naming conventions. The OpenX website tells me that the latest version is ‘2.8.7’ while it looks like they are only sitting on ‘2.6.3’. It does apear that another development team has fallen victim to a PHP application vulnerability.

But lets not forget that ultimately it was the hacker initiated this attack on MetService users.

Time for me to uninstall wordpress yet?

The Best Email Management Tip

This subject has been discussed quite a lot in the internet community such as here, here and here.

I had followed a lot of the tips that I found in the articles around the web. These articles often include many small tricks such as labeling, message rules and immediate replying etc.

But undoubtedly, the single and most effective method I have come across is this:

Unsubscribe, unsubscribe, UNSUBSCRIBE!

I had to actually try this for myself to see how effective it was. It does take time. In my case it was a few months until the last of my unnecessary inbox clutter stopped arriving. Just take it day by day, thinking about every automated email you receive, and unsubscribe if it is not something that requires your guaranteed attention.

The problems with email subscriptions

  1. They trickle in at an often constant rate throughout the day, week or month.
  2. A constant email flow creates distractions throughout the day
  3. The email flow requires sub-daily checking to keep on top of your inbox
  4. They add up over time, often without you realising
  5. Emails are not the most efficient method of digesting subscription style information

This is a rule of thumb. There are certain mission critical services that require your guaranteed attention. For all of those messages that are just interesting and/or not critical, I would advise sticking to feeds and other information collaboration tools.

RIP Flash

This topic has been blogged about lots and lots but I thought I would mix a little university background in there.

It is now (and has been for sometime) possible to completely ignore flash and instead use native web browser technologies. Sure there are a couple exceptions with the more advanced features for flash but I am sure by the end of 2012 flash will be gone – completely.

Is this sad? Kind of.

I learned flash relatively recently at university (2008). Yes that is right. I took a paper that taught adobe flash and action script 3 less than four years ago. The same paper has since shifted its focus more onto web design.
The sad part is that these skills are now useless. That is the way practical computer science works though right? You learn something and then keep on adapting and updating as new technologies emerge.

Am I happy? Very!

Flash was cool. It worked, and you could make cool visualisations, animations and applications. But why! Why do we need this extra tool grafted onto something that is purely capable of handling this itself? Well we no longer do. HTML5 is here and in its pure and native form it can handle rich and interactive web applications.

We still see some people (from adobe) sticking up for their software but the fact of the matter is that with the existence of tools now to convert flash files to pure HTML5. Tools like swifty by google and intelligent and bold moves by apple to cut support for flash on their mobile devices WILL spell the end of flash.

RIP Flash. You had your time.

(Yes I used Photoshop to make this image :-P)

Vodafone and Telecom Twitter Relationship

Vodafone and Telecom had some entertaining tweets between them today.

They were talking about the following picture taken of the Vodafone office in Auckland.

looks chilly over there. any body home @vodafonenz or did you... on Twitpic


View Telecom and Vodafone offices in a larger map

The tweets:

Telecom Tweet to Vodafone 1

Vodafone Tweet to Telecom 1

Telecom to Vodafone Tweet 2

Edit: More tweets

Vodafone to Telecom Tweet 3

Telecom to Vodafone Tweet 3

Telecom to Vodafone Tweet 4

Vodafone to Telecom Tweet 4

Vodafone to Telecom tweet 5

It is good to see a friendly relationship between the two telecommunication rivals.

Network Bandwidth Visualisation for Graphs

Part of my honours project consists of visualising network bandwidth utilisation on a node/edge graph.

Basic information visualisation techniques can be used to make simple, effective and communicative network edge graphics. Current network packages that produce such graphs, from what I have seen, do not attempt to communicate very much information through graph edges.

Lets go through a quick process of making an effective edge.

Say that we have two nodes A and B.Bandwidth 0

A simple way of showing that both A and B are connected to each other in each direction would be a double sided arrow between the two.

Bandwidth 1

Lets change the arrows so that they face towards each other and converge in the middle of the two connected nodes.

Bandwidth 2

So far this just indicates a bidirectional link between the two network devices A and B. Let us use the basic idea that wired networks are essentially just a series of tubes.

Connect both A and B with an empty pipe and a divider in the middle. Then draw arrow heads along the respective pipe sections to show the percentage of the total bandwidth that is currently in use.

This is using the length of the line to indicate the percentage utilisation which is a quantitative variable. The use of length is effective for human recognition of quantitative values. The length clearly shows that about half of the A->B link’s bandwidth is currently in use.

Bandwidth 4

We are not making use of the vertical dimension. Make the height adjust according to the total bandwidth available on a link. For example 100Mbit, 1Gbit, 10Gbit etc.

Again, line thickness is effective for displaying quantitative values. It is obvious that the link A->B has about twice the bandwidth as B->A.

Bandwidth 5

You may have noticed that up to this point I have been avoiding the use of colour for bandwidth quantities. Colour is actually one of the most least useful means for visualising quantitative variables. However it is effective for categorical variables.

Lets say for example that green is good, yellow is intermediate and red is a bad network state. These categories are shown well by adding the respective colours to arrows within the pipe sections. Alternatively the pipe section colours could indicate these network states.

Bandwidth 6

Putting all of this together, we get as our final product:

Bandwidth 8

Google 2 Step Password Verification – Get It!

I just discovered an awesome feature by google for their account login process.

Google Apps Ring

What is 2 Step Verification?

Google’s 2 Step Verification secures your account by requiring you to enter in an additional, randomly generated code which is sent via a txt message to your mobile phone.

Why use it?

This makes it difficult for someone to hack you google account (gmail, document, photos, g+, calendar etc) unless a hacker is able to also steal your physical mobile device.

This in my opinion is extremely important if you are anything like me and keep a lot of important and confidential information locked away in your google accounts. If I were to lose control of this information, I would risk loosing my: web hosting accounts, domain names, credit card details, paypal, trademe, facebook, confidential client information etc.

The 2 Step also service lets you generate application specific passwords that are designed for one time use. So for example, if you use gmail notifier (the small program for windows that lets you know when you have new emails), you can generate a one time use password that grants it access without the 2 Step code.

What if my phone gets lost or stolen?

This is important to note. When you are signing up for the service, it is recommended that you fill in an additional backup phone for use if your’s is not available.  I used a family members phone for this. Also, you are provided with a list of  10 backup codes. These are codes that can be used once only and can be used instead of the txt verification code. Print out these numbers and put them in your wallet or somewhere else safe. Remember that even if someone obtains these number, they mean nothing without also knowing you original google password.

What does a 2 Step TXT look like?

This is all it is:

“Your Google verification code is 123456”

Does it cost?

From what I can tell, the service doesn’t cost you anything in New Zealand. I have not got any charges through to my phone for incoming txt’s received from the google service. I was not even charged for the voice call back service. With the call service you actually get an automated call that reads out the code!

Is it a hassle?

Yes it is. However, you stay logged in to google accounts for 30 days so you will not have to keep generating codes each time you use your computer. I think the added hassle is well worth it considering the security benefits gained.

Waikato University GPA calculator

Grade Point Average (GPA) Calculator:

I wanted to calculate my current grade point average (GPA) to see if I am on track to get a first class honours at Waikato University.

A quick google search did not show any results for a tool that exists publicly on the Waikato Uni’s website.

I decided to quickly code up one in javascript (mostly for procrastination reasons). I will make it available here for current and future students to benefit from. I did my best at getting the parameters right, but do not guarantee it is accurate or that I got these right! If you know it to be calculated any differently then let me know and I will update it.

The tool simply sums the products of each papers points by its grade value and then divides by the sum of all of the points (120 points in a standard full time course).

For Each Row
{ sum += row.points * grade value }

GPA = sum / TotalPoints

This is based off what I found on this Otago University’s page. It is adjusted to suit Waikato Uni’s grade system which gains an A+ for 85% opposed to Otago’s 90% (found here).

You can view the source code here.

And so no one tries to claim that their GPA has been calculated incorrectly:

Disclaimer: This tool will produce an estimate of a grade point average only. The parameters used to produce the GPA have not been verified and are not official. This tool is produced by me and is in no way affiliated with Waikato University. I take no responsibility for the results of using the output of this tool!

Epic Spam Fail

I don’t actually see spam emails very often these days. Gmail does an excellent job of keeping them hidden away and seems to have a very low false positive rate.

Spam Fail

I received an email the other week that failed at trying to convince me that this was a noteworthy email. This SPAMer really needs to run some tests on their software before mass sending.

On second thoughts. No they don’t. Spamming is virtually free…

 

New Zealand Trade Visualisation

My latest University project was to create a visualisation of choice with another class member.

Stephen and I came across a complete set of data from New Zealand to all other trading countries. I must point out here that NZ does a comparatively good job to the rest of the world in terms of data availability. Some countries that will remain unnamed (*cough* Australia *cough*) provide their data in PDF form 🙁

StatsNZ Exports Datatable

This huge table was just screaming out to be visualised. We took inspiration from Minard’s effective French wine export map as well as from a global trade visual. Both of these visualisations use the line width to indicate the quantity of exports between countries. Lines start off thick and then thin out as it branches off into countries. Our goal became to take this static idea and present it in a dynamic way using HTML5 technologies.

Minard - French Wine Exports Map

By combining StatsNZ data and country locations collected using Google’s geolocation API, we were able to produce a JSON datastructure that could easily be adapted to support the visualisation of other countries. The following diagram breaks down the process:

532exports Application Flow

And the final product!

It can also be viewed live in my testing folder here: Export Visualisation

Special thanks to my group member Stephen and to StatsNZ, and the Google Maps API for their awesome data.

Update: The code is now available on github.