Flot Plugins – X Gap Threshold

Live Example
Github Code

You can insert breaks/gaps into a line graph drawn using Flot by inserting coordinate pairs that are set to null.

For example, from the Flot API:

If a null is specified as a point or if one of the coordinates is null or couldn’t be converted to a number, the point is ignored when drawing. As a special case, a null value for lines is interpreted as a line segment end, i.e. the points before and after the null value are not connected.

This works fine at breaking line graphs when the data has a detectable gap in it. That is, when x values have a greater distance between them that is predictable through the data measurement procedures.  For example, say we are polling networking data every 3 minutes, if there is a 6 minute gap between two x values then we should break the line graph at this point to indicate data loss.

While using Flot over summer, I found that breaking line graphs in this way is quite a common requirement. Particularly when working with time series data.

I developed plugin that automates adding null coordinates into the data series based on a given, x gap threshold value. The Flot plugin also allows the gaps to be highlighted in a given color to make them more obvious. Here is a screenshot of the ‘Hello World!’ example.

X Gap Threshold Example Graph

There is a small amount of configuration that is worth mentioning.

The plugin must be configured in the global x-axis options as well as for each individual series that a threshold needs to be set for. The global options enable the plugin and set the gap color. It may make sense to move all the options to the series in future versions so that gaps can be turned on and off and different colors can be used for each series. I leave this up to the reader to change if they require it 🙂

Global X-Axis Options:

xaxis: {
   insertGaps: true,  // enable or disable this plugin
   gapColor: rgba(100,100,100,0.2) // the color to use for gaps
}

Per-Series Options:

var series1 = {
  data: [ ... ],
  label: 'Series 1',
  xGapThresh: 300 // A value of 300 here indicates that a x-gap > 300 will insert a gap
}

Ask in the comments if you have any further questions and be sure to check the basic example.

NetMapJs – A Scalable Network Mapping Tool

I recently posted my final report for my university honours project titled – ‘Improved Network Map Display’.

For part of the project, I implemented a JavaScript network mapping tool that I call ‘NetMapJs’.

I have made the source code of NetMapJs freely available on github.

The main features of NetMapJs are:

  • Support for subnetwork hierarchy
  • Separate layout algorithm support for different subnetworks
  • Rich display of bandwidth data in graph edges
  • Automatic poller to update performance data
  • Semantic zooming which adds more detail to the map as the user zooms in
  • Responsive zooming and panning to navigate through the network map
  • Support for adding overlays such as VLANs
  • Multi-nested interactive overview map to help maintain control when zooming

The following screenshot shows the KAREN network map loaded into NetMapJs. You can view this example running here.

 

NetMapJs makes use of the JIT library for a solid information visualisation framework and Arbor.js for the Force Directed (spring) layout type.

The tool is close to being ready for use in actual networks or a NMSs. Feel free to fork the repository and try it with your network.

Flot Plugins – Event Graphics

Live Example
Github Code

Flot Events System

This flot plugin adds customizable event support. Events graphics contain information that is shown in a popup when the user hovers over them.

This is all done using CSS and div tags.

The following documentation from the script shows the data structures used in the plugin. See it running live for an example of how it works.

 * Events are small icons drawn onto the graph that represent something happening at that time.
 *
 * This plugin adds the following options to flot:
 *
 * options = {
 *      events: {
 *          levels: int   // number of hierarchy levels
 *          data: [],     // array of event objects
 *          types: []     // array of icons
 *          xaxis: int    // the x axis to attach events to
 *      }
 *  };
 *
 *
 * An event is a javascript object in the following form:
 *
 * {
 *      min: startTime,
 *      max: endTime,
 *      eventType: "type",
 *      title: "event title",
 *      description: "event description"
 * }
 *
 * Types is an array of javascript objects in the following form:
 *
 * types: [
 *     {
 *         eventType: "eventType",
 *         level: hierarchicalLevel,
 *         icon: {
               image: "eventImage1.png",
 *             width: 10,
 *             height: 10
 *         }
 *     }
 *  ]

There are a few important points to mention.

The plugin adds two public methods to the flot API.

plot.showEvents(lvlRange)

plot.hideEvents(lvlRange)

Events can be given a hierarchy level. When evens are shown and hidden, a hierarchy range may be passed to the function which only hides or shows events that are in the range.

Event data is added into the flot options file like so:

options = {

events: {
data: []
}
}

See the live example to see how it works in more detail.

Flot Plugins – Autoscale

Github Code
Live Example

This is a plugin that implements a different type of autoscaling.

Flot comes with an autoscaler that calculates the maximum y value based on all the the data points.

On graphs that allow multiple series to be shown and hidden, it is useful to autoscale based only on series that are currently visible on the graph.

This plugin simply, loops over all of the datapoints for the series that are visible, finds the maximum value and multiplies it by the autoscale margin.

// limit to visible serie
if (series.lines.show || series.points.show) {
    var max = Number.NEGATIVE_INFINITY;

    for (var i = 0; i < data.length; i++) {
        max = Math.max(max, data[i][1]);
    }

    max += max * options.yaxis.autoscaleMargin * 10;
    return Math.max(_max, max);
}

Flot Plugins – Easy Time Series Navigation

Over the last two summer breaks I have been working on projects that make use of the flot javascript plotting library.

Part of these projects required a bunch of extra functionality to be added to the standard flot release. I modulated some of the functionality in to flot plugins. By the end of my time playing around with flot I had a bunch of different plugins and have just made all of those public on github to stop them collecting dust!

Plugins developed (in order of interest):

Over a series of blog posts I am going to describe what each plugin is and how it works and hopefully it comes in use for people at some stage.

Easy Time Series Navigation

Github Code
Live Example

The first plugin I am going to explain is the easy time series navigation plugin. The code can be found on github here.

It is developed for use with time series navigation graphs. By that, I mean graphs that can be dragged forward and back in time by clicking and dragging the mouse. This dragging capability is added using the navigation plugin that is included with the standard flot package.

The plugin makes it to trivial to snap to a current range in time. A time range can be either: hour, day, week, month or year. Moving forward and back in time for the given time range is also possible. This is useful if say, we wanted to view the next or previous week from the current graph position.

Here is a screenshot of a flot graph showing latency over time between Maxnet and Auckland University. A single week is shown because I clicked the ‘now’ button. Clicking ‘Week>>’ or ‘<<week’ would move forward and back in time respectively.

Flot Time Snapping Functionality

The plugin just adds the functionality for flot to alter the xaxis position based on the given range and axis number. The actual UI controls are up to you to implement 🙂

There are four public functions added to the flot API to bind your buttons to. I am just going to copy and paste the documentation from the script itself here to explain those.

* This plugin adds four public functions to the flot API
* > nextRange, prevRange, now and snapTo
*
* Each of these function take two of the same parameters: range and xNum.
*
* xNum = the xaxis number (0, 1, 2, …)
* in most cases this will be 0 unless you have multiple xaxis’
*
* range = the time range for snapping.
* possible values are: hour, day, week, month or year
*
* eg.
* nextRange(“week”, 0) – will move the xaxis time range to
* one week in the future.
*
* now(“day”, 0) – will snap the time range to the start
* of the current day (midnight).
*/

The plugin does a bunch of funky time handling stuff that you won’t want to do yourself. The open source javascript date library, datejs, comes with some handy methods that make the code very clean.

Lets take snapping to a given year as an example. If it is not January then we need to move the time to the previous years January.  The Datejs ‘last’ method makes this trivial.

Then we need to move to the first day of January. There is conveniently a ‘moveToFirstDayOfMonth’ method. Finally, the end of the range needs to be one year after January 1st.

case "year":
  if (!startDate.is().january()) startDate.last().january();

  startDate.moveToFirstDayOfMonth().clearTime();
  end = 365 * MS_1DAY;
  break;

There are similar case blocks for the other range types. Then, the xaxis minimum value is set to the start date timestamp and the maximum is set to that value plus the end.

Note that it is possible to specify a time zone offset in the xaxis options.

// don't allow navigating beyond the current time
if (startDate.compareTo(new Date()) > 0) return { min: axis.min, max: axis.max  };

axis.min = startDate.getTime() + plot.getAxes().xaxis.options.tzOffset;
axis.max = axis.min + end;

plot.setupGrid();
plot.draw();

return { min: axis.min, max: axis.max };

Edit: I have added a basic example to show how the API could be included in your code. See the github examples folder.

MetService Hacked – How it Happened

By now, most of you should be aware of the recent attack on the MetService website that hit in a very busy period and infected many visitors computers. It was noticeable in the form of fake virus scanner alerts on infected computers.

Everything that I have read so far explains what the trojan virus does (see this blog post) and the type of vulnerability:

We now know that the ad server was compromised by a malicious attack, through a vulnerability which allowed someone to upload a binary file into the database. This file contained JavaScript code which redirects the browser to a website which downloads malware files to the client machine.

What these sites don’t tell us is what applications on MetService’s ad server resulted in them being compromised.

I decided to do a little poking around while sitting outside in the sun on a quiet Friday.

Firstly, I used the chrome debugger to find out details about their ad server.

Turns out that the address is simply: ads.metservice.com.  With all the magical (php) scripts living in ads.metservice.com/openx-2.6.3/www/delivery. Please turn off directory indexing MetService developers (we really don’t need to see this information).

From this we can see that they are running ‘openx-2.6.3’

So a quick search at on the web for exploits on this version of the ‘OpenX’ ad tool comes up with exploits-7883. This exploit works simply by using the following url to return passwords:

http://ads.metservice.com/openx/www/delivery/fc.php?MAX_type=%20../../../../../../../../../../../../../../../etc/passwd%00

This exploit is a mix of ‘Poison Null Byte‘ and ‘Local File Inclusion‘.

This is most likely how the hacker forced their way through MetService’s security although there are more exploits found in this list which are also possible. Or perhaps I am being too harsh and this is a zero day exploit.

If we assume that it is the security vulnerability mentioned above, then this could have been avoided by keeping OpenX up to date with patches.

MetService does not appear to be keeping their PHP scripts up to date if we go by the folder naming conventions. The OpenX website tells me that the latest version is ‘2.8.7’ while it looks like they are only sitting on ‘2.6.3’. It does apear that another development team has fallen victim to a PHP application vulnerability.

But lets not forget that ultimately it was the hacker initiated this attack on MetService users.

Time for me to uninstall wordpress yet?

Waikato University GPA calculator

Grade Point Average (GPA) Calculator:

I wanted to calculate my current grade point average (GPA) to see if I am on track to get a first class honours at Waikato University.

A quick google search did not show any results for a tool that exists publicly on the Waikato Uni’s website.

I decided to quickly code up one in javascript (mostly for procrastination reasons). I will make it available here for current and future students to benefit from. I did my best at getting the parameters right, but do not guarantee it is accurate or that I got these right! If you know it to be calculated any differently then let me know and I will update it.

The tool simply sums the products of each papers points by its grade value and then divides by the sum of all of the points (120 points in a standard full time course).

For Each Row
{ sum += row.points * grade value }

GPA = sum / TotalPoints

This is based off what I found on this Otago University’s page. It is adjusted to suit Waikato Uni’s grade system which gains an A+ for 85% opposed to Otago’s 90% (found here).

You can view the source code here.

And so no one tries to claim that their GPA has been calculated incorrectly:

Disclaimer: This tool will produce an estimate of a grade point average only. The parameters used to produce the GPA have not been verified and are not official. This tool is produced by me and is in no way affiliated with Waikato University. I take no responsibility for the results of using the output of this tool!

New Zealand Trade Visualisation

My latest University project was to create a visualisation of choice with another class member.

Stephen and I came across a complete set of data from New Zealand to all other trading countries. I must point out here that NZ does a comparatively good job to the rest of the world in terms of data availability. Some countries that will remain unnamed (*cough* Australia *cough*) provide their data in PDF form 🙁

StatsNZ Exports Datatable

This huge table was just screaming out to be visualised. We took inspiration from Minard’s effective French wine export map as well as from a global trade visual. Both of these visualisations use the line width to indicate the quantity of exports between countries. Lines start off thick and then thin out as it branches off into countries. Our goal became to take this static idea and present it in a dynamic way using HTML5 technologies.

Minard - French Wine Exports Map

By combining StatsNZ data and country locations collected using Google’s geolocation API, we were able to produce a JSON datastructure that could easily be adapted to support the visualisation of other countries. The following diagram breaks down the process:

532exports Application Flow

And the final product!

It can also be viewed live in my testing folder here: Export Visualisation

Special thanks to my group member Stephen and to StatsNZ, and the Google Maps API for their awesome data.

Update: The code is now available on github.

Force Directed Emails Visualisation Using arbor.js

I have recently been playing around with ways of visualising my sent emails for an assignment at university. I have over 6 years worth of sent mail sitting in Gmail, collecting electronic dust (and occasionally being dusted off and looked at).

I downloaded all of my sent mail from Gmail (over 3000 conversations), making  use of my university’s internet connection. I made use of thunderbird for this, as it stores emails in flat files on disk. I made a simple python script that takes this raw file and converts it into JSON.

import mailbox
import json

# path to you sent mail mbox
mb = mailbox.mbox('Sent Mail')

fout = file('sent.json', 'w')
items = []
fields = ['date', 'subject', 'to']

for i in range(len(mb)):
    obj = {}
    for item in mb.get_message(i).items():
        if item[0].lower() in fields:
            obj[item[0]] = item[1]
        mb.remove(i)
    items.append(obj)

json.dump(items, fout)
fout.close()

A nice javascript vis library called arbor makes it easy to take a graph and apply a force directed algorithm to it. It allows you to only worry about the visualisation side by separating the layout computation from the graphical display. Using this library and some basic javascript processing I was able to produce the following display.

Sent Emails Visualisation

I have obscured the recipient names for the sake of their privacy only (my own privacy has already been destroyed by the likes of such sites as Facebook – see: Facebook is an appalling spying machine).

The line widths between time periods and recipients relates to the number of emails sent to that person. Nodes can be dragged about and added in by dragging the sliders to alter the time range.

The most interesting part of this process was probably making a wordle based on all of the recipient names for all of the sent emails. The wordles changing from year to year showed the people that I was in the most communication with and perhaps most important in a given time in my life.

To leave you with an idea – someone needs to make a web app to anonymize wordles (ie replace words with random, but unique common words of the same length). A google search did not find anything meaningful.

** EDIT **
Lots of people have been asking for the code. I will provide the link to my github project but it will not include the data file. https://github.com/oughton/email-wordle/

SVG vs Canvas Performance

My honours project this year requires the use of graphical browser technologies to produce a modern network weather map.

I have used SVG and HTML5 canvas in the past and have had mixed performance experiences with them. Over summer I had to make use of SVG to draw a traceroute tree-map which in practice performed a lot quicker running on Google Chrome opposed to Firefox.

I wanted to get a clear picture of the current position that both of these technologies are at in terms of performance for drawing and translating large numbers of nodes.

A quick search around the internet showed a handful of existing tests, but I thought I would give it a go to for the practice.

My testing platforms were:

PC
Windows 7
Intel Core 2 Quad 2.6GHZ
4GB Ram

Mac Book Pro
Mac OS Snow Leopard 10.6.7
Intel i5 2.3 GHz dual core
8GB Ram

I tested the following browsers:

  • Chrome 10.0.648.204
  • Firefox 3.3.16 (I didn’t test FF4.0 but I hear performance is not a lot better)
  • Safari 5.0.4

Test Setup / Method

I wrote a simple script that creates/draws a given number of nodes using either ‘svg’ or ‘canvas’. The nodes are then translated as many times as possible per second.

The framerate is calculated by incrementing a counter ever time all of the nodes have been redraw(canvas)/translated(svg) and then outputting its value after each second via a setInterval function.

function draw() {
    if (diffTime >= 1000) {
        fps = frameCount;
        frameCount = 0.0;
        lastTime = nowTime;

        // draw the 'fps' to the window
    }

    // draw some magical particles

    frameCount++;
}

See the full code here: https://github.com/oughton/Drawing-Browser-Benchmark

There are disadvantages/naiveness to this method. I am only testing translation performance of SVG and redrawing performance of canvas. Event support in SVG is a given and is quick due to DOM nodes existing for each node that is drawn. Canvas requires an event system to be built around the application to get the same event support.

But, nonetheless it gives a good overview of where browsers are at.

Results

I took a serious of measurements at various different numbers of particles and recorded the framerate at that value. This produced a big table of numbers but I will just summaries those in 2 pretty graphs.

These results show Chrome as a clear winner for bothCanvas and SVG performance on my windows 7 pc. The SVG performance on both pc and mac degrades significantly as the number of nodes increase across all of the browsers I tested. Chrome and Safari maintained a somewhat usable performance for SVG around the 1000 node mark where as Firefox began locking up.

Another interesting point is Safaris apparent 90 frames / second limit for canvas. While it did not go over 90 frames /second on my mac, it remained mostly constant over the 2000 particles tested.

From these results it appears that reasonable SVG performance is currently possible, but if you are needing thousands of nodes in your visualization, I would stick with Canvas and implement a basic event system.

** UPDATE **

It was requested that I show results for IE 9 due to the addition of hardware rendering support. I actually did these tests before handing in my final project so it is easy to include the new graph here.

The graph shows that IE 9 canvas rendering speed outperforms all other browsers. IE 9 SVG performance still shows the same rapid decline as the number of particles increase.