Acme Sprockets

My clients are typically concerned about batch process run times. As their business has grown, they want to process more and more data. Or, the users are now more savvy and want that data sooner. Failing to make SLAs is bad, bad, bad.

Generally, a batch stream is made up of a number of serially run jobs. And each job may have have a number of serial activities. But which ones to focus on?

Experience has shown that the largest opportunities are generally found by identifying the largest percentage contributors to the overall processing time. After all, making a 10% improvement in a 20-minute job is only 2 minutes, while a 10% gain in a three-hour job saves 18 minutes.

Plotting the process run times in a Pareto chart makes a easily-understood graphically representation that can help the technical and management teams focus attention on the right processes.

Consider the following example data of 14 batch jobs.

When plotted as a Pareto, it is easy to see that the largest three jobs together take over 60% of the total time.

I've found it easiest to use to use a script-based approach to generating the chart rather than a spreadsheet tool such as Excel or OpenOffice.org Calc. Why? It is very simple (it sorts and accumulates the data form me), and produces consistent output graphs every time. The data is first processed by a custom Ruby script (attached), then plotted using the Ploticus tool. This script and approach, by the way, isn't specific to process run times. One could use it for quality control or other analysis.

To use, you'll need Ruby, Ploticus, and the script in the attachment -- it (pareto.rb) will generate a Pareto chart from tab-delimited, two-column data (on stdin):
(1) label
(2) numeric qty

The script will sort the data by (2), then add the following columns:
(3) percentage component of value column (0-100)
(4) accumulated percentage (last one should be 100)

It will then create the Ploticus script on stdout with the data embedded (which will plot (1), (2), (4)).

To use the script (pareto.rb), do something like the following:

$ ./pareto.rb -t 'Batch Job Runtimes' -y 'minutes' < jobs.txt |pl -stdin -png -o pareto.png

Or if you're on Windows, you'll need to invoke ruby directly so the pipes work. For example:

d:\> ruby pareto.rb -t "Batch Job Runtimes" -y "minutes" < jobs.txt |pl -stdin -png -o pareto.png

The script takes a few command-line arguments. Simply use the traditional "-h" to see them:

$ ./pareto.rb -h
Usage: ./pareto.rb [options]
    -v, --[no-]verbose               Run verbosely
    -y, --ylabel=LABEL               Y-axis label
    -t, --title=TITLE                Graph title
        --qtycolor=COLOR             Color of quantity bars
        --pctcolor=COLOR             Color of cumulative percent line
        --plotheight=INCHES          Height of plot area

I hope you find this helpful.