I’m working on a new project that will be open sourced if I can ever get it to be generically useful. It’s called “sarviz”, and it’s a visualization tool for output from the “sar” UNIX system reporting utility. I know tools like this exist, but please read on, as I’m looking to do something a bit different from what I’ve seen.

A quick, simple explanation of sar

System administrators typically run sar as a cron job, and each day sar will generate a report that lists the values of various system counters for a specified time interval throughout the day. So you end up with a text file that lists, for example, the cpu iowait value every 10 minutes throughout the day. There are maybe a dozen different categories of counters enabled by default, and more that aren’t (like disk-related counters). Anyway, you wind up with a text file that looks something like this:

23:30:01          CPU     %user     %nice   %system   %iowait    %steal     %idle
23:40:02          all      0.32      0.00      0.32      6.57      0.49     92.29
23:40:02            0      0.32      0.00      0.32      6.57      0.49     92.29
23:50:01          all      0.74      0.00      0.82      7.14      0.55     90.76
23:50:01            0      0.74      0.00      0.82      7.14      0.55     90.76
Average:          all      0.82      0.00      0.72     13.54      0.78     84.14
Average:            0      0.82      0.00      0.72     13.54      0.78     84.14

This is just a small part of one section of the file (this box has only one cpu, which is why the ‘all’ and ‘0′ numbers are the same, btw). The whole file on one server, running with default configurations, is 4000 lines long.

There’s a ton of great information in here, but… it all looks like the above. There’s no graphical output to be had. This is bad, because it would be nice to use this (historical) monitoring output for things like capacity planning, problem tracking, etc. You would, of course, want to couple this type of monitoring with something else that’ll do real-time monitoring, alerts, dependencies, escalation, etc.

So I want to write an application that’ll generate graphs of all of this stuff. Furthermore, I thought it would be cool to do something like what planetplanetdoes, which is to say that I want sarviz to run as a cron job, parse all of this stuff, and generate static html files, with an index.html that’ll make it really easy to browse this information either by host, by date, by resource… whatever. Later on I can add features to actually do even more useful stuff like longer-term trending of resource usage (by aggregating across various ’sar’ output files), and more.

Sar is not alone

Sar comes with some friends, and it turns out they can be extremely useful. The best one for my purposes here is called ’sadf’, and it is used to basically format the sar output to make it more useful for programmatic processing. It can output the information in CSV format, or make it ready for insertion into a relational database, but what I’m currently using for sarviz (and it’s early, so this could change) is the XML output capability. With XML output, I won’t have to deal with parsing out column headers, scanning an entire file for information from a single sar run, dealing with the blank lines sar uses by default to make it easier to read on a console, etc. So with sadf I can get output that looks like this:

<timestamp date=”2008-06-15″ time=”07:10:01″ interval=”600″>
<processes per=”second” proc=”0.93″/>
<context-switch per=”second” cswch=”221.50″/>
<cpu-load>
<cpu number=”all” user=”1.77″ nice=”0.00″ system=”0.56″ iowait=”0.04″ steal=”0.08″ idle=”97.55″/>
<cpu number=”0″ user=”1.77″ nice=”0.00″ system=”0.56″ iowait=”0.04″ steal=”0.08″ idle=”97.55″/>
</cpu-load> ….

This is a bit nicer to deal with, and I was excited to use Python’s (now built-in) ElementTree module to do something from scratch after having dealt with it being somewhat abstracted in the Python tools for the GData API (which I used to write a command line client for Google Spreadsheets, for example).

Doing Simple Things with ElementTree

Well, as it turns out, I had kind of a hard time getting started doing what I thought were simple things with ElementTree, so I want to post a few examples of how I did them so that I and others have something to refer to online.

The first thing to know about ElementTree is that there are Element objects, and ElementTree objects. ElementTree objects are made up of a hierarchical collection of Element objects, and Element objects are the things you can actually get attributes from that you’re likely to want. For whatever reason, I was a little confused starting out, because I wanted to get an ElementTree object and then ask that object to “scan the tree and give me all of the “time” attributes of the “timestamp” elements in the tree. You might be able to do this with a one-liner, but I never found a document that said how.

So here’s how to load in an XML file, parse it, and return all of the timestamp elements in that tree (or, rather, this is how I did it, which seems reasonable):

strudel:sa jonesy$ python
Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from xml.etree import ElementTree as ET
>>> tree = ET.parse("sa15.xml")
>>> for ts in tree.findall("host/statistics/timestamp"):
...        isotime = ts.attrib["date"]+"T"+ts.attrib["time"]
...        print isotime

2008-06-16T05:00:01
2008-06-16T05:10:01
2008-06-16T05:20:01
2008-06-16T05:30:01
2008-06-16T05:40:01
2008-06-16T05:50:01
2008-06-16T06:00:01
2008-06-16T06:10:01
2008-06-16T06:20:01
2008-06-16T06:30:01
2008-06-16T06:40:01
2008-06-16T06:50:01
….

So, I imported the ElementTree module, fed my xml file to a method called “parse()”, and that gives me an ElementTree object. In that tree, I then ask for the timestamp elements which are under the root element at “host/statistics/timestamp”. You can then see that I create an ISO8601-formatted timestamp by asking for the “date” and “time” attributes of the timestamp element, and put a “T” between them. I would’ve used something like “T”.join, but there are other attributes in that element, and I only needed two, so I took the easy way out here instead of creating a list first and then doing the join on the list.

Of course, my real interest in the timestamps isn’t to print them, but to get the statistics for each sar run (represented by a timestamp, since sar records statistics for regular time intervals). So now let’s grab the 1-, 5-, and 15-minute load averages according to sar. I want all of this printed on one line along with the timestamp, because this output is going to be graphed using Timeplot, and that’s how Timeplot wants the data. Here goes:


>>>for ts in tree.findall("host/statistics/timestamp"):
...        isotime = ts.attrib["date"] + "T" + ts.attrib["time"]
...        for q in ts.findall("queue"):
...             qstat = [isotime, q.attrib["ldavg-1"], q.attrib["ldavg-5"], q.attrib["ldavg-15"]]
...             print ",".join(qstat)

2008-06-16T05:10:01,0.05,0.12,0.09
2008-06-16T05:20:01,0.03,0.06,0.07
2008-06-16T05:30:01,0.02,0.02,0.03
2008-06-16T05:40:01,0.02,0.06,0.03
2008-06-16T05:50:01,0.03,0.06,0.03
2008-06-16T06:00:01,0.04,0.03,0.00
2008-06-16T06:10:01,0.02,0.06,0.03
2008-06-16T06:20:01,0.06,0.10,0.04
2008-06-16T06:30:01,0.13,0.11,0.06
2008-06-16T06:40:01,0.16,0.12,0.08
2008-06-16T06:50:01,0.04,0.06,0.06

The thing to note here, in case it escaped your eyeball, is that the second call to ‘findall’ feeds an argument relative to the ‘ts’ object rather than the ‘tree’ object.

This data is ready for Timeplot, and now it’s just a matter of somehow generating the files with the appropriate HTML and JavaScript in them to present the information. I have absolutely no clue how to easily use dynamic variables from Python to easily generate static HTML and JavaScript, so what I have in that area of my code is not something I want to share, out of sheer embarrasment. If someone has done that, let me know. PlanetPlanet does not output JavaScript, best I can tell, but it does output HTML, so I’ll be checking that part of the code out (probably uses BeautifulSoup I guess?). Input on that is hereby solicited!

Highslide JS is an open source JavaScript software, offering a Web 2.0 approach to popup windows. It streamlines the use of thumbnail images and HTML popups on web pages. The library offers these features and advantages:

• No plugins like Flash or Java required.
• Popup blockers are no problem. The content expands within the active browser window.
• Single click. After opening the image or HTML popup, the user can scroll further down or leave the page without closing it.
• Compatibility and safe fallback. If the user has disabled JavaScript or is using an old browser, the browser redirects directly to the image itself or to a fallback HTML page.

Project Description
Just by adding this small JavaScript module to the end of your web pages, you get your form's submissions intercepted and, instead of sending request to the server in the usual way, they are done asynchronously and in an AJAX-style smooth way.

As long as it is a client-side library it will work with any server technology: ASP.NET, PHP, JSP, Classic ASP... and even with local HTM files.

I've included extensibility so that you can add easily your own progress indicators (several included) and show error messages the way you prefer (by default it shows an alert).

It's transparent to your server code and could be used to add AJAX capabilities to some applications without writting a single line of code.

Documentation is included.



Features summary

• No-code AJAXification of web apps
• Supports any server technology, including ASP.NET, JSP, PHP, ASP 3.0...
• In ASP.NET it supports all kinds of postbacks: direct and by code.
• Works in any modern browser that supports AJAX.
• Supports cross-posting of forms, that is, you can send the information to any web page in the same domain. If all your web pages have AJAXInterceptor included (for example, you include it in your master page or template) you can hace
• Respects your custom onsumit event handlers.
• Supports browser history so that your users can hit the previous button and get the last rendered page.
• Two versions of the module:
- AJAXInterceptor.js: full commented one. Useful for debugging purposes.
- AJAXINterceptor_r.js: reduced-size version. It downloads faster as it only is 2.6 kB in size. It's better to use this on production apps.
• Automatically show/hide custom progress indicators.
• Support for cancelling operations.
• Support for custom message displaying.
• Supports any form in your page

Obviously this is not substitute at all of full-fledged APIs like Microsoft's ASP.NET AJAX, PHPLiveX or AJAX.NET, but will let you add AJAX support to your apps in a few seconds and without writing any code. Just give it a try!

In the ZIP you will find teh module, a working sample with ASP.NET and a PDF with the help documentation.

This project is just for fun :-)

I will be very glad if you drop me a line in case you use AJAXInterceptor in any real-world application or if you enhance it with new features.

Visit my .NET blog (Spanish) at http://www.jasoft.organd my e-mail marketing blog (English) at http://www.theemailingexperience.com