PBS XML Accounting Toolkit

Rodney Mach < r m a c h at umich edu >

Intro

The PBS accounting toolkit takes standard OpenPBS or PBSPro batch scheduler accounting data for Linux and converts it to an XML representation for easier transformation/rendering to other formats.

Also includes java code to do these transformations as well as supporting XSD, and XSL files for validation and transformation to other formats such as NPACI joblog format, and the Global Grid Forums UR-WG UsageRecord accounting format (see here)

If you use this package, you can create slick accounting reports using the DarkSlide reporting engine to create fancy PDF's to impress your boss!

Latest News

NMI will be including the accounting toolkit in release 5!

Support for special Torque fields will be forthcoming in the next minor release!

Article appears on PBS Accounting toolkit in the October issue of sysadmin magazine!!

Latest version 1.4.3 of Accounting Toolkit supports full draft of the UR-WG UsageRecord accounting format from the GGF for both conversion AND the darkslide reporting engine! Also in addition to PDF reports, Excel reports can also now be generated by the darkslide reporting engine

Requirements

To use the package you will need J2SDK 1.4 or higher from Sun http://java.sun.com/ If Java is not available for your platform, you can generate the XML according to the DTD/Schema in the XML directory in your language of choice, and use the corresponding XSL files to perform any translations.

Example

An example output can be found Here

An example PDF using the corresponding darkslide report engine is here

Usage

Converting accounting data to XML

In /usr/spool/PBS/server_priv/accounting/ directory is the accounting files for OpenPBS. For PBSPro it is /var/spool/PBS/server_priv/accounting/. To convert an accounting file to it's XML representation type (at Unix prompt ) : 
pbstoxml < file.dtd|file.xsd> account_filename output.xml
For example, to convert the accounting file named 20021217 and save it to 20021217.xml you would type 
cd /usr/local/pbsaccounting/bin
pbstoxml /usr/spool/PBS/server_priv/accounting/20021217 20021217.xml
This does not assign a Schema to the document so that you can validate it.

To assign a W3C Schema, type 

pbstoxml ../xml/linux_joblog.xsd 20021217 20021217.xml
The XSD file is included in the distro.

Converting XML to NPACI joblog format

Once you have converted the accounting data to XML, you can convert the XML to npaci joblog format by typing 
pbstojoblog file.xml joblog.txt
For example, to convert the previous 20021217.xml file to an NPACI joblog formatted file you would type 
pbstojoblog 20021217.xml joblog.txt

Converting XML to Global Grid Forums UR-WG UsageRecord format

Once you have converted the accounting data to XML, you can convert it to the UsageRecord XML format as follows 
pbstogrid file.xml grid.xml hostname ppn
For example, to convert the previous 20021217.xml file to a UR-WG XML file you would type 
pbstogrid 20021217.xml grid.xml

Note that this supports the final draft version UsageRecord schema as of October 2003.

Limitations

Record Types

Currently all major OpenPBS records other than CheckPoint records are supported (if you need this send me an example record I will add it in) and I believe most PbsPro records. If you discover unknown record types (a warning is issued on conversion) please send them to me. Any types not supported are because I don't have examples of them

Platforms

I have tested on Linux only.It should work on other platforms, but since they will have different resources etc the code will need to change to support them. If you need support on your platform feel free to contribute any code. I will be adding support for the Origin and supporting schema in a future version.

Resource List

The list of resources for resource list is scheduler specific. I included ones that we use at our site with the default scheduler, if you have other resources you want to add it should be straightforward to support these.

License

The source code is released under the LGPL. Click here for license information

Requesting features, Reporting bugs

Submit any bugs or requests here here.

Developer Document Listing

  1. Full API documentation
  2. Linux Joblog Schema documentation
  3. Unit testing
  4. Unit testing coverage report for release v1.4.2
  5. Building

Installing via RPM

Go to the files section here here and download the RPM for pbsaccounting. If you want to use the darkslide package with it, you will ALSO need to download the darkslide RPM. The current version of pbsaccounting is pbsaccounting-1.4.2-1.i386.rpm, and current release of darkslide is darkslide-1.10-1.i386.rpm

Downloading the code

http://sourceforge.net/projects/pbsaccounting/ has the source code available from CVS where you can build your own distro, includes all the documentation and unit testing. Here is how to get it from CVS (copied from sourceforge docs) 

% export CVS_RSH=ssh
% cvs -d:pserver:anonymous@cvs.pbsaccounting.sourceforge.net:/cvsroot/pbsaccounting login 
% cvs -d:pserver:anonymous@cvs.pbsaccounting.sourceforge.net:/cvsroot/pbsaccounting checkout pbsaccounting
I have noticed that the anonymous checkout is old however, sourceforge doesn't always have the latest CVS checkin available for whatever reason via anonymous CVS

SourceForge