Darkslide. The PBS Report Generator


Darkslide is a report generator lets you create fancy PDF's and Excel files from PBS XML accounting data or URWG XML Accounting data produced by the PBSaccounting toolkit. With darkslide, you can create high quality printable reports with easy to read tables and charts from your existing accounting data. You can also create Excel reports that contain all the data filled in so you can produce your own charts or do your own manipulation with Excel or OpenOffice

Example PDF Output

Here is an example PDF report Here


The XML accounting data as produced by the pbsaccounting package is stored inside a native XML database (Xindice). A Xpath query engine queries the XML to construct data statistics which are passed to the reporting engine for producing PDF, or other output formats.


To use the package you will need J2SDK 1.4 or higher from Sun http://java.sun.com

You will also need to install the PBS accounting package version 1.3 or higher available at here

You will need Xindice XML database from Apache


There are a couple basic steps which are detailed below
  1. Have all your pbs accounting data converted to XML. See the pbsaccounting package for the RPM and instructions
  2. You set up Xindice XML database, and load that XML data into it
  3. You generate your report

Install Xindice

DarkSlide uses the native XML database Xindice to store the accounting data. Here is the procedure (also documented in the Xindice admin guide) to setting up the database to prepare for storing the accounting data.
  1. Download Xindice xml-xindice-1.0.tar.gz from http://xml.apache.org/xindice

  2. Install Xindice and start it up
    % gunzip xml-xindice-1.0.tar.gz
    % tar xf xml-xindice-1.0.tar
    % mv xml-xindice-1.0 /usr/local/
    % nohup /usr/local/xml-xindice-1.0/start &
    Now if everything went well, you should get a message saying "Server Running"

  3. Set important environment variables
    These must be set in your shell, this assumes BASH
    % export PATH=$PATH:/usr/local/xml-xindice-1.0/bin
    % export XINDICE_HOME=/usr/local/xml-xindice-1.0 

  4. Create the collection.
    You now must create a "collection" to store your XML accounting data in. By convention I just use the name of the host that the server is collecting data for, in this case the server is "morpheus". (remember what you use here, as later you will need to edit the config file to reflect the name of the collection you chose )
    % xindiceadmin add_collection -c /db/ -n morpheus
    Created : /db//morpheus

Install Darkslide

To install darkslide, download the RPM from http://sourceforge.net/projects/pbsaccounting/ and install as follows
% rpm -i darkslide-1.10-1.i386.rpm
Then you will need to edit the config QueryEngineConfig.xml file in /usr/local/darkslide-1.10/etc/. The most important parameter is to make sure the <collection> tag has the same collection name you created back in the "Create the collection" step. The default is "morpheus" which is the same name used in these examples, it is arbitrary and I generally use the hostname as mentioned previously.

You may also want to edit the title tag <title> that modifies the title of the report. You should not have to edit any other info on this file.

If you have any problems you may want to lower the debugging tag <debugging> from SEVERE to FINEST to get copious amounts of debugging information.

Load your XML accounting data

Assumingly you already a directory of XML accounting data produced by the PBS accounting toolkit at http://pbsaccounting.sourceforge.net. You can use this directory to do an initial "bulk load" of them into Xindice. This is a one time step.

Assuming all my XML data from a Pbs installation is in /usr/spool/PBS/server_priv/accounting/xml/

% /usr/local/darkslide-1.10/bin/bulkloader --directory=/usr/spool/PBS/server_priv/accounting/xml/ 
Document /usr/spool/PBS/server_priv/accounting/xml/20030331 inserted
Document /usr/spool/PBS/server_priv/accounting/xml/20030401 inserted
Document /usr/spool/PBS/server_priv/accounting/xml/20030402 inserted
Document /usr/spool/PBS/server_priv/accounting/xml/20030403 inserted
Document /usr/spool/PBS/server_priv/accounting/xml/20030404 inserted
Document /usr/spool/PBS/server_priv/accounting/xml/20030405 inserted
You can verify that the database contains the documents by using an Xindice command to query the files in the collection. In this case to query the morpheus collection the command is as follows
% xindiceadmin ld -c /db/morpheus

Generate the PDF report

To generate a report from 4-10-2003 through 04-29-2003 and save report in a file called /tmp/test.pdf type the following command:
 /usr/local/darkslide-1.10/bin/reportgenerator --startdate=04-10-2003 --enddate=04-29-2003 --filename=/tmp/test.pdf
To generate a report for a single day, like the 29th of April, the command would be
 /usr/local/darkslide-1.10/bin/reportgenerator --startdate=04-29-2003 --enddate=04-30-2003 --filename=/tmp/test.pdf

Creating Excel reports

To create excel reports simply give an ".xls" extension to your output file. For example
/usr/local/darkslide-1.10/bin/reportgenerator --startdate=04-10-2003 --enddate=04-29-2003 --filename=/tmp/test.xls

Using URWG XML accounting data instead of PBS XML accounting data

If you populate Xindice with XML accounting data corresponding to the global grid forums URWG XML Schema instead of PBS XML accounting data, you can still use darkslide to generate reports! You must change one line in the configuration file as follows
  1. Open up QueryEngineConfig.xml config file
  2. edit the module tag to tell it to use a different module for parsing XML data. Change <module> to read:
Note you can't mix URWG XML accounting data with PBS XML accounting data.

Daily updates

To load your daily XML accounting data into Xindice, simply create a daily cron job that loads the XML document into Xindice using the xindice commands here:
xindice add_document -c /db/morpheus -f /usr/spool/PBS/server_priv/accounting/xml/filename -n filename

where the filename is the name of the accounting file, like 20030418

Advanced querying

By loading the accounting data into a native XML database like Xindice, you get the power of doing any type of valid XPATH query against your data. For example, if you want to know all the days that the user "rmach" had completed jobs, you can use the xindice xpath command with the appropriate XPATH query as follows:
% xindice xpath -c /db/morpheus -q "//pbs_jobfile/execution_record[user='rmach']/date/day"
To use this mechanism you will need
  1. To know XPATH. A good tutorial is here
  2. To know the XML Schema for the accounting data, see the linux_joblog.xsd included in the pbs accounting toolkit distro.

Developer Document Listing

  1. Full API documentation
  2. Building

Downloading the code

http://sourceforge.net/projects/pbsaccounting/ has the source code available from CVS where you can build your own distro, includes all the documentation and unit testing. Here is how to get it from CVS (copied from sourceforge docs)
% export CVS_RSH=ssh
% cvs -d:pserver:anonymous@cvs.pbsaccounting.sourceforge.net:/cvsroot/pbsaccounting login 
% cvs -d:pserver:anonymous@cvs.pbsaccounting.sourceforge.net:/cvsroot/pbsaccounting checkout pbsaccounting

SourceForge.net Logo