Contents

Update News
Purpose and Features
Structure of SABS
Downloading SABS
Installation Guide
Basic Usage Guide
Advanced Usage

In The News

January 13, 2008

To make things easier for those in the COMP-621 class looking to use SABS for their homework assignments, I have decided to release the raytracer example publically. This includes the code for the mini-raytracer itself, as well as the associated job and benchmark descriptor files.

Here it is, unzip the archive into your SABS installation folder: raytracer.tar.gz (31 KB)

Please keep in mind that you will need to edit task descriptors to match your system's configuration (e.g.: installation directories of your java virtual machine(s)). Do not expect this to run flawlessly "out of the box".

January 25, 2006

Eric Bodden of the Sable Research group has sent me an e-mail to tell me he had made some scripts to automate the use of SABS, and proposed to make those available for others:

Hi. I have written some shellscripts, which others would probably find useful, too. So feel free to distribute them. (see attachements)

sabslastmodified - This runs sabs automatically for the last modified configuration.
exportlast - Exports the last report that is available in all formats into its own subfolder.
showlast - Shows this last generated report in txt format. (less)
autosabs - calls sabslastmodified, exportlast and builds the generated tex file with latex, runs dvips and sends you an email with the attached PS file.

HTH,
Eric

--
Eric Bodden
Sable Research Group
McGill University, Montréal, Canada

So here they are: sabs_auto.tar.gz (1 KB)

Thanks Eric!

January 11, 2006

I did the presentation for the COMP-621 course today and realised I had not put up the presentation online. Hence, I am doing so now. Here are the links to today's intro tutorial as well as my NSERC presentation (which I hope is the latest one, if not, I will be updating it tonight). The NSERC presentation should give you insight on how SABS benchmark extensions can be used to inject AspectJ code into benchmarks. Both presentations are in microsoft powerpoint format, but they work fine in the latest OpenOffice version as well.

sabs_intro.tar.gz (1.1 MB)
sabs_nserc.tar.gz (1.5 MB)

January 9, 2006

It's me again! Long time no see. Everyone was surprised to see me in the Sable lab this morning. Well, I hadn't abandoned you, dear comrades, I've just been quite busy with my studies. Back to the point. I made a new SABS release for Professor Hendren's course. I have added task descriptors to support Soot (both default config and intraprocedural optimizations enabled). I also added a server and interpreter config for the Sun Java VM. And finally, I added a no optimization config for the ABC compiler.

I can't exactly claim that the support for Soot is the most solid ever. Due to a minor glitch (it seems soot won't work on class files in remote directories), I have been forced to use a shortcut to make it run. So the configuration is a little bit peculiar. Nevertheless, it works, and I think that's what's important. I will be preparing a small presentation for Laurie's course tommorow. I hope everything goes well.

August 19, 2005

My work with the raytracer is finally completed. I now have the exported reports I wanted, with all the necessary stats. In the process, I had to fix many bugs in SABS. There was a bug keeping the exporter from working when extensions were used, a bug in SABS causing conflicts with multiple levels of extensions, and a bug in my decimal limiting code. All those are fixed, and I took the liberty of improving upon things. Its now possible to give benchmarks a short name in the same way it can be done with tasks. Statistic names can now contain spaces. And decimals are specified per-statistic, instead of for the entire report, which seemed quite a bit more logical and less hacked. There are other things I would like to improve, but for the moment, I have to do the actual preparation of my presentation for next Wednesday (that is, preparing diagrams, powerpoint slides, and doing a certain amount of practice).

August 17, 2005

I have begun the work on my report material yesterday. Things didn't go too well remotely working from home because of the extreme slowness of the interface using SSH and xterm. For this reason, I will be working here until I have all the raw test material I need for my report (today and possibly tommorow as well). In this process, I found a bug in the launcher which is only present on windows. The bug is in the automatic path detection. I made a new release to fix it. I have also fixed a bug in the extension code that made the extension specific statistics not show up.

August 15, 2005

I have met Laurie Hendren this morning. She has suggested that I make improvements to SABS. The first is the ability to give a short name to benchmarking tasks to be used in the report. This was implemented. The second is the ability to limit the number of significant digits. I was able to implement a way to limit the number of decimals in the exporter. This would need to use some more investigation eventually, however. I have also fixed a bug in the automatic path detection that was introduced by the migration towards the launcher proxy system. The new release is available for download.

August 11, 2005

This morning I had a meeting with Professor Clark Verbrugge. I showed him my work on the exporter and the general state of things. He seemed quite satisfied, and suggested a few improvements to the LaTeX and HTMl file output, which I have implemented this afternoon. We have also discussed my work during the end of the month. Next week I am going to begin writing my Sable report, and the week after, I will work on my presentation with the help of Laurie. I should also show SABS in its current state to Laurie during the next week.

Fortunately, some of the material I will produce for my Sable report will be reusable for my presentation. This presentation will consist mostly of powerpoint slides, where I will show diagrams to explain how SABS works, from a component-based perspective, how everything effectively fits together, and how it is used. I will use the AspectJ raytracer program I created to demonstrate how SABS is used for practical applications. I will do my best to "market" SABS as well as I can during the presentation, especially since there is money at play. I honestly don't care all that much whether I win or not, although there is definitely an incentive for competition, but I definitely want my work to look interesting!

I will also try to take some time to bring the documentation on this site up-to-date, since quite a bit has changed in the way SABS works, and the way it is used. There are new SABS eatures which this site doesn't explain how to use at all, and this needs to be fixed.

August 10, 2005

I found the specification for the CSV format (Comma Separated Values), which is compatible with Excel and other spreadsheet programs, including Gnumeric, and I implemented support for it in the exporter today. It was rather easy (even trivial) to do because of the simplicity of the format. This is good because using this format, it becomes really easy to copy the tables from any spreadsheet program or to create diagrams and other graphics with them. In short, it will be a nice way to pass the data around.

I also found out about a very stupid bug in my java benchmarks. A few of the benchmarks had a return code of 256. It turns out it was because their input needed to be specified on the command line. I fixed this problem, and the benchmarks are now getting used properly. Since I have reached my original goals with the exporter program, I will show it to Clark tommorow, if he is available. Otherwise, I will begin the work on my Sable report and presentation.

August 9, 2005

As promised, I added new benchmarks to the java benchmark collection. I also found that it would be useful if the report exporter warned the user if the report being exported contains errors, so I added that. I was also able to get basic LaTeX exporting support working. Lets just say that it wasn't very easy due to the <sarcasm>very descriptive nature</sarcacm> of the LaTeX error messages, but I figured out what was wrong and got it to work. Tommorow, I will work on Excel format support.

August 8, 2005

Today I fixed a minor bug in the report file creation that was causing entries to appear multiple time. I also improved the organization of report files by sorting entries alphabetically and added support for exporting reports in ASCII text format. I also decided to indicate the task sequence index in the column names of the tables since it would be otherwise impossible to know which task is which when executing the same task multiple times.

Tommorow I will work on adding LaTeX and Excel format support. After which I will add support in SABS for more Java benchmarks so that I can build a more interesting report file with more statistics (which should be more interesting).

August 5, 2005

It works! I now have an implementation of the exporter that exports the tables in a logically arranged manner in HTML format. There are still refinements to do here and there, and other formats need to be implemented, but the most important is done. The system is now able to build tables automatically from the data that was gathered during the benchmarking process. On monday I will perfect this, create some more interesting tables and show Clark my progress. For now, have a good weekend!

August 3, 2005

I finally have a design I am mostly satisfied with. I made changes to the report manager so that it indexes tasks and benchmarks by the object representing them instead of by their file name. I also realised that it might be wise to include some kind of sequence number in the report file, to indicate the order of task execution (this is especially important if the same task is executed twice). I have completed the job report loading code and it seems to work as well. I had to use "late imports", because apparently, Python doesn't support circular dependencies.

I have also found a design that should work for the table creation system. I am looking forward to programming it tommorow. It should definitely work well and make things a good bit simpler than I originally thought. Laurie has also e-mailed me and asked me to write a Sable lab report for my project. I believe this could be a good preparation for my final presentation. This is another incentive for me to finish the exporter by the end of the next week, since I will need time to prepare the report, some proper documentation, and a complete presentation about the project.

August 1, 2005

The design is coming into place and the implementation has begun. I decided to implement the reading of report file into the report manager to preserve a certain degree of symmetry in the implementation. I decided to go for an inheritance design to manage the multiple output formats. Hopefully things will be simple enough. Just a little bit of heuristics will be required to build the tables intelligently. I will be reusing the code from task and benchmark descriptors to load all the data required and try to format things in a human readable manner, with human readable names. Of course, the goal of the exporter isn't to do all the work for us, but it should be capable of doing a good part of the work on its own.

I am aiming to complete this within the next two weeks. This would be ideal as it would leave me some time to prepare my final presentation (and perhaps a small GUI menu application). Tommorow I will begin the serious work on the report reading and I will begin to examine the approaches to take for the table creation process in the table manager class.

July 29, 2005

I begun the work on the format conversion program. I initially wanted to call it "export", but I realised that this was a UNIX command, same goes for "convert", so I settled for "exporter", which I figure is clear enough. I decided that it would be best if the exporter was in the installation directory, along with the SABS benchmarking program, because it would make it easier to share code and keep track of updates. For this reason, I had to modify the launcher program so that it could launch an arbitrary program in the installation directory instead of only SABS.

I made simple shell scripts that act as a proxy by supplying the launcher with the required program name. This design should actually make it easier to add a simple GUI layer on top if I have time to do it. As mentioned earlier, the exporter is going to reuse some of the code from the benchmarking program. Most notably, the XML parser and the config and path management system. The real issue now is to decide how to design things. I am thinking of possibly reusing the report manager to read in existing report files, but I'm not yet sure I like this design. I might actually create a new class to parse them.

Once report files are parsed, it will all be a question of building tables with all the required information. I might want to modify SABS to include more information in its report files, so I can "enrich" the content of exported files a little bit. It would be nice to include human readable names, for example, although it might be wise to simply read the benchmark descriptors for that. Another design issue is to know whether or not I will code everything into one big class that does everything, since everything is so tightly coupled. My current idea is to have a class to read the report files, and one to build the relevant tables. I might then plug-in special classes to handle each possible output format.

July 28, 2005

Today, I have completed the implementation of the text string matching and counting in the output of programs. This is another useful feature, requested by Professor Verbrugge, which is now part of SABS. I have updated the downloadable package as well. I believe SABS is now fairly complete. The next step for me is to begin the work on the output format conversion tool. I will start the design of this tool tommorow. I will start by creating a tool that works without a GUI, in command-line, and if I have the time (and Tk support), I will implement a GUI front-end menu to SABS. In any case, the command-line version will be made easy to use.

July 27, 2005

This morning, I have discussed the possibilities for SABS with professor Clark Verbrugge. He seemed to be satisfied of my progress overall and suggested that I add statistics about the occurence of strings in the output. This doesn't seem too hard to implement. Its basically a variant of my already existing statistical system. Once this is done, I will most likely working on a GUI-based tool that does output format conversion (LaTeX tables, HTML files, Excel files, ASCII tables) from the SABS report files.

This discussion turned out to be quite interesting. He also told me he believed Tk wasn't installed on all lab machines and suggested I try other ones. I have done just that this afternoon, and unfortunately, two machines were not responding, and the other one doesn't seem to have Tk either. Greg will be back in two weeks, I was told, which doesn't help. For this reason, I will start by implementing support for the output-information gathering mechanism first, since this requires no GUI. I will then work on a non-GUI based version of the output format conversion tool. In the end, the only thing that needs a GUI is perhaps a simple front-end menu.

July 26, 2005

I tried to use Tkinter on my home Windows machine, and to my surprise, it actually worked. The Tkinter module really is a standard python module, but for some reason, it won't work on the local machines. I decided to try compiling python locally, and to try the precompiled ActiveState python distribution that comes with extras, and although I got both to work, no luck with Tkinter. My guess is that it detects that the Tk libraries are not installed on these systems. This makes me reconsider the possibilities for the GUI program. If its that complex to get python GUI apps running on linux, I might just want to make it in Java.

I feel a bit paralyzed at the moment, but fortunately, Clark replied to my e-mail yesterday. He is coming back from his vacation and has scheduled an appointment with me in his office tommorow. Hopefully, we can discuss these issues. He might be able to help me find a fix, or take the appropriate decision. I am honestly a bit disappointed, since Python is supposed to be very portable, but oh well. At least I am getting good ideas for the layout of the program in the meantime.

July 25, 2005

I have begun my reasearch about GUI programming in python. Apparently, the standard GUI interface module is the Tkinter python module, which, as the name suggests, interfaces with Tcl/Tk. It seems that the module comes pre-installed with python, making it very easy, and portable over Windows. However, there seems to be a problem with the local Python installation at the Sable labs. For some reason, I can't import the Tkinter package at all, no matter what.

I have visited online programming forums and asked questions about this. Apparently, its really the case that you don't have to install Tkinter separately, and it should come with python. I have also tried to use Tkinter on the remote Windows 2000 machines, but it does not work either, although that might be because they have a very old Python installation (2.2.2). If I can't figure it out, I will notify Greg that there is a problem with the python installation.

EDIT: Apparently, Greg is also on vacation. A friend from the Sable lab suggested that I install the latest python in my user directory. I didn't think that was possible, but that could very well save the day, since neither Greg nor Clark nor Laurie seems to be coming back anytime soon. On a side note, I tried using Tkinter on the Trottier lab machines, and it works. I conclude that the problem really comes from the local systems. If I really can't get it to work, I can always work remotely on the trottier labs computers for the time being. Another possibility is to install "ActiveState ActivePython". I suspect this might be the only version that includes Tkinter, although I still don't have any *clear* information about this matter.

July 22, 2005

Today I have performed some additional testing, and I have prepared a new release. This release is version "1.0" of SABS. That is, the first "complete" release. This comprises a few bugfixes, and many new features, including the specification of the task iteration count inside job files, benchmark extensions, automated gathering of statistics, improved error logging, better outlining of events, the possibility to store benchmark files outside the SABS directory and automatic detection of default installation paths. The new downloadable package is now on this website.

Now that this is complete, and working well (at least within *normal* use conditions), I will move on to the design of tools to make the use of SABS easier. I will make a GUI-based tool to create job descriptors, task descriptors and benchmark descriptors. I might also make a GUI-based installer later on. When these tools are complete, they will be released as a part of the SABS package. They will hopefully make the use of SABS easier for those that are shellophobic as well as the general public. The main motivation of these tools will be to make it an intuitive and step-by-step process, probably in a "wizard-like" manner, with drag and drop lists and file browsing. Fortunately, I will be able to use my current XML parsing code without changes.

July 21, 2005

Today I have comlpeted the code for the extensions. I have also done some very basic testing and debugging. It appears to be working properly. I must admit the code is somewhat "hacked", but as I mentioned before, things didn't turn out as easy as I expected. Nevertheless, it works, and as long as users are careful when using extensions, there should be no problems. Tommorow, I will be doing some more extensive testing and preparing the SABS "1.0" release, that is, the first release to contain all the features I considered to be needed for the system to be complete.

This does not mean that the system is finished. Certainly, I will be making more improvements to the code and fixing more bugs in the future, but when this version is released, I will be moving onto something else for the moment. I would like to update the documentation on this website, so that it is more complete and relevant, since I have made some modifications to the way things are done. I would also like to work on tools to make it easier to use SABS. Perhaps I will learn how to use the python GUI tools and make an easy GUI to setup jobs, task files and benchmark descriptors. I think the members of the Sable Lab would definitely find that more convenient that always editing files by hand.

July 20, 2005

After a more careful examination of the possibilities, I have finally settled on a design for extensions, and begand detailing it and implementing it. I have decided that extensions would be specified in the job file at the time where benchmarks are specified. This has the meaning that extensions are, in fact, creating "new" benchmarks by combining themselves with existing benchmarks.

I have also decided to make the extension class extend the benchmark class. This simplifies code by allowing extensions to be loaded by the same code that loads benchmark descriptors. It also makes implementation easier since it will not be required for extensions to be completely disjoint from benchmarks, while interfacing them in a consistant manner, which implies that extensions still know all of the implementation details of benchmarks. I believe this design should be suitable. I carefully examined the points I needed to take into consideration and I believe I have resolved the important conflicts.

On another note, Nomair suggested that I should give an SABS tutorial when it is complete, and that I should demonstrate everything step-by-step in front of the group. I believe this would be a good idea. It might be a good thing to do this at the end of August as it might encourage more people to adopt SABS.

July 19, 2005

I am now working on the last planned major feature of SABS, that is, benchmark extensions. I have sent an e-mail to Professor Clark Verbrugge asking him if he could send me AspectJ code for aspects that can be added to benchmarks in order to gather extra information. However, I have just learned that he is on vacation until at least next Wednesday. Hence, I will be creating my own test aspects to test extensions. Once extensions are worknig properly, I don't think that there should be any problem substituting any other generic aspects.

I have a fairly complete design in mind for the extensions. I am thinking of giving them their own file format, similar to the benchmark descriptor file format, and that job descriptors should specify the extensions to be applied to all benchmarks at once. Yet I am unsure how this should be specified. The problem here is that aspects are actually modifying the compiled result. Hence, I am wondering if extensions should be specified as a category, to be applied to all benchmarks, or per task application, so that we can run tests with and without extensions, or with a complex combination of them within the same job, for comparison. I am currently leaning towards the latter approach, even though it will be slightly more complex to implement.

There is also a third possible approach I could employ, which would specify extensions to apply to benchmarks at the moment where the benchmarks of a job are specified. This has the advantage that current job descriptors would remain compatible, and that it woud simplify the implementation a bit. At the same time, it would also remain rather flexible. This is definitely an approach to consider as well.

July 15, 2005

I now have a working implementation of automated statistic gathering from programs. Fortunately, it was easier than I expected it to be. I ran into small issues with the XML parsing and some special characters used in regular expressions, but those were easily fixed. I also took the opportunity to improve my XML parsing code somewhat. What this all means is that its now possible to specify in a benchmark that a certain statistic should be gathered from the output of the program when it is executed. This is done by specifying a regular expression to be matched in the output. This regular expression contains a named group that corresponds to the name of the statistic itself. From this, SABS will automatically build a statistical population for this statistic and include it in the report file.

The system also does a minimum of error checking. If the regular expression does not match, for example, an error is reported. I have done some testing and it seems to work well. I was able to gather statistics about the number of megaflops, as well as the internal running time of the Linpack benchmark. I still compute the wall-clock time, but I renamed this statistic to "wallclocktime", to make it more obvious what it is, and avoid conflicts with other statistic names. Keeping the statistics on a per-benchmark basis seems like a good choice, because, for example, the java benchmarks I have, even if they are from the same collection, all report running-time in a different manner.

There is only one last feature important feature for me to implement. This is benchmark extensions. I might have to discuss those with Clark as I'm not yet sure how they should work, and Laurie is not there. In any case, I will begin designing those next week to the best of my knowledge.

July 13, 2005

I received an e-mail from Jorgue Ressia informing me of a bug when using SABS on Windows. Apparently, the ':' character can't be a part of Windows file names. He also suggested that it might be nice to allow benchmark programs to be stored anywhere on the file system instead of forcing them to be in the benchmarks directory. I implemented the bugfix and suggestion today. Those will be part of the next release.

I started looking into python regular expressions, and apparently, its possible to have named groups. This might allow me to implement gathering of statistics more efficiently, without writing my own simplified regex system. There are other design issues involved, but I will keep looking into the problem, and should have an efficient solution soon. Some of the issues involved are whether the statistics should be per-benchmark or per-task. Right now, I'm thinking of making them per-benchmark as it looks rather inflexible and bug-prone to try forcing every program to have the same output. Another issue is where (from what) the output should be captured. I think I might have to define one of the task script steps as a "capture point".

July 12, 2005

Thanks to Bruno Dufour, I now have a standard task file to run StarJ and generate runtime statistics about benchmark programs. This completes the first phase of improvements that were made from user feedback. In addition, I have begun to add new features to SABS. I have modified the code so that it is now possible to specify the iteration count for a task in the job descriptor. This makes it possible to run a set of benchmarks as many times as desired, without having to modify the task file itself (and affecting other jobs). If no iteration count is specified, SABS will use the value already provided in the task descriptor as a default count.

I have updated the downloadable package with these new features. I feel like SABS is finally getting a bit more complete. Tommorow, my goal will be to work on the regex system to catch values to build statistics from in the program output of benchmarks. I have also started the code organization section of the advanced usage guide. It might be wise to add a section about the usage configuration of AspectJ with SABS, I will have to consider that.

July 11, 2005

Bruno Dufour has replied to my e-mail about SABS. He suggested the problem Sokhom and I had been having was due to the local path not being included in the classpath. It turns out this really is the problem. Fortunately, the children shell that Python creates when opening a pipe is a child of the shell from which Python was launched. This means that the classpath variable of the user's shell can be modified to include the current directory '.', and the user will be able to run StarJ properly.

Now that the StarJ problem is resolved, I will be concentrating on the addition of features to SABS. I believe there are 3 things we mostly need:
- Capturing info from programs using regex
- Possibility of specifying the iterations of a task in the job descriptor
- Benchmark descriptor extensions that allow us to "aspectify" a benchmark

July 8, 2005

I did my best to investigate the cause of the problem with StarJ that Sokhom reported. I believe I was able to reproduce the same problem, but I did not find the cause nor any kind of fix. I tried dozens of things, but the problem is just very confusing. Apparently, the analyzer agent works fine if I just run it manually, but if I run it from SABS, in the same conditions (same working directory, ensuring the same JDK is used), it fails to work.

Having ruled out all the possible causes I could imagine, after many failed tries, I decided to e-mail Bruno Dufour, the conceptor of StarJ. Hopefully, he will be able to try it on his own, using the task descriptor I sent him, or at least give me clues to investigate. I really want to fix this, as it would be rather bad if SABS did not have StarJ support (it would reduce its usefullness, and validity as *the* Sable benchmarking framework). I'm still confident we will find a solution. In the worst case, StarJ is open source, and we could always make a custom-tailored build of StarJ for SABS.

July 6, 2005

Today I have completed the error code detection mechanism. I have also updated the downloadable SABS package. SABS will now detect, clearly indicate and report compilation errors as well as runtime errors. I have also made slight updates to the usage guides. Those were mostly consistency updates because of the recent changes I have made to the system and its configuration.

My immediate goal is to investigate the StarJ incompatibility problems. Then, in the coming weeks, to work on more advanced features. I will not begin the code structure section before some required features are implemented. In the coming weeks, once SABS has all the features it needs, I will be working on tools aimed at making SABS easier to use, and converting report files into more usable file formats.

July 5, 2005

I have completed the section about adding support for new tools. I will still have to update it, however, as some of the changes I made today will change the task descriptor format a small bit. I have also added a new section about the structure of the SABS code. I am hoping that no unseen bugs will pose problems to anyone, but for maintainance and future support purposes, I have decided to explain how my code is structured in some detail. This should be enough for someone to continue the development of SABS if I become unavailable.

On another note, I have completed the code to detect compilation errors. It works. In all three currently-supported compilers (Java, ABC, AJC), the errors are detected through the return code. Runtime errors in the benchmarks are also detected. When an error is found, it is logged in the report file, and outputs a clearly visible error message in the console output. In the process of doing this, I have also fixed a bug in my statistics code. I haven't released a new package today because I want to test all other benchmarks, and set their return codes in the benchmark descriptor so that runtime errors can be detected.

I will most likely release the new package tommorow. After this, I will try to find someone to help me find what isn't working with StarJ. Hopefully it won't be too difficult to fix. When all this is done, I will work on combining benchmarks and gathering more statistics. I will need to meet Clark to discuss this.

July 4, 2005

Back from this long weekend, slowly re-gaining the pace of work. At least today the temperature is viable outside. Hopefully the Americans will not get in an interplanetary war like in that Hollywood flick (something about the Boston Tea Party, but with aliens). The news are that Laurie is going away for more than a month. I assume that I will be under the responsibility of Clark while she is away. In any case, he is the one I will inform of my progress when the next update comes.

I did more experimentation today. Fortunately for me, things went rather well. As it turns out, it is possible to catch the return code of programs with the pipe functions in python. It also turns out that when there is a compilation error, in javac, abc and ajc, the return code is nonzero for all three. This gives me an easy way for the framework to detect compilation errors. I still need to settle for an efficient way to specify when and how to apply this error detection mechanism, because not all tasks return 0 on success. I might simply add an entry for the valid return code in the benchmark descriptor execution section.

June 30, 2005

I have done some more research and made some improvements to the system. I have also updated the downloadable package. The main improvements are that the shell error is fixed, the task starts and errors are clearly outlined in the console output of the program and the system is now able to detect default installation paths on its own.

This improvement should make things much easier for anyone that wants to install the system on a single-user machine. The package can be extracted, and if the SABS directory is left as-is, it will be able to detect its own paths, and run without any configuration. Otherwise, for a multi-user install, it will rely on the system configuration file. I have also updated the installation guide with one precision about benchmark descriptors.

The next steps for me are to be able to catch java compilation errors, to investigate the StarJ problem, and to complete the section about adding support for new tools.

June 29, 2005

I assisted to yesterday's Sable meeting. I was happy to get more feedback about SABS. Many people tried it this time. Some interesting remarks were made, the most relevant to me being:

- The start of new task should be outlined better in the text output
- Errors should be very clearly outlined
- It is possible to use $folder/*.java in benchmark descriptors
- When the compilation fails, the system should catch the error
- There appears to be a problem with StarJ (to be investigated)
- The system could auto-detect the default path configuration

I have also looked into the confusing shell error that was coming up. Google helped me find the cause of the problem and I was able to solve it. This will be fixed in the next update of the downloadable package. I also need to complete the section on this page about new tools, and add some precisions, and perhaps a FAQ.

June 28, 2005

I have updated the downloadable package today, as promised. This time, I have full support for compilation and execution, or simple execution of all the SPECjvm98 benchmarks. I have created patch files that fix the code inconsistencies in two of the benchmarks that made them not compile. These patches and instructions on how to apply them are available at the address below:

sabs_specjvm98_patch.tar.gz (The file size is 1 KB)

I have also received feedback from one person. Dayong tried the system and e-mailed me his remarks. He has made some interesting points and made me realise that I should probably try to eliminate the shell error that is produced when running SABS. I have tried to solve this problem today, but did not yet find a solution. I will probably try again tommorow before the meeting. Perhaps someone on the internet has faced a similar problem before. I will also update this page, as promised.

June 27, 2005

I have sent a reminder this morning to encourage everyone to try SABS. This will hopefully get me some feedback. Today I have expanded my scripting system for task descriptors. I have added a new command that can copy the entire content of a directory recursively. This has allowed me to add full support to run all of the SPECjvm98 benchmarks. I have tested it, and everything seems to work fine. I had to use separate task descriptors for compiling and running and simply running, however. Tommorow I will probably make a patch so that all of the SPEC benchmarks that include source code can be compiled. This will complete thefull SPECjvm98 support in SABS.

This is it for now. My todo list for tommorow includes making a patch for SPECjvm98, some instructions on how to apply it, and continue my advanced usage guide on this page to help people add support for new tools to SABS.

June 23, 2005

Today I have improved on my small scripting language for task descriptors. It now supports the passing of arguments in such a way that they can contain spaces in paths, I also find it more readable and crash-safe like this. I have also been looking at options to bring run-support for all of the SPECjvm98 benchmarks. It seems like I will need to add a command to copy all of the source code before compiling it, to avoid polluting the benchmark folder with temporary files. This is what I will be doing next.

After that, I will want to look at patching the errors in the SPECjvm code files. I could do this through diff and patch, or by adding simple text insertion and replacement commands to my small scripting language. This might be considered overkill, however. I presume its better not to overcomplicate the benchmarking framework. I have done minor updates to the guides on the site so they reflect the current state of the framework more accurately, but I will also want to complete the advanced usage information.

This is it for now. I hope some people will try the framework and give me some feedback.

June 22, 2005

The website should be unveiled at the Sable Labs meeting today. This will hopefully encourage more people to try out the benchmarking framework. I have completed a new release in time for this.

Partial support for SPECjvm98 benchmarks has been added, within the limit of what could be supported without modifying the benchmark source code. It is now possible to compile and run 4 of the SPEC benchmarks. Two more could be compiled by issuing a patch to fix minor bugs in the code (possibly features that are no longer supported in javac).

June 15, 2005

Adding support for the SPECjvm benchmarks turns out to be more difficult than I thought. These benchmarks have some strange compiling configuration, and will require access to external input data. I will need to communicate with Clark or Greg to figure out how to compile them in SABS. In the meantime, I have added support for the 10 JOlden benchmarks and included them in the updated downloadable package. These benchmarks take a few minutes to run, which is more useful to test the system and see how it works.

I will not be here this afternoon for the regular meeting because I have an appointment with the doctor. This is not a bad thing, however, since Laurie is currently away at PLDI. She told me that we should discuss the progress of the benchmarking system on friday. This suits me. Hopefully, I will be able to add support for the SPEC benchmarks tommorow.

June 10, 2005

I have written a section of the advanced usage guide that explains how to add support for new benchmarks to the system. I figured that it was needed to explain how to do this since people are necessarily going to want to do this. I left the section about adding support for new tools for later. I also added a simple and discrete counter at the bottom of the page. I consider that the webpage should be ready for our meeting next week. Anyone that wants to try the framework should be able to install it, and if some details are still not clear, I am always available to help, and I will not mind updating the usage guides.

As it was suggested in the last benchmarking meeting, I have modified the handling of the statistics. The default is now to filter only the top 1 and bottom 1 results. I also include the filtered results in the report. Additionally, the default number of iterations is now 10 instead of 20. I have taken the time to fix the code for the automated removal of temporary files as well.

Things seem to be going well. I have already added support for the abc and ajc compilers. What is next on my list is to add support for more benchmarks. This is what I will be working on until the next meeting. In the meantime, I have updated the downloadable package.

June 9, 2005

I spent most of the afternoon working on the website. I have completed a step-by-step installation guide and a basic usage guide. I tried to be as concise and clear as I could be. I have also added the same basic design overview that was found in my powerpoint presentation, for any outsider that would want an overview of what the system is about.

I had a small bike accident and fell hard on the concrete this morning. I didn't feel in such a great shape after that, so I wasn't productive enough to fix the temporary file deletion problem today. The benchmarking framework still works without that, however, and I did update the package. Luckily, my trusty schoolbag absorbed most of the hit when I fell off.

My plans for tommorow are to add a small counter so I can see the web traffic better. I will also fix the temporary file deletion problem, and probably start on my advanced usage guide. Although I am not certain its a good idea to write alot about it now, as certain details about the system might very well change. I plan to have support for more benchmarks (SPEC) and more compilers for the meeting next wednesday.

June 8, 2005

We had a meeting today and it was decided that I would be creating a webpage that would include a simple step-by-step usage guide. This will be my main priority for the moment, since many people are going to be trying out the system. I plan on having a sufficiently complete website tommorow, along with an updated downloadable package.

After that, I will be resuming the work on the handling of temporary files. I still need to clean the contents of the temporary directory properly. After this is done. I will add support for the SPEC benchmarks, and then for the abc and ajc compilers.

Purpose and Features

Why a New Benchmarking Framework?

Design Goals of this Project

Structure of SABS

System Design Overview

Job Descriptor Files

Task Descriptor Files

Benchmark Descriptor Files

Installation Guide

The following is a step-by-step installation guide for Linux and Windows in 5 steps.

1. Obtaining Python

The SABS framework is programmed in the python scripting language, which allows it to run on both Linux and Windows. In order to be able to use it, you will need to have the python interpreter installed on your system. SABS was designed using Python 2.3.5. Hence, we recommend that you download and install version 2.3.5 or higher if you do not have it. The latest python version is always preffered.

If you already have python installed on your system but do not know which version, simply open a command-line terminal and enter the "python" command (without the quotes). This will start a python interpreter session which will output the version number at the top. To exit this interpreter, simply press ctrl+Z on Linux or ctrl+D on Windows. If your Python version is older than version 2.3.5, please upgrade it, as we cannot guarantee full compatibility of SABS with older versions.

You can download the latest version of Python from http://www.python.org.

2. Downloading SABS

The second step is to download the SABS package from our website by clicking on the following link:

sabs.tar.gz (The file size is 132 KB)

3. Extracting the Package

You can extract this package on Linux by typing "tar -xvvzf sabs.tar.gz" (without the quotes) in a command-line terminal. On Windows, the SABS package can be extracted using the Winzip program. If you use another program to extract the package, make sure that it respects the directory structure of the archive. Once the package is extracted, the contents should be under a new "sabs" directory.

4. Installation

Once the SABS package is extracted, you will find that the sabs directory contains 3 subdirectories. The first one is called "docs". This contains the LGPL licensing information, along with other documentation about the framework. The other two are called "install" and "local". These directories contain the benchmarking system itself.

The reason the benchmarking system is divided among these two directories is to provide support for multi-user installations. The "install" directory corresponds to the actual program code, along with the benchmarks that every user of the benchmarking system should have access to. The "local" directory is the user side of the installation, and allows each user to introduce new contents and override global settings without affecting other users in any way.

If you are on a single-user machine, you can simply place the sabs directory and all its contents in home directory and use it from there. On windows, we recommend installing it at "c:/sabs". On Linux, you can simply place the "sabs" under your own home directory, located at "/home/*username*". Where *username* is your user name on the system.

For a proper multi-user installation, the "install" directory needs to be placed in a path such that all users have read access to it. On Windows, this could be "c:/sabs". The "local" directory of each user should ideally be placed in each user's home directory, which is at "c:/documents and settings/*username*". On Linux, the "install" directory should go under "/etc" and the "local" directory should go under "/home/*username*. Rename the "local" and "install" directories as you see fit, but make sure to note their path for the configuration step.

5. Configuration

The last step is to configure the path information of your local installation. At this time, this can only be done by manually editing the XML system configuration file. This file is located at "*local*/system/config/system.xml", where *local* corresponds to the placement of your "local" directory.

This file contains 4 variables you need to edit. These are "installsyspath", "instalbmpath", "localsyspath" and "localbmpath". If you have not modified the internal directory structure of the "local" and "install" directory, then these variables should be set as follows:

<config>
  <var name="installsyspath">*install*/system/</var>
  <var name="installbmpath">*install*/benchmarks/</var>
  <var name="localsyspath">*local*/system/</var>
  <var name="localbmpath">*local*/benchmarks/</var>
</config>

Where *install* and *local* correspond to the paths of the "install" and "local" directories that were originally under the "sabs" folder in the extracted installation package, respectively.

Once this configuration is completed, the benchmarking system should be ready to run.

Basic Usage Guide

Basic vs Adanced Usage

When we say "basic usage", what we really mean is "standard" usage. That is, performing a batch of benchmarks using tools that are already supported by this system. The SABS framework was conceived so that as little configuration as possible would be needed in order to use the system in such a way. The system comes with a set of task descriptor files that tell it how to compile and execute benchmark programs using a set of common compilers and virtual machines. This avoids having to write descriptor files for these tools in order to use them with the system.

If the user wishes to use unsupported compilers, virtual machines or tools, or add his or her own benchmarks to the system, he or she will be required to write task decriptor files for these tools in order to use them with the SABS framework. This is considered advanced usage.

Creating Job Descriptors

In order to perform basic actions (compile and execute) benchmarks in a basic way, the user minimally needs to write a job descriptor file. This file specifies which benchmarks are to be processed by providing the name of their benchmark descriptor file, and specifies which actions to perform on the benchmarks, in order, by specifying a list of task descriptor files.

The job descriptor files are simply XML files. These are located under the "system/jobs" subdirectory of the "*install*" or "*local*" directory. The job descriptors that come with the SABS framework by default are located in the "*install*/system/jobs" directory. If you wish to create your own job descriptor, you should place it in the "*local*/system/jobs" directory. Your new job will override any standard job with the same name, if any exists.

You can see the contents of an example job descriptor file below:

<job>
  <benchmarks>
    <benchmark>linpack</benchmark>
    <benchmark>life</benchmark>
    <benchmark>matrixmult</benchmark>
    <benchmark>qsort</benchmark>
  </benchmarks>
  <tasks>
    <task>setup_create_temp</task>
    <task>comp_javac_std</task>
    <task>exec_sunjvm_std</task>
    <task>setup_delete_temp</task>
  </tasks>
</job>

As you can see, this job descriptor specifies 4 benchmarks to be processed and 4 tasks to execute on each of these 4 benchmarks. When this benchmarking job is run, each task, in the order they are specified, will be applied to each benchmark. The "setup_create_temp" job begins by creating temporary directories for each benchmark. These temporary directories will contain intermediate compilation files and such. The "compile_javac_std" task then compiles each task using the javac compiler with standard options. Each benchmark is then executed in the Sun JVM by the "exec_sunjvm_std" task. To finish the process, the "setup_delete_temp" task deletes the temporary files left over by each benchmark. This task can be ommited if one wishes to keep the compiled benchmark code.

Running Benchmarking Jobs

Running the benchmarking jobs is simply a matter of telling the system which job we wish to run. All that needs to be done is to issue a shell command to launch the job. This is done using the sabs script (on Linux) or the sabs.bat script (on Windows). These scripts are located in the "*local*/system" directory.

For example, if we want to run a test job whose job descriptor file is called "compile_run_test.xml", we would open a command-line terminal, go to the "*local*/system" directory, and issue the "sabs compile_run_test" command. This will run the job, and provide feedback to the user as each task is being performed. When the job is done running, a report file will be produced in the "*local*/system/reports" directory. This report file will be named according to the name of the job, the time the job was run, and the name of the machine it was run on.

In order to simplify the execution of jobs, it could be preferable to add the "local/system" directory to your system path.

Exporting Report Files

Once a benchmarking job has been run, an XML report file is produced. This file contains all the statistical information gathered during the benchmarking process and more, but can be rather cumbersome to work with. For this reason, an exporter program is provided. This program will automatically create human-readable report files containing tables for all the statistics gathered during the benchmarking process. The exporter is able to export files in the HTML, LaTeX, ASCII and CSV formats, and will automatically export to all four of them.

In order to call the exporter, you may issue the command "exporter <report_name>, where <report_name> is the name of the XML report file you wish to export, without extension and without path. For example, if you wanted to export a report file called "comparative_test_18h15m05s_lsv.xml", located in "*local*/system/reports", you would issue the shell command "exporter comparative_test_18h15m05s_lsv" from the "*local*/system" directory. This would produce exported report files of the same name as the XML report files, but with different extensions, in the reports directory.

Advanced Usage

Adding New Benchmarks

Adding a new benchmark to use with the SABS system is simply a matter of placing your benchmark files in the "benchmarks" directory and creating a benchmark descriptor file to tell the system how to use those files. The benchmark descriptor is simply a handle for the system to know how to use your benchmark, or, more specifically, how to apply benchmarking tasks to your benchmark.

The first thing you have to choose is where you wish to place the benchmark files. If you want every user of the system to have access to them, then they should be placed in a subfolder of the "*install*/benchmarks" directory. Otherwise, if you wish to be the only one to have access to them, then your benchmark files should be placed in a subfolder of the "local/benchmarks" directory. For example purpose, we will place the source files of a benchmark called "MyBenchmark" under the "*local*/benchmarks/MyBenchmark" directory.

Once the files of the benchmark have been placed in the appropriate subfolder, a benchmark descriptor must be created to reference those files. The benchmark descriptors are XML files that contain information which will be made accessible to compilation tasks. This file can also be placed in two different location. Either in "*install*/system/benchmarks" or in "*local*/system/benchmarks", depending on where the benchmark files were placed. For our example, we will be adding a benchmark descriptor file called "mybenchmark.xml" under "*local*/system/benchmarks".

Here is the content of our example "mybenchmark.xml" file:

<benchmark>
  <name>My Example Benchmark</name>
  <folder>MyBenchmark</folder>
  <taskinfo class="setup">
    <var name="outfolder">$tempfolder</var>
  </taskinfo>
  <taskinfo class="compilation">
    <var name="classpath">$folder</var>
    <var name="outfolder">$tempfolder</var>
    <var name="codefiles">
      $folder/Main.java
      $folder/Example.java
    </var>
  </taskinfo>
  <taskinfo class="execution">
    <var name="classpath">$tempfolder</var>
    <var name="workdir">$tempfolder</var>
    <var name="mainclass">Main</var>
    <var name="arguments"></var>
    <var name="returncode">0</var>
  </taskinfo>
</benchmark>

The "name" tag specifies the human-readable name of the benchmark. This name is not considered internally by the system. The "folder" tag specifies the subfolder of the "benchmarks" folder in which the files of the benchmark were placed. In our case, since our benchmark was placed under "*local*/benchmarks/MyBenchmark", the folder name has to be "MyBenchmark".

The "taskinfo" tags are very important. They specify information that will be accessible by compilation tasks. Here the task "class" is the category it is part of. For example, the "comp_javac_std" (compilation in javac with standard options), is part of the "compilation" class. Each taskinfo tag specifies variable bindings that will be accessible by the tasks of the "class" they are specified for. We see here that the "codefiles" variable will be accessile by all "compilation" tasks. This has the effect of telling the compiler which files to compile. The other "taskinfo" tag contains variables for the tasks that are part of the "execution" class. These variables tell execution tasks what is the main class used to run the program, and what arguments to pass to the program when running it. The "returncode" variable is the expected return code of the benchmark program. This is used to detect runtime errors.

Please note that it is also possible to list all the code files in the benchmark's folder at once by setting "$folder/*.java" as the content of the "codefiles" variable. It is also possible to list all the code files in multiple folders. Once a benchmark descriptor has been created for a benchmark, it becomes possible to reference it in a job descriptor file in order to use the benchmark. See the section regarding job descriptor files of the basic usage guide for more information.

Adding Support for New Tools

In order to add support for a new tool, a new task descriptor needs to be created. As mentioned before, the SABS software itself has no notion of what a compiler or a Java VM is, and how they are used. A task descriptor serves the purpose of telling the benchmarking framework how to apply a tool to perform some action, of any kind, on an individual benchmark. A task descriptor could tell SABS how to compile source files of any language with a particular compiler, for example, but they could also tell SABS how to perform any kind of post-processing on the output data of a program.

Task descriptors give a large amount of flexibility to SABS, but they also have their limitations, however. One of these is that a task descriptor relies on specific information to be provided by each benchmark descriptor. For example, in order to compile a benchmark, a compilation task will need to have access to a listing of the source files to be compiled. Another limitation is that task descriptors do not act as an interface to a tool, but rather as a simple "hook". As such, task descriptors do not allow for much customization. They only store one possible fixed configuration of the tool. This means that, for example, if you wanted to compile benchmarks with and without optimizations, two task descriptors would need to be created. One to compile with optimizations, and one to compile with the "standard" configuration. While this limitation might be cumbersome to some extent, it also has the advantage of making it more obvious what the purpose of a task is, and how to apply it (there is only one way to apply one benchmarking task).

Creating a task descriptor is as simple as creating a benchmark descriptor. Only one XML file containing the information relevant to the task needs tobe created. As with benchmark descriptors, there are two possible locations for task descriptors. The task descriptor file can either be placed under "*install*/system/tasks", in which case it will be visible to all users of the system, or in "*local*/system/tasks", in which case it will only be visible to the local user, and will override any task from the "*install*/system/tasks" folder that would have the same name as the new one.

For example purpose, here is the content of the "comp_java_std.xml" task descriptor:

<task>
  <class>compilation</class>
  <name>Compilation using javac with standard options</name>
  <script>
    <step type="exec">
      <command>javac -classpath "$classpath" -d "$outfolder" $codefiles</command>
      <returncode>0</returncode>
    </step>
  </script>
  <iterations>1</iterations>
</task>

The "class" entry of the task descriptor specifies what kind of class this task relates to. You can specify any class name you like (even one of your own invention). As specified earlier, the task class specifies which variables of the benchmark descriptor this task can access. Hence, if you create a new task class, you might want to introduce variables of that new class in the benchmarks to which your new task can be applied. The "name" entry is simply the human-readable name of this task. SABS pays no special attention to this field.

The "script" entry specifies how the task is to be applied. In this case, we can see that this script comprises only one step, which executes javac. It gives javac a set of command-line parameters which include some variables (indicated by the dollar sign). These variables are all defined in "taskinfo" blocks of concerned benchmark descriptors. As you can see, a "returncode" value is also specified. This tells SABS what return code is expected to indicate success. In the case of javac, abc and ajc, the return code is 0 to indicate success. It is possible, however, to specify this value using variables that are declared in the benchmark descriptor. Finally, the iterations entry tells the SABS framework to execute this task only one time.

Organization of the SABS programming code

The SABS software is divided into two parts: the program installation (located in the "sabs/install" directory of the downloadable package) and the local installation (located in the "sabs/local" directory of the downloadable package). The "install" directory contains the part of the software to be accessible by all users. The "local" directory is for the local installation, and each user of the system should have a copy of it. The motivation behind this design is to allow each user of the program to override the default options without affecting other users.

The local installation contains directories that mirror the program installation. That is, it contains directories to store benchmarks, benchmark descriptors, task descriptors and job descriptors. When SABS is asked to run a job, or find a benchmark descriptor, it first looks in the local installation directories (which have priority), and then looks in the program installation directory if they were not found. This implies that in order to find all the files, SABS needs to know the paths to the local and program installation directories.

To run SABS, the launcher program (contained in the local directory) is supplied one argument: the benchmarking job to be run. By default, the launcher program will try to find the program installation directory in the default layout of the downloadable package, that is, it will assume that both the "install" and "local" directories are still both under the "sabs" directory. This is typical for a single-user installation of SABS. If the directories are not found, the launcher program will examine an XML config file to see if the user has manually set other program directories. Assuming the directories have been found, the launcher program will then execute the SABS program from the program installation directory, supplying it the name of the job to run, along with the required installation paths.

The program installation directory contains the core of the SABS software. That is, it contains all the code to read the different descriptors, to run the benchmarking jobs, to execute the simple scripts in the task descriptors, to produce report files, etc. All of this code could have been placed directly in the local installation directory, but having it in one central location makes it very easy to upgrade the SABS software if necessary by replacing the code in the program installation directory. On the other hand, the SABS launcher code does not really need to change, as it will always perform the same function. If new functionalities need to be implemented into SABS, this will most likely be done in the SABS core code.

The SABS code is very well commented and Open Source, making it easy for anyone to implement new features when required. The only tool that is needed to work on the code is a text editor to edit the ".py" files, which contain the python source code for SABS. The Python scripting language on its own is rather simple, and can be learned by reading the online references available at www.python.org.