Creating Test Profiles

# Creating A New Phoronix Test Suite / OpenBenchmarking.org Test Profile

A test profile itself consists of a main XML file containing the test's meta-data (test-definition.xml), an XML file if needed to specify required files, download URLs and their MD5/SHA256 hashes and file sizes (downloads.xml), and the install.sh script for installing the test and generating the run-script for execution by the Phoronix Test Suite at run-time. There are also other potential files like pre.sh, interim.sh, and post.sh for execution at pre-run, interim-run, and post-run stages by the Phoronix Test Suite. With the script files comprising test profiles, there is also the option of post-fixing them with e.g. _linux or _windows if wishing to supply different script files based upon the operating system being run during testing, if the setup steps may differ greatly based upon MacOS/Windows/Linux/BSD/Solaris platforms.

Traditionally the easiest way to learn/understand test profile development has been by looking at looking at a basic test like c-ray, tesseract or the hundreds of others that are publicly available. Simply run for example phoronix-test-suite benchmark tesseract and then look at the test profile's contents either via the default locations in ~/.phoronix-test-suite/test-profiles or /var/lib/phoronix-test-suite/test-profiles/ if running as root on non-Windows platforms.

## Versioning Mandate

The Phoronix Test Suite / OpenBenchmarking.org philosophy mandates that with any change, a new version of the test profile be tagged. This is done for reproducibility and being able to ensure the exact test profile state when a set of tests are conducted. Test profiles are versioned in a test-X.Y.Z format where X or Y are bumped whenever a change is made to the test profile that makes the results incomparable to a previous version of the tests (e.g. updating against a new upstream code-base, adjusting the parameters to what is benchmarked, etc). The Phoronix Test Suite then knows not to attempt any comparisons with an incompatible version difference or to go and fetch that specific version of the package. The Z is bumped when just making non-important changes such as just updating the test's meta-data, changing download URLs, etc. This is easy to enforce with the OpenBenchmarking.org infrastructure rather than having to worry about non-human-friendly Git hashes as test profile versions or having to create Git tags after every commit. The test profile version can optionally be specified when running a test, e.g. phoronix-test-suite benchmark scimark2-1.2.1 instead of phoronix-test-suite benchmark scimark2, which would by default choose the latest available test profile version from OpenBenchmarking.org or the latest version on any local Phoromatic Server. The test profile versions are also always written out as part of the Phoronix Test Suite result XML data.

## Writing A Sample Program

Writing a test profile for the Phoronix Test Suite is a relatively quick and easy process for anyone familiar with common Linux commands and the basics of XML. To help you understand the design of the Phoronix Test Suite, this guide covers the steps needed to write a testing profile for a very simple application.

The first step in the profile writing process is to, well, have a piece of software you'd like to use with the Phoronix Test Suite. This software can be closed-source or open-source and be virtually anything as long as it is compatible with an operating system that is supported by the Phoronix Test Suite.

For this guide, the piece of software being used for demonstration is just a simple C++ program that calculates Pi to 8,765,4321 digits using the Leibniz formula. Below is this sample piece of software intended just for demonstration purposes.

> #include <iostream>
> #include <math.h>

> int main()
> {
> double pi = 0;

> for(long int i = 1; i <= 87654321; i++)
> pi += (double) pow(-1, i + 1) / (2 * i - 1);

> pi *= 4;
> std::cout << "Done Calculating Pi..." << endl;
> return 0;
> }

The first step in the actual profile writing process is to name it. If you're looking to ultimately push this profile to be included in the Phoronix Test Suite, its name must be all lower case and consist of just alpha-numeric characters, but can contain dashes (-). A more advanced test profile capability is operating system prefixes, and if using those there is an underscore separating the prefix from the normal profile name. For this sample profile, we're calling it sample-program and the file-name would be sample-program/test-definition.xml. Our (very basic) profile is showcased below.

> <PhoronixTestSuite>
> <TestProfile>
> <Version>1.1.0</Version>
> <TestType>Processor</TestType>
> <SoftwareType>Utility</SoftwareType>
> <License>FREE</License>
> <Status>PRIVATE</Status>
> <Maintainer>Phoronix Media</Maintainer>
> </TestProfile>
> <TestInformation>
> <Title>Sample Pi Program</Title>
> <TimesToRun>3</TimesToRun>
> <ResultScale>Seconds</ResultScale>
> <Proportion>LIB</Proportion>
> <Description>A simple C++ program that calculates Pi to 8,765,4321 digits using the Leibniz formula. This test can be used for showcasing how to write a basic test profile.</Description>
> <ExternalDependencies>build-utilities</ExternalDependencies>
> </TestInformation>
> </PhoronixTestSuite>

This XML profile is what interfaces with the Phoronix Test Suite and provides all the needed information about the test as well as other attributes. For a complete listing of all the supported profile options, look at the specification files in the documentation folder. In the case of sample-program, it lets the Phoronix Test Suite know that it's composed of free software, is designed to test the processor, is intended for private use only, and this profile is maintained by Phoronix Media. In addition, it tells the Phoronix Test Suite to execute this program three times and as no result quantifier is set, the average of the three runs will be taken. This profile also tells the Phoronix Test Suite that the generic build-utilities package is needed, which will attempt to ensure that default system C/C++ compiler and the standard development utilities/libraries are installed on your Linux distribution. This is needed as the C++ source-code will need to be built from source.

The next step is to write the install.sh file, which once called by the Phoronix Test Suite is intended to install the test locally for benchmarking purposes. The install.sh file is technically optional, but is generally used by all tests. Note: The first argument supplied to the install script is the directory that the test needs to be installed to. The install.sh file (in our instance) is to be placed inside test-profiles/sample-program. Below is the install.sh for the sample-program.

> #!/bin/sh
> tar -xjf sample-pi-program-1.tar.bz2
> g++ sample-pi-program.cpp -o sample-pi-program
> echo "#!/bin/sh
> ./sample-pi-program 2>&1
> " > sample-program
> chmod +x sample-program

This install file builds the code with GCC, and then creates a small script that is run by the Phoronix Test Suite. Where does the source-code come into play? Well, it needs to be downloaded now from a web server. The Phoronix Test Suite has built-in support for managing downloads from multiple servers in a random over, fall-back support if one mirror is done, and verification of MD5 check-sums. Below is the downloads.xml file for sample-program that covers all of this.

> <PhoronixTestSuite>
> <Downloads>
> <Package>
> <URL>http://www.phoronix-test-suite.com/benchmark-files/sample-pi-program.cpp</URL>
> <MD5>e90fb790df8d1544696a1439c9b5bd8d</MD5>
> </Package>
> </Downloads>
> </PhoronixTestSuite>

The final step in the profile writing process is to write a parser to strip all information but the reported result from the standard output or $LOG_FILE. In the case of a test profile just measuring how long it takes to run, it is as simple as a results-definition.xml looking like:

> <?xml version="1.0"?>
> <PhoronixTestSuite>
> <SystemMonitor>
> <Sensor>sys.time</Sensor>
> </SystemMonitor>
> </PhoronixTestSuite>

After that, with all the files in their correct locations, just run: phoronix-test-suite benchmark sample-program. The Phoronix Test Suite should now handle the rest by installing the test, running the test, and recording the results (if you so choose). There is no additional work that needs to be done for the results to be recorded in the results viewer or even reporting the results to OpenBenchmarking.org. An up-to-date version of this test profile can be run via phoronix-test-suite benchmark sample-program and then by looking at the test profile source via ~/.phoronix-test-suite/test-profiles/pts/sample-program* or within /var/lib/phoronix-test-suite/test-profiles/pts/ if running as root.