Skip to content

Converting Benchmark Results to Spreadsheets

The conv subcommand allows you to convert results generated by btool eval into an XLSX spreadsheet, which should be used with Excel. Other programs, such as LibreOffice or OpenOffice, should also work. Keep in mind that by default LibreOffice does not automatically recalculate formulas. Therefore, after opening the spreadsheet, use CTRL + SHIFT + F9 to manually recalculate all formulas. You can also enable automatic recalculation in the settings. The spreadsheet will also include some basic plots. When using LibreOffice some features, such as dynamic axis titles, will not work and will have to be changed manually.

To convert your benchmark results to a spreadsheet, use the following command:

btool conv benchmark-results.xml -m "time:t,choices" -o results.xlsx

The name of the resulting .xlsx file is set using the -o, --output option (default out.xlsx). In cases of a missing or wrong file extension the correct file extension is added.

Which benchmark projects to include in the output can be selected via the -p, --project option. By default all projects are selected.

The -m, --measures option specifies which measures to include in the table (default: time:t,timeout:to; -m all selects all measures). Available measures depend on the result parser used during evaluation. Each measure can optionally include a formatting argument after a :. Currently, the supported formatting options are t and to. Both highlight best and worst values for float measures. Use t for most measures, and to for float measures representing booleans, such as timeout.

The --max-col-width option can be used to set the maximum column width of the spreadsheet in terms of pixel. The default is 300.

You can chose to export the instance data to a .parquet file using the -e, --export option. The name of the file will be the same as the specified output, i.e. -o res.xlsx -e -> res.parquet.

For large result files, generating the chart-related sheets can take noticeable time. Use --no-charts to skip creating these sheets.

Spreadsheet Generation

When generating a spreadsheet in XLSX format, normally five sheets are created:

  1. Instance Sheet
    • The instance sheet lists all runs for each benchmark instance (rows) and their results for the selected measures, grouped by system/setting (columns).
    • A summary for each run is included, showing the minimum, maximum, and median values for each float measure.
    • Each column with float measures also has a summary, including the sum, average, standard deviation, distance from the minimum, and counts of best, better, worse, and worst values.
  2. Merged Run Sheet
    • The merged run sheet merges all runs of a single instance depending on a selectable criteria
  3. Class Sheet
    • The class sheet summarizes all runs for each benchmark class, enabling comparisons between classes.
  4. Chart Sheet
    • The chart sheet contains some basics plots (survivor, cactus and cdf)
    • Which measures should be plotted and how runs should be merged can be selected by the user.
    • For additional fine-tuning, such as using log-scale or disabling certain series, the charts have to be adjusted manually.
  5. Helper Sheet
    • The helper sheet contains all data and calculations used for the charts and is by default hidden.
    • The sheet can be unhidden via the sheet selection UI.

Info

All summaries are written as formulas in the .xlsx file. The calculated values are also accessible via the content attribute of the Sheet object.

Both the XLSX representation and the actual content are stored in pandas DataFrames for easier handling and future modifications.