Skip to content

Converting Benchmark Results to Spreadsheets

The conv subcommand allows you to convert results generated by btool eval into an XLSX spreadsheet, which should be used with Excel. Other programs, such as LibreOffice or OpenOffice, should also work. Keep in mind that by default LibreOffice does not automatically recalculate formulas. Therefore, after opening the spreadsheet, use CTRL + SHIFT + F9 to manually recalculate all formulas. You can also enable automatic recalculation in the settings.

To convert your benchmark results to a spreadsheet, use the following command:

btool conv benchmark-results.xml -m "time:t,choices" -o results.xlsx

The name of the resulting .xlsx file is set using the -o, --output option (default out.xlsx). In cases of a missing or wrong file extension the correct file extension is added.

Which benchmark projects to include in the output can be selected via the -p, --project option. By default all projects are selected.

The -m, --measures option specifies which measures to include in the table (default: time:t,timeout:to; -m all selects all measures). Available measures depend on the result parser used during evaluation. Each measure can optionally include a formatting argument after a :. Currently, the supported formatting options are t and to. Both highlight best and worst values for float measures. Use t for most measures, and to for float measures representing booleans, such as timeout.

The --max-col-width option can be used to set the maximum column width of the spreadsheet in terms of pixel. The default is 300.

You can chose to export the instance data to a .parquet file using the -e, --export option. The name of the file will be the same as the specified output, i.e. -o res.xlsx -e -> res.parquet.

The -j, --jupyter-notebook option can be used to generate a .ipynb file, which contains some basic visualization of the instance data, specifically survivor, cactus and CDF diagrams. The notebook can be started using jupyter notebook <notebook.ipynb> and dependencies for the notebook can be installed using pip install .[plot]. This option also automatically enables the -e option.

Spreadsheet Generation

When generating a spreadsheet in XLSX format, two sheets are created:

  1. Instance Sheet
    • The instance sheet lists all runs for each benchmark instance (rows) and their results for the selected measures, grouped by system/setting (columns).
    • A summary for each run is included, showing the minimum, maximum, and median values for each float measure.
    • Each column with float measures also has a summary, including the sum, average, standard deviation, distance from the minimum, and counts of best, better, worse, and worst values.
  2. Class Sheet
    • The class sheet summarizes all runs for each benchmark class, enabling comparisons between classes.

Info

All summaries are written as formulas in the .xlsx file. The calculated values are also accessible via the content attribute of the Sheet object.

Both the XLSX representation and the actual content are stored in pandas DataFrames for easier handling and future modifications.