Skip to content

Converting Benchmark Results to Spreadsheets

The conv subcommand allows you to convert results generated by btool eval into an ODS spreadsheet, which can be opened with LibreOffice, OpenOffice, or Excel.

To convert your benchmark results to a spreadsheet, use the following command:

btool conv benchmark-results.xml -m "time:t,choices" -o results.ods

The name of the resulting .ods file is set using the -o, --output option (default out.ods).

Which benchmark projects to include in the output can be selected via the -p, --project option. By default all projects are selected.

The -m, --measures option specifies which measures to include in the table (default: time:t,timeout:to; -m all selects all measures). Available measures depend on the result parser used during evaluation. Each measure can optionally include a formatting argument after a :. Currently, the supported formatting options are t and to. Both highlight best and worst values for float measures. Use t for most measures, and to for float measures representing booleans, such as timeout.

You can chose to export the instance data to a .parquet file using the -e, --export option. The name of the file will be the same as the specified output, i.e. -o res.ods -e -> res.parquet.

The -j, --jupyter-notebook option can be used to generate a .ipynb file, which contains some basic visualization of the instance data, specifically survivor, cactus and CDF diagrams. The notebook can be started using jupyter notebook <notebook.ipynb> and dependencies for the notebook can be installed using pip install .[plot]. This option also automatically enables the -e option.

Spreadsheet Generation

When generating a spreadsheet in ODS format, two sheets are created:

  1. Instance Sheet
    • The instance sheet lists all runs for each benchmark instance (rows) and their results for the selected measures, grouped by system/setting (columns).
    • A summary for each run is included, showing the minimum, maximum, and median values for each float measure.
    • Each column with float measures also has a summary, including the sum, average, standard deviation, distance from the minimum, and counts of best, better, worse, and worst values.
  2. Class Sheet
    • The class sheet summarizes all runs for each benchmark class, enabling comparisons between classes.

Info

All summaries are written as formulas in the ODS file. The calculated values are also accessible via the content attribute of the Sheet object.

Both the ODS representation and the actual content are stored in pandas DataFrames for easier handling and future modifications.