Skip to content

Converting Benchmark Results to Spreadsheets

The bconv tool allows you to convert results generated by beval into an ODS spreadsheet, which can be opened with LibreOffice, OpenOffice, or Excel.

To convert your benchmark results to a spreadsheet, use the following command:

bconv benchmark-results.xml -m "time:t,choices" -o results.ods

The -m option specifies which measures to include in the table. Available measures depend on the result parser used during evaluation. Each measure can optionally include a formatting argument after a :. Currently, the supported formatting options are t and to. Both highlight best and worst values for float measures. Use t for most measures, and to for float measures representing booleans, such as timeout.

Spreadsheet Generation

When generating a spreadsheet in ODS format, two sheets are created:

  1. Instance Sheet
    • The instance sheet lists all runs for each benchmark instance (rows) and their results for the selected measures, grouped by system/setting (columns).
    • A summary for each run is included, showing the minimum, maximum, and median values for each float measure.
    • Each column with float measures also has a summary, including the sum, average, standard deviation, distance from the minimum, and counts of best, better, worse, and worst values.
  2. Class Sheet
    • The class sheet summarizes all runs for each benchmark class, enabling comparisons between classes.

Info

All summaries are written as formulas in the ODS file. The calculated values are also accessible via the content attribute of the Sheet object.

Both the ODS representation and the actual content are stored in pandas DataFrames for easier handling and future modifications.