Result Classes

Created on Jan 19, 2010

@author: Roland Kaminski

Benchmark dataclass

Represents a benchmark, i.e., a set of instances.

Attributes:

Name	Type	Description
name	str	The name of the benchmark.
classes	dict[int, Class]	Benchmark classes in this benchmark.

__iter__() -> Iterator[Class]

Creates an iterator over all benchmark classes.

BenchmarkMerge

Represents an (ordered) set of benchmark sets.

__init__(benchmarks: set[Benchmark])

Initializes using the given set of benchmarks.

Attributes:

Name	Type	Description
benchmarks	set[Benchmark]	Benchmarks to merge.

__iter__() -> Iterator[Class]

Creates an interator over all benchmark classes in all benchmarks.

Class dataclass

Represents a benchmark class.

Attributes:

Name	Type	Description
benchmark	Benchmark	The benchmark associaed with this class.
name	str	The name of the benchmark.
id	int	A unique id (in the scope of the benchmark).
instances	dict[int, Instance]	Instances belonging to this benchmark class.
values	dict[str, Any]	Mutable dict with helper values.

__iter__() -> Iterator[Instance]

Creates an iterator over all instances in the benchmark class.

__post_init__() -> None

Initialize mutable helper variables.

ClassResult dataclass

Represents the results of all instances of a benchmark class.

Attributes:

Name	Type	Description
benchclass	Class	The benchmark class for the results.
instresults	list[InstanceResult]	Results of instances belonging to the benchmark class.

__iter__() -> Iterator[InstanceResult]

Creates an iterator over all the individual results per instance.

Config dataclass

Represents a config.

Attributes:

Name	Type	Description
name	str	The name of the config.
template	str	A path to the template file.

DistJob dataclass

Bases: Job

Represents a dist job.

Attributes:

Name	Type	Description
name	str	The name of the job.
timeout	int	Timeout of the job.
runs	int	Number of repetitions per instance.
attrib	dict[str, Any]	Arbitrary extra arguments.
script_mode	str	Specifies the script generation mode.
walltime	str	The walltime for a distributed job.

Instance dataclass

Represents a benchmark instance.

Attributes:

Name	Type	Description
benchclass	Class	The class of the instance.
name	str	The name of the benchmark.
id	int	A unique id (in the scope of the benchmark).
max_runs	int	Max number of runs.
values	dict[str, Any]	Mutable dict with helper values.

__post_init__() -> None

Initialize mutable helper variables.

InstanceResult dataclass

Represents the result of an individual instance (with possibly multiple runs).

Attributes:

Name	Type	Description
instance	Instance	The instance for the results.
runs	list[Run]	Results of runs belonging to the instance.

__iter__() -> Iterator[Run]

Creates an iterator over the result of all runs.

Job dataclass

Represents a job.

Attributes:

Name	Type	Description
name	str	The name of the job.
timeout	int	Timeout of the job.
runs	int	Number of repetitions per instance.
attr	dict[str, Any]	Arbitrary extra arguments.

Machine dataclass

Represents a machine.

Attributes:

Name	Type	Description
name	str	The name of the machine.
cpu	str	String describing the CPU.
memory	str	String describing the Memory.

Project dataclass

Describes a project, i.e, a collection of run specifications.

Attributes:

Name	Type	Description
name	str	The name of the project.
job	str	The name of the associated job.
runspecs	list[Runspec]	Run specifications of the project.

__iter__() -> Iterator[Runspec]

Creates an iterator over all run specification in the project.

Result

Stores the benchmark description and its results.

__init__() -> None

Initializes an empty result.

gen_office(out: str, sel_projects: str, measures: list[tuple[str, Any]]) -> None

Prints the current result in open office spreadsheet format.

Attributes:

Name	Type	Description
out	str	The output file to write to.
sel_projects	str	The selected projects ("" for all).
measures	list[tuple[str, Any]]	The measures to extract.

merge(projects: list[Project]) -> BenchmarkMerge

Concatenates the benchmarks in the given projects into one benchmark set.

Attributes:

Name	Type	Description
projects	list[Project]	The projects to merge with.

Run dataclass

Represents the result of an individual run of a benchmark instance.

Attributes:

Name	Type	Description
instresult	InstanceResult	The associated instance result.
number	int	The number of the run.
measures	dict[str, tuple[str, str]]	Concrete measurements.

iter(measures: list[tuple[str, Any]]) -> Iterator[tuple[str, str, str]]

Creates an iterator over all measures captured during the run. Measures can be filter by giving a string set of measure names. If this string set is empty, instead all measures sorted by their keys will be returned.

Attributes:

Name	Type	Description
measures	list[tuple[str, Any]]	Selected measures.

Runspec dataclass

Describes a run specification, i.e, how to run individual systems on a set of instances.

Attributes:

Name	Type	Description
system	System	The system to evaluate.
machine	Machine	The machine to run on.
benchmark	Benchmark	The benchmark set to evaluate.
setting	Setting	The setting to run with.
classresults	list[ClassResult]	The benchmark results.

__iter__() -> Iterator[ClassResult]

Creates an iterator over all results (grouped by benchmark class.)

SeqJob dataclass

Bases: Job

Represents a sequential job.

Attributes:

Name	Type	Description
name	str	The name of the job.
timeout	int	Timeout of the job.
runs	int	Number of repetitions per instance.
attrib	dict[str, Any]	Arbitrary extra arguments.
parallel	int	Number of processes to start in parallel.

Setting dataclass

Represents a setting.

Attributes:

Name	Type	Description
system	System	The system associated with the setting.
name	str	The name of the setting.
cmdline	str	Command line parameters.
tag	str	Tags of the setting.
order	int	An integer denoting the occurrence in the XML file.
attr	dict[str, Any]	Arbitrary extra arguments.

System dataclass

Represents a system.

Attributes:

Name	Type	Description
name	str	The name of the system.
version	str	The version.
config	str	The config (a string).
measures	str	The measurement function (a string).
order	int	An integer denoting the occurrence in the XML file.
settings	dict[str, Setting]	Dictionary of all system settings.