Runscript¶
A runscript defines all aspects of a benchmark set. The following sections will go through all parts of a typical runscript and show what is possible, so you can adapt an existing runscript to your needs or create a new one from scratch.
Some example runscripts can be found here
Folder Structure¶
When running bgen a folder structure is created. The name of the first folder in the tree is given by the root element in the runscript:
<runscript output="output-folder">
...
</runscript>
In this case, the name of the folder is "output-folder". It is recommended that the name should be changed to something meaningful and representative of the benchmark you are running.
The name of the folder second in the hierarchy represents the project, which will be explained later.
The folder third in the hierarchy corresponds to the name of the machine the benchmark was run on. A machine is defined in the runscript as seen below:
<machine name="hpc" cpu="24x8xE5520@2.27GHz" memory="24GB"/>
Info
The cpu and memory attributes are currently unused.
Any number of machines can be defined.
In this folder you will find the start scripts to start the benchmark and the results afterwards. The results are structured as follows:
./results/<benchmark-name>/<system-name>-<system-version>-<setting>/<benchclass>/<instance>/<run>
Configuration¶
At the moment a configuration is only used to reference a run template. Any number of configurations can be defined as below:
<conifg name="seq-generic" template="templates/seq-generic.sh"/>
For more information regarding run templates check here.
System¶
The following line shows, how a system is defined.
<system name="clingo" version="5.8.0" measures="clingo" config="seq-generic" cmdline="--stats">
...
</system>
The values of name and version work together to describe the name of a bash script named <name>-<version>, which has to be placed inside the ./programs directory. In the case of the example above, the solver given to the run template described in the configuration is ./programs/clingo-4.5.4. Since this is a regular bash script, feel free to modify it to your heart's content.
The measure attribute describes a name of a result parser, which will be used to evaluate the results during the beval call. This has no effect on bgen.
The config attribute refers to the name of a configuration, which should be used to run this system.
The cmdline attribute is optional and can be any string, which will be passed to the system regardless of the setting.
A runscript can contain any number of systems, which in turn can contain any number of settings.
Setting¶
Settings are identified by their name and define additional arguments and encodings used by the system.
<setting name="setting-1" cmdline="--quiet=1,0" tag="basic">
<encoding file="encodings/default.lp"/>
<encoding file="extra.lp" tag="extra"/>
</setting>
The tag attribute is another identifier, which is only valid inside this runscript and is used to select multiple setting at once.
Each setting can also contain any number of encoding elements. The file attribute is a relative path from the directory, where bgen is run, to the encoding. If no tag is defined the encoding is passed to the system via the run template for all instances when this setting is selected. A tag can be provided to make the encoding usage instance dependant. Multiple encodings can be selected by using the same tag.
The setting element additionaly supports an optional attribute disttemplate. The default value is "templates/single.dist" which is a reference to single.dist. This attribute is only relevant for distjobs. More information to dist templates can be found here.
Job¶
A job is used to define additional arguments for individual runs. Any number of jobs can be defined. There are two types of jobs, 'default' sequential seqjobs and distjobs for running the benchmarks on a cluster.
<seqjob name="seq-gen" timeout="900" runs="1" parallel="1"/>
<distjob name="dist-gen" timeout="900" runs="1" script_mode="timeout" walltime="23:59:59" cpt="4"/>
The script_mode attribute, defines how runs are grouped and dispatched to the cluster. 'multi' dispatches all runs individually to the cluster for maximum possible parallelization. 'timeout' dispatches groups of runs depending on the set timeout and walltime. If it is not possible to execute all runs sequentially inside the walltime specified, they will be split into as many groups as necessary so that the walltime is never surpassed. For example, if the walltime is 25 hours and you have 100 instances with a timeout of 1 hour each and 1 run each, there will be 4 groups, each with 25 runs, which will be dispatched.
Info
If you have a lot of runs *script_mode* 'multi' can cause issues with the cluster scheduler. Either use 'timeout' or dispatch the generated `.dist` jobs using the `./dispatcher.py`.
A last optional attribute for distjobs is partition, which is the name of the partition used on the cluster. The default value is 'kr'. Other values for this argument can be "short" and "long". If short is used the walltime can not exceed 24 hours.
Benchmark/Instances¶
The benchmark element is mainly used to indicate the location of the instances. Any number of benchmarks can be defined.
<benchmark name="no-pigeons">
...
<benchmark/>
A benchmark is identified by its name and can contain any number of folder or files elements.
<folder path="benchmarks/clasp" enctag="tag1">
<ignore prefix="pigeons"/>
<encoding file="encodings/no-pigeons.lp"/>
<folder/>
It is also possible to specify any amount of encodings which should be called together with all instances in this folder by using the encoding element. Setting-depended encodings can be added by using the optional enctag attribute. A more detailed explanation with examples for encoding support can be found here.
<files path="benchmarks/clasp" enctag="tag1 tag2">
<encoding file="default.lp"/>
<add file="pigeons/pigeonhole11-unsat.lp"/>
</files>
The files element does also support instance- and setting-depended encodings. More information can be found here.
Projects¶
Projects are used to combine all of the previous elements to define a complete benchmark. There are two ways to define projects, using the runtag or the runspec element. A project can contain any number of runtag and runspec elements.
<project name="clingo-basic" job="seq-gen">
<runtag machine="hpc" benchmark="no-pigeons" tag="basic"/>
</project>
Each project is identified by its name. The value of name is also what gives the name of the seconds folder in the overall folder structure. job references a previously defined job to be used as a template for the benchmark.
The runtag element specifies a machine and benchmark (group of instances) to be run. One or multiple settings to be used can be selected by the tag. The special tag all can be used to select all settings. The system to be used is inferred through the setting.
In the above example all instances defined in the 'no-pigeons' benchmark are run using the 'seq-gen' job configuration on machine 'hpc' once for each setting with the tag 'basic'.
<project name="clingo-dist" job="dist-gen">
<runspec machine="hpc" benchmark="no-pigeons" system="clingo" version="5.8.0" setting="setting-1"/>
</project>
Similar to the runtag element, the runspec element can be used to specify a machine and a benchmark. But in contrast to before, only a single system version with a single setting can be referenced.