Experiments

Experiments are the base concept how the tool suite and BenchBuild execute research experiments, e.g., measuring performance or analyzing a software project. They are designed to make the execution of research experiments easy and reproducable by separating experiment specific steps from project specific ones. For example, how a specific project is compiled is the responsibility of the project writer, where how the project is evaluated during a research experiment is the Experiment’s task.

Note

Details on how to run experiments can be found here.

How to add a new experiment to VaRA-TS

Designing a new Experiment is also quite simple.

  • First, create a new python module in the experiments directory and add an experiment class which inherits from benchbuild.experiment.Experiment. If VaRA-TS should provide automatic support for analyzing different versions, i.e., different revisions of a git based project, use VersionExperiment as base class.

  • Second, define two static variables for your experiment: NAME and REPORT_SPEC

  • Third, your experiment needs to pass an additional shorthand parameter.

  • Next, override the actions_for_project method. This method should assign run-time/compile-time extensions and specify the list of actions that should be performed. Each action the experiment does is called a Step and will be executed by BenchBuild in order.

  • Last, add your experiment for testing to the BenchBuild config file vara-root/benchbuild/.benchbuild.yml under plugins/experiments/value. After testing, integrate them into the tool suite by adding it to the experiment list in varats.tools.bb_config.generate_benchbuild_config, so it will be automatically added to the BenchBuild config in the future.

Note

For more information about Experiment’s consider reading the BenchBuild docs.

If you’re looking for a simple example experiment, consider taking a look at JustCompileReport.

Tool suite provided experiments

Experiment utilities

WLLVM module

Module to provide WLLVM/GLLVM support for project compilation and extracting bc files from generated binaries.

WLLVM/GLLVM is a compiler replacement/hook to compile projects with clang, producing LLVM-IR files on the side. This allows us to hook into the build process and to add additional passes/flags, without modifying build files, and later use the generated bc files with LLVM.

class varats.experiment.wllvm.BCFileExtensions(*values)[source]

Bases: Enum

List of possible extensions that specify the way a BC file was created.

An extension should be requested when a BC file needs to fulfill certain requirements, e.g., was compiled with debug metadata or compiled with optimizations.

value: str
DEBUG = 'dbg'
NO_OPT = 'O0'
OPT = 'O2'
TBAA = 'TBAA'
FEATURE = 'feature'
BLAME = 'blame'
BLAME_AST = 'blame_ast'
class varats.experiment.wllvm.RunWLLVM(*extensions, config=None, distcc_hosts=None, **kwargs)[source]

Bases: Extension

This extension implements the WLLVM/GLLVM compiler. GLLVM is used automatically if it is found in the PATH. Otherwise WLLVM is used.

This class is an extension that implements the WLLVM/GLLVM compiler with the required flags LLVM_COMPILER=clang and LLVM_OUTPUFILE=<path>. This compiler is used to transfer the complete project into LLVM-IR.

The distcc_hosts argument will instruct WLLVM/GLLVM to call distcc instead of clang directly. For possible values of distcc_hosts have a look at the “HOST SPECIFICATIONS” in the distcc man page. It is the user’s responsibility to ensure that all distcc hosts are configured to use exactly the same compiler as localhost!

Examples:

  • Usage without distcc: RunWLLVM()

  • Usage with distcc enabled: RunWLLVM(distcc_hosts="localhost 192.168.1.1:3634/64")

class varats.experiment.wllvm.Extract(project, bc_file_extensions=None, handler=None)[source]

Bases: ProjectStep

Extract step to extract a llvm bitcode file(.bc) from the project.

NAME = 'EXTRACT'
DESCRIPTION = 'Extract bitcode out of the execution file.'
BC_CACHE_FOLDER_TEMPLATE = '{cache_dir}/{project_name}/'
project: VProject
static get_bc_file_name(project_name, binary_name, project_version, bc_file_extensions=None)[source]

Parses parameter information into a filename template to name a bitcode file.

Return type:

str

extract()[source]

This step extracts the bitcode of the executable of the project into one file.

Return type:

StepResult

varats.experiment.wllvm.project_bc_files_in_cache(project, required_bc_file_extensions)[source]

Checks if all bc files, corresponding to the projects binaries, are in the cache.

Parameters:

  • project (Project) – the project

  • required_bc_file_extensions (List[BCFileExtensions] | None) – list of required file extensions

Return type:

bool

Returns: True, if all BC files are present, False otherwise.

varats.experiment.wllvm.get_bc_cache_actions(project, bc_file_extensions=None, extraction_error_handler=None, bc_action_creator=<function _create_default_bc_file_creation_actions>)[source]

Builds the action pipeline, if needed, to fill the BC file cache that provides BC files for the compiled binaries of a project.

Parameters:

  • project (Project) – the project to compile

  • bc_file_extensions (List[BCFileExtensions] | None) – list of bc file extensions

  • extraction_error_handler (PEErrorHandler | None) – error handler to report errors during the extraction step

  • bc_action_creator (Callable[[Project, List[BCFileExtensions], PEErrorHandler | None], List[Step]]) – alternative BC cache actions creation callback

Return type:

List[Step]

Returns: required actions to populate the BC cache

varats.experiment.wllvm.get_cached_bc_file_path(project, binary, required_bc_file_extensions=None)[source]

Look up the path to a BC file from the BC cache.

Parameters:

  • project (Project) – the project

  • binary (ProjectBinaryWrapper) – which corresponds to the BC file

  • required_bc_file_extensions (List[BCFileExtensions] | None) – list of required file extensions

Return type:

Path

Returns: path to the cached BC file

varats.experiment.wllvm.is_gllvm_available()[source]
Return type:

bool

Experiment utilities module

Utility module for BenchBuild experiments.

varats.experiment.experiment_util.get_varats_result_folder(project)[source]

Get the project specific path to the varats result folder.

Parameters:

project (Project) – to lookup the result folder for

Return type:

Path

Returns:

path to the project specific result folder

class varats.experiment.experiment_util.PEErrorHandler(result_folder, error_file_name, timeout_duration=None, delete_files=None)[source]

Bases: object

Error handler for process execution errors.

class varats.experiment.experiment_util.FunctionPEErrorWrapper(func, handler)[source]

Bases: object

Wrap a function call with an exception handler.

Parameters:

  • func (Callable[..., Any]) – function to be executed

  • handler (PEErrorHandler) – function to handle exception

varats.experiment.experiment_util.exec_func_with_pe_error_handler(func, handler)[source]

Execute a function call with an exception handler.

Parameters:

  • func (Callable[..., Any]) – function to be executed

  • handler (PEErrorHandler) – function to handle exception

Return type:

None

varats.experiment.experiment_util.get_default_compile_error_wrapped(experiment_handle, project, report_type)[source]

Setup the default project compile function with an error handler.

Parameters:

  • experiment_handle (ExperimentHandle) – handle to the current experiment

  • project (Project) – that will be compiled

  • report_type (Type[BaseReport]) – that should be generated

Return type:

FunctionPEErrorWrapper

Returns:

project compilation function, wrapped with automatic error handling

varats.experiment.experiment_util.create_default_compiler_error_handler(experiment_handle, project, report_type, binary=None)[source]

Create a default PEErrorHandler for compile errors, based on the project, report_type.

Parameters:

  • experiment_handle (ExperimentHandle) – handle to the current experiment

  • project (Project) – currently under analysis

  • report_type (Type[BaseReport]) – that should be generated

  • binary (ProjectBinaryWrapper | None) – if only a specific binary is handled

Return type:

PEErrorHandler

Retruns: a initialized PEErrorHandler

varats.experiment.experiment_util.create_default_analysis_failure_handler(experiment_handle, project, report_type, binary=None, timeout_duration=None)[source]

Create a default PEErrorHandler for analysis failures, based on the project, report_type.

Parameters:

  • experiment_handle (ExperimentHandle) – handle to the current experiment

  • project (Project) – currently under analysis

  • report_type (Type[BaseReport]) – that should be generated

  • binary (ProjectBinaryWrapper | None) – if only a specific binary is handled

  • timeout_duration (str | None) – set timeout

Return type:

PEErrorHandler

Retruns: a initialized PEErrorHandler

varats.experiment.experiment_util.create_default_error_handler(experiment_handle, project, report_type, error_type, binary=None, timeout_duration=None)[source]

Create a default PEErrorHandler based on the project, report_type.

Parameters:

  • experiment_handle (ExperimentHandle) – handle to the current experiment

  • project (Project) – currently under analysis

  • report_type (Type[BaseReport]) – that should be generated

  • error_type (FileStatusExtension) – a FSE describing the problem type

  • timeout_duration (str | None) – set timeout

  • binary (ProjectBinaryWrapper | None) – if only a specific binary is handled

Return type:

PEErrorHandler

Retruns: a initialized PEErrorHandler

varats.experiment.experiment_util.wrap_unlimit_stack_size(cmd)[source]

Wraps a command with prlimit to be executed with max stack size, i.e., setting the soft limit to the hard limit.

Parameters:

cmd (Callable[..., Any]) – command that should be executed with max stack size

Return type:

Any

Returns: wrapped command

class varats.experiment.experiment_util.WithUnlimitedStackSize(*extensions, **kwargs)[source]

Bases: Extension

Sets the stack size of the wrapped command to unlimited (16GB).

class varats.experiment.experiment_util.ExperimentHandle(experiment)[source]

Bases: object

Handle to an experiment that provides helper interfaces for analysis steps to utilize experiment specific data.

get_file_name(report_shorthand, project_name, binary_name, project_revision, project_uuid, extension_type, config_id=None)[source]

Generates a filename for a report file that is generated by the experiment.

Parameters:

  • report_shorthand (str) – unique shorthand for the report

  • project_name (str) – name of the project for which the report was generated

  • binary_name (str) – name of the binary for which the report was generated

  • project_revision (ShortCommitHash) – revision (commit hash)of the analyzed project

  • project_uuid (str) – benchbuild uuid for the experiment run

  • extension_type (FileStatusExtension) – to specify the status of the generated report

Return type:

ReportFilename

Returns:

name for the report file that can later be uniquly identified

report_spec()[source]

Experiment report specification.

Return type:

ReportSpecification

class varats.experiment.experiment_util.VersionExperiment(name=NOTHING, projects=NOTHING, id=NOTHING, schema=NOTHING, container=NOTHING)[source]

Bases: Experiment

Base class for experiments that want to analyze different project revisions.

REPORT_SPEC: ReportSpecification
SHORTHAND: str
classmethod shorthand()[source]

Experiment shorthand.

Return type:

str

classmethod report_spec()[source]

Experiment report specification.

Return type:

ReportSpecification

classmethod file_belongs_to_experiment(file_name)[source]

Checks if the file belongs to this experiment.

Parameters:

file_name (str) – name of the file to check

Return type:

bool

Returns:

True, if the file belongs to this experiment type

get_handle()[source]
Return type:

ExperimentHandle

abstractmethod actions_for_project(project)[source]

Get the actions a project wants to run.

Return type:

MutableSequence[Step]

classmethod sample(prj_cls)[source]

Adapt version sampling process if needed, otherwise fallback to default implementation.

Parameters:

prj_cls (Type[Project]) – project class

Return type:

List[Revision]

Returns:

list of sampled versions

name: str
projects: List[Type[Project]]
container: ContainerImage
class varats.experiment.experiment_util.ZippedReportFolder(result_report_path)[source]

Bases: TemporaryDirectory[str]

Context manager for creating a folder report, i.e., a report file which is actually a folder containing multiple files and other folders.

Example usage: An experiment step can, with this context manager, simply create a folder into which all kinds of data is dropped into. After the completion of the step (leaving the context manager), all files dropped into the folder will be compressed and stored as a single report.

exception varats.experiment.experiment_util.WrongStepCall[source]

Bases: Exception

Throw if the common step method was called.

class varats.experiment.experiment_util.OutputFolderStep(project)[source]

Bases: ProjectStep

Special step class that needs an output folder to write to.

abstractmethod call_with_output_folder(tmp_dir)[source]

Actual call implementation that gets a path to tmp_folder.

Return type:

StepResult

class varats.experiment.experiment_util.ZippedExperimentSteps(output_filepath, actions)[source]

Bases: MultiStep[ZippedStepTy]

Runs multiple actions, providing them a shared tmp folder that afterwards is zipped into an archive.

NAME: ClassVar[str] = 'ZippedSteps'
DESCRIPTION: ClassVar[str] = 'Run multiple actions with a shared tmp folder'
varats.experiment.experiment_util.create_new_success_result_filepath(exp_handle, report_type, project, binary, config_id=None)[source]

Create a result filepath for a successfull report of the executed experiment/project combination.

Parameters:

  • exp_handle (ExperimentHandle) – handle to the current experiment

  • report_type (Type[BaseReport]) – type of the report

  • project (VProject) – current project

  • binary (ProjectBinaryWrapper) – current binary

  • config_id (int | None) – optional id to specify the used configuration

Return type:

ReportFilepath

Returns: formatted success filepath

varats.experiment.experiment_util.create_new_failed_result_filepath(exp_handle, report_type, project, binary, config_id=None)[source]

Create a result filepath for a failed report of the executed experiment/project combination.

Parameters:

  • exp_handle (ExperimentHandle) – handle to the current experiment

  • report_type (Type[BaseReport]) – type of the report

  • project (VProject) – current project

  • binary (ProjectBinaryWrapper) – current binary

  • config_id (int | None) – optional id to specify the used configuration

Return type:

ReportFilepath

Returns: formatted fail filepath

varats.experiment.experiment_util.get_config_patch_steps(project)[source]

Get a list of actions that apply all configuration patches to the project.

Parameters:

project (VProject) – the project to be configured

Return type:

MutableSequence[Step]

Returns:

the actions that configure the project