Experiments

Experiments are the base concept how the tool suite and BenchBuild execute research experiments, e.g., measuring performance or analyzing a software project. They are designed to make the execution of research experiments easy and reproducable by separating experiment specific steps from project specific ones. For example, how a specific project is compiled is the responsibilitty of the project writer, where how the project is evaluated during a research experiment is the Experiment’s task.

Note

Details on how to run experiments can be found here.

Designing a new Experiment is also quite simple.

• First, create a new python module in the experiments directory and add an experiment class which inherits from benchbuild.experiment.Experiment. If VaRA-TS should provide automatic support for analyzing different versions, i.e., different revisions of a git based project, use VersionExperiment as base class.

• Second, define two static variables for your experiment: NAME and REPORT_TYPE

• Next, override the actions_for_project method. This method should assign run-time/compile-time extensions and specify the list of actions that should be performed. Each action the experiment does is called a Step and will be executed by BenchBuild in order.

• Last, add your experiment for testing to the BenchBuild config file vara-root/benchbuild/.benchbuild.yml under plugins/experiments/value. After testing, integrate them into the tool suite by adding it to the experiment list in varats.tools.bb_config.generate_benchbuild_config, so it will be automatically added to the BenchBuild config in the future.

Note

For more information about Experiment’s consider reading the BenchBuild docs.

If you’re looking for a simple example experiment, consider taking a look at JustCompileReport.

• Tool suite provided experiments

• Experiment utilities

Tool suite provided experiments

• Experiment: JustCompile
• Experiment: GenerateBlameReport

Experiment utilities

WLLVM module

Module to provide WLLVM support for project compilation and extracting bc files from generated binaries.

WLLVM is a compiler replacement/hook to compile projects with clang, producing LLVM-IR files on the side. This allows us to hook into the build process and to add additional passes/flags, without modifying build files, and later use the generated bc files with LLVM.

class varats.experiment.wllvm.BCFileExtensions(value)

Bases: enum.Enum

List of possible extensions that specify the way a BC file was created.

An extension should be requested when a BC file needs to fulfill certain requirements, e.g., was compiled with debug metadata or compiled with optimizations.

value: str

DEBUG = 'dbg'

NO_OPT = 'O0'

OPT = 'O2'

TBAA = 'TBAA'

class varats.experiment.wllvm.RunWLLVM(*extensions, config=None, **kwargs)

Bases: benchbuild.extensions.base.Extension

This extension implements the WLLVM compiler.

This class is an extension that implements the WLLVM compiler with the required flags LLVM_COMPILER=clang and LLVM_OUTPUFILE=<path>. This compiler is used to transfer the complete project into LLVM-IR.

class varats.experiment.wllvm.Extract(project, bc_file_extensions=None, handler=None)

Bases: benchbuild.utils.actions.Step

Extract step to extract a llvm bitcode file(.bc) from the project.

NAME: ClassVar[str] = 'EXTRACT'

DESCRIPTION: ClassVar[str] = 'Extract bitcode out of the execution file.'

BC_CACHE_FOLDER_TEMPLATE = '{cache_dir}/{project_name}/'

static get_bc_file_name(project_name, binary_name, project_version, bc_file_extensions=None)

Parses parameter information into a filename template to name a bitcode file.

Return type

str

extract()

This step extracts the bitcode of the executable of the project into one file.

Return type

StepResult

varats.experiment.wllvm.project_bc_files_in_cache(project, required_bc_file_extensions)

Checks if all bc files, corresponding to the projects binaries, are in the cache.

Parameters

• project (Project) – the project

• required_bc_file_extensions (Optional[List[BCFileExtensions]]) – list of required file extensions

Returns: True, if all BC files are present, False otherwise.

Return type

bool

varats.experiment.wllvm.get_bc_cache_actions(project, bc_file_extensions=None, extraction_error_handler=None, bc_action_creator=<function _create_default_bc_file_creation_actions>)

Builds the action pipeline, if needed, to fill the BC file cache that provides BC files for the compiled binaries of a project.

Parameters

• project (Project) – the project to compile

• bc_file_extensions (Optional[List[BCFileExtensions]]) – list of bc file extensions

• extraction_error_handler (Optional[PEErrorHandler]) – error handler to report errors during the extraction step

• bc_action_creator (Callable[[Project, List[BCFileExtensions], Optional[PEErrorHandler]], List[Step]]) – alternative BC cache actions creation callback

Returns: required actions to populate the BC cache

Return type

List[Step]

varats.experiment.wllvm.get_cached_bc_file_path(project, binary, required_bc_file_extensions=None)

Look up the path to a BC file from the BC cache.

Parameters

• project (Project) – the project

• binary (ProjectBinaryWrapper) – which corresponds to the BC file

• required_bc_file_extensions (Optional[List[BCFileExtensions]]) – list of required file extensions

Returns: path to the cached BC file

Return type

Path

---

Experiment utilities module

Utility module for BenchBuild experiments.

class varats.experiment.experiment_util.PEErrorHandler(result_folder, error_file_name, timeout_duration=None, delete_files=None)

Bases: object

Error handler for process execution errors.

class varats.experiment.experiment_util.FunctionPEErrorWrapper(func, handler)

Bases: object

Wrap a function call with an exception handler.

Parameters

• func (Callable[..., Any]) – function to be executed

• handler (PEErrorHandler) – function to handle exception

varats.experiment.experiment_util.exec_func_with_pe_error_handler(func, handler)

Execute a function call with an exception handler.

Parameters

• func (Callable[..., Any]) – function to be executed

• handler (PEErrorHandler) – function to handle exception

Return type

None

varats.experiment.experiment_util.get_default_compile_error_wrapped(project, report_type, result_folder_template)

Setup the default project compile function with an error handler.

Parameters

• project (Project) – that will be compiled

• report_type (Type[BaseReport]) – that should be generated

• result_folder_template (str) – where the results will be placed

Return type

FunctionPEErrorWrapper

Returns

project compilation function, wrapped with automatic error handling

varats.experiment.experiment_util.create_default_compiler_error_handler(project, report_type, output_folder=None, binary=None)

Create a default PEErrorHandler for compile errors, based on the project, report_type.

Parameters

• project (Project) – currently under analysis

• report_type (Type[BaseReport]) – that should be generated

• output_folder (Optional[Path]) – where the errors will be placed

• binary (Optional[ProjectBinaryWrapper]) – if only a specific binary is handled

Retruns: a initialized PEErrorHandler

Return type

PEErrorHandler

varats.experiment.experiment_util.create_default_analysis_failure_handler(project, report_type, output_folder=None, binary=None, timeout_duration=None)

Create a default PEErrorHandler for analysis failures, based on the project, report_type.

Parameters

• project (Project) – currently under analysis

• report_type (Type[BaseReport]) – that should be generated

• output_folder (Optional[Path]) – where the errors will be placed

• binary (Optional[ProjectBinaryWrapper]) – if only a specific binary is handled

• timeout_duration (Optional[str]) – set timeout

Retruns: a initialized PEErrorHandler

Return type

PEErrorHandler

varats.experiment.experiment_util.create_default_error_handler(project, report_type, error_type, output_folder=None, binary=None, timeout_duration=None)

Create a default PEErrorHandler based on the project, report_type.

Parameters

• project (Project) – currently under analysis

• report_type (Type[BaseReport]) – that should be generated

• error_type (FileStatusExtension) – a FSE describing the problem type

• output_folder (Optional[Path]) – where the errors will be placed

• timeout_duration (Optional[str]) – set timeout

• binary (Optional[ProjectBinaryWrapper]) – if only a specific binary is handled

Retruns: a initialized PEErrorHandler

Return type

PEErrorHandler

varats.experiment.experiment_util.wrap_unlimit_stack_size(cmd)

Wraps a command with prlimit to be executed with max stack size, i.e., setting the soft limit to the hard limit.

Parameters

cmd (Callable[..., Any]) – command that should be executed with max stack size

Returns: wrapped command

Return type

Any

class varats.experiment.experiment_util.VersionExperiment(name=NOTHING, projects=NOTHING, id=NOTHING, schema=NOTHING, container=NOTHING)

Bases: benchbuild.experiment.Experiment

Base class for experiments that want to analyze different project revisions.

abstract actions_for_project(project)

Get the actions a project wants to run.

Return type

List[Step]

sample(prj_cls)

Adapt version sampling process if needed, otherwise fallback to default implementation.

Parameters

prj_cls (Type[Project]) – project class

Return type

List[Dict[str, Variant]]

Returns

list of sampled versions

name: str

projects: List[Type[benchbuild.project.Project]]

container: benchbuild.environments.domain.declarative.ContainerImage