Developing with VaRA-TS

First, take a look at our install guide to learn how to set up the tool suite.

The VaRA-TS API Reference contains information about how to work with the tool-suite, as well as how to add your own research tools, Experiments, Reports, Plots, and more.

Debugging with Visual Studio Code shows an example on how to debug benchbuild projects and experiments using the VSCode debugger.

For further information about benchbuild related concepts, like Experiments or Projects, take a look at the benchbuild documentation.

Pre-Commit

We use pre-commit to automatically enforce our code-style guidelines. To activate these checks, you first have to install pre-commit and its hooks:

pip install pre-commit
pre-commit install

Afterwards, the checks will run every time you make a git commit. The same checks also run in our CI and pull requests are required to pass them before merging. It is also possible to run the checks manually:

pre-commit run --all-files

Testing

Tests for the VaRA tool suite are located in the tests directory. The tests are run using pytest but most of the tests are written using unittest syntax.

Mocking the vara configuration

To ensure a flawless test execution, it is important that the order of test execution does not matter, i.e. each single test is idempotent and does not modify the environment. One common problem with this are the vara and benchbuild configurations, as they are stored in a single global object. We therefore provide a helper function replace_config() to set a test-specific vara (and if needed benchbuild) configuration that can be safely modified without affecting other tests. This function can be either used as a decorator similar to unittest.mock or as a context manager.

Test resources

Test resources, like report files or case studies, can be stored in the tests/TEST_INPUTS directory. This directory follows the same structure one would also use for the tool-suites installation environment. The path to this directory can be accessed via the TEST_INPUTS_DIR attribute. If using the replace_config() wrapper or context manager without providing an own configuration, the relevant paths in the replaced configuration object are already pointed to this environment. Keep the data put in this directory as small as possible to avoid bloating the repository.

Warning

Make sure that your tests do not write to this directory to maintain an idempotent test environment. If you really need to write files, use temporary directories instead.

---

Module: helper_utils

Module for test utility functions.

class tests.helper_utils.UnitTestFixture(*args, **kwargs)

Bases: Protocol

A test fixture that can be used with a TestEnvironment.

copy_to_env(path)

Called when entering the test environment.

The new configs are already in place and the cwd is the tmp dir.

Return type:

None

cleanup()

Called when exiting the test environment.

The old configs are in place again, but the cwd is still the tmp dir.

Return type:

None

class tests.helper_utils.FileFixture(src, dst)

Bases: UnitTestFixture

A file or directory that is copied into the test environment.

copy_to_env(path)

Called when entering the test environment.

The new configs are already in place and the cwd is the tmp dir.

Return type:

None

cleanup()

Called when exiting the test environment.

The old configs are in place again, but the cwd is still the tmp dir.

Return type:

None

class tests.helper_utils.RepoFixture(source)

Bases: UnitTestFixture

A git repository that is cloned into the test environment.

The clone uses a local reference to avoid unnecessary traffic.

copy_to_env(path)

Called when entering the test environment.

The new configs are already in place and the cwd is the tmp dir.

Return type:

None

cleanup()

Called when exiting the test environment.

The old configs are in place again, but the cwd is still the tmp dir.

Return type:

None

class tests.helper_utils.UnitTestFixtures

Bases: object

Collection/factory for test fixtures.

PAPER_CONFIGS = <tests.helper_utils.FileFixture object>

RESULT_FILES = <tests.helper_utils.FileFixture object>

PLOTS = <tests.helper_utils.FileFixture object>

TABLES = <tests.helper_utils.FileFixture object>

ARTEFACTS = <tests.helper_utils.FileFixture object>

TEST_PROJECTS = <tests.helper_utils.RepoFixture object>

static create_file_fixture(src, dst)

Creates a file fixture.

Return type:

UnitTestFixture

static create_project_repo_fixture(project)

Creates a repo fixture for the main source of a project.

Return type:

RepoFixture

class tests.helper_utils.TestEnvironment(required_test_inputs)

Bases: object

Test environment implementation.

The wrapped test is run inside a temporary directory that acts as the varats root folder with a fresh default varats and BenchBuild config. The configurations can be accessed via the usual vara_cfg() and bb_cfg() getters.

Parameters:

required_test_inputs (Iterable[UnitTestFixture]) – test inputs to be copied into the test environment

tests.helper_utils.run_in_test_environment(*required_test_inputs)

Run a test in an isolated test environment.

The wrapped test is run inside a temporary directory that acts as the varats root folder with a fresh default varats and BenchBuild config. The configurations can be accessed via the usual vara_cfg() and bb_cfg() getters.

Parameters:

required_test_inputs (UnitTestFixture) – test inputs to be copied into the test environment

Return type:

Callable[..., Any]

Returns:

the wrapped test function

tests.helper_utils.create_test_environment(*required_test_inputs)

Context manager that creates an isolated test environment.

The wrapped test is run inside a temporary directory that acts as the varats root folder with a fresh default varats and BenchBuild config. The configurations can be accessed via the usual vara_cfg() and bb_cfg() getters.

Parameters:

required_test_inputs (UnitTestFixture) – test inputs to be copied into the test environment

Return type:

TestEnvironment

class tests.helper_utils.DummyGit(remote, local, clone=True, limit=10, refspec='HEAD', shallow=True, version_filter=<function Git.<lambda>>)

Bases: Git

A dummy git source that does nothing.

fetch()

Clone the repository, if needed.

This will create a git clone inside the global cache directory.

Parameters:

version (Optional[str], optional) – [description]. Defaults to None.

Returns:

[description]

Return type:

LocalPath

version(target_dir, version='HEAD')

Create a new git worktree pointing to the requested version.

Parameters:

• target_dir (str) – The filesystem path where the new worktree should live.

• version (str) – The desired version the new worktree needs to point to. Defaults to ‘HEAD’.

Returns:

[description]

Return type:

LocalPath

versions()

List all available versions of this source.

Returns:

The list of all available versions.

Return type:

list[Variant]

class tests.helper_utils.ConfigurationHelper

Bases: object

This class is a helper for various tests.

static create_test_config()

This method creates a test configuration.

Return type:

ConfigurationImpl

class tests.helper_utils.BBTestSource(test_versions, local, remote)

Bases: FetchableSource

Source test fixture class.

test_versions: list[str]

property local: str

property remote: str | dict[str, str]

property default: Variant

The default version for this source.

version(target_dir, version)

Fetch the requested version and place it in the target_dir

Parameters:

• target_dir (str) – The filesystem path where the version should be placed in.

• version (str) – The version that should be fetched from the local cache.

Returns:

[description]

Return type:

LocalPath

versions()

List all available versions of this source.

Returns:

The list of all available versions.

Return type:

Iterable[Variant]

---

Debugging with Visual Studio Code

The .vscode/launch.json file in the VaRA-Tool-Suite repository contains a configuration for Visual Studio Code. With F5 the given example is executed. It runs the command benchbuild run -E JustCompile gzip by default, which can be adapted to debug other projects and experiments by changing the arguments that are passed to benchbuild. To step through the JustCompile experiment a breakpoint has to be set in just_compile.py. F9 can be used to set/unset a breakpoint at the current line.

Releasing VaRA-TS and VaRA related tools/libraries

In general, all VaRA related tools and libraries with the same release should be compatible with each other, e.g., vara-11.1.0 for vara-llvm-project needs to work with vara-tool-suite release vara-11.1.0. We try to prevent breaking changes so that the tool suite works with many different VaRA versions, but sometimes breaking changes between releases are necessary.

Prepare release tooling

Before we can create a release, we need to install and prepare our release tools.

Install:

pip install --user wheel tox

Configure pypi credentials

[distutils]
  index-servers =
    testpypi
    pypi

[pypi]
  repository = https://upload.pypi.org/legacy/
  username = __token__
  password = $PYPI_PASSWORD

[testpypi]
  repository = https://test.pypi.org/legacy/
  username = __token__
  password = $PYPI_PASSWORD

vara-tool-suite

Ensure that all branches are in the correct state.

cd VaRA-Tool-Suite
git checkout vara
git pull
git checkout vara-dev
git pull

[Optional]: Update the version numbers, should a larger bump be needed.

git checkout vara-dev
# Update pyproject.toml to the new version
git commit -m "Bump version to $NEW_VERSION"
git push origin vara-dev

Integrate the changes from develop into the release branch vara.

git checkout vara
git merge vara-dev

Build and upload release files with tox and tag the new release, this automatically builds both namespace packages varats and varats-core.

tox -e release
git tag -s vara-X.Y.Z

git push origin vara
git push origin vara-X.Y.Z

Prepare the next version, by default we assume a small bump.

git checkout vara-dev
# Update varats/setup.py and varats-core/setup.py to the next version
# Update varats/setup.py to depend on the new core version
git commit -m "Bump version to $NEW_VERSION"
git push origin vara-dev

vara-feature

Ensure that all branches are in the correct state.

cd vara-feature
git checkout vara
git pull
git checkout vara-dev
git pull

[Optional]: Update the version numbers, should a larger bump be needed.

git checkout vara-dev
# Update setup.py to the next version
git commit --allow-empty -m "Bump version to $NEW_VERSION"
git push origin vara-dev

Integrate the changes from develop into the release branch vara.

git checkout vara
git merge vara-dev

Build and upload release files with tox and tag the new release.

tox -e release
git tag -s vara-X.Y.Z

git push origin vara
git push origin vara-X.Y.Z

Prepare the next version.

git checkout vara-dev
# Update setup.py to the next version
git commit --allow-empty -m "Bump version to $NEW_VERSION"
git push origin vara-dev

VaRA and vara-llvm-project

Releasing

cd vara-llvm-project
git checkout vara-$DEV_VERSION-dev
git pull
git checkout -b vara-$DEV_VERSION
git push --set-upstream origin vara-$DEV_VERSION
git tag -s vara-X.Y.Z
git push origin vara-X.Y.Z

cd vara
git checkout vara-dev
git pull
git checkout vara
git pull
git merge vara-dev
git tag -s vara-X.Y.Z
git push
git push origin vara-X.Y.Z

Preparing new development branches

Before upgrading to the new LLVM base check build and run tests.

cd vara-llvm-project
git checkout vara-$DEV_VERSION-dev
git checkout -b vara-$DEV_VERSION_NEXT-dev
# fetch upstream llvm changes
git fetch upstream
git rebase --onto $NEXT_LLVM_RELEASE_BASE $PREV_LLVM_RELEASE_BASE vara-$DEV_VERSION_NEXT-dev
# Fix regressions and API changes introduced by the new LLVM version.
# Check if the phasar submodule needs an update.
git push --set-upstream origin vara-$DEV_VERSION_NEXT-dev

Set the new development default in varats to $DEV_VERSION_NEXT.

cd vara-tools-suite
vi varats-core/varats/utils/settings.py
# Set vara:version to `XY` of $DEV_VERSION_NEXT