Projects¶

A project is an abstraction of a software project containing information on how to build and execute that software project. The tool suite already ships with some project, but also also allows yoy to add new projects.

How to add a new project to VaRA-TS
List of supported projects

How to add a new project to VaRA-TS¶

To add a new project, create a new class with benchbuild.Project as its superclass. Place the class in one ot the varats/projects/<group> folders, where <group> refers to the language of the project.

Below, you can see a simplified version of the project Gravity with comments explaining the different parts of the project definition. You can use this as a template for your own project:

import typing as tp

import benchbuild as bb
from benchbuild.utils.cmd import make
from benchbuild.utils.settings import get_number_of_jobs
from plumbum import local

from varats.paper_mgmt.paper_config import project_filter_generator
from varats.utils.settings import bb_cfg
from varats.project.project_util import (
    ProjectBinaryWrapper,
    wrap_paths_to_binaries,
    verify_binaries,
)

class Gravity(bb.Project, CVEProviderHook):  # type: ignore
    """Programming language Gravity."""

    NAME = 'gravity'      # The name of the project
    GROUP = 'c_projects'  # The group this project belongs to
    DOMAIN = 'language'   # The application domain of this project

    SOURCE = [
        bb.source.Git(
            remote="https://github.com/marcobambini/gravity.git",
            local="gravity",
            refspec="HEAD",
            limit=None,
            shallow=False,
            version_filter=project_filter_generator("gravity")
        )
    ]

    @property
    def binaries(self) -> tp.List[ProjectBinaryWrapper]:
        """Return a list of binaries generated by the project."""
        return wrap_paths_to_binaries(["gravity"])

    def run_tests(self) -> None:
        """ This function defines tests and benchmarks for the project. """
        pass

    def compile(self) -> None:
        """ Contains instructions on how to build the project. """

        gravity_version_source = local.path(self.source_of(self.primary_source))
        clang = bb.compiler.cc(self)
        with local.cwd(gravity_version_source):
            with local.env(CC=str(clang)):
                bb.watch(make)("-j", get_number_of_jobs(bb_cfg()))

            verify_binaries(self)

Note

Some projects change their build system at some point in time. In such cases, you could check which version you are currently compiling and dispatch the build process to a more specific build function. For an example, have a look at the real Gravity project.

Specifying Project Binaries¶

The binaries() property of a Project exports a list of binaries built by the project, specifying the relative path from the project root to the created binary. When queried by an analysis, a list of ProjectBinaryWrappers is returned, where every wrapper specifies the name, location of the binary, and the BinaryType. To guarantee that all specified binaries were produced during compilation, one should add a verify_binaries() call after the compilation is done. In our example, the verify_binaries() call is directly after the call to make and still located within the gravity_version_source context, meaning the execution is still in the projects source folder.

Binary wrappers can be created automatically with the provided helper functions wrap_paths_to_binaries or wrap_paths_to_binaries_with_name.

Furthermore, keep in mind that, for some projects, binary locations change during project life time. To specify a specific binary locations for a range of project revisions use benchbuild.utils.revision_ranges.

Blocking revisions¶

It will happen that some revisions of a project fail to build because of missing dependencies or because of a bug in the project sources. The tool suite provides a facility to mark such revisions and block them from further experiments.

This can be done with benchbuild’s block_revisions source declaration decorator. This decorator allows you to block single revisions or larger ranges of revisions.

To block revisions, just add the block_revisions decorator around a git source declaration:

from benchbuild.utils.revision_ranges import (
    block_revisions,
    GoodBadSubgraph,
    RevisionRange,
    SingleRevision,
)

...

SOURCE = [
    block_revisions([
        RevisionRange(
            "0b8e0e047fc3d5e18ead3221ad54920f1ad0eedc",
            "8f417752dd14deea64249b5d32b6138ebc877fa9", "nothing to build"
        ),
        GoodBadSubgraph(["e8999a84efbd9c3e739bff7af39500d14e61bfbc"],
                        ["0e918ce0798407dd6c981e1cd26b4ba138d22fab"],
                        "missing -lm"),
        GoodBadSubgraph(["244c5aa91358a5b2472d351e6c7f38ba7da94ef6"],
                        ["371152de2f38534d4da332349d1def83fc66d5bc"],
                        "Visual studio project breaks makefile"),
        GoodBadSubgraph(["112be515b5ef3b67011c7272e5a50ac3a1fcadc4"],
                        ["b9a62dfad41ae06d029493cf4d5757de2a0281b2"],
                        "bug in gravity"),
        SingleRevision(
            "e207f0cc87bf57e9ccb6f0d18ff4fe4d6ef0c096", "bug in gravity"
        ),
        GoodBadSubgraph(["d2a04f92347fb5f2b6fd23bea9b0e12817cd6d8e"],
                        ["e8fbd6a4a2a9618456f1460dc9138b617dc7af4b"],
                        "bug in gravity"),
        GoodBadSubgraph(["968534c5d4f28501b7f34da48cab2c153ae7449b"],
                        ["0caf15328bda90ffb1911077e03b28ea9970208b"],
                        "bug in gravity"),
        GoodBadSubgraph(["e4f95e669a4c5cf2d142d5b0b72a11c117f7092f"],
                        ["09e59da4deff9b35224f4784fae9d0f132be9cea"],
                        "missing -lbsd"),
    ])(
        bb.source.Git(
            remote="https://github.com/marcobambini/gravity.git",
            local="gravity",
            refspec="HEAD",
            limit=None,
            shallow=False,
            version_filter=project_filter_generator("gravity")
        )
    )
]

The decorator takes a list of revision ranges as argument. There are three different kinds of revision ranges:

SingleRevision: a single revision
RevisionRange: all revisions that lie between the two given revisions start and end, i.e., all revisions in the output of git log --pretty=%H --ancestry-path start..end
GoodBadSubgraph: blocks all revisions that contain a bad commit but no good commit

Each revision range can be annotated with a comment to keep track or the reason why the revisions were blocked. This helps not only when reasoning about the validity of experiments, but it also makes it easier to see whether a block can be fixed in the future (e.g., missing dependencies could be resolved by a containerized execution environment).

List of supported projects¶

C - Projects¶

class varats.projects.c_projects.busybox.Busybox(variant=NOTHING, name=NOTHING, domain=NOTHING, group=NOTHING, container=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_uuid=NOTHING, builddir=NOTHING, source=NOTHING, primary_source=NOTHING, compiler_extension=NOTHING, runtime_extension=None)[source]¶: UNIX utility wrapper BusyBox.

class varats.projects.c_projects.coreutils.Coreutils(variant=NOTHING, name=NOTHING, domain=NOTHING, group=NOTHING, container=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_uuid=NOTHING, builddir=NOTHING, source=NOTHING, primary_source=NOTHING, compiler_extension=NOTHING, runtime_extension=None)[source]¶: GNU coretuils / UNIX command-line tools (fetched by Git)

class varats.projects.c_projects.git.Git(variant=NOTHING, name=NOTHING, domain=NOTHING, group=NOTHING, container=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_uuid=NOTHING, builddir=NOTHING, source=NOTHING, primary_source=NOTHING, compiler_extension=NOTHING, runtime_extension=None)[source]¶: Git.

class varats.projects.c_projects.gravity.Gravity(variant=NOTHING, name=NOTHING, domain=NOTHING, group=NOTHING, container=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_uuid=NOTHING, builddir=NOTHING, source=NOTHING, primary_source=NOTHING, compiler_extension=NOTHING, runtime_extension=None)[source]¶: Programming language Gravity.

class varats.projects.c_projects.gzip.Gzip(variant=NOTHING, name=NOTHING, domain=NOTHING, group=NOTHING, container=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_uuid=NOTHING, builddir=NOTHING, source=NOTHING, primary_source=NOTHING, compiler_extension=NOTHING, runtime_extension=None)[source]¶: Compression and decompression tool Gzip (fetched by Git)

class varats.projects.c_projects.libvpx.Libvpx(variant=NOTHING, name=NOTHING, domain=NOTHING, group=NOTHING, container=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_uuid=NOTHING, builddir=NOTHING, source=NOTHING, primary_source=NOTHING, compiler_extension=NOTHING, runtime_extension=None)[source]¶: Codec SDK libvpx (fetched by Git)

class varats.projects.c_projects.lrzip.Lrzip(variant=NOTHING, name=NOTHING, domain=NOTHING, group=NOTHING, container=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_uuid=NOTHING, builddir=NOTHING, source=NOTHING, primary_source=NOTHING, compiler_extension=NOTHING, runtime_extension=None)[source]¶: Compression and decompression tool lrzip (fetched by Git)

class varats.projects.c_projects.openssl.OpenSSL(variant=NOTHING, name=NOTHING, domain=NOTHING, group=NOTHING, container=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_uuid=NOTHING, builddir=NOTHING, source=NOTHING, primary_source=NOTHING, compiler_extension=NOTHING, runtime_extension=None)[source]¶: TLS-framework OpenSSL (fetched by Git)

class varats.projects.c_projects.opus.Opus(variant=NOTHING, name=NOTHING, domain=NOTHING, group=NOTHING, container=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_uuid=NOTHING, builddir=NOTHING, source=NOTHING, primary_source=NOTHING, compiler_extension=NOTHING, runtime_extension=None)[source]¶: Opus is a codec for interactive speech and audio transmission over the Internet.

class varats.projects.c_projects.qemu.Qemu(variant=NOTHING, name=NOTHING, domain=NOTHING, group=NOTHING, container=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_uuid=NOTHING, builddir=NOTHING, source=NOTHING, primary_source=NOTHING, compiler_extension=NOTHING, runtime_extension=None)[source]¶

QEMU, the FAST!

processor emulator.

class varats.projects.c_projects.tmux.Tmux(variant=NOTHING, name=NOTHING, domain=NOTHING, group=NOTHING, container=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_uuid=NOTHING, builddir=NOTHING, source=NOTHING, primary_source=NOTHING, compiler_extension=NOTHING, runtime_extension=None)[source]¶: Terminal multiplexer Tmux.

class varats.projects.c_projects.vim.Vim(variant=NOTHING, name=NOTHING, domain=NOTHING, group=NOTHING, container=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_uuid=NOTHING, builddir=NOTHING, source=NOTHING, primary_source=NOTHING, compiler_extension=NOTHING, runtime_extension=None)[source]¶: Text processing tool vim.

class varats.projects.c_projects.x264.X264(variant=NOTHING, name=NOTHING, domain=NOTHING, group=NOTHING, container=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_uuid=NOTHING, builddir=NOTHING, source=NOTHING, primary_source=NOTHING, compiler_extension=NOTHING, runtime_extension=None)[source]¶: Video encoder x264 (fetched by Git)

class varats.projects.c_projects.xz.Xz(variant=NOTHING, name=NOTHING, domain=NOTHING, group=NOTHING, container=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_uuid=NOTHING, builddir=NOTHING, source=NOTHING, primary_source=NOTHING, compiler_extension=NOTHING, runtime_extension=None)[source]¶: Compression and decompression tool xz (fetched by Git)

C++ - Projects¶

class varats.projects.cpp_projects.doxygen.Doxygen(variant=NOTHING, name=NOTHING, domain=NOTHING, group=NOTHING, container=NOTHING, cflags=NOTHING, ldflags=NOTHING, run_uuid=NOTHING, builddir=NOTHING, source=NOTHING, primary_source=NOTHING, compiler_extension=NOTHING, runtime_extension=None)[source]¶: Doxygen.