Test Writing By Example

Writing A Verifier Test

Since all testing in gem5 right now entirely follows the same format, (run a config of gem5, then compare output to a known standard) Whimsy tries to make this common case simple and the intent explicit. Whimsy provides a general utility function whimsy.gem5.suite.gem5_verify_config() and mixin whimsy.gem5.verifier.Verifier classes.

Let’s create a simple test which can runs gem5 and a config file for all ISAs and optimization versions and checks that the exit status of gem5 was 0.

from testlib import *

verifier = VerifyReturncode(0)

gem5_verify_config(
    name='simple_gem5_returncode_test',

    # Pass our returncode verifier here.
    verifiers=(verifier,),

    # Use the pretend config file in the same directory as this test.
    config=joinpath(getcwd(), 'simple-config.py'),
)

We could then use the list command to look at the tests we have created.

$ ./main.py list . --tests
==========================================================================================
Loading Tests

Discovered 30 tests and 15 testsuites in /home/swilson/Projects/whimsy/docs/examples/simple_returncode_test.py
==========================================================================================
Listing all TestCases.
==========================================================================================
docs/examples:TestCase:simple_gem5_returncode_test [X86 - opt]
docs/examples:TestCase:simple_gem5_returncode_test [X86 - opt] (VerifyReturncode verifier)
docs/examples:TestCase:simple_gem5_returncode_test [SPARC - opt]
docs/examples:TestCase:simple_gem5_returncode_test [SPARC - opt] (VerifyReturncode verifier)
docs/examples:TestCase:simple_gem5_returncode_test [ALPHA - opt]
docs/examples:TestCase:simple_gem5_returncode_test [ALPHA - opt] (VerifyReturncode verifier)
docs/examples:TestCase:simple_gem5_returncode_test [RISCV - opt]
docs/examples:TestCase:simple_gem5_returncode_test [RISCV - opt] (VerifyReturncode verifier)
    ... 22 More tests elided...

A less contrived example is to run gem5 using a config and a test program. Here’s an example of how to do this as well:

from testlib import *

verifiers = (
        # Create a verifier that will check that the output
        # contains the regex 'hello'
        verifier.MatchRegex('hello'),

        # The se.py script is dumb and sets a strange return code on success.
        verifier.VerifyReturncode(1),)
hello_program = TestProgram('hello', 'X86', 'linux')

gem5_verify_config(
    name='test_hello',

    # We now rely on the hello_program to be built before this test is run.
    fixtures=(hello_program,),
    verifiers=verifiers,

    # Use the se.py config from configs/example/se.py
    config=joinpath(config.base_dir, 'configs', 'example','se.py'),

    # Give the config the command and path.
    config_args=['--cmd', hello_program.path],

    # The hello_program only works on the X86 ISA.
    valid_isas=('X86',)
)

The new additions to pick out from this example are:

  • We are handing a tuple of verifiers to gem5_verify_config. We can provide any number of these.
  • We created a TestProgram - a fixture which will be setup before our suite runs. We can also hand any number of these to gem5_verify_config.
  • We can hand config arguments by passing and array of flags/args under the kwarg config_args

Running Your Test

There are now a few ways to run this last suite we’ve just created.

First we could run every test in the directory it’s stored in. Assuming you file is stored in tests/gem5/example/test-hello.py. we would run it by executing the command:

tests/main.py run tests/gem5

If we only want to run this specific suite we need to run by giving the uid:

tests/main.py run tests/gem5 --uid 'gem5/example/test-hello:TestSuite:simple_gem5_returncode_test [X86 - opt]'

If we want to run all the tests with the X86 tag we could run it with one of the tags that was automatically added by gem5_verify_config:

tests/main.py run tests/gem5 --tags X86

A Test From Scratch

The gem5_verify_config method covers all the use cases of the old testing framework as far as I know, however the major reason for creating a new framework is so we have test cases that actually test something. (It’s of my opinion that the old tests are all but useless and should be scrapped save for a couple for top level functional testing.) As such, advanced users should be able to create their own tests easily.

As a ‘simple’ example we’ll duplicate some functionality of gem5_verify_config and create a test that manually spawns gem5 and checks it’s return code.

from testlib import *

# Create a X86/gem5.opt target fixture.
gem5 = Gem5Fixture(constants.x86_tag, constants.opt_tag)

# Use the helper function wrapper which creates a TestCase out of this
# function. The test will automatically get the name of this function. The
# fixtures provided will automatically be given to us by the test runner as
# a dictionary of the format fixture.name -> fixture
@testfunction(fixtures=(gem5,),
              tags=[constants.x86_tag, constants.opt_tag])
def test_gem5_returncode(fixtures):

    # Collect our gem5 fixture using the standard name and get the path of it.
    gem5 = fixtures[constants.gem5_binary_fixture_name].path

    command = [
        gem5,
        config=joinpath(config.base_dir, 'configs', 'example','se.py'),
    ]

    try:
        # Run the given command sending it's output to our log at a low
        # priorirty verbosity level.
        log_call(command)
    except CalledProcessError as e:
        if e.returncode == 1:
            # We can fail by raising an exception
            raise e

        elif e.returncode == 2:
            # We can also fail manually with the fail method.
            test.fail("Return code was 2?!")

    # Returncode was 0
    # When we return this test will be marked as passed.

Since the test function was not placed into a test suite by us, when it is collected by the TestLoader it will automatically be placed into a TestSuite with the name of the module.

Writing Your Own Fixtures

whimsy.fixture.Fixture objects are a major component in writing modular and composable tests while reducing code reuse. There are quite a few Fixture classes built in, but they might not be sufficient.

We’ll pretend we have a test that requires we create a very large empty blob file so gem5 can use it as a disk. (This might be a bit contrived.)

from testlib import *
import os

class DiskGeneratorFixture(Fixture):
    def __init__(self, path, size, name):
        super(DiskGeneratorFixture, self).__init__(
              name,
              # Don't build this at startup, wait until a test that uses this runs.
              lazy_init=True,
              # If multiple test suites use this, don't rebuild this fixture each time.
              build_once=True)

        self.path = path
        self.size = size

    def setup(self):
        # This method is called from the Runner when a TestCase that uses this
        # fixture is about to run.

        super(DiskGeneratorFixture, self).setup()

        # Create the file using the dd program.
        log_call(['dd', 'if=/dev/zero', 'of=%s' % self.path, 'count=%d' % self.size])

    def teardown(self):
        # This method is called after the test or suite that uses this fixture
        # is done running.

        # Remove the file.
        os.remove(self.path)

Migrating an Existing Test

Migrating an old test to whimsy takes a minor amount of work. We can use the same whimsy.gem5.suite.gem5_verify_config() function and whimsy.gem5.verifier.Verifier subclasses we used to create our own tests. Optionally we could add an additional utility function to make this even easier, but I would prefer we keep consistency and not port many of the legacy tests, since most legacy tests serve limited utility.

As an example we’ll migrate the old quick/se/00.hello/arm/linux/simple-atomic-dummychecker test. All paths assume the current working directory is the gem5 base path (i.e., …/gem5/).

Here are the steps:

  1. Look in the directory the old test expects a test.py file to be located in.
  2. Move that file and copy it into the slightly different test location tests/gem5/se/00.hello and change the name to config.py.
  3. Move the reference files to tests/gem5/se/00.hello/ARM/simple-atomic-dummychecker
  4. Assuming that the additional config files which set up the old test are not ported, we need to do so.
    • To do this copy over the old config tests/configs/simple-atomic-dummychecker.py to tests/legacy-configs/simple-atomic-dummychecker.py (We know this is the legacy config name because it is the final name of the old test path.)
  5. Create a test-hello.py file in quick/se/00.hello/ and use gem5_verify_config and verifiers to create a suite that will compare gem5 execution to golden standards.
from testlib import *

ref_path = joinpath(getcwd(), 'ARM', 'simple-atomic-dummychecker')

verifiers = (
        verifier.MatchStdout(joinpath(ref_path, 'simout')),
        verifier.MatchStderr(joinpath(ref_path, 'simerr')),
        verifier.MatchStats(joinpath(ref_path, 'stats.txt')),
        verifier.VerifyReturncode(1),
        )

# The test program still fits, only the path is changed from 'arm' to 'ARM'
hello_program = TestProgram('hello', 'ARM', 'linux')

# This is the path of legacy-configs that share the same base config.
dummychecker = joinpath(config.base_dir,
                        'tests',
                        'legacy-configs',
                        'simple-atomic-dummychecker.py')

gem5_verify_config(
        name='test_hello',
        fixtures=(hello_program,),
        verifiers=verifiers,

        # All legacy configs rely on using the run.py script. It has been slightly
        # updated to make it more generic with less assumptions.
        config=joinpath(config.base_dir, 'tests', 'legacy-configs', 'run.py'),

        # Notice that the legacy run.py arguments have changed. It now forces users
        # to specify config files exactly rather than making assumptions on path.
        config_args=['--executable', hello_program.path,
            '--config', dummychecker,
            '--config', joinpath(getcwd(), 'config.py')],
        valid_isas=('ARM',)
)