Test Writing By Example¶
Writing A Verifier Test¶
Since all testing in gem5 right now entirely follows the same format,
(run a config of gem5, then compare output to a known standard) Whimsy
tries to make this common case simple and the intent explicit. Whimsy
provides a general utility function whimsy.gem5.suite.gem5_verify_config()
and mixin
whimsy.gem5.verifier.Verifier
classes.
Let’s create a simple test which can runs gem5 and a config file for all ISAs and optimization versions and checks that the exit status of gem5 was 0.
from testlib import *
verifier = VerifyReturncode(0)
gem5_verify_config(
name='simple_gem5_returncode_test',
# Pass our returncode verifier here.
verifiers=(verifier,),
# Use the pretend config file in the same directory as this test.
config=joinpath(getcwd(), 'simple-config.py'),
)
We could then use the list command to look at the tests we have created.
$ ./main.py list . --tests
==========================================================================================
Loading Tests
Discovered 30 tests and 15 testsuites in /home/swilson/Projects/whimsy/docs/examples/simple_returncode_test.py
==========================================================================================
Listing all TestCases.
==========================================================================================
docs/examples:TestCase:simple_gem5_returncode_test [X86 - opt]
docs/examples:TestCase:simple_gem5_returncode_test [X86 - opt] (VerifyReturncode verifier)
docs/examples:TestCase:simple_gem5_returncode_test [SPARC - opt]
docs/examples:TestCase:simple_gem5_returncode_test [SPARC - opt] (VerifyReturncode verifier)
docs/examples:TestCase:simple_gem5_returncode_test [ALPHA - opt]
docs/examples:TestCase:simple_gem5_returncode_test [ALPHA - opt] (VerifyReturncode verifier)
docs/examples:TestCase:simple_gem5_returncode_test [RISCV - opt]
docs/examples:TestCase:simple_gem5_returncode_test [RISCV - opt] (VerifyReturncode verifier)
... 22 More tests elided...
A less contrived example is to run gem5 using a config and a test program. Here’s an example of how to do this as well:
from testlib import *
verifiers = (
# Create a verifier that will check that the output
# contains the regex 'hello'
verifier.MatchRegex('hello'),
# The se.py script is dumb and sets a strange return code on success.
verifier.VerifyReturncode(1),)
hello_program = TestProgram('hello', 'X86', 'linux')
gem5_verify_config(
name='test_hello',
# We now rely on the hello_program to be built before this test is run.
fixtures=(hello_program,),
verifiers=verifiers,
# Use the se.py config from configs/example/se.py
config=joinpath(config.base_dir, 'configs', 'example','se.py'),
# Give the config the command and path.
config_args=['--cmd', hello_program.path],
# The hello_program only works on the X86 ISA.
valid_isas=('X86',)
)
The new additions to pick out from this example are:
- We are handing a tuple of verifiers to
gem5_verify_config
. We can provide any number of these. - We created a
TestProgram
- a fixture which will besetup
before our suite runs. We can also hand any number of these togem5_verify_config
. - We can hand config arguments by passing and array of flags/args under
the kwarg
config_args
Running Your Test¶
There are now a few ways to run this last suite we’ve just created.
First we could run every test in the directory it’s stored in. Assuming you
file is stored in tests/gem5/example/test-hello.py
. we would run it
by executing the command:
tests/main.py run tests/gem5
If we only want to run this specific suite we need to run by giving the uid:
tests/main.py run tests/gem5 --uid 'gem5/example/test-hello:TestSuite:simple_gem5_returncode_test [X86 - opt]'
If we want to run all the tests with the X86 tag we could run it with
one of the tags that was automatically added by gem5_verify_config
:
tests/main.py run tests/gem5 --tags X86
A Test From Scratch¶
The gem5_verify_config
method covers all the use cases of the old
testing framework as far as I know, however the major reason for
creating a new framework is so we have test cases that actually test
something. (It’s of my opinion that the old tests are all but useless
and should be scrapped save for a couple for top level functional
testing.) As such, advanced users should be able to create their own
tests easily.
As a ‘simple’ example we’ll duplicate some functionality of
gem5_verify_config
and create a test that manually spawns gem5 and
checks it’s return code.
from testlib import *
# Create a X86/gem5.opt target fixture.
gem5 = Gem5Fixture(constants.x86_tag, constants.opt_tag)
# Use the helper function wrapper which creates a TestCase out of this
# function. The test will automatically get the name of this function. The
# fixtures provided will automatically be given to us by the test runner as
# a dictionary of the format fixture.name -> fixture
@testfunction(fixtures=(gem5,),
tags=[constants.x86_tag, constants.opt_tag])
def test_gem5_returncode(fixtures):
# Collect our gem5 fixture using the standard name and get the path of it.
gem5 = fixtures[constants.gem5_binary_fixture_name].path
command = [
gem5,
config=joinpath(config.base_dir, 'configs', 'example','se.py'),
]
try:
# Run the given command sending it's output to our log at a low
# priorirty verbosity level.
log_call(command)
except CalledProcessError as e:
if e.returncode == 1:
# We can fail by raising an exception
raise e
elif e.returncode == 2:
# We can also fail manually with the fail method.
test.fail("Return code was 2?!")
# Returncode was 0
# When we return this test will be marked as passed.
Since the test function was not placed into a test suite by us, when it
is collected by the TestLoader
it will automatically be placed into
a TestSuite
with the name of the module.
Writing Your Own Fixtures¶
whimsy.fixture.Fixture
objects are a major component in writing
modular and composable tests while reducing code reuse. There are quite a few
Fixture
classes built in, but they might not be sufficient.
We’ll pretend we have a test that requires we create a very large empty blob file so gem5 can use it as a disk. (This might be a bit contrived.)
from testlib import *
import os
class DiskGeneratorFixture(Fixture):
def __init__(self, path, size, name):
super(DiskGeneratorFixture, self).__init__(
name,
# Don't build this at startup, wait until a test that uses this runs.
lazy_init=True,
# If multiple test suites use this, don't rebuild this fixture each time.
build_once=True)
self.path = path
self.size = size
def setup(self):
# This method is called from the Runner when a TestCase that uses this
# fixture is about to run.
super(DiskGeneratorFixture, self).setup()
# Create the file using the dd program.
log_call(['dd', 'if=/dev/zero', 'of=%s' % self.path, 'count=%d' % self.size])
def teardown(self):
# This method is called after the test or suite that uses this fixture
# is done running.
# Remove the file.
os.remove(self.path)
Migrating an Existing Test¶
Migrating an old test to whimsy takes a minor amount of work. We can use the
same whimsy.gem5.suite.gem5_verify_config()
function and
whimsy.gem5.verifier.Verifier
subclasses we used to create our own
tests. Optionally we could add an additional utility function to make this even
easier, but I would prefer we keep consistency and not port many of the legacy
tests, since most legacy tests serve limited utility.
As an example we’ll migrate the old quick/se/00.hello/arm/linux/simple-atomic-dummychecker test. All paths assume the current working directory is the gem5 base path (i.e., …/gem5/).
Here are the steps:
- Look in the directory the old test expects a
test.py
file to be located in. - Move that file and copy it into the slightly different test location
tests/gem5/se/00.hello
and change the name toconfig.py
. - Move the reference files to
tests/gem5/se/00.hello/ARM/simple-atomic-dummychecker
- Assuming that the additional config files which set up the old test are not
ported, we need to do so.
- To do this copy over the old config
tests/configs/simple-atomic-dummychecker.py
totests/legacy-configs/simple-atomic-dummychecker.py
(We know this is the legacy config name because it is the final name of the old test path.)
- To do this copy over the old config
- Create a
test-hello.py
file inquick/se/00.hello/
and usegem5_verify_config
and verifiers to create a suite that will compare gem5 execution to golden standards.
from testlib import *
ref_path = joinpath(getcwd(), 'ARM', 'simple-atomic-dummychecker')
verifiers = (
verifier.MatchStdout(joinpath(ref_path, 'simout')),
verifier.MatchStderr(joinpath(ref_path, 'simerr')),
verifier.MatchStats(joinpath(ref_path, 'stats.txt')),
verifier.VerifyReturncode(1),
)
# The test program still fits, only the path is changed from 'arm' to 'ARM'
hello_program = TestProgram('hello', 'ARM', 'linux')
# This is the path of legacy-configs that share the same base config.
dummychecker = joinpath(config.base_dir,
'tests',
'legacy-configs',
'simple-atomic-dummychecker.py')
gem5_verify_config(
name='test_hello',
fixtures=(hello_program,),
verifiers=verifiers,
# All legacy configs rely on using the run.py script. It has been slightly
# updated to make it more generic with less assumptions.
config=joinpath(config.base_dir, 'tests', 'legacy-configs', 'run.py'),
# Notice that the legacy run.py arguments have changed. It now forces users
# to specify config files exactly rather than making assumptions on path.
config_args=['--executable', hello_program.path,
'--config', dummychecker,
'--config', joinpath(getcwd(), 'config.py')],
valid_isas=('ARM',)
)