Testworks Usage#

Testworks is a Dylan unit testing library.

Quick Start #

For the impatient, this section summarizes most of what you need to know to use Testworks.

Add use testworks; to both your test library and test module.

Tests contain arbitrary code and at least one assertion:

define test test-fn1 ()
  let v = do-something();
  assert-equal("expected-value", fn1(v));
  assert-signals(<error>, fn1(v, key: 7), "regression test for bug 12345");
end;

If there are no assertions in a test it is considered “not implemented”, which is displayed in the output (as a reminder to implement it) but is not considered a failure.

See also: Assertions for other assertion macros.

Benchmarks do not require any assertions and are automatically given the “benchmark” tag:

// Benchmark fn1
define benchmark fn1-benchmark ()
  fn1()
end;

Defining Tests #

Assertions #

An assertion accepts an expression to evaluate and report back on, saying if the expression passed, failed, or crashed (i.e., signaled an error). As an example, in

assert-true(foo > bar)

the expression foo > bar is compared to #f, and the result is recorded by the test harness. Failing (or crashing) cause the test to terminate and skip the remaining assertions in that test.

Note

You may also find check-* macros in existing test suites. These are deprecated assertions that do not cause the test to terminate and require a description of the assertion as the first argument.

See the Testworks Reference for detailed documentation on the available assertion macros:

assert-true

assert-false

assert-equal

assert-not-equal

assert-signals

assert-no-errors

assert-instance?

assert-not-instance?

Each of these takes an optional description, after the required arguments, which will be displayed if the assertion fails. If the description isn’t provided, Testworks makes one from the expressions passed to the assertion macro. For example, assert-true(2 > 3) produces this failure message:

(2 > 3) is true failed [expression "(2 > 3)" evaluates to #f]

In general, Testworks should be pretty good at reporting the actual values that caused the failure so it shouldn’t be necessary to include them in the description all the time. Usually if your test iterates over various inputs it’s a good idea to provide a description so the failing input can be easily identified.

If you do provide a description it may either be a single value to display, as with format-to-string("%s", v), or a format string and corresponding format arguments. These are all valid:

assert-equal(a, b);     // auto-generated description
assert-equal(a, b, a);  // a used as description
assert-equal(a, b, "does %= = %=?", a, b);  // formatted description

Tests #

Tests contain assertions and arbitrary code needed to support those assertions. Each test may be part of a suite. Use the test-definer macro to define a test:

define test NAME (#key EXPECTED-FAILURE?, TAGS)
  BODY
end;

For example:

define test my-test ()
  assert-equal(2, 3);
  assert-equal(#f, #f);
  assert-true(identity(#t), "Check identity function");
end;

The result looks like this:

$ _build/bin/my-test-suite
Running suite my-test-suite:
Running test my-test:
  2 = 3: [2 and 3 are not =.]
   FAILED in 0.000257s and 17KiB

my-test failed
  #f = #f passed
  2 = 3 failed [2 and 3 are not =.]
Ran 2 checks: FAILED (1 failed)
Ran 1 test: FAILED (1 failed)
FAILED in 0.000257 seconds

Note that the third assertion was not executed since the second one failed and terminated my-test.

Tests may be tagged with arbitrary strings, providing a way to select or filter out tests to run:

define test my-test-2 (tags: #["huge"])
  ...huge test that takes a long time...
end test;

define test my-test-3 (tags: #["huge", "verbose"])
  ...test with lots of output...
end test;

Tags can then be passed on the Testworks command-line. For example, this skips both of the above tests:

$ _build/bin/my-test-suite-app --tag=-huge --tag=-verbose

Negative tags take precedence, so --tag=huge --tag=-verbose runs my-test-2 and skips my-test-3.

If the test is expected to fail, or fails under some conditions, Testworks can be made aware of this:

define test failing-test
    (expected-to-fail-reason: "bug 1234")
  assert-true(#f);
end test;

define test fails-on-windows
    (expected-to-fail?: method () $os-name = #"win32" end,
     expected-to-fail-reason: "blah is not implemented for WIN32 platform")
  if ($os-name = #"win32")
    assert-false(#t);
  else
    assert-true(#t);
  end if;
end test;

A test that is expected to fail and then fails is considered to be a passing test. If the test succeeds unexpectedly, it is considered a failing test. When marking a test as expected to fail, expected-to-fail-reason: is required and expected-to-fail?: is optional, and normally unnecessary. An example of a good reason is a bug URL or other bug reference.

Note

When providing a value for expected-to-fail?: always provide a method of no arguments. For example, instead of expected-to-fail?: $os-name == #"win32" use expected-to-fail?: method () $os-name == #"win32" end. The former is equivalent to expected-to-fail?: #f on non-Windows platforms and results in an UNEXPECTED SUCCESS result. This is because the (required) reason string is used as shorthand to indicate that failure is expected even when expected-to-fail?: is #f.

Test setup and teardown is accomplished with normal Dylan code using block () ... cleanup ... end;…

define test foo ()
  block ()
    do-setup-stuff();
    assert-equal(...);
    assert-equal(...);
  cleanup
    do-teardown-stuff()
  end
end;

Benchmarks #

Benchmarks are like tests except for:

They do not require any assertions. (They pass unless they signal an error.)
They are automatically assigned the “benchmark” tag.

The benchmark-definer macro is like test-definer:

define benchmark my-benchmark ()
  ...body...
end;

Benchmarks may be added to suites:

define suite my-benchmarks-suite ()
  benchmark my-benchmark;
end;

Benchmarks and tests may be combined in the same suite, but this is discouraged. It is preferable to have separate libraries for the two since benchmarks often take longer to run and may not necessarily need to be run for every commit.

Suites #

Suites are an optional feature that may be used to organize your tests into a hierarchy. Suites contain tests, benchmarks, and other suites. A suite is defined with the suite-definer macro. The format is:

define suite NAME (#key setup-function, cleanup-function)
    test TEST-NAME;
    benchmark BENCHMARK-NAME;
    suite SUITE-NAME;
end;

For example:

define suite first-suite ()
  test my-test;
  test example-test;
  test my-test-2;
end;

define suite second-suite ()
  suite first-suite;
  test my-test;
end;

Suites can specify setup and cleanup functions via the keyword arguments setup-function and cleanup-function. These can be used for things like establishing database connections, initializing sockets and so on.

A simple example of doing this can be seen in the http-server test suite:

define suite http-test-suite (setup-function: start-sockets)
  suite http-server-test-suite;
  suite http-client-test-suite;
end;

Interface Specification Suites #

The interface-specification-suite-definer macro creates a normal test suite, much like define suite does, but based on an interface specification. For example,

define interface-specification-suite time-specification-suite ()
  sealed instantiable class <time> (<object>);
  constant $utc :: <zone>;
  variable *zone* :: <zone>;
  sealed generic function in-zone (<time>, <zone>) => (<time>);
  function now (#"key", #"zone") => (<time>);
  ...
end;

The specification usually has one clause, or “spec”, for each name exported from your public interface module. Each spec creates a test named test-{name}-specification to verify that the implementation matches the spec for {name}. For example, by checking that the names are bound, that their bindings have the correct types, that functions accept the right number and types of arguments, etc.

Specification suites are otherwise just normal suites. They may include other arbitrary tests and child suites if desired:

define interface-specification-suite time-suite ()
  ...
  test test-time-still-moving-forward;
  suite time-travel-test-suite;
end;

This also means that if your interface is large you may use multiple interface-specification-suite-definer forms and then group them together.

See interface-specification-suite-definer for more details on the various kinds of specs.

Organizing Tests for One Library #

If you don’t use suites, the only organization you need is to name your tests and benchmarks uniquely, and you can safely skip the rest of this section. If you do use suites, read on….

Tests are used to combine related assertions into a unit, and suites further organize related tests and benchmarks. Suites may also contain other suites.

It is common for the test suite for library xxx to export a single test suite named xxx-test-suite, which is further subdivided into sub-suites, tests, and benchmarks as appropriate for that library. Some suites may be exported so that they can be included as a component suite in combined test suites that cover multiple related libraries. (The alternative to this approach is running each library’s tests as a separate executable.)

Note

It is an error for a test to be included in a suite multiple times, even transitively. Doing so would result in a misleading pass/fail ratio, and it is more likely to be a mistake than to be intentional.

The overall structure of a test library that is intended to be included in a combined test library may look something like this:

// --- library.dylan ---

define library xxx-tests
  use common-dylan;
  use testworks;
  use xxx;                 // the library you are testing
  export xxx-tests;        // so other test libs can include it
end;

define module xxx-tests
  use common-dylan;
  use testworks;
  use xxx;                 // the module you are testing
  export xxx-test-suite;   // so other suites can include it
end;

// --- main.dylan ---

define test my-awesome-test ()
  assert-true(...);
  assert-equal(...);
  ...
end;

define benchmark my-awesome-benchmark ()
  awesomely-slow-function();
end;

define suite xxx-test-suite ()
  test my-awesome-test;
  benchmark my-awesome-benchmark;
  suite my-awesome-other-suite;
  ...
end;

Running Tests As A Stand-alone Application #

If you don’t need to export any suites so they can be included in a higher-level combined test suite library (i.e., if you’re happy running your test suite library as an executable) then you can simply call run-test-application to parse the standard testworks command-line options and run the specified tests:

run-test-application();

and you can skip the rest of this section.

If you need to export a suite for use by another library, then you must also define a separate executable library, traditionally named “xxx-test-suite-app”, which calls run-test-application().

Here’s an example of such an application library:

1. The file library.dylan which must use at least the library that exports the test suite, and testworks:

Module:    dylan-user
Synopsis:  An application library for xxx-test-suite

define library xxx-test-suite-app
  use xxx-test-suite;
  use testworks;
end;

define module xxx-test-suite-app
  use xxx-test-suite;
  use testworks;
end;

The file xxx-test-suite-app.dylan which simply contains a call to the method run-test-application:

Module: xxx-test-suite-app

run-test-application();

The file xxx-test-suite-app.lid which specifies the names of the source files:

Library: xxx-test-suite-app
Target-type: executable
Files: library.dylan
       xxx-test-suite-app.dylan

Once a library has been defined in this fashion it can be compiled into an executable with dylan-compiler -build xxx-test-suite-app.lid and run with xxx-test-suite-app --help.

Reports #

The --report and --report-file options can be used to write a full report of test run results so that those results can be compared with subsequent test runs, for example to find regressions. These are the available report types:

failures: Prints out only the list of failures and a summary.
json: Outputs JSON objects that match the suite/test/assertion tree structure, with full detail.
summary (the default): Prints out only a summary of how many assertions, tests and suites were executed, passed, failed or crashed.
surefire: Outputs XML in Surefire format. This elides information about specific assertions. This format is supported by various tools such as Jenkins.
xml: Outputs XML that directly matches the suite/test/assertion tree structure, with full detail.

Comparing Test Results #

* To be filled in *

Quick version:

(master branch)$ my-test-suite –report json –report-file out1.json
(your branch)$ my-test-suite –report json –report-file out2.json
$ testworks-report out1.json out2.json