Testing in Extempore

Extempore’s unit testing library (libs/core/test.xtm) provides a few functions/macros for writing unit tests for xtlang code. The most important of these is the xtmtest macro, which takes (up to) three arguments:

  1. the xtlang function to test (e.g. a bind-func)
  2. a call to said function
  3. (optional) the expected return value

Here’s an example:

;; note the quote (') mark
(xtmtest
 '(bind-func add_two_integers
    (lambda (a:i64 b)
      (+ a b)))
 (add_two_integers 2 3)
 5)

If things don’t work as you’d expect, make sure you’ve quoted the bind-func form.

To test an xtlang function multiple times with different arguments, use xtmtest-result (call this as many times as you like with different arguments and return values).

(xtmtest-result (add_two_integers 1 5) 6)
(xtmtest-result (add_two_integers 10 5000) 5010)

xtlang already has a bunch of tests in the tests/ directory. The general idea is that this directory structure will grow to mirror the libs/ one, so that tests/core/adt.xtm has tests for the code in libs/core/adt.xtm, etc. Some of these test files exist already (although more tests are necessary in these cases), while others need to be created as new tests are written.

In addition, the examples in examples/core and examples/external are an additional level of “testing”—although it’s sometimes hard verify that e.g. the music/graphics sounds/looks ok. Still, running these files from top-to-bottom and making sure they don’t crash and produce the right output is important as well.

Running the tests

The easiest way to run the test suite is using CMake/CTest. By default, there’s a test target created in the Extempore configure process, so that on e.g. OSX/Linux, you can run all the tests & examples with:

    make test

This will take a while, but will give a full report on what works and what’s broken.

If you want more control over how the tests are run, then the CMake configure process will also generate a CTestTestfile.cmake file in your build directory, and you can execute ctest in that directory with additional arguments. One use case of this is to only run tests with a certain label:

    cd /path/to/CTestTestfile.cmake
    ctest -L libs-core

The tests are partitioned into the following (hopefully self-explanatory) labels:

  • libs-core
  • libs-external
  • examples-core
  • examples-external
  • examples-external-shader

To include only the labels matching <regex>, use:

    ctest -L <regex>

To exclude only the labels matching <regex>, use:

    ctest -LE <regex>

Get involved

We really appreciate bug reports, and the best way to submit them these days is in the form of a failing test.

You can just paste the failing test into an email to the mailing list, or you can submit a pull request with the test in it. If you’re not sure where your test should go, you can still submit it as a pull request—just add the test code to the bottom of the tests/failing.xtm file.


Improve this page