tappp.hpp v0.2.0 released

I have written a TAP producer library in C++. “TAP” is the Test Anything Protocol. It is a language-agnostic, line-oriented plain text protocol for reporting success or failure of software tests in realtime. A test in the sense of TAP is any process which outputs TAP. The test uses a library, a database or whatever else must be tested and checks a sequence of assertions about return values, errors happening or not happening and reports the results in TAP format. On the other side, there is the test harness which originally planned the tests to run, started the test processes and is reading their TAP output. The harness is responsible for showing the test status to the user in a sensible way. This can range from showing the TAP output verbatim over per-test summaries to silent automated tests which exit with non-zero should any test fail.

My library only produces TAP. It helps the programmer writing tests to convert higher-level assertions about objects down to TAP, which only knows “[assertion was] ok” and “[assertion was] not ok”, and automatically prints details about false assertions. It can write, but not read TAP. This is because a universal test harness already exists: it is called prove and is normally installed as part of perl, so you likely have it on your system as well. With prove, you can select a bunch of programs to run and as long as they output TAP, it can tell you if the tests succeeded or not.

TAP producers existed for C++ already, notably libperl++ and libtap++. libtap++ was forked from libperl++ to fix the following issues:

The TAP producer should be isolated into a single library, not come as part of a bundle of Perl-related code.
There should be no compile-time boost dependency, or any dependency for that matter because TAP is really simple.

I called my library tappp.hpp (naming is hard). It was written from scratch but also implements the above two guidelines. I wrote it because there were some things that bothered me about libtap++, namely:

The code seemed a little bitrotted. It contained hacky macros and a bunch of code which could have been eliminated with modern C++ conveniences like variadic templates and argument type deduction.
It was not super re-usable as it relied on a single global state, making in particular the “subtest” API of Test::More hard to add.
It would always build a shared library.

The shared library point deserves a bit more explanation. I use TAP for my software or database test suites. These tests are run by developers or continuous integration services before every commit and possibly by the user on installation. They verify that the software works and that no regressions were introduced.

Tests are (re)compiled when needed, run and then are thrown away. Putting the TAP producer into a shared library does not seem to have any benefit over statically compiling it in, especially given that many parts of it are templated anyway. On the contrary, I do not want the test suite to increase the organizational overhead in the main project, which shared libraries (static or dynamic, the latter moreso) do.

tappp.hpp is just a header, a single .hpp file which you can submodule or copy into your project and #include. This is the level of organizational burden that fits the size and importance of a TAP producer inside a test suite inside of what you are really working on.

How to use the library

tappp.hpp consists of two layers. The bottom layer provides a TAP::Context class which encapsulates everything a TAP producer needs to know and whose methods can be used to… produce TAP. Its interface is modelled after the old Test::More Perl module and the Test module in Raku. It contains the simple assertions ok and nok, object comparisons is and isnt, predicate matching like and unlike and exception-related assertions lives, throws and throws_like. It also supports subtests for easier test plan maintenance.

The second “convenience” layer is written on top of the object-oriented layer. It provides a global instance of TAP::Context and free-standing functions which operate on it. This is to make your test code look just like in Test::More, if you can afford using namespace TAP. The full documentation is available from the tappp.hpp repository.

This sample code is taken from the ReadMe file:

#include <tappp.hpp>
#include <vector>
#include <bitset>
#include <stdexcept>
#include <ctime>
#include <cstdlib>

using namespace TAP;

int main(void) {
    plan(10);

    diag("current time is ", time(0));
    pass("the first one's free");

    ok(1 < 255, "integer comparison works");
    is("55", 55, "pluggable comparison",
        [&](std::string s, int i) {
            return s == std::to_string(i);
        }
    );

    std::vector a{5,10,12};
    std::vector b{5,10,15};

    is(a[0], 5, "first element is 5");
    isnt(a[2], b[2], "last elements differ");

    TODO("they do differ, let's see");
    is(a[2], b[2], "give me diagnostics");
    TODO("compiles, works but can't diagnose");
    is(a, b, "differing vectors");

    SUBTEST("exercising exceptions") {
        throws<std::out_of_range>([&] { a.at(3); },
            "index 3 is out of bounds");

        throws_like([&] {
            std::bitset<5> mybitset(std::string("01234"));
        }, "bitset::_M_copy_from_ptr", "bitset takes only bits");

        TODO("research correct exception type!");
        throws<std::domain_error>([&] {
            b.resize(b.max_size() + 1);
        }, "resizing too much leaves domain");

        done_testing();
    }

    b[2] = a[2] = b[2] * 2;
    is(b[2], 30, "changed last element");
    is(a, b, "vectors match now");

    return EXIT_SUCCESS;
}

It showcases a lot of assertions. When this program is compiled and run, it produces the following TAP output:

$ ./t/readme.t
1..10
# current time is 1582508245
ok 1 - the first one's free
ok 2 - integer comparison works
ok 3 - pluggable comparison
ok 4 - first element is 5
ok 5 - last elements differ
not ok 6 - give me diagnostics # TODO they do differ, let's see
# Expected: '15'
#      Got: '12'
not ok 7 - differing vectors # TODO compiles, works but can't diagnose
    ok 1 - index 3 is out of bounds
    ok 2 - bitset takes only bits
    not ok 3 - resizing too much leaves domain # TODO research correct exception type!
    # different exception occurred
    1..3
ok 8 - exercising exceptions
ok 9 - changed last element
ok 10 - vectors match now

Running it under prove parses the TAP output and displays just a summary:

$ prove -e '' t/readme.t
t/readme.t .. ok
All tests successful.
Files=1, Tests=10,  0 wallclock secs ( 0.02 usr +  0.00 sys =  0.02 CPU)
Result: PASS

Since the file t/readme.t is an executable ELF binary, we tell prove to just run it without any interpreter (-e '').

One trick I really like is to use valgrind as an interpreter for a C++ test program:

$ prove -e 'valgrind --quiet --error-exitcode=111 --exit-on-first-error=yes
    --leak-resolution=low --leak-check=full --errors-for-leak-kinds=all'  \
    t/readme.t
t/readme.t .. ok
All tests successful.
Files=1, Tests=10,  1 wallclock secs ( 0.02 usr  0.00 sys +  0.55 cusr  0.01
csys =  0.58 CPU)
Result: PASS

This line uses a specific valgrind invocation which runs the test program under memcheck and suppresses all of valgrind’s normal output. The program will still do its assertions and print TAP which prove reads. But in addition valgrind is instructed to exit with code 111 (arbitrarily chosen, hopefully uncommon) as soon as a memory-related error is detected. This includes jumps depending on uninitialized memory, access of unowned memory, double frees and memory leaks. prove considers tests which exit with non-zero status failed as well, so in particular memory leaks on the code paths exercised by the test programs are covered by my test suite, which is very nice.