C++ unit testing frameworks

We should research and select a C++ unit testing framework that we can encourage developers to use uniformly across the modules. A cursory search yields a framework called doctest that looks promising. This article claims it is “probably the best choice for small projects”

The doctest tutorial shows an overview of how it can be incorporated into the code modules:

A simple example

Suppose we have a factorial() function that we want to test:

int factorial(int number) { return number <= 1 ? number : factorial(number - 1) * number; }

A complete compiling example with a self-registering test looks like this:

#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include "doctest.h"

int factorial(int number) { return number <= 1 ? number : factorial(number - 1) * number; }

TEST_CASE("testing the factorial function") {
    CHECK(factorial(1) == 1);
    CHECK(factorial(2) == 2);
    CHECK(factorial(3) == 6);
    CHECK(factorial(10) == 3628800);
}

This will compile to a complete executable which responds to command line arguments. If you just run it with no arguments it will execute all test cases (in this case - just one), report any failures, report a summary of how many tests passed and failed and returns 0 on success and 1 if anything failed (useful if you just want a yes/no answer to: “did it work”).

If you run this as written it will pass. Everything is good. Right? Well there is still a bug here. We missed to check if factorial(0) == 1 so lets add that check as well:

TEST_CASE("testing the factorial function") {
    CHECK(factorial(0) == 1);
    CHECK(factorial(1) == 1);
    CHECK(factorial(2) == 2);
    CHECK(factorial(3) == 6);
    CHECK(factorial(10) == 3628800);
}

. . .

Personally I do not have experience with C++ unit tests, so we need input from people that do.

@rhaas @cnc6 @hungtan2 @hshah10 @ziyuan.z

Thank you for the suggestion @andrew.manning. I just implemented this into our CMF module as you can see in commits
https://gitlab.com/nsf-muses/module-cmf/cmf-solver/-/commit/2625c64b604dd694f006ed2732d2927fee2077ee
https://gitlab.com/nsf-muses/module-cmf/cmf-solver/-/commit/1edfb5f1be540c347bcdf6d08ed93371ed868747
and they work as intended.

1 Like

Nice work. Your code will be a good example to others.

I should incorporate this into the NSF MUSES / Common · GitLab repo and push a new image that you can then check reproduces your tests properly.

@cnc6 I’m sure you received a GitLab notification, but I updated and rebuilt the base image and tested on your Dockerfile that I modified for the purpose: https://gitlab.com/nsf-muses/module-cmf/cmf-solver/-/merge_requests/2

@andrew.manning I did and I’ve approved the merge. Thanks for the information about the base image.