Convenience Functions for Testing in Go

Wed Sep 4, 2019
~800 Words
Tags: programming, Go

Go’s excellent tooling extends to support for testing. Nevertheless, there are quite a few packages that exist to make it easier to write tests. In particular, Go’s verbose (although accepted) handling of errors is a weakness here, as writing tests can involve a lot of boilerplate. Compare writing a test case using, for example, Catch2, with what would be required if using solely the standard library in Go. Unfortunately, I have not found a utility package for testing that meet my design criteria.

The main design criteria are:

  1. Works with the testing package in the standard library.
  2. Generates logging messages based on the test source code.
  3. Each test should be a single statement.

Working with only the standard library is obviously possible, but note that it only meets the first design criteria (by definition). Another common package is testify, but that packages does not meet requirement #2.

The goal here is not necessarily to promote the package that I wrote. That packages is still small, and needs more use to evaluate the API. However, others might find the technical tricks that were required useful. Two tricks made meeting all of the design criteria possible.

(*T) Helper

This technical detail is not that complicated, but was was not available when I first attempted to build a utility library for use with package testing. The method Helper was added to the types B and T in version 1.9. The effect is to suppress the utility function when reporting an error.

Consider the following test:

1
2
3
func TestFancyCalculation(t *testing.T) {
    catch.Check(t, 1 == 2)
}

The output of which will be:

1
example_test.go:10: check failed: 1 == 2

In the output, the filename and line number refer to the line with the call to catch.Check in the user’s code. This is also where an investigation into the failed test should begin. However, without the call to the method Helper inside of catch.Check, the filename and line number will instead refer to the location of the logging method inside of the utility package, and key information in the messages will be incorrect. This is a serious usability problem.

With the introduction of the method Helper in version 1.9, it was possible to write utility packages for testing that met criteria 1 and 3. However, creating the message for failures still required more input from the caller.

runtime.Caller

If the messages are to be generated completely automatically, the utility packages needs to be able to get a textual representation of the test expressions. In other languages, the preprocessor (in C) or macros can be used to stringify predicate expressions, and then pass those strings as arguments. Whatever its other faults, the venerable assert provides complete messages with zero effort from the caller. However, Go provides neither a preprocessor nor macros, so we need another approach.

A solution is to load and parse the source files1. This involves using the function runtime.Caller, which can provide the path and line number of a function’s callers. With this information, the source code can be read and parsed to recover the arguments to the utility function. Since the utility package can assume that the source code is syntactically correct Go, some shortcuts with the parsing are possible, so that the parsing amounts to little more than counting parenthesis.

There is a weakness with this approach, which is that it can be brittle. For example, if you run the earlier example on the Go playground, you will see an additional error message. The file with the source code could not be opened. This also means that the error message could not be completed, and there will be placeholders where normally the expressions in the calling code would be included. In general, the test binary needs to be run in a context where the source files are accessible, and unchanged. Using go test -c to build the test binary, and then executing that binary at a later date or on a different machine will degrade the error messages.

Summary

Please look at the package catch if you are interested in more details. This package is quite small, but does meet the design goals originally stated. Unfortunately, it also introduces some brittleness where the error messages for failed tests can be degraded.


  1. Thanks to github.com/matryer/is, where I first saw this approach. [return]

Places to join the discussion