Go’s Code Coverage Resolution

Mon Mar 25, 2019
~900 Words
Tags: programming, Go

For C++ development, I developed GCovHTML to generate reports on code coverage. For Go development, I use the builtin tool to generate these reports. The HTML reports generated by Go are nice in that that are a single file, but they do not include as much detail. To evaluate whether or not it would be feasible to process the profile data from Go using my tool, I investigated the format of the coverprofile output files. This turned up an interesting comparison in how Go tracks coverage data, and what that means for reporting.

Straight to the point, the code coverage model in Go does not track line coverage. Additional information may be kept internally, but the output file only includes information for each basic block. This can be seen in the sample cover profile below, which was taken for a small example program. The file format is straightforward, with each line representing a single record. Each record consists of the filename for the source, a source location for the start of the basic block, a source location for the end of the basic block, a count of the number of executable lines (statements) in the basic block, and a flag indicating whether or not the block was executed. The source locations are each a row and byte offset.

1
2
3
4
5
6
mode: set
tmp/coverage.go:7.12,8.15 1 1
tmp/coverage.go:13.2,15.22 3 1
tmp/coverage.go:8.15,11.3 2 1
tmp/coverage.go:18.12,20.27 2 1
tmp/coverage.go:21.2,21.22 1 0

If you compare the records with the source, you can see that the first two records correspond to the function foo, but excluding the deferred function literal. The third record is the function literal used for the defer statement. Finally, records 4 and 5 are the blocks for the function bar. Note that the compiler has split the code for bar into two blocks. The code on line 21 is unreachable after the panic. Interestingly, if the panic is replaced with a return statement, only a single block is generated for bar, even though the code on line 21 remains unreachable.

Line Coverage

When I initially looked at extracting line coverage, the Go compiler focusing on basic blocks made a lot of sense, but only at first glance. By definition, a basic block cannot contain any branches, so all of the instructions in the block will be executed once execution enters the block. This would be more complicated in a language like C++, which supports exceptions. It would also be more complicated in a language like C, with setjmp and longjmp. Any non-local return could break this model, such as provided by panic. Oops. This model is therefore insufficient to properly capture line coverage in Go.

A simple test can show the limitations of the model. Link the example source to a simple test that does nothing but call the function foo, and then check the line coverage. First, look at the output.

1
2
3
foo:A
bar:A
defer

From the output, one can confirm that lines 15 and 21 are not executed. However, if we look at the output from go tool cover [...], we get the following result. The HTML has been wrapped for this blog, but is otherwise as generated. Note that line 15 is marked at executed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
package coverage

import (
        "fmt"
)

func foo() {
        defer func() {
                fmt.Println("defer")
                recover()
        }()

        fmt.Println("foo:A")
        bar()
        fmt.Println("foo:B")
}

func bar() {
        fmt.Println("bar:A")
        panic("non-local return")
        fmt.Println("bar:B")
}

If we replace the panic on line 20 with a return, we get the following, also incorrect, coverage. It’s not clear how line 21 executed. This is less a limit of the model, then a bug in the compiler. It is not clear why the compiler can recognize a panic as a terminating statement, but failed to recognize a return statement. This is especially true given the specification on terminating statement.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
package coverage

import (
        "fmt"
)

func foo() {
        defer func() {
                fmt.Println("defer")
                recover()
        }()

        fmt.Println("foo:A")
        bar()
        fmt.Println("foo:B")
}

func bar() {
        fmt.Println("bar:A")
        return
        fmt.Println("bar:B")
}

Function and Branch Coverage

In addition to line coverage, it is common to report on function coverage and branch coverage. Neither is easily computed using the data provide in the profile provided by Go. If the source files are parsed, then code coverage can be linked to the functions to identify which functions have been called, and which have not. If Go supported the ternary operator, branch coverage would be impossible to extract. However, since the only branches are control flow statements, which will have their associated basic blocks in the source code, branch coverage could (in principle) be extracted as well.

Summary

The tooling for the Go compiler is very good, and includes code coverage. However, the model used does not contain sufficient granularity to accurately capture line coverage information. The line coverage reported may be inaccurate when there are panics. Additionally, there are bugs linked to terminating statements that can lead to incorrect reporting.

Places to join the discussion