Acceptance Tests and Code Coverage

What are acceptance tests?

When writing tests, we often consider three main categories: unit tests, integration tests, and end-to-end (E2E) tests. The boundaries between these test types, the optimal quantity, and their proportional use remain topics of vigorous debate as long as there are developers and code to be tested.

One frequently overlooked type of test is acceptance tests . This may be due to the inherent complexity in creating them compared to the relatively straightforward process of writing unit tests, which evaluate isolated components or functions, or integration tests, which assess the interaction of multiple components.

Acceptance tests serve the crucial role of validating your entire program. They enable you to examine aspects that would otherwise remain untested, such as your application's response to signals, graceful exits, expected variable parsing, and the nature of its output.

Example

To illustrate this concept, let's consider a hypothetical program: one that reads a delay value from the environment, waits for the specified duration, and then generates a random number. In case the user becomes impatient while waiting, triggering an interrupt, the program should handle the interruption gracefully and exit without issues.

    
package main

import (
	"context"
	"fmt"
	"math/rand"
	"os"
	"os/signal"
	"time"
)

func main() {
	if err := run(); err != nil {
		fmt.Fprintf(os.Stderr, "%s\n", err.Error())
		os.Exit(1)
	}
}

func run() error {
	delay, err := time.ParseDuration(os.Getenv("DELAY"))
	if err != nil {
		return fmt.Errorf("invalid DELAY: %v", err)
	}

	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
	defer stop()

	select {
	case <-ctx.Done():
		return fmt.Errorf("interrupted! Exiting")
	case <-time.After(delay):
		fmt.Println(rand.Intn(100))
	}

	return nil
}
    

How do we test it?

package main

import (
	"bytes"
	"os"
	"os/exec"
	"strconv"
	"strings"
	"syscall"
	"testing"
	"time"

	"github.com/stretchr/testify/require"
)

func TestAcceptance(t *testing.T) {
	file, err := os.CreateTemp("", "acceptance-binary-*")
	require.NoError(t, err)

	build := standardCommand("go", "build", "-o", file.Name(), ".")

	require.NoError(t, build.Run())
	require.NoError(t, file.Close())

	acceptanceCMD := func() (cmd *exec.Cmd, stdout *bytes.Buffer, stderr *bytes.Buffer) {
		cmd = exec.Command(file.Name())
		cmd.Env = append(cmd.Env, os.Environ()...)

		stdout = new(bytes.Buffer)
		stderr = new(bytes.Buffer)

		cmd.Stdout = stdout
		cmd.Stderr = stderr

		return
	}

	t.Run("check random number output", func(t *testing.T) {
		acceptance, stdout, stderr := acceptanceCMD()
		acceptance.Env = append(acceptance.Env, "DELAY=0s")

		require.NoError(t, acceptance.Run())

		require.Len(t, stderr.Bytes(), 0)
		require.Greater(t, len(stdout.String()), 0)

		_, err := strconv.Atoi(strings.TrimSpace(stdout.String()))
		require.NoError(t, err)
	})

	t.Run("check delay parsing", func(t *testing.T) {
		acceptance, stdout, stderr := acceptanceCMD()
		acceptance.Env = append(acceptance.Env, "DELAY=two-minutes")

		require.EqualError(t, acceptance.Run(), "exit status 1")

		require.Len(t, stdout.String(), 0)
		require.Equal(t, "invalid DELAY: time: invalid duration \"two-minutes\"\n", stderr.String())
	})

	t.Run("check interrupt handling", func(t *testing.T) {
		acceptance, stdout, stderr := acceptanceCMD()
		acceptance.Env = append(acceptance.Env, "DELAY=1s")

		require.NoError(t, acceptance.Start())

		// Don't interrupt the process before it had a chance to register the signal interceptors
		time.Sleep(50 * time.Millisecond)

		require.NoError(t, acceptance.Process.Signal(syscall.SIGINT))
		require.EqualError(t, acceptance.Wait(), "exit status 1")

		require.Len(t, stdout.String(), 0)
		require.Equal(t, "interrupted! Exiting\n", stderr.String())
	})
}

func standardCommand(name string, args ...string) *exec.Cmd {
	cmd := exec.Command(name, args...)
	cmd.Env = append(cmd.Env, os.Environ()...)
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr
	cmd.Stdin = os.Stdin
	return cmd
}

Coverage

If we wish to add coverage, we need to compile our test binary the -cover or -coverpkg flags. However since we are running our binary compilation from inside a test, let's try to detect if the test is being run in the context of coverage. The go toolchain will use the os argument -test.gocoverdir when running go test with coverage. We can use this fact to build our test binary with the appropriate flags.

isCoverage := slices.ContainsFunc(os.Args, func(arg string) bool { return strings.HasPrefix(arg, "-test.gocoverdir=") })

build := func() *exec.Cmd {
	if isCoverage {
		return standardCommand("go", "build", "-coverpkg=./...", "-o", file.Name(), ".")
	}
	return standardCommand("go", "build", "-o", file.Name(), ".")
}()
  

Now if we run the test with coverage we should see that all of our code is covered, including our main function and our calls to os.Exit .

go test -cover -v .
=== RUN   TestAcceptance
=== RUN   TestAcceptance/check_random_number_output
=== RUN   TestAcceptance/check_delay_parsing
=== RUN   TestAcceptance/check_interrupt_handling
--- PASS: TestAcceptance (0.37s)
    --- PASS: TestAcceptance/check_random_number_output (0.11s)
    --- PASS: TestAcceptance/check_delay_parsing (0.01s)
    --- PASS: TestAcceptance/check_interrupt_handling (0.05s)
PASS
coverage: 100.0% of statements
ok      tstx    (cached)        coverage: 100.0% of statements
  

Happy coverage!