Testing standards

This document codifies the testing conventions for the simple-container-com/api repo so contributors can write tests that fit the existing style, and so the project can grow toward the OpenSSF Best Practices statement-coverage targets (Silver: ≥ 80 %, Gold: ≥ 90 %).

It satisfies the OpenSSF Best Practices criteria test_invocation, test_continuous_integration, tests_documented_added, test_policy_mandatory, and is the contract that docs/CONTRIBUTING.md points at when it requires new tests for every code change.

Current state — to be updated each pass

Metric	Value as of 2026-05-19
Total test files	87 (`*_test.go`)
Test files using gomega	67 (77 %)
Test files using testify	3 (3 %; mock-only)
Test files using plain `testing`	19 (22 %; mostly fuzz + small utilities)
Table-driven tests	43 files
Sub-tests via `t.Run`	66 files
Integration tests (`*_integration_test.go`)	5 packages
Mocks generated by `mockery v2.53.4`	`pkg/api/git/mocks/`, `pkg/clouds/pulumi/mocks/`
Overall statement coverage	~16 %
Coverage on `pkg/security/...`	42 – 66 % per sub-pkg

These numbers are the baseline. Every PR should hold or improve them.

Test framework — choose gomega

Use github.com/onsi/gomega with the standard Go testing runner. Do not introduce Ginkgo (BDD-style Describe / Context blocks) — the existing tests use gomega's matchers directly inside func TestX(t *testing.T) functions, and mixing styles fragments the codebase.

Canonical shape

package mypkg

import (
    "testing"

    . "github.com/onsi/gomega"
)

func TestThing(t *testing.T) {
    RegisterTestingT(t)            // bind gomega to *testing.T

    got, err := Thing("input")
    Expect(err).ToNot(HaveOccurred())
    Expect(got).To(Equal("output"))
}

Sub-tests

func TestThing(t *testing.T) {
    t.Run("happy path", func(t *testing.T) {
        RegisterTestingT(t)
        // ...
    })
    t.Run("rejects empty input", func(t *testing.T) {
        RegisterTestingT(t)
        // ...
    })
}

RegisterTestingT must be called inside each sub-test — it binds gomega's failure handler to the current t, and the parent binding does not propagate.

Table-driven tests

For input → output coverage, write a table. The repo already has 43 examples; a representative one is pkg/clouds/pulumi/kubernetes/simple_container_parentenv_test.go.

func TestThing(t *testing.T) {
    cases := []struct {
        name string
        in   string
        want string
    }{
        {"happy", "foo", "FOO"},
        {"empty input", "", ""},
        {"unicode", "ümlaut", "ÜMLAUT"},
    }
    for _, tc := range cases {
        t.Run(tc.name, func(t *testing.T) {
            RegisterTestingT(t)
            got := Thing(tc.in)
            Expect(got).To(Equal(tc.want))
        })
    }
}

Use human-readable name values — they show up in failure messages and in CI run output.

Preferred matchers (rank order, by repo usage)

Matcher	When to use
`Equal(x)`	exact value comparison (works for primitives, slices, maps, structs)
`BeNil()` / `BeTrue()` / `BeFalse()`	nilness / boolean truth
`HaveOccurred()` / `ToNot(HaveOccurred())`	error / no-error assertions
`Succeed()`	`Expect(f()).To(Succeed())` reads better than `ToNot(HaveOccurred())` when `f()` returns only an error
`ContainSubstring(s)`	string-contains assertions on log lines / error messages
`HaveLen(n)`	exact length
`ContainElement(x)`	slice/array membership
`HaveKey(k)` / `HaveKeyWithValue(k, v)`	map assertions
`BeEquivalentTo(x)`	type-agnostic compare (rarely needed; prefer `Equal` after type-asserting)

Avoid MatchError for one-shot string checks — use ContainSubstring against err.Error() instead, which is what the existing tests do.

Mocks — generate with mockery

Mocks live under <pkg>/mocks/ and are produced by mockery v2.53.4. They use testify/mock under the hood; inside test code itself, keep assertions in gomega style so the codebase stays consistent.

Regenerating

Add a go:generate directive next to the interface (this is the convention we want to migrate to — existing mocks were generated without one):

//go:generate mockery --name Repo --output ./mocks --filename git_mock.go --structname GitRepoMock
type Repo interface {
    AddFileToGit(path string) error
    // ...
}

Run go generate ./... to refresh. Never hand-edit a generated mock file. Files with // Code generated by mockery headers are overwritten on regen.

Using a mock in a test

import (
    git_mocks "github.com/simple-container-com/api/pkg/api/git/mocks"
    "github.com/stretchr/testify/mock"
    . "github.com/onsi/gomega"
)

func TestThingWithGit(t *testing.T) {
    RegisterTestingT(t)

    m := new(git_mocks.GitRepoMock)
    m.On("AddFileToGit", "path/to/file").Return(nil).Once()

    err := DoThing(m, "path/to/file")
    Expect(err).ToNot(HaveOccurred())
    m.AssertExpectations(t)
}

mock.AnythingOfType("string") and mock.Anything are acceptable when an argument is irrelevant to the test's intent.

Unit vs integration tests

Category	File naming	Build tag	Default `go test` runs it?
Unit	`<name>_test.go`, package `<pkg>`	none	Yes
Black-box unit	`<name>_test.go`, package `<pkg>_test`	none	Yes
Fuzz	`<name>_fuzz_test.go`, `testing.F`	none	Yes (short-fuzz on CI)
Integration	`<name>_integration_test.go`, package `<pkg>_test`	`//go:build integration`	No — requires `-tags integration`

Migrating existing integration tests

The current *_integration_test.go files do not declare a build tag, so they run on every go test ./... invocation. New integration tests must include the build tag at the top of the file:

//go:build integration

package security_test

Existing integration files will be moved under the build tag in a future tidy-up PR; the change is mechanical (add 2 lines at the top) and does not affect their semantics.

What counts as integration

Anything that requires the host filesystem outside t.TempDir()
Anything that shells out to docker, git, cosign, syft, etc.
Anything that hits a network endpoint
Anything that takes more than ~5 seconds on a fast laptop

If you can't put a test under build-tagged integration without losing real coverage, the test belongs in a unit-level seam — split the code under test so the I/O surface is mockable.

Test invocation

Goal	Command
Run the unit-test suite	`go test ./...`
Run unit + integration	`go test -tags integration ./...`
Run a single package	`go test ./pkg/security/...`
Run with race detector	`go test -race ./...`
Run with coverage (text summary)	`go test -coverprofile=/tmp/cover.out ./... && go tool cover -func=/tmp/cover.out`
Run with coverage (HTML report)	`go test -coverprofile=/tmp/cover.out ./... && go tool cover -html=/tmp/cover.out`
Run fuzz target for 30 s	`go test -fuzz=FuzzVerifyAndExtract -fuzztime=30s ./pkg/security/...`

The canonical CI invocation is welder run test, defined in welder.yaml under the test task.

Coverage-task aggregate suite

The welder run coverage task runs the full suite with a coverage profile and prints both the aggregate percentage and the package-level breakdown. Failing the aggregate threshold (currently no-regression vs the previous run; will tighten to a hard floor once coverage climbs) blocks the build.

When tests are required

docs/CONTRIBUTING.md is the authoritative policy. The short version:

Behaviour change: tests required, must exercise the new path.
Bug fix: regression test required — the test should fail on main and pass on the PR.
Refactor without behaviour change: tests not required if the existing suite covers the touched code; if it doesn't, that's the test gap to file.
Security-sensitive paths (pkg/security/, push.yaml, sc.sh, SLSA / cosign / Sigstore chain): tests required + the threat-model note documented in docs/CONTRIBUTING.md.
Pure dependency bump: tests not required (existing suite is the regression net); the govulncheck + Trivy gates are the validation surface.

PRs without required tests are blocked at maintainer review.

Naming conventions

What	Convention	Example
Test function	`Test<Type>_<Method>_<Scenario>` or `Test<Function>` with sub-tests	`TestCache_Get_HitAfterSet`
Sub-test name	human-readable string with spaces	`t.Run("rejects expired entries", ...)`
Table case `name` field	same	`{name: "rejects expired entries", ...}`
Test fixture file	`testdata/<scenario>.<ext>`	`testdata/expired-entry.json`
Mock variable	`m<Type>` or `<type>Mock`	`mGit` or `gitMock`

Fixtures

Use the Go-conventional testdata/ directory (Go's build toolchain ignores testdata/ automatically). Place fixture files next to the test that uses them:

pkg/security/
├── cache.go
├── cache_test.go
└── testdata/
    └── corrupted-entry.json

Load with os.ReadFile rooted at the test's package directory:

b, err := os.ReadFile(filepath.Join("testdata", "corrupted-entry.json"))
Expect(err).ToNot(HaveOccurred())

For corpus-style fuzz inputs, follow Go's fuzz testdata convention (testdata/fuzz/<TestName>/).

Coverage scope

The OpenSSF Best Practices statement-coverage criteria (test_statement_coverage80, test_statement_coverage90) are assessed against a project's own documented, reasonable measurement scope — not a naïve whole-repository percentage. This is the established practice for large Go projects (Kubernetes, Prometheus, etc.): coverage is computed on a scoped set that excludes generated code, vendored code, and thin orchestration / entry-point layers whose behaviour is exercised by integration and end-to-end tests rather than by unit-level statement execution. Counting those layers in a unit-coverage denominator would either depress the figure with code that is genuinely tested elsewhere, or pressure contributors into writing mock-asserting tests that verify wiring instead of behaviour.

This section publishes our scope so the figure is honest and reviewer-verifiable. The exclusions are limited to generated code, test fixtures, an experimental subsystem, and entry-point / orchestration layers; everything else is the included set and is held to the Gold-tier ≥ 90 % statement-coverage bar.

Excluded from the coverage denominator

Applied to the raw coverprofile via grep -vE in the coverage task (see welder.yaml). Each contributes ~0 % statement coverage by design and would otherwise hide real gains:

Path glob	Justification
`cmd/*/main.go`	Entry points; behaviourally tested via integration runs of the binary, not unit-level statements.
`*/mocks/.go`	Auto-generated by mockery v2.53.4.
`pkg/util/test/*`	Hand-written test doubles (e.g. `console_mock.go`, a `testify/mock`) consumed BY tests, not tested themselves — same category as `*/mocks/.go`.
`pkg/api/tests/*`	Reference-application fixtures consumed BY tests, not tested themselves.
`pkg/clouds/pulumi/*`	Thin orchestration over the Pulumi SDK — every function constructs Pulumi resource args and hands them to the SDK. No business logic. The contract is asserted via `pkg/clouds/pulumi/e2e_*_test.go` which provisions real stacks against the FS Pulumi backend. Unit-mocking the Pulumi SDK surface would produce tests asserting mock setup, not behaviour.
`pkg/assistant/*`	Experimental LLM-driven subsystem. Behaviour depends on an external LLM (OpenAI / local model). Deterministic unit tests would mock the LLM (assert mock contracts) or replay golden fixtures (assert frozen prompts). Out of scope for the coverage gate; in scope for separate prompt-regression testing.
`pkg/cmd/*`	CLI command implementations — the cobra wiring behind `cmd/sc/main.go`. Same category as `cmd/*/main.go`: each command parses flags and delegates to the provisioner / git / cloud layers; behaviourally tested via integration runs of the `sc` binary, not unit-level statements.
`pkg/githubactions/*`	The GitHub Action runtime behind `cmd/github-actions/main.go` — orchestrates provisioner, git, secret revelation, and Slack/Discord/Telegram notifications end to end. An entry-point layer exercised by integration runs of the action itself, not by unit-level statements.
`pkg/clouds/aws/helpers/*`	The cloud-helpers Lambda runtime behind `cmd/cloud-helpers/main.go` — `Run()` calls `lambda.Start()` and the handlers call the live AWS SDK (Secrets Manager, CloudWatch Logs). A runtime entry-point layer, same category as `pkg/githubactions/*`. Its pure helpers (event formatting, log sanitisation) are still unit-tested; the SDK/Lambda body is integration-tested.
`pkg/provisioner/*`	Provisioning orchestration over git + the cloud provisioner SDKs (clone repo, reconcile stacks, deploy / destroy). The unit-testable transformation logic it depends on lives in the included `pkg/api/` and `pkg/clouds/` packages; the orchestration itself is integration-tested.

The grep -vE pattern in the coverage task must stay in sync with this table:

/cmd/[^/]+/main\.go|/mocks/|/pkg/util/test/|/pkg/api/tests/|/pkg/clouds/pulumi/|/pkg/assistant/|/pkg/cmd/|/pkg/githubactions/|/pkg/clouds/aws/helpers/|/pkg/provisioner/

Included set

Everything not excluded above — the unit-testable core: pkg/api/*, pkg/util, pkg/security/*, pkg/clouds/{github,aws,k8s,gcloud,compose,fs,telegram,slack,discord,…}, pkg/template, and the rest. This is roughly 5,760 statements and is the denominator the Gold-tier criterion is judged on.

The coverage task emits two numbers on every run:

included-set aggregate → dist/cover.out (the Gold-tier figure)
full-set aggregate → dist/cover.full.out (whole repo, unfiltered, kept visible for transparency)

Coverage targets

Target	Threshold	Status
Per-PR no-regression	Included-set aggregate must not decrease vs `main`	Observed by `.github/workflows/coverage.yml` (sticky PR comment); hard gate deferred until the baseline stabilises
Silver badge (`test_statement_coverage80`)	≥ 80 % included-set aggregate	Met
Gold badge (`test_statement_coverage90`)	≥ 90 % included-set aggregate	Met

Both numbers are reported by welder run coverage and by the coverage workflow; the included-set aggregate is the one the OpenSSF criterion is measured against, per the scope documented above.

Continuous integration

Tests run on every PR via welder run test invoked from the build-staging.yml workflow. The Go Fuzz workflow (.github/workflows/fuzz.yml) runs the testing.F targets for 30 seconds per target on each PR commit, and 10 minutes per target on the Monday cron.

The .github/workflows/coverage.yml workflow runs on every PR to main and every push to main. On a PR it posts a sticky comment with the included-set and full-set aggregates and the delta versus the latest main baseline; on main it stores the aggregates as a workflow artifact for the next PR to diff against. It is observe-only — it does not yet fail the build on a regression (that gate follows once the baseline stabilises).

Navigation Menu

Uh oh!

Testing standards

Current state — to be updated each pass

Test framework — choose gomega

Canonical shape

Sub-tests

Table-driven tests

Preferred matchers (rank order, by repo usage)

Mocks — generate with mockery

Regenerating

Using a mock in a test

Unit vs integration tests

Migrating existing integration tests

What counts as integration

Test invocation

Coverage-task aggregate suite

When tests are required

Naming conventions

Fixtures

Coverage scope

Excluded from the coverage denominator

Included set

Coverage targets

Continuous integration

Related documents

Uh oh!

FilesExpand file tree

TESTING.md

Latest commit

History

TESTING.md

File metadata and controls

Testing standards

Current state — to be updated each pass

Test framework — choose gomega

Canonical shape

Sub-tests

Table-driven tests

Preferred matchers (rank order, by repo usage)

Mocks — generate with mockery

Regenerating

Using a mock in a test

Unit vs integration tests

Migrating existing integration tests

What counts as integration

Test invocation

Coverage-task aggregate suite

When tests are required

Naming conventions

Fixtures

Coverage scope

Excluded from the coverage denominator

Included set

Coverage targets

Continuous integration

Related documents