pkg

Go building blocks shared across Latere AI services: authentication, egress credential substitution, LLM wire-dialect translation, telemetry, audit events, transactional email, Postgres migrations, git and subprocess execution, and a set of small concurrency and text utilities. Every package is importable on its own, keeps its dependency surface small, and carries its own tests.

Install

The module path is latere.ai/x/pkg. The source lives at github.com/latere-ai/pkg.

go get latere.ai/x/pkg

Import the package you need, not the module root:

import "latere.ai/x/pkg/md"

Quick start

Parse frontmatter and render Markdown:

package main

import (
	"fmt"
	"log"

	"latere.ai/x/pkg/md"
)

func main() {
	src := []byte("---\ntitle: Notes\n---\n\n# Hello\n\nSome *text*.\n")

	var meta struct{ Title string }
	body, err := md.ParseInto(src, &meta)
	if err != nil {
		log.Fatal(err)
	}

	html, err := md.Render(body)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(meta.Title)
	fmt.Println(string(html))
}

Call the Lux gateway with a typed request:

c := luxsdk.New("https://lux.latere.ai", luxsdk.WithAPIKey(key))
res, err := c.Generate(ctx, &luxsdk.Request{
	Model:    "claude-sonnet-5",
	Messages: []luxsdk.Message{luxsdk.UserText("hello")},
})

Before writing a package

Check this module first. A generic package with a plausible second consumer is written here first, at this module's bar, and consumed from here; a product's internal/ holds only what is specific to that product. Extraction happens when the second consumer appears, not later: the third copy is the one that drifts. What "generic" means in practice: an S3 client, a metrics registry, an error envelope, a probe surface, a retry loop, a host allow-list, a cancellable sleep. What stays in a product: its error codes, its contract header, its store interface, its configuration.

Packages

Infrastructure

Package	What it gives you
audit	A canonical audit-event envelope plus emitters (stdout, OTLP) to serialize it through, so storage adapters stay a per-product concern
authkit	The auth tree. The root holds the one Identity type, the Authenticator interface, and the middleware every service shares; authkit/jwt verifies RS256 tokens against a cached JWKS and authenticates bearer JWTs; authkit/oidc is the OIDC relying party with a Provider for any standard issuer (Latere auth, Keycloak, Google, Cognito) and the Latere Client with encrypted cookie sessions, token refresh, org switching, and a shared /me assembly; authkit/cli holds the token store and device-code login for command-line clients
batch	A generic non-blocking batching pump: producers add without blocking, one goroutine flushes by size or interval and drains on shutdown
drive	Drive workspace client with per-request bearer sources, typed mount errors, snapshot materialization, writeback, and opaque-cursor pagination
egress	Credential substitution at an egress boundary: a workload holds an opaque placeholder, the gateway swaps it for the real secret only toward the hosts the credential is scoped to. The pure engine (Map, Registry, SubstituteHTTPRequest, SubstituteHTTPRequestContext) with static secrets or resolvers minted at substitution time (OAuthClientCredentials), opt-in bounded body substitution, the ingest API and its client carrying static and OAuth client-credentials entries, JWT proxy authentication with a configurable subject claim, and a TLS-terminating CONNECT Gateway with a leaf-minting CA on top
health	The probe surface every service serves on its internal listener: /livez, /readyz with named checks, /version, and /metrics; the decision and the move from healthz are in docs/health.md
hostsandbox	Runs a process on the operator's own machine inside an srt sandbox (Seatbelt on macOS, bubblewrap on Linux), detached, with its output written where the process cannot reach: the stage vocabulary, the settings renderer with the credential deny table, a handle that survives a restart and detects a reused pid, and a preflight whose refusal carries the install commands for the platform; hostsandbox/hostsandboxtest holds the contract every driver of the same seam is held to
email	Transactional mail transport (Mailgun, SMTP, or a log-only fallback) that refuses header injection; subjects and bodies stay with the calling service
llmdialect	Translation between LLM inference wire dialects (Anthropic Messages, OpenAI Chat Completions, OpenAI Responses, lux-native) through a neutral intermediate representation, with an explicit loss report instead of silent drops. Carries provider-executed tools (web search, web fetch) alongside the caller-implemented kind
llmjson	Repairs the JSON a model meant to send: strips a markdown fence and escapes the raw newlines and tabs left inside string values, so a correct answer in the wrong encoding still decodes
luxsdk	First-party Go client for the Lux gateway's native dialect: typed generate, streaming, and token counting
md	YAML frontmatter parsing and GFM-to-HTML rendering
otel	One-call OpenTelemetry bootstrap for traces, metrics, and structured logs, plus HTTP server/client instrumentation and scoped child spans with error recording
s3	S3 REST client in the standard library: put, create-if-absent, conditional get, head, delete, prefixed listing, and presigned GET and PUT, with a Content-Type sent on upload, read back on get and head, and bound into a presigned PUT, signed with Signature Version 4 and retried under retry; s3/s3test is an in-process endpoint for tests that verifies signatures and digests like a provider
pgxmigrate	Applies embedded golang-migrate migrations and reliably closes migrate's own connection pool afterward
scopes	Typed registry of the OAuth/RBAC scopes the Latere auth service issues, for call-site gating and OIDC discovery

Utilities

Smaller pieces with no product knowledge in them. Most were extracted once a second service needed the same thing.

Package	What it gives you
atomicfile	Write-then-rename file replacement, so a reader never observes a half-written file
bearer	Token extraction from Authorization: Bearer with the RFC 7235 case-insensitive scheme, and constant-time comparison
cache	Generic TTL cache with a bounded LRU cap over every entry and an injectable clock
circuitbreaker	Trip-on-failure gate with injectable clocks, non-consuming readiness checks, and cooldown observation
cmdexec	Fluent subprocess builder and a transactional sequencer that rolls back completed steps when a later one fails
dag	Adjacency-list graph operations: deterministic topological sort, cycle detection, reachability, longest path, edge reversal
dircp	Recursive directory copy
envutil	Typed environment reads with defaults: integers, bounded integers, durations, and the conventional boolean spellings
gitutil	The git CLI behind structured results and typed errors: worktrees, rebase with conflict recovery, stashes, branch discovery
hostmatch	One host allow-list rule for every egress surface: exact FQDNs and *.-prefixed wildcards that match any sub-label depth but never the apex; explicit opt-in for exact single-label hosts
httpjson	Strict JSON request decoding (unknown fields and trailing content rejected), response writing, and the {"error": {code, message, details}} envelope every Latere API answers with
metrics	Prometheus text-exposition registry with labeled counters, histograms, and scrape-time gauges, with no client-library dependency; initialize histogram series at zero without an observation
ndjson	NDJSON file reading and appending, plus the terminal-result scan agent output parsers need
pagination	Cursor pagination helpers
pubsub	In-process topic fanout with per-subscriber buffering
ratelimit	Keyed token buckets with refill, burst, rate overrides, retry delay, and safe idle eviction
relpath	Traversal-safe relative paths: validate, join under a base, and symlink-aware containment
retry	Bounded exponential backoff with jitter and optional per-attempt deadlines under the caller's total budget
routine	Periodic fire-and-forget callbacks keyed by UUID, one timer each, with an injectable clock
sanitize	Rune-safe display truncation, byte-budget truncation that never splits a rune, slug generation, and slug validation
semaphore	Cancellable concurrent-work admission with a wait deadline and an idempotent release function
statemachine	Declarative transition table with guarded moves
syncmap	Type-safe sync.Map, with LoadOrStore for the per-key mutex idiom
trackedwg	Wait group that reports what is still outstanding
tree	Generic tree construction and rendering
uniq	Order-preserving deduplication, with a trim-and-drop-empties form for string lists, and a catalog merge that rejects a repeated key instead of dropping it
wait	Cancellable sleep, ticker loop, and poll; wait/waittest polls a condition in a test until it holds
watcher	Filesystem change notification with debouncing

Package-level documentation, including the streaming grammar and per-provider notes, is on pkg.go.dev.

Status

The module is at v0.x. The API is not frozen: a minor version bump may contain a breaking change, so pin an exact version and read the release notes before upgrading. Package layout and the module path are stable.

Releasing

Every tag has a section in CHANGELOG.md, and that section is the body of the GitHub release. Write notes under Unreleased as changes land, then:

make release VERSION=v0.50.0

This moves the Unreleased notes under the version, commits, tags, and pushes. The release workflow refuses a tag with no section, and the pre-push hook installed by make hooks refuses to push one. The rule is lateregate release and lateregate release-notes, shared by every latere.ai repository; the Makefile targets are names for them.

Testing

The suite is hermetic. It needs no database, no credentials, and no outbound network access (HTTP tests run against httptest servers on loopback), and nothing is skipped for missing configuration:

go test ./...

The only conditional skips are two filesystem cases in authkit that cannot run where the OS resolves a user config directory without consulting env vars, or where changing a temporary directory's mode has no effect. Everything else runs everywhere.

make test           # go vet + go test
make test-race      # run tests with the race detector
make test-hermetic  # run tests with only the toolchain on PATH
make cover          # enforce a 90% floor per package
make cover-html     # open the coverage report in a browser
make fuzz           # run every fuzz target for 30s (FUZZTIME to change)
make validate       # the checks specific to what this module promises
make vuln           # fail on vulnerabilities in imported or called code

make cover gates each package, not the module. An average lets a well-tested package carry an untested one: this module passed at 95% overall while pgxmigrate sat at 82.1%, invisible behind it. All 48 packages clear 90% on their own.

make test-hermetic runs the suite with PATH stripped to the Go toolchain and the directories .lateregate.yaml names. A test that depends on what happens to be installed passes on a laptop and fails on a runner, which is the worst order to find out. Three packages here legitimately drive real binaries, cmdexec, gitutil and hostsandbox, and the config says so; the hostsandbox test that needs a real srt skips where none is installed.

make validate is no-tracked-specs, deps, cgo-free, vuln and fuzz: no internal specs in this public module, llmdialect staying stdlib-only, no cgo anywhere, and the two scans.

CI runs all of it on pushes to main and on every pull request, through the shared pipeline in latere-ai/ci. The checks themselves are latere.ai/x/ci-gate, pinned as a tool dependency in go.mod, so every gate runs the same here as on a runner — a gate you can only run in CI tells you too late. make vuln needs jq and installs govulncheck if it is not already present.

Contributing

Issues and pull requests are welcome. A change is easiest to accept when it keeps the dependency surface small (standard library first), comes with tests, and holds the coverage floor that CI checks.

Install the formatting hook once per clone:

make hooks

Every sentence a package emits or carries is written for one reader, and the register follows the reader: a user of the product, a contributor changing it, or a developer debugging a running system. The rule, the surfaces per register, and the shape of an error (one code, one user sentence, one developer detail) are in docs/writing/registers.md.

License

Apache 2.0