fletch

fletch finds and fetches the external references hiding in a file — the packages it installs, the URLs it reaches for, the images it pulls — and retrieves each one safely enough to hand back to an analyzer.

It is the companion to filefacts. filefacts owns parsing and the dependencies a manifest declares. fletch owns the rest: the fuzzy discovery of the undeclared, imperative references — a curl … | sh, an npm install inside a RUN, a URL stashed in a shell variable — and the retrieval of every reference, declared or not. The loop a consumer composes is: analyze a file → find its references → fetch them → analyze what came back. fletch never analyzes; it finds and fetches.

How it works

Three modules, deliberately walled apart so the dangerous half stays auditable:

• find — recognition over filefacts facts. Returns filefacts' declared dependencies plus the imperative ones hunted out of command streams, decoded text, byte-scan strings, and AST module-load calls, along with the calls it saw but could not resolve. Heuristic by nature, so it lives away from the deterministic parsers and the fetch boundary.
• fetch — resolve a reference to a URL through one per-ecosystem table, retrieve it through an SSRF-guarded HTTP client, cache it, verify any pin, and record provenance. Pure mechanism; no recognition logic leaks in.
• registry — look up and normalize a package's registry metadata (publish date, author, download and favorite counts, rating, removal and hold status) across ecosystems into one shape, so a consumer can judge a dependency before paying to fetch and scan its bytes.

Safety

Every input — the scanned file, registry responses, archives, container layers — is treated as hostile.

• SSRF-guarded by construction. A custom DNS resolver refuses any host that resolves to a private, loopback, link-local, or cloud-metadata address (IPv6 must be global unicast), re-checked on every redirect hop, and connects only to the addresses it vetted, so DNS rebinding has nothing to swap. HTTPS only, never through an environment proxy, and never to a port browsers refuse (SMTP, SSH, IRC, …). Container pulls go through the same client. The one exception to HTTPS: a URL a scanned file names verbatim may also be fetched over plain HTTP, since its author already chose its content and implants are commonly served that way. Only the scheme is relaxed for it; the address, port and resolver guards are unchanged, and registry artifacts stay HTTPS.
• Bounded. Every response is size-capped (a request may lower the operator's ceiling, never raise it), every request and every batch has a deadline, and a batch's registry metadata traffic counts against its budget alongside the artifacts — every byte received, including those of a body abandoned as too large. Container layers are spooled to disk and streamed, never held in memory whole, and an image's entry count is capped.
• Pins are verified. A reference that carries a hash (hex or SRI base64) is checked against the bytes retrieved; a mismatch is always reported as one. Immutable artifacts are cached long, mutable @latest tags briefly.
• One registry per record. A repository_url that does not name the type's public registry is refused for metadata, and the same policy decides where the bytes come from, so a record never pairs one registry's bytes with another's reputation.
• Provenance is recorded. Every lookup keeps the raw provider document(s) it was derived from, so a normalized record can be archived and re-parsed later without a re-fetch.

See SECURITY.md for reporting and known limitations.

Ecosystems

• Language registries: npm, JSR, PyPI, Conda, crates.io, RubyGems, Go, Maven, Clojars, NuGet, Composer, Hex, pub.dev, CRAN, CPAN, Terraform.
• Distributions: Debian, Ubuntu, Fedora, openSUSE, RPM Fusion, Alpine, Wolfi, Arch Linux and the AUR, FreeBSD, NetBSD, OpenBSD, Homebrew, Snap.
• Extension and plugin stores: VS Code Marketplace, Open VSX, Firefox add-ons, Chrome Web Store, JetBrains Marketplace, WordPress, ComfyUI, Dify, ClawHub, Hugging Face.
• Sources and images: GitHub repositories; OCI container images (Docker Hub and Quay), pulled and flattened to a single rootfs tarball.

CLI

The binary is a thin companion to the library, so a non-Rust collector can obtain exactly the record the library produces instead of maintaining a parallel, drifting fetcher. It is built with the cli feature, so a library consumer never compiles its dependencies:

cargo install --git https://github.com/atomdrift-project/fletch.git --features cli
fletch registry pkg:npm/left-pad@1.3.0   # resolve + fetch + normalize registry metadata
fletch purl pkg:pypi/requests@2.31.0      # report how the PURL routes — no network I/O
fletch purl -                             # same, batched: one PURL per stdin line
FLETCH_LOG=debug fletch registry …        # diagnostics on stderr (default: warn)

registry prints a {record, sources} JSON envelope: the normalized record alongside the raw provider responses it came from (a response larger than the source limit is named, with truncated: true, but not copied). Exit status is 0 on success, 1 for a usage or I/O error, and 2 (empty stdout, the reason on stderr) when the argument is not a PURL or the registry has no record. purl performs no network I/O — it reports the parsed coordinates, canonical spelling, registry endpoint, and resolved artifact URL.

Library

use fletch::fetch::{BlobCache, HttpFetch};
use fletch::{RefLocator, registry};

let refs = fletch::find::references_in_bytes(&bytes, "Dockerfile").references;
let net = HttpFetch::new()?;
let record = registry::lookup(&RefLocator::Purl(purl), &net, &BlobCache::open()?)?;

Add it as a git dependency (fletch is not on crates.io while its filefacts dependency is git-only):

fletch = { git = "https://github.com/atomdrift-project/fletch.git" }

The offline fetch::Fixtures backend is behind the test-util feature, for a dependent's tests.

PURL identity and artifacts

purl::Purl is the validated representation used at the fetch boundary; its Display form is the one canonical spelling. Purl::parse follows Postel's law for safe, common legacy spellings; Purl::parse_strict implements the purl-spec parse contract without compatibility repairs. Three identities are intentionally distinct:

• Purl::identity — the package lookup key: subpath and the qualifiers that only select an artifact dropped, everything that distinguishes a release (a conda channel, an rpm epoch, a repository_url) kept.
• Purl::release — the identity of one release at a given version.
• Purl::artifact — one exact file: the release plus its selectors and registry-published checksums.

purl::normalize, purl::identity and purl::url_to_purl are the string forms of these for cross-language consumers.

Use fetch::resolve_artifacts when a release can publish more than one file. Its ArtifactMatrix enumerates filenames, URLs, checksums, platform/ABI tags, and runtime constraints, with the candidate a fetch would choose marked preferred.

PURL-to-URL-to-PURL is tested bidirectionally for registries whose artifact URL contains enough information to recover the coordinate. PyPI content-hash paths, mutable tags, and other lossy URL schemes are tested through registry-backed exact artifact identities instead; they are not falsely presented as bijective.

Caching

Fetched blobs are cached on disk and reclaimed in the background — best-effort, self-gated to once a day, non-blocking. Entries past a TTL or beyond the size ceiling (10 GiB default) are swept oldest-first.

On Unix the cache is readable by the owning user alone: fletch refuses to read, write or sweep a cache directory (or shard) owned by another user or writable by group or others, and tightens one it owns. On Windows the cache relies on the per-user OS cache directory's ACL.

Variable	Default	Effect
FLETCH_CACHE_TTL_DAYS	30	Age after which cached blobs are dropped
FLETCH_CACHE_MAX_BYTES	10737418240	Size ceiling before oldest entries are dropped
FLETCH_CACHE_MAX_ENTRIES	262144	Entry ceiling (a blob and its metadata count once), spread over 256 shard directories

These are read once per process.

Registry etiquette

fletch registry sends GITHUB_TOKEN, when set, to the GitHub API, lifting its anonymous limit of 60 requests an hour; library callers pass a token with HttpFetch::with_github_token. It is attached only to repository documents fletch addresses itself (https://api.github.com/repos/{owner}/{repo}), never to a URL a scanned file or a registry response named, and never again once a redirect chain leaves that origin. A private repository's metadata is never reported. Prefer a token without private-repository scope; it exists for the rate limit.

A registry that answers 429, sends Retry-After, or reports its rate limit exhausted is waited out for up to 30 s; a longer pause fails the request, and every request to that host until it ends, without asking the host again. A transient 408 or 5xx is retried for that request alone.

Metrics

Each fetch, registry lookup, metadata read, and registry back-off emits one tracing event on the fletch::metrics target, with stable field names (event, ecosystem, outcome, served, status, bytes, elapsed_ms, …) and bounded label values, so a subscriber can count cache hit rates, outcomes by ecosystem, throttling, and latency. The fields are listed in src/metrics.rs.

Build

make release        # release binary in target/release/fletch
make test           # run the test suite
make lint           # rustfmt + clippy, warnings denied
make test-purl-spec PURL_SPEC_DIR=../purl-spec

Requires Rust 1.97 or later. See CONTRIBUTING.md.

---

Source and issues live on GitHub. Apache 2.0.