Repository navigation
feat(xtask): add the fetch and deps steps to the repository check - #12
Merged
Merged
Conversation
The fetch step downloads the packages of each workspace of the check with `cargo fetch --locked`, once for each target in layout.toml. It is the only step that uses the network; the Cargo commands after it run with CARGO_NET_OFFLINE=true. The deps step reads `cargo metadata --filter-platform` offline for each target and takes the union of the resolutions. It checks the sources, where xtask resolves no crate of layout.toml, the checksums, exact requirements, krepi packages, the directions of the resolved graph, the allow entries of each crate in layout.toml and the allowlist of each workspace, where a package in several versions needs a duplicate reason. It records version drift between the workspaces and the members, export crates, crates.io packages and internal edges of each workspace as facts, and prints the entries to add, replace or remove. The allowlists of xtask and desktop are added. The deps step prints entries with the TOML writer, which adds toml_writer to xtask/Cargo.lock. AGENTS.md states that a change of an external dependency updates the allow entries and the allowlist in the same commit.
Every member of a workspace has the license of a kosmo crate, compared as written: MIT, or MIT AND OFL-1.1 when a font file is tracked in its directory. The license expression of every crates.io package is parsed in the strict mode of the spdx crate, pinned at 0.13.6, and evaluated against the allowed set: MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, Zlib and Unicode-3.0, without WITH or +. A package outside the set, with an expression that is not valid, or without one needs a license exception in the allowlist of the workspace, with the expression as written, an optional reading in the set and a reason. The legacy `/` form, when it is in the set with each `/` read as OR, needs that reading, compared after parsing. An exception that has no reason, records another expression, is not resolved or is not needed fails. xtask/allowlist.toml lists spdx and smallvec, and an exception for arraydeque 0.5.1, whose legacy MIT/Apache-2.0 reads as MIT OR Apache-2.0.
The deps step fails for a crates.io package whose build script declares `links`, or whose build dependencies, through every dependency they bring in for the targets of the workspace, include a package that compiles native code or looks for or binds to system libraries (cc, cmake, cxx-build, autotools, nasm-rs, bindgen, pkg-config, vcpkg, system-deps), or one that can download at build time (reqwest, ureq, curl, attohttpc, minreq, http_req, hyper, isahc, git2). Each failure names the chain of build dependencies. The allowlist entry of the package may state an exception to each of these with its reason: links-exception, native-code-exception and download-exception. An exception without a reason, or one that the package does not need, fails. Other build scripts are left to review.
New tests check that the deps step asks for the metadata of each workspace of the check for each of its targets, and of no other workspace; that the receipt of a failing deps step has every problem in its detail and still holds the numbers of each workspace and the versions that differ; and that the versions of xtask's packages are compared with those of the product workspaces. The tests of the check also assert that a failing step keeps the facts it recorded before it failed, and the tests of the build-script rules pin both lists of build dependencies to the names that CONTRIBUTING.md gives. The deps step now ends through Report::finish, which prints the notes to a writer and then gives Report::verdict, which the receipt test shares. Tests read what the step prints for this repository and for a model that fails.
The weekly advisory job is to be added once the allowlists of the product workspaces together list more than about 20 distinct packages, each name and version counted once. The packages of xtask, which already exceed that number, do not count. The limits of the fetch and deps steps state that build dependencies are resolved for the listed targets and not for other hosts, that the Cargo commands after the fetch step lack the packages that only another host uses and how to fetch them first, that `--all-features` turns on every feature of the members, and that `cargo xtask` downloads xtask's packages outside the fetch step. The example allow entry is marked as one of a planned crate, the fields of an allowlist entry state that its `features` are what an allow entry calls `resolved-features`, and the comment on allow entries in layout.toml states how their features are read and that the deps step prints all the entries of a crate when they differ, or says that they all go.
New tests pin the rules of the deps step that no test held so far: a package declared in two tables of one crate is matched with the allow entry of each by kind and target condition; an allow entry listed twice or with unsorted resolved features fails; two declarations without an entry do not share one entry of another kind, and a dependency and an entry of different names are not paired; a difference of a normal dependency under a target condition names its table; an allowlist entry is compared in `links` and `proc-macro` too; a blank duplicate reason fails; the printed allow entries are written as the layout table writes them; and the printed entries keep the reasons of the allowlist. Further tests check that the checksums come from a tracked Cargo.lock that parses, matched by version; that crates.io packages are counted by version and internal edges once per kind; that the same version in two workspaces is no drift; that the deps step records the time of each `cargo metadata` command; and that build dependencies of a package without a build script are not judged. The fixture gains procedural-macro packages and workspaces whose Cargo.lock is not tracked.
New tests check that a feature that only the first target turns on is resolved too; that an allow entry repeats no feature and has exactly its fields, as allowlist entries and license exceptions do; that a directory beside the repository whose name extends the root's is outside it; that a checksum comes from the crates.io entry of the lock file; that a version missing from the allowlist needs its entry alone; that several listed versions of a package, or several locked versions and one other listed version, are not taken for a changed version; that the entries to add are printed sorted; that an allowlist that is not a UTF-8 regular file is reported; and that drift compares every workspace that resolves a package and lists all the versions of each. The fixture can inspect a tree that a test has changed. For build scripts, a package that is both a build and a normal dependency counts as a build dependency, a failure names the shortest chain, an exception holds for its version alone, and the printed entries keep every exception. A reading outside the allowed set is reported alone, and the test of the metadata command pins the value of CARGO_NET_OFFLINE.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two steps to
cargo xtask checkthat check the dependencies Cargo resolves, after the static layout step and before anything is compiled.cargo fetch --lockedonce per target of the layout table. Every later Cargo command runs withCARGO_NET_OFFLINE=true.cargo metadata --locked --all-features --filter-platform <target>per workspace and target; the union of the resolved packages is checked:path, and packages from crates.io; git, other registries andpathpackages outside the layout table are rejected; krepi's packages are rejected;=x.y.z) for crates.io dependencies, checksums inCargo.lock, and directions of the resolved edges between crates of the layout table;layout.tomlentry with their declared and resolved features, and every crates.io package of a workspace'sCargo.lockis listed in the workspace'sallowlist.tomlwith its version, license, build-script and links marks, resolved features and the crates that reach it; several versions of one package need a stated reason. A package that appears only on platforms outside the target set is neither required nor allowed in the list;spdx, must fall inside the allowed set, or the package is listed in the workspace's license exceptions with the expression it is read as and a reason;links, or whose build dependencies bring in a tool that compiles C or C++, looks for system libraries or downloads at build time, fails unless an exception for that kind gives a reason; other build scripts are left to review;xtask/allowlist.tomllists the 26 crates.io packages of the tool workspace, with one license exception for a legacyMIT/Apache-2.0expression;desktop/allowlist.tomlis empty.CONTRIBUTING.mddescribes both steps, the formats, the license and build-script rules with their name lists, and their limits; the advisory job is now triggered by the packages of the product workspaces.cargo xtask checkpasses locally, and each commit passes on its own.