You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(spec,frontend): state provides from the package that owns a path - #246
A workload inherits tools at versions its author never sees. §9.6 publishes them, but under the distribution's names: a mixin that needs Python has to require deb/python-3.14, which is gone when the base moves to 3.15 and which no Alpine workload provides. A stable name such as python is the author's to choose, but §9.2 will not publish it without a version, and under the version: fallback it becomes python@1.0.0.
This adds providesFrom, a map from capability name to the absolute path of a file in the kit's own filesystem:
Publishing finds the installed package whose file list names the path and states name@version in provides, at that package's version as §9.6 renders it. The field is authored form only: the published descriptor carries the resulting entries and not the field.
Grammar.providesFrom is a new top-level field, added in place under schemaVersion: "3". RELEASES.md says a top-level addition moves schemaVersion because strict decoding makes older readers refuse it. That does not reach published kits here: publishing drops the field, and ValidatePublished refuses a descriptor that still has it, so runtimes and other readers of published descriptors need no change. A frontend built before this change refuses an authored descriptor that uses the field. RELEASES.md records the in-place addition next to the other draft-stage exceptions. No capability version moves.
How a path resolves.
The lookup reads /var/lib/dpkg/info/<package>.list and the F:/R: records of /lib/apk/db/installed directly, because an image assembled from packages keeps the lists without carrying dpkg.
A path a package lists is answered by that package. A path no package lists is followed through symbolic links, which covers update-alternatives and links a recipe made.
Paths are compared through top-level links, so /bin/sh and /usr/bin/sh resolve alike on a merged-/usr system.
A path that is absent, a directory, unowned, owned by two packages, at a version §9.6 cannot name, or at different versions across platforms fails the build. Derived entries are dropped in those cases; these were asked for by name.
Normative statements added (§9.7) and how each is accounted for.
Anchor
Accounting
SPEC-v3 §9.7/not-published
covered by the kit suite's descriptor-valid check, through ValidatePublished
SPEC-v3 §9.7/name-is-authored
waived: grammar rule, judged by the spec validators and schema pin
SPEC-v3 §9.7/workload-only
waived: grammar rule, judged by the spec validators and schema pin
SPEC-v3 §9.7/owned-by-one-package
waived: build-time resolution, judged by spec/provides_from_test.go
SPEC-v3 §9.7/links-followed-when-unowned
waived: same
SPEC-v3 §9.7/version-is-the-owners
waived: same
SPEC-v3 §9.7/agreed-across-platforms
waived: one published manifest cannot show another platform
The build-time duties are waived for the kit suite because a published kit keeps no record of which entry came from a path.
Where the code lives. The lookup is in spec behind a two-method filesystem interface (ImageFiles), so another frontend can vendor it; cmd/frontend adapts a BuildKit reference. ReadDir is used to see symbolic links, since StatFile resolves the whole path first.
Two consequences worth a reviewer's eye.
A stated entry is an ordinary provides entry, so §5.3's one-provider rule applies: a workload that states gh no longer composes with a mixin that provides gh. §9.7 and the authoring skill say so.
A stated entry votes on org.opencontainers.image.version like any authored one. A kit that states several tools needs its own version: to keep that annotation.
Tested. Unit tests in spec and cmd/frontend; schema pin; task validate, lint and test:unit equivalents run by hand in both modules. Of test:tck I ran the statement-accounting tests (TestEvery*), which pass; a full local run of the sandbox suite had not finished when this was opened, so CI is the verdict for it. I also built kits with a frontend image from this branch on debian:bookworm-slim, alpine:3.22, ubuntu:24.04 (the updated ubuntu-docker example) and dhi.io/sbx-templates:shell-docker, and read the published descriptor of each.
Review follow-ups.
3ea2794 removes providesFrom wherever the root mapping gets it, including through a YAML merge key; gives an alias its own copy when it points into what is removed; and makes the published-form check about the key being present, so an empty or null providesFrom is refused.
04f66ba fixes something older than this change that review found in the same code path: entries added at publish no longer override a provides list inherited through a merge key. main has this with the derived deb/ entries alone.
ba77434 fixes two wrong answers in the ownership lookup: a file list belonging to a removed architecture of a package no longer counts because another architecture is installed, and top-level symbolic links are followed through a chain rather than one step.
49800d7 resolves symlinked directories on the way to a requested path, so /usr/lib/jvm/default/bin/java finds the package that installed the real file. Recorded paths are still compared as recorded.
5b1fb1b makes resolution end at the file that is really there: the owner of the resolved path is preferred over a record under the written spelling, and link targets are walked a component at a time so .. is not collapsed as text. Both cases published a wrong version before.
de74933 refuses a path whose directories cannot be walked: a missing component or a file where a directory has to be now ends the lookup, where a .. after it used to lead back to an owned file.
5043867 detaches aliases to an anchored merge sequence before the removal edits it, so a capability config that reuses the alias keeps its content.
8eeb7d5 applies the workload-only rule to a providesFrom that is written but empty, which the schema already refused and validation let through.
d43d0a5 follows merge keys to any depth. An earlier fix capped the walk at 32 levels, below which the field was neither removed nor detected.
Release note
Workload kits can state a provide at the version of the package that owns a file with the new authored-only `providesFrom` field.
A workload inherits tools at versions its author never sees. §9.6 publishes them as deb/ and apk/ entries, but those carry the distribution's names: a mixin that needs Python has to require deb/python-3.14, which is gone when the base moves to 3.15 and which no Alpine workload provides. A stable name such as python is the author's to choose, yet §9.2 will not publish it without a version, and under the version: fallback it becomes python@1.0.0, the kit's release number under a tool's name.
Add providesFrom, a map from capability name to the absolute path of a file in the kit's own filesystem. Publishing finds the installed package whose file list names the path and states name@version in provides, using that package's version as §9.6 renders it. It reads /var/lib/dpkg/info/*.list and the F:/R: records of the apk database directly, because an image assembled from packages keeps the lists without carrying dpkg. A path no package lists is followed through symbolic links, which covers update-alternatives and recipe-made links, and paths are compared through top-level links so both spellings of a merged-/usr path resolve. A path that is absent, a directory, unowned, owned by two packages, at an unnameable version, or at different versions across platforms fails the build: the author asked for the entry by name.
The field is authored form only and workload only. The published descriptor carries the resulting provides entries and not the field, and ValidatePublished refuses one that still has it. That is what allows a top-level addition under schemaVersion 3: runtimes decode published descriptors strictly and never see it. A frontend built before the field refuses an authored descriptor that uses it. RELEASES.md records the in-place addition.
The lookup lives in the spec package behind a two-method filesystem interface so that other frontends can vendor it; cmd/frontend adapts a BuildKit reference to it. The ubuntu-docker example states its Docker and shell tools this way.
Signed-off-by: Christian Dupuis <cd@docker.com>
Sandbox-Id: spiffe://sbx.docker.com/org/57b45934-74c0-11e4-bea4-0242ac11001b/user/1580b874-0d59-41d8-bd81-82a10abc4100/sandbox/74f8ada1-8c4b-4fd6-9802-da94ba278db2
Sandbox-Kit: sha256:dfcafd4d7957b8b6ac4574528241772465187a778e16bdf809d9c85f43e8fed7
Co-authored-by: sbx docker/cdupuis/74f8ada1 <sbx-74f8ada1@users.noreply.docker.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Published YAML can retain empty or merge-inherited providesFrom fields, violating compatibility requirements, while removing the field can break surviving aliases and cause build failures.
Adds authored-only providesFrom declarations so workloads can offer stable tool names at the versions of their owning packages. The shared spec library resolves ownership, and the BuildKit frontend publishes the results as ordinary provides entries.
Changes:
Adds grammar validation and package-file ownership resolution for Debian and Alpine, including cross-platform version agreement.
Integrates resolution and authored-field removal into frontend publishing.
Updates the schema, specification, authoring guidance, example, and tests.
File
Description
tck/sandbox/coverage_test.go
Accounts for the new normative requirements.
spec/validate.go
Adds authored and published providesFrom validation.
spec/types.go
Defines the authored-only descriptor field.
spec/schema_test.go
Pins schema restrictions to the Go grammar.
spec/provides_from.go
Implements ownership resolution, validation, and field removal.
spec/provides_from_test.go
Tests ownership, symlinks, platform agreement, and publication rules.
skills/migrate-kit-to-v3/FIELD-MAPPING.md
Explains when to require stable tool names.
skills/create-kit-v3/SKILL.md
Documents authoring and composition constraints.
schema/kit.schema.json
Adds the field and workload-only restrictions.
RELEASES.md
Records the draft-stage schema compatibility exception.
examples/ubuntu-docker/ubuntu-docker.yaml
Declares stable names for installed tools.
docs/spec/SPEC-v3.md
Specifies path-derived provides and publication behavior.
cmd/frontend/provides_from_test.go
Tests gateway adaptation and platform handling.
cmd/frontend/packages.go
Adapts BuildKit filesystem access for ownership lookup.
cmd/frontend/build.go
Resolves declarations and incorporates their published entries.
…the key in any spelling
Three ways the published form could still carry the field, or break while losing it, all found in review of the change that added it.
WithoutProvidesFrom removed only a key written directly at the top level. A YAML merge key hands the root another mapping's keys, so `<<: {providesFrom: …}` decoded to the same descriptor and kept the field through publishing, and the build then refused its own published form. The field is now removed from every mapping the root merges in, inline or nested. A merge source reached through an alias is copied and the copy is edited, because the anchored mapping belongs to wherever it was defined.
Removing the field also removed any anchor defined under it, leaving an alias elsewhere naming nothing and the document unparseable. Each alias that points into what is removed, or at an anchored mapping that is edited in place, is first replaced by its own copy.
ValidatePublished judged the decoded map's length, so `providesFrom: {}` and a bare `providesFrom:` passed although the key is present and a reader built before the field refuses it. Presence is now read from the raw document, merge keys included.
The rewrite parses its own output before returning it, so a case it cannot handle fails with its cause rather than as a decode error later in the build.
Signed-off-by: Christian Dupuis <cd@docker.com>
Sandbox-Id: spiffe://sbx.docker.com/org/57b45934-74c0-11e4-bea4-0242ac11001b/user/1580b874-0d59-41d8-bd81-82a10abc4100/sandbox/74f8ada1-8c4b-4fd6-9802-da94ba278db2
Sandbox-Kit: sha256:dfcafd4d7957b8b6ac4574528241772465187a778e16bdf809d9c85f43e8fed7
Co-authored-by: sbx docker/cdupuis/74f8ada1 <sbx-74f8ada1@users.noreply.docker.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Derived provides can overwrite inherited declarations, while ownership checks can reject files reached through chained aliases or incorrectly accept files from removed package architectures.
Keep architecture identity when selecting installed package lists
spec/provides_from.go:587
The installed check loses the list's architecture before testing membership. If foo:amd64 is installed but foo:arm64 was removed and retains a .list with conffiles, both lists pass because deb/foo exists in o.versions. A file listed only by the removed architecture can therefore publish a provide at the installed architecture's version instead of failing as unowned. Preserve installed package/architecture identities when selecting dpkg lists; collapse identities only for the version-agreement check.
Resolve chained top-level symlinks before ownership checks
spec/provides_from.go:673
This expands only one top-level link. With /bin -> commands, /commands -> usr/bin, and a package listing /usr/bin/bash, requesting /bin/bash becomes /commands/bash and is reported as unowned even though the filesystem resolves it to the installed file. Resolve successive top-level aliases to a common canonical path, with cycle/depth detection, before comparing ownership records and requested paths.
…ded at publish
withDerivedProvides looked for provides among the root mapping's own keys and created the field when it found none. A merge key can hand the root a provides list from another mapping, and a key created on the root overrides a merged one, so `<<: {provides: [agent@1.0.0]}` published without agent@1.0.0 as soon as a single derived entry was added.
This predates providesFrom: the deb/ entries §9.6 derives are enough to trigger it on any workload with a package database. It was raised in review of the providesFrom change, which adds a second source of entries to the same list.
The created list now starts from the entries the descriptor decodes to. The merged mapping is left as it is, since it may be anchored and shared.
Signed-off-by: Christian Dupuis <cd@docker.com>
Sandbox-Id: spiffe://sbx.docker.com/org/57b45934-74c0-11e4-bea4-0242ac11001b/user/1580b874-0d59-41d8-bd81-82a10abc4100/sandbox/74f8ada1-8c4b-4fd6-9802-da94ba278db2
Sandbox-Kit: sha256:dfcafd4d7957b8b6ac4574528241772465187a778e16bdf809d9c85f43e8fed7
Co-authored-by: sbx docker/cdupuis/74f8ada1 <sbx-74f8ada1@users.noreply.docker.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Ownership resolution can incorrectly treat a removed architecture’s files as installed and fail to match equivalent paths through chained top-level symlinks.
No open issues 🎉
2 previously missed issues
In code that hasn't changed since last review
Verify architecture-specific package installation before using its file list
spec/provides_from.go:587
Dropping the architecture before the installed check lets a removed architecture's leftover file list count as an installed owner. For example, with libfoo:amd64 installed at 2.0 and libfoo:i386 in config-files state at 1.0, libfoo:i386.list passes because deb/libfoo exists. A surviving i386 conffile can then publish a provide at the amd64 package's version despite its owning package being removed. Track installed dpkg identities including architecture when selecting .list files, and collapse to the capability package name only after confirming that specific identity is installed.
Resolve chained top-level symlinks consistently for ownership lookups
spec/provides_from.go:673
This returns after only one top-level link substitution, so equivalent paths can get different ownership keys when top-level links are chained. For example, with /bin -> compat/bin and /compat -> usr, a package listing /bin/tool is indexed as /compat/bin/tool, while providesFrom: {tool: /usr/bin/tool} looks up /usr/bin/tool and fails as unowned even though both paths reach the same installed file. Resolve the top-level link chain consistently for both recorded paths and requested paths, with cycle/depth handling.
… chained top-level links
Two ways the ownership lookup behind providesFrom gave a wrong answer, both found in review.
A co-installable package has one file list per architecture, <package>:<arch>.list, and one architecture can be removed while another stays installed. The lookup dropped the architecture before asking whether the package was installed, so the removed architecture's list, which still holds the conffiles it left behind, passed on the strength of the installed one's name. A leftover conffile could then be stated at the version of a package that does not own it. The status file is now read for which architectures each package is installed for, and an architecture-qualified list owns something only when that architecture is among them.
Top-level symbolic links were followed one step. With /bin pointing at compat/bin and /compat at usr, a recorded /bin/tool was indexed as /compat/bin/tool, where a requested /usr/bin/tool never looked, and an installed command was reported as unowned. The links are now followed until the result no longer starts at one, with a bound so that links leading in a circle end as a path that does not exist.
ReadDpkgStatus's field splitting and installed test are named so the new reader shares them; what it returns is unchanged.
Signed-off-by: Christian Dupuis <cd@docker.com>
Sandbox-Id: spiffe://sbx.docker.com/org/57b45934-74c0-11e4-bea4-0242ac11001b/user/1580b874-0d59-41d8-bd81-82a10abc4100/sandbox/74f8ada1-8c4b-4fd6-9802-da94ba278db2
Sandbox-Kit: sha256:dfcafd4d7957b8b6ac4574528241772465187a778e16bdf809d9c85f43e8fed7
Co-authored-by: sbx docker/cdupuis/74f8ada1 <sbx-74f8ada1@users.noreply.docker.com>
On the two findings in this review, both reproduced and both fixed in ba77434. They came without inline threads, so the answer is here.
Removed architecture counted as an installed owner. With libfoo:amd64 installed at 2.0 and libfoo:i386 in config-files state, a conffile listed only in libfoo:i386.list resolved to foo@2.0. The status file is now read for which architectures each package is installed for, and an architecture-qualified list owns something only when that architecture is installed. A list named without an architecture still belongs to the installed package of that name. A file both installed architectures list still has one owner.
Chained top-level links. With /bin -> compat/bin and /compat -> usr, a recorded /bin/tool was indexed as /compat/bin/tool and a requested /usr/bin/tool came back unowned. Top-level links are now followed until the result no longer starts at one, for recorded and requested paths alike. The walk is bounded, so links that lead in a circle end as "does not exist in the image".
Tests: TestProvidesFromPathsJudgesEachArchitecturesList and TestProvidesFromPathsFollowsChainedTopLevelLinks. I also rebuilt two kits with a frontend image from this commit: on debian:bookworm-slim, where the lists are architecture-qualified (libc6:arm64.list), and on docker.io/sbx/shell, where they are not. Both resolved as before.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because an unpublished finding indicates that intermediate directory symlinks can cause installed files to appear unowned and incorrectly fail the build.
A directory in the middle of a requested path can be a symbolic link: a default JDK, a current release. The lookup compared the path as written with what packages recorded, and only followed a link when the file itself was one, so /usr/lib/jvm/default/bin/java found no owner although a package installed that file as /usr/lib/jvm/java-21/bin/java. The build failed with "belongs to no installed package" for a file a package installed.
The directories of a requested path are now resolved before the file lists are consulted, relative and absolute targets and targets that climb included. The path is still tried as written first, since that is how a package that ships into a linked directory records it. A relative link at the end of the path is resolved against the directory it really is in.
Recorded paths are not resolved this way. That would cost a directory read for every directory in the image; they stay compared as recorded, through top-level links, and §9.7 now says which side gets which treatment.
Signed-off-by: Christian Dupuis <cd@docker.com>
Sandbox-Id: spiffe://sbx.docker.com/org/57b45934-74c0-11e4-bea4-0242ac11001b/user/1580b874-0d59-41d8-bd81-82a10abc4100/sandbox/74f8ada1-8c4b-4fd6-9802-da94ba278db2
Sandbox-Kit: sha256:dfcafd4d7957b8b6ac4574528241772465187a778e16bdf809d9c85f43e8fed7
Co-authored-by: sbx docker/cdupuis/74f8ada1 <sbx-74f8ada1@users.noreply.docker.com>
The review of ba77434 mentions an unpublished finding about intermediate directory symlinks. I checked it and it is real, so it is fixed in 49800d7.
With /usr/lib/jvm/default -> java-21 and a package that records /usr/lib/jvm/java-21/bin/java, providesFrom: {java: /usr/lib/jvm/default/bin/java} failed as unowned. The directories of a requested path are now resolved before the file lists are consulted, including absolute targets and targets that climb with ... The path is still tried as written first, because that is how a package shipping into a linked directory records it.
One limit stays, on purpose: paths a package recorded are compared as recorded, through top-level links only. Resolving every recorded directory would cost a directory read per directory in the image. So a file recorded through a linked directory is found by the spelling the package used, not by its resolved one. §9.7 now states which side gets which treatment, and TestProvidesFromPathsResolvesDirectoryLinks pins both.
Checked with a real build as well: on debian:bookworm-slim, /opt/tools/bash (a linked directory) and /opt/current/bin/awk (two chained directory links, one climbing, then update-alternatives) resolved to bash@5.2.15 and awk@1.3.4.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because an unpublished critical finding indicates that symlink resolution can select the wrong file and publish the wrong package version.
Two readings of a path as text could end at a different file from the one in the image, and state the version of a package that does not own it.
The previous change looked a requested path up as written before looking it up with its directories resolved. A file list can hold a spelling from before a directory on it became a link, so /opt/app/current/bin/tool found a stale record under that name although current now leads to release-2, where another package installed the file. The resolved spelling is now looked up first and the written one only when nothing owns the real file, which keeps the case it was for: a package that ships into a linked directory and records the path that way.
A link's target was joined to its directory and cleaned, which collapses sub/../tool to tool beside the link. In the image, ".." there is the parent of wherever sub leads. Targets are now walked a component at a time through the same directory resolution, for a link at the end of a path and for a directory link alike.
Top-level links are resolved by that same walk when the file lists are read, so an alias names the directory its link finally reaches and a recorded path needs one substitution rather than a loop.
Signed-off-by: Christian Dupuis <cd@docker.com>
Sandbox-Id: spiffe://sbx.docker.com/org/57b45934-74c0-11e4-bea4-0242ac11001b/user/1580b874-0d59-41d8-bd81-82a10abc4100/sandbox/74f8ada1-8c4b-4fd6-9802-da94ba278db2
Sandbox-Kit: sha256:dfcafd4d7957b8b6ac4574528241772465187a778e16bdf809d9c85f43e8fed7
Co-authored-by: sbx docker/cdupuis/74f8ada1 <sbx-74f8ada1@users.noreply.docker.com>
The review of 49800d7 mentions an unpublished critical finding: symlink resolution can select the wrong file and publish the wrong package version. The finding itself is not visible, so I looked for ways that could happen and found two. Both reproduced with a wrong version published, and both are fixed in 5b1fb1b.
A stale record under the written spelling won.49800d7 looked a requested path up as written before looking it up resolved. With old recording /opt/app/current/bin/tool, current -> release-2, and new recording /opt/app/release-2/bin/tool, the result was tool@1.0 where the file there belongs to new at 2.0. The resolved spelling is now looked up first. The written one is the fallback, for a package that ships into a linked directory.
A link target was collapsed as text.tool -> sub/../real-tool with sub -> /srv/other resolved to real-tool beside the link, where the image has /srv/real-tool. Link targets are now walked a component at a time, so .. is the parent of the directory actually reached. This applies to a link at the end of a path, to a directory link, and to top-level aliases.
Tests: TestProvidesFromPathsEndsAtTheRealFile and TestProvidesFromPathsResolvesTopLevelAliasesFully. Real builds with a frontend from this commit on debian:bookworm-slim (including /opt/climb -> tools/../bin/sed, which now resolves to sed@4.9), alpine:3.22 and docker.io/sbx/shell all publish the expected versions.
If the unpublished finding is a third case, I cannot see it from here.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Symlink resolution can incorrectly report package ownership and publish a version when an intermediate path component is missing or is not a directory.
The directory walk reported whether it arrived and the caller ignored the answer, using whatever path came back. When a component was missing the remaining components were appended and the result cleaned, so a ".." after the missing one stepped back out of it: /opt/link pointing at missing/../tool resolved to /opt/tool and published its owner's version for a path the system cannot open.
The walk also accepted any component that was not a link, so a regular file in the middle of a path was walked through as if it were a directory, with the same result for plain/../tool.
A walk that does not arrive now returns no path at all, and nothing is looked up for it. A component has to be a directory or a link to be passed through. A path that ends in a slash, "." or ".." is walked whole as a directory, so a file with a slash after it is refused as well.
Signed-off-by: Christian Dupuis <cd@docker.com>
Sandbox-Id: spiffe://sbx.docker.com/org/57b45934-74c0-11e4-bea4-0242ac11001b/user/1580b874-0d59-41d8-bd81-82a10abc4100/sandbox/74f8ada1-8c4b-4fd6-9802-da94ba278db2
Sandbox-Kit: sha256:dfcafd4d7957b8b6ac4574528241772465187a778e16bdf809d9c85f43e8fed7
Co-authored-by: sbx docker/cdupuis/74f8ada1 <sbx-74f8ada1@users.noreply.docker.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because stripping providesFrom from an anchored merge sequence can silently alter a capability config that reuses its alias.
…se aliases it
Removing providesFrom edits a merge key's list of sources in place: an element is edited, swapped for a copy, or dropped. The removal already gave aliases their own copy of an anchored mapping before changing it, but not of the sequence holding the sources. With `<<: &shared [{providesFrom: …}, {…}]` and a capability config using *shared, the config lost an element when the kit was published, with nothing said.
An anchored sequence is now recorded among the nodes the removal changes, so an alias to it is detached with the sequence as the author wrote it. The test walks every node the removal touches in place and checks that a config aliasing it decodes to what it decoded to before.
Signed-off-by: Christian Dupuis <cd@docker.com>
Sandbox-Id: spiffe://sbx.docker.com/org/57b45934-74c0-11e4-bea4-0242ac11001b/user/1580b874-0d59-41d8-bd81-82a10abc4100/sandbox/74f8ada1-8c4b-4fd6-9802-da94ba278db2
Sandbox-Kit: sha256:dfcafd4d7957b8b6ac4574528241772465187a778e16bdf809d9c85f43e8fed7
Co-authored-by: sbx docker/cdupuis/74f8ada1 <sbx-74f8ada1@users.noreply.docker.com>
The review of de74933 notes, without a published finding, that stripping providesFrom from an anchored merge sequence can silently alter a capability config that reuses its alias. Reproduced, and fixed in 5043867.
With <<: &shared [{providesFrom: {python: /usr/bin/python3}}, {displayName: Second}] and a config holding *shared, the config decoded to one element after publishing where it had two. The removal already detached aliases to an anchored mapping it edits, but not to the sequence that lists the merge sources, which it also changes in place.
An anchored sequence is now among the nodes the removal records as changed, so an alias to it gets the sequence as the author wrote it. TestWithoutProvidesFromLeavesSharedNodesAsTheyWere goes through every node the removal touches in place (the sequence, an element of it, a single anchored source, one reached through a nested merge, a sequence left empty, and the field value) and checks that a config aliasing it decodes to what it decoded to before.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because an unpublished moderate finding identifies inconsistent handling of explicitly empty providesFrom fields on mixins and sets between validation, frontend processing, and the schema.
…n but empty
The schema refuses providesFrom on a mixin or a set by the key being present. Validation applied the rule only when the map had entries, so `providesFrom: {}` or a bare `providesFrom:` on a mixin passed, and the frontend then dropped the field without a word: three answers for one descriptor, and an editor marking wrong what the build accepted.
The rule is now about the field being written. An empty map is visible on the decoded struct. A null is not, since it decodes like no field at all, so ValidateRaw reads presence from the bytes for that spelling, merge keys included. On a workload an empty field still asks for nothing and is dropped at publish.
Signed-off-by: Christian Dupuis <cd@docker.com>
Sandbox-Id: spiffe://sbx.docker.com/org/57b45934-74c0-11e4-bea4-0242ac11001b/user/1580b874-0d59-41d8-bd81-82a10abc4100/sandbox/74f8ada1-8c4b-4fd6-9802-da94ba278db2
Sandbox-Kit: sha256:dfcafd4d7957b8b6ac4574528241772465187a778e16bdf809d9c85f43e8fed7
Co-authored-by: sbx docker/cdupuis/74f8ada1 <sbx-74f8ada1@users.noreply.docker.com>
The review of 5043867 notes, again without a published finding, that an explicitly empty providesFrom on a mixin or a set is handled inconsistently between validation, the frontend and the schema. Confirmed, and fixed in 8eeb7d5.
The schema refuses the key on those kinds whatever is under it. Validation applied the kind rule only when the map had entries, so providesFrom: {} and a bare providesFrom: on a mixin passed, and the frontend then removed the field silently.
Validation now applies the rule to a field that is written. An empty map shows on the decoded struct; a null decodes like no field at all, so ValidateRaw reads presence from the bytes for that spelling, merge keys included. TestProvidesFromKindRuleAppliesToAnEmptyField covers both kinds in each spelling and checks the error is reported once. On a workload an empty field is still accepted and dropped at publish.
One difference remains and is not specific to this field: the schema types providesFrom as an object, so it refuses a null on a workload, where the Go validation accepts it. provides: with nothing under it behaves the same way today.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because an unpublished critical finding shows that deeply nested merges can bypass field removal and validation, allowing published YAML that older strict readers reject.
…providesFrom
The walk that finds providesFrom through YAML merge keys stopped at 32 levels, a bound put there to keep an anchor that aliases itself from recursing forever. The decoder has no such bound: it follows merges as deep as they are nested. A field 33 merges down was therefore neither removed at publish nor seen by the published-form check, and with an empty value nothing else caught it either, so a kit could publish with a key that readers built before the field refuse.
The bound is gone. A walk that would not end is recognized by the same mapping coming round again: presence is judged once per mapping and remembered, which also keeps a mapping merged in many places from being judged many times, and a mapping whose copy is already being stripped is not expanded again beneath itself.
Copying is the one thing that can still grow without limit, when aliases fan out and each level merges the one below it twice. The decoder refuses that document for excessive aliasing before a build reaches the removal; a direct caller now gets an error after a million copied nodes.
Signed-off-by: Christian Dupuis <cd@docker.com>
Sandbox-Id: spiffe://sbx.docker.com/org/57b45934-74c0-11e4-bea4-0242ac11001b/user/1580b874-0d59-41d8-bd81-82a10abc4100/sandbox/74f8ada1-8c4b-4fd6-9802-da94ba278db2
Sandbox-Kit: sha256:dfcafd4d7957b8b6ac4574528241772465187a778e16bdf809d9c85f43e8fed7
Co-authored-by: sbx docker/cdupuis/74f8ada1 <sbx-74f8ada1@users.noreply.docker.com>
The review of 8eeb7d5 notes an unpublished critical finding: deeply nested merges can bypass field removal and validation. Confirmed, and it was my own bound. Fixed in d43d0a5.
The merge walk stopped at 32 levels, which I had added so an anchor that aliases itself could not recurse forever. The decoder has no such limit. With providesFrom: {} 40 merges down, the removal returned the document unchanged and ValidatePublished passed it, so the key would have been published.
There is no depth bound now. A walk that would not end is recognised by the same mapping coming round again: presence is judged once per mapping and remembered, and a mapping whose copy is already being stripped is not expanded again beneath itself. TestProvidesFromIsFoundAtAnyMergeDepth checks removal and the published-form check at depths 1, 31, 32, 33, 40 and 300, with entries, an empty map and a null, and through a chain of 60 aliases.
One bound remains and fails loudly instead of silently: copying. When aliases fan out, each level merging the one below twice, a copy doubles per level. The decoder refuses that document for excessive aliasing before a build gets to the removal; a direct caller gets an error after a million copied nodes. TestWithoutProvidesFromEndsOnDocumentsThatReferToThemselves covers that and a self-merging anchor.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A workload inherits tools at versions its author never sees. §9.6 publishes them, but under the distribution's names: a mixin that needs Python has to require
deb/python-3.14, which is gone when the base moves to 3.15 and which no Alpine workload provides. A stable name such aspythonis the author's to choose, but §9.2 will not publish it without a version, and under theversion:fallback it becomespython@1.0.0.This adds
providesFrom, a map from capability name to the absolute path of a file in the kit's own filesystem:Publishing finds the installed package whose file list names the path and states
name@versioninprovides, at that package's version as §9.6 renders it. The field is authored form only: the published descriptor carries the resulting entries and not the field.Grammar.
providesFromis a new top-level field, added in place underschemaVersion: "3". RELEASES.md says a top-level addition movesschemaVersionbecause strict decoding makes older readers refuse it. That does not reach published kits here: publishing drops the field, andValidatePublishedrefuses a descriptor that still has it, so runtimes and other readers of published descriptors need no change. A frontend built before this change refuses an authored descriptor that uses the field. RELEASES.md records the in-place addition next to the other draft-stage exceptions. No capability version moves.How a path resolves.
/var/lib/dpkg/info/<package>.listand theF:/R:records of/lib/apk/db/installeddirectly, because an image assembled from packages keeps the lists without carryingdpkg.update-alternativesand links a recipe made./bin/shand/usr/bin/shresolve alike on a merged-/usrsystem.Normative statements added (§9.7) and how each is accounted for.
SPEC-v3 §9.7/not-publisheddescriptor-validcheck, throughValidatePublishedSPEC-v3 §9.7/name-is-authoredSPEC-v3 §9.7/workload-onlySPEC-v3 §9.7/owned-by-one-packagespec/provides_from_test.goSPEC-v3 §9.7/links-followed-when-unownedSPEC-v3 §9.7/version-is-the-ownersSPEC-v3 §9.7/agreed-across-platformsThe build-time duties are waived for the kit suite because a published kit keeps no record of which entry came from a path.
Where the code lives. The lookup is in
specbehind a two-method filesystem interface (ImageFiles), so another frontend can vendor it;cmd/frontendadapts a BuildKit reference.ReadDiris used to see symbolic links, sinceStatFileresolves the whole path first.Two consequences worth a reviewer's eye.
providesentry, so §5.3's one-provider rule applies: a workload that statesghno longer composes with a mixin that providesgh. §9.7 and the authoring skill say so.org.opencontainers.image.versionlike any authored one. A kit that states several tools needs its ownversion:to keep that annotation.Tested. Unit tests in
specandcmd/frontend; schema pin;task validate,lintandtest:unitequivalents run by hand in both modules. Oftest:tckI ran the statement-accounting tests (TestEvery*), which pass; a full local run of the sandbox suite had not finished when this was opened, so CI is the verdict for it. I also built kits with a frontend image from this branch ondebian:bookworm-slim,alpine:3.22,ubuntu:24.04(the updatedubuntu-dockerexample) anddhi.io/sbx-templates:shell-docker, and read the published descriptor of each.Review follow-ups.
3ea2794removesprovidesFromwherever the root mapping gets it, including through a YAML merge key; gives an alias its own copy when it points into what is removed; and makes the published-form check about the key being present, so an empty or nullprovidesFromis refused.04f66bafixes something older than this change that review found in the same code path: entries added at publish no longer override aprovideslist inherited through a merge key.mainhas this with the deriveddeb/entries alone.ba77434fixes two wrong answers in the ownership lookup: a file list belonging to a removed architecture of a package no longer counts because another architecture is installed, and top-level symbolic links are followed through a chain rather than one step.49800d7resolves symlinked directories on the way to a requested path, so/usr/lib/jvm/default/bin/javafinds the package that installed the real file. Recorded paths are still compared as recorded.5b1fb1bmakes resolution end at the file that is really there: the owner of the resolved path is preferred over a record under the written spelling, and link targets are walked a component at a time so..is not collapsed as text. Both cases published a wrong version before.de74933refuses a path whose directories cannot be walked: a missing component or a file where a directory has to be now ends the lookup, where a..after it used to lead back to an owned file.5043867detaches aliases to an anchored merge sequence before the removal edits it, so a capability config that reuses the alias keeps its content.8eeb7d5applies the workload-only rule to aprovidesFromthat is written but empty, which the schema already refused and validation let through.d43d0a5follows merge keys to any depth. An earlier fix capped the walk at 32 levels, below which the field was neither removed nor detected.Release note