make test refuses a missing pinned toolchain by name, and names a skip that is asked for (#599)

.github/workflows/ci.yml

@@ -123,6 +123,17 @@ jobs:
           make projection-wire-law-negative-control
           make projection-union-arm-order-negative-control
 
+      # THE TOOLCHAIN GATE's control (issue #599), on the same reasoning and in
+      # the same job. `make test` is the only thing that runs the gate, and no
+      # job on a pull request runs `make test`, so until this step existed the
+      # gate that stops a missing toolchain was itself unwatched on the diff
+      # that changes it. The control needs no toolchain of its own: it points
+      # each registered pin at a path that does not exist and reads the
+      # refusal, which is the state this runner is already in, and takes about
+      # a second.
+      - name: the toolchain gate goes red without the toolchains it pins
+        run: make toolchain-negative-control
+
   # THE GENERATED TREE IS CURRENT (issue #30). The committed generated/ tree is
   # what this repo treats as ground truth — findings are receipted against its
   # line numbers — so a compiler change that moves the emitted text and leaves

docs/CONTRIBUTING.md

@@ -52,6 +52,29 @@ compiles each, and compares the emitted wire against pinned goldens. That
 cross-language bit-identity check is the property this project exists to
 provide, so a change that breaks it is wrong until proven otherwise.
 
+**`make test` refuses a missing pinned toolchain by name, and runs without a
+leg only when you name the skip on purpose.** A pin the Makefile names
+(`NODE`, `DART`, `JAVA`/`JAVAC`, `ELIXIR`/`MIX`/`ELIXIRC`, `DOTNET`) that does
+not resolve stops the run before it starts. Every registered leg is probed, so
+one run names every leg it would have skipped and the path each pin looked in
+rather than sending you back for the next name; to run the chain without those
+legs, name them in `SCHEMA_SKIP_LEGS`, which prints every skip by name and
+which the refusal spells out ready to paste:
+
+```bash
+make test SCHEMA_SKIP_LEGS=dart,java
+```
+
+A gate that passes while its toolchain is missing is a gate with no blade, and
+that is not a hypothetical here: a merge once deleted a clone's `dist` link,
+`make test` passed over four legs in silence, and a red inside one of them rode
+the green run (issue #599). Nothing under `.github` sets `SCHEMA_SKIP_LEGS`:
+certification installs every toolchain and overrides the pins, which resolve
+and pass the gate. `make toolchain` runs the gate alone, and
+`make toolchain-negative-control` proves it still has its blade, on every pin
+of every leg. That control runs inside `make test` and on every pull request,
+in the `go-test` job of `ci.yml`.
+
 The Makefile's `SERIALIZE*` variables override the sibling paths if you keep
 them elsewhere.
 
@@ -112,7 +135,7 @@ touches and two ports landing in one week do not conflict:
 | the target | `compiler/target_<lang>.go` (`target_javascript.go`: a `_js` suffix is a Go build constraint) | its `init` registers the generator |
 | the runtime's claimed names | `internal/tablenames/<lang>.go` | its `init` defines the backend's bit and names |
 | the compiler's tests | `compiler/tables<lang>_test.go` | the package |
-| the build | `make/<lang>.mk` | the Makefile's wildcard include; the file registers its `test-<lang>` leg, its conformance build, its bench unit and its goldens |
+| the build | `make/<lang>.mk` | the Makefile's wildcard include; the file registers its `test-<lang>` leg, its conformance build, its bench unit, its goldens and its pinned toolchain |
 | the conformance leg | `test/conformance/<lang>/driver` and `ci.json` | the harness discovers the driver; `harness matrix` builds the pull-request matrix from the rows |
 | the tables bench leg | `bench/tables/<lang>/leg` | `bench/tables/run.sh` runs every leg |
 | the shape gate's exemptions | `bench/<lang>/SHAPE-GATE.allow`, `bench/tables/<lang>/SHAPE-GATE.allow` | the gate reads every ledger under the tree |
@@ -124,6 +147,17 @@ of the tree and requires the harness, the CI matrix, the bench pass and the
 Makefile to find it with no shared file edited. If a port needs to edit a file
 that lists languages, that is a defect in the registry, not a step.
 
+A port with a pinned toolchain registers it in the same file and the same way:
+`TOOLCHAIN_LEGS += <lang>`, `TOOLCHAIN_PINS_<lang> :=` every pin the leg
+probes, and a `toolchain-<lang>` target carrying one
+`$(call toolchain_probe,...)` per pin. That is what makes `make test` refuse
+the leg by name instead of skipping it, and what makes
+`SCHEMA_SKIP_LEGS=<lang>` a skip anyone can read in the log. The negative
+control points each pin on that list at a path that does not exist IN TURN,
+with the leg's other pins pointed at one that resolves, and requires the
+refusal to name that pin. A pin the target probes but the list leaves out is
+a probe nothing watches, and deleting it keeps the control green.
+
 **Three shared edits are tolerated, and are the whole list.** The port's
 column on [PORTING.md](PORTING.md), the techniques register, is written by
 hand — every technique carried, cited or stated impossible — and its gate

make/checks/packet-arm-defaults.mk

@@ -106,8 +106,8 @@ packet-arm-defaults-cs-negative-control:
 	@cp test/cs/schematest.csproj $(ARM_NC_ROOT)/cs/schema/test/cs/
 	@cp test/cs/src/Program.cs $(ARM_NC_ROOT)/cs/schema/test/cs/src/
 	@ln -s "$(abspath $(SERIALIZE_CS))" $(ARM_NC_ROOT)/cs/serialize.cs
-	cd $(ARM_NC_ROOT)/cs/schema/test/cs && dotnet build --nologo -v quiet -o ../../bin/cs
-	$(call packet_arm_expect,cs,cd $(ARM_NC_ROOT)/cs/schema/test/cs && dotnet ../../bin/cs/schematest.dll,$(ARM_NC_MANAGED_FAILURE))
+	cd $(ARM_NC_ROOT)/cs/schema/test/cs && $(DOTNET) build --nologo -v quiet -o ../../bin/cs
+	$(call packet_arm_expect,cs,cd $(ARM_NC_ROOT)/cs/schema/test/cs && $(DOTNET) ../../bin/cs/schematest.dll,$(ARM_NC_MANAGED_FAILURE))
 
 packet-arm-defaults-java-negative-control:
 	$(call packet_arm_prepare,java,internal/codegen/java/functions.go,java,java)

make/cs.mk

@@ -6,6 +6,21 @@
 # test/cs/schematest.csproj and its ludicrous twin carry the same relative path
 SERIALIZE_CS ?= ../serialize.cs
 
+# The .NET SDK, pinned per project the way every other leg's toolchain is. The
+# SDK VERSION lives in .github/dotnet-version, which both workflows read and
+# internal/ci gates; there is no unpacked copy under dist/ because the SDK
+# installs itself into the machine, so this pin names the COMMAND and the
+# toolchain gate holds it to resolving. Point it at another SDK with
+# DOTNET=/path/to/dotnet.
+DOTNET ?= dotnet
+
+# THE TOOLCHAIN GATE, this leg's half (issue #599; the Makefile's header and
+# docs/CONTRIBUTING.md, "Adding a language"). The C# half of the block gates in
+# the Makefile reads the same pin, so a dotnet that does not resolve stops the
+# chain here rather than in the middle of a two-language control.
+.PHONY: toolchain-cs
+toolchain-cs:
+	@$(call toolchain_probe,cs,DOTNET,$(DOTNET))
 # Packet defaults consume the shared C++ oracle in both build modes.
 build/packet-defaults/cs/.stamp: bin/schema test/packet-defaults/Defaults.schema test/packet-defaults/Plain.schema make/cs.mk
 	./bin/schema generate --lang cs --out build/packet-defaults/cs/defaults test/packet-defaults/Defaults.schema
@@ -14,8 +29,8 @@ build/packet-defaults/cs/.stamp: bin/schema test/packet-defaults/Defaults.schema
 
 .PHONY: packet-defaults-cs packet-defaults-cs-negative-control
 packet-defaults-cs: build/packet-defaults/cs/.stamp packet-defaults-cpp
-	dotnet run --project test/packet-defaults/cs/packet-defaults.csproj -- testdata/wire/packet-defaults
-	dotnet run --configuration Release --project test/packet-defaults/cs/packet-defaults.csproj -- testdata/wire/packet-defaults
+	$(DOTNET) run --project test/packet-defaults/cs/packet-defaults.csproj -- testdata/wire/packet-defaults
+	$(DOTNET) run --configuration Release --project test/packet-defaults/cs/packet-defaults.csproj -- testdata/wire/packet-defaults
 
 packet-defaults-cs-negative-control: packet-defaults-cs
 	@mkdir -p build/packet-defaults/cs-negative
@@ -25,9 +40,9 @@ packet-defaults-cs-negative-control: packet-defaults-cs
 		"$(CURDIR)" "$(CURDIR)" > build/packet-defaults/cs-negative/overlay.json
 	go build -overlay=build/packet-defaults/cs-negative/overlay.json -o build/packet-defaults/cs-negative/schema ./cmd/schema
 	./build/packet-defaults/cs-negative/schema generate --lang cs --out build/packet-defaults/cs-negative/generated test/packet-defaults/Defaults.schema
-	dotnet build test/packet-defaults/cs/packet-defaults.csproj \
+	$(DOTNET) build test/packet-defaults/cs/packet-defaults.csproj \
 		-p:DefaultsDir="$(CURDIR)/build/packet-defaults/cs-negative/generated" -o build/packet-defaults/cs-negative/checker
-	@if dotnet build/packet-defaults/cs-negative/checker/packet-defaults.dll testdata/wire/packet-defaults > build/packet-defaults/cs-negative/log 2>&1; then \
+	@if $(DOTNET) build/packet-defaults/cs-negative/checker/packet-defaults.dll testdata/wire/packet-defaults > build/packet-defaults/cs-negative/log 2>&1; then \
 		echo 'NEGATIVE CONTROL FAILED: missing constructor bytes passed in C#'; exit 1; fi
 	@grep -Fq 'FAILED: packet-default constructor bytes' build/packet-defaults/cs-negative/log || \
 		{ echo 'NEGATIVE CONTROL FAILED: C# failed for another reason'; cat build/packet-defaults/cs-negative/log; exit 1; }
@@ -157,11 +172,11 @@ tables-cs-refuses-pointers: bin/schema
 # million mutants each is ~23 s — inside the 60 s this gate is allowed and ten
 # times the shared N. `make ... N=<n>` still overrides it.
 COOK_CS_N ?= 1000000
-COOK_CS := cd test/cs-cook && dotnet run --no-build --
+COOK_CS := cd test/cs-cook && $(DOTNET) run --no-build --
 
 .PHONY: build-cs-cook
 build-cs-cook: build/tables-generated-cs/.stamp
-	cd test/cs-cook && dotnet build -v q --nologo
+	cd test/cs-cook && $(DOTNET) build -v q --nologo
 
 .PHONY: tables-cook-open-cs
 tables-cook-open-cs: build-cs-cook build/schema_test_cook build/cook-open/.stamp build/cook-open-fixed/.stamp
@@ -185,8 +200,8 @@ tables-cook-open-cs: build-cs-cook build/schema_test_cook build/cook-open/.stamp
 	$(COOK_CS) usage Scene ../../build/cook-open/Scene.cook
 	$(COOK_CS) forge Scene ../../build/cook-open/Scene.cook
 	$(COOK_CS) forge Depot ../../build/cook-open/Depot.cook
-	cd test/cs-cook && SEED=$(SEED) N=$(if $(filter-out 100000,$(N)),$(N),$(COOK_CS_N)) dotnet run --no-build -- fuzz Scene ../../build/cook-open/Scene.cook
-	cd test/cs-cook && SEED=$(SEED) N=$(if $(filter-out 100000,$(N)),$(N),$(COOK_CS_N)) dotnet run --no-build -- fuzz TreeNode ../../build/cook-open/TreeNode.cook
+	cd test/cs-cook && SEED=$(SEED) N=$(if $(filter-out 100000,$(N)),$(N),$(COOK_CS_N)) $(DOTNET) run --no-build -- fuzz Scene ../../build/cook-open/Scene.cook
+	cd test/cs-cook && SEED=$(SEED) N=$(if $(filter-out 100000,$(N)),$(N),$(COOK_CS_N)) $(DOTNET) run --no-build -- fuzz TreeNode ../../build/cook-open/TreeNode.cook
 	$(COOK_CS) accept Scene ../../build/cook-open/Scene.cook
 	# THE BYTE-ORDER LEG's C# half, and it is HALF: a cook written --byte-order
 	# big is refused by the MAGIC here, which is the refusal the page promises.
@@ -226,15 +241,15 @@ define cook_open_cs_sabotage
 		"$(CURDIR)" "$(CURDIR)" > build/cook-open-cs-$(1)/overlay.json
 	@go build -overlay=build/cook-open-cs-$(1)/overlay.json -o build/cook-open-cs-$(1)/schema ./cmd/schema
 	./build/cook-open-cs-$(1)/schema generate --lang cs --out build/cook-open-cs-$(1)/gen tables/pointers
-	@if ( cd test/cs-cook && dotnet build -v q --nologo \
+	@if ( cd test/cs-cook && $(DOTNET) build -v q --nologo \
 			-p:CookGeneratedDir=../../build/cook-open-cs-$(1)/gen \
 			-p:BaseOutputPath=../../build/cook-open-cs-$(1)/bin/ \
 			-p:BaseIntermediateOutputPath=../../build/cook-open-cs-$(1)/obj/ \
 			> ../../build/cook-open-cs-$(1)/build.log 2>&1 ); then :; else \
 		echo "NEGATIVE CONTROL FAILED: the sabotaged emitter's output does not compile"; \
 		cat build/cook-open-cs-$(1)/build.log; exit 1; \
 	fi
-	@if ( cd test/cs-cook && dotnet run --no-build \
+	@if ( cd test/cs-cook && $(DOTNET) run --no-build \
 			-p:CookGeneratedDir=../../build/cook-open-cs-$(1)/gen \
 			-p:BaseOutputPath=../../build/cook-open-cs-$(1)/bin/ \
 			-p:BaseIntermediateOutputPath=../../build/cook-open-cs-$(1)/obj/ \
@@ -268,15 +283,15 @@ tables-cook-open-cs-walk-negative-control: build/cook-open/.stamp
 		"$(CURDIR)" "$(CURDIR)" > build/cook-open-cs-walk/overlay.json
 	@go build -overlay=build/cook-open-cs-walk/overlay.json -o build/cook-open-cs-walk/schema ./cmd/schema
 	./build/cook-open-cs-walk/schema generate --lang cs --out build/cook-open-cs-walk/gen tables/pointers
-	@if ( cd test/cs-cook && dotnet build -v q --nologo \
+	@if ( cd test/cs-cook && $(DOTNET) build -v q --nologo \
 			-p:CookGeneratedDir=../../build/cook-open-cs-walk/gen \
 			-p:BaseOutputPath=../../build/cook-open-cs-walk/bin/ \
 			-p:BaseIntermediateOutputPath=../../build/cook-open-cs-walk/obj/ \
 			> ../../build/cook-open-cs-walk/build.log 2>&1 ); then :; else \
 		echo "NEGATIVE CONTROL FAILED: the sabotaged emitter's output does not compile"; \
 		cat build/cook-open-cs-walk/build.log; exit 1; \
 	fi
-	@if ( cd test/cs-cook && ITERATIONS=10 dotnet run --no-build \
+	@if ( cd test/cs-cook && ITERATIONS=10 $(DOTNET) run --no-build \
 			-p:CookGeneratedDir=../../build/cook-open-cs-walk/gen \
 			-p:BaseOutputPath=../../build/cook-open-cs-walk/bin/ \
 			-p:BaseIntermediateOutputPath=../../build/cook-open-cs-walk/obj/ \
@@ -301,7 +316,7 @@ generated/bench/cs/.stamp: bin/schema $(SCHEMAS_BENCH)
 
 .PHONY: build-conformance-cs
 build-conformance-cs: build/tables-generated-cs/.stamp
-	cd test/conformance/cs && dotnet build -v q --nologo
+	cd test/conformance/cs && $(DOTNET) build -v q --nologo
 
 # THE NEGATIVE CONTROL FOR THE C# WALK (docs/SPEC-TABLES.md §16.5), and it is a
 # different sabotage from the C++ one above on purpose. That one flips a byte of
@@ -358,7 +373,7 @@ tables-cs-leg:
 # (a unit that generates but does not compile is issue #80's lesson), and the
 # packet tests.
 .PHONY: test-cs
-test-cs: build/tables-generated-cs/.stamp generated/bench/tables/cs/.stamp generated/cs/.stamp generated/cs-ludicrous/.stamp generated/bench/cs/.stamp
+test-cs: toolchain-cs build/tables-generated-cs/.stamp generated/bench/tables/cs/.stamp generated/cs/.stamp generated/cs-ludicrous/.stamp generated/bench/cs/.stamp
 	$(MAKE) tables-cs-json-walk
 	$(MAKE) tables-cs-standalone
 	$(MAKE) tables-cs-refuses-pointers
@@ -368,13 +383,15 @@ test-cs: build/tables-generated-cs/.stamp generated/bench/tables/cs/.stamp gener
 	$(MAKE) tables-cook-open-cs-lengths-negative-control
 	$(MAKE) tables-cook-open-cs-root-negative-control
 	$(MAKE) tables-cook-open-cs-walk-negative-control
-	dotnet build bench/tables/cs -c Release --nologo -v quiet
-	cd bench/cs && dotnet build -c Release --nologo -v quiet
-	cd test/cs && dotnet run
-	cd test/cs-ludicrous && dotnet run
+	$(DOTNET) build bench/tables/cs -c Release --nologo -v quiet
+	cd bench/cs && $(DOTNET) build -c Release --nologo -v quiet
+	cd test/cs && $(DOTNET) run
+	cd test/cs-ludicrous && $(DOTNET) run
 
 TEST_LEGS         += test-cs
-CONFORMANCE_LEGS  += build-conformance-cs build-cs-cook
+TOOLCHAIN_LEGS    += cs
+TOOLCHAIN_PINS_cs  := DOTNET
+CONFORMANCE_LEGS  += $(call unless_skipped,cs,build-conformance-cs build-cs-cook)
 BENCH_TABLES_LEGS += generated/bench/tables/cs/.stamp
 GOLDENS_LEGS      += update-goldens-cs
 # Packet UTF-8 content validation, including a compiled mutation control.
@@ -384,17 +401,17 @@ build/packet-text/cs/.stamp: bin/schema test/packet-text/Narrow.schema
 
 .PHONY: packet-utf8-cs packet-utf8-cs-negative-control
 packet-utf8-cs: build/packet-text/cs/.stamp build/packet-text/cpp/driver build/packet-text/harness
-	dotnet build test/packet-text/cs/packet-text.csproj -c Debug -o build/packet-text/cs/debug --nologo
+	$(DOTNET) build test/packet-text/cs/packet-text.csproj -c Debug -o build/packet-text/cs/debug --nologo
 	./build/packet-text/harness dotnet build/packet-text/cs/debug/packet-text.dll
-	dotnet build test/packet-text/cs/packet-text.csproj -c Release -o build/packet-text/cs/release --nologo
+	$(DOTNET) build test/packet-text/cs/packet-text.csproj -c Release -o build/packet-text/cs/release --nologo
 	./build/packet-text/harness dotnet build/packet-text/cs/release/packet-text.dll
 
 packet-utf8-cs-negative-control: packet-utf8-cs
 	@mkdir -p build/packet-text/cs-negative
 	go run ./tools/sabotage -name packet-utf8-cs-read -out build/packet-text/cs-negative/utf8.gotext internal/codegen/csharp/utf8.go
 	@printf '{"Replace":{"%s/internal/codegen/csharp/utf8.go":"%s/build/packet-text/cs-negative/utf8.gotext"}}\n' "$(CURDIR)" "$(CURDIR)" > build/packet-text/cs-negative/overlay.json
 	go run -overlay=build/packet-text/cs-negative/overlay.json ./cmd/schema generate --lang cs --out build/packet-text/cs-negative test/packet-text/Narrow.schema
-	dotnet build test/packet-text/cs/packet-text.csproj -c Release -o build/packet-text/cs-negative/bin -p:TextDir="$(CURDIR)/build/packet-text/cs-negative" --nologo
+	$(DOTNET) build test/packet-text/cs/packet-text.csproj -c Release -o build/packet-text/cs-negative/bin -p:TextDir="$(CURDIR)/build/packet-text/cs-negative" --nologo
 	@if ./build/packet-text/harness -mutations-only dotnet build/packet-text/cs-negative/bin/packet-text.dll > build/packet-text/cs-negative/log 2>&1; then echo 'NEGATIVE CONTROL FAILED: C# UTF-8 removal passed'; exit 1; fi
 	@grep -Fq 'FAILED: packet-text verdict on ' build/packet-text/cs-negative/log || { cat build/packet-text/cs-negative/log; exit 1; }
 	@echo 'packet UTF-8 C# negative control: removed read validation fails bit-flip agreement'
@@ -409,19 +426,19 @@ build/packet-wide/cs/.stamp: bin/schema build/packet-wide/source/WideText.schema
 
 .PHONY: packet-wide-cs packet-wide-cs-negative-control
 packet-wide-cs: build/packet-wide/cs/.stamp build/packet-wide/cpp/driver build/packet-text/harness
-	dotnet build test/packet-wide/cs/packet-wide.csproj -c Debug -o build/packet-wide/cs/debug --nologo
-	dotnet build/packet-wide/cs/debug/packet-wide.dll --contracts
+	$(DOTNET) build test/packet-wide/cs/packet-wide.csproj -c Debug -o build/packet-wide/cs/debug --nologo
+	$(DOTNET) build/packet-wide/cs/debug/packet-wide.dll --contracts
 	./build/packet-text/harness -wide -corpus testdata/conformance/text/wstring.txt -oracle build/packet-wide/cpp/driver dotnet build/packet-wide/cs/debug/packet-wide.dll
-	dotnet build test/packet-wide/cs/packet-wide.csproj -c Release -o build/packet-wide/cs/release --nologo
-	dotnet build/packet-wide/cs/release/packet-wide.dll --contracts
+	$(DOTNET) build test/packet-wide/cs/packet-wide.csproj -c Release -o build/packet-wide/cs/release --nologo
+	$(DOTNET) build/packet-wide/cs/release/packet-wide.dll --contracts
 	./build/packet-text/harness -wide -corpus testdata/conformance/text/wstring.txt -oracle build/packet-wide/cpp/driver dotnet build/packet-wide/cs/release/packet-wide.dll
 
 packet-wide-cs-negative-control: packet-wide-cs
 	@mkdir -p build/packet-wide/cs-negative
 	go run ./tools/sabotage -name packet-wide-cs-pairing -out build/packet-wide/cs-negative/wstring.gotext internal/codegen/csharp/wstring.go
 	@printf '{"Replace":{"%s/internal/codegen/csharp/wstring.go":"%s/build/packet-wide/cs-negative/wstring.gotext"}}\n' "$(CURDIR)" "$(CURDIR)" > build/packet-wide/cs-negative/overlay.json
 	go run -overlay=build/packet-wide/cs-negative/overlay.json ./cmd/schema generate --lang cs --out build/packet-wide/cs-negative build/packet-wide/source/WideText.schema
-	dotnet build test/packet-wide/cs/packet-wide.csproj -c Release -o build/packet-wide/cs-negative/bin -p:TextDir="$(CURDIR)/build/packet-wide/cs-negative" --nologo
+	$(DOTNET) build test/packet-wide/cs/packet-wide.csproj -c Release -o build/packet-wide/cs-negative/bin -p:TextDir="$(CURDIR)/build/packet-wide/cs-negative" --nologo
 	@if ./build/packet-text/harness -wide -corpus testdata/conformance/text/wstring.txt -oracle build/packet-wide/cpp/driver -mutations-only dotnet build/packet-wide/cs-negative/bin/packet-wide.dll > build/packet-wide/cs-negative/log 2>&1; then echo 'NEGATIVE CONTROL FAILED: C# wide pairing removal passed'; exit 1; fi
 	@grep -Fq 'FAILED: packet-text verdict on ' build/packet-wide/cs-negative/log || { cat build/packet-wide/cs-negative/log; exit 1; }
 	@echo 'packet wide C# negative control: removed pairing fails bit-flip agreement'