S12: the command-line guide, a page per command, every example run

.gitattributes

@@ -9,6 +9,10 @@ gradlew        text eol=lf
 *.ps1          text eol=crlf
 # Relix scripts: the tests compare what fmt and optimize write with them byte for byte.
 *.relix        text eol=lf
+# The command-line guide: its command pages are generated, and compared with the
+# generator's output byte for byte.
+docs/guide/command-line/**  text eol=lf
+docs/command-examples/**    text eol=lf
 
 # Binary files.
 *.jar          binary

.github/workflows/release.yml

@@ -1,5 +1,6 @@
 # Releases the command (design §7.3): a platform archive for each of macOS, Linux and
-# Windows, attached with their checksums to this repository's GitHub Release, and the
+# Windows, attached with their checksums to this repository's GitHub Release with the
+# command-line guide's archive (relix-<version>-docs.zip, design §7.4), and the
 # Homebrew formula and Scoop manifest that install them, pushed to the tap
 # (DarkCollective/homebrew-relix) and the bucket (DarkCollective/scoop-relix).
 #
@@ -89,6 +90,12 @@ jobs:
       - name: Build, package and test the archive
         shell: bash
         run: ./gradlew clean build releaseArchive packageTest -PcliVersion="$VERSION" $SKIP_PIN
+      # The command-line guide, the same on every platform, so it is built once. Its
+      # examples have just run in the build above.
+      - name: Build the command-line guide's archive
+        if: matrix.platform == 'linux-x86_64'
+        shell: bash
+        run: ./gradlew docsArchive -PcliVersion="$VERSION"
       - name: Check the archive is this platform's
         shell: bash
         run: |

CONTRIBUTING.md

@@ -189,6 +189,27 @@ because that is the interface a script or a pipeline depends on.
 A change to a command, an option or an exit status updates its help text, which is
 also what the manual page and the site's command reference are generated from.
 
+The command-line section of the programming guide is in
+[`docs/guide/command-line`](docs/guide/command-line/README.md); the site renders it,
+and each release attaches it as `relix-<version>-docs.zip`. Three things keep it true:
+
+- **The command pages are generated.** `docs/guide/command-line/commands/*.md` and
+  `index.json` come from the command's picocli model, followed by the hand-written
+  examples in `docs/command-examples/<command>.md`. After changing an option or one of
+  those files, run `./gradlew commandPages` and commit what it writes;
+  `GuideStructureTest` fails while they are stale.
+- **Every example runs.** A ```` ```shell ```` block is an example, and the plain block on
+  the line after it is what it prints on standard output; `GuideExamplesTest` runs it in
+  a copy of the fixture under `src/test/resources/docs-fixture` (the `~/shop` the pages
+  describe) and compares. An example exits 0 unless it ends `; echo "exit $?"`, which
+  prints its status for the page to show. Write commands a reader would type in a
+  ```` ```shell ```` block, never `bash` or `console`, and keep anything else — an install
+  command, a syntax summary — in a ```` ```text ```` block.
+- **Every link resolves**, to a page or a heading, here or in the engine's manuals:
+  the section sits at `docs/guide/command-line` in the engine's tree, so
+  `../secrets.md` is the guide's page and `../../reference/README.md` the language
+  reference. A new page is listed in the section's `README.md`.
+
 ## Questions
 
 Ask in [Discussions](https://github.com/DarkCollective/relix-cli/discussions).

README.md

@@ -13,6 +13,10 @@ It is a front end to the Relix engine,
 [relix-core](https://github.com/DarkCollective/relix-core), and reaches it only through
 the engine's published API.
 
+Its documentation is the command-line section of the Relix programming guide, in
+[`docs/guide/command-line`](docs/guide/command-line/README.md), published on the
+[Relix site](https://relix.darkcollective.com/guide/index.html).
+
 ## Status
 
 Under development. There is no release yet, and the command line described above is

build.gradle

@@ -57,6 +57,11 @@ sourceSets {
     }
     // The tests of the packaged command: they run its launcher, not its classes.
     packageTest
+    // CommandPagesTest regenerates the command pages to compare them with the checked-in ones.
+    test {
+        compileClasspath += sourceSets.packaging.output
+        runtimeClasspath += sourceSets.packaging.output
+    }
 }
 
 configurations {
@@ -175,6 +180,12 @@ test {
     dependsOn unpackEngineDocs
     inputs.dir(engineDocsDir).withPropertyName('engineDocs')
     systemProperty 'relix.engine.docs', engineDocsDir.get().asFile.absolutePath
+    // The command-line guide's pages, which the docs tests run and check: without these
+    // inputs a docs-only change would leave the tests UP-TO-DATE, passing without running.
+    inputs.dir('docs/guide/command-line').withPropertyName('guide')
+    inputs.dir('docs/command-examples').withPropertyName('commandExamples')
+    systemProperty 'relix.cli.guide', file('docs/guide/command-line').absolutePath
+    systemProperty 'relix.cli.commandExamples', file('docs/command-examples').absolutePath
     finalizedBy tasks.named('jacocoTestReport')
 }
 
@@ -383,6 +394,32 @@ def releaseFiles = tasks.register('releaseFiles', JavaExec) {
     } as CommandLineArgumentProvider)
 }
 
+// ── The command-line guide (design §7.4) ──────────────────────────────────────────────
+//
+//   ./gradlew commandPages      rewrites docs/guide/command-line/commands/*.md and
+//                               index.json from the command's picocli model
+//   ./gradlew docsArchive       build/distributions/relix-<version>-docs.zip, the
+//                               section the site renders in its programming guide
+tasks.register('commandPages', JavaExec) {
+    group = 'documentation'
+    description = 'Rewrites the command pages of docs/guide/command-line from the command model.'
+    classpath = sourceSets.packaging.runtimeClasspath
+    mainClass = 'com.darkcollective.relix.cli.packaging.CommandPages'
+    args = [projectDir.absolutePath]
+}
+
+tasks.register('docsArchive', Zip) {
+    group = 'documentation'
+    description = 'Builds the command-line guide archive the site renders from.'
+    from('docs/guide/command-line') {
+        into 'command-line'
+    }
+    archiveFileName = "relix-${version}-docs.zip"
+    destinationDirectory = layout.buildDirectory.dir('distributions')
+    preserveFileTimestamps = false
+    reproducibleFileOrder = true
+}
+
 def stageRelease = tasks.register('stageRelease', Sync) {
     description = 'Lays out the release archive\'s contents in build/release.'
     dependsOn tasks.named('jlink')

docs/command-examples/bundle.md

@@ -0,0 +1,10 @@
+## Examples
+
+A bundle is one JSON document per script, for a tool such as a query playground:
+
+```shell
+relix bundle -e 'SELECT amount > 100 (Orders)' | jq -c 'keys'
+```
+```
+["diagnostics","namespace","queries"]
+```

docs/command-examples/catalog.md

@@ -0,0 +1,44 @@
+## Examples
+
+```shell
+relix catalog ls -o tsv
+```
+```
+name	kind	arity	file
+BigSpenders	QR	2	~/shop/.relix/catalog/10-views.relix
+Completed	QR	5	~/shop/.relix/catalog/10-views.relix
+Customers	SRC	3	~/shop/.relix/catalog/00-sources.relix
+Orders	SRC	5	~/shop/.relix/catalog/00-sources.relix
+Regions	INL	2	~/.relix/catalog/00-regions.relix
+Spend	QR	2	~/shop/.relix/catalog/10-views.relix
+Stock	SRC	2	~/shop/.relix/catalog/20-warehouse.relix
+```
+
+```shell
+relix catalog schema Customers -o tsv
+```
+```
+column	type
+customer_id	N
+name	S
+city	S
+```
+
+```shell
+relix catalog where Orders -o tsv
+```
+```
+file	line	kind	state
+~/shop/.relix/catalog/00-sources.relix	7	source	wins
+```
+
+```shell
+relix catalog trust --list -o tsv
+```
+```
+directory	state
+~/shop	trusted
+~/shop/reports	trusted
+```
+
+[The catalog](../catalog.md) covers discovery, the overlay and trust.

docs/command-examples/check.md

@@ -0,0 +1,30 @@
+## Examples
+
+A script that analyses prints nothing, and exits 0:
+
+```shell
+relix check scripts/top-customers.relix
+```
+
+A problem is printed on standard error, and the status is 3:
+
+```shell
+relix check scripts/broken.relix 2>&1; echo "exit $?"
+```
+```
+scripts/broken.relix:1:30: error: Undefined relation: 'Ordrs'
+exit 3
+```
+
+`--json` writes the diagnostics to standard output instead, as one document for a tool to
+read:
+
+```shell
+relix check --json scripts/broken.relix; echo "exit $?"
+```
+```
+{"diagnostics":[{"file":"scripts/broken.relix","line":1,"column":30,"severity":"error","message":"Undefined relation: 'Ordrs'"}]}
+exit 3
+```
+
+[Pipelines](../pipelines.md#lint-every-script) checks every script in a tree.

docs/command-examples/completion.md

@@ -0,0 +1,14 @@
+## Examples
+
+The release archives carry each shell's script in `completions/`, and Homebrew installs
+them. To load one by hand, add the line for your shell to its startup file:
+
+```text
+bash        source <(relix completion bash)
+zsh         source <(relix completion zsh)
+fish        relix completion fish > ~/.config/fish/completions/relix.fish
+PowerShell  relix completion powershell | Out-String | Invoke-Expression
+```
+
+The zsh script is also a completion function, so it can be saved as `_relix` in a
+directory on `$fpath` instead.

docs/command-examples/connectors.md

@@ -0,0 +1,22 @@
+## Examples
+
+The connectors that ship with `relix`, and any plugins installed in `~/.relix/connectors`:
+
+```shell
+relix connectors ls
+```
+```
+── connectors ──────────────────────────────────────────────────────────────────
+ type    installed
+ ──────  ─────────
+ clf     true
+ csv     true
+ gedcom  true
+ log     true
+(4 rows)
+```
+
+`relix connectors ls --available` adds the connectors that can be installed, which it
+fetches from the hosted catalog, and `relix connectors install TYPE` installs one. A
+source whose type no installed connector reads — `mongodb`, say — fails with exit status
+5, unless `--allow-download` lets the run fetch the connector.

docs/command-examples/doc.md

@@ -0,0 +1,38 @@
+## Examples
+
+A page is found by its glyph, its keyword or its name:
+
+```shell
+relix doc --markdown σ | head -6
+```
+```
+# Name: Selection (σ / SELECT)
+
+# Syntax:
+σ <condition> (Relation)
+SELECT <condition> (Relation)
+```
+
+With no topic, `relix doc` lists the pages, as rows:
+
+```shell
+relix doc -o tsv | head -4
+```
+```
+topic	category	title	summary
+getting-started	guide	Getting started	Your first script — declare data, name a result, run it
+glossary	guide	Glossary	The vocabulary — relation, tuple, bag vs set, materialisation, pushdown
+σ	operator	Selection	Filter rows by a condition
+```
+
+`--function` lists, or looks up, the functions only:
+
+```shell
+relix doc --function -o tsv | grep -i '^round'
+```
+```
+Round	function	Round	Round to nearest
+```
+
+On a terminal a page is shown through `$PAGER` (`less` when it is not set); `--no-pager`
+writes it straight to standard output.

docs/command-examples/drivers.md

@@ -0,0 +1,20 @@
+## Examples
+
+```shell
+relix drivers ls
+```
+```
+── drivers ─────────────────────────────────────────────────────────────────────
+ name        scheme            installed
+ ──────────  ────────────────  ─────────
+ postgresql  jdbc:postgresql:  false
+ mysql       jdbc:mysql:       false
+(2 rows)
+```
+
+`relix drivers install NAME` downloads one of those drivers, checks its checksum, and
+installs it in `~/.relix/drivers`, or in `$RELIX_DRIVERS` when that is set. A run that
+needs a driver that is not installed fails with exit status 5, unless `--allow-download`
+lets it fetch the driver itself. The H2 driver is part of
+`relix`, and needs no installing. A driver `relix drivers ls` does not list is installed
+by copying its jar into the driver directory.

docs/command-examples/explain.md

@@ -0,0 +1,20 @@
+## Examples
+
+```shell
+relix explain -e 'GROUP city, COUNT(*) -> n (Customers JOIN Orders)'
+```
+```
+Aggregate
+└─ Join NATURAL/HASH build=RIGHT
+   ├─ Scan Customers
+   └─ Scan Orders
+```
+
+`--json` writes each query's plan as a JSON object instead:
+
+```shell
+relix explain --json -e 'SELECT amount > 100 (Orders)' | jq -c 'keys'
+```
+```
+["plan","query","script"]
+```

docs/command-examples/fmt.md

@@ -0,0 +1,61 @@
+## Examples
+
+```shell
+cat scripts/messy.relix
+```
+```
+-- Orders worth a look
+Large:={select amount>100(Orders)};
+query {project order_id,amount (Large)};
+```
+
+`relix fmt` prints it canonically — one statement per line, spaced, each operator in one
+spelling — keeping its comments:
+
+```shell
+relix fmt scripts/messy.relix
+```
+```
+-- Orders worth a look
+Large := { σ amount > 100 (Orders) };
+query { π order_id, amount (Large) };
+```
+
+`--keywords` writes the operators as ASCII keywords instead of glyphs:
+
+```shell
+relix fmt --keywords scripts/messy.relix
+```
+```
+-- Orders worth a look
+Large := { SELECT amount > 100 (Orders) };
+query { PROJECT order_id, amount (Large) };
+```
+
+`--check` changes nothing and names each file that would change, exiting 1 if there is
+one; `-w` rewrites each such file in place:
+
+```shell
+relix fmt --check --keywords scripts/messy.relix scripts/by-city.relix; echo "exit $?"
+```
+```
+scripts/messy.relix
+exit 1
+```
+
+```shell
+relix fmt -w --keywords scripts/messy.relix
+relix fmt --check --keywords scripts/messy.relix; echo "exit $?"
+```
+```
+exit 0
+```
+
+With no file, `relix fmt` formats standard input to standard output:
+
+```shell
+echo 'query {select amount>100(Orders)};' | relix fmt
+```
+```
+query { σ amount > 100 (Orders) };
+```