From bc3c8a4a2d94726b4b19c86cdc8dc13cc4467f78 Mon Sep 17 00:00:00 2001 From: David Tooke Date: Tue, 13 Oct 2026 09:15:39 -0400 Subject: [PATCH] Add pipeline recipes: other programs' output, relix into relix, loops, CI The Pipelines page covered jq, awk, find and xargs, and a one-line data check. It now also shows turning another program's columns into a table (a web log through awk), one relix into another to join tables two projects declare, tee to keep the rows while a second relix summarises them, a shell loop writing one file per value with a bound parameter, comparing two snapshots with EXCEPT and --fail-rows, and a GitHub Actions workflow that installs the latest release with gh and adds a Markdown table to the run's summary. Every new shell example runs in GuideExamplesTest, like the rest of the page. --- docs/guide/command-line/README.md | 2 +- docs/guide/command-line/index.json | 2 +- docs/guide/command-line/pipelines.md | 159 ++++++++++++++++++++++++++- 3 files changed, 159 insertions(+), 4 deletions(-) diff --git a/docs/guide/command-line/README.md b/docs/guide/command-line/README.md index 1aa3479..32c33c6 100644 --- a/docs/guide/command-line/README.md +++ b/docs/guide/command-line/README.md @@ -35,7 +35,7 @@ What a directory gives the commands run in it. | Page | What it covers | |---|---| -| [Pipelines](pipelines.md) | With `jq`, `awk`, `sort` and `head`; with `find` and `xargs`; scripts in here-documents; a data check in CI; optimising a script and reviewing the change | +| [Pipelines](pipelines.md) | With `jq`, `awk`, `sort` and `head`; another program's output as a table; one `relix` into another; `tee`; with `find`, `xargs` and shell loops; scripts in here-documents; data checks in CI, comparing snapshots, and a GitHub Actions workflow; optimising a script and reviewing the change | ## Command reference diff --git a/docs/guide/command-line/index.json b/docs/guide/command-line/index.json index 7830f5e..5f82316 100644 --- a/docs/guide/command-line/index.json +++ b/docs/guide/command-line/index.json @@ -6,7 +6,7 @@ {"path": "output.md", "chapter": "The command line", "title": "Output formats", "summary": "`table`, `tsv`, `csv`, `ndjson`, `json` and `markdown`; NULLs and headers; scripts with several queries; `--fail-empty` and `--fail-rows`; messages; closed pipes and interrupts"}, {"path": "catalog.md", "chapter": "Projects on the command line", "title": "The catalog", "summary": "`.relix/` directories and how they are found; trusting a project; a nearer declaration wins; inspecting the catalog; `--catalog`; `relixrc`"}, {"path": "secrets.md", "chapter": "Projects on the command line", "title": "Secrets and profiles", "summary": "`${NAME}` placeholders; `profiles.json` and `-P`; the environment and `-D`; private profile files; what is never printed"}, - {"path": "pipelines.md", "chapter": "Command-line recipes", "title": "Pipelines", "summary": "With `jq`, `awk`, `sort` and `head`; with `find` and `xargs`; scripts in here-documents; a data check in CI; optimising a script and reviewing the change"}, + {"path": "pipelines.md", "chapter": "Command-line recipes", "title": "Pipelines", "summary": "With `jq`, `awk`, `sort` and `head`; another program's output as a table; one `relix` into another; `tee`; with `find`, `xargs` and shell loops; scripts in here-documents; data checks in CI, comparing snapshots, and a GitHub Actions workflow; optimising a script and reviewing the change"}, {"path": "commands/relix.md", "chapter": "Command reference", "title": "relix", "summary": "The global options every command takes; the exit statuses; the environment variables and files", "command": "relix"}, {"path": "commands/run.md", "chapter": "Command reference", "title": "relix run", "summary": "Runs scripts and writes their queries' rows", "command": "relix run"}, {"path": "commands/check.md", "chapter": "Command reference", "title": "relix check", "summary": "Analyses scripts without running them, and reports their problems", "command": "relix check"}, diff --git a/docs/guide/command-line/pipelines.md b/docs/guide/command-line/pipelines.md index 8f4fe90..0b85f8d 100644 --- a/docs/guide/command-line/pipelines.md +++ b/docs/guide/command-line/pipelines.md @@ -1,8 +1,9 @@ # Pipelines `relix` reads text and writes text, so it works with the programs a shell already has. -This page is a set of recipes: `relix` with `jq` for JSON, with `find` and `xargs` for -many files, with `awk`, `sort` and `head` for its rows, and as a check in CI. +This page is a set of recipes: `relix` with `jq` for JSON, with `awk`, `sort` and `head` +for its rows and for other programs' output, with another `relix`, with `find`, `xargs` +and shell loops for many files, and as a check in CI. The rule that makes all of them work: **standard output carries rows and nothing else.** Messages go to standard error, the format in a pipe is tab-separated values unless `-o` @@ -50,6 +51,68 @@ completed 400 pending 50 ``` +### Another program's output, as a table + +Any program that prints columns can feed a query: turn its output into tab-separated +values with a header line, and bind standard input as `tsv`. Here `awk` splits a web +server's log into its method and path, and `relix` counts the requests to each path: + +```shell +printf 'GET /orders\nGET /customers\nGET /orders\nPOST /orders\n' > access.log +awk 'BEGIN { print "method\tpath" } { print $1 "\t" $2 }' access.log \ + | relix -i Hits=tsv:- -e 'SORT n DESC, path (GROUP path, COUNT(*) -> n (Hits))' +``` +``` +── ────────────────────────────────────────────────────────────── + path n + ────────── ─ + /orders 3 + /customers 1 +(2 rows) +``` + +The last stage writes to the terminal, so it prints a table. The column types are inferred +from the values, so `n` is a number to sort on rather than text. `--schema` gives them instead, as [Binding inputs](inputs.md#the-heading-of-an-input) +shows. + +## One relix into another + +A stage's rows are the next stage's input, so two runs of `relix` compose like any other +two programs. Each run reads the catalog of the directory it runs in, which makes a pipe +the way to join tables two projects declare. The `reports` project declares its own +`Orders`, a fixed sample; the shop's run joins it to the shop's `Customers`: + +```shell +(cd reports && relix -e 'Orders') | relix -i January=tsv:- -e 'PROJECT name, order_id, amount (Customers JOIN January)' -o csv +``` +``` +name,order_id,amount +Alice,101,500 +Dan,102,250 +``` + +The first run writes tab-separated values because its output is a pipe, and the second +reads them because `-i` says `tsv`. Within one project a view is the better tool: it is +one query, so the optimiser sees all of it. + +### Keep the rows, print a summary + +`tee` writes a copy of what passes through it to a file. Here the detail is kept in +`completed.csv`, and a second `relix` summarises it on the way past: + +```shell +relix -e 'Completed' -o csv | tee completed.csv | relix -i C=csv:- -e 'GROUP COUNT(*) -> orders, SUM(amount) -> total (C)' -o tsv +cat completed.csv +``` +``` +orders total +3 400 +order_id,customer_id,amount,status,placed +1,1,120,completed,2026-09-02 +2,1,80,completed,2026-09-14 +4,3,200,completed,2026-10-01 +``` + ## Stopping early When the reader of a pipe stops, `relix` stops too: it cancels the query rather than @@ -98,6 +161,32 @@ data/orders.csv 4 reports/sample-orders.csv 2 ``` +### One output file per value + +A shell loop runs a script once for each value another query lists. The first run prints +each city, one per line with no header; the loop runs the +[parameterised](inputs.md#a-name-value-query-parameters) `by-city.relix` for each one and +writes its customers to a file of its own: + +```shell +relix -e 'PROJECT city (Customers)' --no-header | while read -r city; do + relix -a city="$city" scripts/by-city.relix -o csv > "customers-$city.csv" +done +ls customers-*.csv +cat customers-London.csv +``` +``` +customers-Leeds.csv +customers-London.csv +customers-York.csv +name +Alice +Carol +``` + +The city goes in as a parameter, so a name with a quote or a space in it is a value to +compare with, never part of the script. + ### Check every script's formatting `relix fmt --check` prints the name of each file that is not formatted canonically, and @@ -168,6 +257,72 @@ A CI job runs in a fresh container where the project's `.relix/` cannot have bee trusted by hand; `RELIX_TRUST_ALL=1` trusts it, and a profile or the job's secret environment variables supply the [placeholders](secrets.md). +### Has anything changed since yesterday? + +`EXCEPT` keeps the rows of one input that are not in the other, so over two snapshots of +the same export it lists what is new. With `--fail-rows` the exit status says whether +there was anything: + +```shell +cp data/orders.csv yesterday.csv +{ cat data/orders.csv; echo '5,4,90,pending,2026-10-12'; } > today.csv +relix -i Before=yesterday.csv -i After=today.csv --fail-rows -e 'After EXCEPT Before' -o tsv; echo "exit $?" +``` +``` +order_id customer_id amount status placed +5 4 90 pending 2026-10-12 +exit 1 +``` + +Swap the two inputs to list the rows that went away. + +### In a workflow + +The checks above are each one line of a job, which is all a CI system asks of them. A +GitHub Actions workflow, for instance, installs the Linux archive of the latest release +with `gh`, which its runners carry, then runs the checks, and adds a table to the run's +summary page, a file that takes Markdown: + +```yaml +name: data checks +on: [push, pull_request] + +jobs: + check: + runs-on: ubuntu-latest + env: + RELIX_TRUST_ALL: "1" + GH_TOKEN: ${{ github.token }} + steps: + - uses: actions/checkout@v4 + - name: Install relix + run: | + gh release download -R DarkCollective/relix-cli -p 'relix-*-linux-x86_64.tar.gz' + tar xzf relix-*-linux-x86_64.tar.gz + echo "$PWD"/relix-*/bin >> "$GITHUB_PATH" + - name: Every script analyses + run: find scripts -name '*.relix' -print0 | xargs -0 relix check + - name: Every order has a positive amount + run: relix --fail-rows -e 'SELECT amount <= 0 (Orders)' -o tsv + - name: Big spenders + run: relix -e 'BigSpenders' -o markdown >> "$GITHUB_STEP_SUMMARY" +``` + +To pin a release instead of taking the latest, name its tag after `download`, as in +`gh release download v0.1.1 …`. The Markdown that last step appends is this: + +```shell +relix -e 'BigSpenders' -o markdown +``` +``` +### + +| name | total | +| --- | --- | +| Alice | 200 | +| Carol | 200 | +``` + ## Optimise a script, and review the change `relix optimize` prints a script with each query rewritten by the optimiser, so it works