Project commands for 1.0.0: init, profiles, source and connection

docs/command-examples/connection.md

@@ -0,0 +1,54 @@
+## Examples
+
+```shell
+relix connection ls -o tsv
+```
+```
+name	kind	location	file
+warehouse	jdbc	${WAREHOUSE_URL}	~/shop/.relix/catalog/20-warehouse.relix
+```
+
+Every option is a flag. A secret is a `${NAME}` placeholder, whose value comes from a
+profile or the environment when a query runs:
+
+```shell
+relix connection add archive database --url='${ARCHIVE_URL}' --user='${ARCHIVE_USER}' --password='${ARCHIVE_PASSWORD}'
+cat .relix/catalog/00-connection-archive.relix
+```
+```
+connection archive from jdbc { password: "${ARCHIVE_PASSWORD}", url: "${ARCHIVE_URL}", user: "${ARCHIVE_USER}" };
+```
+
+and a secret given as its value is refused, so it never reaches the catalog:
+
+```shell
+relix connection add scratch database --url=jdbc:h2:mem:scratch --password=hunter2 2>&1; echo "exit $?"
+```
+```
+relix: --password: a secret is written as a ${NAME} placeholder, never its value, which would stay in the catalog, your shell's history and ps; give --password='${NAME}' and set NAME with relix profiles set
+exit 2
+```
+
+`test` opens and closes each connection under the profile:
+
+```shell
+relix connection test -P dev -o tsv; echo "exit $?"
+```
+```
+name	status	problem
+archive	failed	${ARCHIVE_PASSWORD} has no value: set it in a profile (-P), the environment or -D
+warehouse	ok
+exit 5
+```
+
+```shell
+relix connection rm archive
+relix connection test -P dev -o tsv
+```
+```
+name	status	problem
+warehouse	ok
+```
+
+[The catalog](../catalog.md#declaring-sources-and-connections) covers where declarations
+go, and the trust a project keeps across them.

docs/command-examples/init.md

@@ -0,0 +1,29 @@
+## Examples
+
+```shell
+relix init ~/stall --root -P dev -o csv
+find ~/stall/.relix | sort
+cat ~/stall/.relix/relixrc
+```
+```
+~/stall/.relix
+~/stall/.relix/.gitignore
+~/stall/.relix/catalog
+~/stall/.relix/profiles.json
+~/stall/.relix/relixrc
+~/stall/.relix/root
+output = csv
+profile = dev
+```
+
+A directory that has a `.relix/` already is refused, and nothing in it changes:
+
+```shell
+relix init ~/stall 2>&1; echo "exit $?"
+```
+```
+relix: ~/stall/.relix already exists; relix init makes a new project and never changes one
+exit 2
+```
+
+[The catalog](../catalog.md#starting-a-project) covers what each file is for.

docs/command-examples/profiles.md

@@ -0,0 +1,36 @@
+## Examples
+
+```shell
+relix profiles ls -o tsv
+```
+```
+profile	variable	file
+dev	WAREHOUSE_PASSWORD	~/.relix/profiles.json
+dev	WAREHOUSE_URL	~/shop/.relix/profiles.json
+dev	WAREHOUSE_USER	~/shop/.relix/profiles.json
+```
+
+The value comes from standard input, or a prompt that does not echo it on a terminal,
+never from an argument:
+
+```shell
+printf 'jdbc:h2:mem:staging\n' | relix profiles set staging WAREHOUSE_URL
+relix profiles get staging WAREHOUSE_URL
+```
+```
+jdbc:h2:mem:staging
+```
+
+```shell
+relix profiles rm staging
+relix profiles ls -o tsv
+```
+```
+profile	variable	file
+dev	WAREHOUSE_PASSWORD	~/.relix/profiles.json
+dev	WAREHOUSE_URL	~/shop/.relix/profiles.json
+dev	WAREHOUSE_USER	~/shop/.relix/profiles.json
+```
+
+[Secrets and profiles](../secrets.md#setting-profile-values) covers where values come
+from, and what keeps them private.

docs/command-examples/source.md

@@ -0,0 +1,73 @@
+## Examples
+
+```shell
+relix source ls -o tsv
+```
+```
+name	kind	location	file
+Customers	csv	../../data/customers.csv	~/shop/.relix/catalog/00-sources.relix
+Orders	csv	../../data/orders.csv	~/shop/.relix/catalog/00-sources.relix
+Stock	warehouse	STOCK	~/shop/.relix/catalog/20-warehouse.relix
+```
+
+`add` writes one declaration to a file of its own. A relative `LOCATION` is written
+relative to that file, so it still names the same file from anywhere in the project, and
+`--infer-schema` reads the file for its columns:
+
+```shell
+relix source add People csv data/customers.csv --header --infer-schema
+cat .relix/catalog/00-source-People.relix
+```
+```
+source People from csv("../../data/customers.csv") { header: true, schema: { customer_id: number, name: string, city: string } };
+```
+
+`test` reads one row of each source under the profile, and checks the schema it declares
+against what it read:
+
+```shell
+relix source test -P dev -o tsv
+```
+```
+name	status	problem
+Customers	ok
+Orders	ok
+People	ok
+Stock	ok
+```
+
+A source another declaration reads is not removed without `--force`:
+
+```shell
+relix source rm Orders 2>&1; echo "exit $?"
+```
+```
+relix: Orders is read by
+  Completed (~/shop/.relix/catalog/10-views.relix:1)
+remove those first, or give --force
+exit 2
+```
+
+and one declared beside others in a file written by hand is reported, not changed:
+
+```shell
+relix source edit Customers --header=false 2>&1; echo "exit $?"
+```
+```
+relix: ~/shop/.relix/catalog/00-sources.relix:2 declares Customers beside other declarations, which relix source does not edit for you; relix source edit Customers --editor
+exit 2
+```
+
+```shell
+relix source rm People
+relix source ls -o tsv
+```
+```
+name	kind	location	file
+Customers	csv	../../data/customers.csv	~/shop/.relix/catalog/00-sources.relix
+Orders	csv	../../data/orders.csv	~/shop/.relix/catalog/00-sources.relix
+Stock	warehouse	STOCK	~/shop/.relix/catalog/20-warehouse.relix
+```
+
+[The catalog](../catalog.md#declaring-sources-and-connections) covers where declarations
+go, and the trust a project keeps across them.

docs/design.md

@@ -59,8 +59,10 @@ relix [GLOBAL OPTIONS] [RUN OPTIONS] [SCRIPT...]       # `run` is the default co
 | `relix ir` | the IR report | `--ir` / default |
 | `relix provenance` | annotated rows (`--semiring`, `--weight`) | `--provenance` |
 | `relix fmt` | the script, printed canonically (`--glyphs`/`--keywords`) | *new* |
+| `relix init [DIR]` | nothing: creates a project's `.relix/` (`--root`, `-o`, `-P`) and trusts it | *new* (§5.1) |
 | `relix catalog` | `ls`, `schema NAME`, `where NAME`, `files`, `trust` | *new* (§5) |
-| `relix profiles` | the named environments and where each came from | `relix env` |
+| `relix source`, `relix connection` | `add`, `ls`, `edit`, `rm`, `test`: declarations from flags (§5.3) | *new* |
+| `relix profiles` | `ls` (profile, variable, file; never a value), `set`, `get`, `rm` | `relix env` |
 | `relix drivers` | `ls`, `install NAME` | `--download-drivers` |
 | `relix connectors` | `ls`, `install NAME` | `relix connectors`, `--download-connectors` |
 | `relix doc [TOPIC]` | a reference page (paged on a TTY) | `relix help lang …` |
@@ -226,6 +228,12 @@ drains the query for its events and throws the rows away, because both wanted st
     catalog/*.relix
 ```
 
+`relix init [DIR]` creates a project's directory: `catalog/`, a `profiles.json` at mode
+`0600` (holding the profile `-P` names, empty), a `.gitignore` inside `.relix/` that
+holds `profiles.json`, and with `-o`/`-P` a `relixrc`, with `--root` the `root` marker.
+It records the new directory as trusted (§5.4), and refuses, with exit 2, a directory
+that already has a `.relix/`.
+
 ### 5.2 Discovery
 
 1. Walk from the working directory (or `-C DIR`) upward, collecting each `.relix/`
@@ -258,6 +266,17 @@ relix catalog schema Orders  # columns and types
 relix catalog where Orders   # the declaration that won, and the ones it shadowed
 ```
 
+`relix source` and `relix connection` (`add | ls | edit | rm | test`) declare data from
+flags: `add NAME KIND|CONNECTION [LOCATION]`, every other option a `--OPTION=VALUE` flag,
+`--infer-schema` to read a CSV file's or a table's columns. Each `add` writes one
+declaration to `catalog/00-source-<name>.relix` or `00-connection-<name>.relix`, as
+`relix fmt` prints it; one declaration per file is what makes `edit` and `rm` safe, and a
+declaration beside others is reported (file and line) rather than touched. A secret is
+written as a `${…}` placeholder, never a value. `rm` refuses a name another declaration
+reads unless `--force`. `test` reads one row per source (or opens each database
+connection) under the profile and exits 5 when one fails. `--global` writes `~/.relix`.
+A JSON source is schema-on-read and declares no schema, so it takes no `--infer-schema`.
+
 `catalog ls` and `schema` are queries over the engine's own introspection relations
 (`relix.relations`, `relix.columns`), run through the same row renderers, so they come
 out as TSV/NDJSON in a pipe for free.
@@ -275,7 +294,20 @@ a name it would have supplied. `RELIX_TRUST_ALL=1` exists for CI containers.
 
 What is trusted without asking is what the user chose: `~/.relix`, a script named on
 the command line, `-e`, stdin, and `--catalog` files. `relix catalog trust --list` and
-`--revoke DIR` manage the record. `profiles.json` (any level) is refused, as `ssh`
+`--revoke DIR` manage the record.
+
+A command that writes under a `.relix/` (`init`, `profiles set`/`rm`, `source` and
+`connection`) would untrust the project with its own change. So before writing it checks
+that the directory is trusted **and** its files still match the recorded hash; if both
+hold, it records the new hash after writing and the project stays trusted. Otherwise it
+stays untrusted after the write too, so a change someone else made is still reviewed
+before it loads. The command says which happened, on stderr.
+
+`relix profiles set PROFILE VAR` writes a value to the nearest project's `profiles.json`
+(or `~/.relix`'s with `--user`), creating it at `0600` and refusing a looser one. The value
+comes from a prompt that does not echo on a terminal, from stdin otherwise, and never
+from an argument. It warns when git tracks the file. `get` prints the value a run would
+resolve; `ls` lists names only. `profiles.json` (any level) is refused, as `ssh`
 refuses a loose private key, when it is readable by group or others, since it may hold
 credentials; the error names the `chmod` that fixes it. On Windows, where POSIX
 permission bits do not apply, this check is skipped.

docs/guide/command-line/README.md

@@ -28,8 +28,8 @@ What a directory gives the commands run in it.
 
 | Page | What it covers |
 |---|---|
-| [The catalog](catalog.md) | `.relix/` directories and how they are found; trusting a project; a nearer declaration wins; inspecting the catalog; `--catalog`; `relixrc` |
-| [Secrets and profiles](secrets.md) | `${NAME}` placeholders; `profiles.json` and `-P`; the environment and `-D`; private profile files; what is never printed |
+| [The catalog](catalog.md) | `.relix/` directories and how they are found; trusting a project; a nearer declaration wins; inspecting the catalog; `--catalog`; `relixrc`; starting a project with `relix init`; declaring sources and connections |
+| [Secrets and profiles](secrets.md) | `${NAME}` placeholders; `profiles.json` and `-P`; setting values with `relix profiles`; the environment and `-D`; private profile files; what is never printed |
 
 ## Command-line recipes
 
@@ -54,7 +54,11 @@ and examples.
 | [`relix ir`](commands/ir.md) | Prints the symbols, views and queries the analyser made of a script |
 | [`relix provenance`](commands/provenance.md) | Runs each query and annotates each row with where it came from |
 | [`relix fmt`](commands/fmt.md) | Prints scripts canonically, keeping their comments |
+| [`relix init`](commands/init.md) | Creates a project's `.relix/` directory, and trusts it |
 | [`relix catalog`](commands/catalog.md) | Lists the catalog's files, relations, columns and declarations; trusts a project |
+| [`relix source`](commands/source.md) | Declares, lists, changes, removes and tests the catalog's sources |
+| [`relix connection`](commands/connection.md) | Declares, lists, changes, removes and tests the catalog's connections |
+| [`relix profiles`](commands/profiles.md) | Lists, sets and removes the variables of the profiles |
 | [`relix drivers`](commands/drivers.md) | Lists and installs JDBC drivers |
 | [`relix connectors`](commands/connectors.md) | Lists and installs connector plugins |
 | [`relix doc`](commands/doc.md) | Shows a page of the language reference |

docs/guide/command-line/catalog.md

@@ -326,3 +326,86 @@ Bob
 Carol
 Dan
 ```
+
+## Starting a project
+
+`relix init` makes a project's `.relix/` directory, in the working directory or the one it
+names:
+
+```shell
+relix init ~/stall
+find ~/stall/.relix | sort
+```
+```
+~/stall/.relix
+~/stall/.relix/.gitignore
+~/stall/.relix/catalog
+~/stall/.relix/profiles.json
+```
+
+It writes an empty `catalog/`, a `profiles.json` only you can read, and a `.gitignore`
+inside `.relix/` that keeps `profiles.json` out of version control, so no file of yours
+outside the directory is touched. `-o FORMAT` and `-P NAME` write a `relixrc` with those
+defaults, and `--root` writes the `root` marker. Then it trusts the new directory, as
+`relix catalog trust` would, so the first run there loads it without a warning:
+
+```shell
+relix catalog trust --list -o tsv
+```
+```
+directory	state
+~/shop	trusted
+~/shop/reports	trusted
+~/stall	trusted
+```
+
+A directory that already has a `.relix/` is refused with exit status 2: `init` makes a
+new project and never changes one. Started outside your home directory, it warns that
+only the working directory's own `.relix/` is read there.
+
+## Declaring sources and connections
+
+[`relix source`](commands/source.md) and [`relix connection`](commands/connection.md)
+declare data without a catalog file written by hand. The two mirror the language's two
+declarations: a source is a relation, and a connection is what several sources share,
+such as a database and its credentials. Both take `add`, `ls`, `edit`, `rm` and `test`:
+
+```shell
+cd ~/stall
+relix connection add shop database --url='${SHOP_URL}' --user='${SHOP_USER}' --password='${SHOP_PASSWORD}'
+relix source add Orders csv ~/shop/data/orders.csv --infer-schema
+relix source ls -o tsv
+```
+```
+name	kind	location	file
+Orders	csv	~/shop/data/orders.csv	~/stall/.relix/catalog/00-source-Orders.relix
+```
+
+Each `add` writes one declaration to a file of its own, `catalog/00-source-NAME.relix` or
+`catalog/00-connection-NAME.relix`, printed as `relix fmt` prints it, and every option of
+the declaration is a flag: `--OPTION=VALUE`, or `--OPTION` alone for `true`. A secret is
+written as a `${NAME}` placeholder and never as its value, which the command refuses. It
+writes the nearest project's catalog, or with `--global` your own `~/.relix`. A name an
+outer level declares is allowed, as the overlay allows it, and warned about.
+
+One declaration to a file is what lets `edit` and `rm` change it safely. A declaration
+written by hand beside others in its file is reported with its file and line rather than
+touched; `edit NAME --editor` opens it in `$EDITOR` there. `rm` refuses a name another
+declaration reads, unless `--force`. Views and `def`s have no command: they are Relix
+code, and an editor is the tool for code.
+
+Changing a file under a trusted project would untrust it, so these commands, `init` and
+[`relix profiles`](secrets.md#setting-profile-values) look first: a project that is
+trusted, and unchanged since, is trusted again after the change, and any other stays
+untrusted, so a change someone else made is still reviewed. `relix catalog trust --list`
+shows the project still trusted:
+
+```shell
+relix catalog trust --list -o tsv
+```
+```
+directory	state
+~/shop	trusted
+~/shop/reports	trusted
+~/stall	trusted
+```