Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,17 +44,18 @@ A dataset is a named, single-portal, user-managed workspace meant to combine man
- Writes are atomic and paged downloads are resumable: pages append to a per-download segment with the cursor saved, then merge into the store as a new version with an atomic swap. An interrupted `get` picks up from its last cursor, a truncated pull is honestly marked incomplete, and queries only ever see complete, duplicate-free data.
- Datasets are purely local. `cc-data dataset delete` and `purge` only touch your disk.

Every `get`/`query`/`repl` names its dataset as `<portal>/<name>`; a bare `<name>` resolves under the optional `default_portal` config.
Every `get`/`query`/`repl` names its dataset as `<portal>/<name>`; a bare `<name>` resolves under the optional `default_portal` config. Both are hostnames: the environment aliases (`prod`/`staging`/`dev`) expand on `--portal`/`--server`, and are refused in a dataset ref and in `default_portal`, with the hostname to use named in the error. The portal there is also the on-disk folder, so expanding would give one dataset two names and taking it literally would file data under a portal that can never hold a credential. One parser owns every portal value, and it is what guarantees a portal is a single path component: `../wildfire` is refused rather than resolving outside your data root.

## Commands (sketch)

```
cc-data init # install the Claude skill + prompt for login
cc-data login --portal <portal> # browser loopback login; token stored per portal
cc-data logout # revoke the current token server-side
cc-data login [prod|staging|dev] # browser loopback login; the environment sets portal + server
cc-data login --portal <portal|env> # or name the portal directly; token stored per portal
cc-data logout --portal <portal|env> # revoke and remove that portal's stored token
cc-data auth status # stored credentials per portal + default_portal
cc-data reports list --portal <portal>
cc-data reports jobs <run-id> --portal <portal> # list a run's post-processing outputs
cc-data reports list --portal <portal|env>
cc-data reports jobs <run-id> --portal <portal|env> # list a run's post-processing outputs
cc-data dataset create|list|show|rename|edit|delete|purge|reindex ... # list/show take --json (and show --full)
cc-data get report <run-id> --dataset <portal>/<name> [--job <id>]
cc-data get answers <run-id> --dataset <portal>/<name>
Expand All @@ -77,7 +78,7 @@ Every command has real per-command help; `cc-data get answers --help` is the aut
- A presigned attachment URL (from `get attachments --url`) is a short-lived, credential-free capability to one student's file; don't paste it into shared or persistent channels. Prefer downloading into the dataset over `--url`.
- **Manual token entry:** on a headless or SSH host, `cc-data login --token -` reads the token from stdin (piped, or an echo-off prompt on a TTY) — the recommended manual form. The bare `--token <value>` form works but is discouraged: flag values land in shell history and process lists.
- **Dataset-folder trust boundary:** the DuckDB sandbox confines `query`/`repl` to the named dataset folders. On the bundled DuckDB the sandbox resolves symlinks, so this is a *writable-files* boundary, not a symlink escape: anyone who can write into a dataset folder can plant files your queries then read as trusted, so a dataset folder inherits the trust of whoever can write to it.
- **`--server` trust boundary:** `cc-data login --server <origin>` drives the entire login and token exchange against `<origin>`. The CLI enforces an allowlist (a `concord.org`/`concordqa.org` host or subdomain, or a loopback host; http only for loopback) so a social-engineered `--server` cannot capture your login. Widening it is deliberately a code change.
- **`--server` trust boundary:** `cc-data login --server <origin>` drives the entire login and token exchange against `<origin>`. The CLI enforces an allowlist (a `concord.org`/`concordqa.org` host or subdomain, or a loopback host; http only for loopback) so a social-engineered `--server` cannot capture your login. Widening it is deliberately a code change. The environment aliases (`prod`/`staging`/`dev`) expand to origins that are then checked against the same allowlist, so naming an environment is not a way around it.
- Claude never drives the browser login. On an expired or missing token the CLI emits a structured `NOT_AUTHENTICATED` error and a human runs `cc-data login`.

## Development
Expand Down
22 changes: 17 additions & 5 deletions cmd/auth.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,23 +53,35 @@ func renderStatus(res auth.StatusResult, check bool) {
return
}
tw := tabwriter.NewWriter(out, 0, 2, 2, ' ', 0)
// Portal and server sit next to each other: a run belongs to a portal *and*
// the report server behind it, so the pair is what identifies a login.
if check {
fmt.Fprintln(tw, "PORTAL\tBACKEND\tSTORED\tVALID\tLABEL\tLAST USED\tREPORT ACCESS")
fmt.Fprintln(tw, "PORTAL\tSERVER\tBACKEND\tSTORED\tVALID\tLABEL\tLAST USED\tREPORT ACCESS")
} else {
fmt.Fprintln(tw, "PORTAL\tBACKEND\tSTORED")
fmt.Fprintln(tw, "PORTAL\tSERVER\tBACKEND\tSTORED")
}
for _, p := range res.Portals {
if check {
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
p.Portal, p.Backend, fmtTime(p.StoredAt),
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",
p.Portal, serverText(p.Server), p.Backend, fmtTime(p.StoredAt),
validText(p), labelText(p.Label), lastUsedText(p), reportAccessText(p))
} else {
fmt.Fprintf(tw, "%s\t%s\t%s\n", p.Portal, p.Backend, fmtTime(p.StoredAt))
fmt.Fprintf(tw, "%s\t%s\t%s\t%s\n", p.Portal, serverText(p.Server), p.Backend, fmtTime(p.StoredAt))
}
}
tw.Flush()
}

// serverText renders a credential's recorded report server. A credential stored
// before the server was recorded has none, which is what makes its fetches fail;
// showing that as "-" is the point of the column.
func serverText(server string) string {
if server == "" {
return "-"
}
return server
}

func fmtTime(t time.Time) string {
if t.IsZero() {
return "-"
Expand Down
48 changes: 48 additions & 0 deletions cmd/auth_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
package cmd

import (
"bytes"
"strings"
"testing"
"time"

"github.com/concord-consortium/cc-data-cli/internal/auth"
"github.com/concord-consortium/cc-data-cli/internal/config"
"github.com/concord-consortium/cc-data-cli/internal/output"
)

// TestRenderStatusShowsServer pins the report server in the table, not only in
// --json. A run id belongs to a portal and to the server behind it, so this is
// the column that answers "am I pointed at the environment that has my run?".
func TestRenderStatusShowsServer(t *testing.T) {
res := auth.StatusResult{
Portals: []auth.PortalStatus{
{Portal: config.StagingPortal, Server: config.StagingServer, Backend: "keychain", StoredAt: time.Unix(0, 0)},
{Portal: config.DevPortal, Backend: "file", StoredAt: time.Unix(0, 0)},
},
}
for _, check := range []bool{false, true} {
var buf bytes.Buffer
restore := output.SetStreams(&buf, &buf)
renderStatus(res, check)
restore()

got := buf.String()
lines := strings.Split(strings.TrimSpace(got), "\n")
if len(lines) != 3 {
t.Fatalf("check=%v: want a header and two rows, got:\n%s", check, got)
}
if !strings.Contains(lines[0], "SERVER") {
t.Errorf("check=%v: header has no SERVER column: %q", check, lines[0])
}
if !strings.Contains(lines[1], config.StagingServer) {
t.Errorf("check=%v: staging row omits its server: %q", check, lines[1])
}
// A credential with no recorded server reads as "-" in the SERVER column
// (the second field), not merely somewhere on the line.
fields := strings.Fields(lines[2])
if len(fields) < 2 || fields[1] != "-" {
t.Errorf("check=%v: a serverless credential should show - in the SERVER column: %q", check, lines[2])
}
}
}
17 changes: 15 additions & 2 deletions cmd/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,22 @@ func loadRuntime() (*config.Config, string, error) {
return cfg, root, nil
}

// resolveRef parses a dataset ref against the configured default portal.
// resolveRef parses a dataset ref against the configured default portal. Only
// create uses it: a new dataset must name a portal the parser fully accepts.
func resolveRef(cfg *config.Config, raw string) (dataset.Ref, error) {
ref, err := dataset.ParseRef(raw, cfg.DefaultPortal)
ref, err := dataset.ParseRefForConfig(cfg, raw)
if err != nil {
return dataset.Ref{}, output.Usagef("%v", err)
}
return ref, nil
}

// resolveExistingRef is resolveRef for every command that names a dataset
// already on disk (show, delete, purge, rename, edit, reindex, get, query). It
// must reach anything dataset list shows, including a folder under a portal the
// parser would otherwise refuse, so the fix for such a folder is not rm -rf.
func resolveExistingRef(cfg *config.Config, dataRoot, raw string) (dataset.Ref, error) {
ref, err := dataset.ParseRefForExisting(cfg, dataRoot, raw)
if err != nil {
return dataset.Ref{}, output.Usagef("%v", err)
}
Expand Down
18 changes: 11 additions & 7 deletions cmd/dataset.go
Original file line number Diff line number Diff line change
Expand Up @@ -44,14 +44,18 @@ func newDatasetCreateCmd() *cobra.Command {
return err
}
} else {
if cfg.DefaultPortal == "" {
defaultPortal, err := cfg.DefaultPortalValue()
if err != nil {
return output.Usagef("%v", err)
}
if defaultPortal.IsZero() {
return output.Usagef("no dataset name given and no default_portal configured")
}
name, err := dataset.AutoName(root, cfg.DefaultPortal, description)
name, err := dataset.AutoName(root, defaultPortal, description)
if err != nil {
return err
}
ref = dataset.Ref{Portal: cfg.DefaultPortal, Name: name}
ref = dataset.Ref{Portal: defaultPortal, Name: name}
}
echoRef(ref)
if _, err := dataset.Create(root, ref, description); err != nil {
Expand All @@ -78,7 +82,7 @@ func newDatasetRenameCmd() *cobra.Command {
if err != nil {
return err
}
ref, err := resolveRef(cfg, args[0])
ref, err := resolveExistingRef(cfg, root, args[0])
if err != nil {
return err
}
Expand Down Expand Up @@ -112,7 +116,7 @@ func newDatasetEditCmd() *cobra.Command {
if err != nil {
return err
}
ref, err := resolveRef(cfg, args[0])
ref, err := resolveExistingRef(cfg, root, args[0])
if err != nil {
return err
}
Expand Down Expand Up @@ -143,7 +147,7 @@ func newDatasetDeleteCmd() *cobra.Command {
if err != nil {
return err
}
ref, err := resolveRef(cfg, args[0])
ref, err := resolveExistingRef(cfg, root, args[0])
if err != nil {
return err
}
Expand Down Expand Up @@ -178,7 +182,7 @@ func newDatasetPurgeCmd() *cobra.Command {
if err != nil {
return err
}
ref, err := resolveRef(cfg, args[0])
ref, err := resolveExistingRef(cfg, root, args[0])
if err != nil {
return err
}
Expand Down
2 changes: 1 addition & 1 deletion cmd/dataset_reindex.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ func newReindexCmd() *cobra.Command {
if err != nil {
return err
}
ref, err := resolveRef(cfg, args[0])
ref, err := resolveExistingRef(cfg, root, args[0])
if err != nil {
return err
}
Expand Down
2 changes: 1 addition & 1 deletion cmd/dataset_show.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ func newDatasetShowCmd() *cobra.Command {
if err != nil {
return err
}
ref, err := resolveRef(cfg, args[0])
ref, err := resolveExistingRef(cfg, root, args[0])
if err != nil {
return err
}
Expand Down
2 changes: 1 addition & 1 deletion cmd/get.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ func openDatasetForFetch(datasetRef string) (*dataset.Dataset, *api.Client, erro
if err != nil {
return nil, nil, err
}
ref, err := resolveRef(cfg, datasetRef)
ref, err := resolveExistingRef(cfg, root, datasetRef)
if err != nil {
return nil, nil, err
}
Expand Down
Loading
Loading