Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 14 additions & 5 deletions docs/adr/0017-definitions.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,11 +153,20 @@ datamk context -f cell.yaml --terms net_revenue,active_customer

Grammar, closed (ADR 0012 §7 / ADR 0013 §4): `/context` and
`/context/<route>` accept exactly `include` and `terms`; any other parameter
→ 400. `terms` is comma-separated; an empty value, an empty segment, a token
outside `[A-Za-z0-9_.-]`, or more than 64 tokens → 400. Repeated identical
tokens deduplicate. Matching is ASCII-case-insensitive over every term and
alias; a hit resolves to its canonical `term`, and two tokens resolving to
one term yield one entry.
→ 400. `terms` is comma-separated; each token is trimmed; an empty value, an
empty segment, a token outside `[A-Za-z0-9_.-]` plus space, or more than 64
tokens → 400. Repeated identical tokens deduplicate. Matching is on a
normalized key — ASCII-lowercased, every run of whitespace/`_`/`-` folded to
one `_` — over every term and alias, so `white paper`, `White-Paper` and
`white_paper` all hit a `white_paper` term; a hit resolves to its canonical
`term`, and two tokens resolving to one term yield one entry. Uniqueness
(§1) is checked on the same key.

*Amended 2026-10-15:* spaces were previously a 400. A caller — an agent
relaying a person's own words — asks for `white paper`, not `white_paper`;
rewriting the request on the client would have it search for a token the
person never said. The door now takes the words and the lookup folds the
separator, so the glossary's spelling is the glossary's concern alone.

**An unknown term is not an error.** A robot asking for three terms and
getting one wrong must keep the other two, and a 404 would push it back into
Expand Down
2 changes: 1 addition & 1 deletion docs/adr/0018-ossie-ingest.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ Additive; `datamk_context` stays 4. Ossie content carries `from: ossie`
| `/context` | `semantic_models[]`: name, description, file, dataset/bound/metric counts. Always present, `[]` without a source. |
| `/context/<route>` | `exports[].semantic[]`: every dataset bound to the route, with fields, `ai_context`, and per-field verification; the model's relationships touching it (other side named by route or `null`); metrics whose datasets are all bound to this route, in full; other metrics naming the dataset as `{name, model}` pointers. |
| `/context?model=<name>` | one semantic model in full (`semantic_model` block), composable with `terms`. |
| `/context?terms=…` | Ossie dataset, field and metric names plus `ai_context.synonyms` join the lookup index beside `definitions:`; hits land in `semantic_matches[]` `{token, kind, model, dataset, field, description}`. Synonyms outside `[A-Za-z0-9_.-]`/64 chars are kept in the document and warned at `sync` as not addressable. Collisions across models return every hit, each naming its model. |
| `/context?terms=…` | Ossie dataset, field and metric names plus `ai_context.synonyms` join the lookup index beside `definitions:`; hits land in `semantic_matches[]` `{token, kind, model, dataset, field, description}`. Synonyms outside `[A-Za-z0-9_.-]`/64 chars (interior spaces allowed since 2026-10-15; matching folds space/`_`/`-`, ADR 0017 §3) are kept in the document and warned at `sync` as not addressable. Collisions across models return every hit, each naming its model. |

`datamk context --model <name>` mirrors `?model=`. The top-level
`semantic` block on the document carries `synced_at`, `content_sha256`,
Expand Down
20 changes: 9 additions & 11 deletions src/catalog/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -399,22 +399,20 @@ fn sync_semantic(
Ok(())
}

/// ADR 0018 §7: a synonym outside `[A-Za-z0-9_.-]{1,64}` stays in the
/// document (never rejected — datamake ingests, never rewrites, Ossie) but
/// cannot join the `terms=` lookup index; warned here so an author notices
/// at sync time rather than wondering why a lookup silently misses.
/// ADR 0018 §7: a synonym outside the addressable grammar
/// (`context::is_addressable_token`: `[A-Za-z0-9_.-]{1,64}`, interior
/// spaces allowed) stays in the document (never rejected — datamake
/// ingests, never rewrites, Ossie) but cannot join the `terms=` lookup
/// index; warned here so an author notices at sync time rather than
/// wondering why a lookup silently misses.
fn warn_unaddressable_synonyms(doc: &crate::ossie::Document) {
let mut bad: Vec<String> = Vec::new();
let mut check = |ai: &Option<crate::ossie::AiContext>| {
let Some(crate::ossie::AiContext::Detail(d)) = ai else {
return;
};
for s in &d.synonyms {
let ok = !s.is_empty()
&& s.chars().count() <= 64
&& s.chars()
.all(|c| c.is_ascii_alphanumeric() || matches!(c, '_' | '.' | '-'));
if !ok {
if !crate::context::is_addressable_token(s) {
bad.push(s.clone());
}
}
Expand All @@ -434,8 +432,8 @@ fn warn_unaddressable_synonyms(doc: &crate::ossie::Document) {
if !bad.is_empty() {
let shown: Vec<&str> = bad.iter().take(5).map(String::as_str).collect();
eprintln!(
"Warning: {} ai_context.synonyms entries are not addressable ([A-Za-z0-9_.-]{{1,64}}) \
and will not join the `terms=` lookup index: {}{}",
"Warning: {} ai_context.synonyms entries are not addressable ([A-Za-z0-9_.-]{{1,64}}, \
spaces allowed inside) and will not join the `terms=` lookup index: {}{}",
bad.len(),
shown.join(", "),
if bad.len() > shown.len() { ", …" } else { "" }
Expand Down
4 changes: 3 additions & 1 deletion src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,9 @@ pub struct ContextArgs {
#[arg(long, value_name = "ROUTE")]
pub export: Option<String>,
/// Narrow `definitions[]`/`docs[]` to a comma-separated list of terms
/// or aliases (ADR 0017), resolved against the whole cell and composing
/// or aliases (ADR 0017), matched case-, space-, `_`- and `-`-
/// insensitively (quote a multi-word term: --terms "white paper"),
/// resolved against the whole cell and composing
/// with `--export` exactly as `?terms=` composes with
/// `/context/<route>`. Unlike the served door, an unknown term exits
/// non-zero, naming the known ones — a file written by `--out` cannot
Expand Down
9 changes: 6 additions & 3 deletions src/config/schema.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1574,7 +1574,9 @@ impl CellDef {

/// ADR 0017 §1: shape validation for `definitions[]` — term/alias
/// grammar, description length, cell-wide uniqueness (term and alias
/// names compared ASCII-case-insensitively across both sets), and the
/// names compared on `context::term_key` — case-, `_`- and
/// `-`-insensitive — across both sets, since that is the key `?terms=`
/// resolves on and two names sharing it would be unreachable), and the
/// per-cell/per-term caps. Does not resolve `applies_to` against the
/// interface (`validate_applies_to`'s job) — this runs before a
/// `discover:` cell's interface exists.
Expand Down Expand Up @@ -1612,11 +1614,12 @@ impl CellDef {
validate_term_shape(alias, &d.term)?;
}
for name in std::iter::once(&d.term).chain(d.aliases.iter()) {
let key = name.to_ascii_lowercase();
let key = crate::context::term_key(name);
if let Some(prev) = seen.get(&key) {
bail!(
"definitions '{prev}' and '{}' both use the name '{name}' \
(case-insensitive) — term and alias names must be unique cell-wide.",
(case-insensitive; `_` and `-` are the same separator) — term and \
alias names must be unique cell-wide.",
d.term
);
}
Expand Down
133 changes: 109 additions & 24 deletions src/context.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1444,17 +1444,54 @@ fn synonyms_of(ai: &Option<crate::ossie::AiContext>) -> &[String] {
}
}

/// Whether a token is addressable via `?terms=`/`--terms` (ADR 0018 §7):
/// `[A-Za-z0-9_.-]{1,64}`. A synonym outside this grammar is kept in
/// `semantic_model`'s prose but never indexed here — `datamk sync` warns
/// about it at ingest time.
/// Whether a token is addressable via `?terms=`/`--terms` (ADR 0018 §7,
/// amended 2026-10): `[A-Za-z0-9_.-]{1,64}` with interior spaces allowed,
/// so a multi-word business term (`white paper`) is addressable as people
/// actually say it — no leading/trailing whitespace. A synonym outside
/// this grammar is kept in `semantic_model`'s prose but never indexed here
/// — `datamk sync` warns about it at ingest time.
pub(crate) fn is_addressable_token(s: &str) -> bool {
!s.is_empty()
&& s.len() <= 64
&& s.trim() == s
&& s.bytes()
.all(|b| b.is_ascii_alphanumeric() || matches!(b, b'_' | b'.' | b'-' | b' '))
}

/// Whether a name may enter `?model=`'s OpenAPI enum: the `model` door's
/// own grammar (`[A-Za-z0-9_.-]{1,128}`, `serve::validate_context_query`),
/// which — unlike `terms` — admits no spaces. A spaced model name is still
/// addressable through `?terms=` (`is_addressable_token`); it just can't
/// be the one `?model=` selects.
pub(crate) fn is_model_token(s: &str) -> bool {
!s.is_empty()
&& s.len() <= 128
&& s.bytes()
.all(|b| b.is_ascii_alphanumeric() || matches!(b, b'_' | b'.' | b'-'))
}

/// The key `?terms=`/`--terms` lookups match on (ADR 0017 §3, amended
/// 2026-10): ASCII-lowercased, trimmed, with every run of whitespace, `_`
/// or `-` folded to one `_`. `white paper`, `White-Paper` and `white_paper`
/// are one key, so a caller can ask in the words people use and still hit
/// the `white_paper` term the glossary spells with an underscore.
pub(crate) fn term_key(s: &str) -> String {
let mut out = String::with_capacity(s.len());
let mut pending_sep = false;
for c in s.trim().chars() {
if c.is_whitespace() || c == '_' || c == '-' {
pending_sep = true;
continue;
}
if pending_sep {
out.push('_');
pending_sep = false;
}
out.push(c.to_ascii_lowercase());
}
out
}

/// Up to 8 entries of `known` nearest to `token` (case-insensitive, ASCII):
/// prefix matches first, then substring matches, then same-first-3-
/// characters matches, each tier only filling what the previous didn't —
Expand Down Expand Up @@ -1648,8 +1685,9 @@ pub fn resolve_semantic_matches(
) -> Vec<SemanticMatch> {
let mut out = Vec::new();
for tok in tokens {
let key = term_key(tok);
for entry in lookup {
if entry.token.eq_ignore_ascii_case(tok) {
if term_key(&entry.token) == key {
out.push(SemanticMatch {
token: tok.clone(),
kind: entry.kind.clone(),
Expand Down Expand Up @@ -2225,8 +2263,9 @@ impl ContextDocument {
true
}

/// `terms=` (ADR 0017 §3, extended by ADR 0018 §7): case-insensitive
/// lookup over `all_definitions`' terms/aliases AND every addressable
/// `terms=` (ADR 0017 §3, extended by ADR 0018 §7): `term_key`
/// (case-, space-, `_`-, `-`-insensitive) lookup over
/// `all_definitions`' terms/aliases AND every addressable
/// Ossie dataset/field/metric name or synonym (`self.semantic_lookup`).
/// A token may hit both; every Ossie hit is returned (collisions across
/// models are all listed, each naming its model). Resolves against the
Expand All @@ -2251,20 +2290,16 @@ impl ContextDocument {
) -> Vec<String> {
let mut index: IndexMap<String, &str> = IndexMap::new();
for d in all_definitions {
index
.entry(d.term.to_ascii_lowercase())
.or_insert(d.term.as_str());
index.entry(term_key(&d.term)).or_insert(d.term.as_str());
for alias in &d.aliases {
index
.entry(alias.to_ascii_lowercase())
.or_insert(d.term.as_str());
index.entry(term_key(alias)).or_insert(d.term.as_str());
}
}
let mut matched: Vec<String> = Vec::new();
let mut missing: Vec<String> = Vec::new();
let semantic_matches = resolve_semantic_matches(tokens, &self.semantic_lookup);
for tok in tokens {
let definitions_hit = index.get(&tok.to_ascii_lowercase());
let definitions_hit = index.get(&term_key(tok));
let ossie_hit = semantic_matches.iter().any(|m| &m.token == tok);
if let Some(term) = definitions_hit {
if !matched.iter().any(|t| t == term) {
Expand Down Expand Up @@ -5302,6 +5337,43 @@ interface:
);
}

/// ADR 0017 §3 (amended 2026-10): one key per business term however
/// a caller spells the word gap or case.
#[test]
fn term_key_folds_case_whitespace_underscore_and_hyphen() {
for s in [
"white paper",
"White Paper",
"white_paper",
"WHITE-PAPER",
" white paper ",
"white_-_paper",
] {
assert_eq!(term_key(s), "white_paper", "{s:?}");
}
assert_eq!(
term_key("net.revenue"),
"net.revenue",
"`.` is not a separator"
);
assert_eq!(term_key("nr"), "nr");
assert_eq!(term_key(" "), "");
assert!(is_addressable_token("white paper"));
assert!(
!is_addressable_token(" white paper"),
"no leading whitespace"
);
assert!(
!is_addressable_token("white paper "),
"no trailing whitespace"
);
assert!(
!is_addressable_token("white\tpaper"),
"only the ASCII space"
);
assert!(!is_model_token("white paper"), "`?model=` admits no spaces");
}

#[test]
fn terms_resolve_ossie_dataset_field_metric_and_synonym_with_cross_model_collisions() {
let def = sample_def_with_semantic();
Expand Down Expand Up @@ -5344,16 +5416,16 @@ interface:
assert!(hits[0].field.is_none());
}

/// M6: a name outside the addressable token grammar
/// (`[A-Za-z0-9_.-]{1,64}`) is not indexed for `?terms=`/the OpenAPI
/// enums — it still exists in the document (`semantic_model`/
/// `exports[].semantic`), just not in the lookup — mirroring what
/// already applied to synonyms before this fix reached dataset/field/
/// metric/model names too.
/// M6, amended 2026-10: a name with interior spaces IS addressable via
/// `?terms=` (people search in their own words), so it is indexed; a
/// name outside the grammar for another reason (here, a `!`) is not,
/// but still exists in the document (`semantic_model`/
/// `exports[].semantic`). `?model=`'s enum keeps the stricter
/// `is_model_token` grammar — exercised in `serve::openapi`'s tests.
#[test]
fn non_addressable_names_are_not_indexed_but_still_appear_in_the_document() {
let mut ds = ossie_dataset("has space", "orders_daily", &[], vec![]);
ds.fields.push(ossie_field("also bad", "id"));
ds.fields.push(ossie_field("also bad!", "id"));
let model = crate::ossie::SemanticModel {
name: "model with space".to_string(),
description: None,
Expand Down Expand Up @@ -5392,10 +5464,23 @@ interface:
};
let index = crate::ossie::bind::SemanticIndex::build(record, &[]);
let lookup = semantic_lookup_entries(&index);
assert!(lookup.is_empty(), "{lookup:?}");
let tokens: Vec<&str> = lookup.iter().map(|e| e.token.as_str()).collect();
assert_eq!(
tokens,
vec!["model with space", "has space", "metric bad"],
"{lookup:?}"
);
assert!(!is_model_token("model with space"));

// A spaced token resolves however the caller spells the gap.
for spelling in ["has space", "Has_Space", "HAS-SPACE", "has space"] {
let hits = resolve_semantic_matches(&[spelling.to_string()], &lookup);
assert_eq!(hits.len(), 1, "{spelling}: {hits:?}");
assert_eq!(hits[0].kind, "dataset");
}
assert!(resolve_semantic_matches(&["also bad!".to_string()], &lookup).is_empty());

// Still present in the model listing itself — not dropped, just
// not addressable.
// Still present in the model listing itself — not dropped.
assert_eq!(index.models()[0].name, "model with space");
assert_eq!(index.models()[0].datasets[0].name, "has space");
}
Expand Down
Loading
Loading