From 8e7162ea43c6c864ff245fd073fce1121bc0eb07 Mon Sep 17 00:00:00 2001 From: macanderson Date: Wed, 22 Jul 2026 13:31:41 -0700 Subject: [PATCH] =?UTF-8?q?feat(protocol):=20frame=20representations=20?= =?UTF-8?q?=E2=80=94=20full/compact/reference=20(CGEP=20phase=202)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extend ContextFrame so a frame states *how* it carries its content, per the CGEP lifecycle build prompt §"ContextFrame representations". Additive and backward-compatible: `representation` absent ⇒ `full`, and full/legacy frames are byte-identical on the wire. Satisfies frame_representation_witness.rs (both cases). Types (contextgraph-types): - Representation {full|compact|reference}, ContentFidelity, ContentRef, Transform, InlineContentRequirement. - ContextFrame: `content` → Option (absent for references); adds representation, content_fidelity, canonical_content_hash, content_ref, transform, minimum_content_fidelity, inline_content_requirement, canonical_token_cost, tokenizer_ref. `content_digest` kept as the inline-content hash (the spec's content_hash, which feeds FrameId); canonical_content_hash is the full-source hash. `full`/`reference` constructors + representation_invariants(). - Negotiation: ContextQuery.representation_preferences (+ select_representation); Capabilities.representations + resolve (+ representations_consistent). token_cost kept required (u32); canonical_token_cost/tokenizer_ref added additively. Making token_cost optional reopens PR #33's B3 budget-accounting decision on the same field, so it is deferred to that reconciliation. Wire + docs deliverables: - JSON Schema: new $defs + per-representation invariants (allOf if/then); `content` no longer globally required. - examples/reference-messages.json: representation-capable handshake_ack, a query with representation_preferences, and compact + reference frames (all validate). - docs/protocol-surface.md representation section; docs/adr/0005; CHANGELOG. Also unbreaks `main`: contextgraph-host and -conformance did not compile from a half-applied #37 merge (missing ConsentReceipt/EgressScope/FrameId/DropReason imports, a DataFlow literal missing egress_scopes, a CHECK_VERIFY_HONESTY test import, and a stale check-count assertion). Fixed so the workspace builds and the full suite passes. --- CHANGELOG.md | 22 + .../src/bin/contextgraph-example-docs.rs | 76 ++-- .../src/bin/contextgraph-inspect.rs | 1 + contextgraph-conformance/src/lib.rs | 7 +- .../tests/conformance_suite.rs | 7 +- .../tests/golden_fixtures.rs | 14 +- .../tests/stdio_roundtrip.rs | 28 +- .../tests/usage_report.rs | 1 + contextgraph-host/src/compose.rs | 16 +- contextgraph-host/src/host.rs | 17 +- contextgraph-host/src/http.rs | 12 +- contextgraph-host/src/provider.rs | 1 + contextgraph-host/src/stdio.rs | 12 +- contextgraph-host/src/wire.rs | 2 + contextgraph-types/src/capability.rs | 84 ++++ contextgraph-types/src/frame.rs | 403 ++++++++++++++++-- contextgraph-types/src/lib.rs | 5 +- contextgraph-types/src/query.rs | 94 +++- .../tests/frame_representation_witness.rs | 14 +- docs/adr/0005-frame-representations.md | 81 ++++ docs/protocol-surface.md | 58 ++- examples/reference-messages.json | 105 ++++- schema/contextgraph-envelope.schema.json | 118 ++++- 23 files changed, 1050 insertions(+), 128 deletions(-) create mode 100644 docs/adr/0005-frame-representations.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 69a55a3..e76bdb3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,20 @@ which. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1 ## [Unreleased] ### Added +- **Frame representations** on `ContextFrame` — `full` | `compact` | `reference` + (CGEP lifecycle phase 2). A frame now states *how* it carries its content: + `reference` frames carry no inline content, only a `content_ref` resolver + handle and a `canonical_content_hash`; `compact` frames inline a transformed + rendering alongside both. Additive and backward-compatible — `representation` + absent ⇒ `full`, and full/legacy frames are unchanged on the wire. Adds + `content_ref`, `canonical_content_hash`, `content_fidelity`, `transform`, + `minimum_content_fidelity`, `inline_content_requirement`, `canonical_token_cost`, + and `tokenizer_ref`; `content` becomes optional (absent for references). + Negotiated via `ContextQuery.representation_preferences` and + `Capabilities.representations` + `Capabilities.resolve`. Enforced in Rust + (`ContextFrame::representation_invariants`), the JSON Schema, and conformance + tests. See + [docs/adr/0005-frame-representations.md](./docs/adr/0005-frame-representations.md). - [`schema/contextgraph-envelope.schema.json`](./schema/contextgraph-envelope.schema.json) — a machine-readable JSON Schema (Draft 2020-12) for the Context Graph Protocol envelope and all wire types. Validates in any language (`ajv`, Python `jsonschema`, Rust @@ -39,6 +53,14 @@ which. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1 names (`contextgraph-graph`, `contextgraph-example-docs`). ### Fixed +- `contextgraph-host` and `contextgraph-conformance` did not compile from a + half-applied merge of #37 (egress-scope + consent receipts): `host.rs` used + `ConsentReceipt`/`EgressScope` without importing them and a `DataFlow` literal + omitted `egress_scopes`; the conformance crate used `FrameId`/`DropReason` + without importing them, a test omitted a `CHECK_VERIFY_HONESTY` import, and a + check-count assertion was stale (6, now 7). Restored so the workspace builds + and the full test suite passes. (Pre-existing on `main`; unrelated to frame + representations but required to build the branch.) - `docs/index.md`: removed dangling references to `PUBLISHING.md` and `RELEASING.md`, which do not exist in this repository. - `CONTRIBUTING.md`: commit-message examples and issue-tracker links no longer diff --git a/contextgraph-conformance/src/bin/contextgraph-example-docs.rs b/contextgraph-conformance/src/bin/contextgraph-example-docs.rs index 3fe7b16..ca6736f 100644 --- a/contextgraph-conformance/src/bin/contextgraph-example-docs.rs +++ b/contextgraph-conformance/src/bin/contextgraph-example-docs.rs @@ -17,7 +17,7 @@ use clap::{Parser, ValueEnum}; use contextgraph_host::wire::Envelope; use contextgraph_types::capability::QueryCapability; use contextgraph_types::{ - Capabilities, ContextFrame, ContextQueryResult, DataFlow, FrameKind, FrameVerdict, + Capabilities, ContextFrame, ContextQueryResult, DataFlow, EgressScope, FrameKind, FrameVerdict, PROTOCOL_VERSION, Provenance, ProviderInfo, Verdict, VerifyRequest, VerifyResponse, }; @@ -197,6 +197,9 @@ fn capabilities() -> Capabilities { // cannot watch its sources, so it does not advertise `subscribe` — the // exact provider shape verify exists for. verify: true, + // Serves inline full frames only; it does not resolve references. + representations: vec![], + resolve: false, } } @@ -255,62 +258,55 @@ fn canned_frames(misbehave: Option) -> Vec { }; vec![ - ContextFrame { - id: "frm_getting_started".into(), - kind: FrameKind::Doc, - title: "Getting Started".into(), - content: + { + let mut frame = ContextFrame::full( + "frm_getting_started", + FrameKind::Doc, + "Getting Started", "Install the reference binding with `cargo add contextgraph-types`, then implement \ - the four required methods." - .into(), - content_digest: Some(GETTING_STARTED_DIGEST.into()), - uri: Some("file:///docs/getting-started.md".into()), - score: if bad_score { 1.5 } else { 0.82 }, - token_cost, - valid_from: None, - valid_to: None, - recorded_at: None, - provenance: vec![Provenance { + the four required methods.", + if bad_score { 1.5 } else { 0.82 }, + token_cost, + ); + frame.content_digest = Some(GETTING_STARTED_DIGEST.into()); + frame.uri = Some("file:///docs/getting-started.md".into()); + frame.provenance = vec![Provenance { kind: "file".into(), uri: Some("file:///docs/getting-started.md".into()), range: Some("L1-40".into()), digest: None, method: None, by: Some("contextgraph-example-docs".into()), - }], - citation_label: Some(if empty_citation { + }]; + frame.citation_label = Some(if empty_citation { String::new() } else { "getting-started.md L1-40".into() - }), - embedding: None, - relations: vec![], + }); + frame }, - ContextFrame { - id: "frm_configuration".into(), - kind: FrameKind::Doc, - title: "Configuration".into(), - content: "Providers declare their data-flow direction at the handshake so hosts can \ - gate consent before sending any query." - .into(), - content_digest: Some(CONFIGURATION_DIGEST.into()), - uri: Some("file:///docs/configuration.md".into()), - score: 0.61, - token_cost, - valid_from: None, - valid_to: None, - recorded_at: None, - provenance: vec![Provenance { + { + let mut frame = ContextFrame::full( + "frm_configuration", + FrameKind::Doc, + "Configuration", + "Providers declare their data-flow direction at the handshake so hosts can \ + gate consent before sending any query.", + 0.61, + token_cost, + ); + frame.content_digest = Some(CONFIGURATION_DIGEST.into()); + frame.uri = Some("file:///docs/configuration.md".into()); + frame.provenance = vec![Provenance { kind: "file".into(), uri: Some("file:///docs/configuration.md".into()), range: Some("L1-25".into()), digest: None, method: None, by: Some("contextgraph-example-docs".into()), - }], - citation_label: Some("configuration.md L1-25".into()), - embedding: None, - relations: vec![], + }]; + frame.citation_label = Some("configuration.md L1-25".into()); + frame }, ] } diff --git a/contextgraph-conformance/src/bin/contextgraph-inspect.rs b/contextgraph-conformance/src/bin/contextgraph-inspect.rs index 674c4ff..b3bb48f 100644 --- a/contextgraph-conformance/src/bin/contextgraph-inspect.rs +++ b/contextgraph-conformance/src/bin/contextgraph-inspect.rs @@ -198,6 +198,7 @@ async fn fire_query(host: &Host, id: &str, goal: &str) { max_frames: 8, max_tokens: 4096, as_of: None, + representation_preferences: vec![], }; match host.query_provider(id, &query).await { Ok(result) => { diff --git a/contextgraph-conformance/src/lib.rs b/contextgraph-conformance/src/lib.rs index 2e02329..89289d1 100644 --- a/contextgraph-conformance/src/lib.rs +++ b/contextgraph-conformance/src/lib.rs @@ -35,9 +35,11 @@ //! fixture has `--misbehave` flags that trip each one, proving the suite //! catches a broken provider (task deliverable). -use contextgraph_host::{ConsentRecord, ContextProvider, Host, HostError, RawStdioConnection}; +use contextgraph_host::{ + ConsentRecord, ContextProvider, DropReason, Host, HostError, RawStdioConnection, +}; use contextgraph_types::{ - Capabilities, ConsentReceipt, ContextQuery, ContextQueryResult, Grantor, ProviderInfo, + Capabilities, ConsentReceipt, ContextQuery, ContextQueryResult, FrameId, Grantor, ProviderInfo, }; mod report; @@ -468,6 +470,7 @@ pub fn sample_query() -> ContextQuery { max_frames: 8, max_tokens: 4096, as_of: None, + representation_preferences: vec![], } } diff --git a/contextgraph-conformance/tests/conformance_suite.rs b/contextgraph-conformance/tests/conformance_suite.rs index 808468a..816e79f 100644 --- a/contextgraph-conformance/tests/conformance_suite.rs +++ b/contextgraph-conformance/tests/conformance_suite.rs @@ -5,7 +5,8 @@ use contextgraph_conformance::{ CHECK_BUDGET_HONESTY, CHECK_CONSENT_SCOPE, CHECK_FRAME_VALIDITY, CHECK_HANDSHAKE, - CHECK_MALFORMED, CHECK_SHUTDOWN, CheckStatus, ProviderTarget, run_conformance, + CHECK_MALFORMED, CHECK_SHUTDOWN, CHECK_VERIFY_HONESTY, CheckStatus, ProviderTarget, + run_conformance, }; /// Path to the fixture binary, built automatically for integration tests. @@ -37,8 +38,8 @@ async fn a_well_behaved_provider_is_fully_conformant() { "expected conformant; failures: {:?}", report.failures().collect::>() ); - // All six checks ran and passed (none skipped for a stdio provider). - assert_eq!(report.checks.len(), 6); + // All seven checks ran and passed (none skipped for a stdio provider). + assert_eq!(report.checks.len(), 7); for name in [ CHECK_HANDSHAKE, CHECK_CONSENT_SCOPE, diff --git a/contextgraph-conformance/tests/golden_fixtures.rs b/contextgraph-conformance/tests/golden_fixtures.rs index 43220b5..beeaf30 100644 --- a/contextgraph-conformance/tests/golden_fixtures.rs +++ b/contextgraph-conformance/tests/golden_fixtures.rs @@ -18,14 +18,26 @@ const FIXTURE_FILES: [&str; 5] = [ "normalization-vectors.json", "strict-validation.invalid.json", ]; -const FRAME_FIELDS: [&str; 14] = [ +// The pinned `contextgraph-1.0-draft` strict frame profile. `content_digest` is +// intentionally excluded here as it was before the representation work; the nine +// representation/cost fields below are additive and default-absent. +const FRAME_FIELDS: [&str; 23] = [ "id", "kind", "title", "content", "uri", + "representation", + "content_fidelity", + "canonical_content_hash", + "content_ref", + "transform", + "minimum_content_fidelity", + "inline_content_requirement", "score", "token_cost", + "canonical_token_cost", + "tokenizer_ref", "valid_from", "valid_to", "recorded_at", diff --git a/contextgraph-conformance/tests/stdio_roundtrip.rs b/contextgraph-conformance/tests/stdio_roundtrip.rs index 767d678..8f6b6a4 100644 --- a/contextgraph-conformance/tests/stdio_roundtrip.rs +++ b/contextgraph-conformance/tests/stdio_roundtrip.rs @@ -25,6 +25,7 @@ fn query(goal: &str) -> ContextQuery { max_frames: 8, max_tokens: 4096, as_of: None, + representation_preferences: vec![], } } @@ -133,22 +134,17 @@ impl ContextProvider for HealthyProvider { } async fn query(&self, _query: &ContextQuery) -> Result { Ok(ContextQueryResult { - frames: vec![ContextFrame { - id: "frm_healthy".into(), - kind: FrameKind::Doc, - title: "in-process frame".into(), - content: "still here".into(), - content_digest: None, - uri: None, - score: 0.5, - token_cost: 8, - valid_from: None, - valid_to: None, - recorded_at: None, - provenance: vec![], - citation_label: Some("in-process frame".into()), - embedding: None, - relations: vec![], + frames: vec![{ + let mut frame = ContextFrame::full( + "frm_healthy", + FrameKind::Doc, + "in-process frame", + "still here", + 0.5, + 8, + ); + frame.citation_label = Some("in-process frame".into()); + frame }], truncated: false, dropped_estimate: None, diff --git a/contextgraph-conformance/tests/usage_report.rs b/contextgraph-conformance/tests/usage_report.rs index 81bb0f7..6bd278f 100644 --- a/contextgraph-conformance/tests/usage_report.rs +++ b/contextgraph-conformance/tests/usage_report.rs @@ -25,6 +25,7 @@ fn query() -> ContextQuery { max_frames: 8, max_tokens: 4096, as_of: None, + representation_preferences: vec![], } } diff --git a/contextgraph-host/src/compose.rs b/contextgraph-host/src/compose.rs index 813b15e..8b06dba 100644 --- a/contextgraph-host/src/compose.rs +++ b/contextgraph-host/src/compose.rs @@ -81,7 +81,10 @@ fn render_frame(provider_id: &str, frame: &ContextFrame) -> String { provider = provider_id, id = frame.id, kind = frame_kind_name(frame.kind), - content = frame.content, + // A `reference` frame carries no inline content — it must be resolved + // (`context/resolve`, a later phase) before composition; here it renders + // as empty rather than fabricating bytes. + content = frame.content.as_deref().unwrap_or_default(), ) } @@ -95,11 +98,20 @@ mod tests { id: id.into(), kind: FrameKind::Doc, title: id.into(), - content: content.into(), + content: Some(content.into()), content_digest: digest.map(Into::into), uri: None, + representation: Default::default(), + content_fidelity: None, + canonical_content_hash: None, + content_ref: None, + transform: None, + minimum_content_fidelity: None, + inline_content_requirement: None, score: 0.5, token_cost: 10, + canonical_token_cost: None, + tokenizer_ref: None, valid_from: None, valid_to: None, recorded_at: None, diff --git a/contextgraph-host/src/host.rs b/contextgraph-host/src/host.rs index 0ad829c..517afcb 100644 --- a/contextgraph-host/src/host.rs +++ b/contextgraph-host/src/host.rs @@ -15,8 +15,8 @@ use std::collections::HashMap; use std::time::Duration; use contextgraph_types::{ - ContextFrame, ContextQuery, ContextQueryResult, DataFlow, FrameId, ProviderUsage, ServedFrame, - UsageReport, Verdict, VerifyRequest, + ConsentReceipt, ContextFrame, ContextQuery, ContextQueryResult, DataFlow, EgressScope, FrameId, + ProviderUsage, ServedFrame, UsageReport, Verdict, VerifyRequest, }; use crate::consent::{ConsentDecision, ConsentRecord, ConsentStore}; @@ -775,11 +775,20 @@ mod tests { id: id.into(), kind: FrameKind::Doc, title: id.into(), - content: "c".into(), + content: Some("c".into()), content_digest: None, uri: None, + representation: Default::default(), + content_fidelity: None, + canonical_content_hash: None, + content_ref: None, + transform: None, + minimum_content_fidelity: None, + inline_content_requirement: None, score: 0.5, token_cost: cost, + canonical_token_cost: None, + tokenizer_ref: None, valid_from: None, valid_to: None, recorded_at: None, @@ -800,6 +809,7 @@ mod tests { max_frames: 10, max_tokens: 1000, as_of: None, + representation_preferences: vec![], } } @@ -1184,6 +1194,7 @@ mod tests { reads: true, writes: false, egress: false, + egress_scopes: vec![], }, }) } diff --git a/contextgraph-host/src/http.rs b/contextgraph-host/src/http.rs index 70070d1..8bfe437 100644 --- a/contextgraph-host/src/http.rs +++ b/contextgraph-host/src/http.rs @@ -241,11 +241,20 @@ mod tests { id: "frm_h".into(), kind: FrameKind::Doc, title: "remote doc".into(), - content: "remote content".into(), + content: Some("remote content".into()), content_digest: None, uri: Some("https://example.test/doc".into()), + representation: Default::default(), + content_fidelity: None, + canonical_content_hash: None, + content_ref: None, + transform: None, + minimum_content_fidelity: None, + inline_content_requirement: None, score: 0.6, token_cost: 20, + canonical_token_cost: None, + tokenizer_ref: None, valid_from: None, valid_to: None, recorded_at: None, @@ -271,6 +280,7 @@ mod tests { max_frames: 5, max_tokens: 4000, as_of: None, + representation_preferences: vec![], } } diff --git a/contextgraph-host/src/provider.rs b/contextgraph-host/src/provider.rs index 029e05d..c41916c 100644 --- a/contextgraph-host/src/provider.rs +++ b/contextgraph-host/src/provider.rs @@ -118,6 +118,7 @@ mod tests { max_frames: 5, max_tokens: 1000, as_of: None, + representation_preferences: vec![], } } diff --git a/contextgraph-host/src/stdio.rs b/contextgraph-host/src/stdio.rs index 1666b34..7a7625e 100644 --- a/contextgraph-host/src/stdio.rs +++ b/contextgraph-host/src/stdio.rs @@ -454,11 +454,20 @@ mod tests { id: "frm_1".into(), kind: FrameKind::Doc, title: "README".into(), - content: "hello from a stdio provider".into(), + content: Some("hello from a stdio provider".into()), content_digest: None, uri: Some("file:///README.md".into()), + representation: Default::default(), + content_fidelity: None, + canonical_content_hash: None, + content_ref: None, + transform: None, + minimum_content_fidelity: None, + inline_content_requirement: None, score: 0.7, token_cost: 12, + canonical_token_cost: None, + tokenizer_ref: None, valid_from: None, valid_to: None, recorded_at: None, @@ -487,6 +496,7 @@ mod tests { max_frames: 5, max_tokens: 4000, as_of: None, + representation_preferences: vec![], } } diff --git a/contextgraph-host/src/wire.rs b/contextgraph-host/src/wire.rs index e02caeb..9c4e233 100644 --- a/contextgraph-host/src/wire.rs +++ b/contextgraph-host/src/wire.rs @@ -156,6 +156,7 @@ mod tests { max_frames: 1, max_tokens: 1, as_of: None, + representation_preferences: vec![], }, }, Envelope::Frames { @@ -202,6 +203,7 @@ mod tests { max_frames: 5, max_tokens: 2000, as_of: None, + representation_preferences: vec![], }; let env = Envelope::Query { query: query.clone(), diff --git a/contextgraph-types/src/capability.rs b/contextgraph-types/src/capability.rs index 9db5d59..6948f90 100644 --- a/contextgraph-types/src/capability.rs +++ b/contextgraph-types/src/capability.rs @@ -5,6 +5,7 @@ use serde::{Deserialize, Serialize}; +use crate::frame::Representation; use crate::scope::EgressScope; /// Declares what a provider does with data, so a host can gate consent @@ -91,6 +92,42 @@ pub struct Capabilities { /// neither. #[serde(default)] pub verify: bool, + /// The [frame representations](Representation) this provider can return + /// (build prompt §"Capability negotiation"). Empty ⇒ `full` only, the + /// legacy default, so a provider that says nothing keeps working. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub representations: Vec, + /// Whether this provider answers `context/resolve` for a frame's + /// [`content_ref`](crate::ContentRef). Compact/reference support **implies** + /// resolve support ([`representations_consistent`](Self::representations_consistent)). + #[serde(default)] + pub resolve: bool, +} + +impl Capabilities { + /// The representations this provider actually offers, defaulting to `[full]` + /// when it advertised none (the legacy contract). + pub fn offered_representations(&self) -> Vec { + if self.representations.is_empty() { + vec![Representation::Full] + } else { + self.representations.clone() + } + } + + /// Whether the advertised representations are honest: `compact`/`reference` + /// both hand the host a [`content_ref`](crate::ContentRef) to rehydrate, so + /// a provider that cannot [`resolve`](Self::resolve) must not advertise + /// them. A provider offering only `full` is always consistent. + pub fn representations_consistent(&self) -> bool { + if self.resolve { + return true; + } + !self + .representations + .iter() + .any(|rep| matches!(rep, Representation::Compact | Representation::Reference)) + } } #[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)] @@ -223,9 +260,56 @@ mod tests { embeddings_fingerprint: Some("bge-small-v1".into()), subscribe: false, verify: true, + representations: vec![Representation::Full, Representation::Reference], + resolve: true, }; let json = serde_json::to_string(&caps).unwrap(); let back: Capabilities = serde_json::from_str(&json).unwrap(); assert_eq!(back, caps); } + + #[test] + fn representation_capability_defaults_to_full_only_and_is_wire_omitted() { + // A provider that says nothing offers `full` only, and neither new + // field disturbs the pre-representation wire form (resolve defaults + // false and is a plain bool, representations omits when empty). + let caps = Capabilities::default(); + assert_eq!(caps.offered_representations(), vec![Representation::Full]); + assert!(caps.representations_consistent()); + let json = serde_json::to_string(&caps).unwrap(); + assert!(!json.contains("representations")); + + // Absent from the wire ⇒ still full-only, so pre-representation + // providers keep working. + let back: Capabilities = serde_json::from_str(r#"{"upsert":false}"#).unwrap(); + assert_eq!(back.offered_representations(), vec![Representation::Full]); + assert!(!back.resolve); + } + + #[test] + fn reference_or_compact_without_resolve_is_inconsistent() { + // Compact/reference hand the host a content_ref to rehydrate; a + // provider that cannot resolve must not advertise them. + let lying = Capabilities { + representations: vec![Representation::Reference], + resolve: false, + ..Capabilities::default() + }; + assert!(!lying.representations_consistent()); + + let honest = Capabilities { + representations: vec![Representation::Reference], + resolve: true, + ..Capabilities::default() + }; + assert!(honest.representations_consistent()); + + // Advertising only `full` never requires resolve. + let full_only = Capabilities { + representations: vec![Representation::Full], + resolve: false, + ..Capabilities::default() + }; + assert!(full_only.representations_consistent()); + } } diff --git a/contextgraph-types/src/frame.rs b/contextgraph-types/src/frame.rs index 506bdee..746396a 100644 --- a/contextgraph-types/src/frame.rs +++ b/contextgraph-types/src/frame.rs @@ -2,6 +2,16 @@ //! `docs/specs/stella-rust-cli/06-context-protocol.md` §3.4 fixes this exact //! shape; frames, never blobs, carry relevance, cost, and provenance so a //! budgeting, citing host can compose sources honestly. +//! +//! ## Frame representations (CGEP lifecycle, phase 2) +//! +//! A frame states how it carries its content through [`Representation`]: +//! `full` inlines the content (the legacy default), `compact` inlines a +//! transformed rendering alongside a resolver handle, and `reference` carries +//! no inline content at all — only a [`ContentRef`] and a +//! [`canonical_content_hash`](ContextFrame::canonical_content_hash) so a host +//! can rehydrate honestly and verifiably. `representation` absent means `full`, +//! so pre-representation providers and stored frames deserialize unchanged. use serde::{Deserialize, Serialize}; @@ -20,6 +30,88 @@ pub enum FrameKind { Graph, } +/// How a frame carries its content +/// (CGEP lifecycle build prompt, §"ContextFrame representations"). +/// +/// - `full`: canonical inline [`content`](ContextFrame::content) is required. +/// - `compact`: inline content, inline hash, canonical hash, a [`Transform`] +/// identity, and a [`ContentRef`] are all required. +/// - `reference`: inline content is **absent**; a [`ContentRef`] and +/// [`canonical_content_hash`](ContextFrame::canonical_content_hash) are +/// required; the inline content hash and transform are omitted. +/// +/// A `full` frame omits this field on the wire ([`is_full`](Self::is_full)) so +/// legacy frames round-trip byte-for-byte. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum Representation { + #[default] + Full, + Compact, + Reference, +} + +impl Representation { + /// Whether this is the legacy default representation. A `full` frame omits + /// the `representation` field on the wire, so a frame emitted before this + /// field existed round-trips unchanged and `representation` absent means + /// `full`. + pub fn is_full(&self) -> bool { + matches!(self, Representation::Full) + } +} + +/// The fidelity of a frame's carried content relative to its canonical source. +/// A missing value means `exact` for a legacy full frame. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum ContentFidelity { + Exact, + Normalized, + Summarized, + Omitted, +} + +/// Whether a frame's point of use requires the content inline, or accepts a +/// resolvable reference. Blocking constraints, guarded rules, ordered +/// procedures, and executable contracts require inline content at their point +/// of use; this keeps a reference choice from being confused with fidelity. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum InlineContentRequirement { + Required, + ResolvableReferenceAllowed, +} + +/// An opaque resolver handle for a `compact`/`reference` frame's content. +/// +/// [`ContextFrame::uri`] identifies the source resource; `ContentRef::uri` is a +/// **distinct** opaque resolver handle. A `ContentRef` also names the exact +/// [`provider_id`](Self::provider_id) that returned it, so a fan-out host routes +/// resolution back to that provider. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct ContentRef { + /// The exact provider that returned this frame; a fan-out host routes + /// `context/resolve` back to it. + pub provider_id: String, + /// Opaque resolver handle, distinct from [`ContextFrame::uri`]. + pub uri: String, + /// When the handle stops resolving. Absent ⇒ no declared expiry. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub expires_at: Option, +} + +/// The transformation identity a `compact` frame applies to its source to +/// produce the inline rendering, so a consumer knows what it is reading. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct Transform { + /// e.g. `extractive_summary`, `truncation`. + pub method: String, + /// e.g. `provider_default`, or a named implementation. + pub implementation: String, + pub version: String, +} + /// One link in a frame's provenance chain, ordered closest-to-source first. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub struct Provenance { @@ -67,20 +159,64 @@ pub struct ContextFrame { pub title: String, /// Text the host may quote into a prompt. Untrusted data: a conforming /// host delimits this as quoted material, never as instructions. - pub content: String, - /// The provider-declared digest of this frame's content bytes — the third - /// component of its stable [`FrameId`](crate::FrameId) identity, opaque to - /// the protocol (e.g. `sha256:`, matching the `provenance` digests). - /// Absent ⇒ the frame is not verifiable and a host re-queries it rather - /// than reusing it unchecked (`docs/context-reuse.md` §1, §4). + /// + /// Present for `full`/`compact` frames; **absent** for `reference` frames, + /// which carry only a [`content_ref`](Self::content_ref). A reference is + /// never encoded as `content: ""` — the field is omitted entirely. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub content: Option, + /// The provider-declared digest of this frame's **inline** content bytes — + /// the third component of its stable [`FrameId`](crate::FrameId) identity, + /// opaque to the protocol (e.g. `sha256:`). This is the spec's + /// `content_hash` (SHA-256 over the exact inline UTF-8 content) under its + /// established name; see [`canonical_content_hash`](Self::canonical_content_hash) + /// for the full-source hash. Absent ⇒ the frame is not verifiable and a + /// host re-queries it rather than reusing it unchecked + /// (`docs/context-reuse.md` §1, §4). A `reference` frame omits it. #[serde(default, skip_serializing_if = "Option::is_none")] pub content_digest: Option, #[serde(default, skip_serializing_if = "Option::is_none")] pub uri: Option, + /// How this frame carries its content. Absent ⇒ [`Representation::Full`], + /// so legacy frames deserialize unchanged and full frames omit the field. + #[serde(default, skip_serializing_if = "Representation::is_full")] + pub representation: Representation, + /// Fidelity of the carried content relative to the source. Absent ⇒ `exact` + /// for a legacy full frame. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub content_fidelity: Option, + /// SHA-256 over the **complete source content** bytes (distinct from the + /// inline [`content_digest`](Self::content_digest)). Required for + /// `compact`/`reference` frames so a resolved rehydration is verifiable. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub canonical_content_hash: Option, + /// The opaque resolver handle for a `compact`/`reference` frame's content. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub content_ref: Option, + /// The transformation a `compact` frame applied to its source. Omitted for + /// `full`/`reference` frames. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub transform: Option, + /// The lowest content fidelity acceptable at this frame's point of use. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub minimum_content_fidelity: Option, + /// Whether this frame's point of use requires inline content. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub inline_content_requirement: Option, /// Provider-normalized relevance in `[0, 1]`. pub score: f32, - /// Honest, conformance-audited token cost. + /// Honest, conformance-audited token cost of the **inline** rendering. pub token_cost: u32, + /// Token cost of the complete canonical source content, when the provider + /// declares it. If present, [`tokenizer_ref`](Self::tokenizer_ref) SHOULD + /// name the tokenizer it was measured with. Hosts compute model-specific + /// costs when providers omit it. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub canonical_token_cost: Option, + /// Identifies the tokenizer that produced the declared costs + /// (e.g. `openai:o200k_base`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub tokenizer_ref: Option, #[serde(default, skip_serializing_if = "Option::is_none")] pub valid_from: Option, #[serde(default, skip_serializing_if = "Option::is_none")] @@ -98,6 +234,70 @@ pub struct ContextFrame { } impl ContextFrame { + /// A `full`-representation frame carrying inline `content` — the shape every + /// legacy provider emits. The representation/cost/resolver fields default to + /// absent, so a call site need only supply the core, then set extras as + /// needed (the build prompt asks for constructors to reduce source + /// breakage). + pub fn full( + id: impl Into, + kind: FrameKind, + title: impl Into, + content: impl Into, + score: f32, + token_cost: u32, + ) -> Self { + Self { + id: id.into(), + kind, + title: title.into(), + content: Some(content.into()), + content_digest: None, + uri: None, + representation: Representation::Full, + content_fidelity: None, + canonical_content_hash: None, + content_ref: None, + transform: None, + minimum_content_fidelity: None, + inline_content_requirement: None, + score, + token_cost, + canonical_token_cost: None, + tokenizer_ref: None, + valid_from: None, + valid_to: None, + recorded_at: None, + provenance: Vec::new(), + citation_label: None, + embedding: None, + relations: Vec::new(), + } + } + + /// A `reference`-representation frame: no inline content, only a resolver + /// handle and the canonical source hash for honest, verifiable rehydration. + /// `token_cost` is the inline cost (0 — nothing is inlined). + pub fn reference( + id: impl Into, + kind: FrameKind, + title: impl Into, + content_ref: ContentRef, + canonical_content_hash: impl Into, + score: f32, + ) -> Self { + Self { + representation: Representation::Reference, + content: None, + content_ref: Some(content_ref), + canonical_content_hash: Some(canonical_content_hash.into()), + ..Self::full(id, kind, title, String::new(), score, 0) + } + // `..full(..)` seeds every other field; the overrides above make this a + // structurally honest reference (content absent, content_digest None, + // transform None) that satisfies `representation_invariants`. + } + /// Score must be normalized into `[0, 1]` per the protocol contract. /// Conformance suites assert this; providers should self-check too. pub fn has_valid_score(&self) -> bool { @@ -111,6 +311,62 @@ impl ContextFrame { pub fn identity(&self, provider_id: impl Into) -> FrameId { FrameId::new(provider_id, self.id.clone(), self.content_digest.clone()) } + + /// Whether this frame's fields satisfy the invariants of its declared + /// [`representation`](Self::representation). Providers emit conforming + /// frames; hosts reject a frame that lies about its shape (e.g. a + /// `reference` carrying inline content, or a `compact` missing its + /// canonical hash). The `Err` string names the exact violation. + pub fn representation_invariants(&self) -> Result<(), String> { + match self.representation { + Representation::Full => { + if self.content.is_none() { + return Err("full frame requires inline content".into()); + } + } + Representation::Compact => { + if self.content.is_none() { + return Err("compact frame requires inline content".into()); + } + if self.content_digest.is_none() { + return Err( + "compact frame requires an inline content hash (content_digest)".into(), + ); + } + if self.canonical_content_hash.is_none() { + return Err("compact frame requires canonical_content_hash".into()); + } + if self.transform.is_none() { + return Err("compact frame requires a transform identity".into()); + } + if self.content_ref.is_none() { + return Err("compact frame requires content_ref".into()); + } + } + Representation::Reference => { + // "Never encode a reference as content: \"\"" — any inline + // content, empty or not, is a violation. + if self.content.is_some() { + return Err("reference frame must not carry inline content".into()); + } + if self.content_ref.is_none() { + return Err("reference frame requires content_ref".into()); + } + if self.canonical_content_hash.is_none() { + return Err("reference frame requires canonical_content_hash".into()); + } + if self.content_digest.is_some() { + return Err( + "reference frame must omit the inline content hash (content_digest)".into(), + ); + } + if self.transform.is_some() { + return Err("reference frame must omit transform".into()); + } + } + } + Ok(()) + } } #[cfg(test)] @@ -118,30 +374,27 @@ mod tests { use super::*; fn sample_frame() -> ContextFrame { - ContextFrame { - id: "frm_1".into(), - kind: FrameKind::Snippet, - title: "workspace.ts L120-160".into(), - content: "export interface Workspace { ... }".into(), - content_digest: Some("sha256:abc".into()), + let mut frame = ContextFrame::full( + "frm_1", + FrameKind::Snippet, + "workspace.ts L120-160", + "export interface Workspace { ... }", + 0.83, + 412, + ); + frame.content_digest = Some("sha256:abc".into()); + frame.uri = Some("file:///repo/workspace.ts".into()); + frame.recorded_at = Some("2026-07-10T00:00:00Z".into()); + frame.provenance = vec![Provenance { + kind: "file".into(), uri: Some("file:///repo/workspace.ts".into()), - score: 0.83, - token_cost: 412, - valid_from: None, - valid_to: None, - recorded_at: Some("2026-07-10T00:00:00Z".into()), - provenance: vec![Provenance { - kind: "file".into(), - uri: Some("file:///repo/workspace.ts".into()), - range: Some("L120-160".into()), - digest: Some("sha256:abc".into()), - method: None, - by: None, - }], - citation_label: Some("workspace.ts L120-160".into()), - embedding: None, - relations: vec![], - } + range: Some("L120-160".into()), + digest: Some("sha256:abc".into()), + method: None, + by: None, + }]; + frame.citation_label = Some("workspace.ts L120-160".into()); + frame } #[test] @@ -188,4 +441,94 @@ mod tests { assert!(!json.contains("\"provenance\"")); assert!(!json.contains("\"content_digest\"")); } + + #[test] + fn full_frame_omits_representation_on_the_wire() { + // A full frame keeps the legacy wire shape: `representation` is absent, + // so pre-representation consumers see no new field. + let frame = sample_frame(); + assert_eq!(frame.representation, Representation::Full); + let json = serde_json::to_string(&frame).unwrap(); + assert!( + !json.contains("representation"), + "full frames must omit the representation field: {json}" + ); + assert!(frame.representation_invariants().is_ok()); + } + + #[test] + fn reference_frame_omits_content_and_round_trips_its_handle() { + let frame = ContextFrame::reference( + "frm_ref_1", + FrameKind::Doc, + "Deployment runbook", + ContentRef { + provider_id: "provider_example".into(), + uri: "context://provider_example/records/doc_runbook_v1".into(), + expires_at: None, + }, + "sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", + 0.9, + ); + frame + .representation_invariants() + .expect("constructed reference frame must be structurally honest"); + + let json = serde_json::to_string(&frame).unwrap(); + assert!( + !json.contains("\"content\""), + "a reference frame must not carry inline content: {json}" + ); + assert!(json.contains("\"representation\":\"reference\"")); + + let back: ContextFrame = serde_json::from_str(&json).unwrap(); + assert_eq!(back, frame); + assert_eq!(back.representation, Representation::Reference); + assert_eq!( + back.content_ref.as_ref().unwrap().provider_id, + "provider_example" + ); + } + + #[test] + fn a_reference_with_inline_content_violates_its_invariants() { + let mut frame = ContextFrame::reference( + "frm_ref_2", + FrameKind::Doc, + "Runbook", + ContentRef { + provider_id: "p".into(), + uri: "context://p/r".into(), + expires_at: None, + }, + "sha256:aa", + 0.5, + ); + // Even an empty string is a lie for a reference. + frame.content = Some(String::new()); + assert!(frame.representation_invariants().is_err()); + } + + #[test] + fn compact_frame_requires_its_full_metadata_set() { + let mut frame = sample_frame(); + frame.representation = Representation::Compact; + // full()-seeded frame lacks the compact metadata → invalid. + assert!(frame.representation_invariants().is_err()); + + frame.content_digest = Some("sha256:inline".into()); + frame.canonical_content_hash = Some("sha256:canonical".into()); + frame.transform = Some(Transform { + method: "extractive_summary".into(), + implementation: "provider_default".into(), + version: "1".into(), + }); + frame.content_ref = Some(ContentRef { + provider_id: "provider_example".into(), + uri: "context://provider_example/records/x".into(), + expires_at: None, + }); + frame.content = Some("summary…".into()); + assert!(frame.representation_invariants().is_ok()); + } } diff --git a/contextgraph-types/src/lib.rs b/contextgraph-types/src/lib.rs index a43cfd0..1bd030c 100644 --- a/contextgraph-types/src/lib.rs +++ b/contextgraph-types/src/lib.rs @@ -19,7 +19,10 @@ pub mod verify; pub use capability::{Capabilities, DataFlow, ProviderInfo}; pub use consent::{ConsentReceipt, Grantor}; -pub use frame::{ContextFrame, FrameKind, Provenance, Relation}; +pub use frame::{ + ContentFidelity, ContentRef, ContextFrame, FrameEmbedding, FrameKind, InlineContentRequirement, + Provenance, Relation, Representation, Transform, +}; pub use identity::{FrameId, canonical_order}; pub use query::{ContextQuery, ContextQueryResult}; pub use scope::EgressScope; diff --git a/contextgraph-types/src/query.rs b/contextgraph-types/src/query.rs index 854e8eb..01acc39 100644 --- a/contextgraph-types/src/query.rs +++ b/contextgraph-types/src/query.rs @@ -5,7 +5,7 @@ use serde::{Deserialize, Serialize}; -use crate::frame::{ContextFrame, FrameKind}; +use crate::frame::{ContextFrame, FrameKind, Representation}; /// A request to an Context Graph Protocol provider for context frames relevant to a goal. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] @@ -27,6 +27,33 @@ pub struct ContextQuery { /// Pin retrieval to a point in time for bi-temporal facts. #[serde(default, skip_serializing_if = "Option::is_none")] pub as_of: Option, + /// Ordered [frame representation](Representation) preference. The provider + /// returns the first supported representation it can satisfy. Empty on the + /// wire ⇒ the default `[full]`, so pre-representation hosts are unchanged. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub representation_preferences: Vec, +} + +impl ContextQuery { + /// The effective ordered representation preference, defaulting to `[full]` + /// when the host stated none (the legacy behavior). + pub fn preferred_representations(&self) -> Vec { + if self.representation_preferences.is_empty() { + vec![Representation::Full] + } else { + self.representation_preferences.clone() + } + } + + /// The representation a provider should return: the first + /// [preferred](Self::preferred_representations) one it supports. `None` ⇒ + /// none of the requested representations is supported and the provider must + /// answer `unsupported_representation`. + pub fn select_representation(&self, supported: &[Representation]) -> Option { + self.preferred_representations() + .into_iter() + .find(|wanted| supported.contains(wanted)) + } } /// The response to a `context/query` call. @@ -59,23 +86,7 @@ mod tests { use crate::frame::ContextFrame; fn frame_with_cost(id: &str, cost: u32) -> ContextFrame { - ContextFrame { - id: id.into(), - kind: FrameKind::Snippet, - title: id.into(), - content: String::new(), - content_digest: None, - uri: None, - score: 0.5, - token_cost: cost, - valid_from: None, - valid_to: None, - recorded_at: None, - provenance: vec![], - citation_label: None, - embedding: None, - relations: vec![], - } + ContextFrame::full(id, FrameKind::Snippet, id, String::new(), 0.5, cost) } #[test] @@ -89,12 +100,59 @@ mod tests { max_frames: 20, max_tokens: 4000, as_of: None, + representation_preferences: vec![], }; let json = serde_json::to_string(&query).unwrap(); let back: ContextQuery = serde_json::from_str(&json).unwrap(); assert_eq!(back, query); } + #[test] + fn representation_preferences_default_to_full_and_select_first_supported() { + // A host that states nothing gets the legacy `[full]` behavior, and the + // field is omitted from the wire. + let mut query = ContextQuery { + goal: "g".into(), + query_text: None, + embedding: None, + kinds: vec![], + anchors: vec![], + max_frames: 1, + max_tokens: 10, + as_of: None, + representation_preferences: vec![], + }; + assert_eq!( + query.preferred_representations(), + vec![Representation::Full] + ); + assert!( + !serde_json::to_string(&query) + .unwrap() + .contains("representation_preferences") + ); + assert_eq!( + query.select_representation(&[Representation::Full]), + Some(Representation::Full) + ); + + // With an explicit preference, the provider returns the first it can + // satisfy; if none is supported, it must answer unsupported. + query.representation_preferences = vec![Representation::Reference, Representation::Full]; + assert_eq!( + query.select_representation(&[Representation::Full]), + Some(Representation::Full), + ); + assert_eq!( + query.select_representation(&[Representation::Reference, Representation::Full]), + Some(Representation::Reference), + ); + assert_eq!( + query.select_representation(&[Representation::Compact]), + None + ); + } + #[test] fn respects_budget_true_when_under_or_at_limit() { let result = ContextQueryResult { diff --git a/contextgraph-types/tests/frame_representation_witness.rs b/contextgraph-types/tests/frame_representation_witness.rs index 70cd632..0314bcc 100644 --- a/contextgraph-types/tests/frame_representation_witness.rs +++ b/contextgraph-types/tests/frame_representation_witness.rs @@ -57,15 +57,17 @@ fn reference_frame_without_inline_content_deserializes() { // Round-trip back to the wire and check the normative invariants on the // serialized form, so this test does not depend on the chosen Rust API. let value = serde_json::to_value(&frame).expect("frame must serialize"); - let obj = value.as_object().expect("frame serializes as a JSON object"); + let obj = value + .as_object() + .expect("frame serializes as a JSON object"); // Never encode a reference as `content: ""` (or any inline content). match obj.get("content") { None => {} Some(serde_json::Value::Null) => {} - Some(other) => panic!( - "reference frame must not carry inline content on the wire, got: {other}" - ), + Some(other) => { + panic!("reference frame must not carry inline content on the wire, got: {other}") + } } // Reference frames must state their representation and keep the @@ -104,7 +106,9 @@ fn legacy_full_frame_still_deserializes_without_representation() { .expect("legacy full ContextFrames must continue to deserialize"); let value = serde_json::to_value(&frame).expect("frame must serialize"); - let obj = value.as_object().expect("frame serializes as a JSON object"); + let obj = value + .as_object() + .expect("frame serializes as a JSON object"); assert_eq!( obj.get("content").and_then(|v| v.as_str()), Some("export interface Workspace { }"), diff --git a/docs/adr/0005-frame-representations.md b/docs/adr/0005-frame-representations.md new file mode 100644 index 0000000..e9b57c0 --- /dev/null +++ b/docs/adr/0005-frame-representations.md @@ -0,0 +1,81 @@ +# 0005 — Frame representations (full, compact, reference) + +**Status:** Accepted (draft; `contextgraph/1.0-draft`) + +**Context:** CGEP lifecycle work, phase 2 — frame representations. Numbering is +provisional: this ADR lands ahead of PR #33's ADR set (0002–0004); if it +collides on rebase, renumber — the decision, not the number, is what matters. + +## Context + +`ContextFrame` inlines its content as a required `String`. That is honest for a +snippet, but wasteful or impossible for large or already-durable material: a +host that only needs a pointer still pays to move the bytes, and a provider that +holds content behind a resolver cannot return a frame at all without copying it +inline. The lifecycle build prompt (`§ContextFrame representations`) specifies an +additive fix: let a frame *state how it carries its content*. + +A witness test (`frame_representation_witness.rs`) shipped ahead of this change +and failed on `main` itself, because a reference frame carrying no inline content +could not deserialize. This ADR records the implementation that satisfies it. + +## Decision + +Extend the existing `ContextFrame` — do **not** introduce a competing frame +shape — with a `representation` discriminant and its supporting fields: + +- `full` — canonical inline `content` is required. This is the default; a frame + with no `representation` field is `full`, and a `full` frame omits the field on + the wire, so pre-representation providers and stored frames are unchanged. +- `compact` — inline `content` (a transformed rendering) **plus** the inline hash + (`content_digest`), `canonical_content_hash`, a `transform` identity, and a + `content_ref`. +- `reference` — **no** inline content; only a `content_ref` and + `canonical_content_hash`. Never encoded as `content: ""`; the field is omitted. + +Supporting additive fields: `content_fidelity`, `canonical_content_hash`, +`content_ref { provider_id, uri, expires_at? }`, `transform`, +`minimum_content_fidelity`, `inline_content_requirement`, `canonical_token_cost`, +and `tokenizer_ref`. `content` becomes `Option` (absent for references). + +Negotiation is additive too: `ContextQuery.representation_preferences` (absent ⇒ +`[full]`) and `Capabilities.representations` + `Capabilities.resolve`, with the +consistency rule that advertising `compact`/`reference` requires `resolve`. + +The invariants are enforced in three places that must agree: Rust +(`ContextFrame::representation_invariants`), the JSON Schema (`allOf` +conditionals), and the conformance/unit tests. + +### Two names for two hashes + +The spec names the inline-content hash `content_hash` and the full-source hash +`canonical_content_hash`. This repo already has `content_digest` with exactly the +inline-content meaning, and it is the third component of `FrameId`. We **keep +`content_digest`** as the inline hash (it *is* the spec's `content_hash` under an +established name) and **add `canonical_content_hash`** as the distinct +full-source hash. A rename to `content_hash` is a separate, compatibility-aware +change (the build prompt says not to fold renames into feature work). + +### `token_cost` stays required, for now + +The build prompt makes `token_cost` optional and adds `canonical_token_cost` + +`tokenizer_ref`. We add the latter two additively but **keep `token_cost: u32` +required**. Making it optional reopens the exact `budget-honesty` B3 decision +that PR #33 lands (`token_cost` MUST equal `ceil(utf8_len(content)/4)`), on the +same field and the same budget-accounting functions PR #33 rewrites. Deferring +`token_cost`-optionality to that reconciliation avoids a conflict on PR #33's +headline field; the witness fixture is satisfied with `token_cost: 0` present. + +## Consequences + +- `full`, `compact`, and `reference` are **representations of one frame**, not + replacement entities. `CompiledContextFrame`, snapshots, aggregate deltas, and + prompt rendering remain host concerns and are **not** part of this protocol. +- Backward compatible: every new field is optional/default-absent, legacy full + frames deserialize and re-serialize byte-for-byte, and query-only providers are + untouched. +- A `reference` frame must be resolved (`context/resolve`, a later phase) before + its content can be composed; until then a host renders it as empty rather than + fabricating bytes. +- Resolution routing, compaction algorithms, and token allocation stay provider/ + host policy and are not standardized here. diff --git a/docs/protocol-surface.md b/docs/protocol-surface.md index fb38c76..11931b2 100644 --- a/docs/protocol-surface.md +++ b/docs/protocol-surface.md @@ -119,14 +119,26 @@ pub enum FrameKind { Snippet, Symbol, Fact, Doc, Memory, Episode, Graph, } +pub enum Representation { Full, Compact, Reference } // absent ⇒ Full (legacy) + pub struct ContextFrame { pub id: String, // provider-scoped, stable for dedup across queries pub kind: FrameKind, pub title: String, // human label — never a bare uuid - pub content: String, // untrusted data — host must delimit as quoted material + pub content: Option, // untrusted data; ABSENT for a reference frame + pub content_digest: Option, // inline-content hash (the spec's content_hash); feeds FrameId pub uri: Option, + pub representation: Representation, // full | compact | reference (omitted on the wire when full) + pub content_fidelity: Option, // exact | normalized | summarized | omitted + pub canonical_content_hash: Option, // hash of the COMPLETE source content + pub content_ref: Option, // opaque resolver handle (compact/reference) + pub transform: Option, // how a compact frame rendered its source + pub minimum_content_fidelity: Option, + pub inline_content_requirement: Option, pub score: f32, // provider-normalized relevance, [0, 1] - pub token_cost: u32, // honest, conformance-audited token cost + pub token_cost: u32, // honest, conformance-audited inline token cost + pub canonical_token_cost: Option, // cost of the full source (if declared) + pub tokenizer_ref: Option, // e.g. "openai:o200k_base" pub valid_from: Option, pub valid_to: Option, pub recorded_at: Option, @@ -136,6 +148,12 @@ pub struct ContextFrame { pub relations: Vec, } +pub struct ContentRef { // content_ref.uri is distinct from ContextFrame.uri + pub provider_id: String, // the exact provider that returned this frame + pub uri: String, // opaque resolver handle for context/resolve + pub expires_at: Option, +} + pub struct Provenance { pub kind: String, // e.g. "file", "derivation", "episode" (serialized as "type") pub uri: Option, @@ -168,6 +186,42 @@ suite checks both: conformance failure, not a cosmetic gap. (Whole-platform convention: raw ids are never the primary on-screen identifier.) +### Frame representations + +A frame states **how** it carries its content through `representation`. This is +additive: a legacy frame with no `representation` field is a `full` frame, and a +`full` frame omits the field on the wire, so pre-representation providers and +stored frames are unchanged. + +| Representation | Inline `content` | `content_ref` + `canonical_content_hash` | `content_digest` (inline hash) | `transform` | +| --- | --- | --- | --- | --- | +| `full` | **required** | optional | optional | absent | +| `compact` | **required** (a transformed rendering) | **required** | **required** | **required** | +| `reference` | **absent** | **required** | absent | absent | + +- A `reference` frame carries no inline content at all — only a `content_ref` + (an opaque resolver handle) and a `canonical_content_hash` so a host can + rehydrate honestly and verifiably. It is **never** encoded as `content: ""`; + the field is omitted entirely. +- `ContextFrame.uri` identifies the source resource; `content_ref.uri` is a + distinct opaque resolver handle, and `content_ref.provider_id` names the exact + provider a fan-out host routes `context/resolve` back to. +- `content_digest` is the hash of the **inline** content bytes (the spec's + `content_hash`, under its established name; it feeds `FrameId`); + `canonical_content_hash` is the hash of the **complete source** content. + +`ContextFrame::representation_invariants()` enforces the table above in code, and +the JSON Schema enforces it on the wire; providers should self-check, and a host +rejects a frame that lies about its shape. + +**Negotiation.** A host states an ordered `ContextQuery.representation_preferences` +(absent ⇒ `[full]`); the provider returns the first representation it supports +(`ContextQuery::select_representation`) or answers `unsupported_representation`. +A provider advertises `Capabilities.representations` and `Capabilities.resolve`; +because `compact`/`reference` hand the host a `content_ref` to rehydrate, +advertising either **requires** `resolve` support +(`Capabilities::representations_consistent`). + ## Reusing context across turns A single retrieval is only half the story. Reusing retrieved context across the diff --git a/examples/reference-messages.json b/examples/reference-messages.json index cdd6bac..7a4ec01 100644 --- a/examples/reference-messages.json +++ b/examples/reference-messages.json @@ -43,7 +43,9 @@ "reads": true, "writes": false, "egress": true, - "egress_scopes": ["third-party-model"] + "egress_scopes": [ + "third-party-model" + ] } }, "capabilities": { @@ -174,5 +176,106 @@ }, { "type": "shutdown" + }, + { + "type": "handshake_ack", + "protocol_version": "contextgraph/1.0-draft", + "provider": { + "name": "cloud-docs", + "version": "1.3.0", + "data_flow": { + "reads": true, + "writes": false, + "egress": false + } + }, + "capabilities": { + "query": { + "kinds": [ + "doc" + ], + "filters": [] + }, + "upsert": false, + "graph": false, + "embeddings_fingerprint": null, + "subscribe": false, + "verify": true, + "representations": [ + "full", + "compact", + "reference" + ], + "resolve": true + } + }, + { + "type": "query", + "query": { + "goal": "summarize the deployment runbook", + "query_text": "deployment runbook", + "kinds": [ + "doc" + ], + "anchors": [], + "max_frames": 3, + "max_tokens": 512, + "representation_preferences": [ + "compact", + "full" + ] + } + }, + { + "type": "frames", + "result": { + "frames": [ + { + "id": "cloud-docs:runbook-compact", + "kind": "doc", + "title": "Deployment runbook (summary)", + "content": "Roll back with `deploy revert `; page oncall if the error rate exceeds 2%.", + "content_digest": "sha256:11aa22bb33cc44dd", + "uri": "file:///repo/docs/runbook.md", + "representation": "compact", + "content_fidelity": "summarized", + "canonical_content_hash": "sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", + "content_ref": { + "provider_id": "cloud-docs", + "uri": "context://cloud-docs/records/doc_runbook_v1" + }, + "transform": { + "method": "extractive_summary", + "implementation": "provider_default", + "version": "1" + }, + "score": 0.9, + "token_cost": 24, + "canonical_token_cost": 512, + "tokenizer_ref": "openai:o200k_base", + "provenance": [], + "citation_label": "runbook.md (summary)", + "relations": [] + }, + { + "id": "cloud-docs:runbook-ref", + "kind": "doc", + "title": "Deployment runbook", + "uri": "file:///repo/docs/runbook.md", + "representation": "reference", + "canonical_content_hash": "sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", + "content_ref": { + "provider_id": "cloud-docs", + "uri": "context://cloud-docs/records/doc_runbook_v1" + }, + "score": 0.9, + "token_cost": 0, + "provenance": [], + "citation_label": "runbook.md", + "relations": [] + } + ], + "truncated": false + } } ] diff --git a/schema/contextgraph-envelope.schema.json b/schema/contextgraph-envelope.schema.json index 79cffe0..a4a5e48 100644 --- a/schema/contextgraph-envelope.schema.json +++ b/schema/contextgraph-envelope.schema.json @@ -254,6 +254,15 @@ "verify": { "type": "boolean", "$comment": "Whether the provider answers context/verify (docs/context-reuse.md §4). Optional and defaults to false, so a pre-§4 provider is treated as unsupported and the host falls back to re-querying. Independent of `subscribe` — push and pull freshness are complementary." + }, + "representations": { + "type": "array", + "items": { "$ref": "#/$defs/Representation" }, + "$comment": "Frame representations this provider can return. Absent/empty ⇒ full only (the legacy default). Advertising compact or reference requires resolve: true." + }, + "resolve": { + "type": "boolean", + "$comment": "Whether the provider answers context/resolve for a frame's content_ref. Compact/reference support implies resolve support." } }, "required": ["query", "upsert", "graph", "subscribe"], @@ -280,7 +289,12 @@ }, "max_frames": { "$ref": "#/$defs/u32" }, "max_tokens": { "$ref": "#/$defs/u32" }, - "as_of": { "type": ["string", "null"] } + "as_of": { "type": ["string", "null"] }, + "representation_preferences": { + "type": "array", + "items": { "$ref": "#/$defs/Representation" }, + "$comment": "Ordered preference; the provider returns the first representation it supports. Absent/empty ⇒ [full]." + } }, "required": ["goal", "kinds", "anchors", "max_frames", "max_tokens"], "additionalProperties": false @@ -306,6 +320,47 @@ "enum": ["snippet", "symbol", "fact", "doc", "memory", "episode", "graph"] }, + "Representation": { + "type": "string", + "enum": ["full", "compact", "reference"], + "$comment": "How a frame carries content. Absent on the wire ⇒ full. full inlines content; compact inlines a transformed rendering plus a resolver handle; reference carries no inline content, only a content_ref and canonical hash." + }, + + "ContentFidelity": { + "type": "string", + "enum": ["exact", "normalized", "summarized", "omitted"], + "$comment": "Fidelity of carried content relative to the source. Absent ⇒ exact for a legacy full frame." + }, + + "InlineContentRequirement": { + "type": "string", + "enum": ["required", "resolvable_reference_allowed"] + }, + + "ContentRef": { + "type": "object", + "description": "An opaque resolver handle for a compact/reference frame's content. content_ref.uri is distinct from ContextFrame.uri (the source resource); provider_id names the exact provider that returned the frame so a fan-out host routes context/resolve back to it.", + "properties": { + "provider_id": { "type": "string", "minLength": 1 }, + "uri": { "type": "string", "minLength": 1 }, + "expires_at": { "type": ["string", "null"] } + }, + "required": ["provider_id", "uri"], + "additionalProperties": false + }, + + "Transform": { + "type": "object", + "description": "The transformation a compact frame applied to its source, so a consumer knows what it is reading.", + "properties": { + "method": { "type": "string", "minLength": 1 }, + "implementation": { "type": "string", "minLength": 1 }, + "version": { "type": "string", "minLength": 1 } + }, + "required": ["method", "implementation", "version"], + "additionalProperties": false + }, + "ContextFrame": { "type": "object", "description": "The unit of exchange. A frame is not a string — it is a structured record of relevance, cost, provenance, and validity. content is untrusted data: a conforming host delimits it as quoted material and never executes it.", @@ -320,12 +375,31 @@ "$comment": "Provider-declared, opaque digest of the frame's content bytes (e.g. sha256:) — the third component of the frame's stable FrameId identity. Optional: a frame without one is not verifiable and a host re-queries it (docs/context-reuse.md §1, §4)." }, "uri": { "type": ["string", "null"] }, + "representation": { + "$ref": "#/$defs/Representation", + "$comment": "How the frame carries content. Absent ⇒ full (legacy); full frames omit this field on the wire." + }, + "content_fidelity": { "$ref": "#/$defs/ContentFidelity" }, + "canonical_content_hash": { + "type": "string", + "minLength": 1, + "$comment": "SHA-256 over the complete source content bytes — distinct from content_digest (the inline-content hash). Required for compact/reference frames." + }, + "content_ref": { "$ref": "#/$defs/ContentRef" }, + "transform": { "$ref": "#/$defs/Transform" }, + "minimum_content_fidelity": { "$ref": "#/$defs/ContentFidelity" }, + "inline_content_requirement": { "$ref": "#/$defs/InlineContentRequirement" }, "score": { "type": "number", "minimum": 0, "maximum": 1 }, "token_cost": { "$ref": "#/$defs/u32" }, + "canonical_token_cost": { + "$ref": "#/$defs/u32", + "$comment": "Token cost of the complete canonical source content. If token_cost or canonical_token_cost is present, tokenizer_ref should name the tokenizer." + }, + "tokenizer_ref": { "type": "string", "minLength": 1 }, "valid_from": { "type": ["string", "null"] }, "valid_to": { "type": ["string", "null"] }, "recorded_at": { "type": ["string", "null"] }, @@ -344,7 +418,47 @@ "items": { "$ref": "#/$defs/Relation" } } }, - "required": ["id", "kind", "title", "content", "score", "token_cost", "provenance", "relations"], + "required": ["id", "kind", "title", "score", "token_cost", "provenance", "relations"], + "$comment": "content is required for full/compact frames but omitted for reference frames, so it is not globally required; the allOf below enforces the per-representation invariants.", + "allOf": [ + { + "$comment": "reference: no inline content, content_digest, or transform; a resolver handle and canonical hash are required.", + "if": { + "properties": { "representation": { "const": "reference" } }, + "required": ["representation"] + }, + "then": { + "required": ["content_ref", "canonical_content_hash"], + "not": { + "anyOf": [ + { "required": ["content"] }, + { "required": ["content_digest"] }, + { "required": ["transform"] } + ] + } + } + }, + { + "$comment": "compact: inline content plus inline hash, canonical hash, transform identity, and resolver handle.", + "if": { + "properties": { "representation": { "const": "compact" } }, + "required": ["representation"] + }, + "then": { + "required": ["content", "content_digest", "canonical_content_hash", "transform", "content_ref"] + } + }, + { + "$comment": "full (including legacy frames with no representation field): inline content is required.", + "if": { + "not": { + "properties": { "representation": { "enum": ["compact", "reference"] } }, + "required": ["representation"] + } + }, + "then": { "required": ["content"] } + } + ], "additionalProperties": false },