From 79ac5bba2e7767fc16f2568b903b759d3bb5966e Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 16:23:30 +0200 Subject: [PATCH 001/167] docs: establish Symphonia SDD foundation --- AGENTS.md | 14 + README.md | 50 ++- docs/README.md | 64 ++++ docs/architecture/system-architecture.md | 286 ++++++++++++++ ...1-provider-independent-recording-domain.md | 53 +++ .../0002-copy-and-sync-are-distinct.md | 48 +++ .../0003-home-assistant-app-primary.md | 54 +++ docs/decisions/README.md | 12 + docs/development/development-specification.md | 178 +++++++++ docs/domain/domain-model.md | 316 ++++++++++++++++ docs/open-questions.md | 164 +++++++++ docs/product/product-specification.md | 180 +++++++++ .../home-assistant-ecosystem-review.md | 169 +++++++++ docs/providers/provider-research.md | 155 ++++++++ docs/providers/provider-specification.md | 235 ++++++++++++ specs/CATALOG.md | 25 ++ specs/README.md | 132 +++++++ specs/_template.md | 241 ++++++++++++ specs/catalog.json | 313 ++++++++++++++++ specs/catalog.schema.json | 111 ++++++ specs/durable-operations-and-recovery.md | 318 ++++++++++++++++ .../home-assistant-app-runtime-and-ingress.md | 323 ++++++++++++++++ ...library-import-and-provider-projections.md | 327 ++++++++++++++++ specs/one-time-playlist-copy.md | 304 +++++++++++++++ .../provider-connections-and-authorization.md | 348 ++++++++++++++++++ specs/recording-identity-resolution.md | 333 +++++++++++++++++ 26 files changed, 4751 insertions(+), 2 deletions(-) create mode 100644 AGENTS.md create mode 100644 docs/README.md create mode 100644 docs/architecture/system-architecture.md create mode 100644 docs/decisions/0001-provider-independent-recording-domain.md create mode 100644 docs/decisions/0002-copy-and-sync-are-distinct.md create mode 100644 docs/decisions/0003-home-assistant-app-primary.md create mode 100644 docs/decisions/README.md create mode 100644 docs/development/development-specification.md create mode 100644 docs/domain/domain-model.md create mode 100644 docs/open-questions.md create mode 100644 docs/product/product-specification.md create mode 100644 docs/providers/home-assistant-ecosystem-review.md create mode 100644 docs/providers/provider-research.md create mode 100644 docs/providers/provider-specification.md create mode 100644 specs/CATALOG.md create mode 100644 specs/README.md create mode 100644 specs/_template.md create mode 100644 specs/catalog.json create mode 100644 specs/catalog.schema.json create mode 100644 specs/durable-operations-and-recovery.md create mode 100644 specs/home-assistant-app-runtime-and-ingress.md create mode 100644 specs/library-import-and-provider-projections.md create mode 100644 specs/one-time-playlist-copy.md create mode 100644 specs/provider-connections-and-authorization.md create mode 100644 specs/recording-identity-resolution.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..32bafc1 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,14 @@ +## Product specifications and SDDs + +When creating, revising, reviewing, or implementing product behavior: + +1. Read [`specs/README.md`](specs/README.md) and the applicable SDD in full. +2. Consult [`specs/catalog.json`](specs/catalog.json) for capability status, blockers, and evidence. +3. Start new SDDs from [`specs/_template.md`](specs/_template.md). Adapt sections proportionally, but mark non-applicable concerns explicitly instead of deleting them silently. +4. Keep global product/domain/architecture requirements in `docs/`; SDDs assemble those requirements into one vertical, implementation-driving capability contract. Resolve conflicts rather than choosing one document silently. +5. Do not implement a capability whose SDD is not `Ready for implementation`, and do not begin production implementation without explicit owner approval even when the SDD is ready. +6. Update the SDD, catalog evidence, tests, user/operator documentation, and relevant ADRs together when an observable or architectural contract changes. +7. Include concrete user flows, states, failure/recovery behavior, security boundaries, a numeric test budget, acceptance scenarios, and requirement traceability. +8. Treat provider documentation and community implementations as dated evidence, not permanent guarantees. Keep official API research separate from community implementation evidence. + +Until a repository-native specification validator is selected, every SDD change must at minimum pass the manual checks listed in `specs/README.md`. diff --git a/README.md b/README.md index 3484050..c2a947b 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,48 @@ -# symphonia -HomeAssistant App/Service to manage and enjoy your music +# Symphonia + +Symphonia is a self-hosted, provider-independent music library hub, distributed primarily as a Supervisor-managed Home Assistant App. It is intended to help a person understand, move, and eventually synchronize their music across services such as Spotify and YouTube without treating any one provider as the source of the domain model. + +> Your music library belongs to you. Providers are places where representations of that music happen to exist. + +Symphonia is a library-management and interoperability project, not a new music player. Playback, recommendations, social features, transcoding, and multi-room audio are outside the initial scope. Its domain and application core remain independent of Home Assistant so the service can also run and be tested standalone; the deployment and integration pattern follows `vypdev/homeassistant-gateway`. + +## Project status + +**Specification stage. No production application has been implemented.** + +The current work establishes a reviewable source of truth before technology selection or implementation begins. In particular, official provider feasibility still needs validation: Google's public YouTube Data API can manage YouTube video playlists, but the research performed for this specification did not identify an official API exposing the complete YouTube Music library model. + +## Documentation + +Start with the [documentation map](docs/README.md). The horizontal specifications define shared product and architecture rules; the [SDD standard](specs/README.md) and [capability catalog](specs/CATALOG.md) turn those rules into reviewable vertical implementation contracts. + +| Area | Source of truth | +| --- | --- | +| Capability readiness and implementation contracts | [SDD catalog](specs/CATALOG.md) | +| SDD lifecycle, template, and readiness gate | [SDD standard](specs/README.md) | +| Vision, scope, journeys, requirements | [Product specification](docs/product/product-specification.md) | +| Vocabulary, entities, identity, playlists | [Domain model](docs/domain/domain-model.md) | +| System boundaries and operational qualities | [Architecture](docs/architecture/system-architecture.md) | +| Provider contract and capability semantics | [Provider specification](docs/providers/provider-specification.md) | +| Current official-provider evidence | [Provider research](docs/providers/provider-research.md) | +| Existing Home Assistant music projects and design lessons | [Ecosystem review](docs/providers/home-assistant-ecosystem-review.md) | +| Engineering and testing workflow | [Development specification](docs/development/development-specification.md) | +| Decisions already accepted | [Architecture decisions](docs/decisions/README.md) | +| Risks and unresolved choices | [Open questions](docs/open-questions.md) | + +## Product shape + +The intended first useful workflow is: + +1. A local user connects provider accounts. +2. Symphonia imports provider playlists and track representations without losing provenance. +3. It resolves provider tracks to provider-independent recordings, surfacing uncertainty rather than hiding it. +4. The user reviews unresolved items and records durable manual decisions. +5. The user previews and executes a one-time playlist copy. +6. The operation history explains every match, omission, provider write, retry, and failure. + +Persistent synchronization follows only after copy semantics and official provider feasibility are understood. + +## Contributing during specification + +Use requirement identifiers in issues, SDDs, tests, and future commits. Material implementation work starts from the applicable cataloged SDD and its numeric verification budget. A change to accepted behavior must update the relevant horizontal specification and, when it changes an architectural decision, add or supersede an ADR. Do not infer a decision from an open question, and do not start production implementation until the SDD is ready and the owner explicitly approves it. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..c8ccbc4 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,64 @@ +# Documentation map + +**Status:** baseline for review +**Last reviewed:** 2026-09-20 + +This documentation is the horizontal implementation contract for Symphonia. It deliberately separates product intent, domain rules, architecture, provider facts, accepted decisions, and unresolved choices. The vertical, capability-level contracts live in the [SDD catalog](../specs/CATALOG.md) and select from these shared rules without overriding them. + +## Sources of truth + +| Document | Owns | Does not own | +| --- | --- | --- | +| [SDD standard and catalog](../specs/README.md) | Capability boundaries, readiness, end-to-end design, numeric test budgets, acceptance and evidence | Shared product policy or silent overrides of horizontal specifications | +| [Product specification](product/product-specification.md) | Outcomes, scope, journeys, product requirements | Entity design or technology choices | +| [Domain model](domain/domain-model.md) | Ubiquitous language, invariants, identity, copy and sync semantics | Provider API facts | +| [System architecture](architecture/system-architecture.md) | Boundaries, execution model, security and operations | Final implementation stack | +| [Provider specification](providers/provider-specification.md) | Provider port, capabilities, normalized errors | Claims about a specific API | +| [Provider research](providers/provider-research.md) | Dated, sourced facts about provider APIs | Product policy or permanent architecture | +| [Home Assistant music ecosystem review](providers/home-assistant-ecosystem-review.md) | Reusable patterns and cautions from existing HA music projects | Dependency selection or provider guarantees | +| [Development specification](development/development-specification.md) | Specification workflow, testing and delivery gates | Product scope | +| [ADRs](decisions/README.md) | Decisions that have actually been accepted | Proposals and guesses | +| [Open questions](open-questions.md) | Decisions needed, assumptions, risk register, next design work | Accepted requirements | + +## Normative language + +`MUST`, `MUST NOT`, `SHOULD`, `SHOULD NOT`, and `MAY` are normative. Lowercase uses are explanatory. Each requirement has a stable identifier: + +| Prefix | Area | +| --- | --- | +| `SYM-PROD` | Product and user experience | +| `SYM-ACC` | Local and provider accounts | +| `SYM-LIB` | Unified library | +| `SYM-MATCH` | Identity resolution | +| `SYM-PL` | Playlist copy | +| `SYM-SYNC` | Persistent synchronization | +| `SYM-PROV` | Provider abstraction | +| `SYM-ARCH` | Architecture and persistence | +| `SYM-JOB` | Background execution | +| `SYM-SEC` | Security and credentials | +| `SYM-OBS` | Observability | +| `SYM-HA` | Home Assistant | +| `SYM-TEST` | Testing | +| `SYM-DEP` | Deployment | + +Requirement identifiers are never reused. Removed requirements remain in history and should be marked superseded rather than silently renumbered. + +## Status model + +- **Accepted**: an explicit product constraint or recorded ADR; implementation may rely on it. +- **Proposed**: a reviewable direction; implementation must not treat it as settled. +- **Open**: a choice or fact still requiring evidence or owner input. +- **Research snapshot**: a dated external fact that must be revalidated before implementation. + +The current baseline is documentation for review, not approval to implement. The repository must remain free of production application code until the owner explicitly approves implementation. + +## Change workflow + +1. Link a change to one or more requirement identifiers. +2. Select or create the capability SDD from the [catalog](../specs/CATALOG.md); use the [mandatory template](../specs/_template.md). +3. Update product/domain specifications before or with the SDD when shared behavior changes. +4. Record a durable, consequential decision as an ADR; do not use ADRs for routine coding choices. +5. Keep provider facts in dated research and link to official sources. +6. Move answered questions into requirements or ADRs and leave a pointer to the resolution. +7. Do not implement until the SDD passes its readiness gate and the owner explicitly approves implementation. +8. Add acceptance tests traceable to the affected requirements and update catalog evidence with the implementation. diff --git a/docs/architecture/system-architecture.md b/docs/architecture/system-architecture.md new file mode 100644 index 0000000..cac1bcc --- /dev/null +++ b/docs/architecture/system-architecture.md @@ -0,0 +1,286 @@ +# System architecture + +**Status:** Home Assistant-first deployment accepted; logical boundaries proposed; technology stack open +**Last reviewed:** 2026-09-20 + +## Architectural drivers + +The architecture is derived from these needs: + +- long-running, self-hosted operation on modest hardware; +- a primary Supervisor-managed Home Assistant App experience similar to `vypdev/homeassistant-gateway`; +- a core that can run and be tested without Home Assistant; +- provider-independent domain and replaceable adapters; +- durable imports and provider writes that survive restarts; +- explicit uncertainty, partial success, rate limiting, and user action; +- secure storage and rotation of OAuth credentials; +- provider-contract tests without real accounts; +- a future native Home Assistant surface without duplicating domain policy; and +- simple backup, restore, upgrade, and diagnostics. + +These drivers do not yet justify a programming language, web framework, frontend framework, or database product. + +## Context and trust boundaries + +```text + provider OAuth/API boundaries + ┌───────────┬────────────┬─────────┐ + │ Spotify │ YouTube │ future │ + └─────▲─────┴─────▲──────┴────▲────┘ + │ adapters │ │ +┌─────────────────────────┴───────────┴───────────┴──────────────┐ +│ Symphonia service │ +│ web UI/API → application use cases → domain │ +│ │ ↑ │ +│ ├→ durable jobs → provider ports │ +│ └→ repositories / secret store ports │ +└──────────▲───────────────────┬─────────────────────────────────┘ + │ │ + HA Ingress / optional └→ persistent data and backups + companion integration + │ +┌──────────┴──────────┐ +│ Home Assistant │ +│ Supervisor + Core │ +└─────────────────────┘ +``` + +Provider APIs, the browser, Home Assistant/Supervisor, persistent storage, and future external API clients are separate trust boundaries. Network proximity is not authentication. + +## Dependency rule + +Dependencies point inward: + +```text +presentation ──→ application ──→ domain +infrastructure ────────────────→ ports defined inward +composition ──→ concrete implementations +``` + +- **Domain** owns recording identity, playlist semantics, capabilities, plans, policies, and operation states. It imports no provider SDK, Home Assistant package, web framework, database driver, filesystem/config reader, or job framework. +- **Application** orchestrates use cases through ports and owns transaction/idempotency boundaries. It depends on domain types, not concrete adapters. +- **Infrastructure** implements provider, persistence, secret, clock, queue, and Home Assistant ports. +- **Presentation** maps Ingress/HTTP/UI requests and responses. It does not decide match, authorization, retry, or conflict policy. +- **Composition** selects the deployment profile and wires concrete implementations. It is the only layer that knows the full runtime graph. + +This follows the useful boundary pattern in `homeassistant-gateway` without carrying that project's language, frameworks, or non-music policies into Symphonia automatically. + +## Logical components + +| Component | Responsibility | Explicit exclusions | +| --- | --- | --- | +| Web UI | Connection setup, library/playlist views, resolution queue, copy preview, history, diagnostics | Provider tokens, matching policy, direct provider calls | +| HTTP/API presentation | Authenticated input/output mapping, validation shape, request correlation | Domain decisions and raw exception exposure | +| Application use cases | Connect/disconnect, import, resolve, plan copy, execute copy, inspect operations | Provider-specific response types | +| Domain | Provider-independent entities, invariants, capability requirements, matching/copy/sync policy | IO and scheduling | +| Provider adapter host | OAuth/token refresh, pagination, normalized reads/writes/search, error translation | Cross-provider orchestration | +| Resolution engine | Candidate generation/evaluation and evidence production | Unversioned opaque decisions | +| Durable job runner | Leasing, scheduling, checkpoints, retry timing, cancellation | Business-policy invention | +| Persistence adapters | Transactions, migrations, retention, backup-safe storage | Provider API behavior | +| Secret store adapter | Encrypt/decrypt credential material and rotate key references | Returning plaintext to UI/logs | +| Observability | Structured logs, metrics, health/readiness, sanitized diagnostics | Provider payload dumping | +| Home Assistant adapter | Ingress identity and future native integration contract | Owning music domain rules | + +## Deployment model + +### Primary: Home Assistant App + +This is an accepted product/deployment decision, recorded in [ADR 0003](../decisions/0003-home-assistant-app-primary.md). + +- Symphonia is packaged as a Supervisor-managed Home Assistant App (the current name for an add-on). +- The App starts as an `application`, stores durable state under `/data`, exposes its UI through Ingress, and participates in Supervisor backup/update lifecycle. +- Ingress is the administrative UI authentication boundary. The server MUST honor the Ingress base path and MUST NOT assume it is hosted at `/`. +- App permissions, mapped folders, network exposure, and Supervisor/Core API access MUST be least-privilege. Music-provider access alone does not justify Home Assistant API or host filesystem access. +- A published image MUST support the explicitly documented Home Assistant architectures; the initial architecture set is open. +- Provider secrets MUST NOT be placed in App options, image layers, logs, diagnostics, or ordinary backups in plaintext. + +Home Assistant's current App documentation confirms that `/data` is persistent, Ingress can authenticate the UI, and App backup behavior is configurable. These platform facts are linked in [provider/platform research](../providers/provider-research.md#home-assistant-platform). + +### Secondary: standalone service + +To preserve the original “usable without Home Assistant” principle and test the core honestly, the same application core SHOULD have a standalone container/service composition profile. + +The standalone profile has no Ingress. It therefore requires an explicit authentication and network-exposure policy, persistent volume mapping, health endpoints, backup procedure, and OAuth callback configuration. Standalone must not become a second product: domain behavior, migrations, provider adapters, and operation history remain shared. + +Whether standalone packaging ships in the first public release or immediately afterward is open. + +### Optional companion Home Assistant integration + +A companion custom integration MAY later expose native entities, actions, events, and configuration discovery. It must call a stable, authenticated Symphonia API and MUST NOT duplicate matching, sync, credential, or retry logic. + +The integration MAY also be evaluated as a narrow provider-authorization broker so Symphonia can reuse Home Assistant's Application Credentials/config-flow callback machinery. If selected, it must exchange an opaque, one-use connection grant over the authenticated local API; provider operation logic remains in the App. Token ownership, refresh, revocation, backup, failure recovery, and integration/App version skew must be specified before this is accepted. + +Candidate native surface (illustrative, not accepted): + +- sensors for connection health, unmatched count, running operations, and last successful sync; +- actions such as `symphonia.copy_playlist` and future `symphonia.sync`; +- events for operation completed, partial, failed, or user action required. + +Three approaches require an RFC: + +| Approach | Benefits | Costs | +| --- | --- | --- | +| Companion custom integration over local HTTP/WebSocket | Native actions/entities and clean separation; mirrors the gateway pattern | Another artifact and compatibility matrix | +| MQTT discovery/events | Mature decoupling and push model | Adds an MQTT dependency and weakens direct operation correlation | +| App calls Home Assistant APIs directly | Fewer artifacts for events/actions initiated by the App | Couples the service to Home Assistant and does not cleanly provide a native integration surface | + +The companion-integration approach is the current leading direction, not yet an accepted implementation decision. + +## Service/API shape + +The backend needs an authenticated HTTP API for its own web UI and future integration adapter. The API SHOULD be resource/use-case oriented rather than a transparent provider proxy. It must never accept arbitrary provider URLs or expose raw upstream payloads by default. + +Required conceptual endpoints/use cases include: + +- connection list, capability probe, authorization start/callback, refresh, and disconnect; +- import start/status and collection reads; +- unresolved queue, candidate evidence, accept/reject/defer/revoke; +- copy plan, plan read, accept/execute, run status, reconcile/cancel where safe; +- operation history and issue resolution; and +- health, readiness, version, migration state, and sanitized diagnostics. + +Public API versioning, pagination, error envelopes, and event streaming require an API RFC before implementation. + +## Persistence requirements + +- **SYM-ARCH-001:** Durable domain data, accepted plans, manual decisions, job checkpoints, and operation summaries MUST survive process and App restarts. +- **SYM-ARCH-002:** Persistence MUST support atomic state transitions between a job checkpoint and its audit outcome where they share a store. +- **SYM-ARCH-003:** Schema changes MUST use forward migrations, have backup/restore guidance, and be tested from every supported released version. +- **SYM-ARCH-004:** Imported provider payloads MUST carry freshness/retention metadata and be deletable independently from locally owned decisions. +- **SYM-ARCH-005:** Referential design MUST preserve historical operation meaning when a connection or provider item is removed. +- **SYM-ARCH-006:** Backup MUST either quiesce writes or use a transactionally consistent online method. +- **SYM-ARCH-007:** Restore MUST detect incompatible or partial data before starting background writes. + +Conceptual stores: + +- provider connections and credential references; +- recording/provider representation graph and resolution evidence; +- current imported projections plus immutable snapshots needed by active/history policy; +- copy plans and future sync relationships; +- durable job state, checkpoints, leases, and retry schedule; +- operation/audit records and issues; and +- application schema/version/configuration. + +### Database alternatives + +| Option | Fit | Trade-offs | +| --- | --- | --- | +| SQLite under `/data` | Excellent single-user/App simplicity, transactional, easy backup when handled correctly | One-writer behavior and queue/lease design need care; multi-replica is inappropriate | +| PostgreSQL | Strong concurrency and operational tooling | Separate service/configuration is burdensome for a small Home Assistant install | +| Embedded key-value/document store | Simple packaging for some access patterns | Relational identity/history queries and migrations become harder | + +SQLite is the leading MVP candidate because the expected deployment is a single App instance, but it is **not accepted** until concurrency, backup, retention, and migration spikes are complete. + +## Background job and synchronization execution + +Imports and provider writes are asynchronous operations backed by durable state, not process-local tasks. + +- **SYM-JOB-001:** Job states MUST include at least `queued`, `running`, `waiting_rate_limit`, `waiting_user`, `retry_scheduled`, `succeeded`, `partial`, `failed`, and `cancelled`. +- **SYM-JOB-002:** A worker MUST acquire a time-bounded lease; abandoned running work becomes eligible for recovery after lease expiry. +- **SYM-JOB-003:** Every write step MUST have a stable idempotency/reconciliation key and a persisted before/after checkpoint. +- **SYM-JOB-004:** Retry schedules MUST survive restart and MUST respect provider retry hints and a configured attempt/time budget. +- **SYM-JOB-005:** Cancellation MUST be cooperative. Completed provider writes remain recorded and are not described as rolled back unless compensating writes actually succeeded. +- **SYM-JOB-006:** Scheduling MUST be fair across connections and MUST honor provider-, account-, and operation-specific concurrency limits. +- **SYM-JOB-007:** The UI MUST receive progress from persisted operation state; live in-memory events may accelerate display but are not authoritative. +- **SYM-JOB-008:** Scheduler time MUST use UTC instants; user-facing schedules and timestamps use the configured Home Assistant or standalone timezone explicitly. + +Initial process topology alternatives: + +1. **One modular process with API, scheduler, and bounded worker pool.** Simplest App lifecycle; requires careful shutdown and resource isolation. +2. **One image supervising separate API and worker processes.** Better isolation; more complex coordination and health semantics inside an App. +3. **External workflow/queue service.** Powerful but disproportionate for the single-node MVP. + +Option 1 is the proposed starting point. Durable database state—not the process—is the queue authority, so a later split does not change domain semantics. + +## Error model and retries + +Adapters normalize provider failures into stable categories while retaining sanitized provider codes: + +| Category | Default behavior | +| --- | --- | +| `authentication_required` / `authorization_revoked` | No blind retry; mark connection action-required | +| `permission_denied` / `capability_unavailable` | Permanent for the plan; return to planning/user | +| `not_found` / `item_unavailable` | Re-import/reconcile once if staleness is plausible; otherwise item issue | +| `invalid_request` | Permanent implementation/plan failure; no retry storm | +| `rate_limited` | Schedule from `Retry-After`/reset signal or conservative adapter policy | +| `provider_unavailable` / timeout / network | Exponential backoff with jitter and finite budget | +| `conflict` / version changed | Re-import and re-plan or surface conflict; never overwrite silently | +| `unknown_write_outcome` | Reconcile target state before any retry | +| `provider_contract_changed` | Stop affected capability, flag adapter health, require code/evidence update | + +- **SYM-ARCH-008:** Error messages crossing a boundary MUST be sanitized and correlated; tokens, authorization codes, raw headers, and unrestricted payloads MUST be absent. +- **SYM-ARCH-009:** Retry policy MUST be operation-aware. A retryable HTTP status does not by itself make a non-idempotent write safe. +- **SYM-ARCH-010:** Partial success MUST be a first-class terminal or waiting state with item-level facts. + +## Rate-limit model + +- **SYM-ARCH-011:** Each adapter MUST expose known quota dimensions, observed remaining/reset hints, and conservative defaults when limits are undocumented. +- **SYM-ARCH-012:** Rate limiting MUST occur before requests using per-provider/per-connection budgets and after responses using provider hints. +- **SYM-ARCH-013:** Interactive reads MAY receive higher scheduling priority than background refresh, but MUST NOT bypass hard limits. +- **SYM-ARCH-014:** Search-heavy matching MUST use caches and bounded candidate queries consistent with provider retention rules. +- **SYM-ARCH-015:** An operation plan SHOULD estimate known request/quota cost and warn when completion is unlikely within the current budget. + +## Authentication and credentials + +There are three distinct concerns: + +1. **Administrative access to Symphonia.** In Home Assistant mode this is primarily Ingress identity. Standalone authentication is open and MUST NOT default to unauthenticated non-loopback exposure. +2. **Provider OAuth client credentials.** Self-hosters may need to register their own applications. Client secrets and registration configuration are different from user grants. +3. **Provider user grants.** Access/refresh tokens authorize external music operations and need revocation, expiry, and minimum-scope handling. + +- **SYM-SEC-001:** Provider authorization MUST use the currently recommended authorization-code flow for the deployment/client type, CSRF `state`, exact redirect validation, and PKCE where applicable. +- **SYM-SEC-002:** Symphonia MUST request the minimum scopes needed for enabled capabilities and show them before consent. +- **SYM-SEC-003:** Tokens and client secrets MUST be encrypted at rest with key material kept outside the data database/ordinary export. If a platform cannot provide a separate secret, setup MUST make the trade-off explicit. +- **SYM-SEC-004:** Plaintext secrets MUST exist only for the shortest practical call boundary and MUST be redacted from logs, exceptions, metrics, traces, UI state, fixtures, and support bundles. +- **SYM-SEC-005:** Disconnect MUST revoke grants when supported, erase local credential material, disable jobs, and start provider-data cleanup. +- **SYM-SEC-006:** Authorization callbacks MUST bind provider, connection attempt, initiating user/session, state, redirect target, and an expiry; callbacks are single-use. +- **SYM-SEC-007:** Credential refresh MUST use a single-flight lock per connection to avoid destructive refresh-token races. +- **SYM-SEC-008:** Ingress identity headers MUST only be trusted on the verified Ingress listener/path; direct listeners MUST use their own authentication. +- **SYM-SEC-009:** A directly exposed OAuth callback listener MUST expose only the minimum callback surface or independently authenticate every other route. Opening a callback port MUST NOT make the management UI/API anonymously reachable. +- **SYM-SEC-010:** Provider-controlled IDs, names, URIs, metadata, callback parameters, and adapter mappings MUST NOT become local filesystem paths, module/import names, executable URI schemes, arbitrary redirect targets, or unrestricted outbound destinations. +- **SYM-SEC-011:** The App MUST run without root privileges where the platform permits and MUST NOT request host/config/media mounts or broad network privileges without a documented requirement and threat review. + +### OAuth callback risk in Home Assistant mode + +Provider redirect URIs must be exact and externally reachable under provider rules, while Home Assistant Ingress uses a proxied base path and session. The official Home Assistant Spotify integration demonstrates `https://my.home-assistant.io/redirect/oauth` and `/auth/external/callback` through Home Assistant's Application Credentials/config-flow machinery. A Supervisor App does not automatically inherit that machinery; using it would require a companion integration or another explicitly designed broker. + +The project MUST complete an OAuth callback spike for local-only and externally reachable Home Assistant deployments before provider authentication architecture is accepted. It must compare a direct App flow with a minimal companion-integration authorization broker and cover HTTPS, redirect registration, Ingress session continuity, remote access, dynamic paths, user-provided OAuth clients, state/PKCE/single use, denial/error flows, token ownership, integration/App version skew, backup/restore, and reauthorization. Any direct callback port must satisfy `SYM-SEC-009`. Tokens MUST NOT be pasted into App options or browser-export files as a normal workaround. + +See the dated [Home Assistant music ecosystem review](../providers/home-assistant-ecosystem-review.md) for the implementation evidence and security cautions behind these alternatives. + +## Observability + +- **SYM-OBS-001:** Every operation and provider request attempt MUST carry correlation identifiers for operation, run, step, connection, and adapter; provider object IDs are logged only under a documented privacy policy. +- **SYM-OBS-002:** Structured logs MUST record category, sanitized provider code, attempt, duration, and outcome without request/response bodies by default. +- **SYM-OBS-003:** Health means the process is alive; readiness additionally covers migrations, writable persistence, scheduler state, and ability to serve without claiming every provider is online. +- **SYM-OBS-004:** Metrics MUST cover queue depth/age, operation outcomes/duration, provider requests/latency/errors/throttles, resolution states, import freshness, and worker lease recovery. +- **SYM-OBS-005:** Diagnostics MUST be bounded, user-previewable, and safe to share. Secret-redaction tests are release gates. +- **SYM-OBS-006:** Audit history MUST say what Symphonia intended, attempted, observed, and could not determine. + +The MVP may render metrics in its UI and logs; choosing Prometheus/OpenTelemetry or another export is a later implementation decision. + +## Home Assistant contract + +- **SYM-HA-001:** The primary release artifact MUST be installable as a Supervisor-managed Home Assistant App from a documented repository. +- **SYM-HA-002:** The App MUST expose its management UI through Ingress and support Ingress-relative routing. +- **SYM-HA-003:** App restart, Supervisor backup/restore, and upgrade MUST preserve durable state and safely recover jobs. +- **SYM-HA-004:** Domain and application modules MUST NOT import Home Assistant or Supervisor libraries. +- **SYM-HA-005:** Home Assistant unavailability MUST NOT corrupt provider operations; future native surfaces report unavailable and recover. +- **SYM-HA-006:** A companion integration, if built, MUST consume a versioned local Symphonia contract and MUST NOT access the database or token store directly. +- **SYM-HA-007:** Native entities/actions/events MUST expose bounded summaries or identifiers, not playlist contents, tokens, or raw provider errors in state attributes. +- **SYM-HA-008:** Home Assistant automations that trigger mutations MUST create ordinary audited Symphonia operations subject to the same validation, idempotency, and policy as UI requests. +- **SYM-HA-009:** The MVP App panel MUST be admin-only; a future multi-user model MUST define which Home Assistant identity owns provider connections and may approve writes. + +## Technology decisions still open + +| Concern | Reason not yet chosen | Evidence needed | +| --- | --- | --- | +| Backend language/framework | Provider SDK maturity, job ergonomics, footprint, HA App maintainability | Thin vertical spike and maintainer preference | +| UI framework | Ingress routing/auth and complex review UI matter more than popularity | OAuth/Ingress and accessibility prototype | +| SQLite versus PostgreSQL | Concurrency, backup, migration, and library scale unmeasured | Storage/job lease spike and target sizes | +| Job library versus internal durable runner | Retry/idempotency needs are specific; external brokers add operations | Failure/restart spike | +| Secret encryption/key source | HA App secret facilities and portable standalone behavior differ | Threat model and backup/restore test | +| Companion integration transport | Need push, authentication, discovery, and version compatibility | Home Assistant integration RFC | +| Public API/event protocol | Only internal UI needs are currently concrete | UI and companion-integration contract design | + +No implementation agent should infer these choices from examples in `homeassistant-gateway`. diff --git a/docs/decisions/0001-provider-independent-recording-domain.md b/docs/decisions/0001-provider-independent-recording-domain.md new file mode 100644 index 0000000..bb9ea24 --- /dev/null +++ b/docs/decisions/0001-provider-independent-recording-domain.md @@ -0,0 +1,53 @@ +# ADR 0001: Provider-independent recording domain + +- **Status:** accepted +- **Date:** 2026-09-20 + +## Context + +Symphonia must relate music found at multiple providers. If core identities and workflows are shaped around Spotify IDs or YouTube videos, adding providers will leak provider assumptions into matching, playlists, persistence, and UI. Similar titles are also insufficient: a musical work can have covers, live performances, remixes, edits, and remasters. + +ISRC identifies a recording rather than an abstract composition, and even ISRC must be treated as evidence because provider metadata can be absent or inconsistent. + +## Decision + +Symphonia owns provider-independent domain identities. The MVP's primary matchable entity is `Recording` (shown to users as a Symphonia track). Provider tracks are representations linked to a recording by an explicit, evidenced identity link. + +`MusicalWork`, `Recording`, `Release`, and `ProviderTrack` are distinct concepts. Musical-work canonicalization is not required in the MVP, but the model must not collapse different recordings merely because they represent the same work. + +Provider APIs are infrastructure adapters behind capability-aware ports. Provider IDs remain opaque external identities and never become Symphonia IDs. + +## Alternatives considered + +### Use Spotify objects as the core model + +Rejected because Spotify would become a permanent dependency and other providers would be forced into Spotify semantics. + +### Use the lowest common provider fields as the domain + +Rejected because it discards important provenance and cannot express uncertainty, version distinctions, or richer providers. + +### Match musical works rather than recordings + +Rejected for the MVP because playlist interoperability normally needs a specific playable recording/version, not merely a composition. + +### Treat every provider track as unrelated + +Rejected because copying, synchronization, and reusable manual resolution require cross-provider identity. + +## Consequences + +Positive: + +- providers can be replaced and extended; +- manual decisions and operation history survive provider changes; +- ambiguity and contradictory evidence can be represented; +- covers and versions need not be conflated. + +Trade-offs: + +- import requires normalization and identity resolution; +- canonical metadata needs provenance/merge rules; +- provider-specific data must be stored alongside, not forced into, the core entity; +- migrations may be needed as recording/release/work knowledge improves. + diff --git a/docs/decisions/0002-copy-and-sync-are-distinct.md b/docs/decisions/0002-copy-and-sync-are-distinct.md new file mode 100644 index 0000000..d61e816 --- /dev/null +++ b/docs/decisions/0002-copy-and-sync-are-distinct.md @@ -0,0 +1,48 @@ +# ADR 0002: Copy and synchronization are distinct concepts + +- **Status:** accepted +- **Date:** 2026-09-20 + +## Context + +A user may want either a one-time transfer or a continuing relationship. Treating every copy as synchronization creates hidden future behavior; treating sync as repeated stateless copy loses baselines, attribution, direction, and conflict policy. + +The MVP should be useful before complex persistent synchronization is settled. + +## Decision + +Model a playlist copy as a finite `CopyPlan` and `CopyRun` derived from a captured source snapshot. Completion creates no continuing behavior. + +Model future synchronization as a separate durable `SyncRelationship` with participants, direction/mode, baseline, trigger/schedule, and conflict/unmatched policies. Each sync execution is an auditable run of that relationship. + +The MVP implements copy. Sync follows after copy semantics and provider feasibility are validated. + +## Alternatives considered + +### Every copy creates a sync relationship + +Rejected because users could trigger unexpected later writes and the MVP would need conflict/schedule semantics immediately. + +### Sync is a scheduled copy with no durable relationship + +Rejected because the system could not reliably attribute changes, detect concurrent edits, or explain deletion/reordering conflicts. + +### Implement sync before copy + +Rejected because it increases provider-write and recovery risk before matching and copy plans are proven. + +## Consequences + +Positive: + +- one-time intent is explicit and bounded; +- copy can deliver value with a smaller, safer workflow; +- future sync has a correct home for baselines and policy; +- operation history can distinguish a retry from a new recurring run. + +Trade-offs: + +- copy history must retain enough source/target evidence to inform future sync design; +- users who want ongoing mirroring must wait for a later release; +- promoting a copied pair into sync requires an explicit future workflow. + diff --git a/docs/decisions/0003-home-assistant-app-primary.md b/docs/decisions/0003-home-assistant-app-primary.md new file mode 100644 index 0000000..ff0c7a6 --- /dev/null +++ b/docs/decisions/0003-home-assistant-app-primary.md @@ -0,0 +1,54 @@ +# ADR 0003: Home Assistant App is the primary deployment boundary + +- **Status:** accepted +- **Date:** 2026-09-20 + +## Context + +Symphonia is intended to run continuously in a homelab and integrate deeply with Home Assistant. The owner clarified that it will be implemented as a Home Assistant app/service following the pattern established by `vypdev/homeassistant-gateway`. + +Home Assistant Supervisor provides an install/update lifecycle, Ingress UI authentication, persistent App data, backups, and a natural future integration surface. At the same time, music identity, provider access, jobs, and copy policy do not inherently belong to Home Assistant and must remain testable and usable independently. + +## Decision + +The primary supported distribution is a Supervisor-managed Home Assistant App installed from a Home Assistant App repository. It owns the long-running Symphonia service, durable jobs, provider connections, persistence, and management UI exposed through Ingress. + +The domain and application core remain independent of Home Assistant. A standalone composition profile will use the same core so the product can operate without Home Assistant and tests do not require it. Timing of the standalone release remains open. + +A companion Home Assistant custom integration may later expose native entities, actions, and events over a stable authenticated service contract. It will not duplicate domain policy or access the database/secrets directly. + +## Alternatives considered + +### Standalone service first with Home Assistant added later + +Rejected as the primary direction because it postpones the owner's intended installation, lifecycle, Ingress, and homelab experience. + +### Put all logic in a custom integration + +Rejected because long-running provider jobs, complex UI, dependency isolation, and durable service storage fit an App better, while custom integration lifecycle would couple the domain to Home Assistant Core. + +### Require Home Assistant in core modules + +Rejected because it damages testability, standalone usability, and architectural separation without improving provider/domain behavior. + +### App calls Home Assistant directly for every native feature + +Not selected. A companion integration, MQTT, and direct APIs need an RFC. Native surfaces should not force service policy into Home Assistant adapters. + +## Consequences + +Positive: + +- installation, startup, UI authentication, updates, backups, and persistence fit the target homelab; +- Home Assistant users get a first-class operational experience; +- a companion integration can remain thin and native; +- core logic remains portable and testable. + +Trade-offs: + +- Ingress-relative routing and provider OAuth callbacks need explicit design and testing; +- Home Assistant OS/Supervised becomes the primary release matrix; +- standalone mode needs its own authentication/network boundary; +- App and optional integration artifacts need version compatibility; +- provider credentials in Supervisor backups require a deliberate encryption/key/restore model. + diff --git a/docs/decisions/README.md b/docs/decisions/README.md new file mode 100644 index 0000000..0e78b1b --- /dev/null +++ b/docs/decisions/README.md @@ -0,0 +1,12 @@ +# Architecture decision records + +ADRs contain only decisions already justified and accepted by the product brief or owner clarification. Proposals and implementation choices remain in [open questions](../open-questions.md). + +| ADR | Status | Decision | +| --- | --- | --- | +| [0001](0001-provider-independent-recording-domain.md) | Accepted | Provider-independent recording domain | +| [0002](0002-copy-and-sync-are-distinct.md) | Accepted | One-time copy and persistent sync are distinct concepts | +| [0003](0003-home-assistant-app-primary.md) | Accepted | Home Assistant App is the primary deployment boundary | + +An ADR is immutable after acceptance except for typo/link corrections. A changed decision gets a new ADR that supersedes the old one. + diff --git a/docs/development/development-specification.md b/docs/development/development-specification.md new file mode 100644 index 0000000..d8b59ba --- /dev/null +++ b/docs/development/development-specification.md @@ -0,0 +1,178 @@ +# Development and verification specification + +**Status:** proposed baseline +**Last reviewed:** 2026-09-20 + +## Specification-driven workflow + +Implementation begins only when the applicable capability SDD is `Ready for implementation` and the owner explicitly approves the increment. The SDD lifecycle, required content, readiness gate, and catalog rules are defined in the [SDD standard](../../specs/README.md). Start new capability designs from the [mandatory template](../../specs/_template.md), and keep their state in the [catalog](../../specs/CATALOG.md). + +For each proposed increment: + +1. Identify product outcomes and requirement IDs. +2. Resolve any open decision that changes externally observable behavior. +3. Write or update the cataloged SDD, including examples, denial/error behavior, provider capability needs, and a numeric risk-derived test budget. +4. Add or revise an ADR only for a durable consequential decision. +5. Define acceptance tests and fixture data before production code. +6. Review the SDD against every readiness gate and obtain explicit owner approval. +7. Implement inward-facing domain/application behavior first, then adapters/presentation. +8. Run focused tests, the complete local suite, packaging checks, and secret scans. +9. Review the final diff for requirement, SDD, catalog evidence, documentation, migration, and diagnostics consistency. + +Product requirements say **what** must happen; domain documents define terms/invariants; ADRs say **why** a durable choice was made; an SDD says **how** one vertical capability will behave, fail, recover, be verified, and become ready to ship. + +## SDD content + +The complete required structure lives in [`specs/_template.md`](../../specs/_template.md). At minimum, every material SDD contains: + +- status, owner, date, and related requirement IDs; +- user outcome and explicit non-goals; +- preconditions and required connection capabilities; +- normal, empty, ambiguous, partial, denial, timeout, restart, and cancellation examples; +- state transitions and transaction/idempotency boundaries; +- security/privacy and provider-policy implications; +- observable events, metrics, history, and user-facing errors; +- migration/rollback impact; +- a numeric, risk-derived automated test budget and acceptance scenarios; and +- unresolved questions clearly separated from decisions. + +Small implementation changes may update an existing SDD instead of creating one. They may not bypass the applicable contract, weaken its verification budget silently, or use code as the only record of a product or architectural decision. + +## Test strategy + +### Domain unit and property tests + +Fast, deterministic tests cover invariants without database, network, clock, environment, Home Assistant, or provider SDKs. + +Examples: + +- a provider ID never becomes a recording ID; +- order and duplicate occurrences survive playlist transformations; +- a manual rejection prevents automatic relinking; +- plan digests change when inputs/capabilities/policy change; +- job and operation state transitions reject impossible moves; +- matching normalization is Unicode-safe and retains originals; and +- retries never convert an unknown write into a blind duplicate write. + +Property/generative tests SHOULD cover ordered-list diffs, duplicates, batching boundaries, Unicode metadata, duration tolerances, and replay/idempotency. + +### Application/use-case tests + +Use in-memory deterministic ports for providers, repositories, jobs, clock, identifiers, and secret references. Cover every user journey and denial/partial path. Tests assert domain outcomes and port interactions, not framework internals. + +### Provider contract tests + +One shared suite runs against every adapter's offline fake/fixture implementation and concrete mapping layer. It verifies: + +- pagination and no silent truncation; +- media/unavailable/unknown-type handling; +- adapter-manifest classification plus adapter-, connection-, object-, and health-level capability behavior; +- colliding upstream IDs across provider instances and media types; +- normalized errors and retry advice; +- batch/order/duplicate behavior; +- timeout/cancellation; +- secret-safe exceptions/logs; and +- reconciliation after a simulated lost write response. + +Fixtures MUST be synthetic or legally redistributable, minimal, versioned, and free of real tokens/account identifiers. Raw captured payloads require sanitization and policy review before commit. + +### Persistence and migration tests + +Run against the real selected database engine. Cover transactions, constraints, lease contention, restart recovery, consistent backup, restore, retention cleanup, and migrations from every supported release fixture. Migration tests must include failure/interruption behavior. + +### Job/fault tests + +Use a controlled clock and fault-injecting providers to test 429/reset, timeouts before/after writes, 401 refresh races, 403, 404, 5xx, malformed payloads, truncated pagination, process termination at each checkpoint, stale leases, cancellation, and provider revision conflicts. + +### HTTP/UI tests + +Contract tests validate request/response schemas, authentication, authorization, pagination, base paths, correlation, and sanitized errors. Browser tests cover Ingress-relative navigation, narrow/wide layouts, keyboard access, focus/error announcements, OAuth return/error screens, ambiguous-resolution review, copy preview, partial results, reconnect, and restart recovery. + +UI tests MUST not consider an element visible or a request successful proof that the domain outcome occurred; they verify the returned operation and history. + +### Home Assistant App tests + +Artifact tests inspect repository/App metadata, supported architectures, pinned image inputs, Ingress configuration, startup, persistent `/data`, backup settings, non-root execution, mapped folders, exposed ports, least privileges, health, and version consistency. A fake Supervisor/Ingress boundary should exercise path rewriting and trusted-header isolation. If a direct callback listener exists, tests must prove that its management and internal API routes are unreachable. + +At least one disposable Home Assistant OS/Supervised-compatible smoke path must verify install/start/Ingress/restart/backup-restore before stable release. The exact CI environment is an implementation decision. + +### Standalone tests + +If/when shipped, start the published immutable image with a fresh volume and explicit authentication, then verify migration, health/readiness, provider fake workflow, restart persistence, backup/restore, and private-by-default network configuration. + +### Live-provider smoke tests + +Live tests are opt-in and never part of the ordinary suite. They use dedicated accounts, unique prefixed playlists, bounded item counts, conservative quotas, and cleanup that reports rather than hides failure. Destructive cleanup targets only IDs created by that run. Credentials live only in an approved secret facility. + +The live matrix must verify documented behavior, not reverse-engineered endpoints unless a separately accepted unofficial adapter exists. + +## Test requirements + +- **SYM-TEST-001:** Every normative requirement implemented MUST be traceable to at least one automated test or an explicitly documented manual/platform verification. +- **SYM-TEST-002:** Domain and application test suites MUST run without network, provider accounts, Home Assistant, or wall-clock timing. +- **SYM-TEST-003:** Every provider adapter MUST pass the common offline contract suite. +- **SYM-TEST-004:** Provider write workflows MUST test success, rate limit, auth expiry, permanent rejection, partial batch, timeout before write, unknown outcome after write, reconciliation, and restart. +- **SYM-TEST-005:** Secret-canary values MUST be asserted absent from logs, exceptions, API responses, metrics labels, traces, diagnostics, database non-secret fields, and UI snapshots. +- **SYM-TEST-006:** Tests MUST cover duplicated playlist entries, unavailable/deleted items, non-music media, empty playlists, maximum batch boundaries, and source change after planning. +- **SYM-TEST-007:** Matching evaluation MUST use a reviewed corpus containing covers, live/studio, remaster, remix/edit, acoustic, clean/explicit, featured-artist, compilation, localization, and misleading-title cases. +- **SYM-TEST-008:** Resolver changes MUST report corpus regressions by category and MUST NOT silently overwrite manual decisions. +- **SYM-TEST-009:** App packaging and Ingress-relative routing MUST be release-gated. +- **SYM-TEST-010:** Tests that touch external providers MUST be explicitly selected, quota-bounded, and safe to re-run. +- **SYM-TEST-011:** Provider tests MUST cover object-level capability denial and identical upstream IDs from different media types or provider instances without collision. +- **SYM-TEST-012:** App artifact/smoke tests MUST enumerate every listening port and prove that any non-Ingress callback listener cannot reach management routes or trust Ingress identity headers. +- **SYM-TEST-013:** Migration tests MUST use a per-release history, include stable/beta path divergence, and prove failure leaves the prior database recoverable rather than replacing locally authored state with a fresh rescan. + +## Matching evaluation + +Before accepting automatic-match thresholds, create a small human-labeled corpus whose licenses/terms permit the stored fields. Separate candidate-retrieval recall from link precision. False-positive links are generally more harmful than unmatched items because they silently copy the wrong recording, so metrics and review must report both. + +Do not train an ML model on Spotify content; Spotify's current search documentation explicitly prohibits using Spotify content to train machine-learning/AI models. “Learning” in the MVP means deterministic persistence and reuse of user mappings. + +Numeric acceptance thresholds are open until the corpus and error costs are reviewed. + +## Quality gates + +A change is incomplete until, in proportion to its scope: + +- specifications and requirement traceability agree with behavior; +- tests cover success, denial, error, partial, and restart paths; +- provider calls have timeouts, rate-limit handling, and sanitized errors; +- no secret or real private identifier is present in source/fixtures/output; +- migrations, backup, and rollback implications are documented and tested; +- frontend accessibility and Ingress base-path behavior pass; +- App/standalone artifact metadata is consistent; +- local checks and CI pass; and +- the full diff contains no unrelated generated artifacts. + +## Deployment and release expectations + +- **SYM-DEP-001:** The primary release MUST provide a Home Assistant App repository, immutable versioned images, checksums/signing where the selected publication workflow supports it, and upgrade notes. +- **SYM-DEP-002:** The App MUST persist only documented runtime state under `/data` and MUST start successfully after restore without performing provider writes until migrations/recovery complete. +- **SYM-DEP-003:** A release MUST document supported Home Assistant versions and CPU architectures. +- **SYM-DEP-004:** Configuration validation MUST fail closed with actionable messages before workers start. +- **SYM-DEP-005:** The deployment MUST provide health, readiness, version, and sanitized diagnostics. +- **SYM-DEP-006:** Backup and restore documentation MUST cover the database, encryption-key material, provider reauthorization consequences, and active jobs. +- **SYM-DEP-007:** Updates MUST be reversible via documented backup/restore or compatible downgrade policy; unsupported downgrade MUST be detected, not attempted. +- **SYM-DEP-008:** The service MUST handle graceful shutdown by stopping new work, checkpointing or abandoning leases safely, and bounding shutdown time. +- **SYM-DEP-009:** Default App permissions and network exposure MUST be minimal; Docker socket, host network, SSH, arbitrary host filesystem, and privileged mode are prohibited unless a future ADR proves necessity. +- **SYM-DEP-010:** Standalone mode, if released, MUST use the same domain/application code and migration format as the Home Assistant App. + +Release channels, semantic-version policy, supported upgrade window, and exact artifact tooling remain open implementation decisions. + +## Initial traceability examples + +| Requirement | Planned verification | +| --- | --- | +| `SYM-MATCH-006` manual reuse | Domain unit + application workflow + resolver corpus regression test | +| `SYM-PL-005` order/duplicates | Property test + both provider contract suites + live smoke | +| `SYM-PL-006` idempotency | Fault-injected unknown-write/reconciliation/restart test | +| `SYM-PROV-004` pagination | Shared adapter contract at zero, one, boundary, and multi-page counts | +| `SYM-PROV-018` external identity scope | Contract fixtures with cross-type and cross-instance ID collisions | +| `SYM-JOB-004` durable retry | Database/controlled-clock restart test | +| `SYM-SEC-004` secret safety | Canary scan across all output boundaries | +| `SYM-SEC-009` callback isolation | Listener route enumeration + Home Assistant App smoke test | +| `SYM-SEC-010` untrusted provider values | Path/URI/redirect/egress injection tests | +| `SYM-HA-002` Ingress | Base-path HTTP/browser + App smoke test | +| `SYM-DEP-006` restore | Artifact smoke using a backup with active/waiting jobs | + +The SDD catalog and each capability's traceability section are the prospective matrix before code exists. Implementation evidence should be generated or maintained when work is approved; this baseline does not invent test filenames before a stack exists. diff --git a/docs/domain/domain-model.md b/docs/domain/domain-model.md new file mode 100644 index 0000000..91beb84 --- /dev/null +++ b/docs/domain/domain-model.md @@ -0,0 +1,316 @@ +# Domain model + +**Status:** provider-independent core accepted; playlist ownership and matching policy proposed/open +**Last reviewed:** 2026-09-20 + +## Model boundary + +Symphonia models a person's relationship with recordings and provider collections. It does not model audio files or playback. Provider payloads enter through adapters and are translated into this language before use by product workflows. + +This document specifies concepts and invariants, not database tables or API schemas. + +## Ubiquitous language + +| Term | Meaning | Important distinction | +| --- | --- | --- | +| **Musical work** | An abstract composition: melody, lyrics, and authorship independent of a performance. | “Hallelujah” as a composition is not Leonard Cohen's recording or a cover. Not an MVP identity target. | +| **Recording** | A particular recorded performance, mix, edit, or version. This is the primary provider-independent identity in the MVP. | Studio, live, acoustic, cover, remix, remaster, clean, and explicit versions may be different recordings. | +| **Symphonia track** | User-facing shorthand for a `Recording`; not a separate provider-shaped object. | Avoid using unqualified “track” in designs where identity matters. | +| **Release** | A published album, single, EP, or compilation edition containing recordings. | Different releases can contain the same recording; release identity is not recording identity. | +| **Artist credit** | The ordered credited performers attached to a recording or release. | A credit is not yet a canonical person/group identity. | +| **Provider** | A type of external system, such as Spotify or YouTube. | A provider is not an account. | +| **Provider connection** | One authorized external account plus its effective capabilities and credential reference. | More than one connection may eventually exist for one provider. | +| **Provider track** | A provider catalog item that represents or makes a recording available. | It retains the provider ID and may have incomplete or conflicting metadata. A YouTube video is a provider track only when used as a music candidate. | +| **Provider playlist** | An ordered collection owned by a provider/account and imported into Symphonia. | It is not automatically a Symphonia-owned logical playlist. | +| **Playlist entry** | One occurrence at one position in a playlist snapshot. | Repeated tracks are separate entries and must not be collapsed. | +| **Snapshot** | Symphonia's immutable observation of provider state at a point/version. | It does not claim the provider is still unchanged. | +| **Identity link** | The assertion that one provider track represents one recording. | The assertion has origin, evidence, confidence class, and lifecycle. | +| **Candidate** | A possible identity link awaiting automatic or manual disposition. | Candidate does not mean match. | +| **Manual resolution** | A user's accepted or rejected identity assertion. | It outranks later automatic guesses until explicitly invalidated or revoked. | +| **Copy plan** | Immutable, non-mutating instructions derived from one source snapshot and target capabilities. | It is not a sync relationship. | +| **Copy run** | One finite execution of an accepted copy plan. | A retry continues the logical run when safe; a re-plan is a new run. | +| **Sync relationship** | Persistent policy connecting two or more playlist projections. | It has recurring runs, baselines, direction, and conflict semantics. Post-MVP. | +| **Operation** | User-visible unit of work with status, checkpoints, outcomes, and issues. | An operation may contain many provider requests. | +| **Issue** | An actionable mismatch, ambiguity, provider failure, or policy conflict attached to an operation/item. | Exceptions and logs alone are not the product history. | + +## Conceptual relationships + +```text +LocalUser + └─ ProviderConnection ── Provider + ├─ imports ── ProviderTrack ── IdentityLink ── Recording + └─ imports ── ProviderPlaylist + └─ ordered PlaylistEntry ── ProviderTrack + +ProviderPlaylistSnapshot ── source of ── CopyPlan +CopyPlan ── executed as ── CopyRun ── emits ── OperationIssue + +Future: PlaylistProjection(s) ── governed by ── SyncRelationship +``` + +The arrows do not imply persistence ownership. In particular, deleting a connection does not make a recording cease to exist. + +## Core entities and value objects + +### Recording + +The canonical matchable item for the MVP. + +Minimum conceptual attributes: + +- stable Symphonia identifier; +- display title and version descriptors; +- ordered artist credits; +- duration and explicitness when known; +- recording identifiers such as ISRC, each with provenance; +- descriptive release metadata when useful to matching; +- creation/update timestamps; and +- merged/superseded lifecycle state if deduplication is later required. + +A Recording's descriptive fields can be synthesized from provider evidence, but every value SHOULD retain provenance or derivation. Provider metadata disagreement must not be erased by last-write-wins updates. + +### Musical work + +Musical work is included in the language to prevent a category error, not as an MVP aggregate. The MVP MUST NOT merge covers merely because their titles and writers coincide. A later model may connect many recordings to one work. + +### Provider connection + +Minimum conceptual attributes: + +- stable Symphonia identifier; +- provider type and immutable provider-account identity; +- user-chosen display name; +- credential reference, never a token embedded in normal domain payloads; +- requested, granted, and effective capability sets; +- connection health and user-action state; +- token/authorization expiry metadata when available; +- last attempted/successful import times; and +- policy/terms acknowledgement metadata where required. + +### Provider track + +Minimum conceptual attributes: + +- provider type, connection where access is account/market dependent, and provider object ID; +- provider URI/URL when safe to retain; +- provider media kind and availability; +- original metadata snapshot and normalized matching fields; +- provider revision/ETag/update markers when available; +- observed and refresh-by timestamps; and +- policy classification for retention and deletion. + +The external identity key is at least `(provider type, provider object type, provider object ID)` and includes a provider-instance/connection, market, or catalog namespace whenever the provider does not guarantee global uniqueness. It MUST NOT use a mutable title. Whether the same proven-global catalog ID shared through two connections is stored once or projected per connection is a persistence design question; account-specific availability must remain representable. A personal or dynamic collection ID MUST NOT be assumed globally unique merely because it is stable inside one account. + +### Provider playlist and snapshot + +A provider playlist retains its external identity, owner, visibility, description, collaborative flags, provider revision token, and capability constraints. Import creates an immutable snapshot with ordered entries. + +Each entry records: + +- stable snapshot-local entry ID; +- ordinal/position; +- provider item ID and media kind; +- occurrence-specific provider entry ID when exposed; +- added-at/added-by metadata when exposed; +- availability at observation time; and +- eventual recording resolution state. + +An entry can be unavailable or deleted while its historical place in a snapshot remains meaningful. + +### Identity link and candidate + +An accepted identity link contains: + +- provider track ID and recording ID; +- origin: deterministic external ID, heuristic, or manual; +- confidence class; +- positive and negative evidence; +- resolver name and version; +- actor and timestamps; +- status: active, revoked, superseded, or invalidated; and +- invalidation reason where applicable. + +A rejected candidate pair is durable evidence and MUST be consulted by future resolver versions. It is not represented as the absence of a link. + +### Operation, step, and issue + +An operation is the user-facing aggregate for import, resolution, copy, and later sync. It owns: + +- type, initiator, correlation ID, and idempotency key; +- input references and immutable plan/version where applicable; +- state and timestamps; +- resumable steps/checkpoints; +- item outcomes; +- sanitized provider-request metadata; +- issues and their resolution state; and +- result summary. + +Operational events may be pruned under a retention policy, but the final audit summary and manual decisions have separate retention lifecycles. + +## Domain invariants + +1. A provider track has zero or one active identity link to a recording. +2. A recording may have many provider tracks, including multiple tracks from the same provider. +3. A provider ID is never a Symphonia recording ID. +4. An ISRC is evidence for recording identity, not proof of musical-work identity or infallible uniqueness. +5. An imported playlist's order and duplicate occurrences are significant. +6. Provider facts, derived values, and user decisions have separate provenance. +7. A manual decision is never overwritten silently by an automatic resolver. +8. A plan is immutable; changed inputs produce a new plan version. +9. A capability is evaluated for the adapter, connection, affected object, and current provider health at operation time, not assumed globally from provider type. +10. Historical provider writes are append-only audit facts, even when a later compensating action reverses their effect. + +## Identity resolution specification + +### The problem + +Providers describe overlapping but non-identical catalogs. Metadata can be missing, localized, inconsistent, wrongly attributed, market-specific, or duplicated. The resolver must decide whether two representations refer to the same recording without collapsing a musical work's different performances or versions. + +ISRC is valuable because it identifies recordings, but it is not sufficient alone: provider data may omit it, reuse it unexpectedly, attach it at a different version granularity, or expose a video rather than a label-issued recording. Title similarity is weaker and can confuse covers, live performances, edits, and remasters. + +### Resolution pipeline + +The following stages are requirements; their algorithms and thresholds remain open: + +1. **Normalize without destroying source data.** Produce comparison fields for Unicode, punctuation, featured artists, version tokens, and durations while retaining originals. +2. **Generate candidates.** Use exact identifiers where available and provider search/metadata for bounded candidate retrieval. +3. **Evaluate evidence.** Consider identifiers, title, ordered/credited artists, duration tolerance, release context, explicitness, version markers, market availability, and provider type. +4. **Classify.** Produce `exact`, `high_confidence`, `ambiguous`, or `unmatched` as a candidate assessment, then set the provider track's resolution state. +5. **Explain.** Store the material evidence and resolver version. +6. **Apply policy.** Only policy-approved classes may create automatic links; ambiguous results enter the review queue. +7. **Learn decisions.** Persist accepted and rejected pairs for deterministic reuse. “Learn” means rule reuse, not training an ML model on provider content. + +### Confidence semantics + +| Class | Meaning | Permitted MVP behavior | +| --- | --- | --- | +| `exact` | Meets a future, explicit deterministic rule with no known contradictory evidence. | May auto-link once exact rules are approved. | +| `high_confidence` | Strong multi-field evidence but not deterministic. | Auto-link versus review is an open product threshold. | +| `ambiguous` | At least two plausible candidates or material contradictory evidence. | MUST require user resolution before a strict copy. | +| `unmatched` | No acceptable candidate was found within bounded search. | MUST remain visible; MAY be retried after metadata/provider changes. | + +The labels are not percentages. No numeric thresholds are accepted in this baseline. + +### Manual-resolution behavior + +- Accepting a candidate activates or creates an identity link. +- Rejecting a candidate stores a negative assertion scoped to the two identities. +- “Distinct recording” creates a new Recording rather than linking to an inappropriate existing one. +- “Defer” records that the user intentionally postponed the choice and avoids presenting it as a new failure on every screen. +- Revocation preserves who made the original decision and why it changed. +- A provider object that changes identity-like metadata materially is flagged for review; it is not silently rematched over a manual link. + +## Playlist copy model + +Copy is a finite workflow: + +```text +source snapshot → resolution/capability analysis → immutable plan + → user acceptance → target writes → reconciliation → result +``` + +### Plan content + +A copy plan records source playlist/snapshot, target connection, intended target name/visibility, source entry order, target provider track for each ready entry, match evidence, non-ready reason, capabilities observed, unresolved-item policy, and a digest used to detect tampering or accidental version mixing. + +Planning MUST NOT mutate a provider. Target search calls used during planning are reads. + +### Execution behavior + +Execution creates or selects a target only as described in the accepted plan. Writes are checkpointed in provider-safe batches. On retry, Symphonia first determines whether a timed-out write took effect. If provider APIs cannot make a write provably idempotent, the result becomes `needs_reconciliation` rather than risking duplicate entries. + +The following remain open for the copy RFC: + +- strict all-resolved versus best-effort as the default; +- whether an existing target can be appended to in the MVP; +- rollback/cleanup behavior after partial target creation; +- visibility/name collision behavior; and +- user confirmation rules for a re-executed plan. + +## Playlist synchronization model + +Persistent sync is post-MVP but constrains what history copy must retain. + +- **SYM-SYNC-001:** A sync relationship MUST be a durable object separate from its individual runs. +- **SYM-SYNC-002:** A relationship MUST declare participants, direction/mode, field and entry policies, schedule/trigger, unmatched-item policy, and conflict policy. +- **SYM-SYNC-003:** Each run MUST compare current provider snapshots with a known baseline; current-state comparison alone is insufficient to attribute changes. +- **SYM-SYNC-004:** Symphonia MUST distinguish observed external changes, writes made by Symphonia, and uncertain changes. +- **SYM-SYNC-005:** Add, remove, reorder, metadata change, duplicate-entry change, and playlist deletion MUST be distinct change kinds where provider evidence permits. +- **SYM-SYNC-006:** A run MUST be replay-safe and MUST checkpoint provider writes. +- **SYM-SYNC-007:** Conflicts MUST remain visible until a declared policy or user action resolves them. +- **SYM-SYNC-008:** A provider without sufficient change/version evidence MUST declare degraded sync semantics; polling MUST NOT be presented as lossless real-time sync. + +Potential modes are: + +- **source-of-truth/mirror:** one projection defines desired state; target divergence is overwritten or reported according to policy; +- **add-only:** newly observed additions propagate; removals and reorderings do not; +- **bidirectional:** changes on both sides propagate using a common baseline and explicit conflict rules. + +These modes are vocabulary, not accepted MVP behavior. + +## Playlist ownership RFC: provider-owned or Symphonia-owned + +This decision is intentionally open and must be made before persistent sync is designed. + +### Option A — provider-owned playlists with relationships + +Provider playlists remain primary objects. A copy points from one provider snapshot to target writes; a sync relationship connects provider playlists. + +**Advantages** + +- Matches the initial user mental model: “copy Spotify/Rock to YouTube/Rock.” +- Requires less local authoring UI and fewer rules about which edits are authoritative. +- Supports a smaller copy-first MVP and respects provider ownership/visibility semantics. + +**Costs** + +- Bidirectional sync needs careful baselines and conflict attribution. +- Losing/deleting a provider playlist can remove an apparent anchor. +- Relationships with more than two projections and offline editing are awkward. +- A later logical-playlist model may require migration. + +### Option B — Symphonia-owned logical playlists with projections + +A logical playlist is authoritative inside Symphonia; provider playlists are projections of its desired state. + +**Advantages** + +- Gives the user a durable provider-independent object. +- Makes multiple projections and source replacement conceptually clean. +- Provides a natural place for local edits, desired order, and policy. + +**Costs** + +- Requires rules for importing ownership, provider-side edits, projection drift, deletion, and conflicts immediately. +- Risks turning Symphonia into another library editor before basic interoperability works. +- Provider limitations can make a projection unable to represent the logical list exactly. + +### Option C — staged hybrid + +Ship provider-owned copy first, retain immutable snapshots and stable recording identities, then allow a user to promote a relationship or imported playlist into a logical playlist after a dedicated RFC. + +This is the current **proposed direction**, not an accepted decision. It minimizes MVP scope while preserving a migration path. Implementation must avoid naming a provider playlist ID as a logical playlist ID and must retain origin/projection metadata. + +## Lifecycle and deletion + +Provider-sourced metadata is a renewable observation and may be subject to provider-specific refresh/deletion policy. User-authored resolution decisions and operation facts are locally owned, but they must not retain prohibited provider payloads indefinitely by embedding raw metadata. + +Deletion design MUST distinguish: + +- disconnect and revoke credentials; +- delete cached provider data under policy; +- delete a provider object through an authorized operation; +- delete local history; and +- forget/export all local user data. + +Exact retention periods and cascade behavior require the security/privacy RFC. + +## Deliberately unresolved domain questions + +- Can one ProviderTrack map to different Recordings by market or provider metadata revision? +- When are two ISRC-bearing items automatically exact, and what contradictory evidence blocks that? +- Are clean and explicit variants always distinct Recordings for the user's purpose? +- Should releases and artists become canonical entities in the MVP or remain credited metadata? +- What is the stable identity of a YouTube music candidate: video, song entity, or another officially exposed object? +- What library membership means across providers: saved track, liked video/song, followed release, or a provider-specific collection? +- Which playlist ownership option should govern post-copy synchronization? diff --git a/docs/open-questions.md b/docs/open-questions.md new file mode 100644 index 0000000..a09b9a6 --- /dev/null +++ b/docs/open-questions.md @@ -0,0 +1,164 @@ +# Open questions, risks, and next design work + +**Status:** open; nothing here is an accepted decision +**Last reviewed:** 2026-09-20 + +## Decisions requiring owner input + +These are ordered by how soon they block the next specification pass. + +### OQ-001 — What does “YouTube Music provider” mean for the MVP? + +Official research currently proves a YouTube Data API for video playlists, not a full YouTube Music library/catalog API. + +Existing Home Assistant projects prove that useful YouTube Music access is technically possible through reverse-engineered web endpoints, browser cookies, `ytmusicapi`, and proof-of-origin tokens. This reduces implementation-feasibility uncertainty but increases the evidence for authentication, account, policy, completeness, and maintenance risk; see the [ecosystem review](providers/home-assistant-ecosystem-review.md#youtube-music). + +Options after the feasibility spike: + +1. MVP supports official ordinary YouTube playlists and labels limitations clearly. +2. Defer the Google/YouTube side and choose another provider with an official music API. +3. Add a separately named unofficial YouTube Music adapter with explicit stability, authentication, terms, and account-risk warnings. +4. Accept a narrower hybrid, such as copying Spotify recordings to best-match YouTube videos without claiming library parity. + +**Proposed default:** decide only after a test-account spike; never silently substitute reverse-engineered access. + +### OQ-002 — Which playlist owns desired state after the copy MVP? + +Choose provider-owned relationships (Option A), Symphonia-owned logical playlists (Option B), or the staged hybrid described in the [domain model](domain/domain-model.md#playlist-ownership-rfc-provider-owned-or-symphonia-owned). + +**Proposed default:** staged hybrid—provider-owned copy first, retain enough origin/snapshot data to promote later. This is not accepted and persistent sync must not be designed until it is decided. + +### OQ-003 — What is the default treatment of non-ready entries in a copy? + +Options include: + +- strict: no writes until every entry is resolved/supported; +- confirmed best effort: user explicitly accepts omissions; +- create target and pause at unresolved entries; or +- allow placeholders (only if a provider has a meaningful representation). + +**Proposed default:** strict by default with an explicit, itemized best-effort override. Rollback/cleanup after partial provider writes still needs design. + +### OQ-004 — Which provider OAuth boundary should self-hosters use? + +Possibilities include project-owned shared client registrations, bring-your-own client credentials per install, or both. Spotify's five-user Development Mode cap strongly favors bring-your-own credentials for a distributed self-hosted project, but callback registration and support become harder. + +The callback boundary also has two credible shapes: a direct App-owned OAuth flow, or a minimal companion integration that uses Home Assistant Application Credentials/config flows and brokers a one-use connection grant to the App. The latter reuses Home Assistant UX but adds token ownership, backup, revocation, and version-skew questions. Apple Music would add a third, MusicKit-specific token model if promoted into scope. + +This decision requires the Home Assistant OAuth callback spike. It also affects documentation, verification, secret storage, direct port exposure, and whether remote Home Assistant access is needed. + +### OQ-005 — What authentication/exposure does standalone mode use, and when does it ship? + +Ingress supplies the primary App UI boundary. Standalone has no equivalent. Decide whether it ships in the MVP and choose local-only, reverse-proxy identity, built-in local credentials/passkeys, or another bounded mechanism. It must not bind an unauthenticated administrative UI broadly by default. + +### OQ-006 — How many provider connections per type does the MVP UI support? + +The domain supports multiple connections. The smallest UI could allow one Spotify and one Google connection while preserving connection IDs. Multiple family accounts improve usefulness but expand authorization, source/target selection, quota, and deletion UX. + +### OQ-007 — What are the first-release size and hardware targets? + +Provide representative numbers for saved recordings, playlists, largest playlist, provider connections, concurrent operations, target Home Assistant hardware, and acceptable import duration. These are needed before accepting database, pagination/cache, and measurable performance requirements. + +### OQ-008 — What is the local-data/export policy? + +Decide retention for immutable snapshots, operation detail, provider metadata, logs, rejected candidates, and deleted connections; whether the user can export/import mappings; and whether “forget all” is required in the MVP. Provider policy may impose stricter refresh/deletion behavior than product preference. + +### OQ-009 — Which open-source license and contribution policy apply? + +The repository currently has no license. The license should be selected before accepting outside contributions. Contributor handling of provider fixtures, terms, security reports, and trademarks also needs a policy. + +## Research/design gates (not owner preference alone) + +### RG-001 — Official provider feasibility + +Run the dated Spotify and YouTube test-account matrix in [provider research](providers/provider-research.md). Record scopes, account tier, app mode, market, exact endpoints, quota cost, payload gaps, playlist visibility, duplicate/order behavior, and cleanup results. Do not use personal libraries as fixtures. + +If Apple Music is considered as a future provider or fallback for an official music-library surface, run its separate feasibility gates without silently changing the MVP. Community providers may inform test cases but cannot substitute for official-contract evidence. + +### RG-002 — OAuth through a Home Assistant App + +Prototype authorization start/callback/error for Spotify and Google using both a direct App-owned flow and, where viable, a minimal companion-integration broker built on Home Assistant Application Credentials. Test: + +- local and externally reachable Home Assistant URLs; +- HTTPS and exact registered redirects; +- Ingress base path/session and multiple browser users; +- OAuth `state`, PKCE where applicable, single use, expiry, denial, and restart; +- user-provided OAuth client registrations; and +- token ownership, refresh, revocation, App/integration version skew, and backup/restore; +- a callback-only direct listener that does not expose management routes; and +- no secrets in App options, browser-export files, URL query logs, referrers, or diagnostics. + +The result becomes an authentication/secret-storage RFC, not production code. + +### RG-003 — Matching evidence corpus + +Build and review a licensed/synthetic labeled corpus containing exact duplicates, missing/wrong ISRC, covers, live/studio, remasters, remixes/edits, clean/explicit, acoustic, compilations, featured artists, localized metadata, music/lyric videos, and misleading titles. Use it to specify confidence rules and measurable false-link tolerances. + +### RG-004 — Durable operation and storage spike + +Compare SQLite and PostgreSQL for transaction boundaries, leases, crash recovery, unknown writes, snapshot/history queries, online/cold backup under Supervisor, encryption-key restore, representative library size, and a per-migration applied ledger across stable/beta upgrade paths. Prove that a failed migration never replaces irrecoverable manual decisions/audit state with a fresh rescan. No framework selection should precede these results. + +### RG-005 — Home Assistant native surface RFC + +Define what belongs in the App UI versus a companion custom integration. Decide transport/auth/version discovery and a minimal entity/action/event surface. Ensure Home Assistant actions create ordinary audited Symphonia operations and that the integration can be unavailable independently. + +## Future synchronization questions + +These do not block the copy MVP but block sync implementation: + +- Which modes ship first: mirror, add-only, or bidirectional? +- Does order matter in every mode and how are duplicate occurrences identified? +- What baseline identifies “changed on both sides” when a provider lacks a reliable revision token? +- Are target-side additions preserved, propagated, or conflicts in mirror mode? +- How are removals, playlist deletion, privacy change, renames, and unavailable tracks handled? +- What happens when a previously resolved provider item becomes unavailable or changes metadata materially? +- How are loops prevented when Symphonia observes its own delayed provider writes? +- What schedule, jitter, quiet hours, manual trigger, and Home Assistant automation behavior are expected? +- How long can a relationship be degraded before the UI requires intervention? + +## Implementation decisions intentionally deferred + +- backend language and framework; +- UI framework/design system; +- SQLite versus PostgreSQL; +- internal durable runner versus job library; +- secret encryption primitive and key source; +- API/event protocol and versioning; +- companion integration transport; +- App CPU architectures and supported Home Assistant versions; +- release channels and version/support policy; +- exact matching scores/thresholds; and +- telemetry exporters (if any; no external telemetry by default is implied for self-hosting). + +These need evidence and small RFCs; popularity is not evidence. + +## Risk register + +| Risk / assumption | Likelihood | Impact | Current response | +| --- | --- | --- | --- | +| Official YouTube API does not provide the intended YouTube Music model | High | Critical: symmetric MVP promise fails | `RG-001`, then owner choice `OQ-001` | +| Unofficial YouTube Music access requires reusable browser-account cookies and moving private endpoints | High | Critical security/account/support risk | Separate opt-in adapter only; threat model; no silent fallback; `OQ-001` | +| OAuth callback cannot be made reliable/simple through Ingress for bring-your-own clients | Medium-high | Critical: users cannot connect providers safely | `RG-002`; keep callback design open | +| A direct callback port accidentally exposes the management API outside Ingress | Medium | Critical compromise risk | Callback-only listener, independent auth, non-root/least privilege, App artifact tests | +| Spotify Development Mode endpoints/policy change again or required endpoint is restricted | Medium-high | High | Capability probes, dated matrix, graceful disable, avoid hosted assumptions | +| Spotify six-month refresh-token lifetime creates unexpected reconnect failures | High under current docs | Medium-high | Expiry tracking, warnings, `waiting_user`, easy reauthorization | +| Google quota/search budget makes track-by-track matching impractical | High for naive search | High | Batch/cache within policy, plan quota, corpus, bounded candidate strategies | +| YouTube 30-day refresh/deletion policy conflicts with durable local identity/history | Medium | High | Separate provider payloads from user decisions; retention RFC | +| ISRC/similar metadata produces false links across versions | High | High | Conservative rules, evidence, review queue, negative/manual mappings | +| Provider write timeout creates duplicate playlist entries on retry | Medium | High | Checkpoints, stable keys, target reconciliation, unknown-outcome state | +| Provider-owned versus logical playlist choice is delayed until sync code | Medium | High architectural rework | Decide before sync; keep copy/snapshots model-neutral | +| App backups contain usable provider tokens or lose the decryption key | Medium | Critical security/recovery failure | Threat model, external key design, restore tests, sanitized export | +| Single-node embedded storage cannot handle job concurrency/backup safely | Low-medium | Medium | `RG-004`; no multi-replica claim | +| Home Assistant coupling leaks into domain/application | Medium | High maintenance/portability cost | ADR 0003 dependency rule and architecture tests | +| Unknown target library sizes lead to unjustified performance design | High | Medium | `OQ-007`, measurable targets before optimization | +| Future Apple Music support is mistaken for full sync despite no documented remove/reorder operation | Medium | High if scope is promoted | Per-operation/object capability probes; Apple feasibility gates before scope change | + +## Recommended next five specification/design tasks + +1. **Provider feasibility report (`RG-001`).** Prove or narrow the Spotify ↔ YouTube promise using official APIs and dedicated accounts; feed the evidence into the provider, [import](../specs/library-import-and-provider-projections.md), and [copy](../specs/one-time-playlist-copy.md) SDDs. Report unofficial YT Music evidence separately and keep Apple as an explicit future/contingency spike. +2. **Close the [authorization SDD](../specs/provider-connections-and-authorization.md) blockers (`RG-002` + `OQ-004`).** Compare direct App OAuth with a minimal companion-integration broker, then settle callbacks, bring-your-own credentials, encryption, revocation, and backups. +3. **Close the [copy SDD](../specs/one-time-playlist-copy.md) policy blocker (`OQ-003`).** Decide target creation, strict/best-effort behavior, batching, partial failure, reconciliation, cancellation, and exact acceptance examples. +4. **Close the [identity SDD](../specs/recording-identity-resolution.md) evidence blockers (`RG-003`).** Build the corpus and settle normalization, candidate sources, evidence, versioned rules, manual decisions, and measurable safety targets. +5. **Close the [runtime](../specs/home-assistant-app-runtime-and-ingress.md) and [durable-operation](../specs/durable-operations-and-recovery.md) SDD blockers (`RG-004`).** Choose process topology and storage only after crash, lease, migration, backup, and representative-scale evidence. + +After those tasks, revisit playlist ownership (`OQ-002`) before creating any persistent-synchronization SDD. The Home Assistant native surface (`RG-005`) can proceed in parallel once the service API shape is stable, but it is not a prerequisite for the copy MVP. diff --git a/docs/product/product-specification.md b/docs/product/product-specification.md new file mode 100644 index 0000000..2ae6518 --- /dev/null +++ b/docs/product/product-specification.md @@ -0,0 +1,180 @@ +# Product specification + +**Status:** proposed baseline for owner review +**Last reviewed:** 2026-09-20 + +## Vision + +Symphonia is a self-hosted personal music hub through which a person can see and manage their music across providers. Symphonia owns the user's provider-independent view of music; Spotify, YouTube, Apple Music, Plex, Navidrome, and future services are replaceable representations and execution targets. + +The product initially optimizes for trustworthy library interoperability: import, explain, match, review, and copy. It is not primarily a playback surface. + +## Product principles + +1. **The user's model comes first.** Provider objects never become the canonical domain merely because they are convenient. +2. **Uncertainty is visible.** An ambiguous match is a user decision, not a hidden guess. +3. **Writes are previewable and explainable.** A user can see what will happen before a provider is mutated and what happened afterward. +4. **Provider differences remain explicit.** The UI and workflows degrade according to declared capabilities rather than pretending that all providers are equivalent. +5. **Self-hosted is a product constraint.** Operation, backup, upgrades, credentials, and recovery must be understandable to a homelab operator. +6. **Home Assistant is the primary host, not the domain boundary.** The primary package is a Home Assistant App with an Ingress UI, while the same core remains independently runnable and testable. + +## Users + +The MVP supports one local Symphonia user. That user operates the installation and owns the provider connections; in Home Assistant App mode the management surface is initially limited to authenticated administrators. The data model MUST NOT make a provider type synonymous with a single global account; future versions may allow several connections to the same provider. + +Multi-user authorization, sharing between Symphonia users, and hosted SaaS operation are outside the MVP. + +## Goals + +- Provide one inventory of connected provider playlists and library items with clear provenance. +- Recognize when provider-specific items likely represent the same recording. +- Let the user resolve ambiguity and preserve that decision. +- Copy a playlist between supported providers with a complete preview and result. +- Retain enough operation history to diagnose matching and provider failures. +- Establish extension points that make a third provider possible without changing the core domain. +- Run continuously on ordinary self-hosted infrastructure. + +## Explicit non-goals + +The MVP MUST NOT include: + +- audio playback or a competing player experience; +- recommendations, discovery feeds, or AI-generated playlists; +- audio download, ripping, transcoding, streaming, or format management; +- multi-room audio or device control; +- social features or public profiles; +- multi-user tenancy or role-based administration; +- automatic bidirectional playlist synchronization; +- Home Assistant coupling inside the domain/application core or standalone composition; +- an assumption that similar titles imply identical recordings. + +## Core user journeys + +### Connect a provider + +1. The user selects a provider. +2. Symphonia shows the provider's access basis, maturity/support classification, verified capabilities, requested permissions, and known limitations. +3. The user completes the provider's supported authorization flow. +4. Symphonia validates the grant and records a connection without exposing secrets. +5. The first import is scheduled and its progress is visible. + +### Explore the library + +1. The user sees connected providers and import freshness. +2. The user browses playlists grouped or filtered by provider connection. +3. Each item retains its original provider identity and shows its Symphonia recording link or resolution state. + +### Resolve an ambiguous track + +1. The user opens the unresolved-items queue. +2. Symphonia displays source metadata, candidates, confidence classification, and evidence. +3. The user selects a candidate, declares the item distinct, defers it, or marks it unresolvable. +4. Symphonia records the decision, actor, time, evidence context, and algorithm version. +5. Future operations reuse the decision unless the user revokes it or the provider identity becomes invalid. + +### Copy a playlist + +1. The user selects a source playlist and target connection. +2. Symphonia captures the source version and creates a dry-run plan. +3. The plan shows ordered matches, duplicates, ambiguous/unmatched/unavailable items, target capability constraints, and expected writes. +4. The user resolves blockers and chooses an explicit unresolved-item policy. +5. Symphonia executes the accepted plan safely. +6. The user sees the target playlist, item-level results, retries, omissions, and recovery actions. + +### Diagnose an operation + +1. The user opens operation history. +2. Symphonia shows status, timing, source and target, source version, plan version, checkpoints, provider calls summarized without secrets, and item-level outcomes. +3. The user can retry only the recoverable work or start a new plan when the source changed. + +## MVP requirements + +### Product and account + +- **SYM-PROD-001:** The application core and standalone service MUST remain usable without Home Assistant, while the Home Assistant App is the primary supported deployment. +- **SYM-PROD-002:** The application MUST expose uncertainty and provider limitations before a provider write. +- **SYM-PROD-003:** The application MUST distinguish imported provider facts from user-authored decisions and Symphonia-derived data. +- **SYM-ACC-001:** The MVP MUST support one local Symphonia user. +- **SYM-ACC-002:** A provider connection MUST identify both the provider type and provider account; provider type alone is not an account identity. +- **SYM-ACC-003:** Disconnecting a provider MUST stop future provider calls and trigger the applicable provider-data retention workflow without erasing user-authored resolution history unless policy requires it. +- **SYM-ACC-004:** The UI MUST show connection health, last successful import, required user action, and granted functional access. +- **SYM-ACC-005:** The MVP Home Assistant Ingress management surface MUST be restricted to authenticated Home Assistant administrators. +- **SYM-ACC-006:** Before authorization, the UI MUST show whether access uses an official or reverse-engineered contract, adapter maturity/support level, required external dependencies, credential type, and expected reauthorization behavior. + +### Unified library + +- **SYM-LIB-001:** Every imported object MUST retain provider type, provider connection, provider object identifier, source timestamps when available, import time, and raw-data freshness metadata. +- **SYM-LIB-002:** The system MUST represent a recording independently from its provider representations. +- **SYM-LIB-003:** The system MUST preserve playlist order and repeated entries during import. +- **SYM-LIB-004:** Provider deletion or disappearance MUST be represented as availability state; it MUST NOT silently delete the independent recording or audit history. +- **SYM-LIB-005:** The UI MUST allow playlists to be filtered by provider connection and MUST NOT imply that an imported provider playlist is Symphonia-owned. +- **SYM-LIB-006:** The system SHOULD import the connected account's saved or liked tracks when the provider exposes that collection through an approved API. + +### Identity resolution + +- **SYM-MATCH-001:** Each provider track representation MUST be in exactly one resolution state: `resolved`, `ambiguous`, `unmatched`, `deferred`, or `invalid`. +- **SYM-MATCH-002:** A resolution MUST record its origin (`automatic` or `manual`) and the evidence and resolver version used. +- **SYM-MATCH-003:** Automatic matching MUST NOT rely on normalized title alone. +- **SYM-MATCH-004:** Matching MUST preserve distinctions between live, studio, cover, remix, edit, acoustic, remastered, clean, explicit, and other known version indicators when evidence permits. +- **SYM-MATCH-005:** The user MUST be able to accept a candidate, reject a candidate pair, create a distinct recording, defer, or revoke a prior manual resolution. +- **SYM-MATCH-006:** Manual acceptance and rejection decisions MUST be reused in later imports, copy plans, and sync plans until revoked or invalidated with an explanation. +- **SYM-MATCH-007:** A confidence label MUST be accompanied by human-readable evidence; a numeric score alone is insufficient. +- **SYM-MATCH-008:** Thresholds and scoring algorithms MUST remain versioned implementation policy and are not defined by this baseline. + +### Playlist copy + +- **SYM-PL-001:** Copy MUST be modeled as a finite operation, not as a continuing relationship. +- **SYM-PL-002:** Every copy MUST produce a non-mutating plan tied to a captured source playlist version or snapshot. +- **SYM-PL-003:** A plan MUST report every source entry as `ready`, `ambiguous`, `unmatched`, `unsupported`, `unavailable`, or `invalid` before execution. +- **SYM-PL-004:** Execution MUST require an explicit policy for non-ready entries; the MVP default policy is an open question. +- **SYM-PL-005:** Copy MUST preserve relative order and duplicate occurrences among entries that are written, subject to declared target capabilities. +- **SYM-PL-006:** Copy execution MUST have a stable idempotency key and MUST reconcile target state before repeating an uncertain provider write. +- **SYM-PL-007:** A source change after planning MUST be visible. The system MUST either require re-planning or explicitly execute the captured plan; it MUST NOT silently mix versions. +- **SYM-PL-008:** A copy result MUST include the created or selected target, outcome for every source entry, provider errors, and whether safe retry is possible. +- **SYM-PL-009:** Destructive overwrite of an existing target playlist MUST NOT be an implicit copy behavior. + +### Operation history + +- **SYM-PROD-004:** Imports, match-resolution changes, copy plans, copy executions, and future sync runs MUST be attributable and timestamped. +- **SYM-PROD-005:** The MVP MUST expose operation state and item-level failures in the UI. +- **SYM-PROD-006:** User-facing errors MUST say whether the operation is retrying, needs user action, partially succeeded, or permanently failed. + +## MVP boundary + +The first useful release is complete when one local user can: + +- configure one Spotify connection and one validated Google/YouTube connection; +- authenticate with supported provider flows; +- import supported library collections and owned/followed playlists; +- browse provider playlists and their freshness; +- resolve track identities automatically where safe and manually where needed; +- preview and execute a copy in each direction only where the target adapter declares all required capabilities; +- inspect basic operation history; and +- deploy, upgrade, back up, restore, and rotate provider credentials using documented procedures. + +The wording “validated Google/YouTube connection” is deliberate. Symmetric **YouTube Music** library access is not yet proven through an official API; see [provider research](../providers/provider-research.md). If official feasibility fails, the owner must revise the MVP rather than silently adopting a reverse-engineered API. + +## Future scope + +- Persistent one-way, add-only, and bidirectional playlist synchronization. +- Symphonia-owned logical playlists if the ownership RFC selects that model. +- Apple Music, Plex/Plexamp, Navidrome, and other adapters. +- Multiple accounts per provider in the UI. +- Home Assistant entities, actions, and events over a stable Symphonia API. +- More complete album, artist, release, and musical-work modeling. +- Export/import of user-authored mappings and operation history. + +Future scope is not permission to build these items into the MVP. + +## Product success measures + +Targets require real-provider feasibility testing before numeric thresholds are accepted. The MVP MUST at least measure: + +- imported items and playlists per connection; +- resolution-state counts and the proportion automatically resolved; +- manual-resolution reuse; +- copy plan versus execution outcomes; +- provider request, throttle, retry, and failure counts; and +- time since last successful import and operation completion. + +No response-time, matching-accuracy, or scale target is accepted yet; the test corpus and expected homelab library sizes are open questions. diff --git a/docs/providers/home-assistant-ecosystem-review.md b/docs/providers/home-assistant-ecosystem-review.md new file mode 100644 index 0000000..4446474 --- /dev/null +++ b/docs/providers/home-assistant-ecosystem-review.md @@ -0,0 +1,169 @@ +# Home Assistant music ecosystem review + +**Status:** research snapshot and design input, not a dependency or scope decision +**Reviewed:** 2026-09-20 +**Source policy:** project documentation and current source were inspected in addition to official provider documentation; revalidate before implementation + +## Purpose + +This review asks what Symphonia can learn from existing Home Assistant music projects without assuming that their goals, guarantees, or risk tolerance match ours. It complements the official-API-only [provider research](provider-research.md). + +The strongest precedent is Music Assistant: a separate service/App owns the music domain and a comparatively thin Home Assistant integration exposes native surfaces. Existing YouTube Music projects reduce uncertainty about technical access, but they also provide direct evidence that authentication and API stability are the central risks rather than solved details. + +## Executive conclusions + +1. **Keep the App/service as the system of record and the Home Assistant integration thin.** The [Music Assistant integration](https://www.home-assistant.io/integrations/music_assistant/) connects Home Assistant to a separately running server and exposes selected entities/actions. This validates Symphonia's accepted deployment direction without requiring Symphonia to become a playback system. +2. **Adopt a manifest plus effective-capability model.** Music Assistant provider implementations declare features and support multiple provider instances. Symphonia needs the same extensibility, but with more granular, runtime capability descriptors because copy and sync require stronger guarantees than playback and browsing. +3. **Reuse Home Assistant's OAuth conventions where possible, not its provider domain logic.** The official [Spotify integration](https://www.home-assistant.io/integrations/spotify) demonstrates bring-your-own application credentials, multiple accounts, and Home Assistant's external OAuth callback. A companion integration could potentially broker authorization for the App, but this is a spike candidate rather than an accepted design. +4. **Treat unofficial YouTube Music access as a distinct product mode.** Music Assistant and `ytube_music_player` demonstrate useful access through `ytmusicapi`, browser cookies, internal endpoints, and proof-of-origin tokens. They do not turn that surface into a supported Google API. An unofficial adapter would need explicit opt-in, health warnings, separate release gating, and no promise of symmetric copy/sync. +5. **Apple Music is a credible future official-library adapter.** Apple's official API documents library reads, catalog/library search, ISRC, playlist creation, and adding tracks. It does not document playlist-track removal, so new-playlist copy is more plausible than mirror sync. Its user-token acquisition and Home Assistant callback story still require a spike. +6. **Do not inherit playback-first shortcuts.** Symphonia must preserve unavailable entries, expose ambiguous matches, prove pagination completeness, and retain auditable user decisions even where an existing playback product can skip, merge, cap, or rescan data. + +## Projects reviewed + +| Project | What it establishes | Useful pattern for Symphonia | Boundary or warning | +| --- | --- | --- | --- | +| [Home Assistant Spotify](https://www.home-assistant.io/integrations/spotify) | A maintained Home Assistant integration can use application credentials, the HA external OAuth callback, and multiple account entries | Native config flow, reauthentication, callback and credential UX | It is a playback/media-browser integration, not a cross-provider library system | +| [Home Assistant Music Assistant integration](https://www.home-assistant.io/integrations/music_assistant/) | Home Assistant can discover and connect to a separate music server running as an App or container | Service owns domain; integration exposes bounded native actions/entities over an API | Installing an App and installing an integration remain separate lifecycle steps | +| [Music Assistant server](https://github.com/music-assistant/server) | Provider plugins, feature declarations, multiple instances, a normalized internal library, provider mappings, scheduled sync, and versioned SQLite migrations work at real scale | Provider manifest, connection instance, normalized mapping graph, scheduled imports | Playback requirements and automatic merging are not Symphonia requirements | +| [Music Assistant Spotify provider](https://www.music-assistant.io/music-providers/spotify/) | Spotify library/search support and multiple accounts are operationally feasible | Capability probing, account-specific source selection, OAuth lifecycle | Playback engines and their policy/terms trade-offs are out of scope | +| [Music Assistant YouTube Music provider](https://www.music-assistant.io/music-providers/youtube-music/) | Reading a YT Music library/search surface is technically feasible through private web behavior | Isolate the adapter, identify provider-instance-scoped IDs, expose reauthentication health | The project explicitly says there is no official API; cookies expire and a PO-token sidecar is required | +| [`ytube_music_player`](https://github.com/KoljaWindeler/ytube_music_player) | A Home Assistant custom integration can browse/play YT Music via `ytmusicapi` | Additional implementation evidence and failure cases | Its current [browser-auth guide](https://github.com/KoljaWindeler/ytube_music_player/blob/main/QUICK_START_BROWSER_AUTH.md) says OAuth is broken and asks users to export authenticated browser headers | +| [Music Assistant Apple Music provider](https://www.music-assistant.io/music-providers/apple-music/) | Apple libraries and catalog can be represented behind a provider abstraction | Separate catalog/library identifiers and object-level editability | Its playback/auth workarounds are not evidence that every flow is officially supported for Symphonia | +| [`apple-music-custom`](https://github.com/Hackashaq666/apple-music-custom) | A community integration can pair a local companion server with a Home Assistant media-player integration | Another example of a server/integration boundary | It controls the Music app on a macOS host; it is not a cloud-library interoperability adapter | + +No source code has been selected for reuse. Any future reuse proposal must review the exact dependency version, license, security posture, transitive dependencies, and whether importing that implementation would couple Symphonia to playback behavior. + +The review identified official Home Assistant integrations for Spotify and Music Assistant, but did not identify official direct integrations for YouTube Music or Apple Music. The direct examples above are Music Assistant providers or community/HACS integrations, not official Home Assistant Core integrations. Their existence is implementation evidence, not a platform support guarantee. + +## Architecture lessons from Music Assistant + +### Server plus integration is the right split + +The official Home Assistant integration requires a Music Assistant server and can connect to a server hosted as an App or separate container. That closely matches Symphonia's intended shape: + +```text +Home Assistant integration → versioned local API → Symphonia App/service +native actions/entities domain, jobs, adapters, data +``` + +The integration should remain replaceable and unavailable independently. It must not read the App database or become the only way to run provider imports and copies. + +### Provider manifests and instances are worth adopting + +Music Assistant providers have manifests, declared features, configuration, and an instance identity. Its current [YouTube Music manifest](https://github.com/music-assistant/server/blob/dev/music_assistant/providers/ytmusic/manifest.json), for example, labels the provider `beta`, declares `multi_instance`, and pins `ytmusicapi`. + +Symphonia should adopt the concepts, but strengthen them: + +- a manifest describes static adapter identity, provenance, official/unofficial status, version, configuration schema, and declared upper-bound capabilities; +- a provider connection describes the actual account and its scopes, market, health, and probed capabilities; +- an individual playlist or item can further restrict an operation, such as Apple's `canEdit=false`; and +- workflows are enabled only from the intersection of adapter, connection, object, and current-health capabilities. + +A provider object identity may need `(adapter kind, provider instance/connection namespace, object type, provider object ID)`. Music Assistant's current YouTube Music source explicitly notes that some personal playlist IDs are not unique across instances. Symphonia must never assume that an upstream ID is globally unique merely because it looks stable. + +### Provider mappings validate the representation graph + +Music Assistant's [Music Controller](https://github.com/music-assistant/server/tree/dev/music_assistant/controllers/music) aggregates providers into an internal SQLite library and uses provider mappings to relate internal items to provider items. This supports Symphonia's decision to keep provider representations separate from provider-independent recordings. + +The semantic difference matters: Music Assistant may automatically merge items to make playback convenient. Symphonia's mappings influence external writes and therefore require stored evidence, confidence, negative decisions, resolver versions, and manual review. We can borrow the shape, not silent match policy. + +### Migration recovery must reflect irrecoverable local decisions + +Music Assistant's controller documentation records a real divergence between stable and development schema-version histories. This is useful evidence for a per-migration ledger and cross-channel upgrade tests. + +Its ability to fall back to a fresh library database after a failed migration is not generally safe for Symphonia. Provider data may be re-importable, but manual identity decisions, accepted plans, job checkpoints, and audit history are not. Restore or migration failure must stop writes and lead to explicit recovery, not silently discard locally owned state. + +## Provider-specific findings + +### Spotify + +The official Home Assistant Spotify integration reduces OAuth uncertainty. It instructs self-hosters to create a Spotify application with `https://my.home-assistant.io/redirect/oauth`, or `/auth/external/callback` when My Home Assistant is disabled. Home Assistant's [Application Credentials platform](https://developers.home-assistant.io/docs/core/platform/application_credentials/) supplies local client credentials, OAuth/PKCE helpers, token refresh, and reauthentication patterns. + +This yields two candidates for the OAuth spike: + +1. **Direct App flow:** Symphonia owns the callback, token exchange, refresh, and storage. This keeps the provider adapter self-contained but must solve public callback reachability and must not expose an unauthenticated management port. +2. **Companion-integration authorization broker:** a minimal custom integration uses Home Assistant's config flow/application credentials and transfers an opaque, one-use connection grant to the App over an authenticated local contract. This reuses HA callback UX but creates a token-ownership, lifecycle, backup, and versioning boundary that must be threat-modeled. + +The companion integration cannot simply be assumed to make OAuth free: an App does not automatically inherit Home Assistant Core's integration helpers. + +### YouTube Music + +The current Music Assistant provider is unusually valuable as negative as well as positive evidence: + +- its [documentation](https://www.music-assistant.io/music-providers/youtube-music/) explicitly labels the implementation best-effort because YouTube offers no official API for this data/stream surface; +- setup uses a browser cookie and a separate proof-of-origin token generator; +- its [manifest](https://github.com/music-assistant/server/blob/dev/music_assistant/providers/ytmusic/manifest.json) labels the provider beta and depends on `ytmusicapi`; +- its [source](https://github.com/music-assistant/server/blob/dev/music_assistant/providers/ytmusic/__init__.py) namespaces personal playlist IDs by instance, caps some dynamic playlists, lacks paging for playlist tracks, skips unavailable tracks, and currently advertises read/search features rather than playlist-write features; and +- `ytube_music_player` independently documents cookie expiry and a broken OAuth path. + +This changes the uncertainty from “is library access technically possible?” to “can Symphonia responsibly support a moving, unofficial authentication and data contract?” The answer may be yes for an explicitly experimental adapter, but not as an invisible fallback for the official YouTube Data API. + +Minimum conditions for such an adapter would be: + +- an `unofficial_reverse_engineered` access basis, honest maturity/product-support labels, and explicit user acknowledgement; +- no request for a user's primary Google-account cookie without a clear threat model and deletion path; +- dependency and upstream-contract health surfaced in the UI; +- preserved unavailable/unknown entries and explicit incompleteness when paging or caps are uncertain; +- writes disabled until independently proven by contract tests and dedicated accounts; and +- ability to disable/release the adapter independently from official providers. + +### Apple Music + +Apple's official [Apple Music API](https://developer.apple.com/documentation/applemusicapi/) can read personal library resources and the catalog. The API documents [all library songs](https://developer.apple.com/documentation/applemusicapi/get-all-library-songs), paginated responses, catalog/library search, song ISRC, [new library playlist creation](https://developer.apple.com/documentation/applemusicapi/create-a-new-library-playlist), and [adding tracks](https://developer.apple.com/documentation/applemusicapi/add-tracks-to-a-library-playlist). + +Important limitations for Symphonia: + +- personalized calls require both a developer token and a Music User Token; [MusicKit user authentication](https://developer.apple.com/documentation/applemusicapi/user-authentication-for-musickit) must be validated in a self-hosted web/App context; +- library song IDs and catalog song IDs are distinct; both must be retained when present; +- playlist editability is object-specific (`canEdit`), so connection-wide “playlist write” is insufficient; +- Apple documents create and append operations, but the reviewed API surface did not identify an operation to remove playlist entries; and +- Music Assistant reports manual reauthentication and direct-port/cookie fallbacks, which are implementation evidence but not an official contract Symphonia should inherit. + +Therefore Apple Music looks promising for future import and one-time copy-to-new-playlist, but not for strict mirror or bidirectional sync until removal/reorder and authentication are proven. + +## Security lessons + +A 2026 [Music Assistant security advisory](https://github.com/music-assistant/server/security/advisories/GHSA-7jcc-p6xr-835j) described an unauthenticated direct service port combined with user-controlled filesystem paths and root execution. Symphonia does not need a filesystem music provider, but the boundary lessons apply: + +- Ingress authentication does not protect a separately exposed App port. +- A callback listener must expose only the minimum callback surface or have its own authentication; it must never make the management API anonymously reachable. +- Provider IDs, playlist names, URIs, imported metadata, and callback parameters are data, never filesystem paths, import names, executable schemes, or unrestricted outbound URLs. +- The container should run as a non-root user where the App platform permits and have no host/config media mounts without a specific requirement. +- URI schemes, callback destinations, redirect targets, and adapter-controlled outbound hosts need allowlists and canonical validation. + +## Adopt, adapt, and avoid + +### Adopt + +- App/service plus thin Home Assistant integration. +- Provider manifests, multiple connection instances, and declared features. +- Internal provider-representation mappings and versioned migrations. +- Native Home Assistant application-credential/config-flow patterns where a companion integration genuinely owns that boundary. + +### Adapt + +- Replace coarse provider features with operation- and object-level effective capabilities. +- Replace playback-friendly automatic merging with evidence-backed, reversible identity links. +- Replace “rescan after failure” with recovery that protects user-authored state. +- Treat provider quality labels as a first-class support tier visible in planning and diagnostics. + +### Avoid + +- Treating unofficial access as equivalent to an official API. +- Exporting full browser cookies as a normal setup path without an explicit experimental security model. +- Skipping unavailable tracks or returning capped lists as if complete. +- Exposing a direct unauthenticated port merely to make OAuth convenient. +- Letting provider values influence local file paths, arbitrary URI schemes, redirects, or code/module loading. + +## Design consequences for the next RFCs + +This review does not accept a dependency or new provider into the MVP. It narrows the next evidence work: + +1. Extend the provider manifest RFC with independent access-basis, maturity, and product-support classifications so, for example, unofficial-but-mature and official-but-experimental are not conflated. +2. Make capabilities an intersection of adapter, connection, object, and live health—not provider-wide booleans. +3. Include provider instance/connection namespace and object type in external-identity analysis. +4. Compare direct App OAuth with a minimal companion-integration authorization broker in the Home Assistant OAuth spike. +5. Add an Apple Music test-account spike as a future-provider candidate, focusing on Music User Token acquisition, catalog/library IDs, `canEdit`, playlist append, and absence of remove/reorder. +6. Require completeness markers for every import/list operation and preserve unavailable entries. +7. Threat-model every directly exposed App listener and prevent provider-controlled values from acquiring filesystem or executable semantics. diff --git a/docs/providers/provider-research.md b/docs/providers/provider-research.md new file mode 100644 index 0000000..2ea1bfc --- /dev/null +++ b/docs/providers/provider-research.md @@ -0,0 +1,155 @@ +# Provider and platform research + +**Status:** research snapshot, not an architectural decision +**Reviewed:** 2026-09-20 +**Source policy:** official documentation only; revalidate before implementation and every release + +## How to read this document + +This snapshot separates what Symphonia needs from what an official API documents. It does not prove behavior for a particular account, market, application mode, or Home Assistant network setup. A live feasibility spike is required before either provider adapter is committed to the MVP. + +Existing Home Assistant and community implementations are reviewed separately in [Home Assistant music ecosystem review](home-assistant-ecosystem-review.md). They provide valuable implementation evidence but do not replace an official provider contract. + +Legend: + +- **Documented**: an official current page describes the needed primitive. +- **Partial**: a related primitive exists but does not establish Symphonia's full semantics. +- **Not documented**: no official support was identified in the reviewed sources; this is not proof that no private API exists. +- **Unknown**: official documentation is insufficient; test without relying on undocumented behavior. + +## Capability needs versus official support + +| Need | Spotify Web API | YouTube Data API v3 | MVP consequence | +| --- | --- | --- | --- | +| User OAuth and unattended refresh | Documented: authorization-code flows; current docs state 1-hour access tokens and 6-month refresh-token lifetime | Documented: OAuth 2.0 web-server flow with offline access/refresh token | Reauthorization states are normal operations, not exceptional crashes | +| Read account playlists | Documented: current user's owned/followed playlists; scopes affect private/collaborative results | Documented: playlists for authenticated user with `mine=true` | Both can list some account playlists; collection semantics differ | +| Read ordered playlist entries | Documented | Documented for YouTube video `playlistItem` resources | YouTube entries are videos, not documented music-catalog recordings | +| Create playlist | Documented | Documented | Both have a creation primitive | +| Add entries | Documented, up to 100 URIs per documented request | Documented, one resource insertion operation; current cost 50 quota units | Batch/checkpoint and cost models differ materially | +| Remove/reorder/replace | Documented Spotify playlist operations, subject to current endpoint availability/mode | Insert/update/delete playlist-item methods exist; exact duplicate/reorder behavior needs spike | Do not advertise sync from surface similarity alone | +| Saved/liked track library | Documented saved-track endpoints and newer generic library operations | Partial: YouTube exposes a special liked-videos playlist; equivalence to YouTube Music liked songs/library is not documented | “Library” must stay provider-specific until semantics are verified | +| Music catalog search | Documented track search, including `isrc` query filter | Partial: generic YouTube video/channel/playlist search, not a documented YouTube Music catalog search | Cross-provider YouTube matching is the largest feasibility risk | +| ISRC | Documented in Spotify track `external_ids` and search filter | Not documented on YouTube `video`/`playlistItem` resources | YouTube candidates need weaker evidence or another approved source | +| Album/artist/duration metadata | Documented music entities and track duration | Partial: video title/channel/duration; not equivalent to recording/release metadata | Normalization must show evidence quality | +| Revision/change token | Spotify playlist `snapshot_id` documented | ETags exist generally; playlist-change attribution semantics need spike | Polling and stored baselines are still required | +| Playlist webhooks | Not identified in reviewed official Web API docs | No playlist-change push contract identified; push notifications cover channel-resource activity, not a general playlist sync feed | Assume polling until proven otherwise | +| Rate/quota model | Rolling 30-second application limit; 429 and `Retry-After`; exact limit varies by mode | Default quotas documented, including a search-query bucket and general daily units; operations have individual cost | Planner/scheduler must be provider-aware | +| Full YouTube Music library model | N/A | **Not documented.** Reviewed public API is YouTube Data API, centered on videos/channels/playlists | Do not label the official adapter “full YouTube Music” without evidence | + +## Spotify Web API + +### Verified useful primitives + +- [Get Current User's Playlists](https://developer.spotify.com/documentation/web-api/reference/get-a-list-of-current-users-playlists) returns owned or followed playlists and exposes `snapshot_id`; private/collaborative visibility depends on scopes. +- [Create Playlist](https://developer.spotify.com/documentation/web-api/reference/create-playlist) and [Add Items to Playlist](https://developer.spotify.com/documentation/web-api/reference/add-items-to-playlist) support the core target-write workflow. The add endpoint documents a maximum of 100 items per request. +- [Search for Item](https://developer.spotify.com/documentation/web-api/reference/search) supports track searches and an `isrc` field filter. Track results include `external_ids.isrc` where known. +- [Get Track](https://developer.spotify.com/documentation/web-api/reference/get-track) documents duration, artists, album context, and known external IDs including ISRC. +- Spotify's [playlist concepts](https://developer.spotify.com/documentation/web-api/concepts/playlists) document how scopes affect owned/followed, private, and collaborative playlists. + +### Authorization and access constraints + +- Spotify recommends authorization code for a long-running confidential web service and PKCE when a client secret cannot be stored; see [Authorization](https://developer.spotify.com/documentation/web-api/concepts/authorization). +- Current [refresh-token documentation](https://developer.spotify.com/documentation/web-api/tutorials/refreshing-tokens) states that Developer Dashboard refresh tokens last six months and refreshing access does not extend that lifetime. Symphonia must expect scheduled user reauthorization unless policy changes. +- Spotify's February 2026 [developer-access update](https://developer.spotify.com/blog/2026-02-06-update-on-developer-access-and-platform-security) states that Development Mode requires the app owner to have Premium, limits a developer to one Client ID and an app to five authorized users, and limits new Development Mode apps to a smaller endpoint set. The March 9 update postponed endpoint-access changes for existing integrations but not Premium/user/client limits. +- The [February 2026 migration guide](https://developer.spotify.com/documentation/web-api/tutorials/february-2026-migration-guide) and [changelog](https://developer.spotify.com/documentation/web-api/references/changes/february-2026) must be checked against every endpoint selected for the adapter. Development Mode is plausible for a personal install but is not a stable basis for a hosted multi-user product. +- Spotify's [rate-limit documentation](https://developer.spotify.com/documentation/web-api/concepts/rate-limits) describes an application-wide rolling 30-second window, mode-dependent limits, endpoint exceptions, and 429 responses. Exact numeric limits are not generally published; the adapter must learn from responses. + +### Spotify risks to validate + +1. Every required read/write endpoint is available to a newly created September 2026 Development Mode application. +2. A self-hoster can register the callback URL that the Home Assistant App flow requires. +3. Saved-library and playlist content can be stored/used as Symphonia proposes under current developer terms. +4. Snapshot and response behavior is sufficient to reconcile timeouts and repeated entries. +5. Six-month refresh-token expiry gives adequate warning and recovery UX. + +## YouTube and YouTube Music + +### What the official API documents + +- The [YouTube Data API reference](https://developers.google.com/youtube/v3/docs) manages YouTube resources such as videos, channels, playlists, and playlist items. +- Official [playlist guidance](https://developers.google.com/youtube/v3/guides/implementation/playlists) documents `playlists.list` with `mine=true` for the authenticated user's playlists. +- [Playlists](https://developers.google.com/youtube/v3/docs/playlists) can be listed, inserted, updated, and deleted. +- [Playlist items](https://developers.google.com/youtube/v3/docs/playlistItems) can be listed, inserted, updated, and deleted. A playlist item points to a resource such as a YouTube video. +- [PlaylistItems: list](https://developers.google.com/youtube/v3/docs/playlistItems/list) currently costs one general quota unit per call; [PlaylistItems: insert](https://developers.google.com/youtube/v3/docs/playlistItems/insert) currently costs 50 general quota units. +- [Search: list](https://developers.google.com/youtube/v3/docs/search/list) is generic YouTube search. Current documentation describes a default search-query allocation of 100 calls/day in a dedicated bucket; [quota guidance](https://developers.google.com/youtube/v3/getting-started#quota) describes a default 10,000-unit daily allocation for other endpoints, subject to change and extension review. +- Google's [web-server OAuth guide](https://developers.google.com/youtube/v3/guides/auth/server-side-web-apps) documents offline access and refresh tokens for unattended calls. + +### What is not established + +The reviewed official developer documentation did not identify a public **YouTube Music API** that exposes the complete YouTube Music library, song catalog identity, liked songs, albums, or ISRC metadata. The YouTube Data API can create and modify playlists of videos, but official documentation does not promise that: + +- every YouTube Music playlist appears with identical behavior through `playlists.list`; +- a YouTube video corresponds to exactly one musical recording; +- YouTube Music “songs,” uploads, liked songs, and ordinary liked videos share one API model; +- music-specific catalog IDs, album editions, artist credits, explicitness, or ISRC are exposed; or +- writes through the Data API reproduce all YouTube Music UI semantics. + +This absence is an inference from the official surface reviewed on the date above, not a claim about Google's private APIs. Libraries such as `ytmusicapi` use unofficial/reverse-engineered endpoints and therefore have materially different stability, authentication, policy, and maintenance risk. Adopting one requires an explicit ADR and user opt-in; it is not an implicit fallback. + +### Policy and quota constraints + +- [YouTube API Services Developer Policies](https://developers.google.com/youtube/terms/developer-policies) require most stored Authorized API Data not otherwise exempted to be deleted or refreshed within 30 calendar days, and require cleanup after revocation/loss of authorization. Symphonia needs per-field provenance and refresh-by scheduling rather than indefinite raw-cache retention. +- Sensitive-scope production apps can require Google verification; [verification guidance](https://support.google.com/cloud/answer/13464321) and [app audience/user-cap guidance](https://support.google.com/cloud/answer/15549945) must be evaluated for a distributed self-hosted app whose users may bring their own OAuth project. +- Invalid requests also consume quota. Candidate search must be bounded, cached within policy, and planned against daily budgets. + +### YouTube feasibility gates + +Before “YouTube Music provider” becomes an accepted MVP promise, a spike using a dedicated test account MUST answer: + +1. Which playlists created, followed, or edited in YouTube Music are visible through the official Data API? +2. Can they be copied in both directions with order and duplicates intact? +3. How are liked songs, uploads, unavailable videos, music videos, and topic-channel tracks represented? +4. What metadata can distinguish a label recording from a cover, live video, lyric video, edit, or user upload? +5. Are write results reflected in the YouTube Music client as users expect? +6. What OAuth scopes, verification mode, callback URLs, quota, and 30-day refresh behavior apply to a self-hosted install? + +If official support is insufficient, owner decisions are required among narrowing the MVP to ordinary YouTube playlists, deferring Google/YouTube, or accepting a separately labeled unofficial adapter. + +## Apple Music future-provider observations + +Apple Music is outside the current MVP, but the ecosystem review identified a materially stronger official library surface than YouTube Music. It is therefore a useful future-provider or contingency candidate, not an accepted scope change. + +### Documented useful primitives + +- The official [Apple Music API overview](https://developer.apple.com/documentation/applemusicapi/) covers the Apple Music catalog and a user's personal iCloud Music Library, including playlist reads and authorized playlist modification. +- [Get All Library Songs](https://developer.apple.com/documentation/applemusicapi/get-all-library-songs) exposes paginated personal-library songs and may include a distinct `catalogId` alongside the library resource ID. +- Catalog [song attributes](https://developer.apple.com/documentation/applemusicapi/songs/attributes-data.dictionary) include ISRC, duration, artist, album, release date, explicitness, and other useful matching evidence. The Songs API also documents lookup of multiple catalog songs by ISRC. +- The API documents both catalog and [library search](https://developer.apple.com/documentation/applemusicapi/search-for-library-resources). +- [Create a New Library Playlist](https://developer.apple.com/documentation/applemusicapi/create-a-new-library-playlist) and [Add Tracks to a Library Playlist](https://developer.apple.com/documentation/applemusicapi/add-tracks-to-a-library-playlist) support a one-time copy-to-new-playlist workflow. +- Library playlists expose object-specific editability such as `canEdit`; provider support cannot be inferred from the account alone. + +### Authorization and write limitations + +- Personalized requests require a developer token and a Music User Token; see [User Authentication for MusicKit](https://developer.apple.com/documentation/applemusicapi/user-authentication-for-musickit). MusicKit manages user tokens on Apple platforms and the web, but a self-hosted Home Assistant App still needs a dedicated browser/origin/token-lifecycle spike. +- A MusicKit developer registration requires a media identifier and private key. Secret generation, storage, rotation, and the user experience for a distributed self-hosted project are unresolved. +- The reviewed official playlist surface documents creation and appending tracks. It did not identify a documented endpoint to remove or reorder playlist entries. Absence from the reviewed documentation is not proof of impossibility, but strict mirror/bidirectional sync MUST treat those capabilities as `unknown` or `unsupported` until proven. +- Library and catalog IDs are not interchangeable. An adapter needs to retain both and resolve the correct ID type for reads and writes. + +### Apple feasibility gates + +Before proposing Apple Music scope, a dedicated account spike MUST answer: + +1. Can MusicKit on the Web acquire and renew a Music User Token reliably through Home Assistant Ingress without an unauthenticated direct management port? +2. Which user/developer secrets are device-, origin-, App-, or installation-specific, and what can safely survive backup/restore? +3. Are every-page library and playlist imports complete for purchased, uploaded, matched, subscription, and unavailable items? +4. Which resource ID type is required when creating a playlist or appending a catalog/library song? +5. How do `canEdit`, collaborative/shared playlists, storefront, and subscription state alter effective capabilities? +6. Is there any current official removal/reorder primitive, and if not, which copy/sync workflows remain honest? + +## Home Assistant platform + +Home Assistant is the primary deployment platform, not a music provider. + +- [Home Assistant Apps](https://developers.home-assistant.io/docs/apps/) are Supervisor-managed container applications distributed through App repositories. +- [App configuration](https://developers.home-assistant.io/docs/apps/configuration/) documents `/data` persistent storage, startup types, architecture metadata, Ingress, App options, and backup modes. +- [App presentation/Ingress](https://developers.home-assistant.io/docs/apps/presentation/#ingress) documents the authenticated proxied UI boundary and Ingress base-path considerations. +- [App security](https://developers.home-assistant.io/docs/apps/security/) recommends least privilege, avoiding host networking, AppArmor, minimal folder/API access, and careful authentication handling. +- [Application Credentials](https://developers.home-assistant.io/docs/core/platform/application_credentials/) provides OAuth2/config-flow helpers for Home Assistant integrations, including local bring-your-own client credentials and PKCE support. +- The official [Spotify integration](https://www.home-assistant.io/integrations/spotify) uses `https://my.home-assistant.io/redirect/oauth`, or `/auth/external/callback` when My Home Assistant is disabled, and supports multiple account entries. +- A future companion integration can use a config flow and native integration contracts described in the [integration manifest documentation](https://developers.home-assistant.io/docs/creating_integration_manifest/); service/action descriptions and entity platforms remain a separate artifact from the App. + +The provider OAuth callback is not solved merely by enabling Ingress. Application Credentials belongs to Home Assistant integrations, not arbitrary Supervisor Apps. Reusing it would require a companion integration or broker with an explicit token-ownership contract. Exact provider redirect URLs, externally reachable Home Assistant URLs, sessions, user-supplied client registrations, and any direct callback listener require a dedicated spike. + +## Current conclusion + +Spotify is a plausible personal/self-hosted MVP adapter, with meaningful 2026 access and reauthorization constraints. The official YouTube Data API is a plausible **YouTube playlist** adapter, but it is not sufficient evidence for the proposed full **YouTube Music** provider. That distinction is the highest-risk assumption in the current product scope. Apple Music has a promising official library/create/append surface for future scope, but its Home Assistant-compatible authorization flow and missing documented playlist removal/reorder remain open. diff --git a/docs/providers/provider-specification.md b/docs/providers/provider-specification.md new file mode 100644 index 0000000..6dc6fb2 --- /dev/null +++ b/docs/providers/provider-specification.md @@ -0,0 +1,235 @@ +# Provider specification + +**Status:** proposed contract; specific support is a dated research fact +**Last reviewed:** 2026-09-20 + +## Purpose + +Provider adapters translate external authentication, catalog, library, and playlist behavior into Symphonia concepts. The core asks for semantic capabilities and operations; it never branches on `spotify` or `youtube` to decide domain policy. + +This document states what Symphonia needs. [Provider research](provider-research.md) separately records what current official APIs appear to support, while the [Home Assistant music ecosystem review](home-assistant-ecosystem-review.md) records reusable implementation patterns and cautions from existing projects. + +## Provider, connection, and adapter + +- A **Provider** is a named adapter kind and version with a manifest describing its provenance, maturity, configuration, dependencies, and declared capability ceiling. +- A **Provider connection** is one authorized external account with granted scopes, market/account constraints, health, and effective capabilities. +- An **Adapter** implements provider ports for a provider kind. It converts provider DTOs/errors into normalized types and keeps provider-specific payloads outside the core. + +Capabilities belong to a connection at a point in time. They may depend on account tier, scopes, market, API mode, app registration, quota, policy approval, object ownership, or playlist type. + +An effective workflow capability is the intersection of: + +1. the adapter's statically declared capability ceiling; +2. the connection's current account, scopes, configuration, and health; +3. the target object's ownership/type/editability constraints; and +4. live provider availability or policy state. + +### Adapter provenance and support classification + +The manifest keeps independent axes instead of collapsing risk into one “supported” boolean: + +- **access basis:** `official_public_api`, `official_sdk_or_contract`, or `unofficial_reverse_engineered`; +- **maturity:** `experimental`, `beta`, or `stable`; +- **product support:** `disabled`, `best_effort`, or `supported`; +- upstream project/library names and pinned version constraints; +- last API/terms/security review date; and +- whether multiple configured instances are supported. + +The UI may render a concise combined label, but it MUST retain these underlying facts. A mature adapter can still rely on an unofficial access basis, while an official API adapter can remain experimental. + +## Capability descriptor + +A boolean is too weak. Each capability descriptor contains: + +- stable capability ID and schema version; +- support state: `supported`, `unsupported`, `degraded`, `unknown`, or `temporarily_unavailable`; +- evidence source and last-probed time; +- required scopes/authorization and missing grants; +- ownership/media/visibility restrictions; +- batch size, pagination, ordering, duplicate, and atomicity behavior; +- quota cost/rate-limit dimensions when known; +- revision/concurrency token behavior; +- retention or policy constraints; and +- user-facing reason and remediation. + +`unknown` MUST NOT be treated as supported. `degraded` means the operation is possible but cannot meet at least one normal semantic guarantee; the descriptor explains which. + +## Capability catalog + +The initial catalog is intentionally granular: + +### Account and authorization + +- `account.authorize.offline` +- `account.profile.read` +- `account.grant.refresh` +- `account.grant.revoke` + +### Library + +- `library.saved_tracks.read` +- `library.saved_tracks.add` +- `library.saved_tracks.remove` +- `library.albums.read` +- `library.changes.incremental` + +### Playlists + +- `playlist.list.owned` +- `playlist.list.followed` +- `playlist.read.metadata` +- `playlist.read.entries` +- `playlist.revision.read` +- `playlist.create` +- `playlist.update.metadata` +- `playlist.delete` +- `playlist.entries.add` +- `playlist.entries.remove` +- `playlist.entries.reorder` +- `playlist.entries.replace` +- `playlist.duplicates.preserve` +- `playlist.changes.push` + +### Catalog and metadata + +- `catalog.track.get` +- `catalog.track.search` +- `catalog.track.search_by_isrc` +- `metadata.isrc.read` +- `metadata.duration.read` +- `metadata.release.read` +- `metadata.artist_credits.read` +- `metadata.version_markers.read` + +Adapters MAY add namespaced experimental capabilities, but product workflows use only cataloged stable capabilities until the catalog is revised. + +## Capability requirements by workflow + +| Workflow | Required capabilities | Optional enrichment | +| --- | --- | --- | +| Import playlists | list owned or followed, metadata read, entries read | revision read, incremental changes | +| Import saved library | saved tracks read | albums read, incremental changes | +| Match source item | track get plus useful metadata | ISRC, release, search | +| Copy to new playlist | target search/get, playlist create, entries add | metadata update, duplicates preserve, revision read | +| Strict mirror sync | read/revision on both, target add/remove/reorder or replace | push changes, conditional writes | +| Add-only sync | read/revision on source, target search/get/add | push changes | + +A workflow MUST fail during planning with a typed capability explanation if a required capability is absent. It must not discover this after creating a partial target where a preflight probe was possible. + +## Normalized provider ports + +Exact language signatures are deferred, but an adapter must cover these behaviors: + +### Connection lifecycle + +- describe configuration and authorization requirements; +- begin authorization and validate a one-time callback; +- identify the external account without using mutable display names; +- refresh or reauthorize grants; +- probe effective capabilities; and +- revoke/disconnect and describe cleanup obligations. + +### Read/import + +- page through saved/library items and playlists with bounded page sizes; +- fetch playlist metadata, revision evidence, and ordered entries; +- fetch provider track metadata in batches where supported; +- represent deleted, unavailable, private, local-file, episode/non-music, and unknown media explicitly; and +- produce provider cursors/ETags only as opaque adapter-owned values. + +### Search/mapping + +- search for bounded track candidates using supported provider fields; +- fetch enough metadata to explain a candidate; +- return stable external IDs and availability for the authorized account/market; and +- report when search is generic video/content search rather than a music catalog search. + +### Write + +- create a playlist with explicit visibility and description semantics; +- add ordered entries using declared batch sizes; +- where supported, remove, reorder, replace, rename, or delete; +- return revision identifiers and per-item/batch outcomes; and +- provide a reconciliation read for unknown write outcomes. + +## Adapter requirements + +- **SYM-PROV-001:** Domain and application code MUST depend on provider ports and normalized types, never provider SDK types. +- **SYM-PROV-002:** An adapter MUST report effective connection capabilities before a workflow plan is accepted. +- **SYM-PROV-003:** Capability detection MUST combine static documented support, granted scopes, account/object restrictions, configuration, and safe runtime probes where needed. +- **SYM-PROV-004:** An adapter MUST implement pagination without silently truncating results and MUST expose completeness/freshness. +- **SYM-PROV-005:** An adapter MUST preserve provider IDs as opaque strings and MUST NOT infer identity from URL shape. +- **SYM-PROV-006:** Unknown media types and unavailable items MUST survive import as explicit normalized states. +- **SYM-PROV-007:** Adapter calls MUST have bounded connect/read/overall timeouts and cooperative cancellation. +- **SYM-PROV-008:** Provider errors MUST map to the normalized error taxonomy while preserving a sanitized provider code and correlation ID. +- **SYM-PROV-009:** Adapter logging MUST NOT include tokens, authorization codes, client secrets, raw authentication headers, or unrestricted payloads. +- **SYM-PROV-010:** Provider write methods MUST declare idempotency/reconciliation semantics; “retryable” is not a sufficient declaration. +- **SYM-PROV-011:** Rate-limit handling MUST honor provider reset or `Retry-After` signals and share budget state across concurrent work for the connection/provider. +- **SYM-PROV-012:** Adapter contract tests MUST run against deterministic fixtures/fakes without network access. +- **SYM-PROV-013:** Live-provider smoke tests MUST be separately enabled, use dedicated accounts/data, clean up safely, and never gate ordinary contributor tests. +- **SYM-PROV-014:** Raw provider payload storage MUST be minimized, versioned, and governed by provider retention/deletion policy. +- **SYM-PROV-015:** An adapter MUST publish its terms/API research review date and fail visibly when a known incompatible provider contract version is detected. +- **SYM-PROV-016:** Reverse-engineered or unofficial access MUST be a distinct adapter with explicit user opt-in and risk labeling; it MUST NOT masquerade as an official capability. +- **SYM-PROV-017:** Every adapter manifest MUST publish access basis, maturity, product-support level, upstream dependencies, multi-instance support, and last review date. +- **SYM-PROV-018:** An external object identity MUST include its object type and MUST include a provider-instance or connection namespace whenever upstream IDs are not proven globally unique. +- **SYM-PROV-019:** Planning MUST calculate effective capabilities from adapter, connection, object, and live-health constraints; a connection-wide capability MUST NOT override an object-level denial such as a read-only playlist. +- **SYM-PROV-020:** Unofficial or `best_effort` status MUST be visible before authorization and again in any plan that depends on that adapter. + +## Normalized error taxonomy + +Adapters return a stable category plus safe detail: + +```text +authentication_required +authorization_revoked +permission_denied +capability_unavailable +not_found +item_unavailable +invalid_request +rate_limited +provider_unavailable +timeout +network_error +conflict +unknown_write_outcome +provider_contract_changed +``` + +The normalized error includes retry advice, retry-after instant when known, whether user action is required, provider code, and safe operation context. It never includes raw response bodies by default. + +## Import and freshness contract + +An import session records: + +- connection and adapter version; +- requested collections/capabilities; +- start/end time and completeness; +- pages/items observed, skipped, invalid, or inaccessible; +- provider cursors/revisions as opaque values; +- per-object observed-at and refresh-by metadata; +- quota consumed when observable; and +- terminal/warning issues. + +Only a complete import may mark objects missing from the complete provider result as no longer observed. A failed or truncated import MUST NOT mass-delete prior state. + +## Provider-specific configuration + +The adapter may define typed, validated configuration fields such as client ID, redirect registration hints, market, or conservative request budget. The general App options file is not a secret store. Configuration descriptions MUST identify which values are secret and where they are persisted. + +Provider-specific configuration does not leak into recording, playlist, or operation aggregates. Use adapter-owned configuration referenced by the connection. + +## Adding a provider + +A new provider proposal must include: + +1. official versus unofficial status and applicable terms; +2. a dated capability matrix using this catalog; +3. authentication, token lifecycle, callback, and self-hosting analysis; +4. data-retention/deletion obligations; +5. rate/quota behavior and write idempotency analysis; +6. mapping of provider media types to domain types; +7. fixture-based contract tests and an optional live-smoke plan; and +8. UI limitations and user-facing risk language. + +Adding a provider must not require a new field on `Recording` solely because the provider returns it; provider-specific metadata belongs to the representation unless promoted through a provider-independent domain decision. diff --git a/specs/CATALOG.md b/specs/CATALOG.md new file mode 100644 index 0000000..d21fa31 --- /dev/null +++ b/specs/CATALOG.md @@ -0,0 +1,25 @@ +# Capability and SDD catalog + +**Catalog version:** 1 + +This is the human-readable view of [`catalog.json`](./catalog.json). Until generation tooling exists, both files are updated together and checked during review. + +| Capability ID | Status | Primary SDD | Readiness summary | +| --- | --- | --- | --- | +| `home-assistant-app-runtime` | Draft | [Home Assistant App runtime and Ingress](home-assistant-app-runtime-and-ingress.md) | Blocked by storage/recovery, supported platform matrix, and secret-key design | +| `provider-connections-and-authorization` | Draft | [Provider connections and authorization](provider-connections-and-authorization.md) | Blocked by OAuth boundary and provider feasibility spikes | +| `library-import-and-provider-projections` | Draft | [Library import and provider projections](library-import-and-provider-projections.md) | Blocked by provider completeness, retention, and representative scale | +| `recording-identity-resolution` | Draft | [Recording identity resolution](recording-identity-resolution.md) | Blocked by the labeled corpus and accepted automatic-link policy | +| `one-time-playlist-copy` | Draft | [One-time playlist copy](one-time-playlist-copy.md) | Blocked by unresolved-entry policy and proven target write semantics | +| `durable-operations-and-recovery` | Draft | [Durable operations and recovery](durable-operations-and-recovery.md) | Blocked by the persistence/lease/restart spike and operating targets | + +## Deliberately absent + +There is no implementation SDD for persistent playlist synchronization. It remains specified only at the domain/future level until: + +- the one-time copy contract is accepted and proven; +- provider-owned versus Symphonia-owned playlist state is decided; +- change attribution and conflict policy are accepted; and +- provider revision/removal/reorder capabilities are demonstrated. + +Apple Music also remains future provider research rather than a catalog capability. Promoting it requires its feasibility gates and an explicit product-scope decision. diff --git a/specs/README.md b/specs/README.md new file mode 100644 index 0000000..b416ad2 --- /dev/null +++ b/specs/README.md @@ -0,0 +1,132 @@ +# Software Design Document standard + +This directory contains Symphonia's implementation-driving Software Design Documents (SDDs). An SDD is a vertical contract for one product capability: it joins product behavior, domain rules, architecture, provider boundaries, user experience, failure recovery, security, testing, documentation, and acceptance. + +Start new SDDs from [`_template.md`](./_template.md). The capability inventory and readiness state live in [`catalog.json`](./catalog.json); [`CATALOG.md`](./CATALOG.md) is its human-readable companion. + +## Relationship to the shared specifications + +The existing `docs/` documents remain the horizontal sources of truth: + +| Source | Owns | +| --- | --- | +| [`docs/product/product-specification.md`](../docs/product/product-specification.md) | Product scope, journeys, and stable requirement IDs | +| [`docs/domain/domain-model.md`](../docs/domain/domain-model.md) | Shared language, identity, invariants, copy/sync semantics | +| [`docs/architecture/system-architecture.md`](../docs/architecture/system-architecture.md) | System-wide boundaries, security, persistence, and operations | +| [`docs/providers/provider-specification.md`](../docs/providers/provider-specification.md) | Provider port and capability contract | +| [`docs/providers/provider-research.md`](../docs/providers/provider-research.md) | Dated official API evidence | +| [`docs/providers/home-assistant-ecosystem-review.md`](../docs/providers/home-assistant-ecosystem-review.md) | Dated implementation patterns and cautions | +| [`docs/decisions/README.md`](../docs/decisions/README.md) | Accepted architectural decisions | +| [`docs/open-questions.md`](../docs/open-questions.md) | Unresolved owner choices and research gates | + +An SDD does not duplicate or quietly override those sources. It selects the applicable requirements and makes them implementable end to end. If an SDD discovers a missing or conflicting global rule, update the owning document or record an ADR before marking the SDD ready. + +## Status model + +| Status | Meaning | Implementation consequence | +| --- | --- | --- | +| `Draft` | The capability boundary is useful, but evidence or decisions remain open | No production implementation | +| `Ready for review` | The contract is complete enough for owner and specialist review | No production implementation | +| `Ready for implementation` | Blocking decisions are resolved and all readiness gates pass | Implementation may begin only after explicit owner approval | +| `Implemented` | Code, tests, documentation, migrations, and evidence satisfy the SDD | Changes must update all evidence together | +| `Superseded` | A newer SDD or decision replaces this contract | No new work should target it | + +The initial Symphonia SDDs are prospective drafts. There is no implemented behavior to describe as an as-built baseline. + +## Required contract + +Every SDD MUST cover these concerns or state why a concern is not applicable: + +| Concern | Required outcome | +| --- | --- | +| Decision metadata | Status, date, owner, scope, review gates, blockers | +| Problem and evidence | Affected actor, current absence/behavior, primary sources, unknowns | +| Goals and limits | Outcomes, non-goals, fixed safety and product invariants | +| Product journey | Entrypoints, visible surfaces, normal and alternative flows | +| Domain/state model | Shared terms, ownership of facts, states, transitions, invariants | +| Configuration | Safe defaults, bounded alternatives, validation, precedence, persistence, migration | +| Architecture | Dependency direction, pure policy, use cases, ports, adapters, trust/state boundaries | +| UI/UX | Information hierarchy and concrete pending/action/partial/failure/completed examples | +| Failure and recovery | Retained facts, retryability, idempotency, cleanup, operator/user action | +| Security/privacy | Authorization, secrets, untrusted input, abuse/replay, data lifecycle | +| Observability | User status, logs/metrics, correlation, rate-limit and noise behavior | +| Compatibility/rollout | Existing data/jobs, migration, upgrade, rollback, irreversible effects | +| Testing | Numeric risk-derived budget and deterministic evidence | +| Documentation | User, setup, operator, recovery, architecture, and discoverability artifacts | +| Acceptance/DoD | Observable scenarios, requirement traceability, implementation order, gates | + +## Authoring rules + +- Use the existing `SYM-*` requirement IDs. A new normative rule needs an ID in the horizontal document that owns it before the SDD becomes ready. +- Distinguish accepted facts, proposed defaults, configurable alternatives, fixed limits, and open decisions. +- Use diagrams only when they clarify a multi-step flow, state machine, trust boundary, or dependency graph. Every diagram requires a nearby textual equivalent. +- A `MUST` needs an acceptance scenario and planned verification. A `SHOULD` needs a documented exception rule. +- Provider names belong in adapters and evidence, not in provider-independent policy branches. +- Live provider tests supplement but never replace deterministic offline contract tests. +- Partial success and unknown external write outcomes are first-class states, not generic failures. +- User-facing errors follow: impact, cause, next action, retained state. +- Configuration cannot weaken authorization, secret handling, auditability, identity provenance, or idempotency safeguards. + +## Numeric test budget + +Each implementation-driving SDD defines a minimum number of distinct cases distributed across its real risks. The number is a floor, not a substitute for covering every requirement. + +Budgets should cover, where applicable: + +- pure domain/configuration/planning; +- state transitions, concurrency, replay, cancellation, and unknown outcomes; +- application use cases; +- provider/App/persistence adapters and error mapping; +- HTTP/UI/accessibility/sanitization; +- integration, migration, backup/restore, and security abuse cases. + +Parameterized cases count separately only when they represent distinct behavior. No test budget may require live provider calls in the ordinary suite. + +## Readiness gate + +Before changing an SDD to `Ready for implementation`, reviewers must be able to answer yes: + +- Is the user outcome understandable without reading code? +- Are non-goals and fixed safety rules explicit? +- Are all owner choices and research blockers resolved or removed from scope? +- Are provider capabilities treated as runtime evidence rather than assumptions? +- Are architecture and trust boundaries enforceable by tests or static checks? +- Are failure, restart, retry, cancellation, partial success, and cleanup specified? +- Does every normative requirement map to acceptance and verification? +- Is the numeric test budget proportional to the risks? +- Are documentation, migration, backup, and rollback deliverables named? +- Are pending, action-required, partial, failed, and completed UX states concrete? + +Readiness is necessary but not authorization to implement. The owner must still explicitly approve implementation. + +## Current foundation set + +| Capability | SDD | Why it exists before code | +| --- | --- | --- | +| Home Assistant App runtime | [App runtime and Ingress](home-assistant-app-runtime-and-ingress.md) | Packaging, lifecycle, authentication boundary, persistence, backup | +| Provider connections | [Provider connections and authorization](provider-connections-and-authorization.md) | OAuth/cookies, secrets, callback boundary, effective capabilities | +| Provider imports | [Library import and provider projections](library-import-and-provider-projections.md) | Completeness, provenance, unavailable items, retention | +| Recording identity | [Recording identity resolution](recording-identity-resolution.md) | Conservative matching, evidence, manual decisions | +| Playlist copy | [One-time playlist copy](one-time-playlist-copy.md) | Immutable plan, explicit omissions, safe external writes | +| Durable execution | [Durable operations and recovery](durable-operations-and-recovery.md) | Leases, checkpoints, retries, restarts, partial outcomes | + +Persistent synchronization intentionally has no implementation SDD yet. It remains future scope until playlist ownership, conflict semantics, provider revision evidence, and the copy contract are resolved. + +## Manual validation while the repository has no toolchain + +Run after every SDD or catalog change: + +```text +git diff --check +``` + +Also verify: + +1. `catalog.json` parses as JSON and every referenced local path exists. +2. Each catalog capability has exactly one primary SDD. +3. Every SDD uses a catalog ID present in `catalog.json`. +4. Local Markdown links resolve. +5. Requirement IDs referenced by SDDs exist in the horizontal specifications. +6. `CATALOG.md` agrees with `catalog.json`. + +A repository-native validator and generated catalog may be added after the implementation toolchain is selected; that tooling choice must not drive the application stack. diff --git a/specs/_template.md b/specs/_template.md new file mode 100644 index 0000000..1a1866a --- /dev/null +++ b/specs/_template.md @@ -0,0 +1,241 @@ +# + +- Status: Draft +- Date: YYYY-MM-DD +- Catalog capability ID: `` +- Owners: Symphonia maintainers +- Scope: +- Related requirements: `` +- Related decisions/research: +- Required review gates: product UX, architecture, provider feasibility, testing, documentation, security/operations +- Open decisions blocking readiness: + +> Start with [`README.md`](./README.md). Remove this note when the SDD is ready. Mark a non-applicable section explicitly and explain why. + +## 1. Executive summary + +Describe the user-visible outcome, recommended default, and principal safety/correctness rule. + +```text + -> -> +``` + +## 2. Problem, current behavior, and evidence + +### 2.1 Problem + +Who is affected, what they cannot do safely, and why it matters. + +### 2.2 Current behavior + +State whether no implementation exists or describe verified behavior. Separate facts from assumptions. + +### 2.3 Evidence and unknowns + +- Repository specifications/ADRs: +- Official external sources: +- Community implementation evidence: +- Unknowns: + +## 3. Actors, surfaces, and terminology + +| Actor | Goal | Entry point | Visible surfaces | +| --- | --- | --- | --- | +| | | | | + +Define capability-specific terms and refer to the shared domain language. + +## 4. Goals, non-goals, and fixed invariants + +### 4.1 Goals + +1. + +### 4.2 Non-goals + +1. + +### 4.3 Fixed invariants + +1. + +## 5. Product journey + +| Stage | Proposed behavior | User/operator effect | +| --- | --- | --- | +| | | | + +For three or more dependent transitions, include a flow/state diagram and a textual equivalent. + +## 6. Functional behavior and state model + +### 6.1 Happy path + +1. + +### 6.2 Alternative and boundary paths + +- + +### 6.3 State machine + +| State | Entered when | User-visible meaning | Allowed next states | Recovery/owner | +| --- | --- | --- | --- | --- | +| | | | | | + +Specify duplicate, stale, out-of-order, cancellation, and partial behavior where applicable. + +## 7. Configuration contract + +| Input | Type | Recommended default | Allowed values/range | Scope/persistence | +| --- | --- | --- | --- | --- | +| `` | | | | | + +Define validation, precedence, migration, a meaningful alternative, and behavior intentionally not configurable. + +## 8. Clean Architecture design + +### 8.1 Responsibilities and dependency direction + +| Boundary | Owns | Must not own/import | +| --- | --- | --- | +| Domain/pure policy | | Provider/framework details | +| Application | | Concrete SDK/process concerns | +| Adapters/data | | Product policy | +| Infrastructure/composition | | Business decisions | +| Entrypoints/presentation | | Duplicated orchestration | + +### 8.2 Contracts, durable state, and trust boundaries + +- Pure decisions: +- Application contracts: +- Semantic ports: +- Durable state/schema ownership: +- Concurrency/idempotency: +- Trusted/untrusted inputs: +- Error mapping: + +### 8.3 Executable architecture constraints + +- + +## 9. UI/UX and content contract + +### 9.1 Information hierarchy + +1. Current status. +2. Completed facts. +3. What happens next. +4. Required human action or an explicit statement that none is needed. +5. Impact and retained state. +6. Collapsed, sanitized technical evidence. + +### 9.2 Representative states + +Provide concrete content for pending, action required, partial, failed/blocked, and completed states. Use text, not color or icons alone. + +### 9.3 Accessibility and localization + +- Locale source/fallback: +- Keyboard/focus/announcement behavior: +- Narrow/mobile behavior: +- Text alternative for diagrams: +- Sanitization of provider/user content: + +## 10. Failure, recovery, and cleanup + +| Failure/partial state | User impact | Retained facts | Automatic retry | Required action | Cleanup | +| --- | --- | --- | --- | --- | --- | +| | | | | | | + +Errors follow `impact -> cause -> next action -> retained state`. + +## 11. Security, permissions, and privacy + +1. +2. +3. +4. + +## 12. Observability and operational UX + +- User-facing state: +- Logs, metrics, and correlation: +- Pending external dependency versus service failure: +- Rate-limit/retry visibility: +- Notification/noise budget: + +## 13. Compatibility, migration, rollout, and rollback + +- Existing data/in-flight operations: +- Schema/configuration migration: +- Rollout stages: +- Rollback and irreversible effects: + +## 14. Testing strategy and numeric budget + +| Area | Minimum distinct cases | Behaviors/risks covered | +| --- | ---: | --- | +| Domain/configuration/planning | | | +| State/application/idempotency/races | | | +| Adapters/contracts | | | +| HTTP/UI/accessibility/sanitization | | | +| Integration/security/migration | | | +| **Total** | **** | No double counting | + +Define deterministic fakes/clocks/IDs, coverage expectations, contract fixtures, live-smoke isolation, and required human evidence. + +## 15. Documentation and discoverability + +| Audience | Artifact | Required content | Validation/navigation | +| --- | --- | --- | --- | +| User | | | | +| Setup owner | | | | +| Operator | | | | +| Contributor | | | | + +## 16. Acceptance scenarios + +1. Given ``, when ``, then ``. +2. . +3. . +4. . +5. . +6. . +7. . +8. . + +## 17. Requirements traceability + +| Requirement | Policy/use case/adapter/presentation | Test or evidence | Documentation | +| --- | --- | --- | --- | +| `` | | | | + +## 18. Implementation sequence + +1. Resolve blockers and approve contracts/configuration. +2. Add failing architecture/contract tests and pure policies. +3. Add application state/use cases and deterministic tests. +4. Add adapters and contract tests. +5. Add presentation, accessibility, and documentation. +6. Add migration/recovery/security evidence and integration validation. + +## 19. Definition of Done + +- [ ] Status is `Ready for implementation` and explicit owner approval to implement exists. +- [ ] Every normative requirement has acceptance and traceability. +- [ ] Architecture boundaries and external contracts are automatically enforced. +- [ ] Numeric test budget and stated coverage pass. +- [ ] Configuration, migration, backup, rollback, and recovery agree. +- [ ] Pending, action-required, partial, failed, and completed UX states are verified. +- [ ] Accessibility, localization, sanitization, and secret-safety gates pass. +- [ ] User/setup/operator/contributor documentation is complete and discoverable. +- [ ] Catalog status and implementation evidence are current. +- [ ] No readiness-blocking decision remains. + +## 20. References and decisions + +- Primary sources: +- Related SDDs/ADRs: +- Accepted and rejected alternatives: +- Follow-up work outside scope: diff --git a/specs/catalog.json b/specs/catalog.json new file mode 100644 index 0000000..12e5c44 --- /dev/null +++ b/specs/catalog.json @@ -0,0 +1,313 @@ +{ + "$schema": "./catalog.schema.json", + "version": 1, + "capabilities": [ + { + "id": "home-assistant-app-runtime", + "title": "Home Assistant App runtime and Ingress", + "status": "draft", + "scope": "Install, start, authenticate, persist, upgrade, back up, restore, and diagnose the primary Supervisor-managed Symphonia App", + "owner": "Symphonia maintainers", + "specification": "specs/home-assistant-app-runtime-and-ingress.md", + "requirements": [ + "SYM-PROD-001", + "SYM-ACC-005", + "SYM-ARCH-001", + "SYM-ARCH-003", + "SYM-ARCH-006", + "SYM-ARCH-007", + "SYM-SEC-008", + "SYM-SEC-009", + "SYM-SEC-011", + "SYM-OBS-003", + "SYM-OBS-005", + "SYM-HA-001", + "SYM-HA-002", + "SYM-HA-003", + "SYM-HA-004", + "SYM-HA-005", + "SYM-HA-006", + "SYM-HA-007", + "SYM-HA-008", + "SYM-HA-009", + "SYM-TEST-009", + "SYM-TEST-012", + "SYM-TEST-013", + "SYM-DEP-001", + "SYM-DEP-002", + "SYM-DEP-003", + "SYM-DEP-004", + "SYM-DEP-005", + "SYM-DEP-006", + "SYM-DEP-007", + "SYM-DEP-008", + "SYM-DEP-009", + "SYM-DEP-010" + ], + "evidence": [ + "docs/decisions/0003-home-assistant-app-primary.md", + "docs/architecture/system-architecture.md", + "docs/providers/provider-research.md", + "docs/providers/home-assistant-ecosystem-review.md" + ], + "implementationEvidence": { + "code": [], + "tests": [], + "documentation": [] + }, + "blockers": [ + "RG-004 storage and recovery spike", + "Supported Home Assistant versions and CPU architectures", + "Secret-encryption key source and backup/restore behavior", + "OQ-005 standalone release timing and authentication" + ] + }, + { + "id": "provider-connections-and-authorization", + "title": "Provider connections and authorization", + "status": "draft", + "scope": "Disclose provider risk, authorize an account, store and refresh grants, probe effective capabilities, and disconnect safely", + "owner": "Symphonia maintainers", + "specification": "specs/provider-connections-and-authorization.md", + "requirements": [ + "SYM-ACC-002", + "SYM-ACC-003", + "SYM-ACC-004", + "SYM-ACC-006", + "SYM-PROV-002", + "SYM-PROV-003", + "SYM-PROV-008", + "SYM-PROV-009", + "SYM-PROV-015", + "SYM-PROV-016", + "SYM-PROV-017", + "SYM-PROV-019", + "SYM-PROV-020", + "SYM-SEC-001", + "SYM-SEC-002", + "SYM-SEC-003", + "SYM-SEC-004", + "SYM-SEC-005", + "SYM-SEC-006", + "SYM-SEC-007", + "SYM-SEC-008", + "SYM-SEC-009", + "SYM-SEC-010", + "SYM-TEST-005", + "SYM-TEST-011", + "SYM-TEST-012" + ], + "evidence": [ + "docs/providers/provider-specification.md", + "docs/providers/provider-research.md", + "docs/providers/home-assistant-ecosystem-review.md", + "docs/architecture/system-architecture.md" + ], + "implementationEvidence": { + "code": [], + "tests": [], + "documentation": [] + }, + "blockers": [ + "OQ-004 direct App OAuth versus companion-integration authorization broker", + "RG-002 Home Assistant OAuth callback spike", + "RG-001 provider-specific scopes and account constraints", + "Decision on whether an unofficial YouTube Music adapter exists in the MVP" + ] + }, + { + "id": "library-import-and-provider-projections", + "title": "Library import and provider projections", + "status": "draft", + "scope": "Import complete provider collections and ordered playlists while retaining provenance, availability, freshness, and policy metadata", + "owner": "Symphonia maintainers", + "specification": "specs/library-import-and-provider-projections.md", + "requirements": [ + "SYM-PROD-003", + "SYM-LIB-001", + "SYM-LIB-002", + "SYM-LIB-003", + "SYM-LIB-004", + "SYM-LIB-005", + "SYM-LIB-006", + "SYM-PROV-004", + "SYM-PROV-005", + "SYM-PROV-006", + "SYM-PROV-007", + "SYM-PROV-008", + "SYM-PROV-011", + "SYM-PROV-012", + "SYM-PROV-013", + "SYM-PROV-014", + "SYM-PROV-018", + "SYM-ARCH-004", + "SYM-ARCH-005", + "SYM-TEST-003", + "SYM-TEST-006", + "SYM-TEST-010", + "SYM-TEST-011" + ], + "evidence": [ + "docs/domain/domain-model.md", + "docs/providers/provider-specification.md", + "docs/providers/provider-research.md", + "docs/open-questions.md" + ], + "implementationEvidence": { + "code": [], + "tests": [], + "documentation": [] + }, + "blockers": [ + "RG-001 collection semantics and completeness per provider", + "OQ-008 local data retention and export policy", + "OQ-007 representative library sizes and import-duration targets", + "Provider-specific refresh and deletion obligations" + ] + }, + { + "id": "recording-identity-resolution", + "title": "Recording identity resolution", + "status": "draft", + "scope": "Resolve provider tracks to provider-independent recordings with versioned evidence, conservative automation, and durable manual decisions", + "owner": "Symphonia maintainers", + "specification": "specs/recording-identity-resolution.md", + "requirements": [ + "SYM-PROD-002", + "SYM-PROD-003", + "SYM-LIB-002", + "SYM-MATCH-001", + "SYM-MATCH-002", + "SYM-MATCH-003", + "SYM-MATCH-004", + "SYM-MATCH-005", + "SYM-MATCH-006", + "SYM-MATCH-007", + "SYM-MATCH-008", + "SYM-ARCH-014", + "SYM-TEST-007", + "SYM-TEST-008" + ], + "evidence": [ + "docs/decisions/0001-provider-independent-recording-domain.md", + "docs/domain/domain-model.md", + "docs/providers/provider-research.md", + "docs/providers/home-assistant-ecosystem-review.md" + ], + "implementationEvidence": { + "code": [], + "tests": [], + "documentation": [] + }, + "blockers": [ + "RG-003 licensed or synthetic labeled matching corpus", + "Automatic-link precision and false-link tolerance", + "Versioned scoring and confidence policy", + "Provider-specific candidate retrieval feasibility and quota" + ] + }, + { + "id": "one-time-playlist-copy", + "title": "One-time playlist copy", + "status": "draft", + "scope": "Preview and execute a finite playlist copy with an immutable plan, explicit omissions, ordered writes, and item-level outcomes", + "owner": "Symphonia maintainers", + "specification": "specs/one-time-playlist-copy.md", + "requirements": [ + "SYM-PROD-002", + "SYM-PL-001", + "SYM-PL-002", + "SYM-PL-003", + "SYM-PL-004", + "SYM-PL-005", + "SYM-PL-006", + "SYM-PL-007", + "SYM-PL-008", + "SYM-PL-009", + "SYM-PROD-004", + "SYM-PROD-005", + "SYM-PROD-006", + "SYM-PROV-010", + "SYM-PROV-019", + "SYM-ARCH-009", + "SYM-ARCH-010", + "SYM-JOB-003", + "SYM-JOB-004", + "SYM-JOB-005", + "SYM-TEST-004", + "SYM-TEST-006" + ], + "evidence": [ + "docs/decisions/0002-copy-and-sync-are-distinct.md", + "docs/domain/domain-model.md", + "docs/product/product-specification.md", + "docs/providers/provider-research.md" + ], + "implementationEvidence": { + "code": [], + "tests": [], + "documentation": [] + }, + "blockers": [ + "OQ-003 default treatment of non-ready entries", + "RG-001 proven target create/add/order/duplicate behavior", + "Provider-specific unknown-write reconciliation strategy", + "Target naming, visibility, and existing-playlist selection policy" + ] + }, + { + "id": "durable-operations-and-recovery", + "title": "Durable operations and recovery", + "status": "draft", + "scope": "Execute imports and provider writes through durable, observable, lease-based operations that recover safely across restarts", + "owner": "Symphonia maintainers", + "specification": "specs/durable-operations-and-recovery.md", + "requirements": [ + "SYM-PROD-004", + "SYM-PROD-005", + "SYM-PROD-006", + "SYM-ARCH-001", + "SYM-ARCH-002", + "SYM-ARCH-005", + "SYM-ARCH-008", + "SYM-ARCH-009", + "SYM-ARCH-010", + "SYM-JOB-001", + "SYM-JOB-002", + "SYM-JOB-003", + "SYM-JOB-004", + "SYM-JOB-005", + "SYM-JOB-006", + "SYM-JOB-007", + "SYM-JOB-008", + "SYM-OBS-001", + "SYM-OBS-002", + "SYM-OBS-003", + "SYM-OBS-004", + "SYM-OBS-005", + "SYM-OBS-006", + "SYM-TEST-004", + "SYM-TEST-013", + "SYM-DEP-002", + "SYM-DEP-008" + ], + "evidence": [ + "docs/architecture/system-architecture.md", + "docs/development/development-specification.md", + "docs/open-questions.md", + "docs/providers/home-assistant-ecosystem-review.md" + ], + "implementationEvidence": { + "code": [], + "tests": [], + "documentation": [] + }, + "blockers": [ + "RG-004 storage engine and lease/crash-recovery spike", + "OQ-007 representative operation sizes and timing targets", + "Process topology and worker concurrency defaults", + "Retention policy for step detail and operation history" + ] + } + ] +} diff --git a/specs/catalog.schema.json b/specs/catalog.schema.json new file mode 100644 index 0000000..dc79df9 --- /dev/null +++ b/specs/catalog.schema.json @@ -0,0 +1,111 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://symphonia.local/specs/catalog.schema.json", + "title": "Symphonia SDD capability catalog", + "type": "object", + "additionalProperties": false, + "required": ["version", "capabilities"], + "properties": { + "$schema": { + "type": "string" + }, + "version": { + "const": 1 + }, + "capabilities": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "title", + "status", + "scope", + "owner", + "specification", + "requirements", + "evidence", + "implementationEvidence", + "blockers" + ], + "properties": { + "id": { + "type": "string", + "pattern": "^[a-z0-9]+(?:-[a-z0-9]+)*$" + }, + "title": { + "type": "string", + "minLength": 1 + }, + "status": { + "enum": [ + "draft", + "ready-for-review", + "ready-for-implementation", + "implemented", + "superseded" + ] + }, + "scope": { + "type": "string", + "minLength": 1 + }, + "owner": { + "type": "string", + "minLength": 1 + }, + "specification": { + "type": "string", + "pattern": "^specs/.+\\.md$" + }, + "requirements": { + "type": "array", + "items": { + "type": "string", + "pattern": "^SYM-[A-Z]+-[0-9]{3}$" + }, + "uniqueItems": true + }, + "evidence": { + "type": "array", + "items": { + "type": "string" + }, + "uniqueItems": true + }, + "implementationEvidence": { + "type": "object", + "additionalProperties": false, + "required": ["code", "tests", "documentation"], + "properties": { + "code": { + "type": "array", + "items": { "type": "string" }, + "uniqueItems": true + }, + "tests": { + "type": "array", + "items": { "type": "string" }, + "uniqueItems": true + }, + "documentation": { + "type": "array", + "items": { "type": "string" }, + "uniqueItems": true + } + } + }, + "blockers": { + "type": "array", + "items": { + "type": "string" + }, + "uniqueItems": true + } + } + } + } + } +} diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md new file mode 100644 index 0000000..0f113e8 --- /dev/null +++ b/specs/durable-operations-and-recovery.md @@ -0,0 +1,318 @@ +# Durable operations and recovery + +- Status: Draft +- Date: 2026-09-20 +- Catalog capability ID: `durable-operations-and-recovery` +- Owners: Symphonia maintainers +- Scope: execute imports and provider writes through durable, observable operations that recover safely across restarts, rate limits, uncertain writes, and upgrades. +- Related requirements: `SYM-PROD-004`–`SYM-PROD-006`, `SYM-ARCH-001`–`SYM-ARCH-002`, `SYM-ARCH-005`, `SYM-ARCH-008`–`SYM-ARCH-010`, `SYM-JOB-001`–`SYM-JOB-008`, `SYM-OBS-001`–`SYM-OBS-006`, `SYM-TEST-004`, `SYM-TEST-013`, `SYM-DEP-002`, `SYM-DEP-008` +- Related decisions/research: [system architecture](../docs/architecture/system-architecture.md), [development specification](../docs/development/development-specification.md), `RG-004` +- Required review gates: architecture, persistence/recovery, provider contracts, testing, documentation, security/operations +- Open decisions blocking readiness: persistent store; worker/process topology; lease and retention parameters; supported migration strategy; representative operation sizes and timing targets + +## 1. Executive summary + +Symphonia shall execute imports, provider reads, identity work, and playlist writes as durable operations whose authoritative state survives process restarts, Home Assistant App upgrades, temporary provider failures, and rate limits. + +The operation engine shall use explicit state transitions, bounded leases, idempotency keys, checkpoints, and reconciliation. It shall never equate an in-memory task with durable progress or blindly retry a provider mutation whose outcome is unknown. + +## 2. Problem statement and evidence + +Provider work crosses unreliable networks and may take longer than an HTTP request or Home Assistant lifecycle event. A process may stop after a provider accepted a write but before Symphonia recorded the response. Without a durable contract, retrying can duplicate playlists or entries, while abandoning work can leave invisible partial results. + +The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. + +Evidence sources: + +- [Product specification](../docs/product/product-specification.md) +- [System architecture](../docs/architecture/system-architecture.md) +- [Provider specification](../docs/providers/provider-specification.md) +- [Development specification](../docs/development/development-specification.md) + +## 3. Actors and authorization + +- Application services create operations on behalf of an authenticated, authorized actor or a future explicitly approved scheduler. +- Workers claim and advance operations. +- Provider adapters report safe-retry, wait, permanent-failure, and unknown-outcome information. +- Administrators inspect, cancel, resume, or export diagnostics through authorized interfaces. + +Workers do not gain broader provider or household access merely by reading a queued operation. The stored operation references a scoped connection and authorization context. + +## 4. Goals, non-goals, and invariants + +### Goals + +- Persist operation intent before side effects begin. +- Recover automatically from expected restarts and transient failures. +- Prevent concurrent workers from advancing the same step unsafely. +- Make retry timing, progress, partial results, and required user action visible. +- Reconcile unknown provider outcomes before any repeated mutation. +- Support versioned schema and application upgrades. + +### Non-goals + +- A general-purpose distributed workflow platform. +- Exactly-once delivery claims across third-party providers. +- Infinite automatic retry. +- Hiding permanent or ambiguous failures behind a success state. +- Automatic rollback of irreversible provider mutations. + +### Invariants + +- The persistent store, not worker memory, is authoritative. +- Intent and an idempotency key are persisted before an external side effect. +- Only the current lease holder may checkpoint a running step. +- Leases expire and are recoverable; they are never permanent locks. +- A step with unknown external outcome enters reconciliation before retry. +- State transitions are validated, atomic, and auditable. +- Terminal results are immutable except for separately recorded reconciliation or remediation operations. +- Cancellation prevents future work but does not rewrite confirmed history. + +## 5. User journey and operational flow + +1. An application service validates a requested action and stores its immutable intent, actor context, and idempotency key. +2. The same transaction or a recoverable dispatch mechanism makes the operation eligible for work. +3. A worker atomically claims a time-bounded lease. +4. The worker loads the last durable checkpoint and validates operation/schema compatibility. +5. Before each provider side effect, the worker records the intended step and reconciliation data. +6. The worker performs the call, categorizes the result, and atomically checkpoints confirmed progress. +7. It renews the lease only while healthy, releases it when waiting, or records a terminal result. +8. The UI reads persisted state and events rather than subscribing only to a worker process. + +After a crash, another worker waits for lease expiry, claims the operation, and resumes from the last confirmed checkpoint. If a call may have succeeded externally, it reconciles that step first. + +## 6. Functional contract and state model + +### Operation states + +- `queued`: eligible for a worker. +- `running`: actively owned by a valid lease. +- `waiting_rate_limit`: paused until a recorded provider-safe time. +- `waiting_user`: requires an explicit user decision or renewed authorization. +- `retry_scheduled`: transient failure with a bounded next attempt time. +- `succeeded`: all required outcomes confirmed. +- `partial`: work ended with a mix of confirmed success and disclosed permanent failure or exclusion. +- `failed`: the operation cannot safely achieve its required outcome. +- `cancelled`: cancellation was accepted and no more work will be scheduled. + +### Step outcome categories + +- `confirmed_success` +- `safe_retry` +- `rate_limited` +- `authorization_required` +- `permanent_failure` +- `unknown_outcome` +- `cancelled_before_start` + +`unknown_outcome` is not a retry category. It creates a reconciliation step whose result may be confirmed success, confirmed absence followed by safe retry, or `waiting_user`/`failed` if it cannot be determined safely. + +### Transition rules + +- Terminal states cannot transition back to `running`. +- Remediation of a terminal operation creates a linked new operation. +- `running` requires an unexpired lease token and owner. +- `waiting_rate_limit` and `retry_scheduled` store an absolute next-eligible time plus the source of that timing. +- `waiting_user` stores a redacted reason and a bounded set of permitted actions. +- Cancellation is checked before claim, before every side effect, and between provider batches. + +### Idempotency + +The operation request key deduplicates logically identical submissions within a defined scope. Each external mutation step has a stable key or reconciliation fingerprint derived from immutable intent—not from an attempt count. Where a provider lacks idempotency support, the adapter shall supply a deterministic reconciliation strategy or declare the write unsupported. + +## 7. Configuration contract + +| Setting | Scope | Default | Validation and notes | +|---|---|---|---| +| Global worker concurrency | App | Conservative | Bounded by CPU/memory and provider fairness. | +| Per-provider concurrency | Adapter | Provider-specific | Must respect documented and observed limits. | +| Lease duration | Runtime | To be determined by spike | Must exceed normal checkpoint interval and remain recoverable. | +| Lease renewal interval | Runtime | Fraction of lease duration | Renewal failure stops new side effects. | +| Maximum attempts | Operation class | Bounded | Permanent and unknown outcomes bypass blind retry. | +| Backoff policy | Adapter | Exponential with jitter, capped | Honors provider retry hints when safe. | +| Event retention | App | Pending privacy/storage decision | Terminal summary and audit requirements may outlive verbose events. | +| Diagnostic retention | App | Pending policy | Redacted and bounded. | + +Users may cancel operations, but they shall not be offered arbitrary knobs that weaken lease safety, idempotency, retention obligations, or provider rate-limit compliance. + +## 8. Architecture and boundaries + +### Domain layer + +Defines operation intent, states, steps, attempts, outcome categories, cancellation, and transition invariants without depending on a queue library, database driver, Home Assistant, or provider SDK. + +### Application layer + +Creates operations, schedules eligibility, claims leases, dispatches handlers, checkpoints progress, performs reconciliation, and publishes redacted progress views. + +### Ports + +- Transactional operation repository. +- Clock and unique identifier source. +- Eligibility notification/wakeup mechanism. +- Operation handler registry. +- Provider reconciliation and rate-limit advice. +- Audit event sink and redacted diagnostic exporter. + +### Adapters + +- A selected persistent store implements atomic transitions and leases. +- The runtime may use an in-process worker initially, but correctness cannot depend on it staying alive. +- Provider adapters translate API responses into the shared outcome categories. +- Home Assistant App lifecycle hooks request graceful shutdown and expose health; they do not own job truth. + +The persistent store and wakeup mechanism may be one technology or separate ones. The design shall not require an external broker unless evidence justifies that operational cost. + +## 9. UI, UX, and accessibility + +The operations view shall show action type, actor-safe label, creation time, current state, progress numerator/denominator when meaningful, next retry time, and available actions. + +Example running state: + +> Copying playlist: 78 of 124 entries confirmed. You can leave this page; the operation will continue. + +Example rate-limit state: + +> The provider asked Symphonia to wait. The next attempt is scheduled for 14:32. No action is required. + +Example user-action state: + +> This connection needs authorization before the operation can continue. Reauthorize or cancel. + +Example recovery state: + +> Symphonia restarted while a provider request was in progress. It is checking the provider before continuing to avoid duplicate changes. + +Progress shall not move backward without an explicit explanation. Status shall not rely only on color, and live updates shall use non-disruptive accessible announcements. + +## 10. Failure and recovery semantics + +| Failure | Required behavior | Recovery | +|---|---|---| +| Worker crashes before side effect | Lease expires; checkpoint remains authoritative | Another worker safely resumes | +| Worker crashes after external acceptance but before checkpoint | Step remains uncertain | Reconcile externally before deciding success or retry | +| Store unavailable | Stop new external side effects; do not rely on memory-only progress | Resume after durable store health returns | +| Lease renewal fails | Stop starting side effects and relinquish/expire ownership | Another claim after store recovery and lease expiry | +| Duplicate dispatch | Atomic claim permits one active lease | Extra dispatch becomes a no-op | +| Rate limit | Persist provider advice and release active execution | Become eligible at safe time | +| Authorization expires | Persist `waiting_user`; retain checkpoint | Resume after scoped reauthorization | +| Repeated transient failure | Apply bounded backoff and attempt ceiling | End `failed` with remediation guidance | +| Permanent item failures | Continue only when operation policy permits | End `partial` with per-item results | +| Incompatible application/schema upgrade | Do not claim or mutate externally | Run verified migration or require operator action | +| Cancellation races with a provider call | Reconcile the in-flight call; start no later steps | Report confirmed partial state accurately | + +Graceful shutdown stops new claims, lets safe checkpoints finish within a bounded interval, and then releases or permits leases to expire. Correctness must also hold for abrupt termination. + +## 11. Security and privacy + +- Operation payloads store credential references, never raw tokens. +- Payloads, results, and diagnostics are classified and minimized; playlist contents and provider identifiers are household data. +- Every administrative action is authorized and audited. +- Handlers validate payload schema and version before use. +- Logs exclude secrets and avoid full provider response bodies by default. +- Retry and error messages displayed in the UI are redacted and safe for the current actor. +- Queue/store corruption, forged state transitions, and lease theft are included in the threat model. + +## 12. Observability and supportability + +Structured events include operation creation, eligibility, claim, renewal, checkpoint, wait, retry schedule, reconciliation, cancellation, and terminal transition. Each event includes operation ID, operation type, state, attempt count, adapter category, and bounded error code; it excludes credentials and unbounded content. + +Metrics include queue depth, oldest eligible age, running leases, expired lease recoveries, state counts, execution latency, wait duration, attempt counts, unknown outcomes, reconciliation results, and terminal result ratios. Cardinality shall remain bounded. + +Health distinguishes API availability, persistent-store readiness, worker liveness, claim progress, and migration state. A redacted export provides state history, version information, checkpoints, lease history, and categorized errors. + +## 13. Rollout, migration, and compatibility + +Before implementation readiness, a persistence spike shall prove atomic claim, lease expiry, transactional checkpointing, backup/restore behavior under `/data`, and migration safety in the chosen Home Assistant App runtime. + +Operation records and payloads carry explicit schema and handler versions. Upgrade tests cover queued, running, waiting, partial-progress, cancelled, and terminal fixtures. A version that cannot safely resume an operation shall leave it untouched and expose an actionable incompatibility state rather than guessing. + +Initial rollout uses one App instance and conservative concurrency. Multi-instance execution is out of scope unless a later deployment model requires it, but lease semantics shall not depend on process-local mutexes. + +## 14. Numeric test budget + +Minimum planned automated tests: **86**. + +| Area | Minimum | +|---|---:| +| Domain states, transitions, cancellation, and invariants | 20 | +| Claims, leases, races, checkpoints, idempotency, and reconciliation | 24 | +| Persistence and provider outcome adapter contracts | 16 | +| UI states and accessibility | 8 | +| Integration, restart, migration, security, corruption, and redaction | 18 | + +Required deterministic tests include duplicate dispatch, concurrent claim, lease expiry, clock boundaries, crash before/after every checkpoint, store outage, retry exhaustion, rate-limit timing, cancellation races, unknown outcomes, and compatible/incompatible upgrades. + +## 15. Documentation impact + +Implementation shall update: + +- administrator guidance for storage, backup, restore, shutdown, and diagnostics; +- user guidance for progress, waits, cancellation, partial results, and recovery; +- architecture documentation with the selected store and worker topology; +- privacy/retention documentation; +- runbooks for stuck operations, migration failures, and provider reconciliation; +- the SDD catalog and traceability evidence. + +## 16. Acceptance criteria + +1. Operation intent and idempotency data are durable before any external side effect. +2. State transitions are atomic, validated, and auditable. +3. Only one valid lease holder can advance an operation step. +4. A restart at every defined checkpoint resumes without losing confirmed progress. +5. Unknown provider outcomes enter reconciliation and never blind retry. +6. Duplicate submissions and duplicate dispatches do not duplicate logical work. +7. Rate limits and transient failures produce bounded, visible waits. +8. Expired authorization produces `waiting_user` without discarding progress. +9. Cancellation starts no later work and accurately accounts for in-flight outcomes. +10. Store unavailability prevents new external side effects. +11. Supported upgrades preserve or safely refuse every persisted state fixture. +12. Logs, metrics, events, and diagnostics contain no credentials or prohibited content. +13. The numeric test budget and fault-injection suite pass. + +## 17. Requirement traceability + +| Requirement | Design location | Planned verification | +|---|---|---| +| `SYM-JOB-001`–`SYM-JOB-008` | Sections 6-11 | State, lease, checkpoint, retry, fairness, time, cancellation, and reconciliation suites | +| `SYM-PROD-004`–`SYM-PROD-006` | Sections 6, 10, and 13 | Attribution, persisted state, item-failure, and error tests | +| `SYM-ARCH-001`–`SYM-ARCH-002`, `SYM-ARCH-005`, `SYM-ARCH-008`–`SYM-ARCH-010` | Sections 5-14 | Durability, atomicity, history, sanitation, retry, and partial-result tests | +| `SYM-OBS-001`–`SYM-OBS-006` | Section 13 | Correlation, event, metric cardinality, health, diagnostic, and audit tests | +| `SYM-TEST-004`, `SYM-TEST-013` | Sections 14-15 | Provider-write fault and migration release gates | +| `SYM-DEP-002`, `SYM-DEP-008` | Sections 6, 11, and 14 | App restart and graceful/abrupt shutdown tests | + +## 18. Sequence sketch + +```text +Application service -> Operation store: persist intent + idempotency key +Application service -> Wakeup adapter: signal eligibility +Worker -> Operation store: atomically claim bounded lease +Worker -> Operation store: persist step intent + reconciliation fingerprint +Worker -> Provider adapter: perform external call +alt confirmed response + Worker -> Operation store: checkpoint confirmed result +else unknown response + Worker -> Operation store: persist reconciliation-required state + Worker -> Provider adapter: inspect external state + Worker -> Operation store: record reconciled result +end +Worker -> Operation store: wait, continue, or terminal transition +UI -> Operation store: read durable progress view +``` + +## 19. Definition of done + +- Persistent store and worker topology decisions are recorded in ADRs. +- The persistence spike demonstrates required atomicity, restart, and migration behavior. +- Threat model and failure-mode review are complete. +- Operation handlers for the first vertical capability pass the shared contract suite. +- Tests meet the numeric budget and deterministic fault-injection requirements. +- Runbooks, user/admin documentation, and catalog evidence are current. +- Project owner explicitly approves implementation and later release. + +## 20. References + +- [SDD standard](README.md) +- [SDD catalog](CATALOG.md) +- [Home Assistant App runtime and Ingress SDD](home-assistant-app-runtime-and-ingress.md) +- [One-time playlist copy SDD](one-time-playlist-copy.md) diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md new file mode 100644 index 0000000..dfdf853 --- /dev/null +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -0,0 +1,323 @@ +# Home Assistant App runtime and Ingress + +- Status: Draft +- Date: 2026-09-20 +- Catalog capability ID: `home-assistant-app-runtime` +- Owners: Symphonia maintainers +- Scope: define the install, lifecycle, authentication, persistence, recovery, upgrade, backup, and diagnostics contract for the primary Supervisor-managed App. +- Related requirements: `SYM-PROD-001`, `SYM-ACC-005`, `SYM-HA-001`–`SYM-HA-009`, `SYM-DEP-001`–`SYM-DEP-010`, `SYM-SEC-008`–`SYM-SEC-011` +- Related decisions/research: [ADR 0003](../docs/decisions/0003-home-assistant-app-primary.md), [system architecture](../docs/architecture/system-architecture.md), [Home Assistant platform research](../docs/providers/provider-research.md#home-assistant-platform) +- Required review gates: product UX, architecture, Home Assistant platform, testing, documentation, security/operations +- Open decisions blocking readiness: storage/recovery result from `RG-004`; supported Home Assistant versions and CPU architectures; encryption-key/backup contract; standalone release timing from `OQ-005` + +## 1. Executive summary + +Symphonia's primary artifact is a Supervisor-managed Home Assistant App whose administrative UI is available through authenticated Ingress. The App owns the long-running service, durable state, background operations, and provider adapters; Home Assistant is the deployment/authentication boundary, not the music domain. + +The recommended runtime is one least-privileged, non-root App instance with durable `/data`, no host network or broad mounts, and no provider writes until migrations and recovery complete. + +```text +Install App -> validate configuration -> migrate/recover durable state +-> become ready -> administrator opens Ingress UI -> operations run durably +-> backup/upgrade/restart -> recover before any new provider write +``` + +## 2. Problem, current behavior, and evidence + +### 2.1 Problem + +Without a runtime contract, framework selection can accidentally determine authentication, filesystem access, backup semantics, OAuth exposure, or job recovery. A service that appears usable through Ingress can still expose an unauthenticated direct port or resume provider writes before restored state is safe. + +### 2.2 Current behavior + +No Symphonia runtime exists. The accepted behavior is limited to ADR 0003 and global requirements. This SDD is prospective and does not imply that App packaging, images, listeners, or migrations have been implemented. + +### 2.3 Evidence and unknowns + +- Accepted evidence: [ADR 0003](../docs/decisions/0003-home-assistant-app-primary.md). +- Platform evidence: current App, Ingress, persistent `/data`, backup, and security documentation linked from [provider research](../docs/providers/provider-research.md#home-assistant-platform). +- Comparative evidence: Music Assistant uses a server/App plus separate HA integration; see the [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md). +- Security evidence: a direct unauthenticated service port can bypass Ingress, as recorded in the ecosystem review. +- Unknowns: exact image/toolchain, supported architectures, encryption key source, online backup primitive, and first standalone release. + +## 3. Actors, surfaces, and terminology + +| Actor | Goal | Entry point | Visible surfaces | +| --- | --- | --- | --- | +| Home Assistant administrator | Install, configure, operate, upgrade, back up, and restore Symphonia | App repository and App page | App configuration, logs, Ingress panel, backup UI | +| Symphonia user | Manage provider connections and music operations | Ingress panel | Symphonia UI and operation history | +| Operator/contributor | Diagnose lifecycle or packaging faults | App logs/diagnostics and repository | Health/readiness, sanitized bundle, release notes | +| Companion integration | Expose future native HA surfaces | Versioned local API | Entities/actions/events, availability state | + +**Health** means the process can answer a liveness probe. **Readiness** means configuration, storage, migrations, and recovery are safe enough to serve requests without claiming that every provider is online. **Ingress listener** is the trusted proxied management surface. **Direct listener** is any separately exposed callback or standalone endpoint and never inherits Ingress identity. + +## 4. Goals, non-goals, and fixed invariants + +### 4.1 Goals + +1. Provide a predictable install/start/stop/update/backup/restore lifecycle on supported Home Assistant systems. +2. Keep the domain/application core runnable and testable without Home Assistant. +3. Make lifecycle and recovery state understandable without reading raw logs. +4. Fail closed before provider writes when configuration, migration, storage, or recovery is unsafe. + +### 4.2 Non-goals + +1. Choosing the backend, frontend, database, init system, or image-build stack. +2. Defining provider authorization; that belongs to the provider-connections SDD. +3. Shipping standalone mode in the first release. +4. Exposing playback/media-player entities. + +### 4.3 Fixed invariants + +1. Ingress is the only management UI/API authentication boundary in App mode. +2. A direct listener cannot trust Ingress headers or expose management routes anonymously. +3. Domain and application modules cannot import Home Assistant/Supervisor libraries. +4. No startup, restore, or downgrade path may perform provider writes before recovery is complete. +5. `/data` is the only ordinary persistent runtime location; secrets and key material follow the separate accepted secret design. +6. Privileged mode, Docker socket, SSH, host network, and arbitrary host/config/media mounts are prohibited without a superseding ADR. + +## 5. Product journey + +| Stage | Proposed behavior | User/operator effect | +| --- | --- | --- | +| Install | Home Assistant validates repository/App metadata and pulls an immutable versioned image | Unsupported architecture/version fails before start | +| Start | App validates bounded options, storage, schema, and recovery state | UI reports `starting`, `migrating`, or `recovering` truthfully | +| Ready | Ingress routes relative paths to the authenticated management surface | Administrator can use the complete UI without a root-path assumption | +| Degraded | Service remains usable while a provider/integration is unavailable | Local history/diagnostics remain available; affected actions are disabled | +| Backup | Writes are quiesced or a transactionally consistent snapshot is taken | Backup explains token/key and active-operation consequences | +| Upgrade/restore | Migrations and job recovery complete before new writes | Failure blocks readiness and preserves recoverable prior state | +| Stop | New work stops, leases/checkpoints are made recoverable, shutdown is bounded | Supervisor can restart without hidden in-memory authority | + +```mermaid +stateDiagram-v2 + [*] --> Starting + Starting --> Migrating + Migrating --> Recovering + Recovering --> Ready + Ready --> Degraded + Degraded --> Ready + Starting --> Blocked + Migrating --> Blocked + Recovering --> Blocked + Ready --> Stopping + Degraded --> Stopping + Blocked --> Stopping + Stopping --> [*] +``` + +Text equivalent: startup validates configuration, migrates, and recovers before readiness; recoverable external faults may degrade a ready service; unsafe local state blocks readiness; all running states stop through bounded graceful shutdown. + +## 6. Functional behavior and state model + +### 6.1 Happy path + +1. Supervisor starts the immutable image with validated App options and persistent `/data`. +2. The composition root loads configuration without starting workers. +3. Storage opens, schema compatibility is checked, forward migrations run, and interrupted operations are recovered. +4. Readiness becomes true; only then may the scheduler lease work. +5. An authenticated Home Assistant administrator opens the Ingress panel and sees service/provider status. +6. Graceful shutdown stops admissions, checkpoints/abandons leases safely, flushes durable state, and exits within the platform budget. + +### 6.2 Alternative and boundary paths + +- A provider outage degrades only that connection/capability; it does not make local history unavailable. +- Home Assistant Core or the optional integration may be unavailable while the App continues safe provider work. +- A migration or restore incompatibility keeps the service `blocked`; no fresh empty database replaces user-authored state. +- An unsupported downgrade is detected before workers start. +- A callback listener, if selected by the authorization SDD, exposes only its bounded callback routes. + +### 6.3 State machine + +| State | Entered when | User-visible meaning | Allowed next states | Recovery/owner | +| --- | --- | --- | --- | --- | +| `starting` | process and configuration initialization | Symphonia is not ready | `migrating`, `recovering`, `blocked`, `stopping` | automatic or operator fixes configuration | +| `migrating` | schema change is required | Existing data is being upgraded; no writes | `recovering`, `blocked`, `stopping` | restore/upgrade guidance on failure | +| `recovering` | leases/jobs/backup state need reconciliation | Durable state is being made safe | `ready`, `blocked`, `stopping` | automatic bounded recovery | +| `ready` | local invariants pass | UI and eligible operations are available | `degraded`, `stopping` | none | +| `degraded` | external/optional dependency unavailable | Local service works with named limitations | `ready`, `blocked`, `stopping` | retry or user action by cause | +| `blocked` | local safety/configuration/migration failure | No provider writes; action required | `stopping` or restart after correction | administrator | +| `stopping` | Supervisor stop/update | No new work; safe checkpointing | process exit | automatic bounded shutdown | + +## 7. Configuration contract + +No final App option names are accepted yet. The first implementation RFC should minimize configuration and propose at least: + +| Input | Type | Recommended default | Allowed values/range | Scope/persistence | +| --- | --- | --- | --- | --- | +| Log verbosity | enum | `info` | `error`, `warning`, `info`, time-bounded `debug` | App option; reread on restart | +| Display timezone | IANA zone | Home Assistant-configured zone when safely available, otherwise explicit setup value | Valid IANA identifier | persisted; operations store UTC instants | +| Direct callback exposure | bounded mode | `disabled` until the authorization SDD accepts it | `disabled` or one accepted callback-only profile | App option; restart required | + +Invalid or unknown values fail closed before workers start. Authentication, non-root execution, secret redaction, audit history, and mount/network restrictions are not configurable. + +## 8. Clean Architecture design + +### 8.1 Responsibilities and dependency direction + +| Boundary | Owns | Must not own/import | +| --- | --- | --- | +| Domain/application | Music behavior, use cases, semantic ports | Home Assistant, Supervisor, Ingress, container APIs | +| App platform adapter | App options, `/data`, lifecycle, Supervisor-safe metadata | Music identity/copy policy | +| HTTP/Ingress presentation | Base-path routing, trusted identity adaptation, status views | Provider/domain decisions | +| Persistence/job adapters | Schema, transactions, leases, recovery | Presentation or HA identity | +| Composition | Concrete profile and startup/shutdown order | New business rules | +| Companion integration | Versioned local client and native projections | Database/token-store access or duplicate orchestration | + +### 8.2 Contracts, durable state, and trust boundaries + +- Lifecycle use cases: validate/start, migrate, recover, ready, quiesce/backup, graceful stop. +- Ports: configuration, persistence health, migration ledger, operation recovery, clock, diagnostics, platform metadata. +- Durable state: schema version/ledger, operations, leases, application version/config; exact store is open. +- Trust boundaries: Ingress headers are trusted only on the verified listener; App options and restore files are untrusted configuration; direct network input is untrusted. +- Idempotency: startup/recovery can repeat after interruption without duplicating migrations or writes. + +### 8.3 Executable architecture constraints + +- Dependency tests reject Home Assistant/Supervisor imports in domain/application modules. +- Artifact tests parse App metadata and enumerate ports, mounts, privileges, architectures, backup settings, image version, and Ingress configuration. +- Route tests prove Ingress base-path correctness and direct-listener isolation. +- Migration tests use per-release fixtures and fail without replacing the prior database. + +## 9. UI/UX and content contract + +### 9.1 Information hierarchy + +The App panel starts with service status, completed startup fact, next transition, required action, retained state, and a link to bounded diagnostics. + +### 9.2 Representative states + +```text +Starting Symphonia +Status: Recovering 2 interrupted operations. No provider writes are running. +Next: the library opens after recovery completes. No action is required. +``` + +```text +Symphonia needs attention +Status: Startup blocked before provider access. +Cause: the restored database requires a newer Symphonia version. +Action: reinstall version 1.4.0 or restore a compatible backup. +Retained state: the restored data has not been modified. +``` + +```text +Symphonia is ready with limitations +Local library and history are available. Home Assistant native entities are offline. +Next: the App will reconnect automatically; provider operations continue safely. +``` + +### 9.3 Accessibility and localization + +Status is textual and announced when it changes; focus remains stable during polling/push updates. Ingress navigation works at narrow widths and never depends on color. Initial locale follows Home Assistant where available with English fallback; exact supported locales require the UI SDD/toolchain. + +## 10. Failure, recovery, and cleanup + +| Failure/partial state | User impact | Retained facts | Automatic retry | Required action | Cleanup | +| --- | --- | --- | --- | --- | --- | +| Invalid App option | no readiness | durable data untouched | no | correct option | none | +| Migration interruption | no writes | prior DB plus migration evidence | bounded resume only if proven safe | restore/upgrade if blocked | never create fresh DB silently | +| Stale worker lease | delayed operation | job/checkpoint/audit | after lease expiry | none unless repeated | recovered worker records takeover | +| Backup cannot quiesce | backup not declared valid | operations remain authoritative | bounded retry | retry later | no partial backup claim | +| Ingress unavailable | UI unavailable | service/jobs remain durable | platform recovery | inspect HA | no provider rollback | +| Direct listener misconfiguration | authorization disabled | management surface remains closed | no | correct network/callback setup | close listener | + +## 11. Security, permissions, and privacy + +1. The Ingress management surface is admin-only for the MVP. +2. Direct listeners use independent trust rules and cannot access management handlers by path rewriting. +3. The image runs non-root where supported and has no privileged/broad mounts or host networking. +4. App options, logs, diagnostics, backups, and image layers contain no plaintext provider grants. +5. Restore and diagnostic inputs are size-bounded, schema-validated, and never interpreted as code, paths outside owned storage, or arbitrary URLs. + +## 12. Observability and operational UX + +- Liveness and readiness are separate and include reason codes. +- Startup/migration/recovery logs carry an instance/startup correlation ID without secrets. +- Metrics cover startup phase duration, migration/recovery result, stale leases, shutdown duration, and readiness transitions. +- Diagnostics list versions, schema state, queue summaries, listeners, and sanitized configuration—not provider payloads or tokens. +- Stable ready state produces no periodic notifications; only action-required or meaningful transitions surface prominently. + +## 13. Compatibility, migration, rollout, and rollback + +- Every release declares compatible schema range, supported HA versions/architectures, and downgrade behavior. +- Forward migrations are ledgered and tested from every supported release path, including beta/stable divergence. +- Upgrade/restore takes a consistent backup or refuses when it cannot establish one. +- Rollback uses documented compatible image/data rules; an incompatible old image fails closed. +- Standalone packaging, if later released, shares application code and migration format but has its own authentication/network profile. + +## 14. Testing strategy and numeric budget + +Minimum **58 distinct cases**: + +| Area | Minimum distinct cases | Behaviors/risks covered | +| --- | ---: | --- | +| Configuration/lifecycle policy | 10 | valid/invalid/unknown options, phase transitions, shutdown admissions | +| State/recovery/idempotency | 14 | interrupted migrations, stale leases, repeated startup, quiesce, cancellation | +| App/Ingress/artifact contracts | 12 | metadata, architectures, base paths, headers, ports, mounts, non-root | +| HTTP/UI/accessibility/sanitization | 8 | primary states, focus/announcement, narrow layout, hostile diagnostics | +| Integration/security/migration | 14 | install/restart/backup/restore/upgrade/downgrade, direct-port isolation | +| **Total** | **58** | No double counting | + +All ordinary tests use fake Supervisor/Ingress and deterministic storage/clock fixtures. A disposable HA OS/Supervised-compatible smoke environment covers install, Ingress, restart, backup/restore, and upgrade. Manual evidence covers desktop/mobile and light/dark startup/error views. + +## 15. Documentation and discoverability + +| Audience | Artifact | Required content | Validation/navigation | +| --- | --- | --- | --- | +| User | App install/first-run guide | repository install, Ingress entry, normal status | link and smoke path | +| Setup owner | App configuration reference | defaults, network/callback, supported systems | metadata fixture | +| Operator | backup/restore/upgrade/recovery guide | state meanings, blocked recovery, rollback | recovery matrix | +| Contributor | runtime architecture guide | lifecycle, boundaries, artifact checks | dependency/artifact tests | + +## 16. Acceptance scenarios + +1. Given a supported fresh install, startup reaches `ready` only after configuration, migration, and recovery pass; the Ingress UI works under a non-root base path. +2. Given invalid configuration, readiness remains false, no worker starts, and the UI/log states one correction. +3. Given restart during a running operation, the expired lease is recovered from durable state without duplicating a provider write. +4. Given a migration failure, the prior database remains recoverable and no fresh rescan replaces user-authored state. +5. Given a backup request with active work, the App produces a transactionally consistent backup or refuses it explicitly. +6. Given any direct callback port, management routes and Ingress identity headers are unusable on that listener. +7. Given Home Assistant integration unavailability, local App history remains readable and eligible provider jobs are not corrupted. +8. Given an unsupported downgrade, the App blocks before writes and points to compatible restore/upgrade guidance. +9. Given hostile restored/config/diagnostic values, no path escape, arbitrary URL, secret output, or code execution occurs. +10. Given Supervisor stop, new admissions stop and shutdown leaves every lease recoverable within the bounded time. + +## 17. Requirements traceability + +| Requirement | Owner | Test or evidence | Documentation | +| --- | --- | --- | --- | +| `SYM-HA-001`–`SYM-HA-003` | App composition/platform adapter | artifact + disposable HA smoke | install/upgrade/backup guides | +| `SYM-HA-004` | dependency rule | static architecture test | contributor architecture | +| `SYM-SEC-008`–`SYM-SEC-011` | listeners/image composition | route/port/privilege abuse tests | security/setup guide | +| `SYM-ARCH-003`, `SYM-ARCH-006`, `SYM-ARCH-007` | migration/backup use cases | release-fixture migration + restore tests | recovery guide | +| `SYM-DEP-001`–`SYM-DEP-010` | release artifact | metadata/package/smoke matrix | release support policy | +| `SYM-TEST-009`, `SYM-TEST-012`, `SYM-TEST-013` | verification tooling | CI gates | contributor testing guide | + +## 18. Implementation sequence + +1. Complete storage/recovery and secret/backup RFCs; accept platform matrix. +2. Add App metadata/image contract fixtures and architecture boundaries. +3. Implement lifecycle/migration/recovery policies with deterministic tests. +4. Implement platform/persistence adapters and Ingress presentation. +5. Add status UX, diagnostics, accessibility, and documentation. +6. Run disposable HA install/restart/backup/restore/upgrade evidence before stable release. + +## 19. Definition of Done + +- [ ] Blocking runtime/storage/secret/platform decisions are accepted. +- [ ] Every mapped requirement has automated or explicit platform evidence. +- [ ] At least 58 distinct cases and repository coverage gates pass. +- [ ] Domain/application dependency and App artifact boundaries are enforced. +- [ ] Startup, recovery, backup, restore, upgrade, downgrade, and shutdown are proven. +- [ ] Direct listeners cannot bypass Ingress; image privileges/mounts are minimal. +- [ ] Primary lifecycle views pass accessibility, localization fallback, and sanitization review. +- [ ] User/setup/operator/contributor documentation is complete. +- [ ] Catalog implementation evidence is current and explicit owner approval exists. + +## 20. References and decisions + +- Primary sources: Home Assistant App/Ingress/security documentation linked from [provider research](../docs/providers/provider-research.md#home-assistant-platform). +- Related SDDs: [provider authorization](provider-connections-and-authorization.md), [durable operations](durable-operations-and-recovery.md). +- Accepted: App is the primary deployment boundary; core remains HA-independent. +- Rejected: all logic in a custom integration; unauthenticated management port; fresh database fallback after migration failure. +- Follow-up: standalone release and companion integration native surface. diff --git a/specs/library-import-and-provider-projections.md b/specs/library-import-and-provider-projections.md new file mode 100644 index 0000000..c87077c --- /dev/null +++ b/specs/library-import-and-provider-projections.md @@ -0,0 +1,327 @@ +# Library import and provider projections + +- Status: Draft +- Date: 2026-09-20 +- Catalog capability ID: `library-import-and-provider-projections` +- Owners: Symphonia maintainers +- Scope: import approved provider collections and ordered playlists into complete, provenance-rich provider projections without confusing them with provider-independent recordings. +- Related requirements: `SYM-PROD-003`, `SYM-LIB-001`–`SYM-LIB-006`, `SYM-PROV-004`–`SYM-PROV-014`, `SYM-PROV-018`, `SYM-ARCH-004`–`SYM-ARCH-005` +- Related decisions/research: [domain model](../docs/domain/domain-model.md), [provider specification](../docs/providers/provider-specification.md), [provider research](../docs/providers/provider-research.md), `RG-001`, `OQ-007`, `OQ-008` +- Required review gates: product UX, domain, architecture, provider feasibility/policy, testing, documentation, privacy/operations +- Open decisions blocking readiness: proven collection semantics/completeness per MVP provider; retention/export policy; representative library sizes/import targets; provider-specific refresh/deletion obligations + +## 1. Executive summary + +An import is a durable observation of one provider connection, not a merge operation and not evidence that provider items are the same recording. It reads approved collections page by page, retains original identity/order/duplicates/availability/provenance, stages normalized projections, and marks previously observed items missing only after a complete authoritative result. + +The safe default is full, read-only import after a connection succeeds, with no provider mutation and no deletion inferred from a partial/truncated run. + +```text +Connected account -> plan approved collections -> page/batch reads +-> normalize provider projections -> validate completeness +-> atomically publish complete observations or retain partial evidence +-> schedule identity resolution separately +``` + +## 2. Problem, current behavior, and evidence + +### 2.1 Problem + +Provider libraries differ in collection meaning, pagination, unavailable/local/non-music entries, duplicate/order semantics, freshness, quotas, and data-retention rules. A naive import can silently truncate, erase missing items after a failed page, merge unrelated accounts, or store provider data indefinitely in violation of policy. + +### 2.2 Current behavior + +There is no importer or persistence implementation. The domain model and provider contract establish the required representation and completeness semantics; provider-specific facts still require live spikes. + +### 2.3 Evidence and unknowns + +- Shared model: [provider track, playlist, snapshot, and import invariants](../docs/domain/domain-model.md). +- Adapter contract: [pagination, unknown media, completeness, freshness](../docs/providers/provider-specification.md). +- Official API evidence and policy constraints: [provider research](../docs/providers/provider-research.md). +- Comparative warning: existing YT Music implementations may cap dynamic playlists or skip unavailable entries; [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md#youtube-music). +- Unknowns: provider collection parity, target sizes, refresh cadence, raw payload retention, incremental cursor reliability. + +## 3. Actors, surfaces, and terminology + +| Actor | Goal | Entry point | Visible surfaces | +| --- | --- | --- | --- | +| Symphonia user | See current provider playlists/library with honest freshness | First import, refresh action, connection page | Library/playlist views, import progress/history | +| Provider adapter | Translate provider pages/items into normalized observations | Import use case | Contract outcomes and completeness evidence | +| Operator | Diagnose slow, partial, stale, or policy-blocked imports | Operation detail/diagnostics | Counts, pages, quota, reason codes, no raw secrets | +| Resolution engine | Consume provider-track projections after publication | Import completion event | Candidate/resolution queue | + +**Provider projection** is the latest normalized view of an external object for a connection/context. **Import session** is one durable attempt over a declared set of collections. **Complete collection result** means every provider page/range required by the declared contract succeeded without unknown gaps. **Playlist snapshot** is an immutable ordered observation; it is not overwritten in place while an operation depends on it. + +## 4. Goals, non-goals, and fixed invariants + +### 4.1 Goals + +1. Import saved/liked tracks and owned/followed playlists where an approved API exposes them. +2. Preserve provider identity, media kind, order, duplicates, unavailable entries, source timestamps, freshness, and policy metadata. +3. Make partial/truncated/stale results visible and safe. +4. Publish a stable input for resolution and copy planning. + +### 4.2 Non-goals + +1. Deciding recording identity during import. +2. Writing to provider libraries or playlists. +3. Treating generic YouTube videos as a complete YouTube Music catalog. +4. Keeping unrestricted raw API payloads forever. +5. Implementing continuous synchronization/change attribution. + +### 4.3 Fixed invariants + +1. Provider object IDs never become recording IDs. +2. External identity includes media type and any required instance/connection/catalog namespace. +3. Playlist order, duplicate occurrences, unknown media, and unavailable/deleted entries survive normalization. +4. Only a complete authoritative collection result may mark prior items no longer observed. +5. Raw provider data is separable/deletable from user-authored mappings and audit history. +6. Import failure cannot erase the last known complete projection. + +## 5. Product journey + +| Stage | Proposed behavior | User/operator effect | +| --- | --- | --- | +| Plan | Capture connection, adapter version, capability evidence, collections, and request/quota estimate | User sees what will be read and known limitations | +| Queue | Create one durable import operation with per-collection steps | Restart cannot lose intent/progress | +| Read | Page/batch through provider data with bounded timeouts/rate budget | Progress shows observed pages/items and waits | +| Normalize | Retain originals/provenance while mapping known fields/media states | Bad/unknown items become explicit issues, not silent drops | +| Validate | Determine completeness independently for each collection/playlist | One failed playlist need not invalidate every completed collection | +| Publish | Atomically publish complete collection observations and immutable playlist snapshots | Readers never see a half-replaced collection | +| Follow up | Schedule resolution for changed provider tracks | Matching remains separately auditable | + +```mermaid +flowchart LR + A[Import plan] --> B[Durable collection steps] + B --> C[Provider pages/batches] + C --> D[Normalized staged observations] + D --> E{Complete for this collection?} + E -- yes --> F[Publish projection and snapshots] + E -- no --> G[Retain prior projection and partial evidence] + F --> H[Schedule resolution] +``` + +Text equivalent: each planned collection is read into staged normalized observations; complete results replace the current projection atomically, while incomplete results preserve prior truth and record partial evidence; published changes then feed resolution. + +## 6. Functional behavior and state model + +### 6.1 Happy path + +1. A connected, capability-probed account creates an import plan for supported collections. +2. The use case snapshots adapter/version/capability/policy inputs and creates durable steps. +3. The adapter returns bounded pages with opaque continuation/revision evidence and explicit completeness. +4. Each object is validated and normalized with original values/provenance and observation/refresh metadata. +5. Each playlist produces an immutable snapshot with ordered occurrence records, including unavailable/unknown entries. +6. A complete collection transaction publishes the new projection, marks legitimately missing prior objects unavailable/no-longer-observed, and records counts/issues. +7. Changed/new provider tracks are offered to the resolution capability. + +### 6.2 Alternative and boundary paths + +- Empty complete collection is valid and may mark prior observations missing; empty incomplete result cannot. +- One malformed item is an item issue and affects completeness according to the provider contract; it is never silently discarded. +- Rate-limit wait persists and resumes without repeating published pages unsafely. +- Provider revision change during import either restarts/bounds the affected collection or publishes a snapshot explicitly labeled non-atomic/degraded; policy must be provider-specific and visible. +- Import cancellation preserves last complete projection and the partial session audit. +- A provider that cannot paginate or proves only a capped list reports `incomplete/degraded`, not success. + +### 6.3 Import session and collection states + +| State | Entered when | User-visible meaning | Allowed next states | Recovery/owner | +| --- | --- | --- | --- | --- | +| `planned` | collections/capabilities snapshotted | Ready to import | `queued`, `cancelled` | user/system | +| `queued` | durable work admitted | Waiting for worker/budget | `running`, `cancelled` | scheduler | +| `running` | collection lease held | Reading/normalizing provider data | `waiting_rate_limit`, `partial`, `failed`, `succeeded`, `cancelled` | worker | +| `waiting_rate_limit` | provider budget exhausted | Progress retained; resume scheduled | `running`, `failed`, `cancelled` | automatic | +| `partial` | some collections/items published or observed but operation incomplete | Last complete truth plus named partial results | terminal or explicit retry/new import | user/system | +| `failed` | no safe continuation within budget | Last complete projection retained | new import | user/system | +| `succeeded` | every requested collection reached declared terminal result | Current projection/freshness updated | terminal | none | +| `cancelled` | cooperative cancellation completed | No further reads; published complete steps remain | terminal | none | + +Collection outcomes separately record `complete`, `complete_with_item_issues`, `incomplete`, `unsupported`, or `not_requested`. + +## 7. Configuration contract + +Exact names remain open, but behavior is bounded: + +| Input | Type | Recommended default | Allowed values/range | Scope/persistence | +| --- | --- | --- | --- | --- | +| Collections | set | all approved MVP collections supported by the connection | adapter-declared stable collection IDs | snapshotted per import | +| Initial trigger | enum | immediately after verified connection | immediate or explicit manual start | connection preference | +| Periodic refresh | duration/off | unresolved pending provider policy/scale evidence | provider-safe bounded range | connection; next run snapshots | +| Request budget | bounded integer/profile | conservative adapter default | within adapter hard limits | snapshotted per run | +| Raw payload retention | policy | minimum needed for audit/debug under provider terms | accepted retention profiles only | install/connection policy | + +Users cannot configure “ignore pagination errors,” “drop unavailable items,” “assume capped list complete,” or “delete prior state after partial import.” + +## 8. Clean Architecture design + +### 8.1 Responsibilities and dependency direction + +| Boundary | Owns | Must not own/import | +| --- | --- | --- | +| Domain | External identity, provider projection, playlist snapshot/entry, availability, completeness invariants | Provider DTOs/database | +| Application | Plan/run/publish import, collection transaction and follow-up scheduling | Concrete paging/DB/job framework | +| Provider adapter | Page/batch calls, DTO normalization inputs, cursor/revision/error translation | Cross-provider deletion/match policy | +| Persistence adapter | Staging, atomic publication, historical references, retention | Provider API behavior | +| Presentation | Progress/freshness/issues/library projections | Inferring completeness or dropping items | + +### 8.2 Contracts, durable state, and trust boundaries + +- Pure decisions: external identity namespace, collection completeness, publish eligibility, missing-item transition, retention classification. +- Use cases: plan import, execute collection page, publish collection, cancel/retry, inspect session. +- Ports: provider reader, projection/snapshot repository, durable operation scheduler, clock, policy/retention, audit. +- Durable state: import plan/session, collection/page checkpoints, staged observation keys, complete projections, immutable snapshots, issues/freshness. +- Concurrency: at most one publishing import per connection/collection; overlapping read attempts fence publication by plan/session version. +- Untrusted inputs: every provider field, cursor, count, URL, timestamp, media type, and declared total. + +### 8.3 Executable architecture constraints + +- Shared contract suite proves pagination, completeness, identity namespace, unknown/unavailable preservation, timeouts, cancellation, and error mapping. +- Persistence tests prove incomplete sessions cannot mark prior objects missing and readers cannot observe half-publication. +- Dependency tests keep provider DTOs/SDKs outside domain/application and raw payloads outside general API/UI models. +- Policy tests enforce refresh/deletion metadata and provider-specific retention. + +## 9. UI/UX and content contract + +### 9.1 Information hierarchy + +Library/connection pages show current projection freshness and last complete import separately from the latest attempt. Progress is based on persisted counts and collection states, not transient events. + +### 9.2 Representative states + +```text +Importing Spotify +Status: Reading playlists · 18 playlists and 624 entries observed. +Completed: saved tracks. +Next: playlist pages continue after the provider rate-limit window. No action required. +Current library still reflects the complete import from 09:14. +``` + +```text +Import partially completed +Saved tracks and 17 playlists were updated. “Road trip” could not be read after page 3. +Action: Retry that playlist or keep the previous complete snapshot. +Retained state: no entries were removed from “Road trip”. +``` + +```text +Import complete +1,842 saved tracks and 24 playlists were observed. 3 unavailable entries were preserved and need no immediate action. Resolution has been queued for 126 changed tracks. +``` + +### 9.3 Accessibility and localization + +Progress never depends on an indeterminate spinner alone; counts and collection names are textual. Tables retain a list/card alternative at narrow widths. Provider/user text is escaped, bidi-safe where possible, length-bounded, and never rendered as HTML/Markdown. Timestamps show configured locale/timezone with UTC retained internally. + +## 10. Failure, recovery, and cleanup + +| Failure/partial state | User impact | Retained facts | Automatic retry | Required action | Cleanup | +| --- | --- | --- | --- | --- | --- | +| Page timeout before response | import delayed | checkpoint before page | bounded backoff | none unless budget exhausted | staged duplicates deduped by observation key | +| Malformed/unknown item | item issue; collection completeness explicit | raw-minimal/sanitized evidence and position | no blind item loop | adapter fix or user review | retention applies | +| Auth revoked | import waits for user | last complete projection/session checkpoint | no | reconnect | transient page state expires | +| Rate limit | import pauses | progress and retry time | provider-aware | none | no duplicate publication | +| Provider changes mid-read | atomicity degraded/collection restarted | revisions/pages/session | bounded provider policy | retry if exhausted | discard superseded staging | +| Cancellation | no further reads | published complete steps and audit | no | start new import | remove orphan staging after safe horizon | +| Partial/truncated list | freshness not advanced as complete | last complete projection plus partial evidence | bounded retry | user/provider fix | never mass-delete | + +## 11. Security, permissions, and privacy + +1. Imports use minimum read scopes and never acquire write capability merely for convenience. +2. Provider IDs/URLs/names/cursors are opaque data and cannot drive filesystem paths, arbitrary requests, redirects, or code loading. +3. Raw payload storage is minimized, encrypted/classified where required, expiry-tagged, and independently deletable. +4. Logs/metrics avoid full playlist contents and stable provider IDs unless an accepted privacy policy permits bounded correlation. +5. Export/diagnostic views are bounded, previewable, sanitized, and omit tokens/auth headers. + +## 12. Observability and operational UX + +- Import operations expose collection/page/item counts, completeness, prior/current freshness, wait/retry, quota where known, adapter version, and issue categories. +- Metrics cover duration, pages/items, incomplete results, stale age, rate limits, normalization issues, and publication transactions. +- Correlation identifies operation/run/collection/page/connection/adapter without provider payload dumps. +- A waiting rate limit is not a failed import; a partial latest attempt does not overwrite the timestamp of the last complete result. +- Stable periodic success produces history/metrics but no notification unless the user requested it. + +## 13. Compatibility, migration, rollout, and rollback + +- Projection schemas and raw-normalizer versions are explicit; reparsing never erases originals still allowed by retention policy. +- External identity changes require a migration that detects collisions and preserves historical references. +- Adapter upgrades run fixture contract tests before publication and can invalidate/refresh projections without silently remapping recordings. +- Rollback reads only schema versions it declares compatible; otherwise App readiness blocks. +- Initial rollout imports one connection/collection set with write capabilities disabled. + +## 14. Testing strategy and numeric budget + +Minimum **72 distinct cases**: + +| Area | Minimum distinct cases | Behaviors/risks covered | +| --- | ---: | --- | +| Domain/completeness/identity policy | 16 | empty/partial/complete, namespaces, order, duplicates, availability, freshness | +| State/application/idempotency/races | 16 | pagination checkpoints, overlap/fencing, cancellation, resume, atomic publication | +| Provider/persistence contracts | 18 | pages/cursors/revisions, malformed data, caps, rate/auth/errors, staging/transactions | +| HTTP/UI/accessibility/sanitization | 8 | running/waiting/partial/failed/complete, hostile metadata, timestamps, narrow views | +| Integration/security/migration | 14 | large fixtures, restart, retention cleanup, schema/adapter changes, secret/path abuse | +| **Total** | **72** | No double counting | + +Fixtures include zero/one/boundary/multi-page collections, duplicate playlist occurrences, unavailable/deleted/local/non-music/unknown items, colliding IDs across types/instances, revision change mid-read, and provider-declared totals that lie. Live smoke is opt-in and verifies only dated documented behavior with dedicated data. + +## 15. Documentation and discoverability + +| Audience | Artifact | Required content | Validation/navigation | +| --- | --- | --- | --- | +| User | Library/import guide | collections, freshness, manual refresh, unavailable items | UI fixtures | +| Setup owner | Provider import limits | scopes, quota, schedule, retention | adapter capability matrix | +| Operator | Import troubleshooting | partial vs failed, retry, stale data, cleanup | error decision tree | +| Contributor | Import architecture/contract | completeness, staging, identity, fixtures | common contract suite | + +## 16. Acceptance scenarios + +1. Given a complete multi-page collection, every item is published once with provenance/freshness and the collection advances its complete timestamp. +2. Given an empty complete result, prior observations become no-longer-observed; given empty incomplete output, prior state is untouched. +3. Given duplicate playlist entries and unavailable/unknown media, the snapshot retains every occurrence/position and explicit state. +4. Given a timeout/rate limit/restart after page N, resume uses durable checkpoints and cannot publish duplicate or half-replaced state. +5. Given a malformed item or capped/non-pageable provider list, the UI reports explicit item/completeness issues rather than silently claiming success. +6. Given overlapping imports, only the non-superseded fenced session may publish a collection. +7. Given cancellation, no new pages run, complete published steps remain true, incomplete staging cannot delete prior projections, and cleanup is bounded. +8. Given provider disappearance, user-authored resolution/audit data remains while provider payload retention transitions independently. +9. Given hostile provider IDs/URLs/names/cursors, no path/egress/render/log injection or secret disclosure occurs. +10. Given a completed import, only changed/new provider tracks are scheduled for separate resolution with no identity decision made by import. + +## 17. Requirements traceability + +| Requirement | Owner | Test or evidence | Documentation | +| --- | --- | --- | --- | +| `SYM-LIB-001`–`SYM-LIB-006` | domain/import use cases/presentation | projection/snapshot/application fixtures | library/import guide | +| `SYM-PROV-004`–`SYM-PROV-008` | provider contract | shared pagination/media/error suite | provider adapter guide | +| `SYM-PROV-011`–`SYM-PROV-014` | adapter/scheduler/retention | rate/live-isolation/policy tests | provider limits/privacy guide | +| `SYM-PROV-018` | identity policy | cross-type/instance collision fixtures | domain architecture | +| `SYM-ARCH-004`, `SYM-ARCH-005` | persistence/retention | deletion/history/migration tests | retention/recovery guide | +| `SYM-TEST-003`, `SYM-TEST-006`, `SYM-TEST-010`, `SYM-TEST-011` | verification tooling | release gates | contributor testing guide | + +## 18. Implementation sequence + +1. Complete provider feasibility, retention, scale, and collection-scope decisions. +2. Define projection/snapshot/import contracts and failing common adapter tests. +3. Implement pure identity/completeness/publication policy and tests. +4. Implement application use cases with fake provider/persistence/job ports. +5. Implement persistence staging/publication and one provider read adapter. +6. Add progress/library UX, retention cleanup, docs, migration and bounded live evidence. + +## 19. Definition of Done + +- [ ] Collection semantics, retention, scale, and provider-policy blockers are resolved. +- [ ] Every requirement and state maps to acceptance and deterministic tests. +- [ ] At least 72 distinct cases and common adapter/persistence gates pass. +- [ ] Partial/truncated imports cannot erase or masquerade as complete state. +- [ ] Identity namespace, order, duplicates, unavailable/unknown media, and provenance are preserved. +- [ ] Raw payload retention/deletion, diagnostics, and privacy are implemented/documented. +- [ ] Running/waiting/partial/failed/completed UX passes accessibility/sanitization review. +- [ ] Live smoke remains opt-in and catalog evidence is current. +- [ ] Explicit owner approval to implement exists. + +## 20. References and decisions + +- Primary sources: [provider research](../docs/providers/provider-research.md) and provider links therein. +- Related SDDs: [connections](provider-connections-and-authorization.md), [identity resolution](recording-identity-resolution.md), [durable operations](durable-operations-and-recovery.md). +- Accepted: provider projections remain distinct from recordings; only complete results can infer missing items. +- Rejected: silent truncation, skipping unavailable entries, delete-on-partial, matching during import. +- Follow-up: incremental imports and persistent sync after official evidence and baseline semantics exist. diff --git a/specs/one-time-playlist-copy.md b/specs/one-time-playlist-copy.md new file mode 100644 index 0000000..72a8154 --- /dev/null +++ b/specs/one-time-playlist-copy.md @@ -0,0 +1,304 @@ +# One-time playlist copy + +- Status: Draft +- Date: 2026-09-20 +- Catalog capability ID: `one-time-playlist-copy` +- Owners: Symphonia maintainers +- Scope: preview and execute a finite playlist copy using an immutable plan, explicit non-ready policy, ordered writes, reconciliation, and item-level outcomes. +- Related requirements: `SYM-PROD-002`, `SYM-PROD-004`–`SYM-PROD-006`, `SYM-PL-001`–`SYM-PL-009`, `SYM-PROV-010`, `SYM-PROV-019`, `SYM-ARCH-009`–`SYM-ARCH-010`, `SYM-JOB-003`–`SYM-JOB-005`, `SYM-TEST-004`, `SYM-TEST-006` +- Related decisions/research: [ADR 0002](../docs/decisions/0002-copy-and-sync-are-distinct.md), [copy domain model](../docs/domain/domain-model.md), [provider research](../docs/providers/provider-research.md), `OQ-003`, `RG-001` +- Required review gates: product UX, domain, architecture, provider feasibility, testing, documentation, security/operations +- Open decisions blocking readiness: default non-ready policy; proven target write behavior; target collision policy; reconciliation semantics for ambiguous provider writes + +## 1. Executive summary + +Symphonia shall copy one imported playlist to another provider as a finite, reviewable operation. Before any target mutation, it shall capture an immutable source snapshot, resolve provider identities, build a dry-run plan, disclose every non-ready or deliberately omitted item, and require explicit acceptance of that exact plan. + +The operation is intentionally not persistent synchronization. A later source change does not mutate the accepted plan or trigger another copy. Retrying an operation shall be idempotent and shall reconcile uncertain provider outcomes before issuing another write. + +## 2. Problem statement and evidence + +Users need to move a playlist without trusting an opaque best-effort process that may silently substitute tracks, reorder entries, lose duplicates, or create multiple target playlists after a restart. + +The horizontal product and provider specifications require dry runs, durable background work, explicit ambiguity, provider capability checks, and a distinction between canonical recording identity and provider-specific availability. + +Evidence sources: + +- [Product specification](../docs/product/product-specification.md) +- [Domain model](../docs/domain/domain-model.md) +- [Provider specification](../docs/providers/provider-specification.md) +- [System architecture](../docs/architecture/system-architecture.md) +- [Open questions](../docs/open-questions.md) + +## 3. Actors and authorization + +- A Home Assistant administrator configures provider connections and service-level limits. +- An authorized household user selects the source playlist, target provider, and allowed resolution decisions. +- Symphonia plans and executes the copy using only connections that the requesting user is authorized to use. +- Provider adapters expose target capabilities and perform provider-specific reads and writes. + +The exact Home Assistant identity-to-Symphonia-user mapping remains an architectural decision. Until it is resolved, implementations shall not assume that every Ingress visitor may use every stored provider credential. + +## 4. Goals, non-goals, and invariants + +### Goals + +- Produce a complete, stable dry-run plan before target mutation. +- Preserve source entry order and duplicate occurrences when supported. +- Distinguish ready, ambiguous, unmatched, unsupported, unavailable, and invalid entries while retaining resolution evidence and any deliberate omission separately. +- Execute the accepted plan durably and idempotently. +- Make partial success and recovery actions understandable. + +### Non-goals + +- Continuous or bidirectional synchronization. +- Silent fuzzy matching or automatic acceptance below a documented confidence threshold. +- Editing the source playlist. +- Updating an existing target playlist unless a future decision explicitly defines collision and ownership semantics. +- Guaranteeing that every source item exists on the target provider. + +### Invariants + +- No provider write occurs before the user accepts the exact plan digest. +- A plan is immutable after acceptance; changes produce a new plan. +- Every source occurrence has a terminal plan classification. +- Unresolved and ambiguous entries are never silently substituted. +- Unknown provider write outcomes are reconciled before retry. +- Cancellation stops future work but does not claim to undo confirmed provider writes. +- Audit data can relate every target write to the accepted plan and source snapshot. + +## 5. User journey and primary flow + +1. The user selects an imported source playlist and a connected target provider. +2. Symphonia captures the source projection version, entry order, provider capabilities, connection identity, and relevant resolution decisions. +3. Symphonia resolves every source occurrence to a target provider item or a disclosed non-ready classification. +4. Symphonia creates an immutable plan and calculates its digest. +5. The user reviews counts, target settings, ambiguities, unavailable items, and policy consequences. +6. The user resolves eligible ambiguities or excludes items explicitly. Each change creates a revised plan and digest. +7. The user accepts the final plan. +8. A durable operation creates the target playlist, adds entries in planned order, checkpoints progress, and reconciles uncertain results. +9. Symphonia presents a per-item result and an operation summary. + +If the source projection or relevant provider capability changes before acceptance, the plan becomes stale and must be regenerated. Changes after acceptance do not alter the operation. + +## 6. Functional contract and state model + +### Plan entry classifications + +Each source occurrence has exactly one classification required by `SYM-PL-003`: + +- `ready`: one target item is approved for writing; its resolution basis records whether it came from strong automatic evidence or a user decision. +- `ambiguous`: multiple plausible candidates require a decision. +- `unmatched`: no sufficiently supported candidate exists. +- `unsupported`: the provider cannot represent or add the item type. +- `unavailable`: the canonical recording is known but not playable/addable on the target. +- `invalid`: the source occurrence cannot participate because its imported representation is malformed or violates an invariant. + +Classifications apply per occurrence, not just per recording, so duplicates remain visible and ordered. A separate disposition records whether a non-ready occurrence is blocked or deliberately omitted under the accepted policy; omission does not erase or rename its classification. + +### Plan states + +- `draft`: still being calculated or edited. +- `ready_for_review`: calculation is complete and internally consistent. +- `blocked`: the selected policy and one or more non-ready entries prevent acceptance. +- `accepted`: the user accepted the exact digest. +- `stale`: pre-acceptance evidence or capabilities changed. +- `superseded`: another revision replaced this plan. + +### Operation states + +The operation uses the durable job vocabulary defined by `durable-operations-and-recovery`: `queued`, `running`, `waiting_rate_limit`, `waiting_user`, `retry_scheduled`, `succeeded`, `partial`, `failed`, or `cancelled`. + +### Proposed initial acceptance policy + +Until OQ-003 is decided, the proposed default is strict: a plan cannot be accepted while any occurrence is not `ready`. A future explicit best-effort policy may permit itemized omission of `ambiguous`, `unmatched`, `unavailable`, `unsupported`, or `invalid` occurrences after prominent disclosure. This proposal is not final product policy. + +### Target creation and collision policy + +The safe initial behavior is create-only with a provider-visible operation marker when the provider permits it. Symphonia shall not overwrite, clear, or append to a pre-existing playlist solely because its name matches. The final collision policy is a readiness blocker. + +## 7. Configuration contract + +| Setting | Scope | Default | Validation and notes | +|---|---|---|---| +| Target playlist name | Per plan | Source name | Must satisfy target provider rules; normalized value shown before acceptance. | +| Target visibility | Per plan | Private where supported | Available values come from provider capabilities. | +| Non-ready entry policy | Product policy | Strict, proposed | Final policy blocked by OQ-003. | +| Existing target behavior | Product policy | Create only, proposed | No implicit overwrite or append. | +| Batch size | Adapter/runtime | Provider-specific | Bounded by provider limits; not exposed as an arbitrary user tuning knob initially. | +| Retry policy | Adapter/runtime | Conservative | Must distinguish safe retry from unknown outcome. | + +Dry-run review, audit creation, plan digest validation, and reconciliation cannot be disabled. + +## 8. Architecture and boundaries + +### Domain layer + +Owns source snapshots, plan revisions, entry classifications, target intent, accepted digests, and copy outcomes. It contains no Home Assistant or provider SDK types. + +### Application layer + +Coordinates snapshot capture, identity lookup, plan calculation, acceptance, operation dispatch, checkpointing, and reconciliation. + +### Ports + +- Read a stable imported playlist projection. +- Query provider capabilities and connection authorization. +- Resolve a canonical recording to target candidates. +- Create and inspect a target playlist. +- Add ordered batches and reconcile their outcome. +- Persist plans, results, checkpoints, and audit events. + +### Adapters + +- Provider adapters implement capability discovery and writes. +- Persistence adapters store immutable plan revisions and operation state. +- The Home Assistant-facing adapter exposes the web UI and, later, narrowly scoped services if approved. + +Provider-specific workarounds shall remain behind ports and shall not alter domain invariants. + +## 9. UI, UX, and accessibility + +The review page shall show: + +- source playlist and captured version; +- target account, provider, normalized name, and visibility; +- total occurrences and counts for every classification; +- ordered entry details with evidence and alternative candidates; +- consequences of omitting non-ready entries under the selected policy; +- the fact that this is a one-time snapshot, not synchronization; +- the accepted plan digest in an advanced details view. + +Example summary: + +> 124 source entries: 116 ready, 2 ambiguous, 4 unavailable on the target, and 2 unsupported. Nothing has been written yet. + +Example blocked action: + +> Review 2 ambiguous entries before this strict plan can be accepted. + +Example partial result: + +> The target playlist was created with 113 confirmed entries. Three writes could not be confirmed. Symphonia will reconcile them before offering a retry. + +Status shall never rely only on color. Keyboard navigation, visible focus, semantic headings, accessible tables/lists, and screen-reader announcements for material state changes are required. + +## 10. Failure and recovery semantics + +| Failure | Required behavior | User recovery | +|---|---|---| +| Source changes before acceptance | Mark plan stale; issue no writes | Recalculate and review a new revision | +| Connection expires before execution | Pause without losing progress | Reauthorize, then resume | +| Target capability changes | Stop before incompatible writes; record evidence | Re-plan or choose another target | +| Target creation response is unknown | Search/reconcile using stored intent and marker; do not blindly create again | Wait for reconciliation or inspect candidates | +| Entry batch response is unknown | Reconcile target contents/checkpoint before retry | Resume only when safe | +| Rate limit | Enter `waiting_rate_limit` with next eligible time | Automatic bounded resume; cancellation remains available | +| Some items fail permanently | Finish as `partial` with per-item reasons | Create a new remediation plan for failed entries | +| Process or host restarts | Resume from durable checkpoint and lease rules | No manual action unless state becomes uncertain | +| User cancels | Stop scheduling further writes; preserve confirmed results | Review partial result; cancellation is not rollback | + +## 11. Security and privacy + +- Acceptance and execution require authorization for the source projection and target connection. +- UI, logs, events, and exports shall not expose access or refresh tokens. +- Stored target item identifiers and playlist contents are household data and follow the project retention policy. +- Audit records include actor, connection reference, source snapshot, plan digest, timestamps, and outcome, but not provider secrets. +- Provider text, artwork, and URLs are untrusted input and must be safely rendered. +- Any destructive future behavior such as replacement or clearing requires a separate threat review and explicit confirmation contract. + +## 12. Observability and supportability + +Each plan and operation has a non-secret correlation identifier. Structured events cover plan creation, classification totals, acceptance, dispatch, target reconciliation, batch checkpoints, waits, retries, cancellation, and terminal result. + +Metrics may include duration, plan size, classification counts, confirmation latency, retry counts, rate-limit waits, and terminal states. Labels shall not contain playlist names, track titles, user tokens, or unbounded provider identifiers. + +A redacted diagnostic export shall contain the plan version and digest, adapter versions, capability snapshot, state history, checkpoints, and categorized failures. + +## 13. Rollout, migration, and compatibility + +The capability remains unavailable until its dependencies are ready for implementation and the blocking product policies are decided. Initial rollout shall use synthetic provider adapters, then provider sandboxes or tightly controlled accounts before general availability. + +Persisted plans and operations require explicit schema versions. Migrations shall preserve accepted digests and audit history or fail safely before execution. An in-flight operation may resume only when the new version proves checkpoint compatibility. + +## 14. Numeric test budget + +Minimum planned automated tests: **94**. + +| Area | Minimum | +|---|---:| +| Plan calculation, ordering, duplicates, and classifications | 24 | +| State transitions, digest validation, idempotency, and reconciliation | 22 | +| Provider capability and write adapter contracts | 18 | +| UI states and accessibility | 12 | +| Integration, restart, migration, security, and redaction | 18 | + +Required fault injection includes timeouts before and after provider acceptance, duplicate delivery, process death at every write checkpoint, expired authorization, rate limits, capability drift, and ambiguous reconciliation. + +## 15. Documentation impact + +Implementation shall update: + +- user guidance for planning, review, execution, cancellation, and partial results; +- provider support tables with read/write and visibility capabilities; +- administrator guidance for credentials, storage, and diagnostics; +- privacy and retention documentation; +- the SDD catalog and traceability evidence. + +## 16. Acceptance criteria + +1. A user can select an imported playlist and authorized target connection. +2. Planning performs no target mutation. +3. Every source occurrence appears exactly once in the immutable plan, preserving order and duplicates. +4. The UI discloses every ambiguous, unmatched, unavailable, invalid, unsupported, and deliberately omitted occurrence. +5. Acceptance binds to a specific plan digest, source projection version, capabilities snapshot, and target intent. +6. A stale or modified plan cannot execute. +7. Execution survives restart without duplicating the target playlist or confirmed entries. +8. Unknown provider outcomes trigger reconciliation before retry. +9. Cancellation stops future work and accurately reports already confirmed writes. +10. Partial success has per-item explanations and a safe remediation path. +11. Logs and diagnostics contain no provider secrets. +12. The numeric test budget and required fault-injection scenarios pass. + +## 17. Requirement traceability + +| Requirement | Design location | Planned verification | +|---|---|---| +| `SYM-PL-001`–`SYM-PL-009` | Sections 5-11 | Planning, classification, acceptance, execution, collision, and recovery suites | +| `SYM-PROD-002`, `SYM-PROD-004`–`SYM-PROD-006` | Sections 6, 10, and 13 | Disclosure, attribution, state, error, and history tests | +| `SYM-PROV-010`, `SYM-PROV-019` | Sections 7-9 and 11 | Capability and reconciliation adapter contract suites | +| `SYM-ARCH-009`–`SYM-ARCH-010` | Sections 7 and 11 | Operation-aware retry and partial-result tests | +| `SYM-JOB-003`–`SYM-JOB-005` | Sections 7 and 11 | Checkpoint, restart, cancellation, and unknown-outcome tests | +| `SYM-TEST-004`, `SYM-TEST-006` | Section 15 | Required fault and playlist-edge-case release gates | + +## 18. Sequence sketch + +```text +User -> App: choose source and target +App -> Import store: capture stable source projection +App -> Identity service: resolve each occurrence +App -> Provider adapter: read capabilities and target availability +App -> Plan store: persist immutable revision + digest +App -> User: show dry run and blockers +User -> App: accept exact digest +App -> Job store: enqueue durable copy operation +Worker -> Provider adapter: create/reconcile target +Worker -> Provider adapter: add/reconcile ordered batches +Worker -> Job store: checkpoint every confirmed result +App -> User: show terminal and per-item outcomes +``` + +## 19. Definition of done + +- Blocking decisions are resolved and recorded in ADRs or the relevant horizontal specifications. +- Dependencies are at least `Ready for implementation`. +- Threat model and provider write contract reviews are complete. +- Tests meet the numeric budget and fault-injection requirements. +- Documentation and catalog evidence are current. +- Product owner explicitly approves implementation and later release. + +## 20. References + +- [SDD standard](README.md) +- [SDD catalog](CATALOG.md) +- [Recording identity resolution SDD](recording-identity-resolution.md) +- [Durable operations and recovery SDD](durable-operations-and-recovery.md) diff --git a/specs/provider-connections-and-authorization.md b/specs/provider-connections-and-authorization.md new file mode 100644 index 0000000..c7551b6 --- /dev/null +++ b/specs/provider-connections-and-authorization.md @@ -0,0 +1,348 @@ +# Provider connections and authorization + +- Status: Draft +- Date: 2026-09-20 +- Catalog capability ID: `provider-connections-and-authorization` +- Owners: Symphonia maintainers +- Scope: disclose provider risk, authorize one external account, protect and refresh its grant, probe effective capabilities, reauthorize, and disconnect safely. +- Related requirements: `SYM-ACC-002`–`SYM-ACC-004`, `SYM-ACC-006`, `SYM-PROV-002`–`SYM-PROV-003`, `SYM-PROV-008`–`SYM-PROV-009`, `SYM-PROV-015`–`SYM-PROV-020`, `SYM-SEC-001`–`SYM-SEC-010` +- Related decisions/research: [provider specification](../docs/providers/provider-specification.md), [official API research](../docs/providers/provider-research.md), [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md), `OQ-001`, `OQ-004`, `RG-001`, `RG-002` +- Required review gates: product UX, architecture, provider feasibility, testing, documentation, security/privacy +- Open decisions blocking readiness: direct App OAuth versus companion-integration authorization broker; per-provider registration/scopes/token lifecycle; unofficial YouTube Music MVP decision; secret key source and backup contract + +## 1. Executive summary + +Before any external account is connected, Symphonia explains whether the adapter uses an official or reverse-engineered contract, what credentials and dependencies it needs, what access it requests, and how reauthorization works. A successful connection identifies one immutable provider account, stores only encrypted grant material behind a secret reference, probes effective capabilities, and schedules import separately. + +The authorization boundary is not yet selected. The SDD keeps two candidates open: a direct App-owned flow or a minimal Home Assistant companion-integration broker. Neither may expose provider tokens to App options, URLs, logs, diagnostics, or ordinary UI state. + +```text +Choose adapter -> review access basis/scopes/limitations -> configure client credentials +-> authorize with single-use state -> verify immutable account -> store secret reference +-> probe effective capabilities -> connected/action-required -> disconnect and cleanup +``` + +## 2. Problem, current behavior, and evidence + +### 2.1 Problem + +Provider authentication differs in client registration, redirect rules, scopes, refresh, revocation, subscriptions, unofficial cookies, and object-level permissions. Hiding those differences creates unsafe secrets, surprising reauthorization, and workflows that fail only after partial writes. + +### 2.2 Current behavior + +No connection or credential implementation exists. Current facts are requirements and research only. Spotify is the strongest official MVP candidate; full YouTube Music access is not established through an official API; Apple Music is future research. + +### 2.3 Evidence and unknowns + +- Official provider/API facts: [provider research](../docs/providers/provider-research.md). +- Home Assistant OAuth/Application Credentials and existing music projects: [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md). +- Provider-independent contract: [provider specification](../docs/providers/provider-specification.md). +- Unknowns: OAuth ownership, callback reachability, secret encryption key, Google/YouTube product scope, exact scopes, token expiry/revocation behavior under test accounts. + +## 3. Actors, surfaces, and terminology + +| Actor | Goal | Entry point | Visible surfaces | +| --- | --- | --- | --- | +| Home Assistant administrator | Configure provider client credentials and approve account access | Ingress connection setup | Provider disclosure, credential form, authorization return, connection status | +| Provider account holder | Grant/revoke the requested provider access | Provider consent UI | Provider-hosted consent and account selection | +| Symphonia worker | Use a valid grant within declared capabilities | Secret/provider ports | Refresh, probe, import/write calls | +| Companion integration, if selected | Broker HA-native authorization only | HA config flow/local API | HA integration setup and broker status | + +**Adapter manifest** describes access basis, maturity, support, dependencies, and capability ceiling. **Connection attempt** is a short-lived authorization transaction. **Provider connection** is one verified immutable external account plus capability/health state and a credential reference. **Effective capability** is the intersection of adapter, connection, object, and live health constraints. + +## 4. Goals, non-goals, and fixed invariants + +### 4.1 Goals + +1. Make authorization prerequisites, risk, permissions, and reauthorization understandable before consent. +2. Support safe connection, refresh/reauthorize, capability probing, and disconnect. +3. Isolate provider-specific flows behind semantic ports without provider DTO leakage. +4. Keep connection health/action states durable and usable after restart. + +### 4.2 Non-goals + +1. Importing libraries or executing playlist writes. +2. Selecting a shared hosted OAuth application for every future deployment. +3. Treating browser-cookie export as an ordinary OAuth equivalent. +4. Sharing one provider grant across different Symphonia installations without an explicit provider contract. + +### 4.3 Fixed invariants + +1. Provider type alone is never account identity. +2. Unknown/degraded capability is never treated as supported. +3. Authorization callbacks bind attempt, provider, initiating administrator/session, exact redirect, nonce/state, and expiry, and are single-use. +4. Secret plaintext never crosses presentation/log/diagnostic/ordinary backup boundaries. +5. An unofficial adapter is distinct, opt-in, visibly labeled, and independently disableable. +6. Disconnect stops admissions immediately and cannot erase user-authored identity decisions merely because provider payloads are removed. + +## 5. Product journey + +| Stage | Proposed behavior | User/operator effect | +| --- | --- | --- | +| Choose | Show available adapters and support/access classifications | User does not confuse community feasibility with official support | +| Review | Show capabilities, scopes, subscription/client-registration needs, dependencies, token lifetime, and known limits | Informed choice before secrets or consent | +| Configure | Validate non-secret and secret registration inputs through their correct storage boundary | Invalid callbacks/credentials fail before leaving the App | +| Authorize | Start one short-lived signed attempt and leave for provider consent | Return is bound to the initiating flow | +| Verify | Exchange/receive grant, fetch immutable account identity, persist secret reference, probe capabilities | Connection is not declared ready from a token alone | +| Operate | Refresh single-flight; show health/expiry/action required | Background work pauses safely when authorization fails | +| Disconnect | Revoke when supported, delete local grants, disable jobs, apply retention cleanup | User sees retained decisions/history versus removed provider data | + +```mermaid +stateDiagram-v2 + [*] --> NotConfigured + NotConfigured --> Authorizing + Authorizing --> Connected + Authorizing --> NotConfigured + Connected --> Degraded + Connected --> ActionRequired + Degraded --> Connected + Degraded --> ActionRequired + ActionRequired --> Authorizing + Connected --> Disconnecting + Degraded --> Disconnecting + ActionRequired --> Disconnecting + Disconnecting --> Disconnected +``` + +Text equivalent: an unconfigured adapter starts one authorization attempt; verification produces a connected state, which can degrade or require user action; reauthorization returns through a new attempt; disconnect revokes/erases safely and ends disconnected. + +## 6. Functional behavior and state model + +### 6.1 Happy path + +1. The administrator selects an adapter and reads its manifest disclosure. +2. Symphonia validates required client registration and callback prerequisites. +3. A short-lived attempt is created with exact requested scopes and signed/single-use correlation state. +4. Provider consent returns through the accepted callback boundary. +5. The authorization owner exchanges/validates the result, retrieves immutable account identity, stores encrypted grant material, and passes only a secret reference inward. +6. The adapter probes effective capabilities and Symphonia shows `connected` plus any limitations. +7. The import capability schedules a separate durable operation. + +### 6.2 Alternative and boundary paths + +- Consent denial returns to `not_configured` with no connection or stored user grant. +- Missing optional scopes create a connected but reduced capability set only when the user explicitly accepted that mode. +- Expired/revoked credentials become `action_required`; pending writes are not blindly retried. +- Provider outage can be `degraded` without claiming reauthorization is required. +- Multiple accounts of one provider retain separate connection IDs, secret references, account identity, capability probes, and rate budgets. +- An unofficial cookie-based adapter, if ever accepted, uses a different configuration/credential contract and cannot call itself OAuth. + +### 6.3 State machine + +| State | Entered when | User-visible meaning | Allowed next states | Recovery/owner | +| --- | --- | --- | --- | --- | +| `not_configured` | no verified connection | Provider is not connected | `authorizing` | administrator | +| `authorizing` | live unexpired attempt exists | Waiting for consent/return | `connected`, `not_configured` | user completes or cancels | +| `connected` | grant, identity, and capability probe are valid | Eligible capabilities may be used | `degraded`, `action_required`, `disconnecting` | automatic refresh/probe | +| `degraded` | transient provider/dependency/partial capability fault | Some named features unavailable | `connected`, `action_required`, `disconnecting` | retry or administrator | +| `action_required` | expired/revoked/invalid grant or changed required terms | Jobs needing provider access are paused | `authorizing`, `disconnecting` | administrator reauthorizes | +| `disconnecting` | cleanup is underway | No new provider work is admitted | `disconnected` | automatic/bounded user action | +| `disconnected` | local grant erased and cleanup scheduled | No provider access remains | terminal or new independent setup | administrator | + +Authorization attempts also have `created`, `redirected`, `returned`, `consumed`, `denied`, `expired`, and `failed` states; only one transition may consume the callback. + +## 7. Configuration contract + +Exact provider field names remain adapter-owned. Every field declares type, secret status, validation, persistence, and documentation URL. + +| Input | Type | Recommended default | Allowed values/range | Scope/persistence | +| --- | --- | --- | --- | --- | +| OAuth application ownership | enum | bring-your-own for distributed self-hosting | accepted shared registration or per-install credentials | adapter/install; snapshotted for attempt | +| Requested capability profile | enum | minimum MVP import/copy profile | adapter-declared bounded profiles | connection; exact scopes snapshotted | +| Callback mode | enum | unresolved pending `RG-002` | direct App or companion broker after acceptance | install; restart/version contract may apply | +| Unofficial access acknowledgement | boolean | `false` | explicit `true` only for accepted unofficial adapter | connection plus policy version/time | + +Client secrets, refresh tokens, Music User Tokens, browser cookies, and authorization codes are never App options. Redirect targets, scope sets, provider hosts, and token endpoints come from versioned adapter definitions, not arbitrary user URLs unless an accepted provider explicitly requires a bounded configurable endpoint. + +## 8. Clean Architecture design + +### 8.1 Responsibilities and dependency direction + +| Boundary | Owns | Must not own/import | +| --- | --- | --- | +| Domain | Connection identity, health/action state, capability descriptors | OAuth libraries, HTTP, HA config flow | +| Application | begin/complete/reauthorize/disconnect/probe use cases | Provider SDK DTOs, token persistence implementation | +| Provider adapter | Provider authorization metadata, account lookup, refresh/revoke, capability probe | Cross-provider workflow policy | +| Secret adapter | encrypt/store/load/delete/rotate by opaque reference | UI or provider semantics | +| HA broker adapter, optional | Config flow/Application Credentials and one-use local handoff | Provider imports/writes, database access | +| Presentation | Disclosure/forms/status/error mapping | Token values or authorization decisions | + +### 8.2 Contracts, durable state, and trust boundaries + +- Pure decisions: manifest disclosure, scope/capability comparison, state transitions, retry versus action-required classification. +- Use cases: list adapters, begin attempt, complete attempt, probe, refresh, reauthorize, disconnect. +- Ports: provider authorization, secret store, attempt repository, connection repository, clock/ID, cleanup scheduler, audit. +- Durable state: manifest/version acknowledgement, attempt metadata without reusable secrets, connection/account identity, secret reference, capability evidence, expiry/health. +- Concurrency: one callback consumption per attempt; one refresh single-flight per connection; disconnect fences new work. +- Untrusted inputs: provider callbacks/errors, user client credentials, cookie/header material, redirect/query parameters, companion-integration messages. + +### 8.3 Authorization boundary alternatives + +| Alternative | Benefits | Costs/risks | Readiness evidence | +| --- | --- | --- | --- | +| Direct App-owned OAuth | Self-contained provider adapter and standalone parity | Public callback/exposure, exact redirect, token storage all owned by App | `RG-002` callback-only listener and remote/local tests | +| Companion-integration broker | Reuses HA Application Credentials/config-flow callback UX | Second artifact, token ownership/handoff, backup/version skew | Signed one-use handoff threat model and HA integration spike | + +No preference becomes accepted until `RG-002` records evidence and an ADR chooses the boundary. + +### 8.4 Executable architecture constraints + +- Static tests reject provider/HA/secret implementation imports from domain/application policy. +- Adapter contract tests require manifest completeness, typed error mapping, scope/capability probes, and secret-safe failures. +- Route tests prove callbacks cannot reach management handlers or redirect outside an allowlist. +- Canary tests search logs, errors, DB non-secret columns, API/UI, metrics, traces, diagnostics, and backups for credential values. + +## 9. UI/UX and content contract + +### 9.1 Information hierarchy + +Before the primary `Connect` action, show access basis, maturity/support, account/subscription prerequisites, requested functional access, credential type, callback/remote-access needs, external dependencies, reauthorization expectation, and known limitations. + +### 9.2 Representative states + +```text +Spotify · official API · beta adapter +Can read saved tracks and playlists and create playlists after capability verification. +Requires: Spotify Premium and your own Developer application. +Authorization: OAuth; reauthorization may be required when the provider grant expires. +Primary action: Connect Spotify +``` + +```text +YouTube Music · unofficial reverse-engineered access · experimental/best effort +This adapter would require reusable browser-account credentials and may stop working when Google changes private behavior. +No write capability is promised. Use is not enabled in this build. +``` + +```text +Spotify needs reconnection +Impact: imports and playlist writes are paused; existing library/history remain available. +Cause: the provider rejected the expired grant. +Action: Reconnect Spotify. +Retained state: mappings, plans, and audit history are unchanged. +``` + +```text +Disconnected from Spotify +Provider access and local credential material were removed. Provider-derived cached data is scheduled for policy cleanup; manual decisions and operation summaries remain. +``` + +### 9.3 Accessibility and localization + +Risk/support is textual, not icon-only. Consent-return errors receive focus and a live-region announcement without exposing query values. External provider links identify their destination. Provider/user display names are escaped and truncated. English fallback exists for any missing provider translation. + +## 10. Failure, recovery, and cleanup + +| Failure/partial state | User impact | Retained facts | Automatic retry | Required action | Cleanup | +| --- | --- | --- | --- | --- | --- | +| Consent denied/expired attempt | no connection | audit reason only | no | restart flow | erase transient material | +| Callback replay/state mismatch | no connection | security event/correlation | no | restart flow if legitimate | invalidate attempt | +| Token exchange outcome unknown | no usable connection | attempt marked uncertain | read/reconcile only if provider supports | retry authorization otherwise | no duplicate connection | +| Account identity fetch fails | not connected | secret held only within bounded transaction/recovery | bounded transient retry | retry later | delete grant if setup aborts | +| Refresh race | affected jobs wait | current credential reference/version | single-flight owner only | none/action if rejected | losers reread result | +| Provider outage | degraded | connection/capabilities/history | bounded backoff | none unless prolonged | no credential deletion | +| Grant revoked/expired | jobs paused | connection and non-secret history | no blind retry | reauthorize | old grant replaced/erased | +| Disconnect revoke fails | external grant may remain | local disconnect audit | bounded safe retry | revoke in provider UI | local credentials still erased per policy | + +## 11. Security, permissions, and privacy + +1. Use current recommended authorization-code/PKCE behavior for the deployment/client type; never invent a password-collection flow. +2. Callback state is signed/opaque, short-lived, single-use, and bound to initiator, provider, redirect, and attempt. +3. Secret material is encrypted at rest, versioned for rotation, absent from general database exports, and redacted through canary tests. +4. Provider/adapter URLs and schemes are allowlisted; user/provider data cannot choose filesystem paths, modules, callbacks, or unrestricted egress. +5. A companion broker uses mutual/local authentication and a one-use handoff; it never exposes HA-stored credentials through ordinary service/entity state. +6. Browser-cookie access, if accepted, requires a dedicated threat model, explicit primary-account warning, deletion/revocation steps, and independent release kill switch. + +## 12. Observability and operational UX + +- Connection view shows health, capability evidence age, grant/reauthorization horizon when known, last probe/import, and action required. +- Audit records authorization start/result category, account identity hash/reference, capability changes, refresh/reauth/disconnect, and actor—never token/header/body. +- Metrics count attempts/results, refresh outcomes, action-required age, capability changes, and sanitized provider error categories. +- Provider outage is distinct from invalid credentials; a 5xx must not tell users to reconnect. +- Stable connected state emits no periodic notifications; only impending required action, revocation, capability loss, or repeated failure is prominent. + +## 13. Compatibility, migration, rollout, and rollback + +- Adapter manifest/version and acknowledgement version are stored with a connection. +- Scope/capability changes require explicit reauthorization when grants cannot safely expand in place. +- Secret schema/key rotation is transactional and recoverable; rollback never writes old plaintext formats. +- Direct and broker modes do not coexist indefinitely for one connection; migration requires an explicit reauthorization or proven one-time handoff. +- Removing/rolling back an adapter disables its jobs while preserving audit and user decisions. + +## 14. Testing strategy and numeric budget + +Minimum **82 distinct cases**: + +| Area | Minimum distinct cases | Behaviors/risks covered | +| --- | ---: | --- | +| Manifest/configuration/capability policy | 16 | classifications, scopes, account/object/live intersections, invalid combinations | +| Attempt/state/idempotency/races | 20 | expiry, denial, replay, duplicate callback, refresh/disconnect races, restart | +| Provider/secret/broker adapters | 18 | exchange/refresh/revoke/probe, error mapping, rotation, handoff, timeouts | +| HTTP/UI/accessibility/sanitization | 12 | disclosures and all states, focus, hostile names/errors/URLs, locale fallback | +| Integration/security/migration | 16 | direct/broker paths, listener isolation, canary leakage, backup, reauth migration | +| **Total** | **82** | No double counting | + +Provider contract tests are offline and deterministic. Live smoke tests are opt-in, use dedicated accounts/client registrations, bounded scopes/quota, and safe cleanup. Exact secrets are seeded as canaries and asserted absent from every non-secret boundary. Manual evidence reviews provider consent transitions and narrow/mobile disclosures. + +## 15. Documentation and discoverability + +| Audience | Artifact | Required content | Validation/navigation | +| --- | --- | --- | --- | +| User | Connect/reconnect/disconnect guide | support labels, scopes, normal flow, retained data | UI content fixtures | +| Setup owner | Per-provider setup | developer registration, callback, secrets, subscription/dependencies | live-spike matrix | +| Operator | Authentication troubleshooting | outage vs revoked, expiry, rotation, revoke/cleanup | error decision tree | +| Contributor | Adapter/auth architecture | ports, secret boundary, direct/broker contract | architecture/contract tests | + +## 16. Acceptance scenarios + +1. Given an official adapter, setup shows access basis, maturity/support, prerequisites, scopes, callback mode, and reauthorization expectations before `Connect`. +2. Given a valid consent return, exactly one attempt is consumed, immutable account identity is verified, only a secret reference enters connection state, and effective capabilities are shown. +3. Given denial, expired state, replay, wrong initiator, wrong provider, or wrong redirect, no connection is created and transient secrets are removed. +4. Given two simultaneous refresh requests, one exchange occurs and all callers observe one durable result. +5. Given provider outage, the connection becomes degraded without falsely requesting reauthorization; given invalid grant, it becomes action-required without blind retries. +6. Given a read-only playlist or missing scope, planning sees the object/connection denial even when the adapter supports the operation generally. +7. Given disconnect, new calls stop, local grant material is erased, revocation is attempted safely, provider-data cleanup is scheduled, and manual decisions/audit remain. +8. Given an unofficial adapter, authorization cannot begin without explicit risk acknowledgement and the UI never labels it official/supported by Home Assistant. +9. Given hostile callback/provider/error content, no open redirect, path/egress injection, Markdown/HTML injection, or secret output occurs. +10. Given App restart at every attempt phase, no callback is consumed twice and no orphan grant becomes an active connection. +11. Given the broker alternative, a forged/replayed/local unauthenticated handoff is rejected and the integration cannot mutate the App database directly. + +## 17. Requirements traceability + +| Requirement | Owner | Test or evidence | Documentation | +| --- | --- | --- | --- | +| `SYM-ACC-002`–`SYM-ACC-004`, `SYM-ACC-006` | connection use cases/presentation | lifecycle + disclosure fixtures | user connection guide | +| `SYM-PROV-002`, `SYM-PROV-003`, `SYM-PROV-019` | capability policy/adapter | connection/object/live matrix | provider capability reference | +| `SYM-PROV-015`–`SYM-PROV-020` | manifest and presentation | schema/opt-in/label tests | provider support policy | +| `SYM-SEC-001`–`SYM-SEC-007` | auth/secret ports | replay/race/canary/rotation tests | security and provider setup | +| `SYM-SEC-008`–`SYM-SEC-010` | listener/broker/provider adapters | route/redirect/path/egress abuse tests | callback operations guide | +| `SYM-TEST-005`, `SYM-TEST-011`, `SYM-TEST-012` | verification tooling | release gates | contributor testing guide | + +## 18. Implementation sequence + +1. Complete `RG-001`/`RG-002`, secret/backup threat model, and authorization-boundary ADR. +2. Define manifest, attempt, connection, capability, and normalized-error contracts plus architecture tests. +3. Implement pure state/capability/disclosure policies and deterministic tests. +4. Implement application use cases and fake provider/secret/broker ports. +5. Implement one official provider adapter and selected callback boundary behind contract tests. +6. Add UI, reauthorization/disconnect, diagnostics, documentation, and opt-in live smoke evidence. + +## 19. Definition of Done + +- [ ] Authorization-boundary, provider-scope, unofficial-access, and secret/backup blockers are resolved. +- [ ] Every requirement and state maps to acceptance and deterministic verification. +- [ ] At least 82 distinct cases and secret-canary gates pass. +- [ ] Callback replay, redirect, broker/listener isolation, refresh race, rotation, and disconnect are proven. +- [ ] Provider DTOs and secret implementations do not cross inward architecture boundaries. +- [ ] Disclosures and action-required/degraded/disconnected states pass accessibility and sanitization review. +- [ ] Per-provider setup, recovery, revocation, and risk documentation is complete. +- [ ] Live evidence is bounded/opt-in and catalog implementation evidence is current. +- [ ] Explicit owner approval to implement exists. + +## 20. References and decisions + +- Primary sources: official provider and Home Assistant sources in [provider research](../docs/providers/provider-research.md). +- Related SDDs: [App runtime](home-assistant-app-runtime-and-ingress.md), [imports](library-import-and-provider-projections.md), [durable operations](durable-operations-and-recovery.md). +- Accepted: capability and risk disclosure before authorization; opaque secret references; distinct unofficial adapters. +- Open alternatives: direct App OAuth versus HA companion broker. +- Rejected: tokens in App options; pasted callback URLs as tokens; generic retry of invalid grants; treating private APIs as official. diff --git a/specs/recording-identity-resolution.md b/specs/recording-identity-resolution.md new file mode 100644 index 0000000..c7e5626 --- /dev/null +++ b/specs/recording-identity-resolution.md @@ -0,0 +1,333 @@ +# Recording identity resolution + +- Status: Draft +- Date: 2026-09-20 +- Catalog capability ID: `recording-identity-resolution` +- Owners: Symphonia maintainers +- Scope: resolve provider track representations to provider-independent recordings using explainable, versioned evidence and durable manual decisions. +- Related requirements: `SYM-PROD-002`–`SYM-PROD-003`, `SYM-LIB-002`, `SYM-MATCH-001`–`SYM-MATCH-008`, `SYM-ARCH-014`, `SYM-TEST-007`–`SYM-TEST-008` +- Related decisions/research: [ADR 0001](../docs/decisions/0001-provider-independent-recording-domain.md), [identity domain model](../docs/domain/domain-model.md#identity-resolution-specification), [provider research](../docs/providers/provider-research.md), `RG-003` +- Required review gates: product UX, domain/music semantics, architecture, provider feasibility, testing/data licensing, documentation +- Open decisions blocking readiness: reviewed labeled corpus; automatic-link precision/false-link tolerance; versioned scoring/confidence policy; provider candidate retrieval/quota evidence + +## 1. Executive summary + +Symphonia resolves provider representations to recordings conservatively. Exact identifiers such as ISRC are strong evidence but never sole proof; title similarity alone is never sufficient. Automatic decisions are versioned and explainable, ambiguous items enter a review queue, and manual acceptance/rejection is durable and cannot be overwritten silently. + +The recommended MVP default is precision-first: automatically link only policy-approved high-confidence evidence, prefer `ambiguous`/`unmatched` over a wrong recording, and require explicit review before a copy depends on uncertain identity. + +```text +Provider track -> normalize without destroying originals -> generate bounded candidates +-> evaluate identifiers/version/artist/duration/release evidence -> classify +-> safe auto-link OR review queue -> durable accept/reject/distinct/defer decision +-> future imports and plans reuse or explicitly invalidate that decision +``` + +## 2. Problem, current behavior, and evidence + +### 2.1 Problem + +Catalogs contain missing/wrong IDs, localized metadata, covers, live/studio variants, remasters, edits, clean/explicit releases, music videos, user uploads, compilations, and duration differences. A false link silently copies the wrong recording and contaminates future operations; an unmatched item is visible and recoverable. + +### 2.2 Current behavior + +No resolver exists. The provider-independent recording decision is accepted, but candidate algorithms, thresholds, corpus, and precision targets remain open. + +### 2.3 Evidence and unknowns + +- Domain invariants and required pipeline: [domain model](../docs/domain/domain-model.md#identity-resolution-specification). +- Official metadata availability: [provider research](../docs/providers/provider-research.md). +- Comparative evidence: Music Assistant uses provider mappings and automatic search/matching, but Symphonia requires stronger evidence/audit; [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md#provider-mappings-validate-the-representation-graph). +- Unknowns: candidate recall by provider, labeled corpus licensing, acceptable false-positive rate, confidence boundaries, manual-review scale. + +## 3. Actors, surfaces, and terminology + +| Actor | Goal | Entry point | Visible surfaces | +| --- | --- | --- | --- | +| Symphonia user/reviewer | Confirm or reject ambiguous identities | Resolution queue or copy-plan blocker | Candidate comparison, evidence, decision history | +| Resolver | Produce reproducible assessments | New/changed provider projection | Candidate set, evidence, confidence class, version | +| Provider adapter | Retrieve bounded candidates/metadata | Candidate search/get ports | Normalized candidates and capability/quality warnings | +| Copy planner | Use only policy-approved active links | Plan creation | Ready/non-ready entry classification | + +**Recording** is one specific recorded performance/version. **Provider track** is an external representation. **Candidate assessment** is a versioned comparison, not a link. **Identity link** is the active relation between a provider track and recording. **Rejected pair** is durable negative evidence. **Distinct recording** means the reviewer asserts that the provider track should not be merged with current candidates. + +## 4. Goals, non-goals, and fixed invariants + +### 4.1 Goals + +1. Produce safe, explainable, reproducible identity decisions. +2. Preserve meaningful version distinctions and provider evidence provenance. +3. Make ambiguity review efficient without hiding uncertainty. +4. Reuse manual positive and negative decisions across operations. +5. Measure candidate recall and link precision separately. + +### 4.2 Non-goals + +1. Identifying an abstract musical work or merging cover performances. +2. Recommending music or learning from private libraries with an ML model. +3. Guaranteeing every provider item can be resolved. +4. Treating popularity, title equality, or one provider's canonical claim as universal truth. +5. Mutating provider data. + +### 4.3 Fixed invariants + +1. A provider track has zero or one active recording link; a recording may have many provider tracks. +2. ISRC is evidence, not infallible proof; normalized title alone cannot auto-link. +3. Originals and derived comparison values retain provenance. +4. Manual accept/reject/distinct decisions outrank automation until explicitly revoked or invalidated with reason. +5. Every automatic assessment records resolver/rule/data versions and material positive/negative evidence. +6. Covers, live/studio, remix/edit, acoustic, remaster, clean/explicit, and video/upload distinctions are not collapsed when evidence indicates a difference. + +## 5. Product journey + +| Stage | Proposed behavior | User/operator effect | +| --- | --- | --- | +| Normalize | Derive comparison fields while retaining originals and provenance | User can inspect what changed for comparison | +| Generate | Retrieve bounded candidates from identifiers, existing graph, and provider search | Quota/candidate source and completeness are visible | +| Evaluate | Compare identifiers, credits, title/version tokens, duration, release, explicitness, media type, market | Each confidence class has human-readable evidence | +| Apply policy | Auto-link only accepted classes; route ambiguity/unmatched separately | Wrong-link risk is explicit and conservative | +| Review | Accept candidate, reject pair, create distinct recording, defer, or revoke | Decision is attributable and immediately reusable | +| Re-evaluate | New metadata/rule versions can reassess without silently replacing manual truth | Invalidations and supersession are visible | + +```mermaid +stateDiagram-v2 + [*] --> Unmatched + Unmatched --> Ambiguous + Unmatched --> Resolved: safe automatic or manual link + Ambiguous --> Resolved: accept candidate + Ambiguous --> Deferred + Ambiguous --> Unmatched: reject all current candidates + Unmatched --> Resolved: create distinct recording + Deferred --> Ambiguous: review resumed/new evidence + Resolved --> Ambiguous: automatic link invalidated + Resolved --> Unmatched: manual link revoked/no candidates + Resolved --> Resolved: explicit relink/supersession +``` + +Text equivalent: items begin unmatched; candidate evidence may make them ambiguous or safely resolved; reviewers may resolve, reject, create a distinct recording, or defer; later evidence may invalidate automatic links, while manual links change only through explicit revocation/supersession. + +## 6. Functional behavior and state model + +### 6.1 Happy path + +1. A changed provider track is queued with its immutable observed metadata reference. +2. Normalization produces versioned comparison fields without altering the source projection. +3. Candidate generation checks durable accepted/rejected mappings, exact external identifiers, existing recording representations, and bounded provider search where permitted. +4. Evaluation emits candidate evidence and confidence; policy either creates an automatic link or stores a review/unmatched result. +5. A reviewer sees original metadata, material differences, provider availability, and prior decisions. +6. The accepted decision writes an attributable link/rejection/distinct recording and invalidates affected copy plans rather than mutating them. + +### 6.2 Alternative and boundary paths + +- Missing ISRC can still yield candidates; shared ISRC with conflicting version evidence cannot auto-link by identifier alone. +- A YouTube music/lyric/live/user-upload video receives media-type/version evidence and usually weaker confidence than a label catalog track. +- A rejected pair is excluded from future automatic linking but may be shown as previously rejected if materially new evidence appears. +- Provider metadata change triggers versioned reassessment; it does not silently edit historical evidence. +- If candidate search is rate-limited/unavailable, the state remains unmatched/deferred with an incomplete-search reason. +- Deleting a provider connection removes/retains provider payloads per policy while preserving manual decision history as allowed. + +### 6.3 Resolution states + +| State | Entered when | User-visible meaning | Allowed next states | Recovery/owner | +| --- | --- | --- | --- | --- | +| `unmatched` | no policy-acceptable candidate/link | No recording selected | `ambiguous`, `resolved`, `deferred`, `invalid` | new evidence or reviewer | +| `ambiguous` | multiple/insufficient candidates require judgment | Copy is blocked or explicit best effort required | `resolved`, `unmatched`, `deferred`, `invalid` | reviewer | +| `resolved` | one active automatic/manual link exists | Recording identity is available with origin/evidence | `ambiguous`, `unmatched`, `invalid`, `resolved` via explicit supersession | invalidation/reviewer | +| `deferred` | reviewer intentionally postponed | No automatic relink until trigger/policy permits | `ambiguous`, `resolved`, `unmatched`, `invalid` | reviewer/new evidence | +| `invalid` | source representation cannot participate under current model | Item remains auditable but unusable for recording copy | `unmatched` after corrected evidence | provider fix/re-import | + +Candidate assessments separately use `exact`, `high_confidence`, `ambiguous`, and `unmatched`; these do not replace the provider track's resolution state. + +## 7. Configuration contract + +Threshold values are not general user settings in the MVP. They are versioned resolver policy accepted with corpus evidence. + +| Input | Type | Recommended default | Allowed values/range | Scope/persistence | +| --- | --- | --- | --- | --- | +| Automatic-link policy | versioned policy ID | precision-first accepted release policy | reviewed built-in policies only | snapshotted in assessment/link | +| Candidate limit | bounded integer per source | conservative corpus/provider-derived default | adapter/policy hard range | snapshotted per run | +| Search sources | set | existing graph + official provider sources allowed by policy | adapter-declared sources | policy/version | +| Deferred recheck | enum | new material metadata or explicit user action | bounded triggers | decision metadata | + +Users cannot lower safety thresholds ad hoc, enable title-only auto-linking, discard negative decisions, or turn provider-specific identifiers into recording IDs. + +## 8. Clean Architecture design + +### 8.1 Responsibilities and dependency direction + +| Boundary | Owns | Must not own/import | +| --- | --- | --- | +| Domain | Recording, provider track identity, link/rejection invariants, evidence/confidence types | Provider SDK/search/database/UI | +| Application | Resolve/review/revoke/invalidate use cases and plan invalidation | Concrete scoring/search implementation dependencies | +| Resolver policy | Pure normalization/evaluation/classification from normalized evidence | Network, persistence, clock | +| Candidate adapters | Provider/external lookup and metadata normalization | Link acceptance policy | +| Persistence | Evidence/link/rejection/version history | Silent conflict resolution | +| Presentation | Comparison/review/history view models | Mutating links outside use cases | + +### 8.2 Contracts, durable state, and trust boundaries + +- Pure decisions: normalization, evidence comparison, confidence classification, automatic-link eligibility, invalidation. +- Use cases: resolve provider track, review decision, reject pair, create distinct recording, defer, revoke/supersede, inspect history. +- Ports: provider candidate search/get, recording/link/evidence repositories, corpus/policy version, audit, durable operation scheduler. +- Durable state: source observation references, candidate assessments, evidence, active/superseded/revoked links, rejected pairs, actor/time/resolver version. +- Concurrency: decision version/optimistic fence prevents a stale resolver from overwriting a newer manual action. +- Untrusted inputs: all provider metadata, identifiers, artwork/URLs, user-entered notes, external metadata. + +### 8.3 Executable architecture constraints + +- Pure resolver modules cannot import provider SDK, HTTP, database, filesystem, clock, or UI modules. +- Corpus tests report precision/recall and regression by case category and resolver version. +- State tests prove manual decisions win every race with automatic reassessment. +- Presentation tests prove explanations derive from stored structured evidence, not opaque score text. + +## 9. UI/UX and content contract + +### 9.1 Information hierarchy + +The review queue shows source identity/state first, then the best candidates and decisive evidence/differences, followed by one primary action. It never leads with an unexplained numeric score. + +### 9.2 Representative states + +```text +Needs review · “Hallelujah” — Jeff Buckley +Why: two catalog candidates share title and artist. The durations differ by 41 seconds and one is marked live. + +Recommended candidate +Hallelujah · Grace · 6:53 · studio · ISRC USSM19400391 +Evidence: artist exact; title exact; album exact; duration +1s; source has no ISRC. + +Other candidate +Hallelujah (Live at Sin-é) · 9:16 · live +Difference: version marker and duration conflict. + +Primary action: Accept recommended candidate +Other actions: Reject candidate · Create distinct recording · Defer +``` + +```text +Resolved manually +Linked to the studio recording by the Home Assistant administrator on 20 Sep 2026. +This decision will be reused. Revoke or choose a different recording to change it. +``` + +```text +Automatic link invalidated +New provider metadata marks this item as a live version. No replacement was selected. +Impact: new copy plans will treat the item as ambiguous; completed copies/history are unchanged. +Action: Review candidates. +``` + +### 9.3 Accessibility and localization + +Candidate comparison works by keyboard with explicit selected row/action and focus return. Evidence differences use text, not red/green alone. Screen readers receive candidate headings and concise difference summaries. Original provider text and normalized text remain distinguishable. User/provider content is escaped and length-bounded; localized metadata is not forcibly translated. + +## 10. Failure, recovery, and cleanup + +| Failure/partial state | User impact | Retained facts | Automatic retry | Required action | Cleanup | +| --- | --- | --- | --- | --- | --- | +| Candidate search rate-limited | item remains unresolved | source/evidence/search checkpoint | bounded schedule | none unless exhausted | cached results honor retention | +| Provider search unavailable | no fresh candidates | prior assessments labeled stale | bounded | retry/review existing evidence | no link inferred | +| Resolver crash/restart | delayed assessment | operation/source version | durable retry | none | idempotent assessment key | +| Stale automatic result races manual decision | none to manual truth | both attempts/audit | automatic result rejected | none | no silent overwrite | +| Metadata invalidates automatic link | future plans blocked/ambiguous | old link/evidence/status history | reassess once | review if ambiguous | old link superseded, not erased | +| Manual decision becomes impossible | action required | manual history and reason | no automatic replacement | revoke/relink | provider payload retention applies | + +## 11. Security, permissions, and privacy + +1. Only an authenticated authorized administrator may create/revoke manual decisions in the MVP. +2. Provider/user metadata is untrusted content and never controls markup, URLs, paths, queries beyond bounded adapter parameters, or code. +3. Evidence stores only fields justified for resolution/audit and carries provider retention/deletion classification. +4. Automatic decisions and manual actors are auditable; forged/stale commands fail optimistic version checks. +5. Resolver/corpus fixtures contain synthetic or legally redistributable data, never copied private libraries or restricted training datasets. + +## 12. Observability and operational UX + +- Metrics: counts by resolution state/origin/provider, candidate search outcomes, automatic-link rate, manual accept/reject/defer/revoke, invalidations, corpus regressions. +- Audit: provider track, recording/link IDs, actor, source observation, resolver/policy version, evidence categories, decision transition; no raw token/payload. +- Copy plans reference exact active link/evidence versions so later changes do not rewrite history. +- Search waiting/provider outage is distinct from genuine unmatched classification. +- Queue notifications are bounded summaries; individual ambiguous items do not create notification spam. + +## 13. Compatibility, migration, rollout, and rollback + +- Resolver and normalization versions are immutable identifiers; new versions create new assessments. +- Automatic links can be re-evaluated under controlled rollout; manual links/rejections are never bulk-replaced. +- Schema migrations preserve full decision/link status history and rejected pairs. +- Rollback continues to understand links it created or blocks with a compatibility message; it does not reinterpret scores. +- Initial rollout may operate manual-only until corpus thresholds justify automatic linking. + +## 14. Testing strategy and numeric budget + +Minimum **96 distinct cases** due to the high cost of false links: + +| Area | Minimum distinct cases | Behaviors/risks covered | +| --- | ---: | --- | +| Normalization/evidence/classification | 34 | identifiers, Unicode, credits, duration, releases, every version distinction, missing/conflicting data | +| State/application/concurrency | 18 | accept/reject/distinct/defer/revoke, stale resolver, invalidation, replay, plan references | +| Candidate/provider contracts | 14 | bounded search, quota, availability, generic video vs catalog, malformed/duplicate candidates | +| UI/accessibility/sanitization | 12 | unmatched/ambiguous/resolved/invalidated, keyboard/focus, hostile metadata, locale/narrow views | +| Integration/security/migration | 18 | corpus regressions, manual preservation, schema/version changes, retention, permission/forgery | +| **Total** | **96** | No double counting | + +The corpus includes exact duplicates, missing/wrong/reused ISRC, covers, live/studio, remasters, remixes/edits, acoustic, clean/explicit, compilations, featured artists, localized metadata, music/lyric videos, uploads, duration drift, and misleading titles. Candidate recall and accepted-link precision are reported separately. Numeric acceptance thresholds remain a blocker until corpus review. + +## 15. Documentation and discoverability + +| Audience | Artifact | Required content | Validation/navigation | +| --- | --- | --- | --- | +| User | Matching/review guide | recording meaning, evidence, actions, reuse/revoke | review UI fixtures | +| Setup owner | Provider matching limits | metadata/search quality, quota, privacy | provider matrix | +| Operator | Resolver troubleshooting | stuck search, invalidation, corpus/policy version | decision tree | +| Contributor | Resolver architecture/corpus guide | pure policy, evidence schema, licensing, regression rules | architecture/corpus gates | + +## 16. Acceptance scenarios + +1. Given matching high-quality identifiers and no material version conflict, the accepted policy may auto-link and records structured evidence plus resolver/policy version. +2. Given title equality alone, no automatic link is created. +3. Given a shared ISRC but conflicting live/studio, duration, or version evidence, the item becomes ambiguous rather than silently merged. +4. Given cover, remix, remaster, acoustic, clean/explicit, compilation, video, upload, and featured-artist corpus cases, expected distinctions and evidence explanations remain stable. +5. Given manual acceptance or rejection racing automatic reassessment, the manual version wins and the stale result is recorded/rejected without overwrite. +6. Given a rejected pair, future resolution excludes it unless materially new evidence is explicitly shown to the reviewer. +7. Given create-distinct, a provider-independent recording is created without asserting a musical-work relation or merging similar titles. +8. Given automatic-link invalidation, future plans see ambiguity/unmatched while completed plan/history references remain unchanged. +9. Given hostile metadata/URLs/user notes, comparison UI and logs remain escaped, bounded, and non-executable. +10. Given resolver-version rollout/rollback, manual decisions persist and corpus regressions block release under the accepted thresholds. + +## 17. Requirements traceability + +| Requirement | Owner | Test or evidence | Documentation | +| --- | --- | --- | --- | +| `SYM-MATCH-001`–`SYM-MATCH-002` | domain/state/use cases | state/link/evidence tests | matching guide | +| `SYM-MATCH-003`–`SYM-MATCH-004` | resolver policy | labeled corpus category suite | evidence explanation guide | +| `SYM-MATCH-005`–`SYM-MATCH-006` | review use cases/persistence | manual action/race/reuse tests | review/revoke guide | +| `SYM-MATCH-007`–`SYM-MATCH-008` | evidence/policy/presentation | explanation/version fixtures | contributor resolver guide | +| `SYM-ARCH-014` | candidate scheduler/cache | quota/cache/retention tests | provider limits | +| `SYM-TEST-007`–`SYM-TEST-008` | corpus/release tooling | category metrics/regression gate | corpus maintenance guide | + +## 18. Implementation sequence + +1. Build/license/review the corpus and accept error costs, confidence policy, and release thresholds. +2. Define evidence/link/rejection/state schemas and architecture boundaries. +3. Implement pure normalization/evaluation/policy with corpus tests in manual-only mode. +4. Implement application review/invalidation use cases and persistence with race/replay tests. +5. Add candidate adapters behind bounded contracts and quota/retention controls. +6. Add review UI, accessibility, explanations, docs, then controlled automatic-link rollout. + +## 19. Definition of Done + +- [ ] Corpus, precision/recall thresholds, confidence policy, and candidate feasibility blockers are accepted. +- [ ] Every requirement and state maps to acceptance and deterministic/corpus tests. +- [ ] At least 96 distinct cases pass with required category/regression reporting. +- [ ] Manual decisions win all races and survive migrations/resolver changes. +- [ ] False-link safety, evidence/version provenance, invalidation, and plan history are proven. +- [ ] Review UX passes keyboard, screen-reader, narrow layout, localization fallback, and sanitization review. +- [ ] Corpus licensing/privacy and provider data-retention rules are documented. +- [ ] Catalog evidence is current and explicit owner approval to implement exists. + +## 20. References and decisions + +- Primary sources: official provider metadata research in [provider research](../docs/providers/provider-research.md). +- Related SDDs: [imports](library-import-and-provider-projections.md), [playlist copy](one-time-playlist-copy.md), [durable operations](durable-operations-and-recovery.md). +- Accepted: recording identity is provider-independent; uncertainty/manual evidence are first-class. +- Rejected: title-only matching, opaque score-only UX, silent automatic override of manual decisions, work-level cover merging. +- Follow-up: musical-work relationships and additional metadata providers after MVP evidence. From aa62b33b7d25d4b243a0d60455705ed492903f54 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 16:28:22 +0200 Subject: [PATCH 002/167] feat: add copy planning and durable operation core --- .gitignore | 12 + README.md | 4 +- docs/development/development-specification.md | 2 +- docs/development/implementation-baseline.md | 33 +++ pyproject.toml | 22 ++ src/symphonia/__init__.py | 9 + src/symphonia/application/__init__.py | 6 + src/symphonia/application/copy_planning.py | 30 ++ src/symphonia/domain/__init__.py | 22 ++ src/symphonia/domain/models.py | 241 +++++++++++++++ src/symphonia/infrastructure/__init__.py | 11 + .../infrastructure/sqlite_operations.py | 275 ++++++++++++++++++ tests/__init__.py | 1 + tests/test_copy_planning.py | 99 +++++++ tests/test_sqlite_operations.py | 133 +++++++++ 15 files changed, 897 insertions(+), 3 deletions(-) create mode 100644 .gitignore create mode 100644 docs/development/implementation-baseline.md create mode 100644 pyproject.toml create mode 100644 src/symphonia/__init__.py create mode 100644 src/symphonia/application/__init__.py create mode 100644 src/symphonia/application/copy_planning.py create mode 100644 src/symphonia/domain/__init__.py create mode 100644 src/symphonia/domain/models.py create mode 100644 src/symphonia/infrastructure/__init__.py create mode 100644 src/symphonia/infrastructure/sqlite_operations.py create mode 100644 tests/__init__.py create mode 100644 tests/test_copy_planning.py create mode 100644 tests/test_sqlite_operations.py diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..17e1f5f --- /dev/null +++ b/.gitignore @@ -0,0 +1,12 @@ +__pycache__/ +*.py[cod] +*.egg-info/ +.pytest_cache/ +.coverage +coverage.xml +dist/ +build/ +.venv/ +*.sqlite3 +*.db + diff --git a/README.md b/README.md index c2a947b..4f87128 100644 --- a/README.md +++ b/README.md @@ -8,9 +8,9 @@ Symphonia is a library-management and interoperability project, not a new music ## Project status -**Specification stage. No production application has been implemented.** +**Implementation foundation stage. Provider integrations and the Home Assistant App artifact are not implemented yet.** -The current work establishes a reviewable source of truth before technology selection or implementation begins. In particular, official provider feasibility still needs validation: Google's public YouTube Data API can manage YouTube video playlists, but the research performed for this specification did not identify an official API exposing the complete YouTube Music library model. +The current work combines a reviewable source of truth with the first owner-approved, dependency-free domain/persistence slice. Official provider feasibility still needs validation: Google's public YouTube Data API can manage YouTube video playlists, but the research performed for this specification did not identify an official API exposing the complete YouTube Music library model. ## Documentation diff --git a/docs/development/development-specification.md b/docs/development/development-specification.md index d8b59ba..c397c21 100644 --- a/docs/development/development-specification.md +++ b/docs/development/development-specification.md @@ -5,7 +5,7 @@ ## Specification-driven workflow -Implementation begins only when the applicable capability SDD is `Ready for implementation` and the owner explicitly approves the increment. The SDD lifecycle, required content, readiness gate, and catalog rules are defined in the [SDD standard](../../specs/README.md). Start new capability designs from the [mandatory template](../../specs/_template.md), and keep their state in the [catalog](../../specs/CATALOG.md). +Implementation begins only when the applicable capability SDD is `Ready for implementation` and the owner explicitly approves the increment. The current [implementation baseline](implementation-baseline.md) is the explicitly approved exception for dependency-free domain/persistence foundations; it does not authorize provider, OAuth, UI, or Home Assistant capability work while those SDDs remain Draft. The SDD lifecycle, required content, readiness gate, and catalog rules are defined in the [SDD standard](../../specs/README.md). Start new capability designs from the [mandatory template](../../specs/_template.md), and keep their state in the [catalog](../../specs/CATALOG.md). For each proposed increment: diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md new file mode 100644 index 0000000..03a1b65 --- /dev/null +++ b/docs/development/implementation-baseline.md @@ -0,0 +1,33 @@ +# Implementation baseline + +**Status:** owner-approved foundation slice +**Last reviewed:** 2026-09-20 + +The first implementation increment is intentionally narrower than any provider or Home Assistant capability. It proves the provider-independent core and the durable-operation persistence contract without selecting an external web framework, provider SDK, OAuth strategy, or frontend stack. + +## Current slice + +- Python 3.11+ package under `src/symphonia`. +- Dependency-free domain values for ordered playlist snapshots, the six product entry classifications, copy policies, immutable copy plans, and acceptance digests. +- Dependency-free SQLite operation repository proving idempotent creation, time-bounded worker leases, checkpoint ownership, retry scheduling, and expired-lease recovery. +- Deterministic `unittest` coverage under `tests/`. + +The package is an implementation foundation, not a claim that the corresponding capability SDDs are complete. Provider OAuth, provider API adapters, user authentication, Ingress, UI, migrations beyond the initial schema, backup/restore, and operation handlers remain unimplemented and blocked by their SDD decisions. + +## Local verification + +From the repository root: + +```text +PYTHONPATH=src python3 -m unittest discover -s tests -v +``` + +The command must run without network access, provider accounts, Home Assistant, or a pre-existing database. + +## Boundary rules + +- `symphonia.domain` imports no infrastructure, provider, Home Assistant, or web framework modules. +- Application services orchestrate domain values and do not call provider SDKs directly. +- SQLite is an adapter behind the operation repository; it is not exposed as a domain concept. +- The initial Python choice is a reversible implementation baseline, not a final product-stack decision. Any external dependency or deployment commitment requires an SDD/ADR update and tests. + diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..4f14b3c --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,22 @@ +[build-system] +requires = ["setuptools>=68"] +build-backend = "setuptools.build_meta" + +[project] +name = "symphonia" +version = "0.1.0.dev0" +description = "Self-hosted, provider-independent music library hub" +requires-python = ">=3.11" +license = { text = "TBD" } +authors = [{ name = "Symphonia maintainers" }] +dependencies = [] + +[project.scripts] +symphonia = "symphonia.__main__:main" + +[tool.setuptools.packages.find] +where = ["src"] + +[tool.pytest.ini_options] +testpaths = ["tests"] + diff --git a/src/symphonia/__init__.py b/src/symphonia/__init__.py new file mode 100644 index 0000000..2c67b53 --- /dev/null +++ b/src/symphonia/__init__.py @@ -0,0 +1,9 @@ +"""Symphonia application core. + +The package intentionally starts with a dependency-free domain and persistence +slice. Provider and Home Assistant adapters are added behind ports as their +contracts become implementation-ready. +""" + +__version__ = "0.1.0.dev0" + diff --git a/src/symphonia/application/__init__.py b/src/symphonia/application/__init__.py new file mode 100644 index 0000000..5bbe361 --- /dev/null +++ b/src/symphonia/application/__init__.py @@ -0,0 +1,6 @@ +"""Use-case orchestration ports and services.""" + +from .copy_planning import CopyPlanningService + +__all__ = ["CopyPlanningService"] + diff --git a/src/symphonia/application/copy_planning.py b/src/symphonia/application/copy_planning.py new file mode 100644 index 0000000..32ee06a --- /dev/null +++ b/src/symphonia/application/copy_planning.py @@ -0,0 +1,30 @@ +"""Application service for the non-mutating copy planning use case.""" + +from __future__ import annotations + +from dataclasses import dataclass + +from symphonia.domain.models import CopyPlan, CopyPolicy, PlaylistSnapshot, build_copy_plan + + +@dataclass(frozen=True, slots=True) +class CopyPlanningService: + """Coordinates normalized inputs without knowing provider implementations.""" + + def plan( + self, + snapshot: PlaylistSnapshot, + *, + target_provider: str, + target_playlist_name: str, + target_visibility: str = "private", + policy: CopyPolicy = CopyPolicy.STRICT, + ) -> CopyPlan: + return build_copy_plan( + snapshot, + target_provider=target_provider, + target_playlist_name=target_playlist_name, + target_visibility=target_visibility, + policy=policy, + ) + diff --git a/src/symphonia/domain/__init__.py b/src/symphonia/domain/__init__.py new file mode 100644 index 0000000..cb76b7a --- /dev/null +++ b/src/symphonia/domain/__init__.py @@ -0,0 +1,22 @@ +"""Provider-independent domain types.""" + +from .models import ( + CopyPlan, + CopyPlanEntry, + CopyPolicy, + EntryClassification, + PlanAcceptanceError, + PlaylistSnapshot, + SourcePlaylistEntry, +) + +__all__ = [ + "CopyPlan", + "CopyPlanEntry", + "CopyPolicy", + "EntryClassification", + "PlanAcceptanceError", + "PlaylistSnapshot", + "SourcePlaylistEntry", +] + diff --git a/src/symphonia/domain/models.py b/src/symphonia/domain/models.py new file mode 100644 index 0000000..ab1d068 --- /dev/null +++ b/src/symphonia/domain/models.py @@ -0,0 +1,241 @@ +"""Pure domain values used by the first copy-planning slice. + +No provider SDK, HTTP framework, database driver, or Home Assistant package is +allowed in this module. The planner consumes normalized provider-independent +values and emits an immutable plan that can be persisted by an outer layer. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +import hashlib +import json +from typing import Iterable + + +class EntryClassification(str, Enum): + """The six source-entry classifications required by the product model.""" + + READY = "ready" + AMBIGUOUS = "ambiguous" + UNMATCHED = "unmatched" + UNSUPPORTED = "unsupported" + UNAVAILABLE = "unavailable" + INVALID = "invalid" + + +class CopyPolicy(str, Enum): + """How a copy plan treats entries that are not ready to write.""" + + STRICT = "strict" + BEST_EFFORT = "best_effort" + + +class PlanAcceptanceError(ValueError): + """Raised when an immutable plan cannot be accepted safely.""" + + +@dataclass(frozen=True, slots=True) +class SourcePlaylistEntry: + """One occurrence in a captured source playlist snapshot. + + ``position`` is occurrence-specific: two equal provider track IDs at two + positions remain two separate entries. ``target_track_id`` is supplied by + a normalized resolution/projection step and is intentionally opaque. + """ + + occurrence_id: str + position: int + provider_track_id: str + classification: EntryClassification + target_track_id: str | None = None + evidence: tuple[str, ...] = () + reason: str | None = None + + def __post_init__(self) -> None: + if not self.occurrence_id.strip(): + raise ValueError("occurrence_id must not be empty") + if self.position < 0: + raise ValueError("position must be non-negative") + if not self.provider_track_id.strip(): + raise ValueError("provider_track_id must not be empty") + if self.classification is EntryClassification.READY and not self.target_track_id: + raise ValueError("ready entries require a target_track_id") + if self.target_track_id is not None and not self.target_track_id.strip(): + raise ValueError("target_track_id must not be blank") + + +@dataclass(frozen=True, slots=True) +class PlaylistSnapshot: + """Immutable, ordered observation used as the sole planning input.""" + + snapshot_id: str + source_provider: str + source_playlist_id: str + entries: tuple[SourcePlaylistEntry, ...] + + def __post_init__(self) -> None: + for value, field_name in ( + (self.snapshot_id, "snapshot_id"), + (self.source_provider, "source_provider"), + (self.source_playlist_id, "source_playlist_id"), + ): + if not value.strip(): + raise ValueError(f"{field_name} must not be empty") + + positions = [entry.position for entry in self.entries] + occurrence_ids = [entry.occurrence_id for entry in self.entries] + if positions != sorted(positions): + raise ValueError("snapshot entries must be ordered by position") + if len(positions) != len(set(positions)): + raise ValueError("snapshot positions must be unique") + if len(occurrence_ids) != len(set(occurrence_ids)): + raise ValueError("snapshot occurrence IDs must be unique") + + +@dataclass(frozen=True, slots=True) +class CopyPlanEntry: + """Planned disposition for one source occurrence.""" + + occurrence_id: str + position: int + classification: EntryClassification + disposition: str + target_track_id: str | None + reason: str | None + evidence: tuple[str, ...] + + def __post_init__(self) -> None: + if self.disposition not in {"write", "blocked", "omit"}: + raise ValueError("disposition must be write, blocked, or omit") + if self.disposition == "write" and not self.target_track_id: + raise ValueError("write entries require a target_track_id") + + +@dataclass(frozen=True, slots=True) +class CopyPlan: + """Immutable dry-run output with a tamper-evident digest.""" + + source_snapshot_id: str + source_provider: str + source_playlist_id: str + target_provider: str + target_playlist_name: str + target_visibility: str + policy: CopyPolicy + entries: tuple[CopyPlanEntry, ...] + digest: str + + @property + def blocked(self) -> bool: + return any(entry.disposition == "blocked" for entry in self.entries) + + @property + def writable_entries(self) -> tuple[CopyPlanEntry, ...]: + return tuple(entry for entry in self.entries if entry.disposition == "write") + + @property + def omitted_entries(self) -> tuple[CopyPlanEntry, ...]: + return tuple(entry for entry in self.entries if entry.disposition == "omit") + + def accept(self, expected_digest: str) -> "AcceptedCopyPlan": + """Bind execution to this exact plan digest.""" + + if expected_digest != self.digest: + raise PlanAcceptanceError("plan digest does not match the requested acceptance") + if self.blocked: + raise PlanAcceptanceError("plan contains blocked entries") + return AcceptedCopyPlan(plan=self, accepted_digest=self.digest) + + +@dataclass(frozen=True, slots=True) +class AcceptedCopyPlan: + """Accepted immutable plan; execution may only use this value.""" + + plan: CopyPlan + accepted_digest: str + + +def build_copy_plan( + snapshot: PlaylistSnapshot, + *, + target_provider: str, + target_playlist_name: str, + target_visibility: str = "private", + policy: CopyPolicy = CopyPolicy.STRICT, +) -> CopyPlan: + """Build a non-mutating, deterministic copy plan. + + Strict plans block every non-ready occurrence. Best-effort plans explicitly + omit those occurrences while retaining their classification and reason. + No provider write can be performed by this function. + """ + + if not target_provider.strip(): + raise ValueError("target_provider must not be empty") + if not target_playlist_name.strip(): + raise ValueError("target_playlist_name must not be empty") + if not target_visibility.strip(): + raise ValueError("target_visibility must not be empty") + + plan_entries: list[CopyPlanEntry] = [] + for source in snapshot.entries: + ready = source.classification is EntryClassification.READY + disposition = "write" if ready else ("blocked" if policy is CopyPolicy.STRICT else "omit") + reason = source.reason + if not ready and reason is None: + reason = f"source entry is {source.classification.value}" + plan_entries.append( + CopyPlanEntry( + occurrence_id=source.occurrence_id, + position=source.position, + classification=source.classification, + disposition=disposition, + target_track_id=source.target_track_id if ready else None, + reason=reason, + evidence=source.evidence, + ) + ) + + canonical = { + "source_snapshot_id": snapshot.snapshot_id, + "source_provider": snapshot.source_provider, + "source_playlist_id": snapshot.source_playlist_id, + "target_provider": target_provider, + "target_playlist_name": target_playlist_name, + "target_visibility": target_visibility, + "policy": policy.value, + "entries": [ + { + "occurrence_id": entry.occurrence_id, + "position": entry.position, + "classification": entry.classification.value, + "disposition": entry.disposition, + "target_track_id": entry.target_track_id, + "reason": entry.reason, + "evidence": list(entry.evidence), + } + for entry in plan_entries + ], + } + serialized = json.dumps(canonical, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + digest = hashlib.sha256(serialized.encode("utf-8")).hexdigest() + return CopyPlan( + source_snapshot_id=snapshot.snapshot_id, + source_provider=snapshot.source_provider, + source_playlist_id=snapshot.source_playlist_id, + target_provider=target_provider, + target_playlist_name=target_playlist_name, + target_visibility=target_visibility, + policy=policy, + entries=tuple(plan_entries), + digest=digest, + ) + + +def source_entries(entries: Iterable[SourcePlaylistEntry]) -> tuple[SourcePlaylistEntry, ...]: + """Convenience helper for callers constructing a snapshot.""" + + return tuple(entries) + diff --git a/src/symphonia/infrastructure/__init__.py b/src/symphonia/infrastructure/__init__.py new file mode 100644 index 0000000..1af19ce --- /dev/null +++ b/src/symphonia/infrastructure/__init__.py @@ -0,0 +1,11 @@ +"""Infrastructure adapters for the Symphonia core.""" + +from .sqlite_operations import ( + IdempotencyConflict, + OperationNotFound, + OperationRepository, + LeaseConflict, +) + +__all__ = ["IdempotencyConflict", "LeaseConflict", "OperationNotFound", "OperationRepository"] + diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py new file mode 100644 index 0000000..db9cf4d --- /dev/null +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -0,0 +1,275 @@ +"""SQLite persistence for durable operation state. + +This adapter is deliberately small: it proves the transaction/lease contract +needed by the durable-operations SDD before provider handlers and a scheduler +are introduced. The database is the authority; worker memory is not. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime, timedelta, timezone +import json +import sqlite3 +from typing import Any +import uuid + + +def _utc(value: datetime) -> str: + if value.tzinfo is None: + raise ValueError("timestamps must be timezone-aware") + return value.astimezone(timezone.utc).isoformat(timespec="microseconds") + + +def _parse_utc(value: str) -> datetime: + return datetime.fromisoformat(value).astimezone(timezone.utc) + + +class OperationNotFound(LookupError): + pass + + +class IdempotencyConflict(ValueError): + pass + + +class LeaseConflict(RuntimeError): + pass + + +@dataclass(frozen=True, slots=True) +class OperationRecord: + operation_id: str + operation_type: str + state: str + idempotency_key: str + payload: dict[str, Any] + checkpoint: dict[str, Any] + worker_id: str | None + lease_expires_at: datetime | None + next_run_at: datetime | None + created_at: datetime + updated_at: datetime + + +class OperationRepository: + """Transactional operation repository backed by one SQLite database.""" + + def __init__(self, path: str = ":memory:") -> None: + self._connection = sqlite3.connect(path, isolation_level=None) + self._connection.row_factory = sqlite3.Row + self._connection.execute("PRAGMA foreign_keys = ON") + self._connection.execute("PRAGMA busy_timeout = 5000") + self._migrate() + + def close(self) -> None: + self._connection.close() + + def _migrate(self) -> None: + self._connection.executescript( + """ + CREATE TABLE IF NOT EXISTS operations ( + operation_id TEXT PRIMARY KEY, + operation_type TEXT NOT NULL, + state TEXT NOT NULL, + idempotency_key TEXT NOT NULL UNIQUE, + payload_json TEXT NOT NULL, + checkpoint_json TEXT NOT NULL, + worker_id TEXT, + lease_expires_at TEXT, + next_run_at TEXT, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ); + CREATE INDEX IF NOT EXISTS operations_eligibility_idx + ON operations (state, next_run_at, lease_expires_at); + """ + ) + + def create( + self, + *, + operation_type: str, + idempotency_key: str, + payload: dict[str, Any], + now: datetime, + operation_id: str | None = None, + ) -> OperationRecord: + """Create once, or return the identical prior operation by key.""" + + if not operation_type.strip() or not idempotency_key.strip(): + raise ValueError("operation_type and idempotency_key must not be empty") + operation_id = operation_id or str(uuid.uuid4()) + timestamp = _utc(now) + payload_json = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + try: + self._connection.execute( + """ + INSERT INTO operations ( + operation_id, operation_type, state, idempotency_key, + payload_json, checkpoint_json, created_at, updated_at + ) VALUES (?, ?, 'queued', ?, ?, '{}', ?, ?) + """, + (operation_id, operation_type, idempotency_key, payload_json, timestamp, timestamp), + ) + except sqlite3.IntegrityError: + existing = self._by_idempotency(idempotency_key) + if existing is None: + raise + if existing.operation_type != operation_type or existing.payload != payload: + raise IdempotencyConflict("idempotency key is already bound to another operation") + return existing + return self.get(operation_id) + + def get(self, operation_id: str) -> OperationRecord: + row = self._connection.execute( + "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) + ).fetchone() + if row is None: + raise OperationNotFound(operation_id) + return self._record(row) + + def claim( + self, + operation_id: str, + *, + worker_id: str, + now: datetime, + lease_seconds: int = 30, + ) -> OperationRecord: + """Claim queued/retryable work or reclaim a lease that has expired.""" + + if lease_seconds <= 0: + raise ValueError("lease_seconds must be positive") + now_text = _utc(now) + expires_text = _utc(now + timedelta(seconds=lease_seconds)) + self._connection.execute("BEGIN IMMEDIATE") + try: + row = self._connection.execute( + "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) + ).fetchone() + if row is None: + raise OperationNotFound(operation_id) + eligible = row["state"] in {"queued", "retry_scheduled"} + expired = row["state"] == "running" and ( + row["lease_expires_at"] is None or row["lease_expires_at"] <= now_text + ) + if not eligible and not expired: + raise LeaseConflict(f"operation {operation_id} is not eligible for claim") + self._connection.execute( + """ + UPDATE operations + SET state = 'running', worker_id = ?, lease_expires_at = ?, + next_run_at = NULL, updated_at = ? + WHERE operation_id = ? + """, + (worker_id, expires_text, now_text, operation_id), + ) + self._connection.execute("COMMIT") + except Exception: + self._connection.execute("ROLLBACK") + raise + return self.get(operation_id) + + def checkpoint( + self, + operation_id: str, + *, + worker_id: str, + checkpoint: dict[str, Any], + now: datetime, + state: str = "running", + ) -> OperationRecord: + """Persist a checkpoint only for the current, unexpired lease holder.""" + + if state not in {"running", "succeeded", "partial", "failed", "cancelled", "waiting_user"}: + raise ValueError("invalid checkpoint state") + now_text = _utc(now) + checkpoint_json = json.dumps(checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + self._connection.execute("BEGIN IMMEDIATE") + try: + row = self._connection.execute( + "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) + ).fetchone() + if row is None: + raise OperationNotFound(operation_id) + if row["state"] != "running" or row["worker_id"] != worker_id: + raise LeaseConflict("worker does not own a running operation") + if row["lease_expires_at"] is not None and row["lease_expires_at"] <= now_text: + raise LeaseConflict("operation lease has expired") + self._connection.execute( + """ + UPDATE operations + SET state = ?, checkpoint_json = ?, updated_at = ?, + worker_id = CASE WHEN ? IN ('running', 'waiting_user') THEN worker_id ELSE NULL END, + lease_expires_at = CASE WHEN ? IN ('running', 'waiting_user') THEN lease_expires_at ELSE NULL END + WHERE operation_id = ? + """, + (state, checkpoint_json, now_text, state, state, operation_id), + ) + self._connection.execute("COMMIT") + except Exception: + self._connection.execute("ROLLBACK") + raise + return self.get(operation_id) + + def schedule_retry( + self, + operation_id: str, + *, + worker_id: str, + next_run_at: datetime, + checkpoint: dict[str, Any], + now: datetime, + ) -> OperationRecord: + """Release a lease and persist a restart-safe retry time.""" + + now_text = _utc(now) + next_run_text = _utc(next_run_at) + checkpoint_json = json.dumps(checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + self._connection.execute("BEGIN IMMEDIATE") + try: + row = self._connection.execute( + "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) + ).fetchone() + if row is None: + raise OperationNotFound(operation_id) + if row["state"] != "running" or row["worker_id"] != worker_id: + raise LeaseConflict("worker does not own a running operation") + self._connection.execute( + """ + UPDATE operations + SET state = 'retry_scheduled', checkpoint_json = ?, next_run_at = ?, + worker_id = NULL, lease_expires_at = NULL, updated_at = ? + WHERE operation_id = ? + """, + (checkpoint_json, next_run_text, now_text, operation_id), + ) + self._connection.execute("COMMIT") + except Exception: + self._connection.execute("ROLLBACK") + raise + return self.get(operation_id) + + def _by_idempotency(self, idempotency_key: str) -> OperationRecord | None: + row = self._connection.execute( + "SELECT * FROM operations WHERE idempotency_key = ?", (idempotency_key,) + ).fetchone() + return None if row is None else self._record(row) + + @staticmethod + def _record(row: sqlite3.Row) -> OperationRecord: + return OperationRecord( + operation_id=row["operation_id"], + operation_type=row["operation_type"], + state=row["state"], + idempotency_key=row["idempotency_key"], + payload=json.loads(row["payload_json"]), + checkpoint=json.loads(row["checkpoint_json"]), + worker_id=row["worker_id"], + lease_expires_at=None if row["lease_expires_at"] is None else _parse_utc(row["lease_expires_at"]), + next_run_at=None if row["next_run_at"] is None else _parse_utc(row["next_run_at"]), + created_at=_parse_utc(row["created_at"]), + updated_at=_parse_utc(row["updated_at"]), + ) + diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/tests/__init__.py @@ -0,0 +1 @@ + diff --git a/tests/test_copy_planning.py b/tests/test_copy_planning.py new file mode 100644 index 0000000..a78332f --- /dev/null +++ b/tests/test_copy_planning.py @@ -0,0 +1,99 @@ +from __future__ import annotations + +import unittest + +from symphonia.application import CopyPlanningService +from symphonia.domain import ( + CopyPolicy, + EntryClassification, + PlanAcceptanceError, + PlaylistSnapshot, + SourcePlaylistEntry, +) + + +def snapshot(*entries: SourcePlaylistEntry) -> PlaylistSnapshot: + return PlaylistSnapshot( + snapshot_id="snapshot-1", + source_provider="spotify", + source_playlist_id="playlist-1", + entries=tuple(entries), + ) + + +class CopyPlanningTests(unittest.TestCase): + def setUp(self) -> None: + self.service = CopyPlanningService() + + def test_strict_plan_preserves_order_and_duplicate_occurrences(self) -> None: + plan = self.service.plan( + snapshot( + SourcePlaylistEntry("occ-1", 0, "sp-1", EntryClassification.READY, "yt-1"), + SourcePlaylistEntry("occ-2", 1, "sp-1", EntryClassification.READY, "yt-1"), + ), + target_provider="youtube", + target_playlist_name="Rock", + ) + + self.assertFalse(plan.blocked) + self.assertEqual([entry.occurrence_id for entry in plan.writable_entries], ["occ-1", "occ-2"]) + self.assertEqual([entry.position for entry in plan.writable_entries], [0, 1]) + self.assertEqual(plan.writable_entries[0].target_track_id, plan.writable_entries[1].target_track_id) + + def test_strict_plan_blocks_every_non_ready_entry(self) -> None: + plan = self.service.plan( + snapshot( + SourcePlaylistEntry( + "occ-1", + 0, + "sp-1", + EntryClassification.AMBIGUOUS, + reason="two candidates", + ), + SourcePlaylistEntry("occ-2", 1, "sp-2", EntryClassification.UNAVAILABLE), + ), + target_provider="youtube", + target_playlist_name="Rock", + ) + + self.assertTrue(plan.blocked) + self.assertEqual([entry.disposition for entry in plan.entries], ["blocked", "blocked"]) + with self.assertRaises(PlanAcceptanceError): + plan.accept(plan.digest) + + def test_best_effort_retains_classification_and_explicitly_omits(self) -> None: + plan = self.service.plan( + snapshot( + SourcePlaylistEntry("occ-1", 0, "sp-1", EntryClassification.READY, "yt-1"), + SourcePlaylistEntry("occ-2", 1, "sp-2", EntryClassification.UNMATCHED), + ), + target_provider="youtube", + target_playlist_name="Rock", + policy=CopyPolicy.BEST_EFFORT, + ) + + accepted = plan.accept(plan.digest) + self.assertEqual(accepted.accepted_digest, plan.digest) + self.assertEqual(plan.omitted_entries[0].classification, EntryClassification.UNMATCHED) + self.assertEqual(plan.omitted_entries[0].disposition, "omit") + + def test_acceptance_rejects_a_changed_digest(self) -> None: + plan = self.service.plan( + snapshot(SourcePlaylistEntry("occ-1", 0, "sp-1", EntryClassification.READY, "yt-1")), + target_provider="youtube", + target_playlist_name="Rock", + ) + with self.assertRaises(PlanAcceptanceError): + plan.accept("not-the-plan") + + def test_snapshot_rejects_duplicate_positions(self) -> None: + with self.assertRaises(ValueError): + snapshot( + SourcePlaylistEntry("occ-1", 0, "sp-1", EntryClassification.READY, "yt-1"), + SourcePlaylistEntry("occ-2", 0, "sp-2", EntryClassification.READY, "yt-2"), + ) + + +if __name__ == "__main__": + unittest.main() + diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py new file mode 100644 index 0000000..834f042 --- /dev/null +++ b/tests/test_sqlite_operations.py @@ -0,0 +1,133 @@ +from __future__ import annotations + +from datetime import datetime, timedelta, timezone +import unittest + +from symphonia.infrastructure import IdempotencyConflict, LeaseConflict, OperationRepository + + +UTC = timezone.utc + + +class OperationRepositoryTests(unittest.TestCase): + def setUp(self) -> None: + self.repository = OperationRepository() + self.now = datetime(2026, 9, 20, 12, 0, tzinfo=UTC) + + def tearDown(self) -> None: + self.repository.close() + + def test_create_is_idempotent_for_same_payload(self) -> None: + first = self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={"plan_digest": "abc"}, + now=self.now, + ) + second = self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={"plan_digest": "abc"}, + now=self.now + timedelta(seconds=1), + ) + self.assertEqual(first.operation_id, second.operation_id) + self.assertEqual(second.state, "queued") + + def test_same_idempotency_key_cannot_change_intent(self) -> None: + self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={"plan_digest": "abc"}, + now=self.now, + ) + with self.assertRaises(IdempotencyConflict): + self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={"plan_digest": "changed"}, + now=self.now, + ) + + def test_lease_claim_checkpoint_and_terminal_state(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={"plan_digest": "abc"}, + now=self.now, + ) + claimed = self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) + self.assertEqual(claimed.state, "running") + self.assertEqual(claimed.worker_id, "worker-a") + + completed = self.repository.checkpoint( + operation.operation_id, + worker_id="worker-a", + checkpoint={"confirmed": ["occ-1"]}, + now=self.now + timedelta(seconds=1), + state="succeeded", + ) + self.assertEqual(completed.state, "succeeded") + self.assertEqual(completed.checkpoint, {"confirmed": ["occ-1"]}) + self.assertIsNone(completed.worker_id) + + def test_only_lease_owner_can_checkpoint(self) -> None: + operation = self.repository.create( + operation_type="import", + idempotency_key="import-1", + payload={}, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) + with self.assertRaises(LeaseConflict): + self.repository.checkpoint( + operation.operation_id, + worker_id="worker-b", + checkpoint={}, + now=self.now + timedelta(seconds=1), + ) + + def test_expired_lease_can_be_reclaimed_and_retry_survives(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={}, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now, lease_seconds=5) + retry = self.repository.schedule_retry( + operation.operation_id, + worker_id="worker-a", + next_run_at=self.now + timedelta(minutes=1), + checkpoint={"last": "timeout"}, + now=self.now + timedelta(seconds=1), + ) + self.assertEqual(retry.state, "retry_scheduled") + self.assertEqual(retry.next_run_at, self.now + timedelta(minutes=1)) + self.assertIsNone(retry.worker_id) + + reclaimed = self.repository.claim( + operation.operation_id, + worker_id="worker-b", + now=self.now + timedelta(minutes=2), + ) + self.assertEqual(reclaimed.worker_id, "worker-b") + + def test_expired_running_lease_can_be_reclaimed(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={}, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now, lease_seconds=5) + reclaimed = self.repository.claim( + operation.operation_id, + worker_id="worker-b", + now=self.now + timedelta(seconds=6), + ) + self.assertEqual(reclaimed.worker_id, "worker-b") + + +if __name__ == "__main__": + unittest.main() + From 8ee99416cfcdc7cb807a49b875a95e26d612698f Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 16:30:35 +0200 Subject: [PATCH 003/167] feat: add runtime health and readiness surface --- src/symphonia/__main__.py | 33 ++++++++++++++ src/symphonia/runtime/__init__.py | 6 +++ src/symphonia/runtime/http.py | 73 +++++++++++++++++++++++++++++++ tests/test_runtime_http.py | 36 +++++++++++++++ 4 files changed, 148 insertions(+) create mode 100644 src/symphonia/__main__.py create mode 100644 src/symphonia/runtime/__init__.py create mode 100644 src/symphonia/runtime/http.py create mode 100644 tests/test_runtime_http.py diff --git a/src/symphonia/__main__.py b/src/symphonia/__main__.py new file mode 100644 index 0000000..6bbd819 --- /dev/null +++ b/src/symphonia/__main__.py @@ -0,0 +1,33 @@ +"""Command-line entry point for the dependency-free runtime foundation.""" + +from __future__ import annotations + +import argparse +import os + +from symphonia.runtime import create_server + + +def main() -> None: + parser = argparse.ArgumentParser(description="Run the Symphonia runtime foundation") + parser.add_argument("--host", default=os.getenv("SYMPHONIA_HOST", "127.0.0.1")) + parser.add_argument("--port", type=int, default=int(os.getenv("SYMPHONIA_PORT", "8099"))) + parser.add_argument( + "--database", + default=os.getenv("SYMPHONIA_DATABASE", "./symphonia.sqlite3"), + help="SQLite path; Home Assistant App deployments should use /data/symphonia.sqlite3", + ) + args = parser.parse_args() + server = create_server(args.host, args.port, args.database) + try: + server.serve_forever() + except KeyboardInterrupt: + pass + finally: + server.server_close() + server.repository.close() + + +if __name__ == "__main__": + main() + diff --git a/src/symphonia/runtime/__init__.py b/src/symphonia/runtime/__init__.py new file mode 100644 index 0000000..1b5256d --- /dev/null +++ b/src/symphonia/runtime/__init__.py @@ -0,0 +1,6 @@ +"""Minimal process runtime and health endpoints.""" + +from .http import create_server + +__all__ = ["create_server"] + diff --git a/src/symphonia/runtime/http.py b/src/symphonia/runtime/http.py new file mode 100644 index 0000000..6fd543c --- /dev/null +++ b/src/symphonia/runtime/http.py @@ -0,0 +1,73 @@ +"""Dependency-free HTTP health surface for the first runtime slice.""" + +from __future__ import annotations + +from http.server import BaseHTTPRequestHandler, HTTPServer +import json +from typing import Any + +from symphonia import __version__ +from symphonia.infrastructure.sqlite_operations import OperationRepository + + +class SymphoniaHTTPServer(HTTPServer): + allow_reuse_address = True + + def __init__(self, address: tuple[str, int], repository: OperationRepository) -> None: + super().__init__(address, SymphoniaRequestHandler) + self.repository = repository + self.service_version = __version__ + + +class SymphoniaRequestHandler(BaseHTTPRequestHandler): + """Only health/readiness/version are exposed until the API SDD is ready.""" + + server: SymphoniaHTTPServer + + def do_GET(self) -> None: # noqa: N802 - stdlib handler API + status, payload = route_get(self.path, self.server.repository, self.server.service_version) + self._json(status, payload) + + def _json(self, status: int, payload: dict[str, Any]) -> None: + body = json.dumps(payload, ensure_ascii=False, sort_keys=True).encode("utf-8") + self.send_response(status) + self.send_header("Content-Type", "application/json; charset=utf-8") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, format: str, *args: object) -> None: + # Keep the first runtime quiet; structured logging belongs to the + # observability adapter and must not include request payloads by default. + return + + +def create_server(host: str = "127.0.0.1", port: int = 8099, database_path: str = ":memory:") -> SymphoniaHTTPServer: + """Create a server with an already-migrated durable operation store.""" + + repository = OperationRepository(database_path) + return SymphoniaHTTPServer((host, port), repository) + + +def route_get(path: str, repository: OperationRepository, service_version: str = __version__) -> tuple[int, dict[str, Any]]: + """Resolve a GET request without opening a socket. + + Keeping this decision pure-ish makes health/readiness contract tests work + in restricted CI environments and prevents a network permission from being + mistaken for application readiness. + """ + + if path == "/health": + return 200, {"service": "symphonia", "status": "ok", "version": service_version} + if path == "/ready": + try: + healthy = repository.healthcheck() + except Exception: # readiness must fail closed without exposing internals + healthy = False + return (200, {"service": "symphonia", "status": "ready"}) if healthy else ( + 503, + {"service": "symphonia", "status": "not_ready"}, + ) + if path == "/version": + return 200, {"service": "symphonia", "version": service_version} + return 404, {"error": "not_found"} diff --git a/tests/test_runtime_http.py b/tests/test_runtime_http.py new file mode 100644 index 0000000..cf40fce --- /dev/null +++ b/tests/test_runtime_http.py @@ -0,0 +1,36 @@ +from __future__ import annotations + +import unittest + +from symphonia.infrastructure import OperationRepository +from symphonia.runtime.http import route_get + + +class RuntimeHTTPTests(unittest.TestCase): + def setUp(self) -> None: + self.repository = OperationRepository() + + def tearDown(self) -> None: + self.repository.close() + + def test_health_and_readiness_are_public_runtime_checks(self) -> None: + status, payload = route_get("/health", self.repository) + self.assertEqual(status, 200) + self.assertEqual(payload["status"], "ok") + + status, payload = route_get("/ready", self.repository) + self.assertEqual(status, 200) + self.assertEqual(payload["status"], "ready") + + def test_version_and_unknown_routes(self) -> None: + status, payload = route_get("/version", self.repository) + self.assertEqual(status, 200) + self.assertIn("version", payload) + + status, payload = route_get("/admin", self.repository) + self.assertEqual(status, 404) + self.assertEqual(payload, {"error": "not_found"}) + + +if __name__ == "__main__": + unittest.main() From d2a55a24ae43f975cc09240a5de1c20e6c028dc7 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 16:30:50 +0200 Subject: [PATCH 004/167] fix: expose SQLite readiness check --- src/symphonia/infrastructure/sqlite_operations.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index db9cf4d..3bdc861 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -56,7 +56,7 @@ class OperationRepository: """Transactional operation repository backed by one SQLite database.""" def __init__(self, path: str = ":memory:") -> None: - self._connection = sqlite3.connect(path, isolation_level=None) + self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) self._connection.row_factory = sqlite3.Row self._connection.execute("PRAGMA foreign_keys = ON") self._connection.execute("PRAGMA busy_timeout = 5000") @@ -65,6 +65,12 @@ def __init__(self, path: str = ":memory:") -> None: def close(self) -> None: self._connection.close() + def healthcheck(self) -> bool: + """Return whether the migrated store can answer a basic read.""" + + row = self._connection.execute("SELECT 1 AS healthy").fetchone() + return row is not None and row["healthy"] == 1 + def _migrate(self) -> None: self._connection.executescript( """ @@ -272,4 +278,3 @@ def _record(row: sqlite3.Row) -> OperationRecord: created_at=_parse_utc(row["created_at"]), updated_at=_parse_utc(row["updated_at"]), ) - From 4a06d129260de27aebe86fb9e2ec530aca0ff8b2 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 16:33:07 +0200 Subject: [PATCH 005/167] feat: add normalized provider import contracts --- docs/development/implementation-baseline.md | 2 +- src/symphonia/providers/__init__.py | 31 +++++ src/symphonia/providers/contracts.py | 138 ++++++++++++++++++++ src/symphonia/providers/importing.py | 116 ++++++++++++++++ tests/test_provider_import.py | 88 +++++++++++++ 5 files changed, 374 insertions(+), 1 deletion(-) create mode 100644 src/symphonia/providers/__init__.py create mode 100644 src/symphonia/providers/contracts.py create mode 100644 src/symphonia/providers/importing.py create mode 100644 tests/test_provider_import.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 03a1b65..bb4426e 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -10,6 +10,7 @@ The first implementation increment is intentionally narrower than any provider o - Python 3.11+ package under `src/symphonia`. - Dependency-free domain values for ordered playlist snapshots, the six product entry classifications, copy policies, immutable copy plans, and acceptance digests. - Dependency-free SQLite operation repository proving idempotent creation, time-bounded worker leases, checkpoint ownership, retry scheduling, and expired-lease recovery. +- Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - Deterministic `unittest` coverage under `tests/`. The package is an implementation foundation, not a claim that the corresponding capability SDDs are complete. Provider OAuth, provider API adapters, user authentication, Ingress, UI, migrations beyond the initial schema, backup/restore, and operation handlers remain unimplemented and blocked by their SDD decisions. @@ -30,4 +31,3 @@ The command must run without network access, provider accounts, Home Assistant, - Application services orchestrate domain values and do not call provider SDKs directly. - SQLite is an adapter behind the operation repository; it is not exposed as a domain concept. - The initial Python choice is a reversible implementation baseline, not a final product-stack decision. Any external dependency or deployment commitment requires an SDD/ADR update and tests. - diff --git a/src/symphonia/providers/__init__.py b/src/symphonia/providers/__init__.py new file mode 100644 index 0000000..86412de --- /dev/null +++ b/src/symphonia/providers/__init__.py @@ -0,0 +1,31 @@ +"""Normalized provider contracts and import collection helpers.""" + +from .contracts import ( + AccessBasis, + Capability, + MediaKind, + ProviderAdapter, + ProviderCapabilities, + ProviderManifest, + ProviderObjectRef, + ProviderPlaylistEntry, + ProviderPlaylistPage, +) +from .importing import CollectionImportResult, ImportIssue, collect_playlist_pages, to_playlist_snapshot + +__all__ = [ + "AccessBasis", + "Capability", + "CollectionImportResult", + "ImportIssue", + "MediaKind", + "ProviderAdapter", + "ProviderCapabilities", + "ProviderManifest", + "ProviderObjectRef", + "ProviderPlaylistEntry", + "ProviderPlaylistPage", + "collect_playlist_pages", + "to_playlist_snapshot", +] + diff --git a/src/symphonia/providers/contracts.py b/src/symphonia/providers/contracts.py new file mode 100644 index 0000000..b106c73 --- /dev/null +++ b/src/symphonia/providers/contracts.py @@ -0,0 +1,138 @@ +"""Provider-independent ports and normalized provider values. + +Provider adapters translate SDK/API payloads into these values. The domain and +application layers never need to import provider SDK classes or infer identity +from URL shapes. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from typing import Iterable, Protocol + + +class AccessBasis(str, Enum): + OFFICIAL = "official" + UNOFFICIAL = "unofficial" + BEST_EFFORT = "best_effort" + + +class Capability(str, Enum): + READ_PLAYLISTS = "read_playlists" + READ_LIBRARY = "read_library" + SEARCH_TRACKS = "search_tracks" + CREATE_PLAYLIST = "create_playlist" + ADD_PLAYLIST_ENTRIES = "add_playlist_entries" + REORDER_PLAYLIST = "reorder_playlist" + DELETE_PLAYLIST = "delete_playlist" + + +class MediaKind(str, Enum): + TRACK = "track" + VIDEO = "video" + PODCAST = "podcast" + UNKNOWN = "unknown" + + +@dataclass(frozen=True, slots=True) +class ProviderObjectRef: + """Opaque external identity with the required type/namespace boundary.""" + + provider: str + object_type: str + object_id: str + namespace: str + + def __post_init__(self) -> None: + for value, field_name in ( + (self.provider, "provider"), + (self.object_type, "object_type"), + (self.object_id, "object_id"), + (self.namespace, "namespace"), + ): + if not value.strip(): + raise ValueError(f"{field_name} must not be empty") + + @property + def external_key(self) -> tuple[str, str, str, str]: + return (self.provider, self.namespace, self.object_type, self.object_id) + + +@dataclass(frozen=True, slots=True) +class ProviderManifest: + provider: str + display_name: str + access_basis: AccessBasis + maturity: str + support_level: str + upstream_dependencies: tuple[str, ...] = () + reviewed_on: str | None = None + + def __post_init__(self) -> None: + if not self.provider.strip() or not self.display_name.strip(): + raise ValueError("provider and display_name must not be empty") + if not self.maturity.strip() or not self.support_level.strip(): + raise ValueError("maturity and support_level must not be empty") + + +@dataclass(frozen=True, slots=True) +class ProviderCapabilities: + """Effective capabilities after adapter/connection/object/health checks.""" + + enabled: frozenset[Capability] + evidence_version: str + observed_at: str + + def supports(self, capability: Capability) -> bool: + return capability in self.enabled + + +@dataclass(frozen=True, slots=True) +class ProviderPlaylistEntry: + occurrence_id: str + position: int + track: ProviderObjectRef + media_kind: MediaKind + title: str | None = None + available: bool = True + source_added_at: str | None = None + + def __post_init__(self) -> None: + if not self.occurrence_id.strip(): + raise ValueError("occurrence_id must not be empty") + if self.position < 0: + raise ValueError("position must be non-negative") + + +@dataclass(frozen=True, slots=True) +class ProviderPlaylistPage: + playlist: ProviderObjectRef + entries: tuple[ProviderPlaylistEntry, ...] + cursor: str | None + next_cursor: str | None + complete: bool + revision: str | None = None + + def __post_init__(self) -> None: + if self.playlist.object_type != "playlist": + raise ValueError("playlist page requires a playlist object reference") + positions = [entry.position for entry in self.entries] + if positions != sorted(positions): + raise ValueError("page entries must be ordered by position") + if self.complete and self.next_cursor is not None: + raise ValueError("a complete page cannot have a next_cursor") + + +class ProviderAdapter(Protocol): + """Semantic adapter port used by application use cases.""" + + @property + def manifest(self) -> ProviderManifest: ... + + def capabilities(self, connection_id: str) -> ProviderCapabilities: ... + + def read_playlist_pages( + self, connection_id: str, playlist: ProviderObjectRef, cursor: str | None = None + ) -> Iterable[ProviderPlaylistPage]: ... + diff --git a/src/symphonia/providers/importing.py b/src/symphonia/providers/importing.py new file mode 100644 index 0000000..551e2cf --- /dev/null +++ b/src/symphonia/providers/importing.py @@ -0,0 +1,116 @@ +"""Pure import collection and provider-to-domain snapshot conversion.""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum + +from symphonia.domain.models import EntryClassification, PlaylistSnapshot, SourcePlaylistEntry + +from .contracts import ProviderPlaylistEntry, ProviderPlaylistPage + + +class ImportIssue(str, Enum): + NO_PAGES = "no_pages" + MISSING_CONTINUATION = "missing_continuation" + REPEATED_CURSOR = "repeated_cursor" + CONFLICTING_OCCURRENCE = "conflicting_occurrence" + DUPLICATE_OCCURRENCE = "duplicate_occurrence" + INCONSISTENT_PLAYLIST = "inconsistent_playlist" + + +@dataclass(frozen=True, slots=True) +class CollectionImportResult: + playlist: str + provider: str + entries: tuple[ProviderPlaylistEntry, ...] + complete: bool + issues: tuple[ImportIssue, ...] + revision: str | None + + +def collect_playlist_pages(pages: list[ProviderPlaylistPage] | tuple[ProviderPlaylistPage, ...]) -> CollectionImportResult: + """Collect pages without silently truncating or collapsing occurrences. + + Exact repeated occurrences caused by an overlapping page are de-duplicated + by occurrence ID and recorded as an issue. A conflicting repeat, repeated + cursor, missing continuation, or mixed playlist makes the result incomplete. + """ + + if not pages: + return CollectionImportResult("", "", (), False, (ImportIssue.NO_PAGES,), None) + + first = pages[0].playlist + issues: list[ImportIssue] = [] + entries_by_id: dict[str, ProviderPlaylistEntry] = {} + seen_cursors: set[str] = set() + seen_next_cursors: set[str] = set() + complete = True + for index, page in enumerate(pages): + if page.playlist.external_key != first.external_key: + issues.append(ImportIssue.INCONSISTENT_PLAYLIST) + complete = False + if page.cursor is not None and page.cursor in seen_cursors: + issues.append(ImportIssue.REPEATED_CURSOR) + complete = False + if page.cursor is not None: + seen_cursors.add(page.cursor) + if page.next_cursor is not None and page.next_cursor in seen_next_cursors: + issues.append(ImportIssue.REPEATED_CURSOR) + complete = False + if page.next_cursor is not None: + seen_next_cursors.add(page.next_cursor) + + for entry in page.entries: + previous = entries_by_id.get(entry.occurrence_id) + if previous is None: + entries_by_id[entry.occurrence_id] = entry + elif previous == entry: + issues.append(ImportIssue.DUPLICATE_OCCURRENCE) + else: + issues.append(ImportIssue.CONFLICTING_OCCURRENCE) + complete = False + + if index < len(pages) - 1 and page.next_cursor is None: + issues.append(ImportIssue.MISSING_CONTINUATION) + complete = False + if index == len(pages) - 1: + if not page.complete or page.next_cursor is not None: + complete = False + if not page.complete: + issues.append(ImportIssue.MISSING_CONTINUATION) + + ordered = tuple(sorted(entries_by_id.values(), key=lambda entry: entry.position)) + return CollectionImportResult( + playlist=first.object_id, + provider=first.provider, + entries=ordered, + complete=complete and not any(issue in {ImportIssue.CONFLICTING_OCCURRENCE, ImportIssue.REPEATED_CURSOR} for issue in issues), + issues=tuple(dict.fromkeys(issues)), + revision=pages[-1].revision, + ) + + +def to_playlist_snapshot(result: CollectionImportResult, snapshot_id: str) -> PlaylistSnapshot: + """Convert a collected provider result into copy-planner input. + + Import does not perform identity resolution. Available entries therefore + start as ``unmatched``; unavailable entries remain explicit and visible. + """ + + entries = tuple( + SourcePlaylistEntry( + occurrence_id=entry.occurrence_id, + position=entry.position, + provider_track_id=entry.track.object_id, + classification=EntryClassification.UNMATCHED if entry.available else EntryClassification.UNAVAILABLE, + reason=None if entry.available else "provider reported item unavailable", + ) + for entry in result.entries + ) + return PlaylistSnapshot( + snapshot_id=snapshot_id, + source_provider=result.provider, + source_playlist_id=result.playlist, + entries=entries, + ) diff --git a/tests/test_provider_import.py b/tests/test_provider_import.py new file mode 100644 index 0000000..f856fd0 --- /dev/null +++ b/tests/test_provider_import.py @@ -0,0 +1,88 @@ +from __future__ import annotations + +import unittest + +from symphonia.domain import EntryClassification +from symphonia.providers import ( + AccessBasis, + ImportIssue, + MediaKind, + ProviderManifest, + ProviderObjectRef, + ProviderPlaylistEntry, + ProviderPlaylistPage, + collect_playlist_pages, + to_playlist_snapshot, +) + + +def playlist_ref(provider: str = "spotify") -> ProviderObjectRef: + return ProviderObjectRef(provider, "playlist", "playlist-1", "connection-1") + + +def entry(occurrence_id: str, position: int, track_id: str, available: bool = True) -> ProviderPlaylistEntry: + return ProviderPlaylistEntry( + occurrence_id=occurrence_id, + position=position, + track=ProviderObjectRef("spotify", "track", track_id, "connection-1"), + media_kind=MediaKind.TRACK, + available=available, + ) + + +class ProviderContractTests(unittest.TestCase): + def test_external_identity_is_namespaced_by_type_and_connection(self) -> None: + same_upstream_id = ProviderObjectRef("spotify", "track", "same", "connection-a") + another_connection = ProviderObjectRef("spotify", "track", "same", "connection-b") + another_type = ProviderObjectRef("spotify", "video", "same", "connection-a") + self.assertNotEqual(same_upstream_id.external_key, another_connection.external_key) + self.assertNotEqual(same_upstream_id.external_key, another_type.external_key) + + def test_manifest_discloses_access_basis(self) -> None: + manifest = ProviderManifest("spotify", "Spotify", AccessBasis.OFFICIAL, "beta", "limited") + self.assertEqual(manifest.access_basis, AccessBasis.OFFICIAL) + + def test_complete_pages_preserve_order_and_duplicate_occurrences(self) -> None: + result = collect_playlist_pages( + [ + ProviderPlaylistPage(playlist_ref(), (entry("occ-1", 0, "track-1"),), None, "cursor-2", False), + ProviderPlaylistPage( + playlist_ref(), + (entry("occ-2", 1, "track-1"), entry("occ-3", 2, "track-2")), + "cursor-2", + None, + True, + ), + ] + ) + self.assertTrue(result.complete) + self.assertEqual([item.occurrence_id for item in result.entries], ["occ-1", "occ-2", "occ-3"]) + self.assertEqual(result.entries[0].track.object_id, result.entries[1].track.object_id) + + def test_incomplete_page_does_not_look_complete(self) -> None: + result = collect_playlist_pages( + [ProviderPlaylistPage(playlist_ref(), (entry("occ-1", 0, "track-1"),), None, "cursor-2", False)] + ) + self.assertFalse(result.complete) + self.assertEqual(result.issues, (ImportIssue.MISSING_CONTINUATION,)) + + def test_repeated_cursor_and_conflicting_occurrence_are_incomplete(self) -> None: + first = ProviderPlaylistPage(playlist_ref(), (entry("occ-1", 0, "track-1"),), None, "cursor-2", False) + second = ProviderPlaylistPage(playlist_ref(), (entry("occ-1", 0, "track-other"),), "cursor-2", "cursor-2", False) + result = collect_playlist_pages([first, second]) + self.assertFalse(result.complete) + self.assertIn(ImportIssue.CONFLICTING_OCCURRENCE, result.issues) + self.assertIn(ImportIssue.REPEATED_CURSOR, result.issues) + + def test_snapshot_keeps_unavailable_entries_visible(self) -> None: + result = collect_playlist_pages( + [ProviderPlaylistPage(playlist_ref(), (entry("occ-1", 0, "track-1", False),), None, None, True)] + ) + snapshot = to_playlist_snapshot(result, "snapshot-1") + self.assertEqual(snapshot.entries[0].classification, EntryClassification.UNAVAILABLE) + self.assertEqual(snapshot.entries[0].occurrence_id, "occ-1") + + +if __name__ == "__main__": + unittest.main() + From adf8ca953320d56aea1f3a609f55d291de492c95 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 16:33:55 +0200 Subject: [PATCH 006/167] fix: enforce retry timing and lease renewal --- .../infrastructure/sqlite_operations.py | 41 ++++++++++++++++++- tests/test_sqlite_operations.py | 39 +++++++++++++++++- 2 files changed, 78 insertions(+), 2 deletions(-) diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 3bdc861..f7c084b 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -156,7 +156,11 @@ def claim( ).fetchone() if row is None: raise OperationNotFound(operation_id) - eligible = row["state"] in {"queued", "retry_scheduled"} + eligible = row["state"] == "queued" or ( + row["state"] == "retry_scheduled" + and row["next_run_at"] is not None + and row["next_run_at"] <= now_text + ) expired = row["state"] == "running" and ( row["lease_expires_at"] is None or row["lease_expires_at"] <= now_text ) @@ -177,6 +181,41 @@ def claim( raise return self.get(operation_id) + def renew_lease( + self, + operation_id: str, + *, + worker_id: str, + now: datetime, + lease_seconds: int = 30, + ) -> OperationRecord: + """Extend a healthy lease; an expired owner cannot resurrect it.""" + + if lease_seconds <= 0: + raise ValueError("lease_seconds must be positive") + now_text = _utc(now) + expires_text = _utc(now + timedelta(seconds=lease_seconds)) + self._connection.execute("BEGIN IMMEDIATE") + try: + row = self._connection.execute( + "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) + ).fetchone() + if row is None: + raise OperationNotFound(operation_id) + if row["state"] != "running" or row["worker_id"] != worker_id: + raise LeaseConflict("worker does not own a running operation") + if row["lease_expires_at"] is not None and row["lease_expires_at"] <= now_text: + raise LeaseConflict("operation lease has expired") + self._connection.execute( + "UPDATE operations SET lease_expires_at = ?, updated_at = ? WHERE operation_id = ?", + (expires_text, now_text, operation_id), + ) + self._connection.execute("COMMIT") + except Exception: + self._connection.execute("ROLLBACK") + raise + return self.get(operation_id) + def checkpoint( self, operation_id: str, diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 834f042..cd5080a 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -127,7 +127,44 @@ def test_expired_running_lease_can_be_reclaimed(self) -> None: ) self.assertEqual(reclaimed.worker_id, "worker-b") + def test_retry_cannot_be_claimed_before_its_scheduled_time(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={}, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) + self.repository.schedule_retry( + operation.operation_id, + worker_id="worker-a", + next_run_at=self.now + timedelta(minutes=1), + checkpoint={"last": "rate_limited"}, + now=self.now + timedelta(seconds=1), + ) + with self.assertRaises(LeaseConflict): + self.repository.claim( + operation.operation_id, + worker_id="worker-b", + now=self.now + timedelta(seconds=30), + ) + + def test_healthy_worker_can_renew_lease(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={}, + now=self.now, + ) + claimed = self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now, lease_seconds=5) + renewed = self.repository.renew_lease( + operation.operation_id, + worker_id="worker-a", + now=self.now + timedelta(seconds=1), + lease_seconds=60, + ) + self.assertGreater(renewed.lease_expires_at, claimed.lease_expires_at) + if __name__ == "__main__": unittest.main() - From 19eea50bbf1513bd7dcc52ca22a9af9b91755889 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 16:35:10 +0200 Subject: [PATCH 007/167] feat: persist digest-bound copy plans --- docs/development/implementation-baseline.md | 1 + src/symphonia/infrastructure/__init__.py | 12 +- src/symphonia/infrastructure/sqlite_plans.py | 148 +++++++++++++++++++ tests/test_sqlite_plans.py | 73 +++++++++ 4 files changed, 232 insertions(+), 2 deletions(-) create mode 100644 src/symphonia/infrastructure/sqlite_plans.py create mode 100644 tests/test_sqlite_plans.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index bb4426e..5c8cf67 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -11,6 +11,7 @@ The first implementation increment is intentionally narrower than any provider o - Dependency-free domain values for ordered playlist snapshots, the six product entry classifications, copy policies, immutable copy plans, and acceptance digests. - Dependency-free SQLite operation repository proving idempotent creation, time-bounded worker leases, checkpoint ownership, retry scheduling, and expired-lease recovery. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. +- SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Deterministic `unittest` coverage under `tests/`. The package is an implementation foundation, not a claim that the corresponding capability SDDs are complete. Provider OAuth, provider API adapters, user authentication, Ingress, UI, migrations beyond the initial schema, backup/restore, and operation handlers remain unimplemented and blocked by their SDD decisions. diff --git a/src/symphonia/infrastructure/__init__.py b/src/symphonia/infrastructure/__init__.py index 1af19ce..26183fd 100644 --- a/src/symphonia/infrastructure/__init__.py +++ b/src/symphonia/infrastructure/__init__.py @@ -6,6 +6,14 @@ OperationRepository, LeaseConflict, ) +from .sqlite_plans import CopyPlanNotFound, CopyPlanRepository, StoredCopyPlan -__all__ = ["IdempotencyConflict", "LeaseConflict", "OperationNotFound", "OperationRepository"] - +__all__ = [ + "CopyPlanNotFound", + "CopyPlanRepository", + "IdempotencyConflict", + "LeaseConflict", + "OperationNotFound", + "OperationRepository", + "StoredCopyPlan", +] diff --git a/src/symphonia/infrastructure/sqlite_plans.py b/src/symphonia/infrastructure/sqlite_plans.py new file mode 100644 index 0000000..25e3ac2 --- /dev/null +++ b/src/symphonia/infrastructure/sqlite_plans.py @@ -0,0 +1,148 @@ +"""Durable storage for immutable copy plans and their acceptance.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime, timezone +import json +import sqlite3 +from typing import Any + +from symphonia.domain.models import ( + CopyPlan, + CopyPlanEntry, + CopyPolicy, + EntryClassification, + PlanAcceptanceError, +) + + +def _utc(value: datetime) -> str: + if value.tzinfo is None: + raise ValueError("timestamps must be timezone-aware") + return value.astimezone(timezone.utc).isoformat(timespec="microseconds") + + +def _parse_utc(value: str) -> datetime: + return datetime.fromisoformat(value).astimezone(timezone.utc) + + +class CopyPlanNotFound(LookupError): + pass + + +@dataclass(frozen=True, slots=True) +class StoredCopyPlan: + plan: CopyPlan + accepted_at: datetime | None + + +class CopyPlanRepository: + """A small SQLite adapter that never mutates a plan after creation.""" + + def __init__(self, path: str = ":memory:") -> None: + self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) + self._connection.row_factory = sqlite3.Row + self._migrate() + + def close(self) -> None: + self._connection.close() + + def _migrate(self) -> None: + self._connection.executescript( + """ + CREATE TABLE IF NOT EXISTS copy_plans ( + digest TEXT PRIMARY KEY, + plan_json TEXT NOT NULL, + created_at TEXT NOT NULL, + accepted_at TEXT + ); + """ + ) + + def save(self, plan: CopyPlan, *, now: datetime) -> StoredCopyPlan: + payload = _serialize(plan) + plan_json = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + self._connection.execute( + "INSERT OR IGNORE INTO copy_plans (digest, plan_json, created_at) VALUES (?, ?, ?)", + (plan.digest, plan_json, _utc(now)), + ) + stored = self.get(plan.digest) + if stored.plan != plan: + raise ValueError("digest collision or attempted mutation of an existing plan") + return stored + + def get(self, digest: str) -> StoredCopyPlan: + row = self._connection.execute( + "SELECT * FROM copy_plans WHERE digest = ?", (digest,) + ).fetchone() + if row is None: + raise CopyPlanNotFound(digest) + return StoredCopyPlan( + plan=_deserialize(json.loads(row["plan_json"])), + accepted_at=None if row["accepted_at"] is None else _parse_utc(row["accepted_at"]), + ) + + def accept(self, digest: str, *, expected_digest: str, now: datetime) -> StoredCopyPlan: + stored = self.get(digest) + if stored.plan.digest != expected_digest: + raise PlanAcceptanceError("accepted digest does not match the stored plan") + # Reuse domain validation so strict blocked plans and digest changes + # cannot be bypassed by persistence code. + stored.plan.accept(expected_digest) + self._connection.execute( + "UPDATE copy_plans SET accepted_at = COALESCE(accepted_at, ?) WHERE digest = ?", + (_utc(now), digest), + ) + return self.get(digest) + + +def _serialize(plan: CopyPlan) -> dict[str, Any]: + return { + "source_snapshot_id": plan.source_snapshot_id, + "source_provider": plan.source_provider, + "source_playlist_id": plan.source_playlist_id, + "target_provider": plan.target_provider, + "target_playlist_name": plan.target_playlist_name, + "target_visibility": plan.target_visibility, + "policy": plan.policy.value, + "entries": [ + { + "occurrence_id": entry.occurrence_id, + "position": entry.position, + "classification": entry.classification.value, + "disposition": entry.disposition, + "target_track_id": entry.target_track_id, + "reason": entry.reason, + "evidence": list(entry.evidence), + } + for entry in plan.entries + ], + "digest": plan.digest, + } + + +def _deserialize(payload: dict[str, Any]) -> CopyPlan: + return CopyPlan( + source_snapshot_id=payload["source_snapshot_id"], + source_provider=payload["source_provider"], + source_playlist_id=payload["source_playlist_id"], + target_provider=payload["target_provider"], + target_playlist_name=payload["target_playlist_name"], + target_visibility=payload["target_visibility"], + policy=CopyPolicy(payload["policy"]), + entries=tuple( + CopyPlanEntry( + occurrence_id=entry["occurrence_id"], + position=entry["position"], + classification=EntryClassification(entry["classification"]), + disposition=entry["disposition"], + target_track_id=entry["target_track_id"], + reason=entry["reason"], + evidence=tuple(entry["evidence"]), + ) + for entry in payload["entries"] + ), + digest=payload["digest"], + ) + diff --git a/tests/test_sqlite_plans.py b/tests/test_sqlite_plans.py new file mode 100644 index 0000000..47f403c --- /dev/null +++ b/tests/test_sqlite_plans.py @@ -0,0 +1,73 @@ +from __future__ import annotations + +from datetime import datetime, timezone +import unittest + +from symphonia.domain import CopyPolicy, EntryClassification, PlanAcceptanceError, PlaylistSnapshot, SourcePlaylistEntry +from symphonia.domain.models import build_copy_plan +from symphonia.infrastructure import CopyPlanRepository + + +class CopyPlanRepositoryTests(unittest.TestCase): + now = datetime(2026, 9, 20, 12, 0, tzinfo=timezone.utc) + + def setUp(self) -> None: + self.repository = CopyPlanRepository() + + def tearDown(self) -> None: + self.repository.close() + + def ready_plan(self): + snapshot = PlaylistSnapshot( + snapshot_id="snapshot-1", + source_provider="spotify", + source_playlist_id="playlist-1", + entries=(SourcePlaylistEntry("occ-1", 0, "sp-1", EntryClassification.READY, "yt-1"),), + ) + return build_copy_plan( + snapshot, + target_provider="youtube", + target_playlist_name="Rock", + policy=CopyPolicy.STRICT, + ) + + def test_save_and_reload_preserves_immutable_plan_and_digest(self) -> None: + plan = self.ready_plan() + self.repository.save(plan, now=self.now) + stored = self.repository.get(plan.digest) + self.assertEqual(stored.plan, plan) + self.assertIsNone(stored.accepted_at) + + def test_acceptance_is_durable_and_idempotent(self) -> None: + plan = self.ready_plan() + self.repository.save(plan, now=self.now) + first = self.repository.accept(plan.digest, expected_digest=plan.digest, now=self.now) + second = self.repository.accept( + plan.digest, + expected_digest=plan.digest, + now=self.now.replace(hour=13), + ) + self.assertEqual(first.accepted_at, second.accepted_at) + self.assertIsNotNone(first.accepted_at) + + def test_blocked_plan_cannot_be_accepted_through_repository(self) -> None: + snapshot = PlaylistSnapshot( + snapshot_id="snapshot-1", + source_provider="spotify", + source_playlist_id="playlist-1", + entries=(SourcePlaylistEntry("occ-1", 0, "sp-1", EntryClassification.AMBIGUOUS),), + ) + plan = build_copy_plan( + snapshot, + target_provider="youtube", + target_playlist_name="Rock", + policy=CopyPolicy.STRICT, + ) + self.repository.save(plan, now=self.now) + with self.assertRaises(PlanAcceptanceError): + self.repository.accept(plan.digest, expected_digest=plan.digest, now=self.now) + + +if __name__ == "__main__": + unittest.main() + From 7030d17751ff949e82731e07a2585c65f14111ec Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 16:36:43 +0200 Subject: [PATCH 008/167] feat: persist explainable identity decisions --- docs/development/implementation-baseline.md | 1 + src/symphonia/identity/__init__.py | 20 ++++ src/symphonia/identity/models.py | 92 ++++++++++++++++ src/symphonia/infrastructure/__init__.py | 2 + .../infrastructure/sqlite_resolutions.py | 103 ++++++++++++++++++ tests/test_identity_resolution.py | 86 +++++++++++++++ 6 files changed, 304 insertions(+) create mode 100644 src/symphonia/identity/__init__.py create mode 100644 src/symphonia/identity/models.py create mode 100644 src/symphonia/infrastructure/sqlite_resolutions.py create mode 100644 tests/test_identity_resolution.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 5c8cf67..5e968ae 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -12,6 +12,7 @@ The first implementation increment is intentionally narrower than any provider o - Dependency-free SQLite operation repository proving idempotent creation, time-bounded worker leases, checkpoint ownership, retry scheduling, and expired-lease recovery. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. +- Versioned identity assessments and append-only SQLite storage for manual resolution decisions. - Deterministic `unittest` coverage under `tests/`. The package is an implementation foundation, not a claim that the corresponding capability SDDs are complete. Provider OAuth, provider API adapters, user authentication, Ingress, UI, migrations beyond the initial schema, backup/restore, and operation handlers remain unimplemented and blocked by their SDD decisions. diff --git a/src/symphonia/identity/__init__.py b/src/symphonia/identity/__init__.py new file mode 100644 index 0000000..f296a0d --- /dev/null +++ b/src/symphonia/identity/__init__.py @@ -0,0 +1,20 @@ +"""Identity-resolution evidence and durable manual decisions.""" + +from .models import ( + AssessmentClass, + Evidence, + EvidenceKind, + ManualDecision, + ManualDecisionAction, + ResolutionState, +) + +__all__ = [ + "AssessmentClass", + "Evidence", + "EvidenceKind", + "ManualDecision", + "ManualDecisionAction", + "ResolutionState", +] + diff --git a/src/symphonia/identity/models.py b/src/symphonia/identity/models.py new file mode 100644 index 0000000..7f1d78c --- /dev/null +++ b/src/symphonia/identity/models.py @@ -0,0 +1,92 @@ +"""Provider-independent identity-resolution values. + +This module deliberately models evidence and decisions, not a final matching +algorithm. Thresholds and candidate retrieval remain provider/policy work that +must be accepted separately. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum + + +class ResolutionState(str, Enum): + RESOLVED = "resolved" + AMBIGUOUS = "ambiguous" + UNMATCHED = "unmatched" + DEFERRED = "deferred" + INVALID = "invalid" + + +class AssessmentClass(str, Enum): + EXACT = "exact" + HIGH_CONFIDENCE = "high_confidence" + AMBIGUOUS = "ambiguous" + UNMATCHED = "unmatched" + + +class EvidenceKind(str, Enum): + ISRC = "isrc" + TITLE = "title" + ARTIST = "artist" + DURATION = "duration" + VERSION = "version" + RELEASE = "release" + PROVIDER_METADATA = "provider_metadata" + + +class ManualDecisionAction(str, Enum): + ACCEPT = "accept" + REJECT = "reject" + DISTINCT = "distinct" + DEFER = "defer" + REVOKE = "revoke" + + +@dataclass(frozen=True, slots=True) +class Evidence: + kind: EvidenceKind + summary: str + supports: bool + source: str + + def __post_init__(self) -> None: + if not self.summary.strip() or not self.source.strip(): + raise ValueError("evidence summary and source must not be empty") + + +@dataclass(frozen=True, slots=True) +class CandidateAssessment: + provider_track_key: str + candidate_recording_id: str | None + classification: AssessmentClass + evidence: tuple[Evidence, ...] + resolver_version: str + + def __post_init__(self) -> None: + if not self.provider_track_key.strip() or not self.resolver_version.strip(): + raise ValueError("provider_track_key and resolver_version must not be empty") + if self.classification is AssessmentClass.UNMATCHED: + if self.candidate_recording_id is not None: + raise ValueError("unmatched assessments cannot name a candidate") + elif not self.candidate_recording_id or not self.evidence: + raise ValueError("matched assessments require a candidate and evidence") + + +@dataclass(frozen=True, slots=True) +class ManualDecision: + provider_track_key: str + candidate_recording_id: str | None + action: ManualDecisionAction + actor_id: str + reason: str + created_at: str + decision_id: str | None = None + + def __post_init__(self) -> None: + if not self.provider_track_key.strip() or not self.actor_id.strip() or not self.reason.strip(): + raise ValueError("manual decisions require track, actor, and reason") + if self.action in {ManualDecisionAction.ACCEPT, ManualDecisionAction.REJECT} and not self.candidate_recording_id: + raise ValueError("accept/reject decisions require a candidate recording") + diff --git a/src/symphonia/infrastructure/__init__.py b/src/symphonia/infrastructure/__init__.py index 26183fd..64bf216 100644 --- a/src/symphonia/infrastructure/__init__.py +++ b/src/symphonia/infrastructure/__init__.py @@ -7,6 +7,7 @@ LeaseConflict, ) from .sqlite_plans import CopyPlanNotFound, CopyPlanRepository, StoredCopyPlan +from .sqlite_resolutions import ResolutionDecisionRepository __all__ = [ "CopyPlanNotFound", @@ -15,5 +16,6 @@ "LeaseConflict", "OperationNotFound", "OperationRepository", + "ResolutionDecisionRepository", "StoredCopyPlan", ] diff --git a/src/symphonia/infrastructure/sqlite_resolutions.py b/src/symphonia/infrastructure/sqlite_resolutions.py new file mode 100644 index 0000000..b5fe9e7 --- /dev/null +++ b/src/symphonia/infrastructure/sqlite_resolutions.py @@ -0,0 +1,103 @@ +"""Append-only SQLite storage for manual identity decisions.""" + +from __future__ import annotations + +from dataclasses import replace +import json +import sqlite3 +import uuid + +from symphonia.identity.models import ManualDecision, ManualDecisionAction + + +class ResolutionDecisionRepository: + """Preserve every decision; latest state never erases prior authorship.""" + + def __init__(self, path: str = ":memory:") -> None: + self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) + self._connection.row_factory = sqlite3.Row + self._migrate() + + def close(self) -> None: + self._connection.close() + + def _migrate(self) -> None: + self._connection.executescript( + """ + CREATE TABLE IF NOT EXISTS resolution_decisions ( + sequence INTEGER PRIMARY KEY AUTOINCREMENT, + decision_id TEXT NOT NULL UNIQUE, + provider_track_key TEXT NOT NULL, + candidate_recording_id TEXT, + action TEXT NOT NULL, + actor_id TEXT NOT NULL, + reason TEXT NOT NULL, + created_at TEXT NOT NULL, + payload_json TEXT NOT NULL + ); + CREATE INDEX IF NOT EXISTS resolution_decisions_lookup_idx + ON resolution_decisions (provider_track_key, candidate_recording_id, sequence); + """ + ) + + def record(self, decision: ManualDecision) -> ManualDecision: + decision_id = decision.decision_id or str(uuid.uuid4()) + persisted = replace(decision, decision_id=decision_id) + payload = json.dumps( + { + "provider_track_key": persisted.provider_track_key, + "candidate_recording_id": persisted.candidate_recording_id, + "action": persisted.action.value, + "actor_id": persisted.actor_id, + "reason": persisted.reason, + "created_at": persisted.created_at, + }, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + ) + self._connection.execute( + """ + INSERT INTO resolution_decisions ( + decision_id, provider_track_key, candidate_recording_id, action, + actor_id, reason, created_at, payload_json + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + persisted.decision_id, + persisted.provider_track_key, + persisted.candidate_recording_id, + persisted.action.value, + persisted.actor_id, + persisted.reason, + persisted.created_at, + payload, + ), + ) + return persisted + + def latest(self, provider_track_key: str, candidate_recording_id: str | None) -> ManualDecision | None: + row = self._connection.execute( + """ + SELECT * FROM resolution_decisions + WHERE provider_track_key = ? + AND candidate_recording_id IS ? + ORDER BY sequence DESC LIMIT 1 + """, + (provider_track_key, candidate_recording_id), + ).fetchone() + if row is None: + return None + return ManualDecision( + provider_track_key=row["provider_track_key"], + candidate_recording_id=row["candidate_recording_id"], + action=ManualDecisionAction(row["action"]), + actor_id=row["actor_id"], + reason=row["reason"], + created_at=row["created_at"], + decision_id=row["decision_id"], + ) + + def count(self) -> int: + return int(self._connection.execute("SELECT COUNT(*) FROM resolution_decisions").fetchone()[0]) + diff --git a/tests/test_identity_resolution.py b/tests/test_identity_resolution.py new file mode 100644 index 0000000..3abb85d --- /dev/null +++ b/tests/test_identity_resolution.py @@ -0,0 +1,86 @@ +from __future__ import annotations + +from datetime import datetime, timezone +import unittest + +from symphonia.identity import ( + AssessmentClass, + Evidence, + EvidenceKind, + ManualDecision, + ManualDecisionAction, +) +from symphonia.infrastructure import ResolutionDecisionRepository + + +class IdentityResolutionTests(unittest.TestCase): + def test_unmatched_assessment_has_no_candidate(self) -> None: + from symphonia.identity.models import CandidateAssessment + + assessment = CandidateAssessment( + provider_track_key="spotify:connection-1:track-1", + candidate_recording_id=None, + classification=AssessmentClass.UNMATCHED, + evidence=(), + resolver_version="rules-1", + ) + self.assertEqual(assessment.classification, AssessmentClass.UNMATCHED) + + def test_matched_assessment_requires_explainable_evidence(self) -> None: + from symphonia.identity.models import CandidateAssessment + + with self.assertRaises(ValueError): + CandidateAssessment( + provider_track_key="spotify:connection-1:track-1", + candidate_recording_id="recording-1", + classification=AssessmentClass.HIGH_CONFIDENCE, + evidence=(), + resolver_version="rules-1", + ) + + assessment = CandidateAssessment( + provider_track_key="spotify:connection-1:track-1", + candidate_recording_id="recording-1", + classification=AssessmentClass.HIGH_CONFIDENCE, + evidence=(Evidence(EvidenceKind.ISRC, "ISRC matches", True, "provider metadata"),), + resolver_version="rules-1", + ) + self.assertEqual(assessment.evidence[0].kind, EvidenceKind.ISRC) + + def test_manual_decisions_are_append_only_and_latest_is_explicit(self) -> None: + repository = ResolutionDecisionRepository() + try: + first = repository.record( + ManualDecision( + provider_track_key="spotify:connection-1:track-1", + candidate_recording_id="recording-1", + action=ManualDecisionAction.REJECT, + actor_id="local-user", + reason="Live version, not the studio recording", + created_at=datetime.now(timezone.utc).isoformat(), + ) + ) + latest = repository.latest("spotify:connection-1:track-1", "recording-1") + self.assertEqual(latest.decision_id, first.decision_id) + self.assertEqual(latest.action, ManualDecisionAction.REJECT) + + repository.record( + ManualDecision( + provider_track_key="spotify:connection-1:track-1", + candidate_recording_id="recording-1", + action=ManualDecisionAction.ACCEPT, + actor_id="local-user", + reason="Verified the exact recording", + created_at=datetime.now(timezone.utc).isoformat(), + ) + ) + latest = repository.latest("spotify:connection-1:track-1", "recording-1") + self.assertEqual(latest.action, ManualDecisionAction.ACCEPT) + self.assertEqual(repository.count(), 2) + finally: + repository.close() + + +if __name__ == "__main__": + unittest.main() + From 961c656a8b81f6b9ab120190e56d1fd71c1eff1e Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 16:37:32 +0200 Subject: [PATCH 009/167] feat: enqueue only accepted copy plans --- docs/development/implementation-baseline.md | 1 + src/symphonia/application/__init__.py | 4 +- src/symphonia/application/copy_workflow.py | 55 ++++++++++++++ tests/test_copy_workflow.py | 82 +++++++++++++++++++++ 4 files changed, 140 insertions(+), 2 deletions(-) create mode 100644 src/symphonia/application/copy_workflow.py create mode 100644 tests/test_copy_workflow.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 5e968ae..2891c7a 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -13,6 +13,7 @@ The first implementation increment is intentionally narrower than any provider o - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. +- An application workflow that persists plans, requires digest acceptance, and enqueues only accepted plans as durable operations. - Deterministic `unittest` coverage under `tests/`. The package is an implementation foundation, not a claim that the corresponding capability SDDs are complete. Provider OAuth, provider API adapters, user authentication, Ingress, UI, migrations beyond the initial schema, backup/restore, and operation handlers remain unimplemented and blocked by their SDD decisions. diff --git a/src/symphonia/application/__init__.py b/src/symphonia/application/__init__.py index 5bbe361..eb753ba 100644 --- a/src/symphonia/application/__init__.py +++ b/src/symphonia/application/__init__.py @@ -1,6 +1,6 @@ """Use-case orchestration ports and services.""" from .copy_planning import CopyPlanningService +from .copy_workflow import CopyWorkflowService -__all__ = ["CopyPlanningService"] - +__all__ = ["CopyPlanningService", "CopyWorkflowService"] diff --git a/src/symphonia/application/copy_workflow.py b/src/symphonia/application/copy_workflow.py new file mode 100644 index 0000000..6cda10c --- /dev/null +++ b/src/symphonia/application/copy_workflow.py @@ -0,0 +1,55 @@ +"""Application orchestration for planning and admitting a copy operation.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime + +from symphonia.domain.models import CopyPlan, CopyPolicy, PlanAcceptanceError, PlaylistSnapshot +from symphonia.infrastructure.sqlite_operations import OperationRecord, OperationRepository +from symphonia.infrastructure.sqlite_plans import CopyPlanRepository, StoredCopyPlan + +from .copy_planning import CopyPlanningService + + +@dataclass(frozen=True, slots=True) +class CopyWorkflowService: + """Keep plan persistence/acceptance separate from provider execution.""" + + planning: CopyPlanningService + plans: CopyPlanRepository + operations: OperationRepository + + def create_plan( + self, + snapshot: PlaylistSnapshot, + *, + target_provider: str, + target_playlist_name: str, + target_visibility: str, + policy: CopyPolicy, + now: datetime, + ) -> StoredCopyPlan: + plan = self.planning.plan( + snapshot, + target_provider=target_provider, + target_playlist_name=target_playlist_name, + target_visibility=target_visibility, + policy=policy, + ) + return self.plans.save(plan, now=now) + + def accept_plan(self, digest: str, *, now: datetime) -> StoredCopyPlan: + return self.plans.accept(digest, expected_digest=digest, now=now) + + def enqueue_accepted_plan(self, digest: str, *, now: datetime) -> OperationRecord: + stored = self.plans.get(digest) + if stored.accepted_at is None: + raise PlanAcceptanceError("copy plan must be accepted before it can be enqueued") + return self.operations.create( + operation_type="copy_playlist", + idempotency_key=f"copy-plan:{digest}", + payload={"plan_digest": digest}, + now=now, + ) + diff --git a/tests/test_copy_workflow.py b/tests/test_copy_workflow.py new file mode 100644 index 0000000..3f19a6c --- /dev/null +++ b/tests/test_copy_workflow.py @@ -0,0 +1,82 @@ +from __future__ import annotations + +from datetime import datetime, timezone +import unittest + +from symphonia.application import CopyPlanningService, CopyWorkflowService +from symphonia.domain import CopyPolicy, EntryClassification, PlanAcceptanceError, PlaylistSnapshot, SourcePlaylistEntry +from symphonia.infrastructure import CopyPlanRepository, OperationRepository + + +class CopyWorkflowTests(unittest.TestCase): + now = datetime(2026, 9, 20, 12, 0, tzinfo=timezone.utc) + + def setUp(self) -> None: + self.plans = CopyPlanRepository() + self.operations = OperationRepository() + self.workflow = CopyWorkflowService(CopyPlanningService(), self.plans, self.operations) + + def tearDown(self) -> None: + self.plans.close() + self.operations.close() + + def snapshot(self, classification: EntryClassification = EntryClassification.READY) -> PlaylistSnapshot: + return PlaylistSnapshot( + snapshot_id="snapshot-1", + source_provider="spotify", + source_playlist_id="playlist-1", + entries=( + SourcePlaylistEntry( + "occ-1", + 0, + "spotify-track-1", + classification, + "youtube-track-1" if classification is EntryClassification.READY else None, + ), + ), + ) + + def test_workflow_requires_acceptance_before_enqueue(self) -> None: + stored = self.workflow.create_plan( + self.snapshot(), + target_provider="youtube", + target_playlist_name="Rock", + target_visibility="private", + policy=CopyPolicy.STRICT, + now=self.now, + ) + with self.assertRaises(PlanAcceptanceError): + self.workflow.enqueue_accepted_plan(stored.plan.digest, now=self.now) + + def test_workflow_accepts_then_enqueues_idempotently(self) -> None: + stored = self.workflow.create_plan( + self.snapshot(), + target_provider="youtube", + target_playlist_name="Rock", + target_visibility="private", + policy=CopyPolicy.STRICT, + now=self.now, + ) + accepted = self.workflow.accept_plan(stored.plan.digest, now=self.now) + operation = self.workflow.enqueue_accepted_plan(accepted.plan.digest, now=self.now) + duplicate = self.workflow.enqueue_accepted_plan(accepted.plan.digest, now=self.now) + self.assertEqual(operation.operation_id, duplicate.operation_id) + self.assertEqual(operation.state, "queued") + self.assertEqual(operation.payload, {"plan_digest": accepted.plan.digest}) + + def test_blocked_plan_cannot_be_accepted(self) -> None: + stored = self.workflow.create_plan( + self.snapshot(EntryClassification.AMBIGUOUS), + target_provider="youtube", + target_playlist_name="Rock", + target_visibility="private", + policy=CopyPolicy.STRICT, + now=self.now, + ) + with self.assertRaises(PlanAcceptanceError): + self.workflow.accept_plan(stored.plan.digest, now=self.now) + + +if __name__ == "__main__": + unittest.main() + From a532a2b238c8b1e51d24ab6f14bdc8052c3d04dc Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 16:39:09 +0200 Subject: [PATCH 010/167] build: add non-root local runtime container --- .dockerignore | 11 +++++++++ Dockerfile | 26 +++++++++++++++++++++ docs/development/implementation-baseline.md | 11 +++++++++ 3 files changed, 48 insertions(+) create mode 100644 .dockerignore create mode 100644 Dockerfile diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..b921a45 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,11 @@ +.git +.gitignore +.pytest_cache +__pycache__ +*.py[cod] +*.sqlite3 +*.db +docs +specs +tests + diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..413a62c --- /dev/null +++ b/Dockerfile @@ -0,0 +1,26 @@ +FROM python:3.12-slim + +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + PYTHONPATH=/app/src \ + SYMPHONIA_HOST=0.0.0.0 \ + SYMPHONIA_PORT=8099 \ + SYMPHONIA_DATABASE=/data/symphonia.sqlite3 + +WORKDIR /app +COPY src /app/src + +RUN addgroup --system symphonia \ + && adduser --system --ingroup symphonia symphonia \ + && mkdir -p /data \ + && chown -R symphonia:symphonia /app /data + +USER symphonia +VOLUME ["/data"] +EXPOSE 8099 + +HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \ + CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8099/ready', timeout=2)"] + +ENTRYPOINT ["python", "-m", "symphonia"] + diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 2891c7a..b08ff85 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -14,6 +14,17 @@ The first implementation increment is intentionally narrower than any provider o - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. - An application workflow that persists plans, requires digest acceptance, and enqueues only accepted plans as durable operations. + +## Local container profile + +`Dockerfile` packages the dependency-free runtime as a non-root service with the durable volume mounted at `/data`. It is a standalone development/container profile, not yet the published Home Assistant App artifact. + +```text +docker build -t symphonia:dev . +docker run --rm -p 8099:8099 -v symphonia-data:/data symphonia:dev +``` + +The container exposes only the current health/readiness/version surface. A future App manifest must add Ingress, Supervisor metadata, supported architectures, backup declarations, and any direct callback policy only after the runtime SDD blockers are resolved. - Deterministic `unittest` coverage under `tests/`. The package is an implementation foundation, not a claim that the corresponding capability SDDs are complete. Provider OAuth, provider API adapters, user authentication, Ingress, UI, migrations beyond the initial schema, backup/restore, and operation handlers remain unimplemented and blocked by their SDD decisions. From 5ca264b93cc60e83a1250f13cc615e0a334f3b32 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 17:02:45 +0200 Subject: [PATCH 011/167] feat: execute accepted copies with reconciliation --- docs/development/implementation-baseline.md | 1 + src/symphonia/application/__init__.py | 3 +- src/symphonia/application/copy_execution.py | 190 ++++++++++++++++++ .../infrastructure/sqlite_operations.py | 27 ++- src/symphonia/providers/__init__.py | 7 +- src/symphonia/providers/writing.py | 72 +++++++ tests/test_copy_execution.py | 121 +++++++++++ tests/test_sqlite_operations.py | 20 ++ 8 files changed, 437 insertions(+), 4 deletions(-) create mode 100644 src/symphonia/application/copy_execution.py create mode 100644 src/symphonia/providers/writing.py create mode 100644 tests/test_copy_execution.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index b08ff85..33def05 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -14,6 +14,7 @@ The first implementation increment is intentionally narrower than any provider o - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. - An application workflow that persists plans, requires digest acceptance, and enqueues only accepted plans as durable operations. +- A copy executor that creates target playlists through a semantic writer port, checkpoints each occurrence, and reconciles unknown writes before retry. ## Local container profile diff --git a/src/symphonia/application/__init__.py b/src/symphonia/application/__init__.py index eb753ba..e1e0de1 100644 --- a/src/symphonia/application/__init__.py +++ b/src/symphonia/application/__init__.py @@ -2,5 +2,6 @@ from .copy_planning import CopyPlanningService from .copy_workflow import CopyWorkflowService +from .copy_execution import CopyExecutionService -__all__ = ["CopyPlanningService", "CopyWorkflowService"] +__all__ = ["CopyExecutionService", "CopyPlanningService", "CopyWorkflowService"] diff --git a/src/symphonia/application/copy_execution.py b/src/symphonia/application/copy_execution.py new file mode 100644 index 0000000..47c7ba3 --- /dev/null +++ b/src/symphonia/application/copy_execution.py @@ -0,0 +1,190 @@ +"""Durable copy execution against the provider writer port.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime, timedelta +from typing import Any + +from symphonia.domain.models import PlanAcceptanceError +from symphonia.infrastructure.sqlite_operations import OperationRecord, OperationRepository +from symphonia.infrastructure.sqlite_plans import CopyPlanRepository +from symphonia.providers.writing import PlaylistWriter, ProviderWriteError, WriteOutcome + + +@dataclass(frozen=True, slots=True) +class CopyExecutionService: + plans: CopyPlanRepository + operations: OperationRepository + retry_delay_seconds: int = 60 + + def execute( + self, + digest: str, + *, + writer: PlaylistWriter, + worker_id: str, + now: datetime, + lease_seconds: int = 30, + ) -> OperationRecord: + stored = self.plans.get(digest) + if stored.accepted_at is None: + raise PlanAcceptanceError("copy plan must be accepted before execution") + operation = self.operations.create( + operation_type="copy_playlist", + idempotency_key=f"copy-plan:{digest}", + payload={"plan_digest": digest}, + now=now, + ) + if operation.state in {"succeeded", "partial", "failed", "cancelled"}: + return operation + operation = self.operations.claim( + operation.operation_id, + worker_id=worker_id, + now=now, + lease_seconds=lease_seconds, + ) + checkpoint = dict(operation.checkpoint) + confirmed = list(checkpoint.get("confirmed_occurrences", [])) + issues = list(checkpoint.get("issues", [])) + target_id = checkpoint.get("target_playlist_id") + + if target_id is None: + target_key = f"{digest}:target" + try: + target = writer.ensure_target_playlist( + provider=stored.plan.target_provider, + name=stored.plan.target_playlist_name, + visibility=stored.plan.target_visibility, + idempotency_key=target_key, + ) + except ProviderWriteError as error: + if error.outcome is WriteOutcome.RETRYABLE: + return self._schedule_retry(operation, worker_id, checkpoint, now) + if error.outcome is WriteOutcome.UNKNOWN_OUTCOME: + target = writer.reconcile_target_playlist(idempotency_key=target_key) + if target is None: + return self._wait_for_user(operation, worker_id, checkpoint | {"unknown_step": "target"}, now) + else: + issues.append({"step": "target", "detail": error.detail, "provider_code": error.provider_code}) + return self._finish(operation, worker_id, checkpoint | {"issues": issues}, now, "failed") + target_id = target.provider_playlist_id + checkpoint["target_playlist_id"] = target_id + operation = self.operations.checkpoint( + operation.operation_id, + worker_id=worker_id, + checkpoint=checkpoint, + now=now, + state="running", + ) + + for entry in stored.plan.writable_entries: + if entry.occurrence_id in confirmed: + continue + self.operations.renew_lease( + operation.operation_id, + worker_id=worker_id, + now=now, + lease_seconds=lease_seconds, + ) + step_key = f"{digest}:entry:{entry.occurrence_id}" + result = writer.add_entry( + target_playlist_id=target_id, + provider_track_id=entry.target_track_id or "", + idempotency_key=step_key, + ) + if result.outcome is WriteOutcome.UNKNOWN_OUTCOME: + if writer.reconcile_entry( + target_playlist_id=target_id, + provider_track_id=entry.target_track_id or "", + idempotency_key=step_key, + ): + result = type(result)(WriteOutcome.CONFIRMED_SUCCESS, result.provider_code, result.detail) + else: + return self._wait_for_user( + operation, + worker_id, + checkpoint | {"unknown_step": entry.occurrence_id}, + now, + ) + if result.outcome is WriteOutcome.RETRYABLE: + return self._schedule_retry(operation, worker_id, checkpoint, now) + if result.outcome is WriteOutcome.PERMANENT_FAILURE: + issues.append( + { + "step": entry.occurrence_id, + "detail": result.detail or "permanent provider failure", + "provider_code": result.provider_code, + } + ) + checkpoint["issues"] = issues + operation = self.operations.checkpoint( + operation.operation_id, + worker_id=worker_id, + checkpoint=checkpoint, + now=now, + state="running", + ) + continue + confirmed.append(entry.occurrence_id) + checkpoint["confirmed_occurrences"] = confirmed + operation = self.operations.checkpoint( + operation.operation_id, + worker_id=worker_id, + checkpoint=checkpoint, + now=now, + state="running", + ) + + checkpoint["confirmed_occurrences"] = confirmed + checkpoint["omitted_occurrences"] = [entry.occurrence_id for entry in stored.plan.omitted_entries] + checkpoint["issues"] = issues + terminal = "partial" if stored.plan.omitted_entries or issues else "succeeded" + return self._finish(operation, worker_id, checkpoint, now, terminal) + + def _finish( + self, + operation: OperationRecord, + worker_id: str, + checkpoint: dict[str, Any], + now: datetime, + state: str, + ) -> OperationRecord: + return self.operations.checkpoint( + operation.operation_id, + worker_id=worker_id, + checkpoint=checkpoint, + now=now, + state=state, + ) + + def _schedule_retry( + self, + operation: OperationRecord, + worker_id: str, + checkpoint: dict[str, Any], + now: datetime, + ) -> OperationRecord: + return self.operations.schedule_retry( + operation.operation_id, + worker_id=worker_id, + next_run_at=now + timedelta(seconds=self.retry_delay_seconds), + checkpoint=checkpoint, + now=now, + ) + + def _wait_for_user( + self, + operation: OperationRecord, + worker_id: str, + checkpoint: dict[str, Any], + now: datetime, + ) -> OperationRecord: + return self.operations.checkpoint( + operation.operation_id, + worker_id=worker_id, + checkpoint=checkpoint, + now=now, + state="waiting_user", + ) + diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index f7c084b..dff46ea 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -246,8 +246,8 @@ def checkpoint( """ UPDATE operations SET state = ?, checkpoint_json = ?, updated_at = ?, - worker_id = CASE WHEN ? IN ('running', 'waiting_user') THEN worker_id ELSE NULL END, - lease_expires_at = CASE WHEN ? IN ('running', 'waiting_user') THEN lease_expires_at ELSE NULL END + worker_id = CASE WHEN ? = 'running' THEN worker_id ELSE NULL END, + lease_expires_at = CASE WHEN ? = 'running' THEN lease_expires_at ELSE NULL END WHERE operation_id = ? """, (state, checkpoint_json, now_text, state, state, operation_id), @@ -258,6 +258,29 @@ def checkpoint( raise return self.get(operation_id) + def resume(self, operation_id: str, *, now: datetime) -> OperationRecord: + """Re-admit a user-action operation after its external issue is resolved.""" + + now_text = _utc(now) + self._connection.execute("BEGIN IMMEDIATE") + try: + row = self._connection.execute( + "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) + ).fetchone() + if row is None: + raise OperationNotFound(operation_id) + if row["state"] != "waiting_user": + raise LeaseConflict("only waiting_user operations can be resumed") + self._connection.execute( + "UPDATE operations SET state = 'queued', updated_at = ? WHERE operation_id = ?", + (now_text, operation_id), + ) + self._connection.execute("COMMIT") + except Exception: + self._connection.execute("ROLLBACK") + raise + return self.get(operation_id) + def schedule_retry( self, operation_id: str, diff --git a/src/symphonia/providers/__init__.py b/src/symphonia/providers/__init__.py index 86412de..90588cf 100644 --- a/src/symphonia/providers/__init__.py +++ b/src/symphonia/providers/__init__.py @@ -12,6 +12,7 @@ ProviderPlaylistPage, ) from .importing import CollectionImportResult, ImportIssue, collect_playlist_pages, to_playlist_snapshot +from .writing import PlaylistWriter, ProviderWriteError, TargetPlaylist, WriteOutcome, WriteResult __all__ = [ "AccessBasis", @@ -25,7 +26,11 @@ "ProviderObjectRef", "ProviderPlaylistEntry", "ProviderPlaylistPage", + "ProviderWriteError", + "PlaylistWriter", + "TargetPlaylist", + "WriteOutcome", + "WriteResult", "collect_playlist_pages", "to_playlist_snapshot", ] - diff --git a/src/symphonia/providers/writing.py b/src/symphonia/providers/writing.py new file mode 100644 index 0000000..89b7ba7 --- /dev/null +++ b/src/symphonia/providers/writing.py @@ -0,0 +1,72 @@ +"""Normalized target-playlist write contract.""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from typing import Protocol + + +class WriteOutcome(str, Enum): + CONFIRMED_SUCCESS = "confirmed_success" + RETRYABLE = "retryable" + UNKNOWN_OUTCOME = "unknown_outcome" + PERMANENT_FAILURE = "permanent_failure" + + +@dataclass(frozen=True, slots=True) +class TargetPlaylist: + provider_playlist_id: str + + def __post_init__(self) -> None: + if not self.provider_playlist_id.strip(): + raise ValueError("provider_playlist_id must not be empty") + + +@dataclass(frozen=True, slots=True) +class WriteResult: + outcome: WriteOutcome + provider_code: str | None = None + detail: str | None = None + + +class ProviderWriteError(RuntimeError): + """A target-creation failure with an explicit retry/reconciliation class.""" + + def __init__(self, outcome: WriteOutcome, detail: str, provider_code: str | None = None) -> None: + super().__init__(detail) + self.outcome = outcome + self.detail = detail + self.provider_code = provider_code + + +class PlaylistWriter(Protocol): + """Port used by the copy executor; concrete providers stay outside it.""" + + def ensure_target_playlist( + self, + *, + provider: str, + name: str, + visibility: str, + idempotency_key: str, + ) -> TargetPlaylist: ... + + def add_entry( + self, + *, + target_playlist_id: str, + provider_track_id: str, + idempotency_key: str, + ) -> WriteResult: ... + + def reconcile_target_playlist(self, *, idempotency_key: str) -> TargetPlaylist | None: ... + + def reconcile_entry( + self, + *, + target_playlist_id: str, + provider_track_id: str, + idempotency_key: str, + ) -> bool: ... + diff --git a/tests/test_copy_execution.py b/tests/test_copy_execution.py new file mode 100644 index 0000000..e5bb14e --- /dev/null +++ b/tests/test_copy_execution.py @@ -0,0 +1,121 @@ +from __future__ import annotations + +from datetime import datetime, timedelta, timezone +import unittest + +from symphonia.application import CopyExecutionService, CopyPlanningService, CopyWorkflowService +from symphonia.domain import CopyPolicy, EntryClassification, PlaylistSnapshot, SourcePlaylistEntry +from symphonia.infrastructure import CopyPlanRepository, OperationRepository +from symphonia.providers import TargetPlaylist, WriteOutcome, WriteResult + + +NOW = datetime(2026, 9, 20, 12, 0, tzinfo=timezone.utc) + + +class FakeWriter: + def __init__(self) -> None: + self.target = TargetPlaylist("target-playlist-1") + self.added: list[tuple[str, str]] = [] + self.results: dict[str, WriteResult] = {} + self.reconcile_results: dict[str, bool] = {} + self.reconciled: list[str] = [] + + def ensure_target_playlist(self, *, provider: str, name: str, visibility: str, idempotency_key: str) -> TargetPlaylist: + return self.target + + def add_entry(self, *, target_playlist_id: str, provider_track_id: str, idempotency_key: str) -> WriteResult: + self.added.append((idempotency_key, provider_track_id)) + return self.results.get(idempotency_key, WriteResult(WriteOutcome.CONFIRMED_SUCCESS)) + + def reconcile_target_playlist(self, *, idempotency_key: str) -> TargetPlaylist | None: + return self.target + + def reconcile_entry(self, *, target_playlist_id: str, provider_track_id: str, idempotency_key: str) -> bool: + self.reconciled.append(idempotency_key) + return self.reconcile_results.get(idempotency_key, False) + + +class CopyExecutionTests(unittest.TestCase): + def setUp(self) -> None: + self.plans = CopyPlanRepository() + self.operations = OperationRepository() + self.workflow = CopyWorkflowService(CopyPlanningService(), self.plans, self.operations) + self.executor = CopyExecutionService(self.plans, self.operations, retry_delay_seconds=30) + self.writer = FakeWriter() + + def tearDown(self) -> None: + self.plans.close() + self.operations.close() + + def snapshot(self, *, policy: CopyPolicy = CopyPolicy.STRICT, include_unmatched: bool = False) -> tuple[object, CopyPolicy]: + entries = [SourcePlaylistEntry("occ-1", 0, "source-1", EntryClassification.READY, "target-1")] + if include_unmatched: + entries.append(SourcePlaylistEntry("occ-2", 1, "source-2", EntryClassification.UNMATCHED)) + source = PlaylistSnapshot("snapshot-1", "spotify", "playlist-1", tuple(entries)) + return source, policy + + def accepted_digest(self, *, policy: CopyPolicy = CopyPolicy.STRICT, include_unmatched: bool = False) -> str: + source, policy = self.snapshot(policy=policy, include_unmatched=include_unmatched) + stored = self.workflow.create_plan( + source, + target_provider="youtube", + target_playlist_name="Rock", + target_visibility="private", + policy=policy, + now=NOW, + ) + return self.workflow.accept_plan(stored.plan.digest, now=NOW).plan.digest + + def test_success_checkpoints_entries_in_source_order(self) -> None: + digest = self.accepted_digest() + operation = self.executor.execute(digest, writer=self.writer, worker_id="worker-a", now=NOW) + self.assertEqual(operation.state, "succeeded") + self.assertEqual([track for _, track in self.writer.added], ["target-1"]) + self.assertEqual(operation.checkpoint["confirmed_occurrences"], ["occ-1"]) + + def test_best_effort_omission_is_partial_not_success(self) -> None: + digest = self.accepted_digest(policy=CopyPolicy.BEST_EFFORT, include_unmatched=True) + operation = self.executor.execute(digest, writer=self.writer, worker_id="worker-a", now=NOW) + self.assertEqual(operation.state, "partial") + self.assertEqual(operation.checkpoint["omitted_occurrences"], ["occ-2"]) + + def test_unknown_write_is_reconciled_before_success(self) -> None: + digest = self.accepted_digest() + step_key = f"{digest}:entry:occ-1" + self.writer.results[step_key] = WriteResult(WriteOutcome.UNKNOWN_OUTCOME, detail="timeout after request") + self.writer.reconcile_results[step_key] = True + + operation = self.executor.execute(digest, writer=self.writer, worker_id="worker-a", now=NOW) + self.assertEqual(operation.state, "succeeded") + self.assertEqual(self.writer.reconciled, [step_key]) + + def test_unknown_write_without_reconciliation_requires_user(self) -> None: + digest = self.accepted_digest() + step_key = f"{digest}:entry:occ-1" + self.writer.results[step_key] = WriteResult(WriteOutcome.UNKNOWN_OUTCOME, detail="provider timeout") + operation = self.executor.execute(digest, writer=self.writer, worker_id="worker-a", now=NOW) + self.assertEqual(operation.state, "waiting_user") + self.assertEqual(operation.checkpoint["unknown_step"], "occ-1") + + self.operations.resume(operation.operation_id, now=NOW + timedelta(seconds=1)) + self.writer.results[step_key] = WriteResult(WriteOutcome.CONFIRMED_SUCCESS) + resumed = self.executor.execute( + digest, + writer=self.writer, + worker_id="worker-b", + now=NOW + timedelta(seconds=1), + ) + self.assertEqual(resumed.state, "succeeded") + + def test_retryable_write_releases_operation_until_scheduled(self) -> None: + digest = self.accepted_digest() + step_key = f"{digest}:entry:occ-1" + self.writer.results[step_key] = WriteResult(WriteOutcome.RETRYABLE, provider_code="429") + operation = self.executor.execute(digest, writer=self.writer, worker_id="worker-a", now=NOW) + self.assertEqual(operation.state, "retry_scheduled") + self.assertEqual(operation.next_run_at, NOW + timedelta(seconds=30)) + + +if __name__ == "__main__": + unittest.main() + diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index cd5080a..8ca10d8 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -165,6 +165,26 @@ def test_healthy_worker_can_renew_lease(self) -> None: ) self.assertGreater(renewed.lease_expires_at, claimed.lease_expires_at) + def test_waiting_user_operation_releases_lease_and_can_resume(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={}, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) + waiting = self.repository.checkpoint( + operation.operation_id, + worker_id="worker-a", + checkpoint={"unknown_step": "occ-1"}, + now=self.now + timedelta(seconds=1), + state="waiting_user", + ) + self.assertEqual(waiting.state, "waiting_user") + self.assertIsNone(waiting.worker_id) + resumed = self.repository.resume(operation.operation_id, now=self.now + timedelta(seconds=2)) + self.assertEqual(resumed.state, "queued") + if __name__ == "__main__": unittest.main() From 55c0c5b0c5aa91b84b43fe721b6a53a07a0e51f2 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 17:04:44 +0200 Subject: [PATCH 012/167] feat: support cooperative copy cancellation --- src/symphonia/application/copy_execution.py | 23 +++++--- .../infrastructure/sqlite_operations.py | 54 ++++++++++++++++++- tests/test_sqlite_operations.py | 33 ++++++++++++ 3 files changed, 101 insertions(+), 9 deletions(-) diff --git a/src/symphonia/application/copy_execution.py b/src/symphonia/application/copy_execution.py index 47c7ba3..efe2de3 100644 --- a/src/symphonia/application/copy_execution.py +++ b/src/symphonia/application/copy_execution.py @@ -45,11 +45,15 @@ def execute( lease_seconds=lease_seconds, ) checkpoint = dict(operation.checkpoint) + if operation.cancel_requested: + return self._finish(operation, worker_id, checkpoint, now, "cancelled") confirmed = list(checkpoint.get("confirmed_occurrences", [])) issues = list(checkpoint.get("issues", [])) target_id = checkpoint.get("target_playlist_id") if target_id is None: + if self.operations.get(operation.operation_id).cancel_requested: + return self._finish(operation, worker_id, checkpoint, now, "cancelled") target_key = f"{digest}:target" try: target = writer.ensure_target_playlist( @@ -81,12 +85,18 @@ def execute( for entry in stored.plan.writable_entries: if entry.occurrence_id in confirmed: continue - self.operations.renew_lease( - operation.operation_id, - worker_id=worker_id, - now=now, - lease_seconds=lease_seconds, - ) + try: + self.operations.renew_lease( + operation.operation_id, + worker_id=worker_id, + now=now, + lease_seconds=lease_seconds, + ) + except Exception: + latest = self.operations.get(operation.operation_id) + if latest.cancel_requested: + return self._finish(latest, worker_id, checkpoint, now, "running") + raise step_key = f"{digest}:entry:{entry.occurrence_id}" result = writer.add_entry( target_playlist_id=target_id, @@ -187,4 +197,3 @@ def _wait_for_user( now=now, state="waiting_user", ) - diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index dff46ea..ea80741 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -48,6 +48,7 @@ class OperationRecord: worker_id: str | None lease_expires_at: datetime | None next_run_at: datetime | None + cancel_requested: bool created_at: datetime updated_at: datetime @@ -84,6 +85,7 @@ def _migrate(self) -> None: worker_id TEXT, lease_expires_at TEXT, next_run_at TEXT, + cancel_requested INTEGER NOT NULL DEFAULT 0, created_at TEXT NOT NULL, updated_at TEXT NOT NULL ); @@ -156,6 +158,8 @@ def claim( ).fetchone() if row is None: raise OperationNotFound(operation_id) + if row["cancel_requested"]: + raise LeaseConflict("operation cancellation has been requested") eligible = row["state"] == "queued" or ( row["state"] == "retry_scheduled" and row["next_run_at"] is not None @@ -242,15 +246,25 @@ def checkpoint( raise LeaseConflict("worker does not own a running operation") if row["lease_expires_at"] is not None and row["lease_expires_at"] <= now_text: raise LeaseConflict("operation lease has expired") + effective_state = "cancelled" if row["cancel_requested"] else state self._connection.execute( """ UPDATE operations SET state = ?, checkpoint_json = ?, updated_at = ?, worker_id = CASE WHEN ? = 'running' THEN worker_id ELSE NULL END, - lease_expires_at = CASE WHEN ? = 'running' THEN lease_expires_at ELSE NULL END + lease_expires_at = CASE WHEN ? = 'running' THEN lease_expires_at ELSE NULL END, + cancel_requested = CASE WHEN ? = 'cancelled' THEN 0 ELSE cancel_requested END WHERE operation_id = ? """, - (state, checkpoint_json, now_text, state, state, operation_id), + ( + effective_state, + checkpoint_json, + now_text, + effective_state, + effective_state, + effective_state, + operation_id, + ), ) self._connection.execute("COMMIT") except Exception: @@ -258,6 +272,41 @@ def checkpoint( raise return self.get(operation_id) + def cancel(self, operation_id: str, *, now: datetime) -> OperationRecord: + """Request cooperative cancellation and preserve in-flight ownership.""" + + now_text = _utc(now) + self._connection.execute("BEGIN IMMEDIATE") + try: + row = self._connection.execute( + "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) + ).fetchone() + if row is None: + raise OperationNotFound(operation_id) + if row["state"] in {"succeeded", "partial", "failed", "cancelled"}: + self._connection.execute("COMMIT") + return self.get(operation_id) + if row["state"] == "running": + self._connection.execute( + "UPDATE operations SET cancel_requested = 1, updated_at = ? WHERE operation_id = ?", + (now_text, operation_id), + ) + else: + self._connection.execute( + """ + UPDATE operations + SET state = 'cancelled', worker_id = NULL, lease_expires_at = NULL, + cancel_requested = 0, updated_at = ? + WHERE operation_id = ? + """, + (now_text, operation_id), + ) + self._connection.execute("COMMIT") + except Exception: + self._connection.execute("ROLLBACK") + raise + return self.get(operation_id) + def resume(self, operation_id: str, *, now: datetime) -> OperationRecord: """Re-admit a user-action operation after its external issue is resolved.""" @@ -337,6 +386,7 @@ def _record(row: sqlite3.Row) -> OperationRecord: worker_id=row["worker_id"], lease_expires_at=None if row["lease_expires_at"] is None else _parse_utc(row["lease_expires_at"]), next_run_at=None if row["next_run_at"] is None else _parse_utc(row["next_run_at"]), + cancel_requested=bool(row["cancel_requested"]), created_at=_parse_utc(row["created_at"]), updated_at=_parse_utc(row["updated_at"]), ) diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 8ca10d8..f128cc2 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -185,6 +185,39 @@ def test_waiting_user_operation_releases_lease_and_can_resume(self) -> None: resumed = self.repository.resume(operation.operation_id, now=self.now + timedelta(seconds=2)) self.assertEqual(resumed.state, "queued") + def test_queued_cancellation_is_terminal(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={}, + now=self.now, + ) + cancelled = self.repository.cancel(operation.operation_id, now=self.now + timedelta(seconds=1)) + self.assertEqual(cancelled.state, "cancelled") + self.assertFalse(cancelled.cancel_requested) + + def test_running_cancellation_is_acknowledged_at_checkpoint(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={}, + now=self.now, + ) + claimed = self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) + requested = self.repository.cancel(operation.operation_id, now=self.now + timedelta(seconds=1)) + self.assertEqual(requested.state, "running") + self.assertTrue(requested.cancel_requested) + completed = self.repository.checkpoint( + claimed.operation_id, + worker_id="worker-a", + checkpoint={"confirmed": ["occ-1"]}, + now=self.now + timedelta(seconds=2), + state="running", + ) + self.assertEqual(completed.state, "cancelled") + self.assertFalse(completed.cancel_requested) + self.assertIsNone(completed.worker_id) + if __name__ == "__main__": unittest.main() From f4b2c31d24823ef476cf340031b003751cf0ac7f Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 17:05:22 +0200 Subject: [PATCH 013/167] fix: migrate operation store schema forward --- .../infrastructure/sqlite_operations.py | 16 +++++++ tests/test_sqlite_operations.py | 47 +++++++++++++++++++ 2 files changed, 63 insertions(+) diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index ea80741..b83901c 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -56,6 +56,8 @@ class OperationRecord: class OperationRepository: """Transactional operation repository backed by one SQLite database.""" + SCHEMA_VERSION = 2 + def __init__(self, path: str = ":memory:") -> None: self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) self._connection.row_factory = sqlite3.Row @@ -73,6 +75,11 @@ def healthcheck(self) -> bool: return row is not None and row["healthy"] == 1 def _migrate(self) -> None: + current_version = int(self._connection.execute("PRAGMA user_version").fetchone()[0]) + if current_version > self.SCHEMA_VERSION: + raise RuntimeError( + f"operation store schema {current_version} is newer than supported {self.SCHEMA_VERSION}" + ) self._connection.executescript( """ CREATE TABLE IF NOT EXISTS operations ( @@ -93,6 +100,15 @@ def _migrate(self) -> None: ON operations (state, next_run_at, lease_expires_at); """ ) + columns = { + row[1] + for row in self._connection.execute("PRAGMA table_info(operations)").fetchall() + } + if "cancel_requested" not in columns: + self._connection.execute( + "ALTER TABLE operations ADD COLUMN cancel_requested INTEGER NOT NULL DEFAULT 0" + ) + self._connection.execute(f"PRAGMA user_version = {self.SCHEMA_VERSION}") def create( self, diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index f128cc2..0c282c6 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -1,6 +1,8 @@ from __future__ import annotations from datetime import datetime, timedelta, timezone +import sqlite3 +import tempfile import unittest from symphonia.infrastructure import IdempotencyConflict, LeaseConflict, OperationRepository @@ -218,6 +220,51 @@ def test_running_cancellation_is_acknowledged_at_checkpoint(self) -> None: self.assertFalse(completed.cancel_requested) self.assertIsNone(completed.worker_id) + def test_legacy_store_is_migrated_forward_without_losing_operations(self) -> None: + with tempfile.TemporaryDirectory() as directory: + path = f"{directory}/legacy.sqlite3" + connection = sqlite3.connect(path) + connection.executescript( + """ + CREATE TABLE operations ( + operation_id TEXT PRIMARY KEY, + operation_type TEXT NOT NULL, + state TEXT NOT NULL, + idempotency_key TEXT NOT NULL UNIQUE, + payload_json TEXT NOT NULL, + checkpoint_json TEXT NOT NULL, + worker_id TEXT, + lease_expires_at TEXT, + next_run_at TEXT, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ); + PRAGMA user_version = 1; + """ + ) + connection.execute( + """ + INSERT INTO operations ( + operation_id, operation_type, state, idempotency_key, + payload_json, checkpoint_json, created_at, updated_at + ) VALUES ('legacy-1', 'copy', 'queued', 'legacy-key', '{}', '{}', ?, ?) + """, + (self.now.isoformat(), self.now.isoformat()), + ) + connection.commit() + connection.close() + + repository = OperationRepository(path) + try: + record = repository.get("legacy-1") + self.assertFalse(record.cancel_requested) + self.assertEqual( + repository._connection.execute("PRAGMA user_version").fetchone()[0], + OperationRepository.SCHEMA_VERSION, + ) + finally: + repository.close() + if __name__ == "__main__": unittest.main() From 10f224cdd24af4eeff7df8f305ef537fec5c1603 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 17:07:28 +0200 Subject: [PATCH 014/167] feat: persist complete playlist projections --- docs/development/implementation-baseline.md | 1 + src/symphonia/domain/models.py | 9 +- src/symphonia/infrastructure/__init__.py | 4 + .../infrastructure/sqlite_library.py | 195 ++++++++++++++++++ src/symphonia/infrastructure/sqlite_plans.py | 3 +- src/symphonia/providers/importing.py | 5 +- tests/test_copy_planning.py | 20 +- tests/test_provider_import.py | 2 +- tests/test_sqlite_library.py | 68 ++++++ 9 files changed, 302 insertions(+), 5 deletions(-) create mode 100644 src/symphonia/infrastructure/sqlite_library.py create mode 100644 tests/test_sqlite_library.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 33def05..d0c5ede 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -15,6 +15,7 @@ The first implementation increment is intentionally narrower than any provider o - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. - An application workflow that persists plans, requires digest acceptance, and enqueues only accepted plans as durable operations. - A copy executor that creates target playlists through a semantic writer port, checkpoints each occurrence, and reconciles unknown writes before retry. +- SQLite persistence for complete playlist projections that keeps incomplete imports from replacing the last complete snapshot and isolates external IDs by namespace. ## Local container profile diff --git a/src/symphonia/domain/models.py b/src/symphonia/domain/models.py index ab1d068..e224b18 100644 --- a/src/symphonia/domain/models.py +++ b/src/symphonia/domain/models.py @@ -74,6 +74,7 @@ class PlaylistSnapshot: source_provider: str source_playlist_id: str entries: tuple[SourcePlaylistEntry, ...] + source_namespace: str = "" def __post_init__(self) -> None: for value, field_name in ( @@ -83,6 +84,10 @@ def __post_init__(self) -> None: ): if not value.strip(): raise ValueError(f"{field_name} must not be empty") + if not self.source_namespace.strip(): + # Kept optional for legacy in-memory callers; provider-backed + # snapshots should always provide the connection/catalog namespace. + object.__setattr__(self, "source_namespace", "default") positions = [entry.position for entry in self.entries] occurrence_ids = [entry.occurrence_id for entry in self.entries] @@ -126,6 +131,7 @@ class CopyPlan: policy: CopyPolicy entries: tuple[CopyPlanEntry, ...] digest: str + source_namespace: str = "default" @property def blocked(self) -> bool: @@ -202,6 +208,7 @@ def build_copy_plan( "source_snapshot_id": snapshot.snapshot_id, "source_provider": snapshot.source_provider, "source_playlist_id": snapshot.source_playlist_id, + "source_namespace": snapshot.source_namespace, "target_provider": target_provider, "target_playlist_name": target_playlist_name, "target_visibility": target_visibility, @@ -231,6 +238,7 @@ def build_copy_plan( policy=policy, entries=tuple(plan_entries), digest=digest, + source_namespace=snapshot.source_namespace, ) @@ -238,4 +246,3 @@ def source_entries(entries: Iterable[SourcePlaylistEntry]) -> tuple[SourcePlayli """Convenience helper for callers constructing a snapshot.""" return tuple(entries) - diff --git a/src/symphonia/infrastructure/__init__.py b/src/symphonia/infrastructure/__init__.py index 64bf216..a12354d 100644 --- a/src/symphonia/infrastructure/__init__.py +++ b/src/symphonia/infrastructure/__init__.py @@ -8,14 +8,18 @@ ) from .sqlite_plans import CopyPlanNotFound, CopyPlanRepository, StoredCopyPlan from .sqlite_resolutions import ResolutionDecisionRepository +from .sqlite_library import IncompleteCollectionError, PlaylistProjectionRepository, StoredPlaylistSnapshot __all__ = [ "CopyPlanNotFound", "CopyPlanRepository", + "IncompleteCollectionError", "IdempotencyConflict", "LeaseConflict", "OperationNotFound", "OperationRepository", + "PlaylistProjectionRepository", "ResolutionDecisionRepository", "StoredCopyPlan", + "StoredPlaylistSnapshot", ] diff --git a/src/symphonia/infrastructure/sqlite_library.py b/src/symphonia/infrastructure/sqlite_library.py new file mode 100644 index 0000000..35ba483 --- /dev/null +++ b/src/symphonia/infrastructure/sqlite_library.py @@ -0,0 +1,195 @@ +"""Atomic persistence for complete imported playlist snapshots.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime, timezone +import sqlite3 + +from symphonia.domain.models import EntryClassification, PlaylistSnapshot, SourcePlaylistEntry +from symphonia.providers.contracts import ProviderPlaylistEntry +from symphonia.providers.importing import CollectionImportResult + + +def _utc(value: datetime) -> str: + if value.tzinfo is None: + raise ValueError("timestamps must be timezone-aware") + return value.astimezone(timezone.utc).isoformat(timespec="microseconds") + + +def _parse_utc(value: str) -> datetime: + return datetime.fromisoformat(value).astimezone(timezone.utc) + + +class IncompleteCollectionError(ValueError): + pass + + +@dataclass(frozen=True, slots=True) +class StoredPlaylistSnapshot: + snapshot_id: str + provider: str + namespace: str + playlist_id: str + revision: str | None + published_at: datetime + snapshot: PlaylistSnapshot + + +class PlaylistProjectionRepository: + """Keep the last complete projection when a later import is incomplete.""" + + def __init__(self, path: str = ":memory:") -> None: + self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) + self._connection.row_factory = sqlite3.Row + self._migrate() + + def close(self) -> None: + self._connection.close() + + def _migrate(self) -> None: + self._connection.executescript( + """ + CREATE TABLE IF NOT EXISTS playlist_snapshots ( + snapshot_id TEXT PRIMARY KEY, + provider TEXT NOT NULL, + namespace TEXT NOT NULL, + playlist_id TEXT NOT NULL, + revision TEXT, + published_at TEXT NOT NULL, + UNIQUE (provider, namespace, playlist_id, snapshot_id) + ); + CREATE TABLE IF NOT EXISTS playlist_snapshot_entries ( + snapshot_id TEXT NOT NULL REFERENCES playlist_snapshots(snapshot_id), + occurrence_id TEXT NOT NULL, + position INTEGER NOT NULL, + provider_track_id TEXT NOT NULL, + provider_track_namespace TEXT NOT NULL, + media_kind TEXT NOT NULL, + available INTEGER NOT NULL, + PRIMARY KEY (snapshot_id, occurrence_id), + UNIQUE (snapshot_id, position) + ); + CREATE TABLE IF NOT EXISTS current_playlist_snapshots ( + provider TEXT NOT NULL, + namespace TEXT NOT NULL, + playlist_id TEXT NOT NULL, + snapshot_id TEXT NOT NULL REFERENCES playlist_snapshots(snapshot_id), + PRIMARY KEY (provider, namespace, playlist_id) + ); + """ + ) + + def publish( + self, + result: CollectionImportResult, + *, + snapshot_id: str, + published_at: datetime, + ) -> StoredPlaylistSnapshot: + """Publish a complete result atomically; reject incomplete results.""" + + if not result.complete: + raise IncompleteCollectionError("incomplete collection cannot replace the current projection") + if not snapshot_id.strip(): + raise ValueError("snapshot_id must not be empty") + if not result.entries and result.playlist == "": + raise ValueError("result must identify a playlist") + timestamp = _utc(published_at) + self._connection.execute("BEGIN IMMEDIATE") + try: + self._connection.execute( + """ + INSERT INTO playlist_snapshots ( + snapshot_id, provider, namespace, playlist_id, revision, published_at + ) VALUES (?, ?, ?, ?, ?, ?) + """, + (snapshot_id, result.provider, result.namespace, result.playlist, result.revision, timestamp), + ) + self._connection.executemany( + """ + INSERT INTO playlist_snapshot_entries ( + snapshot_id, occurrence_id, position, provider_track_id, + provider_track_namespace, media_kind, available + ) VALUES (?, ?, ?, ?, ?, ?, ?) + """, + [ + ( + snapshot_id, + entry.occurrence_id, + entry.position, + entry.track.object_id, + entry.track.namespace, + entry.media_kind.value, + int(entry.available), + ) + for entry in result.entries + ], + ) + self._connection.execute( + """ + INSERT INTO current_playlist_snapshots (provider, namespace, playlist_id, snapshot_id) + VALUES (?, ?, ?, ?) + ON CONFLICT(provider, namespace, playlist_id) + DO UPDATE SET snapshot_id = excluded.snapshot_id + """, + (result.provider, result.namespace, result.playlist, snapshot_id), + ) + self._connection.execute("COMMIT") + except Exception: + self._connection.execute("ROLLBACK") + raise + return self.get(snapshot_id) + + def get(self, snapshot_id: str) -> StoredPlaylistSnapshot: + row = self._connection.execute( + "SELECT * FROM playlist_snapshots WHERE snapshot_id = ?", (snapshot_id,) + ).fetchone() + if row is None: + raise KeyError(snapshot_id) + entries = self._entries(snapshot_id, row["provider"]) + return StoredPlaylistSnapshot( + snapshot_id=row["snapshot_id"], + provider=row["provider"], + namespace=row["namespace"], + playlist_id=row["playlist_id"], + revision=row["revision"], + published_at=_parse_utc(row["published_at"]), + snapshot=PlaylistSnapshot( + snapshot_id=row["snapshot_id"], + source_provider=row["provider"], + source_playlist_id=row["playlist_id"], + entries=entries, + source_namespace=row["namespace"], + ), + ) + + def current(self, *, provider: str, namespace: str, playlist_id: str) -> StoredPlaylistSnapshot | None: + row = self._connection.execute( + """ + SELECT snapshot_id FROM current_playlist_snapshots + WHERE provider = ? AND namespace = ? AND playlist_id = ? + """, + (provider, namespace, playlist_id), + ).fetchone() + return None if row is None else self.get(row["snapshot_id"]) + + def _entries(self, snapshot_id: str, provider: str) -> tuple[SourcePlaylistEntry, ...]: + rows = self._connection.execute( + """ + SELECT * FROM playlist_snapshot_entries + WHERE snapshot_id = ? ORDER BY position + """, + (snapshot_id,), + ).fetchall() + return tuple( + SourcePlaylistEntry( + occurrence_id=row["occurrence_id"], + position=row["position"], + provider_track_id=row["provider_track_id"], + classification=EntryClassification.UNMATCHED if row["available"] else EntryClassification.UNAVAILABLE, + reason=None if row["available"] else "provider reported item unavailable", + ) + for row in rows + ) + diff --git a/src/symphonia/infrastructure/sqlite_plans.py b/src/symphonia/infrastructure/sqlite_plans.py index 25e3ac2..c7501a9 100644 --- a/src/symphonia/infrastructure/sqlite_plans.py +++ b/src/symphonia/infrastructure/sqlite_plans.py @@ -102,6 +102,7 @@ def _serialize(plan: CopyPlan) -> dict[str, Any]: "source_snapshot_id": plan.source_snapshot_id, "source_provider": plan.source_provider, "source_playlist_id": plan.source_playlist_id, + "source_namespace": plan.source_namespace, "target_provider": plan.target_provider, "target_playlist_name": plan.target_playlist_name, "target_visibility": plan.target_visibility, @@ -144,5 +145,5 @@ def _deserialize(payload: dict[str, Any]) -> CopyPlan: for entry in payload["entries"] ), digest=payload["digest"], + source_namespace=payload.get("source_namespace", "default"), ) - diff --git a/src/symphonia/providers/importing.py b/src/symphonia/providers/importing.py index 551e2cf..292aa02 100644 --- a/src/symphonia/providers/importing.py +++ b/src/symphonia/providers/importing.py @@ -27,6 +27,7 @@ class CollectionImportResult: complete: bool issues: tuple[ImportIssue, ...] revision: str | None + namespace: str = "default" def collect_playlist_pages(pages: list[ProviderPlaylistPage] | tuple[ProviderPlaylistPage, ...]) -> CollectionImportResult: @@ -38,7 +39,7 @@ def collect_playlist_pages(pages: list[ProviderPlaylistPage] | tuple[ProviderPla """ if not pages: - return CollectionImportResult("", "", (), False, (ImportIssue.NO_PAGES,), None) + return CollectionImportResult("", "", (), False, (ImportIssue.NO_PAGES,), None, "default") first = pages[0].playlist issues: list[ImportIssue] = [] @@ -88,6 +89,7 @@ def collect_playlist_pages(pages: list[ProviderPlaylistPage] | tuple[ProviderPla complete=complete and not any(issue in {ImportIssue.CONFLICTING_OCCURRENCE, ImportIssue.REPEATED_CURSOR} for issue in issues), issues=tuple(dict.fromkeys(issues)), revision=pages[-1].revision, + namespace=first.namespace, ) @@ -113,4 +115,5 @@ def to_playlist_snapshot(result: CollectionImportResult, snapshot_id: str) -> Pl source_provider=result.provider, source_playlist_id=result.playlist, entries=entries, + source_namespace=result.namespace, ) diff --git a/tests/test_copy_planning.py b/tests/test_copy_planning.py index a78332f..996d63c 100644 --- a/tests/test_copy_planning.py +++ b/tests/test_copy_planning.py @@ -93,7 +93,25 @@ def test_snapshot_rejects_duplicate_positions(self) -> None: SourcePlaylistEntry("occ-2", 0, "sp-2", EntryClassification.READY, "yt-2"), ) + def test_source_namespace_is_part_of_plan_digest(self) -> None: + first = PlaylistSnapshot( + "snapshot-1", + "spotify", + "playlist-1", + (SourcePlaylistEntry("occ-1", 0, "sp-1", EntryClassification.READY, "yt-1"),), + "connection-a", + ) + second = PlaylistSnapshot( + "snapshot-1", + "spotify", + "playlist-1", + (SourcePlaylistEntry("occ-1", 0, "sp-1", EntryClassification.READY, "yt-1"),), + "connection-b", + ) + first_plan = self.service.plan(first, target_provider="youtube", target_playlist_name="Rock") + second_plan = self.service.plan(second, target_provider="youtube", target_playlist_name="Rock") + self.assertNotEqual(first_plan.digest, second_plan.digest) + if __name__ == "__main__": unittest.main() - diff --git a/tests/test_provider_import.py b/tests/test_provider_import.py index f856fd0..0772b50 100644 --- a/tests/test_provider_import.py +++ b/tests/test_provider_import.py @@ -81,8 +81,8 @@ def test_snapshot_keeps_unavailable_entries_visible(self) -> None: snapshot = to_playlist_snapshot(result, "snapshot-1") self.assertEqual(snapshot.entries[0].classification, EntryClassification.UNAVAILABLE) self.assertEqual(snapshot.entries[0].occurrence_id, "occ-1") + self.assertEqual(snapshot.source_namespace, "connection-1") if __name__ == "__main__": unittest.main() - diff --git a/tests/test_sqlite_library.py b/tests/test_sqlite_library.py new file mode 100644 index 0000000..f17c5a4 --- /dev/null +++ b/tests/test_sqlite_library.py @@ -0,0 +1,68 @@ +from __future__ import annotations + +from datetime import datetime, timezone +import unittest + +from symphonia.domain import EntryClassification +from symphonia.infrastructure import IncompleteCollectionError, PlaylistProjectionRepository +from symphonia.providers import ( + MediaKind, + ProviderObjectRef, + ProviderPlaylistEntry, + ProviderPlaylistPage, + collect_playlist_pages, +) + + +NOW = datetime(2026, 9, 20, 12, 0, tzinfo=timezone.utc) + + +def page(*, namespace: str = "connection-1", complete: bool = True, available: bool = True): + playlist = ProviderObjectRef("spotify", "playlist", "playlist-1", namespace) + track = ProviderObjectRef("spotify", "track", "track-1", namespace) + item = ProviderPlaylistEntry("occ-1", 0, track, MediaKind.TRACK, available=available) + return ProviderPlaylistPage(playlist, (item,), None, None, complete, revision="rev-1") + + +class PlaylistProjectionRepositoryTests(unittest.TestCase): + def setUp(self) -> None: + self.repository = PlaylistProjectionRepository() + + def tearDown(self) -> None: + self.repository.close() + + def test_complete_result_is_published_with_namespace_and_availability(self) -> None: + result = collect_playlist_pages([page(available=False)]) + stored = self.repository.publish(result, snapshot_id="snapshot-1", published_at=NOW) + current = self.repository.current(provider="spotify", namespace="connection-1", playlist_id="playlist-1") + self.assertEqual(stored.snapshot_id, "snapshot-1") + self.assertEqual(current.snapshot.source_namespace, "connection-1") + self.assertEqual(current.snapshot.entries[0].classification, EntryClassification.UNAVAILABLE) + + def test_incomplete_result_cannot_replace_current_projection(self) -> None: + complete = collect_playlist_pages([page()]) + self.repository.publish(complete, snapshot_id="snapshot-1", published_at=NOW) + incomplete = collect_playlist_pages([page(complete=False)]) + with self.assertRaises(IncompleteCollectionError): + self.repository.publish(incomplete, snapshot_id="snapshot-2", published_at=NOW) + current = self.repository.current(provider="spotify", namespace="connection-1", playlist_id="playlist-1") + self.assertEqual(current.snapshot_id, "snapshot-1") + + def test_same_external_playlist_id_isolated_by_namespace(self) -> None: + first = collect_playlist_pages([page(namespace="connection-1")]) + second = collect_playlist_pages([page(namespace="connection-2")]) + self.repository.publish(first, snapshot_id="snapshot-1", published_at=NOW) + self.repository.publish(second, snapshot_id="snapshot-2", published_at=NOW) + self.assertEqual( + self.repository.current(provider="spotify", namespace="connection-1", playlist_id="playlist-1").snapshot_id, + "snapshot-1", + ) + self.assertEqual( + self.repository.current(provider="spotify", namespace="connection-2", playlist_id="playlist-1").snapshot_id, + "snapshot-2", + ) + + +if __name__ == "__main__": + unittest.main() + From 10c662429b32c232d477aab93202e1092ca64d80 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 17:08:10 +0200 Subject: [PATCH 015/167] feat: publish imports through application service --- docs/development/implementation-baseline.md | 1 + src/symphonia/application/__init__.py | 9 +++- src/symphonia/application/library_import.py | 42 +++++++++++++++++ tests/test_library_import.py | 51 +++++++++++++++++++++ 4 files changed, 102 insertions(+), 1 deletion(-) create mode 100644 src/symphonia/application/library_import.py create mode 100644 tests/test_library_import.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index d0c5ede..fa66c80 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -16,6 +16,7 @@ The first implementation increment is intentionally narrower than any provider o - An application workflow that persists plans, requires digest acceptance, and enqueues only accepted plans as durable operations. - A copy executor that creates target playlists through a semantic writer port, checkpoints each occurrence, and reconciles unknown writes before retry. - SQLite persistence for complete playlist projections that keeps incomplete imports from replacing the last complete snapshot and isolates external IDs by namespace. +- An application import use case that reports partial results without advancing freshness or replacing the last complete projection. ## Local container profile diff --git a/src/symphonia/application/__init__.py b/src/symphonia/application/__init__.py index e1e0de1..7be8bbb 100644 --- a/src/symphonia/application/__init__.py +++ b/src/symphonia/application/__init__.py @@ -3,5 +3,12 @@ from .copy_planning import CopyPlanningService from .copy_workflow import CopyWorkflowService from .copy_execution import CopyExecutionService +from .library_import import ImportPublication, LibraryImportService -__all__ = ["CopyExecutionService", "CopyPlanningService", "CopyWorkflowService"] +__all__ = [ + "CopyExecutionService", + "CopyPlanningService", + "CopyWorkflowService", + "ImportPublication", + "LibraryImportService", +] diff --git a/src/symphonia/application/library_import.py b/src/symphonia/application/library_import.py new file mode 100644 index 0000000..1bdc021 --- /dev/null +++ b/src/symphonia/application/library_import.py @@ -0,0 +1,42 @@ +"""Application use case for publishing normalized playlist imports.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime + +from symphonia.infrastructure.sqlite_library import PlaylistProjectionRepository, StoredPlaylistSnapshot +from symphonia.providers.importing import CollectionImportResult, ImportIssue + + +@dataclass(frozen=True, slots=True) +class ImportPublication: + state: str + snapshot: StoredPlaylistSnapshot | None + retained_current: StoredPlaylistSnapshot | None + issues: tuple[ImportIssue, ...] + + +@dataclass(frozen=True, slots=True) +class LibraryImportService: + projections: PlaylistProjectionRepository + + def publish_playlist( + self, + result: CollectionImportResult, + *, + snapshot_id: str, + observed_at: datetime, + ) -> ImportPublication: + """Publish complete data or retain the prior complete projection.""" + + if not result.complete: + retained = self.projections.current( + provider=result.provider, + namespace=result.namespace, + playlist_id=result.playlist, + ) + return ImportPublication("partial", None, retained, result.issues) + snapshot = self.projections.publish(result, snapshot_id=snapshot_id, published_at=observed_at) + return ImportPublication("succeeded", snapshot, snapshot, result.issues) + diff --git a/tests/test_library_import.py b/tests/test_library_import.py new file mode 100644 index 0000000..26e4f0c --- /dev/null +++ b/tests/test_library_import.py @@ -0,0 +1,51 @@ +from __future__ import annotations + +from datetime import datetime, timezone +import unittest + +from symphonia.application import LibraryImportService +from symphonia.infrastructure import PlaylistProjectionRepository +from symphonia.providers import ( + MediaKind, + ProviderObjectRef, + ProviderPlaylistEntry, + ProviderPlaylistPage, + collect_playlist_pages, +) + + +NOW = datetime(2026, 9, 20, 12, 0, tzinfo=timezone.utc) + + +def result(complete: bool): + playlist = ProviderObjectRef("spotify", "playlist", "playlist-1", "connection-1") + track = ProviderObjectRef("spotify", "track", "track-1", "connection-1") + item = ProviderPlaylistEntry("occ-1", 0, track, MediaKind.TRACK) + return collect_playlist_pages([ProviderPlaylistPage(playlist, (item,), None, None, complete)]) + + +class LibraryImportServiceTests(unittest.TestCase): + def setUp(self) -> None: + self.repository = PlaylistProjectionRepository() + self.service = LibraryImportService(self.repository) + + def tearDown(self) -> None: + self.repository.close() + + def test_complete_import_publishes_snapshot(self) -> None: + publication = self.service.publish_playlist(result(True), snapshot_id="snapshot-1", observed_at=NOW) + self.assertEqual(publication.state, "succeeded") + self.assertEqual(publication.snapshot.snapshot_id, "snapshot-1") + self.assertEqual(publication.retained_current.snapshot_id, "snapshot-1") + + def test_partial_import_retains_last_complete_snapshot(self) -> None: + self.service.publish_playlist(result(True), snapshot_id="snapshot-1", observed_at=NOW) + publication = self.service.publish_playlist(result(False), snapshot_id="snapshot-2", observed_at=NOW) + self.assertEqual(publication.state, "partial") + self.assertIsNone(publication.snapshot) + self.assertEqual(publication.retained_current.snapshot_id, "snapshot-1") + + +if __name__ == "__main__": + unittest.main() + From 8d93d825cbb90378cc46da72df92d68cca470241 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:08:40 +0200 Subject: [PATCH 016/167] feat: add durable operation audit events --- docs/development/implementation-baseline.md | 1 + src/symphonia/infrastructure/__init__.py | 2 + .../infrastructure/sqlite_operations.py | 160 ++++++++++++++++++ tests/test_sqlite_operations.py | 46 +++++ 4 files changed, 209 insertions(+) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index fa66c80..7988552 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -10,6 +10,7 @@ The first implementation increment is intentionally narrower than any provider o - Python 3.11+ package under `src/symphonia`. - Dependency-free domain values for ordered playlist snapshots, the six product entry classifications, copy policies, immutable copy plans, and acceptance digests. - Dependency-free SQLite operation repository proving idempotent creation, time-bounded worker leases, checkpoint ownership, retry scheduling, and expired-lease recovery. +- Append-only SQLite operation event history for auditable transitions with sanitized checkpoint summaries. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. diff --git a/src/symphonia/infrastructure/__init__.py b/src/symphonia/infrastructure/__init__.py index a12354d..d9cd02f 100644 --- a/src/symphonia/infrastructure/__init__.py +++ b/src/symphonia/infrastructure/__init__.py @@ -3,6 +3,7 @@ from .sqlite_operations import ( IdempotencyConflict, OperationNotFound, + OperationEvent, OperationRepository, LeaseConflict, ) @@ -17,6 +18,7 @@ "IdempotencyConflict", "LeaseConflict", "OperationNotFound", + "OperationEvent", "OperationRepository", "PlaylistProjectionRepository", "ResolutionDecisionRepository", diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index b83901c..5131029 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -53,6 +53,19 @@ class OperationRecord: updated_at: datetime +@dataclass(frozen=True, slots=True) +class OperationEvent: + """Append-only audit record for an operation state transition.""" + + sequence: int + operation_id: str + event_type: str + state: str + worker_id: str | None + payload: dict[str, Any] + created_at: datetime + + class OperationRepository: """Transactional operation repository backed by one SQLite database.""" @@ -98,6 +111,17 @@ def _migrate(self) -> None: ); CREATE INDEX IF NOT EXISTS operations_eligibility_idx ON operations (state, next_run_at, lease_expires_at); + CREATE TABLE IF NOT EXISTS operation_events ( + sequence INTEGER PRIMARY KEY AUTOINCREMENT, + operation_id TEXT NOT NULL REFERENCES operations(operation_id), + event_type TEXT NOT NULL, + state TEXT NOT NULL, + worker_id TEXT, + payload_json TEXT NOT NULL, + created_at TEXT NOT NULL + ); + CREATE INDEX IF NOT EXISTS operation_events_operation_idx + ON operation_events (operation_id, sequence); """ ) columns = { @@ -126,6 +150,7 @@ def create( operation_id = operation_id or str(uuid.uuid4()) timestamp = _utc(now) payload_json = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + self._connection.execute("BEGIN IMMEDIATE") try: self._connection.execute( """ @@ -136,13 +161,26 @@ def create( """, (operation_id, operation_type, idempotency_key, payload_json, timestamp, timestamp), ) + self._append_event( + operation_id=operation_id, + event_type="created", + state="queued", + worker_id=None, + payload={}, + created_at=timestamp, + ) + self._connection.execute("COMMIT") except sqlite3.IntegrityError: + self._connection.execute("ROLLBACK") existing = self._by_idempotency(idempotency_key) if existing is None: raise if existing.operation_type != operation_type or existing.payload != payload: raise IdempotencyConflict("idempotency key is already bound to another operation") return existing + except Exception: + self._connection.execute("ROLLBACK") + raise return self.get(operation_id) def get(self, operation_id: str) -> OperationRecord: @@ -153,6 +191,21 @@ def get(self, operation_id: str) -> OperationRecord: raise OperationNotFound(operation_id) return self._record(row) + def events(self, operation_id: str) -> tuple[OperationEvent, ...]: + """Return the immutable audit trail in transition order.""" + + self.get(operation_id) + rows = self._connection.execute( + """ + SELECT sequence, operation_id, event_type, state, worker_id, payload_json, created_at + FROM operation_events + WHERE operation_id = ? + ORDER BY sequence ASC + """, + (operation_id,), + ).fetchall() + return tuple(self._event(row) for row in rows) + def claim( self, operation_id: str, @@ -195,6 +248,14 @@ def claim( """, (worker_id, expires_text, now_text, operation_id), ) + self._append_event( + operation_id=operation_id, + event_type="claimed", + state="running", + worker_id=worker_id, + payload={"lease_expires_at": expires_text}, + created_at=now_text, + ) self._connection.execute("COMMIT") except Exception: self._connection.execute("ROLLBACK") @@ -230,6 +291,14 @@ def renew_lease( "UPDATE operations SET lease_expires_at = ?, updated_at = ? WHERE operation_id = ?", (expires_text, now_text, operation_id), ) + self._append_event( + operation_id=operation_id, + event_type="lease_renewed", + state="running", + worker_id=worker_id, + payload={"lease_expires_at": expires_text}, + created_at=now_text, + ) self._connection.execute("COMMIT") except Exception: self._connection.execute("ROLLBACK") @@ -282,6 +351,14 @@ def checkpoint( operation_id, ), ) + self._append_event( + operation_id=operation_id, + event_type="checkpointed", + state=effective_state, + worker_id=worker_id if effective_state == "running" else None, + payload=self._checkpoint_summary(checkpoint), + created_at=now_text, + ) self._connection.execute("COMMIT") except Exception: self._connection.execute("ROLLBACK") @@ -307,6 +384,14 @@ def cancel(self, operation_id: str, *, now: datetime) -> OperationRecord: "UPDATE operations SET cancel_requested = 1, updated_at = ? WHERE operation_id = ?", (now_text, operation_id), ) + self._append_event( + operation_id=operation_id, + event_type="cancellation_requested", + state="running", + worker_id=row["worker_id"], + payload={}, + created_at=now_text, + ) else: self._connection.execute( """ @@ -317,6 +402,14 @@ def cancel(self, operation_id: str, *, now: datetime) -> OperationRecord: """, (now_text, operation_id), ) + self._append_event( + operation_id=operation_id, + event_type="cancelled", + state="cancelled", + worker_id=None, + payload={}, + created_at=now_text, + ) self._connection.execute("COMMIT") except Exception: self._connection.execute("ROLLBACK") @@ -340,6 +433,14 @@ def resume(self, operation_id: str, *, now: datetime) -> OperationRecord: "UPDATE operations SET state = 'queued', updated_at = ? WHERE operation_id = ?", (now_text, operation_id), ) + self._append_event( + operation_id=operation_id, + event_type="resumed", + state="queued", + worker_id=None, + payload={}, + created_at=now_text, + ) self._connection.execute("COMMIT") except Exception: self._connection.execute("ROLLBACK") @@ -378,6 +479,14 @@ def schedule_retry( """, (checkpoint_json, next_run_text, now_text, operation_id), ) + self._append_event( + operation_id=operation_id, + event_type="retry_scheduled", + state="retry_scheduled", + worker_id=None, + payload={"next_run_at": next_run_text, **self._checkpoint_summary(checkpoint)}, + created_at=now_text, + ) self._connection.execute("COMMIT") except Exception: self._connection.execute("ROLLBACK") @@ -390,6 +499,45 @@ def _by_idempotency(self, idempotency_key: str) -> OperationRecord | None: ).fetchone() return None if row is None else self._record(row) + def _append_event( + self, + *, + operation_id: str, + event_type: str, + state: str, + worker_id: str | None, + payload: dict[str, Any], + created_at: str, + ) -> None: + self._connection.execute( + """ + INSERT INTO operation_events ( + operation_id, event_type, state, worker_id, payload_json, created_at + ) VALUES (?, ?, ?, ?, ?, ?) + """, + ( + operation_id, + event_type, + state, + worker_id, + json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")), + created_at, + ), + ) + + @staticmethod + def _checkpoint_summary(checkpoint: dict[str, Any]) -> dict[str, Any]: + """Keep audit data useful while excluding checkpoint values by default.""" + + summary: dict[str, Any] = { + "checkpoint_keys": sorted(str(key) for key in checkpoint), + } + for key in ("confirmed_occurrences", "issues"): + value = checkpoint.get(key) + if isinstance(value, (list, tuple, set)): + summary[f"{key}_count"] = len(value) + return summary + @staticmethod def _record(row: sqlite3.Row) -> OperationRecord: return OperationRecord( @@ -406,3 +554,15 @@ def _record(row: sqlite3.Row) -> OperationRecord: created_at=_parse_utc(row["created_at"]), updated_at=_parse_utc(row["updated_at"]), ) + + @staticmethod + def _event(row: sqlite3.Row) -> OperationEvent: + return OperationEvent( + sequence=row["sequence"], + operation_id=row["operation_id"], + event_type=row["event_type"], + state=row["state"], + worker_id=row["worker_id"], + payload=json.loads(row["payload_json"]), + created_at=_parse_utc(row["created_at"]), + ) diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 0c282c6..1b74faf 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -72,6 +72,42 @@ def test_lease_claim_checkpoint_and_terminal_state(self) -> None: self.assertEqual(completed.checkpoint, {"confirmed": ["occ-1"]}) self.assertIsNone(completed.worker_id) + events = self.repository.events(operation.operation_id) + self.assertEqual( + [event.event_type for event in events], + ["created", "claimed", "checkpointed"], + ) + self.assertEqual([event.state for event in events], ["queued", "running", "succeeded"]) + self.assertEqual(events[1].worker_id, "worker-a") + + def test_checkpoint_events_store_only_sanitized_summary(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={"plan_digest": "abc"}, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) + self.repository.checkpoint( + operation.operation_id, + worker_id="worker-a", + checkpoint={ + "confirmed_occurrences": ["occ-1", "occ-2"], + "issues": [{"code": "provider_error"}], + "secret_token": "must-not-be-audit-payload", + }, + now=self.now + timedelta(seconds=1), + ) + + event = self.repository.events(operation.operation_id)[-1] + self.assertEqual(event.payload["confirmed_occurrences_count"], 2) + self.assertEqual(event.payload["issues_count"], 1) + self.assertEqual( + event.payload["checkpoint_keys"], + ["confirmed_occurrences", "issues", "secret_token"], + ) + self.assertNotIn("must-not-be-audit-payload", event.payload) + def test_only_lease_owner_can_checkpoint(self) -> None: operation = self.repository.create( operation_type="import", @@ -219,6 +255,10 @@ def test_running_cancellation_is_acknowledged_at_checkpoint(self) -> None: self.assertEqual(completed.state, "cancelled") self.assertFalse(completed.cancel_requested) self.assertIsNone(completed.worker_id) + self.assertEqual( + [event.event_type for event in self.repository.events(operation.operation_id)], + ["created", "claimed", "cancellation_requested", "checkpointed"], + ) def test_legacy_store_is_migrated_forward_without_losing_operations(self) -> None: with tempfile.TemporaryDirectory() as directory: @@ -258,6 +298,12 @@ def test_legacy_store_is_migrated_forward_without_losing_operations(self) -> Non try: record = repository.get("legacy-1") self.assertFalse(record.cancel_requested) + self.assertEqual( + repository._connection.execute( + "SELECT COUNT(*) FROM operation_events" + ).fetchone()[0], + 0, + ) self.assertEqual( repository._connection.execute("PRAGMA user_version").fetchone()[0], OperationRepository.SCHEMA_VERSION, From 4af9f511f1df1e516bd5e741a4cdebb8f11cbd24 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:10:23 +0200 Subject: [PATCH 017/167] feat: add atomic next-operation claiming --- docs/development/implementation-baseline.md | 1 + .../infrastructure/sqlite_operations.py | 72 +++++++++++++++++++ tests/test_sqlite_operations.py | 58 +++++++++++++++ 3 files changed, 131 insertions(+) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 7988552..7af38c9 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -10,6 +10,7 @@ The first implementation increment is intentionally narrower than any provider o - Python 3.11+ package under `src/symphonia`. - Dependency-free domain values for ordered playlist snapshots, the six product entry classifications, copy policies, immutable copy plans, and acceptance digests. - Dependency-free SQLite operation repository proving idempotent creation, time-bounded worker leases, checkpoint ownership, retry scheduling, and expired-lease recovery. +- Atomic scheduler-facing claim selection for queued, due-retry, and expired-lease operations. - Append-only SQLite operation event history for auditable transitions with sanitized checkpoint summaries. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 5131029..4937a5d 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -262,6 +262,78 @@ def claim( raise return self.get(operation_id) + def claim_next( + self, + *, + worker_id: str, + now: datetime, + lease_seconds: int = 30, + operation_type: str | None = None, + ) -> OperationRecord | None: + """Atomically claim the oldest queued, due, or expired operation. + + This is the scheduler-facing primitive. It deliberately selects only + durable eligibility; handler dispatch remains an application concern. + """ + + if lease_seconds <= 0: + raise ValueError("lease_seconds must be positive") + now_text = _utc(now) + expires_text = _utc(now + timedelta(seconds=lease_seconds)) + type_clause = " AND operation_type = ?" if operation_type is not None else "" + parameters: tuple[Any, ...] = (now_text, now_text) + if operation_type is not None: + parameters += (operation_type,) + self._connection.execute("BEGIN IMMEDIATE") + try: + row = self._connection.execute( + f""" + SELECT * + FROM operations + WHERE cancel_requested = 0 + AND ( + state = 'queued' + OR (state = 'retry_scheduled' AND next_run_at IS NOT NULL AND next_run_at <= ?) + OR (state = 'running' AND (lease_expires_at IS NULL OR lease_expires_at <= ?)) + ) + {type_clause} + ORDER BY + CASE WHEN state = 'running' THEN COALESCE(lease_expires_at, created_at) + ELSE COALESCE(next_run_at, created_at) + END ASC, + created_at ASC, + operation_id ASC + LIMIT 1 + """, + parameters, + ).fetchone() + if row is None: + self._connection.execute("COMMIT") + return None + operation_id = row["operation_id"] + self._connection.execute( + """ + UPDATE operations + SET state = 'running', worker_id = ?, lease_expires_at = ?, + next_run_at = NULL, updated_at = ? + WHERE operation_id = ? + """, + (worker_id, expires_text, now_text, operation_id), + ) + self._append_event( + operation_id=operation_id, + event_type="claimed", + state="running", + worker_id=worker_id, + payload={"lease_expires_at": expires_text}, + created_at=now_text, + ) + self._connection.execute("COMMIT") + except Exception: + self._connection.execute("ROLLBACK") + raise + return self.get(operation_id) + def renew_lease( self, operation_id: str, diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 1b74faf..f0e5046 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -165,6 +165,64 @@ def test_expired_running_lease_can_be_reclaimed(self) -> None: ) self.assertEqual(reclaimed.worker_id, "worker-b") + def test_claim_next_selects_queued_and_due_retry_work(self) -> None: + queued = self.repository.create( + operation_type="copy", + idempotency_key="copy-queued", + payload={}, + now=self.now, + ) + retry = self.repository.create( + operation_type="import", + idempotency_key="import-retry", + payload={}, + now=self.now + timedelta(seconds=1), + ) + self.repository.claim(retry.operation_id, worker_id="worker-a", now=self.now) + self.repository.schedule_retry( + retry.operation_id, + worker_id="worker-a", + next_run_at=self.now + timedelta(minutes=1), + checkpoint={}, + now=self.now + timedelta(seconds=1), + ) + + claimed = self.repository.claim_next(worker_id="worker-b", now=self.now + timedelta(seconds=2)) + self.assertIsNotNone(claimed) + self.assertEqual(claimed.operation_id, queued.operation_id) + self.assertIsNone( + self.repository.claim_next( + worker_id="worker-b", + now=self.now + timedelta(seconds=30), + operation_type="import", + ) + ) + + due = self.repository.claim_next( + worker_id="worker-c", + now=self.now + timedelta(minutes=2), + operation_type="import", + ) + self.assertIsNotNone(due) + self.assertEqual(due.operation_id, retry.operation_id) + + def test_claim_next_recovers_an_expired_running_lease(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={}, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now, lease_seconds=5) + + recovered = self.repository.claim_next( + worker_id="worker-b", + now=self.now + timedelta(seconds=6), + ) + self.assertIsNotNone(recovered) + self.assertEqual(recovered.operation_id, operation.operation_id) + self.assertEqual(recovered.worker_id, "worker-b") + def test_retry_cannot_be_claimed_before_its_scheduled_time(self) -> None: operation = self.repository.create( operation_type="copy", From 50dbd8769ad17c80d0000a103e94b951e32e9c28 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:11:40 +0200 Subject: [PATCH 018/167] feat: add redacted operation diagnostics --- docs/development/implementation-baseline.md | 1 + .../infrastructure/sqlite_operations.py | 33 +++++++++++++++++++ tests/test_sqlite_operations.py | 29 ++++++++++++++++ 3 files changed, 63 insertions(+) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 7af38c9..4bfd1e6 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -12,6 +12,7 @@ The first implementation increment is intentionally narrower than any provider o - Dependency-free SQLite operation repository proving idempotent creation, time-bounded worker leases, checkpoint ownership, retry scheduling, and expired-lease recovery. - Atomic scheduler-facing claim selection for queued, due-retry, and expired-lease operations. - Append-only SQLite operation event history for auditable transitions with sanitized checkpoint summaries. +- Bounded redacted operation diagnostics for support and a future authenticated operations view. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 4937a5d..60ae5a0 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -206,6 +206,39 @@ def events(self, operation_id: str) -> tuple[OperationEvent, ...]: ).fetchall() return tuple(self._event(row) for row in rows) + def diagnostic(self, operation_id: str, *, event_limit: int = 100) -> dict[str, Any]: + """Return a bounded, redacted support view of one operation.""" + + if event_limit <= 0: + raise ValueError("event_limit must be positive") + record = self.get(operation_id) + all_events = self.events(operation_id) + selected_events = all_events[-event_limit:] + return { + "operation_id": record.operation_id, + "operation_type": record.operation_type, + "state": record.state, + "worker_id": record.worker_id, + "next_run_at": None if record.next_run_at is None else _utc(record.next_run_at), + "cancel_requested": record.cancel_requested, + "created_at": _utc(record.created_at), + "updated_at": _utc(record.updated_at), + "payload_keys": sorted(str(key) for key in record.payload), + "checkpoint": self._checkpoint_summary(record.checkpoint), + "events_truncated": len(selected_events) != len(all_events), + "events": [ + { + "sequence": event.sequence, + "event_type": event.event_type, + "state": event.state, + "worker_id": event.worker_id, + "payload": event.payload, + "created_at": _utc(event.created_at), + } + for event in selected_events + ], + } + def claim( self, operation_id: str, diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index f0e5046..05b7bf7 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -108,6 +108,35 @@ def test_checkpoint_events_store_only_sanitized_summary(self) -> None: ) self.assertNotIn("must-not-be-audit-payload", event.payload) + def test_diagnostic_export_is_bounded_and_redacted(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={"plan_digest": "secret-plan", "access_token": "secret-token"}, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) + self.repository.checkpoint( + operation.operation_id, + worker_id="worker-a", + checkpoint={ + "confirmed_occurrences": ["occ-1"], + "provider_track_id": "provider-secret", + }, + now=self.now + timedelta(seconds=1), + ) + + diagnostic = self.repository.diagnostic(operation.operation_id, event_limit=2) + self.assertEqual(diagnostic["operation_id"], operation.operation_id) + self.assertEqual(diagnostic["payload_keys"], ["access_token", "plan_digest"]) + self.assertEqual(diagnostic["checkpoint"]["confirmed_occurrences_count"], 1) + self.assertTrue(diagnostic["events_truncated"]) + self.assertEqual(len(diagnostic["events"]), 2) + serialized = str(diagnostic) + self.assertNotIn("secret-plan", serialized) + self.assertNotIn("secret-token", serialized) + self.assertNotIn("provider-secret", serialized) + def test_only_lease_owner_can_checkpoint(self) -> None: operation = self.repository.create( operation_type="import", From f769b8cbf9fd41d2a6cc6c7d79a903cd7df079ba Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:13:22 +0200 Subject: [PATCH 019/167] feat: persist provider rate-limit waits --- docs/development/implementation-baseline.md | 1 + src/symphonia/application/copy_execution.py | 21 ++++++++ .../infrastructure/sqlite_operations.py | 51 ++++++++++++++++++- src/symphonia/providers/writing.py | 13 ++++- tests/test_copy_execution.py | 14 ++++- tests/test_sqlite_operations.py | 28 ++++++++++ 6 files changed, 123 insertions(+), 5 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 4bfd1e6..3078321 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -13,6 +13,7 @@ The first implementation increment is intentionally narrower than any provider o - Atomic scheduler-facing claim selection for queued, due-retry, and expired-lease operations. - Append-only SQLite operation event history for auditable transitions with sanitized checkpoint summaries. - Bounded redacted operation diagnostics for support and a future authenticated operations view. +- Explicit provider rate-limit waits with absolute retry times and durable eligibility. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. diff --git a/src/symphonia/application/copy_execution.py b/src/symphonia/application/copy_execution.py index efe2de3..7113742 100644 --- a/src/symphonia/application/copy_execution.py +++ b/src/symphonia/application/copy_execution.py @@ -65,6 +65,8 @@ def execute( except ProviderWriteError as error: if error.outcome is WriteOutcome.RETRYABLE: return self._schedule_retry(operation, worker_id, checkpoint, now) + if error.outcome is WriteOutcome.RATE_LIMITED: + return self._schedule_rate_limit(operation, worker_id, checkpoint, now, error.retry_at) if error.outcome is WriteOutcome.UNKNOWN_OUTCOME: target = writer.reconcile_target_playlist(idempotency_key=target_key) if target is None: @@ -119,6 +121,8 @@ def execute( ) if result.outcome is WriteOutcome.RETRYABLE: return self._schedule_retry(operation, worker_id, checkpoint, now) + if result.outcome is WriteOutcome.RATE_LIMITED: + return self._schedule_rate_limit(operation, worker_id, checkpoint, now, result.retry_at) if result.outcome is WriteOutcome.PERMANENT_FAILURE: issues.append( { @@ -183,6 +187,23 @@ def _schedule_retry( now=now, ) + def _schedule_rate_limit( + self, + operation: OperationRecord, + worker_id: str, + checkpoint: dict[str, Any], + now: datetime, + retry_at: datetime | None, + ) -> OperationRecord: + next_run_at = retry_at if retry_at is not None and retry_at > now else now + timedelta(seconds=self.retry_delay_seconds) + return self.operations.schedule_rate_limit( + operation.operation_id, + worker_id=worker_id, + next_run_at=next_run_at, + checkpoint=checkpoint, + now=now, + ) + def _wait_for_user( self, operation: OperationRecord, diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 60ae5a0..d6be06b 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -263,7 +263,7 @@ def claim( if row["cancel_requested"]: raise LeaseConflict("operation cancellation has been requested") eligible = row["state"] == "queued" or ( - row["state"] == "retry_scheduled" + row["state"] in {"retry_scheduled", "waiting_rate_limit"} and row["next_run_at"] is not None and row["next_run_at"] <= now_text ) @@ -326,7 +326,8 @@ def claim_next( WHERE cancel_requested = 0 AND ( state = 'queued' - OR (state = 'retry_scheduled' AND next_run_at IS NOT NULL AND next_run_at <= ?) + OR (state IN ('retry_scheduled', 'waiting_rate_limit') + AND next_run_at IS NOT NULL AND next_run_at <= ?) OR (state = 'running' AND (lease_expires_at IS NULL OR lease_expires_at <= ?)) ) {type_clause} @@ -598,6 +599,52 @@ def schedule_retry( raise return self.get(operation_id) + def schedule_rate_limit( + self, + operation_id: str, + *, + worker_id: str, + next_run_at: datetime, + checkpoint: dict[str, Any], + now: datetime, + ) -> OperationRecord: + """Release a lease until an absolute provider rate-limit time.""" + + now_text = _utc(now) + next_run_text = _utc(next_run_at) + checkpoint_json = json.dumps(checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + self._connection.execute("BEGIN IMMEDIATE") + try: + row = self._connection.execute( + "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) + ).fetchone() + if row is None: + raise OperationNotFound(operation_id) + if row["state"] != "running" or row["worker_id"] != worker_id: + raise LeaseConflict("worker does not own a running operation") + self._connection.execute( + """ + UPDATE operations + SET state = 'waiting_rate_limit', checkpoint_json = ?, next_run_at = ?, + worker_id = NULL, lease_expires_at = NULL, updated_at = ? + WHERE operation_id = ? + """, + (checkpoint_json, next_run_text, now_text, operation_id), + ) + self._append_event( + operation_id=operation_id, + event_type="rate_limit_wait", + state="waiting_rate_limit", + worker_id=None, + payload={"next_run_at": next_run_text, **self._checkpoint_summary(checkpoint)}, + created_at=now_text, + ) + self._connection.execute("COMMIT") + except Exception: + self._connection.execute("ROLLBACK") + raise + return self.get(operation_id) + def _by_idempotency(self, idempotency_key: str) -> OperationRecord | None: row = self._connection.execute( "SELECT * FROM operations WHERE idempotency_key = ?", (idempotency_key,) diff --git a/src/symphonia/providers/writing.py b/src/symphonia/providers/writing.py index 89b7ba7..478d0d9 100644 --- a/src/symphonia/providers/writing.py +++ b/src/symphonia/providers/writing.py @@ -3,6 +3,7 @@ from __future__ import annotations from dataclasses import dataclass +from datetime import datetime from enum import Enum from typing import Protocol @@ -10,6 +11,7 @@ class WriteOutcome(str, Enum): CONFIRMED_SUCCESS = "confirmed_success" RETRYABLE = "retryable" + RATE_LIMITED = "rate_limited" UNKNOWN_OUTCOME = "unknown_outcome" PERMANENT_FAILURE = "permanent_failure" @@ -28,16 +30,24 @@ class WriteResult: outcome: WriteOutcome provider_code: str | None = None detail: str | None = None + retry_at: datetime | None = None class ProviderWriteError(RuntimeError): """A target-creation failure with an explicit retry/reconciliation class.""" - def __init__(self, outcome: WriteOutcome, detail: str, provider_code: str | None = None) -> None: + def __init__( + self, + outcome: WriteOutcome, + detail: str, + provider_code: str | None = None, + retry_at: datetime | None = None, + ) -> None: super().__init__(detail) self.outcome = outcome self.detail = detail self.provider_code = provider_code + self.retry_at = retry_at class PlaylistWriter(Protocol): @@ -69,4 +79,3 @@ def reconcile_entry( provider_track_id: str, idempotency_key: str, ) -> bool: ... - diff --git a/tests/test_copy_execution.py b/tests/test_copy_execution.py index e5bb14e..fa95257 100644 --- a/tests/test_copy_execution.py +++ b/tests/test_copy_execution.py @@ -115,7 +115,19 @@ def test_retryable_write_releases_operation_until_scheduled(self) -> None: self.assertEqual(operation.state, "retry_scheduled") self.assertEqual(operation.next_run_at, NOW + timedelta(seconds=30)) + def test_rate_limited_write_waits_until_provider_deadline(self) -> None: + digest = self.accepted_digest() + step_key = f"{digest}:entry:occ-1" + self.writer.results[step_key] = WriteResult( + WriteOutcome.RATE_LIMITED, + provider_code="429", + retry_at=NOW + timedelta(minutes=2), + ) + + operation = self.executor.execute(digest, writer=self.writer, worker_id="worker-a", now=NOW) + self.assertEqual(operation.state, "waiting_rate_limit") + self.assertEqual(operation.next_run_at, NOW + timedelta(minutes=2)) + if __name__ == "__main__": unittest.main() - diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 05b7bf7..180fe29 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -274,6 +274,34 @@ def test_retry_cannot_be_claimed_before_its_scheduled_time(self) -> None: now=self.now + timedelta(seconds=30), ) + def test_rate_limit_wait_is_durable_and_claimable_after_deadline(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="copy-1", + payload={}, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) + waiting = self.repository.schedule_rate_limit( + operation.operation_id, + worker_id="worker-a", + next_run_at=self.now + timedelta(minutes=2), + checkpoint={"confirmed_occurrences": ["occ-1"]}, + now=self.now + timedelta(seconds=1), + ) + self.assertEqual(waiting.state, "waiting_rate_limit") + with self.assertRaises(LeaseConflict): + self.repository.claim(operation.operation_id, worker_id="worker-b", now=self.now + timedelta(minutes=1)) + claimed = self.repository.claim_next( + worker_id="worker-b", + now=self.now + timedelta(minutes=2), + ) + self.assertEqual(claimed.operation_id, operation.operation_id) + self.assertEqual( + [event.event_type for event in self.repository.events(operation.operation_id)], + ["created", "claimed", "rate_limit_wait", "claimed"], + ) + def test_healthy_worker_can_renew_lease(self) -> None: operation = self.repository.create( operation_type="copy", From 1bd90a5f86b98f5a3b2cdf42415d2b60cb452476 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:14:21 +0200 Subject: [PATCH 020/167] fix: version operation event schema --- docs/development/implementation-baseline.md | 2 +- src/symphonia/infrastructure/sqlite_operations.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 3078321..17595dc 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -34,7 +34,7 @@ docker run --rm -p 8099:8099 -v symphonia-data:/data symphonia:dev The container exposes only the current health/readiness/version surface. A future App manifest must add Ingress, Supervisor metadata, supported architectures, backup declarations, and any direct callback policy only after the runtime SDD blockers are resolved. - Deterministic `unittest` coverage under `tests/`. -The package is an implementation foundation, not a claim that the corresponding capability SDDs are complete. Provider OAuth, provider API adapters, user authentication, Ingress, UI, migrations beyond the initial schema, backup/restore, and operation handlers remain unimplemented and blocked by their SDD decisions. +The package is an implementation foundation, not a claim that the corresponding capability SDDs are complete. Provider OAuth, provider API adapters, user authentication, Ingress, UI, a complete migration ledger beyond the current forward-compatible SQLite path, backup/restore, and operation handlers remain unimplemented and blocked by their SDD decisions. ## Local verification diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index d6be06b..179c984 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -69,7 +69,7 @@ class OperationEvent: class OperationRepository: """Transactional operation repository backed by one SQLite database.""" - SCHEMA_VERSION = 2 + SCHEMA_VERSION = 3 def __init__(self, path: str = ":memory:") -> None: self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) From 96b5a7393596871f39187cd3a9c676a23b699f60 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:15:41 +0200 Subject: [PATCH 021/167] feat: make playlist snapshot publication idempotent --- docs/development/implementation-baseline.md | 1 + src/symphonia/infrastructure/__init__.py | 8 +++- .../infrastructure/sqlite_library.py | 45 ++++++++++++++++++- tests/test_sqlite_library.py | 45 ++++++++++++++++++- 4 files changed, 95 insertions(+), 4 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 17595dc..e6b0861 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -20,6 +20,7 @@ The first implementation increment is intentionally narrower than any provider o - An application workflow that persists plans, requires digest acceptance, and enqueues only accepted plans as durable operations. - A copy executor that creates target playlists through a semantic writer port, checkpoints each occurrence, and reconciles unknown writes before retry. - SQLite persistence for complete playlist projections that keeps incomplete imports from replacing the last complete snapshot and isolates external IDs by namespace. +- Idempotent snapshot publication that rejects reused IDs with different content and never rolls back a newer current pointer. - An application import use case that reports partial results without advancing freshness or replacing the last complete projection. ## Local container profile diff --git a/src/symphonia/infrastructure/__init__.py b/src/symphonia/infrastructure/__init__.py index d9cd02f..fd62a69 100644 --- a/src/symphonia/infrastructure/__init__.py +++ b/src/symphonia/infrastructure/__init__.py @@ -9,7 +9,12 @@ ) from .sqlite_plans import CopyPlanNotFound, CopyPlanRepository, StoredCopyPlan from .sqlite_resolutions import ResolutionDecisionRepository -from .sqlite_library import IncompleteCollectionError, PlaylistProjectionRepository, StoredPlaylistSnapshot +from .sqlite_library import ( + IncompleteCollectionError, + PlaylistProjectionRepository, + SnapshotConflictError, + StoredPlaylistSnapshot, +) __all__ = [ "CopyPlanNotFound", @@ -22,6 +27,7 @@ "OperationRepository", "PlaylistProjectionRepository", "ResolutionDecisionRepository", + "SnapshotConflictError", "StoredCopyPlan", "StoredPlaylistSnapshot", ] diff --git a/src/symphonia/infrastructure/sqlite_library.py b/src/symphonia/infrastructure/sqlite_library.py index 35ba483..de8922f 100644 --- a/src/symphonia/infrastructure/sqlite_library.py +++ b/src/symphonia/infrastructure/sqlite_library.py @@ -25,6 +25,10 @@ class IncompleteCollectionError(ValueError): pass +class SnapshotConflictError(ValueError): + """Raised when a snapshot ID is reused for different imported content.""" + + @dataclass(frozen=True, slots=True) class StoredPlaylistSnapshot: snapshot_id: str @@ -95,6 +99,15 @@ def publish( raise ValueError("snapshot_id must not be empty") if not result.entries and result.playlist == "": raise ValueError("result must identify a playlist") + existing = self._connection.execute( + "SELECT * FROM playlist_snapshots WHERE snapshot_id = ?", (snapshot_id,) + ).fetchone() + if existing is not None: + if not self._matches_result(existing, result): + raise SnapshotConflictError("snapshot_id is already bound to different imported content") + # Publication is idempotent and must not move a newer current + # pointer backwards if a client retries an older response. + return self.get(snapshot_id) timestamp = _utc(published_at) self._connection.execute("BEGIN IMMEDIATE") try: @@ -141,6 +154,37 @@ def publish( raise return self.get(snapshot_id) + def _matches_result(self, snapshot_row: sqlite3.Row, result: CollectionImportResult) -> bool: + if ( + snapshot_row["provider"] != result.provider + or snapshot_row["namespace"] != result.namespace + or snapshot_row["playlist_id"] != result.playlist + or snapshot_row["revision"] != result.revision + ): + return False + rows = self._connection.execute( + """ + SELECT occurrence_id, position, provider_track_id, + provider_track_namespace, media_kind, available + FROM playlist_snapshot_entries + WHERE snapshot_id = ? + ORDER BY position + """, + (snapshot_row["snapshot_id"],), + ).fetchall() + entries = sorted(result.entries, key=lambda entry: entry.position) + if len(rows) != len(entries): + return False + return all( + row["occurrence_id"] == entry.occurrence_id + and row["position"] == entry.position + and row["provider_track_id"] == entry.track.object_id + and row["provider_track_namespace"] == entry.track.namespace + and row["media_kind"] == entry.media_kind.value + and bool(row["available"]) == entry.available + for row, entry in zip(rows, entries) + ) + def get(self, snapshot_id: str) -> StoredPlaylistSnapshot: row = self._connection.execute( "SELECT * FROM playlist_snapshots WHERE snapshot_id = ?", (snapshot_id,) @@ -192,4 +236,3 @@ def _entries(self, snapshot_id: str, provider: str) -> tuple[SourcePlaylistEntry ) for row in rows ) - diff --git a/tests/test_sqlite_library.py b/tests/test_sqlite_library.py index f17c5a4..8dedc8d 100644 --- a/tests/test_sqlite_library.py +++ b/tests/test_sqlite_library.py @@ -4,7 +4,11 @@ import unittest from symphonia.domain import EntryClassification -from symphonia.infrastructure import IncompleteCollectionError, PlaylistProjectionRepository +from symphonia.infrastructure import ( + IncompleteCollectionError, + PlaylistProjectionRepository, + SnapshotConflictError, +) from symphonia.providers import ( MediaKind, ProviderObjectRef, @@ -48,6 +52,44 @@ def test_incomplete_result_cannot_replace_current_projection(self) -> None: current = self.repository.current(provider="spotify", namespace="connection-1", playlist_id="playlist-1") self.assertEqual(current.snapshot_id, "snapshot-1") + def test_republishing_same_snapshot_is_idempotent(self) -> None: + result = collect_playlist_pages([page()]) + first = self.repository.publish(result, snapshot_id="snapshot-1", published_at=NOW) + second = self.repository.publish(result, snapshot_id="snapshot-1", published_at=NOW.replace(minute=1)) + self.assertEqual(first.snapshot_id, second.snapshot_id) + count = self.repository._connection.execute("SELECT COUNT(*) FROM playlist_snapshots").fetchone()[0] + self.assertEqual(count, 1) + + newer = collect_playlist_pages([page(namespace="connection-1")]) + self.repository.publish(newer, snapshot_id="snapshot-2", published_at=NOW.replace(minute=2)) + self.repository.publish(result, snapshot_id="snapshot-1", published_at=NOW.replace(minute=3)) + current = self.repository.current(provider="spotify", namespace="connection-1", playlist_id="playlist-1") + self.assertEqual(current.snapshot_id, "snapshot-2") + + def test_reusing_snapshot_id_for_different_content_is_rejected(self) -> None: + self.repository.publish(collect_playlist_pages([page()]), snapshot_id="snapshot-1", published_at=NOW) + changed = collect_playlist_pages( + [ + ProviderPlaylistPage( + ProviderObjectRef("spotify", "playlist", "playlist-1", "connection-1"), + ( + ProviderPlaylistEntry( + "occ-1", + 0, + ProviderObjectRef("spotify", "track", "track-2", "connection-1"), + MediaKind.TRACK, + ), + ), + None, + None, + True, + revision="rev-1", + ) + ] + ) + with self.assertRaises(SnapshotConflictError): + self.repository.publish(changed, snapshot_id="snapshot-1", published_at=NOW) + def test_same_external_playlist_id_isolated_by_namespace(self) -> None: first = collect_playlist_pages([page(namespace="connection-1")]) second = collect_playlist_pages([page(namespace="connection-2")]) @@ -65,4 +107,3 @@ def test_same_external_playlist_id_isolated_by_namespace(self) -> None: if __name__ == "__main__": unittest.main() - From 9fbe2db08a46e8d84271509f00c9126722f6658c Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:19:08 +0200 Subject: [PATCH 022/167] feat: persist provider connections safely --- docs/development/implementation-baseline.md | 1 + src/symphonia/infrastructure/__init__.py | 8 + .../infrastructure/sqlite_connections.py | 214 ++++++++++++++++++ src/symphonia/providers/__init__.py | 3 + src/symphonia/providers/connections.py | 56 +++++ tests/test_sqlite_connections.py | 94 ++++++++ 6 files changed, 376 insertions(+) create mode 100644 src/symphonia/infrastructure/sqlite_connections.py create mode 100644 src/symphonia/providers/connections.py create mode 100644 tests/test_sqlite_connections.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index e6b0861..3842738 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -14,6 +14,7 @@ The first implementation increment is intentionally narrower than any provider o - Append-only SQLite operation event history for auditable transitions with sanitized checkpoint summaries. - Bounded redacted operation diagnostics for support and a future authenticated operations view. - Explicit provider rate-limit waits with absolute retry times and durable eligibility. +- Provider connection persistence with account uniqueness, opaque secret references, health states, and effective capability evidence. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. diff --git a/src/symphonia/infrastructure/__init__.py b/src/symphonia/infrastructure/__init__.py index fd62a69..694ab44 100644 --- a/src/symphonia/infrastructure/__init__.py +++ b/src/symphonia/infrastructure/__init__.py @@ -15,17 +15,25 @@ SnapshotConflictError, StoredPlaylistSnapshot, ) +from .sqlite_connections import ( + ConnectionConflict, + ConnectionNotFound, + ProviderConnectionRepository, +) __all__ = [ "CopyPlanNotFound", "CopyPlanRepository", "IncompleteCollectionError", + "ConnectionConflict", + "ConnectionNotFound", "IdempotencyConflict", "LeaseConflict", "OperationNotFound", "OperationEvent", "OperationRepository", "PlaylistProjectionRepository", + "ProviderConnectionRepository", "ResolutionDecisionRepository", "SnapshotConflictError", "StoredCopyPlan", diff --git a/src/symphonia/infrastructure/sqlite_connections.py b/src/symphonia/infrastructure/sqlite_connections.py new file mode 100644 index 0000000..1201ad6 --- /dev/null +++ b/src/symphonia/infrastructure/sqlite_connections.py @@ -0,0 +1,214 @@ +"""SQLite persistence for provider connections and effective capabilities.""" + +from __future__ import annotations + +from datetime import datetime, timezone +import json +import sqlite3 + +from symphonia.providers.connections import ConnectionState, ProviderConnection +from symphonia.providers.contracts import Capability, ProviderCapabilities + + +def _utc(value: datetime) -> str: + if value.tzinfo is None: + raise ValueError("timestamps must be timezone-aware") + return value.astimezone(timezone.utc).isoformat(timespec="microseconds") + + +def _parse_utc(value: str) -> datetime: + return datetime.fromisoformat(value).astimezone(timezone.utc) + + +class ConnectionNotFound(LookupError): + pass + + +class ConnectionConflict(ValueError): + pass + + +class ProviderConnectionRepository: + """Persist account identity and capability evidence, never secret contents.""" + + def __init__(self, path: str = ":memory:") -> None: + self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) + self._connection.row_factory = sqlite3.Row + self._migrate() + + def close(self) -> None: + self._connection.close() + + def _migrate(self) -> None: + self._connection.executescript( + """ + CREATE TABLE IF NOT EXISTS provider_connections ( + connection_id TEXT PRIMARY KEY, + provider TEXT NOT NULL, + provider_account_id TEXT NOT NULL, + state TEXT NOT NULL, + manifest_version TEXT NOT NULL, + secret_ref TEXT, + capabilities_json TEXT, + expires_at TEXT, + health_code TEXT, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + UNIQUE (provider, provider_account_id) + ); + CREATE INDEX IF NOT EXISTS provider_connections_state_idx + ON provider_connections (state, provider, updated_at); + """ + ) + + def create(self, connection: ProviderConnection) -> ProviderConnection: + """Create a connection once; repeated identical creates are idempotent.""" + + payload = _serialize_capabilities(connection.capabilities) + try: + self._connection.execute( + """ + INSERT INTO provider_connections ( + connection_id, provider, provider_account_id, state, manifest_version, + secret_ref, capabilities_json, expires_at, health_code, created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + connection.connection_id, + connection.provider, + connection.provider_account_id, + connection.state.value, + connection.manifest_version, + connection.secret_ref, + payload, + None if connection.expires_at is None else _utc(connection.expires_at), + connection.health_code, + _utc(connection.created_at), + _utc(connection.updated_at), + ), + ) + except sqlite3.IntegrityError as error: + row = self._connection.execute( + "SELECT * FROM provider_connections WHERE connection_id = ?", + (connection.connection_id,), + ).fetchone() + if row is None: + raise ConnectionConflict("provider account is already connected") from error + existing = self._record(row) + if existing != connection: + raise ConnectionConflict("provider account is already connected") from error + return existing + return self.get(connection.connection_id) + + def get(self, connection_id: str) -> ProviderConnection: + row = self._connection.execute( + "SELECT * FROM provider_connections WHERE connection_id = ?", (connection_id,) + ).fetchone() + if row is None: + raise ConnectionNotFound(connection_id) + return self._record(row) + + def list(self, *, provider: str | None = None) -> tuple[ProviderConnection, ...]: + if provider is None: + rows = self._connection.execute( + "SELECT * FROM provider_connections ORDER BY provider, created_at, connection_id" + ).fetchall() + else: + rows = self._connection.execute( + """ + SELECT * FROM provider_connections + WHERE provider = ? + ORDER BY created_at, connection_id + """, + (provider,), + ).fetchall() + return tuple(self._record(row) for row in rows) + + def record_probe( + self, + connection_id: str, + *, + state: ConnectionState, + capabilities: ProviderCapabilities | None, + health_code: str | None, + expires_at: datetime | None, + now: datetime, + ) -> ProviderConnection: + """Atomically publish the latest safe capability/health observation.""" + + current = self.get(connection_id) + if state is not ConnectionState.DISCONNECTED and not current.secret_ref: + raise ConnectionConflict("cannot activate a connection without its secret reference") + self._connection.execute( + """ + UPDATE provider_connections + SET state = ?, capabilities_json = ?, expires_at = ?, health_code = ?, updated_at = ? + WHERE connection_id = ? + """, + ( + state.value, + _serialize_capabilities(capabilities), + None if expires_at is None else _utc(expires_at), + health_code, + _utc(now), + connection_id, + ), + ) + return self.get(connection_id) + + def disconnect(self, connection_id: str, *, now: datetime) -> ProviderConnection: + """Fence future use and remove the local secret reference.""" + + self.get(connection_id) + self._connection.execute( + """ + UPDATE provider_connections + SET state = 'disconnected', secret_ref = NULL, capabilities_json = NULL, + expires_at = NULL, health_code = 'disconnected', updated_at = ? + WHERE connection_id = ? + """, + (_utc(now), connection_id), + ) + return self.get(connection_id) + + @staticmethod + def _record(row: sqlite3.Row) -> ProviderConnection: + return ProviderConnection( + connection_id=row["connection_id"], + provider=row["provider"], + provider_account_id=row["provider_account_id"], + state=ConnectionState(row["state"]), + manifest_version=row["manifest_version"], + secret_ref=row["secret_ref"], + capabilities=_deserialize_capabilities(row["capabilities_json"]), + created_at=_parse_utc(row["created_at"]), + updated_at=_parse_utc(row["updated_at"]), + expires_at=None if row["expires_at"] is None else _parse_utc(row["expires_at"]), + health_code=row["health_code"], + ) + + +def _serialize_capabilities(capabilities: ProviderCapabilities | None) -> str | None: + if capabilities is None: + return None + return json.dumps( + { + "enabled": sorted(capability.value for capability in capabilities.enabled), + "evidence_version": capabilities.evidence_version, + "observed_at": capabilities.observed_at, + }, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + ) + + +def _deserialize_capabilities(payload: str | None) -> ProviderCapabilities | None: + if payload is None: + return None + value = json.loads(payload) + return ProviderCapabilities( + enabled=frozenset(Capability(item) for item in value["enabled"]), + evidence_version=value["evidence_version"], + observed_at=value["observed_at"], + ) diff --git a/src/symphonia/providers/__init__.py b/src/symphonia/providers/__init__.py index 90588cf..2ea286c 100644 --- a/src/symphonia/providers/__init__.py +++ b/src/symphonia/providers/__init__.py @@ -11,6 +11,7 @@ ProviderPlaylistEntry, ProviderPlaylistPage, ) +from .connections import ConnectionState, ProviderConnection from .importing import CollectionImportResult, ImportIssue, collect_playlist_pages, to_playlist_snapshot from .writing import PlaylistWriter, ProviderWriteError, TargetPlaylist, WriteOutcome, WriteResult @@ -18,10 +19,12 @@ "AccessBasis", "Capability", "CollectionImportResult", + "ConnectionState", "ImportIssue", "MediaKind", "ProviderAdapter", "ProviderCapabilities", + "ProviderConnection", "ProviderManifest", "ProviderObjectRef", "ProviderPlaylistEntry", diff --git a/src/symphonia/providers/connections.py b/src/symphonia/providers/connections.py new file mode 100644 index 0000000..eb864e7 --- /dev/null +++ b/src/symphonia/providers/connections.py @@ -0,0 +1,56 @@ +"""Provider connection values shared by application and infrastructure ports.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime +from enum import Enum + +from .contracts import ProviderCapabilities + + +class ConnectionState(str, Enum): + """Durable health/action states for a verified provider account.""" + + CONNECTED = "connected" + DEGRADED = "degraded" + ACTION_REQUIRED = "action_required" + DISCONNECTING = "disconnecting" + DISCONNECTED = "disconnected" + + +@dataclass(frozen=True, slots=True) +class ProviderConnection: + """A verified provider account with only an opaque secret reference.""" + + connection_id: str + provider: str + provider_account_id: str + state: ConnectionState + manifest_version: str + secret_ref: str | None + capabilities: ProviderCapabilities | None + created_at: datetime + updated_at: datetime + expires_at: datetime | None = None + health_code: str | None = None + + def __post_init__(self) -> None: + for value, field_name in ( + (self.connection_id, "connection_id"), + (self.provider, "provider"), + (self.provider_account_id, "provider_account_id"), + (self.manifest_version, "manifest_version"), + ): + if not value.strip(): + raise ValueError(f"{field_name} must not be empty") + if self.state is not ConnectionState.DISCONNECTED and not self.secret_ref: + raise ValueError("active connections require an opaque secret_ref") + if self.secret_ref is not None and not self.secret_ref.strip(): + raise ValueError("secret_ref must not be blank") + for value, field_name in ((self.created_at, "created_at"), (self.updated_at, "updated_at")): + if value.tzinfo is None: + raise ValueError(f"{field_name} must be timezone-aware") + if self.expires_at is not None and self.expires_at.tzinfo is None: + raise ValueError("expires_at must be timezone-aware") + diff --git a/tests/test_sqlite_connections.py b/tests/test_sqlite_connections.py new file mode 100644 index 0000000..080db55 --- /dev/null +++ b/tests/test_sqlite_connections.py @@ -0,0 +1,94 @@ +from __future__ import annotations + +from datetime import datetime, timedelta, timezone +import unittest + +from symphonia.infrastructure import ( + ConnectionConflict, + ConnectionNotFound, + ProviderConnectionRepository, +) +from symphonia.providers import Capability, ConnectionState, ProviderCapabilities, ProviderConnection + + +UTC = timezone.utc +NOW = datetime(2026, 9, 20, 12, 0, tzinfo=UTC) + + +def connection(*, connection_id: str = "spotify-1", account_id: str = "account-1") -> ProviderConnection: + return ProviderConnection( + connection_id=connection_id, + provider="spotify", + provider_account_id=account_id, + state=ConnectionState.CONNECTED, + manifest_version="spotify-2026-09", + secret_ref="secret-ref-1", + capabilities=ProviderCapabilities( + enabled=frozenset({Capability.READ_PLAYLISTS}), + evidence_version="probe-1", + observed_at="2026-09-20T12:00:00Z", + ), + created_at=NOW, + updated_at=NOW, + expires_at=NOW + timedelta(days=30), + health_code=None, + ) + + +class ProviderConnectionRepositoryTests(unittest.TestCase): + def setUp(self) -> None: + self.repository = ProviderConnectionRepository() + + def tearDown(self) -> None: + self.repository.close() + + def test_create_round_trips_capabilities_and_only_persists_secret_reference(self) -> None: + stored = self.repository.create(connection()) + self.assertEqual(stored, connection()) + self.assertEqual(stored.capabilities.enabled, frozenset({Capability.READ_PLAYLISTS})) + raw = self.repository._connection.execute( + "SELECT secret_ref, capabilities_json FROM provider_connections" + ).fetchone() + self.assertEqual(raw["secret_ref"], "secret-ref-1") + self.assertNotIn("access-token", str(raw)) + + def test_same_connection_is_idempotent_but_account_collision_is_rejected(self) -> None: + first = connection() + self.repository.create(first) + self.assertEqual(self.repository.create(first), first) + with self.assertRaises(ConnectionConflict): + self.repository.create(connection(connection_id="spotify-2")) + + def test_probe_updates_health_and_capabilities(self) -> None: + self.repository.create(connection()) + degraded = self.repository.record_probe( + "spotify-1", + state=ConnectionState.DEGRADED, + capabilities=ProviderCapabilities( + enabled=frozenset(), + evidence_version="probe-2", + observed_at="2026-09-20T12:01:00Z", + ), + health_code="provider_unavailable", + expires_at=NOW + timedelta(days=29), + now=NOW + timedelta(minutes=1), + ) + self.assertEqual(degraded.state, ConnectionState.DEGRADED) + self.assertEqual(degraded.health_code, "provider_unavailable") + self.assertEqual(degraded.capabilities.enabled, frozenset()) + + def test_disconnect_erases_local_secret_reference_and_capabilities(self) -> None: + self.repository.create(connection()) + disconnected = self.repository.disconnect("spotify-1", now=NOW + timedelta(minutes=2)) + self.assertEqual(disconnected.state, ConnectionState.DISCONNECTED) + self.assertIsNone(disconnected.secret_ref) + self.assertIsNone(disconnected.capabilities) + self.assertEqual(self.repository.list(provider="spotify")[0].health_code, "disconnected") + + def test_missing_connection_is_explicit(self) -> None: + with self.assertRaises(ConnectionNotFound): + self.repository.get("missing") + + +if __name__ == "__main__": + unittest.main() From c743b9412bce8f101f1ab8018b74c357d2331b19 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:20:49 +0200 Subject: [PATCH 023/167] feat: persist single-use authorization attempts --- docs/development/implementation-baseline.md | 1 + src/symphonia/infrastructure/__init__.py | 10 + .../infrastructure/sqlite_authorization.py | 205 ++++++++++++++++++ src/symphonia/providers/__init__.py | 3 + src/symphonia/providers/authorization.py | 52 +++++ tests/test_sqlite_authorization.py | 76 +++++++ 6 files changed, 347 insertions(+) create mode 100644 src/symphonia/infrastructure/sqlite_authorization.py create mode 100644 src/symphonia/providers/authorization.py create mode 100644 tests/test_sqlite_authorization.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 3842738..156bb1b 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -15,6 +15,7 @@ The first implementation increment is intentionally narrower than any provider o - Bounded redacted operation diagnostics for support and a future authenticated operations view. - Explicit provider rate-limit waits with absolute retry times and durable eligibility. - Provider connection persistence with account uniqueness, opaque secret references, health states, and effective capability evidence. +- Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. diff --git a/src/symphonia/infrastructure/__init__.py b/src/symphonia/infrastructure/__init__.py index 694ab44..2f92894 100644 --- a/src/symphonia/infrastructure/__init__.py +++ b/src/symphonia/infrastructure/__init__.py @@ -20,6 +20,12 @@ ConnectionNotFound, ProviderConnectionRepository, ) +from .sqlite_authorization import ( + AuthorizationAttemptError, + AuthorizationAttemptNotFound, + AuthorizationAttemptRepository, + state_digest, +) __all__ = [ "CopyPlanNotFound", @@ -27,6 +33,9 @@ "IncompleteCollectionError", "ConnectionConflict", "ConnectionNotFound", + "AuthorizationAttemptError", + "AuthorizationAttemptNotFound", + "AuthorizationAttemptRepository", "IdempotencyConflict", "LeaseConflict", "OperationNotFound", @@ -38,4 +47,5 @@ "SnapshotConflictError", "StoredCopyPlan", "StoredPlaylistSnapshot", + "state_digest", ] diff --git a/src/symphonia/infrastructure/sqlite_authorization.py b/src/symphonia/infrastructure/sqlite_authorization.py new file mode 100644 index 0000000..eba9cd8 --- /dev/null +++ b/src/symphonia/infrastructure/sqlite_authorization.py @@ -0,0 +1,205 @@ +"""Durable, single-use authorization-attempt state.""" + +from __future__ import annotations + +from datetime import datetime, timedelta, timezone +import hashlib +import hmac +import sqlite3 + +from symphonia.providers.authorization import AuthorizationAttempt, AuthorizationState + + +def _utc(value: datetime) -> str: + if value.tzinfo is None: + raise ValueError("timestamps must be timezone-aware") + return value.astimezone(timezone.utc).isoformat(timespec="microseconds") + + +def _parse_utc(value: str) -> datetime: + return datetime.fromisoformat(value).astimezone(timezone.utc) + + +def state_digest(state: str) -> str: + if not state or not state.strip(): + raise ValueError("authorization state must not be empty") + return hashlib.sha256(state.encode("utf-8")).hexdigest() + + +class AuthorizationAttemptNotFound(LookupError): + pass + + +class AuthorizationAttemptError(ValueError): + pass + + +class AuthorizationAttemptRepository: + """Store only authorization correlation metadata, never raw state values.""" + + def __init__(self, path: str = ":memory:") -> None: + self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) + self._connection.row_factory = sqlite3.Row + self._migrate() + + def close(self) -> None: + self._connection.close() + + def _migrate(self) -> None: + self._connection.executescript( + """ + CREATE TABLE IF NOT EXISTS authorization_attempts ( + attempt_id TEXT PRIMARY KEY, + provider TEXT NOT NULL, + actor_id TEXT NOT NULL, + redirect_uri TEXT NOT NULL, + state_digest TEXT NOT NULL, + state TEXT NOT NULL, + created_at TEXT NOT NULL, + expires_at TEXT NOT NULL, + completed_at TEXT, + failure_code TEXT + ); + CREATE INDEX IF NOT EXISTS authorization_attempts_expiry_idx + ON authorization_attempts (state, expires_at); + """ + ) + + def create( + self, + *, + attempt_id: str, + provider: str, + actor_id: str, + redirect_uri: str, + raw_state: str, + now: datetime, + ttl: timedelta = timedelta(minutes=10), + ) -> AuthorizationAttempt: + if ttl <= timedelta(0): + raise ValueError("authorization attempt ttl must be positive") + created_at = _utc(now) + expires_at = _utc(now + ttl) + self._connection.execute( + """ + INSERT INTO authorization_attempts ( + attempt_id, provider, actor_id, redirect_uri, state_digest, + state, created_at, expires_at + ) VALUES (?, ?, ?, ?, ?, 'created', ?, ?) + """, + ( + attempt_id, + provider, + actor_id, + redirect_uri, + state_digest(raw_state), + created_at, + expires_at, + ), + ) + return self.get(attempt_id) + + def get(self, attempt_id: str) -> AuthorizationAttempt: + row = self._connection.execute( + "SELECT * FROM authorization_attempts WHERE attempt_id = ?", (attempt_id,) + ).fetchone() + if row is None: + raise AuthorizationAttemptNotFound(attempt_id) + return self._record(row) + + def consume(self, attempt_id: str, *, raw_state: str, now: datetime) -> AuthorizationAttempt: + """Consume a matching, unexpired state exactly once.""" + + now_text = _utc(now) + digest = state_digest(raw_state) + self._connection.execute("BEGIN IMMEDIATE") + try: + row = self._connection.execute( + "SELECT * FROM authorization_attempts WHERE attempt_id = ?", (attempt_id,) + ).fetchone() + if row is None: + raise AuthorizationAttemptNotFound(attempt_id) + if row["state"] != AuthorizationState.CREATED.value: + raise AuthorizationAttemptError("authorization attempt is no longer consumable") + if row["expires_at"] <= now_text: + self._connection.execute( + "UPDATE authorization_attempts SET state = 'expired', completed_at = ? WHERE attempt_id = ?", + (now_text, attempt_id), + ) + self._connection.execute("COMMIT") + raise AuthorizationAttemptError("authorization attempt has expired") + if not hmac.compare_digest(row["state_digest"], digest): + self._connection.execute( + """ + UPDATE authorization_attempts + SET state = 'failed', completed_at = ?, failure_code = 'state_mismatch' + WHERE attempt_id = ? + """, + (now_text, attempt_id), + ) + self._connection.execute("COMMIT") + raise AuthorizationAttemptError("authorization state did not match") + self._connection.execute( + "UPDATE authorization_attempts SET state = 'consumed', completed_at = ? WHERE attempt_id = ?", + (now_text, attempt_id), + ) + self._connection.execute("COMMIT") + except Exception: + if self._connection.in_transaction: + self._connection.execute("ROLLBACK") + raise + return self.get(attempt_id) + + def deny(self, attempt_id: str, *, now: datetime, failure_code: str = "consent_denied") -> AuthorizationAttempt: + return self._complete(attempt_id, AuthorizationState.DENIED, now, failure_code) + + def expire(self, attempt_id: str, *, now: datetime) -> AuthorizationAttempt: + return self._complete(attempt_id, AuthorizationState.EXPIRED, now, "expired") + + def _complete( + self, + attempt_id: str, + state: AuthorizationState, + now: datetime, + failure_code: str, + ) -> AuthorizationAttempt: + now_text = _utc(now) + self._connection.execute("BEGIN IMMEDIATE") + try: + row = self._connection.execute( + "SELECT state FROM authorization_attempts WHERE attempt_id = ?", (attempt_id,) + ).fetchone() + if row is None: + raise AuthorizationAttemptNotFound(attempt_id) + if row["state"] != AuthorizationState.CREATED.value: + raise AuthorizationAttemptError("authorization attempt is already complete") + self._connection.execute( + """ + UPDATE authorization_attempts + SET state = ?, completed_at = ?, failure_code = ? + WHERE attempt_id = ? + """, + (state.value, now_text, failure_code, attempt_id), + ) + self._connection.execute("COMMIT") + except Exception: + if self._connection.in_transaction: + self._connection.execute("ROLLBACK") + raise + return self.get(attempt_id) + + @staticmethod + def _record(row: sqlite3.Row) -> AuthorizationAttempt: + return AuthorizationAttempt( + attempt_id=row["attempt_id"], + provider=row["provider"], + actor_id=row["actor_id"], + redirect_uri=row["redirect_uri"], + state_digest=row["state_digest"], + state=AuthorizationState(row["state"]), + created_at=_parse_utc(row["created_at"]), + expires_at=_parse_utc(row["expires_at"]), + completed_at=None if row["completed_at"] is None else _parse_utc(row["completed_at"]), + failure_code=row["failure_code"], + ) + diff --git a/src/symphonia/providers/__init__.py b/src/symphonia/providers/__init__.py index 2ea286c..7d8c41c 100644 --- a/src/symphonia/providers/__init__.py +++ b/src/symphonia/providers/__init__.py @@ -12,11 +12,14 @@ ProviderPlaylistPage, ) from .connections import ConnectionState, ProviderConnection +from .authorization import AuthorizationAttempt, AuthorizationState from .importing import CollectionImportResult, ImportIssue, collect_playlist_pages, to_playlist_snapshot from .writing import PlaylistWriter, ProviderWriteError, TargetPlaylist, WriteOutcome, WriteResult __all__ = [ "AccessBasis", + "AuthorizationAttempt", + "AuthorizationState", "Capability", "CollectionImportResult", "ConnectionState", diff --git a/src/symphonia/providers/authorization.py b/src/symphonia/providers/authorization.py new file mode 100644 index 0000000..edaa825 --- /dev/null +++ b/src/symphonia/providers/authorization.py @@ -0,0 +1,52 @@ +"""Provider-neutral authorization-attempt values. + +The raw OAuth state/code-verifier values are intentionally not represented by +the durable model. The application may hold them briefly at the boundary, but +the repository stores only a digest and exact callback binding. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime +from enum import Enum + + +class AuthorizationState(str, Enum): + CREATED = "created" + CONSUMED = "consumed" + DENIED = "denied" + EXPIRED = "expired" + FAILED = "failed" + + +@dataclass(frozen=True, slots=True) +class AuthorizationAttempt: + attempt_id: str + provider: str + actor_id: str + redirect_uri: str + state_digest: str + state: AuthorizationState + created_at: datetime + expires_at: datetime + completed_at: datetime | None = None + failure_code: str | None = None + + def __post_init__(self) -> None: + for value, field_name in ( + (self.attempt_id, "attempt_id"), + (self.provider, "provider"), + (self.actor_id, "actor_id"), + (self.redirect_uri, "redirect_uri"), + (self.state_digest, "state_digest"), + ): + if not value.strip(): + raise ValueError(f"{field_name} must not be empty") + if self.created_at.tzinfo is None or self.expires_at.tzinfo is None: + raise ValueError("authorization timestamps must be timezone-aware") + if self.expires_at <= self.created_at: + raise ValueError("authorization attempt must expire after creation") + if self.completed_at is not None and self.completed_at.tzinfo is None: + raise ValueError("completed_at must be timezone-aware") + diff --git a/tests/test_sqlite_authorization.py b/tests/test_sqlite_authorization.py new file mode 100644 index 0000000..4560d36 --- /dev/null +++ b/tests/test_sqlite_authorization.py @@ -0,0 +1,76 @@ +from __future__ import annotations + +from datetime import datetime, timedelta, timezone +import unittest + +from symphonia.infrastructure import ( + AuthorizationAttemptError, + AuthorizationAttemptRepository, + state_digest, +) +from symphonia.providers import AuthorizationState + + +UTC = timezone.utc +NOW = datetime(2026, 9, 20, 12, 0, tzinfo=UTC) + + +class AuthorizationAttemptRepositoryTests(unittest.TestCase): + def setUp(self) -> None: + self.repository = AuthorizationAttemptRepository() + + def tearDown(self) -> None: + self.repository.close() + + def create(self, *, ttl: timedelta = timedelta(minutes=10)): + return self.repository.create( + attempt_id="attempt-1", + provider="spotify", + actor_id="ha-user-1", + redirect_uri="https://ha.example/symphonia/callback", + raw_state="raw-state-secret", + now=NOW, + ttl=ttl, + ) + + def test_persists_only_state_digest_and_exact_callback_binding(self) -> None: + attempt = self.create() + self.assertEqual(attempt.state, AuthorizationState.CREATED) + self.assertEqual(attempt.state_digest, state_digest("raw-state-secret")) + raw = self.repository._connection.execute( + "SELECT state_digest, redirect_uri FROM authorization_attempts" + ).fetchone() + self.assertNotIn("raw-state-secret", str(raw)) + self.assertEqual(raw["redirect_uri"], "https://ha.example/symphonia/callback") + + def test_matching_state_is_single_use(self) -> None: + self.create() + consumed = self.repository.consume("attempt-1", raw_state="raw-state-secret", now=NOW + timedelta(seconds=1)) + self.assertEqual(consumed.state, AuthorizationState.CONSUMED) + with self.assertRaises(AuthorizationAttemptError): + self.repository.consume("attempt-1", raw_state="raw-state-secret", now=NOW + timedelta(seconds=2)) + + def test_mismatch_invalidates_attempt_without_disclosing_expected_state(self) -> None: + self.create() + with self.assertRaises(AuthorizationAttemptError): + self.repository.consume("attempt-1", raw_state="attacker-state", now=NOW + timedelta(seconds=1)) + failed = self.repository.get("attempt-1") + self.assertEqual(failed.state, AuthorizationState.FAILED) + self.assertEqual(failed.failure_code, "state_mismatch") + + def test_expired_attempt_cannot_be_consumed(self) -> None: + self.create(ttl=timedelta(seconds=1)) + with self.assertRaises(AuthorizationAttemptError): + self.repository.consume("attempt-1", raw_state="raw-state-secret", now=NOW + timedelta(seconds=2)) + self.assertEqual(self.repository.get("attempt-1").state, AuthorizationState.EXPIRED) + + def test_denial_is_terminal(self) -> None: + self.create() + denied = self.repository.deny("attempt-1", now=NOW + timedelta(seconds=1)) + self.assertEqual(denied.state, AuthorizationState.DENIED) + with self.assertRaises(AuthorizationAttemptError): + self.repository.expire("attempt-1", now=NOW + timedelta(seconds=2)) + + +if __name__ == "__main__": + unittest.main() From ed21aee7b7724166706e4d46502c5cda183f2189 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:21:43 +0200 Subject: [PATCH 024/167] feat: orchestrate adapter-backed playlist imports --- docs/development/implementation-baseline.md | 1 + src/symphonia/application/library_import.py | 22 ++++++++- tests/test_library_import.py | 53 ++++++++++++++++++++- 3 files changed, 73 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 156bb1b..919cddd 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -16,6 +16,7 @@ The first implementation increment is intentionally narrower than any provider o - Explicit provider rate-limit waits with absolute retry times and durable eligibility. - Provider connection persistence with account uniqueness, opaque secret references, health states, and effective capability evidence. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. +- Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. diff --git a/src/symphonia/application/library_import.py b/src/symphonia/application/library_import.py index 1bdc021..f4aeead 100644 --- a/src/symphonia/application/library_import.py +++ b/src/symphonia/application/library_import.py @@ -6,7 +6,8 @@ from datetime import datetime from symphonia.infrastructure.sqlite_library import PlaylistProjectionRepository, StoredPlaylistSnapshot -from symphonia.providers.importing import CollectionImportResult, ImportIssue +from symphonia.providers.contracts import ProviderAdapter, ProviderObjectRef +from symphonia.providers.importing import CollectionImportResult, ImportIssue, collect_playlist_pages @dataclass(frozen=True, slots=True) @@ -21,6 +22,24 @@ class ImportPublication: class LibraryImportService: projections: PlaylistProjectionRepository + def import_playlist( + self, + adapter: ProviderAdapter, + *, + connection_id: str, + playlist: ProviderObjectRef, + snapshot_id: str, + observed_at: datetime, + cursor: str | None = None, + ) -> ImportPublication: + """Read normalized pages through an adapter and publish the result.""" + + if adapter.manifest.provider != playlist.provider: + raise ValueError("adapter provider does not match playlist provider") + pages = tuple(adapter.read_playlist_pages(connection_id, playlist, cursor)) + result = collect_playlist_pages(pages) + return self.publish_playlist(result, snapshot_id=snapshot_id, observed_at=observed_at) + def publish_playlist( self, result: CollectionImportResult, @@ -39,4 +58,3 @@ def publish_playlist( return ImportPublication("partial", None, retained, result.issues) snapshot = self.projections.publish(result, snapshot_id=snapshot_id, published_at=observed_at) return ImportPublication("succeeded", snapshot, snapshot, result.issues) - diff --git a/tests/test_library_import.py b/tests/test_library_import.py index 26e4f0c..3b452fb 100644 --- a/tests/test_library_import.py +++ b/tests/test_library_import.py @@ -6,7 +6,10 @@ from symphonia.application import LibraryImportService from symphonia.infrastructure import PlaylistProjectionRepository from symphonia.providers import ( + AccessBasis, MediaKind, + ProviderCapabilities, + ProviderManifest, ProviderObjectRef, ProviderPlaylistEntry, ProviderPlaylistPage, @@ -45,7 +48,55 @@ def test_partial_import_retains_last_complete_snapshot(self) -> None: self.assertIsNone(publication.snapshot) self.assertEqual(publication.retained_current.snapshot_id, "snapshot-1") + def test_import_playlist_uses_normalized_adapter_pages(self) -> None: + playlist = ProviderObjectRef("spotify", "playlist", "playlist-1", "connection-1") + + class FakeAdapter: + manifest = ProviderManifest("spotify", "Spotify", AccessBasis.OFFICIAL, "beta", "limited") + + def capabilities(self, connection_id: str) -> ProviderCapabilities: + return ProviderCapabilities(frozenset(), "fixture", "2026-09-20T12:00:00Z") + + def read_playlist_pages(self, connection_id: str, requested: ProviderObjectRef, cursor: str | None = None): + self.assert_connection = connection_id + return [ + ProviderPlaylistPage( + requested, + (ProviderPlaylistEntry("occ-1", 0, ProviderObjectRef("spotify", "track", "track-1", requested.namespace), MediaKind.TRACK),), + cursor, + None, + True, + ) + ] + + publication = self.service.import_playlist( + FakeAdapter(), + connection_id="connection-1", + playlist=playlist, + snapshot_id="snapshot-adapter", + observed_at=NOW, + ) + self.assertEqual(publication.state, "succeeded") + self.assertEqual(publication.snapshot.snapshot_id, "snapshot-adapter") + + def test_import_playlist_rejects_another_provider_adapter(self) -> None: + playlist = ProviderObjectRef("spotify", "playlist", "playlist-1", "connection-1") + + class WrongAdapter: + manifest = ProviderManifest("youtube", "YouTube", AccessBasis.OFFICIAL, "beta", "limited") + + def read_playlist_pages(self, connection_id: str, requested: ProviderObjectRef, cursor: str | None = None): + return () + + with self.assertRaises(ValueError): + self.service.import_playlist( + WrongAdapter(), + connection_id="connection-1", + playlist=playlist, + snapshot_id="snapshot-wrong", + observed_at=NOW, + ) + if __name__ == "__main__": unittest.main() - From 73376b52fc4ca15801a4c95377f9e4930eaa9767 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:22:36 +0200 Subject: [PATCH 025/167] feat: add deterministic provider adapter registry --- docs/development/implementation-baseline.md | 1 + src/symphonia/providers/__init__.py | 4 ++ src/symphonia/providers/registry.py | 44 +++++++++++++++++++++ tests/test_provider_import.py | 23 +++++++++++ 4 files changed, 72 insertions(+) create mode 100644 src/symphonia/providers/registry.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 919cddd..d16dc16 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -17,6 +17,7 @@ The first implementation increment is intentionally narrower than any provider o - Provider connection persistence with account uniqueness, opaque secret references, health states, and effective capability evidence. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. +- Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. diff --git a/src/symphonia/providers/__init__.py b/src/symphonia/providers/__init__.py index 7d8c41c..f819f0e 100644 --- a/src/symphonia/providers/__init__.py +++ b/src/symphonia/providers/__init__.py @@ -13,6 +13,7 @@ ) from .connections import ConnectionState, ProviderConnection from .authorization import AuthorizationAttempt, AuthorizationState +from .registry import ProviderAlreadyRegistered, ProviderNotRegistered, ProviderRegistry from .importing import CollectionImportResult, ImportIssue, collect_playlist_pages, to_playlist_snapshot from .writing import PlaylistWriter, ProviderWriteError, TargetPlaylist, WriteOutcome, WriteResult @@ -26,12 +27,15 @@ "ImportIssue", "MediaKind", "ProviderAdapter", + "ProviderAlreadyRegistered", "ProviderCapabilities", "ProviderConnection", "ProviderManifest", "ProviderObjectRef", + "ProviderNotRegistered", "ProviderPlaylistEntry", "ProviderPlaylistPage", + "ProviderRegistry", "ProviderWriteError", "PlaylistWriter", "TargetPlaylist", diff --git a/src/symphonia/providers/registry.py b/src/symphonia/providers/registry.py new file mode 100644 index 0000000..ee15741 --- /dev/null +++ b/src/symphonia/providers/registry.py @@ -0,0 +1,44 @@ +"""Provider adapter registry used by the composition root.""" + +from __future__ import annotations + +from dataclasses import dataclass + +from .contracts import ProviderAdapter, ProviderManifest + + +class ProviderAlreadyRegistered(ValueError): + pass + + +class ProviderNotRegistered(LookupError): + pass + + +@dataclass(slots=True) +class ProviderRegistry: + """Keep provider discovery explicit and deterministic.""" + + _adapters: dict[str, ProviderAdapter] + + def __init__(self) -> None: + self._adapters = {} + + def register(self, adapter: ProviderAdapter) -> None: + manifest = adapter.manifest + provider = manifest.provider.strip() + if not provider: + raise ValueError("provider manifest must identify a provider") + if provider in self._adapters: + raise ProviderAlreadyRegistered(provider) + self._adapters[provider] = adapter + + def get(self, provider: str) -> ProviderAdapter: + try: + return self._adapters[provider] + except KeyError as error: + raise ProviderNotRegistered(provider) from error + + def manifests(self) -> tuple[ProviderManifest, ...]: + return tuple(self._adapters[key].manifest for key in sorted(self._adapters)) + diff --git a/tests/test_provider_import.py b/tests/test_provider_import.py index 0772b50..493f087 100644 --- a/tests/test_provider_import.py +++ b/tests/test_provider_import.py @@ -11,6 +11,9 @@ ProviderObjectRef, ProviderPlaylistEntry, ProviderPlaylistPage, + ProviderAlreadyRegistered, + ProviderNotRegistered, + ProviderRegistry, collect_playlist_pages, to_playlist_snapshot, ) @@ -31,6 +34,26 @@ def entry(occurrence_id: str, position: int, track_id: str, available: bool = Tr class ProviderContractTests(unittest.TestCase): + def test_provider_registry_discovers_sorted_manifests_and_rejects_duplicates(self) -> None: + class FakeAdapter: + def __init__(self, provider: str) -> None: + self.manifest = ProviderManifest(provider, provider.title(), AccessBasis.OFFICIAL, "beta", "limited") + + def capabilities(self, connection_id: str): + raise NotImplementedError + + def read_playlist_pages(self, connection_id: str, playlist: ProviderObjectRef, cursor: str | None = None): + raise NotImplementedError + + registry = ProviderRegistry() + registry.register(FakeAdapter("youtube")) + registry.register(FakeAdapter("spotify")) + self.assertEqual([manifest.provider for manifest in registry.manifests()], ["spotify", "youtube"]) + with self.assertRaises(ProviderAlreadyRegistered): + registry.register(FakeAdapter("spotify")) + with self.assertRaises(ProviderNotRegistered): + registry.get("apple") + def test_external_identity_is_namespaced_by_type_and_connection(self) -> None: same_upstream_id = ProviderObjectRef("spotify", "track", "same", "connection-a") another_connection = ProviderObjectRef("spotify", "track", "same", "connection-b") From 46288183e2ad0b1cee3d05724eeb233c5907fc1c Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:28:36 +0200 Subject: [PATCH 026/167] feat: add safe Spotify playlist adapter --- docs/development/implementation-baseline.md | 1 + src/symphonia/providers/__init__.py | 7 + src/symphonia/providers/errors.py | 41 +++ src/symphonia/providers/spotify.py | 345 ++++++++++++++++++++ tests/test_spotify_adapter.py | 132 ++++++++ 5 files changed, 526 insertions(+) create mode 100644 src/symphonia/providers/errors.py create mode 100644 src/symphonia/providers/spotify.py create mode 100644 tests/test_spotify_adapter.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index d16dc16..1e17bf0 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -18,6 +18,7 @@ The first implementation increment is intentionally narrower than any provider o - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. +- Offline-testable official Spotify playlist reader with bounded pagination and normalized error categories. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. diff --git a/src/symphonia/providers/__init__.py b/src/symphonia/providers/__init__.py index f819f0e..d4bb1b0 100644 --- a/src/symphonia/providers/__init__.py +++ b/src/symphonia/providers/__init__.py @@ -14,6 +14,8 @@ from .connections import ConnectionState, ProviderConnection from .authorization import AuthorizationAttempt, AuthorizationState from .registry import ProviderAlreadyRegistered, ProviderNotRegistered, ProviderRegistry +from .errors import ProviderApiError, ProviderErrorCategory +from .spotify import JsonResponse, SpotifyAdapter, UrllibJsonClient from .importing import CollectionImportResult, ImportIssue, collect_playlist_pages, to_playlist_snapshot from .writing import PlaylistWriter, ProviderWriteError, TargetPlaylist, WriteOutcome, WriteResult @@ -27,7 +29,9 @@ "ImportIssue", "MediaKind", "ProviderAdapter", + "ProviderApiError", "ProviderAlreadyRegistered", + "ProviderErrorCategory", "ProviderCapabilities", "ProviderConnection", "ProviderManifest", @@ -36,6 +40,9 @@ "ProviderPlaylistEntry", "ProviderPlaylistPage", "ProviderRegistry", + "JsonResponse", + "SpotifyAdapter", + "UrllibJsonClient", "ProviderWriteError", "PlaylistWriter", "TargetPlaylist", diff --git a/src/symphonia/providers/errors.py b/src/symphonia/providers/errors.py new file mode 100644 index 0000000..823a698 --- /dev/null +++ b/src/symphonia/providers/errors.py @@ -0,0 +1,41 @@ +"""Normalized provider error categories shared by adapters.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime +from enum import Enum + + +class ProviderErrorCategory(str, Enum): + AUTHENTICATION_REQUIRED = "authentication_required" + AUTHORIZATION_REVOKED = "authorization_revoked" + PERMISSION_DENIED = "permission_denied" + CAPABILITY_UNAVAILABLE = "capability_unavailable" + NOT_FOUND = "not_found" + ITEM_UNAVAILABLE = "item_unavailable" + INVALID_REQUEST = "invalid_request" + RATE_LIMITED = "rate_limited" + PROVIDER_UNAVAILABLE = "provider_unavailable" + TIMEOUT = "timeout" + NETWORK_ERROR = "network_error" + CONFLICT = "conflict" + UNKNOWN_WRITE_OUTCOME = "unknown_write_outcome" + PROVIDER_CONTRACT_CHANGED = "provider_contract_changed" + + +@dataclass(frozen=True, slots=True) +class ProviderApiError(RuntimeError): + category: ProviderErrorCategory + detail: str + provider_code: str | None = None + retry_at: datetime | None = None + correlation_id: str | None = None + + def __post_init__(self) -> None: + RuntimeError.__init__(self, self.detail) + if not self.detail.strip(): + raise ValueError("provider error detail must not be empty") + if self.correlation_id is not None and not self.correlation_id.strip(): + raise ValueError("correlation_id must not be blank") + diff --git a/src/symphonia/providers/spotify.py b/src/symphonia/providers/spotify.py new file mode 100644 index 0000000..7239ff4 --- /dev/null +++ b/src/symphonia/providers/spotify.py @@ -0,0 +1,345 @@ +"""Offline-testable Spotify Web API adapter for playlist reads. + +The adapter owns only normalized translation and error classification. Token +refresh/storage and the concrete HTTP transport are injected at composition +time, so no secret material enters this module's persistence or logs. +""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping +from dataclasses import dataclass +from datetime import datetime, timedelta, timezone +import json +from typing import Any, Protocol +from urllib.error import HTTPError, URLError +from urllib.parse import urlencode +from urllib.request import Request, urlopen + +from .contracts import ( + AccessBasis, + Capability, + MediaKind, + ProviderAdapter, + ProviderCapabilities, + ProviderManifest, + ProviderObjectRef, + ProviderPlaylistEntry, + ProviderPlaylistPage, +) +from .errors import ProviderApiError, ProviderErrorCategory +from .writing import PlaylistWriter, ProviderWriteError, TargetPlaylist, WriteOutcome, WriteResult + + +@dataclass(frozen=True, slots=True) +class JsonResponse: + status: int + payload: Mapping[str, Any] + headers: Mapping[str, str] + + +class JsonClient(Protocol): + def request( + self, + method: str, + path: str, + *, + token: str, + query: Mapping[str, str], + body: Mapping[str, Any] | None = None, + ) -> JsonResponse: ... + + +class UrllibJsonClient: + """Small standard-library transport with bounded request timeout.""" + + def __init__(self, base_url: str = "https://api.spotify.com/v1", timeout_seconds: float = 10.0) -> None: + if timeout_seconds <= 0: + raise ValueError("timeout_seconds must be positive") + self.base_url = base_url.rstrip("/") + self.timeout_seconds = timeout_seconds + + def request( + self, + method: str, + path: str, + *, + token: str, + query: Mapping[str, str], + body: Mapping[str, Any] | None = None, + ) -> JsonResponse: + url = f"{self.base_url}/{path.lstrip('/')}" + if query: + url = f"{url}?{urlencode(query)}" + encoded_body = None if body is None else json.dumps(body, ensure_ascii=False, separators=(",", ":")).encode("utf-8") + headers = { + "Authorization": f"Bearer {token}", + "Accept": "application/json", + **({"Content-Type": "application/json"} if body is not None else {}), + } + request = Request(url, data=encoded_body, method=method, headers=headers) + try: + with urlopen(request, timeout=self.timeout_seconds) as response: + body = response.read() + payload = json.loads(body.decode("utf-8")) if body else {} + return JsonResponse(response.status, payload, dict(response.headers.items())) + except HTTPError as error: + body = error.read() + try: + payload = json.loads(body.decode("utf-8")) if body else {} + except (UnicodeDecodeError, json.JSONDecodeError): + payload = {} + return JsonResponse(error.code, payload, dict(error.headers.items())) + except TimeoutError as error: + raise ProviderApiError(ProviderErrorCategory.TIMEOUT, "Spotify request timed out") from error + except URLError as error: + raise ProviderApiError(ProviderErrorCategory.NETWORK_ERROR, "Spotify request failed") from error + + +class SpotifyAdapter(ProviderAdapter, PlaylistWriter): + """Translate Spotify playlist pages into Symphonia provider values.""" + + manifest = ProviderManifest( + provider="spotify", + display_name="Spotify", + access_basis=AccessBasis.OFFICIAL, + maturity="beta", + support_level="playlist-read", + upstream_dependencies=("Spotify Web API",), + reviewed_on="2026-09-20", + ) + + def __init__( + self, + client: JsonClient, + token_for_connection: Callable[[str], str], + page_size: int = 50, + connection_id: str | None = None, + ) -> None: + if not 1 <= page_size <= 50: + raise ValueError("Spotify playlist page_size must be between 1 and 50") + self._client = client + self._token_for_connection = token_for_connection + self._page_size = page_size + self._connection_id = connection_id + + def capabilities(self, connection_id: str) -> ProviderCapabilities: + response = self._request(connection_id, "GET", "/me/playlists", {"limit": "1", "offset": "0"}) + return ProviderCapabilities( + enabled=frozenset({Capability.READ_PLAYLISTS}), + evidence_version="spotify-playlist-read-v1", + observed_at=datetime.now(timezone.utc).isoformat(timespec="seconds"), + ) + + def read_playlist_pages( + self, + connection_id: str, + playlist: ProviderObjectRef, + cursor: str | None = None, + ) -> tuple[ProviderPlaylistPage, ...]: + if playlist.provider != self.manifest.provider or playlist.object_type != "playlist": + raise ValueError("Spotify adapter requires a Spotify playlist reference") + offset = self._parse_cursor(cursor) + pages: list[ProviderPlaylistPage] = [] + while True: + response = self._request( + connection_id, + "GET", + f"/playlists/{playlist.object_id}/items", + {"limit": str(self._page_size), "offset": str(offset)}, + ) + items = response.payload.get("items", []) + if not isinstance(items, list): + raise ProviderApiError(ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, "Spotify playlist items were not a list") + entries = tuple( + self._entry(playlist, item, position=offset + index) + for index, item in enumerate(items) + ) + next_url = response.payload.get("next") + if next_url and not entries: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Spotify pagination advanced without returning items", + ) + next_cursor = str(offset + len(entries)) if next_url else None + pages.append( + ProviderPlaylistPage( + playlist=playlist, + entries=entries, + cursor=None if not pages and cursor is None else str(offset), + next_cursor=next_cursor, + complete=next_cursor is None, + revision=response.payload.get("snapshot_id"), + ) + ) + if next_cursor is None: + return tuple(pages) + offset += len(entries) + + def ensure_target_playlist( + self, + *, + provider: str, + name: str, + visibility: str, + idempotency_key: str, + ) -> TargetPlaylist: + if provider != self.manifest.provider: + raise ValueError("Spotify adapter requires a Spotify target provider") + connection_id = self._write_connection_id() + try: + response = self._request( + connection_id, + "POST", + "/me/playlists", + {}, + body={"name": name, "public": visibility == "public"}, + ) + playlist_id = response.payload.get("id") + if not isinstance(playlist_id, str) or not playlist_id.strip(): + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Spotify create-playlist response did not contain an id", + ) + return TargetPlaylist(playlist_id) + except ProviderApiError as error: + raise ProviderWriteError( + self._write_outcome(error), + error.detail, + error.provider_code, + error.retry_at, + ) from error + + def add_entry( + self, + *, + target_playlist_id: str, + provider_track_id: str, + idempotency_key: str, + ) -> WriteResult: + try: + connection_id = self._write_connection_id() + except ProviderWriteError as error: + return WriteResult(error.outcome, provider_code=error.provider_code, detail=error.detail) + try: + self._request( + connection_id, + "POST", + f"/playlists/{target_playlist_id}/items", + {}, + body={"uris": [f"spotify:track:{provider_track_id}"]}, + ) + return WriteResult(WriteOutcome.CONFIRMED_SUCCESS) + except ProviderApiError as error: + return WriteResult( + self._write_outcome(error), + provider_code=error.provider_code, + detail=error.detail, + retry_at=error.retry_at, + ) + + def reconcile_target_playlist(self, *, idempotency_key: str) -> TargetPlaylist | None: + # Spotify does not expose the local idempotency key, so a timed-out + # create cannot be identified safely without a provider-specific + # correlation strategy. The executor therefore pauses for review. + return None + + def reconcile_entry(self, *, target_playlist_id: str, provider_track_id: str, idempotency_key: str) -> bool: + # Existing duplicate occurrences make a positive read insufficient to + # prove which add attempt was accepted. Never claim certainty here. + return False + + def _write_connection_id(self) -> str: + if self._connection_id is None or not self._connection_id.strip(): + raise ProviderWriteError( + WriteOutcome.PERMANENT_FAILURE, + "Spotify writer is not bound to a provider connection", + ) + return self._connection_id + + def _request( + self, + connection_id: str, + method: str, + path: str, + query: Mapping[str, str], + body: Mapping[str, Any] | None = None, + ) -> JsonResponse: + token = self._token_for_connection(connection_id) + if not token.strip(): + raise ProviderApiError(ProviderErrorCategory.AUTHENTICATION_REQUIRED, "Spotify connection has no usable access token") + response = self._client.request(method, path, token=token, query=query, body=body) + if 200 <= response.status < 300: + if not isinstance(response.payload, Mapping): + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Spotify response was not a JSON object", + ) + return response + category = { + 401: ProviderErrorCategory.AUTHENTICATION_REQUIRED, + 403: ProviderErrorCategory.PERMISSION_DENIED, + 404: ProviderErrorCategory.NOT_FOUND, + 429: ProviderErrorCategory.RATE_LIMITED, + }.get(response.status, ProviderErrorCategory.PROVIDER_UNAVAILABLE if response.status >= 500 else ProviderErrorCategory.INVALID_REQUEST) + retry_at = None + retry_after = self._header(response.headers, "retry-after") + if category is ProviderErrorCategory.RATE_LIMITED and retry_after is not None: + try: + retry_at = datetime.now(timezone.utc) + timedelta(seconds=max(0, int(retry_after))) + except ValueError: + retry_at = None + error_body = response.payload.get("error") if isinstance(response.payload, Mapping) else None + provider_code = str(error_body.get("status")) if isinstance(error_body, Mapping) and error_body.get("status") is not None else str(response.status) + raise ProviderApiError(category, "Spotify API request was not accepted", provider_code=provider_code, retry_at=retry_at) + + @staticmethod + def _write_outcome(error: ProviderApiError) -> WriteOutcome: + if error.category is ProviderErrorCategory.RATE_LIMITED: + return WriteOutcome.RATE_LIMITED + if error.category in { + ProviderErrorCategory.TIMEOUT, + ProviderErrorCategory.NETWORK_ERROR, + ProviderErrorCategory.PROVIDER_UNAVAILABLE, + ProviderErrorCategory.UNKNOWN_WRITE_OUTCOME, + }: + return WriteOutcome.UNKNOWN_OUTCOME + return WriteOutcome.PERMANENT_FAILURE + + @staticmethod + def _parse_cursor(cursor: str | None) -> int: + if cursor is None: + return 0 + try: + value = int(cursor) + except ValueError as error: + raise ValueError("Spotify playlist cursor must be an integer offset") from error + if value < 0: + raise ValueError("Spotify playlist cursor must not be negative") + return value + + @staticmethod + def _entry(playlist: ProviderObjectRef, item: Any, *, position: int) -> ProviderPlaylistEntry: + item_payload = (item.get("item") or item.get("track")) if isinstance(item, Mapping) else None + if not isinstance(item_payload, Mapping): + ref = ProviderObjectRef("spotify", "track", f"unavailable:{position}", playlist.namespace) + return ProviderPlaylistEntry(f"{playlist.object_id}:{position}", position, ref, MediaKind.UNKNOWN, available=False) + object_type = str(item_payload.get("type") or "unknown") + object_id = str(item_payload.get("id") or f"unavailable:{position}") + media_kind = {"track": MediaKind.TRACK, "episode": MediaKind.PODCAST}.get(object_type, MediaKind.UNKNOWN) + available = bool(item_payload.get("id")) and media_kind is not MediaKind.UNKNOWN and item_payload.get("is_playable", True) is not False + ref = ProviderObjectRef("spotify", object_type, object_id, playlist.namespace) + return ProviderPlaylistEntry( + occurrence_id=f"{playlist.object_id}:{position}", + position=position, + track=ref, + media_kind=media_kind, + title=item_payload.get("name") if isinstance(item_payload.get("name"), str) else None, + available=available, + source_added_at=item.get("added_at") if isinstance(item, Mapping) and isinstance(item.get("added_at"), str) else None, + ) + + @staticmethod + def _header(headers: Mapping[str, str], name: str) -> str | None: + wanted = name.lower() + return next((value for key, value in headers.items() if key.lower() == wanted), None) diff --git a/tests/test_spotify_adapter.py b/tests/test_spotify_adapter.py new file mode 100644 index 0000000..a800c55 --- /dev/null +++ b/tests/test_spotify_adapter.py @@ -0,0 +1,132 @@ +from __future__ import annotations + +import unittest + +from symphonia.providers import ( + Capability, + JsonResponse, + ProviderApiError, + ProviderErrorCategory, + ProviderObjectRef, + SpotifyAdapter, +) + + +class FakeClient: + def __init__(self, responses: dict[str, JsonResponse]) -> None: + self.responses = responses + self.calls: list[tuple[str, str, str, dict[str, str]]] = [] + + def request(self, method: str, path: str, *, token: str, query: dict[str, str], body=None) -> JsonResponse: + self.calls.append((method, path, token, query)) + self.body = body + return self.responses[query.get("offset", "capabilities")] + + +class SpotifyAdapterTests(unittest.TestCase): + def playlist(self) -> ProviderObjectRef: + return ProviderObjectRef("spotify", "playlist", "playlist-1", "connection-1") + + def test_playlist_pages_preserve_positions_duplicates_and_unavailable_items(self) -> None: + client = FakeClient( + { + "0": JsonResponse( + 200, + { + "snapshot_id": "snapshot-1", + "items": [ + {"added_at": "2026-09-20T12:00:00Z", "item": {"id": "track-1", "type": "track", "name": "One"}}, + {"added_at": "2026-09-20T12:01:00Z", "item": None}, + ], + "next": "https://api.spotify.com/v1/playlists/playlist-1/items?offset=2", + }, + {}, + ), + "2": JsonResponse( + 200, + { + "snapshot_id": "snapshot-1", + "items": [{"item": {"id": "track-1", "type": "track", "name": "One again"}}], + "next": None, + }, + {}, + ), + } + ) + adapter = SpotifyAdapter(client, lambda connection_id: "access-token", page_size=2) + pages = adapter.read_playlist_pages("connection-1", self.playlist()) + self.assertEqual(len(pages), 2) + self.assertEqual([entry.position for page in pages for entry in page.entries], [0, 1, 2]) + self.assertEqual( + [entry.track.object_id for page in pages for entry in page.entries], + ["track-1", "unavailable:1", "track-1"], + ) + self.assertFalse(pages[0].complete) + self.assertTrue(pages[1].complete) + self.assertEqual(client.calls[0][2], "access-token") + + def test_capability_probe_uses_safe_read_endpoint(self) -> None: + client = FakeClient({"0": JsonResponse(200, {"items": []}, {})}) + adapter = SpotifyAdapter(client, lambda connection_id: "access-token", connection_id="connection-1") + capabilities = adapter.capabilities("connection-1") + self.assertTrue(capabilities.supports(Capability.READ_PLAYLISTS)) + self.assertEqual(client.calls[0][1], "/me/playlists") + + def test_rate_limit_is_normalized_with_retry_hint(self) -> None: + client = FakeClient({"0": JsonResponse(429, {"error": {"status": 429}}, {"Retry-After": "10"})}) + adapter = SpotifyAdapter(client, lambda connection_id: "access-token", connection_id="connection-1") + with self.assertRaises(ProviderApiError) as context: + adapter.read_playlist_pages("connection-1", self.playlist()) + self.assertEqual(context.exception.category, ProviderErrorCategory.RATE_LIMITED) + self.assertIsNotNone(context.exception.retry_at) + + def test_invalid_cursor_and_empty_token_fail_closed(self) -> None: + client = FakeClient({"0": JsonResponse(200, {"items": [], "next": None}, {})}) + adapter = SpotifyAdapter(client, lambda connection_id: "") + with self.assertRaises(ProviderApiError): + adapter.read_playlist_pages("connection-1", self.playlist()) + adapter = SpotifyAdapter(client, lambda connection_id: "token") + with self.assertRaises(ValueError): + adapter.read_playlist_pages("connection-1", self.playlist(), cursor="not-an-offset") + + def test_confirmed_writes_use_spotify_json_contract(self) -> None: + client = FakeClient({"capabilities": JsonResponse(201, {"id": "target-1"}, {})}) + adapter = SpotifyAdapter(client, lambda connection_id: "access-token", connection_id="connection-1") + target = adapter.ensure_target_playlist( + provider="spotify", + name="Imported", + visibility="private", + idempotency_key="connection-1", + ) + self.assertEqual(target.provider_playlist_id, "target-1") + self.assertEqual(client.calls[-1][1], "/me/playlists") + self.assertEqual(client.body, {"name": "Imported", "public": False}) + + client.responses["capabilities"] = JsonResponse(201, {"snapshot_id": "snapshot-2"}, {}) + result = adapter.add_entry( + target_playlist_id="target-1", + provider_track_id="track-1", + idempotency_key="entry-1", + ) + self.assertEqual(result.outcome.value, "confirmed_success") + self.assertEqual(client.body, {"uris": ["spotify:track:track-1"]}) + + def test_write_rate_limit_and_unknown_outcome_are_not_blind_retries(self) -> None: + client = FakeClient({"capabilities": JsonResponse(429, {"error": {"status": 429}}, {"Retry-After": "10"})}) + adapter = SpotifyAdapter(client, lambda connection_id: "access-token", connection_id="connection-1") + rate_limited = adapter.add_entry(target_playlist_id="target-1", provider_track_id="track-1", idempotency_key="entry-1") + self.assertEqual(rate_limited.outcome.value, "rate_limited") + self.assertIsNotNone(rate_limited.retry_at) + + class UnknownClient(FakeClient): + def request(self, method: str, path: str, *, token: str, query: dict[str, str], body=None) -> JsonResponse: + raise ProviderApiError(ProviderErrorCategory.TIMEOUT, "request timed out") + + unknown = SpotifyAdapter(UnknownClient({}), lambda connection_id: "access-token", connection_id="connection-1") + result = unknown.add_entry(target_playlist_id="target-1", provider_track_id="track-1", idempotency_key="entry-1") + self.assertEqual(result.outcome.value, "unknown_outcome") + self.assertFalse(unknown.reconcile_entry(target_playlist_id="target-1", provider_track_id="track-1", idempotency_key="entry-1")) + + +if __name__ == "__main__": + unittest.main() From ff703ee912fe80a7d8da7dbe6c443f823f35d83a Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:29:39 +0200 Subject: [PATCH 027/167] feat: orchestrate authorization state boundary --- docs/development/implementation-baseline.md | 1 + src/symphonia/application/__init__.py | 3 + src/symphonia/application/authorization.py | 55 +++++++++++++++++ tests/test_authorization_service.py | 68 +++++++++++++++++++++ 4 files changed, 127 insertions(+) create mode 100644 src/symphonia/application/authorization.py create mode 100644 tests/test_authorization_service.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 1e17bf0..d349d5f 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -16,6 +16,7 @@ The first implementation increment is intentionally narrower than any provider o - Explicit provider rate-limit waits with absolute retry times and durable eligibility. - Provider connection persistence with account uniqueness, opaque secret references, health states, and effective capability evidence. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. +- Application authorization boundary that generates one-use state without persisting the raw value. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. - Offline-testable official Spotify playlist reader with bounded pagination and normalized error categories. diff --git a/src/symphonia/application/__init__.py b/src/symphonia/application/__init__.py index 7be8bbb..f175783 100644 --- a/src/symphonia/application/__init__.py +++ b/src/symphonia/application/__init__.py @@ -4,9 +4,12 @@ from .copy_workflow import CopyWorkflowService from .copy_execution import CopyExecutionService from .library_import import ImportPublication, LibraryImportService +from .authorization import AuthorizationService, AuthorizationStart __all__ = [ "CopyExecutionService", + "AuthorizationService", + "AuthorizationStart", "CopyPlanningService", "CopyWorkflowService", "ImportPublication", diff --git a/src/symphonia/application/authorization.py b/src/symphonia/application/authorization.py new file mode 100644 index 0000000..bf3fc00 --- /dev/null +++ b/src/symphonia/application/authorization.py @@ -0,0 +1,55 @@ +"""Application orchestration for provider-neutral authorization attempts.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime, timedelta +import secrets +import uuid +from typing import Callable + +from symphonia.infrastructure.sqlite_authorization import AuthorizationAttemptRepository +from symphonia.providers.authorization import AuthorizationAttempt + + +@dataclass(frozen=True, slots=True) +class AuthorizationStart: + """Boundary result; ``raw_state`` must not enter logs or persistence.""" + + attempt: AuthorizationAttempt + raw_state: str + + +@dataclass(frozen=True, slots=True) +class AuthorizationService: + attempts: AuthorizationAttemptRepository + state_factory: Callable[[], str] = secrets.token_urlsafe + id_factory: Callable[[], str] = lambda: str(uuid.uuid4()) + + def begin( + self, + *, + provider: str, + actor_id: str, + redirect_uri: str, + now: datetime, + ttl: timedelta = timedelta(minutes=10), + ) -> AuthorizationStart: + raw_state = self.state_factory() + attempt = self.attempts.create( + attempt_id=self.id_factory(), + provider=provider, + actor_id=actor_id, + redirect_uri=redirect_uri, + raw_state=raw_state, + now=now, + ttl=ttl, + ) + return AuthorizationStart(attempt=attempt, raw_state=raw_state) + + def consume(self, attempt_id: str, *, raw_state: str, now: datetime) -> AuthorizationAttempt: + return self.attempts.consume(attempt_id, raw_state=raw_state, now=now) + + def deny(self, attempt_id: str, *, now: datetime, failure_code: str = "consent_denied") -> AuthorizationAttempt: + return self.attempts.deny(attempt_id, now=now, failure_code=failure_code) + diff --git a/tests/test_authorization_service.py b/tests/test_authorization_service.py new file mode 100644 index 0000000..0d9f980 --- /dev/null +++ b/tests/test_authorization_service.py @@ -0,0 +1,68 @@ +from __future__ import annotations + +from datetime import datetime, timezone +import unittest + +from symphonia.application import AuthorizationService +from symphonia.infrastructure import AuthorizationAttemptRepository +from symphonia.providers import AuthorizationState + + +NOW = datetime(2026, 9, 20, 12, 0, tzinfo=timezone.utc) + + +class AuthorizationServiceTests(unittest.TestCase): + def setUp(self) -> None: + self.repository = AuthorizationAttemptRepository() + self.service = AuthorizationService( + self.repository, + state_factory=lambda: "raw-state-only-at-boundary", + id_factory=lambda: "attempt-1", + ) + + def tearDown(self) -> None: + self.repository.close() + + def test_begin_returns_raw_state_but_repository_has_only_digest(self) -> None: + started = self.service.begin( + provider="spotify", + actor_id="ha-user-1", + redirect_uri="https://ha.example/symphonia/callback", + now=NOW, + ) + self.assertEqual(started.raw_state, "raw-state-only-at-boundary") + self.assertEqual(started.attempt.state, AuthorizationState.CREATED) + stored = self.repository.get("attempt-1") + self.assertNotEqual(stored.state_digest, started.raw_state) + raw = self.repository._connection.execute( + "SELECT state_digest FROM authorization_attempts WHERE attempt_id = 'attempt-1'" + ).fetchone()[0] + self.assertNotIn(started.raw_state, raw) + + def test_consume_and_deny_delegate_single_use_transitions(self) -> None: + self.service.begin( + provider="spotify", + actor_id="ha-user-1", + redirect_uri="https://ha.example/symphonia/callback", + now=NOW, + ) + consumed = self.service.consume("attempt-1", raw_state="raw-state-only-at-boundary", now=NOW) + self.assertEqual(consumed.state, AuthorizationState.CONSUMED) + + self.service = AuthorizationService( + self.repository, + state_factory=lambda: "another-state", + id_factory=lambda: "attempt-2", + ) + self.service.begin( + provider="spotify", + actor_id="ha-user-1", + redirect_uri="https://ha.example/symphonia/callback", + now=NOW, + ) + denied = self.service.deny("attempt-2", now=NOW) + self.assertEqual(denied.state, AuthorizationState.DENIED) + + +if __name__ == "__main__": + unittest.main() From af60f47bef588673c8a2a3880d77224d587f6a2b Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:30:31 +0200 Subject: [PATCH 028/167] fix: forbid secrets on disconnected connections --- src/symphonia/providers/connections.py | 3 ++- tests/test_sqlite_connections.py | 14 ++++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/src/symphonia/providers/connections.py b/src/symphonia/providers/connections.py index eb864e7..8435cfe 100644 --- a/src/symphonia/providers/connections.py +++ b/src/symphonia/providers/connections.py @@ -46,6 +46,8 @@ def __post_init__(self) -> None: raise ValueError(f"{field_name} must not be empty") if self.state is not ConnectionState.DISCONNECTED and not self.secret_ref: raise ValueError("active connections require an opaque secret_ref") + if self.state is ConnectionState.DISCONNECTED and self.secret_ref is not None: + raise ValueError("disconnected connections must not retain a secret_ref") if self.secret_ref is not None and not self.secret_ref.strip(): raise ValueError("secret_ref must not be blank") for value, field_name in ((self.created_at, "created_at"), (self.updated_at, "updated_at")): @@ -53,4 +55,3 @@ def __post_init__(self) -> None: raise ValueError(f"{field_name} must be timezone-aware") if self.expires_at is not None and self.expires_at.tzinfo is None: raise ValueError("expires_at must be timezone-aware") - diff --git a/tests/test_sqlite_connections.py b/tests/test_sqlite_connections.py index 080db55..8d91d20 100644 --- a/tests/test_sqlite_connections.py +++ b/tests/test_sqlite_connections.py @@ -89,6 +89,20 @@ def test_missing_connection_is_explicit(self) -> None: with self.assertRaises(ConnectionNotFound): self.repository.get("missing") + def test_disconnected_model_rejects_a_secret_reference(self) -> None: + with self.assertRaises(ValueError): + ProviderConnection( + connection_id="spotify-disconnected", + provider="spotify", + provider_account_id="account-1", + state=ConnectionState.DISCONNECTED, + manifest_version="spotify-2026-09", + secret_ref="must-not-remain", + capabilities=None, + created_at=NOW, + updated_at=NOW, + ) + if __name__ == "__main__": unittest.main() From 81e31ca84f9533d2f0d5c8efc0716ba824695bbb Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:32:49 +0200 Subject: [PATCH 029/167] feat: bind copy plans to target capabilities --- docs/development/implementation-baseline.md | 1 + src/symphonia/application/__init__.py | 3 +- src/symphonia/application/copy_planning.py | 7 ++- src/symphonia/application/copy_workflow.py | 22 +++++++++- src/symphonia/domain/models.py | 20 +++++++++ src/symphonia/infrastructure/sqlite_plans.py | 6 +++ tests/test_copy_workflow.py | 46 +++++++++++++++++++- 7 files changed, 100 insertions(+), 5 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index d349d5f..9a676da 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -18,6 +18,7 @@ The first implementation increment is intentionally narrower than any provider o - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Application authorization boundary that generates one-use state without persisting the raw value. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. +- Copy plans bind target connection and effective write-capability evidence into their digest. - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. - Offline-testable official Spotify playlist reader with bounded pagination and normalized error categories. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. diff --git a/src/symphonia/application/__init__.py b/src/symphonia/application/__init__.py index f175783..fc884d0 100644 --- a/src/symphonia/application/__init__.py +++ b/src/symphonia/application/__init__.py @@ -1,7 +1,7 @@ """Use-case orchestration ports and services.""" from .copy_planning import CopyPlanningService -from .copy_workflow import CopyWorkflowService +from .copy_workflow import CapabilityUnavailableError, CopyWorkflowService from .copy_execution import CopyExecutionService from .library_import import ImportPublication, LibraryImportService from .authorization import AuthorizationService, AuthorizationStart @@ -10,6 +10,7 @@ "CopyExecutionService", "AuthorizationService", "AuthorizationStart", + "CapabilityUnavailableError", "CopyPlanningService", "CopyWorkflowService", "ImportPublication", diff --git a/src/symphonia/application/copy_planning.py b/src/symphonia/application/copy_planning.py index 32ee06a..db9fe7d 100644 --- a/src/symphonia/application/copy_planning.py +++ b/src/symphonia/application/copy_planning.py @@ -19,6 +19,9 @@ def plan( target_playlist_name: str, target_visibility: str = "private", policy: CopyPolicy = CopyPolicy.STRICT, + target_connection_id: str = "default", + target_capabilities: tuple[str, ...] = (), + target_capability_evidence_version: str | None = None, ) -> CopyPlan: return build_copy_plan( snapshot, @@ -26,5 +29,7 @@ def plan( target_playlist_name=target_playlist_name, target_visibility=target_visibility, policy=policy, + target_connection_id=target_connection_id, + target_capabilities=target_capabilities, + target_capability_evidence_version=target_capability_evidence_version, ) - diff --git a/src/symphonia/application/copy_workflow.py b/src/symphonia/application/copy_workflow.py index 6cda10c..5a79102 100644 --- a/src/symphonia/application/copy_workflow.py +++ b/src/symphonia/application/copy_workflow.py @@ -8,10 +8,15 @@ from symphonia.domain.models import CopyPlan, CopyPolicy, PlanAcceptanceError, PlaylistSnapshot from symphonia.infrastructure.sqlite_operations import OperationRecord, OperationRepository from symphonia.infrastructure.sqlite_plans import CopyPlanRepository, StoredCopyPlan +from symphonia.providers.contracts import ProviderCapabilities from .copy_planning import CopyPlanningService +class CapabilityUnavailableError(ValueError): + pass + + @dataclass(frozen=True, slots=True) class CopyWorkflowService: """Keep plan persistence/acceptance separate from provider execution.""" @@ -29,13 +34,29 @@ def create_plan( target_visibility: str, policy: CopyPolicy, now: datetime, + target_connection_id: str = "default", + target_capabilities: ProviderCapabilities | None = None, ) -> StoredCopyPlan: + capability_names: tuple[str, ...] = () + capability_evidence_version = None + if target_capabilities is not None: + required = {"create_playlist", "add_playlist_entries"} + capability_names = tuple(sorted(capability.value for capability in target_capabilities.enabled)) + missing = sorted(required.difference(capability_names)) + if missing: + raise CapabilityUnavailableError( + f"target connection lacks required capabilities: {', '.join(missing)}" + ) + capability_evidence_version = target_capabilities.evidence_version plan = self.planning.plan( snapshot, target_provider=target_provider, target_playlist_name=target_playlist_name, target_visibility=target_visibility, policy=policy, + target_connection_id=target_connection_id, + target_capabilities=capability_names, + target_capability_evidence_version=capability_evidence_version, ) return self.plans.save(plan, now=now) @@ -52,4 +73,3 @@ def enqueue_accepted_plan(self, digest: str, *, now: datetime) -> OperationRecor payload={"plan_digest": digest}, now=now, ) - diff --git a/src/symphonia/domain/models.py b/src/symphonia/domain/models.py index e224b18..1925f3b 100644 --- a/src/symphonia/domain/models.py +++ b/src/symphonia/domain/models.py @@ -132,6 +132,17 @@ class CopyPlan: entries: tuple[CopyPlanEntry, ...] digest: str source_namespace: str = "default" + target_connection_id: str = "default" + target_capabilities: tuple[str, ...] = () + target_capability_evidence_version: str | None = None + + def __post_init__(self) -> None: + if not self.target_connection_id.strip(): + raise ValueError("target_connection_id must not be empty") + if any(not capability.strip() for capability in self.target_capabilities): + raise ValueError("target capabilities must not contain blank values") + if tuple(sorted(set(self.target_capabilities))) != self.target_capabilities: + raise ValueError("target capabilities must be sorted and unique") @property def blocked(self) -> bool: @@ -170,6 +181,9 @@ def build_copy_plan( target_playlist_name: str, target_visibility: str = "private", policy: CopyPolicy = CopyPolicy.STRICT, + target_connection_id: str = "default", + target_capabilities: tuple[str, ...] = (), + target_capability_evidence_version: str | None = None, ) -> CopyPlan: """Build a non-mutating, deterministic copy plan. @@ -213,6 +227,9 @@ def build_copy_plan( "target_playlist_name": target_playlist_name, "target_visibility": target_visibility, "policy": policy.value, + "target_connection_id": target_connection_id, + "target_capabilities": list(target_capabilities), + "target_capability_evidence_version": target_capability_evidence_version, "entries": [ { "occurrence_id": entry.occurrence_id, @@ -239,6 +256,9 @@ def build_copy_plan( entries=tuple(plan_entries), digest=digest, source_namespace=snapshot.source_namespace, + target_connection_id=target_connection_id, + target_capabilities=target_capabilities, + target_capability_evidence_version=target_capability_evidence_version, ) diff --git a/src/symphonia/infrastructure/sqlite_plans.py b/src/symphonia/infrastructure/sqlite_plans.py index c7501a9..6588277 100644 --- a/src/symphonia/infrastructure/sqlite_plans.py +++ b/src/symphonia/infrastructure/sqlite_plans.py @@ -104,6 +104,9 @@ def _serialize(plan: CopyPlan) -> dict[str, Any]: "source_playlist_id": plan.source_playlist_id, "source_namespace": plan.source_namespace, "target_provider": plan.target_provider, + "target_connection_id": plan.target_connection_id, + "target_capabilities": list(plan.target_capabilities), + "target_capability_evidence_version": plan.target_capability_evidence_version, "target_playlist_name": plan.target_playlist_name, "target_visibility": plan.target_visibility, "policy": plan.policy.value, @@ -146,4 +149,7 @@ def _deserialize(payload: dict[str, Any]) -> CopyPlan: ), digest=payload["digest"], source_namespace=payload.get("source_namespace", "default"), + target_connection_id=payload.get("target_connection_id", "default"), + target_capabilities=tuple(payload.get("target_capabilities", ())), + target_capability_evidence_version=payload.get("target_capability_evidence_version"), ) diff --git a/tests/test_copy_workflow.py b/tests/test_copy_workflow.py index 3f19a6c..d00b37d 100644 --- a/tests/test_copy_workflow.py +++ b/tests/test_copy_workflow.py @@ -3,9 +3,10 @@ from datetime import datetime, timezone import unittest -from symphonia.application import CopyPlanningService, CopyWorkflowService +from symphonia.application import CapabilityUnavailableError, CopyPlanningService, CopyWorkflowService from symphonia.domain import CopyPolicy, EntryClassification, PlanAcceptanceError, PlaylistSnapshot, SourcePlaylistEntry from symphonia.infrastructure import CopyPlanRepository, OperationRepository +from symphonia.providers import Capability, ProviderCapabilities class CopyWorkflowTests(unittest.TestCase): @@ -76,7 +77,48 @@ def test_blocked_plan_cannot_be_accepted(self) -> None: with self.assertRaises(PlanAcceptanceError): self.workflow.accept_plan(stored.plan.digest, now=self.now) + def test_plan_binds_target_connection_and_capability_evidence(self) -> None: + capabilities = ProviderCapabilities( + enabled=frozenset({Capability.CREATE_PLAYLIST, Capability.ADD_PLAYLIST_ENTRIES}), + evidence_version="probe-spotify-1", + observed_at="2026-09-20T12:00:00Z", + ) + stored = self.workflow.create_plan( + self.snapshot(), + target_provider="spotify", + target_playlist_name="Rock", + target_visibility="private", + policy=CopyPolicy.STRICT, + target_connection_id="spotify-connection-2", + target_capabilities=capabilities, + now=self.now, + ) + self.assertEqual(stored.plan.target_connection_id, "spotify-connection-2") + self.assertEqual( + stored.plan.target_capabilities, + ("add_playlist_entries", "create_playlist"), + ) + self.assertEqual(stored.plan.target_capability_evidence_version, "probe-spotify-1") + reloaded = self.plans.get(stored.plan.digest) + self.assertEqual(reloaded.plan, stored.plan) + + def test_plan_rejects_target_without_required_write_capabilities(self) -> None: + with self.assertRaises(CapabilityUnavailableError): + self.workflow.create_plan( + self.snapshot(), + target_provider="spotify", + target_playlist_name="Rock", + target_visibility="private", + policy=CopyPolicy.STRICT, + target_connection_id="spotify-connection-2", + target_capabilities=ProviderCapabilities( + enabled=frozenset({Capability.READ_PLAYLISTS}), + evidence_version="probe-spotify-1", + observed_at="2026-09-20T12:00:00Z", + ), + now=self.now, + ) + if __name__ == "__main__": unittest.main() - From 0d87203955c3dd74c542c9adce5397dfc7ff452e Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:33:49 +0200 Subject: [PATCH 030/167] feat: probe and classify provider connections --- docs/development/implementation-baseline.md | 1 + src/symphonia/application/__init__.py | 2 + .../application/provider_connections.py | 82 +++++++++++++++++++ tests/test_provider_connections.py | 81 ++++++++++++++++++ 4 files changed, 166 insertions(+) create mode 100644 src/symphonia/application/provider_connections.py create mode 100644 tests/test_provider_connections.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 9a676da..695d1ee 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -15,6 +15,7 @@ The first implementation increment is intentionally narrower than any provider o - Bounded redacted operation diagnostics for support and a future authenticated operations view. - Explicit provider rate-limit waits with absolute retry times and durable eligibility. - Provider connection persistence with account uniqueness, opaque secret references, health states, and effective capability evidence. +- Application connection service for verified-account registration, capability probes, degraded health, and reauthorization-required classification. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Application authorization boundary that generates one-use state without persisting the raw value. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. diff --git a/src/symphonia/application/__init__.py b/src/symphonia/application/__init__.py index fc884d0..47504ce 100644 --- a/src/symphonia/application/__init__.py +++ b/src/symphonia/application/__init__.py @@ -5,6 +5,7 @@ from .copy_execution import CopyExecutionService from .library_import import ImportPublication, LibraryImportService from .authorization import AuthorizationService, AuthorizationStart +from .provider_connections import ProviderConnectionService __all__ = [ "CopyExecutionService", @@ -15,4 +16,5 @@ "CopyWorkflowService", "ImportPublication", "LibraryImportService", + "ProviderConnectionService", ] diff --git a/src/symphonia/application/provider_connections.py b/src/symphonia/application/provider_connections.py new file mode 100644 index 0000000..f5401d8 --- /dev/null +++ b/src/symphonia/application/provider_connections.py @@ -0,0 +1,82 @@ +"""Application use cases for verified provider connections and probes.""" + +from __future__ import annotations + +from dataclasses import dataclass +from datetime import datetime + +from symphonia.infrastructure.sqlite_connections import ProviderConnectionRepository +from symphonia.providers.connections import ConnectionState, ProviderConnection +from symphonia.providers.errors import ProviderApiError, ProviderErrorCategory +from symphonia.providers.registry import ProviderRegistry + + +@dataclass(frozen=True, slots=True) +class ProviderConnectionService: + connections: ProviderConnectionRepository + providers: ProviderRegistry + + def register_verified( + self, + *, + connection_id: str, + provider: str, + provider_account_id: str, + manifest_version: str, + secret_ref: str, + now: datetime, + ) -> ProviderConnection: + """Persist an already verified account without accepting token material.""" + + adapter = self.providers.get(provider) + if not adapter.manifest.provider == provider: + raise ValueError("provider manifest mismatch") + return self.connections.create( + ProviderConnection( + connection_id=connection_id, + provider=provider, + provider_account_id=provider_account_id, + state=ConnectionState.CONNECTED, + manifest_version=manifest_version, + secret_ref=secret_ref, + capabilities=None, + created_at=now, + updated_at=now, + ) + ) + + def probe(self, connection_id: str, *, now: datetime) -> ProviderConnection: + current = self.connections.get(connection_id) + adapter = self.providers.get(current.provider) + try: + capabilities = adapter.capabilities(connection_id) + except ProviderApiError as error: + state = ( + ConnectionState.ACTION_REQUIRED + if error.category in { + ProviderErrorCategory.AUTHENTICATION_REQUIRED, + ProviderErrorCategory.AUTHORIZATION_REVOKED, + ProviderErrorCategory.PERMISSION_DENIED, + } + else ConnectionState.DEGRADED + ) + return self.connections.record_probe( + connection_id, + state=state, + capabilities=None, + health_code=error.category.value, + expires_at=current.expires_at, + now=now, + ) + return self.connections.record_probe( + connection_id, + state=ConnectionState.CONNECTED, + capabilities=capabilities, + health_code=None, + expires_at=current.expires_at, + now=now, + ) + + def disconnect(self, connection_id: str, *, now: datetime) -> ProviderConnection: + return self.connections.disconnect(connection_id, now=now) + diff --git a/tests/test_provider_connections.py b/tests/test_provider_connections.py new file mode 100644 index 0000000..0010544 --- /dev/null +++ b/tests/test_provider_connections.py @@ -0,0 +1,81 @@ +from __future__ import annotations + +from datetime import datetime, timezone +import unittest + +from symphonia.application import ProviderConnectionService +from symphonia.infrastructure import ProviderConnectionRepository +from symphonia.providers import ( + AccessBasis, + Capability, + ProviderApiError, + ProviderCapabilities, + ProviderErrorCategory, + ProviderManifest, + ProviderRegistry, +) + + +NOW = datetime(2026, 9, 20, 12, 0, tzinfo=timezone.utc) + + +class FakeAdapter: + def __init__(self, error: ProviderApiError | None = None) -> None: + self.manifest = ProviderManifest("spotify", "Spotify", AccessBasis.OFFICIAL, "beta", "fixture") + self.error = error + + def capabilities(self, connection_id: str) -> ProviderCapabilities: + if self.error is not None: + raise self.error + return ProviderCapabilities( + enabled=frozenset({Capability.READ_PLAYLISTS}), + evidence_version="fixture-1", + observed_at="2026-09-20T12:00:00Z", + ) + + def read_playlist_pages(self, connection_id, playlist, cursor=None): + return () + + +class ProviderConnectionServiceTests(unittest.TestCase): + def setUp(self) -> None: + self.connections = ProviderConnectionRepository() + self.registry = ProviderRegistry() + self.adapter = FakeAdapter() + self.registry.register(self.adapter) + self.service = ProviderConnectionService(self.connections, self.registry) + + def tearDown(self) -> None: + self.connections.close() + + def register(self) -> None: + self.service.register_verified( + connection_id="spotify-1", + provider="spotify", + provider_account_id="account-1", + manifest_version="fixture-1", + secret_ref="secret-ref-1", + now=NOW, + ) + + def test_verified_registration_then_probe_publishes_capabilities(self) -> None: + self.register() + probed = self.service.probe("spotify-1", now=NOW) + self.assertEqual(probed.state.value, "connected") + self.assertTrue(probed.capabilities.supports(Capability.READ_PLAYLISTS)) + + def test_auth_error_requires_action_and_provider_outage_degrades(self) -> None: + self.register() + self.adapter.error = ProviderApiError(ProviderErrorCategory.AUTHENTICATION_REQUIRED, "expired grant") + action_required = self.service.probe("spotify-1", now=NOW) + self.assertEqual(action_required.state.value, "action_required") + self.assertEqual(action_required.health_code, "authentication_required") + + self.adapter.error = ProviderApiError(ProviderErrorCategory.PROVIDER_UNAVAILABLE, "provider unavailable") + degraded = self.service.probe("spotify-1", now=NOW) + self.assertEqual(degraded.state.value, "degraded") + self.assertEqual(degraded.health_code, "provider_unavailable") + + +if __name__ == "__main__": + unittest.main() From f98eb9af903b60f77889edfba3ed174f7b1404d1 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:35:34 +0200 Subject: [PATCH 031/167] feat: add scoped YouTube Data playlist reader --- docs/development/implementation-baseline.md | 1 + src/symphonia/providers/__init__.py | 2 + src/symphonia/providers/youtube.py | 166 ++++++++++++++++++++ tests/test_youtube_adapter.py | 83 ++++++++++ 4 files changed, 252 insertions(+) create mode 100644 src/symphonia/providers/youtube.py create mode 100644 tests/test_youtube_adapter.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 695d1ee..308f0c6 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -22,6 +22,7 @@ The first implementation increment is intentionally narrower than any provider o - Copy plans bind target connection and effective write-capability evidence into their digest. - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. - Offline-testable official Spotify playlist reader with bounded pagination and normalized error categories. +- Explicitly scoped official YouTube Data API video-playlist reader; it is not represented as YouTube Music. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. diff --git a/src/symphonia/providers/__init__.py b/src/symphonia/providers/__init__.py index d4bb1b0..e917eba 100644 --- a/src/symphonia/providers/__init__.py +++ b/src/symphonia/providers/__init__.py @@ -16,6 +16,7 @@ from .registry import ProviderAlreadyRegistered, ProviderNotRegistered, ProviderRegistry from .errors import ProviderApiError, ProviderErrorCategory from .spotify import JsonResponse, SpotifyAdapter, UrllibJsonClient +from .youtube import YouTubeDataAdapter from .importing import CollectionImportResult, ImportIssue, collect_playlist_pages, to_playlist_snapshot from .writing import PlaylistWriter, ProviderWriteError, TargetPlaylist, WriteOutcome, WriteResult @@ -43,6 +44,7 @@ "JsonResponse", "SpotifyAdapter", "UrllibJsonClient", + "YouTubeDataAdapter", "ProviderWriteError", "PlaylistWriter", "TargetPlaylist", diff --git a/src/symphonia/providers/youtube.py b/src/symphonia/providers/youtube.py new file mode 100644 index 0000000..1824e24 --- /dev/null +++ b/src/symphonia/providers/youtube.py @@ -0,0 +1,166 @@ +"""Official YouTube Data API adapter for video-playlist reads. + +This is intentionally not a YouTube Music adapter. The provider ID and +manifest make the narrower video-playlist contract visible to planning/UI. +""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping +from datetime import datetime, timedelta, timezone +from typing import Any + +from .contracts import ( + AccessBasis, + Capability, + MediaKind, + ProviderAdapter, + ProviderCapabilities, + ProviderManifest, + ProviderObjectRef, + ProviderPlaylistEntry, + ProviderPlaylistPage, +) +from .errors import ProviderApiError, ProviderErrorCategory +from .spotify import JsonClient, UrllibJsonClient + + +class YouTubeDataAdapter(ProviderAdapter): + """Translate ``playlistItems.list`` pages into normalized video entries.""" + + manifest = ProviderManifest( + provider="youtube_data", + display_name="YouTube Data API (video playlists)", + access_basis=AccessBasis.OFFICIAL, + maturity="experimental", + support_level="video-playlist-read", + upstream_dependencies=("YouTube Data API v3",), + reviewed_on="2026-09-20", + ) + + def __init__( + self, + client: JsonClient | None, + token_for_connection: Callable[[str], str], + page_size: int = 50, + api_key: str | None = None, + ) -> None: + if not 1 <= page_size <= 50: + raise ValueError("YouTube playlist page_size must be between 1 and 50") + self._client = client or UrllibJsonClient("https://www.googleapis.com/youtube/v3") + self._token_for_connection = token_for_connection + self._page_size = page_size + self._api_key = api_key + + def capabilities(self, connection_id: str) -> ProviderCapabilities: + self._request( + connection_id, + "/channels", + {"part": "id", "mine": "true", "maxResults": "1"}, + ) + return ProviderCapabilities( + enabled=frozenset({Capability.READ_PLAYLISTS}), + evidence_version="youtube-data-video-playlist-read-v1", + observed_at=datetime.now(timezone.utc).isoformat(timespec="seconds"), + ) + + def read_playlist_pages( + self, + connection_id: str, + playlist: ProviderObjectRef, + cursor: str | None = None, + ) -> tuple[ProviderPlaylistPage, ...]: + if playlist.provider != self.manifest.provider or playlist.object_type != "playlist": + raise ValueError("YouTube Data adapter requires a youtube_data playlist reference") + page_token = cursor + pages: list[ProviderPlaylistPage] = [] + position = 0 + while True: + query = { + "part": "snippet,contentDetails", + "playlistId": playlist.object_id, + "maxResults": str(self._page_size), + } + if page_token is not None: + query["pageToken"] = page_token + response = self._request(connection_id, "/playlistItems", query) + items = response.payload.get("items", []) + if not isinstance(items, list): + raise ProviderApiError(ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, "YouTube playlist items were not a list") + entries = tuple( + self._entry(playlist, item, position=position + index) + for index, item in enumerate(items) + ) + next_token = response.payload.get("nextPageToken") + if next_token and not entries: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "YouTube pagination advanced without returning items", + ) + pages.append( + ProviderPlaylistPage( + playlist=playlist, + entries=entries, + cursor=page_token, + next_cursor=str(next_token) if next_token else None, + complete=not bool(next_token), + revision=response.payload.get("etag"), + ) + ) + if not next_token: + return tuple(pages) + page_token = str(next_token) + position += len(entries) + + def _request(self, connection_id: str, path: str, query: Mapping[str, str]): + token = self._token_for_connection(connection_id) + if not token.strip(): + raise ProviderApiError(ProviderErrorCategory.AUTHENTICATION_REQUIRED, "YouTube connection has no usable access token") + complete_query = dict(query) + if self._api_key: + complete_query["key"] = self._api_key + response = self._client.request("GET", path, token=token, query=complete_query) + if 200 <= response.status < 300: + if not isinstance(response.payload, Mapping): + raise ProviderApiError(ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, "YouTube response was not a JSON object") + return response + error_payload = response.payload.get("error") if isinstance(response.payload, Mapping) else None + errors = error_payload.get("errors") if isinstance(error_payload, Mapping) else None + reason = errors[0].get("reason") if isinstance(errors, list) and errors and isinstance(errors[0], Mapping) else None + category = { + 401: ProviderErrorCategory.AUTHENTICATION_REQUIRED, + 403: ProviderErrorCategory.PERMISSION_DENIED, + 404: ProviderErrorCategory.NOT_FOUND, + 429: ProviderErrorCategory.RATE_LIMITED, + }.get(response.status, ProviderErrorCategory.PROVIDER_UNAVAILABLE if response.status >= 500 else ProviderErrorCategory.INVALID_REQUEST) + retry_at = None + retry_after = next((value for key, value in response.headers.items() if key.lower() == "retry-after"), None) + if category is ProviderErrorCategory.RATE_LIMITED and retry_after is not None: + try: + retry_at = datetime.now(timezone.utc) + timedelta(seconds=max(0, int(retry_after))) + except ValueError: + retry_at = None + raise ProviderApiError(category, "YouTube Data API request was not accepted", provider_code=str(reason or response.status), retry_at=retry_at) + + @staticmethod + def _entry(playlist: ProviderObjectRef, item: Any, *, position: int) -> ProviderPlaylistEntry: + snippet = item.get("snippet") if isinstance(item, Mapping) else None + content = item.get("contentDetails") if isinstance(item, Mapping) else None + snippet = snippet if isinstance(snippet, Mapping) else {} + content = content if isinstance(content, Mapping) else {} + resource = snippet.get("resourceId") + resource = resource if isinstance(resource, Mapping) else {} + video_id = resource.get("videoId") or content.get("videoId") + available = isinstance(video_id, str) and bool(video_id.strip()) + object_id = video_id if available else f"unavailable:{position}" + occurrence_id = str(item.get("id")) if isinstance(item, Mapping) and item.get("id") else f"{playlist.object_id}:{position}" + return ProviderPlaylistEntry( + occurrence_id=occurrence_id, + position=position, + track=ProviderObjectRef("youtube_data", "video", object_id, playlist.namespace), + media_kind=MediaKind.VIDEO, + title=snippet.get("title") if isinstance(snippet.get("title"), str) else None, + available=available, + source_added_at=snippet.get("publishedAt") if isinstance(snippet.get("publishedAt"), str) else None, + ) + diff --git a/tests/test_youtube_adapter.py b/tests/test_youtube_adapter.py new file mode 100644 index 0000000..fd79b31 --- /dev/null +++ b/tests/test_youtube_adapter.py @@ -0,0 +1,83 @@ +from __future__ import annotations + +import unittest + +from symphonia.providers import ( + Capability, + JsonResponse, + MediaKind, + ProviderObjectRef, + YouTubeDataAdapter, +) + + +class FakeClient: + def __init__(self) -> None: + self.calls = [] + + def request(self, method: str, path: str, *, token: str, query: dict[str, str], body=None) -> JsonResponse: + self.calls.append((method, path, token, query)) + if path == "/channels": + return JsonResponse(200, {"items": [{"id": "channel-1"}]}, {}) + if query.get("pageToken") is None: + return JsonResponse( + 200, + { + "etag": "etag-1", + "items": [ + { + "id": "playlist-item-1", + "snippet": { + "title": "Song video", + "publishedAt": "2026-09-20T12:00:00Z", + "resourceId": {"kind": "youtube#video", "videoId": "video-1"}, + }, + }, + { + "id": "playlist-item-2", + "snippet": {"title": "Deleted video", "resourceId": {"kind": "youtube#video"}}, + }, + ], + "nextPageToken": "page-2", + }, + {}, + ) + return JsonResponse( + 200, + { + "etag": "etag-2", + "items": [ + { + "id": "playlist-item-3", + "contentDetails": {"videoId": "video-2"}, + "snippet": {"title": "Second video", "resourceId": {}}, + } + ], + }, + {}, + ) + + +class YouTubeDataAdapterTests(unittest.TestCase): + def test_official_video_playlist_pages_preserve_unavailable_items(self) -> None: + client = FakeClient() + adapter = YouTubeDataAdapter(client, lambda connection_id: "access-token", page_size=2, api_key="public-key") + playlist = ProviderObjectRef("youtube_data", "playlist", "playlist-1", "google-connection-1") + pages = adapter.read_playlist_pages("google-connection-1", playlist) + self.assertEqual(len(pages), 2) + entries = [entry for page in pages for entry in page.entries] + self.assertEqual([entry.position for entry in entries], [0, 1, 2]) + self.assertEqual(entries[0].media_kind, MediaKind.VIDEO) + self.assertEqual(entries[0].track.object_id, "video-1") + self.assertFalse(entries[1].available) + self.assertEqual(entries[1].track.object_id, "unavailable:1") + self.assertEqual(client.calls[0][3]["key"], "public-key") + + def test_capability_probe_is_separate_from_playlist_read(self) -> None: + adapter = YouTubeDataAdapter(FakeClient(), lambda connection_id: "access-token") + capabilities = adapter.capabilities("google-connection-1") + self.assertEqual(capabilities.enabled, frozenset({Capability.READ_PLAYLISTS})) + + +if __name__ == "__main__": + unittest.main() From 93d9449bb7a241bcbf22e553785effd33434cc08 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:36:48 +0200 Subject: [PATCH 032/167] feat: add experimental Home Assistant App metadata --- README.md | 2 +- addon/README.md | 13 +++++++++ addon/config.yaml | 23 ++++++++++++++++ docs/development/implementation-baseline.md | 1 + tests/test_homeassistant_app_metadata.py | 29 +++++++++++++++++++++ 5 files changed, 67 insertions(+), 1 deletion(-) create mode 100644 addon/README.md create mode 100644 addon/config.yaml create mode 100644 tests/test_homeassistant_app_metadata.py diff --git a/README.md b/README.md index 4f87128..eb46918 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ Symphonia is a library-management and interoperability project, not a new music ## Project status -**Implementation foundation stage. Provider integrations and the Home Assistant App artifact are not implemented yet.** +**Implementation foundation stage. Provider adapters and an experimental Home Assistant App metadata scaffold exist; no published App image, complete UI, or OAuth flow is available yet.** The current work combines a reviewable source of truth with the first owner-approved, dependency-free domain/persistence slice. Official provider feasibility still needs validation: Google's public YouTube Data API can manage YouTube video playlists, but the research performed for this specification did not identify an official API exposing the complete YouTube Music library model. diff --git a/addon/README.md b/addon/README.md new file mode 100644 index 0000000..3adc55f --- /dev/null +++ b/addon/README.md @@ -0,0 +1,13 @@ +# Symphonia Home Assistant App metadata + +This directory is an experimental Supervisor App metadata scaffold. It follows +the App/service boundary used by `vypdev/homeassistant-gateway`: + +- management traffic is Ingress-only (`8099` is not mapped as a direct port); +- only `/data` is persistent; +- the App requests no Home Assistant or Supervisor API access; and +- the service image runs the repository root `Dockerfile` as a non-root user. + +The metadata is not a published release yet. The image reference, supported +architecture matrix, OAuth callback contract, UI, backup/restore behavior and +release signing still require their SDD gates before stable publication. diff --git a/addon/config.yaml b/addon/config.yaml new file mode 100644 index 0000000..71cc890 --- /dev/null +++ b/addon/config.yaml @@ -0,0 +1,23 @@ +name: Symphonia +version: "0.1.0" +slug: symphonia +description: Provider-independent music library hub and playlist migration service +url: https://github.com/vypdev/symphonia +image: ghcr.io/vypdev/symphonia +arch: + - amd64 + - aarch64 +startup: application +boot: auto +stage: experimental +init: false +ingress: true +ingress_port: 8099 +panel_icon: mdi:music-box-multiple +panel_title: Symphonia +map: + - data:rw +options: + log_level: info +schema: + log_level: list(debug|info|warning|error) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 308f0c6..ebc6594 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -23,6 +23,7 @@ The first implementation increment is intentionally narrower than any provider o - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. - Offline-testable official Spotify playlist reader with bounded pagination and normalized error categories. - Explicitly scoped official YouTube Data API video-playlist reader; it is not represented as YouTube Music. +- Experimental Home Assistant App metadata scaffold with Ingress-only management and `/data` persistence. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. diff --git a/tests/test_homeassistant_app_metadata.py b/tests/test_homeassistant_app_metadata.py new file mode 100644 index 0000000..d7feb46 --- /dev/null +++ b/tests/test_homeassistant_app_metadata.py @@ -0,0 +1,29 @@ +from __future__ import annotations + +from pathlib import Path +import unittest + + +ROOT = Path(__file__).resolve().parents[1] + + +class HomeAssistantAppMetadataTests(unittest.TestCase): + def test_experimental_metadata_is_ingress_only_and_least_privilege(self) -> None: + config = (ROOT / "addon" / "config.yaml").read_text(encoding="utf-8") + self.assertIn("stage: experimental", config) + self.assertIn("ingress: true", config) + self.assertIn("ingress_port: 8099", config) + self.assertIn("- data:rw", config) + self.assertNotIn("ports:", config) + self.assertNotIn("homeassistant_api:", config) + self.assertNotIn("hassio_api:", config) + + def test_metadata_declares_only_the_architectures_of_the_experimental_matrix(self) -> None: + config = (ROOT / "addon" / "config.yaml").read_text(encoding="utf-8") + self.assertIn(" - amd64", config) + self.assertIn(" - aarch64", config) + self.assertNotIn(" - armv7", config) + + +if __name__ == "__main__": + unittest.main() From 8bc52e87d4e09ad4b7a118b9c579440501a208f7 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:38:21 +0200 Subject: [PATCH 033/167] feat: support safe Ingress base paths --- docs/development/implementation-baseline.md | 1 + src/symphonia/__main__.py | 8 ++- src/symphonia/runtime/http.py | 57 ++++++++++++++++++--- tests/test_runtime_http.py | 21 ++++++++ 4 files changed, 77 insertions(+), 10 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index ebc6594..6dac0ca 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -24,6 +24,7 @@ The first implementation increment is intentionally narrower than any provider o - Offline-testable official Spotify playlist reader with bounded pagination and normalized error categories. - Explicitly scoped official YouTube Data API video-playlist reader; it is not represented as YouTube Music. - Experimental Home Assistant App metadata scaffold with Ingress-only management and `/data` persistence. +- Ingress-relative health/version routing with normalized, traversal-safe base paths. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. diff --git a/src/symphonia/__main__.py b/src/symphonia/__main__.py index 6bbd819..700b632 100644 --- a/src/symphonia/__main__.py +++ b/src/symphonia/__main__.py @@ -17,8 +17,13 @@ def main() -> None: default=os.getenv("SYMPHONIA_DATABASE", "./symphonia.sqlite3"), help="SQLite path; Home Assistant App deployments should use /data/symphonia.sqlite3", ) + parser.add_argument( + "--ingress-path", + default=os.getenv("SYMPHONIA_INGRESS_PATH", "/"), + help="Ingress base path, for example /local_symphonia", + ) args = parser.parse_args() - server = create_server(args.host, args.port, args.database) + server = create_server(args.host, args.port, args.database, args.ingress_path) try: server.serve_forever() except KeyboardInterrupt: @@ -30,4 +35,3 @@ def main() -> None: if __name__ == "__main__": main() - diff --git a/src/symphonia/runtime/http.py b/src/symphonia/runtime/http.py index 6fd543c..66edffa 100644 --- a/src/symphonia/runtime/http.py +++ b/src/symphonia/runtime/http.py @@ -5,6 +5,7 @@ from http.server import BaseHTTPRequestHandler, HTTPServer import json from typing import Any +from urllib.parse import urlsplit from symphonia import __version__ from symphonia.infrastructure.sqlite_operations import OperationRepository @@ -13,10 +14,11 @@ class SymphoniaHTTPServer(HTTPServer): allow_reuse_address = True - def __init__(self, address: tuple[str, int], repository: OperationRepository) -> None: + def __init__(self, address: tuple[str, int], repository: OperationRepository, ingress_path: str = "/") -> None: super().__init__(address, SymphoniaRequestHandler) self.repository = repository self.service_version = __version__ + self.ingress_path = _normalize_base_path(ingress_path) class SymphoniaRequestHandler(BaseHTTPRequestHandler): @@ -25,7 +27,12 @@ class SymphoniaRequestHandler(BaseHTTPRequestHandler): server: SymphoniaHTTPServer def do_GET(self) -> None: # noqa: N802 - stdlib handler API - status, payload = route_get(self.path, self.server.repository, self.server.service_version) + status, payload = route_get( + self.path, + self.server.repository, + self.server.service_version, + self.server.ingress_path, + ) self._json(status, payload) def _json(self, status: int, payload: dict[str, Any]) -> None: @@ -42,14 +49,24 @@ def log_message(self, format: str, *args: object) -> None: return -def create_server(host: str = "127.0.0.1", port: int = 8099, database_path: str = ":memory:") -> SymphoniaHTTPServer: +def create_server( + host: str = "127.0.0.1", + port: int = 8099, + database_path: str = ":memory:", + ingress_path: str = "/", +) -> SymphoniaHTTPServer: """Create a server with an already-migrated durable operation store.""" repository = OperationRepository(database_path) - return SymphoniaHTTPServer((host, port), repository) + return SymphoniaHTTPServer((host, port), repository, ingress_path) -def route_get(path: str, repository: OperationRepository, service_version: str = __version__) -> tuple[int, dict[str, Any]]: +def route_get( + path: str, + repository: OperationRepository, + service_version: str = __version__, + ingress_path: str = "/", +) -> tuple[int, dict[str, Any]]: """Resolve a GET request without opening a socket. Keeping this decision pure-ish makes health/readiness contract tests work @@ -57,9 +74,12 @@ def route_get(path: str, repository: OperationRepository, service_version: str = mistaken for application readiness. """ - if path == "/health": + relative_path = _relative_path(path, _normalize_base_path(ingress_path)) + if relative_path is None: + return 404, {"error": "not_found"} + if relative_path == "/health": return 200, {"service": "symphonia", "status": "ok", "version": service_version} - if path == "/ready": + if relative_path == "/ready": try: healthy = repository.healthcheck() except Exception: # readiness must fail closed without exposing internals @@ -68,6 +88,27 @@ def route_get(path: str, repository: OperationRepository, service_version: str = 503, {"service": "symphonia", "status": "not_ready"}, ) - if path == "/version": + if relative_path == "/version": return 200, {"service": "symphonia", "version": service_version} return 404, {"error": "not_found"} + + +def _normalize_base_path(value: str) -> str: + if not value or not value.startswith("/"): + raise ValueError("ingress path must start with '/'") + normalized = value.rstrip("/") or "/" + if "//" in normalized or "/.." in normalized or "/./" in normalized: + raise ValueError("ingress path contains an unsafe segment") + return normalized + + +def _relative_path(request_path: str, base_path: str) -> str | None: + path = urlsplit(request_path).path or "/" + if base_path == "/": + return path + if path == base_path: + return "/" + prefix = f"{base_path}/" + if path.startswith(prefix): + return path[len(base_path):] or "/" + return None diff --git a/tests/test_runtime_http.py b/tests/test_runtime_http.py index cf40fce..511bf78 100644 --- a/tests/test_runtime_http.py +++ b/tests/test_runtime_http.py @@ -31,6 +31,27 @@ def test_version_and_unknown_routes(self) -> None: self.assertEqual(status, 404) self.assertEqual(payload, {"error": "not_found"}) + def test_ingress_base_path_is_stripped_without_accepting_sibling_paths(self) -> None: + status, payload = route_get( + "/local_symphonia/ready?poll=1", + self.repository, + ingress_path="/local_symphonia", + ) + self.assertEqual(status, 200) + self.assertEqual(payload["status"], "ready") + + status, payload = route_get( + "/local_symphonia-extra/ready", + self.repository, + ingress_path="/local_symphonia", + ) + self.assertEqual(status, 404) + self.assertEqual(payload, {"error": "not_found"}) + + def test_unsafe_ingress_base_path_is_rejected(self) -> None: + with self.assertRaises(ValueError): + route_get("/ready", self.repository, ingress_path="/bad/../path") + if __name__ == "__main__": unittest.main() From 969652d34c276b2f5b3f02e9d8ea84f172cc1693 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:39:31 +0200 Subject: [PATCH 034/167] feat: intersect layered provider capabilities --- docs/development/implementation-baseline.md | 1 + src/symphonia/providers/__init__.py | 5 +++ src/symphonia/providers/capabilities.py | 48 +++++++++++++++++++++ tests/test_capabilities.py | 41 ++++++++++++++++++ 4 files changed, 95 insertions(+) create mode 100644 src/symphonia/providers/capabilities.py create mode 100644 tests/test_capabilities.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 6dac0ca..2021317 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -16,6 +16,7 @@ The first implementation increment is intentionally narrower than any provider o - Explicit provider rate-limit waits with absolute retry times and durable eligibility. - Provider connection persistence with account uniqueness, opaque secret references, health states, and effective capability evidence. - Application connection service for verified-account registration, capability probes, degraded health, and reauthorization-required classification. +- Pure capability-layer intersection and requirement checks for adapter/connection/object/health constraints. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Application authorization boundary that generates one-use state without persisting the raw value. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. diff --git a/src/symphonia/providers/__init__.py b/src/symphonia/providers/__init__.py index e917eba..adf0696 100644 --- a/src/symphonia/providers/__init__.py +++ b/src/symphonia/providers/__init__.py @@ -17,6 +17,7 @@ from .errors import ProviderApiError, ProviderErrorCategory from .spotify import JsonResponse, SpotifyAdapter, UrllibJsonClient from .youtube import YouTubeDataAdapter +from .capabilities import CapabilityError, CapabilityLayers, missing_capabilities, require_capabilities from .importing import CollectionImportResult, ImportIssue, collect_playlist_pages, to_playlist_snapshot from .writing import PlaylistWriter, ProviderWriteError, TargetPlaylist, WriteOutcome, WriteResult @@ -25,6 +26,8 @@ "AuthorizationAttempt", "AuthorizationState", "Capability", + "CapabilityError", + "CapabilityLayers", "CollectionImportResult", "ConnectionState", "ImportIssue", @@ -51,5 +54,7 @@ "WriteOutcome", "WriteResult", "collect_playlist_pages", + "missing_capabilities", + "require_capabilities", "to_playlist_snapshot", ] diff --git a/src/symphonia/providers/capabilities.py b/src/symphonia/providers/capabilities.py new file mode 100644 index 0000000..4902f4a --- /dev/null +++ b/src/symphonia/providers/capabilities.py @@ -0,0 +1,48 @@ +"""Pure capability intersection and requirement checks.""" + +from __future__ import annotations + +from collections.abc import Iterable +from dataclasses import dataclass + +from .contracts import Capability, ProviderCapabilities + + +class CapabilityError(ValueError): + pass + + +@dataclass(frozen=True, slots=True) +class CapabilityLayers: + adapter: ProviderCapabilities + connection: ProviderCapabilities + object: ProviderCapabilities | None = None + health: ProviderCapabilities | None = None + + def effective(self) -> ProviderCapabilities: + layers = [self.adapter, self.connection] + if self.object is not None: + layers.append(self.object) + if self.health is not None: + layers.append(self.health) + enabled = set(layers[0].enabled) + for layer in layers[1:]: + enabled.intersection_update(layer.enabled) + versions = "+".join(layer.evidence_version for layer in layers) + observed_at = max(layer.observed_at for layer in layers) + return ProviderCapabilities(frozenset(enabled), f"intersection:{versions}", observed_at) + + +def missing_capabilities( + capabilities: ProviderCapabilities, + required: Iterable[Capability], +) -> frozenset[Capability]: + return frozenset(capability for capability in required if capability not in capabilities.enabled) + + +def require_capabilities(capabilities: ProviderCapabilities, required: Iterable[Capability]) -> None: + missing = missing_capabilities(capabilities, required) + if missing: + names = ", ".join(sorted(capability.value for capability in missing)) + raise CapabilityError(f"required capabilities are unavailable: {names}") + diff --git a/tests/test_capabilities.py b/tests/test_capabilities.py new file mode 100644 index 0000000..1d8b061 --- /dev/null +++ b/tests/test_capabilities.py @@ -0,0 +1,41 @@ +from __future__ import annotations + +import unittest + +from symphonia.providers import ( + Capability, + CapabilityError, + CapabilityLayers, + ProviderCapabilities, + missing_capabilities, + require_capabilities, +) + + +def capabilities(enabled: set[Capability], version: str) -> ProviderCapabilities: + return ProviderCapabilities(frozenset(enabled), version, version) + + +class CapabilityTests(unittest.TestCase): + def test_effective_capabilities_intersect_all_layers(self) -> None: + effective = CapabilityLayers( + adapter=capabilities({Capability.READ_PLAYLISTS, Capability.CREATE_PLAYLIST}, "adapter"), + connection=capabilities({Capability.READ_PLAYLISTS, Capability.CREATE_PLAYLIST}, "connection"), + object=capabilities({Capability.READ_PLAYLISTS}, "object"), + health=capabilities({Capability.READ_PLAYLISTS}, "health"), + ).effective() + self.assertEqual(effective.enabled, frozenset({Capability.READ_PLAYLISTS})) + self.assertEqual(effective.evidence_version, "intersection:adapter+connection+object+health") + + def test_requirement_check_reports_missing_capabilities(self) -> None: + current = capabilities({Capability.READ_PLAYLISTS}, "probe") + self.assertEqual( + missing_capabilities(current, {Capability.READ_PLAYLISTS, Capability.CREATE_PLAYLIST}), + frozenset({Capability.CREATE_PLAYLIST}), + ) + with self.assertRaises(CapabilityError): + require_capabilities(current, {Capability.CREATE_PLAYLIST}) + + +if __name__ == "__main__": + unittest.main() From c0effbde3c22d418b8562fff8223b928017b8088 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:40:24 +0200 Subject: [PATCH 035/167] feat: add durable operation runner --- docs/development/implementation-baseline.md | 1 + src/symphonia/application/__init__.py | 2 + src/symphonia/application/operation_runner.py | 46 ++++++++++++++ tests/test_operation_runner.py | 63 +++++++++++++++++++ 4 files changed, 112 insertions(+) create mode 100644 src/symphonia/application/operation_runner.py create mode 100644 tests/test_operation_runner.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 2021317..3cdf9fc 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -17,6 +17,7 @@ The first implementation increment is intentionally narrower than any provider o - Provider connection persistence with account uniqueness, opaque secret references, health states, and effective capability evidence. - Application connection service for verified-account registration, capability probes, degraded health, and reauthorization-required classification. - Pure capability-layer intersection and requirement checks for adapter/connection/object/health constraints. +- Durable operation runner that atomically claims eligible work and fails unwired operation types before side effects. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Application authorization boundary that generates one-use state without persisting the raw value. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. diff --git a/src/symphonia/application/__init__.py b/src/symphonia/application/__init__.py index 47504ce..3200b33 100644 --- a/src/symphonia/application/__init__.py +++ b/src/symphonia/application/__init__.py @@ -6,6 +6,7 @@ from .library_import import ImportPublication, LibraryImportService from .authorization import AuthorizationService, AuthorizationStart from .provider_connections import ProviderConnectionService +from .operation_runner import OperationRunner __all__ = [ "CopyExecutionService", @@ -17,4 +18,5 @@ "ImportPublication", "LibraryImportService", "ProviderConnectionService", + "OperationRunner", ] diff --git a/src/symphonia/application/operation_runner.py b/src/symphonia/application/operation_runner.py new file mode 100644 index 0000000..fd714ce --- /dev/null +++ b/src/symphonia/application/operation_runner.py @@ -0,0 +1,46 @@ +"""Small durable-operation dispatcher for the single-process runtime profile.""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping +from dataclasses import dataclass +from datetime import datetime + +from symphonia.infrastructure.sqlite_operations import OperationRecord, OperationRepository + + +OperationHandler = Callable[[OperationRecord, str, datetime], OperationRecord] + + +@dataclass(frozen=True, slots=True) +class OperationRunner: + operations: OperationRepository + handlers: Mapping[str, OperationHandler] + + def run_once( + self, + *, + worker_id: str, + now: datetime, + lease_seconds: int = 30, + operation_type: str | None = None, + ) -> OperationRecord | None: + operation = self.operations.claim_next( + worker_id=worker_id, + now=now, + lease_seconds=lease_seconds, + operation_type=operation_type, + ) + if operation is None: + return None + handler = self.handlers.get(operation.operation_type) + if handler is None: + return self.operations.checkpoint( + operation.operation_id, + worker_id=worker_id, + checkpoint={**operation.checkpoint, "failure_code": "handler_not_registered"}, + now=now, + state="failed", + ) + return handler(operation, worker_id, now) + diff --git a/tests/test_operation_runner.py b/tests/test_operation_runner.py new file mode 100644 index 0000000..8494134 --- /dev/null +++ b/tests/test_operation_runner.py @@ -0,0 +1,63 @@ +from __future__ import annotations + +from datetime import datetime, timezone +import unittest + +from symphonia.application import OperationRunner +from symphonia.infrastructure import OperationRepository + + +NOW = datetime(2026, 9, 20, 12, 0, tzinfo=timezone.utc) + + +class OperationRunnerTests(unittest.TestCase): + def setUp(self) -> None: + self.repository = OperationRepository() + + def tearDown(self) -> None: + self.repository.close() + + def test_runner_dispatches_only_durable_eligible_work(self) -> None: + operation = self.repository.create( + operation_type="fixture", + idempotency_key="fixture-1", + payload={}, + now=NOW, + ) + calls = [] + + def handler(claimed, worker_id, now): + calls.append((claimed.operation_id, worker_id)) + return self.repository.checkpoint( + claimed.operation_id, + worker_id=worker_id, + checkpoint={"done": True}, + now=now, + state="succeeded", + ) + + runner = OperationRunner(self.repository, {"fixture": handler}) + completed = runner.run_once(worker_id="worker-a", now=NOW) + self.assertEqual(completed.state, "succeeded") + self.assertEqual(calls, [(operation.operation_id, "worker-a")]) + self.assertIsNone(runner.run_once(worker_id="worker-a", now=NOW)) + + def test_missing_handler_fails_before_external_work(self) -> None: + operation = self.repository.create( + operation_type="unwired", + idempotency_key="unwired-1", + payload={}, + now=NOW, + ) + result = OperationRunner(self.repository, {}).run_once(worker_id="worker-a", now=NOW) + self.assertEqual(result.operation_id, operation.operation_id) + self.assertEqual(result.state, "failed") + self.assertEqual(result.checkpoint["failure_code"], "handler_not_registered") + self.assertEqual( + [event.event_type for event in self.repository.events(operation.operation_id)], + ["created", "claimed", "checkpointed"], + ) + + +if __name__ == "__main__": + unittest.main() From 3165d6ad326b98ca57730a884f295cddd1328ec2 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:41:27 +0200 Subject: [PATCH 036/167] feat: dispatch copy execution through operation runner --- docs/development/implementation-baseline.md | 1 + src/symphonia/application/copy_execution.py | 50 ++++++++++++++++++++- tests/test_copy_execution.py | 20 ++++++++- 3 files changed, 69 insertions(+), 2 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 3cdf9fc..78fac10 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -18,6 +18,7 @@ The first implementation increment is intentionally narrower than any provider o - Application connection service for verified-account registration, capability probes, degraded health, and reauthorization-required classification. - Pure capability-layer intersection and requirement checks for adapter/connection/object/health constraints. - Durable operation runner that atomically claims eligible work and fails unwired operation types before side effects. +- Copy executor can run as a claimed operation handler, preserving the same restart/checkpoint semantics under the runner. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Application authorization boundary that generates one-use state without persisting the raw value. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. diff --git a/src/symphonia/application/copy_execution.py b/src/symphonia/application/copy_execution.py index 7113742..7151af7 100644 --- a/src/symphonia/application/copy_execution.py +++ b/src/symphonia/application/copy_execution.py @@ -8,7 +8,7 @@ from symphonia.domain.models import PlanAcceptanceError from symphonia.infrastructure.sqlite_operations import OperationRecord, OperationRepository -from symphonia.infrastructure.sqlite_plans import CopyPlanRepository +from symphonia.infrastructure.sqlite_plans import CopyPlanRepository, StoredCopyPlan from symphonia.providers.writing import PlaylistWriter, ProviderWriteError, WriteOutcome @@ -44,6 +44,54 @@ def execute( now=now, lease_seconds=lease_seconds, ) + return self._execute_claimed( + stored, + operation, + writer=writer, + worker_id=worker_id, + now=now, + lease_seconds=lease_seconds, + ) + + def execute_claimed( + self, + operation: OperationRecord, + *, + writer: PlaylistWriter, + worker_id: str, + now: datetime, + lease_seconds: int = 30, + ) -> OperationRecord: + """Continue a copy operation already claimed by an operation runner.""" + + if operation.state != "running" or operation.worker_id != worker_id: + raise ValueError("copy operation must be running under the supplied worker") + digest = operation.payload.get("plan_digest") + if not isinstance(digest, str) or not digest.strip(): + raise PlanAcceptanceError("copy operation payload has no plan digest") + stored = self.plans.get(digest) + if stored.accepted_at is None: + raise PlanAcceptanceError("copy plan must be accepted before execution") + return self._execute_claimed( + stored, + operation, + writer=writer, + worker_id=worker_id, + now=now, + lease_seconds=lease_seconds, + ) + + def _execute_claimed( + self, + stored: StoredCopyPlan, + operation: OperationRecord, + *, + writer: PlaylistWriter, + worker_id: str, + now: datetime, + lease_seconds: int, + ) -> OperationRecord: + digest = stored.plan.digest checkpoint = dict(operation.checkpoint) if operation.cancel_requested: return self._finish(operation, worker_id, checkpoint, now, "cancelled") diff --git a/tests/test_copy_execution.py b/tests/test_copy_execution.py index fa95257..f0e8600 100644 --- a/tests/test_copy_execution.py +++ b/tests/test_copy_execution.py @@ -3,7 +3,7 @@ from datetime import datetime, timedelta, timezone import unittest -from symphonia.application import CopyExecutionService, CopyPlanningService, CopyWorkflowService +from symphonia.application import CopyExecutionService, CopyPlanningService, CopyWorkflowService, OperationRunner from symphonia.domain import CopyPolicy, EntryClassification, PlaylistSnapshot, SourcePlaylistEntry from symphonia.infrastructure import CopyPlanRepository, OperationRepository from symphonia.providers import TargetPlaylist, WriteOutcome, WriteResult @@ -115,6 +115,24 @@ def test_retryable_write_releases_operation_until_scheduled(self) -> None: self.assertEqual(operation.state, "retry_scheduled") self.assertEqual(operation.next_run_at, NOW + timedelta(seconds=30)) + def test_operation_runner_can_dispatch_claimed_copy_execution(self) -> None: + digest = self.accepted_digest() + self.workflow.enqueue_accepted_plan(digest, now=NOW) + runner = OperationRunner( + self.operations, + { + "copy_playlist": lambda operation, worker_id, now: self.executor.execute_claimed( + operation, + writer=self.writer, + worker_id=worker_id, + now=now, + ) + }, + ) + operation = runner.run_once(worker_id="worker-a", now=NOW) + self.assertEqual(operation.state, "succeeded") + self.assertEqual([track for _, track in self.writer.added], ["target-1"]) + def test_rate_limited_write_waits_until_provider_deadline(self) -> None: digest = self.accepted_digest() step_key = f"{digest}:entry:occ-1" From 4a71aa93287145b010c3b4701cfda4590d18fbe8 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:45:24 +0200 Subject: [PATCH 037/167] feat: add experimental Apple Music playlist reader --- docs/development/implementation-baseline.md | 1 + src/symphonia/providers/__init__.py | 4 + src/symphonia/providers/apple_music.py | 238 ++++++++++++++++++++ src/symphonia/providers/contracts.py | 5 +- tests/test_apple_music_adapter.py | 91 ++++++++ 5 files changed, 336 insertions(+), 3 deletions(-) create mode 100644 src/symphonia/providers/apple_music.py create mode 100644 tests/test_apple_music_adapter.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 78fac10..7f60573 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -26,6 +26,7 @@ The first implementation increment is intentionally narrower than any provider o - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. - Offline-testable official Spotify playlist reader with bounded pagination and normalized error categories. - Explicitly scoped official YouTube Data API video-playlist reader; it is not represented as YouTube Music. +- Experimental official Apple Music library-playlist reader with separate developer/user token inputs. - Experimental Home Assistant App metadata scaffold with Ingress-only management and `/data` persistence. - Ingress-relative health/version routing with normalized, traversal-safe base paths. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. diff --git a/src/symphonia/providers/__init__.py b/src/symphonia/providers/__init__.py index adf0696..d68a509 100644 --- a/src/symphonia/providers/__init__.py +++ b/src/symphonia/providers/__init__.py @@ -17,6 +17,7 @@ from .errors import ProviderApiError, ProviderErrorCategory from .spotify import JsonResponse, SpotifyAdapter, UrllibJsonClient from .youtube import YouTubeDataAdapter +from .apple_music import AppleJsonResponse, AppleMusicAdapter, UrllibAppleMusicClient from .capabilities import CapabilityError, CapabilityLayers, missing_capabilities, require_capabilities from .importing import CollectionImportResult, ImportIssue, collect_playlist_pages, to_playlist_snapshot from .writing import PlaylistWriter, ProviderWriteError, TargetPlaylist, WriteOutcome, WriteResult @@ -48,6 +49,9 @@ "SpotifyAdapter", "UrllibJsonClient", "YouTubeDataAdapter", + "AppleMusicAdapter", + "AppleJsonResponse", + "UrllibAppleMusicClient", "ProviderWriteError", "PlaylistWriter", "TargetPlaylist", diff --git a/src/symphonia/providers/apple_music.py b/src/symphonia/providers/apple_music.py new file mode 100644 index 0000000..f38750f --- /dev/null +++ b/src/symphonia/providers/apple_music.py @@ -0,0 +1,238 @@ +"""Offline-testable Apple Music library-playlist reader.""" + +from __future__ import annotations + +from collections.abc import Callable, Mapping +from datetime import datetime, timedelta, timezone +import json +from typing import Any, Protocol +from urllib.error import HTTPError, URLError +from urllib.parse import urlencode, parse_qs, urlsplit +from urllib.request import Request, urlopen + +from .contracts import ( + AccessBasis, + Capability, + MediaKind, + ProviderAdapter, + ProviderCapabilities, + ProviderManifest, + ProviderObjectRef, + ProviderPlaylistEntry, + ProviderPlaylistPage, +) +from .errors import ProviderApiError, ProviderErrorCategory + + +class AppleJsonClient(Protocol): + def request( + self, + method: str, + path: str, + *, + developer_token: str, + user_token: str, + query: Mapping[str, str], + ) -> "AppleJsonResponse": ... + + +class AppleJsonResponse: + def __init__(self, status: int, payload: Mapping[str, Any], headers: Mapping[str, str]) -> None: + self.status = status + self.payload = payload + self.headers = headers + + +class UrllibAppleMusicClient: + def __init__(self, base_url: str = "https://api.music.apple.com/v1", timeout_seconds: float = 10.0) -> None: + if timeout_seconds <= 0: + raise ValueError("timeout_seconds must be positive") + self.base_url = base_url.rstrip("/") + self.timeout_seconds = timeout_seconds + + def request( + self, + method: str, + path: str, + *, + developer_token: str, + user_token: str, + query: Mapping[str, str], + ) -> AppleJsonResponse: + url = f"{self.base_url}/{path.lstrip('/')}" + if query: + url = f"{url}?{urlencode(query)}" + request = Request( + url, + method=method, + headers={ + "Authorization": f"Bearer {developer_token}", + "Music-User-Token": user_token, + "Accept": "application/json", + }, + ) + try: + with urlopen(request, timeout=self.timeout_seconds) as response: + body = response.read() + payload = json.loads(body.decode("utf-8")) if body else {} + return AppleJsonResponse(response.status, payload, dict(response.headers.items())) + except HTTPError as error: + body = error.read() + try: + payload = json.loads(body.decode("utf-8")) if body else {} + except (UnicodeDecodeError, json.JSONDecodeError): + payload = {} + return AppleJsonResponse(error.code, payload, dict(error.headers.items())) + except TimeoutError as error: + raise ProviderApiError(ProviderErrorCategory.TIMEOUT, "Apple Music request timed out") from error + except URLError as error: + raise ProviderApiError(ProviderErrorCategory.NETWORK_ERROR, "Apple Music request failed") from error + + +class AppleMusicAdapter(ProviderAdapter): + manifest = ProviderManifest( + provider="apple_music", + display_name="Apple Music", + access_basis=AccessBasis.OFFICIAL, + maturity="experimental", + support_level="library-playlist-read", + upstream_dependencies=("Apple Music API", "MusicKit user authentication"), + reviewed_on="2026-09-20", + ) + + def __init__( + self, + client: AppleJsonClient | None, + tokens_for_connection: Callable[[str], tuple[str, str]], + page_size: int = 25, + ) -> None: + if not 1 <= page_size <= 100: + raise ValueError("Apple Music playlist page_size must be between 1 and 100") + self._client = client or UrllibAppleMusicClient() + self._tokens_for_connection = tokens_for_connection + self._page_size = page_size + + def capabilities(self, connection_id: str) -> ProviderCapabilities: + self._request(connection_id, "/me/library/playlists", {"limit": "1", "offset": "0"}) + return ProviderCapabilities( + enabled=frozenset({Capability.READ_PLAYLISTS}), + evidence_version="apple-library-playlist-read-v1", + observed_at=datetime.now(timezone.utc).isoformat(timespec="seconds"), + ) + + def read_playlist_pages( + self, + connection_id: str, + playlist: ProviderObjectRef, + cursor: str | None = None, + ) -> tuple[ProviderPlaylistPage, ...]: + if playlist.provider != self.manifest.provider or playlist.object_type != "library-playlists": + raise ValueError("Apple Music adapter requires an apple_music library-playlists reference") + offset = self._parse_offset(cursor) + pages: list[ProviderPlaylistPage] = [] + while True: + response = self._request( + connection_id, + f"/me/library/playlists/{playlist.object_id}/tracks", + {"limit": str(self._page_size), "offset": str(offset)}, + ) + items = response.payload.get("data", []) + if not isinstance(items, list): + raise ProviderApiError(ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, "Apple Music playlist data was not a list") + entries = tuple(self._entry(playlist, item, offset + index) for index, item in enumerate(items)) + next_url = response.payload.get("next") + if next_url and not entries: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Apple Music pagination advanced without returning items", + ) + next_offset = self._next_offset(next_url, offset + len(entries)) + pages.append( + ProviderPlaylistPage( + playlist=playlist, + entries=entries, + cursor=None if not pages and cursor is None else str(offset), + next_cursor=None if next_offset is None else str(next_offset), + complete=next_offset is None, + revision=response.payload.get("etag"), + ) + ) + if next_offset is None: + return tuple(pages) + offset = next_offset + + def _request(self, connection_id: str, path: str, query: Mapping[str, str]) -> AppleJsonResponse: + developer_token, user_token = self._tokens_for_connection(connection_id) + if not developer_token.strip() or not user_token.strip(): + raise ProviderApiError(ProviderErrorCategory.AUTHENTICATION_REQUIRED, "Apple Music connection has no usable tokens") + response = self._client.request( + "GET", + path, + developer_token=developer_token, + user_token=user_token, + query=query, + ) + if 200 <= response.status < 300: + if not isinstance(response.payload, Mapping): + raise ProviderApiError(ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, "Apple Music response was not a JSON object") + return response + category = { + 401: ProviderErrorCategory.AUTHENTICATION_REQUIRED, + 403: ProviderErrorCategory.PERMISSION_DENIED, + 404: ProviderErrorCategory.NOT_FOUND, + 429: ProviderErrorCategory.RATE_LIMITED, + }.get(response.status, ProviderErrorCategory.PROVIDER_UNAVAILABLE if response.status >= 500 else ProviderErrorCategory.INVALID_REQUEST) + retry_at = None + retry_after = next((value for key, value in response.headers.items() if key.lower() == "retry-after"), None) + if category is ProviderErrorCategory.RATE_LIMITED and retry_after is not None: + try: + retry_at = datetime.now(timezone.utc) + timedelta(seconds=max(0, int(retry_after))) + except ValueError: + retry_at = None + provider_code = str(response.status) + if isinstance(response.payload.get("errors"), list) and response.payload["errors"]: + first = response.payload["errors"][0] + if isinstance(first, Mapping) and first.get("code"): + provider_code = str(first["code"]) + raise ProviderApiError(category, "Apple Music API request was not accepted", provider_code=provider_code, retry_at=retry_at) + + @staticmethod + def _parse_offset(cursor: str | None) -> int: + if cursor is None: + return 0 + try: + value = int(cursor) + except ValueError as error: + raise ValueError("Apple Music playlist cursor must be an integer offset") from error + if value < 0: + raise ValueError("Apple Music playlist cursor must not be negative") + return value + + @staticmethod + def _next_offset(next_url: Any, fallback: int) -> int | None: + if not next_url: + return None + if isinstance(next_url, str): + values = parse_qs(urlsplit(next_url).query).get("offset") + if values: + return AppleMusicAdapter._parse_offset(values[0]) + return fallback + + @staticmethod + def _entry(playlist: ProviderObjectRef, item: Any, position: int) -> ProviderPlaylistEntry: + if not isinstance(item, Mapping): + item = {} + object_id = item.get("id") + available = isinstance(object_id, str) and bool(object_id.strip()) + object_id = object_id if available else f"unavailable:{position}" + object_type = str(item.get("type") or "songs") + attributes = item.get("attributes") if isinstance(item.get("attributes"), Mapping) else {} + return ProviderPlaylistEntry( + occurrence_id=f"{playlist.object_id}:{position}", + position=position, + track=ProviderObjectRef("apple_music", object_type, object_id, playlist.namespace), + media_kind=MediaKind.TRACK, + title=attributes.get("name") if isinstance(attributes.get("name"), str) else None, + available=available, + ) + diff --git a/src/symphonia/providers/contracts.py b/src/symphonia/providers/contracts.py index b106c73..bb2c630 100644 --- a/src/symphonia/providers/contracts.py +++ b/src/symphonia/providers/contracts.py @@ -115,8 +115,8 @@ class ProviderPlaylistPage: revision: str | None = None def __post_init__(self) -> None: - if self.playlist.object_type != "playlist": - raise ValueError("playlist page requires a playlist object reference") + if self.playlist.object_type not in {"playlist", "library-playlists"}: + raise ValueError("playlist page requires a playlist or library-playlists object reference") positions = [entry.position for entry in self.entries] if positions != sorted(positions): raise ValueError("page entries must be ordered by position") @@ -135,4 +135,3 @@ def capabilities(self, connection_id: str) -> ProviderCapabilities: ... def read_playlist_pages( self, connection_id: str, playlist: ProviderObjectRef, cursor: str | None = None ) -> Iterable[ProviderPlaylistPage]: ... - diff --git a/tests/test_apple_music_adapter.py b/tests/test_apple_music_adapter.py new file mode 100644 index 0000000..8932f57 --- /dev/null +++ b/tests/test_apple_music_adapter.py @@ -0,0 +1,91 @@ +from __future__ import annotations + +import unittest + +from symphonia.providers import ( + AppleJsonResponse, + AppleMusicAdapter, + Capability, + MediaKind, + ProviderApiError, + ProviderErrorCategory, + ProviderObjectRef, +) + + +class FakeAppleClient: + def __init__(self) -> None: + self.calls: list[tuple[str, str, str, str, dict[str, str]]] = [] + + def request( + self, + method: str, + path: str, + *, + developer_token: str, + user_token: str, + query: dict[str, str], + ) -> AppleJsonResponse: + self.calls.append((method, path, developer_token, user_token, query)) + if path == "/me/library/playlists": + return AppleJsonResponse(200, {"data": []}, {}) + if query["offset"] == "0": + return AppleJsonResponse( + 200, + { + "data": [ + {"id": "library-song-1", "type": "library-songs", "attributes": {"name": "One"}}, + {"type": "library-songs", "attributes": {"name": "Unavailable"}}, + ], + "next": "https://api.music.apple.com/v1/me/library/playlists/playlist-1/tracks?offset=2", + }, + {}, + ) + return AppleJsonResponse( + 200, + {"data": [{"id": "catalog-song-2", "type": "songs", "attributes": {"name": "Two"}}]}, + {}, + ) + + +class AppleMusicAdapterTests(unittest.TestCase): + def playlist(self) -> ProviderObjectRef: + return ProviderObjectRef("apple_music", "library-playlists", "playlist-1", "apple-connection-1") + + def test_library_playlist_pages_preserve_library_and_catalog_ids(self) -> None: + client = FakeAppleClient() + adapter = AppleMusicAdapter(client, lambda connection_id: ("developer-token", "user-token"), page_size=2) + + pages = adapter.read_playlist_pages("apple-connection-1", self.playlist()) + + self.assertEqual(len(pages), 2) + entries = [entry for page in pages for entry in page.entries] + self.assertEqual([entry.position for entry in entries], [0, 1, 2]) + self.assertEqual(entries[0].media_kind, MediaKind.TRACK) + self.assertEqual(entries[0].track.object_type, "library-songs") + self.assertEqual(entries[0].track.object_id, "library-song-1") + self.assertFalse(entries[1].available) + self.assertEqual(entries[1].track.object_id, "unavailable:1") + self.assertEqual(entries[2].track.object_type, "songs") + self.assertEqual(client.calls[0][2:4], ("developer-token", "user-token")) + + def test_capability_probe_uses_user_library_endpoint(self) -> None: + client = FakeAppleClient() + adapter = AppleMusicAdapter(client, lambda connection_id: ("developer-token", "user-token")) + + capabilities = adapter.capabilities("apple-connection-1") + + self.assertTrue(capabilities.supports(Capability.READ_PLAYLISTS)) + self.assertEqual(client.calls[0][1], "/me/library/playlists") + + def test_missing_developer_or_user_token_fails_closed(self) -> None: + adapter = AppleMusicAdapter(FakeAppleClient(), lambda connection_id: ("", "user-token")) + + with self.assertRaises(ProviderApiError) as context: + adapter.read_playlist_pages("apple-connection-1", self.playlist()) + + self.assertEqual(context.exception.category, ProviderErrorCategory.AUTHENTICATION_REQUIRED) + + +if __name__ == "__main__": + unittest.main() From 1966a72fd1882ce4df3f867a0033b01b61ec1ef7 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:46:22 +0200 Subject: [PATCH 038/167] fix: fail durable operations on handler exceptions --- src/symphonia/application/operation_runner.py | 19 +++++++++++++-- tests/test_operation_runner.py | 24 +++++++++++++++++++ 2 files changed, 41 insertions(+), 2 deletions(-) diff --git a/src/symphonia/application/operation_runner.py b/src/symphonia/application/operation_runner.py index fd714ce..89e246b 100644 --- a/src/symphonia/application/operation_runner.py +++ b/src/symphonia/application/operation_runner.py @@ -42,5 +42,20 @@ def run_once( now=now, state="failed", ) - return handler(operation, worker_id, now) - + try: + return handler(operation, worker_id, now) + except Exception as error: + # A handler must never strand a claimed operation in ``running``. + # Persist only a stable exception class marker: provider details + # may contain credentials, request URLs, or other sensitive data. + failure_code = f"handler_exception:{type(error).__name__}" + latest = self.operations.get(operation.operation_id) + if latest.state == "running" and latest.worker_id == worker_id: + return self.operations.checkpoint( + operation.operation_id, + worker_id=worker_id, + checkpoint={**latest.checkpoint, "failure_code": failure_code}, + now=now, + state="failed", + ) + raise diff --git a/tests/test_operation_runner.py b/tests/test_operation_runner.py index 8494134..1b50b1b 100644 --- a/tests/test_operation_runner.py +++ b/tests/test_operation_runner.py @@ -58,6 +58,30 @@ def test_missing_handler_fails_before_external_work(self) -> None: ["created", "claimed", "checkpointed"], ) + def test_handler_exception_is_terminal_and_does_not_persist_exception_detail(self) -> None: + operation = self.repository.create( + operation_type="fixture", + idempotency_key="fixture-exception", + payload={}, + now=NOW, + ) + + def handler(claimed, worker_id, now): + raise RuntimeError("provider token=secret-token request=https://example.invalid") + + result = OperationRunner(self.repository, {"fixture": handler}).run_once( + worker_id="worker-a", now=NOW + ) + + self.assertEqual(result.operation_id, operation.operation_id) + self.assertEqual(result.state, "failed") + self.assertEqual(result.checkpoint["failure_code"], "handler_exception:RuntimeError") + self.assertNotIn("secret-token", str(result.checkpoint)) + self.assertEqual( + [event.event_type for event in self.repository.events(operation.operation_id)], + ["created", "claimed", "checkpointed"], + ) + if __name__ == "__main__": unittest.main() From 18639a405dc57e6d861b438c5d3d68011fa79347 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:47:35 +0200 Subject: [PATCH 039/167] feat: add cooperative durable operation worker --- src/symphonia/application/__init__.py | 2 + src/symphonia/application/operation_worker.py | 71 +++++++++++++++++++ tests/test_operation_worker.py | 66 +++++++++++++++++ 3 files changed, 139 insertions(+) create mode 100644 src/symphonia/application/operation_worker.py create mode 100644 tests/test_operation_worker.py diff --git a/src/symphonia/application/__init__.py b/src/symphonia/application/__init__.py index 3200b33..2ebaaab 100644 --- a/src/symphonia/application/__init__.py +++ b/src/symphonia/application/__init__.py @@ -7,6 +7,7 @@ from .authorization import AuthorizationService, AuthorizationStart from .provider_connections import ProviderConnectionService from .operation_runner import OperationRunner +from .operation_worker import OperationWorker __all__ = [ "CopyExecutionService", @@ -19,4 +20,5 @@ "LibraryImportService", "ProviderConnectionService", "OperationRunner", + "OperationWorker", ] diff --git a/src/symphonia/application/operation_worker.py b/src/symphonia/application/operation_worker.py new file mode 100644 index 0000000..0b67066 --- /dev/null +++ b/src/symphonia/application/operation_worker.py @@ -0,0 +1,71 @@ +"""Cooperative single-process worker for durable operations.""" + +from __future__ import annotations + +from collections.abc import Callable +from datetime import datetime, timezone +import threading +import time + +from symphonia.infrastructure.sqlite_operations import OperationRecord + +from .operation_runner import OperationRunner + + +Clock = Callable[[], datetime] + + +class OperationWorker: + """Poll an :class:`OperationRunner` without making memory authoritative. + + The worker is deliberately small and single-process. Every iteration + claims through the durable repository, so a stop or crash only delays work + until the lease expires; it cannot turn an in-memory queue into the source + of truth. + """ + + def __init__( + self, + runner: OperationRunner, + *, + worker_id: str, + clock: Clock | None = None, + poll_interval_seconds: float = 1.0, + lease_seconds: int = 30, + ) -> None: + if not worker_id.strip(): + raise ValueError("worker_id must not be empty") + if poll_interval_seconds <= 0: + raise ValueError("poll_interval_seconds must be positive") + if lease_seconds <= 0: + raise ValueError("lease_seconds must be positive") + self._runner = runner + self._worker_id = worker_id + self._clock = clock or (lambda: datetime.now(timezone.utc)) + self._poll_interval_seconds = poll_interval_seconds + self._lease_seconds = lease_seconds + + @property + def worker_id(self) -> str: + return self._worker_id + + def run_once(self, *, operation_type: str | None = None) -> OperationRecord | None: + """Claim and dispatch one eligible operation, if one exists.""" + + return self._runner.run_once( + worker_id=self._worker_id, + now=self._clock(), + lease_seconds=self._lease_seconds, + operation_type=operation_type, + ) + + def run_forever(self, stop_event: threading.Event) -> None: + """Poll until requested to stop; waiting remains interruptible.""" + + while not stop_event.is_set(): + operation = self.run_once() + if operation is None: + stop_event.wait(self._poll_interval_seconds) + + +__all__ = ["OperationWorker"] diff --git a/tests/test_operation_worker.py b/tests/test_operation_worker.py new file mode 100644 index 0000000..782ff0d --- /dev/null +++ b/tests/test_operation_worker.py @@ -0,0 +1,66 @@ +from __future__ import annotations + +from datetime import datetime, timezone +import threading +import unittest + +from symphonia.application import OperationRunner, OperationWorker +from symphonia.infrastructure import OperationRepository + + +NOW = datetime(2026, 9, 20, 12, 0, tzinfo=timezone.utc) + + +class OperationWorkerTests(unittest.TestCase): + def setUp(self) -> None: + self.repository = OperationRepository() + self.operation = self.repository.create( + operation_type="fixture", + idempotency_key="worker-1", + payload={}, + now=NOW, + ) + + def tearDown(self) -> None: + self.repository.close() + + def test_run_once_uses_injected_clock_and_worker_identity(self) -> None: + seen: list[tuple[str, datetime]] = [] + + def handler(operation, worker_id, now): + seen.append((worker_id, now)) + return self.repository.checkpoint( + operation.operation_id, + worker_id=worker_id, + checkpoint={"handled": True}, + now=now, + state="succeeded", + ) + + runner = OperationRunner(self.repository, {"fixture": handler}) + worker = OperationWorker(runner, worker_id="app-worker-1", clock=lambda: NOW) + + result = worker.run_once() + + self.assertEqual(result.operation_id, self.operation.operation_id) + self.assertEqual(result.state, "succeeded") + self.assertEqual(seen, [("app-worker-1", NOW)]) + self.assertIsNone(worker.run_once()) + + def test_run_forever_waits_interruptibly_when_queue_is_empty(self) -> None: + runner = OperationRunner(self.repository, {}) + worker = OperationWorker(runner, worker_id="app-worker-1", clock=lambda: NOW, poll_interval_seconds=0.25) + class StopAfterWait(threading.Event): + def wait(self, timeout=None): + self.set() + return True + + stop_event = StopAfterWait() + + worker.run_forever(stop_event) + + self.assertTrue(stop_event.is_set()) + + +if __name__ == "__main__": + unittest.main() From f160aaa6c0aa2e161d333cb8bff30c782e8b3ccd Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:48:42 +0200 Subject: [PATCH 040/167] fix: preserve provider object types in snapshots --- docs/development/implementation-baseline.md | 1 + src/symphonia/domain/models.py | 3 +++ .../infrastructure/sqlite_library.py | 18 +++++++++++--- src/symphonia/providers/importing.py | 1 + tests/test_sqlite_library.py | 24 +++++++++++++++++++ 5 files changed, 44 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 7f60573..d690c52 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -35,6 +35,7 @@ The first implementation increment is intentionally narrower than any provider o - An application workflow that persists plans, requires digest acceptance, and enqueues only accepted plans as durable operations. - A copy executor that creates target playlists through a semantic writer port, checkpoints each occurrence, and reconciles unknown writes before retry. - SQLite persistence for complete playlist projections that keeps incomplete imports from replacing the last complete snapshot and isolates external IDs by namespace. +- Playlist projections preserve provider object types as part of external identity, including a forward-compatible migration for older rows. - Idempotent snapshot publication that rejects reused IDs with different content and never rolls back a newer current pointer. - An application import use case that reports partial results without advancing freshness or replacing the last complete projection. diff --git a/src/symphonia/domain/models.py b/src/symphonia/domain/models.py index 1925f3b..3ef26eb 100644 --- a/src/symphonia/domain/models.py +++ b/src/symphonia/domain/models.py @@ -52,6 +52,7 @@ class SourcePlaylistEntry: target_track_id: str | None = None evidence: tuple[str, ...] = () reason: str | None = None + provider_track_object_type: str = "track" def __post_init__(self) -> None: if not self.occurrence_id.strip(): @@ -60,6 +61,8 @@ def __post_init__(self) -> None: raise ValueError("position must be non-negative") if not self.provider_track_id.strip(): raise ValueError("provider_track_id must not be empty") + if not self.provider_track_object_type.strip(): + raise ValueError("provider_track_object_type must not be empty") if self.classification is EntryClassification.READY and not self.target_track_id: raise ValueError("ready entries require a target_track_id") if self.target_track_id is not None and not self.target_track_id.strip(): diff --git a/src/symphonia/infrastructure/sqlite_library.py b/src/symphonia/infrastructure/sqlite_library.py index de8922f..4646b99 100644 --- a/src/symphonia/infrastructure/sqlite_library.py +++ b/src/symphonia/infrastructure/sqlite_library.py @@ -68,6 +68,7 @@ def _migrate(self) -> None: occurrence_id TEXT NOT NULL, position INTEGER NOT NULL, provider_track_id TEXT NOT NULL, + provider_track_object_type TEXT NOT NULL DEFAULT 'track', provider_track_namespace TEXT NOT NULL, media_kind TEXT NOT NULL, available INTEGER NOT NULL, @@ -83,6 +84,14 @@ def _migrate(self) -> None: ); """ ) + columns = { + row[1] + for row in self._connection.execute("PRAGMA table_info(playlist_snapshot_entries)").fetchall() + } + if "provider_track_object_type" not in columns: + self._connection.execute( + "ALTER TABLE playlist_snapshot_entries ADD COLUMN provider_track_object_type TEXT NOT NULL DEFAULT 'track'" + ) def publish( self, @@ -123,8 +132,8 @@ def publish( """ INSERT INTO playlist_snapshot_entries ( snapshot_id, occurrence_id, position, provider_track_id, - provider_track_namespace, media_kind, available - ) VALUES (?, ?, ?, ?, ?, ?, ?) + provider_track_object_type, provider_track_namespace, media_kind, available + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) """, [ ( @@ -132,6 +141,7 @@ def publish( entry.occurrence_id, entry.position, entry.track.object_id, + entry.track.object_type, entry.track.namespace, entry.media_kind.value, int(entry.available), @@ -165,7 +175,7 @@ def _matches_result(self, snapshot_row: sqlite3.Row, result: CollectionImportRes rows = self._connection.execute( """ SELECT occurrence_id, position, provider_track_id, - provider_track_namespace, media_kind, available + provider_track_object_type, provider_track_namespace, media_kind, available FROM playlist_snapshot_entries WHERE snapshot_id = ? ORDER BY position @@ -179,6 +189,7 @@ def _matches_result(self, snapshot_row: sqlite3.Row, result: CollectionImportRes row["occurrence_id"] == entry.occurrence_id and row["position"] == entry.position and row["provider_track_id"] == entry.track.object_id + and row["provider_track_object_type"] == entry.track.object_type and row["provider_track_namespace"] == entry.track.namespace and row["media_kind"] == entry.media_kind.value and bool(row["available"]) == entry.available @@ -231,6 +242,7 @@ def _entries(self, snapshot_id: str, provider: str) -> tuple[SourcePlaylistEntry occurrence_id=row["occurrence_id"], position=row["position"], provider_track_id=row["provider_track_id"], + provider_track_object_type=row["provider_track_object_type"], classification=EntryClassification.UNMATCHED if row["available"] else EntryClassification.UNAVAILABLE, reason=None if row["available"] else "provider reported item unavailable", ) diff --git a/src/symphonia/providers/importing.py b/src/symphonia/providers/importing.py index 292aa02..4e5eadf 100644 --- a/src/symphonia/providers/importing.py +++ b/src/symphonia/providers/importing.py @@ -107,6 +107,7 @@ def to_playlist_snapshot(result: CollectionImportResult, snapshot_id: str) -> Pl provider_track_id=entry.track.object_id, classification=EntryClassification.UNMATCHED if entry.available else EntryClassification.UNAVAILABLE, reason=None if entry.available else "provider reported item unavailable", + provider_track_object_type=entry.track.object_type, ) for entry in result.entries ) diff --git a/tests/test_sqlite_library.py b/tests/test_sqlite_library.py index 8dedc8d..520be3d 100644 --- a/tests/test_sqlite_library.py +++ b/tests/test_sqlite_library.py @@ -90,6 +90,30 @@ def test_reusing_snapshot_id_for_different_content_is_rejected(self) -> None: with self.assertRaises(SnapshotConflictError): self.repository.publish(changed, snapshot_id="snapshot-1", published_at=NOW) + def test_reusing_snapshot_id_with_different_provider_object_type_is_rejected(self) -> None: + self.repository.publish(collect_playlist_pages([page()]), snapshot_id="snapshot-1", published_at=NOW) + changed = collect_playlist_pages( + [ + ProviderPlaylistPage( + ProviderObjectRef("spotify", "playlist", "playlist-1", "connection-1"), + ( + ProviderPlaylistEntry( + "occ-1", + 0, + ProviderObjectRef("spotify", "episode", "track-1", "connection-1"), + MediaKind.PODCAST, + ), + ), + None, + None, + True, + revision="rev-1", + ) + ] + ) + with self.assertRaises(SnapshotConflictError): + self.repository.publish(changed, snapshot_id="snapshot-1", published_at=NOW) + def test_same_external_playlist_id_isolated_by_namespace(self) -> None: first = collect_playlist_pages([page(namespace="connection-1")]) second = collect_playlist_pages([page(namespace="connection-2")]) From 8a03add2b09cc165c70e7ac607a1eb59aa78387b Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:50:57 +0200 Subject: [PATCH 041/167] feat: run playlist imports as durable operations --- src/symphonia/application/__init__.py | 2 + .../application/library_import_execution.py | 197 ++++++++++++++++++ .../infrastructure/sqlite_operations.py | 42 ++++ tests/test_library_import_execution.py | 99 +++++++++ tests/test_sqlite_operations.py | 25 +++ 5 files changed, 365 insertions(+) create mode 100644 src/symphonia/application/library_import_execution.py create mode 100644 tests/test_library_import_execution.py diff --git a/src/symphonia/application/__init__.py b/src/symphonia/application/__init__.py index 2ebaaab..f797773 100644 --- a/src/symphonia/application/__init__.py +++ b/src/symphonia/application/__init__.py @@ -4,6 +4,7 @@ from .copy_workflow import CapabilityUnavailableError, CopyWorkflowService from .copy_execution import CopyExecutionService from .library_import import ImportPublication, LibraryImportService +from .library_import_execution import LibraryImportExecutionService from .authorization import AuthorizationService, AuthorizationStart from .provider_connections import ProviderConnectionService from .operation_runner import OperationRunner @@ -18,6 +19,7 @@ "CopyWorkflowService", "ImportPublication", "LibraryImportService", + "LibraryImportExecutionService", "ProviderConnectionService", "OperationRunner", "OperationWorker", diff --git a/src/symphonia/application/library_import_execution.py b/src/symphonia/application/library_import_execution.py new file mode 100644 index 0000000..8020728 --- /dev/null +++ b/src/symphonia/application/library_import_execution.py @@ -0,0 +1,197 @@ +"""Durable operation orchestration for playlist imports.""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass +from datetime import datetime, timedelta, timezone +from typing import Any + +from symphonia.infrastructure.sqlite_operations import OperationRecord, OperationRepository +from symphonia.providers.contracts import ProviderAdapter, ProviderObjectRef +from symphonia.providers.errors import ProviderApiError, ProviderErrorCategory + +from .library_import import ImportPublication, LibraryImportService + + +@dataclass(frozen=True, slots=True) +class LibraryImportExecutionService: + """Make import publication resumable and visible as a durable operation.""" + + imports: LibraryImportService + operations: OperationRepository + retry_delay_seconds: int = 60 + + def __post_init__(self) -> None: + if self.retry_delay_seconds <= 0: + raise ValueError("retry_delay_seconds must be positive") + + def enqueue_playlist( + self, + adapter: ProviderAdapter, + *, + connection_id: str, + playlist: ProviderObjectRef, + snapshot_id: str, + observed_at: datetime, + now: datetime, + ) -> OperationRecord: + """Persist import intent before the first provider read.""" + + if adapter.manifest.provider != playlist.provider: + raise ValueError("adapter provider does not match playlist provider") + if observed_at.tzinfo is None: + raise ValueError("observed_at must be timezone-aware") + payload = { + "provider": playlist.provider, + "connection_id": connection_id, + "playlist": { + "provider": playlist.provider, + "object_type": playlist.object_type, + "object_id": playlist.object_id, + "namespace": playlist.namespace, + }, + "snapshot_id": snapshot_id, + "observed_at": observed_at.astimezone(timezone.utc).isoformat(timespec="microseconds"), + } + idempotency_key = ( + f"import:{connection_id}:{playlist.object_type}:{playlist.object_id}:{snapshot_id}" + ) + return self.operations.create( + operation_type="import_playlist", + idempotency_key=idempotency_key, + payload=payload, + now=now, + ) + + def execute_claimed( + self, + operation: OperationRecord, + *, + adapter: ProviderAdapter, + worker_id: str, + now: datetime, + lease_seconds: int = 30, + ) -> OperationRecord: + """Run one claimed import and turn provider failures into safe states.""" + + if operation.state != "running" or operation.worker_id != worker_id: + raise ValueError("import operation must be running under the supplied worker") + checkpoint = dict(operation.checkpoint) + if operation.cancel_requested: + return self._checkpoint(operation, worker_id, checkpoint, now, "cancelled") + try: + connection_id, playlist, snapshot_id, observed_at = self._payload(operation.payload) + if adapter.manifest.provider != playlist.provider: + raise ValueError("adapter provider does not match operation playlist") + self.operations.renew_lease( + operation.operation_id, + worker_id=worker_id, + now=now, + lease_seconds=lease_seconds, + ) + publication = self.imports.import_playlist( + adapter, + connection_id=connection_id, + playlist=playlist, + snapshot_id=snapshot_id, + observed_at=observed_at, + ) + except ProviderApiError as error: + checkpoint["failure_code"] = error.category.value + if error.category in { + ProviderErrorCategory.AUTHENTICATION_REQUIRED, + ProviderErrorCategory.AUTHORIZATION_REVOKED, + ProviderErrorCategory.PERMISSION_DENIED, + }: + return self._checkpoint(operation, worker_id, checkpoint, now, "waiting_user") + if error.category is ProviderErrorCategory.RATE_LIMITED: + retry_at = error.retry_at if error.retry_at and error.retry_at > now else now + timedelta(seconds=self.retry_delay_seconds) + return self.operations.schedule_rate_limit( + operation.operation_id, + worker_id=worker_id, + next_run_at=retry_at, + checkpoint=checkpoint, + now=now, + ) + if error.category in { + ProviderErrorCategory.PROVIDER_UNAVAILABLE, + ProviderErrorCategory.TIMEOUT, + ProviderErrorCategory.NETWORK_ERROR, + }: + return self.operations.schedule_retry( + operation.operation_id, + worker_id=worker_id, + next_run_at=now + timedelta(seconds=self.retry_delay_seconds), + checkpoint=checkpoint, + now=now, + ) + return self._checkpoint(operation, worker_id, checkpoint, now, "failed") + except Exception as error: + checkpoint["failure_code"] = f"import_exception:{type(error).__name__}" + return self._checkpoint(operation, worker_id, checkpoint, now, "failed") + + return self._complete_publication(operation, worker_id, publication, checkpoint, now) + + def _complete_publication( + self, + operation: OperationRecord, + worker_id: str, + publication: ImportPublication, + checkpoint: dict[str, Any], + now: datetime, + ) -> OperationRecord: + checkpoint.update( + { + "publication_state": publication.state, + "issues": [issue.value for issue in publication.issues], + "published_snapshot_id": None if publication.snapshot is None else publication.snapshot.snapshot_id, + "retained_snapshot_id": None + if publication.retained_current is None + else publication.retained_current.snapshot_id, + } + ) + return self._checkpoint(operation, worker_id, checkpoint, now, publication.state) + + def _checkpoint( + self, + operation: OperationRecord, + worker_id: str, + checkpoint: dict[str, Any], + now: datetime, + state: str, + ) -> OperationRecord: + return self.operations.checkpoint( + operation.operation_id, + worker_id=worker_id, + checkpoint=checkpoint, + now=now, + state=state, + ) + + @staticmethod + def _payload(payload: Mapping[str, Any]) -> tuple[str, ProviderObjectRef, str, datetime]: + connection_id = payload.get("connection_id") + snapshot_id = payload.get("snapshot_id") + observed_at = payload.get("observed_at") + raw_playlist = payload.get("playlist") + if not all(isinstance(value, str) and value.strip() for value in (connection_id, snapshot_id, observed_at)): + raise ValueError("import operation payload is missing required fields") + if not isinstance(raw_playlist, Mapping): + raise ValueError("import operation payload has no playlist reference") + try: + playlist = ProviderObjectRef( + str(raw_playlist["provider"]), + str(raw_playlist["object_type"]), + str(raw_playlist["object_id"]), + str(raw_playlist["namespace"]), + ) + parsed_observed_at = datetime.fromisoformat(observed_at) + except (KeyError, TypeError, ValueError) as error: + raise ValueError("import operation payload has an invalid playlist or timestamp") from error + if parsed_observed_at.tzinfo is None: + raise ValueError("import operation observed_at must be timezone-aware") + return connection_id, playlist, snapshot_id, parsed_observed_at.astimezone(timezone.utc) + + +__all__ = ["LibraryImportExecutionService"] diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 179c984..e543f78 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -576,6 +576,27 @@ def schedule_retry( raise OperationNotFound(operation_id) if row["state"] != "running" or row["worker_id"] != worker_id: raise LeaseConflict("worker does not own a running operation") + if row["cancel_requested"]: + self._connection.execute( + """ + UPDATE operations + SET state = 'cancelled', checkpoint_json = ?, next_run_at = NULL, + worker_id = NULL, lease_expires_at = NULL, + cancel_requested = 0, updated_at = ? + WHERE operation_id = ? + """, + (checkpoint_json, now_text, operation_id), + ) + self._append_event( + operation_id=operation_id, + event_type="cancelled", + state="cancelled", + worker_id=None, + payload=self._checkpoint_summary(checkpoint), + created_at=now_text, + ) + self._connection.execute("COMMIT") + return self.get(operation_id) self._connection.execute( """ UPDATE operations @@ -622,6 +643,27 @@ def schedule_rate_limit( raise OperationNotFound(operation_id) if row["state"] != "running" or row["worker_id"] != worker_id: raise LeaseConflict("worker does not own a running operation") + if row["cancel_requested"]: + self._connection.execute( + """ + UPDATE operations + SET state = 'cancelled', checkpoint_json = ?, next_run_at = NULL, + worker_id = NULL, lease_expires_at = NULL, + cancel_requested = 0, updated_at = ? + WHERE operation_id = ? + """, + (checkpoint_json, now_text, operation_id), + ) + self._append_event( + operation_id=operation_id, + event_type="cancelled", + state="cancelled", + worker_id=None, + payload=self._checkpoint_summary(checkpoint), + created_at=now_text, + ) + self._connection.execute("COMMIT") + return self.get(operation_id) self._connection.execute( """ UPDATE operations diff --git a/tests/test_library_import_execution.py b/tests/test_library_import_execution.py new file mode 100644 index 0000000..0080f97 --- /dev/null +++ b/tests/test_library_import_execution.py @@ -0,0 +1,99 @@ +from __future__ import annotations + +from datetime import datetime, timezone +import unittest + +from symphonia.application import LibraryImportExecutionService, LibraryImportService, OperationRunner +from symphonia.infrastructure import OperationRepository, PlaylistProjectionRepository +from symphonia.providers import ( + AccessBasis, + ProviderApiError, + ProviderErrorCategory, + ProviderManifest, + ProviderObjectRef, + ProviderPlaylistEntry, + ProviderPlaylistPage, + MediaKind, +) + + +NOW = datetime(2026, 9, 20, 12, 0, tzinfo=timezone.utc) + + +class FakeAdapter: + manifest = ProviderManifest("spotify", "Spotify", AccessBasis.OFFICIAL, "beta", "read") + + def __init__(self, *, error: ProviderApiError | None = None, complete: bool = True) -> None: + self.error = error + self.complete = complete + + def capabilities(self, connection_id: str): + raise AssertionError("not used") + + def read_playlist_pages(self, connection_id: str, playlist: ProviderObjectRef, cursor: str | None = None): + if self.error is not None: + raise self.error + entry = ProviderPlaylistEntry( + "occ-1", + 0, + ProviderObjectRef("spotify", "track", "track-1", playlist.namespace), + MediaKind.TRACK, + ) + return (ProviderPlaylistPage(playlist, (entry,), cursor, None, self.complete),) + + +class LibraryImportExecutionTests(unittest.TestCase): + def setUp(self) -> None: + self.operations = OperationRepository() + self.projections = PlaylistProjectionRepository() + self.service = LibraryImportExecutionService(LibraryImportService(self.projections), self.operations) + self.playlist = ProviderObjectRef("spotify", "playlist", "playlist-1", "connection-1") + + def tearDown(self) -> None: + self.projections.close() + self.operations.close() + + def enqueue(self, adapter: FakeAdapter) -> object: + operation = self.service.enqueue_playlist( + adapter, + connection_id="connection-1", + playlist=self.playlist, + snapshot_id="snapshot-1", + observed_at=NOW, + now=NOW, + ) + runner = OperationRunner( + self.operations, + { + "import_playlist": lambda claimed, worker_id, now: self.service.execute_claimed( + claimed, + adapter=adapter, + worker_id=worker_id, + now=now, + ) + }, + ) + return runner.run_once(worker_id="worker-a", now=NOW) + + def test_import_is_published_as_a_succeeded_operation(self) -> None: + result = self.enqueue(FakeAdapter()) + self.assertEqual(result.state, "succeeded") + self.assertEqual(result.checkpoint["published_snapshot_id"], "snapshot-1") + + def test_provider_auth_failure_waits_for_user_without_leaking_detail(self) -> None: + result = self.enqueue( + FakeAdapter(error=ProviderApiError(ProviderErrorCategory.AUTHENTICATION_REQUIRED, "token=secret")) + ) + self.assertEqual(result.state, "waiting_user") + self.assertEqual(result.checkpoint["failure_code"], "authentication_required") + self.assertNotIn("secret", str(result.checkpoint)) + + def test_incomplete_import_is_partial_and_retains_no_new_snapshot(self) -> None: + result = self.enqueue(FakeAdapter(complete=False)) + self.assertEqual(result.state, "partial") + self.assertIsNone(result.checkpoint["published_snapshot_id"]) + self.assertEqual(result.checkpoint["publication_state"], "partial") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 180fe29..68476f4 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -302,6 +302,31 @@ def test_rate_limit_wait_is_durable_and_claimable_after_deadline(self) -> None: ["created", "claimed", "rate_limit_wait", "claimed"], ) + def test_cancellation_race_does_not_schedule_retry(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="copy-cancel-race", + payload={}, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) + self.repository.cancel(operation.operation_id, now=self.now + timedelta(seconds=1)) + + cancelled = self.repository.schedule_retry( + operation.operation_id, + worker_id="worker-a", + next_run_at=self.now + timedelta(minutes=1), + checkpoint={"failure_code": "timeout"}, + now=self.now + timedelta(seconds=2), + ) + + self.assertEqual(cancelled.state, "cancelled") + self.assertIsNone(cancelled.next_run_at) + self.assertEqual( + [event.event_type for event in self.repository.events(operation.operation_id)], + ["created", "claimed", "cancellation_requested", "cancelled"], + ) + def test_healthy_worker_can_renew_lease(self) -> None: operation = self.repository.create( operation_type="copy", From c98250ecc72d5261185415f3069c5e8e7bb40609 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:52:07 +0200 Subject: [PATCH 042/167] feat: add lossless identity normalization --- docs/development/implementation-baseline.md | 1 + src/symphonia/identity/__init__.py | 13 ++- src/symphonia/identity/normalization.py | 116 ++++++++++++++++++++ tests/test_identity_resolution.py | 25 ++++- 4 files changed, 153 insertions(+), 2 deletions(-) create mode 100644 src/symphonia/identity/normalization.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index d690c52..4c2307b 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -32,6 +32,7 @@ The first implementation increment is intentionally narrower than any provider o - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. +- Lossless Unicode-safe identity normalization with explicit version-token and ISRC derived fields; no automatic matching thresholds are assumed. - An application workflow that persists plans, requires digest acceptance, and enqueues only accepted plans as durable operations. - A copy executor that creates target playlists through a semantic writer port, checkpoints each occurrence, and reconciles unknown writes before retry. - SQLite persistence for complete playlist projections that keeps incomplete imports from replacing the last complete snapshot and isolates external IDs by namespace. diff --git a/src/symphonia/identity/__init__.py b/src/symphonia/identity/__init__.py index f296a0d..7637059 100644 --- a/src/symphonia/identity/__init__.py +++ b/src/symphonia/identity/__init__.py @@ -8,6 +8,13 @@ ManualDecisionAction, ResolutionState, ) +from .normalization import ( + NormalizedRecordingMetadata, + normalize_isrc, + normalize_recording_metadata, + normalize_text, + version_tokens, +) __all__ = [ "AssessmentClass", @@ -16,5 +23,9 @@ "ManualDecision", "ManualDecisionAction", "ResolutionState", + "NormalizedRecordingMetadata", + "normalize_isrc", + "normalize_recording_metadata", + "normalize_text", + "version_tokens", ] - diff --git a/src/symphonia/identity/normalization.py b/src/symphonia/identity/normalization.py new file mode 100644 index 0000000..7b888fb --- /dev/null +++ b/src/symphonia/identity/normalization.py @@ -0,0 +1,116 @@ +"""Deterministic, lossless comparison-field normalization. + +Normalization is intentionally not matching. It produces reviewable derived +fields while retaining every original value and does not decide whether two +provider representations are the same recording. +""" + +from __future__ import annotations + +from dataclasses import dataclass +import re +import unicodedata + + +_WHITESPACE = re.compile(r"\s+") +_PUNCTUATION = re.compile(r"[^\w]+", re.UNICODE) +_ISRC = re.compile(r"^[A-Z]{2}[A-Z0-9]{3}\d{7}$") +_VERSION_PATTERNS: tuple[tuple[str, re.Pattern[str]], ...] = ( + ("live", re.compile(r"\blive\b", re.IGNORECASE)), + ("acoustic", re.compile(r"\bacoustic\b", re.IGNORECASE)), + ("remix", re.compile(r"\bremix(?:ed)?\b", re.IGNORECASE)), + ("remaster", re.compile(r"\b(?:digital\s+)?remaster(?:ed)?\b", re.IGNORECASE)), + ("edit", re.compile(r"\b(?:radio|single|extended)\s+edit\b|\bedit\b", re.IGNORECASE)), + ("demo", re.compile(r"\bdemo\b", re.IGNORECASE)), + ("instrumental", re.compile(r"\binstrumental\b", re.IGNORECASE)), + ("karaoke", re.compile(r"\bk karaoke\b|\bkaraoke\b", re.IGNORECASE)), + ("clean", re.compile(r"\bclean\b", re.IGNORECASE)), + ("explicit", re.compile(r"\bexplicit\b", re.IGNORECASE)), +) + + +@dataclass(frozen=True, slots=True) +class NormalizedRecordingMetadata: + """Original and derived fields used by a future resolver policy.""" + + original_title: str + normalized_title: str + original_artists: tuple[str, ...] + normalized_artists: tuple[str, ...] + version_tokens: tuple[str, ...] + original_isrc: str | None = None + normalized_isrc: str | None = None + duration_ms: int | None = None + + def __post_init__(self) -> None: + if not self.original_title.strip(): + raise ValueError("original_title must not be empty") + if not self.normalized_title.strip(): + raise ValueError("normalized_title must not be empty") + if len(self.original_artists) != len(self.normalized_artists): + raise ValueError("original and normalized artist fields must have equal length") + if self.duration_ms is not None and self.duration_ms < 0: + raise ValueError("duration_ms must not be negative") + if self.normalized_isrc is not None and not _ISRC.fullmatch(self.normalized_isrc): + raise ValueError("normalized_isrc is not a valid ISRC") + + +def normalize_text(value: str) -> str: + """Return a Unicode-safe comparison form without changing stored input.""" + + if not isinstance(value, str): + raise TypeError("text to normalize must be a string") + folded = unicodedata.normalize("NFKC", value).casefold() + without_punctuation = _PUNCTUATION.sub(" ", folded) + return _WHITESPACE.sub(" ", without_punctuation).strip() + + +def normalize_isrc(value: str | None) -> str | None: + """Normalize a candidate ISRC, returning ``None`` for malformed input.""" + + if value is None: + return None + compact = re.sub(r"[\s-]+", "", value).upper() + return compact if _ISRC.fullmatch(compact) else None + + +def version_tokens(title: str) -> tuple[str, ...]: + """Extract known version markers in stable policy order.""" + + if not isinstance(title, str): + raise TypeError("title must be a string") + return tuple(name for name, pattern in _VERSION_PATTERNS if pattern.search(title)) + + +def normalize_recording_metadata( + *, + title: str, + artists: tuple[str, ...] | list[str], + isrc: str | None = None, + duration_ms: int | None = None, +) -> NormalizedRecordingMetadata: + """Build comparison fields while keeping provider values intact.""" + + original_artists = tuple(artists) + if not original_artists or any(not artist.strip() for artist in original_artists): + raise ValueError("artists must contain at least one nonblank value") + normalized_isrc = normalize_isrc(isrc) + return NormalizedRecordingMetadata( + original_title=title, + normalized_title=normalize_text(title), + original_artists=original_artists, + normalized_artists=tuple(normalize_text(artist) for artist in original_artists), + version_tokens=version_tokens(title), + original_isrc=isrc, + normalized_isrc=normalized_isrc, + duration_ms=duration_ms, + ) + + +__all__ = [ + "NormalizedRecordingMetadata", + "normalize_isrc", + "normalize_recording_metadata", + "normalize_text", + "version_tokens", +] diff --git a/tests/test_identity_resolution.py b/tests/test_identity_resolution.py index 3abb85d..a9c1358 100644 --- a/tests/test_identity_resolution.py +++ b/tests/test_identity_resolution.py @@ -9,11 +9,35 @@ EvidenceKind, ManualDecision, ManualDecisionAction, + normalize_isrc, + normalize_recording_metadata, + normalize_text, + version_tokens, ) from symphonia.infrastructure import ResolutionDecisionRepository class IdentityResolutionTests(unittest.TestCase): + def test_normalization_is_unicode_safe_and_lossless(self) -> None: + metadata = normalize_recording_metadata( + title="Beyoncé — Halo (Live Edit)", + artists=("Beyoncé", " Jay-Z "), + isrc="us-r1a-99-01234", + ) + + self.assertEqual(metadata.original_title, "Beyoncé — Halo (Live Edit)") + self.assertEqual(metadata.normalized_title, "beyoncé halo live edit") + self.assertEqual(metadata.original_artists, ("Beyoncé", " Jay-Z ")) + self.assertEqual(metadata.normalized_artists, ("beyoncé", "jay z")) + self.assertEqual(metadata.version_tokens, ("live", "edit")) + self.assertEqual(metadata.normalized_isrc, "USR1A9901234") + + def test_isrc_validation_never_turns_malformed_input_into_identity(self) -> None: + self.assertEqual(normalize_isrc("US-R1A-99-01234"), "USR1A9901234") + self.assertIsNone(normalize_isrc("not-an-isrc")) + self.assertEqual(normalize_text(" Café\u00a0del\u00a0Mar "), "café del mar") + self.assertEqual(version_tokens("Song (Acoustic Remix)"), ("acoustic", "remix")) + def test_unmatched_assessment_has_no_candidate(self) -> None: from symphonia.identity.models import CandidateAssessment @@ -83,4 +107,3 @@ def test_manual_decisions_are_append_only_and_latest_is_explicit(self) -> None: if __name__ == "__main__": unittest.main() - From 20c8d4fdfd774a4959a565ea119a8e44c9a11467 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:52:30 +0200 Subject: [PATCH 043/167] docs: align implementation baseline with durable workers --- docs/development/implementation-baseline.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 4c2307b..072239f 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -18,7 +18,9 @@ The first implementation increment is intentionally narrower than any provider o - Application connection service for verified-account registration, capability probes, degraded health, and reauthorization-required classification. - Pure capability-layer intersection and requirement checks for adapter/connection/object/health constraints. - Durable operation runner that atomically claims eligible work and fails unwired operation types before side effects. +- Cooperative single-process operation worker with interruptible polling and injected clock support. - Copy executor can run as a claimed operation handler, preserving the same restart/checkpoint semantics under the runner. +- Playlist import executor persists intent before reads and reports succeeded, partial, waiting-user, retry, and rate-limit outcomes durably. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Application authorization boundary that generates one-use state without persisting the raw value. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. @@ -52,7 +54,7 @@ docker run --rm -p 8099:8099 -v symphonia-data:/data symphonia:dev The container exposes only the current health/readiness/version surface. A future App manifest must add Ingress, Supervisor metadata, supported architectures, backup declarations, and any direct callback policy only after the runtime SDD blockers are resolved. - Deterministic `unittest` coverage under `tests/`. -The package is an implementation foundation, not a claim that the corresponding capability SDDs are complete. Provider OAuth, provider API adapters, user authentication, Ingress, UI, a complete migration ledger beyond the current forward-compatible SQLite path, backup/restore, and operation handlers remain unimplemented and blocked by their SDD decisions. +The package is an implementation foundation, not a claim that the corresponding capability SDDs are complete. Provider OAuth, secret storage, user authentication, Ingress UI, a complete migration ledger beyond the current forward-compatible SQLite path, backup/restore, and production runtime handler wiring remain unimplemented and blocked by their SDD decisions. ## Local verification From fd4168a409a88639229f7b941be3fb0e74e74889 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:54:35 +0200 Subject: [PATCH 044/167] feat: add runtime resource composition --- docs/development/implementation-baseline.md | 1 + src/symphonia/runtime/__init__.py | 4 +- src/symphonia/runtime/resources.py | 72 +++++++++++++++++++++ tests/test_runtime_resources.py | 40 ++++++++++++ 4 files changed, 115 insertions(+), 2 deletions(-) create mode 100644 src/symphonia/runtime/resources.py create mode 100644 tests/test_runtime_resources.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 072239f..22a1414 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -30,6 +30,7 @@ The first implementation increment is intentionally narrower than any provider o - Explicitly scoped official YouTube Data API video-playlist reader; it is not represented as YouTube Music. - Experimental official Apple Music library-playlist reader with separate developer/user token inputs. - Experimental Home Assistant App metadata scaffold with Ingress-only management and `/data` persistence. +- Explicit runtime resource composition and reverse-order shutdown for all durable repositories. - Ingress-relative health/version routing with normalized, traversal-safe base paths. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. diff --git a/src/symphonia/runtime/__init__.py b/src/symphonia/runtime/__init__.py index 1b5256d..5ac53a5 100644 --- a/src/symphonia/runtime/__init__.py +++ b/src/symphonia/runtime/__init__.py @@ -1,6 +1,6 @@ """Minimal process runtime and health endpoints.""" from .http import create_server +from .resources import RuntimeResources -__all__ = ["create_server"] - +__all__ = ["RuntimeResources", "create_server"] diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py new file mode 100644 index 0000000..52a1d99 --- /dev/null +++ b/src/symphonia/runtime/resources.py @@ -0,0 +1,72 @@ +"""Composition root for the dependency-free local runtime.""" + +from __future__ import annotations + +from dataclasses import dataclass + +from symphonia.infrastructure import ( + AuthorizationAttemptRepository, + CopyPlanRepository, + OperationRepository, + PlaylistProjectionRepository, + ProviderConnectionRepository, + ResolutionDecisionRepository, +) + + +@dataclass(slots=True) +class RuntimeResources: + """Own every durable repository opened by one Symphonia process. + + Repositories intentionally remain separate adapters for now. They point + at the same SQLite path in a persistent runtime, while this owner gives + startup/shutdown one explicit lifecycle boundary and leaves room for a + future shared transaction adapter. + """ + + operations: OperationRepository + plans: CopyPlanRepository + connections: ProviderConnectionRepository + authorization: AuthorizationAttemptRepository + projections: PlaylistProjectionRepository + resolutions: ResolutionDecisionRepository + + @classmethod + def open(cls, database_path: str) -> "RuntimeResources": + if not database_path.strip(): + raise ValueError("database_path must not be empty") + opened: list[object] = [] + try: + operations = OperationRepository(database_path) + opened.append(operations) + plans = CopyPlanRepository(database_path) + opened.append(plans) + connections = ProviderConnectionRepository(database_path) + opened.append(connections) + authorization = AuthorizationAttemptRepository(database_path) + opened.append(authorization) + projections = PlaylistProjectionRepository(database_path) + opened.append(projections) + resolutions = ResolutionDecisionRepository(database_path) + opened.append(resolutions) + except Exception: + for repository in reversed(opened): + repository.close() # type: ignore[attr-defined] + raise + return cls(operations, plans, connections, authorization, projections, resolutions) + + def close(self) -> None: + """Close repositories in reverse dependency/startup order.""" + + for repository in ( + self.resolutions, + self.projections, + self.authorization, + self.connections, + self.plans, + self.operations, + ): + repository.close() + + +__all__ = ["RuntimeResources"] diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py new file mode 100644 index 0000000..a961e53 --- /dev/null +++ b/tests/test_runtime_resources.py @@ -0,0 +1,40 @@ +from __future__ import annotations + +from pathlib import Path +import tempfile +import unittest + +from symphonia.runtime import RuntimeResources + + +class RuntimeResourcesTests(unittest.TestCase): + def test_open_creates_all_durable_store_schemas_and_closes_them(self) -> None: + with tempfile.TemporaryDirectory() as directory: + path = str(Path(directory) / "symphonia.sqlite3") + resources = RuntimeResources.open(path) + try: + tables = { + row[0] + for row in resources.operations._connection.execute( + "SELECT name FROM sqlite_master WHERE type = 'table'" + ).fetchall() + } + self.assertIn("operations", tables) + self.assertIn("provider_connections", tables) + self.assertIn("copy_plans", tables) + self.assertIn("playlist_snapshots", tables) + self.assertIn("authorization_attempts", tables) + self.assertIn("resolution_decisions", tables) + finally: + resources.close() + + reopened = RuntimeResources.open(path) + reopened.close() + + def test_empty_database_path_is_rejected_before_opening_stores(self) -> None: + with self.assertRaises(ValueError): + RuntimeResources.open(" ") + + +if __name__ == "__main__": + unittest.main() From 6e0cd0fe05bba0aaca5d27e4d4426be10ce51d43 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:55:38 +0200 Subject: [PATCH 045/167] feat: wire runtime resources into HTTP startup --- src/symphonia/__main__.py | 2 +- src/symphonia/runtime/http.py | 28 ++++++++++++++++++++++++---- 2 files changed, 25 insertions(+), 5 deletions(-) diff --git a/src/symphonia/__main__.py b/src/symphonia/__main__.py index 700b632..68a4f41 100644 --- a/src/symphonia/__main__.py +++ b/src/symphonia/__main__.py @@ -30,7 +30,7 @@ def main() -> None: pass finally: server.server_close() - server.repository.close() + server.close_resources() if __name__ == "__main__": diff --git a/src/symphonia/runtime/http.py b/src/symphonia/runtime/http.py index 66edffa..6597420 100644 --- a/src/symphonia/runtime/http.py +++ b/src/symphonia/runtime/http.py @@ -9,17 +9,33 @@ from symphonia import __version__ from symphonia.infrastructure.sqlite_operations import OperationRepository +from .resources import RuntimeResources class SymphoniaHTTPServer(HTTPServer): allow_reuse_address = True - def __init__(self, address: tuple[str, int], repository: OperationRepository, ingress_path: str = "/") -> None: + def __init__( + self, + address: tuple[str, int], + repository: OperationRepository | None = None, + ingress_path: str = "/", + *, + resources: RuntimeResources | None = None, + ) -> None: + if repository is None and resources is None: + raise ValueError("repository or resources must be supplied") super().__init__(address, SymphoniaRequestHandler) - self.repository = repository + self.resources = resources + self.repository = resources.operations if resources is not None else repository self.service_version = __version__ self.ingress_path = _normalize_base_path(ingress_path) + def close_resources(self) -> None: + if self.resources is not None: + self.resources.close() + self.resources = None + class SymphoniaRequestHandler(BaseHTTPRequestHandler): """Only health/readiness/version are exposed until the API SDD is ready.""" @@ -57,8 +73,12 @@ def create_server( ) -> SymphoniaHTTPServer: """Create a server with an already-migrated durable operation store.""" - repository = OperationRepository(database_path) - return SymphoniaHTTPServer((host, port), repository, ingress_path) + resources = RuntimeResources.open(database_path) + try: + return SymphoniaHTTPServer((host, port), ingress_path=ingress_path, resources=resources) + except Exception: + resources.close() + raise def route_get( From 32be2792970536de5777b84a11889f0fe96784ec Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:56:05 +0200 Subject: [PATCH 046/167] docs: document current provider adapter boundaries --- README.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index eb46918..9dc4c21 100644 --- a/README.md +++ b/README.md @@ -8,10 +8,18 @@ Symphonia is a library-management and interoperability project, not a new music ## Project status -**Implementation foundation stage. Provider adapters and an experimental Home Assistant App metadata scaffold exist; no published App image, complete UI, or OAuth flow is available yet.** +**Implementation foundation stage. Provider adapters, durable import/copy execution, and an experimental Home Assistant App metadata scaffold exist; no published App image, complete UI, or OAuth flow is available yet.** The current work combines a reviewable source of truth with the first owner-approved, dependency-free domain/persistence slice. Official provider feasibility still needs validation: Google's public YouTube Data API can manage YouTube video playlists, but the research performed for this specification did not identify an official API exposing the complete YouTube Music library model. +### Current adapter boundaries + +- Spotify: official playlist reads plus confirmed playlist creation and entry append operations, subject to connection capabilities and provider policy. +- YouTube Data API: official video-playlist reads only; this is deliberately not represented as YouTube Music. +- Apple Music: experimental official library-playlist reads using separate developer and Music User Token inputs; writes and OAuth wiring remain out of scope until their feasibility gates are closed. + +All adapters expose normalized identities and preserve unavailable/duplicate occurrences. Provider credentials are injected behind ports and are not accepted through App options, logs, or ordinary operation payloads. + ## Documentation Start with the [documentation map](docs/README.md). The horizontal specifications define shared product and architecture rules; the [SDD standard](specs/README.md) and [capability catalog](specs/CATALOG.md) turn those rules into reviewable vertical implementation contracts. From ca226d9abe92bc00d1a0cb94f4571ba926993c54 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:57:04 +0200 Subject: [PATCH 047/167] fix: reject repeated provider pagination cursors --- src/symphonia/providers/apple_music.py | 8 +++++++- src/symphonia/providers/youtube.py | 8 +++++++- tests/test_apple_music_adapter.py | 20 +++++++++++++++++++ tests/test_youtube_adapter.py | 27 ++++++++++++++++++++++++++ 4 files changed, 61 insertions(+), 2 deletions(-) diff --git a/src/symphonia/providers/apple_music.py b/src/symphonia/providers/apple_music.py index f38750f..daaff13 100644 --- a/src/symphonia/providers/apple_music.py +++ b/src/symphonia/providers/apple_music.py @@ -130,7 +130,14 @@ def read_playlist_pages( raise ValueError("Apple Music adapter requires an apple_music library-playlists reference") offset = self._parse_offset(cursor) pages: list[ProviderPlaylistPage] = [] + seen_offsets: set[int] = set() while True: + if offset in seen_offsets: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Apple Music pagination repeated an offset", + ) + seen_offsets.add(offset) response = self._request( connection_id, f"/me/library/playlists/{playlist.object_id}/tracks", @@ -235,4 +242,3 @@ def _entry(playlist: ProviderObjectRef, item: Any, position: int) -> ProviderPla title=attributes.get("name") if isinstance(attributes.get("name"), str) else None, available=available, ) - diff --git a/src/symphonia/providers/youtube.py b/src/symphonia/providers/youtube.py index 1824e24..f354510 100644 --- a/src/symphonia/providers/youtube.py +++ b/src/symphonia/providers/youtube.py @@ -75,7 +75,14 @@ def read_playlist_pages( page_token = cursor pages: list[ProviderPlaylistPage] = [] position = 0 + seen_tokens: set[str | None] = set() while True: + if page_token in seen_tokens: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "YouTube pagination repeated a page token", + ) + seen_tokens.add(page_token) query = { "part": "snippet,contentDetails", "playlistId": playlist.object_id, @@ -163,4 +170,3 @@ def _entry(playlist: ProviderObjectRef, item: Any, *, position: int) -> Provider available=available, source_added_at=snippet.get("publishedAt") if isinstance(snippet.get("publishedAt"), str) else None, ) - diff --git a/tests/test_apple_music_adapter.py b/tests/test_apple_music_adapter.py index 8932f57..4e6f8c9 100644 --- a/tests/test_apple_music_adapter.py +++ b/tests/test_apple_music_adapter.py @@ -86,6 +86,26 @@ def test_missing_developer_or_user_token_fails_closed(self) -> None: self.assertEqual(context.exception.category, ProviderErrorCategory.AUTHENTICATION_REQUIRED) + def test_repeated_offset_is_a_provider_contract_failure(self) -> None: + class LoopingClient(FakeAppleClient): + def request(self, method, path, *, developer_token, user_token, query): + self.calls.append((method, path, developer_token, user_token, query)) + return AppleJsonResponse( + 200, + { + "data": [{"id": "song-1", "type": "songs", "attributes": {"name": "One"}}], + "next": "https://api.music.apple.com/v1/me/library/playlists/playlist-1/tracks?offset=0", + }, + {}, + ) + + with self.assertRaises(ProviderApiError) as context: + AppleMusicAdapter( + LoopingClient(), lambda connection_id: ("developer-token", "user-token"), page_size=1 + ).read_playlist_pages("apple-connection-1", self.playlist()) + + self.assertEqual(context.exception.category, ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_youtube_adapter.py b/tests/test_youtube_adapter.py index fd79b31..51203e7 100644 --- a/tests/test_youtube_adapter.py +++ b/tests/test_youtube_adapter.py @@ -7,6 +7,8 @@ JsonResponse, MediaKind, ProviderObjectRef, + ProviderApiError, + ProviderErrorCategory, YouTubeDataAdapter, ) @@ -78,6 +80,31 @@ def test_capability_probe_is_separate_from_playlist_read(self) -> None: capabilities = adapter.capabilities("google-connection-1") self.assertEqual(capabilities.enabled, frozenset({Capability.READ_PLAYLISTS})) + def test_repeated_page_token_is_a_provider_contract_failure(self) -> None: + class LoopingClient(FakeClient): + def request(self, method: str, path: str, *, token: str, query: dict[str, str], body=None) -> JsonResponse: + self.calls.append((method, path, token, query)) + return JsonResponse( + 200, + { + "items": [ + { + "id": "playlist-item-1", + "snippet": {"resourceId": {"videoId": "video-1"}}, + } + ], + "nextPageToken": "same-token", + }, + {}, + ) + + playlist = ProviderObjectRef("youtube_data", "playlist", "playlist-1", "connection-1") + with self.assertRaises(ProviderApiError) as context: + YouTubeDataAdapter(LoopingClient(), lambda connection_id: "access-token").read_playlist_pages( + "connection-1", playlist + ) + self.assertEqual(context.exception.category, ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED) + if __name__ == "__main__": unittest.main() From 7038936a6a66de5444f1b2aeff55f24e1bd57387 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 18:57:55 +0200 Subject: [PATCH 048/167] feat: gate Spotify write capabilities explicitly --- docs/development/implementation-baseline.md | 2 +- src/symphonia/providers/spotify.py | 11 ++++++++--- tests/test_spotify_adapter.py | 15 +++++++++++++++ 3 files changed, 24 insertions(+), 4 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 22a1414..09d8468 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -26,7 +26,7 @@ The first implementation increment is intentionally narrower than any provider o - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. - Copy plans bind target connection and effective write-capability evidence into their digest. - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. -- Offline-testable official Spotify playlist reader with bounded pagination and normalized error categories. +- Offline-testable official Spotify playlist reader/writer with bounded pagination, explicit write-capability gating, and normalized error categories. - Explicitly scoped official YouTube Data API video-playlist reader; it is not represented as YouTube Music. - Experimental official Apple Music library-playlist reader with separate developer/user token inputs. - Experimental Home Assistant App metadata scaffold with Ingress-only management and `/data` persistence. diff --git a/src/symphonia/providers/spotify.py b/src/symphonia/providers/spotify.py index 7239ff4..489ea4a 100644 --- a/src/symphonia/providers/spotify.py +++ b/src/symphonia/providers/spotify.py @@ -104,7 +104,7 @@ class SpotifyAdapter(ProviderAdapter, PlaylistWriter): display_name="Spotify", access_basis=AccessBasis.OFFICIAL, maturity="beta", - support_level="playlist-read", + support_level="playlist-read/write", upstream_dependencies=("Spotify Web API",), reviewed_on="2026-09-20", ) @@ -115,6 +115,7 @@ def __init__( token_for_connection: Callable[[str], str], page_size: int = 50, connection_id: str | None = None, + allow_writes: bool = False, ) -> None: if not 1 <= page_size <= 50: raise ValueError("Spotify playlist page_size must be between 1 and 50") @@ -122,12 +123,16 @@ def __init__( self._token_for_connection = token_for_connection self._page_size = page_size self._connection_id = connection_id + self._allow_writes = allow_writes def capabilities(self, connection_id: str) -> ProviderCapabilities: response = self._request(connection_id, "GET", "/me/playlists", {"limit": "1", "offset": "0"}) + enabled = {Capability.READ_PLAYLISTS} + if self._allow_writes and self._connection_id == connection_id: + enabled.update({Capability.CREATE_PLAYLIST, Capability.ADD_PLAYLIST_ENTRIES}) return ProviderCapabilities( - enabled=frozenset({Capability.READ_PLAYLISTS}), - evidence_version="spotify-playlist-read-v1", + enabled=frozenset(enabled), + evidence_version="spotify-playlist-read-write-v1" if len(enabled) > 1 else "spotify-playlist-read-v1", observed_at=datetime.now(timezone.utc).isoformat(timespec="seconds"), ) diff --git a/tests/test_spotify_adapter.py b/tests/test_spotify_adapter.py index a800c55..c06188d 100644 --- a/tests/test_spotify_adapter.py +++ b/tests/test_spotify_adapter.py @@ -72,6 +72,21 @@ def test_capability_probe_uses_safe_read_endpoint(self) -> None: self.assertTrue(capabilities.supports(Capability.READ_PLAYLISTS)) self.assertEqual(client.calls[0][1], "/me/playlists") + def test_write_capabilities_require_explicit_verified_composition_flag(self) -> None: + client = FakeClient({"0": JsonResponse(200, {"items": []}, {})}) + adapter = SpotifyAdapter( + client, + lambda connection_id: "access-token", + connection_id="connection-1", + allow_writes=True, + ) + + capabilities = adapter.capabilities("connection-1") + + self.assertTrue(capabilities.supports(Capability.CREATE_PLAYLIST)) + self.assertTrue(capabilities.supports(Capability.ADD_PLAYLIST_ENTRIES)) + self.assertEqual(capabilities.evidence_version, "spotify-playlist-read-write-v1") + def test_rate_limit_is_normalized_with_retry_hint(self) -> None: client = FakeClient({"0": JsonResponse(429, {"error": {"status": 429}}, {"Retry-After": "10"})}) adapter = SpotifyAdapter(client, lambda connection_id: "access-token", connection_id="connection-1") From 5ff55367021d086e805382ce0884b979213f55dd Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:00:16 +0200 Subject: [PATCH 049/167] fix: bound transient operation retries --- docs/development/implementation-baseline.md | 1 + src/symphonia/application/copy_execution.py | 17 +++++++++++++ .../application/library_import_execution.py | 8 +++++++ tests/test_copy_execution.py | 12 ++++++++++ tests/test_library_import_execution.py | 24 +++++++++++++++++++ 5 files changed, 62 insertions(+) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 09d8468..64167e3 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -14,6 +14,7 @@ The first implementation increment is intentionally narrower than any provider o - Append-only SQLite operation event history for auditable transitions with sanitized checkpoint summaries. - Bounded redacted operation diagnostics for support and a future authenticated operations view. - Explicit provider rate-limit waits with absolute retry times and durable eligibility. +- Bounded transient retry budgets for copy and import handlers with durable exhaustion outcomes. - Provider connection persistence with account uniqueness, opaque secret references, health states, and effective capability evidence. - Application connection service for verified-account registration, capability probes, degraded health, and reauthorization-required classification. - Pure capability-layer intersection and requirement checks for adapter/connection/object/health constraints. diff --git a/src/symphonia/application/copy_execution.py b/src/symphonia/application/copy_execution.py index 7151af7..4f6bb5b 100644 --- a/src/symphonia/application/copy_execution.py +++ b/src/symphonia/application/copy_execution.py @@ -17,6 +17,13 @@ class CopyExecutionService: plans: CopyPlanRepository operations: OperationRepository retry_delay_seconds: int = 60 + max_retry_attempts: int = 5 + + def __post_init__(self) -> None: + if self.retry_delay_seconds <= 0: + raise ValueError("retry_delay_seconds must be positive") + if self.max_retry_attempts < 0: + raise ValueError("max_retry_attempts must not be negative") def execute( self, @@ -227,6 +234,16 @@ def _schedule_retry( checkpoint: dict[str, Any], now: datetime, ) -> OperationRecord: + retry_attempts = int(checkpoint.get("retry_attempts", 0)) + 1 + checkpoint = {**checkpoint, "retry_attempts": retry_attempts} + if retry_attempts > self.max_retry_attempts: + return self._finish( + operation, + worker_id, + checkpoint | {"failure_code": "retry_exhausted"}, + now, + "failed", + ) return self.operations.schedule_retry( operation.operation_id, worker_id=worker_id, diff --git a/src/symphonia/application/library_import_execution.py b/src/symphonia/application/library_import_execution.py index 8020728..21585c8 100644 --- a/src/symphonia/application/library_import_execution.py +++ b/src/symphonia/application/library_import_execution.py @@ -21,10 +21,13 @@ class LibraryImportExecutionService: imports: LibraryImportService operations: OperationRepository retry_delay_seconds: int = 60 + max_retry_attempts: int = 5 def __post_init__(self) -> None: if self.retry_delay_seconds <= 0: raise ValueError("retry_delay_seconds must be positive") + if self.max_retry_attempts < 0: + raise ValueError("max_retry_attempts must not be negative") def enqueue_playlist( self, @@ -119,6 +122,11 @@ def execute_claimed( ProviderErrorCategory.TIMEOUT, ProviderErrorCategory.NETWORK_ERROR, }: + retry_attempts = int(checkpoint.get("retry_attempts", 0)) + 1 + checkpoint["retry_attempts"] = retry_attempts + if retry_attempts > self.max_retry_attempts: + checkpoint["failure_code"] = "retry_exhausted" + return self._checkpoint(operation, worker_id, checkpoint, now, "failed") return self.operations.schedule_retry( operation.operation_id, worker_id=worker_id, diff --git a/tests/test_copy_execution.py b/tests/test_copy_execution.py index f0e8600..3b3c26b 100644 --- a/tests/test_copy_execution.py +++ b/tests/test_copy_execution.py @@ -115,6 +115,18 @@ def test_retryable_write_releases_operation_until_scheduled(self) -> None: self.assertEqual(operation.state, "retry_scheduled") self.assertEqual(operation.next_run_at, NOW + timedelta(seconds=30)) + def test_retry_budget_turns_repeated_transient_writes_into_failure(self) -> None: + digest = self.accepted_digest() + step_key = f"{digest}:entry:occ-1" + self.writer.results[step_key] = WriteResult(WriteOutcome.RETRYABLE, provider_code="temporary") + executor = CopyExecutionService(self.plans, self.operations, retry_delay_seconds=30, max_retry_attempts=0) + + operation = executor.execute(digest, writer=self.writer, worker_id="worker-a", now=NOW) + + self.assertEqual(operation.state, "failed") + self.assertEqual(operation.checkpoint["failure_code"], "retry_exhausted") + self.assertEqual(operation.checkpoint["retry_attempts"], 1) + def test_operation_runner_can_dispatch_claimed_copy_execution(self) -> None: digest = self.accepted_digest() self.workflow.enqueue_accepted_plan(digest, now=NOW) diff --git a/tests/test_library_import_execution.py b/tests/test_library_import_execution.py index 0080f97..17c142b 100644 --- a/tests/test_library_import_execution.py +++ b/tests/test_library_import_execution.py @@ -94,6 +94,30 @@ def test_incomplete_import_is_partial_and_retains_no_new_snapshot(self) -> None: self.assertIsNone(result.checkpoint["published_snapshot_id"]) self.assertEqual(result.checkpoint["publication_state"], "partial") + def test_transient_import_failure_honors_retry_budget(self) -> None: + service = LibraryImportExecutionService( + LibraryImportService(self.projections), self.operations, max_retry_attempts=0 + ) + created = service.enqueue_playlist( + FakeAdapter(error=ProviderApiError(ProviderErrorCategory.TIMEOUT, "temporary")), + connection_id="connection-1", + playlist=self.playlist, + snapshot_id="snapshot-timeout", + observed_at=NOW, + now=NOW, + ) + claimed = self.operations.claim(created.operation_id, worker_id="worker-a", now=NOW) + + failed = service.execute_claimed( + claimed, + adapter=FakeAdapter(error=ProviderApiError(ProviderErrorCategory.TIMEOUT, "temporary")), + worker_id="worker-a", + now=NOW, + ) + + self.assertEqual(failed.state, "failed") + self.assertEqual(failed.checkpoint["failure_code"], "retry_exhausted") + if __name__ == "__main__": unittest.main() From 31fbee0a4d4b280f20d68ef2ee9299b8f2bb8cdf Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:01:20 +0200 Subject: [PATCH 050/167] fix: bind copy plans to source object types --- src/symphonia/domain/models.py | 5 +++++ src/symphonia/infrastructure/sqlite_plans.py | 2 ++ tests/test_copy_planning.py | 17 +++++++++++++++++ 3 files changed, 24 insertions(+) diff --git a/src/symphonia/domain/models.py b/src/symphonia/domain/models.py index 3ef26eb..14b4750 100644 --- a/src/symphonia/domain/models.py +++ b/src/symphonia/domain/models.py @@ -113,12 +113,15 @@ class CopyPlanEntry: target_track_id: str | None reason: str | None evidence: tuple[str, ...] + source_provider_track_object_type: str = "track" def __post_init__(self) -> None: if self.disposition not in {"write", "blocked", "omit"}: raise ValueError("disposition must be write, blocked, or omit") if self.disposition == "write" and not self.target_track_id: raise ValueError("write entries require a target_track_id") + if not self.source_provider_track_object_type.strip(): + raise ValueError("source_provider_track_object_type must not be empty") @dataclass(frozen=True, slots=True) @@ -218,6 +221,7 @@ def build_copy_plan( target_track_id=source.target_track_id if ready else None, reason=reason, evidence=source.evidence, + source_provider_track_object_type=source.provider_track_object_type, ) ) @@ -242,6 +246,7 @@ def build_copy_plan( "target_track_id": entry.target_track_id, "reason": entry.reason, "evidence": list(entry.evidence), + "source_provider_track_object_type": entry.source_provider_track_object_type, } for entry in plan_entries ], diff --git a/src/symphonia/infrastructure/sqlite_plans.py b/src/symphonia/infrastructure/sqlite_plans.py index 6588277..1dfd10c 100644 --- a/src/symphonia/infrastructure/sqlite_plans.py +++ b/src/symphonia/infrastructure/sqlite_plans.py @@ -119,6 +119,7 @@ def _serialize(plan: CopyPlan) -> dict[str, Any]: "target_track_id": entry.target_track_id, "reason": entry.reason, "evidence": list(entry.evidence), + "source_provider_track_object_type": entry.source_provider_track_object_type, } for entry in plan.entries ], @@ -144,6 +145,7 @@ def _deserialize(payload: dict[str, Any]) -> CopyPlan: target_track_id=entry["target_track_id"], reason=entry["reason"], evidence=tuple(entry["evidence"]), + source_provider_track_object_type=entry.get("source_provider_track_object_type", "track"), ) for entry in payload["entries"] ), diff --git a/tests/test_copy_planning.py b/tests/test_copy_planning.py index 996d63c..5ccb8a8 100644 --- a/tests/test_copy_planning.py +++ b/tests/test_copy_planning.py @@ -112,6 +112,23 @@ def test_source_namespace_is_part_of_plan_digest(self) -> None: second_plan = self.service.plan(second, target_provider="youtube", target_playlist_name="Rock") self.assertNotEqual(first_plan.digest, second_plan.digest) + def test_source_provider_object_type_is_part_of_plan_digest(self) -> None: + track = snapshot( + SourcePlaylistEntry( + "occ-1", 0, "same-id", EntryClassification.READY, "target-1", provider_track_object_type="track" + ) + ) + episode = snapshot( + SourcePlaylistEntry( + "occ-1", 0, "same-id", EntryClassification.READY, "target-1", provider_track_object_type="episode" + ) + ) + + track_plan = self.service.plan(track, target_provider="youtube", target_playlist_name="Rock") + episode_plan = self.service.plan(episode, target_provider="youtube", target_playlist_name="Rock") + + self.assertNotEqual(track_plan.digest, episode_plan.digest) + if __name__ == "__main__": unittest.main() From f4739a0da59dc9537ed658ae0866ebf1201dda66 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:03:25 +0200 Subject: [PATCH 051/167] feat: retain provider metadata in projections --- docs/development/implementation-baseline.md | 1 + src/symphonia/domain/models.py | 4 ++ .../infrastructure/sqlite_library.py | 20 +++++- src/symphonia/providers/importing.py | 2 + tests/test_sqlite_library.py | 64 +++++++++++++++++++ 5 files changed, 88 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 64167e3..a7fdb7d 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -41,6 +41,7 @@ The first implementation increment is intentionally narrower than any provider o - A copy executor that creates target playlists through a semantic writer port, checkpoints each occurrence, and reconciles unknown writes before retry. - SQLite persistence for complete playlist projections that keeps incomplete imports from replacing the last complete snapshot and isolates external IDs by namespace. - Playlist projections preserve provider object types as part of external identity, including a forward-compatible migration for older rows. +- Playlist projections retain normalized provider titles and source-added timestamps for later explainable identity work. - Idempotent snapshot publication that rejects reused IDs with different content and never rolls back a newer current pointer. - An application import use case that reports partial results without advancing freshness or replacing the last complete projection. diff --git a/src/symphonia/domain/models.py b/src/symphonia/domain/models.py index 14b4750..dd1e106 100644 --- a/src/symphonia/domain/models.py +++ b/src/symphonia/domain/models.py @@ -53,6 +53,8 @@ class SourcePlaylistEntry: evidence: tuple[str, ...] = () reason: str | None = None provider_track_object_type: str = "track" + provider_track_title: str | None = None + source_added_at: str | None = None def __post_init__(self) -> None: if not self.occurrence_id.strip(): @@ -63,6 +65,8 @@ def __post_init__(self) -> None: raise ValueError("provider_track_id must not be empty") if not self.provider_track_object_type.strip(): raise ValueError("provider_track_object_type must not be empty") + if self.provider_track_title is not None and not self.provider_track_title.strip(): + raise ValueError("provider_track_title must not be blank") if self.classification is EntryClassification.READY and not self.target_track_id: raise ValueError("ready entries require a target_track_id") if self.target_track_id is not None and not self.target_track_id.strip(): diff --git a/src/symphonia/infrastructure/sqlite_library.py b/src/symphonia/infrastructure/sqlite_library.py index 4646b99..e0efd1a 100644 --- a/src/symphonia/infrastructure/sqlite_library.py +++ b/src/symphonia/infrastructure/sqlite_library.py @@ -69,6 +69,8 @@ def _migrate(self) -> None: position INTEGER NOT NULL, provider_track_id TEXT NOT NULL, provider_track_object_type TEXT NOT NULL DEFAULT 'track', + provider_track_title TEXT, + source_added_at TEXT, provider_track_namespace TEXT NOT NULL, media_kind TEXT NOT NULL, available INTEGER NOT NULL, @@ -92,6 +94,10 @@ def _migrate(self) -> None: self._connection.execute( "ALTER TABLE playlist_snapshot_entries ADD COLUMN provider_track_object_type TEXT NOT NULL DEFAULT 'track'" ) + if "provider_track_title" not in columns: + self._connection.execute("ALTER TABLE playlist_snapshot_entries ADD COLUMN provider_track_title TEXT") + if "source_added_at" not in columns: + self._connection.execute("ALTER TABLE playlist_snapshot_entries ADD COLUMN source_added_at TEXT") def publish( self, @@ -132,8 +138,9 @@ def publish( """ INSERT INTO playlist_snapshot_entries ( snapshot_id, occurrence_id, position, provider_track_id, - provider_track_object_type, provider_track_namespace, media_kind, available - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?) + provider_track_object_type, provider_track_title, source_added_at, + provider_track_namespace, media_kind, available + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, [ ( @@ -142,6 +149,8 @@ def publish( entry.position, entry.track.object_id, entry.track.object_type, + entry.title, + entry.source_added_at, entry.track.namespace, entry.media_kind.value, int(entry.available), @@ -175,7 +184,8 @@ def _matches_result(self, snapshot_row: sqlite3.Row, result: CollectionImportRes rows = self._connection.execute( """ SELECT occurrence_id, position, provider_track_id, - provider_track_object_type, provider_track_namespace, media_kind, available + provider_track_object_type, provider_track_title, source_added_at, + provider_track_namespace, media_kind, available FROM playlist_snapshot_entries WHERE snapshot_id = ? ORDER BY position @@ -190,6 +200,8 @@ def _matches_result(self, snapshot_row: sqlite3.Row, result: CollectionImportRes and row["position"] == entry.position and row["provider_track_id"] == entry.track.object_id and row["provider_track_object_type"] == entry.track.object_type + and row["provider_track_title"] == entry.title + and row["source_added_at"] == entry.source_added_at and row["provider_track_namespace"] == entry.track.namespace and row["media_kind"] == entry.media_kind.value and bool(row["available"]) == entry.available @@ -243,6 +255,8 @@ def _entries(self, snapshot_id: str, provider: str) -> tuple[SourcePlaylistEntry position=row["position"], provider_track_id=row["provider_track_id"], provider_track_object_type=row["provider_track_object_type"], + provider_track_title=row["provider_track_title"], + source_added_at=row["source_added_at"], classification=EntryClassification.UNMATCHED if row["available"] else EntryClassification.UNAVAILABLE, reason=None if row["available"] else "provider reported item unavailable", ) diff --git a/src/symphonia/providers/importing.py b/src/symphonia/providers/importing.py index 4e5eadf..7977cfc 100644 --- a/src/symphonia/providers/importing.py +++ b/src/symphonia/providers/importing.py @@ -108,6 +108,8 @@ def to_playlist_snapshot(result: CollectionImportResult, snapshot_id: str) -> Pl classification=EntryClassification.UNMATCHED if entry.available else EntryClassification.UNAVAILABLE, reason=None if entry.available else "provider reported item unavailable", provider_track_object_type=entry.track.object_type, + provider_track_title=entry.title, + source_added_at=entry.source_added_at, ) for entry in result.entries ) diff --git a/tests/test_sqlite_library.py b/tests/test_sqlite_library.py index 520be3d..3498e06 100644 --- a/tests/test_sqlite_library.py +++ b/tests/test_sqlite_library.py @@ -1,6 +1,8 @@ from __future__ import annotations from datetime import datetime, timezone +import sqlite3 +import tempfile import unittest from symphonia.domain import EntryClassification @@ -43,6 +45,68 @@ def test_complete_result_is_published_with_namespace_and_availability(self) -> N self.assertEqual(current.snapshot.source_namespace, "connection-1") self.assertEqual(current.snapshot.entries[0].classification, EntryClassification.UNAVAILABLE) + def test_complete_result_retains_provider_metadata_for_future_resolution(self) -> None: + playlist = ProviderObjectRef("spotify", "playlist", "playlist-1", "connection-1") + track = ProviderObjectRef("spotify", "track", "track-1", "connection-1") + item = ProviderPlaylistEntry( + "occ-1", + 0, + track, + MediaKind.TRACK, + title="Song title", + source_added_at="2026-09-20T12:00:00Z", + ) + result = collect_playlist_pages([ProviderPlaylistPage(playlist, (item,), None, None, True)]) + + stored = self.repository.publish(result, snapshot_id="snapshot-metadata", published_at=NOW) + + self.assertEqual(stored.snapshot.entries[0].provider_track_title, "Song title") + self.assertEqual(stored.snapshot.entries[0].source_added_at, "2026-09-20T12:00:00Z") + + def test_legacy_projection_schema_gets_metadata_columns(self) -> None: + with tempfile.TemporaryDirectory() as directory: + path = f"{directory}/legacy.sqlite3" + connection = sqlite3.connect(path) + connection.executescript( + """ + CREATE TABLE playlist_snapshots ( + snapshot_id TEXT PRIMARY KEY, + provider TEXT NOT NULL, + namespace TEXT NOT NULL, + playlist_id TEXT NOT NULL, + revision TEXT, + published_at TEXT NOT NULL + ); + CREATE TABLE playlist_snapshot_entries ( + snapshot_id TEXT NOT NULL, + occurrence_id TEXT NOT NULL, + position INTEGER NOT NULL, + provider_track_id TEXT NOT NULL, + provider_track_namespace TEXT NOT NULL, + media_kind TEXT NOT NULL, + available INTEGER NOT NULL + ); + CREATE TABLE current_playlist_snapshots ( + provider TEXT NOT NULL, + namespace TEXT NOT NULL, + playlist_id TEXT NOT NULL, + snapshot_id TEXT NOT NULL + ); + """ + ) + connection.close() + + migrated = PlaylistProjectionRepository(path) + columns = { + row[1] + for row in migrated._connection.execute("PRAGMA table_info(playlist_snapshot_entries)").fetchall() + } + migrated.close() + + self.assertIn("provider_track_object_type", columns) + self.assertIn("provider_track_title", columns) + self.assertIn("source_added_at", columns) + def test_incomplete_result_cannot_replace_current_projection(self) -> None: complete = collect_playlist_pages([page()]) self.repository.publish(complete, snapshot_id="snapshot-1", published_at=NOW) From 7c72be56d1f6240bb21b29bb0623a3b3a59cfc36 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:24:18 +0200 Subject: [PATCH 052/167] fix: bound provider pagination reads --- docs/development/implementation-baseline.md | 1 + src/symphonia/providers/apple_music.py | 9 +++++++++ src/symphonia/providers/spotify.py | 9 +++++++++ src/symphonia/providers/youtube.py | 9 +++++++++ tests/test_apple_music_adapter.py | 8 ++++++++ tests/test_spotify_adapter.py | 21 +++++++++++++++++++++ tests/test_youtube_adapter.py | 9 +++++++++ 7 files changed, 66 insertions(+) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index a7fdb7d..7075ddf 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -30,6 +30,7 @@ The first implementation increment is intentionally narrower than any provider o - Offline-testable official Spotify playlist reader/writer with bounded pagination, explicit write-capability gating, and normalized error categories. - Explicitly scoped official YouTube Data API video-playlist reader; it is not represented as YouTube Music. - Experimental official Apple Music library-playlist reader with separate developer/user token inputs. +- Provider readers enforce bounded page sizes, repeated-cursor detection, and configurable maximum page counts. - Experimental Home Assistant App metadata scaffold with Ingress-only management and `/data` persistence. - Explicit runtime resource composition and reverse-order shutdown for all durable repositories. - Ingress-relative health/version routing with normalized, traversal-safe base paths. diff --git a/src/symphonia/providers/apple_music.py b/src/symphonia/providers/apple_music.py index daaff13..9839543 100644 --- a/src/symphonia/providers/apple_music.py +++ b/src/symphonia/providers/apple_music.py @@ -105,12 +105,16 @@ def __init__( client: AppleJsonClient | None, tokens_for_connection: Callable[[str], tuple[str, str]], page_size: int = 25, + max_pages: int = 10_000, ) -> None: if not 1 <= page_size <= 100: raise ValueError("Apple Music playlist page_size must be between 1 and 100") + if max_pages <= 0: + raise ValueError("Apple Music max_pages must be positive") self._client = client or UrllibAppleMusicClient() self._tokens_for_connection = tokens_for_connection self._page_size = page_size + self._max_pages = max_pages def capabilities(self, connection_id: str) -> ProviderCapabilities: self._request(connection_id, "/me/library/playlists", {"limit": "1", "offset": "0"}) @@ -132,6 +136,11 @@ def read_playlist_pages( pages: list[ProviderPlaylistPage] = [] seen_offsets: set[int] = set() while True: + if len(pages) >= self._max_pages: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Apple Music playlist pagination exceeded the configured page limit", + ) if offset in seen_offsets: raise ProviderApiError( ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, diff --git a/src/symphonia/providers/spotify.py b/src/symphonia/providers/spotify.py index 489ea4a..d0a582a 100644 --- a/src/symphonia/providers/spotify.py +++ b/src/symphonia/providers/spotify.py @@ -116,14 +116,18 @@ def __init__( page_size: int = 50, connection_id: str | None = None, allow_writes: bool = False, + max_pages: int = 10_000, ) -> None: if not 1 <= page_size <= 50: raise ValueError("Spotify playlist page_size must be between 1 and 50") + if max_pages <= 0: + raise ValueError("Spotify max_pages must be positive") self._client = client self._token_for_connection = token_for_connection self._page_size = page_size self._connection_id = connection_id self._allow_writes = allow_writes + self._max_pages = max_pages def capabilities(self, connection_id: str) -> ProviderCapabilities: response = self._request(connection_id, "GET", "/me/playlists", {"limit": "1", "offset": "0"}) @@ -147,6 +151,11 @@ def read_playlist_pages( offset = self._parse_cursor(cursor) pages: list[ProviderPlaylistPage] = [] while True: + if len(pages) >= self._max_pages: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Spotify playlist pagination exceeded the configured page limit", + ) response = self._request( connection_id, "GET", diff --git a/src/symphonia/providers/youtube.py b/src/symphonia/providers/youtube.py index f354510..2f26909 100644 --- a/src/symphonia/providers/youtube.py +++ b/src/symphonia/providers/youtube.py @@ -44,13 +44,17 @@ def __init__( token_for_connection: Callable[[str], str], page_size: int = 50, api_key: str | None = None, + max_pages: int = 10_000, ) -> None: if not 1 <= page_size <= 50: raise ValueError("YouTube playlist page_size must be between 1 and 50") + if max_pages <= 0: + raise ValueError("YouTube max_pages must be positive") self._client = client or UrllibJsonClient("https://www.googleapis.com/youtube/v3") self._token_for_connection = token_for_connection self._page_size = page_size self._api_key = api_key + self._max_pages = max_pages def capabilities(self, connection_id: str) -> ProviderCapabilities: self._request( @@ -77,6 +81,11 @@ def read_playlist_pages( position = 0 seen_tokens: set[str | None] = set() while True: + if len(pages) >= self._max_pages: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "YouTube playlist pagination exceeded the configured page limit", + ) if page_token in seen_tokens: raise ProviderApiError( ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, diff --git a/tests/test_apple_music_adapter.py b/tests/test_apple_music_adapter.py index 4e6f8c9..352e0df 100644 --- a/tests/test_apple_music_adapter.py +++ b/tests/test_apple_music_adapter.py @@ -106,6 +106,14 @@ def request(self, method, path, *, developer_token, user_token, query): self.assertEqual(context.exception.category, ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED) + def test_max_page_limit_fails_closed_before_unbounded_reads(self) -> None: + with self.assertRaises(ProviderApiError) as context: + AppleMusicAdapter( + FakeAppleClient(), lambda connection_id: ("developer-token", "user-token"), max_pages=1 + ).read_playlist_pages("apple-connection-1", self.playlist()) + + self.assertEqual(context.exception.category, ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_spotify_adapter.py b/tests/test_spotify_adapter.py index c06188d..59d4837 100644 --- a/tests/test_spotify_adapter.py +++ b/tests/test_spotify_adapter.py @@ -95,6 +95,27 @@ def test_rate_limit_is_normalized_with_retry_hint(self) -> None: self.assertEqual(context.exception.category, ProviderErrorCategory.RATE_LIMITED) self.assertIsNotNone(context.exception.retry_at) + def test_max_page_limit_fails_closed_before_unbounded_reads(self) -> None: + client = FakeClient( + { + "0": JsonResponse( + 200, + { + "items": [{"item": {"id": "track-1", "type": "track"}}], + "next": "https://api.spotify.com/v1/playlists/playlist-1/items?offset=1", + }, + {}, + ) + } + ) + adapter = SpotifyAdapter(client, lambda connection_id: "access-token", page_size=1, max_pages=1) + + with self.assertRaises(ProviderApiError) as context: + adapter.read_playlist_pages("connection-1", self.playlist()) + + self.assertEqual(context.exception.category, ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED) + self.assertEqual(len(client.calls), 1) + def test_invalid_cursor_and_empty_token_fail_closed(self) -> None: client = FakeClient({"0": JsonResponse(200, {"items": [], "next": None}, {})}) adapter = SpotifyAdapter(client, lambda connection_id: "") diff --git a/tests/test_youtube_adapter.py b/tests/test_youtube_adapter.py index 51203e7..c984883 100644 --- a/tests/test_youtube_adapter.py +++ b/tests/test_youtube_adapter.py @@ -105,6 +105,15 @@ def request(self, method: str, path: str, *, token: str, query: dict[str, str], ) self.assertEqual(context.exception.category, ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED) + def test_max_page_limit_fails_closed_before_unbounded_reads(self) -> None: + playlist = ProviderObjectRef("youtube_data", "playlist", "playlist-1", "connection-1") + adapter = YouTubeDataAdapter(FakeClient(), lambda connection_id: "access-token", max_pages=1) + + with self.assertRaises(ProviderApiError) as context: + adapter.read_playlist_pages("connection-1", playlist) + + self.assertEqual(context.exception.category, ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED) + if __name__ == "__main__": unittest.main() From e941e6777abf5f43557084e3f524a606cbe8d7e9 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:24:56 +0200 Subject: [PATCH 053/167] feat: add bounded operation diagnostics listing --- docs/development/implementation-baseline.md | 1 + .../infrastructure/sqlite_operations.py | 18 ++++++++++++++++++ tests/test_sqlite_operations.py | 19 +++++++++++++++++++ 3 files changed, 38 insertions(+) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 7075ddf..b91e994 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -13,6 +13,7 @@ The first implementation increment is intentionally narrower than any provider o - Atomic scheduler-facing claim selection for queued, due-retry, and expired-lease operations. - Append-only SQLite operation event history for auditable transitions with sanitized checkpoint summaries. - Bounded redacted operation diagnostics for support and a future authenticated operations view. +- Bounded recent-operation diagnostics listing that exposes only redacted support views. - Explicit provider rate-limit waits with absolute retry times and durable eligibility. - Bounded transient retry budgets for copy and import handlers with durable exhaustion outcomes. - Provider connection persistence with account uniqueness, opaque secret references, health states, and effective capability evidence. diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index e543f78..c2f7759 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -239,6 +239,24 @@ def diagnostic(self, operation_id: str, *, event_limit: int = 100) -> dict[str, ], } + def diagnostics(self, *, limit: int = 50, event_limit: int = 20) -> tuple[dict[str, Any], ...]: + """Return a bounded list of redacted operation support views.""" + + if limit <= 0: + raise ValueError("limit must be positive") + if event_limit <= 0: + raise ValueError("event_limit must be positive") + rows = self._connection.execute( + """ + SELECT operation_id + FROM operations + ORDER BY updated_at DESC, operation_id DESC + LIMIT ? + """, + (limit,), + ).fetchall() + return tuple(self.diagnostic(row["operation_id"], event_limit=event_limit) for row in rows) + def claim( self, operation_id: str, diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 68476f4..a888bb1 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -137,6 +137,25 @@ def test_diagnostic_export_is_bounded_and_redacted(self) -> None: self.assertNotIn("secret-token", serialized) self.assertNotIn("provider-secret", serialized) + def test_diagnostics_list_is_bounded_and_redacted(self) -> None: + for index in range(3): + self.repository.create( + operation_type="copy", + idempotency_key=f"diagnostic-{index}", + payload={"access_token": f"secret-{index}"}, + now=self.now + timedelta(seconds=index), + ) + + diagnostics = self.repository.diagnostics(limit=2, event_limit=1) + + self.assertEqual(len(diagnostics), 2) + self.assertTrue(all("payload_keys" in item for item in diagnostics)) + self.assertTrue(all("access_token" in item["payload_keys"] for item in diagnostics)) + self.assertNotIn("secret-", str(diagnostics)) + + with self.assertRaises(ValueError): + self.repository.diagnostics(limit=0) + def test_only_lease_owner_can_checkpoint(self) -> None: operation = self.repository.create( operation_type="import", From afdbd26b01eaff02b2d47840ddc9a2e14e853f7f Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:25:32 +0200 Subject: [PATCH 054/167] fix: audit recovered operation leases --- docs/development/implementation-baseline.md | 1 + .../infrastructure/sqlite_operations.py | 17 +++++++++++++---- tests/test_sqlite_operations.py | 4 ++++ 3 files changed, 18 insertions(+), 4 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index b91e994..663730c 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -16,6 +16,7 @@ The first implementation increment is intentionally narrower than any provider o - Bounded recent-operation diagnostics listing that exposes only redacted support views. - Explicit provider rate-limit waits with absolute retry times and durable eligibility. - Bounded transient retry budgets for copy and import handlers with durable exhaustion outcomes. +- Lease-expiry recovery is audited distinctly from first claims, with prior worker identity retained only as metadata. - Provider connection persistence with account uniqueness, opaque secret references, health states, and effective capability evidence. - Application connection service for verified-account registration, capability probes, degraded health, and reauthorization-required classification. - Pure capability-layer intersection and requirement checks for adapter/connection/object/health constraints. diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index c2f7759..e6fabc6 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -290,6 +290,10 @@ def claim( ) if not eligible and not expired: raise LeaseConflict(f"operation {operation_id} is not eligible for claim") + event_type = "lease_reclaimed" if expired else "claimed" + event_payload = {"lease_expires_at": expires_text} + if expired and row["worker_id"]: + event_payload["previous_worker_id"] = row["worker_id"] self._connection.execute( """ UPDATE operations @@ -301,10 +305,10 @@ def claim( ) self._append_event( operation_id=operation_id, - event_type="claimed", + event_type=event_type, state="running", worker_id=worker_id, - payload={"lease_expires_at": expires_text}, + payload=event_payload, created_at=now_text, ) self._connection.execute("COMMIT") @@ -363,6 +367,11 @@ def claim_next( self._connection.execute("COMMIT") return None operation_id = row["operation_id"] + recovered = row["state"] == "running" + event_type = "lease_reclaimed" if recovered else "claimed" + event_payload = {"lease_expires_at": expires_text} + if recovered and row["worker_id"]: + event_payload["previous_worker_id"] = row["worker_id"] self._connection.execute( """ UPDATE operations @@ -374,10 +383,10 @@ def claim_next( ) self._append_event( operation_id=operation_id, - event_type="claimed", + event_type=event_type, state="running", worker_id=worker_id, - payload={"lease_expires_at": expires_text}, + payload=event_payload, created_at=now_text, ) self._connection.execute("COMMIT") diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index a888bb1..4cd7c7d 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -212,6 +212,9 @@ def test_expired_running_lease_can_be_reclaimed(self) -> None: now=self.now + timedelta(seconds=6), ) self.assertEqual(reclaimed.worker_id, "worker-b") + events = self.repository.events(operation.operation_id) + self.assertEqual(events[-1].event_type, "lease_reclaimed") + self.assertEqual(events[-1].payload["previous_worker_id"], "worker-a") def test_claim_next_selects_queued_and_due_retry_work(self) -> None: queued = self.repository.create( @@ -270,6 +273,7 @@ def test_claim_next_recovers_an_expired_running_lease(self) -> None: self.assertIsNotNone(recovered) self.assertEqual(recovered.operation_id, operation.operation_id) self.assertEqual(recovered.worker_id, "worker-b") + self.assertEqual(self.repository.events(operation.operation_id)[-1].event_type, "lease_reclaimed") def test_retry_cannot_be_claimed_before_its_scheduled_time(self) -> None: operation = self.repository.create( From 11670ce21c0b48083249345b5d0323376d203c0f Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:26:18 +0200 Subject: [PATCH 055/167] fix: make readiness cover all durable stores --- docs/development/implementation-baseline.md | 1 + src/symphonia/runtime/http.py | 6 +++++- src/symphonia/runtime/resources.py | 17 +++++++++++++++++ tests/test_runtime_http.py | 6 ++++++ tests/test_runtime_resources.py | 1 + 5 files changed, 30 insertions(+), 1 deletion(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 663730c..18d93b2 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -35,6 +35,7 @@ The first implementation increment is intentionally narrower than any provider o - Provider readers enforce bounded page sizes, repeated-cursor detection, and configurable maximum page counts. - Experimental Home Assistant App metadata scaffold with Ingress-only management and `/data` persistence. - Explicit runtime resource composition and reverse-order shutdown for all durable repositories. +- Readiness can validate every composed durable store instead of only the operation queue. - Ingress-relative health/version routing with normalized, traversal-safe base paths. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. diff --git a/src/symphonia/runtime/http.py b/src/symphonia/runtime/http.py index 6597420..a5e6e0f 100644 --- a/src/symphonia/runtime/http.py +++ b/src/symphonia/runtime/http.py @@ -4,6 +4,7 @@ from http.server import BaseHTTPRequestHandler, HTTPServer import json +from collections.abc import Callable from typing import Any from urllib.parse import urlsplit @@ -28,6 +29,7 @@ def __init__( super().__init__(address, SymphoniaRequestHandler) self.resources = resources self.repository = resources.operations if resources is not None else repository + self.readiness_check: Callable[[], bool] = resources.healthcheck if resources is not None else repository.healthcheck self.service_version = __version__ self.ingress_path = _normalize_base_path(ingress_path) @@ -48,6 +50,7 @@ def do_GET(self) -> None: # noqa: N802 - stdlib handler API self.server.repository, self.server.service_version, self.server.ingress_path, + self.server.readiness_check, ) self._json(status, payload) @@ -86,6 +89,7 @@ def route_get( repository: OperationRepository, service_version: str = __version__, ingress_path: str = "/", + readiness_check: Callable[[], bool] | None = None, ) -> tuple[int, dict[str, Any]]: """Resolve a GET request without opening a socket. @@ -101,7 +105,7 @@ def route_get( return 200, {"service": "symphonia", "status": "ok", "version": service_version} if relative_path == "/ready": try: - healthy = repository.healthcheck() + healthy = (readiness_check or repository.healthcheck)() except Exception: # readiness must fail closed without exposing internals healthy = False return (200, {"service": "symphonia", "status": "ready"}) if healthy else ( diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 52a1d99..223fd8a 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -68,5 +68,22 @@ def close(self) -> None: ): repository.close() + def healthcheck(self) -> bool: + """Check every durable store without exposing adapter internals.""" + + try: + for repository in ( + self.operations, + self.plans, + self.connections, + self.authorization, + self.projections, + self.resolutions, + ): + repository._connection.execute("SELECT 1").fetchone() # type: ignore[attr-defined] + except Exception: + return False + return True + __all__ = ["RuntimeResources"] diff --git a/tests/test_runtime_http.py b/tests/test_runtime_http.py index 511bf78..4643e1c 100644 --- a/tests/test_runtime_http.py +++ b/tests/test_runtime_http.py @@ -52,6 +52,12 @@ def test_unsafe_ingress_base_path_is_rejected(self) -> None: with self.assertRaises(ValueError): route_get("/ready", self.repository, ingress_path="/bad/../path") + def test_readiness_can_use_the_composed_runtime_healthcheck(self) -> None: + status, payload = route_get("/ready", self.repository, readiness_check=lambda: False) + + self.assertEqual(status, 503) + self.assertEqual(payload["status"], "not_ready") + if __name__ == "__main__": unittest.main() diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index a961e53..59f159d 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -25,6 +25,7 @@ def test_open_creates_all_durable_store_schemas_and_closes_them(self) -> None: self.assertIn("playlist_snapshots", tables) self.assertIn("authorization_attempts", tables) self.assertIn("resolution_decisions", tables) + self.assertTrue(resources.healthcheck()) finally: resources.close() From 71cc06f832ba598f370f4f5da0fe0a6e77e1642e Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:27:38 +0200 Subject: [PATCH 056/167] fix: validate OAuth redirect URIs before persistence --- docs/development/implementation-baseline.md | 1 + src/symphonia/application/authorization.py | 4 ++-- .../infrastructure/sqlite_authorization.py | 4 ++-- src/symphonia/providers/__init__.py | 2 ++ src/symphonia/providers/authorization.py | 20 ++++++++++++++++++- tests/test_authorization_service.py | 15 ++++++++++++++ 6 files changed, 41 insertions(+), 5 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 18d93b2..2302a81 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -26,6 +26,7 @@ The first implementation increment is intentionally narrower than any provider o - Playlist import executor persists intent before reads and reports succeeded, partial, waiting-user, retry, and rate-limit outcomes durably. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Application authorization boundary that generates one-use state without persisting the raw value. +- Authorization callback binding rejects unsafe schemes, fragments, credentials, whitespace, and non-loopback HTTP hosts. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. - Copy plans bind target connection and effective write-capability evidence into their digest. - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. diff --git a/src/symphonia/application/authorization.py b/src/symphonia/application/authorization.py index bf3fc00..8f89088 100644 --- a/src/symphonia/application/authorization.py +++ b/src/symphonia/application/authorization.py @@ -9,7 +9,7 @@ from typing import Callable from symphonia.infrastructure.sqlite_authorization import AuthorizationAttemptRepository -from symphonia.providers.authorization import AuthorizationAttempt +from symphonia.providers.authorization import AuthorizationAttempt, validate_redirect_uri @dataclass(frozen=True, slots=True) @@ -35,6 +35,7 @@ def begin( now: datetime, ttl: timedelta = timedelta(minutes=10), ) -> AuthorizationStart: + validate_redirect_uri(redirect_uri) raw_state = self.state_factory() attempt = self.attempts.create( attempt_id=self.id_factory(), @@ -52,4 +53,3 @@ def consume(self, attempt_id: str, *, raw_state: str, now: datetime) -> Authoriz def deny(self, attempt_id: str, *, now: datetime, failure_code: str = "consent_denied") -> AuthorizationAttempt: return self.attempts.deny(attempt_id, now=now, failure_code=failure_code) - diff --git a/src/symphonia/infrastructure/sqlite_authorization.py b/src/symphonia/infrastructure/sqlite_authorization.py index eba9cd8..b9b4a53 100644 --- a/src/symphonia/infrastructure/sqlite_authorization.py +++ b/src/symphonia/infrastructure/sqlite_authorization.py @@ -7,7 +7,7 @@ import hmac import sqlite3 -from symphonia.providers.authorization import AuthorizationAttempt, AuthorizationState +from symphonia.providers.authorization import AuthorizationAttempt, AuthorizationState, validate_redirect_uri def _utc(value: datetime) -> str: @@ -78,6 +78,7 @@ def create( ) -> AuthorizationAttempt: if ttl <= timedelta(0): raise ValueError("authorization attempt ttl must be positive") + validate_redirect_uri(redirect_uri) created_at = _utc(now) expires_at = _utc(now + ttl) self._connection.execute( @@ -202,4 +203,3 @@ def _record(row: sqlite3.Row) -> AuthorizationAttempt: completed_at=None if row["completed_at"] is None else _parse_utc(row["completed_at"]), failure_code=row["failure_code"], ) - diff --git a/src/symphonia/providers/__init__.py b/src/symphonia/providers/__init__.py index d68a509..69473f7 100644 --- a/src/symphonia/providers/__init__.py +++ b/src/symphonia/providers/__init__.py @@ -13,6 +13,7 @@ ) from .connections import ConnectionState, ProviderConnection from .authorization import AuthorizationAttempt, AuthorizationState +from .authorization import validate_redirect_uri from .registry import ProviderAlreadyRegistered, ProviderNotRegistered, ProviderRegistry from .errors import ProviderApiError, ProviderErrorCategory from .spotify import JsonResponse, SpotifyAdapter, UrllibJsonClient @@ -26,6 +27,7 @@ "AccessBasis", "AuthorizationAttempt", "AuthorizationState", + "validate_redirect_uri", "Capability", "CapabilityError", "CapabilityLayers", diff --git a/src/symphonia/providers/authorization.py b/src/symphonia/providers/authorization.py index edaa825..c0b1c58 100644 --- a/src/symphonia/providers/authorization.py +++ b/src/symphonia/providers/authorization.py @@ -10,6 +10,7 @@ from dataclasses import dataclass from datetime import datetime from enum import Enum +from urllib.parse import urlsplit class AuthorizationState(str, Enum): @@ -20,6 +21,23 @@ class AuthorizationState(str, Enum): FAILED = "failed" +def validate_redirect_uri(value: str) -> str: + """Validate a fixed OAuth callback URI before durable binding.""" + + if not value or not value.strip() or any(character.isspace() for character in value): + raise ValueError("redirect_uri must be a nonblank URI without whitespace") + parsed = urlsplit(value) + if parsed.scheme not in {"http", "https"} or not parsed.hostname: + raise ValueError("redirect_uri must use http(s) and include a host") + if parsed.fragment: + raise ValueError("redirect_uri must not include a fragment") + if parsed.username is not None or parsed.password is not None: + raise ValueError("redirect_uri must not embed credentials") + if parsed.scheme == "http" and parsed.hostname.lower() not in {"localhost", "127.0.0.1", "::1"}: + raise ValueError("http redirect_uri is only allowed for loopback hosts") + return value + + @dataclass(frozen=True, slots=True) class AuthorizationAttempt: attempt_id: str @@ -43,10 +61,10 @@ def __post_init__(self) -> None: ): if not value.strip(): raise ValueError(f"{field_name} must not be empty") + validate_redirect_uri(self.redirect_uri) if self.created_at.tzinfo is None or self.expires_at.tzinfo is None: raise ValueError("authorization timestamps must be timezone-aware") if self.expires_at <= self.created_at: raise ValueError("authorization attempt must expire after creation") if self.completed_at is not None and self.completed_at.tzinfo is None: raise ValueError("completed_at must be timezone-aware") - diff --git a/tests/test_authorization_service.py b/tests/test_authorization_service.py index 0d9f980..a133b66 100644 --- a/tests/test_authorization_service.py +++ b/tests/test_authorization_service.py @@ -63,6 +63,21 @@ def test_consume_and_deny_delegate_single_use_transitions(self) -> None: denied = self.service.deny("attempt-2", now=NOW) self.assertEqual(denied.state, AuthorizationState.DENIED) + def test_redirect_uri_rejects_unsafe_callback_forms(self) -> None: + for redirect_uri in ( + "javascript:alert(1)", + "https://ha.example/callback#fragment", + "https://user:password@ha.example/callback", + "http://remote.example/callback", + ): + with self.subTest(redirect_uri=redirect_uri), self.assertRaises(ValueError): + self.service.begin( + provider="spotify", + actor_id="ha-user-1", + redirect_uri=redirect_uri, + now=NOW, + ) + if __name__ == "__main__": unittest.main() From 9124728adfeccb9a1020eecc23454e121886bf44 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:28:38 +0200 Subject: [PATCH 057/167] fix: redact secrets from provider errors --- docs/development/implementation-baseline.md | 1 + src/symphonia/providers/__init__.py | 3 ++- src/symphonia/providers/errors.py | 21 +++++++++++++++++++-- tests/test_provider_connections.py | 10 ++++++++++ 4 files changed, 32 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 2302a81..5bc6ce0 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -27,6 +27,7 @@ The first implementation increment is intentionally narrower than any provider o - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Application authorization boundary that generates one-use state without persisting the raw value. - Authorization callback binding rejects unsafe schemes, fragments, credentials, whitespace, and non-loopback HTTP hosts. +- Normalized provider errors redact common bearer/token/secret/password/cookie forms at the provider boundary. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. - Copy plans bind target connection and effective write-capability evidence into their digest. - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. diff --git a/src/symphonia/providers/__init__.py b/src/symphonia/providers/__init__.py index 69473f7..2a34350 100644 --- a/src/symphonia/providers/__init__.py +++ b/src/symphonia/providers/__init__.py @@ -15,7 +15,7 @@ from .authorization import AuthorizationAttempt, AuthorizationState from .authorization import validate_redirect_uri from .registry import ProviderAlreadyRegistered, ProviderNotRegistered, ProviderRegistry -from .errors import ProviderApiError, ProviderErrorCategory +from .errors import ProviderApiError, ProviderErrorCategory, redact_error_detail from .spotify import JsonResponse, SpotifyAdapter, UrllibJsonClient from .youtube import YouTubeDataAdapter from .apple_music import AppleJsonResponse, AppleMusicAdapter, UrllibAppleMusicClient @@ -39,6 +39,7 @@ "ProviderApiError", "ProviderAlreadyRegistered", "ProviderErrorCategory", + "redact_error_detail", "ProviderCapabilities", "ProviderConnection", "ProviderManifest", diff --git a/src/symphonia/providers/errors.py b/src/symphonia/providers/errors.py index 823a698..112f4d5 100644 --- a/src/symphonia/providers/errors.py +++ b/src/symphonia/providers/errors.py @@ -5,6 +5,22 @@ from dataclasses import dataclass from datetime import datetime from enum import Enum +import re + + +_BEARER = re.compile(r"(?i)\bBearer\s+[^\s,;]+") +_ASSIGNMENT = re.compile( + r"(?i)\b(token|access[_-]?token|refresh[_-]?token|id[_-]?token|client[_-]?secret|secret|password|cookie|authorization)\s*[:=]\s*[^\s,;]+" +) + + +def redact_error_detail(detail: str) -> str: + """Remove common credential forms before an error leaves the provider port.""" + + if not isinstance(detail, str) or not detail.strip(): + raise ValueError("provider error detail must not be empty") + redacted = _BEARER.sub("Bearer [REDACTED]", detail) + return _ASSIGNMENT.sub(lambda match: f"{match.group(1)}=[REDACTED]", redacted) class ProviderErrorCategory(str, Enum): @@ -33,9 +49,10 @@ class ProviderApiError(RuntimeError): correlation_id: str | None = None def __post_init__(self) -> None: - RuntimeError.__init__(self, self.detail) + redacted_detail = redact_error_detail(self.detail) + object.__setattr__(self, "detail", redacted_detail) + RuntimeError.__init__(self, redacted_detail) if not self.detail.strip(): raise ValueError("provider error detail must not be empty") if self.correlation_id is not None and not self.correlation_id.strip(): raise ValueError("correlation_id must not be blank") - diff --git a/tests/test_provider_connections.py b/tests/test_provider_connections.py index 0010544..c0f40d2 100644 --- a/tests/test_provider_connections.py +++ b/tests/test_provider_connections.py @@ -13,6 +13,7 @@ ProviderErrorCategory, ProviderManifest, ProviderRegistry, + redact_error_detail, ) @@ -38,6 +39,15 @@ def read_playlist_pages(self, connection_id, playlist, cursor=None): class ProviderConnectionServiceTests(unittest.TestCase): + def test_provider_error_detail_redacts_common_credentials(self) -> None: + detail = redact_error_detail("Bearer abc123 token=secret refresh_token=refresh-value") + error = ProviderApiError(ProviderErrorCategory.NETWORK_ERROR, detail) + + self.assertNotIn("abc123", str(error)) + self.assertNotIn("secret", str(error)) + self.assertNotIn("refresh-value", str(error)) + self.assertIn("[REDACTED]", str(error)) + def setUp(self) -> None: self.connections = ProviderConnectionRepository() self.registry = ProviderRegistry() From 90e09ff25e3601f6faaf3ac153127e914333d35a Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:29:29 +0200 Subject: [PATCH 058/167] fix: reject credential keys in operation payloads --- docs/development/implementation-baseline.md | 1 + .../infrastructure/sqlite_operations.py | 17 +++++++++++++++ tests/test_sqlite_operations.py | 21 +++++++++++++------ 3 files changed, 33 insertions(+), 6 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 5bc6ce0..0b79169 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -28,6 +28,7 @@ The first implementation increment is intentionally narrower than any provider o - Application authorization boundary that generates one-use state without persisting the raw value. - Authorization callback binding rejects unsafe schemes, fragments, credentials, whitespace, and non-loopback HTTP hosts. - Normalized provider errors redact common bearer/token/secret/password/cookie forms at the provider boundary. +- Operation persistence rejects common credential-shaped payload keys before SQLite writes. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. - Copy plans bind target connection and effective write-capability evidence into their digest. - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index e6fabc6..08a8f13 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -10,6 +10,7 @@ from dataclasses import dataclass from datetime import datetime, timedelta, timezone import json +import re import sqlite3 from typing import Any import uuid @@ -37,6 +38,21 @@ class LeaseConflict(RuntimeError): pass +_SECRET_PAYLOAD_KEY = re.compile( + r"(?i)(?:^|[_-])(access[_-]?token|refresh[_-]?token|id[_-]?token|client[_-]?secret|password|cookie|authorization|secret[_-]?token)(?:$|[_-])|^(?:secret|token)$" +) + + +def _validate_payload_keys(payload: dict[str, Any]) -> None: + forbidden = sorted( + str(key) + for key in payload + if _SECRET_PAYLOAD_KEY.search(str(key)) + ) + if forbidden: + raise ValueError(f"operation payload contains forbidden credential keys: {', '.join(forbidden)}") + + @dataclass(frozen=True, slots=True) class OperationRecord: operation_id: str @@ -147,6 +163,7 @@ def create( if not operation_type.strip() or not idempotency_key.strip(): raise ValueError("operation_type and idempotency_key must not be empty") + _validate_payload_keys(payload) operation_id = operation_id or str(uuid.uuid4()) timestamp = _utc(now) payload_json = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")) diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 4cd7c7d..6d76430 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -110,9 +110,9 @@ def test_checkpoint_events_store_only_sanitized_summary(self) -> None: def test_diagnostic_export_is_bounded_and_redacted(self) -> None: operation = self.repository.create( - operation_type="copy", - idempotency_key="copy-1", - payload={"plan_digest": "secret-plan", "access_token": "secret-token"}, + operation_type="copy", + idempotency_key="copy-1", + payload={"plan_digest": "secret-plan", "private_value": "secret-token"}, now=self.now, ) self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) @@ -128,7 +128,7 @@ def test_diagnostic_export_is_bounded_and_redacted(self) -> None: diagnostic = self.repository.diagnostic(operation.operation_id, event_limit=2) self.assertEqual(diagnostic["operation_id"], operation.operation_id) - self.assertEqual(diagnostic["payload_keys"], ["access_token", "plan_digest"]) + self.assertEqual(diagnostic["payload_keys"], ["plan_digest", "private_value"]) self.assertEqual(diagnostic["checkpoint"]["confirmed_occurrences_count"], 1) self.assertTrue(diagnostic["events_truncated"]) self.assertEqual(len(diagnostic["events"]), 2) @@ -142,7 +142,7 @@ def test_diagnostics_list_is_bounded_and_redacted(self) -> None: self.repository.create( operation_type="copy", idempotency_key=f"diagnostic-{index}", - payload={"access_token": f"secret-{index}"}, + payload={"private_value": f"secret-{index}"}, now=self.now + timedelta(seconds=index), ) @@ -150,12 +150,21 @@ def test_diagnostics_list_is_bounded_and_redacted(self) -> None: self.assertEqual(len(diagnostics), 2) self.assertTrue(all("payload_keys" in item for item in diagnostics)) - self.assertTrue(all("access_token" in item["payload_keys"] for item in diagnostics)) + self.assertTrue(all("private_value" in item["payload_keys"] for item in diagnostics)) self.assertNotIn("secret-", str(diagnostics)) with self.assertRaises(ValueError): self.repository.diagnostics(limit=0) + def test_operation_payload_rejects_credential_named_fields(self) -> None: + with self.assertRaises(ValueError): + self.repository.create( + operation_type="copy", + idempotency_key="credential-payload", + payload={"access_token": "must-not-persist"}, + now=self.now, + ) + def test_only_lease_owner_can_checkpoint(self) -> None: operation = self.repository.create( operation_type="import", From 82386aeed20ea1154bd80c2901eec44e7b91c70f Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:42:30 +0200 Subject: [PATCH 059/167] feat: add consistent runtime SQLite backups --- docs/development/implementation-baseline.md | 3 +- src/symphonia/runtime/resources.py | 28 +++++++++++++++- tests/test_runtime_resources.py | 36 +++++++++++++++++++++ 3 files changed, 65 insertions(+), 2 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 0b79169..fa56a61 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -39,6 +39,7 @@ The first implementation increment is intentionally narrower than any provider o - Experimental Home Assistant App metadata scaffold with Ingress-only management and `/data` persistence. - Explicit runtime resource composition and reverse-order shutdown for all durable repositories. - Readiness can validate every composed durable store instead of only the operation queue. +- Runtime resources provide a consistent SQLite online-backup helper while the service remains open. - Ingress-relative health/version routing with normalized, traversal-safe base paths. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. @@ -64,7 +65,7 @@ docker run --rm -p 8099:8099 -v symphonia-data:/data symphonia:dev The container exposes only the current health/readiness/version surface. A future App manifest must add Ingress, Supervisor metadata, supported architectures, backup declarations, and any direct callback policy only after the runtime SDD blockers are resolved. - Deterministic `unittest` coverage under `tests/`. -The package is an implementation foundation, not a claim that the corresponding capability SDDs are complete. Provider OAuth, secret storage, user authentication, Ingress UI, a complete migration ledger beyond the current forward-compatible SQLite path, backup/restore, and production runtime handler wiring remain unimplemented and blocked by their SDD decisions. +The package is an implementation foundation, not a claim that the corresponding capability SDDs are complete. Provider OAuth, secret storage, user authentication, Ingress UI, a complete migration ledger beyond the current forward-compatible SQLite path, backup retention/restore policy, and production runtime handler wiring remain unimplemented and blocked by their SDD decisions. ## Local verification diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 223fd8a..8f202ab 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -3,6 +3,8 @@ from __future__ import annotations from dataclasses import dataclass +from pathlib import Path +import sqlite3 from symphonia.infrastructure import ( AuthorizationAttemptRepository, @@ -30,6 +32,7 @@ class RuntimeResources: authorization: AuthorizationAttemptRepository projections: PlaylistProjectionRepository resolutions: ResolutionDecisionRepository + database_path: str = ":memory:" @classmethod def open(cls, database_path: str) -> "RuntimeResources": @@ -53,7 +56,7 @@ def open(cls, database_path: str) -> "RuntimeResources": for repository in reversed(opened): repository.close() # type: ignore[attr-defined] raise - return cls(operations, plans, connections, authorization, projections, resolutions) + return cls(operations, plans, connections, authorization, projections, resolutions, database_path) def close(self) -> None: """Close repositories in reverse dependency/startup order.""" @@ -85,5 +88,28 @@ def healthcheck(self) -> bool: return False return True + def backup_to(self, destination_path: str) -> None: + """Create a consistent SQLite backup of every composed store. + + All repositories share the operation repository's SQLite file. The + online-backup API produces a transactionally consistent copy while + allowing the runtime to keep serving reads and writes. A destination + must be distinct from the live database; callers decide where the + resulting backup should be retained. + """ + + if not destination_path.strip(): + raise ValueError("destination_path must not be empty") + if self.database_path != ":memory:" and destination_path != ":memory:": + if Path(self.database_path).expanduser().resolve() == Path(destination_path).expanduser().resolve(): + raise ValueError("destination_path must differ from the live database") + + destination = sqlite3.connect(destination_path) + try: + self.operations._connection.backup(destination) # type: ignore[attr-defined] + destination.commit() + finally: + destination.close() + __all__ = ["RuntimeResources"] diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index 59f159d..ce6c031 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -3,7 +3,9 @@ from pathlib import Path import tempfile import unittest +from datetime import datetime, timezone +from symphonia.infrastructure import OperationRepository from symphonia.runtime import RuntimeResources @@ -36,6 +38,40 @@ def test_empty_database_path_is_rejected_before_opening_stores(self) -> None: with self.assertRaises(ValueError): RuntimeResources.open(" ") + def test_backup_to_copies_a_consistent_database(self) -> None: + with tempfile.TemporaryDirectory() as directory: + source_path = str(Path(directory) / "symphonia.sqlite3") + backup_path = str(Path(directory) / "backup.sqlite3") + resources = RuntimeResources.open(source_path) + try: + created = resources.operations.create( + operation_type="test", + idempotency_key="backup-key", + payload={"value": "persisted"}, + now=datetime(2026, 9, 20, tzinfo=timezone.utc), + ) + resources.backup_to(backup_path) + finally: + resources.close() + + backup = OperationRepository(backup_path) + try: + restored = backup.get(created.operation_id) + self.assertEqual(restored.payload, {"value": "persisted"}) + self.assertEqual(len(backup.events(created.operation_id)), 1) + finally: + backup.close() + + def test_backup_to_rejects_the_live_database(self) -> None: + with tempfile.TemporaryDirectory() as directory: + source_path = str(Path(directory) / "symphonia.sqlite3") + resources = RuntimeResources.open(source_path) + try: + with self.assertRaises(ValueError): + resources.backup_to(source_path) + finally: + resources.close() + if __name__ == "__main__": unittest.main() From 05d8d93255d3e03f9544efcaf27929f103060c20 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:43:30 +0200 Subject: [PATCH 060/167] feat: add aggregate operation queue summaries --- docs/development/implementation-baseline.md | 1 + .../infrastructure/sqlite_operations.py | 33 ++++++++++++++++ tests/test_sqlite_operations.py | 38 +++++++++++++++++++ 3 files changed, 72 insertions(+) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index fa56a61..efd5dbb 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -14,6 +14,7 @@ The first implementation increment is intentionally narrower than any provider o - Append-only SQLite operation event history for auditable transitions with sanitized checkpoint summaries. - Bounded redacted operation diagnostics for support and a future authenticated operations view. - Bounded recent-operation diagnostics listing that exposes only redacted support views. +- Aggregate operation queue summaries expose state counts and eligibility without payload data. - Explicit provider rate-limit waits with absolute retry times and durable eligibility. - Bounded transient retry budgets for copy and import handlers with durable exhaustion outcomes. - Lease-expiry recovery is audited distinctly from first claims, with prior worker identity retained only as metadata. diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 08a8f13..86c7bf0 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -274,6 +274,39 @@ def diagnostics(self, *, limit: int = 50, event_limit: int = 20) -> tuple[dict[s ).fetchall() return tuple(self.diagnostic(row["operation_id"], event_limit=event_limit) for row in rows) + def queue_summary(self, *, now: datetime) -> dict[str, Any]: + """Return aggregate queue health without exposing operation payloads.""" + + now_text = _utc(now) + state_rows = self._connection.execute( + "SELECT state, COUNT(*) AS count FROM operations GROUP BY state" + ).fetchall() + states = {str(row["state"]): int(row["count"]) for row in state_rows} + eligible = self._connection.execute( + """ + SELECT COUNT(*) AS count + FROM operations + WHERE cancel_requested = 0 + AND ( + state = 'queued' + OR (state IN ('retry_scheduled', 'waiting_rate_limit') + AND next_run_at IS NOT NULL AND next_run_at <= ?) + OR (state = 'running' + AND (lease_expires_at IS NULL OR lease_expires_at <= ?)) + ) + """, + (now_text, now_text), + ).fetchone() + cancellation_rows = self._connection.execute( + "SELECT COUNT(*) AS count FROM operations WHERE cancel_requested = 1" + ).fetchone() + return { + "total": sum(states.values()), + "states": states, + "eligible_count": int(eligible["count"]), + "cancellation_requested_count": int(cancellation_rows["count"]), + } + def claim( self, operation_id: str, diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 6d76430..3a1ae9c 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -156,6 +156,44 @@ def test_diagnostics_list_is_bounded_and_redacted(self) -> None: with self.assertRaises(ValueError): self.repository.diagnostics(limit=0) + def test_queue_summary_is_aggregate_and_counts_only_eligible_work(self) -> None: + queued = self.repository.create( + operation_type="copy", + idempotency_key="summary-queued", + payload={"plan_digest": "opaque"}, + now=self.now, + ) + running = self.repository.create( + operation_type="copy", + idempotency_key="summary-running", + payload={}, + now=self.now, + ) + self.repository.claim(running.operation_id, worker_id="worker-a", now=self.now, lease_seconds=60) + retry = self.repository.create( + operation_type="import", + idempotency_key="summary-retry", + payload={}, + now=self.now, + ) + self.repository.claim(retry.operation_id, worker_id="worker-a", now=self.now) + self.repository.schedule_retry( + retry.operation_id, + worker_id="worker-a", + next_run_at=self.now - timedelta(seconds=1), + checkpoint={"private": "not returned"}, + now=self.now, + ) + + summary = self.repository.queue_summary(now=self.now) + + self.assertEqual(summary["total"], 3) + self.assertEqual(summary["states"], {"queued": 1, "retry_scheduled": 1, "running": 1}) + self.assertEqual(summary["eligible_count"], 2) + self.assertEqual(summary["cancellation_requested_count"], 0) + self.assertNotIn("opaque", str(summary)) + self.assertNotIn("private", str(summary)) + def test_operation_payload_rejects_credential_named_fields(self) -> None: with self.assertRaises(ValueError): self.repository.create( From abd84abb0f1b681635d7a307f62067cfb6862ade Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:45:05 +0200 Subject: [PATCH 061/167] feat: validate runtime configuration before startup --- docs/development/implementation-baseline.md | 1 + src/symphonia/__main__.py | 15 ++++- src/symphonia/runtime/__init__.py | 3 +- src/symphonia/runtime/config.py | 64 +++++++++++++++++++++ tests/test_runtime_config.py | 52 +++++++++++++++++ 5 files changed, 131 insertions(+), 4 deletions(-) create mode 100644 src/symphonia/runtime/config.py create mode 100644 tests/test_runtime_config.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index efd5dbb..c7718dd 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -41,6 +41,7 @@ The first implementation increment is intentionally narrower than any provider o - Explicit runtime resource composition and reverse-order shutdown for all durable repositories. - Readiness can validate every composed durable store instead of only the operation queue. - Runtime resources provide a consistent SQLite online-backup helper while the service remains open. +- Runtime configuration validates host, port, database path, and Ingress base path before startup. - Ingress-relative health/version routing with normalized, traversal-safe base paths. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. diff --git a/src/symphonia/__main__.py b/src/symphonia/__main__.py index 68a4f41..0fd3913 100644 --- a/src/symphonia/__main__.py +++ b/src/symphonia/__main__.py @@ -5,13 +5,13 @@ import argparse import os -from symphonia.runtime import create_server +from symphonia.runtime import RuntimeConfig, create_server def main() -> None: parser = argparse.ArgumentParser(description="Run the Symphonia runtime foundation") parser.add_argument("--host", default=os.getenv("SYMPHONIA_HOST", "127.0.0.1")) - parser.add_argument("--port", type=int, default=int(os.getenv("SYMPHONIA_PORT", "8099"))) + parser.add_argument("--port", default=os.getenv("SYMPHONIA_PORT", "8099")) parser.add_argument( "--database", default=os.getenv("SYMPHONIA_DATABASE", "./symphonia.sqlite3"), @@ -23,7 +23,16 @@ def main() -> None: help="Ingress base path, for example /local_symphonia", ) args = parser.parse_args() - server = create_server(args.host, args.port, args.database, args.ingress_path) + try: + config = RuntimeConfig( + host=args.host, + port=args.port, + database_path=args.database, + ingress_path=args.ingress_path, + ) + except ValueError as error: + parser.error(str(error)) + server = create_server(config.host, config.port, config.database_path, config.ingress_path) try: server.serve_forever() except KeyboardInterrupt: diff --git a/src/symphonia/runtime/__init__.py b/src/symphonia/runtime/__init__.py index 5ac53a5..8577ebf 100644 --- a/src/symphonia/runtime/__init__.py +++ b/src/symphonia/runtime/__init__.py @@ -1,6 +1,7 @@ """Minimal process runtime and health endpoints.""" from .http import create_server +from .config import RuntimeConfig from .resources import RuntimeResources -__all__ = ["RuntimeResources", "create_server"] +__all__ = ["RuntimeConfig", "RuntimeResources", "create_server"] diff --git a/src/symphonia/runtime/config.py b/src/symphonia/runtime/config.py new file mode 100644 index 0000000..907ba88 --- /dev/null +++ b/src/symphonia/runtime/config.py @@ -0,0 +1,64 @@ +"""Validated process configuration for the local and Home Assistant runtimes.""" + +from __future__ import annotations + +from dataclasses import dataclass +import os +from collections.abc import Mapping + + +def _normalize_ingress_path(value: str) -> str: + if not value or not value.startswith("/"): + raise ValueError("ingress path must start with '/'") + normalized = value.rstrip("/") or "/" + if "//" in normalized or "/.." in normalized or "/./" in normalized: + raise ValueError("ingress path contains an unsafe segment") + return normalized + + +def _parse_port(value: object) -> int: + try: + port = int(value) # type: ignore[arg-type] + except (TypeError, ValueError) as error: + raise ValueError("port must be an integer") from error + if not 1 <= port <= 65_535: + raise ValueError("port must be between 1 and 65535") + return port + + +@dataclass(frozen=True, slots=True) +class RuntimeConfig: + """Configuration that is safe to hand to the runtime composition root.""" + + host: str = "127.0.0.1" + port: int = 8099 + database_path: str = "./symphonia.sqlite3" + ingress_path: str = "/" + + def __post_init__(self) -> None: + if not isinstance(self.host, str): + raise ValueError("host must be a non-empty value without whitespace") + host = self.host.strip() + if not host or any(char.isspace() for char in host): + raise ValueError("host must be a non-empty value without whitespace") + object.__setattr__(self, "host", host) + object.__setattr__(self, "port", _parse_port(self.port)) + if not isinstance(self.database_path, str) or not self.database_path.strip(): + raise ValueError("database_path must not be empty") + if "\x00" in self.database_path: + raise ValueError("database_path must not contain NUL bytes") + object.__setattr__(self, "database_path", self.database_path.strip()) + object.__setattr__(self, "ingress_path", _normalize_ingress_path(self.ingress_path)) + + @classmethod + def from_environment(cls, environ: Mapping[str, str] | None = None) -> "RuntimeConfig": + values = os.environ if environ is None else environ + return cls( + host=values.get("SYMPHONIA_HOST", "127.0.0.1"), + port=_parse_port(values.get("SYMPHONIA_PORT", "8099")), + database_path=values.get("SYMPHONIA_DATABASE", "./symphonia.sqlite3"), + ingress_path=values.get("SYMPHONIA_INGRESS_PATH", "/"), + ) + + +__all__ = ["RuntimeConfig"] diff --git a/tests/test_runtime_config.py b/tests/test_runtime_config.py new file mode 100644 index 0000000..ed751bd --- /dev/null +++ b/tests/test_runtime_config.py @@ -0,0 +1,52 @@ +from __future__ import annotations + +import unittest + +from symphonia.runtime import RuntimeConfig + + +class RuntimeConfigTests(unittest.TestCase): + def test_defaults_are_stable_for_local_runtime(self) -> None: + config = RuntimeConfig() + + self.assertEqual(config.host, "127.0.0.1") + self.assertEqual(config.port, 8099) + self.assertEqual(config.database_path, "./symphonia.sqlite3") + self.assertEqual(config.ingress_path, "/") + + def test_environment_values_are_parsed_and_normalized(self) -> None: + config = RuntimeConfig.from_environment( + { + "SYMPHONIA_HOST": " 0.0.0.0 ", + "SYMPHONIA_PORT": "8100", + "SYMPHONIA_DATABASE": " /data/symphonia.sqlite3 ", + "SYMPHONIA_INGRESS_PATH": "/local_symphonia/", + } + ) + + self.assertEqual(config.host, "0.0.0.0") + self.assertEqual(config.port, 8100) + self.assertEqual(config.database_path, "/data/symphonia.sqlite3") + self.assertEqual(config.ingress_path, "/local_symphonia") + + def test_invalid_configuration_fails_before_runtime_start(self) -> None: + invalid_values = ( + {"port": 0}, + {"port": 65_536}, + {"port": "not-a-number"}, + {"host": "host with spaces"}, + {"database_path": " "}, + {"database_path": "bad\x00path"}, + {"ingress_path": "relative"}, + {"ingress_path": "/bad/../path"}, + ) + for values in invalid_values: + with self.subTest(values=values), self.assertRaises(ValueError): + RuntimeConfig(**values) + + with self.assertRaisesRegex(ValueError, "port"): + RuntimeConfig.from_environment({"SYMPHONIA_PORT": "invalid"}) + + +if __name__ == "__main__": + unittest.main() From f6baebdc88b3e64ce70cdb58c8bdfa768b873183 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:46:18 +0200 Subject: [PATCH 062/167] fix: encapsulate durable store readiness checks --- .../infrastructure/sqlite_authorization.py | 9 +++++++++ src/symphonia/infrastructure/sqlite_connections.py | 9 +++++++++ src/symphonia/infrastructure/sqlite_library.py | 9 +++++++++ src/symphonia/infrastructure/sqlite_plans.py | 9 +++++++++ src/symphonia/infrastructure/sqlite_resolutions.py | 10 +++++++++- src/symphonia/runtime/resources.py | 3 ++- tests/test_runtime_resources.py | 13 +++++++++++++ 7 files changed, 60 insertions(+), 2 deletions(-) diff --git a/src/symphonia/infrastructure/sqlite_authorization.py b/src/symphonia/infrastructure/sqlite_authorization.py index b9b4a53..1a56a69 100644 --- a/src/symphonia/infrastructure/sqlite_authorization.py +++ b/src/symphonia/infrastructure/sqlite_authorization.py @@ -45,6 +45,15 @@ def __init__(self, path: str = ":memory:") -> None: def close(self) -> None: self._connection.close() + def healthcheck(self) -> bool: + """Return whether the migrated authorization store can be read.""" + + try: + row = self._connection.execute("SELECT 1 AS healthy").fetchone() + except sqlite3.Error: + return False + return row is not None and row["healthy"] == 1 + def _migrate(self) -> None: self._connection.executescript( """ diff --git a/src/symphonia/infrastructure/sqlite_connections.py b/src/symphonia/infrastructure/sqlite_connections.py index 1201ad6..95beda2 100644 --- a/src/symphonia/infrastructure/sqlite_connections.py +++ b/src/symphonia/infrastructure/sqlite_connections.py @@ -39,6 +39,15 @@ def __init__(self, path: str = ":memory:") -> None: def close(self) -> None: self._connection.close() + def healthcheck(self) -> bool: + """Return whether the migrated connection store can be read.""" + + try: + row = self._connection.execute("SELECT 1 AS healthy").fetchone() + except sqlite3.Error: + return False + return row is not None and row["healthy"] == 1 + def _migrate(self) -> None: self._connection.executescript( """ diff --git a/src/symphonia/infrastructure/sqlite_library.py b/src/symphonia/infrastructure/sqlite_library.py index e0efd1a..be5a8d8 100644 --- a/src/symphonia/infrastructure/sqlite_library.py +++ b/src/symphonia/infrastructure/sqlite_library.py @@ -51,6 +51,15 @@ def __init__(self, path: str = ":memory:") -> None: def close(self) -> None: self._connection.close() + def healthcheck(self) -> bool: + """Return whether the migrated projection store can be read.""" + + try: + row = self._connection.execute("SELECT 1 AS healthy").fetchone() + except sqlite3.Error: + return False + return row is not None and row["healthy"] == 1 + def _migrate(self) -> None: self._connection.executescript( """ diff --git a/src/symphonia/infrastructure/sqlite_plans.py b/src/symphonia/infrastructure/sqlite_plans.py index 1dfd10c..3c56a29 100644 --- a/src/symphonia/infrastructure/sqlite_plans.py +++ b/src/symphonia/infrastructure/sqlite_plans.py @@ -48,6 +48,15 @@ def __init__(self, path: str = ":memory:") -> None: def close(self) -> None: self._connection.close() + def healthcheck(self) -> bool: + """Return whether the migrated plan store can be read.""" + + try: + row = self._connection.execute("SELECT 1 AS healthy").fetchone() + except sqlite3.Error: + return False + return row is not None and row["healthy"] == 1 + def _migrate(self) -> None: self._connection.executescript( """ diff --git a/src/symphonia/infrastructure/sqlite_resolutions.py b/src/symphonia/infrastructure/sqlite_resolutions.py index b5fe9e7..f33f258 100644 --- a/src/symphonia/infrastructure/sqlite_resolutions.py +++ b/src/symphonia/infrastructure/sqlite_resolutions.py @@ -21,6 +21,15 @@ def __init__(self, path: str = ":memory:") -> None: def close(self) -> None: self._connection.close() + def healthcheck(self) -> bool: + """Return whether the migrated decision store can be read.""" + + try: + row = self._connection.execute("SELECT 1 AS healthy").fetchone() + except sqlite3.Error: + return False + return row is not None and row["healthy"] == 1 + def _migrate(self) -> None: self._connection.executescript( """ @@ -100,4 +109,3 @@ def latest(self, provider_track_key: str, candidate_recording_id: str | None) -> def count(self) -> int: return int(self._connection.execute("SELECT COUNT(*) FROM resolution_decisions").fetchone()[0]) - diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 8f202ab..91ae7f0 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -83,7 +83,8 @@ def healthcheck(self) -> bool: self.projections, self.resolutions, ): - repository._connection.execute("SELECT 1").fetchone() # type: ignore[attr-defined] + if not repository.healthcheck(): + return False except Exception: return False return True diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index ce6c031..5291649 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -38,6 +38,19 @@ def test_empty_database_path_is_rejected_before_opening_stores(self) -> None: with self.assertRaises(ValueError): RuntimeResources.open(" ") + def test_readiness_fails_closed_when_one_store_is_closed(self) -> None: + with tempfile.TemporaryDirectory() as directory: + resources = RuntimeResources.open(str(Path(directory) / "symphonia.sqlite3")) + try: + resources.plans.close() + self.assertFalse(resources.healthcheck()) + finally: + resources.resolutions.close() + resources.projections.close() + resources.authorization.close() + resources.connections.close() + resources.operations.close() + def test_backup_to_copies_a_consistent_database(self) -> None: with tempfile.TemporaryDirectory() as directory: source_path = str(Path(directory) / "symphonia.sqlite3") From 1178f9bbf893ca2ef31d85ca25144d7f6ce2f186 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:46:56 +0200 Subject: [PATCH 063/167] fix: fail cleanly on invalid ingress configuration --- src/symphonia/runtime/config.py | 2 +- tests/test_runtime_config.py | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/src/symphonia/runtime/config.py b/src/symphonia/runtime/config.py index 907ba88..a4c5afd 100644 --- a/src/symphonia/runtime/config.py +++ b/src/symphonia/runtime/config.py @@ -8,7 +8,7 @@ def _normalize_ingress_path(value: str) -> str: - if not value or not value.startswith("/"): + if not isinstance(value, str) or not value or not value.startswith("/"): raise ValueError("ingress path must start with '/'") normalized = value.rstrip("/") or "/" if "//" in normalized or "/.." in normalized or "/./" in normalized: diff --git a/tests/test_runtime_config.py b/tests/test_runtime_config.py index ed751bd..e718198 100644 --- a/tests/test_runtime_config.py +++ b/tests/test_runtime_config.py @@ -39,6 +39,7 @@ def test_invalid_configuration_fails_before_runtime_start(self) -> None: {"database_path": "bad\x00path"}, {"ingress_path": "relative"}, {"ingress_path": "/bad/../path"}, + {"ingress_path": 1}, ) for values in invalid_values: with self.subTest(values=values), self.assertRaises(ValueError): From b744405af6c7444c85e8ffb8031b970b6c727a90 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:53:26 +0200 Subject: [PATCH 064/167] feat: expose queue age and expired lease diagnostics --- docs/development/implementation-baseline.md | 2 +- .../infrastructure/sqlite_operations.py | 28 ++++++++++++++++++- tests/test_sqlite_operations.py | 21 ++++++++++++++ 3 files changed, 49 insertions(+), 2 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index c7718dd..6d3d270 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -14,7 +14,7 @@ The first implementation increment is intentionally narrower than any provider o - Append-only SQLite operation event history for auditable transitions with sanitized checkpoint summaries. - Bounded redacted operation diagnostics for support and a future authenticated operations view. - Bounded recent-operation diagnostics listing that exposes only redacted support views. -- Aggregate operation queue summaries expose state counts and eligibility without payload data. +- Aggregate operation queue summaries expose state counts, eligible age, and expired leases without payload data. - Explicit provider rate-limit waits with absolute retry times and durable eligibility. - Bounded transient retry budgets for copy and import handlers with durable exhaustion outcomes. - Lease-expiry recovery is audited distinctly from first claims, with prior worker identity retained only as metadata. diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 86c7bf0..80a04ba 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -284,7 +284,13 @@ def queue_summary(self, *, now: datetime) -> dict[str, Any]: states = {str(row["state"]): int(row["count"]) for row in state_rows} eligible = self._connection.execute( """ - SELECT COUNT(*) AS count + SELECT COUNT(*) AS count, + MIN( + CASE WHEN state = 'running' + THEN COALESCE(lease_expires_at, created_at) + ELSE COALESCE(next_run_at, created_at) + END + ) AS oldest_eligible_at FROM operations WHERE cancel_requested = 0 AND ( @@ -297,13 +303,33 @@ def queue_summary(self, *, now: datetime) -> dict[str, Any]: """, (now_text, now_text), ).fetchone() + expired_leases = self._connection.execute( + """ + SELECT COUNT(*) AS count + FROM operations + WHERE cancel_requested = 0 + AND state = 'running' + AND (lease_expires_at IS NULL OR lease_expires_at <= ?) + """, + (now_text,), + ).fetchone() cancellation_rows = self._connection.execute( "SELECT COUNT(*) AS count FROM operations WHERE cancel_requested = 1" ).fetchone() + oldest_eligible_at = eligible["oldest_eligible_at"] + oldest_eligible_age_seconds = None + if oldest_eligible_at is not None: + oldest_eligible_age_seconds = max( + 0, + int((now - _parse_utc(oldest_eligible_at)).total_seconds()), + ) return { "total": sum(states.values()), "states": states, "eligible_count": int(eligible["count"]), + "expired_lease_count": int(expired_leases["count"]), + "oldest_eligible_at": oldest_eligible_at, + "oldest_eligible_age_seconds": oldest_eligible_age_seconds, "cancellation_requested_count": int(cancellation_rows["count"]), } diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 3a1ae9c..32066d1 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -190,10 +190,31 @@ def test_queue_summary_is_aggregate_and_counts_only_eligible_work(self) -> None: self.assertEqual(summary["total"], 3) self.assertEqual(summary["states"], {"queued": 1, "retry_scheduled": 1, "running": 1}) self.assertEqual(summary["eligible_count"], 2) + self.assertEqual(summary["expired_lease_count"], 0) + self.assertEqual( + summary["oldest_eligible_at"], + (self.now - timedelta(seconds=1)).isoformat(timespec="microseconds"), + ) + self.assertEqual(summary["oldest_eligible_age_seconds"], 1) self.assertEqual(summary["cancellation_requested_count"], 0) self.assertNotIn("opaque", str(summary)) self.assertNotIn("private", str(summary)) + def test_queue_summary_counts_expired_leases(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="summary-expired", + payload={}, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now, lease_seconds=1) + + summary = self.repository.queue_summary(now=self.now + timedelta(seconds=2)) + + self.assertEqual(summary["eligible_count"], 1) + self.assertEqual(summary["expired_lease_count"], 1) + self.assertEqual(summary["oldest_eligible_age_seconds"], 1) + def test_operation_payload_rejects_credential_named_fields(self) -> None: with self.assertRaises(ValueError): self.repository.create( From b3ba9dcc609aa0b327173305445f9ae72ac58134 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:54:09 +0200 Subject: [PATCH 065/167] feat: add safe runtime diagnostics view --- docs/development/implementation-baseline.md | 1 + src/symphonia/runtime/resources.py | 27 ++++++++++++++ tests/test_runtime_resources.py | 40 +++++++++++++++++++++ 3 files changed, 68 insertions(+) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 6d3d270..d0ecbef 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -42,6 +42,7 @@ The first implementation increment is intentionally narrower than any provider o - Readiness can validate every composed durable store instead of only the operation queue. - Runtime resources provide a consistent SQLite online-backup helper while the service remains open. - Runtime configuration validates host, port, database path, and Ingress base path before startup. +- Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. - Ingress-relative health/version routing with normalized, traversal-safe base paths. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 91ae7f0..71367ed 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -3,8 +3,10 @@ from __future__ import annotations from dataclasses import dataclass +from datetime import datetime from pathlib import Path import sqlite3 +from typing import Any from symphonia.infrastructure import ( AuthorizationAttemptRepository, @@ -112,5 +114,30 @@ def backup_to(self, destination_path: str) -> None: finally: destination.close() + def diagnostics( + self, + *, + now: datetime, + operation_limit: int = 50, + event_limit: int = 20, + ) -> dict[str, Any]: + """Return a bounded support view without database paths or payloads.""" + + if operation_limit <= 0: + raise ValueError("operation_limit must be positive") + if event_limit <= 0: + raise ValueError("event_limit must be positive") + ready = self.healthcheck() + if not ready: + return {"ready": False} + return { + "ready": True, + "queue": self.operations.queue_summary(now=now), + "operations": self.operations.diagnostics( + limit=operation_limit, + event_limit=event_limit, + ), + } + __all__ = ["RuntimeResources"] diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index 5291649..5fe3a1a 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -85,6 +85,46 @@ def test_backup_to_rejects_the_live_database(self) -> None: finally: resources.close() + def test_diagnostics_combine_safe_queue_and_operation_views(self) -> None: + with tempfile.TemporaryDirectory() as directory: + resources = RuntimeResources.open(str(Path(directory) / "symphonia.sqlite3")) + try: + resources.operations.create( + operation_type="test", + idempotency_key="diagnostic-key", + payload={"plan_digest": "must-not-appear"}, + now=datetime(2026, 9, 20, tzinfo=timezone.utc), + ) + diagnostics = resources.diagnostics( + now=datetime(2026, 9, 20, 0, 0, 1, tzinfo=timezone.utc), + operation_limit=1, + event_limit=1, + ) + self.assertTrue(diagnostics["ready"]) + self.assertEqual(diagnostics["queue"]["total"], 1) + self.assertEqual(len(diagnostics["operations"]), 1) + self.assertNotIn("must-not-appear", str(diagnostics)) + with self.assertRaises(ValueError): + resources.diagnostics(now=datetime(2026, 9, 20, tzinfo=timezone.utc), operation_limit=0) + finally: + resources.close() + + def test_diagnostics_fail_closed_when_a_store_is_unavailable(self) -> None: + with tempfile.TemporaryDirectory() as directory: + resources = RuntimeResources.open(str(Path(directory) / "symphonia.sqlite3")) + resources.plans.close() + try: + self.assertEqual( + resources.diagnostics(now=datetime(2026, 9, 20, tzinfo=timezone.utc)), + {"ready": False}, + ) + finally: + resources.resolutions.close() + resources.projections.close() + resources.authorization.close() + resources.connections.close() + resources.operations.close() + if __name__ == "__main__": unittest.main() From 426190dd8b8b741e9e1702d4fb10fb84f205c6b7 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:55:24 +0200 Subject: [PATCH 066/167] feat: add safe provider connection health summaries --- docs/development/implementation-baseline.md | 1 + .../infrastructure/sqlite_connections.py | 21 ++++++++++++++++++ src/symphonia/runtime/resources.py | 1 + tests/test_runtime_resources.py | 1 + tests/test_sqlite_connections.py | 22 +++++++++++++++++++ 5 files changed, 46 insertions(+) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index d0ecbef..feb1d1d 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -43,6 +43,7 @@ The first implementation increment is intentionally narrower than any provider o - Runtime resources provide a consistent SQLite online-backup helper while the service remains open. - Runtime configuration validates host, port, database path, and Ingress base path before startup. - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. +- Provider connection diagnostics expose only provider/state counts, never account or credential data. - Ingress-relative health/version routing with normalized, traversal-safe base paths. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. diff --git a/src/symphonia/infrastructure/sqlite_connections.py b/src/symphonia/infrastructure/sqlite_connections.py index 95beda2..a01c309 100644 --- a/src/symphonia/infrastructure/sqlite_connections.py +++ b/src/symphonia/infrastructure/sqlite_connections.py @@ -5,6 +5,7 @@ from datetime import datetime, timezone import json import sqlite3 +from typing import Any from symphonia.providers.connections import ConnectionState, ProviderConnection from symphonia.providers.contracts import Capability, ProviderCapabilities @@ -133,6 +134,26 @@ def list(self, *, provider: str | None = None) -> tuple[ProviderConnection, ...] ).fetchall() return tuple(self._record(row) for row in rows) + def health_summary(self) -> dict[str, Any]: + """Return provider/state counts without account or credential data.""" + + rows = self._connection.execute( + """ + SELECT provider, state, COUNT(*) AS count + FROM provider_connections + GROUP BY provider, state + ORDER BY provider, state + """ + ).fetchall() + by_provider: dict[str, dict[str, int]] = {} + total = 0 + for row in rows: + provider = str(row["provider"]) + count = int(row["count"]) + by_provider.setdefault(provider, {})[str(row["state"])] = count + total += count + return {"total": total, "by_provider": by_provider} + def record_probe( self, connection_id: str, diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 71367ed..76cb5ed 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -133,6 +133,7 @@ def diagnostics( return { "ready": True, "queue": self.operations.queue_summary(now=now), + "connections": self.connections.health_summary(), "operations": self.operations.diagnostics( limit=operation_limit, event_limit=event_limit, diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index 5fe3a1a..4f02382 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -102,6 +102,7 @@ def test_diagnostics_combine_safe_queue_and_operation_views(self) -> None: ) self.assertTrue(diagnostics["ready"]) self.assertEqual(diagnostics["queue"]["total"], 1) + self.assertEqual(diagnostics["connections"], {"total": 0, "by_provider": {}}) self.assertEqual(len(diagnostics["operations"]), 1) self.assertNotIn("must-not-appear", str(diagnostics)) with self.assertRaises(ValueError): diff --git a/tests/test_sqlite_connections.py b/tests/test_sqlite_connections.py index 8d91d20..e54dd99 100644 --- a/tests/test_sqlite_connections.py +++ b/tests/test_sqlite_connections.py @@ -85,6 +85,28 @@ def test_disconnect_erases_local_secret_reference_and_capabilities(self) -> None self.assertIsNone(disconnected.capabilities) self.assertEqual(self.repository.list(provider="spotify")[0].health_code, "disconnected") + def test_health_summary_contains_only_provider_state_counts(self) -> None: + self.repository.create(connection()) + self.repository.create( + ProviderConnection( + connection_id="spotify-2", + provider="spotify", + provider_account_id="account-2", + state=ConnectionState.ACTION_REQUIRED, + manifest_version="v1", + secret_ref="opaque-secret-2", + capabilities=None, + created_at=NOW, + updated_at=NOW, + ) + ) + + summary = self.repository.health_summary() + + self.assertEqual(summary, {"total": 2, "by_provider": {"spotify": {"action_required": 1, "connected": 1}}}) + self.assertNotIn("account-1", str(summary)) + self.assertNotIn("opaque-secret", str(summary)) + def test_missing_connection_is_explicit(self) -> None: with self.assertRaises(ConnectionNotFound): self.repository.get("missing") From 1f89c14276c1949bd1f445fcc3f4a937d76ab10b Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:56:01 +0200 Subject: [PATCH 067/167] fix: fail closed during runtime diagnostics races --- src/symphonia/runtime/resources.py | 26 +++++++++++++++----------- tests/test_runtime_resources.py | 18 ++++++++++++++++++ 2 files changed, 33 insertions(+), 11 deletions(-) diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 76cb5ed..4b13a88 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -127,18 +127,22 @@ def diagnostics( raise ValueError("operation_limit must be positive") if event_limit <= 0: raise ValueError("event_limit must be positive") - ready = self.healthcheck() - if not ready: + try: + if not self.healthcheck(): + return {"ready": False} + return { + "ready": True, + "queue": self.operations.queue_summary(now=now), + "connections": self.connections.health_summary(), + "operations": self.operations.diagnostics( + limit=operation_limit, + event_limit=event_limit, + ), + } + except Exception: + # A concurrent close or adapter failure must not expose internals + # or make a support endpoint look healthier than the stores are. return {"ready": False} - return { - "ready": True, - "queue": self.operations.queue_summary(now=now), - "connections": self.connections.health_summary(), - "operations": self.operations.diagnostics( - limit=operation_limit, - event_limit=event_limit, - ), - } __all__ = ["RuntimeResources"] diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index 4f02382..56576f1 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -126,6 +126,24 @@ def test_diagnostics_fail_closed_when_a_store_is_unavailable(self) -> None: resources.connections.close() resources.operations.close() + def test_diagnostics_fail_closed_if_a_store_fails_after_readiness(self) -> None: + with tempfile.TemporaryDirectory() as directory: + resources = RuntimeResources.open(str(Path(directory) / "symphonia.sqlite3")) + try: + original = resources.operations.queue_summary + + def fail_after_readiness(*, now): + raise RuntimeError("internal database detail") + + resources.operations.queue_summary = fail_after_readiness # type: ignore[method-assign] + self.assertEqual( + resources.diagnostics(now=datetime(2026, 9, 20, tzinfo=timezone.utc)), + {"ready": False}, + ) + resources.operations.queue_summary = original # type: ignore[method-assign] + finally: + resources.close() + if __name__ == "__main__": unittest.main() From 5d754b7212d4d6c5a2d62c687ee72a5ff7d6fd2e Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 19:56:38 +0200 Subject: [PATCH 068/167] test: guard Home Assistant container contract --- tests/test_homeassistant_app_metadata.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/tests/test_homeassistant_app_metadata.py b/tests/test_homeassistant_app_metadata.py index d7feb46..2631a22 100644 --- a/tests/test_homeassistant_app_metadata.py +++ b/tests/test_homeassistant_app_metadata.py @@ -24,6 +24,14 @@ def test_metadata_declares_only_the_architectures_of_the_experimental_matrix(sel self.assertIn(" - aarch64", config) self.assertNotIn(" - armv7", config) + def test_container_contract_is_persistent_non_root_and_probeable(self) -> None: + dockerfile = (ROOT / "Dockerfile").read_text(encoding="utf-8") + self.assertIn("SYMPHONIA_DATABASE=/data/symphonia.sqlite3", dockerfile) + self.assertIn("USER symphonia", dockerfile) + self.assertIn('VOLUME ["/data"]', dockerfile) + self.assertIn("HEALTHCHECK", dockerfile) + self.assertIn("/ready", dockerfile) + if __name__ == "__main__": unittest.main() From 6090035f466c69cd6afd61ccbdb92db30404d9e2 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 20:00:48 +0200 Subject: [PATCH 069/167] feat: add safe playlist projection summaries --- docs/development/implementation-baseline.md | 1 + .../infrastructure/sqlite_library.py | 25 +++++++++++++++++++ src/symphonia/runtime/resources.py | 1 + tests/test_runtime_resources.py | 10 ++++++++ tests/test_sqlite_library.py | 16 ++++++++++++ 5 files changed, 53 insertions(+) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index feb1d1d..23d56e2 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -44,6 +44,7 @@ The first implementation increment is intentionally narrower than any provider o - Runtime configuration validates host, port, database path, and Ingress base path before startup. - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. - Provider connection diagnostics expose only provider/state counts, never account or credential data. +- Import diagnostics expose bounded snapshot freshness and availability counts without playlist content. - Ingress-relative health/version routing with normalized, traversal-safe base paths. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. diff --git a/src/symphonia/infrastructure/sqlite_library.py b/src/symphonia/infrastructure/sqlite_library.py index be5a8d8..e06a58d 100644 --- a/src/symphonia/infrastructure/sqlite_library.py +++ b/src/symphonia/infrastructure/sqlite_library.py @@ -5,6 +5,7 @@ from dataclasses import dataclass from datetime import datetime, timezone import sqlite3 +from typing import Any from symphonia.domain.models import EntryClassification, PlaylistSnapshot, SourcePlaylistEntry from symphonia.providers.contracts import ProviderPlaylistEntry @@ -250,6 +251,30 @@ def current(self, *, provider: str, namespace: str, playlist_id: str) -> StoredP ).fetchone() return None if row is None else self.get(row["snapshot_id"]) + def summary(self) -> dict[str, Any]: + """Return bounded import freshness/counts without playlist contents.""" + + snapshots = self._connection.execute( + "SELECT COUNT(*) AS count, MAX(published_at) AS latest_published_at FROM playlist_snapshots" + ).fetchone() + current = self._connection.execute( + "SELECT COUNT(*) AS count FROM current_playlist_snapshots" + ).fetchone() + entries = self._connection.execute( + """ + SELECT COUNT(*) AS count, + COALESCE(SUM(CASE WHEN available = 0 THEN 1 ELSE 0 END), 0) AS unavailable_count + FROM playlist_snapshot_entries + """ + ).fetchone() + return { + "snapshot_count": int(snapshots["count"]), + "current_playlist_count": int(current["count"]), + "entry_count": int(entries["count"]), + "unavailable_entry_count": int(entries["unavailable_count"]), + "latest_published_at": snapshots["latest_published_at"], + } + def _entries(self, snapshot_id: str, provider: str) -> tuple[SourcePlaylistEntry, ...]: rows = self._connection.execute( """ diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 4b13a88..5b91a11 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -134,6 +134,7 @@ def diagnostics( "ready": True, "queue": self.operations.queue_summary(now=now), "connections": self.connections.health_summary(), + "projections": self.projections.summary(), "operations": self.operations.diagnostics( limit=operation_limit, event_limit=event_limit, diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index 56576f1..a07279d 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -103,6 +103,16 @@ def test_diagnostics_combine_safe_queue_and_operation_views(self) -> None: self.assertTrue(diagnostics["ready"]) self.assertEqual(diagnostics["queue"]["total"], 1) self.assertEqual(diagnostics["connections"], {"total": 0, "by_provider": {}}) + self.assertEqual( + diagnostics["projections"], + { + "snapshot_count": 0, + "current_playlist_count": 0, + "entry_count": 0, + "unavailable_entry_count": 0, + "latest_published_at": None, + }, + ) self.assertEqual(len(diagnostics["operations"]), 1) self.assertNotIn("must-not-appear", str(diagnostics)) with self.assertRaises(ValueError): diff --git a/tests/test_sqlite_library.py b/tests/test_sqlite_library.py index 3498e06..dc7a51b 100644 --- a/tests/test_sqlite_library.py +++ b/tests/test_sqlite_library.py @@ -63,6 +63,22 @@ def test_complete_result_retains_provider_metadata_for_future_resolution(self) - self.assertEqual(stored.snapshot.entries[0].provider_track_title, "Song title") self.assertEqual(stored.snapshot.entries[0].source_added_at, "2026-09-20T12:00:00Z") + def test_summary_reports_import_freshness_without_playlist_content(self) -> None: + self.repository.publish( + collect_playlist_pages([page(available=False)]), + snapshot_id="snapshot-summary", + published_at=NOW, + ) + + summary = self.repository.summary() + + self.assertEqual(summary["snapshot_count"], 1) + self.assertEqual(summary["current_playlist_count"], 1) + self.assertEqual(summary["entry_count"], 1) + self.assertEqual(summary["unavailable_entry_count"], 1) + self.assertEqual(summary["latest_published_at"], "2026-09-20T12:00:00.000000+00:00") + self.assertNotIn("playlist-1", str(summary)) + def test_legacy_projection_schema_gets_metadata_columns(self) -> None: with tempfile.TemporaryDirectory() as directory: path = f"{directory}/legacy.sqlite3" From 343fea9f6369e562f2d093a20496e2960c66993a Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 20:01:23 +0200 Subject: [PATCH 070/167] feat: add safe resolution decision summaries --- docs/development/implementation-baseline.md | 1 + .../infrastructure/sqlite_resolutions.py | 15 +++++++++++++++ src/symphonia/runtime/resources.py | 1 + tests/test_identity_resolution.py | 5 +++++ tests/test_runtime_resources.py | 1 + 5 files changed, 23 insertions(+) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 23d56e2..f8500e0 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -45,6 +45,7 @@ The first implementation increment is intentionally narrower than any provider o - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. - Provider connection diagnostics expose only provider/state counts, never account or credential data. - Import diagnostics expose bounded snapshot freshness and availability counts without playlist content. +- Resolution diagnostics expose only manual-decision action counts, never track, actor, or reason data. - Ingress-relative health/version routing with normalized, traversal-safe base paths. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. diff --git a/src/symphonia/infrastructure/sqlite_resolutions.py b/src/symphonia/infrastructure/sqlite_resolutions.py index f33f258..663e362 100644 --- a/src/symphonia/infrastructure/sqlite_resolutions.py +++ b/src/symphonia/infrastructure/sqlite_resolutions.py @@ -6,6 +6,7 @@ import json import sqlite3 import uuid +from typing import Any from symphonia.identity.models import ManualDecision, ManualDecisionAction @@ -109,3 +110,17 @@ def latest(self, provider_track_key: str, candidate_recording_id: str | None) -> def count(self) -> int: return int(self._connection.execute("SELECT COUNT(*) FROM resolution_decisions").fetchone()[0]) + + def summary(self) -> dict[str, Any]: + """Return decision counts without track, candidate, actor, or reason data.""" + + rows = self._connection.execute( + """ + SELECT action, COUNT(*) AS count + FROM resolution_decisions + GROUP BY action + ORDER BY action + """ + ).fetchall() + by_action = {str(row["action"]): int(row["count"]) for row in rows} + return {"total": sum(by_action.values()), "by_action": by_action} diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 5b91a11..a8fff93 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -135,6 +135,7 @@ def diagnostics( "queue": self.operations.queue_summary(now=now), "connections": self.connections.health_summary(), "projections": self.projections.summary(), + "resolutions": self.resolutions.summary(), "operations": self.operations.diagnostics( limit=operation_limit, event_limit=event_limit, diff --git a/tests/test_identity_resolution.py b/tests/test_identity_resolution.py index a9c1358..9daab4a 100644 --- a/tests/test_identity_resolution.py +++ b/tests/test_identity_resolution.py @@ -101,6 +101,11 @@ def test_manual_decisions_are_append_only_and_latest_is_explicit(self) -> None: latest = repository.latest("spotify:connection-1:track-1", "recording-1") self.assertEqual(latest.action, ManualDecisionAction.ACCEPT) self.assertEqual(repository.count(), 2) + self.assertEqual( + repository.summary(), + {"total": 2, "by_action": {"accept": 1, "reject": 1}}, + ) + self.assertNotIn("recording-1", str(repository.summary())) finally: repository.close() diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index a07279d..c65aa4d 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -113,6 +113,7 @@ def test_diagnostics_combine_safe_queue_and_operation_views(self) -> None: "latest_published_at": None, }, ) + self.assertEqual(diagnostics["resolutions"], {"total": 0, "by_action": {}}) self.assertEqual(len(diagnostics["operations"]), 1) self.assertNotIn("must-not-appear", str(diagnostics)) with self.assertRaises(ValueError): From 83df573de2911eb80a835dd224a5ba0b6297ca74 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 20:02:31 +0200 Subject: [PATCH 071/167] fix: redact credentials at the write boundary --- src/symphonia/providers/writing.py | 11 +++++++++-- tests/test_copy_execution.py | 21 ++++++++++++++++++++- 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/src/symphonia/providers/writing.py b/src/symphonia/providers/writing.py index 478d0d9..d20b83a 100644 --- a/src/symphonia/providers/writing.py +++ b/src/symphonia/providers/writing.py @@ -7,6 +7,8 @@ from enum import Enum from typing import Protocol +from .errors import redact_error_detail + class WriteOutcome(str, Enum): CONFIRMED_SUCCESS = "confirmed_success" @@ -32,6 +34,10 @@ class WriteResult: detail: str | None = None retry_at: datetime | None = None + def __post_init__(self) -> None: + if self.detail is not None: + object.__setattr__(self, "detail", redact_error_detail(self.detail)) + class ProviderWriteError(RuntimeError): """A target-creation failure with an explicit retry/reconciliation class.""" @@ -43,9 +49,10 @@ def __init__( provider_code: str | None = None, retry_at: datetime | None = None, ) -> None: - super().__init__(detail) + redacted_detail = redact_error_detail(detail) + super().__init__(redacted_detail) self.outcome = outcome - self.detail = detail + self.detail = redacted_detail self.provider_code = provider_code self.retry_at = retry_at diff --git a/tests/test_copy_execution.py b/tests/test_copy_execution.py index 3b3c26b..bff5571 100644 --- a/tests/test_copy_execution.py +++ b/tests/test_copy_execution.py @@ -6,7 +6,7 @@ from symphonia.application import CopyExecutionService, CopyPlanningService, CopyWorkflowService, OperationRunner from symphonia.domain import CopyPolicy, EntryClassification, PlaylistSnapshot, SourcePlaylistEntry from symphonia.infrastructure import CopyPlanRepository, OperationRepository -from symphonia.providers import TargetPlaylist, WriteOutcome, WriteResult +from symphonia.providers import ProviderWriteError, TargetPlaylist, WriteOutcome, WriteResult NOW = datetime(2026, 9, 20, 12, 0, tzinfo=timezone.utc) @@ -158,6 +158,25 @@ def test_rate_limited_write_waits_until_provider_deadline(self) -> None: self.assertEqual(operation.state, "waiting_rate_limit") self.assertEqual(operation.next_run_at, NOW + timedelta(minutes=2)) + def test_write_boundary_redacts_credentials_in_errors_and_results(self) -> None: + error = ProviderWriteError( + WriteOutcome.PERMANENT_FAILURE, + "Bearer abc123 token=secret refresh_token=refresh-value", + ) + result = WriteResult( + WriteOutcome.PERMANENT_FAILURE, + detail="authorization=header-value password=hunter2", + ) + + for value in (str(error), error.detail, result.detail): + self.assertNotIn("abc123", value) + self.assertNotIn("secret", value) + self.assertNotIn("refresh-value", value) + self.assertNotIn("header-value", value) + self.assertNotIn("hunter2", value) + self.assertIn("[REDACTED]", str(error)) + self.assertIn("[REDACTED]", result.detail) + if __name__ == "__main__": unittest.main() From f92900f492ee93840c772251ffdc6744d57e0eed Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 20:03:23 +0200 Subject: [PATCH 072/167] feat: preflight SQLite backups before restore --- docs/development/implementation-baseline.md | 1 + src/symphonia/runtime/resources.py | 43 +++++++++++++++++++++ tests/test_runtime_resources.py | 13 +++++++ 3 files changed, 57 insertions(+) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index f8500e0..fe5a4d7 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -41,6 +41,7 @@ The first implementation increment is intentionally narrower than any provider o - Explicit runtime resource composition and reverse-order shutdown for all durable repositories. - Readiness can validate every composed durable store instead of only the operation queue. - Runtime resources provide a consistent SQLite online-backup helper while the service remains open. +- Runtime resources can preflight backup integrity and required durable tables read-only before restore design is selected. - Runtime configuration validates host, port, database path, and Ingress base path before startup. - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. - Provider connection diagnostics expose only provider/state counts, never account or credential data. diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index a8fff93..7db41d7 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -7,6 +7,7 @@ from pathlib import Path import sqlite3 from typing import Any +from urllib.parse import quote from symphonia.infrastructure import ( AuthorizationAttemptRepository, @@ -114,6 +115,48 @@ def backup_to(self, destination_path: str) -> None: finally: destination.close() + @classmethod + def validate_backup(cls, backup_path: str) -> bool: + """Validate a backup read-only before a future restore operation.""" + + if not isinstance(backup_path, str) or not backup_path.strip(): + raise ValueError("backup_path must not be empty") + if backup_path == ":memory:": + return False + path = Path(backup_path).expanduser().resolve() + if not path.is_file(): + return False + uri = f"file:{quote(str(path))}?mode=ro" + required_tables = { + "operations", + "operation_events", + "copy_plans", + "provider_connections", + "authorization_attempts", + "playlist_snapshots", + "playlist_snapshot_entries", + "current_playlist_snapshots", + "resolution_decisions", + } + connection: sqlite3.Connection | None = None + try: + connection = sqlite3.connect(uri, uri=True) + integrity = connection.execute("PRAGMA integrity_check").fetchone() + if integrity is None or integrity[0] != "ok": + return False + tables = { + row[0] + for row in connection.execute( + "SELECT name FROM sqlite_master WHERE type = 'table'" + ).fetchall() + } + return required_tables.issubset(tables) + except sqlite3.Error: + return False + finally: + if connection is not None: + connection.close() + def diagnostics( self, *, diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index c65aa4d..82d9dee 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -67,6 +67,8 @@ def test_backup_to_copies_a_consistent_database(self) -> None: finally: resources.close() + self.assertTrue(RuntimeResources.validate_backup(backup_path)) + backup = OperationRepository(backup_path) try: restored = backup.get(created.operation_id) @@ -85,6 +87,17 @@ def test_backup_to_rejects_the_live_database(self) -> None: finally: resources.close() + def test_validate_backup_rejects_missing_or_corrupt_files(self) -> None: + with tempfile.TemporaryDirectory() as directory: + missing = str(Path(directory) / "missing.sqlite3") + corrupt = Path(directory) / "corrupt.sqlite3" + corrupt.write_text("not a sqlite database", encoding="utf-8") + + self.assertFalse(RuntimeResources.validate_backup(missing)) + self.assertFalse(RuntimeResources.validate_backup(str(corrupt))) + with self.assertRaises(ValueError): + RuntimeResources.validate_backup(" ") + def test_diagnostics_combine_safe_queue_and_operation_views(self) -> None: with tempfile.TemporaryDirectory() as directory: resources = RuntimeResources.open(str(Path(directory) / "symphonia.sqlite3")) From 5110af52a4e2c932455e4393a3f21c3680e8a2d5 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 20:03:53 +0200 Subject: [PATCH 073/167] fix: refuse backups from unhealthy runtimes --- src/symphonia/runtime/resources.py | 2 ++ tests/test_runtime_resources.py | 17 +++++++++++++++++ 2 files changed, 19 insertions(+) diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 7db41d7..28aaed2 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -104,6 +104,8 @@ def backup_to(self, destination_path: str) -> None: if not destination_path.strip(): raise ValueError("destination_path must not be empty") + if not self.healthcheck(): + raise RuntimeError("cannot back up an unhealthy runtime") if self.database_path != ":memory:" and destination_path != ":memory:": if Path(self.database_path).expanduser().resolve() == Path(destination_path).expanduser().resolve(): raise ValueError("destination_path must differ from the live database") diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index 82d9dee..59edf81 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -87,6 +87,23 @@ def test_backup_to_rejects_the_live_database(self) -> None: finally: resources.close() + def test_backup_to_fails_closed_when_a_store_is_unhealthy(self) -> None: + with tempfile.TemporaryDirectory() as directory: + source_path = str(Path(directory) / "symphonia.sqlite3") + destination_path = str(Path(directory) / "backup.sqlite3") + resources = RuntimeResources.open(source_path) + try: + resources.plans.close() + with self.assertRaisesRegex(RuntimeError, "unhealthy"): + resources.backup_to(destination_path) + self.assertFalse(Path(destination_path).exists()) + finally: + resources.resolutions.close() + resources.projections.close() + resources.authorization.close() + resources.connections.close() + resources.operations.close() + def test_validate_backup_rejects_missing_or_corrupt_files(self) -> None: with tempfile.TemporaryDirectory() as directory: missing = str(Path(directory) / "missing.sqlite3") From 34ba9cfc2c119f181be648a88b8be850919956ac Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 20:12:50 +0200 Subject: [PATCH 074/167] fix: publish SQLite backups atomically --- src/symphonia/runtime/resources.py | 39 ++++++++++++++++++++++++------ tests/test_runtime_resources.py | 36 +++++++++++++++++++++++++++ 2 files changed, 68 insertions(+), 7 deletions(-) diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 28aaed2..a869a94 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -4,8 +4,10 @@ from dataclasses import dataclass from datetime import datetime +import os from pathlib import Path import sqlite3 +import tempfile from typing import Any from urllib.parse import quote @@ -104,18 +106,41 @@ def backup_to(self, destination_path: str) -> None: if not destination_path.strip(): raise ValueError("destination_path must not be empty") + if self.database_path == ":memory:": + raise ValueError("backups require a persistent database path") + if destination_path == ":memory:": + raise ValueError("destination_path must be a persistent filesystem path") if not self.healthcheck(): raise RuntimeError("cannot back up an unhealthy runtime") - if self.database_path != ":memory:" and destination_path != ":memory:": - if Path(self.database_path).expanduser().resolve() == Path(destination_path).expanduser().resolve(): - raise ValueError("destination_path must differ from the live database") + live_path = Path(self.database_path).expanduser().resolve() + destination_path_object = Path(destination_path).expanduser().resolve() + if live_path == destination_path_object: + raise ValueError("destination_path must differ from the live database") - destination = sqlite3.connect(destination_path) + temporary_path: str | None = None try: - self.operations._connection.backup(destination) # type: ignore[attr-defined] - destination.commit() + with tempfile.NamedTemporaryFile( + mode="wb", + prefix=f".{destination_path_object.name}.", + suffix=".tmp", + dir=destination_path_object.parent, + delete=False, + ) as temporary: + temporary_path = temporary.name + destination = sqlite3.connect(temporary_path) + try: + self.operations._connection.backup(destination) # type: ignore[attr-defined] + destination.commit() + finally: + destination.close() + os.replace(temporary_path, destination_path_object) + temporary_path = None finally: - destination.close() + if temporary_path is not None: + try: + os.unlink(temporary_path) + except FileNotFoundError: + pass @classmethod def validate_backup(cls, backup_path: str) -> bool: diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index 59edf81..c5468e7 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -87,6 +87,42 @@ def test_backup_to_rejects_the_live_database(self) -> None: finally: resources.close() + def test_backup_to_rejects_in_memory_runtime_and_destination(self) -> None: + resources = RuntimeResources.open(":memory:") + try: + with self.assertRaises(ValueError): + resources.backup_to("backup.sqlite3") + finally: + resources.close() + + with tempfile.TemporaryDirectory() as directory: + source_path = str(Path(directory) / "symphonia.sqlite3") + resources = RuntimeResources.open(source_path) + try: + with self.assertRaises(ValueError): + resources.backup_to(":memory:") + finally: + resources.close() + + def test_backup_to_replaces_an_existing_destination_atomically(self) -> None: + with tempfile.TemporaryDirectory() as directory: + source_path = str(Path(directory) / "symphonia.sqlite3") + backup_path = Path(directory) / "backup.sqlite3" + backup_path.write_text("old backup", encoding="utf-8") + resources = RuntimeResources.open(source_path) + try: + resources.operations.create( + operation_type="test", + idempotency_key="atomic-backup", + payload={}, + now=datetime(2026, 9, 20, tzinfo=timezone.utc), + ) + resources.backup_to(str(backup_path)) + finally: + resources.close() + + self.assertTrue(RuntimeResources.validate_backup(str(backup_path))) + def test_backup_to_fails_closed_when_a_store_is_unhealthy(self) -> None: with tempfile.TemporaryDirectory() as directory: source_path = str(Path(directory) / "symphonia.sqlite3") From f1db298621bcf3a5bcf94c06a8436fa326360aa7 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 20:13:41 +0200 Subject: [PATCH 075/167] fix: reject nested credential payloads --- docs/development/implementation-baseline.md | 2 +- .../infrastructure/sqlite_operations.py | 44 ++++++++++++++++--- tests/test_sqlite_operations.py | 20 +++++++++ 3 files changed, 58 insertions(+), 8 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index fe5a4d7..30ebc26 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -29,7 +29,7 @@ The first implementation increment is intentionally narrower than any provider o - Application authorization boundary that generates one-use state without persisting the raw value. - Authorization callback binding rejects unsafe schemes, fragments, credentials, whitespace, and non-loopback HTTP hosts. - Normalized provider errors redact common bearer/token/secret/password/cookie forms at the provider boundary. -- Operation persistence rejects common credential-shaped payload keys before SQLite writes. +- Operation persistence recursively rejects credential-shaped payload keys and cyclic structures before SQLite writes. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. - Copy plans bind target connection and effective write-capability evidence into their digest. - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 80a04ba..cc955b7 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -7,6 +7,7 @@ from __future__ import annotations +from collections.abc import Mapping from dataclasses import dataclass from datetime import datetime, timedelta, timezone import json @@ -43,14 +44,43 @@ class LeaseConflict(RuntimeError): ) -def _validate_payload_keys(payload: dict[str, Any]) -> None: - forbidden = sorted( - str(key) - for key in payload - if _SECRET_PAYLOAD_KEY.search(str(key)) - ) +def _validate_payload_keys(payload: Any) -> None: + forbidden: list[str] = [] + active_containers: set[int] = set() + + def walk(value: Any, path: str = "") -> None: + if isinstance(value, Mapping): + identity = id(value) + if identity in active_containers: + raise ValueError("operation payload must not contain cyclic structures") + active_containers.add(identity) + try: + for key, nested in value.items(): + key_text = str(key) + key_path = key_text if not path else f"{path}.{key_text}" + if _SECRET_PAYLOAD_KEY.search(key_text): + forbidden.append(key_path) + walk(nested, key_path) + finally: + active_containers.remove(identity) + return + if isinstance(value, (list, tuple)): + identity = id(value) + if identity in active_containers: + raise ValueError("operation payload must not contain cyclic structures") + active_containers.add(identity) + try: + for index, nested in enumerate(value): + walk(nested, f"{path}[{index}]") + finally: + active_containers.remove(identity) + + walk(payload) if forbidden: - raise ValueError(f"operation payload contains forbidden credential keys: {', '.join(forbidden)}") + raise ValueError( + "operation payload contains forbidden credential keys: " + + ", ".join(sorted(forbidden)) + ) @dataclass(frozen=True, slots=True) diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 32066d1..f29f809 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -224,6 +224,26 @@ def test_operation_payload_rejects_credential_named_fields(self) -> None: now=self.now, ) + def test_operation_payload_rejects_nested_credentials(self) -> None: + with self.assertRaisesRegex(ValueError, r"provider.credentials\[0\]\.access_token"): + self.repository.create( + operation_type="copy", + idempotency_key="nested-credential-payload", + payload={"provider": {"credentials": [{"access_token": "must-not-persist"}]}}, + now=self.now, + ) + + def test_operation_payload_rejects_cyclic_structures_before_json_encoding(self) -> None: + payload: dict[str, object] = {} + payload["nested"] = payload + with self.assertRaisesRegex(ValueError, "cyclic"): + self.repository.create( + operation_type="copy", + idempotency_key="cyclic-payload", + payload=payload, + now=self.now, + ) + def test_only_lease_owner_can_checkpoint(self) -> None: operation = self.repository.create( operation_type="import", From ede5b9fae3fcd63b6602ffe236af62ec5c7b158c Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 20:14:27 +0200 Subject: [PATCH 076/167] fix: redact provider error codes --- docs/development/implementation-baseline.md | 2 +- src/symphonia/providers/errors.py | 2 ++ src/symphonia/providers/writing.py | 4 +++- tests/test_copy_execution.py | 3 +++ tests/test_provider_connections.py | 7 ++++++- 5 files changed, 15 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 30ebc26..df3c27b 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -28,7 +28,7 @@ The first implementation increment is intentionally narrower than any provider o - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Application authorization boundary that generates one-use state without persisting the raw value. - Authorization callback binding rejects unsafe schemes, fragments, credentials, whitespace, and non-loopback HTTP hosts. -- Normalized provider errors redact common bearer/token/secret/password/cookie forms at the provider boundary. +- Normalized provider errors and provider codes redact common bearer/token/secret/password/cookie forms at the provider boundary. - Operation persistence recursively rejects credential-shaped payload keys and cyclic structures before SQLite writes. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. - Copy plans bind target connection and effective write-capability evidence into their digest. diff --git a/src/symphonia/providers/errors.py b/src/symphonia/providers/errors.py index 112f4d5..2b9138f 100644 --- a/src/symphonia/providers/errors.py +++ b/src/symphonia/providers/errors.py @@ -51,6 +51,8 @@ class ProviderApiError(RuntimeError): def __post_init__(self) -> None: redacted_detail = redact_error_detail(self.detail) object.__setattr__(self, "detail", redacted_detail) + if self.provider_code is not None: + object.__setattr__(self, "provider_code", redact_error_detail(self.provider_code)) RuntimeError.__init__(self, redacted_detail) if not self.detail.strip(): raise ValueError("provider error detail must not be empty") diff --git a/src/symphonia/providers/writing.py b/src/symphonia/providers/writing.py index d20b83a..0ed95d6 100644 --- a/src/symphonia/providers/writing.py +++ b/src/symphonia/providers/writing.py @@ -37,6 +37,8 @@ class WriteResult: def __post_init__(self) -> None: if self.detail is not None: object.__setattr__(self, "detail", redact_error_detail(self.detail)) + if self.provider_code is not None: + object.__setattr__(self, "provider_code", redact_error_detail(self.provider_code)) class ProviderWriteError(RuntimeError): @@ -53,7 +55,7 @@ def __init__( super().__init__(redacted_detail) self.outcome = outcome self.detail = redacted_detail - self.provider_code = provider_code + self.provider_code = None if provider_code is None else redact_error_detail(provider_code) self.retry_at = retry_at diff --git a/tests/test_copy_execution.py b/tests/test_copy_execution.py index bff5571..b1eae32 100644 --- a/tests/test_copy_execution.py +++ b/tests/test_copy_execution.py @@ -165,6 +165,7 @@ def test_write_boundary_redacts_credentials_in_errors_and_results(self) -> None: ) result = WriteResult( WriteOutcome.PERMANENT_FAILURE, + provider_code="token=provider-secret", detail="authorization=header-value password=hunter2", ) @@ -174,8 +175,10 @@ def test_write_boundary_redacts_credentials_in_errors_and_results(self) -> None: self.assertNotIn("refresh-value", value) self.assertNotIn("header-value", value) self.assertNotIn("hunter2", value) + self.assertNotIn("provider-secret", value) self.assertIn("[REDACTED]", str(error)) self.assertIn("[REDACTED]", result.detail) + self.assertIn("[REDACTED]", result.provider_code) if __name__ == "__main__": diff --git a/tests/test_provider_connections.py b/tests/test_provider_connections.py index c0f40d2..b2e5d11 100644 --- a/tests/test_provider_connections.py +++ b/tests/test_provider_connections.py @@ -41,11 +41,16 @@ def read_playlist_pages(self, connection_id, playlist, cursor=None): class ProviderConnectionServiceTests(unittest.TestCase): def test_provider_error_detail_redacts_common_credentials(self) -> None: detail = redact_error_detail("Bearer abc123 token=secret refresh_token=refresh-value") - error = ProviderApiError(ProviderErrorCategory.NETWORK_ERROR, detail) + error = ProviderApiError( + ProviderErrorCategory.NETWORK_ERROR, + detail, + provider_code="authorization=header-secret", + ) self.assertNotIn("abc123", str(error)) self.assertNotIn("secret", str(error)) self.assertNotIn("refresh-value", str(error)) + self.assertNotIn("header-secret", error.provider_code) self.assertIn("[REDACTED]", str(error)) def setUp(self) -> None: From e9bd96058bbff8a634323ec291214632dd8dff3f Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 20:15:10 +0200 Subject: [PATCH 077/167] feat: report expired provider connections --- docs/development/implementation-baseline.md | 2 +- .../infrastructure/sqlite_connections.py | 17 +++++++++++++++-- src/symphonia/runtime/resources.py | 2 +- tests/test_runtime_resources.py | 5 ++++- tests/test_sqlite_connections.py | 7 +++++++ 5 files changed, 28 insertions(+), 5 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index df3c27b..babbd64 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -44,7 +44,7 @@ The first implementation increment is intentionally narrower than any provider o - Runtime resources can preflight backup integrity and required durable tables read-only before restore design is selected. - Runtime configuration validates host, port, database path, and Ingress base path before startup. - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. -- Provider connection diagnostics expose only provider/state counts, never account or credential data. +- Provider connection diagnostics expose provider/state and expiry counts, never account or credential data. - Import diagnostics expose bounded snapshot freshness and availability counts without playlist content. - Resolution diagnostics expose only manual-decision action counts, never track, actor, or reason data. - Ingress-relative health/version routing with normalized, traversal-safe base paths. diff --git a/src/symphonia/infrastructure/sqlite_connections.py b/src/symphonia/infrastructure/sqlite_connections.py index a01c309..9795afb 100644 --- a/src/symphonia/infrastructure/sqlite_connections.py +++ b/src/symphonia/infrastructure/sqlite_connections.py @@ -134,7 +134,7 @@ def list(self, *, provider: str | None = None) -> tuple[ProviderConnection, ...] ).fetchall() return tuple(self._record(row) for row in rows) - def health_summary(self) -> dict[str, Any]: + def health_summary(self, *, now: datetime | None = None) -> dict[str, Any]: """Return provider/state counts without account or credential data.""" rows = self._connection.execute( @@ -152,7 +152,20 @@ def health_summary(self) -> dict[str, Any]: count = int(row["count"]) by_provider.setdefault(provider, {})[str(row["state"])] = count total += count - return {"total": total, "by_provider": by_provider} + summary: dict[str, Any] = {"total": total, "by_provider": by_provider} + if now is not None: + expired = self._connection.execute( + """ + SELECT COUNT(*) AS count + FROM provider_connections + WHERE state != 'disconnected' + AND expires_at IS NOT NULL + AND expires_at <= ? + """, + (_utc(now),), + ).fetchone() + summary["expired_count"] = int(expired["count"]) + return summary def record_probe( self, diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index a869a94..b5f9ce1 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -203,7 +203,7 @@ def diagnostics( return { "ready": True, "queue": self.operations.queue_summary(now=now), - "connections": self.connections.health_summary(), + "connections": self.connections.health_summary(now=now), "projections": self.projections.summary(), "resolutions": self.resolutions.summary(), "operations": self.operations.diagnostics( diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index c5468e7..0cb888a 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -168,7 +168,10 @@ def test_diagnostics_combine_safe_queue_and_operation_views(self) -> None: ) self.assertTrue(diagnostics["ready"]) self.assertEqual(diagnostics["queue"]["total"], 1) - self.assertEqual(diagnostics["connections"], {"total": 0, "by_provider": {}}) + self.assertEqual( + diagnostics["connections"], + {"total": 0, "by_provider": {}, "expired_count": 0}, + ) self.assertEqual( diagnostics["projections"], { diff --git a/tests/test_sqlite_connections.py b/tests/test_sqlite_connections.py index e54dd99..b719631 100644 --- a/tests/test_sqlite_connections.py +++ b/tests/test_sqlite_connections.py @@ -107,6 +107,13 @@ def test_health_summary_contains_only_provider_state_counts(self) -> None: self.assertNotIn("account-1", str(summary)) self.assertNotIn("opaque-secret", str(summary)) + def test_health_summary_counts_expired_active_connections_when_time_is_supplied(self) -> None: + self.repository.create(connection()) + + summary = self.repository.health_summary(now=NOW + timedelta(days=31)) + + self.assertEqual(summary["expired_count"], 1) + def test_missing_connection_is_explicit(self) -> None: with self.assertRaises(ConnectionNotFound): self.repository.get("missing") From 59063d643d23fbbd6825725c504c9284953da84f Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 21:12:45 +0200 Subject: [PATCH 078/167] docs: define Home Assistant-native UI contract --- README.md | 1 + addon/README.md | 7 +- docs/README.md | 2 + docs/architecture/system-architecture.md | 30 +- .../0004-home-assistant-native-ui.md | 63 +++ docs/decisions/README.md | 2 +- docs/development/development-specification.md | 13 + docs/development/implementation-baseline.md | 2 +- docs/open-questions.md | 19 +- .../home-assistant-ui-specification.md | 182 +++++++++ docs/product/product-specification.md | 7 + .../home-assistant-ecosystem-review.md | 44 +++ docs/providers/provider-research.md | 7 + specs/CATALOG.md | 1 + specs/README.md | 3 + specs/_template.md | 3 + specs/catalog.json | 50 +++ specs/durable-operations-and-recovery.md | 9 +- .../home-assistant-app-runtime-and-ingress.md | 12 +- specs/home-assistant-native-ui.md | 364 ++++++++++++++++++ ...library-import-and-provider-projections.md | 8 +- specs/one-time-playlist-copy.md | 9 +- .../provider-connections-and-authorization.md | 8 +- specs/recording-identity-resolution.md | 8 +- 24 files changed, 837 insertions(+), 17 deletions(-) create mode 100644 docs/decisions/0004-home-assistant-native-ui.md create mode 100644 docs/product/home-assistant-ui-specification.md create mode 100644 specs/home-assistant-native-ui.md diff --git a/README.md b/README.md index 9dc4c21..16ac909 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,7 @@ Start with the [documentation map](docs/README.md). The horizontal specification | Capability readiness and implementation contracts | [SDD catalog](specs/CATALOG.md) | | SDD lifecycle, template, and readiness gate | [SDD standard](specs/README.md) | | Vision, scope, journeys, requirements | [Product specification](docs/product/product-specification.md) | +| Home Assistant-native UI and component contract | [UI specification](docs/product/home-assistant-ui-specification.md) and [UI foundation SDD](specs/home-assistant-native-ui.md) | | Vocabulary, entities, identity, playlists | [Domain model](docs/domain/domain-model.md) | | System boundaries and operational qualities | [Architecture](docs/architecture/system-architecture.md) | | Provider contract and capability semantics | [Provider specification](docs/providers/provider-specification.md) | diff --git a/addon/README.md b/addon/README.md index 3adc55f..701efec 100644 --- a/addon/README.md +++ b/addon/README.md @@ -9,5 +9,8 @@ the App/service boundary used by `vypdev/homeassistant-gateway`: - the service image runs the repository root `Dockerfile` as a non-root user. The metadata is not a published release yet. The image reference, supported -architecture matrix, OAuth callback contract, UI, backup/restore behavior and -release signing still require their SDD gates before stable publication. +architecture matrix, OAuth callback contract, Home Assistant-native UI, +backup/restore behavior and release signing still require their SDD gates +before stable publication. The future UI is governed by the +[Home Assistant-native UI foundation](../specs/home-assistant-native-ui.md); the +metadata scaffold does not imply that UI has been implemented. diff --git a/docs/README.md b/docs/README.md index c8ccbc4..8bcc22d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,6 +11,7 @@ This documentation is the horizontal implementation contract for Symphonia. It d | --- | --- | --- | | [SDD standard and catalog](../specs/README.md) | Capability boundaries, readiness, end-to-end design, numeric test budgets, acceptance and evidence | Shared product policy or silent overrides of horizontal specifications | | [Product specification](product/product-specification.md) | Outcomes, scope, journeys, product requirements | Entity design or technology choices | +| [Home Assistant-native UI specification](product/home-assistant-ui-specification.md) | Cross-cutting UI direction, component families, host context, accessibility, responsive and visual compatibility requirements | Feature-specific content/state or frontend framework selection | | [Domain model](domain/domain-model.md) | Ubiquitous language, invariants, identity, copy and sync semantics | Provider API facts | | [System architecture](architecture/system-architecture.md) | Boundaries, execution model, security and operations | Final implementation stack | | [Provider specification](providers/provider-specification.md) | Provider port, capabilities, normalized errors | Claims about a specific API | @@ -27,6 +28,7 @@ This documentation is the horizontal implementation contract for Symphonia. It d | Prefix | Area | | --- | --- | | `SYM-PROD` | Product and user experience | +| `SYM-UI` | Home Assistant-native UI and component compatibility | | `SYM-ACC` | Local and provider accounts | | `SYM-LIB` | Unified library | | `SYM-MATCH` | Identity resolution | diff --git a/docs/architecture/system-architecture.md b/docs/architecture/system-architecture.md index cac1bcc..4c9f2ee 100644 --- a/docs/architecture/system-architecture.md +++ b/docs/architecture/system-architecture.md @@ -9,6 +9,7 @@ The architecture is derived from these needs: - long-running, self-hosted operation on modest hardware; - a primary Supervisor-managed Home Assistant App experience similar to `vypdev/homeassistant-gateway`; +- a management UI that follows current Home Assistant component, theme, responsive, and interaction patterns without depending on private frontend internals; - a core that can run and be tested without Home Assistant; - provider-independent domain and replaceable adapters; - durable imports and provider writes that survive restarts; @@ -18,7 +19,7 @@ The architecture is derived from these needs: - a future native Home Assistant surface without duplicating domain policy; and - simple backup, restore, upgrade, and diagnostics. -These drivers do not yet justify a programming language, web framework, frontend framework, or database product. +These drivers do not yet justify a programming language, web framework, frontend framework, or database product. The observable UI direction and compatibility-layer boundary are accepted separately in [ADR 0004](../decisions/0004-home-assistant-native-ui.md); that decision does not select a framework. ## Context and trust boundaries @@ -69,7 +70,7 @@ This follows the useful boundary pattern in `homeassistant-gateway` without carr | Component | Responsibility | Explicit exclusions | | --- | --- | --- | -| Web UI | Connection setup, library/playlist views, resolution queue, copy preview, history, diagnostics | Provider tokens, matching policy, direct provider calls | +| Web UI | Home Assistant-native-adjacent shell and component compatibility layer; connection setup, library/playlist views, resolution queue, copy preview, history, diagnostics | Provider tokens, matching policy, direct provider calls, private HA frontend modules | | HTTP/API presentation | Authenticated input/output mapping, validation shape, request correlation | Domain decisions and raw exception exposure | | Application use cases | Connect/disconnect, import, resolve, plan copy, execute copy, inspect operations | Provider-specific response types | | Domain | Provider-independent entities, invariants, capability requirements, matching/copy/sync policy | IO and scheduling | @@ -81,6 +82,27 @@ This follows the useful boundary pattern in `homeassistant-gateway` without carr | Observability | Structured logs, metrics, health/readiness, sanitized diagnostics | Provider payload dumping | | Home Assistant adapter | Ingress identity and future native integration contract | Owning music domain rules | +## Presentation and Home Assistant-native UI boundary + +The cross-cutting UI contract lives in the [Home Assistant-native UI specification](../product/home-assistant-ui-specification.md), is accepted by [ADR 0004](../decisions/0004-home-assistant-native-ui.md), and is made implementation-driving by the [UI foundation SDD](../../specs/home-assistant-native-ui.md). + +```text +public HA/App context ─→ validated context adapter ─→ semantic UI tokens +browser/standalone fallback ────────────────────────┘ + ↓ +feature view model ─→ feature composition ─→ presentation-only components + ↑ │ │ +application API/use case ───┘ └→ catalog/reference tests +``` + +Text equivalent: a validated adapter maps only supported Home Assistant App context, with deterministic browser/standalone fallbacks, into semantic tokens. Feature views combine application-owned view models with a presentation-only component package; catalog, accessibility, responsive, Ingress, and visual-reference tests verify the result. + +The compatibility package owns tokens, component geometry, accessibility interaction, and layout helpers. It may depend on the selected frontend framework and presentation assets, but it must not import API clients, application/domain models, provider adapters, Home Assistant private modules, or parent-page DOM/storage. Feature views own orchestration and route state but consume component families through one public package boundary rather than styling parallel one-off controls. + +Official Home Assistant documentation, design-portal examples, and current source are the primary evolving reference. Pinned `homeassistant-gateway` sources and screenshots are implementation evidence only. Any direct reuse of a Home Assistant component requires a documented public/versioned contract, supported-version matrix, fallback, and rollback path; superficial runtime availability in the parent frame is not a contract. + +Theme, locale, direction, timezone, safe-area, and base-path facts are untrusted until the context adapter validates their supported message/origin/schema. They affect presentation and formatting only; they never grant authorization or change domain/application policy. Standalone composition supplies the same semantic inputs from its own authenticated profile and must preserve the same feature states and actions. + ## Deployment model ### Primary: Home Assistant App @@ -276,11 +298,11 @@ The MVP may render metrics in its UI and logs; choosing Prometheus/OpenTelemetry | Concern | Reason not yet chosen | Evidence needed | | --- | --- | --- | | Backend language/framework | Provider SDK maturity, job ergonomics, footprint, HA App maintainability | Thin vertical spike and maintainer preference | -| UI framework | Ingress routing/auth and complex review UI matter more than popularity | OAuth/Ingress and accessibility prototype | +| UI framework/build tooling | The Home Assistant-native component, accessibility, catalog, Ingress, and standalone contracts are accepted, but the implementation technology remains reversible | `RG-006` prototype proving public context, package boundaries, bundle/compatibility cost, catalog and browser evidence | | SQLite versus PostgreSQL | Concurrency, backup, migration, and library scale unmeasured | Storage/job lease spike and target sizes | | Job library versus internal durable runner | Retry/idempotency needs are specific; external brokers add operations | Failure/restart spike | | Secret encryption/key source | HA App secret facilities and portable standalone behavior differ | Threat model and backup/restore test | | Companion integration transport | Need push, authentication, discovery, and version compatibility | Home Assistant integration RFC | | Public API/event protocol | Only internal UI needs are currently concrete | UI and companion-integration contract design | -No implementation agent should infer these choices from examples in `homeassistant-gateway`. +No implementation agent should infer these technology choices from examples in `homeassistant-gateway`. The Gateway informs the accepted presentation boundary and verification approach, not an automatic dependency or stack selection. diff --git a/docs/decisions/0004-home-assistant-native-ui.md b/docs/decisions/0004-home-assistant-native-ui.md new file mode 100644 index 0000000..f3d86ed --- /dev/null +++ b/docs/decisions/0004-home-assistant-native-ui.md @@ -0,0 +1,63 @@ +# ADR 0004: The App UI follows Home Assistant-native interaction and visual patterns + +- **Status:** accepted +- **Date:** 2026-09-20 + +## Context + +Symphonia's primary management surface runs inside Home Assistant through Ingress. A generic SaaS-style dashboard would make the App feel detached from its host and would force users to learn a second set of navigation, form, status, and feedback conventions. + +The owner clarified that Symphonia's UI and components must be as close as practical to native Home Assistant, following the approach demonstrated by [`vypdev/homeassistant-gateway`](https://github.com/vypdev/homeassistant-gateway). The Gateway is useful dated implementation evidence: at commit [`1ed75be`](https://github.com/vypdev/homeassistant-gateway/tree/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42) it uses a compatibility token layer, presentation-only primitives, a component catalog, responsive and visual tests, and committed Home Assistant visual references. + +Home Assistant's official frontend remains the primary reference, but its built-in components are not a stable public dependency for an independently served App. Home Assistant's own 2026.4 developer notice warns custom-card authors that those component APIs may change and recommends independent components; the same churn risk applies more strongly to an App that would reach across its iframe boundary. + +## Decision + +Symphonia's App UI will be **Home Assistant-native-adjacent**: it will reproduce the host's current interaction model, information density, component families, terminology, theme behavior, responsive conventions, and restrained operational visual language as closely as practical without claiming to be part of the Home Assistant frontend. + +Symphonia will own a small presentation-only compatibility layer for its component families and semantic design tokens. That layer will: + +- follow official Home Assistant design guidance, design-portal examples, current frontend source, and dated reference captures; +- expose Symphonia-owned primitives for buttons, icon buttons, fields, selectors, check controls, tabs, cards, settings/list rows, status indicators, alerts, dialogs, progress, empty/loading states, tables, and responsive layout; +- consume public theme, locale, direction, safe-area, and Ingress context only when the supported Home Assistant contract exposes them, with deterministic standalone/browser fallbacks; +- keep application state, API calls, provider rules, and use-case orchestration outside presentation primitives; and +- remain independently deployable and testable without importing undocumented or private Home Assistant frontend modules. + +Visual fidelity is a compatibility objective, not permission to copy unstable implementation details blindly. Semantic behavior, accessibility, current Home Assistant terminology, light/dark theme parity, mobile behavior, and state clarity take precedence over pixel identity to one release. + +The frontend framework remains a separate implementation decision. A framework is acceptable only if it can satisfy the compatibility-layer, Ingress, accessibility, performance, catalog, and test contracts without coupling the domain/application core to Home Assistant. + +## Consequences + +Positive: + +- the App feels coherent when opened inside Home Assistant; +- familiar controls and wording lower the learning cost; +- an owned compatibility layer isolates Symphonia from Home Assistant frontend churn; +- the same presentation package can render predictably in Ingress and standalone test/catalog contexts; +- dated references and visual tests make “looks native” reviewable instead of subjective. + +Trade-offs: + +- the component layer and visual reference set require ongoing maintenance as Home Assistant evolves; +- direct use of convenient Home Assistant internals is prohibited unless a later public, versioned contract justifies it; +- every supported release needs light/dark, narrow/wide, keyboard, accessibility, and Ingress-context evidence; +- pixel-level divergence may be necessary when accessibility, security, responsiveness, or an independent deployment boundary requires it. + +## Rejected alternatives + +### Generic product-branded SaaS dashboard + +Rejected because it conflicts with the primary Home Assistant host experience and the owner's explicit direction. + +### Import Home Assistant private components directly + +Rejected as the default because the external component APIs are not a stable supported contract and current migrations can rename or replace controls between monthly releases. + +### Freeze a pixel copy of one Home Assistant release + +Rejected because it would age immediately, discourage semantic/accessibility improvements, and confuse resemblance with compatibility. + +### Duplicate presentation behavior separately in every feature view + +Rejected because inconsistent states, focus behavior, spacing, and responsive rules would undermine the native-adjacent goal and make upgrades unreviewable. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 0e78b1b..6dd6aa0 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -7,6 +7,6 @@ ADRs contain only decisions already justified and accepted by the product brief | [0001](0001-provider-independent-recording-domain.md) | Accepted | Provider-independent recording domain | | [0002](0002-copy-and-sync-are-distinct.md) | Accepted | One-time copy and persistent sync are distinct concepts | | [0003](0003-home-assistant-app-primary.md) | Accepted | Home Assistant App is the primary deployment boundary | +| [0004](0004-home-assistant-native-ui.md) | Accepted | App UI follows Home Assistant-native interaction and visual patterns through an owned compatibility layer | An ADR is immutable after acceptance except for typo/link corrections. A changed decision gets a new ADR that supersedes the old one. - diff --git a/docs/development/development-specification.md b/docs/development/development-specification.md index c397c21..4a93e23 100644 --- a/docs/development/development-specification.md +++ b/docs/development/development-specification.md @@ -88,6 +88,15 @@ Use a controlled clock and fault-injecting providers to test 429/reset, timeouts Contract tests validate request/response schemas, authentication, authorization, pagination, base paths, correlation, and sanitized errors. Browser tests cover Ingress-relative navigation, narrow/wide layouts, keyboard access, focus/error announcements, OAuth return/error screens, ambiguous-resolution review, copy preview, partial results, reconnect, and restart recovery. +The [Home Assistant-native UI specification](../product/home-assistant-ui-specification.md) and [UI foundation SDD](../../specs/home-assistant-native-ui.md) add four distinct evidence layers: + +1. presentation-package contract tests for component semantics, variants, tokens, focus, validation, and announcements; +2. a deterministic component catalog covering light/dark, narrow/wide, long/RTL text, disabled/loading/empty/error/destructive states, reduced motion, and increased contrast; +3. responsive geometry and Ingress tests proving no unexpected document overflow, no clipped essential actions, bounded internal table/diagnostic scrolling, safe-area handling, arbitrary base paths, deep links, refresh, assets, and push transports; and +4. reviewed visual comparison against a dated official Home Assistant reference manifest with public-data provenance and documented intentional divergences. + +Canonical pixel baselines may use one declared browser engine, but all supported engines must pass behavior, accessibility, and geometry contracts. Generated screenshot updates are never self-approving and source-string assertions do not substitute for browser accessibility/interaction tests. + UI tests MUST not consider an element visible or a request successful proof that the domain outcome occurred; they verify the returned operation and history. ### Home Assistant App tests @@ -121,6 +130,8 @@ The live matrix must verify documented behavior, not reverse-engineered endpoint - **SYM-TEST-011:** Provider tests MUST cover object-level capability denial and identical upstream IDs from different media types or provider instances without collision. - **SYM-TEST-012:** App artifact/smoke tests MUST enumerate every listening port and prove that any non-Ingress callback listener cannot reach management routes or trust Ingress identity headers. - **SYM-TEST-013:** Migration tests MUST use a per-release history, include stable/beta path divergence, and prove failure leaves the prior database recoverable rather than replacing locally authored state with a fresh rescan. +- **SYM-TEST-014:** Every shared UI component family and representative feature state MUST pass deterministic catalog, keyboard/focus/semantics, accessibility, theme, localization, responsive-geometry, and hostile-content checks before release. +- **SYM-TEST-015:** Every supported Home Assistant release profile MUST pass arbitrary Ingress base-path, public host-context validation/fallback, theme/locale/direction/safe-area, and reviewed visual-reference compatibility gates; private frontend availability MUST NOT be a test prerequisite. ## Matching evaluation @@ -140,6 +151,7 @@ A change is incomplete until, in proportion to its scope: - no secret or real private identifier is present in source/fixtures/output; - migrations, backup, and rollback implications are documented and tested; - frontend accessibility and Ingress base-path behavior pass; +- the shared component catalog, dated Home Assistant visual references, supported HA/browser matrix, and any intentional UI divergences are current and reviewed; - App/standalone artifact metadata is consistent; - local checks and CI pass; and - the full diff contains no unrelated generated artifacts. @@ -173,6 +185,7 @@ Release channels, semantic-version policy, supported upgrade window, and exact a | `SYM-SEC-009` callback isolation | Listener route enumeration + Home Assistant App smoke test | | `SYM-SEC-010` untrusted provider values | Path/URI/redirect/egress injection tests | | `SYM-HA-002` Ingress | Base-path HTTP/browser + App smoke test | +| `SYM-UI-001`–`SYM-UI-015` Home Assistant-native UI | UI package/catalog + a11y/keyboard + responsive/Ingress/context + reviewed dated visual evidence | | `SYM-DEP-006` restore | Artifact smoke using a backup with active/waiting jobs | The SDD catalog and each capability's traceability section are the prospective matrix before code exists. Implementation evidence should be generated or maintained when work is approved; this baseline does not invent test filenames before a stack exists. diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index babbd64..a64b0cb 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -72,7 +72,7 @@ docker run --rm -p 8099:8099 -v symphonia-data:/data symphonia:dev The container exposes only the current health/readiness/version surface. A future App manifest must add Ingress, Supervisor metadata, supported architectures, backup declarations, and any direct callback policy only after the runtime SDD blockers are resolved. - Deterministic `unittest` coverage under `tests/`. -The package is an implementation foundation, not a claim that the corresponding capability SDDs are complete. Provider OAuth, secret storage, user authentication, Ingress UI, a complete migration ledger beyond the current forward-compatible SQLite path, backup retention/restore policy, and production runtime handler wiring remain unimplemented and blocked by their SDD decisions. +The package is an implementation foundation, not a claim that the corresponding capability SDDs are complete. Provider OAuth, secret storage, user authentication, Ingress UI, a complete migration ledger beyond the current forward-compatible SQLite path, backup retention/restore policy, and production runtime handler wiring remain unimplemented and blocked by their SDD decisions. The Home Assistant-native UI direction and compatibility-layer boundary are documented in [ADR 0004](../decisions/0004-home-assistant-native-ui.md) and the [UI foundation SDD](../../specs/home-assistant-native-ui.md), but this foundation slice does not authorize or include frontend implementation. ## Local verification diff --git a/docs/open-questions.md b/docs/open-questions.md index a09b9a6..2bab073 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -102,6 +102,20 @@ Compare SQLite and PostgreSQL for transaction boundaries, leases, crash recovery Define what belongs in the App UI versus a companion custom integration. Decide transport/auth/version discovery and a minimal entity/action/event surface. Ensure Home Assistant actions create ordinary audited Symphonia operations and that the integration can be unavailable independently. +### RG-006 — Home Assistant UI compatibility and host-context spike + +The owner has accepted the Home Assistant-native-adjacent direction, independent compatibility layer, and no-private-frontend-dependency boundary in [ADR 0004](decisions/0004-home-assistant-native-ui.md). Before the [UI foundation SDD](../specs/home-assistant-native-ui.md) becomes `Ready for implementation`, a documentation/prototype spike must: + +- declare the supported Home Assistant and evergreen-browser matrix; +- verify arbitrary Ingress base paths, deep links, refresh, assets, HTTP, and any WebSocket/SSE transport; +- capture the exact supported public contract for theme, locale, direction, timezone, safe-area insets, and context changes, including origin/message/schema validation and deterministic fallback; +- compare candidate frontend/build approaches against component-package isolation, bundle/old-device cost, accessibility, localization, catalog, and standalone reuse; +- create a dated official Home Assistant reference manifest with public-data provenance, light/dark availability, phone/wide viewports, human review ownership, and immutable history; +- prototype representative navigation, card, button, form, status/alert, dialog, progress/empty, settings/list row, dense data, and ordered-evidence components without production feature behavior; and +- prove keyboard/focus/announcement, reduced-motion, contrast, long/RTL text, safe-area/zoom/virtual-keyboard, no document overflow, and bounded table/diagnostic scrolling. + +The result selects the presentation implementation/tooling and supported matrix. It does not authorize production UI work until the SDD is ready and the owner explicitly approves implementation. + ## Future synchronization questions These do not block the copy MVP but block sync implementation: @@ -119,7 +133,7 @@ These do not block the copy MVP but block sync implementation: ## Implementation decisions intentionally deferred - backend language and framework; -- UI framework/design system; +- UI framework/build tooling and supported HA/browser matrix; the Home Assistant-native design-system contract itself is accepted by ADR 0004; - SQLite versus PostgreSQL; - internal durable runner versus job library; - secret encryption primitive and key source; @@ -150,6 +164,7 @@ These need evidence and small RFCs; popularity is not evidence. | App backups contain usable provider tokens or lose the decryption key | Medium | Critical security/recovery failure | Threat model, external key design, restore tests, sanitized export | | Single-node embedded storage cannot handle job concurrency/backup safely | Low-medium | Medium | `RG-004`; no multi-replica claim | | Home Assistant coupling leaks into domain/application | Medium | High maintenance/portability cost | ADR 0003 dependency rule and architecture tests | +| Native-looking UI depends on unstable private Home Assistant components or drifts into an unrelated SaaS design | Medium | High compatibility and product-coherence cost | ADR 0004, owned compatibility layer, `RG-006`, dated official references, catalog/a11y/responsive/visual release gates | | Unknown target library sizes lead to unjustified performance design | High | Medium | `OQ-007`, measurable targets before optimization | | Future Apple Music support is mistaken for full sync despite no documented remove/reorder operation | Medium | High if scope is promoted | Per-operation/object capability probes; Apple feasibility gates before scope change | @@ -162,3 +177,5 @@ These need evidence and small RFCs; popularity is not evidence. 5. **Close the [runtime](../specs/home-assistant-app-runtime-and-ingress.md) and [durable-operation](../specs/durable-operations-and-recovery.md) SDD blockers (`RG-004`).** Choose process topology and storage only after crash, lease, migration, backup, and representative-scale evidence. After those tasks, revisit playlist ownership (`OQ-002`) before creating any persistent-synchronization SDD. The Home Assistant native surface (`RG-005`) can proceed in parallel once the service API shape is stable, but it is not a prerequisite for the copy MVP. + +The UI compatibility spike (`RG-006`) can also proceed in parallel as non-production evidence. It must settle the supported host/browser/context/tooling matrix before any production frontend work, while feature content and behavior continue to be owned by their existing SDDs. diff --git a/docs/product/home-assistant-ui-specification.md b/docs/product/home-assistant-ui-specification.md new file mode 100644 index 0000000..e19c881 --- /dev/null +++ b/docs/product/home-assistant-ui-specification.md @@ -0,0 +1,182 @@ +# Home Assistant-native UI specification + +**Status:** accepted product direction; implementation contract ready for specialist review +**Last reviewed:** 2026-09-20 + +## Purpose and scope + +This document owns the cross-cutting product contract for Symphonia's web UI. Feature SDDs still own their information, actions, states, failures, and content; this specification defines how those surfaces fit Home Assistant visually and behaviorally. + +The primary target is the authenticated Home Assistant Ingress panel. A standalone profile may use the same presentation layer, but it must not become a visually or behaviorally different product. + +## Decision and evidence status + +The owner has accepted the requirement that Symphonia's UI and components be as close as practical to native Home Assistant. [ADR 0004](../decisions/0004-home-assistant-native-ui.md) records the architectural consequences. + +Evidence reviewed on 2026-09-20: + +- official [Home Assistant frontend design guidance](https://developers.home-assistant.io/docs/frontend/design/) and [design portal](https://design.home-assistant.io/); +- official [frontend architecture](https://developers.home-assistant.io/docs/frontend/architecture/), which uses composable web components and unidirectional data flow; +- the official [`ha-card` source](https://github.com/home-assistant/frontend/blob/dev/src/components/ha-card.ts) as a current example of semantic tokens, opaque surfaces, border, radius, slots, and optional elevation; +- official [2026.4 component guidance](https://developers.home-assistant.io/blog/2026/03/25/frontend-component-updates-2026.4/), which warns custom-card authors not to depend on unstable built-in component APIs; +- official [2026.8 App/custom-panel guidance](https://developers.home-assistant.io/blog/2026/07/31/frontend-component-updates-2026.8/), including App-iframe safe-area propagation; +- `vypdev/homeassistant-gateway` commit [`1ed75be`](https://github.com/vypdev/homeassistant-gateway/tree/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42), especially its [frontend direction](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/frontend-design.md), [design system](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/frontend-design-system.md), [UI catalog](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/frontend-ui-catalog.md), [testing strategy](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/frontend-testing-strategy.md), and dated [Home Assistant reference set](https://github.com/vypdev/homeassistant-gateway/tree/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/ui-reference/home-assistant). + +The official sources define the target. The Gateway is community implementation evidence, not a permanent Home Assistant guarantee and not a dependency selection. + +## Terminology + +- **Home Assistant-native-adjacent** means recognizably consistent with Home Assistant's current visual language and interaction model while remaining an independently served Symphonia UI. +- **Compatibility layer** means Symphonia-owned semantic tokens, presentation primitives, and layout helpers that isolate feature views from framework and host-specific details. +- **Component family** means one reusable semantic control or container category, not a one-off styled element. +- **Reference capture** means a dated screenshot of an official public Home Assistant surface used for human comparison, never as the only behavioral specification. +- **Visual parity** means comparable hierarchy, density, geometry, feedback, and theme behavior; it does not mean pixel equality with every Home Assistant version. + +## Product principles + +1. **Familiar before distinctive.** Use Home Assistant navigation, wording, control hierarchy, and status conventions before inventing a Symphonia-specific pattern. +2. **Operational before decorative.** The interface is quiet and information-led: opaque surfaces, moderate borders/radii, restrained elevation, and motion only where it explains state. +3. **Semantic before pixel-identical.** Accessibility, correct interaction, responsive behavior, and current host semantics outrank a frozen screenshot match. +4. **One compatibility layer.** Feature views compose approved families; they do not recreate buttons, fields, cards, dialogs, or status treatments ad hoc. +5. **Independent by construction.** Home Assistant private modules and undocumented DOM/CSS internals are not runtime dependencies. +6. **All product states are designed.** Loading, empty, ready, degraded, partial, waiting, action-required, blocked, failed, and completed states are first-class surfaces. + +## Information architecture and shell + +The Ingress panel must feel like one focused Home Assistant application, not a miniature operating system inside Home Assistant. + +- A page has one clear title, an optional concise explanation, contextual actions, and a stable content region. +- Primary navigation uses a Home Assistant-like flat tab, list/settings, or compact mobile navigation pattern chosen by information depth; it does not mix several competing navigation systems. +- The current section is expressed textually and programmatically, not only by color or icon. +- Deep links and browser history work under a non-root Ingress base path. +- The shell reserves safe-area insets supplied by supported Home Assistant App iframe contracts and has a no-inset fallback. +- A narrow viewport changes layout and action placement without removing essential status, evidence, or recovery actions. + +## Required component families + +| Symphonia family | Home Assistant reference behavior | Required variants and states | +| --- | --- | --- | +| Page heading and toolbar | Settings and operational panel headings with contextual actions | title, description, leading/back action, primary/secondary actions, wrapping narrow state | +| Primary navigation/tabs | Flat HA tabs with a clear active indicator; compact mobile adaptation | selected, unselected, disabled, keyboard arrow/Home/End behavior, overflow/scroll | +| Card and section | `ha-card`-like opaque surface using semantic background, border, radius, spacing, optional restrained elevation | default, actionable, warning/blocked, disabled, nested section | +| Settings/list row | HA settings menu row with icon, primary text, supporting text, chevron/action | default, selected, disabled, destructive, multiline/narrow | +| Button | HA action hierarchy and size/density | primary/brand, secondary/neutral, destructive, success/warning where justified, text/link, icon-leading, loading, disabled | +| Icon button | MDI-style icon inheriting `currentColor` with accessible name and tooltip | default, hover/focus, pressed/toggled, disabled, destructive | +| Text/search/password field | Current HA form-field shape, label, help, validation, semantic form background | empty, populated, required, disabled, read-only, invalid, loading where applicable | +| Select/check/switch controls | HA-like choice hierarchy with explicit labels and descriptions | selected, mixed where applicable, disabled, invalid, grouped | +| Status chip/badge | Compact metadata only; text plus semantic tone | neutral, success/ready, warning/action-required, danger/blocked, unknown | +| Alert/feedback | Feedback adjacent to the affected scope | information, success, warning, error; polite/assertive announcement as appropriate | +| Dialog/adaptive confirmation | Labelled modal on wide screens and mobile-appropriate bounded surface | confirmation, destructive confirmation, loading, validation error, cancellation/focus return | +| Progress/loading/empty | HA-like progress and empty surfaces that explain what is happening | determinate, indeterminate with textual state, skeleton only when meaningful, empty with action, retryable error | +| Data list/table/result row | Dense operational data with semantic headings and bounded internal scrolling | populated, empty, loading, partial, row action, selected, responsive list alternative | +| Tags and evidence groups | Compact metadata, not primary actions | wrapping, overflow-safe, removable only when semantically an input | + +Pills are reserved for compact status/metadata. Cards are not nested repeatedly for decoration. Destructive styling is reserved for destructive or irreversible effects. Provider artwork and branding may identify provider content, but must not replace Home Assistant interaction conventions. + +## Tokens, themes, icons, and motion + +The compatibility layer owns semantic aliases for canvas, surface levels, divider/border, primary and secondary text, brand/action, success, warning, danger, focus, typography, spacing, control/card radii, and optional elevation. + +- When supported public Home Assistant context or iframe properties provide theme information, the UI maps them into the compatibility layer; it never reads Home Assistant private storage or traverses the parent DOM. +- Light and dark modes change tokens, not component geometry or information hierarchy. +- A deterministic fallback theme exists for standalone mode and for absent/malformed host context. +- Hard-coded colors are allowed only inside the token definition or for provider-owned brand artwork reviewed for contrast; feature components consume semantic tokens. +- Material Design Icons, or the selected compatible icon set, use consistent optical size and inherit `currentColor`; an icon never carries the sole accessible label. +- Permanent gradients, glassmorphism, decorative dot fields, ambient animation, and novelty effects are outside the accepted visual language. +- Motion is brief and state-explanatory; `prefers-reduced-motion` removes non-essential animation without hiding progress or focus. +- `prefers-contrast`/forced-colors behavior must preserve boundaries, focus, status text, and actions. + +## Locale, direction, dates, and content + +- The UI uses a supported public Home Assistant locale/timezone hint when available, then a browser/standalone setting, then English. +- Missing regional translations fall back from region to base language and then English. +- Component layout supports longer translated text, bidirectional provider metadata, and right-to-left document direction where the selected locale requires it. +- User-facing dates, numbers, durations, and time zones follow the resolved locale while durable values remain unambiguous UTC or typed quantities. +- Home Assistant terminology is preferred when it describes the same concept. Symphonia domain terms remain explicit where substituting a Home Assistant term would change meaning. +- Provider/user content is escaped, length-bounded, and never interpreted as HTML or Markdown. + +## Accessibility and interaction + +- The target is WCAG 2.2 AA for all supported flows. +- Every action is keyboard operable; focus is visible, ordered, and restored after dialogs or transient views close. +- Tabs, dialogs, tables, lists, fields, validation, progress, and live status use native semantics or complete ARIA patterns. +- Color, icon, position, animation, and shape are never the only carrier of status or required action. +- Loading updates are polite and non-disruptive; blocking/error transitions receive appropriate focus or announcement without repeatedly stealing focus during polling. +- Pointer targets and spacing remain usable on Home Assistant mobile/Companion App viewports and do not require hover. +- A disabled action explains its unmet prerequisite nearby or through accessible help; it is not a silent dead end. + +## Responsive and Ingress behavior + +Required reference viewports include at least a small phone, a common phone, tablet/narrow desktop, and wide desktop; exact browser support belongs to the release matrix. + +- There is no unexpected document-level horizontal overflow. +- Tables, code, and diagnostic payloads may scroll only inside explicit bounded containers; essential actions remain reachable outside those containers. +- Multi-column content collapses predictably; result rows and form actions may stack without changing action order or semantics. +- Insets, browser zoom, large text, virtual keyboard, and narrow embedded-height cases must not cover primary actions or focused fields. +- Ingress routing, asset URLs, API URLs, deep links, refresh, WebSocket/SSE, and OAuth return/error views cannot assume `/`. + +## State and feedback contract + +Every feature view maps durable/application facts into this shared sequence: + +1. current status; +2. completed facts; +3. what happens next; +4. required human action or an explicit statement that none is required; +5. impact and retained state; and +6. collapsed, sanitized technical evidence. + +Errors follow `impact -> cause -> next action -> retained state`. Retry timing is absolute and localized when known. A spinner alone is never sufficient progress. A partial result is never styled or worded as success, and an external dependency outage is not presented as local data loss. + +## Security and privacy boundaries + +- Presentation primitives are pure with respect to provider calls, secrets, authorization, and domain decisions. +- The browser never receives provider refresh tokens, client secrets, raw cookies, or server-only diagnostic payloads. +- URLs from provider/user content use explicit allowed schemes and safe link attributes; untrusted content cannot choose navigation, callback, filesystem, module, or request destinations. +- Clipboard/download actions preview and label the bounded, sanitized content they expose. +- Visual snapshots, component fixtures, translations, and reference captures contain synthetic or public demonstration data only. + +## Component catalog and visual reference policy + +A component family is not ready until its catalog demonstrates: + +- normal, disabled, loading, empty, invalid/error, and destructive variants where applicable; +- light and dark themes; +- keyboard focus and visible focus treatment; +- narrow and wide layouts plus long/translated content; +- accessibility inspection with no known critical/serious violation; and +- deterministic fixtures without network, Home Assistant instance, real account, or secret. + +The project keeps a dated visual reference manifest pointing to official Home Assistant public-demo or design-portal surfaces and records the Home Assistant version/date, viewport, theme, provenance, and reviewer. Official source and documentation resolve ambiguity; screenshots do not define hidden behavior. + +Visual snapshots are reviewed evidence, not self-approving output. A generated baseline change requires a human explanation of whether it follows Home Assistant evolution, fixes a defect, or intentionally diverges for accessibility/product reasons. + +## Requirements + +- **SYM-UI-001:** The primary App UI MUST be Home Assistant-native-adjacent in information hierarchy, terminology, density, component geometry, feedback, and responsive behavior, as defined by dated official references. +- **SYM-UI-002:** Feature views MUST compose an owned presentation-only compatibility layer and MUST NOT import undocumented/private Home Assistant frontend modules or depend on parent DOM/CSS internals. +- **SYM-UI-003:** The compatibility layer MUST provide the component families and required variants in this specification; one-off controls duplicating an existing family require a documented exception. +- **SYM-UI-004:** Light, dark, high-contrast/forced-colors where supported, and reduced-motion behavior MUST preserve semantics, focus, status, and usable actions. +- **SYM-UI-005:** Every user-facing capability MUST render loading/pending, empty where applicable, ready, degraded/partial, action-required, failed/blocked, and completed outcomes that exist in its SDD using text in addition to color or iconography. +- **SYM-UI-006:** The UI MUST meet WCAG 2.2 AA for supported flows and MUST provide keyboard operation, visible focus, correct control semantics, focus management, and bounded live announcements. +- **SYM-UI-007:** The UI MUST work under an arbitrary non-root Ingress base path and MUST adapt to supported safe-area, narrow/mobile, zoom, long-text, and embedded-height conditions without hiding essential state or actions. +- **SYM-UI-008:** Theme, locale, direction, timezone, and safe-area context MUST use supported public Home Assistant/App contracts when available and deterministic browser/standalone fallbacks otherwise; private Home Assistant storage/DOM access is prohibited. +- **SYM-UI-009:** Provider, user, localization, and diagnostic content MUST be escaped, length-bounded, overflow-safe, and non-executable; secrets and unrestricted raw payloads MUST be absent from UI state, fixtures, snapshots, clipboard, and downloads. +- **SYM-UI-010:** Every public component family MUST have a deterministic catalog covering its applicable variants, themes, focus, accessibility, and responsive states before feature views adopt it. +- **SYM-UI-011:** Release evidence MUST include behavioral, accessibility, responsive, and reviewed visual comparison against a dated official Home Assistant reference manifest; snapshot regeneration alone is insufficient approval. +- **SYM-UI-012:** Application/API/domain state and use-case orchestration MUST remain outside presentation primitives, and the UI MUST treat persisted application state rather than transient browser events as authoritative. +- **SYM-UI-013:** Standalone and Ingress profiles MUST share component and feature semantics; host context MAY change tokens, navigation integration, and authentication framing but MUST NOT create a second product behavior. +- **SYM-UI-014:** Decorative effects MUST NOT compete with operational status, evidence, or actions; permanent ambient gradients, glassmorphism, decorative animation, and novelty backgrounds are not part of the accepted visual language. +- **SYM-UI-015:** Supported Home Assistant/frontend compatibility MUST be versioned and revalidated when official component, token, App-iframe, or safe-area contracts change. + +## Exceptions and review + +A deliberate divergence from current Home Assistant behavior must record: + +1. affected component/view and Home Assistant reference; +2. accessibility, security, product, or technical reason; +3. user-visible consequence; +4. fallback and compatibility impact; and +5. approval in the applicable SDD or ADR when the divergence is durable. + +Provider branding alone is not an exception. A future public, versioned Home Assistant component package may justify direct reuse, but adoption requires a compatibility/rollback review rather than bypassing this contract. diff --git a/docs/product/product-specification.md b/docs/product/product-specification.md index 2ae6518..ca4303f 100644 --- a/docs/product/product-specification.md +++ b/docs/product/product-specification.md @@ -17,6 +17,7 @@ The product initially optimizes for trustworthy library interoperability: import 4. **Provider differences remain explicit.** The UI and workflows degrade according to declared capabilities rather than pretending that all providers are equivalent. 5. **Self-hosted is a product constraint.** Operation, backup, upgrades, credentials, and recovery must be understandable to a homelab operator. 6. **Home Assistant is the primary host, not the domain boundary.** The primary package is a Home Assistant App with an Ingress UI, while the same core remains independently runnable and testable. +7. **The App should feel native to its host.** UI hierarchy, components, terminology, density, themes, responsive behavior, and feedback follow current Home Assistant patterns through an independent compatibility layer; see the [Home Assistant-native UI specification](home-assistant-ui-specification.md). ## Users @@ -33,6 +34,7 @@ Multi-user authorization, sharing between Symphonia users, and hosted SaaS opera - Retain enough operation history to diagnose matching and provider failures. - Establish extension points that make a third provider possible without changing the core domain. - Run continuously on ordinary self-hosted infrastructure. +- Provide a Home Assistant-native-adjacent management experience that remains accessible and semantically consistent in Ingress and standalone profiles. ## Explicit non-goals @@ -46,6 +48,7 @@ The MVP MUST NOT include: - multi-user tenancy or role-based administration; - automatic bidirectional playlist synchronization; - Home Assistant coupling inside the domain/application core or standalone composition; +- a generic detached SaaS dashboard, direct dependency on private Home Assistant frontend internals, or decorative styling that competes with operational state; - an assumption that similar titles imply identical recordings. ## Core user journeys @@ -89,6 +92,10 @@ The MVP MUST NOT include: ## MVP requirements +### Home Assistant-native UI + +The cross-cutting `SYM-UI-001`–`SYM-UI-015` requirements live in the [Home Assistant-native UI specification](home-assistant-ui-specification.md). Every feature SDD with a web surface must map its feature-specific states and actions onto that shared component, accessibility, responsive, Ingress, localization, sanitization, and visual-compatibility contract. [ADR 0004](../decisions/0004-home-assistant-native-ui.md) records the accepted decision to use an owned compatibility layer rather than private Home Assistant frontend modules. + ### Product and account - **SYM-PROD-001:** The application core and standalone service MUST remain usable without Home Assistant, while the Home Assistant App is the primary supported deployment. diff --git a/docs/providers/home-assistant-ecosystem-review.md b/docs/providers/home-assistant-ecosystem-review.md index 4446474..adfdb5d 100644 --- a/docs/providers/home-assistant-ecosystem-review.md +++ b/docs/providers/home-assistant-ecosystem-review.md @@ -18,11 +18,13 @@ The strongest precedent is Music Assistant: a separate service/App owns the musi 4. **Treat unofficial YouTube Music access as a distinct product mode.** Music Assistant and `ytube_music_player` demonstrate useful access through `ytmusicapi`, browser cookies, internal endpoints, and proof-of-origin tokens. They do not turn that surface into a supported Google API. An unofficial adapter would need explicit opt-in, health warnings, separate release gating, and no promise of symmetric copy/sync. 5. **Apple Music is a credible future official-library adapter.** Apple's official API documents library reads, catalog/library search, ISRC, playlist creation, and adding tracks. It does not document playlist-track removal, so new-playlist copy is more plausible than mirror sync. Its user-token acquisition and Home Assistant callback story still require a spike. 6. **Do not inherit playback-first shortcuts.** Symphonia must preserve unavailable entries, expose ambiguous matches, prove pagination completeness, and retain auditable user decisions even where an existing playback product can skip, merge, cap, or rescan data. +7. **Adopt the Gateway's independent Home Assistant-native UI pattern, not its stack by implication.** At the pinned commit reviewed, `vypdev/homeassistant-gateway` reproduces Home Assistant component families and operational density through its own tokens/primitives, catalog, visual references, and responsive/accessibility tests while avoiding private Home Assistant frontend imports. Symphonia has accepted that boundary in [ADR 0004](../decisions/0004-home-assistant-native-ui.md); framework selection remains open. ## Projects reviewed | Project | What it establishes | Useful pattern for Symphonia | Boundary or warning | | --- | --- | --- | --- | +| [`vypdev/homeassistant-gateway`](https://github.com/vypdev/homeassistant-gateway/tree/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42) | A Supervisor App can present a coherent HA-native-adjacent Ingress UI using an owned component compatibility layer, catalog, dated official-demo references, and browser/visual tests | Semantic tokens, presentation-only primitives, shared component families, light/dark and mobile evidence, explicit private-HA dependency boundary | Community implementation reviewed at one commit; its Lit/Vite/Storybook choices and exact CSS are not automatically Symphonia decisions | | [Home Assistant Spotify](https://www.home-assistant.io/integrations/spotify) | A maintained Home Assistant integration can use application credentials, the HA external OAuth callback, and multiple account entries | Native config flow, reauthentication, callback and credential UX | It is a playback/media-browser integration, not a cross-provider library system | | [Home Assistant Music Assistant integration](https://www.home-assistant.io/integrations/music_assistant/) | Home Assistant can discover and connect to a separate music server running as an App or container | Service owns domain; integration exposes bounded native actions/entities over an API | Installing an App and installing an integration remain separate lifecycle steps | | [Music Assistant server](https://github.com/music-assistant/server) | Provider plugins, feature declarations, multiple instances, a normalized internal library, provider mappings, scheduled sync, and versioned SQLite migrations work at real scale | Provider manifest, connection instance, normalized mapping graph, scheduled imports | Playback requirements and automatic merging are not Symphonia requirements | @@ -122,6 +124,44 @@ Important limitations for Symphonia: Therefore Apple Music looks promising for future import and one-time copy-to-new-playlist, but not for strict mirror or bidirectional sync until removal/reorder and authentication are proven. +## Home Assistant-native UI lessons from `homeassistant-gateway` + +The Gateway repository was inspected at commit [`1ed75be9f8fabdab386db0fe4320cfb0f67d4f42`](https://github.com/vypdev/homeassistant-gateway/tree/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42), committed 2026-08-27. This pin matters: the repository is active, while Home Assistant component names and tokens also evolve. + +### Evidence inspected + +- [`docs/frontend-design.md`](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/frontend-design.md) chooses a native-adjacent rather than SaaS-like surface: familiar density/terminology, solid surfaces, moderate borders, restrained elevation, no ambient gradients/glassmorphism, limited motion, light/dark mapping, and explicit accessibility rules. +- [`docs/frontend-design-system.md`](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/frontend-design-system.md) defines a presentation-only primitive layer and keeps application state, API calls, and domain-specific options in owning views/controllers. +- [`docs/frontend-ui-catalog.md`](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/frontend-ui-catalog.md) requires an isolated catalog for buttons, icon buttons, tabs, cards, sections, metrics, toolbars, result rows, fields/selects, status chips, alerts, loading/empty states, dialogs, and responsive layouts. +- [`frontend/src/ui/ui-primitives.ts`](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/frontend/src/ui/ui-primitives.ts) and [`ui-layouts.ts`](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/frontend/src/ui/ui-layouts.ts) show concrete semantic behavior: button loading disables repeat action and exposes `aria-busy`; tabs implement roles/selection/keyboard navigation; fields associate help/errors; alerts use live semantics; dialogs use stable labels/descriptions; responsive lists/tables and settings rows are reusable. +- [`docs/frontend-testing-strategy.md`](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/frontend-testing-strategy.md) layers pure/runtime, controller, HTTP adapter, UX-structure, responsive geometry, flow, visual, and production-bundle evidence. It explicitly checks page overflow, element clipping, allowed internal scrolling, active navigation, and multiple browser engines. +- [`docs/ui-reference/home-assistant/README.md`](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/ui-reference/home-assistant/README.md) records 20 public-demo Home Assistant screenshots with capture date, viewport, public-data provenance, inspection policy, and a warning that source/documentation must also be checked. +- [`docs/frontend-and-credentials.md`](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/frontend-and-credentials.md) keeps normal management Ingress-only and makes credential values/status boundaries explicit in the UI contract. + +### Patterns to adopt + +1. Own a small semantic compatibility layer rather than styling raw controls independently in each view. +2. Use official Home Assistant source/design/docs as the current target and dated public-demo captures as reviewable visual evidence. +3. Keep UI primitives presentation-only; route/API/controller/application/domain state remains outside them. +4. Treat light/dark, narrow/wide, focus, disabled, loading, error, empty, partial, and completed variants as catalog requirements, not polish after feature implementation. +5. Verify responsive geometry explicitly: no document overflow or clipped actions; tables/diagnostics may scroll only in bounded containers. +6. Keep the visual layer quiet: opaque surfaces, semantic borders/tokens, restrained elevation and motion, status text alongside color/icons. +7. Require human review for screenshot updates and retain reports/traces on failure. + +### Patterns to adapt + +- Symphonia needs richer item-level uncertainty, matching evidence, playlist order/duplicate displays, long-running durable operations, and provider risk disclosure than the Gateway. Its component catalog must therefore cover dense ordered lists, evidence comparisons, immutable-plan review, and partial/reconciliation states. +- The Gateway's exact Lit, Vite, Storybook, palette, radii, navigation, and CSS values are evidence, not accepted Symphonia dependencies. The selected implementation must satisfy the [UI specification](../product/home-assistant-ui-specification.md) and [UI foundation SDD](../../specs/home-assistant-native-ui.md). +- Home Assistant 2026.8 documents safe-area propagation for App iframes; Symphonia must validate the supported version range and context/origin/schema instead of assuming the latest behavior everywhere. +- The Gateway uses fixed public-demo reference captures. Symphonia should keep an immutable manifest/history so one newly captured Home Assistant release does not erase why an older supported release differs. + +### Patterns to avoid + +- Importing Home Assistant components merely because their tags happen to exist in a parent page. +- Reading parent DOM, private CSS, `.storage`, cookies, or undocumented frontend state to obtain theme/locale/identity. +- Treating snapshot generation as approval, source-string assertions as accessibility tests, or Chromium pixel identity as cross-browser correctness. +- Copying the Gateway's product-specific navigation, policy labels, or credential flows into music workflows. + ## Security lessons A 2026 [Music Assistant security advisory](https://github.com/music-assistant/server/security/advisories/GHSA-7jcc-p6xr-835j) described an unauthenticated direct service port combined with user-controlled filesystem paths and root execution. Symphonia does not need a filesystem music provider, but the boundary lessons apply: @@ -140,6 +180,7 @@ A 2026 [Music Assistant security advisory](https://github.com/music-assistant/se - Provider manifests, multiple connection instances, and declared features. - Internal provider-representation mappings and versioned migrations. - Native Home Assistant application-credential/config-flow patterns where a companion integration genuinely owns that boundary. +- Home Assistant-native-adjacent UI through an independent semantic token/component layer, deterministic catalog, and dated official visual references. ### Adapt @@ -147,6 +188,7 @@ A 2026 [Music Assistant security advisory](https://github.com/music-assistant/se - Replace playback-friendly automatic merging with evidence-backed, reversible identity links. - Replace “rescan after failure” with recovery that protects user-authored state. - Treat provider quality labels as a first-class support tier visible in planning and diagnostics. +- Extend the Gateway's operational components for Symphonia's uncertainty evidence, ordered occurrences, long-running durable state, and item-level partial outcomes. ### Avoid @@ -155,6 +197,7 @@ A 2026 [Music Assistant security advisory](https://github.com/music-assistant/se - Skipping unavailable tracks or returning capped lists as if complete. - Exposing a direct unauthenticated port merely to make OAuth convenient. - Letting provider values influence local file paths, arbitrary URI schemes, redirects, or code/module loading. +- Depending on private Home Assistant frontend modules/parent DOM or accepting visual snapshots without semantic, accessibility, responsive, and human-review evidence. ## Design consequences for the next RFCs @@ -167,3 +210,4 @@ This review does not accept a dependency or new provider into the MVP. It narrow 5. Add an Apple Music test-account spike as a future-provider candidate, focusing on Music User Token acquisition, catalog/library IDs, `canEdit`, playlist append, and absence of remove/reorder. 6. Require completeness markers for every import/list operation and preserve unavailable entries. 7. Threat-model every directly exposed App listener and prevent provider-controlled values from acquiring filesystem or executable semantics. +8. Treat the [UI specification](../product/home-assistant-ui-specification.md) and [UI foundation SDD](../../specs/home-assistant-native-ui.md) as the shared presentation contract; complete `RG-006` before selecting or implementing the frontend stack. diff --git a/docs/providers/provider-research.md b/docs/providers/provider-research.md index 2ea1bfc..8842794 100644 --- a/docs/providers/provider-research.md +++ b/docs/providers/provider-research.md @@ -143,6 +143,11 @@ Home Assistant is the primary deployment platform, not a music provider. - [Home Assistant Apps](https://developers.home-assistant.io/docs/apps/) are Supervisor-managed container applications distributed through App repositories. - [App configuration](https://developers.home-assistant.io/docs/apps/configuration/) documents `/data` persistent storage, startup types, architecture metadata, Ingress, App options, and backup modes. - [App presentation/Ingress](https://developers.home-assistant.io/docs/apps/presentation/#ingress) documents the authenticated proxied UI boundary and Ingress base-path considerations. +- [Frontend design guidance](https://developers.home-assistant.io/docs/frontend/design/) identifies the official design portal as the maintained place to inspect reusable components, card states, light/dark comparisons, and Home Assistant wording. +- [Frontend architecture](https://developers.home-assistant.io/docs/frontend/architecture/) documents a web-component, panel, dialog, unidirectional-data-flow, and decentralized-routing architecture. This is useful reference evidence, not a requirement that an independently served App import the complete frontend. +- Home Assistant's [2026.4 component update](https://developers.home-assistant.io/blog/2026/03/25/frontend-component-updates-2026.4/) explicitly warns custom-card authors that built-in component APIs can change and recommends independent components. Symphonia applies that churn warning to its stronger iframe boundary and therefore mirrors public semantics/tokens through its own compatibility layer rather than depending on private bundles. +- Home Assistant's [2026.8 component/App update](https://developers.home-assistant.io/blog/2026/07/31/frontend-component-updates-2026.8/) documents safe-area handling and propagated inset values for custom panels and App iframes. Supported-version behavior still requires `RG-006` rather than assuming every installed Home Assistant version exposes the same context. +- The current official [`ha-card` source](https://github.com/home-assistant/frontend/blob/dev/src/components/ha-card.ts) demonstrates semantic theme tokens, opaque surface, border, radius, slotted content/actions, and optional elevation. It is dated design evidence, not a stable runtime dependency. - [App security](https://developers.home-assistant.io/docs/apps/security/) recommends least privilege, avoiding host networking, AppArmor, minimal folder/API access, and careful authentication handling. - [Application Credentials](https://developers.home-assistant.io/docs/core/platform/application_credentials/) provides OAuth2/config-flow helpers for Home Assistant integrations, including local bring-your-own client credentials and PKCE support. - The official [Spotify integration](https://www.home-assistant.io/integrations/spotify) uses `https://my.home-assistant.io/redirect/oauth`, or `/auth/external/callback` when My Home Assistant is disabled, and supports multiple account entries. @@ -150,6 +155,8 @@ Home Assistant is the primary deployment platform, not a music provider. The provider OAuth callback is not solved merely by enabling Ingress. Application Credentials belongs to Home Assistant integrations, not arbitrary Supervisor Apps. Reusing it would require a companion integration or broker with an explicit token-ownership contract. Exact provider redirect URLs, externally reachable Home Assistant URLs, sessions, user-supplied client registrations, and any direct callback listener require a dedicated spike. +Likewise, an Ingress iframe does not automatically provide a stable importable Home Assistant component library. Symphonia's accepted [UI contract](../product/home-assistant-ui-specification.md) treats official components, tokens, and demo/design-portal views as evolving reference evidence; theme/locale/direction/safe-area/base-path context must cross a documented, validated App boundary with deterministic fallback. + ## Current conclusion Spotify is a plausible personal/self-hosted MVP adapter, with meaningful 2026 access and reauthorization constraints. The official YouTube Data API is a plausible **YouTube playlist** adapter, but it is not sufficient evidence for the proposed full **YouTube Music** provider. That distinction is the highest-risk assumption in the current product scope. Apple Music has a promising official library/create/append surface for future scope, but its Home Assistant-compatible authorization flow and missing documented playlist removal/reorder remain open. diff --git a/specs/CATALOG.md b/specs/CATALOG.md index d21fa31..2ba29ad 100644 --- a/specs/CATALOG.md +++ b/specs/CATALOG.md @@ -7,6 +7,7 @@ This is the human-readable view of [`catalog.json`](./catalog.json). Until gener | Capability ID | Status | Primary SDD | Readiness summary | | --- | --- | --- | --- | | `home-assistant-app-runtime` | Draft | [Home Assistant App runtime and Ingress](home-assistant-app-runtime-and-ingress.md) | Blocked by storage/recovery, supported platform matrix, and secret-key design | +| `home-assistant-native-ui` | Ready for review | [Home Assistant-native UI foundation](home-assistant-native-ui.md) | Product direction is accepted; blocked from implementation readiness by the supported matrix, frontend/build choice, public host-context validation, and visual-reference procedure | | `provider-connections-and-authorization` | Draft | [Provider connections and authorization](provider-connections-and-authorization.md) | Blocked by OAuth boundary and provider feasibility spikes | | `library-import-and-provider-projections` | Draft | [Library import and provider projections](library-import-and-provider-projections.md) | Blocked by provider completeness, retention, and representative scale | | `recording-identity-resolution` | Draft | [Recording identity resolution](recording-identity-resolution.md) | Blocked by the labeled corpus and accepted automatic-link policy | diff --git a/specs/README.md b/specs/README.md index b416ad2..f41fc2a 100644 --- a/specs/README.md +++ b/specs/README.md @@ -11,6 +11,7 @@ The existing `docs/` documents remain the horizontal sources of truth: | Source | Owns | | --- | --- | | [`docs/product/product-specification.md`](../docs/product/product-specification.md) | Product scope, journeys, and stable requirement IDs | +| [`docs/product/home-assistant-ui-specification.md`](../docs/product/home-assistant-ui-specification.md) | Cross-cutting UI/component, host-context, accessibility, responsive, and visual compatibility requirements | | [`docs/domain/domain-model.md`](../docs/domain/domain-model.md) | Shared language, identity, invariants, copy/sync semantics | | [`docs/architecture/system-architecture.md`](../docs/architecture/system-architecture.md) | System-wide boundaries, security, persistence, and operations | | [`docs/providers/provider-specification.md`](../docs/providers/provider-specification.md) | Provider port and capability contract | @@ -66,6 +67,7 @@ Every SDD MUST cover these concerns or state why a concern is not applicable: - Partial success and unknown external write outcomes are first-class states, not generic failures. - User-facing errors follow: impact, cause, next action, retained state. - Configuration cannot weaken authorization, secret handling, auditability, identity provenance, or idempotency safeguards. +- Every web-surface SDD maps its feature-specific states and actions onto the Home Assistant-native UI foundation; it does not create a parallel design system or depend on private Home Assistant frontend modules. ## Numeric test budget @@ -104,6 +106,7 @@ Readiness is necessary but not authorization to implement. The owner must still | Capability | SDD | Why it exists before code | | --- | --- | --- | | Home Assistant App runtime | [App runtime and Ingress](home-assistant-app-runtime-and-ingress.md) | Packaging, lifecycle, authentication boundary, persistence, backup | +| Home Assistant-native UI | [UI foundation](home-assistant-native-ui.md) | Shared component families, host context, accessibility, responsive behavior, catalog, visual compatibility | | Provider connections | [Provider connections and authorization](provider-connections-and-authorization.md) | OAuth/cookies, secrets, callback boundary, effective capabilities | | Provider imports | [Library import and provider projections](library-import-and-provider-projections.md) | Completeness, provenance, unavailable items, retention | | Recording identity | [Recording identity resolution](recording-identity-resolution.md) | Conservative matching, evidence, manual decisions | diff --git a/specs/_template.md b/specs/_template.md index 1a1866a..b455406 100644 --- a/specs/_template.md +++ b/specs/_template.md @@ -121,6 +121,8 @@ Define validation, precedence, migration, a meaningful alternative, and behavior ## 9. UI/UX and content contract +If this capability has a web surface, state how its feature-specific content/actions/states compose the [Home Assistant-native UI foundation](home-assistant-native-ui.md), which shared component families it uses, and any intentional divergence. If no web surface exists, mark this explicitly and explain why. + ### 9.1 Information hierarchy 1. Current status. @@ -228,6 +230,7 @@ Define deterministic fakes/clocks/IDs, coverage expectations, contract fixtures, - [ ] Numeric test budget and stated coverage pass. - [ ] Configuration, migration, backup, rollback, and recovery agree. - [ ] Pending, action-required, partial, failed, and completed UX states are verified. +- [ ] Every web surface uses the shared Home Assistant-native component/catalog/host-context contract, or an intentional divergence is documented and approved. - [ ] Accessibility, localization, sanitization, and secret-safety gates pass. - [ ] User/setup/operator/contributor documentation is complete and discoverable. - [ ] Catalog status and implementation evidence are current. diff --git a/specs/catalog.json b/specs/catalog.json index 12e5c44..043ffcb 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -62,6 +62,56 @@ "OQ-005 standalone release timing and authentication" ] }, + { + "id": "home-assistant-native-ui", + "title": "Home Assistant-native UI foundation", + "status": "ready-for-review", + "scope": "Provide the shared Home Assistant-native-adjacent shell, component families, semantic tokens, host-context adaptation, accessibility, responsive behavior, component catalog, and visual compatibility evidence", + "owner": "Symphonia maintainers", + "specification": "specs/home-assistant-native-ui.md", + "requirements": [ + "SYM-UI-001", + "SYM-UI-002", + "SYM-UI-003", + "SYM-UI-004", + "SYM-UI-005", + "SYM-UI-006", + "SYM-UI-007", + "SYM-UI-008", + "SYM-UI-009", + "SYM-UI-010", + "SYM-UI-011", + "SYM-UI-012", + "SYM-UI-013", + "SYM-UI-014", + "SYM-UI-015", + "SYM-JOB-007", + "SYM-SEC-004", + "SYM-SEC-008", + "SYM-TEST-005", + "SYM-TEST-009", + "SYM-TEST-014", + "SYM-TEST-015" + ], + "evidence": [ + "docs/decisions/0004-home-assistant-native-ui.md", + "docs/product/home-assistant-ui-specification.md", + "docs/architecture/system-architecture.md", + "docs/providers/provider-research.md", + "docs/providers/home-assistant-ecosystem-review.md" + ], + "implementationEvidence": { + "code": [], + "tests": [], + "documentation": [] + }, + "blockers": [ + "Supported Home Assistant and browser matrix", + "Frontend framework and build-tool selection", + "RG-006 verified theme, locale, direction, safe-area, and Ingress context contract", + "Accepted dated visual-reference capture and update procedure" + ] + }, { "id": "provider-connections-and-authorization", "title": "Provider connections and authorization", diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 0f113e8..40ac08f 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -6,7 +6,7 @@ - Owners: Symphonia maintainers - Scope: execute imports and provider writes through durable, observable operations that recover safely across restarts, rate limits, uncertain writes, and upgrades. - Related requirements: `SYM-PROD-004`–`SYM-PROD-006`, `SYM-ARCH-001`–`SYM-ARCH-002`, `SYM-ARCH-005`, `SYM-ARCH-008`–`SYM-ARCH-010`, `SYM-JOB-001`–`SYM-JOB-008`, `SYM-OBS-001`–`SYM-OBS-006`, `SYM-TEST-004`, `SYM-TEST-013`, `SYM-DEP-002`, `SYM-DEP-008` -- Related decisions/research: [system architecture](../docs/architecture/system-architecture.md), [development specification](../docs/development/development-specification.md), `RG-004` +- Related decisions/research: [system architecture](../docs/architecture/system-architecture.md), [development specification](../docs/development/development-specification.md), [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [UI foundation](home-assistant-native-ui.md), `RG-004` - Required review gates: architecture, persistence/recovery, provider contracts, testing, documentation, security/operations - Open decisions blocking readiness: persistent store; worker/process topology; lease and retention parameters; supported migration strategy; representative operation sizes and timing targets @@ -165,6 +165,8 @@ The persistent store and wakeup mechanism may be one technology or separate ones ## 9. UI, UX, and accessibility +Operation list/detail, progress, wait, recovery, cancellation, reconciliation, and terminal-result surfaces conform to the [Home Assistant-native UI foundation](home-assistant-native-ui.md). They use shared statuses, alerts, progress/loading, cards/sections, responsive result rows/tables, buttons, and adaptive dialogs. Visual updates are projections of persisted operation state; component animation or in-memory events can improve immediacy but never invent progress or success. + The operations view shall show action type, actor-safe label, creation time, current state, progress numerator/denominator when meaningful, next retry time, and available actions. Example running state: @@ -243,6 +245,8 @@ Minimum planned automated tests: **86**. Required deterministic tests include duplicate dispatch, concurrent claim, lease expiry, clock boundaries, crash before/after every checkpoint, store outage, retry exhaustion, rate-limit timing, cancellation races, unknown outcomes, and compatible/incompatible upgrades. +The eight feature-specific UI cases supplement the UI-foundation budget and inherit its component-catalog, host-context, accessibility, responsive, theme/localization, hostile-content, and dated visual-reference gates. + ## 15. Documentation impact Implementation shall update: @@ -269,6 +273,7 @@ Implementation shall update: 11. Supported upgrades preserve or safely refuse every persisted state fixture. 12. Logs, metrics, events, and diagnostics contain no credentials or prohibited content. 13. The numeric test budget and fault-injection suite pass. +14. Queued, running, rate-limited, retry-scheduled, waiting-user, recovering, reconciling, partial, failed, cancelled, and succeeded fixtures use shared Home Assistant-native components; updates preserve focus, never move progress backward without explanation, and remain truthful after disconnect/reload at narrow and wide widths. ## 17. Requirement traceability @@ -316,3 +321,5 @@ UI -> Operation store: read durable progress view - [SDD catalog](CATALOG.md) - [Home Assistant App runtime and Ingress SDD](home-assistant-app-runtime-and-ingress.md) - [One-time playlist copy SDD](one-time-playlist-copy.md) +- [Home Assistant-native UI foundation SDD](home-assistant-native-ui.md) +- [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md) diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index dfdf853..e09fd7f 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -6,7 +6,7 @@ - Owners: Symphonia maintainers - Scope: define the install, lifecycle, authentication, persistence, recovery, upgrade, backup, and diagnostics contract for the primary Supervisor-managed App. - Related requirements: `SYM-PROD-001`, `SYM-ACC-005`, `SYM-HA-001`–`SYM-HA-009`, `SYM-DEP-001`–`SYM-DEP-010`, `SYM-SEC-008`–`SYM-SEC-011` -- Related decisions/research: [ADR 0003](../docs/decisions/0003-home-assistant-app-primary.md), [system architecture](../docs/architecture/system-architecture.md), [Home Assistant platform research](../docs/providers/provider-research.md#home-assistant-platform) +- Related decisions/research: [ADR 0003](../docs/decisions/0003-home-assistant-app-primary.md), [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [system architecture](../docs/architecture/system-architecture.md), [Home Assistant platform research](../docs/providers/provider-research.md#home-assistant-platform), [UI foundation](home-assistant-native-ui.md) - Required review gates: product UX, architecture, Home Assistant platform, testing, documentation, security/operations - Open decisions blocking readiness: storage/recovery result from `RG-004`; supported Home Assistant versions and CPU architectures; encryption-key/backup contract; standalone release timing from `OQ-005` @@ -180,6 +180,8 @@ Invalid or unknown values fail closed before workers start. Authentication, non- ## 9. UI/UX and content contract +All lifecycle surfaces conform to the [Home Assistant-native UI foundation](home-assistant-native-ui.md). The shell, heading/toolbar, navigation, cards/sections, buttons, statuses, alerts, progress/loading/empty states, dialogs, and diagnostics containers come from the shared compatibility layer; this SDD owns their lifecycle content and allowed actions, not a parallel visual system. Host theme/locale/direction/safe-area context uses only the validated public adapter and does not change readiness or authorization. + ### 9.1 Information hierarchy The App panel starts with service status, completed startup fact, next transition, required action, retained state, and a link to bounded diagnostics. @@ -208,7 +210,7 @@ Next: the App will reconnect automatically; provider operations continue safely. ### 9.3 Accessibility and localization -Status is textual and announced when it changes; focus remains stable during polling/push updates. Ingress navigation works at narrow widths and never depends on color. Initial locale follows Home Assistant where available with English fallback; exact supported locales require the UI SDD/toolchain. +Status is textual and announced when it changes; focus remains stable during polling/push updates. Ingress navigation works at narrow widths and never depends on color. Initial locale follows validated Home Assistant context where available with English fallback; the exact locale set remains part of the UI foundation's supported-matrix and translation-plan blockers. ## 10. Failure, recovery, and cleanup @@ -260,6 +262,8 @@ Minimum **58 distinct cases**: All ordinary tests use fake Supervisor/Ingress and deterministic storage/clock fixtures. A disposable HA OS/Supervised-compatible smoke environment covers install, Ingress, restart, backup/restore, and upgrade. Manual evidence covers desktop/mobile and light/dark startup/error views. +The eight feature-specific UI cases above supplement rather than replace the 84-case UI-foundation budget. The runtime release also inherits component-catalog, arbitrary-base-path, host-context fallback, safe-area, accessibility, responsive-geometry, hostile-content, and dated visual-reference gates from that SDD. + ## 15. Documentation and discoverability | Audience | Artifact | Required content | Validation/navigation | @@ -281,6 +285,7 @@ All ordinary tests use fake Supervisor/Ingress and deterministic storage/clock f 8. Given an unsupported downgrade, the App blocks before writes and points to compatible restore/upgrade guidance. 9. Given hostile restored/config/diagnostic values, no path escape, arbitrary URL, secret output, or code execution occurs. 10. Given Supervisor stop, new admissions stop and shutdown leaves every lease recoverable within the bounded time. +11. Given supported Home Assistant theme/locale/direction/safe-area context or its absence, every lifecycle state uses the shared Home Assistant-native components, preserves focus/status/actions at phone and wide widths, and falls back without touching readiness or authorization. ## 17. Requirements traceability @@ -318,6 +323,7 @@ All ordinary tests use fake Supervisor/Ingress and deterministic storage/clock f - Primary sources: Home Assistant App/Ingress/security documentation linked from [provider research](../docs/providers/provider-research.md#home-assistant-platform). - Related SDDs: [provider authorization](provider-connections-and-authorization.md), [durable operations](durable-operations-and-recovery.md). -- Accepted: App is the primary deployment boundary; core remains HA-independent. +- Related UI contract: [Home Assistant-native UI foundation](home-assistant-native-ui.md) and [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md). +- Accepted: App is the primary deployment boundary; core remains HA-independent; UI follows the owned Home Assistant-native compatibility layer. - Rejected: all logic in a custom integration; unauthenticated management port; fresh database fallback after migration failure. - Follow-up: standalone release and companion integration native surface. diff --git a/specs/home-assistant-native-ui.md b/specs/home-assistant-native-ui.md new file mode 100644 index 0000000..db9cb10 --- /dev/null +++ b/specs/home-assistant-native-ui.md @@ -0,0 +1,364 @@ +# Home Assistant-native UI foundation + +- Status: Ready for review +- Date: 2026-09-20 +- Catalog capability ID: `home-assistant-native-ui` +- Owners: Symphonia maintainers +- Scope: establish the shared shell, component families, semantic tokens, host-context adaptation, accessibility, responsive behavior, component catalog, and visual compatibility evidence for every Symphonia web view. +- Related requirements: `SYM-UI-001`–`SYM-UI-015`, `SYM-JOB-007`, `SYM-SEC-004`, `SYM-SEC-008`, `SYM-TEST-005`, `SYM-TEST-009`, `SYM-TEST-014`, `SYM-TEST-015` +- Related decisions/research: [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [UI specification](../docs/product/home-assistant-ui-specification.md), [official platform research](../docs/providers/provider-research.md#home-assistant-platform), [Gateway/HA UI evidence](../docs/providers/home-assistant-ecosystem-review.md#home-assistant-native-ui-lessons-from-homeassistant-gateway) +- Required review gates: product UX, Home Assistant platform, frontend architecture, accessibility, localization, testing/visual QA, documentation, security/privacy +- Open decisions blocking implementation readiness: supported Home Assistant/browser matrix; frontend framework/build selection; verified theme/locale/direction/safe-area context contract across supported Home Assistant versions; accepted visual-reference capture/update procedure + +## 1. Executive summary + +Every Symphonia web view will be built from one Home Assistant-native-adjacent compatibility layer. The layer owns semantic tokens, accessible presentation primitives, layout behavior, theme/locale/safe-area adaptation, and a deterministic component catalog. Feature views own content and use-case state but cannot invent duplicate controls or depend on private Home Assistant frontend internals. + +The safe default is an independent, quiet operational UI that follows current Home Assistant patterns closely, works under arbitrary Ingress paths, and falls back deterministically in standalone mode. + +```text +Public HA/App context + browser fallback -> semantic compatibility tokens +-> reusable component families -> feature view models from durable application state +-> Ingress/standalone shell -> behavioral, accessibility, responsive and visual evidence +``` + +## 2. Problem, current behavior, and evidence + +### 2.1 Problem + +Without a shared UI contract, each feature can implement different cards, buttons, forms, statuses, navigation, and responsive rules. The result may function but feel foreign inside Home Assistant, hide important partial/recovery states, and acquire an unstable dependency on the host frontend. + +“Looks native” is otherwise subjective. It needs dated references, reusable component contracts, explicit allowed divergences, and acceptance evidence. + +### 2.2 Current behavior + +No complete Symphonia UI or UI package exists. The repository has an experimental Ingress metadata scaffold and prospective feature SDDs. This SDD does not select or authorize a frontend framework or production implementation. + +### 2.3 Evidence and unknowns + +- Accepted repository direction: [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md) and the [UI specification](../docs/product/home-assistant-ui-specification.md). +- Official evidence: Home Assistant design portal, frontend architecture/source, independent-component warning, current App/Ingress and safe-area contracts linked from the UI specification and platform research. +- Community evidence: `vypdev/homeassistant-gateway` commit `1ed75be` demonstrates a presentation-only compatibility layer, HA-like component families, component catalog, official-demo reference captures, accessibility/responsive tests, and visual baselines. +- Unknowns: exact supported Home Assistant/browser versions; selected frontend/build tools; the public context actually available to an Ingress App at each supported version; long-term token mapping; reference capture automation and review ownership. + +## 3. Actors, surfaces, and terminology + +| Actor | Goal | Entry point | Visible surfaces | +| --- | --- | --- | --- | +| Home Assistant administrator/user | Use Symphonia without learning an alien UI | Ingress panel | shell, navigation, connections, library, matching, copy, operations, diagnostics | +| Standalone user, if released | Use the same product semantics outside HA | authenticated standalone URL | same feature views with fallback host framing | +| Product/accessibility reviewer | Inspect every state before feature integration | component catalog and reference manifest | variants, themes, focus, viewports, long text, failures | +| Contributor | Compose consistent views without duplicating presentation behavior | public UI package entry point | tokens, components, layouts, stories/fixtures | +| Release operator | Detect host/frontend compatibility regressions | release evidence and supported matrix | Ingress smoke, screenshots, accessibility/responsive reports | + +**Compatibility token** is a semantic Symphonia variable optionally mapped from supported host context. **Primitive** is presentation-only reusable behavior. **Catalog story/fixture** is deterministic isolated evidence. **Reference manifest** records dated official Home Assistant visual sources and provenance. **Intentional divergence** is a reviewed difference from the current official reference. + +## 4. Goals, non-goals, and fixed invariants + +### 4.1 Goals + +1. Make every App view recognizably coherent with current Home Assistant. +2. Provide one reusable, accessible component vocabulary for all feature SDDs. +3. Isolate Home Assistant UI churn behind public context adapters and semantic tokens. +4. Prove behavior across themes, viewports, localization, accessibility settings, Ingress paths, and failure states. +5. Keep feature/application state independent from presentation and host-specific code. + +### 4.2 Non-goals + +1. Pixel-copying one Home Assistant release indefinitely. +2. Importing or forking the complete Home Assistant frontend. +3. Selecting a frontend framework in this SDD. +4. Defining each feature's domain content, actions, or state machine. +5. Making Symphonia a Lovelace card or replacing the Home Assistant shell. +6. Creating a decorative brand system that competes with operational status. + +### 4.3 Fixed invariants + +1. Feature views use the approved compatibility layer for existing component families. +2. Private Home Assistant modules, parent DOM traversal, private storage, and undocumented CSS selectors are prohibited dependencies. +3. No theme, viewport, locale, provider content, or animation preference may remove semantic status, focus, or required actions. +4. Presentation components contain no provider calls, secrets, domain policy, persistence, or operation orchestration. +5. Ingress and standalone render the same product state and action semantics. +6. Visual similarity never weakens accessibility, security, sanitization, or truthfulness. + +## 5. Product journey + +| Stage | Proposed behavior | User/operator effect | +| --- | --- | --- | +| Resolve host context | Accept supported theme/locale/direction/safe-area/base-path facts and validate them | UI matches its host without trusting arbitrary parent state | +| Bootstrap | Apply fallback tokens immediately, then bounded host context without geometry flash that hides content | First meaningful state is readable and stable | +| Navigate | Render one responsive shell and current-section semantics | User understands location at wide and narrow widths | +| Interact | Compose standard component families with feature view models | Controls behave consistently across journeys | +| Update | Re-render from authoritative application state with bounded announcements | Long operations remain understandable after navigation/reconnect | +| Review | Exercise catalog, accessibility, responsive, and official-reference comparison | “Native-adjacent” has inspectable evidence | +| Upgrade | Revalidate supported HA versions/tokens/context and intentional divergences | Host changes cannot silently break the App UI | + +```mermaid +flowchart LR + A[Supported HA/App context] --> C[Context validation and semantic token map] + B[Browser/standalone fallback] --> C + C --> D[Presentation-only component families] + E[Feature view model from application API] --> F[Feature composition] + D --> F + F --> G[Ingress or standalone shell] + G --> H[Catalog, a11y, responsive and visual evidence] +``` + +Text equivalent: validated public host context or deterministic fallback feeds semantic tokens; reusable presentation families combine with feature-owned view models inside one shell; catalog and browser evidence verify the result. + +## 6. Functional behavior and state model + +### 6.1 Happy path + +1. The shell resolves base path and renders readable fallback tokens without blocking on Home Assistant context. +2. A context adapter accepts only the documented host message/origin/shape and maps supported theme, locale, direction, timezone, and safe-area facts. +3. Navigation loads a feature route and preserves history under the Ingress prefix. +4. The feature requests application state; loading is textual and non-destructive. +5. The view composes only approved component families and renders the durable state plus available actions. +6. Updates replace relevant view state, preserve focus where possible, and announce only material transitions. +7. The same fixtures render in catalog, light/dark and narrow/wide checks, and reviewed visual comparisons. + +### 6.2 Alternative and boundary paths + +- Missing, malformed, unsupported, or late host context keeps the deterministic fallback and records a sanitized compatibility reason. +- Host theme/locale changes update tokens/content without losing unsaved form state or focus. +- Unknown routes show a native-adjacent not-found surface with a safe return action; they never escape the Ingress base path. +- Connection loss preserves last safe state, labels it stale, and offers bounded reconnect behavior. +- Large tables/diagnostics scroll inside explicit containers or switch to a responsive record layout; the document does not overflow horizontally. +- Long translations, RTL, zoom, safe-area insets, and virtual keyboards reflow without covering primary actions. +- Unsupported future Home Assistant context fields are ignored rather than treated as trusted configuration. + +### 6.3 UI shell/context states + +| State | Entered when | User-visible meaning | Allowed next states | Recovery/owner | +| --- | --- | --- | --- | --- | +| `bootstrapping` | shell/assets loaded; application/context pending | Symphonia is opening; no outcome implied | `loading`, `ready`, `degraded`, `blocked` | automatic | +| `loading` | route/application request active | Named data/state is being loaded | `ready`, `empty`, `degraded`, `blocked` | automatic/retry | +| `ready` | view state and applicable context are valid | Current feature is available | `loading`, `degraded`, route change | user/system | +| `empty` | valid response has no applicable items | Nothing exists yet and next action is explicit | `loading`, `ready` | user/system | +| `degraded` | stale/offline/host-context limitation but safe local view remains | Named limitation; retained data/actions explicit | `ready`, `blocked` | retry/user action | +| `blocked` | unsafe/auth/application failure prevents the view/action | Impact, cause, next action, retained state | `loading`, `ready` after correction | user/operator | + +Feature states such as partial, waiting-user, failed, and completed remain owned by their SDDs and use the shared state/feedback components. + +## 7. Configuration contract + +| Input | Type | Recommended default | Allowed values/range | Scope/persistence | +| --- | --- | --- | --- | --- | +| Theme mode | enum | follow validated Home Assistant context, then browser preference | `host/auto`, `light`, `dark` only if product permits override | user/browser preference; never domain state | +| Locale | BCP 47 tag | validated HA locale, browser locale, then English | packaged supported locales with base-language fallback | user/browser; server retains typed values | +| Text direction | enum | derived from locale/public host context | `ltr`, `rtl` | derived, not arbitrary per component | +| Safe-area handling | enum | supported host insets with zero fallback | host-managed or App-managed accepted profile | deployment/profile | +| Visual reference manifest | versioned document | latest reviewed supported HA release/reference date | immutable historical entries plus current pointer | repository/release evidence | +| Motion | media/user preference | system reduced-motion contract | normal or reduced | browser preference | + +Users cannot configure away visible focus, semantic errors, required confirmations, accessible names, secret redaction, or the Home Assistant-native component hierarchy. Feature code cannot override tokens with unreviewed hard-coded palettes. + +## 8. Clean Architecture design + +### 8.1 Responsibilities and dependency direction + +| Boundary | Owns | Must not own/import | +| --- | --- | --- | +| Product UI policy | component/state/compatibility invariants and permitted divergences | framework APIs, provider/domain logic | +| Application/API presentation | authenticated view models, action DTOs, persisted-state mapping | component geometry/theme decisions | +| UI compatibility package | semantic tokens, primitives, layouts, interaction/accessibility behavior | API clients, application/domain models, provider calls | +| Feature views/controllers | compose view models, call use cases, route-level state/focus | duplicate primitives, domain decisions | +| HA context adapter | validated public message/context mapping, base path, safe area | parent DOM/private storage, music policy | +| Shell/composition | routing, localization loading, theme selection, package wiring | feature business rules | +| Catalog/reference tooling | deterministic stories, public-demo reference provenance, visual/a11y/responsive evidence | live HA/provider/user data | + +### 8.2 Contracts, durable state, and trust boundaries + +- Pure decisions: token fallback/mapping, component variants, responsive layout selection, status tone/content, locale fallback, safe-area geometry. +- Application contracts: feature view models and commands with explicit loading/empty/partial/action/error facts. +- Ports/adapters: host-context source, locale catalog loader, history/base-path router, clock only for formatted relative time, diagnostic correlation. +- Durable state: none owned by primitives; user-safe preferences may persist separately, while feature truth remains server-side. +- Concurrency: stale route/API/context results are fenced or cancelled; late results cannot overwrite a newer navigation/action. +- Trusted/untrusted inputs: parent messages, URL/base path, theme names/tokens, locale tags, translations, provider/user content, diagnostic text, and API errors are untrusted until validated. +- Error mapping: UI receives stable safe categories and prepared user actions, never raw exceptions/provider bodies. + +### 8.3 Executable architecture constraints + +- Static dependency checks prohibit API/controller/application/domain/HA-private imports from the UI compatibility package and catalog fixtures. +- A single public package entry point exposes approved component families; compatibility re-exports contain no separate implementation. +- Token linting rejects hard-coded feature colors/spacing/radii outside approved token/provider-brand definitions. +- Route tests enumerate non-root base paths, refresh, deep links, assets, HTTP, and push transports. +- Context tests validate allowed origin/message/schema and fallback behavior; unsupported fields remain inert. +- Component semantics are exercised by interaction/accessibility tests, not source-string assertions alone. + +## 9. UI/UX and content contract + +### 9.1 Information hierarchy + +1. Home Assistant-like page heading/current section and contextual action. +2. Current status and primary user goal. +3. Completed facts and material evidence. +4. What happens next and whether action is required. +5. Impact, retained state, and safe recovery action. +6. Collapsed sanitized technical details. + +### 9.2 Representative states + +```text +Opening Symphonia +Loading connection status. Your running operations continue in the background. +``` + +```text +No provider connections +Connect a supported music service to import playlists. Review access basis and permissions before leaving Symphonia. +Primary action: Add connection +``` + +```text +Library available with limitations +The latest Spotify import is incomplete; the complete library from 09:14 remains visible. +Action: Retry the failed playlist. No existing entries were removed. +``` + +```text +Symphonia needs attention +This view cannot confirm current operation state because the service is unavailable. +Action: Retry connection or open diagnostics. +Retained state: the last confirmed status is shown as stale; no provider write is being inferred. +``` + +```text +Copy completed with omissions +113 entries were confirmed and 4 explicitly accepted omissions remain listed. +Primary action: Review result +``` + +### 9.3 Accessibility and localization + +- Locale/direction/timezone use validated public HA context with region-to-language-to-English fallback. +- Navigation, tabs, dialogs, fields, lists/tables, row actions, loading, and feedback follow native HTML or complete ARIA patterns. +- Focus survives passive refresh; dialogs trap and restore focus; route/action errors move focus only when required to recover. +- Mobile/narrow layout preserves headings, current status, primary action, and recovery facts; bounded containers own intentional scrolling. +- Provider/user/translation content is escaped, bidi-isolated where needed, length-bounded, and never executable markup. + +## 10. Failure, recovery, and cleanup + +| Failure/partial state | User impact | Retained facts | Automatic retry | Required action | Cleanup | +| --- | --- | --- | --- | --- | --- | +| Host context absent/malformed | fallback theme/locale/safe area | feature state unaffected | bounded re-subscribe if documented | none | discard invalid message | +| Theme/locale update fails | prior valid presentation remains | selected route/form state | reload catalog/context once | choose fallback/reload if persistent | no preference corruption | +| Route/asset base-path error | view/action unavailable | server operations continue | no loop | return/reopen App; diagnostics | invalidate bad client route cache | +| API disconnect during read | current data labeled stale | last confirmed view and correlation | bounded reconnect | retry if exhausted | cancel superseded requests | +| API disconnect after action | outcome unknown in browser | server operation/idempotency reference | fetch authoritative state | inspect operation; never repeat blindly | clear transient button loading after reconciliation | +| Component render/translation defect | affected content unusable | application state server-side | error boundary once | reload/report safe diagnostic | no secret/raw state dump | +| Unsupported HA frontend change | possible visual/context mismatch | independent fallback works | no speculative adaptation | release compatibility update | record divergence/reference update | + +## 11. Security, permissions, and privacy + +1. Ingress identity is trusted only at the verified server boundary; theme/locale messages do not grant authorization. +2. Parent-window messaging validates origin, type, correlation, and bounded schema; no wildcard data is executed or persisted blindly. +3. Provider/user/diagnostic strings cannot become markup, CSS, script, unsafe URLs, route destinations, or component definitions. +4. UI fixtures, screenshots, clipboard, downloads, browser logs, and error boundaries exclude secrets and real private account/library data. +5. Destructive actions use explicit consequences and confirmation; visual similarity cannot weaken server authorization or idempotency. + +## 12. Observability and operational UX + +- A sanitized client compatibility view exposes UI version, resolved theme/locale/direction/safe-area profile, route/base-path mode, connectivity, and reference-manifest version—not tokens, parent messages, or content. +- Client errors use bounded codes/correlation and aggregate component/route/context failures without provider/user text as metric labels. +- Loading/reconnect announcements are rate-limited; stable healthy state emits no repeated toast/notification. +- Visual mismatch is a release/test issue, not a user alert unless it makes a supported flow unusable. +- Browser state never supersedes the server's durable operation status. + +## 13. Compatibility, migration, rollout, and rollback + +- The UI package, token schema, translation catalog, API view models, and reference manifest are versioned. +- Each release declares supported Home Assistant and evergreen browser ranges; an unsupported host gets a clear compatibility warning only when evidence shows material risk. +- Token/context adapters tolerate absent new fields and ignore unknown fields; breaking renames require an explicit compatibility mapping and fixture. +- Component migrations occur family by family through the public package entry point; feature views do not straddle independent duplicate implementations indefinitely. +- Initial rollout may begin with catalog/reference evidence before feature views, then one operational vertical slice, then all remaining surfaces. +- Rollback serves a compatible prior UI/API asset set; mixed incompatible asset/API versions fail clearly and refresh safely rather than issuing malformed actions. + +## 14. Testing strategy and numeric budget + +Minimum **84 distinct cases**: + +| Area | Minimum distinct cases | Behaviors/risks covered | +| --- | ---: | --- | +| Tokens/context/configuration/pure policy | 14 | host/fallback theme, locale, RTL, safe area, invalid context, token mapping, preferences | +| Component interaction and accessibility | 24 | every required family; keyboard/focus/ARIA; loading/disabled/error/destructive; announcements | +| Responsive/theme/locale geometry | 16 | light/dark/contrast/reduced motion, phone/tablet/desktop, zoom, long/RTL text, overflow | +| Shell/feature/Ingress/security contracts | 18 | arbitrary base path, deep link/refresh/assets/push, stale/cancelled requests, hostile content, secret canaries | +| Visual/reference/compatibility/release | 12 | dated HA comparisons, intentional divergence, snapshot review, supported host/browser upgrade/rollback | +| **Total** | **84** | No double counting | + +Pure tests use fixed context messages, tokens, locales, translations, routes, and view models. Component/browser fixtures use no network and synthetic public-safe data. Browser behavior runs across the supported engine matrix; canonical pixel baselines may use one declared engine, while every engine must pass semantics and geometry contracts. + +Required human evidence reviews official Home Assistant references versus the catalog and representative feature screens in light/dark, phone/wide, long-text, focus, action-required, partial, blocked, and completed states. Baseline updates require written review; they never self-approve. + +## 15. Documentation and discoverability + +| Audience | Artifact | Required content | Validation/navigation | +| --- | --- | --- | --- | +| User | UI/accessibility preferences guide | host theme/locale behavior, navigation, progress, reduced motion, fallback | linked from Help/About | +| Setup owner | Ingress/standalone presentation guide | base path, safe area, supported HA/browser matrix, compatibility diagnostics | install/upgrade checks | +| Operator | UI troubleshooting | blank/stale/context/theme/base-path symptoms and safe recovery | decision tree + sanitized diagnostic fixture | +| Contributor | component and content guide | tokens, families, composition, wording, accessibility, exceptions | catalog + architecture/lint gates | +| Reviewer | visual-reference manifest/runbook | provenance, capture version/date/viewports/themes, comparison and approval | release evidence index | + +## 16. Acceptance scenarios + +1. Given a supported Home Assistant Ingress panel, the shell adopts validated theme, locale, direction, safe-area, and base-path context while remaining recognizably consistent with official references. +2. Given absent, malformed, late, or unknown host context, a deterministic accessible fallback renders and no untrusted field changes authorization, navigation, CSS, or executable behavior. +3. Given light, dark, increased contrast/forced colors, or reduced motion, status, focus, structure, and all actions remain perceivable and usable. +4. Given keyboard-only use, every component family and representative feature flow supports visible focus, correct order, activation, dialog focus return, and non-disruptive announcements. +5. Given a phone viewport, safe-area insets, zoom, virtual keyboard, and long translated text, no essential state/action is clipped and no unexpected document-level horizontal overflow occurs. +6. Given an arbitrary Ingress prefix, direct/deep navigation, refresh, assets, HTTP calls, and any push connection resolve within that prefix and cannot escape to `/` accidentally. +7. Given loading, empty, ready, degraded/partial, action-required, failed/blocked, and completed feature fixtures, the UI uses shared families and truthful text rather than spinner/color/icon alone. +8. Given provider/user/translation/diagnostic strings containing HTML, Markdown, bidi controls, long tokens, unsafe URLs, or secret canaries, rendering and output remain bounded, inert, and secret-free. +9. Given two overlapping route/action/context requests, the stale result cannot overwrite the latest view or trigger a repeated mutation. +10. Given the standalone profile, the same feature states/actions/components render with documented fallback framing rather than a second product contract. +11. Given a proposed new component, it is rejected from feature use until its catalog covers applicable variants, themes, focus, accessibility, responsive, and long-content states. +12. Given a visual snapshot change, release evidence identifies the dated Home Assistant reference and records intended parity, accessibility/product divergence, or regression correction before approval. +13. Given a Home Assistant frontend/component migration, compatibility tests prove independent fallback and the supported matrix/reference manifest are updated before release. +14. Given a browser action whose response is lost, the UI reloads authoritative server state and never infers success or blindly repeats an irreversible operation. + +## 17. Requirements traceability + +| Requirement | Policy/use case/adapter/presentation | Test or evidence | Documentation | +| --- | --- | --- | --- | +| `SYM-UI-001`–`SYM-UI-003`, `SYM-UI-014` | product UI policy + compatibility package | component catalog, dependency/lint gates, dated HA comparison | component/content guide + ADR 0004 | +| `SYM-UI-004`, `SYM-UI-006` | tokens and component interaction | theme/contrast/motion + keyboard/ARIA/a11y browser matrix | accessibility/preferences guide | +| `SYM-UI-005`, `SYM-UI-012` | feature presentation + application view models | shared state fixtures, durable-state/reconnect races | feature SDDs and user guides | +| `SYM-UI-007`–`SYM-UI-008` | shell, router, HA context adapter | Ingress prefix, context schema/origin, safe-area/viewport tests | Ingress presentation guide | +| `SYM-UI-009` | presentation/output boundaries | hostile-content and secret-canary suite | security/content guide | +| `SYM-UI-010`–`SYM-UI-011`, `SYM-UI-015` | catalog/reference/release tooling | catalog completeness, visual review, HA/browser compatibility matrix | reference manifest/runbook | +| `SYM-UI-013` | composition profiles | Ingress versus standalone fixture parity | deployment guides | +| `SYM-TEST-005`, `SYM-TEST-009`, `SYM-TEST-014`, `SYM-TEST-015` | verification tooling | release gates | contributor testing guide | + +## 18. Implementation sequence + +1. Accept the supported Home Assistant/browser matrix, public context contract, visual-reference procedure, and frontend/build decision. +2. Establish reference manifest, semantic tokens, fallback context, package boundary, architecture/lint checks, and catalog harness. +3. Add foundational components: headings/toolbars, navigation/tabs, cards/sections, buttons/icon buttons, fields/choices, status/alerts, loading/empty, and layouts. +4. Add dialogs, settings/list rows, responsive data displays, safe-area/base-path shell, localization/direction, and full interaction/accessibility tests. +5. Integrate one vertical operational view and prove durable-state mapping, reconnect, hostile content, Ingress, standalone fallback, and representative visuals. +6. Migrate remaining feature views through the public package; complete compatibility matrix, documentation, visual review, and release evidence. + +## 19. Definition of Done + +- [ ] Status is `Ready for implementation` and explicit owner approval to implement exists. +- [ ] Supported HA/browser matrix, frontend/build, context/safe-area, and reference procedures are accepted. +- [ ] Every `SYM-UI-*` requirement maps to acceptance and deterministic or explicit human evidence. +- [ ] At least 84 distinct cases and architecture/lint/secret gates pass. +- [ ] Every required component family is available only through the public package and complete in the catalog. +- [ ] Representative feature states pass light/dark, contrast, reduced-motion, keyboard, screen-reader, narrow/wide, long/RTL text, and sanitization review. +- [ ] Ingress prefix, deep links, assets, reconnect/push, safe area, and standalone fallback are proven. +- [ ] Dated official reference manifest and reviewed visual evidence are current; all intentional divergences are documented. +- [ ] User/setup/operator/contributor/reviewer documentation is complete and discoverable. +- [ ] Catalog status/implementation evidence are current; no private HA dependency or readiness blocker remains. + +## 20. References and decisions + +- Primary sources: official Home Assistant links in the [UI specification](../docs/product/home-assistant-ui-specification.md) and [platform research](../docs/providers/provider-research.md#home-assistant-platform). +- Community evidence: pinned `vypdev/homeassistant-gateway` sources listed in the UI specification and ecosystem review. +- Related SDDs: [App runtime/Ingress](home-assistant-app-runtime-and-ingress.md), [provider connections](provider-connections-and-authorization.md), [imports](library-import-and-provider-projections.md), [identity resolution](recording-identity-resolution.md), [playlist copy](one-time-playlist-copy.md), [durable operations](durable-operations-and-recovery.md). +- Accepted: Home Assistant-native-adjacent direction, owned compatibility layer, no private HA frontend runtime dependency, shared Ingress/standalone semantics, dated catalog/reference evidence. +- Open: framework/build stack, supported matrix, public App context details, reference capture/update automation. +- Rejected: generic SaaS dashboard, private HA component imports by default, frozen pixel copy, decorative ambient UI, feature-local duplicate primitives. diff --git a/specs/library-import-and-provider-projections.md b/specs/library-import-and-provider-projections.md index c87077c..928ab46 100644 --- a/specs/library-import-and-provider-projections.md +++ b/specs/library-import-and-provider-projections.md @@ -6,7 +6,7 @@ - Owners: Symphonia maintainers - Scope: import approved provider collections and ordered playlists into complete, provenance-rich provider projections without confusing them with provider-independent recordings. - Related requirements: `SYM-PROD-003`, `SYM-LIB-001`–`SYM-LIB-006`, `SYM-PROV-004`–`SYM-PROV-014`, `SYM-PROV-018`, `SYM-ARCH-004`–`SYM-ARCH-005` -- Related decisions/research: [domain model](../docs/domain/domain-model.md), [provider specification](../docs/providers/provider-specification.md), [provider research](../docs/providers/provider-research.md), `RG-001`, `OQ-007`, `OQ-008` +- Related decisions/research: [domain model](../docs/domain/domain-model.md), [provider specification](../docs/providers/provider-specification.md), [provider research](../docs/providers/provider-research.md), [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [UI foundation](home-assistant-native-ui.md), `RG-001`, `OQ-007`, `OQ-008` - Required review gates: product UX, domain, architecture, provider feasibility/policy, testing, documentation, privacy/operations - Open decisions blocking readiness: proven collection semantics/completeness per MVP provider; retention/export policy; representative library sizes/import targets; provider-specific refresh/deletion obligations @@ -183,6 +183,8 @@ Users cannot configure “ignore pagination errors,” “drop unavailable items ## 9. UI/UX and content contract +Library, playlist, freshness, progress, and import-result views conform to the [Home Assistant-native UI foundation](home-assistant-native-ui.md). They use shared headings/toolbars, filters/fields, status/alerts, progress and empty states, cards/sections, result rows, and responsive data displays. Dense tables may switch to list/record layouts or scroll only inside bounded containers; a provider-like artwork grid cannot hide completeness, freshness, provenance, unavailable entries, or required recovery. + ### 9.1 Information hierarchy Library/connection pages show current projection freshness and last complete import separately from the latest attempt. Progress is based on persisted counts and collection states, not transient events. @@ -264,6 +266,8 @@ Minimum **72 distinct cases**: Fixtures include zero/one/boundary/multi-page collections, duplicate playlist occurrences, unavailable/deleted/local/non-music/unknown items, colliding IDs across types/instances, revision change mid-read, and provider-declared totals that lie. Live smoke is opt-in and verifies only dated documented behavior with dedicated data. +The eight feature-specific UI cases supplement the UI-foundation budget and inherit its component-catalog, host-context, accessibility, responsive geometry, theme/localization, hostile-content, and dated visual-reference gates. + ## 15. Documentation and discoverability | Audience | Artifact | Required content | Validation/navigation | @@ -285,6 +289,7 @@ Fixtures include zero/one/boundary/multi-page collections, duplicate playlist oc 8. Given provider disappearance, user-authored resolution/audit data remains while provider payload retention transitions independently. 9. Given hostile provider IDs/URLs/names/cursors, no path/egress/render/log injection or secret disclosure occurs. 10. Given a completed import, only changed/new provider tracks are scheduled for separate resolution with no identity decision made by import. +11. Given empty, importing, rate-limited, stale, partial, failed, and complete fixtures with long provider text and dense ordered entries, the shared Home Assistant-native components preserve textual completeness/freshness, keyboard access, bounded scrolling, and essential actions at narrow and wide widths. ## 17. Requirements traceability @@ -322,6 +327,7 @@ Fixtures include zero/one/boundary/multi-page collections, duplicate playlist oc - Primary sources: [provider research](../docs/providers/provider-research.md) and provider links therein. - Related SDDs: [connections](provider-connections-and-authorization.md), [identity resolution](recording-identity-resolution.md), [durable operations](durable-operations-and-recovery.md). +- Related UI contract: [Home Assistant-native UI foundation](home-assistant-native-ui.md) and [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md). - Accepted: provider projections remain distinct from recordings; only complete results can infer missing items. - Rejected: silent truncation, skipping unavailable entries, delete-on-partial, matching during import. - Follow-up: incremental imports and persistent sync after official evidence and baseline semantics exist. diff --git a/specs/one-time-playlist-copy.md b/specs/one-time-playlist-copy.md index 72a8154..a451dee 100644 --- a/specs/one-time-playlist-copy.md +++ b/specs/one-time-playlist-copy.md @@ -6,7 +6,7 @@ - Owners: Symphonia maintainers - Scope: preview and execute a finite playlist copy using an immutable plan, explicit non-ready policy, ordered writes, reconciliation, and item-level outcomes. - Related requirements: `SYM-PROD-002`, `SYM-PROD-004`–`SYM-PROD-006`, `SYM-PL-001`–`SYM-PL-009`, `SYM-PROV-010`, `SYM-PROV-019`, `SYM-ARCH-009`–`SYM-ARCH-010`, `SYM-JOB-003`–`SYM-JOB-005`, `SYM-TEST-004`, `SYM-TEST-006` -- Related decisions/research: [ADR 0002](../docs/decisions/0002-copy-and-sync-are-distinct.md), [copy domain model](../docs/domain/domain-model.md), [provider research](../docs/providers/provider-research.md), `OQ-003`, `RG-001` +- Related decisions/research: [ADR 0002](../docs/decisions/0002-copy-and-sync-are-distinct.md), [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [copy domain model](../docs/domain/domain-model.md), [provider research](../docs/providers/provider-research.md), [UI foundation](home-assistant-native-ui.md), `OQ-003`, `RG-001` - Required review gates: product UX, domain, architecture, provider feasibility, testing, documentation, security/operations - Open decisions blocking readiness: default non-ready policy; proven target write behavior; target collision policy; reconciliation semantics for ambiguous provider writes @@ -159,6 +159,8 @@ Provider-specific workarounds shall remain behind ports and shall not alter doma ## 9. UI, UX, and accessibility +Source/target selection, dry-run summary, ordered occurrence review, ambiguity resolution links, acceptance, execution, reconciliation, and result views conform to the [Home Assistant-native UI foundation](home-assistant-native-ui.md). They use shared headings/toolbars, fields/selectors, cards/sections, status/alerts, progress, responsive data rows/tables, buttons, and adaptive confirmation dialogs. The immutable-plan and item-evidence density may extend ordinary settings patterns, but must preserve Home Assistant tokens, focus, action hierarchy, and documented intentional divergences. + The review page shall show: - source playlist and captured version; @@ -234,6 +236,8 @@ Minimum planned automated tests: **94**. Required fault injection includes timeouts before and after provider acceptance, duplicate delivery, process death at every write checkpoint, expired authorization, rate limits, capability drift, and ambiguous reconciliation. +The twelve feature-specific UI cases supplement the UI-foundation budget and inherit its component-catalog, host-context, accessibility, responsive, theme/localization, hostile-content, and dated visual-reference gates. + ## 15. Documentation impact Implementation shall update: @@ -258,6 +262,7 @@ Implementation shall update: 10. Partial success has per-item explanations and a safe remediation path. 11. Logs and diagnostics contain no provider secrets. 12. The numeric test budget and required fault-injection scenarios pass. +13. Plan selection, review, blocked, accepted, running, waiting, reconciling, partial, cancelled, failed, and completed fixtures use the shared Home Assistant-native components and preserve every ordered occurrence, textual consequence, focus/action, and safe recovery path at phone and wide widths. ## 17. Requirement traceability @@ -302,3 +307,5 @@ App -> User: show terminal and per-item outcomes - [SDD catalog](CATALOG.md) - [Recording identity resolution SDD](recording-identity-resolution.md) - [Durable operations and recovery SDD](durable-operations-and-recovery.md) +- [Home Assistant-native UI foundation SDD](home-assistant-native-ui.md) +- [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md) diff --git a/specs/provider-connections-and-authorization.md b/specs/provider-connections-and-authorization.md index c7551b6..11ca4f3 100644 --- a/specs/provider-connections-and-authorization.md +++ b/specs/provider-connections-and-authorization.md @@ -6,7 +6,7 @@ - Owners: Symphonia maintainers - Scope: disclose provider risk, authorize one external account, protect and refresh its grant, probe effective capabilities, reauthorize, and disconnect safely. - Related requirements: `SYM-ACC-002`–`SYM-ACC-004`, `SYM-ACC-006`, `SYM-PROV-002`–`SYM-PROV-003`, `SYM-PROV-008`–`SYM-PROV-009`, `SYM-PROV-015`–`SYM-PROV-020`, `SYM-SEC-001`–`SYM-SEC-010` -- Related decisions/research: [provider specification](../docs/providers/provider-specification.md), [official API research](../docs/providers/provider-research.md), [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md), `OQ-001`, `OQ-004`, `RG-001`, `RG-002` +- Related decisions/research: [provider specification](../docs/providers/provider-specification.md), [official API research](../docs/providers/provider-research.md), [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md), [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [UI foundation](home-assistant-native-ui.md), `OQ-001`, `OQ-004`, `RG-001`, `RG-002` - Required review gates: product UX, architecture, provider feasibility, testing, documentation, security/privacy - Open decisions blocking readiness: direct App OAuth versus companion-integration authorization broker; per-provider registration/scopes/token lifecycle; unofficial YouTube Music MVP decision; secret key source and backup contract @@ -194,6 +194,8 @@ No preference becomes accepted until `RG-002` records evidence and an ADR choose ## 9. UI/UX and content contract +Connection, disclosure, credential, callback-result, status, and disconnect views conform to the [Home Assistant-native UI foundation](home-assistant-native-ui.md). They use the shared settings/list rows, cards/sections, fields/choices, buttons, status chips, adjacent alerts, adaptive confirmation dialogs, and loading/empty states. Provider branding identifies the provider but does not replace Home Assistant interaction hierarchy. Risk, unofficial access, permission scope, and reconnection remain textual and cannot be reduced to a colored badge. + ### 9.1 Information hierarchy Before the primary `Connect` action, show access basis, maturity/support, account/subscription prerequisites, requested functional access, credential type, callback/remote-access needs, external dependencies, reauthorization expectation, and known limitations. @@ -284,6 +286,8 @@ Minimum **82 distinct cases**: Provider contract tests are offline and deterministic. Live smoke tests are opt-in, use dedicated accounts/client registrations, bounded scopes/quota, and safe cleanup. Exact secrets are seeded as canaries and asserted absent from every non-secret boundary. Manual evidence reviews provider consent transitions and narrow/mobile disclosures. +The twelve feature-specific UI cases supplement the UI-foundation budget and inherit its catalog, host-context, theme, accessibility, responsive, hostile-content, and dated visual-reference release gates. + ## 15. Documentation and discoverability | Audience | Artifact | Required content | Validation/navigation | @@ -306,6 +310,7 @@ Provider contract tests are offline and deterministic. Live smoke tests are opt- 9. Given hostile callback/provider/error content, no open redirect, path/egress injection, Markdown/HTML injection, or secret output occurs. 10. Given App restart at every attempt phase, no callback is consumed twice and no orphan grant becomes an active connection. 11. Given the broker alternative, a forged/replayed/local unauthenticated handoff is rejected and the integration cannot mutate the App database directly. +12. Given official, unofficial, connected, degraded, action-required, authorizing, and disconnected fixtures, the shared Home Assistant-native component families preserve information hierarchy, keyboard/focus behavior, narrow layout, theme parity, and textual risk without exposing secret material. ## 17. Requirements traceability @@ -343,6 +348,7 @@ Provider contract tests are offline and deterministic. Live smoke tests are opt- - Primary sources: official provider and Home Assistant sources in [provider research](../docs/providers/provider-research.md). - Related SDDs: [App runtime](home-assistant-app-runtime-and-ingress.md), [imports](library-import-and-provider-projections.md), [durable operations](durable-operations-and-recovery.md). +- Related UI contract: [Home Assistant-native UI foundation](home-assistant-native-ui.md) and [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md). - Accepted: capability and risk disclosure before authorization; opaque secret references; distinct unofficial adapters. - Open alternatives: direct App OAuth versus HA companion broker. - Rejected: tokens in App options; pasted callback URLs as tokens; generic retry of invalid grants; treating private APIs as official. diff --git a/specs/recording-identity-resolution.md b/specs/recording-identity-resolution.md index c7e5626..f309cd8 100644 --- a/specs/recording-identity-resolution.md +++ b/specs/recording-identity-resolution.md @@ -6,7 +6,7 @@ - Owners: Symphonia maintainers - Scope: resolve provider track representations to provider-independent recordings using explainable, versioned evidence and durable manual decisions. - Related requirements: `SYM-PROD-002`–`SYM-PROD-003`, `SYM-LIB-002`, `SYM-MATCH-001`–`SYM-MATCH-008`, `SYM-ARCH-014`, `SYM-TEST-007`–`SYM-TEST-008` -- Related decisions/research: [ADR 0001](../docs/decisions/0001-provider-independent-recording-domain.md), [identity domain model](../docs/domain/domain-model.md#identity-resolution-specification), [provider research](../docs/providers/provider-research.md), `RG-003` +- Related decisions/research: [ADR 0001](../docs/decisions/0001-provider-independent-recording-domain.md), [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [identity domain model](../docs/domain/domain-model.md#identity-resolution-specification), [provider research](../docs/providers/provider-research.md), [UI foundation](home-assistant-native-ui.md), `RG-003` - Required review gates: product UX, domain/music semantics, architecture, provider feasibility, testing/data licensing, documentation - Open decisions blocking readiness: reviewed labeled corpus; automatic-link precision/false-link tolerance; versioned scoring/confidence policy; provider candidate retrieval/quota evidence @@ -182,6 +182,8 @@ Users cannot lower safety thresholds ad hoc, enable title-only auto-linking, dis ## 9. UI/UX and content contract +Resolution queue, candidate comparison, evidence, history, and confirmation surfaces conform to the [Home Assistant-native UI foundation](home-assistant-native-ui.md). They compose shared cards/sections, status/alerts, tabs where justified, dense responsive result rows/tables, buttons, and adaptive dialogs. Music evidence may require richer comparison layouts than ordinary Home Assistant settings, but those layouts must keep Home Assistant density, tokens, focus, and action hierarchy and document any intentional divergence. + ### 9.1 Information hierarchy The review queue shows source identity/state first, then the best candidates and decisive evidence/differences, followed by one primary action. It never leads with an unexplained numeric score. @@ -271,6 +273,8 @@ Minimum **96 distinct cases** due to the high cost of false links: The corpus includes exact duplicates, missing/wrong/reused ISRC, covers, live/studio, remasters, remixes/edits, acoustic, clean/explicit, compilations, featured artists, localized metadata, music/lyric videos, uploads, duration drift, and misleading titles. Candidate recall and accepted-link precision are reported separately. Numeric acceptance thresholds remain a blocker until corpus review. +The twelve feature-specific UI cases supplement the UI-foundation budget and inherit its catalog, host-context, theme/localization, accessibility, responsive, hostile-content, and dated visual-reference gates. + ## 15. Documentation and discoverability | Audience | Artifact | Required content | Validation/navigation | @@ -292,6 +296,7 @@ The corpus includes exact duplicates, missing/wrong/reused ISRC, covers, live/st 8. Given automatic-link invalidation, future plans see ambiguity/unmatched while completed plan/history references remain unchanged. 9. Given hostile metadata/URLs/user notes, comparison UI and logs remain escaped, bounded, and non-executable. 10. Given resolver-version rollout/rollback, manual decisions persist and corpus regressions block release under the accepted thresholds. +11. Given unmatched, ambiguous, manually resolved, automatically invalidated, and provider-search-degraded fixtures, candidate/evidence views use shared Home Assistant-native components, remain keyboard-complete and comparison-readable on narrow screens, and never reduce evidence to color, artwork, or score alone. ## 17. Requirements traceability @@ -328,6 +333,7 @@ The corpus includes exact duplicates, missing/wrong/reused ISRC, covers, live/st - Primary sources: official provider metadata research in [provider research](../docs/providers/provider-research.md). - Related SDDs: [imports](library-import-and-provider-projections.md), [playlist copy](one-time-playlist-copy.md), [durable operations](durable-operations-and-recovery.md). +- Related UI contract: [Home Assistant-native UI foundation](home-assistant-native-ui.md) and [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md). - Accepted: recording identity is provider-independent; uncertainty/manual evidence are first-class. - Rejected: title-only matching, opaque score-only UX, silent automatic override of manual decisions, work-level cover merging. - Follow-up: musical-work relationships and additional metadata providers after MVP evidence. From 946b313c044f640bcca0d176bf9d2d31f0d54484 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 21:14:15 +0200 Subject: [PATCH 079/167] fix: protect durable checkpoints from credentials --- docs/development/implementation-baseline.md | 2 +- .../infrastructure/sqlite_operations.py | 3 + tests/test_sqlite_operations.py | 58 ++++++++++++++++++- 3 files changed, 59 insertions(+), 4 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index a64b0cb..6bccb51 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -29,7 +29,7 @@ The first implementation increment is intentionally narrower than any provider o - Application authorization boundary that generates one-use state without persisting the raw value. - Authorization callback binding rejects unsafe schemes, fragments, credentials, whitespace, and non-loopback HTTP hosts. - Normalized provider errors and provider codes redact common bearer/token/secret/password/cookie forms at the provider boundary. -- Operation persistence recursively rejects credential-shaped payload keys and cyclic structures before SQLite writes. +- Operation payloads and durable checkpoints recursively reject credential-shaped keys and cyclic structures before SQLite writes. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. - Copy plans bind target connection and effective write-capability evidence into their digest. - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index cc955b7..130975d 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -557,6 +557,7 @@ def checkpoint( if state not in {"running", "succeeded", "partial", "failed", "cancelled", "waiting_user"}: raise ValueError("invalid checkpoint state") + _validate_payload_keys(checkpoint) now_text = _utc(now) checkpoint_json = json.dumps(checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":")) self._connection.execute("BEGIN IMMEDIATE") @@ -697,6 +698,7 @@ def schedule_retry( ) -> OperationRecord: """Release a lease and persist a restart-safe retry time.""" + _validate_payload_keys(checkpoint) now_text = _utc(now) next_run_text = _utc(next_run_at) checkpoint_json = json.dumps(checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":")) @@ -764,6 +766,7 @@ def schedule_rate_limit( ) -> OperationRecord: """Release a lease until an absolute provider rate-limit time.""" + _validate_payload_keys(checkpoint) now_text = _utc(now) next_run_text = _utc(next_run_at) checkpoint_json = json.dumps(checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":")) diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index f29f809..46c719f 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -94,7 +94,7 @@ def test_checkpoint_events_store_only_sanitized_summary(self) -> None: checkpoint={ "confirmed_occurrences": ["occ-1", "occ-2"], "issues": [{"code": "provider_error"}], - "secret_token": "must-not-be-audit-payload", + "private_value": "must-not-be-an-audit-payload", }, now=self.now + timedelta(seconds=1), ) @@ -104,9 +104,61 @@ def test_checkpoint_events_store_only_sanitized_summary(self) -> None: self.assertEqual(event.payload["issues_count"], 1) self.assertEqual( event.payload["checkpoint_keys"], - ["confirmed_occurrences", "issues", "secret_token"], + ["confirmed_occurrences", "issues", "private_value"], ) - self.assertNotIn("must-not-be-audit-payload", event.payload) + self.assertNotIn("must-not-be-an-audit-payload", event.payload) + + def test_checkpoint_rejects_nested_credentials_before_persistence(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="checkpoint-credential", + payload={}, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) + + with self.assertRaisesRegex(ValueError, "access_token"): + self.repository.checkpoint( + operation.operation_id, + worker_id="worker-a", + checkpoint={"provider": {"access_token": "must-not-persist"}}, + now=self.now + timedelta(seconds=1), + ) + + self.assertEqual(self.repository.get(operation.operation_id).checkpoint, {}) + + def test_retry_and_rate_limit_checkpoints_reject_credentials(self) -> None: + retry_operation = self.repository.create( + operation_type="copy", + idempotency_key="retry-checkpoint-credential", + payload={}, + now=self.now, + ) + self.repository.claim(retry_operation.operation_id, worker_id="worker-a", now=self.now) + with self.assertRaises(ValueError): + self.repository.schedule_retry( + retry_operation.operation_id, + worker_id="worker-a", + next_run_at=self.now + timedelta(minutes=1), + checkpoint={"refresh_token": "must-not-persist"}, + now=self.now, + ) + + rate_limit_operation = self.repository.create( + operation_type="copy", + idempotency_key="rate-limit-checkpoint-credential", + payload={}, + now=self.now, + ) + self.repository.claim(rate_limit_operation.operation_id, worker_id="worker-a", now=self.now) + with self.assertRaises(ValueError): + self.repository.schedule_rate_limit( + rate_limit_operation.operation_id, + worker_id="worker-a", + next_run_at=self.now + timedelta(minutes=1), + checkpoint={"client_secret": "must-not-persist"}, + now=self.now, + ) def test_diagnostic_export_is_bounded_and_redacted(self) -> None: operation = self.repository.create( From 3f486e9763a860744d4aab5506aab69b6c035c50 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 21:55:30 +0200 Subject: [PATCH 080/167] fix: validate backup schema and foreign keys --- src/symphonia/runtime/resources.py | 47 +++++++++++++++++++++++++++++- tests/test_runtime_resources.py | 18 ++++++++++++ 2 files changed, 64 insertions(+), 1 deletion(-) diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index b5f9ce1..81c4e02 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -165,6 +165,40 @@ def validate_backup(cls, backup_path: str) -> bool: "current_playlist_snapshots", "resolution_decisions", } + required_columns = { + "operations": { + "operation_id", "operation_type", "state", "idempotency_key", + "payload_json", "checkpoint_json", "worker_id", "lease_expires_at", + "next_run_at", "cancel_requested", "created_at", "updated_at", + }, + "operation_events": { + "sequence", "operation_id", "event_type", "state", "worker_id", + "payload_json", "created_at", + }, + "copy_plans": {"digest", "plan_json", "created_at", "accepted_at"}, + "provider_connections": { + "connection_id", "provider", "provider_account_id", "state", + "manifest_version", "secret_ref", "capabilities_json", "expires_at", + "health_code", "created_at", "updated_at", + }, + "authorization_attempts": { + "attempt_id", "provider", "actor_id", "redirect_uri", "state_digest", + "state", "created_at", "expires_at", "completed_at", "failure_code", + }, + "playlist_snapshots": { + "snapshot_id", "provider", "namespace", "playlist_id", "revision", "published_at", + }, + "playlist_snapshot_entries": { + "snapshot_id", "occurrence_id", "position", "provider_track_id", + "provider_track_object_type", "provider_track_title", "source_added_at", + "provider_track_namespace", "media_kind", "available", + }, + "current_playlist_snapshots": {"provider", "namespace", "playlist_id", "snapshot_id"}, + "resolution_decisions": { + "sequence", "decision_id", "provider_track_key", "candidate_recording_id", + "action", "actor_id", "reason", "created_at", "payload_json", + }, + } connection: sqlite3.Connection | None = None try: connection = sqlite3.connect(uri, uri=True) @@ -177,7 +211,18 @@ def validate_backup(cls, backup_path: str) -> bool: "SELECT name FROM sqlite_master WHERE type = 'table'" ).fetchall() } - return required_tables.issubset(tables) + if not required_tables.issubset(tables): + return False + if connection.execute("PRAGMA foreign_key_check").fetchall(): + return False + for table, columns in required_columns.items(): + actual_columns = { + row[1] + for row in connection.execute(f"PRAGMA table_info({table})").fetchall() + } + if not columns.issubset(actual_columns): + return False + return True except sqlite3.Error: return False finally: diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index 0cb888a..2992795 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -4,6 +4,7 @@ import tempfile import unittest from datetime import datetime, timezone +import sqlite3 from symphonia.infrastructure import OperationRepository from symphonia.runtime import RuntimeResources @@ -151,6 +152,23 @@ def test_validate_backup_rejects_missing_or_corrupt_files(self) -> None: with self.assertRaises(ValueError): RuntimeResources.validate_backup(" ") + def test_validate_backup_rejects_schema_shaped_but_incompatible_files(self) -> None: + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "partial.sqlite3" + connection = sqlite3.connect(path) + try: + for table in ( + "operations", "operation_events", "copy_plans", "provider_connections", + "authorization_attempts", "playlist_snapshots", "playlist_snapshot_entries", + "current_playlist_snapshots", "resolution_decisions", + ): + connection.execute(f"CREATE TABLE {table} (placeholder TEXT)") + connection.commit() + finally: + connection.close() + + self.assertFalse(RuntimeResources.validate_backup(str(path))) + def test_diagnostics_combine_safe_queue_and_operation_views(self) -> None: with tempfile.TemporaryDirectory() as directory: resources = RuntimeResources.open(str(Path(directory) / "symphonia.sqlite3")) From 8147c2b6c4e9254915bd07a26b43b839864c6168 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 21:56:44 +0200 Subject: [PATCH 081/167] fix: cap support diagnostic sizes --- docs/development/implementation-baseline.md | 2 +- src/symphonia/infrastructure/sqlite_operations.py | 14 ++++++++------ src/symphonia/runtime/resources.py | 2 ++ tests/test_runtime_resources.py | 2 ++ tests/test_sqlite_operations.py | 4 ++++ 5 files changed, 17 insertions(+), 7 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 6bccb51..df52577 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -12,7 +12,7 @@ The first implementation increment is intentionally narrower than any provider o - Dependency-free SQLite operation repository proving idempotent creation, time-bounded worker leases, checkpoint ownership, retry scheduling, and expired-lease recovery. - Atomic scheduler-facing claim selection for queued, due-retry, and expired-lease operations. - Append-only SQLite operation event history for auditable transitions with sanitized checkpoint summaries. -- Bounded redacted operation diagnostics for support and a future authenticated operations view. +- Bounded redacted operation diagnostics for support and a future authenticated operations view, with hard item/event caps. - Bounded recent-operation diagnostics listing that exposes only redacted support views. - Aggregate operation queue summaries expose state counts, eligible age, and expired leases without payload data. - Explicit provider rate-limit waits with absolute retry times and durable eligibility. diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 130975d..2cc4e92 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -42,6 +42,8 @@ class LeaseConflict(RuntimeError): _SECRET_PAYLOAD_KEY = re.compile( r"(?i)(?:^|[_-])(access[_-]?token|refresh[_-]?token|id[_-]?token|client[_-]?secret|password|cookie|authorization|secret[_-]?token)(?:$|[_-])|^(?:secret|token)$" ) +_MAX_DIAGNOSTIC_OPERATIONS = 100 +_MAX_DIAGNOSTIC_EVENTS = 100 def _validate_payload_keys(payload: Any) -> None: @@ -256,8 +258,8 @@ def events(self, operation_id: str) -> tuple[OperationEvent, ...]: def diagnostic(self, operation_id: str, *, event_limit: int = 100) -> dict[str, Any]: """Return a bounded, redacted support view of one operation.""" - if event_limit <= 0: - raise ValueError("event_limit must be positive") + if not 0 < event_limit <= _MAX_DIAGNOSTIC_EVENTS: + raise ValueError(f"event_limit must be between 1 and {_MAX_DIAGNOSTIC_EVENTS}") record = self.get(operation_id) all_events = self.events(operation_id) selected_events = all_events[-event_limit:] @@ -289,10 +291,10 @@ def diagnostic(self, operation_id: str, *, event_limit: int = 100) -> dict[str, def diagnostics(self, *, limit: int = 50, event_limit: int = 20) -> tuple[dict[str, Any], ...]: """Return a bounded list of redacted operation support views.""" - if limit <= 0: - raise ValueError("limit must be positive") - if event_limit <= 0: - raise ValueError("event_limit must be positive") + if not 0 < limit <= _MAX_DIAGNOSTIC_OPERATIONS: + raise ValueError(f"limit must be between 1 and {_MAX_DIAGNOSTIC_OPERATIONS}") + if not 0 < event_limit <= _MAX_DIAGNOSTIC_EVENTS: + raise ValueError(f"event_limit must be between 1 and {_MAX_DIAGNOSTIC_EVENTS}") rows = self._connection.execute( """ SELECT operation_id diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 81c4e02..60d0cb1 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -242,6 +242,8 @@ def diagnostics( raise ValueError("operation_limit must be positive") if event_limit <= 0: raise ValueError("event_limit must be positive") + if operation_limit > 100 or event_limit > 100: + raise ValueError("diagnostic limits must not exceed 100") try: if not self.healthcheck(): return {"ready": False} diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index 2992795..d7f10f7 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -205,6 +205,8 @@ def test_diagnostics_combine_safe_queue_and_operation_views(self) -> None: self.assertNotIn("must-not-appear", str(diagnostics)) with self.assertRaises(ValueError): resources.diagnostics(now=datetime(2026, 9, 20, tzinfo=timezone.utc), operation_limit=0) + with self.assertRaises(ValueError): + resources.diagnostics(now=datetime(2026, 9, 20, tzinfo=timezone.utc), event_limit=101) finally: resources.close() diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 46c719f..6bc6c16 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -188,6 +188,8 @@ def test_diagnostic_export_is_bounded_and_redacted(self) -> None: self.assertNotIn("secret-plan", serialized) self.assertNotIn("secret-token", serialized) self.assertNotIn("provider-secret", serialized) + with self.assertRaises(ValueError): + self.repository.diagnostic(operation.operation_id, event_limit=101) def test_diagnostics_list_is_bounded_and_redacted(self) -> None: for index in range(3): @@ -207,6 +209,8 @@ def test_diagnostics_list_is_bounded_and_redacted(self) -> None: with self.assertRaises(ValueError): self.repository.diagnostics(limit=0) + with self.assertRaises(ValueError): + self.repository.diagnostics(limit=101) def test_queue_summary_is_aggregate_and_counts_only_eligible_work(self) -> None: queued = self.repository.create( From 0c9de2884a368c8793501215cc24c684936b6342 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 21:57:32 +0200 Subject: [PATCH 082/167] feat: support runtime resource context management --- docs/development/implementation-baseline.md | 1 + src/symphonia/runtime/resources.py | 6 ++++++ tests/test_runtime_resources.py | 8 ++++++++ 3 files changed, 15 insertions(+) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index df52577..5353de8 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -39,6 +39,7 @@ The first implementation increment is intentionally narrower than any provider o - Provider readers enforce bounded page sizes, repeated-cursor detection, and configurable maximum page counts. - Experimental Home Assistant App metadata scaffold with Ingress-only management and `/data` persistence. - Explicit runtime resource composition and reverse-order shutdown for all durable repositories. +- Runtime resources support explicit and context-manager lifecycle shutdown. - Readiness can validate every composed durable store instead of only the operation queue. - Runtime resources provide a consistent SQLite online-backup helper while the service remains open. - Runtime resources can preflight backup integrity and required durable tables read-only before restore design is selected. diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 60d0cb1..915ed65 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -76,6 +76,12 @@ def close(self) -> None: ): repository.close() + def __enter__(self) -> "RuntimeResources": + return self + + def __exit__(self, _exception_type: object, _exception: object, _traceback: object) -> None: + self.close() + def healthcheck(self) -> bool: """Check every durable store without exposing adapter internals.""" diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index d7f10f7..c45d905 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -39,6 +39,14 @@ def test_empty_database_path_is_rejected_before_opening_stores(self) -> None: with self.assertRaises(ValueError): RuntimeResources.open(" ") + def test_resources_support_context_manager_lifecycle(self) -> None: + with tempfile.TemporaryDirectory() as directory: + resources = RuntimeResources.open(str(Path(directory) / "symphonia.sqlite3")) + with resources as managed: + self.assertIs(managed, resources) + self.assertTrue(managed.healthcheck()) + self.assertFalse(resources.healthcheck()) + def test_readiness_fails_closed_when_one_store_is_closed(self) -> None: with tempfile.TemporaryDirectory() as directory: resources = RuntimeResources.open(str(Path(directory) / "symphonia.sqlite3")) From ddd1e345a6521e6322b8437f91e7b1626e5a29ad Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 22:06:00 +0200 Subject: [PATCH 083/167] docs: trace foundation evidence without advancing readiness --- specs/CATALOG.md | 4 ++ specs/catalog.json | 41 ++++++++++++++++--- specs/durable-operations-and-recovery.md | 2 + .../home-assistant-app-runtime-and-ingress.md | 10 ++++- 4 files changed, 49 insertions(+), 8 deletions(-) diff --git a/specs/CATALOG.md b/specs/CATALOG.md index 2ba29ad..b3e7bd7 100644 --- a/specs/CATALOG.md +++ b/specs/CATALOG.md @@ -14,6 +14,10 @@ This is the human-readable view of [`catalog.json`](./catalog.json). Until gener | `one-time-playlist-copy` | Draft | [One-time playlist copy](one-time-playlist-copy.md) | Blocked by unresolved-entry policy and proven target write semantics | | `durable-operations-and-recovery` | Draft | [Durable operations and recovery](durable-operations-and-recovery.md) | Blocked by the persistence/lease/restart spike and operating targets | +## Foundation evidence boundary + +`home-assistant-app-runtime` and `durable-operations-and-recovery` have owner-approved foundation evidence in `catalog.json`. The listed code, tests, and documentation cover only dependency-free persistence, lifecycle composition, packaging, redaction, diagnostics, backup preflight, and deterministic worker mechanics. They do not change either capability's `Draft` status or clear its blockers; OAuth, secret ownership, complete migration/restore, production topology, UI, and provider-vertical acceptance remain gated by their SDDs. + ## Deliberately absent There is no implementation SDD for persistent playlist synchronization. It remains specified only at the domain/future level until: diff --git a/specs/catalog.json b/specs/catalog.json index 043ffcb..dd1f401 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -51,9 +51,23 @@ "docs/providers/home-assistant-ecosystem-review.md" ], "implementationEvidence": { - "code": [], - "tests": [], - "documentation": [] + "code": [ + "src/symphonia/runtime/config.py", + "src/symphonia/runtime/http.py", + "src/symphonia/runtime/resources.py", + "Dockerfile", + "addon/config.yaml" + ], + "tests": [ + "tests/test_runtime_config.py", + "tests/test_runtime_http.py", + "tests/test_runtime_resources.py", + "tests/test_homeassistant_app_metadata.py" + ], + "documentation": [ + "docs/development/implementation-baseline.md", + "specs/home-assistant-app-runtime-and-ingress.md" + ] }, "blockers": [ "RG-004 storage and recovery spike", @@ -348,9 +362,24 @@ "docs/providers/home-assistant-ecosystem-review.md" ], "implementationEvidence": { - "code": [], - "tests": [], - "documentation": [] + "code": [ + "src/symphonia/infrastructure/sqlite_operations.py", + "src/symphonia/application/operation_runner.py", + "src/symphonia/application/operation_worker.py", + "src/symphonia/application/copy_execution.py", + "src/symphonia/application/library_import_execution.py" + ], + "tests": [ + "tests/test_sqlite_operations.py", + "tests/test_operation_runner.py", + "tests/test_operation_worker.py", + "tests/test_copy_execution.py", + "tests/test_library_import_execution.py" + ], + "documentation": [ + "docs/development/implementation-baseline.md", + "specs/durable-operations-and-recovery.md" + ] }, "blockers": [ "RG-004 storage engine and lease/crash-recovery spike", diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 40ac08f..fc5625e 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,6 +22,8 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics, provider-independent handlers, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. + Evidence sources: - [Product specification](../docs/product/product-specification.md) diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index e09fd7f..5ad83d0 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,9 +30,15 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -No Symphonia runtime exists. The accepted behavior is limited to ADR 0003 and global requirements. This SDD is prospective and does not imply that App packaging, images, listeners, or migrations have been implemented. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing, composed SQLite stores, transactionally consistent backup/preflight helpers, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. -### 2.3 Evidence and unknowns +This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. + +### 2.3 Foundation evidence boundary + +The current foundation is intentionally limited to reversible composition, persistence, packaging, and safety contracts. It is useful for later spikes and does not resolve `RG-004`, the supported Home Assistant matrix, encryption-key/backup ownership, or standalone-release decisions. New App behavior must wait for those gates and explicit owner approval. + +### 2.4 Evidence and unknowns - Accepted evidence: [ADR 0003](../docs/decisions/0003-home-assistant-app-primary.md). - Platform evidence: current App, Ingress, persistent `/data`, backup, and security documentation linked from [provider research](../docs/providers/provider-research.md#home-assistant-platform). From 30f2afec9a7ea14d1ab65de587d27c2291620f1f Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 22:08:01 +0200 Subject: [PATCH 084/167] test: enforce dependency-free architecture boundaries --- docs/development/implementation-baseline.md | 1 + specs/catalog.json | 6 ++- tests/test_architecture_boundaries.py | 56 +++++++++++++++++++++ 3 files changed, 61 insertions(+), 2 deletions(-) create mode 100644 tests/test_architecture_boundaries.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 5353de8..e3c9c64 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -72,6 +72,7 @@ docker run --rm -p 8099:8099 -v symphonia-data:/data symphonia:dev The container exposes only the current health/readiness/version surface. A future App manifest must add Ingress, Supervisor metadata, supported architectures, backup declarations, and any direct callback policy only after the runtime SDD blockers are resolved. - Deterministic `unittest` coverage under `tests/`. +- Static AST boundary tests keep the domain free of adapters/host frameworks and keep the foundation limited to Python's standard library plus local `symphonia` modules. The package is an implementation foundation, not a claim that the corresponding capability SDDs are complete. Provider OAuth, secret storage, user authentication, Ingress UI, a complete migration ledger beyond the current forward-compatible SQLite path, backup retention/restore policy, and production runtime handler wiring remain unimplemented and blocked by their SDD decisions. The Home Assistant-native UI direction and compatibility-layer boundary are documented in [ADR 0004](../decisions/0004-home-assistant-native-ui.md) and the [UI foundation SDD](../../specs/home-assistant-native-ui.md), but this foundation slice does not authorize or include frontend implementation. diff --git a/specs/catalog.json b/specs/catalog.json index dd1f401..772bfd7 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -62,7 +62,8 @@ "tests/test_runtime_config.py", "tests/test_runtime_http.py", "tests/test_runtime_resources.py", - "tests/test_homeassistant_app_metadata.py" + "tests/test_homeassistant_app_metadata.py", + "tests/test_architecture_boundaries.py" ], "documentation": [ "docs/development/implementation-baseline.md", @@ -374,7 +375,8 @@ "tests/test_operation_runner.py", "tests/test_operation_worker.py", "tests/test_copy_execution.py", - "tests/test_library_import_execution.py" + "tests/test_library_import_execution.py", + "tests/test_architecture_boundaries.py" ], "documentation": [ "docs/development/implementation-baseline.md", diff --git a/tests/test_architecture_boundaries.py b/tests/test_architecture_boundaries.py new file mode 100644 index 0000000..d5d0a95 --- /dev/null +++ b/tests/test_architecture_boundaries.py @@ -0,0 +1,56 @@ +from __future__ import annotations + +import ast +from pathlib import Path +import sys +import unittest + + +ROOT = Path(__file__).resolve().parents[1] +SOURCE_ROOT = ROOT / "src" / "symphonia" + + +def _absolute_imports(path: Path) -> list[str]: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + imports: list[str] = [] + for node in ast.walk(tree): + if isinstance(node, ast.Import): + imports.extend(alias.name for alias in node.names) + elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: + imports.append(node.module) + return imports + + +class ArchitectureBoundaryTests(unittest.TestCase): + def test_domain_does_not_import_adapters_or_host_frameworks(self) -> None: + forbidden_prefixes = ( + "symphonia.infrastructure", + "symphonia.providers", + "symphonia.runtime", + "homeassistant", + "supervisor", + "aiohttp", + "fastapi", + "flask", + "starlette", + ) + violations = [] + for path in sorted((SOURCE_ROOT / "domain").glob("*.py")): + for module in _absolute_imports(path): + if module.startswith(forbidden_prefixes): + violations.append(f"{path.relative_to(ROOT)} imports {module}") + self.assertEqual(violations, []) + + def test_foundation_source_uses_only_stdlib_and_local_modules(self) -> None: + stdlib = set(sys.stdlib_module_names) + violations = [] + for path in sorted(SOURCE_ROOT.rglob("*.py")): + for module in _absolute_imports(path): + root = module.split(".", 1)[0] + if root not in stdlib and root != "symphonia": + violations.append(f"{path.relative_to(ROOT)} imports {module}") + self.assertEqual(violations, []) + + +if __name__ == "__main__": + unittest.main() From 9c5b68012211486c87fb6bbf0bde22b3a8c5776e Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 20 Sep 2026 22:11:20 +0200 Subject: [PATCH 085/167] fix: harden runtime configuration boundaries --- docs/development/implementation-baseline.md | 1 + specs/home-assistant-app-runtime-and-ingress.md | 2 ++ src/symphonia/runtime/config.py | 14 ++++++++++++-- src/symphonia/runtime/http.py | 4 ++++ tests/test_runtime_config.py | 11 +++++++++++ tests/test_runtime_http.py | 5 +++-- 6 files changed, 33 insertions(+), 4 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index e3c9c64..00e03b2 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -44,6 +44,7 @@ The first implementation increment is intentionally narrower than any provider o - Runtime resources provide a consistent SQLite online-backup helper while the service remains open. - Runtime resources can preflight backup integrity and required durable tables read-only before restore design is selected. - Runtime configuration validates host, port, database path, and Ingress base path before startup. +- Runtime configuration rejects control characters, non-integral ports, and query/fragment-bearing Ingress paths before startup. - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. - Provider connection diagnostics expose provider/state and expiry counts, never account or credential data. - Import diagnostics expose bounded snapshot freshness and availability counts without playlist content. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 5ad83d0..6af7200 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -156,6 +156,8 @@ No final App option names are accepted yet. The first implementation RFC should Invalid or unknown values fail closed before workers start. Authentication, non-root execution, secret redaction, audit history, and mount/network restrictions are not configurable. +The current foundation profile applies this boundary before server creation: host and database values reject control characters, ports must be integral and within the valid TCP range, and the Ingress base path must be a normalized path without query, fragment, control, or traversal segments. These checks are implementation evidence only; final App option names and supported deployment values remain open. + ## 8. Clean Architecture design ### 8.1 Responsibilities and dependency direction diff --git a/src/symphonia/runtime/config.py b/src/symphonia/runtime/config.py index a4c5afd..15a7b73 100644 --- a/src/symphonia/runtime/config.py +++ b/src/symphonia/runtime/config.py @@ -10,6 +10,10 @@ def _normalize_ingress_path(value: str) -> str: if not isinstance(value, str) or not value or not value.startswith("/"): raise ValueError("ingress path must start with '/'") + if any(ord(char) < 0x20 or ord(char) == 0x7F for char in value): + raise ValueError("ingress path must not contain control characters") + if "?" in value or "#" in value: + raise ValueError("ingress path must contain only a path") normalized = value.rstrip("/") or "/" if "//" in normalized or "/.." in normalized or "/./" in normalized: raise ValueError("ingress path contains an unsafe segment") @@ -17,6 +21,10 @@ def _normalize_ingress_path(value: str) -> str: def _parse_port(value: object) -> int: + if isinstance(value, bool): + raise ValueError("port must be an integer") + if isinstance(value, float) and not value.is_integer(): + raise ValueError("port must be an integer") try: port = int(value) # type: ignore[arg-type] except (TypeError, ValueError) as error: @@ -41,12 +49,14 @@ def __post_init__(self) -> None: host = self.host.strip() if not host or any(char.isspace() for char in host): raise ValueError("host must be a non-empty value without whitespace") + if any(ord(char) < 0x20 or ord(char) == 0x7F for char in host): + raise ValueError("host must not contain control characters") object.__setattr__(self, "host", host) object.__setattr__(self, "port", _parse_port(self.port)) if not isinstance(self.database_path, str) or not self.database_path.strip(): raise ValueError("database_path must not be empty") - if "\x00" in self.database_path: - raise ValueError("database_path must not contain NUL bytes") + if any(ord(char) < 0x20 or ord(char) == 0x7F for char in self.database_path): + raise ValueError("database_path must not contain control characters") object.__setattr__(self, "database_path", self.database_path.strip()) object.__setattr__(self, "ingress_path", _normalize_ingress_path(self.ingress_path)) diff --git a/src/symphonia/runtime/http.py b/src/symphonia/runtime/http.py index a5e6e0f..435a435 100644 --- a/src/symphonia/runtime/http.py +++ b/src/symphonia/runtime/http.py @@ -120,6 +120,10 @@ def route_get( def _normalize_base_path(value: str) -> str: if not value or not value.startswith("/"): raise ValueError("ingress path must start with '/'") + if any(ord(char) < 0x20 or ord(char) == 0x7F for char in value): + raise ValueError("ingress path must not contain control characters") + if "?" in value or "#" in value: + raise ValueError("ingress path must contain only a path") normalized = value.rstrip("/") or "/" if "//" in normalized or "/.." in normalized or "/./" in normalized: raise ValueError("ingress path contains an unsafe segment") diff --git a/tests/test_runtime_config.py b/tests/test_runtime_config.py index e718198..9946f9d 100644 --- a/tests/test_runtime_config.py +++ b/tests/test_runtime_config.py @@ -37,8 +37,10 @@ def test_invalid_configuration_fails_before_runtime_start(self) -> None: {"host": "host with spaces"}, {"database_path": " "}, {"database_path": "bad\x00path"}, + {"database_path": "bad\npath"}, {"ingress_path": "relative"}, {"ingress_path": "/bad/../path"}, + {"ingress_path": "/local_symphonia?query"}, {"ingress_path": 1}, ) for values in invalid_values: @@ -48,6 +50,15 @@ def test_invalid_configuration_fails_before_runtime_start(self) -> None: with self.assertRaisesRegex(ValueError, "port"): RuntimeConfig.from_environment({"SYMPHONIA_PORT": "invalid"}) + def test_rejects_invalid_host_and_non_integral_port(self) -> None: + for values in ( + {"host": "127.0.0.1\x00"}, + {"port": True}, + {"port": 8099.5}, + ): + with self.subTest(values=values), self.assertRaises(ValueError): + RuntimeConfig(**values) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_runtime_http.py b/tests/test_runtime_http.py index 4643e1c..806c4c0 100644 --- a/tests/test_runtime_http.py +++ b/tests/test_runtime_http.py @@ -49,8 +49,9 @@ def test_ingress_base_path_is_stripped_without_accepting_sibling_paths(self) -> self.assertEqual(payload, {"error": "not_found"}) def test_unsafe_ingress_base_path_is_rejected(self) -> None: - with self.assertRaises(ValueError): - route_get("/ready", self.repository, ingress_path="/bad/../path") + for ingress_path in ("/bad/../path", "/local_symphonia?query", "/local\x00symphonia"): + with self.assertRaises(ValueError): + route_get("/ready", self.repository, ingress_path=ingress_path) def test_readiness_can_use_the_composed_runtime_healthcheck(self) -> None: status, payload = route_get("/ready", self.repository, readiness_check=lambda: False) From 1f11b0a997f2d53451a8308e523c59d2b2e66930 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 21 Sep 2026 02:06:02 +0200 Subject: [PATCH 086/167] fix: make runtime shutdown idempotent --- docs/development/implementation-baseline.md | 1 + specs/home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/resources.py | 9 ++++++++- tests/test_runtime_resources.py | 10 ++++++++++ 4 files changed, 20 insertions(+), 2 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 00e03b2..de23854 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -40,6 +40,7 @@ The first implementation increment is intentionally narrower than any provider o - Experimental Home Assistant App metadata scaffold with Ingress-only management and `/data` persistence. - Explicit runtime resource composition and reverse-order shutdown for all durable repositories. - Runtime resources support explicit and context-manager lifecycle shutdown. +- Runtime resource shutdown is idempotent and remains not-ready after closure, so repeated Supervisor/finally cleanup cannot reopen or report healthy stores. - Readiness can validate every composed durable store instead of only the operation queue. - Runtime resources provide a consistent SQLite online-backup helper while the service remains open. - Runtime resources can preflight backup integrity and required durable tables read-only before restore design is selected. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 6af7200..6bd3a9c 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing, composed SQLite stores, transactionally consistent backup/preflight helpers, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing, composed SQLite stores, transactionally consistent backup/preflight helpers, idempotent resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 915ed65..6880a0c 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -2,7 +2,7 @@ from __future__ import annotations -from dataclasses import dataclass +from dataclasses import dataclass, field from datetime import datetime import os from pathlib import Path @@ -38,6 +38,7 @@ class RuntimeResources: projections: PlaylistProjectionRepository resolutions: ResolutionDecisionRepository database_path: str = ":memory:" + _closed: bool = field(default=False, init=False, repr=False) @classmethod def open(cls, database_path: str) -> "RuntimeResources": @@ -66,6 +67,10 @@ def open(cls, database_path: str) -> "RuntimeResources": def close(self) -> None: """Close repositories in reverse dependency/startup order.""" + if self._closed: + return + self._closed = True + for repository in ( self.resolutions, self.projections, @@ -85,6 +90,8 @@ def __exit__(self, _exception_type: object, _exception: object, _traceback: obje def healthcheck(self) -> bool: """Check every durable store without exposing adapter internals.""" + if self._closed: + return False try: for repository in ( self.operations, diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index c45d905..e88d9a4 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -47,6 +47,16 @@ def test_resources_support_context_manager_lifecycle(self) -> None: self.assertTrue(managed.healthcheck()) self.assertFalse(resources.healthcheck()) + def test_close_is_idempotent_and_keeps_runtime_not_ready(self) -> None: + with tempfile.TemporaryDirectory() as directory: + resources = RuntimeResources.open(str(Path(directory) / "symphonia.sqlite3")) + resources.close() + resources.close() + + self.assertFalse(resources.healthcheck()) + with self.assertRaisesRegex(RuntimeError, "unhealthy"): + resources.backup_to(str(Path(directory) / "backup.sqlite3")) + def test_readiness_fails_closed_when_one_store_is_closed(self) -> None: with tempfile.TemporaryDirectory() as directory: resources = RuntimeResources.open(str(Path(directory) / "symphonia.sqlite3")) From a03ddb88fbf1b167db456ac849a34cb8d44e9db2 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 21 Sep 2026 02:07:56 +0200 Subject: [PATCH 087/167] fix: enforce foreign keys across sqlite stores --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- src/symphonia/infrastructure/sqlite_authorization.py | 1 + src/symphonia/infrastructure/sqlite_connections.py | 1 + src/symphonia/infrastructure/sqlite_library.py | 1 + src/symphonia/infrastructure/sqlite_plans.py | 1 + src/symphonia/infrastructure/sqlite_resolutions.py | 1 + tests/test_runtime_resources.py | 12 ++++++++++++ 8 files changed, 19 insertions(+), 1 deletion(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index de23854..6699d1b 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -39,6 +39,7 @@ The first implementation increment is intentionally narrower than any provider o - Provider readers enforce bounded page sizes, repeated-cursor detection, and configurable maximum page counts. - Experimental Home Assistant App metadata scaffold with Ingress-only management and `/data` persistence. - Explicit runtime resource composition and reverse-order shutdown for all durable repositories. +- Every SQLite repository enables foreign-key enforcement at connection startup; backup preflight remains a separate integrity check. - Runtime resources support explicit and context-manager lifecycle shutdown. - Runtime resource shutdown is idempotent and remains not-ready after closure, so repeated Supervisor/finally cleanup cannot reopen or report healthy stores. - Readiness can validate every composed durable store instead of only the operation queue. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index fc5625e..247acc3 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics, provider-independent handlers, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics, provider-independent handlers, foreign-key enforcement, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_authorization.py b/src/symphonia/infrastructure/sqlite_authorization.py index 1a56a69..7bbb5ef 100644 --- a/src/symphonia/infrastructure/sqlite_authorization.py +++ b/src/symphonia/infrastructure/sqlite_authorization.py @@ -40,6 +40,7 @@ class AuthorizationAttemptRepository: def __init__(self, path: str = ":memory:") -> None: self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) self._connection.row_factory = sqlite3.Row + self._connection.execute("PRAGMA foreign_keys = ON") self._migrate() def close(self) -> None: diff --git a/src/symphonia/infrastructure/sqlite_connections.py b/src/symphonia/infrastructure/sqlite_connections.py index 9795afb..b6bd7bc 100644 --- a/src/symphonia/infrastructure/sqlite_connections.py +++ b/src/symphonia/infrastructure/sqlite_connections.py @@ -35,6 +35,7 @@ class ProviderConnectionRepository: def __init__(self, path: str = ":memory:") -> None: self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) self._connection.row_factory = sqlite3.Row + self._connection.execute("PRAGMA foreign_keys = ON") self._migrate() def close(self) -> None: diff --git a/src/symphonia/infrastructure/sqlite_library.py b/src/symphonia/infrastructure/sqlite_library.py index e06a58d..45e3d45 100644 --- a/src/symphonia/infrastructure/sqlite_library.py +++ b/src/symphonia/infrastructure/sqlite_library.py @@ -47,6 +47,7 @@ class PlaylistProjectionRepository: def __init__(self, path: str = ":memory:") -> None: self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) self._connection.row_factory = sqlite3.Row + self._connection.execute("PRAGMA foreign_keys = ON") self._migrate() def close(self) -> None: diff --git a/src/symphonia/infrastructure/sqlite_plans.py b/src/symphonia/infrastructure/sqlite_plans.py index 3c56a29..1915006 100644 --- a/src/symphonia/infrastructure/sqlite_plans.py +++ b/src/symphonia/infrastructure/sqlite_plans.py @@ -43,6 +43,7 @@ class CopyPlanRepository: def __init__(self, path: str = ":memory:") -> None: self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) self._connection.row_factory = sqlite3.Row + self._connection.execute("PRAGMA foreign_keys = ON") self._migrate() def close(self) -> None: diff --git a/src/symphonia/infrastructure/sqlite_resolutions.py b/src/symphonia/infrastructure/sqlite_resolutions.py index 663e362..d2fcc48 100644 --- a/src/symphonia/infrastructure/sqlite_resolutions.py +++ b/src/symphonia/infrastructure/sqlite_resolutions.py @@ -17,6 +17,7 @@ class ResolutionDecisionRepository: def __init__(self, path: str = ":memory:") -> None: self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) self._connection.row_factory = sqlite3.Row + self._connection.execute("PRAGMA foreign_keys = ON") self._migrate() def close(self) -> None: diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index e88d9a4..29d8086 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -29,6 +29,18 @@ def test_open_creates_all_durable_store_schemas_and_closes_them(self) -> None: self.assertIn("authorization_attempts", tables) self.assertIn("resolution_decisions", tables) self.assertTrue(resources.healthcheck()) + for repository in ( + resources.operations, + resources.plans, + resources.connections, + resources.authorization, + resources.projections, + resources.resolutions, + ): + self.assertEqual( + repository._connection.execute("PRAGMA foreign_keys").fetchone()[0], # type: ignore[attr-defined] + 1, + ) finally: resources.close() From fc4803ebe9a940752d5bd4afdf13c1050ee48de3 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 21 Sep 2026 02:08:36 +0200 Subject: [PATCH 088/167] test: verify sqlite orphan rejection --- tests/test_runtime_resources.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index 29d8086..e46de31 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -51,6 +51,21 @@ def test_empty_database_path_is_rejected_before_opening_stores(self) -> None: with self.assertRaises(ValueError): RuntimeResources.open(" ") + def test_foreign_keys_reject_orphan_projection_rows(self) -> None: + with tempfile.TemporaryDirectory() as directory: + resources = RuntimeResources.open(str(Path(directory) / "symphonia.sqlite3")) + try: + with self.assertRaises(sqlite3.IntegrityError): + resources.projections._connection.execute( # type: ignore[attr-defined] + """ + INSERT INTO current_playlist_snapshots ( + provider, namespace, playlist_id, snapshot_id + ) VALUES ('test', 'test', 'playlist', 'missing-snapshot') + """ + ) + finally: + resources.close() + def test_resources_support_context_manager_lifecycle(self) -> None: with tempfile.TemporaryDirectory() as directory: resources = RuntimeResources.open(str(Path(directory) / "symphonia.sqlite3")) From 3c0a4954218d71393244c3f411b210c279cdf2be Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 21 Sep 2026 02:17:48 +0200 Subject: [PATCH 089/167] refactor: centralize sqlite connection policy --- docs/development/implementation-baseline.md | 1 + specs/catalog.json | 1 + specs/durable-operations-and-recovery.md | 2 +- .../infrastructure/sqlite_authorization.py | 6 +++--- src/symphonia/infrastructure/sqlite_common.py | 18 ++++++++++++++++++ .../infrastructure/sqlite_connections.py | 6 +++--- src/symphonia/infrastructure/sqlite_library.py | 6 +++--- .../infrastructure/sqlite_operations.py | 7 +++---- src/symphonia/infrastructure/sqlite_plans.py | 6 +++--- .../infrastructure/sqlite_resolutions.py | 6 +++--- tests/test_runtime_resources.py | 4 ++++ 11 files changed, 43 insertions(+), 20 deletions(-) create mode 100644 src/symphonia/infrastructure/sqlite_common.py diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 6699d1b..5173183 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -40,6 +40,7 @@ The first implementation increment is intentionally narrower than any provider o - Experimental Home Assistant App metadata scaffold with Ingress-only management and `/data` persistence. - Explicit runtime resource composition and reverse-order shutdown for all durable repositories. - Every SQLite repository enables foreign-key enforcement at connection startup; backup preflight remains a separate integrity check. +- A shared SQLite connection policy applies the same five-second busy timeout and row-factory settings to every durable store. - Runtime resources support explicit and context-manager lifecycle shutdown. - Runtime resource shutdown is idempotent and remains not-ready after closure, so repeated Supervisor/finally cleanup cannot reopen or report healthy stores. - Readiness can validate every composed durable store instead of only the operation queue. diff --git a/specs/catalog.json b/specs/catalog.json index 772bfd7..6f4df94 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -364,6 +364,7 @@ ], "implementationEvidence": { "code": [ + "src/symphonia/infrastructure/sqlite_common.py", "src/symphonia/infrastructure/sqlite_operations.py", "src/symphonia/application/operation_runner.py", "src/symphonia/application/operation_worker.py", diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 247acc3..673c77f 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics, provider-independent handlers, foreign-key enforcement, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics, provider-independent handlers, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_authorization.py b/src/symphonia/infrastructure/sqlite_authorization.py index 7bbb5ef..a07cc29 100644 --- a/src/symphonia/infrastructure/sqlite_authorization.py +++ b/src/symphonia/infrastructure/sqlite_authorization.py @@ -9,6 +9,8 @@ from symphonia.providers.authorization import AuthorizationAttempt, AuthorizationState, validate_redirect_uri +from .sqlite_common import connect + def _utc(value: datetime) -> str: if value.tzinfo is None: @@ -38,9 +40,7 @@ class AuthorizationAttemptRepository: """Store only authorization correlation metadata, never raw state values.""" def __init__(self, path: str = ":memory:") -> None: - self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) - self._connection.row_factory = sqlite3.Row - self._connection.execute("PRAGMA foreign_keys = ON") + self._connection = connect(path) self._migrate() def close(self) -> None: diff --git a/src/symphonia/infrastructure/sqlite_common.py b/src/symphonia/infrastructure/sqlite_common.py new file mode 100644 index 0000000..b940a9d --- /dev/null +++ b/src/symphonia/infrastructure/sqlite_common.py @@ -0,0 +1,18 @@ +"""Shared connection policy for the dependency-free SQLite adapters.""" + +from __future__ import annotations + +import sqlite3 + + +def connect(path: str) -> sqlite3.Connection: + """Open a repository connection with the runtime safety defaults.""" + + connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) + connection.row_factory = sqlite3.Row + connection.execute("PRAGMA foreign_keys = ON") + connection.execute("PRAGMA busy_timeout = 5000") + return connection + + +__all__ = ["connect"] diff --git a/src/symphonia/infrastructure/sqlite_connections.py b/src/symphonia/infrastructure/sqlite_connections.py index b6bd7bc..f8daa5c 100644 --- a/src/symphonia/infrastructure/sqlite_connections.py +++ b/src/symphonia/infrastructure/sqlite_connections.py @@ -10,6 +10,8 @@ from symphonia.providers.connections import ConnectionState, ProviderConnection from symphonia.providers.contracts import Capability, ProviderCapabilities +from .sqlite_common import connect + def _utc(value: datetime) -> str: if value.tzinfo is None: @@ -33,9 +35,7 @@ class ProviderConnectionRepository: """Persist account identity and capability evidence, never secret contents.""" def __init__(self, path: str = ":memory:") -> None: - self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) - self._connection.row_factory = sqlite3.Row - self._connection.execute("PRAGMA foreign_keys = ON") + self._connection = connect(path) self._migrate() def close(self) -> None: diff --git a/src/symphonia/infrastructure/sqlite_library.py b/src/symphonia/infrastructure/sqlite_library.py index 45e3d45..0f08c0c 100644 --- a/src/symphonia/infrastructure/sqlite_library.py +++ b/src/symphonia/infrastructure/sqlite_library.py @@ -11,6 +11,8 @@ from symphonia.providers.contracts import ProviderPlaylistEntry from symphonia.providers.importing import CollectionImportResult +from .sqlite_common import connect + def _utc(value: datetime) -> str: if value.tzinfo is None: @@ -45,9 +47,7 @@ class PlaylistProjectionRepository: """Keep the last complete projection when a later import is incomplete.""" def __init__(self, path: str = ":memory:") -> None: - self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) - self._connection.row_factory = sqlite3.Row - self._connection.execute("PRAGMA foreign_keys = ON") + self._connection = connect(path) self._migrate() def close(self) -> None: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 2cc4e92..abdd274 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -16,6 +16,8 @@ from typing import Any import uuid +from .sqlite_common import connect + def _utc(value: datetime) -> str: if value.tzinfo is None: @@ -120,10 +122,7 @@ class OperationRepository: SCHEMA_VERSION = 3 def __init__(self, path: str = ":memory:") -> None: - self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) - self._connection.row_factory = sqlite3.Row - self._connection.execute("PRAGMA foreign_keys = ON") - self._connection.execute("PRAGMA busy_timeout = 5000") + self._connection = connect(path) self._migrate() def close(self) -> None: diff --git a/src/symphonia/infrastructure/sqlite_plans.py b/src/symphonia/infrastructure/sqlite_plans.py index 1915006..e2bef30 100644 --- a/src/symphonia/infrastructure/sqlite_plans.py +++ b/src/symphonia/infrastructure/sqlite_plans.py @@ -16,6 +16,8 @@ PlanAcceptanceError, ) +from .sqlite_common import connect + def _utc(value: datetime) -> str: if value.tzinfo is None: @@ -41,9 +43,7 @@ class CopyPlanRepository: """A small SQLite adapter that never mutates a plan after creation.""" def __init__(self, path: str = ":memory:") -> None: - self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) - self._connection.row_factory = sqlite3.Row - self._connection.execute("PRAGMA foreign_keys = ON") + self._connection = connect(path) self._migrate() def close(self) -> None: diff --git a/src/symphonia/infrastructure/sqlite_resolutions.py b/src/symphonia/infrastructure/sqlite_resolutions.py index d2fcc48..55f2c50 100644 --- a/src/symphonia/infrastructure/sqlite_resolutions.py +++ b/src/symphonia/infrastructure/sqlite_resolutions.py @@ -10,14 +10,14 @@ from symphonia.identity.models import ManualDecision, ManualDecisionAction +from .sqlite_common import connect + class ResolutionDecisionRepository: """Preserve every decision; latest state never erases prior authorship.""" def __init__(self, path: str = ":memory:") -> None: - self._connection = sqlite3.connect(path, isolation_level=None, check_same_thread=False) - self._connection.row_factory = sqlite3.Row - self._connection.execute("PRAGMA foreign_keys = ON") + self._connection = connect(path) self._migrate() def close(self) -> None: diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index e46de31..2ade9de 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -41,6 +41,10 @@ def test_open_creates_all_durable_store_schemas_and_closes_them(self) -> None: repository._connection.execute("PRAGMA foreign_keys").fetchone()[0], # type: ignore[attr-defined] 1, ) + self.assertEqual( + repository._connection.execute("PRAGMA busy_timeout").fetchone()[0], # type: ignore[attr-defined] + 5000, + ) finally: resources.close() From 55b9c7fbcc10c6646a18f30bb90c31ee73f9e92a Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 21 Sep 2026 02:19:39 +0200 Subject: [PATCH 090/167] fix: require object-shaped durable payloads --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- .../infrastructure/sqlite_operations.py | 14 ++++++++--- tests/test_sqlite_operations.py | 25 +++++++++++++++++++ 4 files changed, 37 insertions(+), 5 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 5173183..f1b3144 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -30,6 +30,7 @@ The first implementation increment is intentionally narrower than any provider o - Authorization callback binding rejects unsafe schemes, fragments, credentials, whitespace, and non-loopback HTTP hosts. - Normalized provider errors and provider codes redact common bearer/token/secret/password/cookie forms at the provider boundary. - Operation payloads and durable checkpoints recursively reject credential-shaped keys and cyclic structures before SQLite writes. +- Durable operation intents, checkpoints, retries, and rate-limit waits require JSON-object payloads before SQLite writes. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. - Copy plans bind target connection and effective write-capability evidence into their digest. - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 673c77f..e6b40a5 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics, provider-independent handlers, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics, provider-independent handlers, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, JSON-object payload validation, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index abdd274..9559263 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -87,6 +87,12 @@ def walk(value: Any, path: str = "") -> None: ) +def _validate_object_payload(payload: Any, *, label: str) -> None: + if not isinstance(payload, dict): + raise ValueError(f"{label} must be a JSON object") + _validate_payload_keys(payload) + + @dataclass(frozen=True, slots=True) class OperationRecord: operation_id: str @@ -194,7 +200,7 @@ def create( if not operation_type.strip() or not idempotency_key.strip(): raise ValueError("operation_type and idempotency_key must not be empty") - _validate_payload_keys(payload) + _validate_object_payload(payload, label="operation payload") operation_id = operation_id or str(uuid.uuid4()) timestamp = _utc(now) payload_json = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")) @@ -558,7 +564,7 @@ def checkpoint( if state not in {"running", "succeeded", "partial", "failed", "cancelled", "waiting_user"}: raise ValueError("invalid checkpoint state") - _validate_payload_keys(checkpoint) + _validate_object_payload(checkpoint, label="checkpoint") now_text = _utc(now) checkpoint_json = json.dumps(checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":")) self._connection.execute("BEGIN IMMEDIATE") @@ -699,7 +705,7 @@ def schedule_retry( ) -> OperationRecord: """Release a lease and persist a restart-safe retry time.""" - _validate_payload_keys(checkpoint) + _validate_object_payload(checkpoint, label="checkpoint") now_text = _utc(now) next_run_text = _utc(next_run_at) checkpoint_json = json.dumps(checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":")) @@ -767,7 +773,7 @@ def schedule_rate_limit( ) -> OperationRecord: """Release a lease until an absolute provider rate-limit time.""" - _validate_payload_keys(checkpoint) + _validate_object_payload(checkpoint, label="checkpoint") now_text = _utc(now) next_run_text = _utc(next_run_at) checkpoint_json = json.dumps(checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":")) diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 6bc6c16..3af3cd1 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -300,6 +300,31 @@ def test_operation_payload_rejects_cyclic_structures_before_json_encoding(self) now=self.now, ) + def test_operation_payload_and_checkpoint_must_be_json_objects(self) -> None: + with self.assertRaisesRegex(ValueError, "operation payload must be a JSON object"): + self.repository.create( + operation_type="copy", + idempotency_key="list-payload", + payload=[], # type: ignore[arg-type] + now=self.now, + ) + + operation = self.repository.create( + operation_type="copy", + idempotency_key="list-checkpoint", + payload={}, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) + with self.assertRaisesRegex(ValueError, "checkpoint must be a JSON object"): + self.repository.checkpoint( + operation.operation_id, + worker_id="worker-a", + checkpoint=[], # type: ignore[arg-type] + now=self.now + timedelta(seconds=1), + ) + self.assertEqual(self.repository.get(operation.operation_id).state, "running") + def test_only_lease_owner_can_checkpoint(self) -> None: operation = self.repository.create( operation_type="import", From 52880af8e005c47608b5f8c3c4986f1d05668d0b Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 21 Sep 2026 02:20:57 +0200 Subject: [PATCH 091/167] fix: reject non-string durable payload keys --- docs/development/implementation-baseline.md | 2 +- specs/durable-operations-and-recovery.md | 2 +- src/symphonia/infrastructure/sqlite_operations.py | 2 ++ tests/test_sqlite_operations.py | 9 +++++++++ 4 files changed, 13 insertions(+), 2 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index f1b3144..d332d8c 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -30,7 +30,7 @@ The first implementation increment is intentionally narrower than any provider o - Authorization callback binding rejects unsafe schemes, fragments, credentials, whitespace, and non-loopback HTTP hosts. - Normalized provider errors and provider codes redact common bearer/token/secret/password/cookie forms at the provider boundary. - Operation payloads and durable checkpoints recursively reject credential-shaped keys and cyclic structures before SQLite writes. -- Durable operation intents, checkpoints, retries, and rate-limit waits require JSON-object payloads before SQLite writes. +- Durable operation intents, checkpoints, retries, and rate-limit waits require JSON-object payloads with string keys before SQLite writes. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. - Copy plans bind target connection and effective write-capability evidence into their digest. - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index e6b40a5..723717b 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics, provider-independent handlers, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, JSON-object payload validation, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics, provider-independent handlers, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, JSON-object/string-key payload validation, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 9559263..8b02232 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -60,6 +60,8 @@ def walk(value: Any, path: str = "") -> None: active_containers.add(identity) try: for key, nested in value.items(): + if not isinstance(key, str): + raise ValueError("operation payload object keys must be strings") key_text = str(key) key_path = key_text if not path else f"{path}.{key_text}" if _SECRET_PAYLOAD_KEY.search(key_text): diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 3af3cd1..3414007 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -325,6 +325,15 @@ def test_operation_payload_and_checkpoint_must_be_json_objects(self) -> None: ) self.assertEqual(self.repository.get(operation.operation_id).state, "running") + def test_operation_payload_rejects_non_string_object_keys(self) -> None: + with self.assertRaisesRegex(ValueError, "keys must be strings"): + self.repository.create( + operation_type="copy", + idempotency_key="numeric-key", + payload={1: "must-not-be-coerced"}, # type: ignore[dict-item] + now=self.now, + ) + def test_only_lease_owner_can_checkpoint(self) -> None: operation = self.repository.create( operation_type="import", From b7a63786e199be21396e1dc07e523b287b59a830 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 21 Sep 2026 02:23:40 +0200 Subject: [PATCH 092/167] fix: bound diagnostic key lists --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- .../infrastructure/sqlite_operations.py | 14 +++++++++-- tests/test_sqlite_operations.py | 23 +++++++++++++++++++ 4 files changed, 37 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index d332d8c..c930621 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -13,6 +13,7 @@ The first implementation increment is intentionally narrower than any provider o - Atomic scheduler-facing claim selection for queued, due-retry, and expired-lease operations. - Append-only SQLite operation event history for auditable transitions with sanitized checkpoint summaries. - Bounded redacted operation diagnostics for support and a future authenticated operations view, with hard item/event caps. +- Diagnostic payload/checkpoint key lists are capped at 100 entries and mark truncation. - Bounded recent-operation diagnostics listing that exposes only redacted support views. - Aggregate operation queue summaries expose state counts, eligible age, and expired leases without payload data. - Explicit provider rate-limit waits with absolute retry times and durable eligibility. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 723717b..d445b67 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics, provider-independent handlers, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, JSON-object/string-key payload validation, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists, provider-independent handlers, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, JSON-object/string-key payload validation, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 8b02232..86ddca8 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -46,6 +46,7 @@ class LeaseConflict(RuntimeError): ) _MAX_DIAGNOSTIC_OPERATIONS = 100 _MAX_DIAGNOSTIC_EVENTS = 100 +_MAX_DIAGNOSTIC_KEYS = 100 def _validate_payload_keys(payload: Any) -> None: @@ -270,6 +271,7 @@ def diagnostic(self, operation_id: str, *, event_limit: int = 100) -> dict[str, record = self.get(operation_id) all_events = self.events(operation_id) selected_events = all_events[-event_limit:] + payload_keys, payload_keys_truncated = self._bounded_keys(record.payload) return { "operation_id": record.operation_id, "operation_type": record.operation_type, @@ -279,7 +281,8 @@ def diagnostic(self, operation_id: str, *, event_limit: int = 100) -> dict[str, "cancel_requested": record.cancel_requested, "created_at": _utc(record.created_at), "updated_at": _utc(record.updated_at), - "payload_keys": sorted(str(key) for key in record.payload), + "payload_keys": payload_keys, + "payload_keys_truncated": payload_keys_truncated, "checkpoint": self._checkpoint_summary(record.checkpoint), "events_truncated": len(selected_events) != len(all_events), "events": [ @@ -868,8 +871,10 @@ def _append_event( def _checkpoint_summary(checkpoint: dict[str, Any]) -> dict[str, Any]: """Keep audit data useful while excluding checkpoint values by default.""" + checkpoint_keys, checkpoint_keys_truncated = OperationRepository._bounded_keys(checkpoint) summary: dict[str, Any] = { - "checkpoint_keys": sorted(str(key) for key in checkpoint), + "checkpoint_keys": checkpoint_keys, + "checkpoint_keys_truncated": checkpoint_keys_truncated, } for key in ("confirmed_occurrences", "issues"): value = checkpoint.get(key) @@ -877,6 +882,11 @@ def _checkpoint_summary(checkpoint: dict[str, Any]) -> dict[str, Any]: summary[f"{key}_count"] = len(value) return summary + @staticmethod + def _bounded_keys(value: dict[str, Any]) -> tuple[list[str], bool]: + keys = sorted(str(key) for key in value) + return keys[:_MAX_DIAGNOSTIC_KEYS], len(keys) > _MAX_DIAGNOSTIC_KEYS + @staticmethod def _record(row: sqlite3.Row) -> OperationRecord: return OperationRecord( diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 3414007..a650cf5 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -212,6 +212,29 @@ def test_diagnostics_list_is_bounded_and_redacted(self) -> None: with self.assertRaises(ValueError): self.repository.diagnostics(limit=101) + def test_diagnostic_key_lists_are_bounded_and_mark_truncation(self) -> None: + payload = {f"key-{index:03d}": index for index in range(101)} + operation = self.repository.create( + operation_type="copy", + idempotency_key="bounded-diagnostic-keys", + payload=payload, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) + self.repository.checkpoint( + operation.operation_id, + worker_id="worker-a", + checkpoint={f"checkpoint-{index:03d}": index for index in range(101)}, + now=self.now + timedelta(seconds=1), + ) + + diagnostic = self.repository.diagnostic(operation.operation_id) + + self.assertEqual(len(diagnostic["payload_keys"]), 100) + self.assertTrue(diagnostic["payload_keys_truncated"]) + self.assertEqual(len(diagnostic["checkpoint"]["checkpoint_keys"]), 100) + self.assertTrue(diagnostic["checkpoint"]["checkpoint_keys_truncated"]) + def test_queue_summary_is_aggregate_and_counts_only_eligible_work(self) -> None: queued = self.repository.create( operation_type="copy", From 2150fedd551bbca1b642bec64d82b962306807ed Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 21 Sep 2026 02:24:49 +0200 Subject: [PATCH 093/167] fix: bound diagnostic event reads --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- .../infrastructure/sqlite_operations.py | 18 ++++++++++--- tests/test_sqlite_operations.py | 25 +++++++++++++++++++ 4 files changed, 42 insertions(+), 4 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index c930621..2af3001 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -14,6 +14,7 @@ The first implementation increment is intentionally narrower than any provider o - Append-only SQLite operation event history for auditable transitions with sanitized checkpoint summaries. - Bounded redacted operation diagnostics for support and a future authenticated operations view, with hard item/event caps. - Diagnostic payload/checkpoint key lists are capped at 100 entries and mark truncation. +- Diagnostic event queries fetch only the requested window plus one row to determine truncation. - Bounded recent-operation diagnostics listing that exposes only redacted support views. - Aggregate operation queue summaries expose state counts, eligible age, and expired leases without payload data. - Explicit provider rate-limit waits with absolute retry times and durable eligibility. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index d445b67..43c1a96 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists, provider-independent handlers, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, JSON-object/string-key payload validation, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, JSON-object/string-key payload validation, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 86ddca8..4591c5e 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -269,8 +269,20 @@ def diagnostic(self, operation_id: str, *, event_limit: int = 100) -> dict[str, if not 0 < event_limit <= _MAX_DIAGNOSTIC_EVENTS: raise ValueError(f"event_limit must be between 1 and {_MAX_DIAGNOSTIC_EVENTS}") record = self.get(operation_id) - all_events = self.events(operation_id) - selected_events = all_events[-event_limit:] + event_rows = self._connection.execute( + """ + SELECT sequence, operation_id, event_type, state, worker_id, payload_json, created_at + FROM operation_events + WHERE operation_id = ? + ORDER BY sequence DESC + LIMIT ? + """, + (operation_id, event_limit + 1), + ).fetchall() + events_truncated = len(event_rows) > event_limit + selected_events = tuple( + self._event(row) for row in reversed(event_rows[:event_limit]) + ) payload_keys, payload_keys_truncated = self._bounded_keys(record.payload) return { "operation_id": record.operation_id, @@ -284,7 +296,7 @@ def diagnostic(self, operation_id: str, *, event_limit: int = 100) -> dict[str, "payload_keys": payload_keys, "payload_keys_truncated": payload_keys_truncated, "checkpoint": self._checkpoint_summary(record.checkpoint), - "events_truncated": len(selected_events) != len(all_events), + "events_truncated": events_truncated, "events": [ { "sequence": event.sequence, diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index a650cf5..df40aa5 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -235,6 +235,31 @@ def test_diagnostic_key_lists_are_bounded_and_mark_truncation(self) -> None: self.assertEqual(len(diagnostic["checkpoint"]["checkpoint_keys"]), 100) self.assertTrue(diagnostic["checkpoint"]["checkpoint_keys_truncated"]) + def test_diagnostic_reads_only_a_bounded_event_window(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="bounded-event-window", + payload={}, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) + for index in range(12): + self.repository.checkpoint( + operation.operation_id, + worker_id="worker-a", + checkpoint={"step": index}, + now=self.now + timedelta(seconds=index + 1), + ) + + diagnostic = self.repository.diagnostic(operation.operation_id, event_limit=5) + + self.assertTrue(diagnostic["events_truncated"]) + self.assertEqual(len(diagnostic["events"]), 5) + self.assertEqual( + [event["event_type"] for event in diagnostic["events"]], + ["checkpointed"] * 5, + ) + def test_queue_summary_is_aggregate_and_counts_only_eligible_work(self) -> None: queued = self.repository.create( operation_type="copy", From c1788a3d953e7c86635effa869938e407d004664 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 21 Sep 2026 02:44:48 +0200 Subject: [PATCH 094/167] fix: migrate operation schema transactionally --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- .../infrastructure/sqlite_operations.py | 95 +++++++++++-------- tests/test_sqlite_operations.py | 7 ++ 4 files changed, 66 insertions(+), 39 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 2af3001..203f4bd 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -58,6 +58,7 @@ The first implementation increment is intentionally narrower than any provider o - Ingress-relative health/version routing with normalized, traversal-safe base paths. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. +- The operation store applies its schema DDL, legacy column migration, and `user_version` marker in one transaction. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. - Lossless Unicode-safe identity normalization with explicit version-token and ISRC derived fields; no automatic matching thresholds are assumed. - An application workflow that persists plans, requires digest acceptance, and enqueues only accepted plans as durable operations. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 43c1a96..c2b3429 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, JSON-object/string-key payload validation, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, JSON-object/string-key payload validation, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 4591c5e..6fb2ca2 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -149,46 +149,65 @@ def _migrate(self) -> None: raise RuntimeError( f"operation store schema {current_version} is newer than supported {self.SCHEMA_VERSION}" ) - self._connection.executescript( - """ - CREATE TABLE IF NOT EXISTS operations ( - operation_id TEXT PRIMARY KEY, - operation_type TEXT NOT NULL, - state TEXT NOT NULL, - idempotency_key TEXT NOT NULL UNIQUE, - payload_json TEXT NOT NULL, - checkpoint_json TEXT NOT NULL, - worker_id TEXT, - lease_expires_at TEXT, - next_run_at TEXT, - cancel_requested INTEGER NOT NULL DEFAULT 0, - created_at TEXT NOT NULL, - updated_at TEXT NOT NULL - ); - CREATE INDEX IF NOT EXISTS operations_eligibility_idx - ON operations (state, next_run_at, lease_expires_at); - CREATE TABLE IF NOT EXISTS operation_events ( - sequence INTEGER PRIMARY KEY AUTOINCREMENT, - operation_id TEXT NOT NULL REFERENCES operations(operation_id), - event_type TEXT NOT NULL, - state TEXT NOT NULL, - worker_id TEXT, - payload_json TEXT NOT NULL, - created_at TEXT NOT NULL - ); - CREATE INDEX IF NOT EXISTS operation_events_operation_idx - ON operation_events (operation_id, sequence); - """ - ) - columns = { - row[1] - for row in self._connection.execute("PRAGMA table_info(operations)").fetchall() - } - if "cancel_requested" not in columns: + self._connection.execute("BEGIN IMMEDIATE") + try: + self._connection.execute( + """ + CREATE TABLE IF NOT EXISTS operations ( + operation_id TEXT PRIMARY KEY, + operation_type TEXT NOT NULL, + state TEXT NOT NULL, + idempotency_key TEXT NOT NULL UNIQUE, + payload_json TEXT NOT NULL, + checkpoint_json TEXT NOT NULL, + worker_id TEXT, + lease_expires_at TEXT, + next_run_at TEXT, + cancel_requested INTEGER NOT NULL DEFAULT 0, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL + ) + """ + ) self._connection.execute( - "ALTER TABLE operations ADD COLUMN cancel_requested INTEGER NOT NULL DEFAULT 0" + """ + CREATE INDEX IF NOT EXISTS operations_eligibility_idx + ON operations (state, next_run_at, lease_expires_at) + """ ) - self._connection.execute(f"PRAGMA user_version = {self.SCHEMA_VERSION}") + self._connection.execute( + """ + CREATE TABLE IF NOT EXISTS operation_events ( + sequence INTEGER PRIMARY KEY AUTOINCREMENT, + operation_id TEXT NOT NULL REFERENCES operations(operation_id), + event_type TEXT NOT NULL, + state TEXT NOT NULL, + worker_id TEXT, + payload_json TEXT NOT NULL, + created_at TEXT NOT NULL + ) + """ + ) + self._connection.execute( + """ + CREATE INDEX IF NOT EXISTS operation_events_operation_idx + ON operation_events (operation_id, sequence) + """ + ) + columns = { + row[1] + for row in self._connection.execute("PRAGMA table_info(operations)").fetchall() + } + if "cancel_requested" not in columns: + self._connection.execute( + "ALTER TABLE operations ADD COLUMN cancel_requested INTEGER NOT NULL DEFAULT 0" + ) + self._connection.execute(f"PRAGMA user_version = {self.SCHEMA_VERSION}") + self._connection.execute("COMMIT") + except Exception: + if self._connection.in_transaction: + self._connection.execute("ROLLBACK") + raise def create( self, diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index df40aa5..c04a0b8 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -700,6 +700,13 @@ def test_legacy_store_is_migrated_forward_without_losing_operations(self) -> Non finally: repository.close() + def test_fresh_operation_store_commits_schema_and_version_together(self) -> None: + self.assertEqual( + self.repository._connection.execute("PRAGMA user_version").fetchone()[0], # type: ignore[attr-defined] + self.repository.SCHEMA_VERSION, + ) + self.assertFalse(self.repository._connection.in_transaction) # type: ignore[attr-defined] + if __name__ == "__main__": unittest.main() From 39b0c8740777b1ce78aa7e75b8e3ea6cbb0cb6da Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 21 Sep 2026 12:21:12 +0200 Subject: [PATCH 095/167] fix: reject future operation schemas in backups --- docs/development/implementation-baseline.md | 1 + .../home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/resources.py | 5 +++++ tests/test_runtime_resources.py | 21 +++++++++++++++++++ 4 files changed, 28 insertions(+), 1 deletion(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 203f4bd..58cf7a7 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -49,6 +49,7 @@ The first implementation increment is intentionally narrower than any provider o - Readiness can validate every composed durable store instead of only the operation queue. - Runtime resources provide a consistent SQLite online-backup helper while the service remains open. - Runtime resources can preflight backup integrity and required durable tables read-only before restore design is selected. +- Backup preflight rejects operation databases whose schema version is newer than the running foundation. - Runtime configuration validates host, port, database path, and Ingress base path before startup. - Runtime configuration rejects control characters, non-integral ports, and query/fragment-bearing Ingress paths before startup. - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 6bd3a9c..f87de53 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing, composed SQLite stores, transactionally consistent backup/preflight helpers, idempotent resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing, composed SQLite stores, transactionally consistent backup/preflight helpers with future-schema rejection, idempotent resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 6880a0c..e7ac78a 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -218,6 +218,11 @@ def validate_backup(cls, backup_path: str) -> bool: integrity = connection.execute("PRAGMA integrity_check").fetchone() if integrity is None or integrity[0] != "ok": return False + operation_schema_version = int( + connection.execute("PRAGMA user_version").fetchone()[0] + ) + if operation_schema_version > OperationRepository.SCHEMA_VERSION: + return False tables = { row[0] for row in connection.execute( diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index 2ade9de..f97b4d9 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -218,6 +218,27 @@ def test_validate_backup_rejects_schema_shaped_but_incompatible_files(self) -> N self.assertFalse(RuntimeResources.validate_backup(str(path))) + def test_validate_backup_rejects_a_future_operation_schema(self) -> None: + with tempfile.TemporaryDirectory() as directory: + source_path = str(Path(directory) / "symphonia.sqlite3") + backup_path = Path(directory) / "backup.sqlite3" + resources = RuntimeResources.open(source_path) + try: + resources.backup_to(str(backup_path)) + finally: + resources.close() + + connection = sqlite3.connect(backup_path) + try: + connection.execute( + f"PRAGMA user_version = {OperationRepository.SCHEMA_VERSION + 1}" + ) + connection.commit() + finally: + connection.close() + + self.assertFalse(RuntimeResources.validate_backup(str(backup_path))) + def test_diagnostics_combine_safe_queue_and_operation_views(self) -> None: with tempfile.TemporaryDirectory() as directory: resources = RuntimeResources.open(str(Path(directory) / "symphonia.sqlite3")) From 185b012a75781a6d172c3e226a3e6b8f6e97d4c4 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 21 Sep 2026 12:22:40 +0200 Subject: [PATCH 096/167] fix: validate operation handler results --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- src/symphonia/application/operation_runner.py | 7 ++++++- tests/test_operation_runner.py | 20 +++++++++++++++++++ 4 files changed, 28 insertions(+), 2 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 58cf7a7..2ca86ea 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -24,6 +24,7 @@ The first implementation increment is intentionally narrower than any provider o - Application connection service for verified-account registration, capability probes, degraded health, and reauthorization-required classification. - Pure capability-layer intersection and requirement checks for adapter/connection/object/health constraints. - Durable operation runner that atomically claims eligible work and fails unwired operation types before side effects. +- The operation runner validates handler result type and operation identity before returning a claimed result. - Cooperative single-process operation worker with interruptible polling and injected clock support. - Copy executor can run as a claimed operation handler, preserving the same restart/checkpoint semantics under the runner. - Playlist import executor persists intent before reads and reports succeeded, partial, waiting-user, retry, and rate-limit outcomes durably. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index c2b3429..d160dbb 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, JSON-object/string-key payload validation, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, JSON-object/string-key payload validation, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/application/operation_runner.py b/src/symphonia/application/operation_runner.py index 89e246b..836a315 100644 --- a/src/symphonia/application/operation_runner.py +++ b/src/symphonia/application/operation_runner.py @@ -43,7 +43,12 @@ def run_once( state="failed", ) try: - return handler(operation, worker_id, now) + result = handler(operation, worker_id, now) + if not isinstance(result, OperationRecord): + raise TypeError("operation handler must return OperationRecord") + if result.operation_id != operation.operation_id: + raise ValueError("operation handler returned a different operation") + return result except Exception as error: # A handler must never strand a claimed operation in ``running``. # Persist only a stable exception class marker: provider details diff --git a/tests/test_operation_runner.py b/tests/test_operation_runner.py index 1b50b1b..1ac1242 100644 --- a/tests/test_operation_runner.py +++ b/tests/test_operation_runner.py @@ -82,6 +82,26 @@ def handler(claimed, worker_id, now): ["created", "claimed", "checkpointed"], ) + def test_invalid_handler_result_is_terminal_and_does_not_strand_work(self) -> None: + operation = self.repository.create( + operation_type="fixture", + idempotency_key="fixture-invalid-result", + payload={}, + now=NOW, + ) + + def handler(_claimed, _worker_id, _now): + return None + + result = OperationRunner(self.repository, {"fixture": handler}).run_once( + worker_id="worker-a", now=NOW + ) + + self.assertEqual(result.operation_id, operation.operation_id) + self.assertEqual(result.state, "failed") + self.assertEqual(result.checkpoint["failure_code"], "handler_exception:TypeError") + self.assertEqual(self.repository.get(operation.operation_id).state, "failed") + if __name__ == "__main__": unittest.main() From 1ad205e4015eaa0587e19ab8309b54ab769a12c5 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Tue, 22 Sep 2026 10:12:42 +0200 Subject: [PATCH 097/167] fix: harden runtime json response headers --- docs/development/implementation-baseline.md | 1 + .../home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/http.py | 3 +++ tests/test_runtime_http.py | 27 ++++++++++++++++++- 4 files changed, 31 insertions(+), 2 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 2ca86ea..706f895 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -58,6 +58,7 @@ The first implementation increment is intentionally narrower than any provider o - Import diagnostics expose bounded snapshot freshness and availability counts without playlist content. - Resolution diagnostics expose only manual-decision action counts, never track, actor, or reason data. - Ingress-relative health/version routing with normalized, traversal-safe base paths. +- The current JSON health/readiness/version surface disables caching, MIME sniffing, and referrer propagation. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - The operation store applies its schema DDL, legacy column migration, and `user_version` marker in one transaction. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index f87de53..a9e7485 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing, composed SQLite stores, transactionally consistent backup/preflight helpers with future-schema rejection, idempotent resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing with bounded response headers, composed SQLite stores, transactionally consistent backup/preflight helpers with future-schema rejection, idempotent resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/runtime/http.py b/src/symphonia/runtime/http.py index 435a435..e8439d8 100644 --- a/src/symphonia/runtime/http.py +++ b/src/symphonia/runtime/http.py @@ -59,6 +59,9 @@ def _json(self, status: int, payload: dict[str, Any]) -> None: self.send_response(status) self.send_header("Content-Type", "application/json; charset=utf-8") self.send_header("Content-Length", str(len(body))) + self.send_header("Cache-Control", "no-store") + self.send_header("X-Content-Type-Options", "nosniff") + self.send_header("Referrer-Policy", "no-referrer") self.end_headers() self.wfile.write(body) diff --git a/tests/test_runtime_http.py b/tests/test_runtime_http.py index 806c4c0..278b9e2 100644 --- a/tests/test_runtime_http.py +++ b/tests/test_runtime_http.py @@ -1,9 +1,10 @@ from __future__ import annotations +from io import BytesIO import unittest from symphonia.infrastructure import OperationRepository -from symphonia.runtime.http import route_get +from symphonia.runtime.http import SymphoniaRequestHandler, route_get class RuntimeHTTPTests(unittest.TestCase): @@ -59,6 +60,30 @@ def test_readiness_can_use_the_composed_runtime_healthcheck(self) -> None: self.assertEqual(status, 503) self.assertEqual(payload["status"], "not_ready") + def test_json_surface_sets_no_cache_and_content_sniffing_headers(self) -> None: + class FakeHandler: + def __init__(self) -> None: + self.status = None + self.headers = {} + self.wfile = BytesIO() + + def send_response(self, status: int) -> None: + self.status = status + + def send_header(self, name: str, value: str) -> None: + self.headers[name] = value + + def end_headers(self) -> None: + return + + handler = FakeHandler() + SymphoniaRequestHandler._json(handler, 200, {"status": "ok"}) # type: ignore[arg-type] + + self.assertEqual(handler.status, 200) + self.assertEqual(handler.headers["Cache-Control"], "no-store") + self.assertEqual(handler.headers["X-Content-Type-Options"], "nosniff") + self.assertEqual(handler.headers["Referrer-Policy"], "no-referrer") + if __name__ == "__main__": unittest.main() From 64f4f613a4ca7ae541fc33079008447c303a07e5 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Tue, 22 Sep 2026 12:00:00 +0200 Subject: [PATCH 098/167] feat: harden runtime and durable recovery foundation --- .github/workflows/verify.yml | 28 ++ Makefile | 14 + README.md | 10 + docs/architecture/system-architecture.md | 13 +- .../0003-home-assistant-app-primary.md | 8 +- docs/development/implementation-baseline.md | 21 +- docs/development/oauth-callback-spike.md | 33 +++ docs/development/storage-recovery-spike.md | 33 +++ docs/open-questions.md | 33 ++- .../home-assistant-ecosystem-review.md | 2 +- docs/providers/provider-research.md | 19 +- specs/CATALOG.md | 2 +- specs/README.md | 6 + specs/catalog.json | 22 +- specs/durable-operations-and-recovery.md | 2 +- .../home-assistant-app-runtime-and-ingress.md | 2 +- .../provider-connections-and-authorization.md | 44 ++-- src/symphonia/application/authorization.py | 52 ++++ src/symphonia/application/operation_worker.py | 13 +- src/symphonia/domain/models.py | 34 +++ .../infrastructure/sqlite_authorization.py | 48 +++- src/symphonia/infrastructure/sqlite_common.py | 26 +- .../infrastructure/sqlite_connections.py | 22 +- .../infrastructure/sqlite_library.py | 10 +- .../infrastructure/sqlite_operations.py | 115 ++++++-- src/symphonia/infrastructure/sqlite_plans.py | 24 +- .../infrastructure/sqlite_resolutions.py | 22 +- src/symphonia/providers/apple_music.py | 70 ++++- src/symphonia/providers/authorization.py | 19 +- src/symphonia/providers/contracts.py | 65 ++++- src/symphonia/providers/errors.py | 26 +- src/symphonia/providers/spotify.py | 86 +++++- src/symphonia/providers/youtube.py | 19 +- src/symphonia/runtime/http.py | 12 +- src/symphonia/runtime/resources.py | 10 +- tests/test_apple_music_adapter.py | 31 +++ tests/test_authorization_service.py | 38 +++ tests/test_copy_planning.py | 26 ++ tests/test_identity_resolution.py | 22 ++ tests/test_oauth_callback_spike.py | 174 ++++++++++++ tests/test_operation_worker.py | 13 + tests/test_provider_connections.py | 17 +- tests/test_provider_import.py | 75 ++++++ tests/test_runtime_http.py | 65 ++++- tests/test_runtime_resources.py | 95 +++++++ tests/test_spec_validator.py | 14 + tests/test_spotify_adapter.py | 80 ++++++ tests/test_sqlite_authorization.py | 134 ++++++++++ tests/test_sqlite_connections.py | 11 + tests/test_sqlite_operations.py | 198 ++++++++++++++ tests/test_sqlite_plans.py | 31 ++- tests/test_storage_recovery_spike.py | 29 ++ tests/test_youtube_adapter.py | 30 +++ tools/__init__.py | 1 + tools/oauth_callback_spike.py | 174 ++++++++++++ tools/storage_recovery_spike.py | 100 +++++++ tools/validate_specs.py | 248 ++++++++++++++++++ 57 files changed, 2431 insertions(+), 140 deletions(-) create mode 100644 .github/workflows/verify.yml create mode 100644 Makefile create mode 100644 docs/development/oauth-callback-spike.md create mode 100644 docs/development/storage-recovery-spike.md create mode 100644 tests/test_oauth_callback_spike.py create mode 100644 tests/test_spec_validator.py create mode 100644 tests/test_storage_recovery_spike.py create mode 100644 tools/__init__.py create mode 100644 tools/oauth_callback_spike.py create mode 100644 tools/storage_recovery_spike.py create mode 100644 tools/validate_specs.py diff --git a/.github/workflows/verify.yml b/.github/workflows/verify.yml new file mode 100644 index 0000000..e17cc29 --- /dev/null +++ b/.github/workflows/verify.yml @@ -0,0 +1,28 @@ +name: Verify + +on: + push: + pull_request: + +permissions: + contents: read + +jobs: + verify: + runs-on: ubuntu-latest + strategy: + matrix: + python-version: ["3.11", "3.12", "3.13"] + steps: + - name: Check out repository + uses: actions/checkout@v4 + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + - name: Check specification consistency + run: PYTHONPATH=. python tools/validate_specs.py + - name: Check patch whitespace + run: git diff --check + - name: Run offline test suite + run: PYTHONPATH=src:. python -m unittest discover -s tests -v diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..f118bdb --- /dev/null +++ b/Makefile @@ -0,0 +1,14 @@ +.PHONY: verify specs test spike-storage + +specs: + PYTHONPATH=. python3 tools/validate_specs.py + +test: + PYTHONPATH=src:. python3 -m unittest discover -s tests -v + +spike-storage: + PYTHONPATH=src:. python3 tools/storage_recovery_spike.py --operations 1000 + +verify: specs + git diff --check + $(MAKE) test diff --git a/README.md b/README.md index 16ac909..ca2902a 100644 --- a/README.md +++ b/README.md @@ -55,3 +55,13 @@ Persistent synchronization follows only after copy semantics and official provid ## Contributing during specification Use requirement identifiers in issues, SDDs, tests, and future commits. Material implementation work starts from the applicable cataloged SDD and its numeric verification budget. A change to accepted behavior must update the relevant horizontal specification and, when it changes an architectural decision, add or supersede an ADR. Do not infer a decision from an open question, and do not start production implementation until the SDD is ready and the owner explicitly approves it. + +Before opening a change, run `make verify`. It checks the SDD/catalog links and +traceability, patch whitespace, and the offline test suite without provider +accounts, Home Assistant, or network access. + +The two current evidence spikes can be run independently: `make spike-storage` +exercises restart/lease recovery and SQLite backup validation, while the +[OAuth callback spike](docs/development/oauth-callback-spike.md) documents the +direct App-owned callback boundary. Both are research evidence, not published +production endpoints. diff --git a/docs/architecture/system-architecture.md b/docs/architecture/system-architecture.md index 4c9f2ee..87ee230 100644 --- a/docs/architecture/system-architecture.md +++ b/docs/architecture/system-architecture.md @@ -130,7 +130,13 @@ Whether standalone packaging ships in the first public release or immediately af A companion custom integration MAY later expose native entities, actions, events, and configuration discovery. It must call a stable, authenticated Symphonia API and MUST NOT duplicate matching, sync, credential, or retry logic. -The integration MAY also be evaluated as a narrow provider-authorization broker so Symphonia can reuse Home Assistant's Application Credentials/config-flow callback machinery. If selected, it must exchange an opaque, one-use connection grant over the authenticated local API; provider operation logic remains in the App. Token ownership, refresh, revocation, backup, failure recovery, and integration/App version skew must be specified before this is accepted. +The MVP does not require the companion integration to broker provider +authorization. Provider authorization is owned by the App's adapters following +the App-plus-Ingress pattern. A future proposal MAY evaluate a narrow broker +to reuse Home Assistant's Application Credentials/config-flow machinery; if +selected, it must exchange an opaque, one-use connection grant over the +authenticated local API, and token ownership, refresh, revocation, +backup/recovery, and version skew must be specified before acceptance. Candidate native surface (illustrative, not accepted): @@ -146,7 +152,8 @@ Three approaches require an RFC: | MQTT discovery/events | Mature decoupling and push model | Adds an MQTT dependency and weakens direct operation correlation | | App calls Home Assistant APIs directly | Fewer artifacts for events/actions initiated by the App | Couples the service to Home Assistant and does not cleanly provide a native integration surface | -The companion-integration approach is the current leading direction, not yet an accepted implementation decision. +The App-plus-Ingress approach is the accepted primary direction. The +companion-integration approach is deferred to a future native-surface RFC. ## Service/API shape @@ -266,7 +273,7 @@ There are three distinct concerns: Provider redirect URIs must be exact and externally reachable under provider rules, while Home Assistant Ingress uses a proxied base path and session. The official Home Assistant Spotify integration demonstrates `https://my.home-assistant.io/redirect/oauth` and `/auth/external/callback` through Home Assistant's Application Credentials/config-flow machinery. A Supervisor App does not automatically inherit that machinery; using it would require a companion integration or another explicitly designed broker. -The project MUST complete an OAuth callback spike for local-only and externally reachable Home Assistant deployments before provider authentication architecture is accepted. It must compare a direct App flow with a minimal companion-integration authorization broker and cover HTTPS, redirect registration, Ingress session continuity, remote access, dynamic paths, user-provided OAuth clients, state/PKCE/single use, denial/error flows, token ownership, integration/App version skew, backup/restore, and reauthorization. Any direct callback port must satisfy `SYM-SEC-009`. Tokens MUST NOT be pasted into App options or browser-export files as a normal workaround. +The project MUST complete an OAuth callback spike for local-only and externally reachable Home Assistant deployments before provider authentication architecture is accepted. It must validate the direct App flow and cover HTTPS, redirect registration, Ingress session continuity, remote access, dynamic paths, user-provided OAuth clients, state/PKCE/single use, denial/error flows, token ownership, backup/restore, and reauthorization. Any direct callback port must satisfy `SYM-SEC-009`. Tokens MUST NOT be pasted into App options or browser-export files as a normal workaround. See the dated [Home Assistant music ecosystem review](../providers/home-assistant-ecosystem-review.md) for the implementation evidence and security cautions behind these alternatives. diff --git a/docs/decisions/0003-home-assistant-app-primary.md b/docs/decisions/0003-home-assistant-app-primary.md index ff0c7a6..ef11e17 100644 --- a/docs/decisions/0003-home-assistant-app-primary.md +++ b/docs/decisions/0003-home-assistant-app-primary.md @@ -2,6 +2,7 @@ - **Status:** accepted - **Date:** 2026-09-20 +- **Last reviewed:** 2026-09-22 ## Context @@ -13,6 +14,12 @@ Home Assistant Supervisor provides an install/update lifecycle, Ingress UI authe The primary supported distribution is a Supervisor-managed Home Assistant App installed from a Home Assistant App repository. It owns the long-running Symphonia service, durable jobs, provider connections, persistence, and management UI exposed through Ingress. +The implementation follows the `vypdev/homeassistant-gateway` deployment +pattern: the App is the product boundary and Ingress is the administrative +entry point. Provider authorization is owned by the App's provider adapters; +the MVP does not require a companion integration to broker OAuth. A companion +integration remains a later, optional native-surface extension. + The domain and application core remain independent of Home Assistant. A standalone composition profile will use the same core so the product can operate without Home Assistant and tests do not require it. Timing of the standalone release remains open. A companion Home Assistant custom integration may later expose native entities, actions, and events over a stable authenticated service contract. It will not duplicate domain policy or access the database/secrets directly. @@ -51,4 +58,3 @@ Trade-offs: - standalone mode needs its own authentication/network boundary; - App and optional integration artifacts need version compatibility; - provider credentials in Supervisor backups require a deliberate encryption/key/restore model. - diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 706f895..e2e27da 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -1,7 +1,7 @@ # Implementation baseline **Status:** owner-approved foundation slice -**Last reviewed:** 2026-09-20 +**Last reviewed:** 2026-09-22 The first implementation increment is intentionally narrower than any provider or Home Assistant capability. It proves the provider-independent core and the durable-operation persistence contract without selecting an external web framework, provider SDK, OAuth strategy, or frontend stack. @@ -29,18 +29,26 @@ The first implementation increment is intentionally narrower than any provider o - Copy executor can run as a claimed operation handler, preserving the same restart/checkpoint semantics under the runner. - Playlist import executor persists intent before reads and reports succeeded, partial, waiting-user, retry, and rate-limit outcomes durably. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. +- Direct-callback authorization can resolve a durable attempt by the returned state digest after restart without persisting raw state; ambiguous digests fail closed. +- Authorization consumption is covered across independent SQLite connections so callback replay races produce exactly one consumed attempt. +- Authorization state is bounded before hashing, required identifiers are validated before SQLite writes, and durable failure codes use a bounded safe alphabet. - Application authorization boundary that generates one-use state without persisting the raw value. - Authorization callback binding rejects unsafe schemes, fragments, credentials, whitespace, and non-loopback HTTP hosts. -- Normalized provider errors and provider codes redact common bearer/token/secret/password/cookie forms at the provider boundary. +- Normalized provider errors and provider codes redact common bearer/token/secret/password/cookie forms at the provider boundary, including quoted JSON-like and query-like assignments. - Operation payloads and durable checkpoints recursively reject credential-shaped keys and cyclic structures before SQLite writes. - Durable operation intents, checkpoints, retries, and rate-limit waits require JSON-object payloads with string keys before SQLite writes. +- Durable operation JSON rejects non-finite numbers on write and read, and operation/worker identifiers are validated before lease transitions. +- Shared SQLite JSON helpers apply deterministic serialization and reject non-standard numbers in capability and plan payloads as well as operation state. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. - Copy plans bind target connection and effective write-capability evidence into their digest. - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. +- Concrete provider manifests expose upstream dependencies and a dated research review marker. - Offline-testable official Spotify playlist reader/writer with bounded pagination, explicit write-capability gating, and normalized error categories. +- Spotify write adapters reject blank playlist/entry identifiers and unknown visibility values before issuing provider requests. - Explicitly scoped official YouTube Data API video-playlist reader; it is not represented as YouTube Music. - Experimental official Apple Music library-playlist reader with separate developer/user token inputs. - Provider readers enforce bounded page sizes, repeated-cursor detection, and configurable maximum page counts. +- Provider readers fail closed on malformed/non-absolute continuation links, backward offsets, and non-textual access/page tokens. - Experimental Home Assistant App metadata scaffold with Ingress-only management and `/data` persistence. - Explicit runtime resource composition and reverse-order shutdown for all durable repositories. - Every SQLite repository enables foreign-key enforcement at connection startup; backup preflight remains a separate integrity check. @@ -50,18 +58,27 @@ The first implementation increment is intentionally narrower than any provider o - Readiness can validate every composed durable store instead of only the operation queue. - Runtime resources provide a consistent SQLite online-backup helper while the service remains open. - Runtime resources can preflight backup integrity and required durable tables read-only before restore design is selected. +- Backup publication validates the temporary SQLite copy before atomically replacing the destination. +- Runtime backups reject missing parent directories and directory destinations before creating a temporary artifact. - Backup preflight rejects operation databases whose schema version is newer than the running foundation. - Runtime configuration validates host, port, database path, and Ingress base path before startup. - Runtime configuration rejects control characters, non-integral ports, and query/fragment-bearing Ingress paths before startup. - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. +- Runtime readiness validates SQLite integrity, foreign-key references, durable operation state/event values, authorization attempts, provider capability JSON, plan JSON, and resolution payloads; corruption fails closed. - Provider connection diagnostics expose provider/state and expiry counts, never account or credential data. - Import diagnostics expose bounded snapshot freshness and availability counts without playlist content. - Resolution diagnostics expose only manual-decision action counts, never track, actor, or reason data. - Ingress-relative health/version routing with normalized, traversal-safe base paths. +- The composed runtime HTTP server has an offline route contract and a loopback smoke path for real health/readiness lifecycle checks. - The current JSON health/readiness/version surface disables caching, MIME sniffing, and referrer propagation. +- The runtime JSON surface rejects non-standard numeric values before writing a response. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. +- Normalized provider identity, manifest, capability, entry, and page values reject non-textual or malformed boundary data before it reaches planning. +- Normalized provider pages reject wrong enum/runtime types and duplicate positions before collection; dated manifest evidence must use an ISO date. - SQLite storage for immutable copy plans, including durable digest-bound acceptance. +- Copy-plan reads and acceptance recompute the digest over execution-relevant content, rejecting tampering even when the stored digest field is unchanged. - The operation store applies its schema DDL, legacy column migration, and `user_version` marker in one transaction. +- SQLite operation tests exercise real multi-connection races: one operation cannot be claimed twice and concurrent scheduler workers claim distinct queue items. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. - Lossless Unicode-safe identity normalization with explicit version-token and ISRC derived fields; no automatic matching thresholds are assumed. - An application workflow that persists plans, requires digest acceptance, and enqueues only accepted plans as durable operations. diff --git a/docs/development/oauth-callback-spike.md b/docs/development/oauth-callback-spike.md new file mode 100644 index 0000000..5108a1e --- /dev/null +++ b/docs/development/oauth-callback-spike.md @@ -0,0 +1,33 @@ +# Direct App OAuth callback spike + +**Status:** research prototype; not production authorization code +**Reviewed:** 2026-09-22 + +This spike follows the App-plus-Ingress direction used by Symphonia's primary +deployment model. Provider adapters remain inside the App. A future companion +integration is not required for the MVP authorization path. + +## What it proves locally + +- callback state resolves from a durable SHA-256 digest after repository restart; +- the callback is single-use and provider-bound before consumption; +- denial becomes a terminal attempt state without persisting provider error text; +- unknown paths, duplicate state parameters, replay, and wrong-provider callbacks fail safely; +- the HTTP prototype exposes only the configured callback route and refuses broad + host binding by default; and +- the response never reflects authorization codes or provider error descriptions. + +The transient authorization code is returned only to the in-process exchange +owner and is marked non-representational in the prototype result. A future +provider adapter must exchange it immediately, keep the resulting grant behind +the secret boundary, and never place either value in logs, URLs, diagnostics, +ordinary operation payloads, or backups. + +## Still required before production + +`RG-002` remains open. A Home Assistant test matrix must verify externally +reachable HTTPS redirects, arbitrary Ingress paths, remote access, browser +session continuity, user-provided client registrations, PKCE where applicable, +refresh/revocation, restart at every attempt phase, and callback-listener +isolation in the actual App network topology. The loopback-only server in this +spike is deliberately insufficient evidence for those claims. diff --git a/docs/development/storage-recovery-spike.md b/docs/development/storage-recovery-spike.md new file mode 100644 index 0000000..c738648 --- /dev/null +++ b/docs/development/storage-recovery-spike.md @@ -0,0 +1,33 @@ +# SQLite recovery and backup spike + +**Status:** research evidence only +**Last reviewed:** 2026-09-22 + +The repository now contains a small offline spike at +[`tools/storage_recovery_spike.py`](../../tools/storage_recovery_spike.py). It +creates a disposable persistent runtime, queues a bounded number of operations, +claims one lease, closes the runtime, reopens it as a different worker, waits +until the lease has expired, reclaims and completes the operation, then creates +and validates an online SQLite backup. + +Run it from the repository root with: + +```text +PYTHONPATH=src:. python3 tools/storage_recovery_spike.py --operations 1000 +``` + +The JSON result reports only fixture counts, state transition evidence, file +sizes, backup validity, and elapsed time. It contains no database path, +operation payload, account identifier, or credential. The elapsed time is an +observation for the machine and fixture size used; it is not a product SLO or +a restore benchmark. + +This spike supports the foundation claims that operation leases survive a +process restart, concurrent SQLite workers respect the lease boundary, and +the composed runtime can produce a structurally validated backup. The unit +suite also reopens a backup containing operation, provider-connection, and +authorization-attempt rows through the normal runtime composition root and +readiness now fails closed when durable state contains invalid JSON or enum +values. It does not yet select backup retention, encryption, +Supervisor backup declarations, restore UX, or a production scheduler. Those +remain part of the App runtime and durable execution SDD gates. diff --git a/docs/open-questions.md b/docs/open-questions.md index 2bab073..d5fd272 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -1,7 +1,7 @@ # Open questions, risks, and next design work **Status:** open; nothing here is an accepted decision -**Last reviewed:** 2026-09-20 +**Last reviewed:** 2026-09-22 ## Decisions requiring owner input @@ -39,13 +39,18 @@ Options include: **Proposed default:** strict by default with an explicit, itemized best-effort override. Rollback/cleanup after partial provider writes still needs design. -### OQ-004 — Which provider OAuth boundary should self-hosters use? +### OQ-004 — Which provider OAuth credentials and callback profile should self-hosters use? Possibilities include project-owned shared client registrations, bring-your-own client credentials per install, or both. Spotify's five-user Development Mode cap strongly favors bring-your-own credentials for a distributed self-hosted project, but callback registration and support become harder. -The callback boundary also has two credible shapes: a direct App-owned OAuth flow, or a minimal companion integration that uses Home Assistant Application Credentials/config flows and brokers a one-use connection grant to the App. The latter reuses Home Assistant UX but adds token ownership, backup, revocation, and version-skew questions. Apple Music would add a third, MusicKit-specific token model if promoted into scope. - -This decision requires the Home Assistant OAuth callback spike. It also affects documentation, verification, secret storage, direct port exposure, and whether remote Home Assistant access is needed. +Owner direction (2026-09-22): follow the `vypdev/homeassistant-gateway` +deployment pattern. The Supervisor App owns the provider adapters and the +Ingress UI/API; a companion integration is not required to broker OAuth for +the MVP. The working authorization shape is therefore a direct App-owned, +callback-only flow. `RG-002` still has to prove callback reachability, exact +redirect registration, state/PKCE/single-use behavior, secret ownership, and +local/remote Home Assistant operation before the implementation contract is +ready. ### OQ-005 — What authentication/exposure does standalone mode use, and when does it ship? @@ -73,11 +78,17 @@ The repository currently has no license. The license should be selected before a Run the dated Spotify and YouTube test-account matrix in [provider research](providers/provider-research.md). Record scopes, account tier, app mode, market, exact endpoints, quota cost, payload gaps, playlist visibility, duplicate/order behavior, and cleanup results. Do not use personal libraries as fixtures. +The official documentation was revalidated on 2026-09-22. That refresh confirms +the documented Spotify playlist write limits and authorization lifetime, and +that the public YouTube Data API remains a video-playlist surface rather than +proof of full YouTube Music library parity. A dedicated test-account run is +still required before this gate can close. + If Apple Music is considered as a future provider or fallback for an official music-library surface, run its separate feasibility gates without silently changing the MVP. Community providers may inform test cases but cannot substitute for official-contract evidence. ### RG-002 — OAuth through a Home Assistant App -Prototype authorization start/callback/error for Spotify and Google using both a direct App-owned flow and, where viable, a minimal companion-integration broker built on Home Assistant Application Credentials. Test: +Prototype authorization start/callback/error for Spotify and Google using a direct App-owned, callback-only flow. Test: - local and externally reachable Home Assistant URLs; - HTTPS and exact registered redirects; @@ -89,6 +100,9 @@ Prototype authorization start/callback/error for Spotify and Google using both a - no secrets in App options, browser-export files, URL query logs, referrers, or diagnostics. The result becomes an authentication/secret-storage RFC, not production code. +The current local evidence is recorded in the [direct App OAuth callback +spike](development/oauth-callback-spike.md); it proves route isolation and +durable single-use state but does not close the Home Assistant topology gate. ### RG-003 — Matching evidence corpus @@ -98,6 +112,11 @@ Build and review a licensed/synthetic labeled corpus containing exact duplicates Compare SQLite and PostgreSQL for transaction boundaries, leases, crash recovery, unknown writes, snapshot/history queries, online/cold backup under Supervisor, encryption-key restore, representative library size, and a per-migration applied ledger across stable/beta upgrade paths. Prove that a failed migration never replaces irrecoverable manual decisions/audit state with a fresh rescan. No framework selection should precede these results. +The current local evidence includes a [reproducible SQLite recovery and backup +spike](development/storage-recovery-spike.md) covering an expired lease across +runtime restart and read-only backup validation. It is an initial evidence +point, not a storage choice or a production SLO. + ### RG-005 — Home Assistant native surface RFC Define what belongs in the App UI versus a companion custom integration. Decide transport/auth/version discovery and a minimal entity/action/event surface. Ensure Home Assistant actions create ordinary audited Symphonia operations and that the integration can be unavailable independently. @@ -171,7 +190,7 @@ These need evidence and small RFCs; popularity is not evidence. ## Recommended next five specification/design tasks 1. **Provider feasibility report (`RG-001`).** Prove or narrow the Spotify ↔ YouTube promise using official APIs and dedicated accounts; feed the evidence into the provider, [import](../specs/library-import-and-provider-projections.md), and [copy](../specs/one-time-playlist-copy.md) SDDs. Report unofficial YT Music evidence separately and keep Apple as an explicit future/contingency spike. -2. **Close the [authorization SDD](../specs/provider-connections-and-authorization.md) blockers (`RG-002` + `OQ-004`).** Compare direct App OAuth with a minimal companion-integration broker, then settle callbacks, bring-your-own credentials, encryption, revocation, and backups. +2. **Close the [authorization SDD](../specs/provider-connections-and-authorization.md) blockers (`RG-002` + `OQ-004`).** Validate the direct App callback flow, then settle bring-your-own credentials, encryption, revocation, and backups. 3. **Close the [copy SDD](../specs/one-time-playlist-copy.md) policy blocker (`OQ-003`).** Decide target creation, strict/best-effort behavior, batching, partial failure, reconciliation, cancellation, and exact acceptance examples. 4. **Close the [identity SDD](../specs/recording-identity-resolution.md) evidence blockers (`RG-003`).** Build the corpus and settle normalization, candidate sources, evidence, versioned rules, manual decisions, and measurable safety targets. 5. **Close the [runtime](../specs/home-assistant-app-runtime-and-ingress.md) and [durable-operation](../specs/durable-operations-and-recovery.md) SDD blockers (`RG-004`).** Choose process topology and storage only after crash, lease, migration, backup, and representative-scale evidence. diff --git a/docs/providers/home-assistant-ecosystem-review.md b/docs/providers/home-assistant-ecosystem-review.md index adfdb5d..9d02ba2 100644 --- a/docs/providers/home-assistant-ecosystem-review.md +++ b/docs/providers/home-assistant-ecosystem-review.md @@ -206,7 +206,7 @@ This review does not accept a dependency or new provider into the MVP. It narrow 1. Extend the provider manifest RFC with independent access-basis, maturity, and product-support classifications so, for example, unofficial-but-mature and official-but-experimental are not conflated. 2. Make capabilities an intersection of adapter, connection, object, and live health—not provider-wide booleans. 3. Include provider instance/connection namespace and object type in external-identity analysis. -4. Compare direct App OAuth with a minimal companion-integration authorization broker in the Home Assistant OAuth spike. +4. Validate the direct App-owned, callback-only OAuth flow in the Home Assistant OAuth spike; keep a companion broker as a separately scoped future option. 5. Add an Apple Music test-account spike as a future-provider candidate, focusing on Music User Token acquisition, catalog/library IDs, `canEdit`, playlist append, and absence of remove/reorder. 6. Require completeness markers for every import/list operation and preserve unavailable entries. 7. Threat-model every directly exposed App listener and prevent provider-controlled values from acquiring filesystem or executable semantics. diff --git a/docs/providers/provider-research.md b/docs/providers/provider-research.md index 8842794..0b8c69d 100644 --- a/docs/providers/provider-research.md +++ b/docs/providers/provider-research.md @@ -1,7 +1,7 @@ # Provider and platform research **Status:** research snapshot, not an architectural decision -**Reviewed:** 2026-09-20 +**Reviewed:** 2026-09-22 **Source policy:** official documentation only; revalidate before implementation and every release ## How to read this document @@ -10,6 +10,23 @@ This snapshot separates what Symphonia needs from what an official API documents Existing Home Assistant and community implementations are reviewed separately in [Home Assistant music ecosystem review](home-assistant-ecosystem-review.md). They provide valuable implementation evidence but do not replace an official provider contract. +## 2026-09-22 verification update + +The official references were rechecked before the next design pass. This is a +documentation refresh, not a completed live-provider feasibility spike. + +| Provider/platform | Reconfirmed official evidence | Consequence for Symphonia | +| --- | --- | --- | +| Spotify | Playlist creation remains a separate empty-playlist operation; playlist item insertion accepts at most 100 items per request; the create operation defaults to public unless the request explicitly selects private visibility and has the required scope. | Keep target creation, visibility, batching, and checkpointing as separate capability decisions. The adapter must default to the product's safe private policy rather than inheriting the API default. | +| Spotify authorization | Access tokens remain short-lived and the current refresh-token guidance documents a six-month lifetime for Developer Dashboard apps. | Reauthorization is a normal durable state and must be visible before an import or write is dispatched. | +| YouTube Data API | The official surface models playlists as collections of videos; playlist-item insertion is an OAuth-protected write and the documented default quota is 10,000 units/day for most endpoints, with writes commonly costing 50 units. | The official adapter can be called YouTube Data, not YouTube Music. Search and write budgets must be planned explicitly; a playlist/video ID is not a recording identity. | +| Apple Music | The official API exposes a personal iCloud Music Library, playlist reads, playlist creation, and adding tracks, using a developer token plus Music User Token. | Apple remains a future/contingency provider. Its token and origin lifecycle still require a dedicated feasibility spike before MVP promotion. | +| Home Assistant Apps | Ingress is the authenticated UI boundary, requires the App to allow only the Supervisor ingress source, and exposes the ingress path through a request header. | The App can keep management routes Ingress-only, but this does not solve provider OAuth callback ownership or token storage. | + +These checks reinforce the existing conclusion: the repository may prepare +official Spotify and ordinary YouTube adapter contracts, but it must not claim +full YouTube Music library parity or silently use an unofficial endpoint. + Legend: - **Documented**: an official current page describes the needed primitive. diff --git a/specs/CATALOG.md b/specs/CATALOG.md index b3e7bd7..47c882a 100644 --- a/specs/CATALOG.md +++ b/specs/CATALOG.md @@ -8,7 +8,7 @@ This is the human-readable view of [`catalog.json`](./catalog.json). Until gener | --- | --- | --- | --- | | `home-assistant-app-runtime` | Draft | [Home Assistant App runtime and Ingress](home-assistant-app-runtime-and-ingress.md) | Blocked by storage/recovery, supported platform matrix, and secret-key design | | `home-assistant-native-ui` | Ready for review | [Home Assistant-native UI foundation](home-assistant-native-ui.md) | Product direction is accepted; blocked from implementation readiness by the supported matrix, frontend/build choice, public host-context validation, and visual-reference procedure | -| `provider-connections-and-authorization` | Draft | [Provider connections and authorization](provider-connections-and-authorization.md) | Blocked by OAuth boundary and provider feasibility spikes | +| `provider-connections-and-authorization` | Draft | [Provider connections and authorization](provider-connections-and-authorization.md) | Blocked by direct callback reachability, secret/backup design, and provider feasibility spikes | | `library-import-and-provider-projections` | Draft | [Library import and provider projections](library-import-and-provider-projections.md) | Blocked by provider completeness, retention, and representative scale | | `recording-identity-resolution` | Draft | [Recording identity resolution](recording-identity-resolution.md) | Blocked by the labeled corpus and accepted automatic-link policy | | `one-time-playlist-copy` | Draft | [One-time playlist copy](one-time-playlist-copy.md) | Blocked by unresolved-entry policy and proven target write semantics | diff --git a/specs/README.md b/specs/README.md index f41fc2a..cf1935c 100644 --- a/specs/README.md +++ b/specs/README.md @@ -121,8 +121,14 @@ Run after every SDD or catalog change: ```text git diff --check +PYTHONPATH=. python3 tools/validate_specs.py ``` +The validator checks that `catalog.json` parses, referenced files exist, each +SDD has one known catalog capability ID, catalog statuses and paths agree with +`CATALOG.md`, requirement IDs are present in the horizontal specifications, +and local Markdown links resolve. + Also verify: 1. `catalog.json` parses as JSON and every referenced local path exists. diff --git a/specs/catalog.json b/specs/catalog.json index 6f4df94..95c0620 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -55,6 +55,11 @@ "src/symphonia/runtime/config.py", "src/symphonia/runtime/http.py", "src/symphonia/runtime/resources.py", + "src/symphonia/infrastructure/sqlite_authorization.py", + "src/symphonia/infrastructure/sqlite_connections.py", + "src/symphonia/infrastructure/sqlite_library.py", + "src/symphonia/infrastructure/sqlite_plans.py", + "src/symphonia/infrastructure/sqlite_resolutions.py", "Dockerfile", "addon/config.yaml" ], @@ -62,6 +67,10 @@ "tests/test_runtime_config.py", "tests/test_runtime_http.py", "tests/test_runtime_resources.py", + "tests/test_sqlite_authorization.py", + "tests/test_sqlite_connections.py", + "tests/test_sqlite_plans.py", + "tests/test_identity_resolution.py", "tests/test_homeassistant_app_metadata.py", "tests/test_architecture_boundaries.py" ], @@ -174,7 +183,7 @@ "documentation": [] }, "blockers": [ - "OQ-004 direct App OAuth versus companion-integration authorization broker", + "RG-002 direct App callback reachability and OAuth boundary validation", "RG-002 Home Assistant OAuth callback spike", "RG-001 provider-specific scopes and account constraints", "Decision on whether an unofficial YouTube Music adapter exists in the MVP" @@ -366,6 +375,12 @@ "code": [ "src/symphonia/infrastructure/sqlite_common.py", "src/symphonia/infrastructure/sqlite_operations.py", + "src/symphonia/infrastructure/sqlite_authorization.py", + "src/symphonia/infrastructure/sqlite_connections.py", + "src/symphonia/infrastructure/sqlite_library.py", + "src/symphonia/infrastructure/sqlite_plans.py", + "src/symphonia/infrastructure/sqlite_resolutions.py", + "src/symphonia/domain/models.py", "src/symphonia/application/operation_runner.py", "src/symphonia/application/operation_worker.py", "src/symphonia/application/copy_execution.py", @@ -373,6 +388,11 @@ ], "tests": [ "tests/test_sqlite_operations.py", + "tests/test_sqlite_authorization.py", + "tests/test_sqlite_connections.py", + "tests/test_sqlite_plans.py", + "tests/test_identity_resolution.py", + "tests/test_copy_planning.py", "tests/test_operation_runner.py", "tests/test_operation_worker.py", "tests/test_copy_execution.py", diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index d160dbb..19c8772 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, JSON-object/string-key payload validation, and deterministic worker tests. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, JSON-object/string-key payload validation, and deterministic worker tests. Its readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, and invalid credential-bearing payload keys; real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index a9e7485..d58f305 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing with bounded response headers, composed SQLite stores, transactionally consistent backup/preflight helpers with future-schema rejection, idempotent resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing with bounded response headers, composed SQLite stores with semantic readiness checks, transactionally consistent backup/preflight helpers with future-schema rejection and temporary-copy validation before publication, idempotent resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/specs/provider-connections-and-authorization.md b/specs/provider-connections-and-authorization.md index 11ca4f3..7486c5d 100644 --- a/specs/provider-connections-and-authorization.md +++ b/specs/provider-connections-and-authorization.md @@ -8,13 +8,18 @@ - Related requirements: `SYM-ACC-002`–`SYM-ACC-004`, `SYM-ACC-006`, `SYM-PROV-002`–`SYM-PROV-003`, `SYM-PROV-008`–`SYM-PROV-009`, `SYM-PROV-015`–`SYM-PROV-020`, `SYM-SEC-001`–`SYM-SEC-010` - Related decisions/research: [provider specification](../docs/providers/provider-specification.md), [official API research](../docs/providers/provider-research.md), [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md), [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [UI foundation](home-assistant-native-ui.md), `OQ-001`, `OQ-004`, `RG-001`, `RG-002` - Required review gates: product UX, architecture, provider feasibility, testing, documentation, security/privacy -- Open decisions blocking readiness: direct App OAuth versus companion-integration authorization broker; per-provider registration/scopes/token lifecycle; unofficial YouTube Music MVP decision; secret key source and backup contract +- Open decisions blocking readiness: direct App callback reachability and provider-specific registration/scopes/token lifecycle; unofficial YouTube Music MVP decision; secret key source and backup contract ## 1. Executive summary Before any external account is connected, Symphonia explains whether the adapter uses an official or reverse-engineered contract, what credentials and dependencies it needs, what access it requests, and how reauthorization works. A successful connection identifies one immutable provider account, stores only encrypted grant material behind a secret reference, probes effective capabilities, and schedules import separately. -The authorization boundary is not yet selected. The SDD keeps two candidates open: a direct App-owned flow or a minimal Home Assistant companion-integration broker. Neither may expose provider tokens to App options, URLs, logs, diagnostics, or ordinary UI state. +Following the accepted App-plus-Ingress deployment direction, the MVP uses a +direct App-owned, callback-only authorization flow. A future companion +integration may provide a native authorization broker, but it is not required +for the MVP and cannot be assumed by provider workflows. Neither boundary may +expose provider tokens to App options, URLs, logs, diagnostics, or ordinary UI +state. ```text Choose adapter -> review access basis/scopes/limitations -> configure client credentials @@ -30,14 +35,21 @@ Provider authentication differs in client registration, redirect rules, scopes, ### 2.2 Current behavior -No connection or credential implementation exists. Current facts are requirements and research only. Spotify is the strongest official MVP candidate; full YouTube Music access is not established through an official API; Apple Music is future research. +The foundation now persists provider-neutral authorization attempts with hashed +and bounded state, exact redirect binding, expiry, single-use consumption, +bounded durable outcome codes, and callback state lookup that survives restart. +Invalid attempt metadata is rejected before SQLite writes. Provider token +exchange, secret storage, account verification, and the provider-specific +callback adapters do not yet exist. Spotify is the strongest official MVP +candidate; full YouTube Music access is not established through an official +API; Apple Music is future research. ### 2.3 Evidence and unknowns - Official provider/API facts: [provider research](../docs/providers/provider-research.md). - Home Assistant OAuth/Application Credentials and existing music projects: [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md). - Provider-independent contract: [provider specification](../docs/providers/provider-specification.md). -- Unknowns: OAuth ownership, callback reachability, secret encryption key, Google/YouTube product scope, exact scopes, token expiry/revocation behavior under test accounts. +- Unknowns: callback reachability, secret encryption key, Google/YouTube product scope, exact scopes, token expiry/revocation behavior under test accounts, and provider-specific client-registration policy. ## 3. Actors, surfaces, and terminology @@ -149,7 +161,7 @@ Exact provider field names remain adapter-owned. Every field declares type, secr | --- | --- | --- | --- | --- | | OAuth application ownership | enum | bring-your-own for distributed self-hosting | accepted shared registration or per-install credentials | adapter/install; snapshotted for attempt | | Requested capability profile | enum | minimum MVP import/copy profile | adapter-declared bounded profiles | connection; exact scopes snapshotted | -| Callback mode | enum | unresolved pending `RG-002` | direct App or companion broker after acceptance | install; restart/version contract may apply | +| Callback mode | enum | direct App callback-only, pending `RG-002` | one accepted direct callback profile | install; restart required if the callback profile changes | | Unofficial access acknowledgement | boolean | `false` | explicit `true` only for accepted unofficial adapter | connection plus policy version/time | Client secrets, refresh tokens, Music User Tokens, browser cookies, and authorization codes are never App options. Redirect targets, scope sets, provider hosts, and token endpoints come from versioned adapter definitions, not arbitrary user URLs unless an accepted provider explicitly requires a bounded configurable endpoint. @@ -164,7 +176,7 @@ Client secrets, refresh tokens, Music User Tokens, browser cookies, and authoriz | Application | begin/complete/reauthorize/disconnect/probe use cases | Provider SDK DTOs, token persistence implementation | | Provider adapter | Provider authorization metadata, account lookup, refresh/revoke, capability probe | Cross-provider workflow policy | | Secret adapter | encrypt/store/load/delete/rotate by opaque reference | UI or provider semantics | -| HA broker adapter, optional | Config flow/Application Credentials and one-use local handoff | Provider imports/writes, database access | +| Future HA broker adapter | Config flow/Application Credentials and one-use local handoff | MVP provider imports/writes, database access | | Presentation | Disclosure/forms/status/error mapping | Token values or authorization decisions | ### 8.2 Contracts, durable state, and trust boundaries @@ -181,9 +193,11 @@ Client secrets, refresh tokens, Music User Tokens, browser cookies, and authoriz | Alternative | Benefits | Costs/risks | Readiness evidence | | --- | --- | --- | --- | | Direct App-owned OAuth | Self-contained provider adapter and standalone parity | Public callback/exposure, exact redirect, token storage all owned by App | `RG-002` callback-only listener and remote/local tests | -| Companion-integration broker | Reuses HA Application Credentials/config-flow callback UX | Second artifact, token ownership/handoff, backup/version skew | Signed one-use handoff threat model and HA integration spike | +| Companion-integration broker (future) | Reuses HA Application Credentials/config-flow callback UX | Second artifact, token ownership/handoff, backup/version skew | Separate native-surface RFC; not an MVP dependency | -No preference becomes accepted until `RG-002` records evidence and an ADR chooses the boundary. +The direct App callback is the MVP working direction. `RG-002` must still +prove its local/remote reachability, listener isolation, and secret/recovery +properties before the SDD can become ready for implementation. ### 8.4 Executable architecture constraints @@ -279,7 +293,7 @@ Minimum **82 distinct cases**: | --- | ---: | --- | | Manifest/configuration/capability policy | 16 | classifications, scopes, account/object/live intersections, invalid combinations | | Attempt/state/idempotency/races | 20 | expiry, denial, replay, duplicate callback, refresh/disconnect races, restart | -| Provider/secret/broker adapters | 18 | exchange/refresh/revoke/probe, error mapping, rotation, handoff, timeouts | +| Provider/secret/callback adapters | 18 | exchange/refresh/revoke/probe, error mapping, rotation, listener isolation, timeouts | | HTTP/UI/accessibility/sanitization | 12 | disclosures and all states, focus, hostile names/errors/URLs, locale fallback | | Integration/security/migration | 16 | direct/broker paths, listener isolation, canary leakage, backup, reauth migration | | **Total** | **82** | No double counting | @@ -309,7 +323,7 @@ The twelve feature-specific UI cases supplement the UI-foundation budget and inh 8. Given an unofficial adapter, authorization cannot begin without explicit risk acknowledgement and the UI never labels it official/supported by Home Assistant. 9. Given hostile callback/provider/error content, no open redirect, path/egress injection, Markdown/HTML injection, or secret output occurs. 10. Given App restart at every attempt phase, no callback is consumed twice and no orphan grant becomes an active connection. -11. Given the broker alternative, a forged/replayed/local unauthenticated handoff is rejected and the integration cannot mutate the App database directly. +11. Given a direct callback listener, management routes and Ingress identity headers are unavailable on that listener, and a forged/replayed callback cannot create a connection. 12. Given official, unofficial, connected, degraded, action-required, authorizing, and disconnected fixtures, the shared Home Assistant-native component families preserve information hierarchy, keyboard/focus behavior, narrow layout, theme parity, and textual risk without exposing secret material. ## 17. Requirements traceability @@ -320,7 +334,7 @@ The twelve feature-specific UI cases supplement the UI-foundation budget and inh | `SYM-PROV-002`, `SYM-PROV-003`, `SYM-PROV-019` | capability policy/adapter | connection/object/live matrix | provider capability reference | | `SYM-PROV-015`–`SYM-PROV-020` | manifest and presentation | schema/opt-in/label tests | provider support policy | | `SYM-SEC-001`–`SYM-SEC-007` | auth/secret ports | replay/race/canary/rotation tests | security and provider setup | -| `SYM-SEC-008`–`SYM-SEC-010` | listener/broker/provider adapters | route/redirect/path/egress abuse tests | callback operations guide | +| `SYM-SEC-008`–`SYM-SEC-010` | listener/callback/provider adapters | route/redirect/path/egress abuse tests | callback operations guide | | `SYM-TEST-005`, `SYM-TEST-011`, `SYM-TEST-012` | verification tooling | release gates | contributor testing guide | ## 18. Implementation sequence @@ -328,8 +342,8 @@ The twelve feature-specific UI cases supplement the UI-foundation budget and inh 1. Complete `RG-001`/`RG-002`, secret/backup threat model, and authorization-boundary ADR. 2. Define manifest, attempt, connection, capability, and normalized-error contracts plus architecture tests. 3. Implement pure state/capability/disclosure policies and deterministic tests. -4. Implement application use cases and fake provider/secret/broker ports. -5. Implement one official provider adapter and selected callback boundary behind contract tests. +4. Implement application use cases and fake provider/secret/callback ports. +5. Implement one official provider adapter and the direct callback boundary behind contract tests. 6. Add UI, reauthorization/disconnect, diagnostics, documentation, and opt-in live smoke evidence. ## 19. Definition of Done @@ -337,7 +351,7 @@ The twelve feature-specific UI cases supplement the UI-foundation budget and inh - [ ] Authorization-boundary, provider-scope, unofficial-access, and secret/backup blockers are resolved. - [ ] Every requirement and state maps to acceptance and deterministic verification. - [ ] At least 82 distinct cases and secret-canary gates pass. -- [ ] Callback replay, redirect, broker/listener isolation, refresh race, rotation, and disconnect are proven. +- [ ] Callback replay, redirect, listener isolation, refresh race, rotation, and disconnect are proven. - [ ] Provider DTOs and secret implementations do not cross inward architecture boundaries. - [ ] Disclosures and action-required/degraded/disconnected states pass accessibility and sanitization review. - [ ] Per-provider setup, recovery, revocation, and risk documentation is complete. @@ -350,5 +364,5 @@ The twelve feature-specific UI cases supplement the UI-foundation budget and inh - Related SDDs: [App runtime](home-assistant-app-runtime-and-ingress.md), [imports](library-import-and-provider-projections.md), [durable operations](durable-operations-and-recovery.md). - Related UI contract: [Home Assistant-native UI foundation](home-assistant-native-ui.md) and [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md). - Accepted: capability and risk disclosure before authorization; opaque secret references; distinct unofficial adapters. -- Open alternatives: direct App OAuth versus HA companion broker. +- MVP direction: direct App-owned callback-only OAuth; a companion broker is deferred. - Rejected: tokens in App options; pasted callback URLs as tokens; generic retry of invalid grants; treating private APIs as official. diff --git a/src/symphonia/application/authorization.py b/src/symphonia/application/authorization.py index 8f89088..c4d1516 100644 --- a/src/symphonia/application/authorization.py +++ b/src/symphonia/application/authorization.py @@ -51,5 +51,57 @@ def begin( def consume(self, attempt_id: str, *, raw_state: str, now: datetime) -> AuthorizationAttempt: return self.attempts.consume(attempt_id, raw_state=raw_state, now=now) + def consume_callback(self, *, raw_state: str, now: datetime) -> AuthorizationAttempt: + """Consume a provider callback using only its returned state value.""" + + return self._consume_callback(raw_state=raw_state, now=now) + + def resolve_callback(self, *, raw_state: str) -> AuthorizationAttempt: + """Resolve callback metadata before validating provider-specific routing.""" + + return self.attempts.get_by_state(raw_state) + + def _consume_callback( + self, + *, + raw_state: str, + now: datetime, + expected_provider: str | None = None, + ) -> AuthorizationAttempt: + attempt = self.resolve_callback(raw_state=raw_state) + if expected_provider is not None and attempt.provider != expected_provider: + raise ValueError("authorization callback provider did not match the attempt") + return self.attempts.consume(attempt.attempt_id, raw_state=raw_state, now=now) + + def consume_callback_for_provider( + self, + *, + raw_state: str, + provider: str, + now: datetime, + ) -> AuthorizationAttempt: + """Consume callback state only when it belongs to the routed provider.""" + + if not provider.strip(): + raise ValueError("provider must not be blank") + return self._consume_callback(raw_state=raw_state, now=now, expected_provider=provider) + + def deny_callback( + self, + *, + raw_state: str, + provider: str, + now: datetime, + failure_code: str = "consent_denied", + ) -> AuthorizationAttempt: + """Record provider denial only for the matching, durable attempt.""" + + if not provider.strip(): + raise ValueError("provider must not be blank") + attempt = self.resolve_callback(raw_state=raw_state) + if attempt.provider != provider: + raise ValueError("authorization callback provider did not match the attempt") + return self.attempts.deny(attempt.attempt_id, now=now, failure_code=failure_code) + def deny(self, attempt_id: str, *, now: datetime, failure_code: str = "consent_denied") -> AuthorizationAttempt: return self.attempts.deny(attempt_id, now=now, failure_code=failure_code) diff --git a/src/symphonia/application/operation_worker.py b/src/symphonia/application/operation_worker.py index 0b67066..c50e0fc 100644 --- a/src/symphonia/application/operation_worker.py +++ b/src/symphonia/application/operation_worker.py @@ -4,8 +4,8 @@ from collections.abc import Callable from datetime import datetime, timezone +import math import threading -import time from symphonia.infrastructure.sqlite_operations import OperationRecord @@ -33,11 +33,16 @@ def __init__( poll_interval_seconds: float = 1.0, lease_seconds: int = 30, ) -> None: - if not worker_id.strip(): + if not isinstance(worker_id, str) or not worker_id.strip(): raise ValueError("worker_id must not be empty") - if poll_interval_seconds <= 0: + if ( + isinstance(poll_interval_seconds, bool) + or not isinstance(poll_interval_seconds, (int, float)) + or not math.isfinite(poll_interval_seconds) + or poll_interval_seconds <= 0 + ): raise ValueError("poll_interval_seconds must be positive") - if lease_seconds <= 0: + if isinstance(lease_seconds, bool) or not isinstance(lease_seconds, int) or lease_seconds <= 0: raise ValueError("lease_seconds must be positive") self._runner = runner self._worker_id = worker_id diff --git a/src/symphonia/domain/models.py b/src/symphonia/domain/models.py index dd1e106..de770ee 100644 --- a/src/symphonia/domain/models.py +++ b/src/symphonia/domain/models.py @@ -166,11 +166,45 @@ def writable_entries(self) -> tuple[CopyPlanEntry, ...]: def omitted_entries(self) -> tuple[CopyPlanEntry, ...]: return tuple(entry for entry in self.entries if entry.disposition == "omit") + def recompute_digest(self) -> str: + """Recompute the immutable plan digest from every execution-relevant field.""" + + canonical = { + "source_snapshot_id": self.source_snapshot_id, + "source_provider": self.source_provider, + "source_playlist_id": self.source_playlist_id, + "source_namespace": self.source_namespace, + "target_provider": self.target_provider, + "target_playlist_name": self.target_playlist_name, + "target_visibility": self.target_visibility, + "policy": self.policy.value, + "target_connection_id": self.target_connection_id, + "target_capabilities": list(self.target_capabilities), + "target_capability_evidence_version": self.target_capability_evidence_version, + "entries": [ + { + "occurrence_id": entry.occurrence_id, + "position": entry.position, + "classification": entry.classification.value, + "disposition": entry.disposition, + "target_track_id": entry.target_track_id, + "reason": entry.reason, + "evidence": list(entry.evidence), + "source_provider_track_object_type": entry.source_provider_track_object_type, + } + for entry in self.entries + ], + } + serialized = json.dumps(canonical, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + return hashlib.sha256(serialized.encode("utf-8")).hexdigest() + def accept(self, expected_digest: str) -> "AcceptedCopyPlan": """Bind execution to this exact plan digest.""" if expected_digest != self.digest: raise PlanAcceptanceError("plan digest does not match the requested acceptance") + if self.recompute_digest() != self.digest: + raise PlanAcceptanceError("plan content does not match its digest") if self.blocked: raise PlanAcceptanceError("plan contains blocked entries") return AcceptedCopyPlan(plan=self, accepted_digest=self.digest) diff --git a/src/symphonia/infrastructure/sqlite_authorization.py b/src/symphonia/infrastructure/sqlite_authorization.py index a07cc29..735e468 100644 --- a/src/symphonia/infrastructure/sqlite_authorization.py +++ b/src/symphonia/infrastructure/sqlite_authorization.py @@ -7,7 +7,13 @@ import hmac import sqlite3 -from symphonia.providers.authorization import AuthorizationAttempt, AuthorizationState, validate_redirect_uri +from symphonia.providers.authorization import ( + MAX_AUTHORIZATION_STATE_LENGTH, + AuthorizationAttempt, + AuthorizationState, + validate_failure_code, + validate_redirect_uri, +) from .sqlite_common import connect @@ -23,7 +29,11 @@ def _parse_utc(value: str) -> datetime: def state_digest(state: str) -> str: - if not state or not state.strip(): + if ( + not isinstance(state, str) + or not state.strip() + or len(state) > MAX_AUTHORIZATION_STATE_LENGTH + ): raise ValueError("authorization state must not be empty") return hashlib.sha256(state.encode("utf-8")).hexdigest() @@ -47,13 +57,17 @@ def close(self) -> None: self._connection.close() def healthcheck(self) -> bool: - """Return whether the migrated authorization store can be read.""" + """Return whether schema and authorization attempts are readable.""" try: - row = self._connection.execute("SELECT 1 AS healthy").fetchone() - except sqlite3.Error: + integrity = self._connection.execute("PRAGMA integrity_check(1)").fetchone() + if integrity is None or integrity[0] != "ok": + return False + for row in self._connection.execute("SELECT * FROM authorization_attempts").fetchall(): + self._record(row) + except (sqlite3.Error, TypeError, ValueError): return False - return row is not None and row["healthy"] == 1 + return True def _migrate(self) -> None: self._connection.executescript( @@ -88,7 +102,15 @@ def create( ) -> AuthorizationAttempt: if ttl <= timedelta(0): raise ValueError("authorization attempt ttl must be positive") + for value, field_name in ( + (attempt_id, "attempt_id"), + (provider, "provider"), + (actor_id, "actor_id"), + ): + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"{field_name} must not be empty") validate_redirect_uri(redirect_uri) + state_digest(raw_state) created_at = _utc(now) expires_at = _utc(now + ttl) self._connection.execute( @@ -118,6 +140,19 @@ def get(self, attempt_id: str) -> AuthorizationAttempt: raise AuthorizationAttemptNotFound(attempt_id) return self._record(row) + def get_by_state(self, raw_state: str) -> AuthorizationAttempt: + """Resolve callback state without persisting or returning raw state.""" + + digest = state_digest(raw_state) + rows = self._connection.execute( + "SELECT * FROM authorization_attempts WHERE state_digest = ? LIMIT 2", (digest,) + ).fetchall() + if not rows: + raise AuthorizationAttemptNotFound("authorization state") + if len(rows) > 1: + raise AuthorizationAttemptError("authorization state is ambiguous") + return self._record(rows[0]) + def consume(self, attempt_id: str, *, raw_state: str, now: datetime) -> AuthorizationAttempt: """Consume a matching, unexpired state exactly once.""" @@ -162,6 +197,7 @@ def consume(self, attempt_id: str, *, raw_state: str, now: datetime) -> Authoriz return self.get(attempt_id) def deny(self, attempt_id: str, *, now: datetime, failure_code: str = "consent_denied") -> AuthorizationAttempt: + validate_failure_code(failure_code) return self._complete(attempt_id, AuthorizationState.DENIED, now, failure_code) def expire(self, attempt_id: str, *, now: datetime) -> AuthorizationAttempt: diff --git a/src/symphonia/infrastructure/sqlite_common.py b/src/symphonia/infrastructure/sqlite_common.py index b940a9d..05e4398 100644 --- a/src/symphonia/infrastructure/sqlite_common.py +++ b/src/symphonia/infrastructure/sqlite_common.py @@ -2,7 +2,31 @@ from __future__ import annotations +import json import sqlite3 +from typing import Any + + +def _reject_non_finite_json(value: str) -> None: + raise ValueError(f"non-standard JSON constant is not allowed: {value}") + + +def dump_json(value: Any) -> str: + """Serialize adapter payloads using one strict, deterministic policy.""" + + return json.dumps( + value, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ) + + +def load_json(value: str) -> Any: + """Read adapter JSON without accepting NaN or Infinity extensions.""" + + return json.loads(value, parse_constant=_reject_non_finite_json) def connect(path: str) -> sqlite3.Connection: @@ -15,4 +39,4 @@ def connect(path: str) -> sqlite3.Connection: return connection -__all__ = ["connect"] +__all__ = ["connect", "dump_json", "load_json"] diff --git a/src/symphonia/infrastructure/sqlite_connections.py b/src/symphonia/infrastructure/sqlite_connections.py index f8daa5c..967fa6d 100644 --- a/src/symphonia/infrastructure/sqlite_connections.py +++ b/src/symphonia/infrastructure/sqlite_connections.py @@ -3,14 +3,13 @@ from __future__ import annotations from datetime import datetime, timezone -import json import sqlite3 from typing import Any from symphonia.providers.connections import ConnectionState, ProviderConnection from symphonia.providers.contracts import Capability, ProviderCapabilities -from .sqlite_common import connect +from .sqlite_common import connect, dump_json, load_json def _utc(value: datetime) -> str: @@ -42,13 +41,17 @@ def close(self) -> None: self._connection.close() def healthcheck(self) -> bool: - """Return whether the migrated connection store can be read.""" + """Return whether schema and persisted connection values are readable.""" try: - row = self._connection.execute("SELECT 1 AS healthy").fetchone() - except sqlite3.Error: + integrity = self._connection.execute("PRAGMA integrity_check(1)").fetchone() + if integrity is None or integrity[0] != "ok": + return False + for row in self._connection.execute("SELECT * FROM provider_connections").fetchall(): + self._record(row) + except (sqlite3.Error, TypeError, ValueError): return False - return row is not None and row["healthy"] == 1 + return True def _migrate(self) -> None: self._connection.executescript( @@ -235,22 +238,19 @@ def _record(row: sqlite3.Row) -> ProviderConnection: def _serialize_capabilities(capabilities: ProviderCapabilities | None) -> str | None: if capabilities is None: return None - return json.dumps( + return dump_json( { "enabled": sorted(capability.value for capability in capabilities.enabled), "evidence_version": capabilities.evidence_version, "observed_at": capabilities.observed_at, }, - ensure_ascii=False, - sort_keys=True, - separators=(",", ":"), ) def _deserialize_capabilities(payload: str | None) -> ProviderCapabilities | None: if payload is None: return None - value = json.loads(payload) + value = load_json(payload) return ProviderCapabilities( enabled=frozenset(Capability(item) for item in value["enabled"]), evidence_version=value["evidence_version"], diff --git a/src/symphonia/infrastructure/sqlite_library.py b/src/symphonia/infrastructure/sqlite_library.py index 0f08c0c..05acc92 100644 --- a/src/symphonia/infrastructure/sqlite_library.py +++ b/src/symphonia/infrastructure/sqlite_library.py @@ -54,13 +54,17 @@ def close(self) -> None: self._connection.close() def healthcheck(self) -> bool: - """Return whether the migrated projection store can be read.""" + """Return whether schema and projection references are readable.""" try: - row = self._connection.execute("SELECT 1 AS healthy").fetchone() + integrity = self._connection.execute("PRAGMA integrity_check(1)").fetchone() + if integrity is None or integrity[0] != "ok": + return False + if self._connection.execute("PRAGMA foreign_key_check").fetchone() is not None: + return False except sqlite3.Error: return False - return row is not None and row["healthy"] == 1 + return True def _migrate(self) -> None: self._connection.executescript( diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 6fb2ca2..aa391b3 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -47,6 +47,40 @@ class LeaseConflict(RuntimeError): _MAX_DIAGNOSTIC_OPERATIONS = 100 _MAX_DIAGNOSTIC_EVENTS = 100 _MAX_DIAGNOSTIC_KEYS = 100 +_OPERATION_STATES = frozenset( + { + "queued", + "running", + "waiting_rate_limit", + "waiting_user", + "retry_scheduled", + "succeeded", + "partial", + "failed", + "cancelled", + } +) + + +def _require_text(value: Any, *, label: str) -> str: + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"{label} must be a non-empty string") + return value + + +def _reject_non_finite_json(value: str) -> None: + raise ValueError(f"non-standard JSON constant is not allowed: {value}") + + +def _load_json(value: str) -> Any: + return json.loads(value, parse_constant=_reject_non_finite_json) + + +def _load_json_object(value: str, *, label: str) -> dict[str, Any]: + parsed = _load_json(value) + if not isinstance(parsed, dict): + raise ValueError(f"{label} must be a JSON object") + return parsed def _validate_payload_keys(payload: Any) -> None: @@ -138,10 +172,21 @@ def close(self) -> None: self._connection.close() def healthcheck(self) -> bool: - """Return whether the migrated store can answer a basic read.""" + """Return whether schema and durable operation values are readable.""" - row = self._connection.execute("SELECT 1 AS healthy").fetchone() - return row is not None and row["healthy"] == 1 + try: + integrity = self._connection.execute("PRAGMA integrity_check(1)").fetchone() + if integrity is None or integrity[0] != "ok": + return False + if self._connection.execute("PRAGMA foreign_key_check").fetchone() is not None: + return False + for row in self._connection.execute("SELECT * FROM operations").fetchall(): + self._record(row) + for row in self._connection.execute("SELECT * FROM operation_events").fetchall(): + self._event(row) + return True + except (sqlite3.Error, TypeError, ValueError): + return False def _migrate(self) -> None: current_version = int(self._connection.execute("PRAGMA user_version").fetchone()[0]) @@ -220,12 +265,17 @@ def create( ) -> OperationRecord: """Create once, or return the identical prior operation by key.""" - if not operation_type.strip() or not idempotency_key.strip(): - raise ValueError("operation_type and idempotency_key must not be empty") + _require_text(operation_type, label="operation_type") + _require_text(idempotency_key, label="idempotency_key") _validate_object_payload(payload, label="operation payload") - operation_id = operation_id or str(uuid.uuid4()) + if operation_id is None: + operation_id = str(uuid.uuid4()) + else: + _require_text(operation_id, label="operation_id") timestamp = _utc(now) - payload_json = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + payload_json = json.dumps( + payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"), allow_nan=False + ) self._connection.execute("BEGIN IMMEDIATE") try: self._connection.execute( @@ -416,6 +466,8 @@ def claim( ) -> OperationRecord: """Claim queued/retryable work or reclaim a lease that has expired.""" + _require_text(operation_id, label="operation_id") + _require_text(worker_id, label="worker_id") if lease_seconds <= 0: raise ValueError("lease_seconds must be positive") now_text = _utc(now) @@ -480,6 +532,9 @@ def claim_next( durable eligibility; handler dispatch remains an application concern. """ + _require_text(worker_id, label="worker_id") + if operation_type is not None: + _require_text(operation_type, label="operation_type") if lease_seconds <= 0: raise ValueError("lease_seconds must be positive") now_text = _utc(now) @@ -554,6 +609,8 @@ def renew_lease( ) -> OperationRecord: """Extend a healthy lease; an expired owner cannot resurrect it.""" + _require_text(operation_id, label="operation_id") + _require_text(worker_id, label="worker_id") if lease_seconds <= 0: raise ValueError("lease_seconds must be positive") now_text = _utc(now) @@ -598,11 +655,15 @@ def checkpoint( ) -> OperationRecord: """Persist a checkpoint only for the current, unexpired lease holder.""" + _require_text(operation_id, label="operation_id") + _require_text(worker_id, label="worker_id") if state not in {"running", "succeeded", "partial", "failed", "cancelled", "waiting_user"}: raise ValueError("invalid checkpoint state") _validate_object_payload(checkpoint, label="checkpoint") now_text = _utc(now) - checkpoint_json = json.dumps(checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + checkpoint_json = json.dumps( + checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":"), allow_nan=False + ) self._connection.execute("BEGIN IMMEDIATE") try: row = self._connection.execute( @@ -741,10 +802,14 @@ def schedule_retry( ) -> OperationRecord: """Release a lease and persist a restart-safe retry time.""" + _require_text(operation_id, label="operation_id") + _require_text(worker_id, label="worker_id") _validate_object_payload(checkpoint, label="checkpoint") now_text = _utc(now) next_run_text = _utc(next_run_at) - checkpoint_json = json.dumps(checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + checkpoint_json = json.dumps( + checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":"), allow_nan=False + ) self._connection.execute("BEGIN IMMEDIATE") try: row = self._connection.execute( @@ -809,10 +874,14 @@ def schedule_rate_limit( ) -> OperationRecord: """Release a lease until an absolute provider rate-limit time.""" + _require_text(operation_id, label="operation_id") + _require_text(worker_id, label="worker_id") _validate_object_payload(checkpoint, label="checkpoint") now_text = _utc(now) next_run_text = _utc(next_run_at) - checkpoint_json = json.dumps(checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + checkpoint_json = json.dumps( + checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":"), allow_nan=False + ) self._connection.execute("BEGIN IMMEDIATE") try: row = self._connection.execute( @@ -893,7 +962,9 @@ def _append_event( event_type, state, worker_id, - json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")), + json.dumps( + payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"), allow_nan=False + ), created_at, ), ) @@ -920,13 +991,22 @@ def _bounded_keys(value: dict[str, Any]) -> tuple[list[str], bool]: @staticmethod def _record(row: sqlite3.Row) -> OperationRecord: + state = row["state"] + if state not in _OPERATION_STATES: + raise ValueError(f"operation store contains invalid state: {state!r}") + if row["cancel_requested"] not in (0, 1): + raise ValueError("operation store contains an invalid cancellation flag") + payload = _load_json_object(row["payload_json"], label="operation payload") + checkpoint = _load_json_object(row["checkpoint_json"], label="operation checkpoint") + _validate_payload_keys(payload) + _validate_payload_keys(checkpoint) return OperationRecord( operation_id=row["operation_id"], operation_type=row["operation_type"], - state=row["state"], + state=state, idempotency_key=row["idempotency_key"], - payload=json.loads(row["payload_json"]), - checkpoint=json.loads(row["checkpoint_json"]), + payload=payload, + checkpoint=checkpoint, worker_id=row["worker_id"], lease_expires_at=None if row["lease_expires_at"] is None else _parse_utc(row["lease_expires_at"]), next_run_at=None if row["next_run_at"] is None else _parse_utc(row["next_run_at"]), @@ -937,12 +1017,15 @@ def _record(row: sqlite3.Row) -> OperationRecord: @staticmethod def _event(row: sqlite3.Row) -> OperationEvent: + state = row["state"] + if state not in _OPERATION_STATES: + raise ValueError(f"operation event contains invalid state: {state!r}") return OperationEvent( sequence=row["sequence"], operation_id=row["operation_id"], event_type=row["event_type"], - state=row["state"], + state=state, worker_id=row["worker_id"], - payload=json.loads(row["payload_json"]), + payload=_load_json_object(row["payload_json"], label="operation event payload"), created_at=_parse_utc(row["created_at"]), ) diff --git a/src/symphonia/infrastructure/sqlite_plans.py b/src/symphonia/infrastructure/sqlite_plans.py index e2bef30..92ed1ca 100644 --- a/src/symphonia/infrastructure/sqlite_plans.py +++ b/src/symphonia/infrastructure/sqlite_plans.py @@ -4,7 +4,6 @@ from dataclasses import dataclass from datetime import datetime, timezone -import json import sqlite3 from typing import Any @@ -16,7 +15,7 @@ PlanAcceptanceError, ) -from .sqlite_common import connect +from .sqlite_common import connect, dump_json, load_json def _utc(value: datetime) -> str: @@ -50,13 +49,19 @@ def close(self) -> None: self._connection.close() def healthcheck(self) -> bool: - """Return whether the migrated plan store can be read.""" + """Return whether schema and immutable plan values are readable.""" try: - row = self._connection.execute("SELECT 1 AS healthy").fetchone() - except sqlite3.Error: + integrity = self._connection.execute("PRAGMA integrity_check(1)").fetchone() + if integrity is None or integrity[0] != "ok": + return False + for row in self._connection.execute("SELECT digest, plan_json FROM copy_plans").fetchall(): + plan = _deserialize(load_json(row["plan_json"])) + if plan.digest != row["digest"] or plan.recompute_digest() != row["digest"]: + return False + except (sqlite3.Error, TypeError, ValueError, KeyError): return False - return row is not None and row["healthy"] == 1 + return True def _migrate(self) -> None: self._connection.executescript( @@ -72,7 +77,7 @@ def _migrate(self) -> None: def save(self, plan: CopyPlan, *, now: datetime) -> StoredCopyPlan: payload = _serialize(plan) - plan_json = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + plan_json = dump_json(payload) self._connection.execute( "INSERT OR IGNORE INTO copy_plans (digest, plan_json, created_at) VALUES (?, ?, ?)", (plan.digest, plan_json, _utc(now)), @@ -88,8 +93,11 @@ def get(self, digest: str) -> StoredCopyPlan: ).fetchone() if row is None: raise CopyPlanNotFound(digest) + plan = _deserialize(load_json(row["plan_json"])) + if plan.digest != row["digest"] or plan.recompute_digest() != row["digest"]: + raise ValueError("stored plan content does not match its digest") return StoredCopyPlan( - plan=_deserialize(json.loads(row["plan_json"])), + plan=plan, accepted_at=None if row["accepted_at"] is None else _parse_utc(row["accepted_at"]), ) diff --git a/src/symphonia/infrastructure/sqlite_resolutions.py b/src/symphonia/infrastructure/sqlite_resolutions.py index 55f2c50..a466132 100644 --- a/src/symphonia/infrastructure/sqlite_resolutions.py +++ b/src/symphonia/infrastructure/sqlite_resolutions.py @@ -3,14 +3,13 @@ from __future__ import annotations from dataclasses import replace -import json import sqlite3 import uuid from typing import Any from symphonia.identity.models import ManualDecision, ManualDecisionAction -from .sqlite_common import connect +from .sqlite_common import connect, dump_json, load_json class ResolutionDecisionRepository: @@ -24,13 +23,19 @@ def close(self) -> None: self._connection.close() def healthcheck(self) -> bool: - """Return whether the migrated decision store can be read.""" + """Return whether schema and decision payloads are readable.""" try: - row = self._connection.execute("SELECT 1 AS healthy").fetchone() - except sqlite3.Error: + integrity = self._connection.execute("PRAGMA integrity_check(1)").fetchone() + if integrity is None or integrity[0] != "ok": + return False + for row in self._connection.execute("SELECT payload_json FROM resolution_decisions").fetchall(): + payload = load_json(row["payload_json"]) + if not isinstance(payload, dict): + return False + except (sqlite3.Error, TypeError, ValueError): return False - return row is not None and row["healthy"] == 1 + return True def _migrate(self) -> None: self._connection.executescript( @@ -54,7 +59,7 @@ def _migrate(self) -> None: def record(self, decision: ManualDecision) -> ManualDecision: decision_id = decision.decision_id or str(uuid.uuid4()) persisted = replace(decision, decision_id=decision_id) - payload = json.dumps( + payload = dump_json( { "provider_track_key": persisted.provider_track_key, "candidate_recording_id": persisted.candidate_recording_id, @@ -63,9 +68,6 @@ def record(self, decision: ManualDecision) -> ManualDecision: "reason": persisted.reason, "created_at": persisted.created_at, }, - ensure_ascii=False, - sort_keys=True, - separators=(",", ":"), ) self._connection.execute( """ diff --git a/src/symphonia/providers/apple_music.py b/src/symphonia/providers/apple_music.py index 9839543..24d1d89 100644 --- a/src/symphonia/providers/apple_music.py +++ b/src/symphonia/providers/apple_music.py @@ -97,7 +97,7 @@ class AppleMusicAdapter(ProviderAdapter): maturity="experimental", support_level="library-playlist-read", upstream_dependencies=("Apple Music API", "MusicKit user authentication"), - reviewed_on="2026-09-20", + reviewed_on="2026-09-22", ) def __init__( @@ -107,9 +107,9 @@ def __init__( page_size: int = 25, max_pages: int = 10_000, ) -> None: - if not 1 <= page_size <= 100: + if isinstance(page_size, bool) or not isinstance(page_size, int) or not 1 <= page_size <= 100: raise ValueError("Apple Music playlist page_size must be between 1 and 100") - if max_pages <= 0: + if isinstance(max_pages, bool) or not isinstance(max_pages, int) or max_pages <= 0: raise ValueError("Apple Music max_pages must be positive") self._client = client or UrllibAppleMusicClient() self._tokens_for_connection = tokens_for_connection @@ -170,7 +170,9 @@ def read_playlist_pages( cursor=None if not pages and cursor is None else str(offset), next_cursor=None if next_offset is None else str(next_offset), complete=next_offset is None, - revision=response.payload.get("etag"), + revision=response.payload.get("etag") + if isinstance(response.payload.get("etag"), str) + else None, ) ) if next_offset is None: @@ -178,8 +180,19 @@ def read_playlist_pages( offset = next_offset def _request(self, connection_id: str, path: str, query: Mapping[str, str]) -> AppleJsonResponse: - developer_token, user_token = self._tokens_for_connection(connection_id) - if not developer_token.strip() or not user_token.strip(): + try: + developer_token, user_token = self._tokens_for_connection(connection_id) + except (TypeError, ValueError) as error: + raise ProviderApiError( + ProviderErrorCategory.AUTHENTICATION_REQUIRED, + "Apple Music connection did not provide a token pair", + ) from error + if ( + not isinstance(developer_token, str) + or not isinstance(user_token, str) + or not developer_token.strip() + or not user_token.strip() + ): raise ProviderApiError(ProviderErrorCategory.AUTHENTICATION_REQUIRED, "Apple Music connection has no usable tokens") response = self._client.request( "GET", @@ -216,9 +229,11 @@ def _request(self, connection_id: str, path: str, query: Mapping[str, str]) -> A def _parse_offset(cursor: str | None) -> int: if cursor is None: return 0 + if not isinstance(cursor, str) or not cursor.strip(): + raise ValueError("Apple Music playlist cursor must be a non-empty string offset") try: value = int(cursor) - except ValueError as error: + except (TypeError, ValueError) as error: raise ValueError("Apple Music playlist cursor must be an integer offset") from error if value < 0: raise ValueError("Apple Music playlist cursor must not be negative") @@ -228,11 +243,42 @@ def _parse_offset(cursor: str | None) -> int: def _next_offset(next_url: Any, fallback: int) -> int | None: if not next_url: return None - if isinstance(next_url, str): - values = parse_qs(urlsplit(next_url).query).get("offset") - if values: - return AppleMusicAdapter._parse_offset(values[0]) - return fallback + if not isinstance(next_url, str): + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Apple Music pagination next link was not a URL", + ) + try: + parsed = urlsplit(next_url) + except ValueError as error: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Apple Music pagination next link was malformed", + ) from error + if parsed.scheme not in {"http", "https"} or not parsed.netloc: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Apple Music pagination next link was not an absolute URL", + ) + values = parse_qs(parsed.query, keep_blank_values=True).get("offset") + if values is None or len(values) != 1: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Apple Music pagination next link did not contain one offset", + ) + try: + next_offset = AppleMusicAdapter._parse_offset(values[0]) + except ValueError as error: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Apple Music pagination next link contained an invalid offset", + ) from error + if next_offset < fallback: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Apple Music pagination moved backwards", + ) + return next_offset @staticmethod def _entry(playlist: ProviderObjectRef, item: Any, position: int) -> ProviderPlaylistEntry: diff --git a/src/symphonia/providers/authorization.py b/src/symphonia/providers/authorization.py index c0b1c58..782a23a 100644 --- a/src/symphonia/providers/authorization.py +++ b/src/symphonia/providers/authorization.py @@ -10,9 +10,14 @@ from dataclasses import dataclass from datetime import datetime from enum import Enum +import re from urllib.parse import urlsplit +MAX_AUTHORIZATION_STATE_LENGTH = 512 +_FAILURE_CODE = re.compile(r"^[a-z0-9][a-z0-9_.-]{0,63}$") + + class AuthorizationState(str, Enum): CREATED = "created" CONSUMED = "consumed" @@ -24,7 +29,7 @@ class AuthorizationState(str, Enum): def validate_redirect_uri(value: str) -> str: """Validate a fixed OAuth callback URI before durable binding.""" - if not value or not value.strip() or any(character.isspace() for character in value): + if not isinstance(value, str) or not value.strip() or any(character.isspace() for character in value): raise ValueError("redirect_uri must be a nonblank URI without whitespace") parsed = urlsplit(value) if parsed.scheme not in {"http", "https"} or not parsed.hostname: @@ -38,6 +43,14 @@ def validate_redirect_uri(value: str) -> str: return value +def validate_failure_code(value: str) -> str: + """Keep durable authorization outcomes bounded and safe for diagnostics.""" + + if not isinstance(value, str) or _FAILURE_CODE.fullmatch(value) is None: + raise ValueError("failure_code must be a bounded lowercase code") + return value + + @dataclass(frozen=True, slots=True) class AuthorizationAttempt: attempt_id: str @@ -59,7 +72,7 @@ def __post_init__(self) -> None: (self.redirect_uri, "redirect_uri"), (self.state_digest, "state_digest"), ): - if not value.strip(): + if not isinstance(value, str) or not value.strip(): raise ValueError(f"{field_name} must not be empty") validate_redirect_uri(self.redirect_uri) if self.created_at.tzinfo is None or self.expires_at.tzinfo is None: @@ -68,3 +81,5 @@ def __post_init__(self) -> None: raise ValueError("authorization attempt must expire after creation") if self.completed_at is not None and self.completed_at.tzinfo is None: raise ValueError("completed_at must be timezone-aware") + if self.failure_code is not None: + validate_failure_code(self.failure_code) diff --git a/src/symphonia/providers/contracts.py b/src/symphonia/providers/contracts.py index bb2c630..bdc4f58 100644 --- a/src/symphonia/providers/contracts.py +++ b/src/symphonia/providers/contracts.py @@ -8,10 +8,16 @@ from __future__ import annotations from dataclasses import dataclass +from datetime import date from enum import Enum from typing import Iterable, Protocol +def _require_text(value: object, *, field_name: str) -> None: + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"{field_name} must be a non-empty string") + + class AccessBasis(str, Enum): OFFICIAL = "official" UNOFFICIAL = "unofficial" @@ -51,8 +57,7 @@ def __post_init__(self) -> None: (self.object_id, "object_id"), (self.namespace, "namespace"), ): - if not value.strip(): - raise ValueError(f"{field_name} must not be empty") + _require_text(value, field_name=field_name) @property def external_key(self) -> tuple[str, str, str, str]: @@ -70,10 +75,20 @@ class ProviderManifest: reviewed_on: str | None = None def __post_init__(self) -> None: - if not self.provider.strip() or not self.display_name.strip(): - raise ValueError("provider and display_name must not be empty") - if not self.maturity.strip() or not self.support_level.strip(): - raise ValueError("maturity and support_level must not be empty") + if not isinstance(self.access_basis, AccessBasis): + raise ValueError("access_basis must be an AccessBasis value") + _require_text(self.provider, field_name="provider") + _require_text(self.display_name, field_name="display_name") + _require_text(self.maturity, field_name="maturity") + _require_text(self.support_level, field_name="support_level") + if any(not isinstance(item, str) or not item.strip() for item in self.upstream_dependencies): + raise ValueError("upstream_dependencies must contain non-empty strings") + if self.reviewed_on is not None: + _require_text(self.reviewed_on, field_name="reviewed_on") + try: + date.fromisoformat(self.reviewed_on) + except ValueError as error: + raise ValueError("reviewed_on must be an ISO date") from error @dataclass(frozen=True, slots=True) @@ -84,6 +99,12 @@ class ProviderCapabilities: evidence_version: str observed_at: str + def __post_init__(self) -> None: + if any(not isinstance(item, Capability) for item in self.enabled): + raise ValueError("enabled must contain Capability values") + _require_text(self.evidence_version, field_name="evidence_version") + _require_text(self.observed_at, field_name="observed_at") + def supports(self, capability: Capability) -> bool: return capability in self.enabled @@ -99,10 +120,21 @@ class ProviderPlaylistEntry: source_added_at: str | None = None def __post_init__(self) -> None: - if not self.occurrence_id.strip(): - raise ValueError("occurrence_id must not be empty") + _require_text(self.occurrence_id, field_name="occurrence_id") + if isinstance(self.position, bool) or not isinstance(self.position, int): + raise ValueError("position must be an integer") if self.position < 0: raise ValueError("position must be non-negative") + if not isinstance(self.track, ProviderObjectRef): + raise ValueError("track must be a ProviderObjectRef") + if not isinstance(self.media_kind, MediaKind): + raise ValueError("media_kind must be a MediaKind value") + if self.title is not None and not isinstance(self.title, str): + raise ValueError("title must be a string when provided") + if self.source_added_at is not None and not isinstance(self.source_added_at, str): + raise ValueError("source_added_at must be a string when provided") + if not isinstance(self.available, bool): + raise ValueError("available must be a boolean") @dataclass(frozen=True, slots=True) @@ -115,13 +147,30 @@ class ProviderPlaylistPage: revision: str | None = None def __post_init__(self) -> None: + if not isinstance(self.playlist, ProviderObjectRef): + raise ValueError("playlist must be a ProviderObjectRef") if self.playlist.object_type not in {"playlist", "library-playlists"}: raise ValueError("playlist page requires a playlist or library-playlists object reference") + if not isinstance(self.entries, tuple): + raise ValueError("entries must be a tuple") + if not isinstance(self.complete, bool): + raise ValueError("complete must be a boolean") + if any(not isinstance(entry, ProviderPlaylistEntry) for entry in self.entries): + raise ValueError("entries must contain ProviderPlaylistEntry values") positions = [entry.position for entry in self.entries] if positions != sorted(positions): raise ValueError("page entries must be ordered by position") + if len(positions) != len(set(positions)): + raise ValueError("page entry positions must be unique") if self.complete and self.next_cursor is not None: raise ValueError("a complete page cannot have a next_cursor") + for value, field_name in ( + (self.cursor, "cursor"), + (self.next_cursor, "next_cursor"), + (self.revision, "revision"), + ): + if value is not None and not isinstance(value, str): + raise ValueError(f"{field_name} must be a string when provided") class ProviderAdapter(Protocol): diff --git a/src/symphonia/providers/errors.py b/src/symphonia/providers/errors.py index 2b9138f..d8a890d 100644 --- a/src/symphonia/providers/errors.py +++ b/src/symphonia/providers/errors.py @@ -9,9 +9,28 @@ _BEARER = re.compile(r"(?i)\bBearer\s+[^\s,;]+") -_ASSIGNMENT = re.compile( - r"(?i)\b(token|access[_-]?token|refresh[_-]?token|id[_-]?token|client[_-]?secret|secret|password|cookie|authorization)\s*[:=]\s*[^\s,;]+" +_CREDENTIAL_NAME = ( + r"(?:token|access[_-]?token|refresh[_-]?token|id[_-]?token|" + r"client[_-]?secret|secret|password|cookie|authorization)" ) +_ASSIGNMENT_QUOTED = re.compile( + rf"(?i)(?P[\"']?{_CREDENTIAL_NAME}[\"']?)" + r"\s*(?P[:=])\s*" + r"(?P[\"'])(?P(?:\\.|[^\"'])*)(?P=quote)" +) +_ASSIGNMENT_UNQUOTED = re.compile( + rf"(?i)(?P[\"']?{_CREDENTIAL_NAME}[\"']?)" + r"\s*(?P[:=])\s*" + r"(?P[^,\s;}\"'&]+)" +) + + +def _redact_assignment(match: re.Match[str]) -> str: + quote = match.groupdict().get("quote") or "" + return ( + f"{match.group('key')}{match.group('separator')}" + f"{quote}[REDACTED]{quote}" + ) def redact_error_detail(detail: str) -> str: @@ -20,7 +39,8 @@ def redact_error_detail(detail: str) -> str: if not isinstance(detail, str) or not detail.strip(): raise ValueError("provider error detail must not be empty") redacted = _BEARER.sub("Bearer [REDACTED]", detail) - return _ASSIGNMENT.sub(lambda match: f"{match.group(1)}=[REDACTED]", redacted) + redacted = _ASSIGNMENT_QUOTED.sub(_redact_assignment, redacted) + return _ASSIGNMENT_UNQUOTED.sub(_redact_assignment, redacted) class ProviderErrorCategory(str, Enum): diff --git a/src/symphonia/providers/spotify.py b/src/symphonia/providers/spotify.py index d0a582a..b0f39ce 100644 --- a/src/symphonia/providers/spotify.py +++ b/src/symphonia/providers/spotify.py @@ -13,7 +13,7 @@ import json from typing import Any, Protocol from urllib.error import HTTPError, URLError -from urllib.parse import urlencode +from urllib.parse import parse_qs, urlencode, urlsplit from urllib.request import Request, urlopen from .contracts import ( @@ -106,7 +106,7 @@ class SpotifyAdapter(ProviderAdapter, PlaylistWriter): maturity="beta", support_level="playlist-read/write", upstream_dependencies=("Spotify Web API",), - reviewed_on="2026-09-20", + reviewed_on="2026-09-22", ) def __init__( @@ -118,9 +118,9 @@ def __init__( allow_writes: bool = False, max_pages: int = 10_000, ) -> None: - if not 1 <= page_size <= 50: + if isinstance(page_size, bool) or not isinstance(page_size, int) or not 1 <= page_size <= 50: raise ValueError("Spotify playlist page_size must be between 1 and 50") - if max_pages <= 0: + if isinstance(max_pages, bool) or not isinstance(max_pages, int) or max_pages <= 0: raise ValueError("Spotify max_pages must be positive") self._client = client self._token_for_connection = token_for_connection @@ -150,12 +150,19 @@ def read_playlist_pages( raise ValueError("Spotify adapter requires a Spotify playlist reference") offset = self._parse_cursor(cursor) pages: list[ProviderPlaylistPage] = [] + seen_offsets: set[int] = set() while True: if len(pages) >= self._max_pages: raise ProviderApiError( ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, "Spotify playlist pagination exceeded the configured page limit", ) + if offset in seen_offsets: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Spotify pagination repeated an offset", + ) + seen_offsets.add(offset) response = self._request( connection_id, "GET", @@ -175,7 +182,8 @@ def read_playlist_pages( ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, "Spotify pagination advanced without returning items", ) - next_cursor = str(offset + len(entries)) if next_url else None + next_offset = self._next_offset(next_url, offset + len(entries)) + next_cursor = str(next_offset) if next_offset is not None else None pages.append( ProviderPlaylistPage( playlist=playlist, @@ -183,12 +191,16 @@ def read_playlist_pages( cursor=None if not pages and cursor is None else str(offset), next_cursor=next_cursor, complete=next_cursor is None, - revision=response.payload.get("snapshot_id"), + revision=response.payload.get("snapshot_id") + if isinstance(response.payload.get("snapshot_id"), str) + else None, ) ) if next_cursor is None: return tuple(pages) - offset += len(entries) + if next_offset is None: + raise AssertionError("Spotify pagination cursor was unexpectedly empty") + offset = next_offset def ensure_target_playlist( self, @@ -200,6 +212,12 @@ def ensure_target_playlist( ) -> TargetPlaylist: if provider != self.manifest.provider: raise ValueError("Spotify adapter requires a Spotify target provider") + if not isinstance(name, str) or not name.strip(): + raise ValueError("Spotify playlist name must not be empty") + if not isinstance(idempotency_key, str) or not idempotency_key.strip(): + raise ValueError("Spotify playlist idempotency_key must not be empty") + if visibility not in {"private", "public"}: + raise ValueError("Spotify playlist visibility must be private or public") connection_id = self._write_connection_id() try: response = self._request( @@ -231,6 +249,15 @@ def add_entry( provider_track_id: str, idempotency_key: str, ) -> WriteResult: + if ( + not isinstance(target_playlist_id, str) + or not target_playlist_id.strip() + or not isinstance(provider_track_id, str) + or not provider_track_id.strip() + or not isinstance(idempotency_key, str) + or not idempotency_key.strip() + ): + raise ValueError("Spotify write identifiers must not be empty") try: connection_id = self._write_connection_id() except ProviderWriteError as error: @@ -280,7 +307,7 @@ def _request( body: Mapping[str, Any] | None = None, ) -> JsonResponse: token = self._token_for_connection(connection_id) - if not token.strip(): + if not isinstance(token, str) or not token.strip(): raise ProviderApiError(ProviderErrorCategory.AUTHENTICATION_REQUIRED, "Spotify connection has no usable access token") response = self._client.request(method, path, token=token, query=query, body=body) if 200 <= response.status < 300: @@ -324,6 +351,8 @@ def _write_outcome(error: ProviderApiError) -> WriteOutcome: def _parse_cursor(cursor: str | None) -> int: if cursor is None: return 0 + if not isinstance(cursor, str) or not cursor.strip(): + raise ValueError("Spotify playlist cursor must be a non-empty string offset") try: value = int(cursor) except ValueError as error: @@ -332,6 +361,47 @@ def _parse_cursor(cursor: str | None) -> int: raise ValueError("Spotify playlist cursor must not be negative") return value + @classmethod + def _next_offset(cls, next_url: Any, fallback: int) -> int | None: + if not next_url: + return None + if not isinstance(next_url, str): + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Spotify pagination next link was not a URL", + ) + try: + parsed = urlsplit(next_url) + except ValueError as error: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Spotify pagination next link was malformed", + ) from error + if parsed.scheme not in {"http", "https"} or not parsed.netloc: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Spotify pagination next link was not an absolute URL", + ) + values = parse_qs(parsed.query, keep_blank_values=True).get("offset") + if values is None or len(values) != 1: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Spotify pagination next link did not contain one offset", + ) + try: + next_offset = cls._parse_cursor(values[0]) + except ValueError as error: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Spotify pagination next link contained an invalid offset", + ) from error + if next_offset < fallback: + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "Spotify pagination moved backwards", + ) + return next_offset + @staticmethod def _entry(playlist: ProviderObjectRef, item: Any, *, position: int) -> ProviderPlaylistEntry: item_payload = (item.get("item") or item.get("track")) if isinstance(item, Mapping) else None diff --git a/src/symphonia/providers/youtube.py b/src/symphonia/providers/youtube.py index 2f26909..ccc52a5 100644 --- a/src/symphonia/providers/youtube.py +++ b/src/symphonia/providers/youtube.py @@ -35,7 +35,7 @@ class YouTubeDataAdapter(ProviderAdapter): maturity="experimental", support_level="video-playlist-read", upstream_dependencies=("YouTube Data API v3",), - reviewed_on="2026-09-20", + reviewed_on="2026-09-22", ) def __init__( @@ -46,9 +46,9 @@ def __init__( api_key: str | None = None, max_pages: int = 10_000, ) -> None: - if not 1 <= page_size <= 50: + if isinstance(page_size, bool) or not isinstance(page_size, int) or not 1 <= page_size <= 50: raise ValueError("YouTube playlist page_size must be between 1 and 50") - if max_pages <= 0: + if isinstance(max_pages, bool) or not isinstance(max_pages, int) or max_pages <= 0: raise ValueError("YouTube max_pages must be positive") self._client = client or UrllibJsonClient("https://www.googleapis.com/youtube/v3") self._token_for_connection = token_for_connection @@ -76,6 +76,8 @@ def read_playlist_pages( ) -> tuple[ProviderPlaylistPage, ...]: if playlist.provider != self.manifest.provider or playlist.object_type != "playlist": raise ValueError("YouTube Data adapter requires a youtube_data playlist reference") + if cursor is not None and (not isinstance(cursor, str) or not cursor.strip()): + raise ValueError("YouTube playlist cursor must be a non-empty string token") page_token = cursor pages: list[ProviderPlaylistPage] = [] position = 0 @@ -108,6 +110,11 @@ def read_playlist_pages( for index, item in enumerate(items) ) next_token = response.payload.get("nextPageToken") + if next_token is not None and (not isinstance(next_token, str) or not next_token.strip()): + raise ProviderApiError( + ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, + "YouTube pagination token was invalid", + ) if next_token and not entries: raise ProviderApiError( ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED, @@ -120,7 +127,9 @@ def read_playlist_pages( cursor=page_token, next_cursor=str(next_token) if next_token else None, complete=not bool(next_token), - revision=response.payload.get("etag"), + revision=response.payload.get("etag") + if isinstance(response.payload.get("etag"), str) + else None, ) ) if not next_token: @@ -130,7 +139,7 @@ def read_playlist_pages( def _request(self, connection_id: str, path: str, query: Mapping[str, str]): token = self._token_for_connection(connection_id) - if not token.strip(): + if not isinstance(token, str) or not token.strip(): raise ProviderApiError(ProviderErrorCategory.AUTHENTICATION_REQUIRED, "YouTube connection has no usable access token") complete_query = dict(query) if self._api_key: diff --git a/src/symphonia/runtime/http.py b/src/symphonia/runtime/http.py index e8439d8..55171e3 100644 --- a/src/symphonia/runtime/http.py +++ b/src/symphonia/runtime/http.py @@ -24,14 +24,17 @@ def __init__( *, resources: RuntimeResources | None = None, ) -> None: + if repository is not None and resources is not None: + raise ValueError("repository and resources are mutually exclusive") if repository is None and resources is None: raise ValueError("repository or resources must be supplied") + normalized_ingress_path = _normalize_base_path(ingress_path) super().__init__(address, SymphoniaRequestHandler) self.resources = resources self.repository = resources.operations if resources is not None else repository self.readiness_check: Callable[[], bool] = resources.healthcheck if resources is not None else repository.healthcheck self.service_version = __version__ - self.ingress_path = _normalize_base_path(ingress_path) + self.ingress_path = normalized_ingress_path def close_resources(self) -> None: if self.resources is not None: @@ -55,7 +58,12 @@ def do_GET(self) -> None: # noqa: N802 - stdlib handler API self._json(status, payload) def _json(self, status: int, payload: dict[str, Any]) -> None: - body = json.dumps(payload, ensure_ascii=False, sort_keys=True).encode("utf-8") + body = json.dumps( + payload, + ensure_ascii=False, + sort_keys=True, + allow_nan=False, + ).encode("utf-8") self.send_response(status) self.send_header("Content-Type", "application/json; charset=utf-8") self.send_header("Content-Length", str(len(body))) diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index e7ac78a..02be36b 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -42,7 +42,7 @@ class RuntimeResources: @classmethod def open(cls, database_path: str) -> "RuntimeResources": - if not database_path.strip(): + if not isinstance(database_path, str) or not database_path.strip(): raise ValueError("database_path must not be empty") opened: list[object] = [] try: @@ -117,7 +117,7 @@ def backup_to(self, destination_path: str) -> None: resulting backup should be retained. """ - if not destination_path.strip(): + if not isinstance(destination_path, str) or not destination_path.strip(): raise ValueError("destination_path must not be empty") if self.database_path == ":memory:": raise ValueError("backups require a persistent database path") @@ -129,6 +129,10 @@ def backup_to(self, destination_path: str) -> None: destination_path_object = Path(destination_path).expanduser().resolve() if live_path == destination_path_object: raise ValueError("destination_path must differ from the live database") + if not destination_path_object.parent.is_dir(): + raise ValueError("destination_path parent directory must exist") + if destination_path_object.exists() and destination_path_object.is_dir(): + raise ValueError("destination_path must be a file path") temporary_path: str | None = None try: @@ -146,6 +150,8 @@ def backup_to(self, destination_path: str) -> None: destination.commit() finally: destination.close() + if not self.validate_backup(temporary_path): + raise RuntimeError("SQLite backup integrity validation failed") os.replace(temporary_path, destination_path_object) temporary_path = None finally: diff --git a/tests/test_apple_music_adapter.py b/tests/test_apple_music_adapter.py index 352e0df..4d2a5f4 100644 --- a/tests/test_apple_music_adapter.py +++ b/tests/test_apple_music_adapter.py @@ -86,6 +86,31 @@ def test_missing_developer_or_user_token_fails_closed(self) -> None: self.assertEqual(context.exception.category, ProviderErrorCategory.AUTHENTICATION_REQUIRED) + for token_pair in ((None, "user-token"), ("developer-token",), ("developer-token", {})): + adapter = AppleMusicAdapter(FakeAppleClient(), lambda connection_id, pair=token_pair: pair) # type: ignore[arg-type] + with self.subTest(token_pair=token_pair), self.assertRaises(ProviderApiError) as context: + adapter.read_playlist_pages("apple-connection-1", self.playlist()) + self.assertEqual(context.exception.category, ProviderErrorCategory.AUTHENTICATION_REQUIRED) + + def test_malformed_next_link_fails_closed_without_fallback_pagination(self) -> None: + class MalformedClient(FakeAppleClient): + def request(self, method, path, *, developer_token, user_token, query): + self.calls.append((method, path, developer_token, user_token, query)) + return AppleJsonResponse( + 200, + { + "data": [{"id": "song-1", "type": "songs", "attributes": {"name": "One"}}], + "next": "https://api.music.apple.com/v1/me/library/playlists/playlist-1/tracks?offset=not-an-integer", + }, + {}, + ) + + with self.assertRaises(ProviderApiError) as context: + AppleMusicAdapter( + MalformedClient(), lambda connection_id: ("developer-token", "user-token") + ).read_playlist_pages("apple-connection-1", self.playlist()) + self.assertEqual(context.exception.category, ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED) + def test_repeated_offset_is_a_provider_contract_failure(self) -> None: class LoopingClient(FakeAppleClient): def request(self, method, path, *, developer_token, user_token, query): @@ -114,6 +139,12 @@ def test_max_page_limit_fails_closed_before_unbounded_reads(self) -> None: self.assertEqual(context.exception.category, ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED) + def test_non_textual_cursor_is_rejected_before_provider_request(self) -> None: + with self.assertRaises(ValueError): + AppleMusicAdapter( + FakeAppleClient(), lambda connection_id: ("developer-token", "user-token") + ).read_playlist_pages("apple-connection-1", self.playlist(), cursor=True) # type: ignore[arg-type] + if __name__ == "__main__": unittest.main() diff --git a/tests/test_authorization_service.py b/tests/test_authorization_service.py index a133b66..4a8f991 100644 --- a/tests/test_authorization_service.py +++ b/tests/test_authorization_service.py @@ -63,6 +63,36 @@ def test_consume_and_deny_delegate_single_use_transitions(self) -> None: denied = self.service.deny("attempt-2", now=NOW) self.assertEqual(denied.state, AuthorizationState.DENIED) + def test_callback_consumption_resolves_the_durable_attempt_by_state(self) -> None: + self.service.begin( + provider="spotify", + actor_id="ha-user-1", + redirect_uri="https://ha.example/symphonia/callback", + now=NOW, + ) + + consumed = self.service.consume_callback(raw_state="raw-state-only-at-boundary", now=NOW) + + self.assertEqual(consumed.attempt_id, "attempt-1") + self.assertEqual(consumed.state, AuthorizationState.CONSUMED) + + def test_callback_provider_binding_is_checked_before_consumption(self) -> None: + self.service.begin( + provider="spotify", + actor_id="ha-user-1", + redirect_uri="https://ha.example/symphonia/callback", + now=NOW, + ) + + with self.assertRaises(ValueError): + self.service.consume_callback_for_provider( + raw_state="raw-state-only-at-boundary", + provider="google", + now=NOW, + ) + + self.assertEqual(self.repository.get("attempt-1").state, AuthorizationState.CREATED) + def test_redirect_uri_rejects_unsafe_callback_forms(self) -> None: for redirect_uri in ( "javascript:alert(1)", @@ -78,6 +108,14 @@ def test_redirect_uri_rejects_unsafe_callback_forms(self) -> None: now=NOW, ) + with self.assertRaises(ValueError): + self.service.begin( + provider="spotify", + actor_id="ha-user-1", + redirect_uri=None, # type: ignore[arg-type] + now=NOW, + ) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_copy_planning.py b/tests/test_copy_planning.py index 5ccb8a8..0ffd355 100644 --- a/tests/test_copy_planning.py +++ b/tests/test_copy_planning.py @@ -36,6 +36,7 @@ def test_strict_plan_preserves_order_and_duplicate_occurrences(self) -> None: ) self.assertFalse(plan.blocked) + self.assertEqual(plan.recompute_digest(), plan.digest) self.assertEqual([entry.occurrence_id for entry in plan.writable_entries], ["occ-1", "occ-2"]) self.assertEqual([entry.position for entry in plan.writable_entries], [0, 1]) self.assertEqual(plan.writable_entries[0].target_track_id, plan.writable_entries[1].target_track_id) @@ -86,6 +87,31 @@ def test_acceptance_rejects_a_changed_digest(self) -> None: with self.assertRaises(PlanAcceptanceError): plan.accept("not-the-plan") + def test_acceptance_rejects_tampered_plan_content(self) -> None: + plan = self.service.plan( + snapshot(SourcePlaylistEntry("occ-1", 0, "sp-1", EntryClassification.READY, "yt-1")), + target_provider="youtube", + target_playlist_name="Rock", + ) + tampered = type(plan)( + source_snapshot_id=plan.source_snapshot_id, + source_provider=plan.source_provider, + source_playlist_id=plan.source_playlist_id, + target_provider=plan.target_provider, + target_playlist_name="Tampered", + target_visibility=plan.target_visibility, + policy=plan.policy, + entries=plan.entries, + digest=plan.digest, + source_namespace=plan.source_namespace, + target_connection_id=plan.target_connection_id, + target_capabilities=plan.target_capabilities, + target_capability_evidence_version=plan.target_capability_evidence_version, + ) + + with self.assertRaises(PlanAcceptanceError): + tampered.accept(tampered.digest) + def test_snapshot_rejects_duplicate_positions(self) -> None: with self.assertRaises(ValueError): snapshot( diff --git a/tests/test_identity_resolution.py b/tests/test_identity_resolution.py index 9daab4a..5c6bfc4 100644 --- a/tests/test_identity_resolution.py +++ b/tests/test_identity_resolution.py @@ -109,6 +109,28 @@ def test_manual_decisions_are_append_only_and_latest_is_explicit(self) -> None: finally: repository.close() + def test_resolution_healthcheck_fails_closed_on_corrupt_payload(self) -> None: + repository = ResolutionDecisionRepository() + try: + repository.record( + ManualDecision( + provider_track_key="spotify:connection-1:track-1", + candidate_recording_id="recording-1", + action=ManualDecisionAction.ACCEPT, + actor_id="local-user", + reason="Verified the exact recording", + created_at=datetime.now(timezone.utc).isoformat(), + ) + ) + repository._connection.execute( # type: ignore[attr-defined] + "UPDATE resolution_decisions SET payload_json = ?", + ("[]",), + ) + + self.assertFalse(repository.healthcheck()) + finally: + repository.close() + if __name__ == "__main__": unittest.main() diff --git a/tests/test_oauth_callback_spike.py b/tests/test_oauth_callback_spike.py new file mode 100644 index 0000000..b7fecaf --- /dev/null +++ b/tests/test_oauth_callback_spike.py @@ -0,0 +1,174 @@ +from __future__ import annotations + +from datetime import datetime, timezone +import http.client +import threading +import unittest + +from symphonia.application import AuthorizationService +from symphonia.infrastructure import AuthorizationAttemptRepository +from symphonia.providers import AuthorizationState +from tools.oauth_callback_spike import create_callback_server, handle_callback + + +NOW = datetime(2026, 9, 22, 12, 0, tzinfo=timezone.utc) + + +class OAuthCallbackSpikeTests(unittest.TestCase): + def setUp(self) -> None: + self.repository = AuthorizationAttemptRepository() + self.service = AuthorizationService( + self.repository, + state_factory=lambda: "state-for-callback", + id_factory=lambda: "attempt-1", + ) + self.service.begin( + provider="spotify", + actor_id="ha-user-1", + redirect_uri="https://ha.example/symphonia/oauth/callback", + now=NOW, + ) + + def tearDown(self) -> None: + self.repository.close() + + def test_success_consumes_state_and_does_not_echo_code(self) -> None: + result = handle_callback( + "/oauth/callback?state=state-for-callback&code=secret-provider-code", + provider="spotify", + callback_path="/oauth/callback", + authorization=self.service, + now=NOW, + ) + + self.assertEqual(result.status, 200) + self.assertEqual(result.authorization_code, "secret-provider-code") + self.assertEqual(result.attempt.state, AuthorizationState.CONSUMED) + self.assertNotIn("secret-provider-code", result.message) + + def test_wrong_route_is_not_a_management_or_callback_surface(self) -> None: + result = handle_callback( + "/health", + provider="spotify", + callback_path="/oauth/callback", + authorization=self.service, + now=NOW, + ) + + self.assertEqual(result.status, 404) + self.assertEqual(self.repository.get("attempt-1").state, AuthorizationState.CREATED) + + def test_wrong_provider_does_not_consume_state(self) -> None: + result = handle_callback( + "/oauth/callback?state=state-for-callback&code=provider-code", + provider="youtube_data", + callback_path="/oauth/callback", + authorization=self.service, + now=NOW, + ) + + self.assertEqual(result.status, 400) + self.assertEqual(self.repository.get("attempt-1").state, AuthorizationState.CREATED) + + def test_replay_is_rejected_without_returning_the_code(self) -> None: + first = handle_callback( + "/oauth/callback?state=state-for-callback&code=provider-code", + provider="spotify", + callback_path="/oauth/callback", + authorization=self.service, + now=NOW, + ) + replay = handle_callback( + "/oauth/callback?state=state-for-callback&code=provider-code", + provider="spotify", + callback_path="/oauth/callback", + authorization=self.service, + now=NOW, + ) + + self.assertEqual(first.status, 200) + self.assertEqual(replay.status, 400) + self.assertIsNone(replay.authorization_code) + self.assertNotIn("provider-code", replay.message) + + def test_denial_records_a_safe_terminal_state(self) -> None: + result = handle_callback( + "/oauth/callback?state=state-for-callback&error=access_denied&error_description=secret-detail", + provider="spotify", + callback_path="/oauth/callback", + authorization=self.service, + now=NOW, + ) + + self.assertEqual(result.status, 200) + self.assertEqual(result.attempt.state, AuthorizationState.DENIED) + self.assertNotIn("secret-detail", result.message) + + def test_duplicate_state_parameter_is_rejected_without_consumption(self) -> None: + result = handle_callback( + "/oauth/callback?state=one&state=two&code=provider-code", + provider="spotify", + callback_path="/oauth/callback", + authorization=self.service, + now=NOW, + ) + + self.assertEqual(result.status, 400) + self.assertEqual(self.repository.get("attempt-1").state, AuthorizationState.CREATED) + + def test_spike_server_is_loopback_only(self) -> None: + with self.assertRaises(ValueError): + create_callback_server( + authorization=self.service, + provider="spotify", + clock=lambda: NOW, + host="0.0.0.0", + ) + + def test_loopback_http_boundary_consumes_callback_without_leaking_code(self) -> None: + try: + server = create_callback_server( + authorization=self.service, + provider="spotify", + clock=lambda: NOW, + host="127.0.0.1", + port=0, + ) + except PermissionError: + self.skipTest("the test sandbox does not permit local socket binding") + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + host, port = server.server_address + connection = http.client.HTTPConnection(host, port, timeout=2) + try: + connection.request("GET", "/health") + wrong_route = connection.getresponse() + self.assertEqual(wrong_route.status, 404) + self.assertEqual(self.repository.get("attempt-1").state, AuthorizationState.CREATED) + + connection.request( + "GET", + "/oauth/callback?state=state-for-callback&code=secret-http-code", + ) + callback = connection.getresponse() + body = callback.read().decode("utf-8") + finally: + connection.close() + + self.assertEqual(callback.status, 200) + self.assertNotIn("secret-http-code", body) + self.assertEqual( + callback.getheader("Cache-Control"), + "no-store", + ) + self.assertEqual(self.repository.get("attempt-1").state, AuthorizationState.CONSUMED) + finally: + server.shutdown() + server.server_close() + thread.join(timeout=2) + self.assertFalse(thread.is_alive()) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_operation_worker.py b/tests/test_operation_worker.py index 782ff0d..74bd0a4 100644 --- a/tests/test_operation_worker.py +++ b/tests/test_operation_worker.py @@ -61,6 +61,19 @@ def wait(self, timeout=None): self.assertTrue(stop_event.is_set()) + def test_worker_rejects_invalid_identity_and_timing_values(self) -> None: + runner = OperationRunner(self.repository, {}) + invalid_values = ( + {"worker_id": None}, + {"worker_id": "worker", "poll_interval_seconds": True}, + {"worker_id": "worker", "poll_interval_seconds": float("nan")}, + {"worker_id": "worker", "lease_seconds": True}, + {"worker_id": "worker", "lease_seconds": 0.5}, + ) + for values in invalid_values: + with self.subTest(values=values), self.assertRaises(ValueError): + OperationWorker(runner, **values) # type: ignore[arg-type] + if __name__ == "__main__": unittest.main() diff --git a/tests/test_provider_connections.py b/tests/test_provider_connections.py index b2e5d11..013c7df 100644 --- a/tests/test_provider_connections.py +++ b/tests/test_provider_connections.py @@ -40,7 +40,11 @@ def read_playlist_pages(self, connection_id, playlist, cursor=None): class ProviderConnectionServiceTests(unittest.TestCase): def test_provider_error_detail_redacts_common_credentials(self) -> None: - detail = redact_error_detail("Bearer abc123 token=secret refresh_token=refresh-value") + detail = redact_error_detail( + 'Bearer abc123 token=secret refresh_token="refresh-value" ' + 'json={"access_token":"json-secret","client_secret": "client-value"} ' + "cookie='cookie-value' password=\"two words\" authorization: 'header secret'" + ) error = ProviderApiError( ProviderErrorCategory.NETWORK_ERROR, detail, @@ -48,10 +52,17 @@ def test_provider_error_detail_redacts_common_credentials(self) -> None: ) self.assertNotIn("abc123", str(error)) - self.assertNotIn("secret", str(error)) self.assertNotIn("refresh-value", str(error)) + self.assertNotIn("json-secret", str(error)) + self.assertNotIn("client-value", str(error)) + self.assertNotIn("cookie-value", str(error)) + self.assertNotIn("two words", str(error)) + self.assertNotIn("header secret", str(error)) + self.assertIn('access_token":"[REDACTED]"', str(error)) + self.assertIn('client_secret":"[REDACTED]"', str(error)) + self.assertIn("cookie='[REDACTED]'", str(error)) + self.assertEqual(error.provider_code, "authorization=[REDACTED]") self.assertNotIn("header-secret", error.provider_code) - self.assertIn("[REDACTED]", str(error)) def setUp(self) -> None: self.connections = ProviderConnectionRepository() diff --git a/tests/test_provider_import.py b/tests/test_provider_import.py index 493f087..7811d5e 100644 --- a/tests/test_provider_import.py +++ b/tests/test_provider_import.py @@ -1,5 +1,6 @@ from __future__ import annotations +from datetime import date import unittest from symphonia.domain import EntryClassification @@ -14,6 +15,9 @@ ProviderAlreadyRegistered, ProviderNotRegistered, ProviderRegistry, + AppleMusicAdapter, + SpotifyAdapter, + YouTubeDataAdapter, collect_playlist_pages, to_playlist_snapshot, ) @@ -61,9 +65,80 @@ def test_external_identity_is_namespaced_by_type_and_connection(self) -> None: self.assertNotEqual(same_upstream_id.external_key, another_connection.external_key) self.assertNotEqual(same_upstream_id.external_key, another_type.external_key) + def test_normalized_provider_values_reject_non_textual_identity_fields(self) -> None: + with self.assertRaises(ValueError): + ProviderObjectRef(None, "track", "track-1", "connection-1") # type: ignore[arg-type] + with self.assertRaises(ValueError): + ProviderPlaylistEntry( + occurrence_id=None, # type: ignore[arg-type] + position=0, + track=ProviderObjectRef("spotify", "track", "track-1", "connection-1"), + media_kind=MediaKind.TRACK, + ) + with self.assertRaises(ValueError): + ProviderPlaylistPage( + playlist_ref(), + (), + None, + None, + True, + revision=42, # type: ignore[arg-type] + ) + + def test_normalized_provider_values_reject_wrong_runtime_types(self) -> None: + with self.assertRaises(ValueError): + ProviderManifest("spotify", "Spotify", "official", "beta", "limited") # type: ignore[arg-type] + with self.assertRaises(ValueError): + ProviderPlaylistEntry( + occurrence_id="occ-1", + position=True, # type: ignore[arg-type] + track=ProviderObjectRef("spotify", "track", "track-1", "connection-1"), + media_kind=MediaKind.TRACK, + ) + with self.assertRaises(ValueError): + ProviderPlaylistEntry( + occurrence_id="occ-1", + position=0, + track=None, # type: ignore[arg-type] + media_kind=MediaKind.TRACK, + ) + with self.assertRaises(ValueError): + ProviderPlaylistEntry( + occurrence_id="occ-1", + position=0, + track=ProviderObjectRef("spotify", "track", "track-1", "connection-1"), + media_kind="track", # type: ignore[arg-type] + ) + with self.assertRaises(ValueError): + ProviderPlaylistPage(playlist_ref(), [], None, None, True) # type: ignore[arg-type] + with self.assertRaises(ValueError): + ProviderPlaylistPage( + playlist_ref(), + (entry("occ-1", 0, "track-1"), entry("occ-2", 0, "track-2")), + None, + None, + True, + ) + def test_manifest_discloses_access_basis(self) -> None: manifest = ProviderManifest("spotify", "Spotify", AccessBasis.OFFICIAL, "beta", "limited") self.assertEqual(manifest.access_basis, AccessBasis.OFFICIAL) + with self.assertRaises(ValueError): + ProviderManifest( + "spotify", + "Spotify", + AccessBasis.OFFICIAL, + "beta", + "limited", + reviewed_on="2026-9-2", + ) + + def test_concrete_adapter_manifests_have_dated_provenance(self) -> None: + for adapter in (SpotifyAdapter, YouTubeDataAdapter, AppleMusicAdapter): + manifest = adapter.manifest + self.assertTrue(manifest.upstream_dependencies) + self.assertIsNotNone(manifest.reviewed_on) + date.fromisoformat(manifest.reviewed_on or "") def test_complete_pages_preserve_order_and_duplicate_occurrences(self) -> None: result = collect_playlist_pages( diff --git a/tests/test_runtime_http.py b/tests/test_runtime_http.py index 278b9e2..030397a 100644 --- a/tests/test_runtime_http.py +++ b/tests/test_runtime_http.py @@ -1,10 +1,14 @@ from __future__ import annotations from io import BytesIO +import http.client +from pathlib import Path +import tempfile +import threading import unittest from symphonia.infrastructure import OperationRepository -from symphonia.runtime.http import SymphoniaRequestHandler, route_get +from symphonia.runtime.http import SymphoniaRequestHandler, create_server, route_get class RuntimeHTTPTests(unittest.TestCase): @@ -84,6 +88,65 @@ def end_headers(self) -> None: self.assertEqual(handler.headers["X-Content-Type-Options"], "nosniff") self.assertEqual(handler.headers["Referrer-Policy"], "no-referrer") + def test_json_surface_rejects_non_standard_numbers(self) -> None: + class FakeHandler: + def __init__(self) -> None: + self.wfile = BytesIO() + + def send_response(self, status: int) -> None: + return + + def send_header(self, name: str, value: str) -> None: + return + + def end_headers(self) -> None: + return + + with self.assertRaises(ValueError): + SymphoniaRequestHandler._json( # type: ignore[arg-type] + FakeHandler(), + 200, + {"value": float("nan")}, + ) + + def test_composed_runtime_http_smoke_exposes_health_and_readiness(self) -> None: + with tempfile.TemporaryDirectory() as directory: + try: + server = create_server( + host="127.0.0.1", + port=0, + database_path=str(Path(directory) / "symphonia.sqlite3"), + ingress_path="/symphonia", + ) + except PermissionError: + self.skipTest("the test sandbox does not permit local socket binding") + + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + host, port = server.server_address + connection = http.client.HTTPConnection(host, port, timeout=2) + try: + connection.request("GET", "/symphonia/health") + health = connection.getresponse() + health_body = health.read().decode("utf-8") + connection.request("GET", "/symphonia/ready") + ready = connection.getresponse() + ready_body = ready.read().decode("utf-8") + finally: + connection.close() + + self.assertEqual(health.status, 200) + self.assertIn('"status": "ok"', health_body) + self.assertEqual(ready.status, 200) + self.assertIn('"status": "ready"', ready_body) + finally: + server.shutdown() + server.server_close() + server.close_resources() + thread.join(timeout=2) + self.assertFalse(thread.is_alive()) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_runtime_resources.py b/tests/test_runtime_resources.py index f97b4d9..0a0019e 100644 --- a/tests/test_runtime_resources.py +++ b/tests/test_runtime_resources.py @@ -5,8 +5,15 @@ import unittest from datetime import datetime, timezone import sqlite3 +from unittest.mock import patch from symphonia.infrastructure import OperationRepository +from symphonia.providers import ( + Capability, + ConnectionState, + ProviderCapabilities, + ProviderConnection, +) from symphonia.runtime import RuntimeResources @@ -101,6 +108,24 @@ def test_readiness_fails_closed_when_one_store_is_closed(self) -> None: resources.connections.close() resources.operations.close() + def test_readiness_fails_closed_when_operation_state_is_corrupt(self) -> None: + with tempfile.TemporaryDirectory() as directory: + resources = RuntimeResources.open(str(Path(directory) / "symphonia.sqlite3")) + try: + operation = resources.operations.create( + operation_type="test", + idempotency_key="corrupt-runtime-state", + payload={}, + now=datetime(2026, 9, 20, tzinfo=timezone.utc), + ) + resources.operations._connection.execute( # type: ignore[attr-defined] + "UPDATE operations SET state = ? WHERE operation_id = ?", + ("corrupt", operation.operation_id), + ) + self.assertFalse(resources.healthcheck()) + finally: + resources.close() + def test_backup_to_copies_a_consistent_database(self) -> None: with tempfile.TemporaryDirectory() as directory: source_path = str(Path(directory) / "symphonia.sqlite3") @@ -113,6 +138,31 @@ def test_backup_to_copies_a_consistent_database(self) -> None: payload={"value": "persisted"}, now=datetime(2026, 9, 20, tzinfo=timezone.utc), ) + resources.connections.create( + ProviderConnection( + connection_id="spotify-1", + provider="spotify", + provider_account_id="account-1", + state=ConnectionState.CONNECTED, + manifest_version="spotify-2026-09", + secret_ref="opaque-secret-ref", + capabilities=ProviderCapabilities( + enabled=frozenset({Capability.READ_PLAYLISTS}), + evidence_version="probe-1", + observed_at="2026-09-20T12:00:00Z", + ), + created_at=datetime(2026, 9, 20, tzinfo=timezone.utc), + updated_at=datetime(2026, 9, 20, tzinfo=timezone.utc), + ) + ) + attempt = resources.authorization.create( + attempt_id="attempt-1", + provider="spotify", + actor_id="ha-user-1", + redirect_uri="https://ha.example/symphonia/callback", + raw_state="callback-state", + now=datetime(2026, 9, 20, tzinfo=timezone.utc), + ) resources.backup_to(backup_path) finally: resources.close() @@ -127,6 +177,16 @@ def test_backup_to_copies_a_consistent_database(self) -> None: finally: backup.close() + restored_resources = RuntimeResources.open(backup_path) + try: + self.assertEqual(restored_resources.connections.get("spotify-1").secret_ref, "opaque-secret-ref") + self.assertEqual( + restored_resources.authorization.get_by_state("callback-state").attempt_id, + attempt.attempt_id, + ) + finally: + restored_resources.close() + def test_backup_to_rejects_the_live_database(self) -> None: with tempfile.TemporaryDirectory() as directory: source_path = str(Path(directory) / "symphonia.sqlite3") @@ -154,6 +214,21 @@ def test_backup_to_rejects_in_memory_runtime_and_destination(self) -> None: finally: resources.close() + def test_backup_to_rejects_invalid_destination_shape_before_writing(self) -> None: + with tempfile.TemporaryDirectory() as directory: + source_path = str(Path(directory) / "symphonia.sqlite3") + resources = RuntimeResources.open(source_path) + try: + with self.assertRaises(ValueError): + resources.backup_to(str(Path(directory) / "missing" / "backup.sqlite3")) + with self.assertRaises(ValueError): + resources.backup_to(directory) + finally: + resources.close() + + with self.assertRaises(ValueError): + RuntimeResources.open(None) # type: ignore[arg-type] + def test_backup_to_replaces_an_existing_destination_atomically(self) -> None: with tempfile.TemporaryDirectory() as directory: source_path = str(Path(directory) / "symphonia.sqlite3") @@ -173,6 +248,26 @@ def test_backup_to_replaces_an_existing_destination_atomically(self) -> None: self.assertTrue(RuntimeResources.validate_backup(str(backup_path))) + def test_backup_to_does_not_publish_an_invalid_temporary_copy(self) -> None: + with tempfile.TemporaryDirectory() as directory: + source_path = str(Path(directory) / "symphonia.sqlite3") + backup_path = Path(directory) / "backup.sqlite3" + backup_path.write_text("previous backup", encoding="utf-8") + resources = RuntimeResources.open(source_path) + try: + resources.operations.create( + operation_type="test", + idempotency_key="invalid-backup", + payload={}, + now=datetime(2026, 9, 20, tzinfo=timezone.utc), + ) + with patch.object(RuntimeResources, "validate_backup", return_value=False): + with self.assertRaisesRegex(RuntimeError, "integrity"): + resources.backup_to(str(backup_path)) + self.assertEqual(backup_path.read_text(encoding="utf-8"), "previous backup") + finally: + resources.close() + def test_backup_to_fails_closed_when_a_store_is_unhealthy(self) -> None: with tempfile.TemporaryDirectory() as directory: source_path = str(Path(directory) / "symphonia.sqlite3") diff --git a/tests/test_spec_validator.py b/tests/test_spec_validator.py new file mode 100644 index 0000000..c313d1f --- /dev/null +++ b/tests/test_spec_validator.py @@ -0,0 +1,14 @@ +from pathlib import Path +import unittest + +from tools.validate_specs import validate + + +class SpecValidatorTests(unittest.TestCase): + def test_repository_specifications_are_consistent(self) -> None: + root = Path(__file__).resolve().parents[1] + self.assertEqual(validate(root), []) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_spotify_adapter.py b/tests/test_spotify_adapter.py index 59d4837..d4de8b3 100644 --- a/tests/test_spotify_adapter.py +++ b/tests/test_spotify_adapter.py @@ -116,14 +116,94 @@ def test_max_page_limit_fails_closed_before_unbounded_reads(self) -> None: self.assertEqual(context.exception.category, ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED) self.assertEqual(len(client.calls), 1) + def test_repeated_offset_fails_closed_before_unbounded_reads(self) -> None: + class LoopingClient: + def __init__(self) -> None: + self.calls = [] + + def request(self, method: str, path: str, *, token: str, query: dict[str, str], body=None) -> JsonResponse: + self.calls.append((method, path, token, query)) + return JsonResponse( + 200, + { + "items": [{"item": {"id": "track-1", "type": "track"}}], + "next": "https://api.spotify.com/v1/playlists/playlist-1/items?offset=0", + }, + {}, + ) + + with self.assertRaises(ProviderApiError) as context: + SpotifyAdapter(LoopingClient(), lambda connection_id: "access-token", page_size=1).read_playlist_pages( + "connection-1", self.playlist() + ) + + self.assertEqual(context.exception.category, ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED) + def test_invalid_cursor_and_empty_token_fail_closed(self) -> None: client = FakeClient({"0": JsonResponse(200, {"items": [], "next": None}, {})}) adapter = SpotifyAdapter(client, lambda connection_id: "") with self.assertRaises(ProviderApiError): adapter.read_playlist_pages("connection-1", self.playlist()) + adapter = SpotifyAdapter(client, lambda connection_id: None) # type: ignore[arg-type] + with self.assertRaises(ProviderApiError) as context: + adapter.read_playlist_pages("connection-1", self.playlist()) + self.assertEqual(context.exception.category, ProviderErrorCategory.AUTHENTICATION_REQUIRED) adapter = SpotifyAdapter(client, lambda connection_id: "token") with self.assertRaises(ValueError): adapter.read_playlist_pages("connection-1", self.playlist(), cursor="not-an-offset") + with self.assertRaises(ValueError): + adapter.read_playlist_pages("connection-1", self.playlist(), cursor=True) # type: ignore[arg-type] + + def test_malformed_next_links_fail_closed_without_fallback_pagination(self) -> None: + for next_url in ( + "not-a-url", + "https://api.spotify.com/v1/playlists/playlist-1/items?limit=1", + "https://api.spotify.com/v1/playlists/playlist-1/items?offset=1&offset=2", + "https://api.spotify.com/v1/playlists/playlist-1/items?offset=0", + "https://api.spotify.com/v1/playlists/playlist-1/items?offset=not-an-integer", + ): + client = FakeClient( + { + "0": JsonResponse( + 200, + {"items": [{"item": {"id": "track-1", "type": "track"}}], "next": next_url}, + {}, + ) + } + ) + with self.subTest(next_url=next_url), self.assertRaises(ProviderApiError) as context: + SpotifyAdapter(client, lambda connection_id: "token").read_playlist_pages( + "connection-1", self.playlist(), + ) + self.assertEqual(context.exception.category, ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED) + + def test_target_visibility_must_be_explicit(self) -> None: + adapter = SpotifyAdapter( + FakeClient({"capabilities": JsonResponse(201, {"id": "target-1"}, {})}), + lambda connection_id: "access-token", + connection_id="connection-1", + ) + with self.assertRaises(ValueError): + adapter.ensure_target_playlist( + provider="spotify", + name="Imported", + visibility="unlisted", + idempotency_key="target-1", + ) + + for kwargs in ( + {"name": "", "visibility": "private", "idempotency_key": "target-1"}, + {"name": "Imported", "visibility": "private", "idempotency_key": ""}, + ): + with self.subTest(kwargs=kwargs), self.assertRaises(ValueError): + adapter.ensure_target_playlist(provider="spotify", **kwargs) + + with self.assertRaises(ValueError): + adapter.add_entry( + target_playlist_id="", + provider_track_id="track-1", + idempotency_key="entry-1", + ) def test_confirmed_writes_use_spotify_json_contract(self) -> None: client = FakeClient({"capabilities": JsonResponse(201, {"id": "target-1"}, {})}) diff --git a/tests/test_sqlite_authorization.py b/tests/test_sqlite_authorization.py index 4560d36..b35ba1b 100644 --- a/tests/test_sqlite_authorization.py +++ b/tests/test_sqlite_authorization.py @@ -1,6 +1,9 @@ from __future__ import annotations from datetime import datetime, timedelta, timezone +from pathlib import Path +import tempfile +import threading import unittest from symphonia.infrastructure import ( @@ -43,6 +46,51 @@ def test_persists_only_state_digest_and_exact_callback_binding(self) -> None: self.assertNotIn("raw-state-secret", str(raw)) self.assertEqual(raw["redirect_uri"], "https://ha.example/symphonia/callback") + def test_healthcheck_fails_closed_on_corrupt_attempt_state(self) -> None: + self.create() + self.repository._connection.execute( # type: ignore[attr-defined] + "UPDATE authorization_attempts SET state = ? WHERE attempt_id = ?", + ("corrupt", "attempt-1"), + ) + + self.assertFalse(self.repository.healthcheck()) + + def test_callback_state_resolves_by_digest_after_repository_restart(self) -> None: + with tempfile.TemporaryDirectory() as directory: + path = str(Path(directory) / "authorization.sqlite3") + repository = AuthorizationAttemptRepository(path) + repository.create( + attempt_id="attempt-1", + provider="spotify", + actor_id="ha-user-1", + redirect_uri="https://ha.example/symphonia/callback", + raw_state="callback-state", + now=NOW, + ) + repository.close() + + restarted = AuthorizationAttemptRepository(path) + try: + resolved = restarted.get_by_state("callback-state") + self.assertEqual(resolved.attempt_id, "attempt-1") + self.assertNotEqual(resolved.state_digest, "callback-state") + finally: + restarted.close() + + def test_ambiguous_callback_state_fails_closed(self) -> None: + self.create() + self.repository.create( + attempt_id="attempt-2", + provider="spotify", + actor_id="ha-user-1", + redirect_uri="https://ha.example/symphonia/callback", + raw_state="raw-state-secret", + now=NOW, + ) + + with self.assertRaises(AuthorizationAttemptError): + self.repository.get_by_state("raw-state-secret") + def test_matching_state_is_single_use(self) -> None: self.create() consumed = self.repository.consume("attempt-1", raw_state="raw-state-secret", now=NOW + timedelta(seconds=1)) @@ -50,6 +98,58 @@ def test_matching_state_is_single_use(self) -> None: with self.assertRaises(AuthorizationAttemptError): self.repository.consume("attempt-1", raw_state="raw-state-secret", now=NOW + timedelta(seconds=2)) + def test_two_sqlite_connections_can_consume_a_state_only_once(self) -> None: + with tempfile.TemporaryDirectory() as directory: + path = str(Path(directory) / "authorization.sqlite3") + seed = AuthorizationAttemptRepository(path) + seed.create( + attempt_id="attempt-1", + provider="spotify", + actor_id="ha-user-1", + redirect_uri="https://ha.example/symphonia/callback", + raw_state="raw-state-secret", + now=NOW, + ) + seed.close() + + start = threading.Barrier(2) + outcomes: list[str] = [] + outcomes_lock = threading.Lock() + + def consume() -> None: + repository = AuthorizationAttemptRepository(path) + try: + start.wait(timeout=5) + repository.consume( + "attempt-1", + raw_state="raw-state-secret", + now=NOW + timedelta(seconds=1), + ) + outcome = "consumed" + except AuthorizationAttemptError: + outcome = "rejected" + except Exception as error: # pragma: no cover - keeps thread failures observable + outcome = f"error:{type(error).__name__}" + finally: + repository.close() + with outcomes_lock: + outcomes.append(outcome) + + threads = [threading.Thread(target=consume) for _ in range(2)] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=10) + + self.assertTrue(all(not thread.is_alive() for thread in threads)) + self.assertEqual(sorted(outcomes), ["consumed", "rejected"]) + + verifier = AuthorizationAttemptRepository(path) + try: + self.assertEqual(verifier.get("attempt-1").state, AuthorizationState.CONSUMED) + finally: + verifier.close() + def test_mismatch_invalidates_attempt_without_disclosing_expected_state(self) -> None: self.create() with self.assertRaises(AuthorizationAttemptError): @@ -71,6 +171,40 @@ def test_denial_is_terminal(self) -> None: with self.assertRaises(AuthorizationAttemptError): self.repository.expire("attempt-1", now=NOW + timedelta(seconds=2)) + def test_invalid_creation_values_are_rejected_before_persistence(self) -> None: + for field, value in ( + ("attempt_id", " "), + ("provider", " "), + ("actor_id", " "), + ("raw_state", "s" * 513), + ): + values = { + "attempt_id": "attempt-1", + "provider": "spotify", + "actor_id": "ha-user-1", + "redirect_uri": "https://ha.example/symphonia/callback", + "raw_state": "raw-state-secret", + "now": NOW, + } + values[field] = value + with self.subTest(field=field), self.assertRaises(ValueError): + self.repository.create(**values) + count = self.repository._connection.execute( + "SELECT COUNT(*) FROM authorization_attempts" + ).fetchone()[0] + self.assertEqual(count, 0) + + def test_failure_code_is_bounded_and_safe_for_durable_storage(self) -> None: + self.create() + for failure_code in ("provider detail", "TOKEN=secret", "x" * 65, ""): + with self.subTest(failure_code=failure_code), self.assertRaises(ValueError): + self.repository.deny( + "attempt-1", + now=NOW + timedelta(seconds=1), + failure_code=failure_code, + ) + self.assertEqual(self.repository.get("attempt-1").state, AuthorizationState.CREATED) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_sqlite_connections.py b/tests/test_sqlite_connections.py index b719631..e29bbbf 100644 --- a/tests/test_sqlite_connections.py +++ b/tests/test_sqlite_connections.py @@ -52,6 +52,17 @@ def test_create_round_trips_capabilities_and_only_persists_secret_reference(self self.assertEqual(raw["secret_ref"], "secret-ref-1") self.assertNotIn("access-token", str(raw)) + def test_capability_json_rejects_non_standard_numbers_on_read(self) -> None: + self.repository.create(connection()) + self.repository._connection.execute( # type: ignore[attr-defined] + "UPDATE provider_connections SET capabilities_json = ?", + ('{"enabled": [], "evidence_version": NaN, "observed_at": "probe"}',), + ) + + self.assertFalse(self.repository.healthcheck()) + with self.assertRaises(ValueError): + self.repository.get("spotify-1") + def test_same_connection_is_idempotent_but_account_collision_is_rejected(self) -> None: first = connection() self.repository.create(first) diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index c04a0b8..529efae 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -3,6 +3,7 @@ from datetime import datetime, timedelta, timezone import sqlite3 import tempfile +import threading import unittest from symphonia.infrastructure import IdempotencyConflict, LeaseConflict, OperationRepository @@ -80,6 +81,117 @@ def test_lease_claim_checkpoint_and_terminal_state(self) -> None: self.assertEqual([event.state for event in events], ["queued", "running", "succeeded"]) self.assertEqual(events[1].worker_id, "worker-a") + def test_two_sqlite_connections_cannot_claim_same_operation(self) -> None: + """Exercise the BEGIN IMMEDIATE lease boundary with real connections.""" + + with tempfile.TemporaryDirectory() as temporary_directory: + database_path = f"{temporary_directory}/operations.sqlite3" + seed_repository = OperationRepository(database_path) + operation = seed_repository.create( + operation_type="copy", + idempotency_key="concurrent-claim", + payload={}, + now=self.now, + ) + seed_repository.close() + + start = threading.Barrier(2) + outcomes: list[tuple[str, str]] = [] + outcomes_lock = threading.Lock() + + def attempt_claim(worker_id: str) -> None: + repository = OperationRepository(database_path) + try: + start.wait(timeout=5) + repository.claim(operation.operation_id, worker_id=worker_id, now=self.now) + outcome = (worker_id, "claimed") + except LeaseConflict: + outcome = (worker_id, "conflict") + except Exception as error: # pragma: no cover - keeps thread failures observable + outcome = (worker_id, f"error:{type(error).__name__}") + finally: + repository.close() + with outcomes_lock: + outcomes.append(outcome) + + threads = [ + threading.Thread(target=attempt_claim, args=(worker_id,)) + for worker_id in ("worker-a", "worker-b") + ] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=10) + + self.assertTrue(all(not thread.is_alive() for thread in threads)) + statuses = {worker_id: status for worker_id, status in outcomes} + self.assertEqual(set(statuses), {"worker-a", "worker-b"}) + self.assertEqual(sorted(statuses.values()), ["claimed", "conflict"]) + + verifier = OperationRepository(database_path) + try: + record = verifier.get(operation.operation_id) + self.assertEqual(record.state, "running") + self.assertIn(record.worker_id, {"worker-a", "worker-b"}) + self.assertEqual( + [event.event_type for event in verifier.events(operation.operation_id)], + ["created", "claimed"], + ) + finally: + verifier.close() + + def test_two_sqlite_connections_claim_distinct_queue_items(self) -> None: + """Exercise scheduler selection under concurrent workers.""" + + with tempfile.TemporaryDirectory() as temporary_directory: + database_path = f"{temporary_directory}/operations.sqlite3" + seed_repository = OperationRepository(database_path) + first = seed_repository.create( + operation_type="copy", + idempotency_key="concurrent-next-1", + payload={}, + now=self.now, + ) + second = seed_repository.create( + operation_type="copy", + idempotency_key="concurrent-next-2", + payload={}, + now=self.now + timedelta(seconds=1), + ) + seed_repository.close() + + start = threading.Barrier(2) + outcomes: list[tuple[str, str | None]] = [] + outcomes_lock = threading.Lock() + + def claim_next(worker_id: str) -> None: + repository = OperationRepository(database_path) + try: + start.wait(timeout=5) + claimed = repository.claim_next(worker_id=worker_id, now=self.now) + outcome = (worker_id, None if claimed is None else claimed.operation_id) + except Exception as error: # pragma: no cover - keeps thread failures observable + outcome = (worker_id, f"error:{type(error).__name__}") + finally: + repository.close() + with outcomes_lock: + outcomes.append(outcome) + + threads = [ + threading.Thread(target=claim_next, args=(worker_id,)) + for worker_id in ("worker-a", "worker-b") + ] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=10) + + self.assertTrue(all(not thread.is_alive() for thread in threads)) + self.assertEqual({worker_id for worker_id, _ in outcomes}, {"worker-a", "worker-b"}) + claimed_ids = [operation_id for _, operation_id in outcomes] + self.assertEqual(set(claimed_ids), {first.operation_id, second.operation_id}) + self.assertEqual(len(claimed_ids), len(set(claimed_ids))) + def test_checkpoint_events_store_only_sanitized_summary(self) -> None: operation = self.repository.create( operation_type="copy", @@ -373,6 +485,92 @@ def test_operation_payload_and_checkpoint_must_be_json_objects(self) -> None: ) self.assertEqual(self.repository.get(operation.operation_id).state, "running") + def test_non_finite_numbers_are_rejected_before_durable_json_writes(self) -> None: + with self.assertRaises(ValueError): + self.repository.create( + operation_type="copy", + idempotency_key="nan-payload", + payload={"value": float("nan")}, + now=self.now, + ) + + operation = self.repository.create( + operation_type="copy", + idempotency_key="nan-checkpoint", + payload={}, + now=self.now, + ) + self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) + with self.assertRaises(ValueError): + self.repository.checkpoint( + operation.operation_id, + worker_id="worker-a", + checkpoint={"value": float("inf")}, + now=self.now, + ) + self.assertEqual(self.repository.get(operation.operation_id).checkpoint, {}) + + self.repository._connection.execute( + "UPDATE operations SET payload_json = ? WHERE operation_id = ?", + ('{"value": NaN}', operation.operation_id), + ) + with self.assertRaises(ValueError): + self.repository.get(operation.operation_id) + + def test_healthcheck_fails_closed_on_invalid_state_or_json(self) -> None: + operation = self.repository.create( + operation_type="copy", + idempotency_key="healthcheck-corruption", + payload={}, + now=self.now, + ) + + self.repository._connection.execute( # type: ignore[attr-defined] + "UPDATE operations SET state = ? WHERE operation_id = ?", + ("not-a-real-state", operation.operation_id), + ) + self.assertFalse(self.repository.healthcheck()) + + self.repository._connection.execute( # type: ignore[attr-defined] + "UPDATE operations SET state = ?, payload_json = ? WHERE operation_id = ?", + ("queued", "[]", operation.operation_id), + ) + self.assertFalse(self.repository.healthcheck()) + + self.repository._connection.execute( # type: ignore[attr-defined] + "UPDATE operations SET payload_json = ? WHERE operation_id = ?", + ('{"access_token":"must-not-be-readable"}', operation.operation_id), + ) + self.assertFalse(self.repository.healthcheck()) + + def test_operation_and_worker_identifiers_require_non_empty_text(self) -> None: + for field, value in ( + ("operation_type", None), + ("idempotency_key", " "), + ("operation_id", ""), + ): + values = { + "operation_type": "copy", + "idempotency_key": "identifier-test", + "payload": {}, + "now": self.now, + } + if field == "operation_id": + values[field] = value + else: + values[field] = value + with self.subTest(field=field), self.assertRaises(ValueError): + self.repository.create(**values) + + operation = self.repository.create( + operation_type="copy", + idempotency_key="worker-identifier-test", + payload={}, + now=self.now, + ) + with self.assertRaises(ValueError): + self.repository.claim(operation.operation_id, worker_id=None, now=self.now) # type: ignore[arg-type] + def test_operation_payload_rejects_non_string_object_keys(self) -> None: with self.assertRaisesRegex(ValueError, "keys must be strings"): self.repository.create( diff --git a/tests/test_sqlite_plans.py b/tests/test_sqlite_plans.py index 47f403c..2bbc12c 100644 --- a/tests/test_sqlite_plans.py +++ b/tests/test_sqlite_plans.py @@ -38,6 +38,36 @@ def test_save_and_reload_preserves_immutable_plan_and_digest(self) -> None: self.assertEqual(stored.plan, plan) self.assertIsNone(stored.accepted_at) + def test_plan_json_rejects_non_standard_numbers_on_read(self) -> None: + plan = self.ready_plan() + self.repository.save(plan, now=self.now) + self.repository._connection.execute( # type: ignore[attr-defined] + "UPDATE copy_plans SET plan_json = ? WHERE digest = ?", + ('{"source_snapshot_id": NaN}', plan.digest), + ) + + self.assertFalse(self.repository.healthcheck()) + with self.assertRaises(ValueError): + self.repository.get(plan.digest) + + def test_plan_json_rejects_content_tampering_with_an_unchanged_digest(self) -> None: + plan = self.ready_plan() + self.repository.save(plan, now=self.now) + raw = self.repository._connection.execute( # type: ignore[attr-defined] + "SELECT plan_json FROM copy_plans WHERE digest = ?", + (plan.digest,), + ).fetchone()["plan_json"] + tampered = raw.replace('"target_playlist_name":"Rock"', '"target_playlist_name":"Tampered"') + self.assertNotEqual(raw, tampered) + self.repository._connection.execute( # type: ignore[attr-defined] + "UPDATE copy_plans SET plan_json = ? WHERE digest = ?", + (tampered, plan.digest), + ) + + self.assertFalse(self.repository.healthcheck()) + with self.assertRaises(ValueError): + self.repository.get(plan.digest) + def test_acceptance_is_durable_and_idempotent(self) -> None: plan = self.ready_plan() self.repository.save(plan, now=self.now) @@ -70,4 +100,3 @@ def test_blocked_plan_cannot_be_accepted_through_repository(self) -> None: if __name__ == "__main__": unittest.main() - diff --git a/tests/test_storage_recovery_spike.py b/tests/test_storage_recovery_spike.py new file mode 100644 index 0000000..3fa375a --- /dev/null +++ b/tests/test_storage_recovery_spike.py @@ -0,0 +1,29 @@ +from __future__ import annotations + +import unittest + +from tools.storage_recovery_spike import run_spike + + +class StorageRecoverySpikeTests(unittest.TestCase): + def test_spike_recovers_expired_lease_and_validates_backup(self) -> None: + evidence = run_spike(operation_count=8) + + self.assertEqual(evidence["operation_count"], 8) + self.assertEqual(evidence["recovered_operation_id"], "spike-operation-0") + self.assertEqual(evidence["recovered_worker_id"], "worker-after-restart") + self.assertEqual(evidence["recovered_state"], "succeeded") + self.assertEqual(evidence["audit_event_count"], 4) + self.assertTrue(evidence["backup_valid"]) + self.assertGreater(evidence["source_bytes"], 0) + self.assertEqual(evidence["source_bytes"], evidence["backup_bytes"]) + + def test_spike_bounds_disposable_fixture_size(self) -> None: + with self.assertRaises(ValueError): + run_spike(operation_count=0) + with self.assertRaises(ValueError): + run_spike(operation_count=10_001) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_youtube_adapter.py b/tests/test_youtube_adapter.py index c984883..626a8d0 100644 --- a/tests/test_youtube_adapter.py +++ b/tests/test_youtube_adapter.py @@ -105,6 +105,29 @@ def request(self, method: str, path: str, *, token: str, query: dict[str, str], ) self.assertEqual(context.exception.category, ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED) + def test_non_textual_page_token_is_a_provider_contract_failure(self) -> None: + class MalformedClient(FakeClient): + def request(self, method: str, path: str, *, token: str, query: dict[str, str], body=None) -> JsonResponse: + self.calls.append((method, path, token, query)) + if path == "/channels": + return JsonResponse(200, {"items": [{"id": "channel-1"}]}, {}) + return JsonResponse(200, {"items": [], "nextPageToken": {"unexpected": "object"}}, {}) + + playlist = ProviderObjectRef("youtube_data", "playlist", "playlist-1", "connection-1") + with self.assertRaises(ProviderApiError) as context: + YouTubeDataAdapter(MalformedClient(), lambda connection_id: "access-token").read_playlist_pages( + "connection-1", playlist + ) + self.assertEqual(context.exception.category, ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED) + + def test_non_textual_access_token_fails_as_authentication_required(self) -> None: + playlist = ProviderObjectRef("youtube_data", "playlist", "playlist-1", "connection-1") + with self.assertRaises(ProviderApiError) as context: + YouTubeDataAdapter(FakeClient(), lambda connection_id: None).read_playlist_pages( # type: ignore[arg-type] + "connection-1", playlist + ) + self.assertEqual(context.exception.category, ProviderErrorCategory.AUTHENTICATION_REQUIRED) + def test_max_page_limit_fails_closed_before_unbounded_reads(self) -> None: playlist = ProviderObjectRef("youtube_data", "playlist", "playlist-1", "connection-1") adapter = YouTubeDataAdapter(FakeClient(), lambda connection_id: "access-token", max_pages=1) @@ -114,6 +137,13 @@ def test_max_page_limit_fails_closed_before_unbounded_reads(self) -> None: self.assertEqual(context.exception.category, ProviderErrorCategory.PROVIDER_CONTRACT_CHANGED) + def test_non_textual_cursor_is_rejected_before_provider_request(self) -> None: + playlist = ProviderObjectRef("youtube_data", "playlist", "playlist-1", "connection-1") + with self.assertRaises(ValueError): + YouTubeDataAdapter(FakeClient(), lambda connection_id: "access-token").read_playlist_pages( + "connection-1", playlist, cursor={"unexpected": "object"} # type: ignore[arg-type] + ) + if __name__ == "__main__": unittest.main() diff --git a/tools/__init__.py b/tools/__init__.py new file mode 100644 index 0000000..34b4078 --- /dev/null +++ b/tools/__init__.py @@ -0,0 +1 @@ +"""Repository maintenance tools for Symphonia.""" diff --git a/tools/oauth_callback_spike.py b/tools/oauth_callback_spike.py new file mode 100644 index 0000000..819500d --- /dev/null +++ b/tools/oauth_callback_spike.py @@ -0,0 +1,174 @@ +"""Callback-only OAuth boundary spike for the direct App flow. + +This module deliberately lives under ``tools`` until the authorization SDD is +ready for production implementation. It proves route isolation, durable +state consumption, provider binding, and secret-safe callback responses using +only the standard library. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from datetime import datetime +from http.server import BaseHTTPRequestHandler, HTTPServer +from collections.abc import Mapping +from typing import Any +from urllib.parse import parse_qs, urlsplit + +from symphonia.application.authorization import AuthorizationService +from symphonia.infrastructure.sqlite_authorization import ( + AuthorizationAttemptError, + AuthorizationAttemptNotFound, +) +from symphonia.providers.authorization import AuthorizationAttempt + + +@dataclass(frozen=True, slots=True) +class OAuthCallbackResult: + """Safe HTTP result plus transient handoff data for the exchange owner.""" + + status: int + message: str + attempt: AuthorizationAttempt | None = None + authorization_code: str | None = field(default=None, repr=False) + + +def handle_callback( + request_path: str, + *, + provider: str, + callback_path: str, + authorization: AuthorizationService, + now: datetime, +) -> OAuthCallbackResult: + """Handle only one provider callback path without echoing query values.""" + + if not provider.strip(): + raise ValueError("provider must not be blank") + callback_path = _normalize_callback_path(callback_path) + parsed = urlsplit(request_path) + if parsed.path != callback_path or parsed.fragment: + return OAuthCallbackResult(404, "Not found.") + + query = parse_qs(parsed.query, keep_blank_values=True) + try: + state = _single_query_value(query, "state") + error = _single_query_value(query, "error") + code = _single_query_value(query, "code", max_length=8192) + except ValueError: + return OAuthCallbackResult(400, "The authorization callback is invalid or expired.") + if state is None: + return OAuthCallbackResult(400, "The authorization callback is invalid or expired.") + + if error is not None and code is not None: + return OAuthCallbackResult(400, "The authorization callback is invalid or expired.") + + try: + if error is not None: + attempt = authorization.deny_callback( + raw_state=state, + provider=provider, + now=now, + failure_code="provider_consent_denied", + ) + return OAuthCallbackResult(200, "Authorization was cancelled. You can close this window.", attempt=attempt) + if code is None: + return OAuthCallbackResult(400, "The authorization callback is invalid or expired.") + attempt = authorization.consume_callback_for_provider( + raw_state=state, + provider=provider, + now=now, + ) + except (AuthorizationAttemptError, AuthorizationAttemptNotFound, ValueError): + return OAuthCallbackResult(400, "The authorization callback is invalid or expired.") + + return OAuthCallbackResult( + 200, + "Authorization received. You can close this window.", + attempt=attempt, + authorization_code=code, + ) + + +class CallbackOnlyServer(HTTPServer): + """A spike server exposing exactly one callback route.""" + + def __init__( + self, + address: tuple[str, int], + *, + provider: str, + callback_path: str, + authorization: AuthorizationService, + clock: Any, + ) -> None: + self.provider = provider + self.callback_path = _normalize_callback_path(callback_path) + self.authorization = authorization + self.clock = clock + super().__init__(address, CallbackOnlyRequestHandler) + + +class CallbackOnlyRequestHandler(BaseHTTPRequestHandler): + server: CallbackOnlyServer + + def do_GET(self) -> None: # noqa: N802 - stdlib handler API + result = handle_callback( + self.path, + provider=self.server.provider, + callback_path=self.server.callback_path, + authorization=self.server.authorization, + now=self.server.clock(), + ) + body = result.message.encode("utf-8") + self.send_response(result.status) + self.send_header("Content-Type", "text/plain; charset=utf-8") + self.send_header("Content-Length", str(len(body))) + self.send_header("Cache-Control", "no-store") + self.send_header("X-Content-Type-Options", "nosniff") + self.send_header("Referrer-Policy", "no-referrer") + self.end_headers() + self.wfile.write(body) + + def log_message(self, format: str, *args: object) -> None: + return + + +def create_callback_server( + *, + authorization: AuthorizationService, + provider: str, + callback_path: str = "/oauth/callback", + host: str = "127.0.0.1", + port: int = 0, + clock: Any, +) -> CallbackOnlyServer: + """Create a loopback-only spike server; no accidental public bind.""" + + if host not in {"127.0.0.1", "localhost", "::1"}: + raise ValueError("the callback spike only permits a loopback host") + return CallbackOnlyServer( + (host, port), + provider=provider, + callback_path=callback_path, + authorization=authorization, + clock=clock, + ) + + +def _normalize_callback_path(value: str) -> str: + if not value or not value.startswith("/") or "?" in value or "#" in value: + raise ValueError("callback_path must be an absolute path without query or fragment") + normalized = value.rstrip("/") or "/" + if "//" in normalized or "/../" in normalized or normalized.endswith("/.."): + raise ValueError("callback_path contains an unsafe segment") + return normalized + + +def _single_query_value(query: Mapping[str, list[str]], name: str, *, max_length: int = 512) -> str | None: + values = query.get(name) + if values is None: + return None + if len(values) != 1 or not values[0].strip() or len(values[0]) > max_length: + raise ValueError(f"callback query parameter {name} is invalid") + return values[0] diff --git a/tools/storage_recovery_spike.py b/tools/storage_recovery_spike.py new file mode 100644 index 0000000..bc38225 --- /dev/null +++ b/tools/storage_recovery_spike.py @@ -0,0 +1,100 @@ +"""Reproducible SQLite recovery/backup spike for the runtime foundation. + +This is evidence tooling, not a production benchmark or a restore command. It +creates a disposable persistent store, simulates a worker restart with an +expired lease, and validates an online backup using the runtime's existing +preflight checks. +""" + +from __future__ import annotations + +import argparse +from datetime import datetime, timedelta, timezone +import json +from pathlib import Path +import tempfile +import time +from typing import Any + +from symphonia.runtime import RuntimeResources + + +BASE_TIME = datetime(2026, 9, 22, 12, 0, tzinfo=timezone.utc) + + +def run_spike(*, operation_count: int = 1000) -> dict[str, Any]: + """Run the disposable recovery scenario and return secret-free evidence.""" + + if not 1 <= operation_count <= 10_000: + raise ValueError("operation_count must be between 1 and 10000") + + started = time.perf_counter() + with tempfile.TemporaryDirectory(prefix="symphonia-storage-spike-") as directory: + source_path = Path(directory) / "symphonia.sqlite3" + backup_path = Path(directory) / "backup.sqlite3" + + with RuntimeResources.open(str(source_path)) as resources: + for index in range(operation_count): + resources.operations.create( + operation_type="spike.copy", + idempotency_key=f"spike-key-{index}", + operation_id=f"spike-operation-{index}", + payload={"fixture_index": index}, + now=BASE_TIME, + ) + claimed = resources.operations.claim( + "spike-operation-0", + worker_id="worker-before-restart", + now=BASE_TIME, + lease_seconds=1, + ) + + with RuntimeResources.open(str(source_path)) as resources: + recovered = resources.operations.claim( + claimed.operation_id, + worker_id="worker-after-restart", + now=BASE_TIME + timedelta(seconds=2), + lease_seconds=30, + ) + recovered_worker_id = recovered.worker_id + completed = resources.operations.checkpoint( + recovered.operation_id, + worker_id="worker-after-restart", + checkpoint={"recovered": True}, + now=BASE_TIME + timedelta(seconds=3), + state="succeeded", + ) + audit_event_count = len(resources.operations.events(recovered.operation_id)) + resources.backup_to(str(backup_path)) + + source_bytes = source_path.stat().st_size + backup_bytes = backup_path.stat().st_size + backup_valid = RuntimeResources.validate_backup(str(backup_path)) + elapsed_ms = round((time.perf_counter() - started) * 1000, 2) + return { + "operation_count": operation_count, + "recovered_operation_id": recovered.operation_id, + "recovered_worker_id": recovered_worker_id, + "recovered_state": completed.state, + "audit_event_count": audit_event_count, + "source_bytes": source_bytes, + "backup_bytes": backup_bytes, + "backup_valid": backup_valid, + "elapsed_ms": elapsed_ms, + } + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--operations", + type=int, + default=1000, + help="number of disposable queued operations to create (1-10000)", + ) + args = parser.parse_args() + print(json.dumps(run_spike(operation_count=args.operations), sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/tools/validate_specs.py b/tools/validate_specs.py new file mode 100644 index 0000000..05f9a63 --- /dev/null +++ b/tools/validate_specs.py @@ -0,0 +1,248 @@ +"""Validate the repository-native SDD/catalog consistency rules. + +This intentionally uses only the Python standard library. It checks the +manual rules documented in ``specs/README.md`` so a future implementation can +rely on the same checks locally and in CI without selecting an application +toolchain first. +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +from pathlib import Path +from typing import Any + + +CAPABILITY_ID_RE = re.compile(r"^\s*-\s+Catalog capability ID:\s+`([^`]+)`\s*$", re.MULTILINE) +REQUIREMENT_RE = re.compile(r"\bSYM-[A-Z]+-[0-9]{3}\b") +CATALOG_ROW_RE = re.compile( + r"^\|\s*`([^`]+)`\s*\|\s*([^|]+?)\s*\|\s*\[[^]]+\]\(([^)]+)\)\s*\|", + re.MULTILINE, +) +MARKDOWN_LINK_RE = re.compile(r"(?]+)>|([^\s)]+))(?:\s+[^)]*)?\)") + +CATALOG_STATUSES = { + "draft": "Draft", + "ready-for-review": "Ready for review", + "ready-for-implementation": "Ready for implementation", + "implemented": "Implemented", + "superseded": "Superseded", +} +EXCLUDED_SPEC_MARKDOWN = {"README.md", "CATALOG.md", "_template.md"} + + +def _add(errors: list[str], message: str) -> None: + errors.append(message) + + +def _local_path(root: Path, raw: str) -> Path: + return root / raw + + +def _validate_catalog_shape(root: Path, catalog: Any, errors: list[str]) -> list[dict[str, Any]]: + if not isinstance(catalog, dict): + _add(errors, "catalog.json must contain an object") + return [] + if catalog.get("version") != 1: + _add(errors, "catalog.json version must be 1") + capabilities = catalog.get("capabilities") + if not isinstance(capabilities, list) or not capabilities: + _add(errors, "catalog.json capabilities must be a non-empty array") + return [] + + ids: set[str] = set() + specifications: set[str] = set() + for index, capability in enumerate(capabilities): + prefix = f"catalog capability {index}" + if not isinstance(capability, dict): + _add(errors, f"{prefix} must be an object") + continue + for field in ( + "id", + "title", + "status", + "scope", + "owner", + "specification", + "requirements", + "evidence", + "implementationEvidence", + "blockers", + ): + if field not in capability: + _add(errors, f"{prefix} is missing {field!r}") + capability_id = capability.get("id") + if not isinstance(capability_id, str) or not capability_id: + _add(errors, f"{prefix} has an invalid id") + elif capability_id in ids: + _add(errors, f"duplicate catalog capability id: {capability_id}") + else: + ids.add(capability_id) + status = capability.get("status") + if status not in CATALOG_STATUSES: + _add(errors, f"{prefix} has invalid status: {status!r}") + specification = capability.get("specification") + if not isinstance(specification, str) or not specification: + _add(errors, f"{prefix} has an invalid specification path") + elif specification in specifications: + _add(errors, f"multiple capabilities use primary SDD: {specification}") + else: + specifications.add(specification) + requirements = capability.get("requirements") + if not isinstance(requirements, list) or len(requirements) != len(set(requirements)): + _add(errors, f"{prefix} requirements must be a unique array") + + for field in ("evidence", "blockers"): + if not isinstance(capability.get(field), list): + _add(errors, f"{prefix} {field} must be an array") + implementation = capability.get("implementationEvidence") + if not isinstance(implementation, dict): + _add(errors, f"{prefix} implementationEvidence must be an object") + else: + for field in ("code", "tests", "documentation"): + if not isinstance(implementation.get(field), list): + _add(errors, f"{prefix} implementationEvidence.{field} must be an array") + return [capability for capability in capabilities if isinstance(capability, dict)] + + +def _validate_paths(root: Path, capabilities: list[dict[str, Any]], errors: list[str]) -> None: + for capability in capabilities: + capability_id = capability.get("id", "") + paths: list[str] = [] + for field in ("specification", "evidence"): + values = capability.get(field, []) + paths.extend([values] if isinstance(values, str) else values if isinstance(values, list) else []) + implementation = capability.get("implementationEvidence", {}) + if isinstance(implementation, dict): + for values in implementation.values(): + if isinstance(values, list): + paths.extend(values) + for raw_path in paths: + if not isinstance(raw_path, str): + _add(errors, f"{capability_id} contains a non-string path") + continue + path = _local_path(root, raw_path) + if not path.exists(): + _add(errors, f"{capability_id} references missing path: {raw_path}") + + +def _validate_sdds(root: Path, capabilities: list[dict[str, Any]], errors: list[str]) -> None: + by_id = {capability.get("id"): capability for capability in capabilities} + for capability in capabilities: + capability_id = capability.get("id") + specification = capability.get("specification") + if not isinstance(capability_id, str) or not isinstance(specification, str): + continue + path = _local_path(root, specification) + if not path.is_file(): + continue + text = path.read_text(encoding="utf-8") + matches = CAPABILITY_ID_RE.findall(text) + if matches != [capability_id]: + _add(errors, f"{specification} must declare exactly catalog capability ID {capability_id!r}") + status_match = re.search(r"^\s*-\s+Status:\s+(.+?)\s*$", text, re.MULTILINE) + expected_status = CATALOG_STATUSES.get(capability.get("status")) + if status_match and expected_status and status_match.group(1) != expected_status: + _add(errors, f"{specification} status {status_match.group(1)!r} disagrees with catalog {expected_status!r}") + + for path in sorted((root / "specs").glob("*.md")): + if path.name in EXCLUDED_SPEC_MARKDOWN: + continue + text = path.read_text(encoding="utf-8") + matches = CAPABILITY_ID_RE.findall(text) + if len(matches) != 1: + _add(errors, f"{path.relative_to(root)} must declare exactly one catalog capability ID") + elif matches[0] not in by_id: + _add(errors, f"{path.relative_to(root)} uses unknown catalog capability ID {matches[0]!r}") + + +def _validate_requirements(root: Path, capabilities: list[dict[str, Any]], errors: list[str]) -> None: + known: set[str] = set() + for path in (root / "docs").rglob("*.md"): + known.update(REQUIREMENT_RE.findall(path.read_text(encoding="utf-8"))) + for capability in capabilities: + for requirement in capability.get("requirements", []): + if requirement not in known: + _add(errors, f"{capability.get('id', '')} references unknown requirement {requirement}") + + +def _validate_catalog_markdown(root: Path, capabilities: list[dict[str, Any]], errors: list[str]) -> None: + path = root / "specs" / "CATALOG.md" + if not path.is_file(): + _add(errors, "specs/CATALOG.md is missing") + return + rows = {match.group(1): (match.group(2).strip(), match.group(3)) for match in CATALOG_ROW_RE.finditer(path.read_text(encoding="utf-8"))} + expected = {capability.get("id"): capability for capability in capabilities} + if set(rows) != set(expected): + _add(errors, "specs/CATALOG.md capability rows disagree with catalog.json") + for capability_id, capability in expected.items(): + row = rows.get(capability_id) + if row is None: + continue + expected_status = CATALOG_STATUSES.get(capability.get("status")) + expected_path = Path(capability.get("specification", "")).name + if row != (expected_status, expected_path): + _add(errors, f"specs/CATALOG.md row for {capability_id} disagrees with catalog.json") + + +def _validate_markdown_links(root: Path, errors: list[str]) -> None: + for path in sorted(root.rglob("*.md")): + if ".git" in path.parts: + continue + text = path.read_text(encoding="utf-8") + for match in MARKDOWN_LINK_RE.finditer(text): + target = match.group(1) or match.group(2) + if not target or target.startswith(("http://", "https://", "mailto:", "#", "codex://")): + continue + target_path = target.split("#", 1)[0].split("?", 1)[0] + if not target_path: + continue + candidate = (path.parent / target_path).resolve() + try: + candidate.relative_to(root.resolve()) + except ValueError: + _add(errors, f"{path.relative_to(root)} links outside repository: {target}") + continue + if not candidate.exists(): + _add(errors, f"{path.relative_to(root)} links to missing path: {target}") + + +def validate(root: Path) -> list[str]: + """Return all catalog/SDD consistency errors for ``root``.""" + + errors: list[str] = [] + catalog_path = root / "specs" / "catalog.json" + try: + catalog = json.loads(catalog_path.read_text(encoding="utf-8")) + except FileNotFoundError: + return ["specs/catalog.json is missing"] + except json.JSONDecodeError as error: + return [f"specs/catalog.json is invalid JSON: {error}"] + capabilities = _validate_catalog_shape(root, catalog, errors) + _validate_paths(root, capabilities, errors) + _validate_sdds(root, capabilities, errors) + _validate_requirements(root, capabilities, errors) + _validate_catalog_markdown(root, capabilities, errors) + _validate_markdown_links(root, errors) + return errors + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description="Validate Symphonia SDD/catalog consistency") + parser.add_argument("--root", type=Path, default=Path(__file__).resolve().parents[1]) + args = parser.parse_args(argv) + errors = validate(args.root.resolve()) + if errors: + print(f"spec validation failed with {len(errors)} error(s):", file=sys.stderr) + for error in errors: + print(f"- {error}", file=sys.stderr) + return 1 + print("spec validation passed") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From bd00868d8bcc4ca168d6a1889e8aea69ed73f25d Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Thu, 24 Sep 2026 15:35:46 +0200 Subject: [PATCH 099/167] fix: close all runtime resources on shutdown errors --- docs/development/implementation-baseline.md | 2 +- specs/home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/resources.py | 12 +++++++++--- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index e2e27da..28f0e65 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -54,7 +54,7 @@ The first implementation increment is intentionally narrower than any provider o - Every SQLite repository enables foreign-key enforcement at connection startup; backup preflight remains a separate integrity check. - A shared SQLite connection policy applies the same five-second busy timeout and row-factory settings to every durable store. - Runtime resources support explicit and context-manager lifecycle shutdown. -- Runtime resource shutdown is idempotent and remains not-ready after closure, so repeated Supervisor/finally cleanup cannot reopen or report healthy stores. +- Runtime resource shutdown attempts every repository close even if one raises, remains retryable after a partial close, and is idempotent after full closure. - Readiness can validate every composed durable store instead of only the operation queue. - Runtime resources provide a consistent SQLite online-backup helper while the service remains open. - Runtime resources can preflight backup integrity and required durable tables read-only before restore design is selected. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index d58f305..ff73d83 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing with bounded response headers, composed SQLite stores with semantic readiness checks, transactionally consistent backup/preflight helpers with future-schema rejection and temporary-copy validation before publication, idempotent resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing with bounded response headers, composed SQLite stores with semantic readiness checks, transactionally consistent backup/preflight helpers with future-schema rejection and temporary-copy validation before publication, retryable reverse-order resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 02be36b..807d250 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -69,8 +69,7 @@ def close(self) -> None: if self._closed: return - self._closed = True - + first_error: Exception | None = None for repository in ( self.resolutions, self.projections, @@ -79,7 +78,14 @@ def close(self) -> None: self.plans, self.operations, ): - repository.close() + try: + repository.close() + except Exception as error: + if first_error is None: + first_error = error + if first_error is not None: + raise first_error + self._closed = True def __enter__(self) -> "RuntimeResources": return self From e091285198a9c1c076f92ef1110227f2a6eed66a Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Thu, 24 Sep 2026 15:38:08 +0200 Subject: [PATCH 100/167] fix: reject non-origin runtime request targets --- docs/development/implementation-baseline.md | 2 +- specs/home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/http.py | 14 +++++++++++++- 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 28f0e65..d6d308f 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -68,7 +68,7 @@ The first implementation increment is intentionally narrower than any provider o - Provider connection diagnostics expose provider/state and expiry counts, never account or credential data. - Import diagnostics expose bounded snapshot freshness and availability counts without playlist content. - Resolution diagnostics expose only manual-decision action counts, never track, actor, or reason data. -- Ingress-relative health/version routing with normalized, traversal-safe base paths. +- Ingress-relative health/version routing accepts only origin-form request targets, rejects absolute/network-path targets and fragments, and uses normalized traversal-safe base paths. - The composed runtime HTTP server has an offline route contract and a loopback smoke path for real health/readiness lifecycle checks. - The current JSON health/readiness/version surface disables caching, MIME sniffing, and referrer propagation. - The runtime JSON surface rejects non-standard numeric values before writing a response. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index ff73d83..ae7d8ee 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing with bounded response headers, composed SQLite stores with semantic readiness checks, transactionally consistent backup/preflight helpers with future-schema rejection and temporary-copy validation before publication, retryable reverse-order resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing that rejects absolute-form and fragment-bearing request targets, bounded response headers, composed SQLite stores with semantic readiness checks, transactionally consistent backup/preflight helpers with future-schema rejection and temporary-copy validation before publication, retryable reverse-order resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/runtime/http.py b/src/symphonia/runtime/http.py index 55171e3..f1591a3 100644 --- a/src/symphonia/runtime/http.py +++ b/src/symphonia/runtime/http.py @@ -142,7 +142,19 @@ def _normalize_base_path(value: str) -> str: def _relative_path(request_path: str, base_path: str) -> str | None: - path = urlsplit(request_path).path or "/" + if ( + not isinstance(request_path, str) + or not request_path.startswith("/") + or request_path.startswith("//") + ): + return None + try: + parsed = urlsplit(request_path) + except ValueError: + return None + if parsed.scheme or parsed.netloc or parsed.fragment: + return None + path = parsed.path or "/" if base_path == "/": return path if path == base_path: From e6aac03073c20d475a4bf7002156fba4dc193124 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Thu, 24 Sep 2026 15:39:29 +0200 Subject: [PATCH 101/167] fix: unwind partial runtime startup safely --- docs/development/implementation-baseline.md | 1 + specs/home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/resources.py | 14 ++++++++++++-- 3 files changed, 14 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index d6d308f..ca28b97 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -55,6 +55,7 @@ The first implementation increment is intentionally narrower than any provider o - A shared SQLite connection policy applies the same five-second busy timeout and row-factory settings to every durable store. - Runtime resources support explicit and context-manager lifecycle shutdown. - Runtime resource shutdown attempts every repository close even if one raises, remains retryable after a partial close, and is idempotent after full closure. +- Partial runtime startup unwinds every repository already opened, retaining the startup exception as primary if cleanup also fails. - Readiness can validate every composed durable store instead of only the operation queue. - Runtime resources provide a consistent SQLite online-backup helper while the service remains open. - Runtime resources can preflight backup integrity and required durable tables read-only before restore design is selected. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index ae7d8ee..8e9afca 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing that rejects absolute-form and fragment-bearing request targets, bounded response headers, composed SQLite stores with semantic readiness checks, transactionally consistent backup/preflight helpers with future-schema rejection and temporary-copy validation before publication, retryable reverse-order resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing that rejects absolute-form and fragment-bearing request targets, bounded response headers, composed SQLite stores with semantic readiness checks, transactionally consistent backup/preflight helpers with future-schema rejection and temporary-copy validation before publication, reverse-order cleanup after partial startup, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 807d250..7cafcde 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -58,9 +58,19 @@ def open(cls, database_path: str) -> "RuntimeResources": opened.append(projections) resolutions = ResolutionDecisionRepository(database_path) opened.append(resolutions) - except Exception: + except Exception as startup_error: + cleanup_error_types: list[str] = [] for repository in reversed(opened): - repository.close() # type: ignore[attr-defined] + try: + repository.close() # type: ignore[attr-defined] + except Exception as cleanup_error: + cleanup_error_types.append(type(cleanup_error).__name__) + if cleanup_error_types: + error_types = ", ".join(cleanup_error_types) + startup_error.add_note( + "startup cleanup also encountered repository close errors " + f"({error_types})" + ) raise return cls(operations, plans, connections, authorization, projections, resolutions, database_path) From 18faaa6d1b292a32e4faee08d89f8fcf0d3edab1 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Thu, 24 Sep 2026 15:41:32 +0200 Subject: [PATCH 102/167] fix: close sqlite connections when migrations fail --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- .../home-assistant-app-runtime-and-ingress.md | 2 +- .../infrastructure/sqlite_authorization.py | 4 ++-- src/symphonia/infrastructure/sqlite_common.py | 22 ++++++++++++++++++- .../infrastructure/sqlite_connections.py | 4 ++-- .../infrastructure/sqlite_library.py | 4 ++-- .../infrastructure/sqlite_operations.py | 4 ++-- src/symphonia/infrastructure/sqlite_plans.py | 4 ++-- .../infrastructure/sqlite_resolutions.py | 4 ++-- 10 files changed, 36 insertions(+), 15 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index ca28b97..a617ad3 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -56,6 +56,7 @@ The first implementation increment is intentionally narrower than any provider o - Runtime resources support explicit and context-manager lifecycle shutdown. - Runtime resource shutdown attempts every repository close even if one raises, remains retryable after a partial close, and is idempotent after full closure. - Partial runtime startup unwinds every repository already opened, retaining the startup exception as primary if cleanup also fails. +- Every SQLite store closes its newly opened connection when schema initialization fails, preserving the migration error as primary. - Readiness can validate every composed durable store instead of only the operation queue. - Runtime resources provide a consistent SQLite online-backup helper while the service remains open. - Runtime resources can preflight backup integrity and required durable tables read-only before restore design is selected. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 19c8772..ed831f1 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, JSON-object/string-key payload validation, and deterministic worker tests. Its readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, and invalid credential-bearing payload keys; real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Its readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, and invalid credential-bearing payload keys; real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 8e9afca..69a34ee 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing that rejects absolute-form and fragment-bearing request targets, bounded response headers, composed SQLite stores with semantic readiness checks, transactionally consistent backup/preflight helpers with future-schema rejection and temporary-copy validation before publication, reverse-order cleanup after partial startup, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing that rejects absolute-form and fragment-bearing request targets, bounded response headers, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection and temporary-copy validation before publication, reverse-order cleanup after partial startup, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/infrastructure/sqlite_authorization.py b/src/symphonia/infrastructure/sqlite_authorization.py index 735e468..d7bd65d 100644 --- a/src/symphonia/infrastructure/sqlite_authorization.py +++ b/src/symphonia/infrastructure/sqlite_authorization.py @@ -15,7 +15,7 @@ validate_redirect_uri, ) -from .sqlite_common import connect +from .sqlite_common import connect, initialize_with_cleanup def _utc(value: datetime) -> str: @@ -51,7 +51,7 @@ class AuthorizationAttemptRepository: def __init__(self, path: str = ":memory:") -> None: self._connection = connect(path) - self._migrate() + initialize_with_cleanup(self._connection, self._migrate) def close(self) -> None: self._connection.close() diff --git a/src/symphonia/infrastructure/sqlite_common.py b/src/symphonia/infrastructure/sqlite_common.py index 05e4398..0d5cbc4 100644 --- a/src/symphonia/infrastructure/sqlite_common.py +++ b/src/symphonia/infrastructure/sqlite_common.py @@ -4,6 +4,7 @@ import json import sqlite3 +from collections.abc import Callable from typing import Any @@ -39,4 +40,23 @@ def connect(path: str) -> sqlite3.Connection: return connection -__all__ = ["connect", "dump_json", "load_json"] +def initialize_with_cleanup( + connection: sqlite3.Connection, + initialize: Callable[[], None], +) -> None: + """Close a newly opened connection if its schema initialization fails.""" + + try: + initialize() + except BaseException as initialization_error: + try: + connection.close() + except BaseException as cleanup_error: + initialization_error.add_note( + "SQLite connection cleanup also failed " + f"({type(cleanup_error).__name__})" + ) + raise + + +__all__ = ["connect", "dump_json", "initialize_with_cleanup", "load_json"] diff --git a/src/symphonia/infrastructure/sqlite_connections.py b/src/symphonia/infrastructure/sqlite_connections.py index 967fa6d..dd2541f 100644 --- a/src/symphonia/infrastructure/sqlite_connections.py +++ b/src/symphonia/infrastructure/sqlite_connections.py @@ -9,7 +9,7 @@ from symphonia.providers.connections import ConnectionState, ProviderConnection from symphonia.providers.contracts import Capability, ProviderCapabilities -from .sqlite_common import connect, dump_json, load_json +from .sqlite_common import connect, dump_json, initialize_with_cleanup, load_json def _utc(value: datetime) -> str: @@ -35,7 +35,7 @@ class ProviderConnectionRepository: def __init__(self, path: str = ":memory:") -> None: self._connection = connect(path) - self._migrate() + initialize_with_cleanup(self._connection, self._migrate) def close(self) -> None: self._connection.close() diff --git a/src/symphonia/infrastructure/sqlite_library.py b/src/symphonia/infrastructure/sqlite_library.py index 05acc92..1a3e5a8 100644 --- a/src/symphonia/infrastructure/sqlite_library.py +++ b/src/symphonia/infrastructure/sqlite_library.py @@ -11,7 +11,7 @@ from symphonia.providers.contracts import ProviderPlaylistEntry from symphonia.providers.importing import CollectionImportResult -from .sqlite_common import connect +from .sqlite_common import connect, initialize_with_cleanup def _utc(value: datetime) -> str: @@ -48,7 +48,7 @@ class PlaylistProjectionRepository: def __init__(self, path: str = ":memory:") -> None: self._connection = connect(path) - self._migrate() + initialize_with_cleanup(self._connection, self._migrate) def close(self) -> None: self._connection.close() diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index aa391b3..3c141d8 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -16,7 +16,7 @@ from typing import Any import uuid -from .sqlite_common import connect +from .sqlite_common import connect, initialize_with_cleanup def _utc(value: datetime) -> str: @@ -166,7 +166,7 @@ class OperationRepository: def __init__(self, path: str = ":memory:") -> None: self._connection = connect(path) - self._migrate() + initialize_with_cleanup(self._connection, self._migrate) def close(self) -> None: self._connection.close() diff --git a/src/symphonia/infrastructure/sqlite_plans.py b/src/symphonia/infrastructure/sqlite_plans.py index 92ed1ca..ff86254 100644 --- a/src/symphonia/infrastructure/sqlite_plans.py +++ b/src/symphonia/infrastructure/sqlite_plans.py @@ -15,7 +15,7 @@ PlanAcceptanceError, ) -from .sqlite_common import connect, dump_json, load_json +from .sqlite_common import connect, dump_json, initialize_with_cleanup, load_json def _utc(value: datetime) -> str: @@ -43,7 +43,7 @@ class CopyPlanRepository: def __init__(self, path: str = ":memory:") -> None: self._connection = connect(path) - self._migrate() + initialize_with_cleanup(self._connection, self._migrate) def close(self) -> None: self._connection.close() diff --git a/src/symphonia/infrastructure/sqlite_resolutions.py b/src/symphonia/infrastructure/sqlite_resolutions.py index a466132..618b89c 100644 --- a/src/symphonia/infrastructure/sqlite_resolutions.py +++ b/src/symphonia/infrastructure/sqlite_resolutions.py @@ -9,7 +9,7 @@ from symphonia.identity.models import ManualDecision, ManualDecisionAction -from .sqlite_common import connect, dump_json, load_json +from .sqlite_common import connect, dump_json, initialize_with_cleanup, load_json class ResolutionDecisionRepository: @@ -17,7 +17,7 @@ class ResolutionDecisionRepository: def __init__(self, path: str = ":memory:") -> None: self._connection = connect(path) - self._migrate() + initialize_with_cleanup(self._connection, self._migrate) def close(self) -> None: self._connection.close() From c49ca76902be9a4acf97c148ffda5772514b0c6e Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Thu, 24 Sep 2026 15:42:38 +0200 Subject: [PATCH 103/167] fix: reject symlinked backup destinations --- docs/development/implementation-baseline.md | 2 +- specs/home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/resources.py | 7 ++++++- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index a617ad3..35da6d2 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -61,7 +61,7 @@ The first implementation increment is intentionally narrower than any provider o - Runtime resources provide a consistent SQLite online-backup helper while the service remains open. - Runtime resources can preflight backup integrity and required durable tables read-only before restore design is selected. - Backup publication validates the temporary SQLite copy before atomically replacing the destination. -- Runtime backups reject missing parent directories and directory destinations before creating a temporary artifact. +- Runtime backups reject missing parent directories, directory destinations, and final-component symlinks before creating a temporary artifact. - Backup preflight rejects operation databases whose schema version is newer than the running foundation. - Runtime configuration validates host, port, database path, and Ingress base path before startup. - Runtime configuration rejects control characters, non-integral ports, and query/fragment-bearing Ingress paths before startup. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 69a34ee..2a42017 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing that rejects absolute-form and fragment-bearing request targets, bounded response headers, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection and temporary-copy validation before publication, reverse-order cleanup after partial startup, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing that rejects absolute-form and fragment-bearing request targets, bounded response headers, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, reverse-order cleanup after partial startup, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 7cafcde..3f65b60 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -142,7 +142,12 @@ def backup_to(self, destination_path: str) -> None: if not self.healthcheck(): raise RuntimeError("cannot back up an unhealthy runtime") live_path = Path(self.database_path).expanduser().resolve() - destination_path_object = Path(destination_path).expanduser().resolve() + destination_candidate = Path(destination_path).expanduser() + if destination_candidate.is_symlink(): + raise ValueError("destination_path must not be a symbolic link") + destination_path_object = ( + destination_candidate.parent.resolve() / destination_candidate.name + ) if live_path == destination_path_object: raise ValueError("destination_path must differ from the live database") if not destination_path_object.parent.is_dir(): From 774f49fc31490be17c753a62a4411034f209b7c2 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Thu, 24 Sep 2026 15:44:58 +0200 Subject: [PATCH 104/167] fix: centralize ingress path validation --- docs/development/implementation-baseline.md | 2 +- .../home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/config.py | 10 ++++++---- src/symphonia/runtime/http.py | 18 +++--------------- 4 files changed, 11 insertions(+), 21 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 35da6d2..e7af293 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -64,7 +64,7 @@ The first implementation increment is intentionally narrower than any provider o - Runtime backups reject missing parent directories, directory destinations, and final-component symlinks before creating a temporary artifact. - Backup preflight rejects operation databases whose schema version is newer than the running foundation. - Runtime configuration validates host, port, database path, and Ingress base path before startup. -- Runtime configuration rejects control characters, non-integral ports, and query/fragment-bearing Ingress paths before startup. +- Runtime configuration and HTTP routing share one Ingress path normalizer that rejects control characters, query/fragments, backslashes, repeated slashes, and literal dot segments. - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. - Runtime readiness validates SQLite integrity, foreign-key references, durable operation state/event values, authorization attempts, provider capability JSON, plan JSON, and resolution payloads; corruption fails closed. - Provider connection diagnostics expose provider/state and expiry counts, never account or credential data. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 2a42017..4daf158 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, Ingress-relative health/readiness/version routing that rejects absolute-form and fragment-bearing request targets, bounded response headers, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, reverse-order cleanup after partial startup, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing, bounded response headers, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, reverse-order cleanup after partial startup, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/runtime/config.py b/src/symphonia/runtime/config.py index 15a7b73..900f624 100644 --- a/src/symphonia/runtime/config.py +++ b/src/symphonia/runtime/config.py @@ -7,15 +7,17 @@ from collections.abc import Mapping -def _normalize_ingress_path(value: str) -> str: +def normalize_ingress_path(value: object) -> str: if not isinstance(value, str) or not value or not value.startswith("/"): raise ValueError("ingress path must start with '/'") if any(ord(char) < 0x20 or ord(char) == 0x7F for char in value): raise ValueError("ingress path must not contain control characters") if "?" in value or "#" in value: raise ValueError("ingress path must contain only a path") + if "\\" in value or "//" in value: + raise ValueError("ingress path contains an unsafe separator") normalized = value.rstrip("/") or "/" - if "//" in normalized or "/.." in normalized or "/./" in normalized: + if any(segment in {".", ".."} for segment in normalized.split("/")): raise ValueError("ingress path contains an unsafe segment") return normalized @@ -58,7 +60,7 @@ def __post_init__(self) -> None: if any(ord(char) < 0x20 or ord(char) == 0x7F for char in self.database_path): raise ValueError("database_path must not contain control characters") object.__setattr__(self, "database_path", self.database_path.strip()) - object.__setattr__(self, "ingress_path", _normalize_ingress_path(self.ingress_path)) + object.__setattr__(self, "ingress_path", normalize_ingress_path(self.ingress_path)) @classmethod def from_environment(cls, environ: Mapping[str, str] | None = None) -> "RuntimeConfig": @@ -71,4 +73,4 @@ def from_environment(cls, environ: Mapping[str, str] | None = None) -> "RuntimeC ) -__all__ = ["RuntimeConfig"] +__all__ = ["RuntimeConfig", "normalize_ingress_path"] diff --git a/src/symphonia/runtime/http.py b/src/symphonia/runtime/http.py index f1591a3..78320ef 100644 --- a/src/symphonia/runtime/http.py +++ b/src/symphonia/runtime/http.py @@ -10,6 +10,7 @@ from symphonia import __version__ from symphonia.infrastructure.sqlite_operations import OperationRepository +from .config import normalize_ingress_path from .resources import RuntimeResources @@ -28,7 +29,7 @@ def __init__( raise ValueError("repository and resources are mutually exclusive") if repository is None and resources is None: raise ValueError("repository or resources must be supplied") - normalized_ingress_path = _normalize_base_path(ingress_path) + normalized_ingress_path = normalize_ingress_path(ingress_path) super().__init__(address, SymphoniaRequestHandler) self.resources = resources self.repository = resources.operations if resources is not None else repository @@ -109,7 +110,7 @@ def route_get( mistaken for application readiness. """ - relative_path = _relative_path(path, _normalize_base_path(ingress_path)) + relative_path = _relative_path(path, normalize_ingress_path(ingress_path)) if relative_path is None: return 404, {"error": "not_found"} if relative_path == "/health": @@ -128,19 +129,6 @@ def route_get( return 404, {"error": "not_found"} -def _normalize_base_path(value: str) -> str: - if not value or not value.startswith("/"): - raise ValueError("ingress path must start with '/'") - if any(ord(char) < 0x20 or ord(char) == 0x7F for char in value): - raise ValueError("ingress path must not contain control characters") - if "?" in value or "#" in value: - raise ValueError("ingress path must contain only a path") - normalized = value.rstrip("/") or "/" - if "//" in normalized or "/.." in normalized or "/./" in normalized: - raise ValueError("ingress path contains an unsafe segment") - return normalized - - def _relative_path(request_path: str, base_path: str) -> str | None: if ( not isinstance(request_path, str) From 788b136bf886d87ed004d9071aea0694a553352f Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Fri, 25 Sep 2026 23:38:59 +0200 Subject: [PATCH 105/167] refactor: encapsulate sqlite backup in repository --- docs/development/implementation-baseline.md | 2 +- src/symphonia/infrastructure/sqlite_operations.py | 5 +++++ src/symphonia/runtime/resources.py | 2 +- 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index e7af293..583e9cb 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -58,7 +58,7 @@ The first implementation increment is intentionally narrower than any provider o - Partial runtime startup unwinds every repository already opened, retaining the startup exception as primary if cleanup also fails. - Every SQLite store closes its newly opened connection when schema initialization fails, preserving the migration error as primary. - Readiness can validate every composed durable store instead of only the operation queue. -- Runtime resources provide a consistent SQLite online-backup helper while the service remains open. +- Runtime resources request a consistent online backup through the operation repository adapter while the service remains open, without reaching into its private connection. - Runtime resources can preflight backup integrity and required durable tables read-only before restore design is selected. - Backup publication validates the temporary SQLite copy before atomically replacing the destination. - Runtime backups reject missing parent directories, directory destinations, and final-component symlinks before creating a temporary artifact. diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 3c141d8..74a3fc8 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -171,6 +171,11 @@ def __init__(self, path: str = ":memory:") -> None: def close(self) -> None: self._connection.close() + def backup_to(self, destination: sqlite3.Connection) -> None: + """Copy this store's consistent SQLite snapshot to a destination.""" + + self._connection.backup(destination) + def healthcheck(self) -> bool: """Return whether schema and durable operation values are readable.""" diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index 3f65b60..ac0ec30 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -167,7 +167,7 @@ def backup_to(self, destination_path: str) -> None: temporary_path = temporary.name destination = sqlite3.connect(temporary_path) try: - self.operations._connection.backup(destination) # type: ignore[attr-defined] + self.operations.backup_to(destination) destination.commit() finally: destination.close() From 93982b479b8a613a388e3bfea46bdf02c2f971e8 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Fri, 25 Sep 2026 23:41:02 +0200 Subject: [PATCH 106/167] fix: preserve cleanup on interrupted startup --- docs/development/implementation-baseline.md | 5 +++-- specs/home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/http.py | 10 ++++++++-- src/symphonia/runtime/resources.py | 8 ++++---- 4 files changed, 16 insertions(+), 9 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 583e9cb..a2dc74c 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -54,8 +54,9 @@ The first implementation increment is intentionally narrower than any provider o - Every SQLite repository enables foreign-key enforcement at connection startup; backup preflight remains a separate integrity check. - A shared SQLite connection policy applies the same five-second busy timeout and row-factory settings to every durable store. - Runtime resources support explicit and context-manager lifecycle shutdown. -- Runtime resource shutdown attempts every repository close even if one raises, remains retryable after a partial close, and is idempotent after full closure. -- Partial runtime startup unwinds every repository already opened, retaining the startup exception as primary if cleanup also fails. +- Runtime resource shutdown attempts every repository close even if one raises or is interrupted, remains retryable after a partial close, and is idempotent after full closure. +- Partial runtime startup unwinds every repository already opened, including on interruption, retaining the startup exception as primary if cleanup also fails. +- HTTP server construction closes composed resources on any startup failure without replacing the original cause with a cleanup error. - Every SQLite store closes its newly opened connection when schema initialization fails, preserving the migration error as primary. - Readiness can validate every composed durable store instead of only the operation queue. - Runtime resources request a consistent online backup through the operation repository adapter while the service remains open, without reaching into its private connection. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 4daf158..906941a 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing, bounded response headers, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, reverse-order cleanup after partial startup, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing, bounded response headers, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/runtime/http.py b/src/symphonia/runtime/http.py index 78320ef..d67ca14 100644 --- a/src/symphonia/runtime/http.py +++ b/src/symphonia/runtime/http.py @@ -91,8 +91,14 @@ def create_server( resources = RuntimeResources.open(database_path) try: return SymphoniaHTTPServer((host, port), ingress_path=ingress_path, resources=resources) - except Exception: - resources.close() + except BaseException as startup_error: + try: + resources.close() + except BaseException as cleanup_error: + startup_error.add_note( + "runtime resource cleanup also failed " + f"({type(cleanup_error).__name__})" + ) raise diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index ac0ec30..e065b95 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -58,12 +58,12 @@ def open(cls, database_path: str) -> "RuntimeResources": opened.append(projections) resolutions = ResolutionDecisionRepository(database_path) opened.append(resolutions) - except Exception as startup_error: + except BaseException as startup_error: cleanup_error_types: list[str] = [] for repository in reversed(opened): try: repository.close() # type: ignore[attr-defined] - except Exception as cleanup_error: + except BaseException as cleanup_error: cleanup_error_types.append(type(cleanup_error).__name__) if cleanup_error_types: error_types = ", ".join(cleanup_error_types) @@ -79,7 +79,7 @@ def close(self) -> None: if self._closed: return - first_error: Exception | None = None + first_error: BaseException | None = None for repository in ( self.resolutions, self.projections, @@ -90,7 +90,7 @@ def close(self) -> None: ): try: repository.close() - except Exception as error: + except BaseException as error: if first_error is None: first_error = error if first_error is not None: From d4e502b01566afcc4035345287e430320d8539e5 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Fri, 25 Sep 2026 23:42:02 +0200 Subject: [PATCH 107/167] fix: validate runtime config before opening stores --- docs/development/implementation-baseline.md | 2 +- specs/home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/http.py | 16 +++++++++++++--- 3 files changed, 15 insertions(+), 5 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index a2dc74c..a975c7a 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -64,7 +64,7 @@ The first implementation increment is intentionally narrower than any provider o - Backup publication validates the temporary SQLite copy before atomically replacing the destination. - Runtime backups reject missing parent directories, directory destinations, and final-component symlinks before creating a temporary artifact. - Backup preflight rejects operation databases whose schema version is newer than the running foundation. -- Runtime configuration validates host, port, database path, and Ingress base path before startup. +- Runtime configuration and the public server factory validate host, port, database path, and Ingress base path before opening stores or binding a socket. - Runtime configuration and HTTP routing share one Ingress path normalizer that rejects control characters, query/fragments, backslashes, repeated slashes, and literal dot segments. - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. - Runtime readiness validates SQLite integrity, foreign-key references, durable operation state/event values, authorization attempts, provider capability JSON, plan JSON, and resolution payloads; corruption fails closed. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 906941a..cbefd03 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing, bounded response headers, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing, bounded response headers, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/runtime/http.py b/src/symphonia/runtime/http.py index d67ca14..e419de7 100644 --- a/src/symphonia/runtime/http.py +++ b/src/symphonia/runtime/http.py @@ -10,7 +10,7 @@ from symphonia import __version__ from symphonia.infrastructure.sqlite_operations import OperationRepository -from .config import normalize_ingress_path +from .config import RuntimeConfig, normalize_ingress_path from .resources import RuntimeResources @@ -88,9 +88,19 @@ def create_server( ) -> SymphoniaHTTPServer: """Create a server with an already-migrated durable operation store.""" - resources = RuntimeResources.open(database_path) + config = RuntimeConfig( + host=host, + port=port, + database_path=database_path, + ingress_path=ingress_path, + ) + resources = RuntimeResources.open(config.database_path) try: - return SymphoniaHTTPServer((host, port), ingress_path=ingress_path, resources=resources) + return SymphoniaHTTPServer( + (config.host, config.port), + ingress_path=config.ingress_path, + resources=resources, + ) except BaseException as startup_error: try: resources.close() From 666a39d1986c02f27fad4bccb131369b4dfe3500 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Fri, 25 Sep 2026 23:43:52 +0200 Subject: [PATCH 108/167] fix: shut down runtime cleanly on sigterm --- docs/development/implementation-baseline.md | 1 + .../home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/__main__.py | 20 ++++++++++++++++--- 3 files changed, 19 insertions(+), 4 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index a975c7a..a7bb9c9 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -57,6 +57,7 @@ The first implementation increment is intentionally narrower than any provider o - Runtime resource shutdown attempts every repository close even if one raises or is interrupted, remains retryable after a partial close, and is idempotent after full closure. - Partial runtime startup unwinds every repository already opened, including on interruption, retaining the startup exception as primary if cleanup also fails. - HTTP server construction closes composed resources on any startup failure without replacing the original cause with a cleanup error. +- The CLI treats `SIGTERM` as a graceful stop and closes the HTTP listener and composed resources before restoring the prior signal handler. - Every SQLite store closes its newly opened connection when schema initialization fails, preserving the migration error as primary. - Readiness can validate every composed durable store instead of only the operation queue. - Runtime resources request a consistent online backup through the operation repository adapter while the service remains open, without reaching into its private connection. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index cbefd03..18e5038 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing, bounded response headers, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing, bounded response headers, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/__main__.py b/src/symphonia/__main__.py index 0fd3913..a284a7d 100644 --- a/src/symphonia/__main__.py +++ b/src/symphonia/__main__.py @@ -4,10 +4,16 @@ import argparse import os +import signal +from types import FrameType from symphonia.runtime import RuntimeConfig, create_server +def _handle_sigterm(_signum: int, _frame: FrameType | None) -> None: + raise KeyboardInterrupt + + def main() -> None: parser = argparse.ArgumentParser(description="Run the Symphonia runtime foundation") parser.add_argument("--host", default=os.getenv("SYMPHONIA_HOST", "127.0.0.1")) @@ -32,14 +38,22 @@ def main() -> None: ) except ValueError as error: parser.error(str(error)) - server = create_server(config.host, config.port, config.database_path, config.ingress_path) + previous_sigterm_handler = signal.signal(signal.SIGTERM, _handle_sigterm) + server = None try: + server = create_server(config.host, config.port, config.database_path, config.ingress_path) server.serve_forever() except KeyboardInterrupt: pass finally: - server.server_close() - server.close_resources() + try: + if server is not None: + try: + server.server_close() + finally: + server.close_resources() + finally: + signal.signal(signal.SIGTERM, previous_sigterm_handler) if __name__ == "__main__": From 5fd7ef4bfa71b6153bfba86a925e6c60c3a4b6a1 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Fri, 25 Sep 2026 23:45:30 +0200 Subject: [PATCH 109/167] fix: sanitize default runtime http errors --- docs/development/implementation-baseline.md | 1 + .../home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/http.py | 29 ++++++++++++++++++- 3 files changed, 30 insertions(+), 2 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index a7bb9c9..2698809 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -75,6 +75,7 @@ The first implementation increment is intentionally narrower than any provider o - Ingress-relative health/version routing accepts only origin-form request targets, rejects absolute/network-path targets and fragments, and uses normalized traversal-safe base paths. - The composed runtime HTTP server has an offline route contract and a loopback smoke path for real health/readiness lifecycle checks. - The current JSON health/readiness/version surface disables caching, MIME sniffing, and referrer propagation. +- The HTTP handler suppresses the Python version header and returns generic JSON for parser/method errors, closing the connection without reflecting details. - The runtime JSON surface rejects non-standard numeric values before writing a response. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - Normalized provider identity, manifest, capability, entry, and page values reject non-textual or malformed boundary data before it reaches planning. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 18e5038..83ed00e 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing, bounded response headers, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing, bounded response headers and generic JSON parser/method errors without Python version disclosure, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/runtime/http.py b/src/symphonia/runtime/http.py index e419de7..75e12fd 100644 --- a/src/symphonia/runtime/http.py +++ b/src/symphonia/runtime/http.py @@ -46,6 +46,9 @@ def close_resources(self) -> None: class SymphoniaRequestHandler(BaseHTTPRequestHandler): """Only health/readiness/version are exposed until the API SDD is ready.""" + server_version = "Symphonia" + sys_version = "" + server: SymphoniaHTTPServer def do_GET(self) -> None: # noqa: N802 - stdlib handler API @@ -58,7 +61,28 @@ def do_GET(self) -> None: # noqa: N802 - stdlib handler API ) self._json(status, payload) - def _json(self, status: int, payload: dict[str, Any]) -> None: + def send_error( + self, + code: int, + message: str | None = None, + explain: str | None = None, + ) -> None: + del message, explain + error = "not_implemented" if code == 501 else ( + "server_error" if code >= 500 else "bad_request" + ) + self._json(code, {"error": error}, close_connection=True) + + def version_string(self) -> str: + return self.server_version + + def _json( + self, + status: int, + payload: dict[str, Any], + *, + close_connection: bool = False, + ) -> None: body = json.dumps( payload, ensure_ascii=False, @@ -71,6 +95,9 @@ def _json(self, status: int, payload: dict[str, Any]) -> None: self.send_header("Cache-Control", "no-store") self.send_header("X-Content-Type-Options", "nosniff") self.send_header("Referrer-Policy", "no-referrer") + if close_connection: + self.close_connection = True + self.send_header("Connection", "close") self.end_headers() self.wfile.write(body) From 17c7c2ca9aacbc01be1b260ef13fd4ab854e78c8 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Fri, 25 Sep 2026 23:46:43 +0200 Subject: [PATCH 110/167] fix: align container probe with runtime config --- Dockerfile | 3 +-- docs/development/implementation-baseline.md | 1 + specs/home-assistant-app-runtime-and-ingress.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Dockerfile b/Dockerfile index 413a62c..b9cc4af 100644 --- a/Dockerfile +++ b/Dockerfile @@ -20,7 +20,6 @@ VOLUME ["/data"] EXPOSE 8099 HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \ - CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8099/ready', timeout=2)"] + CMD ["python", "-c", "import os,urllib.request; port=os.getenv('SYMPHONIA_PORT','8099'); base=os.getenv('SYMPHONIA_INGRESS_PATH','/').rstrip('/'); urllib.request.urlopen(f'http://127.0.0.1:{port}{base}/ready', timeout=2)"] ENTRYPOINT ["python", "-m", "symphonia"] - diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 2698809..9ad6ff4 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -76,6 +76,7 @@ The first implementation increment is intentionally narrower than any provider o - The composed runtime HTTP server has an offline route contract and a loopback smoke path for real health/readiness lifecycle checks. - The current JSON health/readiness/version surface disables caching, MIME sniffing, and referrer propagation. - The HTTP handler suppresses the Python version header and returns generic JSON for parser/method errors, closing the connection without reflecting details. +- The container health probe follows the configured runtime port and Ingress base path. - The runtime JSON surface rejects non-standard numeric values before writing a response. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - Normalized provider identity, manifest, capability, entry, and page values reject non-textual or malformed boundary data before it reaches planning. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 83ed00e..23d4ffc 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing, bounded response headers and generic JSON parser/method errors without Python version disclosure, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing, a container health probe that follows the configured port and Ingress path, bounded response headers and generic JSON parser/method errors without Python version disclosure, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. From 3be58f6bc2d908869587bfd41c25b6f5889fecfa Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Fri, 25 Sep 2026 23:47:50 +0200 Subject: [PATCH 111/167] ci: check whitespace in pushed patch range --- .github/workflows/verify.yml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.github/workflows/verify.yml b/.github/workflows/verify.yml index e17cc29..b4bd51b 100644 --- a/.github/workflows/verify.yml +++ b/.github/workflows/verify.yml @@ -16,6 +16,8 @@ jobs: steps: - name: Check out repository uses: actions/checkout@v4 + with: + fetch-depth: 0 - name: Set up Python uses: actions/setup-python@v5 with: @@ -23,6 +25,8 @@ jobs: - name: Check specification consistency run: PYTHONPATH=. python tools/validate_specs.py - name: Check patch whitespace - run: git diff --check + env: + PATCH_BASE: ${{ github.event.pull_request.base.sha || github.event.before }} + run: git diff --check "${PATCH_BASE}...HEAD" - name: Run offline test suite run: PYTHONPATH=src:. python -m unittest discover -s tests -v From fd8b636ddd70b511af9ee3b8caf389047c157ddb Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Fri, 25 Sep 2026 23:49:05 +0200 Subject: [PATCH 112/167] fix: bound runtime http connection time --- docs/development/implementation-baseline.md | 1 + specs/home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/http.py | 7 +++++++ 3 files changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 9ad6ff4..3d370d5 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -73,6 +73,7 @@ The first implementation increment is intentionally narrower than any provider o - Import diagnostics expose bounded snapshot freshness and availability counts without playlist content. - Resolution diagnostics expose only manual-decision action counts, never track, actor, or reason data. - Ingress-relative health/version routing accepts only origin-form request targets, rejects absolute/network-path targets and fragments, and uses normalized traversal-safe base paths. +- The synchronous health server caps each accepted connection at a two-second socket timeout so an incomplete client cannot hold the only request loop indefinitely. - The composed runtime HTTP server has an offline route contract and a loopback smoke path for real health/readiness lifecycle checks. - The current JSON health/readiness/version surface disables caching, MIME sniffing, and referrer propagation. - The HTTP handler suppresses the Python version header and returns generic JSON for parser/method errors, closing the connection without reflecting details. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 23d4ffc..8b91c20 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing, a container health probe that follows the configured port and Ingress path, bounded response headers and generic JSON parser/method errors without Python version disclosure, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing with a bounded per-connection timeout, a container health probe that follows the configured port and Ingress path, bounded response headers and generic JSON parser/method errors without Python version disclosure, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/runtime/http.py b/src/symphonia/runtime/http.py index 75e12fd..522c98a 100644 --- a/src/symphonia/runtime/http.py +++ b/src/symphonia/runtime/http.py @@ -5,6 +5,7 @@ from http.server import BaseHTTPRequestHandler, HTTPServer import json from collections.abc import Callable +from socket import socket from typing import Any from urllib.parse import urlsplit @@ -16,6 +17,7 @@ class SymphoniaHTTPServer(HTTPServer): allow_reuse_address = True + request_timeout_seconds = 2.0 def __init__( self, @@ -37,6 +39,11 @@ def __init__( self.service_version = __version__ self.ingress_path = normalized_ingress_path + def get_request(self) -> tuple[socket, tuple[str, int]]: + request, client_address = super().get_request() + request.settimeout(self.request_timeout_seconds) + return request, client_address + def close_resources(self) -> None: if self.resources is not None: self.resources.close() From cf9521bc04ee1e7e2bdc9468146dc58e945ed9f9 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Fri, 25 Sep 2026 23:49:47 +0200 Subject: [PATCH 113/167] docs: refresh implementation baseline review date --- docs/development/implementation-baseline.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 3d370d5..a788616 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -1,7 +1,7 @@ # Implementation baseline **Status:** owner-approved foundation slice -**Last reviewed:** 2026-09-22 +**Last reviewed:** 2026-09-25 The first implementation increment is intentionally narrower than any provider or Home Assistant capability. It proves the provider-independent core and the durable-operation persistence contract without selecting an external web framework, provider SDK, OAuth strategy, or frontend stack. From bd3bb7d707b66cb9b3c55c36ba9844beeb8af2b1 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 10:10:09 +0200 Subject: [PATCH 114/167] fix: align container probe with bind host --- Dockerfile | 2 +- docs/development/implementation-baseline.md | 1 + specs/home-assistant-app-runtime-and-ingress.md | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/Dockerfile b/Dockerfile index b9cc4af..dbfa6e1 100644 --- a/Dockerfile +++ b/Dockerfile @@ -20,6 +20,6 @@ VOLUME ["/data"] EXPOSE 8099 HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \ - CMD ["python", "-c", "import os,urllib.request; port=os.getenv('SYMPHONIA_PORT','8099'); base=os.getenv('SYMPHONIA_INGRESS_PATH','/').rstrip('/'); urllib.request.urlopen(f'http://127.0.0.1:{port}{base}/ready', timeout=2)"] + CMD ["python", "-c", "import http.client,os,sys; host=os.getenv('SYMPHONIA_HOST','127.0.0.1'); host={'0.0.0.0':'127.0.0.1','::':'::1'}.get(host,host); port=int(os.getenv('SYMPHONIA_PORT','8099')); base=os.getenv('SYMPHONIA_INGRESS_PATH','/').rstrip('/'); connection=http.client.HTTPConnection(host,port,timeout=2); connection.request('GET',f'{base}/ready'); response=connection.getresponse(); response.read(); sys.exit(response.status != 200)"] ENTRYPOINT ["python", "-m", "symphonia"] diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index a788616..7554687 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -78,6 +78,7 @@ The first implementation increment is intentionally narrower than any provider o - The current JSON health/readiness/version surface disables caching, MIME sniffing, and referrer propagation. - The HTTP handler suppresses the Python version header and returns generic JSON for parser/method errors, closing the connection without reflecting details. - The container health probe follows the configured runtime port and Ingress base path. +- The container health probe follows the configured listener host, mapping IPv4/IPv6 wildcard binds to their loopback equivalents. - The runtime JSON surface rejects non-standard numeric values before writing a response. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - Normalized provider identity, manifest, capability, entry, and page values reject non-textual or malformed boundary data before it reaches planning. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 8b91c20..0897dd6 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing with a bounded per-connection timeout, a container health probe that follows the configured port and Ingress path, bounded response headers and generic JSON parser/method errors without Python version disclosure, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing with a bounded per-connection timeout, a container health probe that follows the configured host (mapping wildcard binds to loopback), port, and Ingress path, bounded response headers and generic JSON parser/method errors without Python version disclosure, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. From acf3f351078d870c2e35e38945e42ffa9b5f44ee Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 10:11:29 +0200 Subject: [PATCH 115/167] ci: handle first-push whitespace checks --- .github/workflows/verify.yml | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/.github/workflows/verify.yml b/.github/workflows/verify.yml index b4bd51b..a4941fe 100644 --- a/.github/workflows/verify.yml +++ b/.github/workflows/verify.yml @@ -27,6 +27,20 @@ jobs: - name: Check patch whitespace env: PATCH_BASE: ${{ github.event.pull_request.base.sha || github.event.before }} - run: git diff --check "${PATCH_BASE}...HEAD" + DEFAULT_BRANCH: ${{ github.event.repository.default_branch }} + run: | + if [[ "$PATCH_BASE" =~ ^0+$ ]]; then + DEFAULT_REF="refs/remotes/origin/$DEFAULT_BRANCH" + if git show-ref --verify --quiet "$DEFAULT_REF"; then + if PATCH_BASE="$(git merge-base HEAD "$DEFAULT_REF")"; then + git diff --check "${PATCH_BASE}...HEAD" + exit 0 + fi + fi + EMPTY_TREE="$(git hash-object -t tree /dev/null)" + git diff --check "$EMPTY_TREE" HEAD + else + git diff --check "${PATCH_BASE}...HEAD" + fi - name: Run offline test suite run: PYTHONPATH=src:. python -m unittest discover -s tests -v From bf21a5b2fb41e77041db911580e9b32519cfa768 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 10:12:36 +0200 Subject: [PATCH 116/167] fix: validate container readiness response --- Dockerfile | 2 +- docs/development/implementation-baseline.md | 2 +- specs/home-assistant-app-runtime-and-ingress.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Dockerfile b/Dockerfile index dbfa6e1..7f1ec46 100644 --- a/Dockerfile +++ b/Dockerfile @@ -20,6 +20,6 @@ VOLUME ["/data"] EXPOSE 8099 HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \ - CMD ["python", "-c", "import http.client,os,sys; host=os.getenv('SYMPHONIA_HOST','127.0.0.1'); host={'0.0.0.0':'127.0.0.1','::':'::1'}.get(host,host); port=int(os.getenv('SYMPHONIA_PORT','8099')); base=os.getenv('SYMPHONIA_INGRESS_PATH','/').rstrip('/'); connection=http.client.HTTPConnection(host,port,timeout=2); connection.request('GET',f'{base}/ready'); response=connection.getresponse(); response.read(); sys.exit(response.status != 200)"] + CMD ["python", "-c", "import http.client,json,os,sys; host=os.getenv('SYMPHONIA_HOST','127.0.0.1'); host={'0.0.0.0':'127.0.0.1','::':'::1'}.get(host,host); port=int(os.getenv('SYMPHONIA_PORT','8099')); base=os.getenv('SYMPHONIA_INGRESS_PATH','/').rstrip('/'); connection=http.client.HTTPConnection(host,port,timeout=2); connection.request('GET',f'{base}/ready'); response=connection.getresponse(); payload=json.loads(response.read()); sys.exit(response.status != 200 or payload.get('service') != 'symphonia' or payload.get('status') != 'ready')"] ENTRYPOINT ["python", "-m", "symphonia"] diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 7554687..417f91e 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -78,7 +78,7 @@ The first implementation increment is intentionally narrower than any provider o - The current JSON health/readiness/version surface disables caching, MIME sniffing, and referrer propagation. - The HTTP handler suppresses the Python version header and returns generic JSON for parser/method errors, closing the connection without reflecting details. - The container health probe follows the configured runtime port and Ingress base path. -- The container health probe follows the configured listener host, mapping IPv4/IPv6 wildcard binds to their loopback equivalents. +- The container health probe follows the configured listener host, mapping IPv4/IPv6 wildcard binds to their loopback equivalents, and requires the Symphonia ready JSON contract rather than any HTTP 200 response. - The runtime JSON surface rejects non-standard numeric values before writing a response. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - Normalized provider identity, manifest, capability, entry, and page values reject non-textual or malformed boundary data before it reaches planning. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 0897dd6..d8ddb60 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing with a bounded per-connection timeout, a container health probe that follows the configured host (mapping wildcard binds to loopback), port, and Ingress path, bounded response headers and generic JSON parser/method errors without Python version disclosure, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing with a bounded per-connection timeout, a container health probe that follows the configured host (mapping wildcard binds to loopback), port, and Ingress path and validates Symphonia's ready response, bounded response headers and generic JSON parser/method errors without Python version disclosure, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. From 1a72be52cc878be75b1f42432dffc3f5ae2a9fa0 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 10:13:59 +0200 Subject: [PATCH 117/167] fix: preserve backup failure during cleanup --- docs/development/implementation-baseline.md | 1 + specs/home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/resources.py | 8 +++++++- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 417f91e..b30b776 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -63,6 +63,7 @@ The first implementation increment is intentionally narrower than any provider o - Runtime resources request a consistent online backup through the operation repository adapter while the service remains open, without reaching into its private connection. - Runtime resources can preflight backup integrity and required durable tables read-only before restore design is selected. - Backup publication validates the temporary SQLite copy before atomically replacing the destination. +- Failed backup cleanup preserves the primary backup error and annotates a secondary temporary-file removal failure. - Runtime backups reject missing parent directories, directory destinations, and final-component symlinks before creating a temporary artifact. - Backup preflight rejects operation databases whose schema version is newer than the running foundation. - Runtime configuration and the public server factory validate host, port, database path, and Ingress base path before opening stores or binding a socket. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index d8ddb60..d7c5dd8 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing with a bounded per-connection timeout, a container health probe that follows the configured host (mapping wildcard binds to loopback), port, and Ingress path and validates Symphonia's ready response, bounded response headers and generic JSON parser/method errors without Python version disclosure, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, and temporary-copy validation before publication, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing with a bounded per-connection timeout, a container health probe that follows the configured host (mapping wildcard binds to loopback), port, and Ingress path and validates Symphonia's ready response, bounded response headers and generic JSON parser/method errors without Python version disclosure, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, temporary-copy validation before publication, and primary-error preservation when temporary backup cleanup also fails, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index e065b95..c3fe5db 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -175,12 +175,18 @@ def backup_to(self, destination_path: str) -> None: raise RuntimeError("SQLite backup integrity validation failed") os.replace(temporary_path, destination_path_object) temporary_path = None - finally: + except BaseException as backup_error: if temporary_path is not None: try: os.unlink(temporary_path) except FileNotFoundError: pass + except BaseException as cleanup_error: + backup_error.add_note( + "temporary backup cleanup also failed " + f"({type(cleanup_error).__name__})" + ) + raise @classmethod def validate_backup(cls, backup_path: str) -> bool: From fe352e77055ec6226d75227bf261d272a1859f05 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 10:15:07 +0200 Subject: [PATCH 118/167] fix: handle dropped runtime http clients quietly --- docs/development/implementation-baseline.md | 1 + .../home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/http.py | 24 +++++++++++-------- 3 files changed, 16 insertions(+), 11 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index b30b776..a886522 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -78,6 +78,7 @@ The first implementation increment is intentionally narrower than any provider o - The composed runtime HTTP server has an offline route contract and a loopback smoke path for real health/readiness lifecycle checks. - The current JSON health/readiness/version surface disables caching, MIME sniffing, and referrer propagation. - The HTTP handler suppresses the Python version header and returns generic JSON for parser/method errors, closing the connection without reflecting details. +- Client disconnects or response-write timeouts close the request quietly instead of emitting a server traceback. - The container health probe follows the configured runtime port and Ingress base path. - The container health probe follows the configured listener host, mapping IPv4/IPv6 wildcard binds to their loopback equivalents, and requires the Symphonia ready JSON contract rather than any HTTP 200 response. - The runtime JSON surface rejects non-standard numeric values before writing a response. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index d7c5dd8..55f49c1 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing with a bounded per-connection timeout, a container health probe that follows the configured host (mapping wildcard binds to loopback), port, and Ingress path and validates Symphonia's ready response, bounded response headers and generic JSON parser/method errors without Python version disclosure, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, temporary-copy validation before publication, and primary-error preservation when temporary backup cleanup also fails, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing with a bounded per-connection timeout, a container health probe that follows the configured host (mapping wildcard binds to loopback), port, and Ingress path and validates Symphonia's ready response, bounded response headers and generic JSON parser/method errors without Python version disclosure, quiet handling of client disconnects during response writes, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, temporary-copy validation before publication, and primary-error preservation when temporary backup cleanup also fails, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. diff --git a/src/symphonia/runtime/http.py b/src/symphonia/runtime/http.py index 522c98a..bfd947a 100644 --- a/src/symphonia/runtime/http.py +++ b/src/symphonia/runtime/http.py @@ -96,17 +96,21 @@ def _json( sort_keys=True, allow_nan=False, ).encode("utf-8") - self.send_response(status) - self.send_header("Content-Type", "application/json; charset=utf-8") - self.send_header("Content-Length", str(len(body))) - self.send_header("Cache-Control", "no-store") - self.send_header("X-Content-Type-Options", "nosniff") - self.send_header("Referrer-Policy", "no-referrer") - if close_connection: + try: + self.send_response(status) + self.send_header("Content-Type", "application/json; charset=utf-8") + self.send_header("Content-Length", str(len(body))) + self.send_header("Cache-Control", "no-store") + self.send_header("X-Content-Type-Options", "nosniff") + self.send_header("Referrer-Policy", "no-referrer") + if close_connection: + self.close_connection = True + self.send_header("Connection", "close") + self.end_headers() + self.wfile.write(body) + except (ConnectionError, TimeoutError): + # A client timing out or disconnecting is not a server fault. self.close_connection = True - self.send_header("Connection", "close") - self.end_headers() - self.wfile.write(body) def log_message(self, format: str, *args: object) -> None: # Keep the first runtime quiet; structured logging belongs to the From 2cbb2f8d611b0dc9246fb7e5baa08eaaacb12555 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 14:21:38 +0200 Subject: [PATCH 119/167] fix: reject lossy runtime port coercions --- docs/development/implementation-baseline.md | 1 + src/symphonia/runtime/config.py | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index a886522..3d38510 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -67,6 +67,7 @@ The first implementation increment is intentionally narrower than any provider o - Runtime backups reject missing parent directories, directory destinations, and final-component symlinks before creating a temporary artifact. - Backup preflight rejects operation databases whose schema version is newer than the running foundation. - Runtime configuration and the public server factory validate host, port, database path, and Ingress base path before opening stores or binding a socket. +- Port parsing accepts integer values, numeric strings, and integral floats while rejecting booleans and arbitrary integer-coercible objects that could silently truncate. - Runtime configuration and HTTP routing share one Ingress path normalizer that rejects control characters, query/fragments, backslashes, repeated slashes, and literal dot segments. - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. - Runtime readiness validates SQLite integrity, foreign-key references, durable operation state/event values, authorization attempts, provider capability JSON, plan JSON, and resolution payloads; corruption fails closed. diff --git a/src/symphonia/runtime/config.py b/src/symphonia/runtime/config.py index 900f624..dc93054 100644 --- a/src/symphonia/runtime/config.py +++ b/src/symphonia/runtime/config.py @@ -23,7 +23,7 @@ def normalize_ingress_path(value: object) -> str: def _parse_port(value: object) -> int: - if isinstance(value, bool): + if isinstance(value, bool) or not isinstance(value, (int, float, str)): raise ValueError("port must be an integer") if isinstance(value, float) and not value.is_integer(): raise ValueError("port must be an integer") From 7e23f8fbd67dd41267d6524eebc6c02c4d7f0382 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 14:22:42 +0200 Subject: [PATCH 120/167] fix: reject unicode controls in runtime config --- docs/development/implementation-baseline.md | 2 +- specs/home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/config.py | 15 ++++++++++----- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 3d38510..9e482c3 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -68,7 +68,7 @@ The first implementation increment is intentionally narrower than any provider o - Backup preflight rejects operation databases whose schema version is newer than the running foundation. - Runtime configuration and the public server factory validate host, port, database path, and Ingress base path before opening stores or binding a socket. - Port parsing accepts integer values, numeric strings, and integral floats while rejecting booleans and arbitrary integer-coercible objects that could silently truncate. -- Runtime configuration and HTTP routing share one Ingress path normalizer that rejects control characters, query/fragments, backslashes, repeated slashes, and literal dot segments. +- Runtime host, database, and Ingress values reject Unicode C0/C1 control characters before host whitespace normalization; the shared Ingress normalizer also rejects query/fragments, backslashes, repeated slashes, and literal dot segments. - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. - Runtime readiness validates SQLite integrity, foreign-key references, durable operation state/event values, authorization attempts, provider capability JSON, plan JSON, and resolution payloads; corruption fails closed. - Provider connection diagnostics expose provider/state and expiry counts, never account or credential data. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 55f49c1..888f0ca 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -156,7 +156,7 @@ No final App option names are accepted yet. The first implementation RFC should Invalid or unknown values fail closed before workers start. Authentication, non-root execution, secret redaction, audit history, and mount/network restrictions are not configurable. -The current foundation profile applies this boundary before server creation: host and database values reject control characters, ports must be integral and within the valid TCP range, and the Ingress base path must be a normalized path without query, fragment, control, or traversal segments. These checks are implementation evidence only; final App option names and supported deployment values remain open. +The current foundation profile applies this boundary before server creation: host and database values reject Unicode C0/C1 control characters (before host whitespace normalization), ports must be integral and within the valid TCP range, and the Ingress base path must be a normalized path without query, fragment, control, or traversal segments. These checks are implementation evidence only; final App option names and supported deployment values remain open. ## 8. Clean Architecture design diff --git a/src/symphonia/runtime/config.py b/src/symphonia/runtime/config.py index dc93054..8952d49 100644 --- a/src/symphonia/runtime/config.py +++ b/src/symphonia/runtime/config.py @@ -2,15 +2,20 @@ from __future__ import annotations +from collections.abc import Mapping from dataclasses import dataclass import os -from collections.abc import Mapping +import unicodedata + + +def _has_control_characters(value: str) -> bool: + return any(unicodedata.category(char) == "Cc" for char in value) def normalize_ingress_path(value: object) -> str: if not isinstance(value, str) or not value or not value.startswith("/"): raise ValueError("ingress path must start with '/'") - if any(ord(char) < 0x20 or ord(char) == 0x7F for char in value): + if _has_control_characters(value): raise ValueError("ingress path must not contain control characters") if "?" in value or "#" in value: raise ValueError("ingress path must contain only a path") @@ -48,16 +53,16 @@ class RuntimeConfig: def __post_init__(self) -> None: if not isinstance(self.host, str): raise ValueError("host must be a non-empty value without whitespace") + if _has_control_characters(self.host): + raise ValueError("host must not contain control characters") host = self.host.strip() if not host or any(char.isspace() for char in host): raise ValueError("host must be a non-empty value without whitespace") - if any(ord(char) < 0x20 or ord(char) == 0x7F for char in host): - raise ValueError("host must not contain control characters") object.__setattr__(self, "host", host) object.__setattr__(self, "port", _parse_port(self.port)) if not isinstance(self.database_path, str) or not self.database_path.strip(): raise ValueError("database_path must not be empty") - if any(ord(char) < 0x20 or ord(char) == 0x7F for char in self.database_path): + if _has_control_characters(self.database_path): raise ValueError("database_path must not contain control characters") object.__setattr__(self, "database_path", self.database_path.strip()) object.__setattr__(self, "ingress_path", normalize_ingress_path(self.ingress_path)) From 7bbaf25e445e58460ba0c7e8f615f8d299479400 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 14:23:42 +0200 Subject: [PATCH 121/167] fix: normalize health probe listener host --- Dockerfile | 2 +- docs/development/implementation-baseline.md | 2 +- specs/home-assistant-app-runtime-and-ingress.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Dockerfile b/Dockerfile index 7f1ec46..0c511e0 100644 --- a/Dockerfile +++ b/Dockerfile @@ -20,6 +20,6 @@ VOLUME ["/data"] EXPOSE 8099 HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \ - CMD ["python", "-c", "import http.client,json,os,sys; host=os.getenv('SYMPHONIA_HOST','127.0.0.1'); host={'0.0.0.0':'127.0.0.1','::':'::1'}.get(host,host); port=int(os.getenv('SYMPHONIA_PORT','8099')); base=os.getenv('SYMPHONIA_INGRESS_PATH','/').rstrip('/'); connection=http.client.HTTPConnection(host,port,timeout=2); connection.request('GET',f'{base}/ready'); response=connection.getresponse(); payload=json.loads(response.read()); sys.exit(response.status != 200 or payload.get('service') != 'symphonia' or payload.get('status') != 'ready')"] + CMD ["python", "-c", "import http.client,json,os,sys; host=os.getenv('SYMPHONIA_HOST','127.0.0.1').strip(); host={'0.0.0.0':'127.0.0.1','::':'::1'}.get(host,host); port=int(os.getenv('SYMPHONIA_PORT','8099')); base=os.getenv('SYMPHONIA_INGRESS_PATH','/').rstrip('/'); connection=http.client.HTTPConnection(host,port,timeout=2); connection.request('GET',f'{base}/ready'); response=connection.getresponse(); payload=json.loads(response.read()); sys.exit(response.status != 200 or payload.get('service') != 'symphonia' or payload.get('status') != 'ready')"] ENTRYPOINT ["python", "-m", "symphonia"] diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 9e482c3..9fb54aa 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -81,7 +81,7 @@ The first implementation increment is intentionally narrower than any provider o - The HTTP handler suppresses the Python version header and returns generic JSON for parser/method errors, closing the connection without reflecting details. - Client disconnects or response-write timeouts close the request quietly instead of emitting a server traceback. - The container health probe follows the configured runtime port and Ingress base path. -- The container health probe follows the configured listener host, mapping IPv4/IPv6 wildcard binds to their loopback equivalents, and requires the Symphonia ready JSON contract rather than any HTTP 200 response. +- The container health probe follows the normalized configured listener host, mapping IPv4/IPv6 wildcard binds to their loopback equivalents, and requires the Symphonia ready JSON contract rather than any HTTP 200 response. - The runtime JSON surface rejects non-standard numeric values before writing a response. - Normalized provider contracts and a dependency-free playlist-page collector proving opaque identity namespaces, completeness, pagination safety, duplicate occurrences, and unavailable-item preservation. - Normalized provider identity, manifest, capability, entry, and page values reject non-textual or malformed boundary data before it reaches planning. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 888f0ca..3b5e42f 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -30,7 +30,7 @@ Without a runtime contract, framework selection can accidentally determine authe ### 2.2 Current behavior -An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing with a bounded per-connection timeout, a container health probe that follows the configured host (mapping wildcard binds to loopback), port, and Ingress path and validates Symphonia's ready response, bounded response headers and generic JSON parser/method errors without Python version disclosure, quiet handling of client disconnects during response writes, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, temporary-copy validation before publication, and primary-error preservation when temporary backup cleanup also fails, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. +An owner-approved dependency-free runtime foundation exists, but the complete Supervisor-managed App described here does not. The foundation includes a non-root container metadata scaffold, `/data` persistence, validated process configuration applied before stores open or sockets bind, one shared Ingress path normalizer for configuration and HTTP routing, origin-form-only health/readiness/version routing with a bounded per-connection timeout, a container health probe that follows the normalized configured host (mapping wildcard binds to loopback), port, and Ingress path and validates Symphonia's ready response, bounded response headers and generic JSON parser/method errors without Python version disclosure, quiet handling of client disconnects during response writes, composed SQLite stores with semantic readiness checks and connection cleanup on migration failure, transactionally consistent backup/preflight helpers with future-schema rejection, safe destination-path handling, temporary-copy validation before publication, and primary-error preservation when temporary backup cleanup also fails, interruption-safe reverse-order cleanup after partial startup, `SIGTERM`-driven graceful CLI shutdown, retryable resource shutdown, and deterministic tests. It does not provide the accepted authentication, migration ledger, restore workflow, platform matrix, or complete lifecycle UI required by this SDD. This SDD remains prospective and `Draft`; the foundation evidence below must not be read as implementation readiness or as proof that the complete App capability exists. From 2d4eb7dea9b4c15ed411d5e1b7f56eb3708eb123 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 14:25:04 +0200 Subject: [PATCH 122/167] fix: share sqlite path validation across runtime entrypoints --- docs/development/implementation-baseline.md | 2 +- specs/home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/__init__.py | 4 ++-- src/symphonia/runtime/config.py | 16 ++++++++++------ src/symphonia/runtime/resources.py | 4 ++-- 5 files changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 9fb54aa..9053e80 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -66,7 +66,7 @@ The first implementation increment is intentionally narrower than any provider o - Failed backup cleanup preserves the primary backup error and annotates a secondary temporary-file removal failure. - Runtime backups reject missing parent directories, directory destinations, and final-component symlinks before creating a temporary artifact. - Backup preflight rejects operation databases whose schema version is newer than the running foundation. -- Runtime configuration and the public server factory validate host, port, database path, and Ingress base path before opening stores or binding a socket. +- Runtime configuration and the public server factory validate host, port, database path, and Ingress base path before opening stores or binding a socket; direct resource opening shares the database-path validator. - Port parsing accepts integer values, numeric strings, and integral floats while rejecting booleans and arbitrary integer-coercible objects that could silently truncate. - Runtime host, database, and Ingress values reject Unicode C0/C1 control characters before host whitespace normalization; the shared Ingress normalizer also rejects query/fragments, backslashes, repeated slashes, and literal dot segments. - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index 3b5e42f..a78df0d 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -156,7 +156,7 @@ No final App option names are accepted yet. The first implementation RFC should Invalid or unknown values fail closed before workers start. Authentication, non-root execution, secret redaction, audit history, and mount/network restrictions are not configurable. -The current foundation profile applies this boundary before server creation: host and database values reject Unicode C0/C1 control characters (before host whitespace normalization), ports must be integral and within the valid TCP range, and the Ingress base path must be a normalized path without query, fragment, control, or traversal segments. These checks are implementation evidence only; final App option names and supported deployment values remain open. +The current foundation profile applies this boundary before opening stores or creating a server: host and database values reject Unicode C0/C1 control characters (before host whitespace normalization), ports must be integral and within the valid TCP range, and the Ingress base path must be a normalized path without query, fragment, control, or traversal segments. Runtime configuration and direct resource opening share database-path normalization. These checks are implementation evidence only; final App option names and supported deployment values remain open. ## 8. Clean Architecture design diff --git a/src/symphonia/runtime/__init__.py b/src/symphonia/runtime/__init__.py index 8577ebf..a92d6f2 100644 --- a/src/symphonia/runtime/__init__.py +++ b/src/symphonia/runtime/__init__.py @@ -1,7 +1,7 @@ """Minimal process runtime and health endpoints.""" +from .config import RuntimeConfig, normalize_database_path from .http import create_server -from .config import RuntimeConfig from .resources import RuntimeResources -__all__ = ["RuntimeConfig", "RuntimeResources", "create_server"] +__all__ = ["RuntimeConfig", "RuntimeResources", "create_server", "normalize_database_path"] diff --git a/src/symphonia/runtime/config.py b/src/symphonia/runtime/config.py index 8952d49..59dfb40 100644 --- a/src/symphonia/runtime/config.py +++ b/src/symphonia/runtime/config.py @@ -41,6 +41,14 @@ def _parse_port(value: object) -> int: return port +def normalize_database_path(value: object) -> str: + if not isinstance(value, str) or not value.strip(): + raise ValueError("database_path must not be empty") + if _has_control_characters(value): + raise ValueError("database_path must not contain control characters") + return value.strip() + + @dataclass(frozen=True, slots=True) class RuntimeConfig: """Configuration that is safe to hand to the runtime composition root.""" @@ -60,11 +68,7 @@ def __post_init__(self) -> None: raise ValueError("host must be a non-empty value without whitespace") object.__setattr__(self, "host", host) object.__setattr__(self, "port", _parse_port(self.port)) - if not isinstance(self.database_path, str) or not self.database_path.strip(): - raise ValueError("database_path must not be empty") - if _has_control_characters(self.database_path): - raise ValueError("database_path must not contain control characters") - object.__setattr__(self, "database_path", self.database_path.strip()) + object.__setattr__(self, "database_path", normalize_database_path(self.database_path)) object.__setattr__(self, "ingress_path", normalize_ingress_path(self.ingress_path)) @classmethod @@ -78,4 +82,4 @@ def from_environment(cls, environ: Mapping[str, str] | None = None) -> "RuntimeC ) -__all__ = ["RuntimeConfig", "normalize_ingress_path"] +__all__ = ["RuntimeConfig", "normalize_database_path", "normalize_ingress_path"] diff --git a/src/symphonia/runtime/resources.py b/src/symphonia/runtime/resources.py index c3fe5db..9a9f657 100644 --- a/src/symphonia/runtime/resources.py +++ b/src/symphonia/runtime/resources.py @@ -19,6 +19,7 @@ ProviderConnectionRepository, ResolutionDecisionRepository, ) +from .config import normalize_database_path @dataclass(slots=True) @@ -42,8 +43,7 @@ class RuntimeResources: @classmethod def open(cls, database_path: str) -> "RuntimeResources": - if not isinstance(database_path, str) or not database_path.strip(): - raise ValueError("database_path must not be empty") + database_path = normalize_database_path(database_path) opened: list[object] = [] try: operations = OperationRepository(database_path) From 6f2151e6eacb93f8110f32f6a9b5c77d7d59a26d Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 14:26:39 +0200 Subject: [PATCH 123/167] fix: normalize sqlite home paths consistently --- docs/development/implementation-baseline.md | 2 +- specs/home-assistant-app-runtime-and-ingress.md | 2 +- src/symphonia/runtime/config.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 9053e80..7af260e 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -66,7 +66,7 @@ The first implementation increment is intentionally narrower than any provider o - Failed backup cleanup preserves the primary backup error and annotates a secondary temporary-file removal failure. - Runtime backups reject missing parent directories, directory destinations, and final-component symlinks before creating a temporary artifact. - Backup preflight rejects operation databases whose schema version is newer than the running foundation. -- Runtime configuration and the public server factory validate host, port, database path, and Ingress base path before opening stores or binding a socket; direct resource opening shares the database-path validator. +- Runtime configuration and the public server factory validate host, port, database path, and Ingress base path before opening stores or binding a socket; direct resource opening shares the database-path validator, including consistent `~` expansion for SQLite and backup path comparisons. - Port parsing accepts integer values, numeric strings, and integral floats while rejecting booleans and arbitrary integer-coercible objects that could silently truncate. - Runtime host, database, and Ingress values reject Unicode C0/C1 control characters before host whitespace normalization; the shared Ingress normalizer also rejects query/fragments, backslashes, repeated slashes, and literal dot segments. - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index a78df0d..f1652f9 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -156,7 +156,7 @@ No final App option names are accepted yet. The first implementation RFC should Invalid or unknown values fail closed before workers start. Authentication, non-root execution, secret redaction, audit history, and mount/network restrictions are not configurable. -The current foundation profile applies this boundary before opening stores or creating a server: host and database values reject Unicode C0/C1 control characters (before host whitespace normalization), ports must be integral and within the valid TCP range, and the Ingress base path must be a normalized path without query, fragment, control, or traversal segments. Runtime configuration and direct resource opening share database-path normalization. These checks are implementation evidence only; final App option names and supported deployment values remain open. +The current foundation profile applies this boundary before opening stores or creating a server: host and database values reject Unicode C0/C1 control characters (before host whitespace normalization), ports must be integral and within the valid TCP range, and the Ingress base path must be a normalized path without query, fragment, control, or traversal segments. Runtime configuration and direct resource opening share database-path normalization, including expansion of a leading home-directory marker so SQLite and backup path checks identify the same file. These checks are implementation evidence only; final App option names and supported deployment values remain open. ## 8. Clean Architecture design diff --git a/src/symphonia/runtime/config.py b/src/symphonia/runtime/config.py index 59dfb40..c367f06 100644 --- a/src/symphonia/runtime/config.py +++ b/src/symphonia/runtime/config.py @@ -46,7 +46,7 @@ def normalize_database_path(value: object) -> str: raise ValueError("database_path must not be empty") if _has_control_characters(value): raise ValueError("database_path must not contain control characters") - return value.strip() + return os.path.expanduser(value.strip()) @dataclass(frozen=True, slots=True) From 8f5cdafb9a0f030c18fbf1ad95ca3396f3b95238 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 16:17:01 +0200 Subject: [PATCH 124/167] style: remove trailing blank lines for initial CI check --- .dockerignore | 1 - .gitignore | 1 - docs/decisions/0001-provider-independent-recording-domain.md | 1 - docs/decisions/0002-copy-and-sync-are-distinct.md | 1 - pyproject.toml | 1 - src/symphonia/__init__.py | 1 - src/symphonia/application/provider_connections.py | 1 - src/symphonia/domain/__init__.py | 1 - src/symphonia/identity/models.py | 1 - src/symphonia/providers/capabilities.py | 1 - src/symphonia/providers/registry.py | 1 - 11 files changed, 11 deletions(-) diff --git a/.dockerignore b/.dockerignore index b921a45..a691515 100644 --- a/.dockerignore +++ b/.dockerignore @@ -8,4 +8,3 @@ __pycache__ docs specs tests - diff --git a/.gitignore b/.gitignore index 17e1f5f..b6577ee 100644 --- a/.gitignore +++ b/.gitignore @@ -9,4 +9,3 @@ build/ .venv/ *.sqlite3 *.db - diff --git a/docs/decisions/0001-provider-independent-recording-domain.md b/docs/decisions/0001-provider-independent-recording-domain.md index bb9ea24..d3c03b8 100644 --- a/docs/decisions/0001-provider-independent-recording-domain.md +++ b/docs/decisions/0001-provider-independent-recording-domain.md @@ -50,4 +50,3 @@ Trade-offs: - canonical metadata needs provenance/merge rules; - provider-specific data must be stored alongside, not forced into, the core entity; - migrations may be needed as recording/release/work knowledge improves. - diff --git a/docs/decisions/0002-copy-and-sync-are-distinct.md b/docs/decisions/0002-copy-and-sync-are-distinct.md index d61e816..f279a3a 100644 --- a/docs/decisions/0002-copy-and-sync-are-distinct.md +++ b/docs/decisions/0002-copy-and-sync-are-distinct.md @@ -45,4 +45,3 @@ Trade-offs: - copy history must retain enough source/target evidence to inform future sync design; - users who want ongoing mirroring must wait for a later release; - promoting a copied pair into sync requires an explicit future workflow. - diff --git a/pyproject.toml b/pyproject.toml index 4f14b3c..ff8077e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -19,4 +19,3 @@ where = ["src"] [tool.pytest.ini_options] testpaths = ["tests"] - diff --git a/src/symphonia/__init__.py b/src/symphonia/__init__.py index 2c67b53..cd0b307 100644 --- a/src/symphonia/__init__.py +++ b/src/symphonia/__init__.py @@ -6,4 +6,3 @@ """ __version__ = "0.1.0.dev0" - diff --git a/src/symphonia/application/provider_connections.py b/src/symphonia/application/provider_connections.py index f5401d8..7723508 100644 --- a/src/symphonia/application/provider_connections.py +++ b/src/symphonia/application/provider_connections.py @@ -79,4 +79,3 @@ def probe(self, connection_id: str, *, now: datetime) -> ProviderConnection: def disconnect(self, connection_id: str, *, now: datetime) -> ProviderConnection: return self.connections.disconnect(connection_id, now=now) - diff --git a/src/symphonia/domain/__init__.py b/src/symphonia/domain/__init__.py index cb76b7a..98305b2 100644 --- a/src/symphonia/domain/__init__.py +++ b/src/symphonia/domain/__init__.py @@ -19,4 +19,3 @@ "PlaylistSnapshot", "SourcePlaylistEntry", ] - diff --git a/src/symphonia/identity/models.py b/src/symphonia/identity/models.py index 7f1d78c..59131b0 100644 --- a/src/symphonia/identity/models.py +++ b/src/symphonia/identity/models.py @@ -89,4 +89,3 @@ def __post_init__(self) -> None: raise ValueError("manual decisions require track, actor, and reason") if self.action in {ManualDecisionAction.ACCEPT, ManualDecisionAction.REJECT} and not self.candidate_recording_id: raise ValueError("accept/reject decisions require a candidate recording") - diff --git a/src/symphonia/providers/capabilities.py b/src/symphonia/providers/capabilities.py index 4902f4a..482e41e 100644 --- a/src/symphonia/providers/capabilities.py +++ b/src/symphonia/providers/capabilities.py @@ -45,4 +45,3 @@ def require_capabilities(capabilities: ProviderCapabilities, required: Iterable[ if missing: names = ", ".join(sorted(capability.value for capability in missing)) raise CapabilityError(f"required capabilities are unavailable: {names}") - diff --git a/src/symphonia/providers/registry.py b/src/symphonia/providers/registry.py index ee15741..4bab8b4 100644 --- a/src/symphonia/providers/registry.py +++ b/src/symphonia/providers/registry.py @@ -41,4 +41,3 @@ def get(self, provider: str) -> ProviderAdapter: def manifests(self) -> tuple[ProviderManifest, ...]: return tuple(self._adapters[key].manifest for key in sorted(self._adapters)) - From 7345fec8eafa53151c39dd38d7ce6b3f1e4d5995 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 16:22:10 +0200 Subject: [PATCH 125/167] fix: unwind interrupted operation transactions safely --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- .../infrastructure/sqlite_operations.py | 81 +++++++++++-------- 3 files changed, 50 insertions(+), 34 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 7af260e..88c8b56 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -89,6 +89,7 @@ The first implementation increment is intentionally narrower than any provider o - SQLite storage for immutable copy plans, including durable digest-bound acceptance. - Copy-plan reads and acceptance recompute the digest over execution-relevant content, rejecting tampering even when the stored digest field is unchanged. - The operation store applies its schema DDL, legacy column migration, and `user_version` marker in one transaction. +- Operation-store transactions include transaction start in their protected scope, roll back on process-level interruptions when possible, and preserve the primary error if rollback also fails. - SQLite operation tests exercise real multi-connection races: one operation cannot be claimed twice and concurrent scheduler workers claim distinct queue items. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. - Lossless Unicode-safe identity normalization with explicit version-token and ISRC derived fields; no automatic matching thresholds are assumed. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index ed831f1..1edbe30 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Its readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, and invalid credential-bearing payload keys; real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, and invalid credential-bearing payload keys; real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 74a3fc8..bbeeedf 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -83,6 +83,20 @@ def _load_json_object(value: str, *, label: str) -> dict[str, Any]: return parsed +def _rollback_after_error(connection: sqlite3.Connection, error: BaseException) -> None: + """Release an interrupted transaction without hiding its primary failure.""" + + if not connection.in_transaction: + return + try: + connection.execute("ROLLBACK") + except BaseException as rollback_error: + error.add_note( + "SQLite transaction rollback also failed " + f"({type(rollback_error).__name__})" + ) + + def _validate_payload_keys(payload: Any) -> None: forbidden: list[str] = [] active_containers: set[int] = set() @@ -199,8 +213,8 @@ def _migrate(self) -> None: raise RuntimeError( f"operation store schema {current_version} is newer than supported {self.SCHEMA_VERSION}" ) - self._connection.execute("BEGIN IMMEDIATE") try: + self._connection.execute("BEGIN IMMEDIATE") self._connection.execute( """ CREATE TABLE IF NOT EXISTS operations ( @@ -254,9 +268,8 @@ def _migrate(self) -> None: ) self._connection.execute(f"PRAGMA user_version = {self.SCHEMA_VERSION}") self._connection.execute("COMMIT") - except Exception: - if self._connection.in_transaction: - self._connection.execute("ROLLBACK") + except BaseException as error: + _rollback_after_error(self._connection, error) raise def create( @@ -281,8 +294,8 @@ def create( payload_json = json.dumps( payload, ensure_ascii=False, sort_keys=True, separators=(",", ":"), allow_nan=False ) - self._connection.execute("BEGIN IMMEDIATE") try: + self._connection.execute("BEGIN IMMEDIATE") self._connection.execute( """ INSERT INTO operations ( @@ -301,16 +314,18 @@ def create( created_at=timestamp, ) self._connection.execute("COMMIT") - except sqlite3.IntegrityError: - self._connection.execute("ROLLBACK") + except sqlite3.IntegrityError as error: + _rollback_after_error(self._connection, error) + if self._connection.in_transaction: + raise existing = self._by_idempotency(idempotency_key) if existing is None: raise if existing.operation_type != operation_type or existing.payload != payload: raise IdempotencyConflict("idempotency key is already bound to another operation") return existing - except Exception: - self._connection.execute("ROLLBACK") + except BaseException as error: + _rollback_after_error(self._connection, error) raise return self.get(operation_id) @@ -477,8 +492,8 @@ def claim( raise ValueError("lease_seconds must be positive") now_text = _utc(now) expires_text = _utc(now + timedelta(seconds=lease_seconds)) - self._connection.execute("BEGIN IMMEDIATE") try: + self._connection.execute("BEGIN IMMEDIATE") row = self._connection.execute( "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) ).fetchone() @@ -518,8 +533,8 @@ def claim( created_at=now_text, ) self._connection.execute("COMMIT") - except Exception: - self._connection.execute("ROLLBACK") + except BaseException as error: + _rollback_after_error(self._connection, error) raise return self.get(operation_id) @@ -548,8 +563,8 @@ def claim_next( parameters: tuple[Any, ...] = (now_text, now_text) if operation_type is not None: parameters += (operation_type,) - self._connection.execute("BEGIN IMMEDIATE") try: + self._connection.execute("BEGIN IMMEDIATE") row = self._connection.execute( f""" SELECT * @@ -599,8 +614,8 @@ def claim_next( created_at=now_text, ) self._connection.execute("COMMIT") - except Exception: - self._connection.execute("ROLLBACK") + except BaseException as error: + _rollback_after_error(self._connection, error) raise return self.get(operation_id) @@ -620,8 +635,8 @@ def renew_lease( raise ValueError("lease_seconds must be positive") now_text = _utc(now) expires_text = _utc(now + timedelta(seconds=lease_seconds)) - self._connection.execute("BEGIN IMMEDIATE") try: + self._connection.execute("BEGIN IMMEDIATE") row = self._connection.execute( "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) ).fetchone() @@ -644,8 +659,8 @@ def renew_lease( created_at=now_text, ) self._connection.execute("COMMIT") - except Exception: - self._connection.execute("ROLLBACK") + except BaseException as error: + _rollback_after_error(self._connection, error) raise return self.get(operation_id) @@ -669,8 +684,8 @@ def checkpoint( checkpoint_json = json.dumps( checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":"), allow_nan=False ) - self._connection.execute("BEGIN IMMEDIATE") try: + self._connection.execute("BEGIN IMMEDIATE") row = self._connection.execute( "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) ).fetchone() @@ -709,8 +724,8 @@ def checkpoint( created_at=now_text, ) self._connection.execute("COMMIT") - except Exception: - self._connection.execute("ROLLBACK") + except BaseException as error: + _rollback_after_error(self._connection, error) raise return self.get(operation_id) @@ -718,8 +733,8 @@ def cancel(self, operation_id: str, *, now: datetime) -> OperationRecord: """Request cooperative cancellation and preserve in-flight ownership.""" now_text = _utc(now) - self._connection.execute("BEGIN IMMEDIATE") try: + self._connection.execute("BEGIN IMMEDIATE") row = self._connection.execute( "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) ).fetchone() @@ -760,8 +775,8 @@ def cancel(self, operation_id: str, *, now: datetime) -> OperationRecord: created_at=now_text, ) self._connection.execute("COMMIT") - except Exception: - self._connection.execute("ROLLBACK") + except BaseException as error: + _rollback_after_error(self._connection, error) raise return self.get(operation_id) @@ -769,8 +784,8 @@ def resume(self, operation_id: str, *, now: datetime) -> OperationRecord: """Re-admit a user-action operation after its external issue is resolved.""" now_text = _utc(now) - self._connection.execute("BEGIN IMMEDIATE") try: + self._connection.execute("BEGIN IMMEDIATE") row = self._connection.execute( "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) ).fetchone() @@ -791,8 +806,8 @@ def resume(self, operation_id: str, *, now: datetime) -> OperationRecord: created_at=now_text, ) self._connection.execute("COMMIT") - except Exception: - self._connection.execute("ROLLBACK") + except BaseException as error: + _rollback_after_error(self._connection, error) raise return self.get(operation_id) @@ -815,8 +830,8 @@ def schedule_retry( checkpoint_json = json.dumps( checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":"), allow_nan=False ) - self._connection.execute("BEGIN IMMEDIATE") try: + self._connection.execute("BEGIN IMMEDIATE") row = self._connection.execute( "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) ).fetchone() @@ -863,8 +878,8 @@ def schedule_retry( created_at=now_text, ) self._connection.execute("COMMIT") - except Exception: - self._connection.execute("ROLLBACK") + except BaseException as error: + _rollback_after_error(self._connection, error) raise return self.get(operation_id) @@ -887,8 +902,8 @@ def schedule_rate_limit( checkpoint_json = json.dumps( checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":"), allow_nan=False ) - self._connection.execute("BEGIN IMMEDIATE") try: + self._connection.execute("BEGIN IMMEDIATE") row = self._connection.execute( "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) ).fetchone() @@ -935,8 +950,8 @@ def schedule_rate_limit( created_at=now_text, ) self._connection.execute("COMMIT") - except Exception: - self._connection.execute("ROLLBACK") + except BaseException as error: + _rollback_after_error(self._connection, error) raise return self.get(operation_id) From ff9cd395c653e43e4f0bf66663c234bde2597cf9 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 16:24:17 +0200 Subject: [PATCH 126/167] fix: reject api keys in durable operation payloads --- docs/development/implementation-baseline.md | 2 +- specs/durable-operations-and-recovery.md | 2 +- src/symphonia/infrastructure/sqlite_operations.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 88c8b56..911be16 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -36,7 +36,7 @@ The first implementation increment is intentionally narrower than any provider o - Authorization callback binding rejects unsafe schemes, fragments, credentials, whitespace, and non-loopback HTTP hosts. - Normalized provider errors and provider codes redact common bearer/token/secret/password/cookie forms at the provider boundary, including quoted JSON-like and query-like assignments. - Operation payloads and durable checkpoints recursively reject credential-shaped keys and cyclic structures before SQLite writes. -- Durable operation intents, checkpoints, retries, and rate-limit waits require JSON-object payloads with string keys before SQLite writes. +- Durable operation intents, checkpoints, retries, and rate-limit waits require JSON-object payloads with string keys and reject token-, password-, authorization-, and API-key-shaped fields before SQLite writes. - Durable operation JSON rejects non-finite numbers on write and read, and operation/worker identifiers are validated before lease transitions. - Shared SQLite JSON helpers apply deterministic serialization and reject non-standard numbers in capability and plan payloads as well as operation state. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 1edbe30..b5e258d 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, and invalid credential-bearing payload keys; real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, and invalid credential-bearing payload keys; writes and reads reject token-, password-, authorization-, and API-key-shaped payload fields. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index bbeeedf..b1bda5e 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -42,7 +42,7 @@ class LeaseConflict(RuntimeError): _SECRET_PAYLOAD_KEY = re.compile( - r"(?i)(?:^|[_-])(access[_-]?token|refresh[_-]?token|id[_-]?token|client[_-]?secret|password|cookie|authorization|secret[_-]?token)(?:$|[_-])|^(?:secret|token)$" + r"(?i)(?:^|[_-])(access[_-]?token|refresh[_-]?token|id[_-]?token|client[_-]?secret|api[_-]?key|password|cookie|authorization|secret[_-]?token)(?:$|[_-])|^(?:secret|token)$" ) _MAX_DIAGNOSTIC_OPERATIONS = 100 _MAX_DIAGNOSTIC_EVENTS = 100 From 33e410ef05e4dc1328feaab0a4b0a4175c3e6fb4 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 16:26:25 +0200 Subject: [PATCH 127/167] fix: validate credentials in durable audit events --- docs/development/implementation-baseline.md | 2 +- specs/durable-operations-and-recovery.md | 2 +- .../infrastructure/sqlite_operations.py | 25 +++++++++++-------- 3 files changed, 17 insertions(+), 12 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 911be16..f023cd5 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -36,7 +36,7 @@ The first implementation increment is intentionally narrower than any provider o - Authorization callback binding rejects unsafe schemes, fragments, credentials, whitespace, and non-loopback HTTP hosts. - Normalized provider errors and provider codes redact common bearer/token/secret/password/cookie forms at the provider boundary, including quoted JSON-like and query-like assignments. - Operation payloads and durable checkpoints recursively reject credential-shaped keys and cyclic structures before SQLite writes. -- Durable operation intents, checkpoints, retries, and rate-limit waits require JSON-object payloads with string keys and reject token-, password-, authorization-, and API-key-shaped fields before SQLite writes. +- Durable operation intents, checkpoints, retries, rate-limit waits, and audit events require JSON-object payloads with string keys and reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads. - Durable operation JSON rejects non-finite numbers on write and read, and operation/worker identifiers are validated before lease transitions. - Shared SQLite JSON helpers apply deterministic serialization and reject non-standard numbers in capability and plan payloads as well as operation state. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index b5e258d..536c4ab 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, and invalid credential-bearing payload keys; writes and reads reject token-, password-, authorization-, and API-key-shaped payload fields. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index b1bda5e..4c838fa 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -44,6 +44,7 @@ class LeaseConflict(RuntimeError): _SECRET_PAYLOAD_KEY = re.compile( r"(?i)(?:^|[_-])(access[_-]?token|refresh[_-]?token|id[_-]?token|client[_-]?secret|api[_-]?key|password|cookie|authorization|secret[_-]?token)(?:$|[_-])|^(?:secret|token)$" ) +_CAMEL_CASE_BOUNDARY = re.compile(r"(?<=[a-z0-9])(?=[A-Z])") _MAX_DIAGNOSTIC_OPERATIONS = 100 _MAX_DIAGNOSTIC_EVENTS = 100 _MAX_DIAGNOSTIC_KEYS = 100 @@ -97,7 +98,7 @@ def _rollback_after_error(connection: sqlite3.Connection, error: BaseException) ) -def _validate_payload_keys(payload: Any) -> None: +def _validate_payload_keys(payload: Any, *, label: str = "operation payload") -> None: forbidden: list[str] = [] active_containers: set[int] = set() @@ -105,15 +106,16 @@ def walk(value: Any, path: str = "") -> None: if isinstance(value, Mapping): identity = id(value) if identity in active_containers: - raise ValueError("operation payload must not contain cyclic structures") + raise ValueError(f"{label} must not contain cyclic structures") active_containers.add(identity) try: for key, nested in value.items(): if not isinstance(key, str): - raise ValueError("operation payload object keys must be strings") + raise ValueError(f"{label} object keys must be strings") key_text = str(key) key_path = key_text if not path else f"{path}.{key_text}" - if _SECRET_PAYLOAD_KEY.search(key_text): + normalized_key = _CAMEL_CASE_BOUNDARY.sub("_", key_text) + if _SECRET_PAYLOAD_KEY.search(normalized_key): forbidden.append(key_path) walk(nested, key_path) finally: @@ -122,7 +124,7 @@ def walk(value: Any, path: str = "") -> None: if isinstance(value, (list, tuple)): identity = id(value) if identity in active_containers: - raise ValueError("operation payload must not contain cyclic structures") + raise ValueError(f"{label} must not contain cyclic structures") active_containers.add(identity) try: for index, nested in enumerate(value): @@ -133,7 +135,7 @@ def walk(value: Any, path: str = "") -> None: walk(payload) if forbidden: raise ValueError( - "operation payload contains forbidden credential keys: " + f"{label} contains forbidden credential keys: " + ", ".join(sorted(forbidden)) ) @@ -141,7 +143,7 @@ def walk(value: Any, path: str = "") -> None: def _validate_object_payload(payload: Any, *, label: str) -> None: if not isinstance(payload, dict): raise ValueError(f"{label} must be a JSON object") - _validate_payload_keys(payload) + _validate_payload_keys(payload, label=label) @dataclass(frozen=True, slots=True) @@ -971,6 +973,7 @@ def _append_event( payload: dict[str, Any], created_at: str, ) -> None: + _validate_object_payload(payload, label="operation event payload") self._connection.execute( """ INSERT INTO operation_events ( @@ -1018,8 +1021,8 @@ def _record(row: sqlite3.Row) -> OperationRecord: raise ValueError("operation store contains an invalid cancellation flag") payload = _load_json_object(row["payload_json"], label="operation payload") checkpoint = _load_json_object(row["checkpoint_json"], label="operation checkpoint") - _validate_payload_keys(payload) - _validate_payload_keys(checkpoint) + _validate_payload_keys(payload, label="operation payload") + _validate_payload_keys(checkpoint, label="operation checkpoint") return OperationRecord( operation_id=row["operation_id"], operation_type=row["operation_type"], @@ -1040,12 +1043,14 @@ def _event(row: sqlite3.Row) -> OperationEvent: state = row["state"] if state not in _OPERATION_STATES: raise ValueError(f"operation event contains invalid state: {state!r}") + payload = _load_json_object(row["payload_json"], label="operation event payload") + _validate_payload_keys(payload, label="operation event payload") return OperationEvent( sequence=row["sequence"], operation_id=row["operation_id"], event_type=row["event_type"], state=state, worker_id=row["worker_id"], - payload=_load_json_object(row["payload_json"], label="operation event payload"), + payload=payload, created_at=_parse_utc(row["created_at"]), ) From c607edacd68c7884c1e8c49797c15c00a8165e29 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 16:27:29 +0200 Subject: [PATCH 128/167] fix: avoid echoing sensitive operation keys --- docs/development/implementation-baseline.md | 2 +- specs/durable-operations-and-recovery.md | 2 +- .../infrastructure/sqlite_operations.py | 21 ++++++++----------- 3 files changed, 11 insertions(+), 14 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index f023cd5..6b2137d 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -36,7 +36,7 @@ The first implementation increment is intentionally narrower than any provider o - Authorization callback binding rejects unsafe schemes, fragments, credentials, whitespace, and non-loopback HTTP hosts. - Normalized provider errors and provider codes redact common bearer/token/secret/password/cookie forms at the provider boundary, including quoted JSON-like and query-like assignments. - Operation payloads and durable checkpoints recursively reject credential-shaped keys and cyclic structures before SQLite writes. -- Durable operation intents, checkpoints, retries, rate-limit waits, and audit events require JSON-object payloads with string keys and reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads. +- Durable operation intents, checkpoints, retries, rate-limit waits, and audit events require JSON-object payloads with string keys and reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, without echoing rejected key names in errors. - Durable operation JSON rejects non-finite numbers on write and read, and operation/worker identifiers are validated before lease transitions. - Shared SQLite JSON helpers apply deterministic serialization and reject non-standard numbers in capability and plan payloads as well as operation state. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 536c4ab..c5738b4 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 4c838fa..32d6354 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -99,10 +99,11 @@ def _rollback_after_error(connection: sqlite3.Connection, error: BaseException) def _validate_payload_keys(payload: Any, *, label: str = "operation payload") -> None: - forbidden: list[str] = [] + forbidden_key_found = False active_containers: set[int] = set() - def walk(value: Any, path: str = "") -> None: + def walk(value: Any) -> None: + nonlocal forbidden_key_found if isinstance(value, Mapping): identity = id(value) if identity in active_containers: @@ -113,11 +114,10 @@ def walk(value: Any, path: str = "") -> None: if not isinstance(key, str): raise ValueError(f"{label} object keys must be strings") key_text = str(key) - key_path = key_text if not path else f"{path}.{key_text}" normalized_key = _CAMEL_CASE_BOUNDARY.sub("_", key_text) if _SECRET_PAYLOAD_KEY.search(normalized_key): - forbidden.append(key_path) - walk(nested, key_path) + forbidden_key_found = True + walk(nested) finally: active_containers.remove(identity) return @@ -127,17 +127,14 @@ def walk(value: Any, path: str = "") -> None: raise ValueError(f"{label} must not contain cyclic structures") active_containers.add(identity) try: - for index, nested in enumerate(value): - walk(nested, f"{path}[{index}]") + for nested in value: + walk(nested) finally: active_containers.remove(identity) walk(payload) - if forbidden: - raise ValueError( - f"{label} contains forbidden credential keys: " - + ", ".join(sorted(forbidden)) - ) + if forbidden_key_found: + raise ValueError(f"{label} contains credential-shaped keys") def _validate_object_payload(payload: Any, *, label: str) -> None: From 427ab8edfa495cc476162fba6486619e60d2b8cf Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 16:28:55 +0200 Subject: [PATCH 129/167] fix: reject naive operation timestamps --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- src/symphonia/infrastructure/sqlite_operations.py | 5 ++++- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 6b2137d..da2a16e 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -38,6 +38,7 @@ The first implementation increment is intentionally narrower than any provider o - Operation payloads and durable checkpoints recursively reject credential-shaped keys and cyclic structures before SQLite writes. - Durable operation intents, checkpoints, retries, rate-limit waits, and audit events require JSON-object payloads with string keys and reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, without echoing rejected key names in errors. - Durable operation JSON rejects non-finite numbers on write and read, and operation/worker identifiers are validated before lease transitions. +- Persisted operation and audit-event timestamps must include a timezone; readiness fails closed instead of interpreting naive timestamps in the host's local zone. - Shared SQLite JSON helpers apply deterministic serialization and reject non-standard numbers in capability and plan payloads as well as operation state. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. - Copy plans bind target connection and effective write-capability evidence into their digest. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index c5738b4..79b6c63 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, timezone-less operation/event timestamps, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 32d6354..b37865b 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -26,7 +26,10 @@ def _utc(value: datetime) -> str: def _parse_utc(value: str) -> datetime: - return datetime.fromisoformat(value).astimezone(timezone.utc) + parsed = datetime.fromisoformat(value) + if parsed.tzinfo is None or parsed.utcoffset() is None: + raise ValueError("timestamps must be timezone-aware") + return parsed.astimezone(timezone.utc) class OperationNotFound(LookupError): From a902f905937fdda0ddc297927af341062646a397 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 16:29:56 +0200 Subject: [PATCH 130/167] fix: validate durable lease durations strictly --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- src/symphonia/infrastructure/sqlite_operations.py | 15 +++++++++------ 3 files changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index da2a16e..3376780 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -39,6 +39,7 @@ The first implementation increment is intentionally narrower than any provider o - Durable operation intents, checkpoints, retries, rate-limit waits, and audit events require JSON-object payloads with string keys and reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, without echoing rejected key names in errors. - Durable operation JSON rejects non-finite numbers on write and read, and operation/worker identifiers are validated before lease transitions. - Persisted operation and audit-event timestamps must include a timezone; readiness fails closed instead of interpreting naive timestamps in the host's local zone. +- Lease durations passed to the durable operation repository must be positive integers; booleans and lossy/coercible values are rejected consistently with the worker boundary. - Shared SQLite JSON helpers apply deterministic serialization and reject non-standard numbers in capability and plan payloads as well as operation state. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. - Copy plans bind target connection and effective write-capability evidence into their digest. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 79b6c63..fe1c938 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, timezone-less operation/event timestamps, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, timezone-less operation/event timestamps, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index b37865b..1be9702 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -72,6 +72,12 @@ def _require_text(value: Any, *, label: str) -> str: return value +def _require_positive_int(value: Any, *, label: str) -> int: + if isinstance(value, bool) or not isinstance(value, int) or value <= 0: + raise ValueError(f"{label} must be a positive integer") + return value + + def _reject_non_finite_json(value: str) -> None: raise ValueError(f"non-standard JSON constant is not allowed: {value}") @@ -490,8 +496,7 @@ def claim( _require_text(operation_id, label="operation_id") _require_text(worker_id, label="worker_id") - if lease_seconds <= 0: - raise ValueError("lease_seconds must be positive") + _require_positive_int(lease_seconds, label="lease_seconds") now_text = _utc(now) expires_text = _utc(now + timedelta(seconds=lease_seconds)) try: @@ -557,8 +562,7 @@ def claim_next( _require_text(worker_id, label="worker_id") if operation_type is not None: _require_text(operation_type, label="operation_type") - if lease_seconds <= 0: - raise ValueError("lease_seconds must be positive") + _require_positive_int(lease_seconds, label="lease_seconds") now_text = _utc(now) expires_text = _utc(now + timedelta(seconds=lease_seconds)) type_clause = " AND operation_type = ?" if operation_type is not None else "" @@ -633,8 +637,7 @@ def renew_lease( _require_text(operation_id, label="operation_id") _require_text(worker_id, label="worker_id") - if lease_seconds <= 0: - raise ValueError("lease_seconds must be positive") + _require_positive_int(lease_seconds, label="lease_seconds") now_text = _utc(now) expires_text = _utc(now + timedelta(seconds=lease_seconds)) try: From 6fa762824a40df82367dcc558e405d1c7a55b786 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 16:30:41 +0200 Subject: [PATCH 131/167] perf: stream durable operation readiness checks --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- src/symphonia/infrastructure/sqlite_operations.py | 4 ++-- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 3376780..42b53c7 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -73,6 +73,7 @@ The first implementation increment is intentionally narrower than any provider o - Runtime host, database, and Ingress values reject Unicode C0/C1 control characters before host whitespace normalization; the shared Ingress normalizer also rejects query/fragments, backslashes, repeated slashes, and literal dot segments. - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. - Runtime readiness validates SQLite integrity, foreign-key references, durable operation state/event values, authorization attempts, provider capability JSON, plan JSON, and resolution payloads; corruption fails closed. +- Durable operation readiness validates operation and event rows incrementally instead of materializing the full ledger in memory. - Provider connection diagnostics expose provider/state and expiry counts, never account or credential data. - Import diagnostics expose bounded snapshot freshness and availability counts without playlist content. - Resolution diagnostics expose only manual-decision action counts, never track, actor, or reason data. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index fe1c938..9838017 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness checks also fail closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, timezone-less operation/event timestamps, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, timezone-less operation/event timestamps, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 1be9702..5a544e8 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -207,9 +207,9 @@ def healthcheck(self) -> bool: return False if self._connection.execute("PRAGMA foreign_key_check").fetchone() is not None: return False - for row in self._connection.execute("SELECT * FROM operations").fetchall(): + for row in self._connection.execute("SELECT * FROM operations"): self._record(row) - for row in self._connection.execute("SELECT * FROM operation_events").fetchall(): + for row in self._connection.execute("SELECT * FROM operation_events"): self._event(row) return True except (sqlite3.Error, TypeError, ValueError): From c19d73329ebfb2c1e3be389bbca263bc33277ab7 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 18:35:21 +0200 Subject: [PATCH 132/167] fix: serialize shared operation repository access --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- .../infrastructure/sqlite_operations.py | 48 ++++++++++++++++++- 3 files changed, 48 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 42b53c7..ac6be27 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -93,6 +93,7 @@ The first implementation increment is intentionally narrower than any provider o - Copy-plan reads and acceptance recompute the digest over execution-relevant content, rejecting tampering even when the stored digest field is unchanged. - The operation store applies its schema DDL, legacy column migration, and `user_version` marker in one transaction. - Operation-store transactions include transaction start in their protected scope, roll back on process-level interruptions when possible, and preserve the primary error if rollback also fails. +- The durable operation repository serializes complete calls on its shared SQLite connection; cross-connection and cross-process correctness remains enforced by SQLite transactions and leases. - SQLite operation tests exercise real multi-connection races: one operation cannot be claimed twice and concurrent scheduler workers claim distinct queue items. - Versioned identity assessments and append-only SQLite storage for manual resolution decisions. - Lossless Unicode-safe identity normalization with explicit version-token and ISRC derived fields; no automatic matching thresholds are assumed. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 9838017..fab2409 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, timezone-less operation/event timestamps, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, timezone-less operation/event timestamps, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 5a544e8..caa573f 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -7,13 +7,15 @@ from __future__ import annotations -from collections.abc import Mapping +from collections.abc import Callable, Mapping from dataclasses import dataclass from datetime import datetime, timedelta, timezone +from functools import wraps import json import re import sqlite3 -from typing import Any +from threading import RLock +from typing import Any, Concatenate, ParamSpec, TypeVar import uuid from .sqlite_common import connect, initialize_with_cleanup @@ -107,6 +109,27 @@ def _rollback_after_error(connection: sqlite3.Connection, error: BaseException) ) +_RepositoryArgs = ParamSpec("_RepositoryArgs") +_RepositoryResult = TypeVar("_RepositoryResult") + + +def _serialize_repository_access( + method: Callable[Concatenate[Any, _RepositoryArgs], _RepositoryResult], +) -> Callable[Concatenate[Any, _RepositoryArgs], _RepositoryResult]: + """Keep each use of a shared SQLite connection within one local critical section.""" + + @wraps(method) + def wrapped( + self: Any, + *args: _RepositoryArgs.args, + **kwargs: _RepositoryArgs.kwargs, + ) -> _RepositoryResult: + with self._connection_lock: + return method(self, *args, **kwargs) + + return wrapped + + def _validate_payload_keys(payload: Any, *, label: str = "operation payload") -> None: forbidden_key_found = False active_containers: set[int] = set() @@ -187,17 +210,21 @@ class OperationRepository: SCHEMA_VERSION = 3 def __init__(self, path: str = ":memory:") -> None: + self._connection_lock = RLock() self._connection = connect(path) initialize_with_cleanup(self._connection, self._migrate) + @_serialize_repository_access def close(self) -> None: self._connection.close() + @_serialize_repository_access def backup_to(self, destination: sqlite3.Connection) -> None: """Copy this store's consistent SQLite snapshot to a destination.""" self._connection.backup(destination) + @_serialize_repository_access def healthcheck(self) -> bool: """Return whether schema and durable operation values are readable.""" @@ -215,6 +242,7 @@ def healthcheck(self) -> bool: except (sqlite3.Error, TypeError, ValueError): return False + @_serialize_repository_access def _migrate(self) -> None: current_version = int(self._connection.execute("PRAGMA user_version").fetchone()[0]) if current_version > self.SCHEMA_VERSION: @@ -280,6 +308,7 @@ def _migrate(self) -> None: _rollback_after_error(self._connection, error) raise + @_serialize_repository_access def create( self, *, @@ -337,6 +366,7 @@ def create( raise return self.get(operation_id) + @_serialize_repository_access def get(self, operation_id: str) -> OperationRecord: row = self._connection.execute( "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) @@ -345,6 +375,7 @@ def get(self, operation_id: str) -> OperationRecord: raise OperationNotFound(operation_id) return self._record(row) + @_serialize_repository_access def events(self, operation_id: str) -> tuple[OperationEvent, ...]: """Return the immutable audit trail in transition order.""" @@ -360,6 +391,7 @@ def events(self, operation_id: str) -> tuple[OperationEvent, ...]: ).fetchall() return tuple(self._event(row) for row in rows) + @_serialize_repository_access def diagnostic(self, operation_id: str, *, event_limit: int = 100) -> dict[str, Any]: """Return a bounded, redacted support view of one operation.""" @@ -407,6 +439,7 @@ def diagnostic(self, operation_id: str, *, event_limit: int = 100) -> dict[str, ], } + @_serialize_repository_access def diagnostics(self, *, limit: int = 50, event_limit: int = 20) -> tuple[dict[str, Any], ...]: """Return a bounded list of redacted operation support views.""" @@ -425,6 +458,7 @@ def diagnostics(self, *, limit: int = 50, event_limit: int = 20) -> tuple[dict[s ).fetchall() return tuple(self.diagnostic(row["operation_id"], event_limit=event_limit) for row in rows) + @_serialize_repository_access def queue_summary(self, *, now: datetime) -> dict[str, Any]: """Return aggregate queue health without exposing operation payloads.""" @@ -484,6 +518,7 @@ def queue_summary(self, *, now: datetime) -> dict[str, Any]: "cancellation_requested_count": int(cancellation_rows["count"]), } + @_serialize_repository_access def claim( self, operation_id: str, @@ -545,6 +580,7 @@ def claim( raise return self.get(operation_id) + @_serialize_repository_access def claim_next( self, *, @@ -625,6 +661,7 @@ def claim_next( raise return self.get(operation_id) + @_serialize_repository_access def renew_lease( self, operation_id: str, @@ -669,6 +706,7 @@ def renew_lease( raise return self.get(operation_id) + @_serialize_repository_access def checkpoint( self, operation_id: str, @@ -734,6 +772,7 @@ def checkpoint( raise return self.get(operation_id) + @_serialize_repository_access def cancel(self, operation_id: str, *, now: datetime) -> OperationRecord: """Request cooperative cancellation and preserve in-flight ownership.""" @@ -785,6 +824,7 @@ def cancel(self, operation_id: str, *, now: datetime) -> OperationRecord: raise return self.get(operation_id) + @_serialize_repository_access def resume(self, operation_id: str, *, now: datetime) -> OperationRecord: """Re-admit a user-action operation after its external issue is resolved.""" @@ -816,6 +856,7 @@ def resume(self, operation_id: str, *, now: datetime) -> OperationRecord: raise return self.get(operation_id) + @_serialize_repository_access def schedule_retry( self, operation_id: str, @@ -888,6 +929,7 @@ def schedule_retry( raise return self.get(operation_id) + @_serialize_repository_access def schedule_rate_limit( self, operation_id: str, @@ -960,12 +1002,14 @@ def schedule_rate_limit( raise return self.get(operation_id) + @_serialize_repository_access def _by_idempotency(self, idempotency_key: str) -> OperationRecord | None: row = self._connection.execute( "SELECT * FROM operations WHERE idempotency_key = ?", (idempotency_key,) ).fetchone() return None if row is None else self._record(row) + @_serialize_repository_access def _append_event( self, *, From 94f94ba5cbfb49b36e5171c3dcdf6a7ebe4e6d9e Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 18:36:57 +0200 Subject: [PATCH 133/167] fix: reject timestamps without a defined offset --- docs/development/implementation-baseline.md | 2 +- specs/durable-operations-and-recovery.md | 2 +- src/symphonia/infrastructure/sqlite_operations.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index ac6be27..b2ea5fb 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -38,7 +38,7 @@ The first implementation increment is intentionally narrower than any provider o - Operation payloads and durable checkpoints recursively reject credential-shaped keys and cyclic structures before SQLite writes. - Durable operation intents, checkpoints, retries, rate-limit waits, and audit events require JSON-object payloads with string keys and reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, without echoing rejected key names in errors. - Durable operation JSON rejects non-finite numbers on write and read, and operation/worker identifiers are validated before lease transitions. -- Persisted operation and audit-event timestamps must include a timezone; readiness fails closed instead of interpreting naive timestamps in the host's local zone. +- Operation and audit-event timestamps must have a defined UTC offset on write and read; the repository never interprets naive timestamps using the host's local zone. - Lease durations passed to the durable operation repository must be positive integers; booleans and lossy/coercible values are rejected consistently with the worker boundary. - Shared SQLite JSON helpers apply deterministic serialization and reject non-standard numbers in capability and plan payloads as well as operation state. - Adapter-backed playlist import orchestration that preserves normalized pagination/completeness guarantees. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index fab2409..f3a7efe 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, timezone-less operation/event timestamps, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, timestamps without a defined offset, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index caa573f..86fa066 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -22,7 +22,7 @@ def _utc(value: datetime) -> str: - if value.tzinfo is None: + if value.tzinfo is None or value.utcoffset() is None: raise ValueError("timestamps must be timezone-aware") return value.astimezone(timezone.utc).isoformat(timespec="microseconds") From f26c3f4c29954d78d92ce2746d8ecbe4fd61f7ac Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 18:38:42 +0200 Subject: [PATCH 134/167] fix: return persisted operation after handler dispatch --- docs/development/implementation-baseline.md | 2 +- specs/durable-operations-and-recovery.md | 2 +- src/symphonia/application/operation_runner.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index b2ea5fb..8daf7b5 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -24,7 +24,7 @@ The first implementation increment is intentionally narrower than any provider o - Application connection service for verified-account registration, capability probes, degraded health, and reauthorization-required classification. - Pure capability-layer intersection and requirement checks for adapter/connection/object/health constraints. - Durable operation runner that atomically claims eligible work and fails unwired operation types before side effects. -- The operation runner validates handler result type and operation identity before returning a claimed result. +- The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a potentially stale handler object. - Cooperative single-process operation worker with interruptible polling and injected clock support. - Copy executor can run as a claimed operation handler, preserving the same restart/checkpoint semantics under the runner. - Playlist import executor persists intent before reads and reports succeeded, partial, waiting-user, retry, and rate-limit outcomes durably. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index f3a7efe..594fa56 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, timestamps without a defined offset, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a possibly stale handler object. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, timestamps without a defined offset, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/application/operation_runner.py b/src/symphonia/application/operation_runner.py index 836a315..b2553f2 100644 --- a/src/symphonia/application/operation_runner.py +++ b/src/symphonia/application/operation_runner.py @@ -48,7 +48,7 @@ def run_once( raise TypeError("operation handler must return OperationRecord") if result.operation_id != operation.operation_id: raise ValueError("operation handler returned a different operation") - return result + return self.operations.get(operation.operation_id) except Exception as error: # A handler must never strand a claimed operation in ``running``. # Persist only a stable exception class marker: provider details From 14d19c0a3f0b5c86fea40fdab0ca2fd01a166592 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 18:43:05 +0200 Subject: [PATCH 135/167] fix: validate operation ids at repository boundaries --- docs/development/implementation-baseline.md | 2 +- specs/durable-operations-and-recovery.md | 2 +- src/symphonia/infrastructure/sqlite_operations.py | 5 +++++ 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 8daf7b5..61de2ce 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -37,7 +37,7 @@ The first implementation increment is intentionally narrower than any provider o - Normalized provider errors and provider codes redact common bearer/token/secret/password/cookie forms at the provider boundary, including quoted JSON-like and query-like assignments. - Operation payloads and durable checkpoints recursively reject credential-shaped keys and cyclic structures before SQLite writes. - Durable operation intents, checkpoints, retries, rate-limit waits, and audit events require JSON-object payloads with string keys and reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, without echoing rejected key names in errors. -- Durable operation JSON rejects non-finite numbers on write and read, and operation/worker identifiers are validated before lease transitions. +- Durable operation JSON rejects non-finite numbers on write and read, and repository entrypoints validate operation/worker identifiers before reads and state transitions. - Operation and audit-event timestamps must have a defined UTC offset on write and read; the repository never interprets naive timestamps using the host's local zone. - Lease durations passed to the durable operation repository must be positive integers; booleans and lossy/coercible values are rejected consistently with the worker boundary. - Shared SQLite JSON helpers apply deterministic serialization and reject non-standard numbers in capability and plan payloads as well as operation state. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 594fa56..a920269 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a possibly stale handler object. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, timestamps without a defined offset, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Repository entrypoints validate non-empty operation IDs before reads and state transitions. The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a possibly stale handler object. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, timestamps without a defined offset, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 86fa066..3a1963a 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -368,6 +368,7 @@ def create( @_serialize_repository_access def get(self, operation_id: str) -> OperationRecord: + _require_text(operation_id, label="operation_id") row = self._connection.execute( "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) ).fetchone() @@ -379,6 +380,7 @@ def get(self, operation_id: str) -> OperationRecord: def events(self, operation_id: str) -> tuple[OperationEvent, ...]: """Return the immutable audit trail in transition order.""" + _require_text(operation_id, label="operation_id") self.get(operation_id) rows = self._connection.execute( """ @@ -395,6 +397,7 @@ def events(self, operation_id: str) -> tuple[OperationEvent, ...]: def diagnostic(self, operation_id: str, *, event_limit: int = 100) -> dict[str, Any]: """Return a bounded, redacted support view of one operation.""" + _require_text(operation_id, label="operation_id") if not 0 < event_limit <= _MAX_DIAGNOSTIC_EVENTS: raise ValueError(f"event_limit must be between 1 and {_MAX_DIAGNOSTIC_EVENTS}") record = self.get(operation_id) @@ -776,6 +779,7 @@ def checkpoint( def cancel(self, operation_id: str, *, now: datetime) -> OperationRecord: """Request cooperative cancellation and preserve in-flight ownership.""" + _require_text(operation_id, label="operation_id") now_text = _utc(now) try: self._connection.execute("BEGIN IMMEDIATE") @@ -828,6 +832,7 @@ def cancel(self, operation_id: str, *, now: datetime) -> OperationRecord: def resume(self, operation_id: str, *, now: datetime) -> OperationRecord: """Re-admit a user-action operation after its external issue is resolved.""" + _require_text(operation_id, label="operation_id") now_text = _utc(now) try: self._connection.execute("BEGIN IMMEDIATE") From 251014a6cecb5eed8810932980c54740b22d7d71 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 18:44:11 +0200 Subject: [PATCH 136/167] fix: fail readiness closed on pathological records --- docs/development/implementation-baseline.md | 2 +- specs/durable-operations-and-recovery.md | 2 +- src/symphonia/infrastructure/sqlite_operations.py | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 61de2ce..89fb7de 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -72,7 +72,7 @@ The first implementation increment is intentionally narrower than any provider o - Port parsing accepts integer values, numeric strings, and integral floats while rejecting booleans and arbitrary integer-coercible objects that could silently truncate. - Runtime host, database, and Ingress values reject Unicode C0/C1 control characters before host whitespace normalization; the shared Ingress normalizer also rejects query/fragments, backslashes, repeated slashes, and literal dot segments. - Runtime resources provide a bounded, payload-free aggregate diagnostics view for future authenticated surfaces. -- Runtime readiness validates SQLite integrity, foreign-key references, durable operation state/event values, authorization attempts, provider capability JSON, plan JSON, and resolution payloads; corruption fails closed. +- Runtime readiness validates SQLite integrity, foreign-key references, durable operation state/event values, authorization attempts, provider capability JSON, plan JSON, and resolution payloads; malformed, out-of-range, or excessively nested durable values fail closed. - Durable operation readiness validates operation and event rows incrementally instead of materializing the full ledger in memory. - Provider connection diagnostics expose provider/state and expiry counts, never account or credential data. - Import diagnostics expose bounded snapshot freshness and availability counts without playlist content. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index a920269..651dd17 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Repository entrypoints validate non-empty operation IDs before reads and state transitions. The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a possibly stale handler object. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed JSON objects, timestamps without a defined offset, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Repository entrypoints validate non-empty operation IDs before reads and state transitions. The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a possibly stale handler object. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed or excessively nested JSON objects, timestamps without a defined or representable UTC offset, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 3a1963a..6953a01 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -239,7 +239,7 @@ def healthcheck(self) -> bool: for row in self._connection.execute("SELECT * FROM operation_events"): self._event(row) return True - except (sqlite3.Error, TypeError, ValueError): + except (sqlite3.Error, TypeError, ValueError, OverflowError, RecursionError): return False @_serialize_repository_access From 2b51a1aeaaaeff3c6b9eaea6ab3b0397911de03a Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 18:45:37 +0200 Subject: [PATCH 137/167] fix: validate checkpoint state types --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- src/symphonia/infrastructure/sqlite_operations.py | 9 ++++++++- 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 89fb7de..952f537 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -38,6 +38,7 @@ The first implementation increment is intentionally narrower than any provider o - Operation payloads and durable checkpoints recursively reject credential-shaped keys and cyclic structures before SQLite writes. - Durable operation intents, checkpoints, retries, rate-limit waits, and audit events require JSON-object payloads with string keys and reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, without echoing rejected key names in errors. - Durable operation JSON rejects non-finite numbers on write and read, and repository entrypoints validate operation/worker identifiers before reads and state transitions. +- Checkpoint transitions reject non-string and unknown state values before opening a transaction. - Operation and audit-event timestamps must have a defined UTC offset on write and read; the repository never interprets naive timestamps using the host's local zone. - Lease durations passed to the durable operation repository must be positive integers; booleans and lossy/coercible values are rejected consistently with the worker boundary. - Shared SQLite JSON helpers apply deterministic serialization and reject non-standard numbers in capability and plan payloads as well as operation state. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 651dd17..2d4a018 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Repository entrypoints validate non-empty operation IDs before reads and state transitions. The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a possibly stale handler object. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed or excessively nested JSON objects, timestamps without a defined or representable UTC offset, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints with strict state validation, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Repository entrypoints validate non-empty operation IDs before reads and state transitions. The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a possibly stale handler object. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed or excessively nested JSON objects, timestamps without a defined or representable UTC offset, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 6953a01..baf2665 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -723,7 +723,14 @@ def checkpoint( _require_text(operation_id, label="operation_id") _require_text(worker_id, label="worker_id") - if state not in {"running", "succeeded", "partial", "failed", "cancelled", "waiting_user"}: + if not isinstance(state, str) or state not in { + "running", + "succeeded", + "partial", + "failed", + "cancelled", + "waiting_user", + }: raise ValueError("invalid checkpoint state") _validate_object_payload(checkpoint, label="checkpoint") now_text = _utc(now) From fb897881eca44c769c9bd52827f727aa52a5267d Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 20:18:45 +0200 Subject: [PATCH 138/167] fix: validate diagnostic query limits --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- .../infrastructure/sqlite_operations.py | 19 +++++++++++++------ 3 files changed, 15 insertions(+), 7 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 952f537..ddd50be 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -13,6 +13,7 @@ The first implementation increment is intentionally narrower than any provider o - Atomic scheduler-facing claim selection for queued, due-retry, and expired-lease operations. - Append-only SQLite operation event history for auditable transitions with sanitized checkpoint summaries. - Bounded redacted operation diagnostics for support and a future authenticated operations view, with hard item/event caps. +- Diagnostic operation/event limits require bounded positive integers and reject booleans and fractional values. - Diagnostic payload/checkpoint key lists are capped at 100 entries and mark truncation. - Diagnostic event queries fetch only the requested window plus one row to determine truncation. - Bounded recent-operation diagnostics listing that exposes only redacted support views. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 2d4a018..c1fbeb6 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints with strict state validation, retries, cancellation, recovery events, bounded diagnostics with capped key lists and bounded event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Repository entrypoints validate non-empty operation IDs before reads and state transitions. The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a possibly stale handler object. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed or excessively nested JSON objects, timestamps without a defined or representable UTC offset, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints with strict state validation, retries, cancellation, recovery events, bounded diagnostics with strict integer limits and capped key lists/event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Repository entrypoints validate non-empty operation IDs before reads and state transitions. The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a possibly stale handler object. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed or excessively nested JSON objects, timestamps without a defined or representable UTC offset, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index baf2665..3e20e14 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -80,6 +80,12 @@ def _require_positive_int(value: Any, *, label: str) -> int: return value +def _require_bounded_int(value: Any, *, label: str, maximum: int) -> int: + if isinstance(value, bool) or not isinstance(value, int) or not 1 <= value <= maximum: + raise ValueError(f"{label} must be an integer between 1 and {maximum}") + return value + + def _reject_non_finite_json(value: str) -> None: raise ValueError(f"non-standard JSON constant is not allowed: {value}") @@ -398,8 +404,9 @@ def diagnostic(self, operation_id: str, *, event_limit: int = 100) -> dict[str, """Return a bounded, redacted support view of one operation.""" _require_text(operation_id, label="operation_id") - if not 0 < event_limit <= _MAX_DIAGNOSTIC_EVENTS: - raise ValueError(f"event_limit must be between 1 and {_MAX_DIAGNOSTIC_EVENTS}") + _require_bounded_int( + event_limit, label="event_limit", maximum=_MAX_DIAGNOSTIC_EVENTS + ) record = self.get(operation_id) event_rows = self._connection.execute( """ @@ -446,10 +453,10 @@ def diagnostic(self, operation_id: str, *, event_limit: int = 100) -> dict[str, def diagnostics(self, *, limit: int = 50, event_limit: int = 20) -> tuple[dict[str, Any], ...]: """Return a bounded list of redacted operation support views.""" - if not 0 < limit <= _MAX_DIAGNOSTIC_OPERATIONS: - raise ValueError(f"limit must be between 1 and {_MAX_DIAGNOSTIC_OPERATIONS}") - if not 0 < event_limit <= _MAX_DIAGNOSTIC_EVENTS: - raise ValueError(f"event_limit must be between 1 and {_MAX_DIAGNOSTIC_EVENTS}") + _require_bounded_int(limit, label="limit", maximum=_MAX_DIAGNOSTIC_OPERATIONS) + _require_bounded_int( + event_limit, label="event_limit", maximum=_MAX_DIAGNOSTIC_EVENTS + ) rows = self._connection.execute( """ SELECT operation_id From f086eb8bfcd85bc8b6c743ef9306904a104d2766 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 20:19:35 +0200 Subject: [PATCH 139/167] perf: bound diagnostic key selection memory --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- src/symphonia/infrastructure/sqlite_operations.py | 3 ++- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index ddd50be..58b2496 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -15,6 +15,7 @@ The first implementation increment is intentionally narrower than any provider o - Bounded redacted operation diagnostics for support and a future authenticated operations view, with hard item/event caps. - Diagnostic operation/event limits require bounded positive integers and reject booleans and fractional values. - Diagnostic payload/checkpoint key lists are capped at 100 entries and mark truncation. +- Diagnostic key summaries select their bounded sorted window without sorting/materializing the complete key set. - Diagnostic event queries fetch only the requested window plus one row to determine truncation. - Bounded recent-operation diagnostics listing that exposes only redacted support views. - Aggregate operation queue summaries expose state counts, eligible age, and expired leases without payload data. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index c1fbeb6..cd6970d 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints with strict state validation, retries, cancellation, recovery events, bounded diagnostics with strict integer limits and capped key lists/event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Repository entrypoints validate non-empty operation IDs before reads and state transitions. The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a possibly stale handler object. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed or excessively nested JSON objects, timestamps without a defined or representable UTC offset, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints with strict state validation, retries, cancellation, recovery events, bounded diagnostics with strict integer limits, capped key lists and bounded-memory key summaries/event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Repository entrypoints validate non-empty operation IDs before reads and state transitions. The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a possibly stale handler object. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed or excessively nested JSON objects, timestamps without a defined or representable UTC offset, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 3e20e14..15b80eb 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -11,6 +11,7 @@ from dataclasses import dataclass from datetime import datetime, timedelta, timezone from functools import wraps +import heapq import json import re import sqlite3 @@ -1075,7 +1076,7 @@ def _checkpoint_summary(checkpoint: dict[str, Any]) -> dict[str, Any]: @staticmethod def _bounded_keys(value: dict[str, Any]) -> tuple[list[str], bool]: - keys = sorted(str(key) for key in value) + keys = heapq.nsmallest(_MAX_DIAGNOSTIC_KEYS + 1, (str(key) for key in value)) return keys[:_MAX_DIAGNOSTIC_KEYS], len(keys) > _MAX_DIAGNOSTIC_KEYS @staticmethod From 2c3b10fa612f33d5bde6460b042600140c705941 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 20:21:39 +0200 Subject: [PATCH 140/167] docs: track cancellation recovery decision --- docs/open-questions.md | 8 +++++++- specs/CATALOG.md | 2 +- specs/catalog.json | 3 ++- specs/durable-operations-and-recovery.md | 4 +++- 4 files changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/open-questions.md b/docs/open-questions.md index d5fd272..8fac186 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -1,7 +1,7 @@ # Open questions, risks, and next design work **Status:** open; nothing here is an accepted decision -**Last reviewed:** 2026-09-22 +**Last reviewed:** 2026-09-26 ## Decisions requiring owner input @@ -72,6 +72,12 @@ Decide retention for immutable snapshots, operation detail, provider metadata, l The repository currently has no license. The license should be selected before accepting outside contributions. Contributor handling of provider fixtures, terms, security reports, and trademarks also needs a policy. +### OQ-010 — How should cancellation recover after a worker loses an uncertain external write? + +If cancellation is requested while a provider write is in flight and the worker disappears before recording the response, the system cannot safely assume the write did not happen. Choose between a durable reconciliation-required phase that resumes only reconciliation, or a `waiting_user` state that blocks until an explicit manual action. The operation must not be reported as fully cancelled while the external outcome remains unknown. + +**Proposed default:** persist the in-flight step and reconcile it before finalizing cancellation; use `waiting_user` if the provider cannot safely confirm the outcome. The current operation state model does not yet define this recovery transition. + ## Research/design gates (not owner preference alone) ### RG-001 — Official provider feasibility diff --git a/specs/CATALOG.md b/specs/CATALOG.md index 47c882a..7deeec8 100644 --- a/specs/CATALOG.md +++ b/specs/CATALOG.md @@ -12,7 +12,7 @@ This is the human-readable view of [`catalog.json`](./catalog.json). Until gener | `library-import-and-provider-projections` | Draft | [Library import and provider projections](library-import-and-provider-projections.md) | Blocked by provider completeness, retention, and representative scale | | `recording-identity-resolution` | Draft | [Recording identity resolution](recording-identity-resolution.md) | Blocked by the labeled corpus and accepted automatic-link policy | | `one-time-playlist-copy` | Draft | [One-time playlist copy](one-time-playlist-copy.md) | Blocked by unresolved-entry policy and proven target write semantics | -| `durable-operations-and-recovery` | Draft | [Durable operations and recovery](durable-operations-and-recovery.md) | Blocked by the persistence/lease/restart spike and operating targets | +| `durable-operations-and-recovery` | Draft | [Durable operations and recovery](durable-operations-and-recovery.md) | Blocked by persistence/lease/restart evidence, operating targets, and cancellation recovery for uncertain writes (`OQ-010`) | ## Foundation evidence boundary diff --git a/specs/catalog.json b/specs/catalog.json index 95c0620..934a02c 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -408,7 +408,8 @@ "RG-004 storage engine and lease/crash-recovery spike", "OQ-007 representative operation sizes and timing targets", "Process topology and worker concurrency defaults", - "Retention policy for step detail and operation history" + "Retention policy for step detail and operation history", + "OQ-010 cancellation recovery after an uncertain in-flight provider write" ] } ] diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index cd6970d..21e11a0 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -8,7 +8,7 @@ - Related requirements: `SYM-PROD-004`–`SYM-PROD-006`, `SYM-ARCH-001`–`SYM-ARCH-002`, `SYM-ARCH-005`, `SYM-ARCH-008`–`SYM-ARCH-010`, `SYM-JOB-001`–`SYM-JOB-008`, `SYM-OBS-001`–`SYM-OBS-006`, `SYM-TEST-004`, `SYM-TEST-013`, `SYM-DEP-002`, `SYM-DEP-008` - Related decisions/research: [system architecture](../docs/architecture/system-architecture.md), [development specification](../docs/development/development-specification.md), [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [UI foundation](home-assistant-native-ui.md), `RG-004` - Required review gates: architecture, persistence/recovery, provider contracts, testing, documentation, security/operations -- Open decisions blocking readiness: persistent store; worker/process topology; lease and retention parameters; supported migration strategy; representative operation sizes and timing targets +- Open decisions blocking readiness: persistent store; worker/process topology; lease and retention parameters; supported migration strategy; representative operation sizes and timing targets; cancellation after lease expiry with an uncertain external outcome (`OQ-010`) ## 1. Executive summary @@ -207,6 +207,8 @@ Progress shall not move backward without an explicit explanation. Status shall n Graceful shutdown stops new claims, lets safe checkpoints finish within a bounded interval, and then releases or permits leases to expire. Correctness must also hold for abrupt termination. +If a cancellation request survives a worker loss while a provider outcome is uncertain, the ordinary handler must not be dispatched again and the operation must not be reported as fully cancelled until reconciliation or the owner-approved `OQ-010` manual recovery path resolves the in-flight step. + ## 11. Security and privacy - Operation payloads store credential references, never raw tokens. From bc0e57cb2ca39243cf4155ca58b524968fa56f32 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 20:23:06 +0200 Subject: [PATCH 141/167] feat: report storage spike phase timings --- docs/development/storage-recovery-spike.md | 11 +++--- docs/open-questions.md | 5 ++- specs/catalog.json | 1 + specs/durable-operations-and-recovery.md | 1 + tools/storage_recovery_spike.py | 44 +++++++++++++++++++++- 5 files changed, 53 insertions(+), 9 deletions(-) diff --git a/docs/development/storage-recovery-spike.md b/docs/development/storage-recovery-spike.md index c738648..2497ebe 100644 --- a/docs/development/storage-recovery-spike.md +++ b/docs/development/storage-recovery-spike.md @@ -16,11 +16,12 @@ Run it from the repository root with: PYTHONPATH=src:. python3 tools/storage_recovery_spike.py --operations 1000 ``` -The JSON result reports only fixture counts, state transition evidence, file -sizes, backup validity, and elapsed time. It contains no database path, -operation payload, account identifier, or credential. The elapsed time is an -observation for the machine and fixture size used; it is not a product SLO or -a restore benchmark. +The JSON result reports only fixture counts, state transition evidence, SQLite +version, file sizes, backup validity, total elapsed time, and separate timings +for runtime open, operation creation, claim/recovery, checkpoint, and backup +creation/validation. It contains no database path, operation payload, account +identifier, or credential. Timings are observations for the machine and +fixture size used; they are not product SLOs or a restore benchmark. This spike supports the foundation claims that operation leases survive a process restart, concurrent SQLite workers respect the lease boundary, and diff --git a/docs/open-questions.md b/docs/open-questions.md index 8fac186..2a9425e 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -120,8 +120,9 @@ Compare SQLite and PostgreSQL for transaction boundaries, leases, crash recovery The current local evidence includes a [reproducible SQLite recovery and backup spike](development/storage-recovery-spike.md) covering an expired lease across -runtime restart and read-only backup validation. It is an initial evidence -point, not a storage choice or a production SLO. +runtime restart and read-only backup validation, with separate phase timings +and the SQLite runtime version. It is an initial evidence point, not a storage +choice, representative-scale conclusion, or production SLO. ### RG-005 — Home Assistant native surface RFC diff --git a/specs/catalog.json b/specs/catalog.json index 934a02c..b97cd70 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -368,6 +368,7 @@ "evidence": [ "docs/architecture/system-architecture.md", "docs/development/development-specification.md", + "docs/development/storage-recovery-spike.md", "docs/open-questions.md", "docs/providers/home-assistant-ecosystem-review.md" ], diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 21e11a0..b86de6b 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -30,6 +30,7 @@ Evidence sources: - [System architecture](../docs/architecture/system-architecture.md) - [Provider specification](../docs/providers/provider-specification.md) - [Development specification](../docs/development/development-specification.md) +- [SQLite recovery and backup spike](../docs/development/storage-recovery-spike.md) ## 3. Actors and authorization diff --git a/tools/storage_recovery_spike.py b/tools/storage_recovery_spike.py index bc38225..2156baa 100644 --- a/tools/storage_recovery_spike.py +++ b/tools/storage_recovery_spike.py @@ -12,6 +12,7 @@ from datetime import datetime, timedelta, timezone import json from pathlib import Path +import sqlite3 import tempfile import time from typing import Any @@ -25,15 +26,25 @@ def run_spike(*, operation_count: int = 1000) -> dict[str, Any]: """Run the disposable recovery scenario and return secret-free evidence.""" - if not 1 <= operation_count <= 10_000: - raise ValueError("operation_count must be between 1 and 10000") + if ( + isinstance(operation_count, bool) + or not isinstance(operation_count, int) + or not 1 <= operation_count <= 10_000 + ): + raise ValueError("operation_count must be an integer between 1 and 10000") started = time.perf_counter() + phase_ms: dict[str, float] = {} with tempfile.TemporaryDirectory(prefix="symphonia-storage-spike-") as directory: source_path = Path(directory) / "symphonia.sqlite3" backup_path = Path(directory) / "backup.sqlite3" + phase_started = time.perf_counter() with RuntimeResources.open(str(source_path)) as resources: + phase_ms["initial_runtime_open"] = round( + (time.perf_counter() - phase_started) * 1000, 2 + ) + phase_started = time.perf_counter() for index in range(operation_count): resources.operations.create( operation_type="spike.copy", @@ -42,21 +53,37 @@ def run_spike(*, operation_count: int = 1000) -> dict[str, Any]: payload={"fixture_index": index}, now=BASE_TIME, ) + phase_ms["operation_creation"] = round( + (time.perf_counter() - phase_started) * 1000, 2 + ) + phase_started = time.perf_counter() claimed = resources.operations.claim( "spike-operation-0", worker_id="worker-before-restart", now=BASE_TIME, lease_seconds=1, ) + phase_ms["initial_claim"] = round( + (time.perf_counter() - phase_started) * 1000, 2 + ) + phase_started = time.perf_counter() with RuntimeResources.open(str(source_path)) as resources: + phase_ms["restart_runtime_open"] = round( + (time.perf_counter() - phase_started) * 1000, 2 + ) + phase_started = time.perf_counter() recovered = resources.operations.claim( claimed.operation_id, worker_id="worker-after-restart", now=BASE_TIME + timedelta(seconds=2), lease_seconds=30, ) + phase_ms["expired_lease_recovery"] = round( + (time.perf_counter() - phase_started) * 1000, 2 + ) recovered_worker_id = recovered.worker_id + phase_started = time.perf_counter() completed = resources.operations.checkpoint( recovered.operation_id, worker_id="worker-after-restart", @@ -64,15 +91,27 @@ def run_spike(*, operation_count: int = 1000) -> dict[str, Any]: now=BASE_TIME + timedelta(seconds=3), state="succeeded", ) + phase_ms["completion_checkpoint"] = round( + (time.perf_counter() - phase_started) * 1000, 2 + ) audit_event_count = len(resources.operations.events(recovered.operation_id)) + phase_started = time.perf_counter() resources.backup_to(str(backup_path)) + phase_ms["backup_creation"] = round( + (time.perf_counter() - phase_started) * 1000, 2 + ) source_bytes = source_path.stat().st_size backup_bytes = backup_path.stat().st_size + phase_started = time.perf_counter() backup_valid = RuntimeResources.validate_backup(str(backup_path)) + phase_ms["backup_validation"] = round( + (time.perf_counter() - phase_started) * 1000, 2 + ) elapsed_ms = round((time.perf_counter() - started) * 1000, 2) return { "operation_count": operation_count, + "sqlite_version": sqlite3.sqlite_version, "recovered_operation_id": recovered.operation_id, "recovered_worker_id": recovered_worker_id, "recovered_state": completed.state, @@ -80,6 +119,7 @@ def run_spike(*, operation_count: int = 1000) -> dict[str, Any]: "source_bytes": source_bytes, "backup_bytes": backup_bytes, "backup_valid": backup_valid, + "phase_ms": phase_ms, "elapsed_ms": elapsed_ms, } From c1e11d9900f348cbe50fcd20b7a093aab038ace2 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 20:24:52 +0200 Subject: [PATCH 142/167] feat: parameterize synthetic storage spike payloads --- docs/development/storage-recovery-spike.md | 10 +++- docs/open-questions.md | 6 ++- tools/storage_recovery_spike.py | 53 +++++++++++++++++++--- 3 files changed, 59 insertions(+), 10 deletions(-) diff --git a/docs/development/storage-recovery-spike.md b/docs/development/storage-recovery-spike.md index 2497ebe..ffd9f2b 100644 --- a/docs/development/storage-recovery-spike.md +++ b/docs/development/storage-recovery-spike.md @@ -1,7 +1,7 @@ # SQLite recovery and backup spike **Status:** research evidence only -**Last reviewed:** 2026-09-22 +**Last reviewed:** 2026-09-26 The repository now contains a small offline spike at [`tools/storage_recovery_spike.py`](../../tools/storage_recovery_spike.py). It @@ -13,7 +13,7 @@ and validates an online SQLite backup. Run it from the repository root with: ```text -PYTHONPATH=src:. python3 tools/storage_recovery_spike.py --operations 1000 +PYTHONPATH=src:. python3 tools/storage_recovery_spike.py --operations 1000 --payload-bytes 256 ``` The JSON result reports only fixture counts, state transition evidence, SQLite @@ -23,6 +23,12 @@ creation/validation. It contains no database path, operation payload, account identifier, or credential. Timings are observations for the machine and fixture size used; they are not product SLOs or a restore benchmark. +`--payload-bytes` adds only repeated synthetic padding to each queued operation +so storage and write timings can be sampled at several fixture sizes. Runs are +capped at 64 MiB of total synthetic payload; the padding is not a model of real +playlist contents, and the cap is tooling protection rather than a product +limit. + This spike supports the foundation claims that operation leases survive a process restart, concurrent SQLite workers respect the lease boundary, and the composed runtime can produce a structurally validated backup. The unit diff --git a/docs/open-questions.md b/docs/open-questions.md index 2a9425e..edebe3a 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -121,8 +121,10 @@ Compare SQLite and PostgreSQL for transaction boundaries, leases, crash recovery The current local evidence includes a [reproducible SQLite recovery and backup spike](development/storage-recovery-spike.md) covering an expired lease across runtime restart and read-only backup validation, with separate phase timings -and the SQLite runtime version. It is an initial evidence point, not a storage -choice, representative-scale conclusion, or production SLO. +and the SQLite runtime version. Its synthetic per-operation payload size is +configurable for sensitivity runs, but is not representative data. It is an +initial evidence point, not a storage choice, representative-scale conclusion, +or production SLO. ### RG-005 — Home Assistant native surface RFC diff --git a/tools/storage_recovery_spike.py b/tools/storage_recovery_spike.py index 2156baa..1b07609 100644 --- a/tools/storage_recovery_spike.py +++ b/tools/storage_recovery_spike.py @@ -21,17 +21,39 @@ BASE_TIME = datetime(2026, 9, 22, 12, 0, tzinfo=timezone.utc) +MAX_OPERATION_COUNT = 10_000 +MAX_PAYLOAD_BYTES_PER_OPERATION = 65_536 +MAX_TOTAL_PAYLOAD_BYTES = 64 * 1024 * 1024 -def run_spike(*, operation_count: int = 1000) -> dict[str, Any]: +def run_spike( + *, operation_count: int = 1000, payload_bytes: int = 0 +) -> dict[str, Any]: """Run the disposable recovery scenario and return secret-free evidence.""" if ( isinstance(operation_count, bool) or not isinstance(operation_count, int) - or not 1 <= operation_count <= 10_000 + or not 1 <= operation_count <= MAX_OPERATION_COUNT ): - raise ValueError("operation_count must be an integer between 1 and 10000") + raise ValueError( + f"operation_count must be an integer between 1 and {MAX_OPERATION_COUNT}" + ) + if ( + isinstance(payload_bytes, bool) + or not isinstance(payload_bytes, int) + or not 0 <= payload_bytes <= MAX_PAYLOAD_BYTES_PER_OPERATION + ): + raise ValueError( + "payload_bytes must be an integer between 0 and " + f"{MAX_PAYLOAD_BYTES_PER_OPERATION}" + ) + total_payload_bytes = operation_count * payload_bytes + if total_payload_bytes > MAX_TOTAL_PAYLOAD_BYTES: + raise ValueError( + f"total synthetic payload must not exceed {MAX_TOTAL_PAYLOAD_BYTES} bytes" + ) + synthetic_padding = "x" * payload_bytes started = time.perf_counter() phase_ms: dict[str, float] = {} @@ -46,11 +68,14 @@ def run_spike(*, operation_count: int = 1000) -> dict[str, Any]: ) phase_started = time.perf_counter() for index in range(operation_count): + payload = {"fixture_index": index} + if synthetic_padding: + payload["synthetic_padding"] = synthetic_padding resources.operations.create( operation_type="spike.copy", idempotency_key=f"spike-key-{index}", operation_id=f"spike-operation-{index}", - payload={"fixture_index": index}, + payload=payload, now=BASE_TIME, ) phase_ms["operation_creation"] = round( @@ -111,6 +136,8 @@ def run_spike(*, operation_count: int = 1000) -> dict[str, Any]: elapsed_ms = round((time.perf_counter() - started) * 1000, 2) return { "operation_count": operation_count, + "payload_bytes_per_operation": payload_bytes, + "total_synthetic_payload_bytes": total_payload_bytes, "sqlite_version": sqlite3.sqlite_version, "recovered_operation_id": recovered.operation_id, "recovered_worker_id": recovered_worker_id, @@ -130,10 +157,24 @@ def main() -> None: "--operations", type=int, default=1000, - help="number of disposable queued operations to create (1-10000)", + help=f"number of disposable queued operations to create (1-{MAX_OPERATION_COUNT})", + ) + parser.add_argument( + "--payload-bytes", + type=int, + default=0, + help=( + "synthetic padding bytes stored in each fixture operation " + f"(0-{MAX_PAYLOAD_BYTES_PER_OPERATION}; 64 MiB total run cap)" + ), ) args = parser.parse_args() - print(json.dumps(run_spike(operation_count=args.operations), sort_keys=True)) + print( + json.dumps( + run_spike(operation_count=args.operations, payload_bytes=args.payload_bytes), + sort_keys=True, + ) + ) if __name__ == "__main__": From 0c1ad66ae9c353ce49f30a47376b282d8c41b0da Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 22:21:21 +0200 Subject: [PATCH 143/167] feat: quarantine cancelled operations with unknown outcomes --- docs/development/implementation-baseline.md | 3 +- docs/open-questions.md | 6 +- specs/CATALOG.md | 2 +- specs/catalog.json | 2 +- specs/durable-operations-and-recovery.md | 20 +- .../infrastructure/sqlite_operations.py | 229 +++++++++++++++++- 6 files changed, 236 insertions(+), 26 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 58b2496..251b721 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -1,7 +1,7 @@ # Implementation baseline **Status:** owner-approved foundation slice -**Last reviewed:** 2026-09-25 +**Last reviewed:** 2026-09-26 The first implementation increment is intentionally narrower than any provider or Home Assistant capability. It proves the provider-independent core and the durable-operation persistence contract without selecting an external web framework, provider SDK, OAuth strategy, or frontend stack. @@ -22,6 +22,7 @@ The first implementation increment is intentionally narrower than any provider o - Explicit provider rate-limit waits with absolute retry times and durable eligibility. - Bounded transient retry budgets for copy and import handlers with durable exhaustion outcomes. - Lease-expiry recovery is audited distinctly from first claims, with prior worker identity retained only as metadata. +- An expired lease with cancellation requested is quarantined in `waiting_user` with durable reconciliation evidence, never sent through the ordinary handler, and can be finalized only through an explicit no-effect/effect-confirmed repository resolution; authenticated caller wiring remains out of scope. - Provider connection persistence with account uniqueness, opaque secret references, health states, and effective capability evidence. - Application connection service for verified-account registration, capability probes, degraded health, and reauthorization-required classification. - Pure capability-layer intersection and requirement checks for adapter/connection/object/health constraints. diff --git a/docs/open-questions.md b/docs/open-questions.md index edebe3a..15cf369 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -72,11 +72,11 @@ Decide retention for immutable snapshots, operation detail, provider metadata, l The repository currently has no license. The license should be selected before accepting outside contributions. Contributor handling of provider fixtures, terms, security reports, and trademarks also needs a policy. -### OQ-010 — How should cancellation recover after a worker loses an uncertain external write? +### OQ-010 — How should cancellation recover after a worker loses an uncertain external write? — Resolved design -If cancellation is requested while a provider write is in flight and the worker disappears before recording the response, the system cannot safely assume the write did not happen. Choose between a durable reconciliation-required phase that resumes only reconciliation, or a `waiting_user` state that blocks until an explicit manual action. The operation must not be reported as fully cancelled while the external outcome remains unknown. +**Decision (2026-09-26, selected under owner delegation):** use `waiting_user` as the provider-independent safety boundary. If a cancellation-requested worker lease expires, quarantine the operation with a durable reconciliation marker; never dispatch its ordinary handler again and never report it as cancelled while the external outcome is unknown. An authorized operator must establish `no_effect` (terminal `cancelled`) or `effect_confirmed` (terminal `partial`) before clearing the cancellation request. If neither fact can be established, it stays in `waiting_user`. -**Proposed default:** persist the in-flight step and reconcile it before finalizing cancellation; use `waiting_user` if the provider cannot safely confirm the outcome. The current operation state model does not yet define this recovery transition. +This foundation deliberately does not guess at provider reconciliation or expose an unauthenticated resolution endpoint. A future provider-aware reconciliation handler may resolve the outcome automatically only after its provider contract is proven. The repository primitive for explicit resolution is not itself an authorization boundary; any caller must enforce the approved operator identity and action policy. The durable-operation SDD remains blocked on that provider/UI vertical and its other readiness gates, but OQ-010 no longer represents an unresolved policy choice. ## Research/design gates (not owner preference alone) diff --git a/specs/CATALOG.md b/specs/CATALOG.md index 7deeec8..7cba0b6 100644 --- a/specs/CATALOG.md +++ b/specs/CATALOG.md @@ -12,7 +12,7 @@ This is the human-readable view of [`catalog.json`](./catalog.json). Until gener | `library-import-and-provider-projections` | Draft | [Library import and provider projections](library-import-and-provider-projections.md) | Blocked by provider completeness, retention, and representative scale | | `recording-identity-resolution` | Draft | [Recording identity resolution](recording-identity-resolution.md) | Blocked by the labeled corpus and accepted automatic-link policy | | `one-time-playlist-copy` | Draft | [One-time playlist copy](one-time-playlist-copy.md) | Blocked by unresolved-entry policy and proven target write semantics | -| `durable-operations-and-recovery` | Draft | [Durable operations and recovery](durable-operations-and-recovery.md) | Blocked by persistence/lease/restart evidence, operating targets, and cancellation recovery for uncertain writes (`OQ-010`) | +| `durable-operations-and-recovery` | Draft | [Durable operations and recovery](durable-operations-and-recovery.md) | Blocked by persistence/lease/restart evidence, operating targets, and the provider-aware/authorized vertical for uncertain-write reconciliation | ## Foundation evidence boundary diff --git a/specs/catalog.json b/specs/catalog.json index b97cd70..2ba97fa 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -410,7 +410,7 @@ "OQ-007 representative operation sizes and timing targets", "Process topology and worker concurrency defaults", "Retention policy for step detail and operation history", - "OQ-010 cancellation recovery after an uncertain in-flight provider write" + "Provider-aware reconciliation and authorized manual-resolution surface" ] } ] diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index b86de6b..8ac52bb 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -1,14 +1,14 @@ # Durable operations and recovery - Status: Draft -- Date: 2026-09-20 +- Date: 2026-09-26 - Catalog capability ID: `durable-operations-and-recovery` - Owners: Symphonia maintainers - Scope: execute imports and provider writes through durable, observable operations that recover safely across restarts, rate limits, uncertain writes, and upgrades. - Related requirements: `SYM-PROD-004`–`SYM-PROD-006`, `SYM-ARCH-001`–`SYM-ARCH-002`, `SYM-ARCH-005`, `SYM-ARCH-008`–`SYM-ARCH-010`, `SYM-JOB-001`–`SYM-JOB-008`, `SYM-OBS-001`–`SYM-OBS-006`, `SYM-TEST-004`, `SYM-TEST-013`, `SYM-DEP-002`, `SYM-DEP-008` - Related decisions/research: [system architecture](../docs/architecture/system-architecture.md), [development specification](../docs/development/development-specification.md), [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [UI foundation](home-assistant-native-ui.md), `RG-004` - Required review gates: architecture, persistence/recovery, provider contracts, testing, documentation, security/operations -- Open decisions blocking readiness: persistent store; worker/process topology; lease and retention parameters; supported migration strategy; representative operation sizes and timing targets; cancellation after lease expiry with an uncertain external outcome (`OQ-010`) +- Open decisions blocking readiness: persistent store; worker/process topology; lease and retention parameters; supported migration strategy; representative operation sizes and timing targets; provider-aware reconciliation and an authorized manual-resolution surface ## 1. Executive summary @@ -22,7 +22,7 @@ Provider work crosses unreliable networks and may take longer than an HTTP reque The product, architecture, and provider specifications require resumability, rate-limit awareness, explicit partial outcomes, auditable state, and safe operation in the Home Assistant App lifecycle. -An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints with strict state validation, retries, cancellation, recovery events, bounded diagnostics with strict integer limits, capped key lists and bounded-memory key summaries/event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker tests. Repository entrypoints validate non-empty operation IDs before reads and state transitions. The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a possibly stale handler object. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed or excessively nested JSON objects, timestamps without a defined or representable UTC offset, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, and the complete provider/UI vertical slice are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. +An owner-approved foundation slice now proves a dependency-free SQLite operation repository, transactional schema/legacy-column migration, leases with strict positive-integer duration validation, checkpoints with strict state validation, retries, cancellation, recovery events, bounded diagnostics with strict integer limits, capped key lists and bounded-memory key summaries/event reads, provider-independent handlers with result validation, shared SQLite connection policy with foreign-key enforcement and bounded lock waits, connection cleanup when store initialization fails, JSON-object/string-key payload validation, and deterministic worker mechanics. Repository entrypoints validate non-empty operation IDs before reads and state transitions. The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a possibly stale handler object. Complete repository calls serialize access to a shared SQLite connection; cross-connection and cross-process correctness continues to rely on SQLite transactions and leases. Operation transactions place `BEGIN` inside the protected scope, attempt rollback on process-level interruption, and preserve the primary exception if rollback also fails. Readiness streams through persisted operations and audit events rather than materializing the ledger, and fails closed on SQLite integrity or foreign-key corruption, invalid persisted operation/event states, malformed or excessively nested JSON objects, timestamps without a defined or representable UTC offset, and invalid credential-bearing payload keys; operation, checkpoint, and audit-event payloads reject token-, password-, authorization-, and API-key-shaped fields in snake, kebab, and camel case on writes and reads, while errors do not echo rejected key names. Cancellation recovery now quarantines an expired cancellation-requested lease in `waiting_user`, records why reconciliation is required, prevents ordinary resume/handler dispatch, and supports an explicit repository resolution to truthful `cancelled` or `partial` outcomes. This is a persistence primitive only: authorization and a user/operator surface are not implemented. Real multi-connection claim/replay and backup-reopen evidence cover the recovery path. The capability SDD remains `Draft`: persistent-store and worker-topology decisions, retention, migration strategy, representative sizing, provider-aware reconciliation, authorization/UI, and the complete provider vertical are still open. This evidence does not authorize production operation policy or claim that the full capability is implemented. Evidence sources: @@ -204,11 +204,11 @@ Progress shall not move backward without an explicit explanation. Status shall n | Repeated transient failure | Apply bounded backoff and attempt ceiling | End `failed` with remediation guidance | | Permanent item failures | Continue only when operation policy permits | End `partial` with per-item results | | Incompatible application/schema upgrade | Do not claim or mutate externally | Run verified migration or require operator action | -| Cancellation races with a provider call | Reconcile the in-flight call; start no later steps | Report confirmed partial state accurately | +| Cancellation races with a provider call | Reconcile the in-flight call; if the worker lease expires, quarantine in `waiting_user` and never dispatch the ordinary handler | Report confirmed partial state accurately; an unresolved outcome is not terminal `cancelled` | Graceful shutdown stops new claims, lets safe checkpoints finish within a bounded interval, and then releases or permits leases to expire. Correctness must also hold for abrupt termination. -If a cancellation request survives a worker loss while a provider outcome is uncertain, the ordinary handler must not be dispatched again and the operation must not be reported as fully cancelled until reconciliation or the owner-approved `OQ-010` manual recovery path resolves the in-flight step. +If a cancellation request survives worker loss while a provider outcome is uncertain, the repository moves the expired operation to `waiting_user`, records `reconciliation_required`, and clears its lease without clearing the cancellation request. It is never eligible for ordinary claim or resume. A trusted caller may resolve only an established `no_effect` outcome to `cancelled` or `effect_confirmed` to `partial`; unresolved outcomes remain in `waiting_user`. The repository primitive is not an authorization boundary, and no caller is wired until the provider-specific and authenticated operator contracts are approved. ## 11. Security and privacy @@ -224,7 +224,7 @@ If a cancellation request survives a worker loss while a provider outcome is unc Structured events include operation creation, eligibility, claim, renewal, checkpoint, wait, retry schedule, reconciliation, cancellation, and terminal transition. Each event includes operation ID, operation type, state, attempt count, adapter category, and bounded error code; it excludes credentials and unbounded content. -Metrics include queue depth, oldest eligible age, running leases, expired lease recoveries, state counts, execution latency, wait duration, attempt counts, unknown outcomes, reconciliation results, and terminal result ratios. Cardinality shall remain bounded. +Metrics include queue depth, oldest eligible age, running leases, expired lease recoveries, cancellation-reconciliation-required count, state counts, execution latency, wait duration, attempt counts, unknown outcomes, reconciliation results, and terminal result ratios. Cardinality shall remain bounded. Health distinguishes API availability, persistent-store readiness, worker liveness, claim progress, and migration state. A redacted export provides state history, version information, checkpoints, lease history, and categorized errors. @@ -238,17 +238,17 @@ Initial rollout uses one App instance and conservative concurrency. Multi-instan ## 14. Numeric test budget -Minimum planned automated tests: **86**. +Minimum planned automated tests: **88**. | Area | Minimum | |---|---:| | Domain states, transitions, cancellation, and invariants | 20 | -| Claims, leases, races, checkpoints, idempotency, and reconciliation | 24 | +| Claims, leases, races, checkpoints, idempotency, and reconciliation | 26 | | Persistence and provider outcome adapter contracts | 16 | | UI states and accessibility | 8 | | Integration, restart, migration, security, corruption, and redaction | 18 | -Required deterministic tests include duplicate dispatch, concurrent claim, lease expiry, clock boundaries, crash before/after every checkpoint, store outage, retry exhaustion, rate-limit timing, cancellation races, unknown outcomes, and compatible/incompatible upgrades. +Required deterministic tests include duplicate dispatch, concurrent claim, lease expiry, clock boundaries, crash before/after every checkpoint, store outage, retry exhaustion, rate-limit timing, cancellation races, unknown outcomes, cancellation quarantine and manual outcome resolution, and compatible/incompatible upgrades. The eight feature-specific UI cases supplement the UI-foundation budget and inherit its component-catalog, host-context, accessibility, responsive, theme/localization, hostile-content, and dated visual-reference gates. @@ -273,7 +273,7 @@ Implementation shall update: 6. Duplicate submissions and duplicate dispatches do not duplicate logical work. 7. Rate limits and transient failures produce bounded, visible waits. 8. Expired authorization produces `waiting_user` without discarding progress. -9. Cancellation starts no later work and accurately accounts for in-flight outcomes. +9. Cancellation starts no later work and accurately accounts for in-flight outcomes. A cancellation-requested operation whose worker lease expires during an uncertain provider write becomes `waiting_user`, is not dispatched through its ordinary handler, and remains non-resumable until explicit resolution records `no_effect` or `effect_confirmed`. 10. Store unavailability prevents new external side effects. 11. Supported upgrades preserve or safely refuse every persisted state fixture. 12. Logs, metrics, events, and diagnostics contain no credentials or prohibited content. diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 15b80eb..ed02242 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -503,12 +503,24 @@ def queue_summary(self, *, now: datetime) -> dict[str, Any]: """ SELECT COUNT(*) AS count FROM operations - WHERE cancel_requested = 0 - AND state = 'running' + WHERE state = 'running' AND (lease_expires_at IS NULL OR lease_expires_at <= ?) """, (now_text,), ).fetchone() + cancellation_recovery = self._connection.execute( + """ + SELECT COUNT(*) AS count + FROM operations + WHERE cancel_requested = 1 + AND ( + state = 'waiting_user' + OR (state = 'running' + AND (lease_expires_at IS NULL OR lease_expires_at <= ?)) + ) + """, + (now_text,), + ).fetchone() cancellation_rows = self._connection.execute( "SELECT COUNT(*) AS count FROM operations WHERE cancel_requested = 1" ).fetchone() @@ -527,6 +539,7 @@ def queue_summary(self, *, now: datetime) -> dict[str, Any]: "oldest_eligible_at": oldest_eligible_at, "oldest_eligible_age_seconds": oldest_eligible_age_seconds, "cancellation_requested_count": int(cancellation_rows["count"]), + "cancellation_recovery_required_count": int(cancellation_recovery["count"]), } @_serialize_repository_access @@ -600,10 +613,11 @@ def claim_next( lease_seconds: int = 30, operation_type: str | None = None, ) -> OperationRecord | None: - """Atomically claim the oldest queued, due, or expired operation. + """Quarantine expired cancelled work, then claim one eligible operation. - This is the scheduler-facing primitive. It deliberately selects only - durable eligibility; handler dispatch remains an application concern. + Cancellation recovery never dispatches an ordinary handler. The + scheduler-facing claim still selects only durable eligibility; handler + dispatch remains an application concern. """ _require_text(worker_id, label="worker_id") @@ -618,6 +632,60 @@ def claim_next( parameters += (operation_type,) try: self._connection.execute("BEGIN IMMEDIATE") + # A cancelled worker may have disappeared during an external + # request. Quarantine one expired lease for explicit resolution; + # never send it through the ordinary operation handler again. + uncertain = self._connection.execute( + """ + SELECT operation_id, checkpoint_json + FROM operations + WHERE cancel_requested = 1 + AND state = 'running' + AND (lease_expires_at IS NULL OR lease_expires_at <= ?) + ORDER BY COALESCE(lease_expires_at, created_at), created_at, operation_id + LIMIT 1 + """, + (now_text,), + ).fetchone() + if uncertain is not None: + operation_id = uncertain["operation_id"] + checkpoint = _load_json_object( + uncertain["checkpoint_json"], label="operation checkpoint" + ) + checkpoint.update( + { + "reconciliation_required": True, + "recovery_reason": "cancelled_worker_lease_expired", + } + ) + checkpoint_json = json.dumps( + checkpoint, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ) + self._connection.execute( + """ + UPDATE operations + SET state = 'waiting_user', checkpoint_json = ?, + worker_id = NULL, lease_expires_at = NULL, + next_run_at = NULL, updated_at = ? + WHERE operation_id = ? + """, + (checkpoint_json, now_text, operation_id), + ) + self._append_event( + operation_id=operation_id, + event_type="cancellation_reconciliation_required", + state="waiting_user", + worker_id=None, + payload={ + "recovery_reason": "cancelled_worker_lease_expired", + **self._checkpoint_summary(checkpoint), + }, + created_at=now_text, + ) row = self._connection.execute( f""" SELECT * @@ -742,9 +810,6 @@ def checkpoint( raise ValueError("invalid checkpoint state") _validate_object_payload(checkpoint, label="checkpoint") now_text = _utc(now) - checkpoint_json = json.dumps( - checkpoint, ensure_ascii=False, sort_keys=True, separators=(",", ":"), allow_nan=False - ) try: self._connection.execute("BEGIN IMMEDIATE") row = self._connection.execute( @@ -756,14 +821,38 @@ def checkpoint( raise LeaseConflict("worker does not own a running operation") if row["lease_expires_at"] is not None and row["lease_expires_at"] <= now_text: raise LeaseConflict("operation lease has expired") - effective_state = "cancelled" if row["cancel_requested"] else state + effective_state = state + if row["cancel_requested"] and state in {"running", "cancelled", "waiting_user"}: + outcome_is_uncertain = ( + checkpoint.get("unknown_step") is not None + or checkpoint.get("reconciliation_required") is True + ) + if outcome_is_uncertain: + effective_state = "waiting_user" + checkpoint = { + **checkpoint, + "reconciliation_required": True, + "recovery_reason": "cancelled_during_unknown_outcome", + } + else: + effective_state = "cancelled" + checkpoint_json = json.dumps( + checkpoint, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ) self._connection.execute( """ UPDATE operations SET state = ?, checkpoint_json = ?, updated_at = ?, worker_id = CASE WHEN ? = 'running' THEN worker_id ELSE NULL END, lease_expires_at = CASE WHEN ? = 'running' THEN lease_expires_at ELSE NULL END, - cancel_requested = CASE WHEN ? = 'cancelled' THEN 0 ELSE cancel_requested END + cancel_requested = CASE + WHEN ? IN ('cancelled', 'succeeded', 'partial', 'failed') THEN 0 + ELSE cancel_requested + END WHERE operation_id = ? """, ( @@ -806,6 +895,50 @@ def cancel(self, operation_id: str, *, now: datetime) -> OperationRecord: if row["state"] in {"succeeded", "partial", "failed", "cancelled"}: self._connection.execute("COMMIT") return self.get(operation_id) + if row["state"] == "waiting_user" and row["cancel_requested"]: + # Repeated cancellation is not a resolution of an unknown + # provider outcome. Preserve the actionable quarantine. + self._connection.execute("COMMIT") + return self.get(operation_id) + if row["state"] == "waiting_user": + checkpoint = _load_json_object( + row["checkpoint_json"], label="operation checkpoint" + ) + if ( + checkpoint.get("unknown_step") is not None + or checkpoint.get("reconciliation_required") is True + ): + checkpoint.update( + { + "reconciliation_required": True, + "recovery_reason": "cancelled_while_unknown_outcome", + } + ) + checkpoint_json = json.dumps( + checkpoint, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ) + self._connection.execute( + """ + UPDATE operations + SET checkpoint_json = ?, cancel_requested = 1, updated_at = ? + WHERE operation_id = ? + """, + (checkpoint_json, now_text, operation_id), + ) + self._append_event( + operation_id=operation_id, + event_type="cancellation_requested", + state="waiting_user", + worker_id=None, + payload={"recovery_reason": "cancelled_while_unknown_outcome"}, + created_at=now_text, + ) + self._connection.execute("COMMIT") + return self.get(operation_id) if row["state"] == "running": self._connection.execute( "UPDATE operations SET cancel_requested = 1, updated_at = ? WHERE operation_id = ?", @@ -858,6 +991,16 @@ def resume(self, operation_id: str, *, now: datetime) -> OperationRecord: raise OperationNotFound(operation_id) if row["state"] != "waiting_user": raise LeaseConflict("only waiting_user operations can be resumed") + checkpoint = _load_json_object( + row["checkpoint_json"], label="operation checkpoint" + ) + if ( + row["cancel_requested"] + or checkpoint.get("reconciliation_required") is True + ): + raise LeaseConflict( + "cancellation recovery must be resolved before operation can resume" + ) self._connection.execute( "UPDATE operations SET state = 'queued', updated_at = ? WHERE operation_id = ?", (now_text, operation_id), @@ -876,6 +1019,72 @@ def resume(self, operation_id: str, *, now: datetime) -> OperationRecord: raise return self.get(operation_id) + @_serialize_repository_access + def resolve_cancelled_outcome( + self, + operation_id: str, + *, + outcome: str, + now: datetime, + ) -> OperationRecord: + """Persist an explicit manual resolution of an uncertain cancelled write. + + Callers must authorize the human/operator action before invoking this + repository primitive. An unresolved outcome stays in ``waiting_user`` + and cannot be resumed through the ordinary operation path. + """ + + _require_text(operation_id, label="operation_id") + if not isinstance(outcome, str) or outcome not in {"no_effect", "effect_confirmed"}: + raise ValueError("outcome must be 'no_effect' or 'effect_confirmed'") + now_text = _utc(now) + try: + self._connection.execute("BEGIN IMMEDIATE") + row = self._connection.execute( + "SELECT * FROM operations WHERE operation_id = ?", (operation_id,) + ).fetchone() + if row is None: + raise OperationNotFound(operation_id) + if row["state"] != "waiting_user" or not row["cancel_requested"]: + raise LeaseConflict("operation has no unresolved cancellation outcome") + checkpoint = _load_json_object( + row["checkpoint_json"], label="operation checkpoint" + ) + if checkpoint.get("reconciliation_required") is not True: + raise LeaseConflict("operation is not awaiting cancellation reconciliation") + checkpoint["cancellation_resolution"] = outcome + checkpoint_json = json.dumps( + checkpoint, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ) + resolved_state = "cancelled" if outcome == "no_effect" else "partial" + self._connection.execute( + """ + UPDATE operations + SET state = ?, checkpoint_json = ?, cancel_requested = 0, + worker_id = NULL, lease_expires_at = NULL, + next_run_at = NULL, updated_at = ? + WHERE operation_id = ? + """, + (resolved_state, checkpoint_json, now_text, operation_id), + ) + self._append_event( + operation_id=operation_id, + event_type="cancellation_reconciled", + state=resolved_state, + worker_id=None, + payload={"outcome": outcome}, + created_at=now_text, + ) + self._connection.execute("COMMIT") + except BaseException as error: + _rollback_after_error(self._connection, error) + raise + return self.get(operation_id) + @_serialize_repository_access def schedule_retry( self, From 8b860d0d62b128950208c94789335e03f0abc5af Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 22:22:36 +0200 Subject: [PATCH 144/167] feat: expose safe cancellation recovery diagnostics --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 2 +- .../infrastructure/sqlite_operations.py | 17 +++++++++++++++++ 3 files changed, 19 insertions(+), 1 deletion(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 251b721..1778ba3 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -23,6 +23,7 @@ The first implementation increment is intentionally narrower than any provider o - Bounded transient retry budgets for copy and import handlers with durable exhaustion outcomes. - Lease-expiry recovery is audited distinctly from first claims, with prior worker identity retained only as metadata. - An expired lease with cancellation requested is quarantined in `waiting_user` with durable reconciliation evidence, never sent through the ordinary handler, and can be finalized only through an explicit no-effect/effect-confirmed repository resolution; authenticated caller wiring remains out of scope. +- Redacted operation diagnostics expose a bounded unknown-step indicator and only allowlisted reconciliation/recovery outcome metadata, never unknown step IDs or checkpoint values. - Provider connection persistence with account uniqueness, opaque secret references, health states, and effective capability evidence. - Application connection service for verified-account registration, capability probes, degraded health, and reauthorization-required classification. - Pure capability-layer intersection and requirement checks for adapter/connection/object/health constraints. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 8ac52bb..d21299f 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -222,7 +222,7 @@ If a cancellation request survives worker loss while a provider outcome is uncer ## 12. Observability and supportability -Structured events include operation creation, eligibility, claim, renewal, checkpoint, wait, retry schedule, reconciliation, cancellation, and terminal transition. Each event includes operation ID, operation type, state, attempt count, adapter category, and bounded error code; it excludes credentials and unbounded content. +Structured events include operation creation, eligibility, claim, renewal, checkpoint, wait, retry schedule, reconciliation, cancellation, and terminal transition. Each event includes operation ID, operation type, state, attempt count, adapter category, and bounded error code; it excludes credentials and unbounded content. Redacted operation diagnostics may expose only a boolean unknown-step indicator, a validated reconciliation-required flag, and allowlisted recovery/resolution enums; they never expose the unknown step identifier or checkpoint value. Metrics include queue depth, oldest eligible age, running leases, expired lease recoveries, cancellation-reconciliation-required count, state counts, execution latency, wait duration, attempt counts, unknown outcomes, reconciliation results, and terminal result ratios. Cardinality shall remain bounded. diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index ed02242..1ef5bcc 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -1276,7 +1276,24 @@ def _checkpoint_summary(checkpoint: dict[str, Any]) -> dict[str, Any]: summary: dict[str, Any] = { "checkpoint_keys": checkpoint_keys, "checkpoint_keys_truncated": checkpoint_keys_truncated, + "unknown_step_present": checkpoint.get("unknown_step") is not None, } + reconciliation_required = checkpoint.get("reconciliation_required") + if isinstance(reconciliation_required, bool): + summary["reconciliation_required"] = reconciliation_required + recovery_reason = checkpoint.get("recovery_reason") + if isinstance(recovery_reason, str) and recovery_reason in { + "cancelled_worker_lease_expired", + "cancelled_during_unknown_outcome", + "cancelled_while_unknown_outcome", + }: + summary["recovery_reason"] = recovery_reason + cancellation_resolution = checkpoint.get("cancellation_resolution") + if isinstance(cancellation_resolution, str) and cancellation_resolution in { + "no_effect", + "effect_confirmed", + }: + summary["cancellation_resolution"] = cancellation_resolution for key in ("confirmed_occurrences", "issues"): value = checkpoint.get(key) if isinstance(value, (list, tuple, set)): From 003e57c534e4b13f3d084e3289c5312be029fa18 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 22:26:23 +0200 Subject: [PATCH 145/167] feat: cover cancelled lease recovery in storage spike --- docs/development/storage-recovery-spike.md | 38 ++++++++----- specs/catalog.json | 1 + specs/durable-operations-and-recovery.md | 2 +- tools/storage_recovery_spike.py | 65 ++++++++++++++++++++-- 4 files changed, 84 insertions(+), 22 deletions(-) diff --git a/docs/development/storage-recovery-spike.md b/docs/development/storage-recovery-spike.md index ffd9f2b..b80992f 100644 --- a/docs/development/storage-recovery-spike.md +++ b/docs/development/storage-recovery-spike.md @@ -6,9 +6,12 @@ The repository now contains a small offline spike at [`tools/storage_recovery_spike.py`](../../tools/storage_recovery_spike.py). It creates a disposable persistent runtime, queues a bounded number of operations, -claims one lease, closes the runtime, reopens it as a different worker, waits -until the lease has expired, reclaims and completes the operation, then creates -and validates an online SQLite backup. +claims one ordinary lease and one cancellation-requested lease, closes the +runtime, reopens it as a different worker, waits until both leases have expired, +then verifies that the ordinary lease is reclaimed while the cancelled +operation is quarantined in `waiting_user` and cannot be resumed. It completes +the ordinary operation, then creates and validates an online SQLite backup +containing both recovery outcomes. Run it from the repository root with: @@ -16,12 +19,14 @@ Run it from the repository root with: PYTHONPATH=src:. python3 tools/storage_recovery_spike.py --operations 1000 --payload-bytes 256 ``` -The JSON result reports only fixture counts, state transition evidence, SQLite -version, file sizes, backup validity, total elapsed time, and separate timings -for runtime open, operation creation, claim/recovery, checkpoint, and backup -creation/validation. It contains no database path, operation payload, account -identifier, or credential. Timings are observations for the machine and -fixture size used; they are not product SLOs or a restore benchmark. +The JSON result reports the configured fixture count plus one dedicated +cancellation fixture, ordinary and cancellation-recovery state evidence, +SQLite version, file sizes, backup validity, total elapsed time, and separate +timings for runtime open, operation creation, claim/recovery, cancellation +fixture setup, checkpoint, and backup creation/validation. It contains no +database path, operation payload, account identifier, or credential. Timings +are observations for the machine and fixture size used; they are not product +SLOs or a restore benchmark. `--payload-bytes` adds only repeated synthetic padding to each queued operation so storage and write timings can be sampled at several fixture sizes. Runs are @@ -30,11 +35,14 @@ playlist contents, and the cap is tooling protection rather than a product limit. This spike supports the foundation claims that operation leases survive a -process restart, concurrent SQLite workers respect the lease boundary, and -the composed runtime can produce a structurally validated backup. The unit -suite also reopens a backup containing operation, provider-connection, and +process restart, cancellation-requested expired leases are not resumed as +ordinary work, and the composed runtime can produce a structurally validated +backup. The unit suite also reopens a backup containing operation, provider-connection, and authorization-attempt rows through the normal runtime composition root and readiness now fails closed when durable state contains invalid JSON or enum -values. It does not yet select backup retention, encryption, -Supervisor backup declarations, restore UX, or a production scheduler. Those -remain part of the App runtime and durable execution SDD gates. +values. The cancellation scenario is repository-level evidence only: it does +not exercise a provider API, prove provider reconciliation, or provide +authenticated operator resolution. The spike does not yet select backup +retention, encryption, Supervisor backup declarations, restore UX, or a +production scheduler. Those remain part of the App runtime and durable +execution SDD gates. diff --git a/specs/catalog.json b/specs/catalog.json index 2ba97fa..8f3aff1 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -376,6 +376,7 @@ "code": [ "src/symphonia/infrastructure/sqlite_common.py", "src/symphonia/infrastructure/sqlite_operations.py", + "tools/storage_recovery_spike.py", "src/symphonia/infrastructure/sqlite_authorization.py", "src/symphonia/infrastructure/sqlite_connections.py", "src/symphonia/infrastructure/sqlite_library.py", diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index d21299f..7f5f351 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -230,7 +230,7 @@ Health distinguishes API availability, persistent-store readiness, worker livene ## 13. Rollout, migration, and compatibility -Before implementation readiness, a persistence spike shall prove atomic claim, lease expiry, transactional checkpointing, backup/restore behavior under `/data`, and migration safety in the chosen Home Assistant App runtime. +Before implementation readiness, a persistence spike shall prove atomic claim, ordinary and cancellation-requested lease expiry, transactional checkpointing, backup/restore behavior under `/data`, and migration safety in the chosen Home Assistant App runtime. The local SQLite spike now exercises cancellation quarantine and verifies that unresolved cancellation cannot be resumed; it remains research evidence, not representative Home Assistant/App-scale or provider evidence. Operation records and payloads carry explicit schema and handler versions. Upgrade tests cover queued, running, waiting, partial-progress, cancelled, and terminal fixtures. A version that cannot safely resume an operation shall leave it untouched and expose an actionable incompatibility state rather than guessing. diff --git a/tools/storage_recovery_spike.py b/tools/storage_recovery_spike.py index 1b07609..47eb2e1 100644 --- a/tools/storage_recovery_spike.py +++ b/tools/storage_recovery_spike.py @@ -1,9 +1,9 @@ """Reproducible SQLite recovery/backup spike for the runtime foundation. This is evidence tooling, not a production benchmark or a restore command. It -creates a disposable persistent store, simulates a worker restart with an -expired lease, and validates an online backup using the runtime's existing -preflight checks. +creates a disposable persistent store, simulates ordinary and cancellation- +requested worker restarts with expired leases, and validates an online backup +using the runtime's existing preflight checks. """ from __future__ import annotations @@ -17,6 +17,7 @@ import time from typing import Any +from symphonia.infrastructure.sqlite_operations import LeaseConflict from symphonia.runtime import RuntimeResources @@ -72,7 +73,7 @@ def run_spike( if synthetic_padding: payload["synthetic_padding"] = synthetic_padding resources.operations.create( - operation_type="spike.copy", + operation_type="spike.recovery" if index == 0 else "spike.queued", idempotency_key=f"spike-key-{index}", operation_id=f"spike-operation-{index}", payload=payload, @@ -91,6 +92,27 @@ def run_spike( phase_ms["initial_claim"] = round( (time.perf_counter() - phase_started) * 1000, 2 ) + phase_started = time.perf_counter() + cancelled = resources.operations.create( + operation_type="spike.recovery", + idempotency_key="spike-cancelled-key", + operation_id="spike-cancelled-operation", + payload={"fixture": "cancelled-expired-lease"}, + now=BASE_TIME, + ) + cancelled = resources.operations.claim( + cancelled.operation_id, + worker_id="worker-before-restart", + now=BASE_TIME, + lease_seconds=1, + ) + resources.operations.cancel( + cancelled.operation_id, + now=BASE_TIME + timedelta(milliseconds=500), + ) + phase_ms["cancellation_fixture_setup"] = round( + (time.perf_counter() - phase_started) * 1000, 2 + ) phase_started = time.perf_counter() with RuntimeResources.open(str(source_path)) as resources: @@ -98,15 +120,37 @@ def run_spike( (time.perf_counter() - phase_started) * 1000, 2 ) phase_started = time.perf_counter() - recovered = resources.operations.claim( - claimed.operation_id, + recovered = resources.operations.claim_next( worker_id="worker-after-restart", now=BASE_TIME + timedelta(seconds=2), lease_seconds=30, + operation_type="spike.recovery", ) + if recovered is None or recovered.operation_id != claimed.operation_id: + raise RuntimeError("ordinary expired fixture was not reclaimed") phase_ms["expired_lease_recovery"] = round( (time.perf_counter() - phase_started) * 1000, 2 ) + cancellation_recovered = resources.operations.get(cancelled.operation_id) + cancellation_audit_event_count = len( + resources.operations.events(cancelled.operation_id) + ) + try: + resources.operations.resume( + cancelled.operation_id, + now=BASE_TIME + timedelta(seconds=2), + ) + except LeaseConflict: + cancellation_resume_blocked = True + else: + cancellation_resume_blocked = False + if ( + cancellation_recovered.state != "waiting_user" + or not cancellation_recovered.cancel_requested + or cancellation_recovered.checkpoint.get("reconciliation_required") is not True + or not cancellation_resume_blocked + ): + raise RuntimeError("cancelled expired fixture was not safely quarantined") recovered_worker_id = recovered.worker_id phase_started = time.perf_counter() completed = resources.operations.checkpoint( @@ -136,13 +180,22 @@ def run_spike( elapsed_ms = round((time.perf_counter() - started) * 1000, 2) return { "operation_count": operation_count, + "cancellation_fixture_count": 1, "payload_bytes_per_operation": payload_bytes, "total_synthetic_payload_bytes": total_payload_bytes, "sqlite_version": sqlite3.sqlite_version, "recovered_operation_id": recovered.operation_id, "recovered_worker_id": recovered_worker_id, "recovered_state": completed.state, + "cancellation_recovery_state": cancellation_recovered.state, + "cancellation_still_requested": cancellation_recovered.cancel_requested, + "cancellation_reconciliation_required": cancellation_recovered.checkpoint.get( + "reconciliation_required" + ) + is True, + "cancellation_resume_blocked": cancellation_resume_blocked, "audit_event_count": audit_event_count, + "cancellation_audit_event_count": cancellation_audit_event_count, "source_bytes": source_bytes, "backup_bytes": backup_bytes, "backup_valid": backup_valid, From 09a9744619d0167cfd7f66069dfe59ff09e992fb Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sat, 26 Sep 2026 22:28:26 +0200 Subject: [PATCH 146/167] fix: preserve uncertain cancellation across retry releases --- docs/development/implementation-baseline.md | 2 +- specs/durable-operations-and-recovery.md | 4 +- .../infrastructure/sqlite_operations.py | 80 ++++++++++++++++--- 3 files changed, 74 insertions(+), 12 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 1778ba3..6e6f81a 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -22,7 +22,7 @@ The first implementation increment is intentionally narrower than any provider o - Explicit provider rate-limit waits with absolute retry times and durable eligibility. - Bounded transient retry budgets for copy and import handlers with durable exhaustion outcomes. - Lease-expiry recovery is audited distinctly from first claims, with prior worker identity retained only as metadata. -- An expired lease with cancellation requested is quarantined in `waiting_user` with durable reconciliation evidence, never sent through the ordinary handler, and can be finalized only through an explicit no-effect/effect-confirmed repository resolution; authenticated caller wiring remains out of scope. +- An expired lease with cancellation requested is quarantined in `waiting_user` with durable reconciliation evidence, never sent through the ordinary handler, and can be finalized only through an explicit no-effect/effect-confirmed repository resolution; checkpoint, retry, and rate-limit release paths preserve this quarantine; authenticated caller wiring remains out of scope. - Redacted operation diagnostics expose a bounded unknown-step indicator and only allowlisted reconciliation/recovery outcome metadata, never unknown step IDs or checkpoint values. - Provider connection persistence with account uniqueness, opaque secret references, health states, and effective capability evidence. - Application connection service for verified-account registration, capability probes, degraded health, and reauthorization-required classification. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 7f5f351..0b0a622 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -208,7 +208,7 @@ Progress shall not move backward without an explicit explanation. Status shall n Graceful shutdown stops new claims, lets safe checkpoints finish within a bounded interval, and then releases or permits leases to expire. Correctness must also hold for abrupt termination. -If a cancellation request survives worker loss while a provider outcome is uncertain, the repository moves the expired operation to `waiting_user`, records `reconciliation_required`, and clears its lease without clearing the cancellation request. It is never eligible for ordinary claim or resume. A trusted caller may resolve only an established `no_effect` outcome to `cancelled` or `effect_confirmed` to `partial`; unresolved outcomes remain in `waiting_user`. The repository primitive is not an authorization boundary, and no caller is wired until the provider-specific and authenticated operator contracts are approved. +If a cancellation request survives worker loss while a provider outcome is uncertain, the repository moves the expired operation to `waiting_user`, records `reconciliation_required`, and clears its lease without clearing the cancellation request. It is never eligible for ordinary claim or resume. The same quarantine applies if a worker reaches a retry or rate-limit release path with an uncertain checkpoint after cancellation was requested; such an operation is not scheduled for another ordinary attempt. A trusted caller may resolve only an established `no_effect` outcome to `cancelled` or `effect_confirmed` to `partial`; unresolved outcomes remain in `waiting_user`. The repository primitive is not an authorization boundary, and no caller is wired until the provider-specific and authenticated operator contracts are approved. ## 11. Security and privacy @@ -273,7 +273,7 @@ Implementation shall update: 6. Duplicate submissions and duplicate dispatches do not duplicate logical work. 7. Rate limits and transient failures produce bounded, visible waits. 8. Expired authorization produces `waiting_user` without discarding progress. -9. Cancellation starts no later work and accurately accounts for in-flight outcomes. A cancellation-requested operation whose worker lease expires during an uncertain provider write becomes `waiting_user`, is not dispatched through its ordinary handler, and remains non-resumable until explicit resolution records `no_effect` or `effect_confirmed`. +9. Cancellation starts no later work and accurately accounts for in-flight outcomes. A cancellation-requested operation whose worker lease expires during an uncertain provider write becomes `waiting_user`, is not dispatched through its ordinary handler or retry/rate-limit path, and remains non-resumable until explicit resolution records `no_effect` or `effect_confirmed`. 10. Store unavailability prevents new external side effects. 11. Supported upgrades preserve or safely refuse every persisted state fixture. 12. Logs, metrics, events, and diagnostics contain no credentials or prohibited content. diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 1ef5bcc..7cfa884 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -182,6 +182,13 @@ def _validate_object_payload(payload: Any, *, label: str) -> None: _validate_payload_keys(payload, label=label) +def _checkpoint_requires_reconciliation(checkpoint: dict[str, Any]) -> bool: + return ( + checkpoint.get("unknown_step") is not None + or checkpoint.get("reconciliation_required") is True + ) + + @dataclass(frozen=True, slots=True) class OperationRecord: operation_id: str @@ -823,11 +830,7 @@ def checkpoint( raise LeaseConflict("operation lease has expired") effective_state = state if row["cancel_requested"] and state in {"running", "cancelled", "waiting_user"}: - outcome_is_uncertain = ( - checkpoint.get("unknown_step") is not None - or checkpoint.get("reconciliation_required") is True - ) - if outcome_is_uncertain: + if _checkpoint_requires_reconciliation(checkpoint): effective_state = "waiting_user" checkpoint = { **checkpoint, @@ -904,10 +907,7 @@ def cancel(self, operation_id: str, *, now: datetime) -> OperationRecord: checkpoint = _load_json_object( row["checkpoint_json"], label="operation checkpoint" ) - if ( - checkpoint.get("unknown_step") is not None - or checkpoint.get("reconciliation_required") is True - ): + if _checkpoint_requires_reconciliation(checkpoint): checkpoint.update( { "reconciliation_required": True, @@ -1115,6 +1115,15 @@ def schedule_retry( if row["state"] != "running" or row["worker_id"] != worker_id: raise LeaseConflict("worker does not own a running operation") if row["cancel_requested"]: + if _checkpoint_requires_reconciliation(checkpoint): + self._quarantine_cancelled_outcome( + operation_id, + checkpoint=checkpoint, + recovery_reason="cancelled_during_unknown_outcome", + now_text=now_text, + ) + self._connection.execute("COMMIT") + return self.get(operation_id) self._connection.execute( """ UPDATE operations @@ -1188,6 +1197,15 @@ def schedule_rate_limit( if row["state"] != "running" or row["worker_id"] != worker_id: raise LeaseConflict("worker does not own a running operation") if row["cancel_requested"]: + if _checkpoint_requires_reconciliation(checkpoint): + self._quarantine_cancelled_outcome( + operation_id, + checkpoint=checkpoint, + recovery_reason="cancelled_during_unknown_outcome", + now_text=now_text, + ) + self._connection.execute("COMMIT") + return self.get(operation_id) self._connection.execute( """ UPDATE operations @@ -1238,6 +1256,50 @@ def _by_idempotency(self, idempotency_key: str) -> OperationRecord | None: ).fetchone() return None if row is None else self._record(row) + def _quarantine_cancelled_outcome( + self, + operation_id: str, + *, + checkpoint: dict[str, Any], + recovery_reason: str, + now_text: str, + ) -> None: + """Persist reconciliation quarantine inside the caller's transaction.""" + + checkpoint = { + **checkpoint, + "reconciliation_required": True, + "recovery_reason": recovery_reason, + } + _validate_object_payload(checkpoint, label="operation checkpoint") + checkpoint_json = json.dumps( + checkpoint, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ) + self._connection.execute( + """ + UPDATE operations + SET state = 'waiting_user', checkpoint_json = ?, next_run_at = NULL, + worker_id = NULL, lease_expires_at = NULL, updated_at = ? + WHERE operation_id = ? + """, + (checkpoint_json, now_text, operation_id), + ) + self._append_event( + operation_id=operation_id, + event_type="cancellation_reconciliation_required", + state="waiting_user", + worker_id=None, + payload={ + "recovery_reason": recovery_reason, + **self._checkpoint_summary(checkpoint), + }, + created_at=now_text, + ) + @_serialize_repository_access def _append_event( self, From 684ae0ae6a1ed79846252b8bd2bf42edbb06fd45 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 18:54:26 +0200 Subject: [PATCH 147/167] fix: stop before side effects after cancellation request --- docs/development/implementation-baseline.md | 1 + specs/durable-operations-and-recovery.md | 4 ++-- src/symphonia/application/copy_execution.py | 15 ++++++++++++--- .../application/library_import_execution.py | 16 +++++++++++++++- .../infrastructure/sqlite_operations.py | 4 +++- 5 files changed, 33 insertions(+), 7 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 6e6f81a..187f7e8 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -21,6 +21,7 @@ The first implementation increment is intentionally narrower than any provider o - Aggregate operation queue summaries expose state counts, eligible age, and expired leases without payload data. - Explicit provider rate-limit waits with absolute retry times and durable eligibility. - Bounded transient retry budgets for copy and import handlers with durable exhaustion outcomes. +- Lease renewal refuses cancellation-requested work; copy and import executors checkpoint a cooperative stop before beginning their next provider step when they still hold a valid lease. - Lease-expiry recovery is audited distinctly from first claims, with prior worker identity retained only as metadata. - An expired lease with cancellation requested is quarantined in `waiting_user` with durable reconciliation evidence, never sent through the ordinary handler, and can be finalized only through an explicit no-effect/effect-confirmed repository resolution; checkpoint, retry, and rate-limit release paths preserve this quarantine; authenticated caller wiring remains out of scope. - Redacted operation diagnostics expose a bounded unknown-step indicator and only allowlisted reconciliation/recovery outcome metadata, never unknown step IDs or checkpoint values. diff --git a/specs/durable-operations-and-recovery.md b/specs/durable-operations-and-recovery.md index 0b0a622..1bbfcb4 100644 --- a/specs/durable-operations-and-recovery.md +++ b/specs/durable-operations-and-recovery.md @@ -117,7 +117,7 @@ After a crash, another worker waits for lease expiry, claims the operation, and - `running` requires an unexpired lease token and owner. - `waiting_rate_limit` and `retry_scheduled` store an absolute next-eligible time plus the source of that timing. - `waiting_user` stores a redacted reason and a bounded set of permitted actions. -- Cancellation is checked before claim, before every side effect, and between provider batches. +- Cancellation is checked before claim, before every side effect, and between provider batches. The lease-renewal gate used before a new step refuses a cancellation-requested operation; while its lease remains valid, the handler checkpoints the cancellation without starting that step. ### Idempotency @@ -197,7 +197,7 @@ Progress shall not move backward without an explicit explanation. Status shall n | Worker crashes before side effect | Lease expires; checkpoint remains authoritative | Another worker safely resumes | | Worker crashes after external acceptance but before checkpoint | Step remains uncertain | Reconcile externally before deciding success or retry | | Store unavailable | Stop new external side effects; do not rely on memory-only progress | Resume after durable store health returns | -| Lease renewal fails | Stop starting side effects and relinquish/expire ownership | Another claim after store recovery and lease expiry | +| Lease renewal fails or cancellation is requested | Stop before another side effect; checkpoint cancellation only while the lease is still valid | A still-valid worker records the safe stop; otherwise lease expiry recovery takes over | | Duplicate dispatch | Atomic claim permits one active lease | Extra dispatch becomes a no-op | | Rate limit | Persist provider advice and release active execution | Become eligible at safe time | | Authorization expires | Persist `waiting_user`; retain checkpoint | Resume after scoped reauthorization | diff --git a/src/symphonia/application/copy_execution.py b/src/symphonia/application/copy_execution.py index 4f6bb5b..dcae410 100644 --- a/src/symphonia/application/copy_execution.py +++ b/src/symphonia/application/copy_execution.py @@ -7,7 +7,11 @@ from typing import Any from symphonia.domain.models import PlanAcceptanceError -from symphonia.infrastructure.sqlite_operations import OperationRecord, OperationRepository +from symphonia.infrastructure.sqlite_operations import ( + LeaseConflict, + OperationRecord, + OperationRepository, +) from symphonia.infrastructure.sqlite_plans import CopyPlanRepository, StoredCopyPlan from symphonia.providers.writing import PlaylistWriter, ProviderWriteError, WriteOutcome @@ -149,9 +153,14 @@ def _execute_claimed( now=now, lease_seconds=lease_seconds, ) - except Exception: + except LeaseConflict: latest = self.operations.get(operation.operation_id) - if latest.cancel_requested: + if ( + latest.cancel_requested + and latest.worker_id == worker_id + and latest.lease_expires_at is not None + and latest.lease_expires_at > now + ): return self._finish(latest, worker_id, checkpoint, now, "running") raise step_key = f"{digest}:entry:{entry.occurrence_id}" diff --git a/src/symphonia/application/library_import_execution.py b/src/symphonia/application/library_import_execution.py index 21585c8..fa2e81a 100644 --- a/src/symphonia/application/library_import_execution.py +++ b/src/symphonia/application/library_import_execution.py @@ -7,7 +7,11 @@ from datetime import datetime, timedelta, timezone from typing import Any -from symphonia.infrastructure.sqlite_operations import OperationRecord, OperationRepository +from symphonia.infrastructure.sqlite_operations import ( + LeaseConflict, + OperationRecord, + OperationRepository, +) from symphonia.providers.contracts import ProviderAdapter, ProviderObjectRef from symphonia.providers.errors import ProviderApiError, ProviderErrorCategory @@ -100,6 +104,16 @@ def execute_claimed( snapshot_id=snapshot_id, observed_at=observed_at, ) + except LeaseConflict: + latest = self.operations.get(operation.operation_id) + if ( + latest.cancel_requested + and latest.worker_id == worker_id + and latest.lease_expires_at is not None + and latest.lease_expires_at > now + ): + return self._checkpoint(latest, worker_id, checkpoint, now, "cancelled") + raise except ProviderApiError as error: checkpoint["failure_code"] = error.category.value if error.category in { diff --git a/src/symphonia/infrastructure/sqlite_operations.py b/src/symphonia/infrastructure/sqlite_operations.py index 7cfa884..4107db5 100644 --- a/src/symphonia/infrastructure/sqlite_operations.py +++ b/src/symphonia/infrastructure/sqlite_operations.py @@ -756,7 +756,7 @@ def renew_lease( now: datetime, lease_seconds: int = 30, ) -> OperationRecord: - """Extend a healthy lease; an expired owner cannot resurrect it.""" + """Extend a healthy, non-cancelled lease; stale owners cannot revive it.""" _require_text(operation_id, label="operation_id") _require_text(worker_id, label="worker_id") @@ -772,6 +772,8 @@ def renew_lease( raise OperationNotFound(operation_id) if row["state"] != "running" or row["worker_id"] != worker_id: raise LeaseConflict("worker does not own a running operation") + if row["cancel_requested"]: + raise LeaseConflict("operation cancellation has been requested") if row["lease_expires_at"] is not None and row["lease_expires_at"] <= now_text: raise LeaseConflict("operation lease has expired") self._connection.execute( From dd845667025453161c080217798bf065de9055aa Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 19:06:29 +0200 Subject: [PATCH 148/167] fix: reconcile in-flight playlist writes before retry --- docs/development/implementation-baseline.md | 3 +- specs/catalog.json | 12 +- specs/one-time-playlist-copy.md | 9 +- src/symphonia/application/copy_execution.py | 301 +++++++++++++++++--- 4 files changed, 283 insertions(+), 42 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 187f7e8..a383aa0 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -1,7 +1,7 @@ # Implementation baseline **Status:** owner-approved foundation slice -**Last reviewed:** 2026-09-26 +**Last reviewed:** 2026-09-27 The first implementation increment is intentionally narrower than any provider or Home Assistant capability. It proves the provider-independent core and the durable-operation persistence contract without selecting an external web framework, provider SDK, OAuth strategy, or frontend stack. @@ -32,6 +32,7 @@ The first implementation increment is intentionally narrower than any provider o - The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a potentially stale handler object. - Cooperative single-process operation worker with interruptible polling and injected clock support. - Copy executor can run as a claimed operation handler, preserving the same restart/checkpoint semantics under the runner. +- Copy execution durably marks a target-create or entry-add step in-flight before the provider call; a resumed step is reconciled before any repeated mutation, and inconclusive evidence or reconciliation errors stay in `waiting_user`. This is a foundation safety behavior only; provider reconciliation strategy and the full copy contract remain blocked in the Draft SDD. - Playlist import executor persists intent before reads and reports succeeded, partial, waiting-user, retry, and rate-limit outcomes durably. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Direct-callback authorization can resolve a durable attempt by the returned state digest after restart without persisting raw state; ambiguous digests fail closed. diff --git a/specs/catalog.json b/specs/catalog.json index 8f3aff1..09fee28 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -315,12 +315,18 @@ "docs/decisions/0002-copy-and-sync-are-distinct.md", "docs/domain/domain-model.md", "docs/product/product-specification.md", - "docs/providers/provider-research.md" + "docs/providers/provider-research.md", + "docs/development/implementation-baseline.md" ], "implementationEvidence": { - "code": [], + "code": [ + "src/symphonia/application/copy_execution.py" + ], "tests": [], - "documentation": [] + "documentation": [ + "docs/development/implementation-baseline.md", + "specs/one-time-playlist-copy.md" + ] }, "blockers": [ "OQ-003 default treatment of non-ready entries", diff --git a/specs/one-time-playlist-copy.md b/specs/one-time-playlist-copy.md index a451dee..5049fa8 100644 --- a/specs/one-time-playlist-copy.md +++ b/specs/one-time-playlist-copy.md @@ -1,7 +1,7 @@ # One-time playlist copy - Status: Draft -- Date: 2026-09-20 +- Date: 2026-09-27 - Catalog capability ID: `one-time-playlist-copy` - Owners: Symphonia maintainers - Scope: preview and execute a finite playlist copy using an immutable plan, explicit non-ready policy, ordered writes, reconciliation, and item-level outcomes. @@ -29,6 +29,7 @@ Evidence sources: - [Provider specification](../docs/providers/provider-specification.md) - [System architecture](../docs/architecture/system-architecture.md) - [Open questions](../docs/open-questions.md) +- [Owner-approved implementation baseline](../docs/development/implementation-baseline.md) (foundation evidence only; it does not close this SDD's blockers or authorize the full capability) ## 3. Actors and authorization @@ -192,8 +193,8 @@ Status shall never rely only on color. Keyboard navigation, visible focus, seman | Source changes before acceptance | Mark plan stale; issue no writes | Recalculate and review a new revision | | Connection expires before execution | Pause without losing progress | Reauthorize, then resume | | Target capability changes | Stop before incompatible writes; record evidence | Re-plan or choose another target | -| Target creation response is unknown | Search/reconcile using stored intent and marker; do not blindly create again | Wait for reconciliation or inspect candidates | -| Entry batch response is unknown | Reconcile target contents/checkpoint before retry | Resume only when safe | +| Target creation may have an unknown outcome, including process loss during the call | Persist the target step as in-flight before the create call; on every resume reconcile the stored idempotency key before any create call; an inconclusive result stays `waiting_user` | Wait for provider-aware reconciliation or inspect candidates; never blindly create again | +| Entry write may have an unknown outcome, including process loss during the call | Persist the occurrence as in-flight before the add call; on every resume reconcile that occurrence before any add call; only positive reconciliation confirms it, while inconclusive evidence stays `waiting_user` | Resume only after positive reconciliation; otherwise wait for provider-aware evidence or operator review | | Rate limit | Enter `waiting_rate_limit` with next eligible time | Automatic bounded resume; cancellation remains available | | Some items fail permanently | Finish as `partial` with per-item reasons | Create a new remediation plan for failed entries | | Process or host restarts | Resume from durable checkpoint and lease rules | No manual action unless state becomes uncertain | @@ -257,7 +258,7 @@ Implementation shall update: 5. Acceptance binds to a specific plan digest, source projection version, capabilities snapshot, and target intent. 6. A stale or modified plan cannot execute. 7. Execution survives restart without duplicating the target playlist or confirmed entries. -8. Unknown provider outcomes trigger reconciliation before retry. +8. Target and entry steps are durably marked in-flight before each provider mutation; after restart/resume, reconciliation runs before any repeated create/add call, and inconclusive evidence or reconciliation errors remain `waiting_user` without another mutation. 9. Cancellation stops future work and accurately reports already confirmed writes. 10. Partial success has per-item explanations and a safe remediation path. 11. Logs and diagnostics contain no provider secrets. diff --git a/src/symphonia/application/copy_execution.py b/src/symphonia/application/copy_execution.py index dcae410..91199e7 100644 --- a/src/symphonia/application/copy_execution.py +++ b/src/symphonia/application/copy_execution.py @@ -109,11 +109,113 @@ def _execute_claimed( confirmed = list(checkpoint.get("confirmed_occurrences", [])) issues = list(checkpoint.get("issues", [])) target_id = checkpoint.get("target_playlist_id") + unknown_step = checkpoint.get("unknown_step") + writable_entries = stored.plan.writable_entries + writable_ids = {entry.occurrence_id for entry in writable_entries} + failed_steps = { + issue.get("step") + for issue in issues + if isinstance(issue, dict) and isinstance(issue.get("step"), str) + } + + if target_id is None and confirmed: + return self._wait_for_user( + operation, + worker_id, + checkpoint | {"failure_code": "invalid_target_checkpoint"}, + now, + ) + + if unknown_step is not None: + first_unconfirmed = next( + ( + entry.occurrence_id + for entry in writable_entries + if entry.occurrence_id not in confirmed and entry.occurrence_id not in failed_steps + ), + None, + ) + if unknown_step != "target" and ( + not isinstance(unknown_step, str) + or unknown_step not in writable_ids + or unknown_step in confirmed + or unknown_step != first_unconfirmed + or target_id is None + ): + return self._wait_for_user( + operation, + worker_id, + checkpoint | {"failure_code": "invalid_unknown_step_checkpoint"}, + now, + ) + if unknown_step == "target" and target_id is None and confirmed: + return self._wait_for_user( + operation, + worker_id, + checkpoint | {"failure_code": "invalid_unknown_step_checkpoint"}, + now, + ) + + target_key = f"{digest}:target" + if unknown_step == "target": + stopped = self._renew_or_stop( + operation, + worker_id, + checkpoint, + now, + lease_seconds, + ) + if stopped is not None: + return stopped + try: + target = writer.reconcile_target_playlist(idempotency_key=target_key) + except Exception: + return self._wait_for_user(operation, worker_id, checkpoint, now) + if target is None or (target_id is not None and target.provider_playlist_id != target_id): + return self._wait_for_user(operation, worker_id, checkpoint, now) + target_id = target.provider_playlist_id + checkpoint["target_playlist_id"] = target_id + checkpoint.pop("unknown_step", None) + operation = self.operations.checkpoint( + operation.operation_id, + worker_id=worker_id, + checkpoint=checkpoint, + now=now, + state="running", + ) + if operation.state != "running": + return operation + unknown_step = None if target_id is None: - if self.operations.get(operation.operation_id).cancel_requested: - return self._finish(operation, worker_id, checkpoint, now, "cancelled") - target_key = f"{digest}:target" + stopped = self._renew_or_stop( + operation, + worker_id, + checkpoint, + now, + lease_seconds, + ) + if stopped is not None: + return stopped + checkpoint["unknown_step"] = "target" + operation = self.operations.checkpoint( + operation.operation_id, + worker_id=worker_id, + checkpoint=checkpoint, + now=now, + state="running", + ) + if operation.state != "running": + return operation + stopped = self._renew_or_stop( + operation, + worker_id, + checkpoint, + now, + lease_seconds, + ) + if stopped is not None: + return stopped try: target = writer.ensure_target_playlist( provider=stored.plan.target_provider, @@ -123,18 +225,36 @@ def _execute_claimed( ) except ProviderWriteError as error: if error.outcome is WriteOutcome.RETRYABLE: + checkpoint.pop("unknown_step", None) return self._schedule_retry(operation, worker_id, checkpoint, now) if error.outcome is WriteOutcome.RATE_LIMITED: + checkpoint.pop("unknown_step", None) return self._schedule_rate_limit(operation, worker_id, checkpoint, now, error.retry_at) if error.outcome is WriteOutcome.UNKNOWN_OUTCOME: - target = writer.reconcile_target_playlist(idempotency_key=target_key) + stopped = self._renew_or_stop( + operation, + worker_id, + checkpoint, + now, + lease_seconds, + ) + if stopped is not None: + return stopped + try: + target = writer.reconcile_target_playlist(idempotency_key=target_key) + except Exception: + return self._wait_for_user(operation, worker_id, checkpoint, now) if target is None: - return self._wait_for_user(operation, worker_id, checkpoint | {"unknown_step": "target"}, now) + return self._wait_for_user(operation, worker_id, checkpoint, now) else: + checkpoint.pop("unknown_step", None) issues.append({"step": "target", "detail": error.detail, "provider_code": error.provider_code}) return self._finish(operation, worker_id, checkpoint | {"issues": issues}, now, "failed") + except Exception: + return self._wait_for_user(operation, worker_id, checkpoint, now) target_id = target.provider_playlist_id checkpoint["target_playlist_id"] = target_id + checkpoint.pop("unknown_step", None) operation = self.operations.checkpoint( operation.operation_id, worker_id=worker_id, @@ -142,52 +262,127 @@ def _execute_claimed( now=now, state="running", ) + if operation.state != "running": + return operation - for entry in stored.plan.writable_entries: + for entry in writable_entries: if entry.occurrence_id in confirmed: continue - try: - self.operations.renew_lease( + step_key = f"{digest}:entry:{entry.occurrence_id}" + if unknown_step == entry.occurrence_id: + stopped = self._renew_or_stop( + operation, + worker_id, + checkpoint, + now, + lease_seconds, + ) + if stopped is not None: + return stopped + try: + reconciled = writer.reconcile_entry( + target_playlist_id=target_id, + provider_track_id=entry.target_track_id or "", + idempotency_key=step_key, + ) + except Exception: + return self._wait_for_user(operation, worker_id, checkpoint, now) + if not reconciled: + return self._wait_for_user(operation, worker_id, checkpoint, now) + confirmed.append(entry.occurrence_id) + checkpoint["confirmed_occurrences"] = confirmed + checkpoint.pop("unknown_step", None) + operation = self.operations.checkpoint( operation.operation_id, worker_id=worker_id, + checkpoint=checkpoint, now=now, - lease_seconds=lease_seconds, + state="running", ) - except LeaseConflict: - latest = self.operations.get(operation.operation_id) - if ( - latest.cancel_requested - and latest.worker_id == worker_id - and latest.lease_expires_at is not None - and latest.lease_expires_at > now - ): - return self._finish(latest, worker_id, checkpoint, now, "running") - raise - step_key = f"{digest}:entry:{entry.occurrence_id}" - result = writer.add_entry( - target_playlist_id=target_id, - provider_track_id=entry.target_track_id or "", - idempotency_key=step_key, + if operation.state != "running": + return operation + unknown_step = None + continue + + stopped = self._renew_or_stop( + operation, + worker_id, + checkpoint, + now, + lease_seconds, ) - if result.outcome is WriteOutcome.UNKNOWN_OUTCOME: - if writer.reconcile_entry( + if stopped is not None: + return stopped + checkpoint["unknown_step"] = entry.occurrence_id + operation = self.operations.checkpoint( + operation.operation_id, + worker_id=worker_id, + checkpoint=checkpoint, + now=now, + state="running", + ) + if operation.state != "running": + return operation + stopped = self._renew_or_stop( + operation, + worker_id, + checkpoint, + now, + lease_seconds, + ) + if stopped is not None: + return stopped + try: + result = writer.add_entry( target_playlist_id=target_id, provider_track_id=entry.target_track_id or "", idempotency_key=step_key, - ): - result = type(result)(WriteOutcome.CONFIRMED_SUCCESS, result.provider_code, result.detail) - else: - return self._wait_for_user( - operation, - worker_id, - checkpoint | {"unknown_step": entry.occurrence_id}, - now, + ) + except Exception: + return self._wait_for_user(operation, worker_id, checkpoint, now) + if result.outcome is WriteOutcome.UNKNOWN_OUTCOME: + stopped = self._renew_or_stop( + operation, + worker_id, + checkpoint, + now, + lease_seconds, + ) + if stopped is not None: + return stopped + try: + reconciled = writer.reconcile_entry( + target_playlist_id=target_id, + provider_track_id=entry.target_track_id or "", + idempotency_key=step_key, + ) + except Exception: + return self._wait_for_user(operation, worker_id, checkpoint, now) + if reconciled: + confirmed.append(entry.occurrence_id) + checkpoint["confirmed_occurrences"] = confirmed + checkpoint.pop("unknown_step", None) + operation = self.operations.checkpoint( + operation.operation_id, + worker_id=worker_id, + checkpoint=checkpoint, + now=now, + state="running", ) + if operation.state != "running": + return operation + unknown_step = None + continue + else: + return self._wait_for_user(operation, worker_id, checkpoint, now) if result.outcome is WriteOutcome.RETRYABLE: + checkpoint.pop("unknown_step", None) return self._schedule_retry(operation, worker_id, checkpoint, now) if result.outcome is WriteOutcome.RATE_LIMITED: + checkpoint.pop("unknown_step", None) return self._schedule_rate_limit(operation, worker_id, checkpoint, now, result.retry_at) if result.outcome is WriteOutcome.PERMANENT_FAILURE: + checkpoint.pop("unknown_step", None) issues.append( { "step": entry.occurrence_id, @@ -203,9 +398,12 @@ def _execute_claimed( now=now, state="running", ) + if operation.state != "running": + return operation continue confirmed.append(entry.occurrence_id) checkpoint["confirmed_occurrences"] = confirmed + checkpoint.pop("unknown_step", None) operation = self.operations.checkpoint( operation.operation_id, worker_id=worker_id, @@ -213,6 +411,8 @@ def _execute_claimed( now=now, state="running", ) + if operation.state != "running": + return operation checkpoint["confirmed_occurrences"] = confirmed checkpoint["omitted_occurrences"] = [entry.occurrence_id for entry in stored.plan.omitted_entries] @@ -220,6 +420,39 @@ def _execute_claimed( terminal = "partial" if stored.plan.omitted_entries or issues else "succeeded" return self._finish(operation, worker_id, checkpoint, now, terminal) + def _renew_or_stop( + self, + operation: OperationRecord, + worker_id: str, + checkpoint: dict[str, Any], + now: datetime, + lease_seconds: int, + ) -> OperationRecord | None: + try: + self.operations.renew_lease( + operation.operation_id, + worker_id=worker_id, + now=now, + lease_seconds=lease_seconds, + ) + except LeaseConflict: + latest = self.operations.get(operation.operation_id) + if ( + latest.cancel_requested + and latest.worker_id == worker_id + and latest.lease_expires_at is not None + and latest.lease_expires_at > now + ): + return self.operations.checkpoint( + operation.operation_id, + worker_id=worker_id, + checkpoint=checkpoint, + now=now, + state="running", + ) + raise + return None + def _finish( self, operation: OperationRecord, From 2b0e7a257263d3f7843ae7a9ec58b852a6dd2ac0 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 19:09:27 +0200 Subject: [PATCH 149/167] docs: define provider write reconciliation evidence --- docs/development/implementation-baseline.md | 2 +- docs/providers/provider-research.md | 16 +++++++++++++++- docs/providers/provider-specification.md | 4 +++- specs/catalog.json | 1 + specs/one-time-playlist-copy.md | 5 +++-- 5 files changed, 23 insertions(+), 5 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index a383aa0..4dd419c 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -32,7 +32,7 @@ The first implementation increment is intentionally narrower than any provider o - The operation runner validates handler result type and identity, then returns the current persisted record rather than trusting a potentially stale handler object. - Cooperative single-process operation worker with interruptible polling and injected clock support. - Copy executor can run as a claimed operation handler, preserving the same restart/checkpoint semantics under the runner. -- Copy execution durably marks a target-create or entry-add step in-flight before the provider call; a resumed step is reconciled before any repeated mutation, and inconclusive evidence or reconciliation errors stay in `waiting_user`. This is a foundation safety behavior only; provider reconciliation strategy and the full copy contract remain blocked in the Draft SDD. +- Copy execution durably marks a target-create or entry-add step in-flight before the provider call; a resumed step is reconciled before any repeated mutation, and inconclusive evidence or reconciliation errors stay in `waiting_user`. The current boolean entry-reconciliation port treats `false` only as inconclusive and cannot represent proven no-effect; a richer provider contract and the full copy capability remain blocked in the Draft SDD. - Playlist import executor persists intent before reads and reports succeeded, partial, waiting-user, retry, and rate-limit outcomes durably. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Direct-callback authorization can resolve a durable attempt by the returned state digest after restart without persisting raw state; ambiguous digests fail closed. diff --git a/docs/providers/provider-research.md b/docs/providers/provider-research.md index 0b8c69d..792b57d 100644 --- a/docs/providers/provider-research.md +++ b/docs/providers/provider-research.md @@ -1,7 +1,7 @@ # Provider and platform research **Status:** research snapshot, not an architectural decision -**Reviewed:** 2026-09-22 +**Reviewed:** 2026-09-27 **Source policy:** official documentation only; revalidate before implementation and every release ## How to read this document @@ -10,6 +10,20 @@ This snapshot separates what Symphonia needs from what an official API documents Existing Home Assistant and community implementations are reviewed separately in [Home Assistant music ecosystem review](home-assistant-ecosystem-review.md). They provide valuable implementation evidence but do not replace an official provider contract. +## 2026-09-27 write-outcome and reconciliation update + +This is a dated review of official API documentation only, not a live provider feasibility spike. The official pages describe mutation requests and successful responses, but do not document a general client idempotency-key contract or a provider-independent way to prove whether a timed-out request took effect. + +| Provider | Officially documented behavior | Recovery consequence for Symphonia | +| --- | --- | --- | +| Spotify | [Create Playlist](https://developer.spotify.com/documentation/web-api/reference/create-playlist) returns a playlist resource; names need not be unique and public visibility defaults to `true`. [Add Items](https://developer.spotify.com/documentation/web-api/reference/add-items-to-playlist) appends or inserts ordered URIs, accepts up to 100 per request, and returns a `snapshot_id`. The request reference does not list a client idempotency key or promise duplicate suppression. | A snapshot identifies playlist state/version, not which timed-out occurrence was applied. Duplicate tracks and concurrent edits prevent inferring a particular write from presence alone. Creation by name cannot safely reconcile a timeout. | +| YouTube Data API | [PlaylistItems: insert](https://developers.google.com/youtube/v3/docs/playlistItems/insert) creates one playlist-item resource from `playlistId` and `resourceId`, can accept a position, returns the created resource on success, and costs 50 quota units. A requested position requires manual playlist sorting. The reference does not document an idempotency key or replay guarantee. | A returned playlist-item ID is useful only when the response is received. On timeout, matching a video in the playlist does not identify which duplicate occurrence belongs to the attempt; a dedicated-account spike must prove any stronger strategy. | +| Apple Music | [Add Tracks to a Library Playlist](https://developer.apple.com/documentation/applemusicapi/add-tracks-to-a-library-playlist) appends tracks and reports HTTP 204 on success; its documentation warns that a new resource may take time to appear. The request contract does not document an idempotency key or a reconciliation guarantee. | An immediate read that does not show the track cannot prove no effect. Treat timeout/read lag as inconclusive and do not repeat automatically. | + +**Inference, not a provider guarantee:** until a dedicated-account spike demonstrates reliable positive and no-effect reconciliation for an exact operation, Symphonia must checkpoint a provider step as in-flight before the mutation, reconcile before any replay, and leave inconclusive results in `waiting_user`. Absence of an idempotency guarantee from these pages is not proof that no provider-specific strategy exists. + +The YouTube `playlistItems.insert` reference reported a last-updated date of 2026-09-14 UTC when reviewed here; Spotify and Apple pages were re-opened on 2026-09-27. + ## 2026-09-22 verification update The official references were rechecked before the next design pass. This is a diff --git a/docs/providers/provider-specification.md b/docs/providers/provider-specification.md index 6dc6fb2..0d00964 100644 --- a/docs/providers/provider-specification.md +++ b/docs/providers/provider-specification.md @@ -1,7 +1,7 @@ # Provider specification **Status:** proposed contract; specific support is a dated research fact -**Last reviewed:** 2026-09-20 +**Last reviewed:** 2026-09-27 ## Purpose @@ -152,6 +152,8 @@ Exact language signatures are deferred, but an adapter must cover these behavior - return revision identifiers and per-item/batch outcomes; and - provide a reconciliation read for unknown write outcomes. +For reconciliation, adapters and application ports MUST distinguish an effect-confirmed result, a proven no-effect result, and an inconclusive result. An absent item in an eventually consistent or duplicate-bearing read is not proof of no effect. A repeated mutation is safe only after a provider-specific idempotency guarantee or positive proof of no effect; inconclusive outcomes remain action-required and MUST NOT be replayed automatically. These guarantees require dated official evidence and a dedicated-account feasibility spike where behavior depends on account or playlist state. + ## Adapter requirements - **SYM-PROV-001:** Domain and application code MUST depend on provider ports and normalized types, never provider SDK types. diff --git a/specs/catalog.json b/specs/catalog.json index 09fee28..80c5de4 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -316,6 +316,7 @@ "docs/domain/domain-model.md", "docs/product/product-specification.md", "docs/providers/provider-research.md", + "docs/providers/provider-specification.md", "docs/development/implementation-baseline.md" ], "implementationEvidence": { diff --git a/specs/one-time-playlist-copy.md b/specs/one-time-playlist-copy.md index 5049fa8..8764e30 100644 --- a/specs/one-time-playlist-copy.md +++ b/specs/one-time-playlist-copy.md @@ -29,6 +29,7 @@ Evidence sources: - [Provider specification](../docs/providers/provider-specification.md) - [System architecture](../docs/architecture/system-architecture.md) - [Open questions](../docs/open-questions.md) +- [Dated official write/reconciliation research](../docs/providers/provider-research.md#2026-09-27-write-outcome-and-reconciliation-update) - [Owner-approved implementation baseline](../docs/development/implementation-baseline.md) (foundation evidence only; it does not close this SDD's blockers or authorize the full capability) ## 3. Actors and authorization @@ -147,7 +148,7 @@ Coordinates snapshot capture, identity lookup, plan calculation, acceptance, ope - Query provider capabilities and connection authorization. - Resolve a canonical recording to target candidates. - Create and inspect a target playlist. -- Add ordered batches and reconcile their outcome. +- Add ordered batches and reconcile their outcome as `effect_confirmed`, `no_effect`, or `inconclusive`; the current foundation's boolean entry port maps `false` to `inconclusive` and cannot yet represent proven no-effect. - Persist plans, results, checkpoints, and audit events. ### Adapters @@ -194,7 +195,7 @@ Status shall never rely only on color. Keyboard navigation, visible focus, seman | Connection expires before execution | Pause without losing progress | Reauthorize, then resume | | Target capability changes | Stop before incompatible writes; record evidence | Re-plan or choose another target | | Target creation may have an unknown outcome, including process loss during the call | Persist the target step as in-flight before the create call; on every resume reconcile the stored idempotency key before any create call; an inconclusive result stays `waiting_user` | Wait for provider-aware reconciliation or inspect candidates; never blindly create again | -| Entry write may have an unknown outcome, including process loss during the call | Persist the occurrence as in-flight before the add call; on every resume reconcile that occurrence before any add call; only positive reconciliation confirms it, while inconclusive evidence stays `waiting_user` | Resume only after positive reconciliation; otherwise wait for provider-aware evidence or operator review | +| Entry write may have an unknown outcome, including process loss during the call | Persist the occurrence as in-flight before the add call; on every resume reconcile that occurrence before any add call; only positive confirmation or a proven no-effect result permits progress; inconclusive evidence stays `waiting_user` | Resume only after provider-aware evidence; otherwise wait for operator review | | Rate limit | Enter `waiting_rate_limit` with next eligible time | Automatic bounded resume; cancellation remains available | | Some items fail permanently | Finish as `partial` with per-item reasons | Create a new remediation plan for failed entries | | Process or host restarts | Resume from durable checkpoint and lease rules | No manual action unless state becomes uncertain | From 6e477a0f48e4b74406589adca57dab5e8701cf90 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 19:11:48 +0200 Subject: [PATCH 150/167] docs: align import SDD with foundation evidence --- docs/development/implementation-baseline.md | 2 +- specs/catalog.json | 23 +++++++++++++++---- ...library-import-and-provider-projections.md | 10 +++++--- 3 files changed, 27 insertions(+), 8 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 4dd419c..de1542f 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -33,7 +33,7 @@ The first implementation increment is intentionally narrower than any provider o - Cooperative single-process operation worker with interruptible polling and injected clock support. - Copy executor can run as a claimed operation handler, preserving the same restart/checkpoint semantics under the runner. - Copy execution durably marks a target-create or entry-add step in-flight before the provider call; a resumed step is reconciled before any repeated mutation, and inconclusive evidence or reconciliation errors stay in `waiting_user`. The current boolean entry-reconciliation port treats `false` only as inconclusive and cannot represent proven no-effect; a richer provider contract and the full copy capability remain blocked in the Draft SDD. -- Playlist import executor persists intent before reads and reports succeeded, partial, waiting-user, retry, and rate-limit outcomes durably. +- Playlist import executor persists intent before reads and reports succeeded, partial, waiting-user, retry, and rate-limit outcomes durably. Current scope is one playlist: pages are materialized in memory, and transient retries re-read from the beginning because page-level durable staging/checkpoints are not implemented; full library-import behavior remains blocked in its Draft SDD. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Direct-callback authorization can resolve a durable attempt by the returned state digest after restart without persisting raw state; ambiguous digests fail closed. - Authorization consumption is covered across independent SQLite connections so callback replay races produce exactly one consumed attempt. diff --git a/specs/catalog.json b/specs/catalog.json index 80c5de4..aacbc22 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -225,17 +225,32 @@ "docs/domain/domain-model.md", "docs/providers/provider-specification.md", "docs/providers/provider-research.md", - "docs/open-questions.md" + "docs/open-questions.md", + "docs/development/implementation-baseline.md" ], "implementationEvidence": { - "code": [], - "tests": [], - "documentation": [] + "code": [ + "src/symphonia/providers/importing.py", + "src/symphonia/application/library_import.py", + "src/symphonia/application/library_import_execution.py", + "src/symphonia/infrastructure/sqlite_library.py" + ], + "tests": [ + "tests/test_provider_import.py", + "tests/test_library_import.py", + "tests/test_library_import_execution.py", + "tests/test_sqlite_library.py" + ], + "documentation": [ + "docs/development/implementation-baseline.md", + "specs/library-import-and-provider-projections.md" + ] }, "blockers": [ "RG-001 collection semantics and completeness per provider", "OQ-008 local data retention and export policy", "OQ-007 representative library sizes and import-duration targets", + "Durable page-level staging/checkpoints and memory bounds", "Provider-specific refresh and deletion obligations" ] }, diff --git a/specs/library-import-and-provider-projections.md b/specs/library-import-and-provider-projections.md index 928ab46..d695c3c 100644 --- a/specs/library-import-and-provider-projections.md +++ b/specs/library-import-and-provider-projections.md @@ -1,14 +1,14 @@ # Library import and provider projections - Status: Draft -- Date: 2026-09-20 +- Date: 2026-09-27 - Catalog capability ID: `library-import-and-provider-projections` - Owners: Symphonia maintainers - Scope: import approved provider collections and ordered playlists into complete, provenance-rich provider projections without confusing them with provider-independent recordings. - Related requirements: `SYM-PROD-003`, `SYM-LIB-001`–`SYM-LIB-006`, `SYM-PROV-004`–`SYM-PROV-014`, `SYM-PROV-018`, `SYM-ARCH-004`–`SYM-ARCH-005` - Related decisions/research: [domain model](../docs/domain/domain-model.md), [provider specification](../docs/providers/provider-specification.md), [provider research](../docs/providers/provider-research.md), [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [UI foundation](home-assistant-native-ui.md), `RG-001`, `OQ-007`, `OQ-008` - Required review gates: product UX, domain, architecture, provider feasibility/policy, testing, documentation, privacy/operations -- Open decisions blocking readiness: proven collection semantics/completeness per MVP provider; retention/export policy; representative library sizes/import targets; provider-specific refresh/deletion obligations +- Open decisions blocking readiness: proven collection semantics/completeness per MVP provider; retention/export policy; representative library sizes/import targets; durable page-level staging/checkpoints and memory bounds; provider-specific refresh/deletion obligations ## 1. Executive summary @@ -31,7 +31,9 @@ Provider libraries differ in collection meaning, pagination, unavailable/local/n ### 2.2 Current behavior -There is no importer or persistence implementation. The domain model and provider contract establish the required representation and completeness semantics; provider-specific facts still require live spikes. +There is no complete multi-collection library importer. The owner-approved foundation currently supports one playlist at a time: normalized provider pages are materialized in memory, checked for completeness, and a complete result is published as an immutable SQLite snapshot; an incomplete result retains the prior complete snapshot. Snapshot publication is idempotent for the same content and rejects reuse of a snapshot ID for different content. + +The durable wrapper persists import intent before provider reads and records terminal publication details, authentication waits, rate-limit waits, and bounded transient retries. It does not durably checkpoint individual pages or stage them incrementally; after a transient failure it re-reads the playlist from the beginning. Saved-track collections, multi-collection sessions, retention/deletion scheduling, export policy, and representative-size guarantees are not part of this foundation evidence. This paragraph is an as-built foundation boundary, not a claim that the SDD capability is implemented. ### 2.3 Evidence and unknowns @@ -39,6 +41,8 @@ There is no importer or persistence implementation. The domain model and provide - Adapter contract: [pagination, unknown media, completeness, freshness](../docs/providers/provider-specification.md). - Official API evidence and policy constraints: [provider research](../docs/providers/provider-research.md). - Comparative warning: existing YT Music implementations may cap dynamic playlists or skip unavailable entries; [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md#youtube-music). +- As-built foundation evidence: [page collector](../src/symphonia/providers/importing.py), [playlist import use case](../src/symphonia/application/library_import.py), [durable import executor](../src/symphonia/application/library_import_execution.py), [atomic snapshot repository](../src/symphonia/infrastructure/sqlite_library.py), and the [implementation baseline](../docs/development/implementation-baseline.md). +- Existing deterministic evidence for this foundation slice: [provider import contracts](../tests/test_provider_import.py), [import use case](../tests/test_library_import.py), [durable import outcomes](../tests/test_library_import_execution.py), and [snapshot persistence](../tests/test_sqlite_library.py). Listing these files is not a statement that the complete SDD test budget has passed. - Unknowns: provider collection parity, target sizes, refresh cadence, raw payload retention, incremental cursor reliability. ## 3. Actors, surfaces, and terminology From c1c2ce4c2ce9915931c7744881aafb61c7946b9b Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 19:14:14 +0200 Subject: [PATCH 151/167] docs: record authorization foundation boundaries --- specs/catalog.json | 29 ++++++++++++++--- .../provider-connections-and-authorization.md | 31 +++++++++++++------ 2 files changed, 46 insertions(+), 14 deletions(-) diff --git a/specs/catalog.json b/specs/catalog.json index aacbc22..5ce1ce6 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -175,17 +175,38 @@ "docs/providers/provider-specification.md", "docs/providers/provider-research.md", "docs/providers/home-assistant-ecosystem-review.md", - "docs/architecture/system-architecture.md" + "docs/architecture/system-architecture.md", + "docs/development/oauth-callback-spike.md", + "docs/development/implementation-baseline.md" ], "implementationEvidence": { - "code": [], - "tests": [], - "documentation": [] + "code": [ + "src/symphonia/providers/authorization.py", + "src/symphonia/providers/connections.py", + "src/symphonia/application/authorization.py", + "src/symphonia/application/provider_connections.py", + "src/symphonia/infrastructure/sqlite_authorization.py", + "src/symphonia/infrastructure/sqlite_connections.py", + "tools/oauth_callback_spike.py" + ], + "tests": [ + "tests/test_authorization_service.py", + "tests/test_sqlite_authorization.py", + "tests/test_provider_connections.py", + "tests/test_sqlite_connections.py", + "tests/test_oauth_callback_spike.py" + ], + "documentation": [ + "docs/development/oauth-callback-spike.md", + "docs/development/implementation-baseline.md", + "specs/provider-connections-and-authorization.md" + ] }, "blockers": [ "RG-002 direct App callback reachability and OAuth boundary validation", "RG-002 Home Assistant OAuth callback spike", "RG-001 provider-specific scopes and account constraints", + "Authenticated actor/session binding and provider token/secret lifecycle integration", "Decision on whether an unofficial YouTube Music adapter exists in the MVP" ] }, diff --git a/specs/provider-connections-and-authorization.md b/specs/provider-connections-and-authorization.md index 7486c5d..8d29425 100644 --- a/specs/provider-connections-and-authorization.md +++ b/specs/provider-connections-and-authorization.md @@ -1,14 +1,14 @@ # Provider connections and authorization - Status: Draft -- Date: 2026-09-20 +- Date: 2026-09-27 - Catalog capability ID: `provider-connections-and-authorization` - Owners: Symphonia maintainers - Scope: disclose provider risk, authorize one external account, protect and refresh its grant, probe effective capabilities, reauthorize, and disconnect safely. - Related requirements: `SYM-ACC-002`–`SYM-ACC-004`, `SYM-ACC-006`, `SYM-PROV-002`–`SYM-PROV-003`, `SYM-PROV-008`–`SYM-PROV-009`, `SYM-PROV-015`–`SYM-PROV-020`, `SYM-SEC-001`–`SYM-SEC-010` - Related decisions/research: [provider specification](../docs/providers/provider-specification.md), [official API research](../docs/providers/provider-research.md), [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md), [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [UI foundation](home-assistant-native-ui.md), `OQ-001`, `OQ-004`, `RG-001`, `RG-002` - Required review gates: product UX, architecture, provider feasibility, testing, documentation, security/privacy -- Open decisions blocking readiness: direct App callback reachability and provider-specific registration/scopes/token lifecycle; unofficial YouTube Music MVP decision; secret key source and backup contract +- Open decisions blocking readiness: direct App callback reachability and provider-specific registration/scopes/token lifecycle; authenticated actor/session binding; provider token exchange/refresh/revocation and encrypted secret-store integration; unofficial YouTube Music MVP decision; secret key source and backup contract ## 1. Executive summary @@ -35,20 +35,31 @@ Provider authentication differs in client registration, redirect rules, scopes, ### 2.2 Current behavior -The foundation now persists provider-neutral authorization attempts with hashed -and bounded state, exact redirect binding, expiry, single-use consumption, -bounded durable outcome codes, and callback state lookup that survives restart. -Invalid attempt metadata is rejected before SQLite writes. Provider token -exchange, secret storage, account verification, and the provider-specific -callback adapters do not yet exist. Spotify is the strongest official MVP -candidate; full YouTube Music access is not established through an official -API; Apple Music is future research. +The owner-approved foundation persists provider-neutral authorization attempts +with hashed and bounded state, exact redirect binding, expiry, single-use +consumption, bounded durable outcome codes, and callback state lookup that +survives restart. Connection persistence stores an opaque secret reference and +capability evidence, and its application service can register an already +verified account and classify a capability-probe failure. The separate +standard-library callback spike is loopback-only and demonstrates route/state +handling; it is not wired into the runtime and does not prove remotely +reachable Home Assistant App callback behavior. + +Provider token exchange, refresh/revocation adapters, encrypted secret +storage, end-to-end account verification, authenticated actor/session binding, +and provider-specific callback integration do not yet exist. Invalid attempt +metadata is rejected before SQLite writes. Spotify is the strongest official +MVP candidate; full YouTube Music access is not established through an +official API; Apple Music is future research. This is foundation evidence, +not a claim that the connection capability is implemented. ### 2.3 Evidence and unknowns - Official provider/API facts: [provider research](../docs/providers/provider-research.md). - Home Assistant OAuth/Application Credentials and existing music projects: [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md). - Provider-independent contract: [provider specification](../docs/providers/provider-specification.md). +- As-built foundation evidence: [authorization application](../src/symphonia/application/authorization.py), [attempt repository](../src/symphonia/infrastructure/sqlite_authorization.py), [connection service](../src/symphonia/application/provider_connections.py), [connection repository](../src/symphonia/infrastructure/sqlite_connections.py), [direct-callback spike](../docs/development/oauth-callback-spike.md), and [implementation baseline](../docs/development/implementation-baseline.md). +- Existing deterministic foundation evidence: [authorization attempt tests](../tests/test_authorization_service.py), [durable attempt tests](../tests/test_sqlite_authorization.py), [connection service tests](../tests/test_provider_connections.py), [connection persistence tests](../tests/test_sqlite_connections.py), and [loopback callback-spike tests](../tests/test_oauth_callback_spike.py). These do not prove the full SDD test budget or remote App topology. - Unknowns: callback reachability, secret encryption key, Google/YouTube product scope, exact scopes, token expiry/revocation behavior under test accounts, and provider-specific client-registration policy. ## 3. Actors, surfaces, and terminology From 8156a32b997d5227475af6606cbdbddc4b3fad77 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 19:18:20 +0200 Subject: [PATCH 152/167] test: keep inconclusive copy writes pending on resume --- tests/test_copy_execution.py | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/tests/test_copy_execution.py b/tests/test_copy_execution.py index b1eae32..7a8954b 100644 --- a/tests/test_copy_execution.py +++ b/tests/test_copy_execution.py @@ -105,7 +105,23 @@ def test_unknown_write_without_reconciliation_requires_user(self) -> None: worker_id="worker-b", now=NOW + timedelta(seconds=1), ) - self.assertEqual(resumed.state, "succeeded") + self.assertEqual(resumed.state, "waiting_user") + self.assertEqual(resumed.checkpoint["unknown_step"], "occ-1") + self.assertEqual(self.writer.added, [(step_key, "target-1")]) + self.assertEqual(self.writer.reconciled, [step_key, step_key]) + + self.writer.reconcile_results[step_key] = True + self.operations.resume(operation.operation_id, now=NOW + timedelta(seconds=2)) + reconciled = self.executor.execute( + digest, + writer=self.writer, + worker_id="worker-c", + now=NOW + timedelta(seconds=2), + ) + self.assertEqual(reconciled.state, "succeeded") + self.assertEqual(reconciled.checkpoint["confirmed_occurrences"], ["occ-1"]) + self.assertNotIn("unknown_step", reconciled.checkpoint) + self.assertEqual(self.writer.added, [(step_key, "target-1")]) def test_retryable_write_releases_operation_until_scheduled(self) -> None: digest = self.accepted_digest() From 64c5cae0b9480c6b5962f8a9d6bd9649e56061f2 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 19:20:41 +0200 Subject: [PATCH 153/167] fix: preserve permanent copy failures across reconciliation --- docs/development/implementation-baseline.md | 1 + specs/catalog.json | 4 +- specs/one-time-playlist-copy.md | 2 +- src/symphonia/application/copy_execution.py | 3 +- tests/test_copy_execution.py | 53 +++++++++++++++++++-- 5 files changed, 56 insertions(+), 7 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index de1542f..bf796b6 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -33,6 +33,7 @@ The first implementation increment is intentionally narrower than any provider o - Cooperative single-process operation worker with interruptible polling and injected clock support. - Copy executor can run as a claimed operation handler, preserving the same restart/checkpoint semantics under the runner. - Copy execution durably marks a target-create or entry-add step in-flight before the provider call; a resumed step is reconciled before any repeated mutation, and inconclusive evidence or reconciliation errors stay in `waiting_user`. The current boolean entry-reconciliation port treats `false` only as inconclusive and cannot represent proven no-effect; a richer provider contract and the full copy capability remain blocked in the Draft SDD. +- Copy execution retains permanently failed occurrences as itemized issues across waits and restarts; resuming a later uncertain write does not repeat those failed writes. - Playlist import executor persists intent before reads and reports succeeded, partial, waiting-user, retry, and rate-limit outcomes durably. Current scope is one playlist: pages are materialized in memory, and transient retries re-read from the beginning because page-level durable staging/checkpoints are not implemented; full library-import behavior remains blocked in its Draft SDD. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Direct-callback authorization can resolve a durable attempt by the returned state digest after restart without persisting raw state; ambiguous digests fail closed. diff --git a/specs/catalog.json b/specs/catalog.json index 5ce1ce6..3ce6dd1 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -359,7 +359,9 @@ "code": [ "src/symphonia/application/copy_execution.py" ], - "tests": [], + "tests": [ + "tests/test_copy_execution.py" + ], "documentation": [ "docs/development/implementation-baseline.md", "specs/one-time-playlist-copy.md" diff --git a/specs/one-time-playlist-copy.md b/specs/one-time-playlist-copy.md index 8764e30..b3e4ee8 100644 --- a/specs/one-time-playlist-copy.md +++ b/specs/one-time-playlist-copy.md @@ -197,7 +197,7 @@ Status shall never rely only on color. Keyboard navigation, visible focus, seman | Target creation may have an unknown outcome, including process loss during the call | Persist the target step as in-flight before the create call; on every resume reconcile the stored idempotency key before any create call; an inconclusive result stays `waiting_user` | Wait for provider-aware reconciliation or inspect candidates; never blindly create again | | Entry write may have an unknown outcome, including process loss during the call | Persist the occurrence as in-flight before the add call; on every resume reconcile that occurrence before any add call; only positive confirmation or a proven no-effect result permits progress; inconclusive evidence stays `waiting_user` | Resume only after provider-aware evidence; otherwise wait for operator review | | Rate limit | Enter `waiting_rate_limit` with next eligible time | Automatic bounded resume; cancellation remains available | -| Some items fail permanently | Finish as `partial` with per-item reasons | Create a new remediation plan for failed entries | +| Some items fail permanently | Checkpoint each failure as a terminal item issue; never resend that occurrence when a later step waits or resumes; finish as `partial` with per-item reasons | Create a new remediation plan for failed entries | | Process or host restarts | Resume from durable checkpoint and lease rules | No manual action unless state becomes uncertain | | User cancels | Stop scheduling further writes; preserve confirmed results | Review partial result; cancellation is not rollback | diff --git a/src/symphonia/application/copy_execution.py b/src/symphonia/application/copy_execution.py index 91199e7..5143b63 100644 --- a/src/symphonia/application/copy_execution.py +++ b/src/symphonia/application/copy_execution.py @@ -266,7 +266,7 @@ def _execute_claimed( return operation for entry in writable_entries: - if entry.occurrence_id in confirmed: + if entry.occurrence_id in confirmed or entry.occurrence_id in failed_steps: continue step_key = f"{digest}:entry:{entry.occurrence_id}" if unknown_step == entry.occurrence_id: @@ -383,6 +383,7 @@ def _execute_claimed( return self._schedule_rate_limit(operation, worker_id, checkpoint, now, result.retry_at) if result.outcome is WriteOutcome.PERMANENT_FAILURE: checkpoint.pop("unknown_step", None) + failed_steps.add(entry.occurrence_id) issues.append( { "step": entry.occurrence_id, diff --git a/tests/test_copy_execution.py b/tests/test_copy_execution.py index 7a8954b..1d8e48b 100644 --- a/tests/test_copy_execution.py +++ b/tests/test_copy_execution.py @@ -47,15 +47,35 @@ def tearDown(self) -> None: self.plans.close() self.operations.close() - def snapshot(self, *, policy: CopyPolicy = CopyPolicy.STRICT, include_unmatched: bool = False) -> tuple[object, CopyPolicy]: + def snapshot( + self, + *, + policy: CopyPolicy = CopyPolicy.STRICT, + include_unmatched: bool = False, + include_second_ready: bool = False, + ) -> tuple[PlaylistSnapshot, CopyPolicy]: entries = [SourcePlaylistEntry("occ-1", 0, "source-1", EntryClassification.READY, "target-1")] + if include_second_ready: + entries.append(SourcePlaylistEntry("occ-2", 1, "source-2", EntryClassification.READY, "target-2")) if include_unmatched: - entries.append(SourcePlaylistEntry("occ-2", 1, "source-2", EntryClassification.UNMATCHED)) + entries.append( + SourcePlaylistEntry(f"occ-{len(entries) + 1}", len(entries), "source-3", EntryClassification.UNMATCHED) + ) source = PlaylistSnapshot("snapshot-1", "spotify", "playlist-1", tuple(entries)) return source, policy - def accepted_digest(self, *, policy: CopyPolicy = CopyPolicy.STRICT, include_unmatched: bool = False) -> str: - source, policy = self.snapshot(policy=policy, include_unmatched=include_unmatched) + def accepted_digest( + self, + *, + policy: CopyPolicy = CopyPolicy.STRICT, + include_unmatched: bool = False, + include_second_ready: bool = False, + ) -> str: + source, policy = self.snapshot( + policy=policy, + include_unmatched=include_unmatched, + include_second_ready=include_second_ready, + ) stored = self.workflow.create_plan( source, target_provider="youtube", @@ -123,6 +143,31 @@ def test_unknown_write_without_reconciliation_requires_user(self) -> None: self.assertNotIn("unknown_step", reconciled.checkpoint) self.assertEqual(self.writer.added, [(step_key, "target-1")]) + def test_resume_skips_permanent_failure_before_reconciling_later_unknown_write(self) -> None: + digest = self.accepted_digest(include_second_ready=True) + first_key = f"{digest}:entry:occ-1" + second_key = f"{digest}:entry:occ-2" + self.writer.results[first_key] = WriteResult(WriteOutcome.PERMANENT_FAILURE, provider_code="unavailable") + self.writer.results[second_key] = WriteResult(WriteOutcome.UNKNOWN_OUTCOME, detail="provider timeout") + + operation = self.executor.execute(digest, writer=self.writer, worker_id="worker-a", now=NOW) + self.assertEqual(operation.state, "waiting_user") + self.assertEqual(operation.checkpoint["unknown_step"], "occ-2") + self.assertEqual(self.writer.added, [(first_key, "target-1"), (second_key, "target-2")]) + + self.operations.resume(operation.operation_id, now=NOW + timedelta(seconds=1)) + pending = self.executor.execute(digest, writer=self.writer, worker_id="worker-b", now=NOW + timedelta(seconds=1)) + self.assertEqual(pending.state, "waiting_user") + self.assertEqual(self.writer.added, [(first_key, "target-1"), (second_key, "target-2")]) + + self.writer.reconcile_results[second_key] = True + self.operations.resume(operation.operation_id, now=NOW + timedelta(seconds=2)) + reconciled = self.executor.execute(digest, writer=self.writer, worker_id="worker-c", now=NOW + timedelta(seconds=2)) + self.assertEqual(reconciled.state, "partial") + self.assertEqual(reconciled.checkpoint["confirmed_occurrences"], ["occ-2"]) + self.assertEqual([issue["step"] for issue in reconciled.checkpoint["issues"]], ["occ-1"]) + self.assertEqual(self.writer.added, [(first_key, "target-1"), (second_key, "target-2")]) + def test_retryable_write_releases_operation_until_scheduled(self) -> None: digest = self.accepted_digest() step_key = f"{digest}:entry:occ-1" From eb3fb5efcbde2314814b2731ad031c5db1169596 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 19:21:36 +0200 Subject: [PATCH 154/167] test: align runtime and storage checks with safety contracts --- tests/test_runtime_http.py | 6 +++++- tests/test_sqlite_operations.py | 6 ++++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/tests/test_runtime_http.py b/tests/test_runtime_http.py index 030397a..00499d7 100644 --- a/tests/test_runtime_http.py +++ b/tests/test_runtime_http.py @@ -3,6 +3,7 @@ from io import BytesIO import http.client from pathlib import Path +import socket import tempfile import threading import unittest @@ -112,9 +113,12 @@ def end_headers(self) -> None: def test_composed_runtime_http_smoke_exposes_health_and_readiness(self) -> None: with tempfile.TemporaryDirectory() as directory: try: + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as probe: + probe.bind(("127.0.0.1", 0)) + port = probe.getsockname()[1] server = create_server( host="127.0.0.1", - port=0, + port=port, database_path=str(Path(directory) / "symphonia.sqlite3"), ingress_path="/symphonia", ) diff --git a/tests/test_sqlite_operations.py b/tests/test_sqlite_operations.py index 529efae..01ec351 100644 --- a/tests/test_sqlite_operations.py +++ b/tests/test_sqlite_operations.py @@ -229,13 +229,14 @@ def test_checkpoint_rejects_nested_credentials_before_persistence(self) -> None: ) self.repository.claim(operation.operation_id, worker_id="worker-a", now=self.now) - with self.assertRaisesRegex(ValueError, "access_token"): + with self.assertRaisesRegex(ValueError, "checkpoint contains credential-shaped keys") as raised: self.repository.checkpoint( operation.operation_id, worker_id="worker-a", checkpoint={"provider": {"access_token": "must-not-persist"}}, now=self.now + timedelta(seconds=1), ) + self.assertNotIn("access_token", str(raised.exception)) self.assertEqual(self.repository.get(operation.operation_id).checkpoint, {}) @@ -441,13 +442,14 @@ def test_operation_payload_rejects_credential_named_fields(self) -> None: ) def test_operation_payload_rejects_nested_credentials(self) -> None: - with self.assertRaisesRegex(ValueError, r"provider.credentials\[0\]\.access_token"): + with self.assertRaisesRegex(ValueError, "operation payload contains credential-shaped keys") as raised: self.repository.create( operation_type="copy", idempotency_key="nested-credential-payload", payload={"provider": {"credentials": [{"access_token": "must-not-persist"}]}}, now=self.now, ) + self.assertNotIn("access_token", str(raised.exception)) def test_operation_payload_rejects_cyclic_structures_before_json_encoding(self) -> None: payload: dict[str, object] = {} From 5131e454875ceef82eca6e430692db5d1dc4ee59 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 19:23:03 +0200 Subject: [PATCH 155/167] test: require target reconciliation before resumed creation --- tests/test_copy_execution.py | 41 ++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/tests/test_copy_execution.py b/tests/test_copy_execution.py index 1d8e48b..bb7da32 100644 --- a/tests/test_copy_execution.py +++ b/tests/test_copy_execution.py @@ -109,6 +109,47 @@ def test_unknown_write_is_reconciled_before_success(self) -> None: self.assertEqual(operation.state, "succeeded") self.assertEqual(self.writer.reconciled, [step_key]) + def test_unknown_target_creation_is_not_repeated_until_reconciled(self) -> None: + class UnknownTargetWriter(FakeWriter): + def __init__(self) -> None: + super().__init__() + self.create_attempts = 0 + self.target_reconciliation_attempts = 0 + self.reconciled_target: TargetPlaylist | None = None + + def ensure_target_playlist( + self, *, provider: str, name: str, visibility: str, idempotency_key: str + ) -> TargetPlaylist: + self.create_attempts += 1 + raise ProviderWriteError(WriteOutcome.UNKNOWN_OUTCOME, "timeout after create request") + + def reconcile_target_playlist(self, *, idempotency_key: str) -> TargetPlaylist | None: + self.target_reconciliation_attempts += 1 + return self.reconciled_target + + digest = self.accepted_digest() + writer = UnknownTargetWriter() + operation = self.executor.execute(digest, writer=writer, worker_id="worker-a", now=NOW) + self.assertEqual(operation.state, "waiting_user") + self.assertEqual(operation.checkpoint["unknown_step"], "target") + self.assertNotIn("target_playlist_id", operation.checkpoint) + self.assertEqual(writer.create_attempts, 1) + self.assertEqual(writer.added, []) + + self.operations.resume(operation.operation_id, now=NOW + timedelta(seconds=1)) + pending = self.executor.execute(digest, writer=writer, worker_id="worker-b", now=NOW + timedelta(seconds=1)) + self.assertEqual(pending.state, "waiting_user") + self.assertEqual(writer.create_attempts, 1) + self.assertEqual(writer.target_reconciliation_attempts, 2) + + writer.reconciled_target = writer.target + self.operations.resume(operation.operation_id, now=NOW + timedelta(seconds=2)) + reconciled = self.executor.execute(digest, writer=writer, worker_id="worker-c", now=NOW + timedelta(seconds=2)) + self.assertEqual(reconciled.state, "succeeded") + self.assertEqual(reconciled.checkpoint["target_playlist_id"], writer.target.provider_playlist_id) + self.assertEqual(writer.create_attempts, 1) + self.assertEqual(writer.added, [(f"{digest}:entry:occ-1", "target-1")]) + def test_unknown_write_without_reconciliation_requires_user(self) -> None: digest = self.accepted_digest() step_key = f"{digest}:entry:occ-1" From 8566d37a06af11e66973156c78c39eaae81abe48 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 19:25:37 +0200 Subject: [PATCH 156/167] fix: reject inconsistent copy progress before provider writes --- docs/development/implementation-baseline.md | 1 + specs/one-time-playlist-copy.md | 1 + src/symphonia/application/copy_execution.py | 50 ++++++++++++--- tests/test_copy_execution.py | 70 +++++++++++++++++++++ 4 files changed, 113 insertions(+), 9 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index bf796b6..a8c2656 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -34,6 +34,7 @@ The first implementation increment is intentionally narrower than any provider o - Copy executor can run as a claimed operation handler, preserving the same restart/checkpoint semantics under the runner. - Copy execution durably marks a target-create or entry-add step in-flight before the provider call; a resumed step is reconciled before any repeated mutation, and inconclusive evidence or reconciliation errors stay in `waiting_user`. The current boolean entry-reconciliation port treats `false` only as inconclusive and cannot represent proven no-effect; a richer provider contract and the full copy capability remain blocked in the Draft SDD. - Copy execution retains permanently failed occurrences as itemized issues across waits and restarts; resuming a later uncertain write does not repeat those failed writes. +- Before resumed writes, copy execution rejects malformed, duplicated, out-of-plan, out-of-order, or contradictory item progress and invalid target identifiers into `waiting_user` for review. - Playlist import executor persists intent before reads and reports succeeded, partial, waiting-user, retry, and rate-limit outcomes durably. Current scope is one playlist: pages are materialized in memory, and transient retries re-read from the beginning because page-level durable staging/checkpoints are not implemented; full library-import behavior remains blocked in its Draft SDD. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Direct-callback authorization can resolve a durable attempt by the returned state digest after restart without persisting raw state; ambiguous digests fail closed. diff --git a/specs/one-time-playlist-copy.md b/specs/one-time-playlist-copy.md index b3e4ee8..8207968 100644 --- a/specs/one-time-playlist-copy.md +++ b/specs/one-time-playlist-copy.md @@ -199,6 +199,7 @@ Status shall never rely only on color. Keyboard navigation, visible focus, seman | Rate limit | Enter `waiting_rate_limit` with next eligible time | Automatic bounded resume; cancellation remains available | | Some items fail permanently | Checkpoint each failure as a terminal item issue; never resend that occurrence when a later step waits or resumes; finish as `partial` with per-item reasons | Create a new remediation plan for failed entries | | Process or host restarts | Resume from durable checkpoint and lease rules | No manual action unless state becomes uncertain | +| Stored copy progress contradicts the accepted plan | Stop before another provider mutation and retain the checkpoint in `waiting_user` | Investigate or restore compatible evidence; do not infer success or retry from malformed progress | | User cancels | Stop scheduling further writes; preserve confirmed results | Review partial result; cancellation is not rollback | ## 11. Security and privacy diff --git a/src/symphonia/application/copy_execution.py b/src/symphonia/application/copy_execution.py index 5143b63..51a9b38 100644 --- a/src/symphonia/application/copy_execution.py +++ b/src/symphonia/application/copy_execution.py @@ -106,17 +106,49 @@ def _execute_claimed( checkpoint = dict(operation.checkpoint) if operation.cancel_requested: return self._finish(operation, worker_id, checkpoint, now, "cancelled") - confirmed = list(checkpoint.get("confirmed_occurrences", [])) - issues = list(checkpoint.get("issues", [])) - target_id = checkpoint.get("target_playlist_id") - unknown_step = checkpoint.get("unknown_step") writable_entries = stored.plan.writable_entries writable_ids = {entry.occurrence_id for entry in writable_entries} - failed_steps = { - issue.get("step") - for issue in issues - if isinstance(issue, dict) and isinstance(issue.get("step"), str) - } + raw_confirmed = checkpoint.get("confirmed_occurrences", []) + raw_issues = checkpoint.get("issues", []) + target_id = checkpoint.get("target_playlist_id") + unknown_step = checkpoint.get("unknown_step") + if ( + not isinstance(raw_confirmed, list) + or not all(isinstance(item, str) for item in raw_confirmed) + or not isinstance(raw_issues, list) + or (target_id is not None and (not isinstance(target_id, str) or not target_id.strip())) + ): + return self._wait_for_user( + operation, + worker_id, + checkpoint | {"failure_code": "invalid_copy_checkpoint"}, + now, + ) + confirmed = list(raw_confirmed) + issues = list(raw_issues) + confirmed_ids = set(confirmed) + issue_steps = [issue.get("step") if isinstance(issue, dict) else None for issue in issues] + if any(not isinstance(step, str) or step not in writable_ids for step in issue_steps): + return self._wait_for_user( + operation, + worker_id, + checkpoint | {"failure_code": "invalid_copy_checkpoint"}, + now, + ) + failed_steps = set(issue_steps) + if ( + len(confirmed_ids) != len(confirmed) + or not confirmed_ids <= writable_ids + or confirmed != [entry.occurrence_id for entry in writable_entries if entry.occurrence_id in confirmed_ids] + or len(failed_steps) != len(issue_steps) + or confirmed_ids & failed_steps + ): + return self._wait_for_user( + operation, + worker_id, + checkpoint | {"failure_code": "invalid_copy_checkpoint"}, + now, + ) if target_id is None and confirmed: return self._wait_for_user( diff --git a/tests/test_copy_execution.py b/tests/test_copy_execution.py index bb7da32..f93578a 100644 --- a/tests/test_copy_execution.py +++ b/tests/test_copy_execution.py @@ -86,6 +86,23 @@ def accepted_digest( ) return self.workflow.accept_plan(stored.plan.digest, now=NOW).plan.digest + def resume_with_checkpoint(self, digest: str, checkpoint: dict[str, object]) -> None: + operation = self.operations.create( + operation_type="copy_playlist", + idempotency_key=f"copy-plan:{digest}", + payload={"plan_digest": digest}, + now=NOW, + ) + self.operations.claim(operation.operation_id, worker_id="seeding-worker", now=NOW) + self.operations.checkpoint( + operation.operation_id, + worker_id="seeding-worker", + checkpoint=checkpoint, + now=NOW, + state="waiting_user", + ) + self.operations.resume(operation.operation_id, now=NOW + timedelta(seconds=1)) + def test_success_checkpoints_entries_in_source_order(self) -> None: digest = self.accepted_digest() operation = self.executor.execute(digest, writer=self.writer, worker_id="worker-a", now=NOW) @@ -209,6 +226,59 @@ def test_resume_skips_permanent_failure_before_reconciling_later_unknown_write(s self.assertEqual([issue["step"] for issue in reconciled.checkpoint["issues"]], ["occ-1"]) self.assertEqual(self.writer.added, [(first_key, "target-1"), (second_key, "target-2")]) + def test_duplicate_confirmed_checkpoint_waits_without_provider_writes(self) -> None: + digest = self.accepted_digest() + self.resume_with_checkpoint( + digest, + {"target_playlist_id": "target-playlist-1", "confirmed_occurrences": ["occ-1", "occ-1"]}, + ) + + operation = self.executor.execute(digest, writer=self.writer, worker_id="worker-b", now=NOW + timedelta(seconds=1)) + + self.assertEqual(operation.state, "waiting_user") + self.assertEqual(operation.checkpoint["failure_code"], "invalid_copy_checkpoint") + self.assertEqual(self.writer.added, []) + + def test_conflicting_failure_checkpoint_waits_without_provider_writes(self) -> None: + digest = self.accepted_digest() + self.resume_with_checkpoint( + digest, + { + "target_playlist_id": "target-playlist-1", + "confirmed_occurrences": ["occ-1"], + "issues": [{"step": "occ-1", "detail": "unavailable"}], + }, + ) + + operation = self.executor.execute(digest, writer=self.writer, worker_id="worker-b", now=NOW + timedelta(seconds=1)) + + self.assertEqual(operation.state, "waiting_user") + self.assertEqual(operation.checkpoint["failure_code"], "invalid_copy_checkpoint") + self.assertEqual(self.writer.added, []) + + def test_out_of_order_confirmations_wait_without_provider_writes(self) -> None: + digest = self.accepted_digest(include_second_ready=True) + self.resume_with_checkpoint( + digest, + {"target_playlist_id": "target-playlist-1", "confirmed_occurrences": ["occ-2", "occ-1"]}, + ) + + operation = self.executor.execute(digest, writer=self.writer, worker_id="worker-b", now=NOW + timedelta(seconds=1)) + + self.assertEqual(operation.state, "waiting_user") + self.assertEqual(operation.checkpoint["failure_code"], "invalid_copy_checkpoint") + self.assertEqual(self.writer.added, []) + + def test_malformed_target_identifier_waits_without_provider_writes(self) -> None: + digest = self.accepted_digest() + self.resume_with_checkpoint(digest, {"target_playlist_id": ["not", "an", "id"]}) + + operation = self.executor.execute(digest, writer=self.writer, worker_id="worker-b", now=NOW + timedelta(seconds=1)) + + self.assertEqual(operation.state, "waiting_user") + self.assertEqual(operation.checkpoint["failure_code"], "invalid_copy_checkpoint") + self.assertEqual(self.writer.added, []) + def test_retryable_write_releases_operation_until_scheduled(self) -> None: digest = self.accepted_digest() step_key = f"{digest}:entry:occ-1" From 8430cd8252ceb632823d73022adb810964cb8c46 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 19:27:25 +0200 Subject: [PATCH 157/167] fix: validate persisted import identity before adapter reads --- docs/development/implementation-baseline.md | 1 + ...library-import-and-provider-projections.md | 2 + .../application/library_import_execution.py | 10 ++-- tests/test_library_import_execution.py | 57 +++++++++++++++++++ 4 files changed, 66 insertions(+), 4 deletions(-) diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index a8c2656..8501e42 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -36,6 +36,7 @@ The first implementation increment is intentionally narrower than any provider o - Copy execution retains permanently failed occurrences as itemized issues across waits and restarts; resuming a later uncertain write does not repeat those failed writes. - Before resumed writes, copy execution rejects malformed, duplicated, out-of-plan, out-of-order, or contradictory item progress and invalid target identifiers into `waiting_user` for review. - Playlist import executor persists intent before reads and reports succeeded, partial, waiting-user, retry, and rate-limit outcomes durably. Current scope is one playlist: pages are materialized in memory, and transient retries re-read from the beginning because page-level durable staging/checkpoints are not implemented; full library-import behavior remains blocked in its Draft SDD. +- Resumed playlist imports reject non-textual stored playlist identifiers and conflicting provider bindings before any adapter read; malformed intent is not coerced into a new external identity. - Provider-neutral authorization attempts with hashed state, exact redirect binding, expiry, and single-use consumption. - Direct-callback authorization can resolve a durable attempt by the returned state digest after restart without persisting raw state; ambiguous digests fail closed. - Authorization consumption is covered across independent SQLite connections so callback replay races produce exactly one consumed attempt. diff --git a/specs/library-import-and-provider-projections.md b/specs/library-import-and-provider-projections.md index d695c3c..4ae3d0b 100644 --- a/specs/library-import-and-provider-projections.md +++ b/specs/library-import-and-provider-projections.md @@ -35,6 +35,8 @@ There is no complete multi-collection library importer. The owner-approved found The durable wrapper persists import intent before provider reads and records terminal publication details, authentication waits, rate-limit waits, and bounded transient retries. It does not durably checkpoint individual pages or stage them incrementally; after a transient failure it re-reads the playlist from the beginning. Saved-track collections, multi-collection sessions, retention/deletion scheduling, export policy, and representative-size guarantees are not part of this foundation evidence. This paragraph is an as-built foundation boundary, not a claim that the SDD capability is implemented. +The foundation executor checks stored playlist identifier types and the operation/playlist provider binding before invoking an adapter; malformed persisted intent fails without a provider read. + ### 2.3 Evidence and unknowns - Shared model: [provider track, playlist, snapshot, and import invariants](../docs/domain/domain-model.md). diff --git a/src/symphonia/application/library_import_execution.py b/src/symphonia/application/library_import_execution.py index fa2e81a..f0fee2b 100644 --- a/src/symphonia/application/library_import_execution.py +++ b/src/symphonia/application/library_import_execution.py @@ -203,14 +203,16 @@ def _payload(payload: Mapping[str, Any]) -> tuple[str, ProviderObjectRef, str, d raise ValueError("import operation payload has no playlist reference") try: playlist = ProviderObjectRef( - str(raw_playlist["provider"]), - str(raw_playlist["object_type"]), - str(raw_playlist["object_id"]), - str(raw_playlist["namespace"]), + raw_playlist["provider"], + raw_playlist["object_type"], + raw_playlist["object_id"], + raw_playlist["namespace"], ) parsed_observed_at = datetime.fromisoformat(observed_at) except (KeyError, TypeError, ValueError) as error: raise ValueError("import operation payload has an invalid playlist or timestamp") from error + if payload.get("provider") != playlist.provider: + raise ValueError("import operation provider does not match playlist") if parsed_observed_at.tzinfo is None: raise ValueError("import operation observed_at must be timezone-aware") return connection_id, playlist, snapshot_id, parsed_observed_at.astimezone(timezone.utc) diff --git a/tests/test_library_import_execution.py b/tests/test_library_import_execution.py index 17c142b..0e9ad1b 100644 --- a/tests/test_library_import_execution.py +++ b/tests/test_library_import_execution.py @@ -26,11 +26,13 @@ class FakeAdapter: def __init__(self, *, error: ProviderApiError | None = None, complete: bool = True) -> None: self.error = error self.complete = complete + self.read_calls = 0 def capabilities(self, connection_id: str): raise AssertionError("not used") def read_playlist_pages(self, connection_id: str, playlist: ProviderObjectRef, cursor: str | None = None): + self.read_calls += 1 if self.error is not None: raise self.error entry = ProviderPlaylistEntry( @@ -118,6 +120,61 @@ def test_transient_import_failure_honors_retry_budget(self) -> None: self.assertEqual(failed.state, "failed") self.assertEqual(failed.checkpoint["failure_code"], "retry_exhausted") + def test_malformed_persisted_playlist_id_never_reaches_adapter(self) -> None: + adapter = FakeAdapter() + created = self.operations.create( + operation_type="import_playlist", + idempotency_key="malformed-import-payload", + payload={ + "provider": "spotify", + "connection_id": "connection-1", + "playlist": { + "provider": "spotify", + "object_type": "playlist", + "object_id": ["not", "an", "id"], + "namespace": "connection-1", + }, + "snapshot_id": "snapshot-malformed", + "observed_at": NOW.isoformat(), + }, + now=NOW, + ) + claimed = self.operations.claim(created.operation_id, worker_id="worker-a", now=NOW) + + result = self.service.execute_claimed(claimed, adapter=adapter, worker_id="worker-a", now=NOW) + + self.assertEqual(result.state, "failed") + self.assertEqual(adapter.read_calls, 0) + self.assertIsNone( + self.projections.current(provider="spotify", namespace="connection-1", playlist_id="playlist-1") + ) + + def test_conflicting_persisted_provider_never_reaches_adapter(self) -> None: + adapter = FakeAdapter() + created = self.operations.create( + operation_type="import_playlist", + idempotency_key="conflicting-import-provider", + payload={ + "provider": "youtube", + "connection_id": "connection-1", + "playlist": { + "provider": "spotify", + "object_type": "playlist", + "object_id": "playlist-1", + "namespace": "connection-1", + }, + "snapshot_id": "snapshot-conflicting", + "observed_at": NOW.isoformat(), + }, + now=NOW, + ) + claimed = self.operations.claim(created.operation_id, worker_id="worker-a", now=NOW) + + result = self.service.execute_claimed(claimed, adapter=adapter, worker_id="worker-a", now=NOW) + + self.assertEqual(result.state, "failed") + self.assertEqual(adapter.read_calls, 0) + if __name__ == "__main__": unittest.main() From 1ce6881e35f5ec7210ba4c845bece3b5fa9c32fc Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 19:39:10 +0200 Subject: [PATCH 158/167] refactor: share provider JSON transport and guard architecture --- .gitignore | 2 + docs/README.md | 1 + docs/development/implementation-baseline.md | 1 + docs/development/quality-audit.md | 35 ++++++++ specs/catalog.json | 2 + ...library-import-and-provider-projections.md | 2 + src/symphonia/providers/http_json.py | 86 +++++++++++++++++++ src/symphonia/providers/spotify.py | 74 +--------------- src/symphonia/providers/youtube.py | 7 +- tests/test_architecture_boundaries.py | 81 ++++++++++++++++- tests/test_youtube_adapter.py | 18 ++++ 11 files changed, 232 insertions(+), 77 deletions(-) create mode 100644 docs/development/quality-audit.md create mode 100644 src/symphonia/providers/http_json.py diff --git a/.gitignore b/.gitignore index b6577ee..321f050 100644 --- a/.gitignore +++ b/.gitignore @@ -9,3 +9,5 @@ build/ .venv/ *.sqlite3 *.db +.repowise/ +graphify-out/ diff --git a/docs/README.md b/docs/README.md index 8bcc22d..d3f067c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -18,6 +18,7 @@ This documentation is the horizontal implementation contract for Symphonia. It d | [Provider research](providers/provider-research.md) | Dated, sourced facts about provider APIs | Product policy or permanent architecture | | [Home Assistant music ecosystem review](providers/home-assistant-ecosystem-review.md) | Reusable patterns and cautions from existing HA music projects | Dependency selection or provider guarantees | | [Development specification](development/development-specification.md) | Specification workflow, testing and delivery gates | Product scope | +| [Local quality audit](development/quality-audit.md) | Reproducible offline RepoWise/Graphify review and current architecture/test debt | A release or SDD readiness claim | | [ADRs](decisions/README.md) | Decisions that have actually been accepted | Proposals and guesses | | [Open questions](open-questions.md) | Decisions needed, assumptions, risk register, next design work | Accepted requirements | diff --git a/docs/development/implementation-baseline.md b/docs/development/implementation-baseline.md index 8501e42..3c8869c 100644 --- a/docs/development/implementation-baseline.md +++ b/docs/development/implementation-baseline.md @@ -56,6 +56,7 @@ The first implementation increment is intentionally narrower than any provider o - Deterministic provider adapter registry with manifest discovery and duplicate-provider protection. - Concrete provider manifests expose upstream dependencies and a dated research review marker. - Offline-testable official Spotify playlist reader/writer with bounded pagination, explicit write-capability gating, and normalized error categories. +- Spotify and YouTube Data share a provider-neutral JSON transport; the YouTube adapter no longer imports the Spotify adapter, and default transport errors identify the correct provider without echoing tokens. - Spotify write adapters reject blank playlist/entry identifiers and unknown visibility values before issuing provider requests. - Explicitly scoped official YouTube Data API video-playlist reader; it is not represented as YouTube Music. - Experimental official Apple Music library-playlist reader with separate developer/user token inputs. diff --git a/docs/development/quality-audit.md b/docs/development/quality-audit.md new file mode 100644 index 0000000..d4cab36 --- /dev/null +++ b/docs/development/quality-audit.md @@ -0,0 +1,35 @@ +# Local quality audit + +**Snapshot:** 2026-09-27 on `develop`. This is a prioritization aid, not a release gate or a claim that a Draft SDD is implemented. The [development specification](development-specification.md) and [SDD catalog](../../specs/CATALOG.md) remain authoritative. + +## Reproduce without provider access or an LLM + +From the repository root, with the optional `repowise` and `graphify` CLIs installed: + +```text +repowise health . --refactoring-targets +repowise dead-code . --safe-only +audit_dir="$(mktemp -d /tmp/symphonia-graph-audit.XXXXXX)" +graphify extract . --code-only --no-cluster --out "$audit_dir" +graphify god-nodes --top 20 --graph "$audit_dir/graphify-out/graph.json" +graphify affected OperationRepository --depth 1 --graph "$audit_dir/graphify-out/graph.json" +make verify +``` + +Use a fresh temporary output directory for each graph extraction. `--code-only` keeps Graphify's extraction local and skips LLM/document processing. RepoWise's `health` command also runs locally without an LLM. Neither tool is required by `make verify`; generated `.repowise/` and `graphify-out/` data are ignored by Git. Do not run LLM-backed generation, repository-history secret scans, or upload/export commands with private provider fixtures without a separate privacy review. + +## Findings and interpretation + +- RepoWise 0.45.0 reported `copy_execution.py` as a maintainability hotspot (score 1.65/10, maximum cyclomatic complexity 79), followed by `sqlite_operations.py` (score 1.65/10, 1,308 nonblank lines). These are heuristic scores, not failures by themselves. In particular, the copy executor coordinates uncertain external writes and must be refactored only with restart/reconciliation characterization tests. +- Its reported line and branch coverage were `null`: the current test command runs deterministic `unittest` cases but produces no coverage report. Test count must not be presented as coverage. Add measured branch coverage before using a numeric coverage gate. +- After the transport extraction, Graphify's local AST graph contained 1,111 nodes and 2,872 edges. `OperationRepository` (56 edges) remained the most connected node. A change to operation persistence has a broad impact across the copy/import runners, runtime, and tests; inspect reverse dependencies before modifying its contract. +- Graphify makes the existing direct application-to-SQLite imports visible. The owner-approved foundation permits the current concrete composition, but an implementation-ready architecture should define repository ports and keep concrete stores at the composition boundary. This remains design debt, not a silently accepted dependency rule. +- The graph also exposed a YouTube Data adapter dependency on `spotify.py` for JSON transport. The transport now lives in `providers/http_json.py`, with an architecture guard against sibling-adapter imports and an offline test for correctly attributed, secret-safe failures. +- RepoWise marked `source_entries` as an unused export. It is a public convenience helper; static usage alone is insufficient evidence to remove a public API. + +## Quality work in order + +1. Preserve and expand fault-injection tests for unknown writes, cancellation, restart, and malformed checkpoints before reducing `copy_execution.py` complexity. +2. Define application repository ports and a composition rule in the relevant SDD/architecture documents before removing the current direct SQLite dependencies. +3. Measure line and branch coverage with an explicitly selected development tool, then target missing high-risk branches instead of a global percentage alone. +4. Keep architecture import tests and `make verify` as the dependency-free baseline; review graph/health deltas as advisory evidence rather than automatically deleting or rewriting code. diff --git a/specs/catalog.json b/specs/catalog.json index 3ce6dd1..e9a0260 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -252,12 +252,14 @@ "implementationEvidence": { "code": [ "src/symphonia/providers/importing.py", + "src/symphonia/providers/http_json.py", "src/symphonia/application/library_import.py", "src/symphonia/application/library_import_execution.py", "src/symphonia/infrastructure/sqlite_library.py" ], "tests": [ "tests/test_provider_import.py", + "tests/test_youtube_adapter.py", "tests/test_library_import.py", "tests/test_library_import_execution.py", "tests/test_sqlite_library.py" diff --git a/specs/library-import-and-provider-projections.md b/specs/library-import-and-provider-projections.md index 4ae3d0b..8ca047c 100644 --- a/specs/library-import-and-provider-projections.md +++ b/specs/library-import-and-provider-projections.md @@ -37,6 +37,8 @@ The durable wrapper persists import intent before provider reads and records ter The foundation executor checks stored playlist identifier types and the operation/playlist provider binding before invoking an adapter; malformed persisted intent fails without a provider read. +The official Spotify and YouTube Data foundation adapters use a shared JSON transport instead of importing one adapter through the other. Offline tests confirm provider-specific, secret-safe timeout and network error classification; this does not establish the full provider import capability. + ### 2.3 Evidence and unknowns - Shared model: [provider track, playlist, snapshot, and import invariants](../docs/domain/domain-model.md). diff --git a/src/symphonia/providers/http_json.py b/src/symphonia/providers/http_json.py new file mode 100644 index 0000000..c6f9e70 --- /dev/null +++ b/src/symphonia/providers/http_json.py @@ -0,0 +1,86 @@ +"""Provider-neutral JSON HTTP transport shared by official adapters.""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass +import json +from typing import Any, Protocol +from urllib.error import HTTPError, URLError +from urllib.parse import urlencode +from urllib.request import Request, urlopen + +from .errors import ProviderApiError, ProviderErrorCategory + + +@dataclass(frozen=True, slots=True) +class JsonResponse: + status: int + payload: Mapping[str, Any] + headers: Mapping[str, str] + + +class JsonClient(Protocol): + def request( + self, + method: str, + path: str, + *, + token: str, + query: Mapping[str, str], + body: Mapping[str, Any] | None = None, + ) -> JsonResponse: ... + + +class UrllibJsonClient: + """Bounded JSON transport; the default endpoint preserves Spotify callers.""" + + def __init__( + self, + base_url: str = "https://api.spotify.com/v1", + timeout_seconds: float = 10.0, + provider_label: str = "Spotify", + ) -> None: + if timeout_seconds <= 0: + raise ValueError("timeout_seconds must be positive") + if not isinstance(provider_label, str) or not provider_label.strip(): + raise ValueError("provider_label must be non-empty text") + self.base_url = base_url.rstrip("/") + self.timeout_seconds = timeout_seconds + self.provider_label = provider_label + + def request( + self, + method: str, + path: str, + *, + token: str, + query: Mapping[str, str], + body: Mapping[str, Any] | None = None, + ) -> JsonResponse: + url = f"{self.base_url}/{path.lstrip('/')}" + if query: + url = f"{url}?{urlencode(query)}" + encoded_body = None if body is None else json.dumps(body, ensure_ascii=False, separators=(",", ":")).encode("utf-8") + headers = { + "Authorization": f"Bearer {token}", + "Accept": "application/json", + **({"Content-Type": "application/json"} if body is not None else {}), + } + request = Request(url, data=encoded_body, method=method, headers=headers) + try: + with urlopen(request, timeout=self.timeout_seconds) as response: + body = response.read() + payload = json.loads(body.decode("utf-8")) if body else {} + return JsonResponse(response.status, payload, dict(response.headers.items())) + except HTTPError as error: + body = error.read() + try: + payload = json.loads(body.decode("utf-8")) if body else {} + except (UnicodeDecodeError, json.JSONDecodeError): + payload = {} + return JsonResponse(error.code, payload, dict(error.headers.items())) + except TimeoutError as error: + raise ProviderApiError(ProviderErrorCategory.TIMEOUT, f"{self.provider_label} request timed out") from error + except URLError as error: + raise ProviderApiError(ProviderErrorCategory.NETWORK_ERROR, f"{self.provider_label} request failed") from error diff --git a/src/symphonia/providers/spotify.py b/src/symphonia/providers/spotify.py index b0f39ce..2586fb0 100644 --- a/src/symphonia/providers/spotify.py +++ b/src/symphonia/providers/spotify.py @@ -8,13 +8,9 @@ from __future__ import annotations from collections.abc import Callable, Mapping -from dataclasses import dataclass from datetime import datetime, timedelta, timezone -import json -from typing import Any, Protocol -from urllib.error import HTTPError, URLError -from urllib.parse import parse_qs, urlencode, urlsplit -from urllib.request import Request, urlopen +from typing import Any +from urllib.parse import parse_qs, urlsplit from .contracts import ( AccessBasis, @@ -28,74 +24,10 @@ ProviderPlaylistPage, ) from .errors import ProviderApiError, ProviderErrorCategory +from .http_json import JsonClient, JsonResponse, UrllibJsonClient from .writing import PlaylistWriter, ProviderWriteError, TargetPlaylist, WriteOutcome, WriteResult -@dataclass(frozen=True, slots=True) -class JsonResponse: - status: int - payload: Mapping[str, Any] - headers: Mapping[str, str] - - -class JsonClient(Protocol): - def request( - self, - method: str, - path: str, - *, - token: str, - query: Mapping[str, str], - body: Mapping[str, Any] | None = None, - ) -> JsonResponse: ... - - -class UrllibJsonClient: - """Small standard-library transport with bounded request timeout.""" - - def __init__(self, base_url: str = "https://api.spotify.com/v1", timeout_seconds: float = 10.0) -> None: - if timeout_seconds <= 0: - raise ValueError("timeout_seconds must be positive") - self.base_url = base_url.rstrip("/") - self.timeout_seconds = timeout_seconds - - def request( - self, - method: str, - path: str, - *, - token: str, - query: Mapping[str, str], - body: Mapping[str, Any] | None = None, - ) -> JsonResponse: - url = f"{self.base_url}/{path.lstrip('/')}" - if query: - url = f"{url}?{urlencode(query)}" - encoded_body = None if body is None else json.dumps(body, ensure_ascii=False, separators=(",", ":")).encode("utf-8") - headers = { - "Authorization": f"Bearer {token}", - "Accept": "application/json", - **({"Content-Type": "application/json"} if body is not None else {}), - } - request = Request(url, data=encoded_body, method=method, headers=headers) - try: - with urlopen(request, timeout=self.timeout_seconds) as response: - body = response.read() - payload = json.loads(body.decode("utf-8")) if body else {} - return JsonResponse(response.status, payload, dict(response.headers.items())) - except HTTPError as error: - body = error.read() - try: - payload = json.loads(body.decode("utf-8")) if body else {} - except (UnicodeDecodeError, json.JSONDecodeError): - payload = {} - return JsonResponse(error.code, payload, dict(error.headers.items())) - except TimeoutError as error: - raise ProviderApiError(ProviderErrorCategory.TIMEOUT, "Spotify request timed out") from error - except URLError as error: - raise ProviderApiError(ProviderErrorCategory.NETWORK_ERROR, "Spotify request failed") from error - - class SpotifyAdapter(ProviderAdapter, PlaylistWriter): """Translate Spotify playlist pages into Symphonia provider values.""" diff --git a/src/symphonia/providers/youtube.py b/src/symphonia/providers/youtube.py index ccc52a5..8d455b0 100644 --- a/src/symphonia/providers/youtube.py +++ b/src/symphonia/providers/youtube.py @@ -22,7 +22,7 @@ ProviderPlaylistPage, ) from .errors import ProviderApiError, ProviderErrorCategory -from .spotify import JsonClient, UrllibJsonClient +from .http_json import JsonClient, UrllibJsonClient class YouTubeDataAdapter(ProviderAdapter): @@ -50,7 +50,10 @@ def __init__( raise ValueError("YouTube playlist page_size must be between 1 and 50") if isinstance(max_pages, bool) or not isinstance(max_pages, int) or max_pages <= 0: raise ValueError("YouTube max_pages must be positive") - self._client = client or UrllibJsonClient("https://www.googleapis.com/youtube/v3") + self._client = client or UrllibJsonClient( + "https://www.googleapis.com/youtube/v3", + provider_label="YouTube Data", + ) self._token_for_connection = token_for_connection self._page_size = page_size self._api_key = api_key diff --git a/tests/test_architecture_boundaries.py b/tests/test_architecture_boundaries.py index d5d0a95..a7816de 100644 --- a/tests/test_architecture_boundaries.py +++ b/tests/test_architecture_boundaries.py @@ -10,18 +10,50 @@ SOURCE_ROOT = ROOT / "src" / "symphonia" -def _absolute_imports(path: Path) -> list[str]: - tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) +def _resolved_imports(tree: ast.AST, package: str) -> list[str]: + package_parts = package.split(".") imports: list[str] = [] for node in ast.walk(tree): if isinstance(node, ast.Import): imports.extend(alias.name for alias in node.names) - elif isinstance(node, ast.ImportFrom) and node.level == 0 and node.module: - imports.append(node.module) + elif isinstance(node, ast.ImportFrom): + if node.level: + if node.level > len(package_parts): + raise ValueError("relative import escapes the package") + base_parts = package_parts[: len(package_parts) - node.level + 1] + if node.module: + base_parts.extend(node.module.split(".")) + base = ".".join(base_parts) + else: + base = node.module or "" + if base: + imports.append(base) + imports.extend(f"{base}.{alias.name}" for alias in node.names if alias.name != "*") return imports +def _absolute_imports(path: Path) -> list[str]: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) + relative_parts = path.relative_to(SOURCE_ROOT).with_suffix("").parts + package_parts = ("symphonia", *relative_parts[:-1]) + return _resolved_imports(tree, ".".join(package_parts)) + + class ArchitectureBoundaryTests(unittest.TestCase): + def test_relative_and_package_imports_are_resolved_before_checking_boundaries(self) -> None: + imports = _resolved_imports( + ast.parse( + "from ..infrastructure import sqlite_operations\n" + "from .. import runtime\n" + "from symphonia import providers\n" + ), + "symphonia.domain", + ) + self.assertIn("symphonia.infrastructure", imports) + self.assertIn("symphonia.infrastructure.sqlite_operations", imports) + self.assertIn("symphonia.runtime", imports) + self.assertIn("symphonia.providers", imports) + def test_domain_does_not_import_adapters_or_host_frameworks(self) -> None: forbidden_prefixes = ( "symphonia.infrastructure", @@ -51,6 +83,47 @@ def test_foundation_source_uses_only_stdlib_and_local_modules(self) -> None: violations.append(f"{path.relative_to(ROOT)} imports {module}") self.assertEqual(violations, []) + def test_provider_and_identity_layers_do_not_import_outer_layers(self) -> None: + forbidden_prefixes = ( + "symphonia.application", + "symphonia.infrastructure", + "symphonia.runtime", + ) + violations = [] + for layer in ("providers", "identity"): + for path in sorted((SOURCE_ROOT / layer).glob("*.py")): + for module in _absolute_imports(path): + if module.startswith(forbidden_prefixes): + violations.append(f"{path.relative_to(ROOT)} imports {module}") + self.assertEqual(violations, []) + + def test_application_does_not_import_runtime_or_concrete_provider_adapters(self) -> None: + forbidden_prefixes = ( + "symphonia.runtime", + "symphonia.providers.spotify", + "symphonia.providers.youtube", + "symphonia.providers.apple_music", + ) + violations = [] + for path in sorted((SOURCE_ROOT / "application").glob("*.py")): + for module in _absolute_imports(path): + if module == "symphonia.providers" or module.startswith(forbidden_prefixes): + violations.append(f"{path.relative_to(ROOT)} imports {module}") + self.assertEqual(violations, []) + + def test_provider_adapters_do_not_import_sibling_adapters(self) -> None: + adapter_names = {"spotify", "youtube", "apple_music"} + violations = [] + for adapter_name in sorted(adapter_names): + path = SOURCE_ROOT / "providers" / f"{adapter_name}.py" + forbidden = tuple( + f"symphonia.providers.{other}" for other in adapter_names - {adapter_name} + ) + for module in _absolute_imports(path): + if module.startswith(forbidden): + violations.append(f"{path.relative_to(ROOT)} imports {module}") + self.assertEqual(violations, []) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_youtube_adapter.py b/tests/test_youtube_adapter.py index 626a8d0..2b173c9 100644 --- a/tests/test_youtube_adapter.py +++ b/tests/test_youtube_adapter.py @@ -1,6 +1,8 @@ from __future__ import annotations import unittest +from unittest.mock import patch +from urllib.error import URLError from symphonia.providers import ( Capability, @@ -61,6 +63,22 @@ def request(self, method: str, path: str, *, token: str, query: dict[str, str], class YouTubeDataAdapterTests(unittest.TestCase): + def test_default_transport_reports_youtube_network_failures_without_secrets(self) -> None: + adapter = YouTubeDataAdapter(None, lambda connection_id: "token=private-value") + for failure, category in ( + (TimeoutError("private-value"), ProviderErrorCategory.TIMEOUT), + (URLError("private-value"), ProviderErrorCategory.NETWORK_ERROR), + ): + with self.subTest(category=category), patch( + "symphonia.providers.http_json.urlopen", side_effect=failure + ): + with self.assertRaises(ProviderApiError) as raised: + adapter.capabilities("google-connection-1") + self.assertEqual(raised.exception.category, category) + self.assertIn("YouTube Data", str(raised.exception)) + self.assertNotIn("Spotify", str(raised.exception)) + self.assertNotIn("private-value", str(raised.exception)) + def test_official_video_playlist_pages_preserve_unavailable_items(self) -> None: client = FakeClient() adapter = YouTubeDataAdapter(client, lambda connection_id: "access-token", page_size=2, api_key="public-key") From 1112a6ab83032489dc914d76a9ea198e9ab07d2f Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 19:41:47 +0200 Subject: [PATCH 159/167] refactor: centralize copy confirmations and constant-time replay checks --- docs/development/quality-audit.md | 1 + src/symphonia/application/copy_execution.py | 61 +++++++++++---------- tests/test_copy_execution.py | 34 ++++++++++++ 3 files changed, 66 insertions(+), 30 deletions(-) diff --git a/docs/development/quality-audit.md b/docs/development/quality-audit.md index d4cab36..0f0fa29 100644 --- a/docs/development/quality-audit.md +++ b/docs/development/quality-audit.md @@ -21,6 +21,7 @@ Use a fresh temporary output directory for each graph extraction. `--code-only` ## Findings and interpretation - RepoWise 0.45.0 reported `copy_execution.py` as a maintainability hotspot (score 1.65/10, maximum cyclomatic complexity 79), followed by `sqlite_operations.py` (score 1.65/10, 1,308 nonblank lines). These are heuristic scores, not failures by themselves. In particular, the copy executor coordinates uncertain external writes and must be refactored only with restart/reconciliation characterization tests. +- A bounded copy-executor change consolidated the repeated confirmation checkpoint and replaced list membership in the write loop with a synchronized set. RepoWise no longer reports its `membership_test_against_list_in_loop` marker; the maximum complexity remains 79, so the broader state-machine refactor is still open. - Its reported line and branch coverage were `null`: the current test command runs deterministic `unittest` cases but produces no coverage report. Test count must not be presented as coverage. Add measured branch coverage before using a numeric coverage gate. - After the transport extraction, Graphify's local AST graph contained 1,111 nodes and 2,872 edges. `OperationRepository` (56 edges) remained the most connected node. A change to operation persistence has a broad impact across the copy/import runners, runtime, and tests; inspect reverse dependencies before modifying its contract. - Graphify makes the existing direct application-to-SQLite imports visible. The owner-approved foundation permits the current concrete composition, but an implementation-ready architecture should define repository ports and keep concrete stores at the composition boundary. This remains design debt, not a silently accepted dependency rule. diff --git a/src/symphonia/application/copy_execution.py b/src/symphonia/application/copy_execution.py index 51a9b38..b56754e 100644 --- a/src/symphonia/application/copy_execution.py +++ b/src/symphonia/application/copy_execution.py @@ -163,14 +163,14 @@ def _execute_claimed( ( entry.occurrence_id for entry in writable_entries - if entry.occurrence_id not in confirmed and entry.occurrence_id not in failed_steps + if entry.occurrence_id not in confirmed_ids and entry.occurrence_id not in failed_steps ), None, ) if unknown_step != "target" and ( not isinstance(unknown_step, str) or unknown_step not in writable_ids - or unknown_step in confirmed + or unknown_step in confirmed_ids or unknown_step != first_unconfirmed or target_id is None ): @@ -298,7 +298,7 @@ def _execute_claimed( return operation for entry in writable_entries: - if entry.occurrence_id in confirmed or entry.occurrence_id in failed_steps: + if entry.occurrence_id in confirmed_ids or entry.occurrence_id in failed_steps: continue step_key = f"{digest}:entry:{entry.occurrence_id}" if unknown_step == entry.occurrence_id: @@ -321,15 +321,8 @@ def _execute_claimed( return self._wait_for_user(operation, worker_id, checkpoint, now) if not reconciled: return self._wait_for_user(operation, worker_id, checkpoint, now) - confirmed.append(entry.occurrence_id) - checkpoint["confirmed_occurrences"] = confirmed - checkpoint.pop("unknown_step", None) - operation = self.operations.checkpoint( - operation.operation_id, - worker_id=worker_id, - checkpoint=checkpoint, - now=now, - state="running", + operation = self._record_confirmation( + operation, worker_id, checkpoint, confirmed, confirmed_ids, entry.occurrence_id, now ) if operation.state != "running": return operation @@ -391,15 +384,8 @@ def _execute_claimed( except Exception: return self._wait_for_user(operation, worker_id, checkpoint, now) if reconciled: - confirmed.append(entry.occurrence_id) - checkpoint["confirmed_occurrences"] = confirmed - checkpoint.pop("unknown_step", None) - operation = self.operations.checkpoint( - operation.operation_id, - worker_id=worker_id, - checkpoint=checkpoint, - now=now, - state="running", + operation = self._record_confirmation( + operation, worker_id, checkpoint, confirmed, confirmed_ids, entry.occurrence_id, now ) if operation.state != "running": return operation @@ -434,15 +420,8 @@ def _execute_claimed( if operation.state != "running": return operation continue - confirmed.append(entry.occurrence_id) - checkpoint["confirmed_occurrences"] = confirmed - checkpoint.pop("unknown_step", None) - operation = self.operations.checkpoint( - operation.operation_id, - worker_id=worker_id, - checkpoint=checkpoint, - now=now, - state="running", + operation = self._record_confirmation( + operation, worker_id, checkpoint, confirmed, confirmed_ids, entry.occurrence_id, now ) if operation.state != "running": return operation @@ -453,6 +432,28 @@ def _execute_claimed( terminal = "partial" if stored.plan.omitted_entries or issues else "succeeded" return self._finish(operation, worker_id, checkpoint, now, terminal) + def _record_confirmation( + self, + operation: OperationRecord, + worker_id: str, + checkpoint: dict[str, Any], + confirmed: list[str], + confirmed_ids: set[str], + occurrence_id: str, + now: datetime, + ) -> OperationRecord: + confirmed.append(occurrence_id) + confirmed_ids.add(occurrence_id) + checkpoint["confirmed_occurrences"] = confirmed + checkpoint.pop("unknown_step", None) + return self.operations.checkpoint( + operation.operation_id, + worker_id=worker_id, + checkpoint=checkpoint, + now=now, + state="running", + ) + def _renew_or_stop( self, operation: OperationRecord, diff --git a/tests/test_copy_execution.py b/tests/test_copy_execution.py index f93578a..22aaaed 100644 --- a/tests/test_copy_execution.py +++ b/tests/test_copy_execution.py @@ -110,6 +110,40 @@ def test_success_checkpoints_entries_in_source_order(self) -> None: self.assertEqual([track for _, track in self.writer.added], ["target-1"]) self.assertEqual(operation.checkpoint["confirmed_occurrences"], ["occ-1"]) + def test_duplicate_target_tracks_keep_distinct_ordered_occurrences(self) -> None: + source = PlaylistSnapshot( + "snapshot-duplicates", + "spotify", + "playlist-1", + ( + SourcePlaylistEntry("occ-1", 0, "source-1", EntryClassification.READY, "target-shared"), + SourcePlaylistEntry("occ-2", 1, "source-1", EntryClassification.READY, "target-shared"), + SourcePlaylistEntry("occ-3", 2, "source-2", EntryClassification.READY, "target-other"), + ), + ) + stored = self.workflow.create_plan( + source, + target_provider="youtube", + target_playlist_name="Duplicates", + target_visibility="private", + policy=CopyPolicy.STRICT, + now=NOW, + ) + digest = self.workflow.accept_plan(stored.plan.digest, now=NOW).plan.digest + + operation = self.executor.execute(digest, writer=self.writer, worker_id="worker-a", now=NOW) + + self.assertEqual(operation.state, "succeeded") + self.assertEqual(operation.checkpoint["confirmed_occurrences"], ["occ-1", "occ-2", "occ-3"]) + self.assertEqual( + self.writer.added, + [ + (f"{digest}:entry:occ-1", "target-shared"), + (f"{digest}:entry:occ-2", "target-shared"), + (f"{digest}:entry:occ-3", "target-other"), + ], + ) + def test_best_effort_omission_is_partial_not_success(self) -> None: digest = self.accepted_digest(policy=CopyPolicy.BEST_EFFORT, include_unmatched=True) operation = self.executor.execute(digest, writer=self.writer, worker_id="worker-a", now=NOW) From ccd476bc7c15929bae9b99ee375852911847dd60 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 20:01:40 +0200 Subject: [PATCH 160/167] docs: add isolated Home Assistant-adjacent UI review fixture --- README.md | 1 + docs/development/ui-spike/README.md | 23 +++++ docs/development/ui-spike/demo.js | 25 ++++++ docs/development/ui-spike/index.html | 65 ++++++++++++++ docs/development/ui-spike/styles.css | 124 +++++++++++++++++++++++++++ specs/catalog.json | 3 +- specs/home-assistant-native-ui.md | 5 +- tests/test_ui_spike.py | 78 +++++++++++++++++ 8 files changed, 321 insertions(+), 3 deletions(-) create mode 100644 docs/development/ui-spike/README.md create mode 100644 docs/development/ui-spike/demo.js create mode 100644 docs/development/ui-spike/index.html create mode 100644 docs/development/ui-spike/styles.css create mode 100644 tests/test_ui_spike.py diff --git a/README.md b/README.md index ca2902a..83d9b48 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,7 @@ Start with the [documentation map](docs/README.md). The horizontal specification | SDD lifecycle, template, and readiness gate | [SDD standard](specs/README.md) | | Vision, scope, journeys, requirements | [Product specification](docs/product/product-specification.md) | | Home Assistant-native UI and component contract | [UI specification](docs/product/home-assistant-ui-specification.md) and [UI foundation SDD](specs/home-assistant-native-ui.md) | +| Non-production Home Assistant-adjacent UI review | [Synthetic UI fixture](docs/development/ui-spike/README.md) | | Vocabulary, entities, identity, playlists | [Domain model](docs/domain/domain-model.md) | | System boundaries and operational qualities | [Architecture](docs/architecture/system-architecture.md) | | Provider contract and capability semantics | [Provider specification](docs/providers/provider-specification.md) | diff --git a/docs/development/ui-spike/README.md b/docs/development/ui-spike/README.md new file mode 100644 index 0000000..0c2ea2a --- /dev/null +++ b/docs/development/ui-spike/README.md @@ -0,0 +1,23 @@ +# Home Assistant-adjacent UI spike (review fixture) + +This is a **non-production, synthetic-data-only** visual/interaction fixture for the [UI foundation SDD](../../../specs/home-assistant-native-ui.md). It is not served by the App, has no application/API calls, does not perform copy or connect providers, and does not select Symphonia's frontend framework. Open `index.html` locally to inspect the shell. The controls only change the local fixture view/theme. + +Automated browser inspection of this local file was blocked by the current browser security policy on 2026-09-27. The repository tests check structure and isolation, but **no Symphonia screenshot, visual parity, responsive layout, or browser accessibility result has been approved**. A reviewer must perform the inspection below in an authorized environment; a future release cannot substitute static tests for it. + +## Evidence and intentional choices + +- Reviewed 2026-09-27 against the public [Home Assistant demo reference set](https://github.com/vypdev/homeassistant-gateway/tree/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/ui-reference/home-assistant) captured 2026-08-05 (especially `menu-settings-light.png`, `menu-automations-light.png`, and `menu-automations-mobile.png`). Those captures are dated evidence, **not** a current supported-version matrix or proof of dark theme behavior. +- Compared with the [Gateway component catalog](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/frontend-ui-catalog.md) at pinned commit `1ed75be9f8fabdab386db0fe4320cfb0f67d4f42`: opaque surfaces, quiet borders, flat selected navigation, compact status metadata, restrained action hierarchy. This fixture uses independently written CSS and native HTML controls; it does not copy Gateway components or assets. +- The top bar represents the **App interior**, not a replacement Home Assistant sidebar. Home Assistant owns its outer navigation; duplicating that shell inside Ingress would create a nested app. This is an intentional compositional distinction from Gateway's Home Assistant *reference* story. +- State copy follows the SDD: partial import identifies retained data and retry; copy preview never implies a write; completed/blocked operation examples show confirmed facts and recovery. All names/counts are fabricated. + +## Inspection checklist + +1. Open `index.html` and inspect Connections, Library, Copy, and Activity in a 1280×720 viewport and at 390×844. Check that the page does not scroll horizontally and the current section/action remains visible. +2. Toggle dark theme. Recheck status text, borders, focus rings, and contrast. Keyboard Tab through navigation, theme control, and demo actions; use Enter/Space where appropriate. +3. Zoom to 200%, inspect long text, and enable reduced motion/forced colors if available. These are exploratory checks, not a WCAG certification. +4. Compare light-mode density, card geometry, navigation indicator, controls, and status treatment with the pinned official-demo captures. Record any future divergence before treating this spike as an implementation input. + +## Remaining gates + +`RG-006` still needs a real supported Home Assistant/browser matrix, verified public host-context/Ingress contract, frontend/build selection, and approved capture/update/reviewer procedure. This spike does not satisfy the SDD's 84-case production test budget or authorize production UI implementation. diff --git a/docs/development/ui-spike/demo.js b/docs/development/ui-spike/demo.js new file mode 100644 index 0000000..874c2a8 --- /dev/null +++ b/docs/development/ui-spike/demo.js @@ -0,0 +1,25 @@ +// Isolated visual fixture. No network, storage, or application-side effects. +const views = [...document.querySelectorAll('.view')]; +const links = [...document.querySelectorAll('[data-section]')]; +const validSections = new Set(views.map((view) => view.id)); + +function showSection() { + const requested = window.location.hash.slice(1); + const section = validSections.has(requested) ? requested : 'connections'; + for (const view of views) view.hidden = view.id !== section; + for (const link of links) { + if (link.dataset.section === section) link.setAttribute('aria-current', 'page'); + else link.removeAttribute('aria-current'); + } + document.title = `${document.getElementById(`${section}-title`).textContent} · Symphonia UI fixture`; +} + +window.addEventListener('hashchange', showSection); +showSection(); + +const themeToggle = document.getElementById('theme-toggle'); +themeToggle.addEventListener('click', () => { + const dark = document.documentElement.classList.toggle('theme-dark'); + themeToggle.setAttribute('aria-label', `Switch to ${dark ? 'light' : 'dark'} theme`); + themeToggle.title = `Switch to ${dark ? 'light' : 'dark'} theme`; +}); diff --git a/docs/development/ui-spike/index.html b/docs/development/ui-spike/index.html new file mode 100644 index 0000000..cae2261 --- /dev/null +++ b/docs/development/ui-spike/index.html @@ -0,0 +1,65 @@ + + + + + + + Symphonia · UI review fixture + + + + + +
+
+ +
SymphoniaMusic library
+ UI review fixture + +
+
+ +
+
+

Connections

Manage music services available to Symphonia.

+

Preview only. Connecting a provider is not available in this fixture.

+
+
Source serviceLibrary imported today at 09:14 · 12 playlists
Connected
+
Target serviceReady for a reviewed, one-time copy
Connected
+
+

Before copying

Review source freshness and unresolved recordings before writing to a destination.

+
Connections do not start a copy

A copy requires a separate preview, explicit omissions, and confirmation. The original playlist remains unchanged.

+
+ + + + + + +
+
Symphonia UI review fixture · synthetic data · no provider or Home Assistant connection
+ + diff --git a/docs/development/ui-spike/styles.css b/docs/development/ui-spike/styles.css new file mode 100644 index 0000000..36428a1 --- /dev/null +++ b/docs/development/ui-spike/styles.css @@ -0,0 +1,124 @@ +:root { + color-scheme: light; + font: 14px/1.45 system-ui, -apple-system, BlinkMacSystemFont, "Roboto", sans-serif; + --canvas: #fafafa; + --surface: #fff; + --surface-muted: #f3f3f3; + --border: #e5e5e5; + --text: #1a1a1a; + --secondary: #616161; + --accent: #008db8; + --accent-text: #fff; + --success: #006d35; + --warning: #a64500; + --danger: #b42336; + --focus: #007eac; + --card-radius: 12px; + --button-radius: 999px; +} + +:root.theme-dark { + color-scheme: dark; + --canvas: #141414; + --surface: #242424; + --surface-muted: #343434; + --border: #555; + --text: #f7f7f7; + --secondary: #d0d0d0; + --accent: #49c7ef; + --accent-text: #101820; + --success: #72dc92; + --warning: #ffab65; + --danger: #ff8998; + --focus: #89ddff; +} + +*, *::before, *::after { box-sizing: border-box; } +html { background: var(--canvas); } +body { min-width: 0; min-height: 100vh; margin: 0; color: var(--text); background: var(--canvas); } +button, a { font: inherit; } +button:focus-visible, a:focus-visible { outline: 3px solid var(--focus); outline-offset: 3px; } +button:disabled { opacity: .6; cursor: not-allowed; } +.skip-link { position: absolute; z-index: 10; top: -60px; left: 8px; padding: 8px 12px; color: var(--text); background: var(--surface); } +.skip-link:focus { top: 8px; } +.app-bar { border-bottom: 1px solid var(--border); background: var(--surface); } +.app-bar-inner, .section-nav-inner { display: flex; align-items: center; width: min(1080px, 100%); margin: auto; } +.app-bar-inner { min-height: 60px; gap: 12px; padding: 0 20px; } +.app-mark { display: grid; place-items: center; width: 34px; height: 34px; border-radius: 50%; color: var(--accent-text); background: var(--accent); font-size: 22px; line-height: 1; } +.app-identity { display: grid; line-height: 1.2; } +.app-identity strong { font-size: 17px; font-weight: 600; } +.app-identity span { color: var(--secondary); font-size: 12px; } +.fixture-label { margin-left: auto; color: var(--secondary); font-size: 12px; } +.icon-button { display: grid; place-items: center; width: 42px; height: 42px; border: 0; border-radius: 50%; color: var(--text); background: transparent; cursor: pointer; font-size: 23px; } +.icon-button:hover { background: var(--surface-muted); } +.section-nav { border-bottom: 1px solid var(--border); background: var(--surface); } +.section-nav-inner { gap: 0; overflow-x: auto; scrollbar-width: thin; padding: 0 12px; } +.section-nav a { display: inline-flex; align-items: center; flex: none; min-height: 52px; padding: 0 18px; border-bottom: 2px solid transparent; color: var(--secondary); text-decoration: none; white-space: nowrap; } +.section-nav a:hover { color: var(--text); background: var(--surface-muted); } +.section-nav a[aria-current="page"] { border-bottom-color: var(--accent); color: var(--text); font-weight: 600; } +main { width: min(1080px, 100%); margin: 0 auto; padding: 26px 20px 72px; } +.view[hidden] { display: none; } +.page-heading { display: flex; align-items: flex-start; justify-content: space-between; flex-wrap: wrap; gap: 16px; margin-bottom: 22px; } +h1, h2, p { margin: 0; } +h1 { font-size: 28px; font-weight: 500; line-height: 1.2; } +h2 { font-size: 18px; font-weight: 500; line-height: 1.3; } +.page-heading p, .section-heading p, .copy-summary p { margin-top: 5px; color: var(--secondary); } +.button { min-height: 40px; padding: 8px 17px; border: 1px solid var(--border); border-radius: var(--button-radius); background: var(--surface); color: var(--text); white-space: nowrap; } +.button.primary { border-color: var(--accent); color: var(--accent-text); background: var(--accent); } +.fixture-note, .disabled-explanation { margin: -4px 0 17px; color: var(--secondary); font-size: 12px; } +.card { min-width: 0; border: 1px solid var(--border); border-radius: var(--card-radius); background: var(--surface); } +.settings-card { max-width: 760px; padding: 0 16px; } +.settings-row { display: flex; align-items: center; min-width: 0; gap: 15px; min-height: 76px; padding: 12px 2px; } +.settings-row + .settings-row { border-top: 1px solid var(--border); } +.provider-icon, .round-icon { display: grid; place-items: center; flex: none; width: 39px; height: 39px; border-radius: 50%; color: #fff; background: #1577b6; font-weight: 700; } +.secondary-icon { background: #6166b5; } +.round-icon { width: 28px; height: 28px; background: var(--accent); color: var(--accent-text); font-family: Georgia, serif; } +.row-copy { display: grid; min-width: 0; gap: 2px; flex: 1; } +.row-copy strong, .result-row strong { font-weight: 550; overflow-wrap: anywhere; } +.row-copy span, .result-row div span { color: var(--secondary); font-size: 13px; overflow-wrap: anywhere; } +.status { display: inline-flex; align-items: center; align-self: center; flex: none; max-width: 100%; min-height: 26px; padding: 3px 10px; border: 1px solid var(--border); border-radius: 999px; background: var(--surface-muted); color: var(--secondary); font-size: 12px; line-height: 1.2; text-align: center; } +.status.success { color: var(--success); border-color: var(--success); background: var(--surface); } +.status.warning { color: var(--warning); border-color: var(--warning); background: var(--surface); } +.status.danger { color: var(--danger); border-color: var(--danger); background: var(--surface); } +.section-heading { margin: 28px 0 12px; } +.information-card { display: flex; align-items: flex-start; gap: 12px; max-width: 760px; padding: 16px; } +.information-card p { margin-top: 4px; color: var(--secondary); } +.alert { margin-bottom: 18px; padding: 14px 16px; border: 1px solid var(--warning); border-radius: var(--card-radius); background: var(--surface); } +.alert strong { color: var(--warning); } +.alert p { margin-top: 5px; color: var(--text); } +.metric-grid { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 12px; margin: 0 0 20px; } +.metric { display: grid; align-content: start; gap: 2px; min-height: 125px; padding: 16px; } +.metric span, .metric small { color: var(--secondary); } +.metric strong { font-size: 27px; line-height: 1.2; } +.list-card { padding: 0 16px; } +.card-heading, .result-row { display: flex; align-items: center; justify-content: space-between; gap: 16px; padding: 14px 0; } +.card-heading { border-bottom: 1px solid var(--border); } +.result-row + .result-row { border-top: 1px solid var(--border); } +.result-row div { display: grid; min-width: 0; gap: 3px; } +.step-list { display: flex; flex-wrap: wrap; gap: 8px; margin-bottom: 20px; } +.step { padding: 6px 10px; border-bottom: 2px solid var(--border); color: var(--secondary); } +.step.done { color: var(--success); } +.step.active { border-bottom-color: var(--accent); color: var(--text); font-weight: 600; } +.copy-summary { display: flex; justify-content: space-between; align-items: center; flex-wrap: wrap; gap: 12px; padding: 16px; margin-bottom: 16px; } +.form-actions { display: flex; justify-content: flex-end; flex-wrap: wrap; gap: 8px; margin-top: 20px; } +.disabled-explanation { margin: 8px 0 0; text-align: right; } +footer { padding: 18px 20px max(18px, env(safe-area-inset-bottom)); border-top: 1px solid var(--border); color: var(--secondary); font-size: 12px; text-align: center; } +@media (max-width: 620px) { + .fixture-label { display: none; } + .icon-button { margin-left: auto; } + .app-bar-inner { padding-inline: 14px; } + .section-nav-inner { padding-inline: 4px; } + .section-nav a { padding-inline: 13px; } + main { padding: 20px 14px 56px; } + h1 { font-size: 25px; } + .metric-grid { grid-template-columns: 1fr; gap: 8px; } + .metric { min-height: 0; grid-template-columns: 1fr auto; align-items: baseline; } + .metric small { grid-column: 1 / -1; } + .settings-row, .result-row { align-items: flex-start; flex-wrap: wrap; } + .settings-row .status, .result-row .status { margin-left: 54px; } + .result-row .status { margin-left: 0; } + .card-heading { align-items: flex-start; flex-wrap: wrap; } + .form-actions .button { flex: 1 1 auto; } +} +@media (prefers-reduced-motion: reduce) { *, *::before, *::after { scroll-behavior: auto !important; animation-duration: 0s !important; transition-duration: 0s !important; } } +@media (forced-colors: active) { .card, .alert, .status, .section-nav a[aria-current="page"] { border: 1px solid CanvasText; } .button:focus-visible, a:focus-visible { outline-color: Highlight; } } diff --git a/specs/catalog.json b/specs/catalog.json index e9a0260..a75824b 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -122,7 +122,8 @@ "docs/product/home-assistant-ui-specification.md", "docs/architecture/system-architecture.md", "docs/providers/provider-research.md", - "docs/providers/home-assistant-ecosystem-review.md" + "docs/providers/home-assistant-ecosystem-review.md", + "docs/development/ui-spike/README.md" ], "implementationEvidence": { "code": [], diff --git a/specs/home-assistant-native-ui.md b/specs/home-assistant-native-ui.md index db9cb10..a3d5771 100644 --- a/specs/home-assistant-native-ui.md +++ b/specs/home-assistant-native-ui.md @@ -1,7 +1,7 @@ # Home Assistant-native UI foundation - Status: Ready for review -- Date: 2026-09-20 +- Date: 2026-09-27 - Catalog capability ID: `home-assistant-native-ui` - Owners: Symphonia maintainers - Scope: establish the shared shell, component families, semantic tokens, host-context adaptation, accessibility, responsive behavior, component catalog, and visual compatibility evidence for every Symphonia web view. @@ -32,13 +32,14 @@ Without a shared UI contract, each feature can implement different cards, button ### 2.2 Current behavior -No complete Symphonia UI or UI package exists. The repository has an experimental Ingress metadata scaffold and prospective feature SDDs. This SDD does not select or authorize a frontend framework or production implementation. +No complete Symphonia UI or UI package exists. The repository has an experimental Ingress metadata scaffold, prospective feature SDDs, and an isolated [synthetic UI review fixture](../docs/development/ui-spike/README.md). The fixture is not served by the App, does not connect to application state, and has not passed browser visual/accessibility review. This SDD does not select or authorize a frontend framework or production implementation. ### 2.3 Evidence and unknowns - Accepted repository direction: [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md) and the [UI specification](../docs/product/home-assistant-ui-specification.md). - Official evidence: Home Assistant design portal, frontend architecture/source, independent-component warning, current App/Ingress and safe-area contracts linked from the UI specification and platform research. - Community evidence: `vypdev/homeassistant-gateway` commit `1ed75be` demonstrates a presentation-only compatibility layer, HA-like component families, component catalog, official-demo reference captures, accessibility/responsive tests, and visual baselines. +- Symphonia review evidence: the [isolated fixture](../docs/development/ui-spike/README.md) exercises a proposed App-interior shell and synthetic connection, library, copy, and operation states. Its structural tests are not production component, Ingress, visual, or WCAG evidence. - Unknowns: exact supported Home Assistant/browser versions; selected frontend/build tools; the public context actually available to an Ingress App at each supported version; long-term token mapping; reference capture automation and review ownership. ## 3. Actors, surfaces, and terminology diff --git a/tests/test_ui_spike.py b/tests/test_ui_spike.py new file mode 100644 index 0000000..4a9afe8 --- /dev/null +++ b/tests/test_ui_spike.py @@ -0,0 +1,78 @@ +"""Offline review-fixture guardrails; not production UI acceptance tests.""" + +from html.parser import HTMLParser +from pathlib import Path +import re +import unittest + + +FIXTURE = Path(__file__).resolve().parents[1] / "docs/development/ui-spike" + + +class FixtureParser(HTMLParser): + def __init__(self) -> None: + super().__init__() + self.elements: list[tuple[str, dict[str, str | None]]] = [] + + def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None: + self.elements.append((tag, dict(attrs))) + + +class UiSpikeContractTests(unittest.TestCase): + @classmethod + def setUpClass(cls) -> None: + cls.html = (FIXTURE / "index.html").read_text(encoding="utf-8") + cls.css = (FIXTURE / "styles.css").read_text(encoding="utf-8") + cls.script = (FIXTURE / "demo.js").read_text(encoding="utf-8") + cls.parser = FixtureParser() + cls.parser.feed(cls.html) + + def test_fixture_sections_have_unique_ids_and_navigation_targets(self) -> None: + elements = self.parser.elements + ids = [attrs["id"] for _, attrs in elements if attrs.get("id")] + self.assertEqual(len(ids), len(set(ids))) + views = {attrs["id"] for _, attrs in elements if "view" in (attrs.get("class") or "").split()} + links = {attrs["data-section"] for tag, attrs in elements if tag == "a" and attrs.get("data-section")} + self.assertEqual(views, links) + self.assertEqual(views, {"connections", "library", "copy", "activity"}) + for _, attrs in elements: + if attrs.get("aria-labelledby"): + self.assertIn(attrs["aria-labelledby"], ids) + self.assertIn(('main', {'id': 'main', 'tabindex': '-1'}), elements) + + def test_fixture_cannot_issue_actions_or_load_remote_assets(self) -> None: + for tag, attrs in self.parser.elements: + if tag == "button" and attrs.get("id") != "theme-toggle": + self.assertIn("disabled", attrs) + for name in ("href", "src", "action"): + value = attrs.get(name) + if value is None: + continue + self.assertFalse(value.startswith(("http:", "https:", "//", "/", "javascript:")), (tag, name, value)) + self.assertFalse(any(name.lower().startswith("on") for name in attrs), (tag, attrs)) + self.assertNotIn("fetch(", self.script) + self.assertNotIn("XMLHttpRequest", self.script) + self.assertNotIn("localStorage", self.script) + self.assertNotIn("sessionStorage", self.script) + self.assertNotIn("innerHTML", self.script) + + def test_theme_and_responsive_fallbacks_are_explicit(self) -> None: + self.assertIn(":root.theme-dark", self.css) + self.assertIn("@media (max-width: 620px)", self.css) + self.assertIn("prefers-reduced-motion: reduce", self.css) + self.assertIn("forced-colors: active", self.css) + self.assertIn("safe-area-inset-bottom", self.css) + self.assertIn(":focus-visible", self.css) + self.assertIn("aria-current", self.script) + self.assertNotRegex(self.css, r"url\(\s*['\"]?https?://") + + def test_visible_statuses_are_words_not_color_only(self) -> None: + statuses = re.findall(r'([^<]+)', self.html) + self.assertGreaterEqual(len(statuses), 10) + self.assertTrue(all(status.strip() for status in statuses)) + self.assertIn("No destination changes", self.html) + self.assertIn("no success is inferred", self.html) + + +if __name__ == "__main__": + unittest.main() From f2ff581830bc5c4bf6ecf37ff41e87d8eb9ca5b8 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 20:02:14 +0200 Subject: [PATCH 161/167] test: characterize copy recovery after lost provider responses --- docs/development/quality-audit.md | 1 + tests/test_copy_execution.py | 97 +++++++++++++++++++++++++++++++ 2 files changed, 98 insertions(+) diff --git a/docs/development/quality-audit.md b/docs/development/quality-audit.md index 0f0fa29..79ec094 100644 --- a/docs/development/quality-audit.md +++ b/docs/development/quality-audit.md @@ -22,6 +22,7 @@ Use a fresh temporary output directory for each graph extraction. `--code-only` - RepoWise 0.45.0 reported `copy_execution.py` as a maintainability hotspot (score 1.65/10, maximum cyclomatic complexity 79), followed by `sqlite_operations.py` (score 1.65/10, 1,308 nonblank lines). These are heuristic scores, not failures by themselves. In particular, the copy executor coordinates uncertain external writes and must be refactored only with restart/reconciliation characterization tests. - A bounded copy-executor change consolidated the repeated confirmation checkpoint and replaced list membership in the write loop with a synchronized set. RepoWise no longer reports its `membership_test_against_list_in_loop` marker; the maximum complexity remains 79, so the broader state-machine refactor is still open. +- Fault-injection characterization now covers process loss immediately after target creation and after an entry write, with a new worker lease; the entry case also closes and reopens file-backed SQLite stores. Both prove reconciliation precedes any repeat write. These tests protect a future state-machine extraction, but do not make the high-complexity executor itself simpler. - Its reported line and branch coverage were `null`: the current test command runs deterministic `unittest` cases but produces no coverage report. Test count must not be presented as coverage. Add measured branch coverage before using a numeric coverage gate. - After the transport extraction, Graphify's local AST graph contained 1,111 nodes and 2,872 edges. `OperationRepository` (56 edges) remained the most connected node. A change to operation persistence has a broad impact across the copy/import runners, runtime, and tests; inspect reverse dependencies before modifying its contract. - Graphify makes the existing direct application-to-SQLite imports visible. The owner-approved foundation permits the current concrete composition, but an implementation-ready architecture should define repository ports and keep concrete stores at the composition boundary. This remains design debt, not a silently accepted dependency rule. diff --git a/tests/test_copy_execution.py b/tests/test_copy_execution.py index 22aaaed..8ef692b 100644 --- a/tests/test_copy_execution.py +++ b/tests/test_copy_execution.py @@ -1,6 +1,8 @@ from __future__ import annotations from datetime import datetime, timedelta, timezone +from pathlib import Path +from tempfile import TemporaryDirectory import unittest from symphonia.application import CopyExecutionService, CopyPlanningService, CopyWorkflowService, OperationRunner @@ -235,6 +237,101 @@ def test_unknown_write_without_reconciliation_requires_user(self) -> None: self.assertNotIn("unknown_step", reconciled.checkpoint) self.assertEqual(self.writer.added, [(step_key, "target-1")]) + def test_process_loss_after_entry_write_reconciles_without_duplicate(self) -> None: + """A lost response must survive SQLite reopen and a new worker lease.""" + + class SimulatedProcessLoss(BaseException): + pass + + class CrashAfterEntry(FakeWriter): + def add_entry(self, *, target_playlist_id: str, provider_track_id: str, idempotency_key: str) -> WriteResult: + self.added.append((idempotency_key, provider_track_id)) + raise SimulatedProcessLoss() + + with TemporaryDirectory() as directory: + path = str(Path(directory) / "copy.sqlite3") + plans = CopyPlanRepository(path) + operations = OperationRepository(path) + try: + workflow = CopyWorkflowService(CopyPlanningService(), plans, operations) + source, policy = self.snapshot() + stored = workflow.create_plan( + source, + target_provider="youtube", + target_playlist_name="Rock", + target_visibility="private", + policy=policy, + now=NOW, + ) + digest = workflow.accept_plan(stored.plan.digest, now=NOW).plan.digest + queued = workflow.enqueue_accepted_plan(digest, now=NOW) + writer = CrashAfterEntry() + with self.assertRaises(SimulatedProcessLoss): + CopyExecutionService(plans, operations).execute( + digest, writer=writer, worker_id="worker-before-crash", now=NOW + ) + interrupted = operations.get(queued.operation_id) + self.assertEqual(interrupted.state, "running") + self.assertEqual(interrupted.checkpoint["unknown_step"], "occ-1") + finally: + operations.close() + plans.close() + + plans = CopyPlanRepository(path) + operations = OperationRepository(path) + try: + step_key = f"{digest}:entry:occ-1" + writer.reconcile_results[step_key] = True + resumed = CopyExecutionService(plans, operations).execute( + digest, + writer=writer, + worker_id="worker-after-restart", + now=NOW + timedelta(seconds=31), + ) + self.assertEqual(resumed.state, "succeeded") + self.assertEqual(resumed.checkpoint["confirmed_occurrences"], ["occ-1"]) + self.assertEqual(writer.reconciled, [step_key]) + self.assertEqual(writer.added, [(step_key, "target-1")]) + finally: + operations.close() + plans.close() + + def test_process_loss_after_target_creation_reconciles_before_any_entry(self) -> None: + class SimulatedProcessLoss(BaseException): + pass + + class CrashAfterTarget(FakeWriter): + def __init__(self) -> None: + super().__init__() + self.create_attempts = 0 + self.reconcile_attempts = 0 + + def ensure_target_playlist( + self, *, provider: str, name: str, visibility: str, idempotency_key: str + ) -> TargetPlaylist: + self.create_attempts += 1 + raise SimulatedProcessLoss() + + def reconcile_target_playlist(self, *, idempotency_key: str) -> TargetPlaylist | None: + self.reconcile_attempts += 1 + return self.target + + digest = self.accepted_digest() + writer = CrashAfterTarget() + with self.assertRaises(SimulatedProcessLoss): + self.executor.execute(digest, writer=writer, worker_id="worker-before-crash", now=NOW) + + recovered = CopyExecutionService(self.plans, self.operations).execute( + digest, + writer=writer, + worker_id="worker-after-restart", + now=NOW + timedelta(seconds=31), + ) + self.assertEqual(recovered.state, "succeeded") + self.assertEqual(writer.create_attempts, 1) + self.assertEqual(writer.reconcile_attempts, 1) + self.assertEqual(writer.added, [(f"{digest}:entry:occ-1", "target-1")]) + def test_resume_skips_permanent_failure_before_reconciling_later_unknown_write(self) -> None: digest = self.accepted_digest(include_second_ready=True) first_key = f"{digest}:entry:occ-1" From a2871fd2e28b89c979ac23389a7737ad8c9cc202 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Sun, 27 Sep 2026 21:22:52 +0200 Subject: [PATCH 162/167] docs: record Lit UI decision and verify isolated RG-006 spike --- .github/workflows/verify.yml | 27 + .gitignore | 1 + README.md | 1 + docs/architecture/system-architecture.md | 9 +- docs/decisions/0005-lit-typescript-vite-ui.md | 33 + docs/decisions/README.md | 1 + docs/development/ui-lit-spike/README.md | 9 + docs/development/ui-lit-spike/index.html | 12 + .../ui-lit-spike/package-lock.json | 1277 +++++++++++++++++ docs/development/ui-lit-spike/package.json | 21 + .../ui-lit-spike/scripts/check-boundary.mjs | 13 + .../ui-lit-spike/scripts/verify-output.mjs | 14 + .../ui-lit-spike/src/appearance.test.ts | 16 + .../ui-lit-spike/src/appearance.ts | 8 + docs/development/ui-lit-spike/src/main.ts | 82 ++ docs/development/ui-lit-spike/tsconfig.json | 14 + docs/development/ui-lit-spike/vite.config.ts | 6 + docs/development/ui-spike/README.md | 6 +- .../ui-spike/host-context-evidence.md | 40 + .../ui-spike/visual-reference-plan.md | 32 + docs/open-questions.md | 10 +- .../home-assistant-ui-specification.md | 4 +- .../home-assistant-ecosystem-review.md | 6 +- specs/CATALOG.md | 2 +- specs/README.md | 5 +- specs/catalog.json | 12 +- specs/home-assistant-native-ui.md | 27 +- tests/test_spec_validator.py | 21 +- tools/validate_specs.py | 29 +- 29 files changed, 1697 insertions(+), 41 deletions(-) create mode 100644 docs/decisions/0005-lit-typescript-vite-ui.md create mode 100644 docs/development/ui-lit-spike/README.md create mode 100644 docs/development/ui-lit-spike/index.html create mode 100644 docs/development/ui-lit-spike/package-lock.json create mode 100644 docs/development/ui-lit-spike/package.json create mode 100644 docs/development/ui-lit-spike/scripts/check-boundary.mjs create mode 100644 docs/development/ui-lit-spike/scripts/verify-output.mjs create mode 100644 docs/development/ui-lit-spike/src/appearance.test.ts create mode 100644 docs/development/ui-lit-spike/src/appearance.ts create mode 100644 docs/development/ui-lit-spike/src/main.ts create mode 100644 docs/development/ui-lit-spike/tsconfig.json create mode 100644 docs/development/ui-lit-spike/vite.config.ts create mode 100644 docs/development/ui-spike/host-context-evidence.md create mode 100644 docs/development/ui-spike/visual-reference-plan.md diff --git a/.github/workflows/verify.yml b/.github/workflows/verify.yml index a4941fe..8dc6a16 100644 --- a/.github/workflows/verify.yml +++ b/.github/workflows/verify.yml @@ -44,3 +44,30 @@ jobs: fi - name: Run offline test suite run: PYTHONPATH=src:. python -m unittest discover -s tests -v + + ui-lit-spike: + runs-on: ubuntu-latest + defaults: + run: + working-directory: docs/development/ui-lit-spike + steps: + - name: Check out repository + uses: actions/checkout@v4 + - name: Set up Node.js + uses: actions/setup-node@v4 + with: + node-version: "24" + cache: npm + cache-dependency-path: docs/development/ui-lit-spike/package-lock.json + - name: Install pinned spike dependencies + run: npm ci --ignore-scripts + - name: Check types + run: npm run check + - name: Check presentation boundary + run: npm run check-boundary + - name: Run pure fallback tests + run: npm test + - name: Build isolated UI spike + run: npm run build + - name: Inspect relative assets + run: npm run verify-output diff --git a/.gitignore b/.gitignore index 321f050..cf09668 100644 --- a/.gitignore +++ b/.gitignore @@ -6,6 +6,7 @@ __pycache__/ coverage.xml dist/ build/ +node_modules/ .venv/ *.sqlite3 *.db diff --git a/README.md b/README.md index 83d9b48..198f204 100644 --- a/README.md +++ b/README.md @@ -31,6 +31,7 @@ Start with the [documentation map](docs/README.md). The horizontal specification | Vision, scope, journeys, requirements | [Product specification](docs/product/product-specification.md) | | Home Assistant-native UI and component contract | [UI specification](docs/product/home-assistant-ui-specification.md) and [UI foundation SDD](specs/home-assistant-native-ui.md) | | Non-production Home Assistant-adjacent UI review | [Synthetic UI fixture](docs/development/ui-spike/README.md) | +| Non-production Lit/TypeScript/Vite feasibility | [Isolated build spike](docs/development/ui-lit-spike/README.md) | | Vocabulary, entities, identity, playlists | [Domain model](docs/domain/domain-model.md) | | System boundaries and operational qualities | [Architecture](docs/architecture/system-architecture.md) | | Provider contract and capability semantics | [Provider specification](docs/providers/provider-specification.md) | diff --git a/docs/architecture/system-architecture.md b/docs/architecture/system-architecture.md index 87ee230..b547174 100644 --- a/docs/architecture/system-architecture.md +++ b/docs/architecture/system-architecture.md @@ -1,7 +1,7 @@ # System architecture -**Status:** Home Assistant-first deployment accepted; logical boundaries proposed; technology stack open -**Last reviewed:** 2026-09-20 +**Status:** Home Assistant-first deployment and UI stack direction accepted; logical boundaries proposed; other technology choices open +**Last reviewed:** 2026-09-27 ## Architectural drivers @@ -19,7 +19,7 @@ The architecture is derived from these needs: - a future native Home Assistant surface without duplicating domain policy; and - simple backup, restore, upgrade, and diagnostics. -These drivers do not yet justify a programming language, web framework, frontend framework, or database product. The observable UI direction and compatibility-layer boundary are accepted separately in [ADR 0004](../decisions/0004-home-assistant-native-ui.md); that decision does not select a framework. +These drivers do not yet justify a backend framework or database product. The UI direction and compatibility-layer boundary are accepted in [ADR 0004](../decisions/0004-home-assistant-native-ui.md); the owner subsequently selected Lit, TypeScript, and Vite in [ADR 0005](../decisions/0005-lit-typescript-vite-ui.md). That selection does not close the UI compatibility/release evidence gate. ## Context and trust boundaries @@ -305,11 +305,10 @@ The MVP may render metrics in its UI and logs; choosing Prometheus/OpenTelemetry | Concern | Reason not yet chosen | Evidence needed | | --- | --- | --- | | Backend language/framework | Provider SDK maturity, job ergonomics, footprint, HA App maintainability | Thin vertical spike and maintainer preference | -| UI framework/build tooling | The Home Assistant-native component, accessibility, catalog, Ingress, and standalone contracts are accepted, but the implementation technology remains reversible | `RG-006` prototype proving public context, package boundaries, bundle/compatibility cost, catalog and browser evidence | | SQLite versus PostgreSQL | Concurrency, backup, migration, and library scale unmeasured | Storage/job lease spike and target sizes | | Job library versus internal durable runner | Retry/idempotency needs are specific; external brokers add operations | Failure/restart spike | | Secret encryption/key source | HA App secret facilities and portable standalone behavior differ | Threat model and backup/restore test | | Companion integration transport | Need push, authentication, discovery, and version compatibility | Home Assistant integration RFC | | Public API/event protocol | Only internal UI needs are currently concrete | UI and companion-integration contract design | -No implementation agent should infer these technology choices from examples in `homeassistant-gateway`. The Gateway informs the accepted presentation boundary and verification approach, not an automatic dependency or stack selection. +The remaining choices must not be inferred from `homeassistant-gateway`. The UI stack is a separate, explicit owner decision in ADR 0005, not an automatic dependency selection. `RG-006` still requires public-context, package-boundary, bundle, Ingress, catalog, browser, and visual evidence before production UI work. diff --git a/docs/decisions/0005-lit-typescript-vite-ui.md b/docs/decisions/0005-lit-typescript-vite-ui.md new file mode 100644 index 0000000..d08fbeb --- /dev/null +++ b/docs/decisions/0005-lit-typescript-vite-ui.md @@ -0,0 +1,33 @@ +# ADR 0005: Use Lit, TypeScript, and Vite for the App presentation layer + +- **Status:** accepted +- **Date:** 2026-09-27 +- **Scope:** future Symphonia web UI and component catalog; not the Python/domain core + +## Context + +[ADR 0004](0004-home-assistant-native-ui.md) requires an independently served, Home Assistant-native-adjacent App UI with a Symphonia-owned presentation-only compatibility layer. The owner explicitly approved Lit + TypeScript + Vite on 2026-09-27 after reviewing the isolated UI fixture and the approach of `vypdev/homeassistant-gateway`. + +Lit provides standards-based custom elements and reactive templates. TypeScript can type component inputs and host-context boundaries. Vite can build browser-resolvable static assets; it does **not** by itself solve arbitrary Ingress base paths, session/authentication, localization, safe areas, or visual parity. The [Gateway's pinned frontend](https://github.com/vypdev/homeassistant-gateway/tree/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/frontend) is implementation evidence, not a library to copy or an authority for Symphonia's behavior. [Lit documentation](https://lit.dev/docs/components/overview/) and [Vite production build documentation](https://vite.dev/guide/build.html) are the primary tooling references. + +## Decision + +When the [UI foundation SDD](../../specs/home-assistant-native-ui.md) becomes `Ready for implementation`, the production presentation package will use **Lit 3, TypeScript, and Vite**, subject to a reproducible lockfile and the then-supported versions. Lit component primitives will be presentation-only; feature controllers and API clients remain outside the public component package. The built UI will be independently served inside Home Assistant Ingress and use the same feature/component semantics in a standalone profile. + +The implementation will not import private Home Assistant frontend modules, traverse the parent DOM, or assume Home Assistant theme/locale data is forwarded to App iframes. A validated public host-context adapter and deterministic browser/standalone fallbacks remain separate contracts. The App's parent owns the Home Assistant shell; Symphonia owns only its interior navigation and content. + +This decision selects a framework/build direction, **not** a package manager, visual baseline, supported HA/browser matrix, validated host-context protocol, or permission to begin production UI. `RG-006` and the SDD readiness/owner-approval gates still apply. If bundle cost, browser support, accessibility, or Ingress compatibility cannot be demonstrated, a new ADR must supersede this one. + +## Consequences and verification + +- Favor native HTML semantics within Lit templates; component encapsulation must not hide labels, focus, or announcement behavior. +- A single public UI entry point, dependency-direction checks, deterministic catalog fixtures, type checking, browser/a11y tests, and reviewed visual references become release evidence. +- Vite output must work below arbitrary non-root Ingress prefixes for assets, navigation, refresh, API calls, and any push transport. Hash routes in the current static fixture are not proof of this. +- Build size and first meaningful render on a modest Home Assistant device must be measured before an implementation-ready matrix is accepted; no performance claim is made by this ADR. +- Dependency versions and license/security review are performed when the package is created; Gateway's versions are not inherited silently. + +## Alternatives considered + +- **Continue with plain HTML/CSS/JS:** suitable for the isolated review fixture, but does not supply the agreed typed reusable component package and catalog architecture for the full UI. +- **Use Home Assistant's private built-in components:** rejected by ADR 0004 because their external API and iframe availability are not a stable App contract. +- **Use another SPA framework:** not ruled out technically; the owner chose Lit's web-component approach for alignment with the Gateway pattern. A future evidence-based reversal would require a superseding ADR. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 6dd6aa0..ee6591c 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -8,5 +8,6 @@ ADRs contain only decisions already justified and accepted by the product brief | [0002](0002-copy-and-sync-are-distinct.md) | Accepted | One-time copy and persistent sync are distinct concepts | | [0003](0003-home-assistant-app-primary.md) | Accepted | Home Assistant App is the primary deployment boundary | | [0004](0004-home-assistant-native-ui.md) | Accepted | App UI follows Home Assistant-native interaction and visual patterns through an owned compatibility layer | +| [0005](0005-lit-typescript-vite-ui.md) | Accepted | Future App presentation layer uses Lit, TypeScript, and Vite, subject to UI readiness gates | An ADR is immutable after acceptance except for typo/link corrections. A changed decision gets a new ADR that supersedes the old one. diff --git a/docs/development/ui-lit-spike/README.md b/docs/development/ui-lit-spike/README.md new file mode 100644 index 0000000..98f3cb8 --- /dev/null +++ b/docs/development/ui-lit-spike/README.md @@ -0,0 +1,9 @@ +# Isolated Lit/TypeScript/Vite build spike + +This is **non-production RG-006 evidence**, not an App route, production component package, or SDD completion claim. It tests the owner-approved [ADR 0005](../../decisions/0005-lit-typescript-vite-ui.md) stack with synthetic UI data and no provider, Home Assistant, or Symphonia API access. The earlier [HTML/CSS review fixture](../ui-spike/README.md) remains the broader visual concept; this smaller spike verifies tooling and a presentation-only component boundary. + +From this directory, run `npm ci --ignore-scripts`, `npm run check`, `npm test`, `npm run check-boundary`, `npm run build`, and `npm run verify-output`. CI repeats these checks. The lockfile pins the evaluated dependencies; npm is used **for this spike**, not chosen as the final package-manager policy. `vite.config.ts` emits relative asset URLs (`base: './'`) and `verify-output` checks them. This does not prove server routing, Ingress authentication, browser behavior, or Home Assistant visual parity. + +The UI has one independently owned custom element, semantic tokens, native buttons, and explicit light/dark fixture control. Its imported `appearance.ts` is a pure fallback decision with deterministic tests. There are no API clients or host-private imports. Component-catalog breadth, WCAG/browser results, real base-path tests, HA context validation, and the 84-case SDD budget remain open. + +Build size and dependency versions are recorded in [RG-006 host-context evidence](../ui-spike/host-context-evidence.md) after running the pinned build. Do not copy the package into the App before the UI SDD is `Ready for implementation` and the owner approves that stage. diff --git a/docs/development/ui-lit-spike/index.html b/docs/development/ui-lit-spike/index.html new file mode 100644 index 0000000..3f0f87e --- /dev/null +++ b/docs/development/ui-lit-spike/index.html @@ -0,0 +1,12 @@ + + + + + + Symphonia · Lit UI build spike + + + + + + diff --git a/docs/development/ui-lit-spike/package-lock.json b/docs/development/ui-lit-spike/package-lock.json new file mode 100644 index 0000000..3430205 --- /dev/null +++ b/docs/development/ui-lit-spike/package-lock.json @@ -0,0 +1,1277 @@ +{ + "name": "symphonia-ui-lit-spike", + "version": "0.0.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "symphonia-ui-lit-spike", + "version": "0.0.0", + "dependencies": { + "lit": "3.3.3" + }, + "devDependencies": { + "@types/node": "24.19.0", + "typescript": "7.0.2", + "vite": "8.3.1" + } + }, + "node_modules/@lit-labs/ssr-dom-shim": { + "version": "1.6.0", + "resolved": "https://registry.npmjs.org/@lit-labs/ssr-dom-shim/-/ssr-dom-shim-1.6.0.tgz", + "integrity": "sha512-VHb0ALPMTlgKjM6yIxxoQNnpKyUKLD04VzeQdsiXkMqkvYlAHxq9glGLmgbb889/1GsohSOAjvQYoiBppXFqrQ==", + "license": "BSD-3-Clause" + }, + "node_modules/@lit/reactive-element": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/@lit/reactive-element/-/reactive-element-2.1.2.tgz", + "integrity": "sha512-pbCDiVMnne1lYUIaYNN5wrwQXDtHaYtg7YEFPeW+hws6U47WeFvISGUWekPGKWOP1ygrs0ef0o1VJMk1exos5A==", + "license": "BSD-3-Clause", + "dependencies": { + "@lit-labs/ssr-dom-shim": "^1.5.0" + } + }, + "node_modules/@oxc-project/types": { + "version": "0.151.0", + "resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.151.0.tgz", + "integrity": "sha512-J1yXrIlNDZVzE3ada310xeAw7nH8yCAyLPuUIsjKatFPmfn5bS1oW+cM+QsGOtVWd5nhSpbwZWx/rue+r5Z+PA==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/oxc-project" + } + }, + "node_modules/@rolldown/binding-android-arm-eabi": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm-eabi/-/binding-android-arm-eabi-1.2.11.tgz", + "integrity": "sha512-A5kXfGKvKWWZE0TtPrfsvT+q4Y5d1QG8gGUzpYjGydM+fARM9MuX90PrXYXe0XbsDVgyxxNzHo6giCj90bsFNw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-android-arm64": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.2.11.tgz", + "integrity": "sha512-z6cTycz+iJ4PVkuL4HHW4DfTfoeU/2nqYYuSOrTmH7yHK5Y0LCOnA03V4ZNxavyVaU1oOqUgIg2klN/s+USGOA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-darwin-arm64": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.2.11.tgz", + "integrity": "sha512-jShvqNtP6vDC6/A5JOAzbVV+DkgHqhl/ScVCJEbt+TUY6QYz7YnXcrg3sLtFBniro0f/Ld50ZwCWA6f7KYD1nQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-darwin-x64": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.2.11.tgz", + "integrity": "sha512-f2i2xiNWq1Z1l2++q2fuhZRdLAT3aqxD6vRNm1RAxpUoBcdqNB3C0s1Bt+K+PbEx2F5F4gQp6hqKkphCY/xF9w==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-freebsd-x64": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.2.11.tgz", + "integrity": "sha512-4Ir5FSOKIAMr4r0kExpt1s3bMgzJU3rA45AYOHtQpls0oNeqcYBKrWMlckrYH4KCfGLfkfn1tN1dmZPMVsdXow==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-arm-gnueabihf": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.2.11.tgz", + "integrity": "sha512-/gnRDM+39BROzAN/k1OZjDPnDMcZxB/0EUxKjONO5yVkNEvlsoMDrxGNKgZi/ttFriS2gwlDNzB65pvNbFOXIQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-arm64-gnu": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.2.11.tgz", + "integrity": "sha512-PFaK8HwvAHbaKbBcDNQihjMKYvFnA5hiENx/l5tphTDz1E0WFp32l0A7aq7lyUwGsRw/xSrNIy/gIK4thrSCrw==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-arm64-musl": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.2.11.tgz", + "integrity": "sha512-AskzJUIKRLPxkruR1wLKewGbOw+EYfU/9lOrBFj4AFrEA8hPpKFnODWNu2WLaNs0QNkEb9QIJufmVZZIL/bJlg==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-ppc64-gnu": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.2.11.tgz", + "integrity": "sha512-qlUGAheh2yh8afH7QBgx0PrRHN85hKnNd78x8MeMhXivuevgd8vgf6/CstOzmNKY/lLTHvNTrPy98cLnAugzJw==", + "cpu": [ + "ppc64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-s390x-gnu": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.2.11.tgz", + "integrity": "sha512-secpEad+0vCbSfn8upFySkDskv+bGPk3THSDS9Y89yc4rb4kzqHp8Dmyd9BkQW4SnhNXBZCl/6CrO//hZahNJQ==", + "cpu": [ + "s390x" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-x64-gnu": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.2.11.tgz", + "integrity": "sha512-mOVBT3dPpkWm8XBWPmU4bf+U6dYDLeMo/9ojUmis4N0L5uu10qra5vOyngZ7/PSdoE4G9KvRt4bloRxNjLas7A==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-linux-x64-musl": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.2.11.tgz", + "integrity": "sha512-Is78i9A8Ui4SqcxUwFJ9uMmjDn58IbVTjFWYdQestFEgeuEmHMLGNriXnVJKkwG2YiZjw8cP0zCTyDMdDGtOOg==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-openharmony-arm64": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.2.11.tgz", + "integrity": "sha512-dUCXneZ87INUMyQ0D+C0HrEBNUPNXHaPmU5GTjyKTJEiussw9Kaj5Ln8UztPe4epV/ffvgNBEadksdYhmW6xJA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-win32-arm64-msvc": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.2.11.tgz", + "integrity": "sha512-jByxb6qfd+bH1xUd0qnfFnb17i9sWBPY2tOavJ0l3tdr3OTu+Kvtm8cd/JV5nFt657b1VqGltxg9olOEfofXWw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/binding-win32-x64-msvc": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.2.11.tgz", + "integrity": "sha512-/PzKqzAJ03i19oy2ItPvyvaVjOjBCNnfaJs8yvUdGBKmiESgnrJSQ2awd81QzFbbnAmu7YO9ZnJrDCb9VSJPRA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/pluginutils": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.1.tgz", + "integrity": "sha512-2j9bGt5Jh8hj+vPtgzPtl72j0yRxHAyumoo6TNfAjsLB04UtpSvPbPcDcBMxz7n+9CYB0c1GxQFxYRg2jimqGw==", + "dev": true, + "license": "MIT" + }, + "node_modules/@types/node": { + "version": "24.19.0", + "resolved": "https://registry.npmjs.org/@types/node/-/node-24.19.0.tgz", + "integrity": "sha512-zY+5tKxXdhGh1PYI0ac+7juvEu4OI6vWtVVoj5i2m42jxAY1U+zHGt6QCyOFwykdP62sM3MJ9stoYYUw5aCWew==", + "dev": true, + "license": "MIT", + "dependencies": { + "undici-types": ">=7.24.0 <7.24.7" + } + }, + "node_modules/@types/trusted-types": { + "version": "2.0.7", + "resolved": "https://registry.npmjs.org/@types/trusted-types/-/trusted-types-2.0.7.tgz", + "integrity": "sha512-ScaPdn1dQczgbl0QFTeTOmVHFULt394XJgOQNoyVhZ6r2vLnMLJfBPd53SB52T/3G36VI1/g2MZaX0cwDuXsfw==", + "license": "MIT" + }, + "node_modules/@typescript/typescript-aix-ppc64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-aix-ppc64/-/typescript-aix-ppc64-7.0.2.tgz", + "integrity": "sha512-MTKKkWB7p/0E9xi1d1tHtZ5PiLkGEMIq88pK2CubZjOsLtYTLqhgIgi6zepFa+9GHZ6h05NMCkQxGKiPXMxXtQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-darwin-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-darwin-arm64/-/typescript-darwin-arm64-7.0.2.tgz", + "integrity": "sha512-gowzar9MwS/aRWp6f3a4KUqzRjAZjOsmGNCM6LcTgXum+dBfgsBVMN+AgvOCCbguXyick6LJhpBszxMebJ8syA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-darwin-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-darwin-x64/-/typescript-darwin-x64-7.0.2.tgz", + "integrity": "sha512-SZ9xZInqApNlNGc9s0W1VSsktYSOe9cFqNOIqmN1Gs8SmkjKZYFt017G4VwPxASInODuAdbTW7sXiFUf893RgA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-freebsd-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-freebsd-arm64/-/typescript-freebsd-arm64-7.0.2.tgz", + "integrity": "sha512-W5NH4y/J0plIIS5b2xvTEkU7JFxyqdMAOgf+Ilhl0vHQXKO5dZoxd+C/jEtq56c4F3wk71RB4BMRQ2XdI+bwYQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-freebsd-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-freebsd-x64/-/typescript-freebsd-x64-7.0.2.tgz", + "integrity": "sha512-UMGDx5sTpzNw3WiPebH7l90IWfJggEd+egHt/q6p7/Cm3zqoV7VxkGXt+3DxPIw8CcmvAB0j3sVVfbhX+M4Tpw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-arm": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-arm/-/typescript-linux-arm-7.0.2.tgz", + "integrity": "sha512-gffT3xPz9sR7j/YJExkyPntrI0P2EP9XbOyWzth2/Gs0RstK+90RBcO0ncXoXy/beYll1SXw846Nf2zdnEz0QQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-arm64/-/typescript-linux-arm64-7.0.2.tgz", + "integrity": "sha512-Qh4eU4/y3yDjnfjjyPYihMj5/ODIlmt+Bzu17OI+fiSRDW57QmU5SiN63exPRNJPKUzcc1INa1NXdrJ+MqHjUQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-loong64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-loong64/-/typescript-linux-loong64-7.0.2.tgz", + "integrity": "sha512-uEHck9i8hoAzXPiYRib1O7miOnz23SxIeVl6F4LXox+qov1K35jHcEW6VHKvZI+pyvl7fZEP4MCU5LYvIq1GuQ==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-mips64el": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-mips64el/-/typescript-linux-mips64el-7.0.2.tgz", + "integrity": "sha512-R4KvAMnE43W5Qeqb0Ly56O3mWMWIAgsMyz36DCaycd5nbg/9kzm0liw3JocfRqyJY0KPmzFjbswozXyW0DnIYA==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-ppc64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-ppc64/-/typescript-linux-ppc64-7.0.2.tgz", + "integrity": "sha512-DORx5b3sd/4S7eayxm4FQv+A7CrkUIGRaHiwI8oiHTAI1fAPWhF4J0vAlkC8biAlHSVVwxMQ3tjZ2/DVbnQiiA==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-riscv64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-riscv64/-/typescript-linux-riscv64-7.0.2.tgz", + "integrity": "sha512-wf0jqEDOjrPRnKwYRyyJDRo11KMbvMFrU+q4zqKyChODBzvlkbhNQfKvLxQCcwTpdDaXSHZTVuh0JoCrKCUMHQ==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-s390x": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-s390x/-/typescript-linux-s390x-7.0.2.tgz", + "integrity": "sha512-IkwJc3L7yhytWd/ewjyxNDfOmswCm9GWMJT/ue/dU4aZNbwZeYAetq42VyLmsmSjvoX7z74X6ZaYCtzAr0EuGw==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-linux-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-linux-x64/-/typescript-linux-x64-7.0.2.tgz", + "integrity": "sha512-EYdf2cNg7rgCWJnxCdJ+F3V39O8ihb37eHAu1LK8oAFizgTQbPOK7zHHXbPt8rX24COqODXeI3sIf0fCXG7H/A==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-netbsd-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-netbsd-arm64/-/typescript-netbsd-arm64-7.0.2.tgz", + "integrity": "sha512-+polYF4MF04aPpO5FTkHran9yUQDSXqy5GiSDKpsll5jy3l3+g9QLhpf39T+ePtefhXLOGrLl0QIjkQP6VnelA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-netbsd-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-netbsd-x64/-/typescript-netbsd-x64-7.0.2.tgz", + "integrity": "sha512-8YIT0EHM/3dq10ZOVF/A7pc/YSMtbcecct4rWtexrnSCHOPcpC2KTLXfTCR6vDpnSiY12heNb1GiN/wu+T/FyA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-openbsd-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-openbsd-arm64/-/typescript-openbsd-arm64-7.0.2.tgz", + "integrity": "sha512-APT8+ClYnuYm1u9+kgGXoMj2VzWzcymwh2gNSQVySHfkRDGOTVkoWLjCmOQSaO+PoqQ57B0flRp9SA+7GnnkzQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-openbsd-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-openbsd-x64/-/typescript-openbsd-x64-7.0.2.tgz", + "integrity": "sha512-yX7s+Q0Dln0Dt9tEzZsAjXXR/+ytBM7AlglaqyeMPxQszJ1JhlJdZ6jLA+IzldHtflX81em7lDao1xXu+aRRkg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-sunos-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-sunos-x64/-/typescript-sunos-x64-7.0.2.tgz", + "integrity": "sha512-dLJDGaLZ1D4HPQn62u1n8mBDkJREwMsAkCdkwd4Ieqw+x3TUyTsqY0YiBCtE6H6OzzgGk3iuZ3vFWRS+E8/d1g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-win32-arm64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-win32-arm64/-/typescript-win32-arm64-7.0.2.tgz", + "integrity": "sha512-Gyl1Vy6OsWesLzmq+EP0Fb7b4Nid5232AvcA2SFcdYreldpNtYFFofPjnt62y9hQy7VTaZp65ICJjuAQRaVcIQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/@typescript/typescript-win32-x64": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/@typescript/typescript-win32-x64/-/typescript-win32-x64-7.0.2.tgz", + "integrity": "sha512-0BQ3HkAHHlKLSp1qRvf3SUhGpGsDuhB/jgFw75guyqbxJqEaS0Cw/VFO8i2nHglJUzQCRtMMR/IBAKE3ETMC4g==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "Apache-2.0", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=16.20.0" + } + }, + "node_modules/detect-libc": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz", + "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=8" + } + }, + "node_modules/fdir": { + "version": "6.5.0", + "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz", + "integrity": "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12.0.0" + }, + "peerDependencies": { + "picomatch": "^3 || ^4" + }, + "peerDependenciesMeta": { + "picomatch": { + "optional": true + } + } + }, + "node_modules/fsevents": { + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", + "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/lightningcss": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss/-/lightningcss-1.33.0.tgz", + "integrity": "sha512-WkUDrojuJs0xkgGf2udWxa3yGBRxPtxUkB79i6aCZLRgc7PM8fZe9TosfPDcvEpQZbuFASnHYmRLBLUbmLOIIA==", + "dev": true, + "license": "MPL-2.0", + "dependencies": { + "detect-libc": "^2.0.3" + }, + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + }, + "optionalDependencies": { + "lightningcss-android-arm64": "1.33.0", + "lightningcss-darwin-arm64": "1.33.0", + "lightningcss-darwin-x64": "1.33.0", + "lightningcss-freebsd-x64": "1.33.0", + "lightningcss-linux-arm-gnueabihf": "1.33.0", + "lightningcss-linux-arm64-gnu": "1.33.0", + "lightningcss-linux-arm64-musl": "1.33.0", + "lightningcss-linux-x64-gnu": "1.33.0", + "lightningcss-linux-x64-musl": "1.33.0", + "lightningcss-win32-arm64-msvc": "1.33.0", + "lightningcss-win32-x64-msvc": "1.33.0" + } + }, + "node_modules/lightningcss-android-arm64": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-android-arm64/-/lightningcss-android-arm64-1.33.0.tgz", + "integrity": "sha512-gEpRTalKdosp4Bb8qWtc2iOgE5SeIHlpS1up9bFq2wAyYhl1UdTObYiHe98zEM9SQvSoqQZ1IQD0JNpg3Ml5pg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-darwin-arm64": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-darwin-arm64/-/lightningcss-darwin-arm64-1.33.0.tgz", + "integrity": "sha512-Sciaz8eenNTKn9b3t7+xr0ipTp9YxKQY4npwQ3mrRuL0BAVHBLyZxofhaKBAVtzmtRZ/zTyo0/to4B1uWG/Djg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-darwin-x64": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-darwin-x64/-/lightningcss-darwin-x64-1.33.0.tgz", + "integrity": "sha512-Z5UPAxzrjlWNNyGy6i65cJzzvgJ5D3T6wMvs+gWpY9d7qRhANrxqAp6LhxIgZhWEw18RfJTGcRxjuLIBr+m8XQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-freebsd-x64": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-freebsd-x64/-/lightningcss-freebsd-x64-1.33.0.tgz", + "integrity": "sha512-QQM/Ti/hQajJwCY+RiWuCZ9sdtI/XQk7nDK5vC8kkdwixezOlDgvDx7+RT+QjK6FcFT4MpsuoBnHIo/O3StRRg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-linux-arm-gnueabihf": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-arm-gnueabihf/-/lightningcss-linux-arm-gnueabihf-1.33.0.tgz", + "integrity": "sha512-N7FVBe6iS24MlM6R/4RBTxGhQheZGs7tiQ9U32UtF75NzP5Q7xWPRqLBCKxlRQRk3rY1jCIPLzx7WzOhuUIRLQ==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-linux-arm64-gnu": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-gnu/-/lightningcss-linux-arm64-gnu-1.33.0.tgz", + "integrity": "sha512-j2v/itmy4HlNxlc6voKXYgBqNi0Ng2LShg4z7GufpEgs05P+2suBVyi9I6YHq5uoVFx9ETin3eCEhLVyXGQnKg==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MPL-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-linux-arm64-musl": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-musl/-/lightningcss-linux-arm64-musl-1.33.0.tgz", + "integrity": "sha512-yiO5ROMuYQgXbC60yjZU5CYSFZGKXL0HFATXt9mHJn1+zW55oCtMI9NfcVhYLMFDL7gV7oBPon/EmMMGg2OvtQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MPL-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-linux-x64-gnu": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-x64-gnu/-/lightningcss-linux-x64-gnu-1.33.0.tgz", + "integrity": "sha512-ar+Ju7LmcN0Jo4FpL4hpFybwNG9/3A/Br5KW2n2jyODg3MEZXaDYADdemoNS+BDNfMgKvylJLj4S5tyRActuAg==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MPL-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-linux-x64-musl": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-x64-musl/-/lightningcss-linux-x64-musl-1.33.0.tgz", + "integrity": "sha512-RYiYbkokw0trfKqqzfF55lginwEPrD3OJDfTuJzFs1MK6iFnDenaz1fqLLtX4ITG3OktJQXOeTaw1awrBAlZPw==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MPL-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-win32-arm64-msvc": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-win32-arm64-msvc/-/lightningcss-win32-arm64-msvc-1.33.0.tgz", + "integrity": "sha512-1K+MPfLSFVpphzpdbfkhlWk6wBrTObBzS2T6db10PNOZgR9GoVsAWzwNyuhUYYbTp23j+4RrncfujZ4uAzXvwA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-win32-x64-msvc": { + "version": "1.33.0", + "resolved": "https://registry.npmjs.org/lightningcss-win32-x64-msvc/-/lightningcss-win32-x64-msvc-1.33.0.tgz", + "integrity": "sha512-OlEICDx/Xl0FqSp4bry8zFnCvGpig3Gl4gCquvYwHuqJKEC1+n9NgDniFvqHGmMv1ZkqDJrDqKKSykTDX+ehuA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lit": { + "version": "3.3.3", + "resolved": "https://registry.npmjs.org/lit/-/lit-3.3.3.tgz", + "integrity": "sha512-fycuvZg/hkpozL00lm1pEJH5nN/lr9ZXd6mJI2HSN4+Bzc+LDNdEApJ6HFbPkdFNHLvOplIIuJvxkS4XUxqirw==", + "license": "BSD-3-Clause", + "dependencies": { + "@lit/reactive-element": "^2.1.0", + "lit-element": "^4.2.0", + "lit-html": "^3.3.0" + } + }, + "node_modules/lit-element": { + "version": "4.2.2", + "resolved": "https://registry.npmjs.org/lit-element/-/lit-element-4.2.2.tgz", + "integrity": "sha512-aFKhNToWxoyhkNDmWZwEva2SlQia+jfG0fjIWV//YeTaWrVnOxD89dPKfigCUspXFmjzOEUQpOkejH5Ly6sG0w==", + "license": "BSD-3-Clause", + "dependencies": { + "@lit-labs/ssr-dom-shim": "^1.5.0", + "@lit/reactive-element": "^2.1.0", + "lit-html": "^3.3.0" + } + }, + "node_modules/lit-html": { + "version": "3.3.3", + "resolved": "https://registry.npmjs.org/lit-html/-/lit-html-3.3.3.tgz", + "integrity": "sha512-el8M6jK2o3RXBnrSHX3ZKrsN8zEV63pSExTO1wYJz7QndGYZ8353e2a5PPX+qHe2aGayfnchQmkAojaWAREOIA==", + "license": "BSD-3-Clause", + "dependencies": { + "@types/trusted-types": "^2.0.2" + } + }, + "node_modules/nanoid": { + "version": "3.3.19", + "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.19.tgz", + "integrity": "sha512-Y2tUNy4ouw6tq5oDSKeQYGOyhkUBhNOcGV/02KC+6kd9eDGqdZd++mjMiIDilrBYvjEnCYvVtsuHCuP+okSfug==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "bin": { + "nanoid": "bin/nanoid.cjs" + }, + "engines": { + "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" + } + }, + "node_modules/picocolors": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", + "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", + "dev": true, + "license": "ISC" + }, + "node_modules/picomatch": { + "version": "4.0.7", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.7.tgz", + "integrity": "sha512-qcJu88Q2IWqJsDD529JKMdwGm/dvInW4HvQnRwiH9JtihJvzGOscDtHE3x1pBKeUOTysQ8kVmLnJ2kJu7yhcGA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/jonschlinkert" + } + }, + "node_modules/postcss": { + "version": "8.5.28", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.28.tgz", + "integrity": "sha512-RRuzqDtt5Y9h3quz5hWhK+TPnsmVs6WwSU6LkJMeY4HstUEDuYTG8UJSdawMRzmzAtV+KEoG8N3Qg2qLy5vM/A==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/postcss/" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/postcss" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "nanoid": "^3.3.18", + "picocolors": "^1.1.1", + "source-map-js": "^1.2.1" + }, + "engines": { + "node": "^10 || ^12 || >=14" + } + }, + "node_modules/rolldown": { + "version": "1.2.11", + "resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.2.11.tgz", + "integrity": "sha512-qpSwIyz0jHQq5qXBTNxFmE6664rJ7O+4TvPFOiOaBSrz8IOHc1koKKSqTM2H6u1UG1+TveuC6vaDHKXFOvb1Kw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@oxc-project/types": "=0.151.0", + "@rolldown/pluginutils": "^1.0.0" + }, + "bin": { + "rolldown": "bin/cli.mjs" + }, + "engines": { + "node": "^20.19.0 || >=22.12.0" + }, + "optionalDependencies": { + "@rolldown/binding-android-arm-eabi": "1.2.11", + "@rolldown/binding-android-arm64": "1.2.11", + "@rolldown/binding-darwin-arm64": "1.2.11", + "@rolldown/binding-darwin-x64": "1.2.11", + "@rolldown/binding-freebsd-x64": "1.2.11", + "@rolldown/binding-linux-arm-gnueabihf": "1.2.11", + "@rolldown/binding-linux-arm64-gnu": "1.2.11", + "@rolldown/binding-linux-arm64-musl": "1.2.11", + "@rolldown/binding-linux-ppc64-gnu": "1.2.11", + "@rolldown/binding-linux-s390x-gnu": "1.2.11", + "@rolldown/binding-linux-x64-gnu": "1.2.11", + "@rolldown/binding-linux-x64-musl": "1.2.11", + "@rolldown/binding-openharmony-arm64": "1.2.11", + "@rolldown/binding-win32-arm64-msvc": "1.2.11", + "@rolldown/binding-win32-x64-msvc": "1.2.11" + } + }, + "node_modules/source-map-js": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", + "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==", + "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/tinyglobby": { + "version": "0.2.17", + "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.17.tgz", + "integrity": "sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==", + "dev": true, + "license": "MIT", + "dependencies": { + "fdir": "^6.5.0", + "picomatch": "^4.0.4" + }, + "engines": { + "node": ">=12.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/SuperchupuDev" + } + }, + "node_modules/typescript": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-7.0.2.tgz", + "integrity": "sha512-8FYau96o3NKOhbjKi/qNvG/W5jhzxkbdm5sj9AbZ/5T5sWqn3hJgLfGx27sRKZWTvyzCP8dLRBTf5tBTSRVUNA==", + "dev": true, + "license": "Apache-2.0", + "bin": { + "tsc": "bin/tsc" + }, + "engines": { + "node": ">=16.20.0" + }, + "optionalDependencies": { + "@typescript/typescript-aix-ppc64": "7.0.2", + "@typescript/typescript-darwin-arm64": "7.0.2", + "@typescript/typescript-darwin-x64": "7.0.2", + "@typescript/typescript-freebsd-arm64": "7.0.2", + "@typescript/typescript-freebsd-x64": "7.0.2", + "@typescript/typescript-linux-arm": "7.0.2", + "@typescript/typescript-linux-arm64": "7.0.2", + "@typescript/typescript-linux-loong64": "7.0.2", + "@typescript/typescript-linux-mips64el": "7.0.2", + "@typescript/typescript-linux-ppc64": "7.0.2", + "@typescript/typescript-linux-riscv64": "7.0.2", + "@typescript/typescript-linux-s390x": "7.0.2", + "@typescript/typescript-linux-x64": "7.0.2", + "@typescript/typescript-netbsd-arm64": "7.0.2", + "@typescript/typescript-netbsd-x64": "7.0.2", + "@typescript/typescript-openbsd-arm64": "7.0.2", + "@typescript/typescript-openbsd-x64": "7.0.2", + "@typescript/typescript-sunos-x64": "7.0.2", + "@typescript/typescript-win32-arm64": "7.0.2", + "@typescript/typescript-win32-x64": "7.0.2" + } + }, + "node_modules/undici-types": { + "version": "7.24.6", + "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.24.6.tgz", + "integrity": "sha512-WRNW+sJgj5OBN4/0JpHFqtqzhpbnV0GuB+OozA9gCL7a993SmU+1JBZCzLNxYsbMfIeDL+lTsphD5jN5N+n0zg==", + "dev": true, + "license": "MIT" + }, + "node_modules/vite": { + "version": "8.3.1", + "resolved": "https://registry.npmjs.org/vite/-/vite-8.3.1.tgz", + "integrity": "sha512-/bvH9E9tmCXRGp2uXY3WbOldqpTwFkbha/8ANaEQ6VkxhH60KyqLwgZq6lG2y+4uT55x9+9eUHMpQ7uGnOCKjA==", + "dev": true, + "license": "MIT", + "dependencies": { + "lightningcss": "^1.33.0", + "picomatch": "^4.0.7", + "postcss": "^8.5.28", + "rolldown": "~1.2.9", + "tinyglobby": "^0.2.17" + }, + "bin": { + "vite": "bin/vite.js" + }, + "engines": { + "node": "^20.19.0 || >=22.12.0" + }, + "funding": { + "url": "https://github.com/vitejs/vite?sponsor=1" + }, + "optionalDependencies": { + "fsevents": "~2.3.3" + }, + "peerDependencies": { + "@types/node": "^20.19.0 || >=22.12.0", + "@vitejs/devtools": "^0.7.1", + "esbuild": "^0.27.0 || ^0.28.0", + "jiti": ">=1.21.0", + "less": "^4.0.0", + "sass": "^1.70.0", + "sass-embedded": "^1.70.0", + "stylus": ">=0.54.8", + "sugarss": "^5.0.0", + "terser": "^5.16.0", + "tsx": "^4.8.1", + "yaml": "^2.4.2" + }, + "peerDependenciesMeta": { + "@types/node": { + "optional": true + }, + "@vitejs/devtools": { + "optional": true + }, + "esbuild": { + "optional": true + }, + "jiti": { + "optional": true + }, + "less": { + "optional": true + }, + "sass": { + "optional": true + }, + "sass-embedded": { + "optional": true + }, + "stylus": { + "optional": true + }, + "sugarss": { + "optional": true + }, + "terser": { + "optional": true + }, + "tsx": { + "optional": true + }, + "yaml": { + "optional": true + } + } + } + } +} diff --git a/docs/development/ui-lit-spike/package.json b/docs/development/ui-lit-spike/package.json new file mode 100644 index 0000000..28d7524 --- /dev/null +++ b/docs/development/ui-lit-spike/package.json @@ -0,0 +1,21 @@ +{ + "name": "symphonia-ui-lit-spike", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "check": "tsc --noEmit", + "test": "node --experimental-strip-types --test src/appearance.test.ts", + "build": "vite build", + "verify-output": "node scripts/verify-output.mjs", + "check-boundary": "node scripts/check-boundary.mjs" + }, + "dependencies": { + "lit": "3.3.3" + }, + "devDependencies": { + "@types/node": "24.19.0", + "typescript": "7.0.2", + "vite": "8.3.1" + } +} diff --git a/docs/development/ui-lit-spike/scripts/check-boundary.mjs b/docs/development/ui-lit-spike/scripts/check-boundary.mjs new file mode 100644 index 0000000..7325c9b --- /dev/null +++ b/docs/development/ui-lit-spike/scripts/check-boundary.mjs @@ -0,0 +1,13 @@ +import assert from "node:assert/strict"; +import { readdirSync, readFileSync } from "node:fs"; + +for (const file of readdirSync("src").filter((name) => name.endsWith(".ts") && !name.endsWith(".test.ts"))) { + const source = readFileSync(`src/${file}`, "utf8"); + const imports = [...source.matchAll(/\b(?:import|export)\s+(?:[^;]*?\s+from\s+)?["']([^"']+)["']/g)] + .map((match) => match[1]); + for (const specifier of imports) { + assert.ok(specifier === "lit" || specifier.startsWith("./"), `${file} imports non-presentation module ${specifier}`); + assert.ok(!specifier.includes(".."), `${file} escapes its presentation package`); + } + assert.doesNotMatch(source, /\b(?:fetch|XMLHttpRequest|localStorage|sessionStorage)\b|window\.parent/, `${file} crosses the fixture boundary`); +} diff --git a/docs/development/ui-lit-spike/scripts/verify-output.mjs b/docs/development/ui-lit-spike/scripts/verify-output.mjs new file mode 100644 index 0000000..82a43cf --- /dev/null +++ b/docs/development/ui-lit-spike/scripts/verify-output.mjs @@ -0,0 +1,14 @@ +import assert from "node:assert/strict"; +import { readFileSync, statSync } from "node:fs"; +import { gzipSync } from "node:zlib"; + +const html = readFileSync("dist/index.html", "utf8"); +const scripts = [...html.matchAll(/]*\bsrc="([^"]+)"/g)].map((match) => match[1]); +assert.equal(scripts.length, 1, "one bundled entry asset is expected"); +assert.match(scripts[0], /^\.\/assets\/[A-Za-z0-9._-]+\.js$/); +assert.doesNotMatch(html, /(?:src|href)="(?:\/|https?:|\/\/)/, "built asset URLs must remain relative"); + +const asset = scripts[0].slice(2); +const bytes = readFileSync(`dist/${asset}`); +assert.ok(statSync(`dist/${asset}`).isFile()); +console.log(`Spike asset: ${bytes.length} bytes, ${gzipSync(bytes).length} bytes gzip`); diff --git a/docs/development/ui-lit-spike/src/appearance.test.ts b/docs/development/ui-lit-spike/src/appearance.test.ts new file mode 100644 index 0000000..a65f29a --- /dev/null +++ b/docs/development/ui-lit-spike/src/appearance.test.ts @@ -0,0 +1,16 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { resolveTheme } from "./appearance.ts"; + +test("auto follows the browser light preference", () => { + assert.equal(resolveTheme("auto", false), "light"); +}); + +test("auto follows the browser dark preference", () => { + assert.equal(resolveTheme("auto", true), "dark"); +}); + +test("an explicit preference does not depend on the browser", () => { + assert.equal(resolveTheme("light", true), "light"); + assert.equal(resolveTheme("dark", false), "dark"); +}); diff --git a/docs/development/ui-lit-spike/src/appearance.ts b/docs/development/ui-lit-spike/src/appearance.ts new file mode 100644 index 0000000..0b79cda --- /dev/null +++ b/docs/development/ui-lit-spike/src/appearance.ts @@ -0,0 +1,8 @@ +export type Theme = "light" | "dark"; +export type Preference = Theme | "auto"; + +// Browser fallback only. No Home Assistant theme property has been verified. +export function resolveTheme(preference: Preference, prefersDark: boolean): Theme { + if (preference === "light" || preference === "dark") return preference; + return prefersDark ? "dark" : "light"; +} diff --git a/docs/development/ui-lit-spike/src/main.ts b/docs/development/ui-lit-spike/src/main.ts new file mode 100644 index 0000000..ec8d1db --- /dev/null +++ b/docs/development/ui-lit-spike/src/main.ts @@ -0,0 +1,82 @@ +import { LitElement, css, html } from "lit"; +import { resolveTheme, type Preference } from "./appearance.ts"; + +type Section = "connections" | "library" | "activity"; +const sections: readonly Section[] = ["connections", "library", "activity"]; + +class SymUiSpike extends LitElement { + static properties = { + preference: { state: true }, + section: { state: true }, + systemDark: { state: true }, + }; + + private preference: Preference = "auto"; + private section: Section = "connections"; + private readonly colorScheme = window.matchMedia("(prefers-color-scheme: dark)"); + private systemDark = this.colorScheme.matches; + private readonly onColorSchemeChange = (event: MediaQueryListEvent): void => { + this.systemDark = event.matches; + }; + + connectedCallback(): void { + super.connectedCallback(); + this.colorScheme.addEventListener("change", this.onColorSchemeChange); + } + + disconnectedCallback(): void { + this.colorScheme.removeEventListener("change", this.onColorSchemeChange); + super.disconnectedCallback(); + } + + private selectSection(section: Section): void { + this.section = section; + } + + private toggleTheme(): void { + this.preference = this.theme === "dark" ? "light" : "dark"; + } + + private get theme(): "light" | "dark" { + return resolveTheme(this.preference, this.systemDark); + } + + render() { + const theme = this.theme; + return html` +
+
SymphoniaUI tooling spike · synthetic data
+ +
+

${this.section === "connections" ? "Connections" : this.section === "library" ? "Library" : "Activity"}

+

Presentation-only fixture. No provider or Home Assistant connection.

+ ${this.section === "connections" ? html`

Source service

Connected · last complete import at 09:14

` : this.section === "library" ? html`

Library available with limitations

12 playlists retained. Retry the incomplete import; no existing entries were removed.

` : html`

Copy needs reconciliation

Destination outcome is unknown. Inspect the durable operation before retrying; no success is inferred.

`} +
+
+ `; + } + + static styles = css` + :host { display: block; font: 14px/1.45 system-ui, sans-serif; } + *, *::before, *::after { box-sizing: border-box; } + .light { --canvas: #fafafa; --surface: #fff; --border: #e5e5e5; --text: #1a1a1a; --secondary: #616161; --accent: #008db8; } + .dark { --canvas: #141414; --surface: #242424; --border: #555; --text: #f7f7f7; --secondary: #d0d0d0; --accent: #49c7ef; } + .light, .dark { min-height: 100vh; color: var(--text); background: var(--canvas); } + header, nav { display: flex; align-items: center; gap: 16px; padding: 12px max(16px, calc((100% - 960px) / 2)); border-bottom: 1px solid var(--border); background: var(--surface); } + header span { margin-inline-start: auto; color: var(--secondary); font-size: 12px; } + button { min-height: 40px; border: 0; color: inherit; background: transparent; cursor: pointer; } + button:focus-visible { outline: 3px solid var(--accent); outline-offset: 2px; } + nav { gap: 0; overflow-x: auto; } + nav button { padding: 0 14px; border-bottom: 2px solid transparent; text-transform: capitalize; } + nav button[aria-current="page"] { border-bottom-color: var(--accent); font-weight: 700; } + main { max-width: 992px; margin: auto; padding: 24px 16px; } + h1 { margin: 0; font-size: 28px; font-weight: 500; } + main > p, article p { color: var(--secondary); } + article { max-width: 640px; margin-top: 24px; padding: 16px; border: 1px solid var(--border); border-radius: 12px; background: var(--surface); } + h2 { margin: 0; font-size: 18px; font-weight: 500; } + @media (max-width: 600px) { header span { display: none; } header button { margin-inline-start: auto; } } + @media (forced-colors: active) { article, nav button[aria-current="page"] { border-color: CanvasText; } } + `; +} + +customElements.define("sym-ui-spike", SymUiSpike); diff --git a/docs/development/ui-lit-spike/tsconfig.json b/docs/development/ui-lit-spike/tsconfig.json new file mode 100644 index 0000000..c78743d --- /dev/null +++ b/docs/development/ui-lit-spike/tsconfig.json @@ -0,0 +1,14 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "ESNext", + "moduleResolution": "Bundler", + "lib": ["ES2022", "DOM"], + "strict": true, + "noEmit": true, + "allowImportingTsExtensions": true, + "skipLibCheck": true, + "types": ["node"] + }, + "include": ["src", "vite.config.ts"] +} diff --git a/docs/development/ui-lit-spike/vite.config.ts b/docs/development/ui-lit-spike/vite.config.ts new file mode 100644 index 0000000..5122589 --- /dev/null +++ b/docs/development/ui-lit-spike/vite.config.ts @@ -0,0 +1,6 @@ +import { defineConfig } from "vite"; + +export default defineConfig({ + base: "./", + build: { outDir: "dist" }, +}); diff --git a/docs/development/ui-spike/README.md b/docs/development/ui-spike/README.md index 0c2ea2a..a646fee 100644 --- a/docs/development/ui-spike/README.md +++ b/docs/development/ui-spike/README.md @@ -1,9 +1,11 @@ # Home Assistant-adjacent UI spike (review fixture) -This is a **non-production, synthetic-data-only** visual/interaction fixture for the [UI foundation SDD](../../../specs/home-assistant-native-ui.md). It is not served by the App, has no application/API calls, does not perform copy or connect providers, and does not select Symphonia's frontend framework. Open `index.html` locally to inspect the shell. The controls only change the local fixture view/theme. +This is a **non-production, synthetic-data-only** visual/interaction fixture for the [UI foundation SDD](../../../specs/home-assistant-native-ui.md). It is not served by the App, has no application/API calls, and does not perform copy or connect providers. It predates the [Lit/TypeScript/Vite decision](../../decisions/0005-lit-typescript-vite-ui.md) and is not a test of that stack. Open `index.html` locally to inspect the shell. The controls only change the local fixture view/theme. Automated browser inspection of this local file was blocked by the current browser security policy on 2026-09-27. The repository tests check structure and isolation, but **no Symphonia screenshot, visual parity, responsive layout, or browser accessibility result has been approved**. A reviewer must perform the inspection below in an authorized environment; a future release cannot substitute static tests for it. +The [host-context evidence](host-context-evidence.md) records what current official sources expose, and the [visual-reference procedure](visual-reference-plan.md) is a proposal awaiting reviewer approval. + ## Evidence and intentional choices - Reviewed 2026-09-27 against the public [Home Assistant demo reference set](https://github.com/vypdev/homeassistant-gateway/tree/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/ui-reference/home-assistant) captured 2026-08-05 (especially `menu-settings-light.png`, `menu-automations-light.png`, and `menu-automations-mobile.png`). Those captures are dated evidence, **not** a current supported-version matrix or proof of dark theme behavior. @@ -20,4 +22,4 @@ Automated browser inspection of this local file was blocked by the current brows ## Remaining gates -`RG-006` still needs a real supported Home Assistant/browser matrix, verified public host-context/Ingress contract, frontend/build selection, and approved capture/update/reviewer procedure. This spike does not satisfy the SDD's 84-case production test budget or authorize production UI implementation. +`RG-006` still needs a real supported Home Assistant/browser matrix, verified public [host-context/Ingress contract](host-context-evidence.md), Lit/Vite build/compatibility evidence, and approved capture/update/reviewer procedure. This spike does not satisfy the SDD's 84-case production test budget or authorize production UI implementation. diff --git a/docs/development/ui-spike/host-context-evidence.md b/docs/development/ui-spike/host-context-evidence.md new file mode 100644 index 0000000..e06ac6e --- /dev/null +++ b/docs/development/ui-spike/host-context-evidence.md @@ -0,0 +1,40 @@ +# RG-006 host-context and compatibility evidence + +**Reviewed:** 2026-09-27. **Status:** partial research; no supported HA/browser matrix or production App-context claim yet. This is evidence for the [UI foundation SDD](../../../specs/home-assistant-native-ui.md), not an implementation contract. + +## What the public sources actually establish + +| Concern | Official evidence | Consequence for Symphonia | +| --- | --- | --- | +| App Ingress | [App presentation guidance](https://developers.home-assistant.io/docs/apps/presentation/#ingress) says Ingress authenticates and proxies the UI, exposes `X-Ingress-Path`, and supports HTTP, streaming, and WebSockets; [App configuration](https://developers.home-assistant.io/docs/apps/configuration/) defines `ingress`, `ingress_port`, and `ingress_entry`. | Resolve a server-validated base path per request; never assume `/`. These docs do not prove our router/build/asset behavior. | +| Safe area, default | [2026.8 component update](https://developers.home-assistant.io/blog/2026/07/31/frontend-component-updates-2026.8/) says App iframes receive host-managed safe-area padding by default. | Begin with host-managed padding. Do not add the same insets in the App interior by default. | +| Safe area, opt-in | The same update documents `home-assistant/subscribe-properties` with `handleSafeArea: true` and a resulting `safeAreaInsets` property. The [current App panel source](https://github.com/home-assistant/frontend/blob/dev/src/panels/app/ha-panel-app.ts) shows `top`, `right`, `bottom`, and `left` values as CSS-length strings. | Opt-in only if a verified design needs full-bleed content and after validating source, origin, shape, units, and double-padding behavior. The static fixture's `env(safe-area-inset-bottom)` is exploratory, not evidence that the iframe receives HA insets. | +| Other App properties | The current App panel source sends `type: home-assistant/properties`, `narrow`, `route`, and `safeAreaInsets` after subscription. It posts to the iframe using `"*"`; this is source-code evidence, not a promise across releases. | Treat the message as untrusted UI hints, not identity/authorization. Do not assume a correlation ID or a theme/locale/timezone field. Validate `event.source`, same-origin expectation, type, and bounded fields against a supported-version fixture before use. | +| Theme, locale, direction, timezone | The inspected public App-properties message contains none of these fields. No public App iframe contract for them was verified in this research. | Use deterministic browser/standalone fallbacks (`prefers-color-scheme`, `navigator.language`, `Intl` timezone, locale-derived direction) until another *public, versioned and tested* source is proven. This may differ from Home Assistant's per-user settings and must be disclosed. Never read parent DOM or private storage to close the gap. | +| Tooling | [Lit components](https://lit.dev/docs/components/overview/) are custom elements; [Lit tooling guidance](https://lit.dev/docs/tools/development/) supports TypeScript and Vite; [Vite build documentation](https://vite.dev/guide/build.html) describes production assets. [ADR 0005](../../decisions/0005-lit-typescript-vite-ui.md) records owner approval. | The stack decision is closed. Ingress asset/base-path behavior, bundle cost, browser matrix, accessibility, and catalog build remain unverified. | + +The [isolated Lit build spike](../ui-lit-spike/README.md) used Node 24.20.0, Lit 3.3.3, TypeScript 7.0.2, Vite 8.3.1, and a pinned npm lockfile on 2026-09-27. Type-checking, three pure fallback tests, a presentation-import guard, Vite build, and relative-asset inspection passed. Its single JS asset was **19,327 bytes / 7,529 bytes gzip**. This is a tiny synthetic sample, not a budget or performance result for the full component catalog, a modest HA device, or Ingress. The CI job reruns these checks on Node 24; browser and device evidence remain open. + +The `dev` branch of Home Assistant frontend is mutable. Before releasing, pin the exact supported tag/commit and verify the resulting browser behavior. The 2026.8 blog establishes the introduction of safe-area behavior; it does not establish that every earlier or later installation exposes identical properties. + +## Candidate matrix — not yet a support promise + +| Target | Candidate | Evidence still required | +| --- | --- | --- | +| Home Assistant Core/App panel | 2026.8 and a current supported release | Real Supervisor/App Ingress tests for default and opted-in safe area, property changes, origin/source, reload/deep link, and user theme/locale behavior. | +| Desktop browser | Current stable Chromium, Firefox, Safari | Keyboard/a11y, theme, zoom, RTL, responsive and route/asset checks inside Ingress; record exact versions and OS. | +| Companion App | Current Android WebView and iOS WKWebView | Narrow/safe-area/virtual-keyboard, orientation, session and deep-link tests; record app/OS/webview versions. | +| Standalone fallback | Current stable browser matrix | Same feature fixture semantics without Home Assistant framing; authenticated route behavior when that profile is designed. | + +An unsupported or absent host property is not an error by itself: the UI remains readable with fallback tokens and host-managed safe area. A confirmed compatibility hazard gets a versioned warning; no speculative warning based only on a version string. + +## Required test sequence before marking the SDD ready + +1. Record a disposable HA installation's Core, Supervisor, frontend commit/build, Companion App (if applicable), browser, OS, theme, locale, direction, and viewport. Use public/demo data only for visual capture. +2. Open the App through a non-root Ingress prefix; inspect `X-Ingress-Path` at the trusted server boundary. Reload and deep-link each fixture route; verify relative assets, HTTP, and any WebSocket/SSE path stay under that prefix. +3. In 2026.8+, compare default host-managed padding against `handleSafeArea: true` subscription on phone/orientation changes. Validate that top/bottom/side insets are applied exactly once and that the host header is not duplicated inside the App. +4. Capture the exact `home-assistant/properties` message fields from an authorized disposable instance without saving private data. Reject wrong source/origin, malformed objects, oversized values, unsafe CSS lengths, and unexpected fields in adapter tests. Check what happens on unsubscribe/reload. +5. Change Home Assistant user theme, locale, and timezone. Record whether the iframe receives any supported public signal. If it does not, retain and document browser/standalone fallbacks; do not infer equality with the HA user preference. +6. Run component/browser/accessibility and visual-reference review at phone, tablet, and desktop sizes, light/dark, forced colors, reduced motion, 200% zoom, long/RTL text, and virtual keyboard. Record explicit reviewer approval or a divergence with rationale. + +The [fixture review checklist](README.md) is a starting point only. No live Home Assistant probe, screenshot, or browser visual/accessibility approval was performed in this evidence pass. diff --git a/docs/development/ui-spike/visual-reference-plan.md b/docs/development/ui-spike/visual-reference-plan.md new file mode 100644 index 0000000..8301945 --- /dev/null +++ b/docs/development/ui-spike/visual-reference-plan.md @@ -0,0 +1,32 @@ +# Proposed Home Assistant visual-reference procedure + +**Drafted:** 2026-09-27. **Approval:** pending product/UX and accessibility review. This is a procedure proposal for `RG-006`, not an accepted baseline or evidence that Symphonia visually matches Home Assistant. + +## Candidate references + +The public [Home Assistant demo](https://demo.home-assistant.io/) is the primary capture target. The pinned [Gateway reference set](https://github.com/vypdev/homeassistant-gateway/tree/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/ui-reference/home-assistant) documents public-demo captures dated 2026-08-05, including Settings and Automations at 1280×720 and 390×844. It does not identify a Home Assistant release for each image and includes no approved dark-theme reference; therefore it can guide **candidate** hierarchy/density review but cannot itself become Symphonia's versioned official baseline. + +Candidate screen pairs for the first review: + +| Home Assistant surface | Symphonia fixture/product surface | Compare | +| --- | --- | --- | +| Settings landing, wide and phone | Connections | heading, settings rows, spacing, action hierarchy, mobile reflow | +| Automations list, wide and phone | Library and Activity | flat navigation, filters/list density, row/status treatment, partial/empty states | +| Developer Tools States, wide and phone | Copy review and later diagnostics | dense evidence, bounded scrolling, labels, actions and responsive alternatives | + +These are **comparative analogues**, not claims that music operations are Home Assistant automations or that one screenshot defines interaction behavior. + +## Capture manifest fields + +For every accepted image, record: `reference_id`, official URL and route, Home Assistant Core/frontend version or exact source commit, capture UTC timestamp, viewport and device pixel ratio, browser/OS version, theme, locale and direction, public-data provenance, screenshot path and SHA-256, capture command/configuration, and reviewer. Keep immutable historical entries; a new release appends a new entry and changes a separate current pointer. Never overwrite an old accepted image or store account, token, session cookie, private dashboard, provider library, or browser-profile export. + +Capture the official reference and the matching Symphonia catalog/feature fixture in the same declared browser engine and geometry, with fonts loaded and transient loading finished. If the public demo cannot supply a dark theme, mark dark comparison unavailable; do not relabel a browser color-scheme override as an official dark capture. Use official documentation/source and interaction tests for behavior that screenshots cannot establish. + +## Review and change control + +1. The contributor records the manifest and captures, then compares hierarchy, density, geometry, type, borders, navigation, focus, status and actions at wide and phone sizes. They document each observed mismatch, not only pixel diffs. +2. A separate product/UX reviewer accepts parity or records an intentional divergence with reference, reason, user-visible consequence and fallback. An accessibility reviewer checks focus, keyboard, text scaling and contrast where a visual divergence affects interaction. +3. A baseline update requires a written reason: Home Assistant evolution, Symphonia defect correction, or intentional accessibility/product divergence. Regenerating snapshots never self-approves a change. +4. Before each release, revalidate the declared Home Assistant/browser matrix and current reference pointer. A new HA component or iframe contract triggers review of the affected families and the support matrix, not an automatic screenshot replacement. + +The current static [UI fixture](README.md) has no approved browser capture. Browser automation of its local `file:` URL was blocked in the current environment on 2026-09-27; this procedure must be executed in an authorized test environment before the SDD can advance. diff --git a/docs/open-questions.md b/docs/open-questions.md index 15cf369..3f9602d 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -132,17 +132,17 @@ Define what belongs in the App UI versus a companion custom integration. Decide ### RG-006 — Home Assistant UI compatibility and host-context spike -The owner has accepted the Home Assistant-native-adjacent direction, independent compatibility layer, and no-private-frontend-dependency boundary in [ADR 0004](decisions/0004-home-assistant-native-ui.md). Before the [UI foundation SDD](../specs/home-assistant-native-ui.md) becomes `Ready for implementation`, a documentation/prototype spike must: +The owner has accepted the Home Assistant-native-adjacent direction, independent compatibility layer, and no-private-frontend-dependency boundary in [ADR 0004](decisions/0004-home-assistant-native-ui.md), and selected Lit + TypeScript + Vite in [ADR 0005](decisions/0005-lit-typescript-vite-ui.md). [Current official-source research](development/ui-spike/host-context-evidence.md) finds no theme/locale/timezone fields in the inspected App-properties message. Before the [UI foundation SDD](../specs/home-assistant-native-ui.md) becomes `Ready for implementation`, a documentation/prototype spike must: - declare the supported Home Assistant and evergreen-browser matrix; - verify arbitrary Ingress base paths, deep links, refresh, assets, HTTP, and any WebSocket/SSE transport; - capture the exact supported public contract for theme, locale, direction, timezone, safe-area insets, and context changes, including origin/message/schema validation and deterministic fallback; -- compare candidate frontend/build approaches against component-package isolation, bundle/old-device cost, accessibility, localization, catalog, and standalone reuse; +- verify the selected Lit/Vite approach against component-package isolation, bundle/old-device cost, accessibility, localization, catalog, and standalone reuse; - create a dated official Home Assistant reference manifest with public-data provenance, light/dark availability, phone/wide viewports, human review ownership, and immutable history; - prototype representative navigation, card, button, form, status/alert, dialog, progress/empty, settings/list row, dense data, and ordered-evidence components without production feature behavior; and - prove keyboard/focus/announcement, reduced-motion, contrast, long/RTL text, safe-area/zoom/virtual-keyboard, no document overflow, and bounded table/diagnostic scrolling. -The result selects the presentation implementation/tooling and supported matrix. It does not authorize production UI work until the SDD is ready and the owner explicitly approves implementation. +The tooling choice is accepted; the result must still establish a supported matrix and validated host/visual contracts. It does not authorize production UI work until the SDD is ready and the owner explicitly approves implementation. ## Future synchronization questions @@ -161,7 +161,7 @@ These do not block the copy MVP but block sync implementation: ## Implementation decisions intentionally deferred - backend language and framework; -- UI framework/build tooling and supported HA/browser matrix; the Home Assistant-native design-system contract itself is accepted by ADR 0004; +- supported HA/browser matrix and Lit/Vite compatibility evidence; the UI stack is accepted by ADR 0005 and the design-system contract by ADR 0004; - SQLite versus PostgreSQL; - internal durable runner versus job library; - secret encryption primitive and key source; @@ -206,4 +206,4 @@ These need evidence and small RFCs; popularity is not evidence. After those tasks, revisit playlist ownership (`OQ-002`) before creating any persistent-synchronization SDD. The Home Assistant native surface (`RG-005`) can proceed in parallel once the service API shape is stable, but it is not a prerequisite for the copy MVP. -The UI compatibility spike (`RG-006`) can also proceed in parallel as non-production evidence. It must settle the supported host/browser/context/tooling matrix before any production frontend work, while feature content and behavior continue to be owned by their existing SDDs. +The UI compatibility spike (`RG-006`) can also proceed in parallel as non-production evidence. It must prove the supported host/browser/context matrix and selected Lit/Vite tooling before any production frontend work, while feature content and behavior continue to be owned by their existing SDDs. diff --git a/docs/product/home-assistant-ui-specification.md b/docs/product/home-assistant-ui-specification.md index e19c881..bf05429 100644 --- a/docs/product/home-assistant-ui-specification.md +++ b/docs/product/home-assistant-ui-specification.md @@ -11,7 +11,7 @@ The primary target is the authenticated Home Assistant Ingress panel. A standalo ## Decision and evidence status -The owner has accepted the requirement that Symphonia's UI and components be as close as practical to native Home Assistant. [ADR 0004](../decisions/0004-home-assistant-native-ui.md) records the architectural consequences. +The owner has accepted the requirement that Symphonia's UI and components be as close as practical to native Home Assistant. [ADR 0004](../decisions/0004-home-assistant-native-ui.md) records the architectural consequences, and [ADR 0005](../decisions/0005-lit-typescript-vite-ui.md) selects Lit, TypeScript, and Vite for the future presentation layer. [RG-006 evidence](../development/ui-spike/host-context-evidence.md) keeps public host-context behavior separate from unverified theme/locale assumptions. Evidence reviewed on 2026-09-20: @@ -22,7 +22,7 @@ Evidence reviewed on 2026-09-20: - official [2026.8 App/custom-panel guidance](https://developers.home-assistant.io/blog/2026/07/31/frontend-component-updates-2026.8/), including App-iframe safe-area propagation; - `vypdev/homeassistant-gateway` commit [`1ed75be`](https://github.com/vypdev/homeassistant-gateway/tree/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42), especially its [frontend direction](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/frontend-design.md), [design system](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/frontend-design-system.md), [UI catalog](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/frontend-ui-catalog.md), [testing strategy](https://github.com/vypdev/homeassistant-gateway/blob/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/frontend-testing-strategy.md), and dated [Home Assistant reference set](https://github.com/vypdev/homeassistant-gateway/tree/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42/docs/ui-reference/home-assistant). -The official sources define the target. The Gateway is community implementation evidence, not a permanent Home Assistant guarantee and not a dependency selection. +The official sources define the target. The Gateway is community implementation evidence, not a permanent Home Assistant guarantee; the later explicit owner decision in ADR 0005, not the Gateway itself, selects the stack. ## Terminology diff --git a/docs/providers/home-assistant-ecosystem-review.md b/docs/providers/home-assistant-ecosystem-review.md index 9d02ba2..091bf6e 100644 --- a/docs/providers/home-assistant-ecosystem-review.md +++ b/docs/providers/home-assistant-ecosystem-review.md @@ -18,7 +18,7 @@ The strongest precedent is Music Assistant: a separate service/App owns the musi 4. **Treat unofficial YouTube Music access as a distinct product mode.** Music Assistant and `ytube_music_player` demonstrate useful access through `ytmusicapi`, browser cookies, internal endpoints, and proof-of-origin tokens. They do not turn that surface into a supported Google API. An unofficial adapter would need explicit opt-in, health warnings, separate release gating, and no promise of symmetric copy/sync. 5. **Apple Music is a credible future official-library adapter.** Apple's official API documents library reads, catalog/library search, ISRC, playlist creation, and adding tracks. It does not document playlist-track removal, so new-playlist copy is more plausible than mirror sync. Its user-token acquisition and Home Assistant callback story still require a spike. 6. **Do not inherit playback-first shortcuts.** Symphonia must preserve unavailable entries, expose ambiguous matches, prove pagination completeness, and retain auditable user decisions even where an existing playback product can skip, merge, cap, or rescan data. -7. **Adopt the Gateway's independent Home Assistant-native UI pattern, not its stack by implication.** At the pinned commit reviewed, `vypdev/homeassistant-gateway` reproduces Home Assistant component families and operational density through its own tokens/primitives, catalog, visual references, and responsive/accessibility tests while avoiding private Home Assistant frontend imports. Symphonia has accepted that boundary in [ADR 0004](../decisions/0004-home-assistant-native-ui.md); framework selection remains open. +7. **Adopt the Gateway's independent Home Assistant-native UI pattern, not its stack by implication.** At the pinned commit reviewed, `vypdev/homeassistant-gateway` reproduces Home Assistant component families and operational density through its own tokens/primitives, catalog, visual references, and responsive/accessibility tests while avoiding private Home Assistant frontend imports. Symphonia accepted that boundary in [ADR 0004](../decisions/0004-home-assistant-native-ui.md); the owner later selected Lit/TypeScript/Vite explicitly in [ADR 0005](../decisions/0005-lit-typescript-vite-ui.md), not as an inference from this review. ## Projects reviewed @@ -151,7 +151,7 @@ The Gateway repository was inspected at commit [`1ed75be9f8fabdab386db0fe4320cfb ### Patterns to adapt - Symphonia needs richer item-level uncertainty, matching evidence, playlist order/duplicate displays, long-running durable operations, and provider risk disclosure than the Gateway. Its component catalog must therefore cover dense ordered lists, evidence comparisons, immutable-plan review, and partial/reconciliation states. -- The Gateway's exact Lit, Vite, Storybook, palette, radii, navigation, and CSS values are evidence, not accepted Symphonia dependencies. The selected implementation must satisfy the [UI specification](../product/home-assistant-ui-specification.md) and [UI foundation SDD](../../specs/home-assistant-native-ui.md). +- The Gateway's exact dependency versions, Storybook setup, palette, radii, navigation, and CSS values are evidence, not accepted Symphonia dependencies. Lit/TypeScript/Vite was accepted later by an independent owner decision ([ADR 0005](../decisions/0005-lit-typescript-vite-ui.md)); its implementation must still satisfy the [UI specification](../product/home-assistant-ui-specification.md) and [UI foundation SDD](../../specs/home-assistant-native-ui.md). - Home Assistant 2026.8 documents safe-area propagation for App iframes; Symphonia must validate the supported version range and context/origin/schema instead of assuming the latest behavior everywhere. - The Gateway uses fixed public-demo reference captures. Symphonia should keep an immutable manifest/history so one newly captured Home Assistant release does not erase why an older supported release differs. @@ -210,4 +210,4 @@ This review does not accept a dependency or new provider into the MVP. It narrow 5. Add an Apple Music test-account spike as a future-provider candidate, focusing on Music User Token acquisition, catalog/library IDs, `canEdit`, playlist append, and absence of remove/reorder. 6. Require completeness markers for every import/list operation and preserve unavailable entries. 7. Threat-model every directly exposed App listener and prevent provider-controlled values from acquiring filesystem or executable semantics. -8. Treat the [UI specification](../product/home-assistant-ui-specification.md) and [UI foundation SDD](../../specs/home-assistant-native-ui.md) as the shared presentation contract; complete `RG-006` before selecting or implementing the frontend stack. +8. Treat the [UI specification](../product/home-assistant-ui-specification.md) and [UI foundation SDD](../../specs/home-assistant-native-ui.md) as the shared presentation contract; complete `RG-006` proof of the selected stack before production frontend implementation. diff --git a/specs/CATALOG.md b/specs/CATALOG.md index 7cba0b6..c355e8b 100644 --- a/specs/CATALOG.md +++ b/specs/CATALOG.md @@ -7,7 +7,7 @@ This is the human-readable view of [`catalog.json`](./catalog.json). Until gener | Capability ID | Status | Primary SDD | Readiness summary | | --- | --- | --- | --- | | `home-assistant-app-runtime` | Draft | [Home Assistant App runtime and Ingress](home-assistant-app-runtime-and-ingress.md) | Blocked by storage/recovery, supported platform matrix, and secret-key design | -| `home-assistant-native-ui` | Ready for review | [Home Assistant-native UI foundation](home-assistant-native-ui.md) | Product direction is accepted; blocked from implementation readiness by the supported matrix, frontend/build choice, public host-context validation, and visual-reference procedure | +| `home-assistant-native-ui` | Ready for review | [Home Assistant-native UI foundation](home-assistant-native-ui.md) | Lit/TypeScript/Vite is accepted; blocked by supported matrix, public host-context/fallback proof, build/browser evidence, and visual-reference review | | `provider-connections-and-authorization` | Draft | [Provider connections and authorization](provider-connections-and-authorization.md) | Blocked by direct callback reachability, secret/backup design, and provider feasibility spikes | | `library-import-and-provider-projections` | Draft | [Library import and provider projections](library-import-and-provider-projections.md) | Blocked by provider completeness, retention, and representative scale | | `recording-identity-resolution` | Draft | [Recording identity resolution](recording-identity-resolution.md) | Blocked by the labeled corpus and accepted automatic-link policy | diff --git a/specs/README.md b/specs/README.md index cf1935c..731fdca 100644 --- a/specs/README.md +++ b/specs/README.md @@ -127,7 +127,10 @@ PYTHONPATH=. python3 tools/validate_specs.py The validator checks that `catalog.json` parses, referenced files exist, each SDD has one known catalog capability ID, catalog statuses and paths agree with `CATALOG.md`, requirement IDs are present in the horizontal specifications, -and local Markdown links resolve. +and local Markdown links resolve. Generated/vendor directories such as +`node_modules`, `dist`, and virtual environments are excluded from Markdown +scanning; owned documentation is still checked even when dependencies are +installed locally. Also verify: diff --git a/specs/catalog.json b/specs/catalog.json index a75824b..c4a5eb7 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -119,11 +119,15 @@ ], "evidence": [ "docs/decisions/0004-home-assistant-native-ui.md", + "docs/decisions/0005-lit-typescript-vite-ui.md", "docs/product/home-assistant-ui-specification.md", "docs/architecture/system-architecture.md", "docs/providers/provider-research.md", "docs/providers/home-assistant-ecosystem-review.md", - "docs/development/ui-spike/README.md" + "docs/development/ui-spike/README.md", + "docs/development/ui-spike/host-context-evidence.md", + "docs/development/ui-spike/visual-reference-plan.md", + "docs/development/ui-lit-spike/README.md" ], "implementationEvidence": { "code": [], @@ -132,9 +136,9 @@ }, "blockers": [ "Supported Home Assistant and browser matrix", - "Frontend framework and build-tool selection", - "RG-006 verified theme, locale, direction, safe-area, and Ingress context contract", - "Accepted dated visual-reference capture and update procedure" + "RG-006 verified public host context and deterministic theme, locale, direction, timezone, and safe-area fallbacks", + "Lit/Vite Ingress, bundle, accessibility, and browser compatibility evidence", + "Accepted dated visual-reference capture/update procedure and reviewed visuals" ] }, { diff --git a/specs/home-assistant-native-ui.md b/specs/home-assistant-native-ui.md index a3d5771..d7a810c 100644 --- a/specs/home-assistant-native-ui.md +++ b/specs/home-assistant-native-ui.md @@ -6,9 +6,9 @@ - Owners: Symphonia maintainers - Scope: establish the shared shell, component families, semantic tokens, host-context adaptation, accessibility, responsive behavior, component catalog, and visual compatibility evidence for every Symphonia web view. - Related requirements: `SYM-UI-001`–`SYM-UI-015`, `SYM-JOB-007`, `SYM-SEC-004`, `SYM-SEC-008`, `SYM-TEST-005`, `SYM-TEST-009`, `SYM-TEST-014`, `SYM-TEST-015` -- Related decisions/research: [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [UI specification](../docs/product/home-assistant-ui-specification.md), [official platform research](../docs/providers/provider-research.md#home-assistant-platform), [Gateway/HA UI evidence](../docs/providers/home-assistant-ecosystem-review.md#home-assistant-native-ui-lessons-from-homeassistant-gateway) +- Related decisions/research: [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [ADR 0005](../docs/decisions/0005-lit-typescript-vite-ui.md), [UI specification](../docs/product/home-assistant-ui-specification.md), [official platform research](../docs/providers/provider-research.md#home-assistant-platform), [Gateway/HA UI evidence](../docs/providers/home-assistant-ecosystem-review.md#home-assistant-native-ui-lessons-from-homeassistant-gateway), [RG-006 host-context evidence](../docs/development/ui-spike/host-context-evidence.md) - Required review gates: product UX, Home Assistant platform, frontend architecture, accessibility, localization, testing/visual QA, documentation, security/privacy -- Open decisions blocking implementation readiness: supported Home Assistant/browser matrix; frontend framework/build selection; verified theme/locale/direction/safe-area context contract across supported Home Assistant versions; accepted visual-reference capture/update procedure +- Open decisions blocking implementation readiness: supported Home Assistant/browser matrix; verified public host-context/Ingress contract and deterministic fallbacks across that matrix; Lit/Vite compatibility and bundle evidence; accepted visual-reference capture/update procedure and visual/accessibility review ## 1. Executive summary @@ -32,7 +32,7 @@ Without a shared UI contract, each feature can implement different cards, button ### 2.2 Current behavior -No complete Symphonia UI or UI package exists. The repository has an experimental Ingress metadata scaffold, prospective feature SDDs, and an isolated [synthetic UI review fixture](../docs/development/ui-spike/README.md). The fixture is not served by the App, does not connect to application state, and has not passed browser visual/accessibility review. This SDD does not select or authorize a frontend framework or production implementation. +No complete Symphonia UI or UI package exists. The repository has an experimental Ingress metadata scaffold, prospective feature SDDs, an isolated [synthetic UI review fixture](../docs/development/ui-spike/README.md), and a separate [Lit build spike](../docs/development/ui-lit-spike/README.md). Neither is served by the App or connected to application state; neither has passed browser visual/accessibility review. [ADR 0005](../docs/decisions/0005-lit-typescript-vite-ui.md) selects Lit, TypeScript, and Vite for future implementation; this SDD does not authorize production UI yet. ### 2.3 Evidence and unknowns @@ -40,7 +40,10 @@ No complete Symphonia UI or UI package exists. The repository has an experimenta - Official evidence: Home Assistant design portal, frontend architecture/source, independent-component warning, current App/Ingress and safe-area contracts linked from the UI specification and platform research. - Community evidence: `vypdev/homeassistant-gateway` commit `1ed75be` demonstrates a presentation-only compatibility layer, HA-like component families, component catalog, official-demo reference captures, accessibility/responsive tests, and visual baselines. - Symphonia review evidence: the [isolated fixture](../docs/development/ui-spike/README.md) exercises a proposed App-interior shell and synthetic connection, library, copy, and operation states. Its structural tests are not production component, Ingress, visual, or WCAG evidence. -- Unknowns: exact supported Home Assistant/browser versions; selected frontend/build tools; the public context actually available to an Ingress App at each supported version; long-term token mapping; reference capture automation and review ownership. +- The [isolated Lit build spike](../docs/development/ui-lit-spike/README.md) proves a small typed/presentation-only build, relative output asset, and pure browser fallback tests. It does not prove full bundle cost, Ingress, real host context, or browser/accessibility behavior. +- The [visual-reference procedure](../docs/development/ui-spike/visual-reference-plan.md) is drafted but has no accepted Symphonia baseline or reviewer approval. +- Current [official-source review](../docs/development/ui-spike/host-context-evidence.md) finds `narrow`, `route`, and `safeAreaInsets` in the App properties message, but no theme, locale, direction, or timezone field. These cannot be assumed to follow Home Assistant user settings inside an iframe. +- Unknowns: exact supported Home Assistant/browser versions; the public context actually available to an Ingress App at each supported version; Lit/Vite compatibility and bundle cost; long-term token mapping; reference capture automation and review ownership. ## 3. Actors, surfaces, and terminology @@ -112,7 +115,7 @@ Text equivalent: validated public host context or deterministic fallback feeds s ### 6.1 Happy path 1. The shell resolves base path and renders readable fallback tokens without blocking on Home Assistant context. -2. A context adapter accepts only the documented host message/origin/shape and maps supported theme, locale, direction, timezone, and safe-area facts. +2. A context adapter accepts only documented, validated public host fields. Current App-properties research establishes narrow/route/safe-area hints, not theme/locale/direction/timezone; unsupported fields use deterministic browser/standalone fallbacks. 3. Navigation loads a feature route and preserves history under the Ingress prefix. 4. The feature requests application state; loading is textual and non-destructive. 5. The view composes only approved component families and renders the durable state plus available actions. @@ -146,8 +149,8 @@ Feature states such as partial, waiting-user, failed, and completed remain owned | Input | Type | Recommended default | Allowed values/range | Scope/persistence | | --- | --- | --- | --- | --- | -| Theme mode | enum | follow validated Home Assistant context, then browser preference | `host/auto`, `light`, `dark` only if product permits override | user/browser preference; never domain state | -| Locale | BCP 47 tag | validated HA locale, browser locale, then English | packaged supported locales with base-language fallback | user/browser; server retains typed values | +| Theme mode | enum | validated public HA theme context **if proven**, otherwise browser preference | `host/auto`, `light`, `dark` only if product permits override | user/browser preference; never domain state | +| Locale | BCP 47 tag | validated public HA locale **if proven**, browser locale, then English | packaged supported locales with base-language fallback | user/browser; server retains typed values | | Text direction | enum | derived from locale/public host context | `ltr`, `rtl` | derived, not arbitrary per component | | Safe-area handling | enum | supported host insets with zero fallback | host-managed or App-managed accepted profile | deployment/profile | | Visual reference manifest | versioned document | latest reviewed supported HA release/reference date | immutable historical entries plus current pointer | repository/release evidence | @@ -254,7 +257,7 @@ Primary action: Review result ## 11. Security, permissions, and privacy 1. Ingress identity is trusted only at the verified server boundary; theme/locale messages do not grant authorization. -2. Parent-window messaging validates origin, type, correlation, and bounded schema; no wildcard data is executed or persisted blindly. +2. Parent-window messaging validates source, origin, type, and bounded schema; correlation is required only if the supported public protocol supplies one. The inspected App-properties message supplies no correlation field, so stale updates require local fencing rather than a fictional host token. No wildcard data is executed or persisted blindly. 3. Provider/user/diagnostic strings cannot become markup, CSS, script, unsafe URLs, route destinations, or component definitions. 4. UI fixtures, screenshots, clipboard, downloads, browser logs, and error boundaries exclude secrets and real private account/library data. 5. Destructive actions use explicit consequences and confirmation; visual similarity cannot weaken server authorization or idempotency. @@ -335,7 +338,7 @@ Required human evidence reviews official Home Assistant references versus the ca ## 18. Implementation sequence -1. Accept the supported Home Assistant/browser matrix, public context contract, visual-reference procedure, and frontend/build decision. +1. Verify the supported Home Assistant/browser matrix, public context contract and fallbacks, Lit/Vite build/compatibility evidence, and visual-reference procedure. The frontend/build choice is recorded in ADR 0005; its compatibility evidence remains open. 2. Establish reference manifest, semantic tokens, fallback context, package boundary, architecture/lint checks, and catalog harness. 3. Add foundational components: headings/toolbars, navigation/tabs, cards/sections, buttons/icon buttons, fields/choices, status/alerts, loading/empty, and layouts. 4. Add dialogs, settings/list rows, responsive data displays, safe-area/base-path shell, localization/direction, and full interaction/accessibility tests. @@ -345,7 +348,7 @@ Required human evidence reviews official Home Assistant references versus the ca ## 19. Definition of Done - [ ] Status is `Ready for implementation` and explicit owner approval to implement exists. -- [ ] Supported HA/browser matrix, frontend/build, context/safe-area, and reference procedures are accepted. +- [ ] Supported HA/browser matrix, Lit/Vite compatibility/build evidence, context/safe-area/fallbacks, and reference procedures are accepted. - [ ] Every `SYM-UI-*` requirement maps to acceptance and deterministic or explicit human evidence. - [ ] At least 84 distinct cases and architecture/lint/secret gates pass. - [ ] Every required component family is available only through the public package and complete in the catalog. @@ -360,6 +363,6 @@ Required human evidence reviews official Home Assistant references versus the ca - Primary sources: official Home Assistant links in the [UI specification](../docs/product/home-assistant-ui-specification.md) and [platform research](../docs/providers/provider-research.md#home-assistant-platform). - Community evidence: pinned `vypdev/homeassistant-gateway` sources listed in the UI specification and ecosystem review. - Related SDDs: [App runtime/Ingress](home-assistant-app-runtime-and-ingress.md), [provider connections](provider-connections-and-authorization.md), [imports](library-import-and-provider-projections.md), [identity resolution](recording-identity-resolution.md), [playlist copy](one-time-playlist-copy.md), [durable operations](durable-operations-and-recovery.md). -- Accepted: Home Assistant-native-adjacent direction, owned compatibility layer, no private HA frontend runtime dependency, shared Ingress/standalone semantics, dated catalog/reference evidence. -- Open: framework/build stack, supported matrix, public App context details, reference capture/update automation. +- Accepted: Home Assistant-native-adjacent direction, owned compatibility layer, no private HA frontend runtime dependency, shared Ingress/standalone semantics, Lit + TypeScript + Vite direction (ADR 0005), dated catalog/reference evidence. +- Open: supported matrix, verified public App context and fallbacks, Lit/Vite bundle/Ingress evidence, reference capture/update automation and visual review. - Rejected: generic SaaS dashboard, private HA component imports by default, frozen pixel copy, decorative ambient UI, feature-local duplicate primitives. diff --git a/tests/test_spec_validator.py b/tests/test_spec_validator.py index c313d1f..2162446 100644 --- a/tests/test_spec_validator.py +++ b/tests/test_spec_validator.py @@ -1,7 +1,8 @@ from pathlib import Path +from tempfile import TemporaryDirectory import unittest -from tools.validate_specs import validate +from tools.validate_specs import _validate_markdown_links, _validate_requirements, validate class SpecValidatorTests(unittest.TestCase): @@ -9,6 +10,24 @@ def test_repository_specifications_are_consistent(self) -> None: root = Path(__file__).resolve().parents[1] self.assertEqual(validate(root), []) + def test_generated_dependency_markdown_is_not_treated_as_ours(self) -> None: + with TemporaryDirectory() as directory: + root = Path(directory) + docs = root / "docs" + docs.mkdir() + (docs / "guide.md").write_text("[missing](missing.md)", encoding="utf-8") + vendor = docs / "node_modules" / "package" + vendor.mkdir(parents=True) + (vendor / "README.md").write_text("[also missing](elsewhere.md) SYM-UI-999", encoding="utf-8") + + errors: list[str] = [] + _validate_markdown_links(root, errors) + self.assertEqual(errors, ["docs/guide.md links to missing path: missing.md"]) + + errors = [] + _validate_requirements(root, [{"id": "ui", "requirements": ["SYM-UI-999"]}], errors) + self.assertEqual(errors, ["ui references unknown requirement SYM-UI-999"]) + if __name__ == "__main__": unittest.main() diff --git a/tools/validate_specs.py b/tools/validate_specs.py index 05f9a63..3b3660f 100644 --- a/tools/validate_specs.py +++ b/tools/validate_specs.py @@ -10,10 +10,11 @@ import argparse import json +import os import re import sys from pathlib import Path -from typing import Any +from typing import Any, Iterator CAPABILITY_ID_RE = re.compile(r"^\s*-\s+Catalog capability ID:\s+`([^`]+)`\s*$", re.MULTILINE) @@ -32,6 +33,26 @@ "superseded": "Superseded", } EXCLUDED_SPEC_MARKDOWN = {"README.md", "CATALOG.md", "_template.md"} +GENERATED_MARKDOWN_DIRS = { + ".git", + ".repowise", + ".venv", + "__pycache__", + "build", + "dist", + "graphify-out", + "node_modules", +} + + +def _repository_markdown(root: Path) -> Iterator[Path]: + """Yield owned Markdown, excluding generated/vendor trees at any depth.""" + + for directory, children, files in os.walk(root): + children[:] = sorted(child for child in children if child not in GENERATED_MARKDOWN_DIRS) + for name in sorted(files): + if name.endswith(".md"): + yield Path(directory) / name def _add(errors: list[str], message: str) -> None: @@ -161,7 +182,7 @@ def _validate_sdds(root: Path, capabilities: list[dict[str, Any]], errors: list[ def _validate_requirements(root: Path, capabilities: list[dict[str, Any]], errors: list[str]) -> None: known: set[str] = set() - for path in (root / "docs").rglob("*.md"): + for path in _repository_markdown(root / "docs"): known.update(REQUIREMENT_RE.findall(path.read_text(encoding="utf-8"))) for capability in capabilities: for requirement in capability.get("requirements", []): @@ -189,9 +210,7 @@ def _validate_catalog_markdown(root: Path, capabilities: list[dict[str, Any]], e def _validate_markdown_links(root: Path, errors: list[str]) -> None: - for path in sorted(root.rglob("*.md")): - if ".git" in path.parts: - continue + for path in _repository_markdown(root): text = path.read_text(encoding="utf-8") for match in MARKDOWN_LINK_RE.finditer(text): target = match.group(1) or match.group(2) From b20a8e9ae34bdac87064a166604f38be64fbbe4b Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 28 Sep 2026 09:33:08 +0200 Subject: [PATCH 163/167] docs: specify listening capability and narrow HA playback broker --- docs/README.md | 4 +- docs/architecture/system-architecture.md | 48 ++- ...6-narrow-home-assistant-playback-broker.md | 35 ++ docs/decisions/README.md | 1 + docs/development/quality-audit.md | 8 + docs/domain/domain-model.md | 11 +- docs/open-questions.md | 16 +- .../home-assistant-ui-specification.md | 6 +- docs/product/product-specification.md | 41 +- .../home-assistant-ecosystem-review.md | 16 +- .../playback-integration-source-review.md | 90 +++++ docs/providers/provider-research.md | 16 +- docs/providers/provider-specification.md | 29 +- specs/CATALOG.md | 1 + specs/README.md | 2 + specs/catalog.json | 54 +++ .../home-assistant-app-runtime-and-ingress.md | 21 +- specs/home-assistant-native-ui.md | 27 +- specs/listening-and-playback-control.md | 372 ++++++++++++++++++ .../provider-connections-and-authorization.md | 12 +- 20 files changed, 757 insertions(+), 53 deletions(-) create mode 100644 docs/decisions/0006-narrow-home-assistant-playback-broker.md create mode 100644 docs/providers/playback-integration-source-review.md create mode 100644 specs/listening-and-playback-control.md diff --git a/docs/README.md b/docs/README.md index d3f067c..f07dc71 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,7 +1,7 @@ # Documentation map **Status:** baseline for review -**Last reviewed:** 2026-09-20 +**Last reviewed:** 2026-09-28 This documentation is the horizontal implementation contract for Symphonia. It deliberately separates product intent, domain rules, architecture, provider facts, accepted decisions, and unresolved choices. The vertical, capability-level contracts live in the [SDD catalog](../specs/CATALOG.md) and select from these shared rules without overriding them. @@ -17,6 +17,7 @@ This documentation is the horizontal implementation contract for Symphonia. It d | [Provider specification](providers/provider-specification.md) | Provider port, capabilities, normalized errors | Claims about a specific API | | [Provider research](providers/provider-research.md) | Dated, sourced facts about provider APIs | Product policy or permanent architecture | | [Home Assistant music ecosystem review](providers/home-assistant-ecosystem-review.md) | Reusable patterns and cautions from existing HA music projects | Dependency selection or provider guarantees | +| [Playback integration source review](providers/playback-integration-source-review.md) | Dated source-code findings, operation limits, risk and live proof matrix for HA Spotify, Music Assistant, and YT Music projects | Official API guarantees, release support, or SDD readiness | | [Development specification](development/development-specification.md) | Specification workflow, testing and delivery gates | Product scope | | [Local quality audit](development/quality-audit.md) | Reproducible offline RepoWise/Graphify review and current architecture/test debt | A release or SDD readiness claim | | [ADRs](decisions/README.md) | Decisions that have actually been accepted | Proposals and guesses | @@ -34,6 +35,7 @@ This documentation is the horizontal implementation contract for Symphonia. It d | `SYM-LIB` | Unified library | | `SYM-MATCH` | Identity resolution | | `SYM-PL` | Playlist copy | +| `SYM-PLAY` | Listening and playback control | | `SYM-SYNC` | Persistent synchronization | | `SYM-PROV` | Provider abstraction | | `SYM-ARCH` | Architecture and persistence | diff --git a/docs/architecture/system-architecture.md b/docs/architecture/system-architecture.md index b547174..78a173f 100644 --- a/docs/architecture/system-architecture.md +++ b/docs/architecture/system-architecture.md @@ -1,7 +1,7 @@ # System architecture **Status:** Home Assistant-first deployment and UI stack direction accepted; logical boundaries proposed; other technology choices open -**Last reviewed:** 2026-09-27 +**Last reviewed:** 2026-09-28 ## Architectural drivers @@ -12,6 +12,7 @@ The architecture is derived from these needs: - a management UI that follows current Home Assistant component, theme, responsive, and interaction patterns without depending on private frontend internals; - a core that can run and be tested without Home Assistant; - provider-independent domain and replaceable adapters; +- a listening surface that observes and controls explicitly selected external players without decoding or relaying audio; - durable imports and provider writes that survive restarts; - explicit uncertainty, partial success, rate limiting, and user action; - secure storage and rotation of OAuth credentials; @@ -48,6 +49,17 @@ These drivers do not yet justify a backend framework or database product. The UI Provider APIs, the browser, Home Assistant/Supervisor, persistent storage, and future external API clients are separate trust boundaries. Network proximity is not authentication. +The diagram's provider arrows describe library/copy adapters, not automatic playback access. [ADR 0006](../decisions/0006-narrow-home-assistant-playback-broker.md) chooses a narrow companion integration for Home Assistant listening instead of granting the App broad Core API authority. An imported provider account and an HA playback entity may be different accounts. See the [listening SDD](../../specs/listening-and-playback-control.md) for the proposed binding and state contract. + +```text +Ingress browser -> Symphonia listening API -> playback port + <-> authenticated narrow companion broker + -> one allowed HA media_player -> output +Library provider connection -- optional explicit, evidenced binding --^ +``` + +Text equivalent: the browser requests a typed listening use case; the App's playback adapter communicates with an authenticated, music-scoped companion broker, which alone accesses the selected HA player. A library connection may be explicitly associated for compatible media references but never supplies broker/Core credentials or chooses the output by itself. + ## Dependency rule Dependencies point inward: @@ -58,9 +70,9 @@ infrastructure ────────────────→ ports defined composition ──→ concrete implementations ``` -- **Domain** owns recording identity, playlist semantics, capabilities, plans, policies, and operation states. It imports no provider SDK, Home Assistant package, web framework, database driver, filesystem/config reader, or job framework. +- **Domain** owns recording identity, playlist semantics, capabilities, plans, policies, and operation states. Pure listening policy owns target identity, command eligibility, and observation freshness without modeling audio streams. It imports no provider SDK, Home Assistant package, web framework, database driver, filesystem/config reader, or job framework. - **Application** orchestrates use cases through ports and owns transaction/idempotency boundaries. It depends on domain types, not concrete adapters. -- **Infrastructure** implements provider, persistence, secret, clock, queue, and Home Assistant ports. +- **Infrastructure** implements provider, playback-source/player, persistence, secret, clock, queue, and Home Assistant ports. - **Presentation** maps Ingress/HTTP/UI requests and responses. It does not decide match, authorization, retry, or conflict policy. - **Composition** selects the deployment profile and wires concrete implementations. It is the only layer that knows the full runtime graph. @@ -70,9 +82,9 @@ This follows the useful boundary pattern in `homeassistant-gateway` without carr | Component | Responsibility | Explicit exclusions | | --- | --- | --- | -| Web UI | Home Assistant-native-adjacent shell and component compatibility layer; connection setup, library/playlist views, resolution queue, copy preview, history, diagnostics | Provider tokens, matching policy, direct provider calls, private HA frontend modules | +| Web UI | Home Assistant-native-adjacent shell and component compatibility layer; listening/now-playing, connection setup, library/playlist views, resolution queue, copy preview, history, diagnostics | Provider tokens, matching policy, direct provider or Home Assistant calls, private HA frontend modules | | HTTP/API presentation | Authenticated input/output mapping, validation shape, request correlation | Domain decisions and raw exception exposure | -| Application use cases | Connect/disconnect, import, resolve, plan copy, execute copy, inspect operations | Provider-specific response types | +| Application use cases | Connect/disconnect, import, resolve, plan copy, execute copy, inspect operations, list playback targets, observe session, issue one bounded command | Provider-specific or Home Assistant response types | | Domain | Provider-independent entities, invariants, capability requirements, matching/copy/sync policy | IO and scheduling | | Provider adapter host | OAuth/token refresh, pagination, normalized reads/writes/search, error translation | Cross-provider orchestration | | Resolution engine | Candidate generation/evaluation and evidence production | Unversioned opaque decisions | @@ -81,6 +93,8 @@ This follows the useful boundary pattern in `homeassistant-gateway` without carr | Secret store adapter | Encrypt/decrypt credential material and rotate key references | Returning plaintext to UI/logs | | Observability | Structured logs, metrics, health/readiness, sanitized diagnostics | Provider payload dumping | | Home Assistant adapter | Ingress identity and future native integration contract | Owning music domain rules | +| Playback adapter | Normalize broker-reported observations/capabilities and dispatch typed listening requests for an explicit player | Import/write credentials, Core API credentials, arbitrary HA service calls, audio processing | +| Companion playback broker | Within HA, discover/observe allowed entities and gate/dispatch typed commands to one exact entity | Symphonia provider grants/database, music-domain policy, generic Core proxy | ## Presentation and Home Assistant-native UI boundary @@ -113,6 +127,7 @@ This is an accepted product/deployment decision, recorded in [ADR 0003](../decis - The App starts as an `application`, stores durable state under `/data`, exposes its UI through Ingress, and participates in Supervisor backup/update lifecycle. - Ingress is the administrative UI authentication boundary. The server MUST honor the Ingress base path and MUST NOT assume it is hosted at `/`. - App permissions, mapped folders, network exposure, and Supervisor/Core API access MUST be least-privilege. Music-provider access alone does not justify Home Assistant API or host filesystem access. +- For HA listening, [ADR 0006](../decisions/0006-narrow-home-assistant-playback-broker.md) selects a narrow companion broker. The App must not enable `homeassistant_api: true` or use `SUPERVISOR_TOKEN` to call Core for this feature. Broker pairing/authentication, versioning, caller identity, and transport remain SDD gates; network proximity and Ingress alone do not authenticate broker traffic. Library/copy remains available without the broker. - A published image MUST support the explicitly documented Home Assistant architectures; the initial architecture set is open. - Provider secrets MUST NOT be placed in App options, image layers, logs, diagnostics, or ordinary backups in plaintext. @@ -126,17 +141,17 @@ The standalone profile has no Ingress. It therefore requires an explicit authent Whether standalone packaging ships in the first public release or immediately afterward is open. -### Optional companion Home Assistant integration +### Companion Home Assistant integration -A companion custom integration MAY later expose native entities, actions, events, and configuration discovery. It must call a stable, authenticated Symphonia API and MUST NOT duplicate matching, sync, credential, or retry logic. +A companion custom integration is required for the selected HA playback path under [ADR 0006](../decisions/0006-narrow-home-assistant-playback-broker.md); it MAY later expose other native entities, actions, events, and configuration discovery. It must use a stable, authenticated, versioned Symphonia contract and MUST NOT duplicate matching, sync, provider-credential, or durable-write retry logic. The exact transport and pairing protocol remain unselected. The MVP does not require the companion integration to broker provider authorization. Provider authorization is owned by the App's adapters following -the App-plus-Ingress pattern. A future proposal MAY evaluate a narrow broker -to reuse Home Assistant's Application Credentials/config-flow machinery; if -selected, it must exchange an opaque, one-use connection grant over the -authenticated local API, and token ownership, refresh, revocation, -backup/recovery, and version skew must be specified before acceptance. +the App-plus-Ingress pattern. The playback broker decision does not make that +integration an OAuth broker. A future proposal MAY evaluate reusing Home +Assistant's Application Credentials/config-flow machinery; if selected, it +must separately specify token ownership, refresh, revocation, backup/recovery, +and version skew before acceptance. Candidate native surface (illustrative, not accepted): @@ -144,7 +159,7 @@ Candidate native surface (illustrative, not accepted): - actions such as `symphonia.copy_playlist` and future `symphonia.sync`; - events for operation completed, partial, failed, or user action required. -Three approaches require an RFC: +Three approaches were compared for native surfaces; ADR 0006 selects the companion for listening, while the exact transport/native projections still need an RFC: | Approach | Benefits | Costs | | --- | --- | --- | @@ -152,8 +167,7 @@ Three approaches require an RFC: | MQTT discovery/events | Mature decoupling and push model | Adds an MQTT dependency and weakens direct operation correlation | | App calls Home Assistant APIs directly | Fewer artifacts for events/actions initiated by the App | Couples the service to Home Assistant and does not cleanly provide a native integration surface | -The App-plus-Ingress approach is the accepted primary direction. The -companion-integration approach is deferred to a future native-surface RFC. +The App-plus-Ingress approach remains the accepted primary product direction. The companion's playback responsibility is selected; native projections beyond listening remain deferred to a future native-surface RFC. ## Service/API shape @@ -162,6 +176,7 @@ The backend needs an authenticated HTTP API for its own web UI and future integr Required conceptual endpoints/use cases include: - connection list, capability probe, authorization start/callback, refresh, and disconnect; +- configured playback source/target list, explicit binding status, now-playing observation, browse/select supported media, and bounded command/reconciliation status; - import start/status and collection reads; - unresolved queue, candidate evidence, accept/reject/defer/revoke; - copy plan, plan read, accept/execute, run status, reconcile/cancel where safe; @@ -299,6 +314,9 @@ The MVP may render metrics in its UI and logs; choosing Prometheus/OpenTelemetry - **SYM-HA-007:** Native entities/actions/events MUST expose bounded summaries or identifiers, not playlist contents, tokens, or raw provider errors in state attributes. - **SYM-HA-008:** Home Assistant automations that trigger mutations MUST create ordinary audited Symphonia operations subject to the same validation, idempotency, and policy as UI requests. - **SYM-HA-009:** The MVP App panel MUST be admin-only; a future multi-user model MUST define which Home Assistant identity owns provider connections and may approve writes. +- **SYM-HA-010:** Any App-to-Core playback integration MUST be explicitly enabled and threat-reviewed; Core credentials remain server-side, and the adapter MUST allowlist player entities, read methods, and media-player actions rather than exposing a generic Home Assistant API relay to the UI. +- **SYM-HA-011:** The Home Assistant listening path MUST use a narrow, authenticated, versioned companion integration that alone accesses Core playback entities. The App MUST NOT enable broad Supervisor-to-Core API access for listening; the broker MUST restrict each read/command to an exact allowed entity and typed operation, and its absence or incompatibility MUST disable only listening without replaying commands. +- **SYM-ARCH-016:** Playback observation/commands MUST remain separate from imported library snapshots and durable playlist-write execution; observing or commanding an external player MUST NOT silently mutate canonical recording/playlist facts or enter copy-job retry policy. ## Technology decisions still open diff --git a/docs/decisions/0006-narrow-home-assistant-playback-broker.md b/docs/decisions/0006-narrow-home-assistant-playback-broker.md new file mode 100644 index 0000000..b03e2c8 --- /dev/null +++ b/docs/decisions/0006-narrow-home-assistant-playback-broker.md @@ -0,0 +1,35 @@ +# ADR 0006: Use a narrow Home Assistant companion broker for playback + +- **Status:** accepted +- **Date:** 2026-09-28 +- **Scope:** Home Assistant playback observation and control; not provider OAuth or the independent music core + +## Context + +The owner wants listening and playback controls inside Symphonia's Home Assistant App. [ADR 0003](0003-home-assistant-app-primary.md) makes the App the owner of durable music data and Ingress UI while leaving a companion integration possible. The [playback source review](../providers/playback-integration-source-review.md) shows that HA's Spotify and Music Assistant entities have integration-specific feature, identity, output, queue, and caller-context limits. + +The Supervisor Core API proxy would let the App call Home Assistant, but `homeassistant_api: true` plus a server-side Supervisor token is broader authority than a music feature needs. App-side allowlisting would reduce accidental use but would not narrow that credential's underlying Core permissions. A broker can expose only typed playback operations and keep Core access inside an HA custom integration. + +## Decision + +For Home Assistant playback, build a **narrow companion custom integration** as the broker between the Symphonia App and explicitly allowed existing `media_player` entities. The App remains the product/UI, provider-authorization, library/copy, and listening-policy owner. The broker owns HA entity discovery/observation and dispatch of a small, versioned set of typed media-player operations. It must not own Symphonia's database, provider grants, matching/copy policy, or audio streaming. + +The listening feature will **not** enable `homeassistant_api: true` or give the App a general Core API credential as its initial playback path. The browser never receives a Core or broker credential. Only the installed broker may invoke Home Assistant services, and it must enforce an exact entity allowlist, operation/media-kind allowlist, current feature/state checks, and safe caller/account attribution. It may deny a request even when HA advertises a feature. + +The broker transport, mutual authentication/pairing, secret rotation/revocation, version negotiation, reconnect behavior, user attribution, state freshness, command correlation, deployment discovery, and failure semantics are **not selected by this ADR**. They are release-blocking contracts in the [listening SDD](../../specs/listening-and-playback-control.md). No generic HA service call, arbitrary entity selector, URL, local path, or title-search request may cross the broker boundary. An accepted broker command is not proof of playback effect. + +The companion integration is required **for HA listening**, but remains optional for App-only library/import/copy use. This does not change ADR 0003's decision that provider OAuth belongs to App adapters in the current plan. A later use of HA Application Credentials would need a separate decision. + +## Alternatives considered + +- **App-to-Core Supervisor proxy:** fewer artifacts but grants broad Core authority to the App. Not selected for listening. +- **Direct Spotify/MA playback APIs in the App:** may solve some device-ID limits but adds separate grants, account/policy obligations, and provider-specific session behavior; not selected as a silent fallback. +- **MQTT/general event bus:** useful for some projections, but introduces another service and does not by itself define authentication, exactly-one target, command reconciliation, or caller attribution. + +## Consequences and verification + +- App and broker are versioned separately; missing/incompatible broker degrades only listening, not library/copy. +- The broker needs a minimal authenticated local contract and a threat review covering pairing/replay, entity authorization, upgrade/rollback, Ingress isolation, logs, and backup/restore. +- Each supported integration needs a versioned operation matrix. Stock Spotify idle start and exact output are not promised until proven; MA user/queue behavior needs separate proof; unofficial YT Music remains opt-in. +- Contract tests and a disposable Home Assistant installation must demonstrate one-target dispatch, rejected arbitrary Core authority, secret isolation, stale/unknown outcomes, and safe reconnect without replay. +- This ADR chooses the authority boundary only. It does **not** mark the listening SDD `Ready for implementation` or authorize production broker code while its other gates remain open. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index ee6591c..5087fe6 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -9,5 +9,6 @@ ADRs contain only decisions already justified and accepted by the product brief | [0003](0003-home-assistant-app-primary.md) | Accepted | Home Assistant App is the primary deployment boundary | | [0004](0004-home-assistant-native-ui.md) | Accepted | App UI follows Home Assistant-native interaction and visual patterns through an owned compatibility layer | | [0005](0005-lit-typescript-vite-ui.md) | Accepted | Future App presentation layer uses Lit, TypeScript, and Vite, subject to UI readiness gates | +| [0006](0006-narrow-home-assistant-playback-broker.md) | Accepted | Home Assistant playback uses a narrow companion broker instead of broad App-to-Core authority | An ADR is immutable after acceptance except for typo/link corrections. A changed decision gets a new ADR that supersedes the old one. diff --git a/docs/development/quality-audit.md b/docs/development/quality-audit.md index 79ec094..404cd23 100644 --- a/docs/development/quality-audit.md +++ b/docs/development/quality-audit.md @@ -35,3 +35,11 @@ Use a fresh temporary output directory for each graph extraction. `--code-only` 2. Define application repository ports and a composition rule in the relevant SDD/architecture documents before removing the current direct SQLite dependencies. 3. Measure line and branch coverage with an explicitly selected development tool, then target missing high-risk branches instead of a global percentage alone. 4. Keep architecture import tests and `make verify` as the dependency-free baseline; review graph/health deltas as advisory evidence rather than automatically deleting or rewriting code. + +## Follow-up (2026-09-28) + +Re-ran the documented offline commands against the local `develop` working tree while defining the listening broker. RepoWise still identifies `copy_execution.py` (1.6/10) and `sqlite_operations.py` (1.6/10) as XL hotspots; its safe-only dead-code pass found **zero** candidates. Its apparent highest refactoring ratio on `application/__init__.py` is an `untested_hotspot` heuristic, not proof of a defect or permission to remove a public export. No production refactor follows from a score alone. + +Graphify's `--code-only --no-cluster` extraction produced **1,181 nodes and 3,060 edges**. `OperationRepository` remains the most connected symbol (57 edges); its one-hop dependents include runtime HTTP/resources, copy and import execution, the operation runner, and their tests. This reinforces the existing plan to define repository ports and preserve restart/reconciliation characterization before changing operation persistence. The broker design is kept out of domain/application imports and is not wired into this graph while its SDD is Draft. + +The local `make verify` run passed **227 `unittest` cases (2 skipped)** plus specification/whitespace checks. The isolated Lit spike also passed `npm ci --ignore-scripts`, TypeScript, presentation-boundary checks, three pure fallback tests, build, and relative-asset verification. Neither result proves Home Assistant integration, live playback, visual parity, or CI on GitHub. RepoWise still has no measured branch-coverage input; that remains a separate tooling decision rather than a fabricated coverage percentage. diff --git a/docs/domain/domain-model.md b/docs/domain/domain-model.md index 91beb84..df1f3b4 100644 --- a/docs/domain/domain-model.md +++ b/docs/domain/domain-model.md @@ -1,11 +1,11 @@ # Domain model **Status:** provider-independent core accepted; playlist ownership and matching policy proposed/open -**Last reviewed:** 2026-09-20 +**Last reviewed:** 2026-09-28 ## Model boundary -Symphonia models a person's relationship with recordings and provider collections. It does not model audio files or playback. Provider payloads enter through adapters and are translated into this language before use by product workflows. +Symphonia models a person's relationship with recordings and provider collections. It does not model audio files, decode streams, or own a playback queue. A separate, transient listening context may refer to an external playback session and player without turning observed now-playing metadata into a canonical recording or imported playlist fact. Provider payloads enter through adapters and are translated into this language before use by product workflows. This document specifies concepts and invariants, not database tables or API schemas. @@ -20,6 +20,11 @@ This document specifies concepts and invariants, not database tables or API sche | **Artist credit** | The ordered credited performers attached to a recording or release. | A credit is not yet a canonical person/group identity. | | **Provider** | A type of external system, such as Spotify or YouTube. | A provider is not an account. | | **Provider connection** | One authorized external account plus its effective capabilities and credential reference. | More than one connection may eventually exist for one provider. | +| **Playback source** | One configured integration/account able to browse or start music through an approved playback adapter. | A library provider connection does not configure or authorize it automatically. | +| **Player target** | One explicit Home Assistant `media_player` entity or other accepted playback endpoint. | A player can expose several input sources; it is not a provider account or necessarily a physical speaker. | +| **Output device** | The destination chosen through a player/source's supported output-selection contract. | A source name is not a universally stable device ID; some players have no selectable output. | +| **Playback session observation** | Ephemeral, timestamped account/player state received from a playback adapter. | It may be delayed or unavailable and does not prove that an imported playlist is current. | +| **Playback source binding** | Explicit association between a listening source/player and an optional library connection, with identity evidence or a visible unverified label. | Matching names or provider kinds are not proof of the same account. | | **Provider track** | A provider catalog item that represents or makes a recording available. | It retains the provider ID and may have incomplete or conflicting metadata. A YouTube video is a provider track only when used as a music candidate. | | **Provider playlist** | An ordered collection owned by a provider/account and imported into Symphonia. | It is not automatically a Symphonia-owned logical playlist. | | **Playlist entry** | One occurrence at one position in a playlist snapshot. | Repeated tracks are separate entries and must not be collapsed. | @@ -50,6 +55,8 @@ Future: PlaylistProjection(s) ── governed by ── SyncRelationship The arrows do not imply persistence ownership. In particular, deleting a connection does not make a recording cease to exist. +Listening is an adjacent context: `PlaybackSource -> PlayerTarget -> observed PlaybackSession`; an optional explicit binding points to `ProviderConnection`. It does not own `Recording`, `ProviderPlaylist`, `CopyPlan`, or `Operation`. Starting a provider playlist needs an exact source-compatible reference; track title similarity or an automatic identity link is insufficient. No playback observation is promoted into library history without a separately approved policy. + ## Core entities and value objects ### Recording diff --git a/docs/open-questions.md b/docs/open-questions.md index 3f9602d..f180a7c 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -1,7 +1,7 @@ # Open questions, risks, and next design work **Status:** open; nothing here is an accepted decision -**Last reviewed:** 2026-09-26 +**Last reviewed:** 2026-09-28 ## Decisions requiring owner input @@ -78,6 +78,12 @@ The repository currently has no license. The license should be selected before a This foundation deliberately does not guess at provider reconciliation or expose an unauthenticated resolution endpoint. A future provider-aware reconciliation handler may resolve the outcome automatically only after its provider contract is proven. The repository primitive for explicit resolution is not itself an authorization boundary; any caller must enforce the approved operator identity and action policy. The durable-operation SDD remains blocked on that provider/UI vertical and its other readiness gates, but OQ-010 no longer represents an unresolved policy choice. +### OQ-011 — Which playback sources and Home Assistant authority ship first? + +The owner wants a Home Assistant-integrated listening surface alongside playlist management; see the [listening SDD](../specs/listening-and-playback-control.md). **Authority boundary resolved (2026-09-28):** the owner chose a narrow companion playback broker, recorded in [ADR 0006](decisions/0006-narrow-home-assistant-playback-broker.md), instead of granting the App broad Supervisor Core API authority. The first supported source/output matrix and broker transport, pairing, versioning, user attribution, and threat model remain open. This is independent of `OQ-004`'s provider OAuth callback decision: provider import credentials do not authorize HA player control. + +**Proposed staged default, conditional on proof:** first evaluate the official Home Assistant Spotify entity via the broker on an already active, uniquely identifiable compatible output. The [source review](providers/playback-integration-source-review.md) shows that stock HA Spotify drops `PLAY_MEDIA` while idle, selects devices by name and may not support exact-output promises. Decide whether this deliberately limited path is useful or a separately authorized device-ID-capable profile is needed before promising start-from-idle or “play here.” Then consider Music Assistant players and its unofficial YouTube Music source with independent account/caller/queue and support-risk review. `ytube_music_player` is optional community evidence, not an automatic dependency. Multiple active players are selected explicitly. There is no fallback to a generic Core API relay. + ## Research/design gates (not owner preference alone) ### RG-001 — Official provider feasibility @@ -144,6 +150,10 @@ The owner has accepted the Home Assistant-native-adjacent direction, independent The tooling choice is accepted; the result must still establish a supported matrix and validated host/visual contracts. It does not authorize production UI work until the SDD is ready and the owner explicitly approves implementation. +### RG-007 — Listening source, target, and command feasibility + +Use a disposable Home Assistant installation and dedicated accounts/devices after the broker protocol/security decision. Record the exact versions and account identities for HA Spotify, Music Assistant, and any proposed community player. Run the [dated source-review proof matrix](providers/playback-integration-source-review.md#rg-007-proof-matrix-before-release): verify Spotify idle/active/restricted and duplicate-name device behavior, WebSocket versus service browsing and service feature gates, exact-output/start feasibility, MA queue versus external source and per-user/default-user attribution through the **actual companion-broker route**, playlist reference mapping, private/unavailable/large playlist behavior, state-update/device-discovery lag, accepted-but-unobserved commands, external-controller races, HA restart, and credential recovery. Prove whether account identity can be machine-verified; if not, specify a visible user-confirmed but unverified binding and forbid automatic private-playlist mapping. Test hostile entity/media IDs, MA free-text/URL/path fallback rejection, broker pairing/version/replay/secret isolation, and ensure no broker credential reaches Ingress/browser state. Decide freshness/command-confirmation budgets from these observations. Keep community implementation findings separate from official platform evidence and do not infer control of native YouTube Music app sessions. + ## Future synchronization questions These do not block the copy MVP but block sync implementation: @@ -193,6 +203,8 @@ These need evidence and small RFCs; popularity is not evidence. | Single-node embedded storage cannot handle job concurrency/backup safely | Low-medium | Medium | `RG-004`; no multi-replica claim | | Home Assistant coupling leaks into domain/application | Medium | High maintenance/portability cost | ADR 0003 dependency rule and architecture tests | | Native-looking UI depends on unstable private Home Assistant components or drifts into an unrelated SaaS design | Medium | High compatibility and product-coherence cost | ADR 0004, owned compatibility layer, `RG-006`, dated official references, catalog/a11y/responsive/visual release gates | +| Listening UI confuses library account, HA source, and speaker or shows accepted commands as confirmed playback | High without explicit contract | High: wrong output/account and misleading controls | `OQ-011`, `RG-007`, explicit binding, exact references, observed-effect reconciliation | +| Broker pairing/operation scope fails or unofficial YT Music credentials cross into browser/diagnostics | Medium | Critical privacy/control risk | ADR 0006 narrow broker, pairing/replay/secret threat review, entity/action allowlist, no cookie handoff | | Unknown target library sizes lead to unjustified performance design | High | Medium | `OQ-007`, measurable targets before optimization | | Future Apple Music support is mistaken for full sync despite no documented remove/reorder operation | Medium | High if scope is promoted | Per-operation/object capability probes; Apple feasibility gates before scope change | @@ -202,7 +214,7 @@ These need evidence and small RFCs; popularity is not evidence. 2. **Close the [authorization SDD](../specs/provider-connections-and-authorization.md) blockers (`RG-002` + `OQ-004`).** Validate the direct App callback flow, then settle bring-your-own credentials, encryption, revocation, and backups. 3. **Close the [copy SDD](../specs/one-time-playlist-copy.md) policy blocker (`OQ-003`).** Decide target creation, strict/best-effort behavior, batching, partial failure, reconciliation, cancellation, and exact acceptance examples. 4. **Close the [identity SDD](../specs/recording-identity-resolution.md) evidence blockers (`RG-003`).** Build the corpus and settle normalization, candidate sources, evidence, versioned rules, manual decisions, and measurable safety targets. -5. **Close the [runtime](../specs/home-assistant-app-runtime-and-ingress.md) and [durable-operation](../specs/durable-operations-and-recovery.md) SDD blockers (`RG-004`).** Choose process topology and storage only after crash, lease, migration, backup, and representative-scale evidence. +5. **Define the [listening](../specs/listening-and-playback-control.md) vertical (`OQ-011`/`RG-007`) alongside [runtime](../specs/home-assistant-app-runtime-and-ingress.md) and [durable-operation](../specs/durable-operations-and-recovery.md) blockers (`RG-004`).** Specify and threat-review the selected companion-broker protocol and prove source/account/player/output behavior before promising Spotify or YouTube Music controls; choose process topology/storage only after crash, lease, migration, backup, and representative-scale evidence. After those tasks, revisit playlist ownership (`OQ-002`) before creating any persistent-synchronization SDD. The Home Assistant native surface (`RG-005`) can proceed in parallel once the service API shape is stable, but it is not a prerequisite for the copy MVP. diff --git a/docs/product/home-assistant-ui-specification.md b/docs/product/home-assistant-ui-specification.md index bf05429..f2a7c66 100644 --- a/docs/product/home-assistant-ui-specification.md +++ b/docs/product/home-assistant-ui-specification.md @@ -1,7 +1,7 @@ # Home Assistant-native UI specification **Status:** accepted product direction; implementation contract ready for specialist review -**Last reviewed:** 2026-09-20 +**Last reviewed:** 2026-09-28 ## Purpose and scope @@ -70,6 +70,9 @@ The Ingress panel must feel like one focused Home Assistant application, not a m | Progress/loading/empty | HA-like progress and empty surfaces that explain what is happening | determinate, indeterminate with textual state, skeleton only when meaningful, empty with action, retryable error | | Data list/table/result row | Dense operational data with semantic headings and bounded internal scrolling | populated, empty, loading, partial, row action, selected, responsive list alternative | | Tags and evidence groups | Compact metadata, not primary actions | wrapping, overflow-safe, removable only when semantically an input | +| Now-playing and transport controls | HA media-player card conventions within the App's independent component layer | selected player/source, artwork/metadata/position, play/pause/stop/skip, pending/disabled/stale/unavailable, multiple active players, narrow layout | + +The listening route uses a clear source → content → output sequence and a persistent-in-route now-playing region. It does not pretend that an imported playlist is a playable queue or that all `media_player` entities support the same controls. Missing source, output, account binding, or exact media reference receives a specific explanation and recovery action. A command that Home Assistant accepted but whose effect is not yet observed remains pending/uncertain rather than showing a false success state. Pills are reserved for compact status/metadata. Cards are not nested repeatedly for decoration. Destructive styling is reserved for destructive or irreversible effects. Provider artwork and branding may identify provider content, but must not replace Home Assistant interaction conventions. @@ -168,6 +171,7 @@ Visual snapshots are reviewed evidence, not self-approving output. A generated b - **SYM-UI-013:** Standalone and Ingress profiles MUST share component and feature semantics; host context MAY change tokens, navigation integration, and authentication framing but MUST NOT create a second product behavior. - **SYM-UI-014:** Decorative effects MUST NOT compete with operational status, evidence, or actions; permanent ambient gradients, glassmorphism, decorative animation, and novelty backgrounds are not part of the accepted visual language. - **SYM-UI-015:** Supported Home Assistant/frontend compatibility MUST be versioned and revalidated when official component, token, App-iframe, or safe-area contracts change. +- **SYM-UI-016:** The listening view MUST compose the shared now-playing/transport family with explicit source and target selection, capability-specific disabled reasons, observation freshness, and keyboard/screen-reader state changes; passive progress updates MUST NOT repeatedly steal focus or announce every tick. ## Exceptions and review diff --git a/docs/product/product-specification.md b/docs/product/product-specification.md index ca4303f..c31bd2d 100644 --- a/docs/product/product-specification.md +++ b/docs/product/product-specification.md @@ -1,13 +1,13 @@ # Product specification **Status:** proposed baseline for owner review -**Last reviewed:** 2026-09-20 +**Last reviewed:** 2026-09-28 ## Vision Symphonia is a self-hosted personal music hub through which a person can see and manage their music across providers. Symphonia owns the user's provider-independent view of music; Spotify, YouTube, Apple Music, Plex, Navidrome, and future services are replaceable representations and execution targets. -The product initially optimizes for trustworthy library interoperability: import, explain, match, review, and copy. It is not primarily a playback surface. +The product combines a Home Assistant-integrated listening surface with trustworthy library interoperability. A user can choose music and an output, see what is playing, and control an existing playback session where a configured integration supports it; Symphonia also imports, explains, matches, reviews, and copies playlists. Symphonia is a playback **controller**, not a replacement audio-streaming engine. Library access and playback access are separate capabilities even when they refer to the same service. ## Product principles @@ -18,6 +18,7 @@ The product initially optimizes for trustworthy library interoperability: import 5. **Self-hosted is a product constraint.** Operation, backup, upgrades, credentials, and recovery must be understandable to a homelab operator. 6. **Home Assistant is the primary host, not the domain boundary.** The primary package is a Home Assistant App with an Ingress UI, while the same core remains independently runnable and testable. 7. **The App should feel native to its host.** UI hierarchy, components, terminology, density, themes, responsive behavior, and feedback follow current Home Assistant patterns through an independent compatibility layer; see the [Home Assistant-native UI specification](home-assistant-ui-specification.md). +8. **Listening is grounded in an explicit source and output.** An imported provider playlist, an authorized playback account, a Home Assistant `media_player`, and a physical output are not interchangeable. The UI never infers one from a matching provider name. ## Users @@ -28,6 +29,7 @@ Multi-user authorization, sharing between Symphonia users, and hosted SaaS opera ## Goals - Provide one inventory of connected provider playlists and library items with clear provenance. +- Provide a familiar listening view for configured playback sources: browse/select available music, choose a supported output, see current playback, and use only supported controls. - Recognize when provider-specific items likely represent the same recording. - Let the user resolve ambiguity and preserve that decision. - Copy a playlist between supported providers with a complete preview and result. @@ -40,10 +42,10 @@ Multi-user authorization, sharing between Symphonia users, and hosted SaaS opera The MVP MUST NOT include: -- audio playback or a competing player experience; +- decoding or delivering audio in Symphonia, or becoming a competing streaming backend; - recommendations, discovery feeds, or AI-generated playlists; - audio download, ripping, transcoding, streaming, or format management; -- multi-room audio or device control; +- inventing multi-room synchronization, discovering arbitrary devices outside configured playback integrations, or guaranteeing control of sessions not exposed to Home Assistant; - social features or public profiles; - multi-user tenancy or role-based administration; - automatic bidirectional playlist synchronization; @@ -51,6 +53,8 @@ The MVP MUST NOT include: - a generic detached SaaS dashboard, direct dependency on private Home Assistant frontend internals, or decorative styling that competes with operational state; - an assumption that similar titles imply identical recordings. +Provider-backed browsing/search for a selected playback source is part of listening; the non-goal above excludes Symphonia-owned recommendation or discovery algorithms. + ## Core user journeys ### Connect a provider @@ -67,6 +71,14 @@ The MVP MUST NOT include: 2. The user browses playlists grouped or filtered by provider connection. 3. Each item retains its original provider identity and shows its Symphonia recording link or resolution state. +### Listen and control + +1. The user sees connected library providers separately from configured playback sources and available outputs. Missing playback setup is actionable, not presented as a broken library connection. +2. The user chooses a source and explicit target player/output, then browses content supported by that source. Imported playlists can be started only when an exact, permitted playback reference is available for that source/account; otherwise the UI offers the source's media browser or explains the limitation. +3. Before replacing current playback, the UI names the target and the content that will start. A successful command is confirmed by observed player state, not by an accepted request alone. +4. The listening view shows the selected session's title, artist, artwork, progress and playlist/context only when exposed, with provenance and freshness. The user can pause/resume, skip, stop, change source/output, or start another playlist only when the current player advertises and actually accepts the action. +5. If several players are active, the user chooses which one to control. If Home Assistant or the playback integration is unavailable, Symphonia keeps library and playlist-management features available and marks playback state stale or unavailable. + ### Resolve an ambiguous track 1. The user opens the unresolved-items queue. @@ -94,7 +106,7 @@ The MVP MUST NOT include: ### Home Assistant-native UI -The cross-cutting `SYM-UI-001`–`SYM-UI-015` requirements live in the [Home Assistant-native UI specification](home-assistant-ui-specification.md). Every feature SDD with a web surface must map its feature-specific states and actions onto that shared component, accessibility, responsive, Ingress, localization, sanitization, and visual-compatibility contract. [ADR 0004](../decisions/0004-home-assistant-native-ui.md) records the accepted decision to use an owned compatibility layer rather than private Home Assistant frontend modules. +The cross-cutting `SYM-UI-001`–`SYM-UI-016` requirements live in the [Home Assistant-native UI specification](home-assistant-ui-specification.md). Every feature SDD with a web surface must map its feature-specific states and actions onto that shared component, accessibility, responsive, Ingress, localization, sanitization, and visual-compatibility contract. [ADR 0004](../decisions/0004-home-assistant-native-ui.md) records the accepted decision to use an owned compatibility layer rather than private Home Assistant frontend modules. ### Product and account @@ -108,6 +120,18 @@ The cross-cutting `SYM-UI-001`–`SYM-UI-015` requirements live in the [Home Ass - **SYM-ACC-005:** The MVP Home Assistant Ingress management surface MUST be restricted to authenticated Home Assistant administrators. - **SYM-ACC-006:** Before authorization, the UI MUST show whether access uses an official or reverse-engineered contract, adapter maturity/support level, required external dependencies, credential type, and expected reauthorization behavior. +### Listening and playback + +- **SYM-PLAY-001:** Symphonia MUST distinguish a library provider connection, a playback source/account, a Home Assistant player entity, and an output device; linking them MUST be explicit and must disclose unverified account identity. +- **SYM-PLAY-002:** The listening view MUST show the currently selected player's reported playback state and available title, artist, artwork, position, and playlist/context with source, observation time, and stale/unavailable status; unknown fields MUST remain unknown rather than being inferred from imported data. +- **SYM-PLAY-003:** Playback controls MUST be enabled from effective, current player/source capabilities and authorization; unsupported, unknown, or unavailable actions MUST be disabled with a reason. +- **SYM-PLAY-004:** Starting a track, album, or playlist MUST name a selected target, require an exact source-compatible media reference or supported media-browser selection, and warn when it will replace current playback. A display-title match is insufficient. +- **SYM-PLAY-005:** A playback command MUST have a bounded pending state and reconcile against observed state. Accepted transport requests MUST NOT be labeled as confirmed playback; lost/ambiguous responses MUST NOT be blindly replayed. +- **SYM-PLAY-006:** The App MUST keep library/copy functions usable when Home Assistant playback access is unavailable and MUST NOT silently substitute direct provider playback credentials or an unofficial integration. +- **SYM-PLAY-007:** The UI MUST distinguish pause, resume, stop, skip, output selection, and starting a different playlist; an action MUST target only the explicitly selected, authorized player and must never broadcast to every player by default. +- **SYM-PLAY-008:** Playback-session data MUST be ephemeral by default, sanitized in UI/logs/diagnostics, and excluded from durable import/copy history unless a separate user-visible policy is approved. +- **SYM-PLAY-009:** The listening view MUST offer source-backed browsing of the playable media kinds exposed by the configured source (such as tracks, albums, and playlists), and search only where that source supports it; unavailable kinds/search MUST be explained rather than replaced by Symphonia's imported catalog or guessed matches. + ### Unified library - **SYM-LIB-001:** Every imported object MUST retain provider type, provider connection, provider object identifier, source timestamps when available, import time, and raw-data freshness metadata. @@ -154,6 +178,7 @@ The first useful release is complete when one local user can: - authenticate with supported provider flows; - import supported library collections and owned/followed playlists; - browse provider playlists and their freshness; +- discover at least one explicitly linked Home Assistant playback source/player, browse its playable content, start a supported track or playlist on a chosen output, observe its current session, and use the controls it supports; starting an imported playlist requires a verified compatible reference; - resolve track identities automatically where safe and manually where needed; - preview and execute a copy in each direction only where the target adapter declares all required capabilities; - inspect basic operation history; and @@ -161,6 +186,8 @@ The first useful release is complete when one local user can: The wording “validated Google/YouTube connection” is deliberate. Symmetric **YouTube Music** library access is not yet proven through an official API; see [provider research](../providers/provider-research.md). If official feasibility fails, the owner must revise the MVP rather than silently adopting a reverse-engineered API. +This listening goal does **not** promise a YouTube Music session outside a configured Home Assistant/Music Assistant/community player, or that connecting Symphonia to Spotify automatically configures Home Assistant's Spotify integration. If the first-release playback source/output pairing or account-binding contract cannot be proven, the release boundary requires an explicit owner revision; a screenshot-only listening view does not satisfy it. + ## Future scope - Persistent one-way, add-only, and bidirectional playlist synchronization. @@ -168,6 +195,7 @@ The wording “validated Google/YouTube connection” is deliberate. Symmetric * - Apple Music, Plex/Plexamp, Navidrome, and other adapters. - Multiple accounts per provider in the UI. - Home Assistant entities, actions, and events over a stable Symphonia API. +- Direct provider playback adapters where a supported contract materially improves coverage beyond Home Assistant players, subject to separate scopes, permissions, and policy review. - More complete album, artist, release, and musical-work modeling. - Export/import of user-authored mappings and operation history. @@ -182,6 +210,7 @@ Targets require real-provider feasibility testing before numeric thresholds are - manual-resolution reuse; - copy plan versus execution outcomes; - provider request, throttle, retry, and failure counts; and -- time since last successful import and operation completion. +- time since last successful import and operation completion; and +- configured/active playback targets, state freshness, supported versus unavailable controls, command confirmation/ambiguity, and source/output setup failures without recording listening history by default. No response-time, matching-accuracy, or scale target is accepted yet; the test corpus and expected homelab library sizes are open questions. diff --git a/docs/providers/home-assistant-ecosystem-review.md b/docs/providers/home-assistant-ecosystem-review.md index 091bf6e..dd485b3 100644 --- a/docs/providers/home-assistant-ecosystem-review.md +++ b/docs/providers/home-assistant-ecosystem-review.md @@ -1,7 +1,7 @@ # Home Assistant music ecosystem review **Status:** research snapshot and design input, not a dependency or scope decision -**Reviewed:** 2026-09-20 +**Reviewed:** 2026-09-28 (playback-source addendum; earlier implementation findings retain their original snapshot) **Source policy:** project documentation and current source were inspected in addition to official provider documentation; revalidate before implementation ## Purpose @@ -12,7 +12,7 @@ The strongest precedent is Music Assistant: a separate service/App owns the musi ## Executive conclusions -1. **Keep the App/service as the system of record and the Home Assistant integration thin.** The [Music Assistant integration](https://www.home-assistant.io/integrations/music_assistant/) connects Home Assistant to a separately running server and exposes selected entities/actions. This validates Symphonia's accepted deployment direction without requiring Symphonia to become a playback system. +1. **Keep the App/service as the system of record and the Home Assistant integration thin.** The [Music Assistant integration](https://www.home-assistant.io/integrations/music_assistant/) connects Home Assistant to a separately running server and exposes selected entities/actions. This validates Symphonia's accepted deployment direction without requiring Symphonia to become an audio-streaming engine; a separate playback-controller capability is now proposed. 2. **Adopt a manifest plus effective-capability model.** Music Assistant provider implementations declare features and support multiple provider instances. Symphonia needs the same extensibility, but with more granular, runtime capability descriptors because copy and sync require stronger guarantees than playback and browsing. 3. **Reuse Home Assistant's OAuth conventions where possible, not its provider domain logic.** The official [Spotify integration](https://www.home-assistant.io/integrations/spotify) demonstrates bring-your-own application credentials, multiple accounts, and Home Assistant's external OAuth callback. A companion integration could potentially broker authorization for the App, but this is a spike candidate rather than an accepted design. 4. **Treat unofficial YouTube Music access as a distinct product mode.** Music Assistant and `ytube_music_player` demonstrate useful access through `ytmusicapi`, browser cookies, internal endpoints, and proof-of-origin tokens. They do not turn that surface into a supported Google API. An unofficial adapter would need explicit opt-in, health warnings, separate release gating, and no promise of symmetric copy/sync. @@ -27,7 +27,7 @@ The strongest precedent is Music Assistant: a separate service/App owns the musi | [`vypdev/homeassistant-gateway`](https://github.com/vypdev/homeassistant-gateway/tree/1ed75be9f8fabdab386db0fe4320cfb0f67d4f42) | A Supervisor App can present a coherent HA-native-adjacent Ingress UI using an owned component compatibility layer, catalog, dated official-demo references, and browser/visual tests | Semantic tokens, presentation-only primitives, shared component families, light/dark and mobile evidence, explicit private-HA dependency boundary | Community implementation reviewed at one commit; its Lit/Vite/Storybook choices and exact CSS are not automatically Symphonia decisions | | [Home Assistant Spotify](https://www.home-assistant.io/integrations/spotify) | A maintained Home Assistant integration can use application credentials, the HA external OAuth callback, and multiple account entries | Native config flow, reauthentication, callback and credential UX | It is a playback/media-browser integration, not a cross-provider library system | | [Home Assistant Music Assistant integration](https://www.home-assistant.io/integrations/music_assistant/) | Home Assistant can discover and connect to a separate music server running as an App or container | Service owns domain; integration exposes bounded native actions/entities over an API | Installing an App and installing an integration remain separate lifecycle steps | -| [Music Assistant server](https://github.com/music-assistant/server) | Provider plugins, feature declarations, multiple instances, a normalized internal library, provider mappings, scheduled sync, and versioned SQLite migrations work at real scale | Provider manifest, connection instance, normalized mapping graph, scheduled imports | Playback requirements and automatic merging are not Symphonia requirements | +| [Music Assistant server](https://github.com/music-assistant/server) | Provider plugins, feature declarations, multiple instances, a normalized internal library, provider mappings, scheduled sync, and versioned SQLite migrations work at real scale | Provider manifest, connection instance, normalized mapping graph, scheduled imports | Its audio engine, queue ownership, and automatic merging are not Symphonia requirements | | [Music Assistant Spotify provider](https://www.music-assistant.io/music-providers/spotify/) | Spotify library/search support and multiple accounts are operationally feasible | Capability probing, account-specific source selection, OAuth lifecycle | Playback engines and their policy/terms trade-offs are out of scope | | [Music Assistant YouTube Music provider](https://www.music-assistant.io/music-providers/youtube-music/) | Reading a YT Music library/search surface is technically feasible through private web behavior | Isolate the adapter, identify provider-instance-scoped IDs, expose reauthentication health | The project explicitly says there is no official API; cookies expire and a PO-token sidecar is required | | [`ytube_music_player`](https://github.com/KoljaWindeler/ytube_music_player) | A Home Assistant custom integration can browse/play YT Music via `ytmusicapi` | Additional implementation evidence and failure cases | Its current [browser-auth guide](https://github.com/KoljaWindeler/ytube_music_player/blob/main/QUICK_START_BROWSER_AUTH.md) says OAuth is broken and asks users to export authenticated browser headers | @@ -38,6 +38,16 @@ No source code has been selected for reuse. Any future reuse proposal must revie The review identified official Home Assistant integrations for Spotify and Music Assistant, but did not identify official direct integrations for YouTube Music or Apple Music. The direct examples above are Music Assistant providers or community/HACS integrations, not official Home Assistant Core integrations. Their existence is implementation evidence, not a platform support guarantee. +## Playback source evidence (2026-09-28) + +This addendum records implementation/project evidence separately from the [official API research](provider-research.md#2026-09-28-playback-and-home-assistant-api-update). The deeper [source-code review and proof matrix](playback-integration-source-review.md) identifies version-sensitive operation limits and remaining live tests. The official [HA Music Assistant integration](https://www.home-assistant.io/integrations/music_assistant/) exposes each MA player/group as a `media_player` with current item, progress, and controls; it also notes duplicate HA/MA versions of imported players and that remote content may have limited metadata. A listening UI must therefore select one exact player identity and show provenance rather than merging same-named entities. + +Music Assistant's [YouTube Music source guide](https://www.music-assistant.io/music-providers/youtube-music/) explicitly calls its access best-effort/unofficial, says free accounts are unsupported, and currently requires a login cookie plus PO-token generator. It can present library/playlists inside MA and feed MA playback, but that does not establish observation/control of every session started in Google's official mobile/web clients. It also does not make the Symphonia library adapter automatically available to MA. + +The community [`ytube_music_player` README](https://github.com/KoljaWindeler/ytube_music_player/blob/main/README.md) describes browsing YT Music content, buffering a playlist, and forwarding stream URLs to a selected remote HA player. Its `media_player` can start a playlist and select a remote player. This is useful evidence for optional per-integration commands, **not** a general YouTube Music Connect/remote-session API. Its authentication and upstream behavior must be revalidated before any supported-product claim; the conflicting OAuth/browser-header guidance in this repository is itself a reason for a dedicated setup/security spike. + +Inference: Symphonia can offer one consistent, capability-driven listening route over approved HA entities without embedding MA or a YT Music scraper. Library-provider connection, playback-source account, HA entity, and physical output remain separately configured and explicitly bound. The exact Spotify/MA/community mapping, account-proof method, permission scope, and supported first-release matrix remain design/test gates, not accepted guarantees. + ## Architecture lessons from Music Assistant ### Server plus integration is the right split diff --git a/docs/providers/playback-integration-source-review.md b/docs/providers/playback-integration-source-review.md new file mode 100644 index 0000000..0ef7726 --- /dev/null +++ b/docs/providers/playback-integration-source-review.md @@ -0,0 +1,90 @@ +# Playback integration source review + +**Reviewed:** 2026-09-28. **Purpose:** reduce implementation uncertainty for [Listening and playback control](../../specs/listening-and-playback-control.md), not approve that Draft SDD or select a dependency. This is a reading of the linked project source on mutable `dev`/`main` branches; exact deployed releases, account behavior and device behavior still require the `RG-007` live matrix. Recheck and pin revisions before implementation. Official provider **API** evidence stays in [provider research](provider-research.md); this document analyzes integration implementations. No code or authentication material is proposed for reuse. + +## Source inventory and evidence boundary + +| Project | Source inspected | What it establishes | +| --- | --- | --- | +| Home Assistant Core, Spotify | [`media_player.py`](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/media_player.py), [`browse_media.py`](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/browse_media.py), [`coordinator.py`](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/coordinator.py) | Current Core entity behavior, feature flags, browsing, command dispatch, polling; not a universal Spotify Connect guarantee. | +| Home Assistant Core, Music Assistant | [`media_player.py`](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/media_player.py), [`media_browser.py`](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/media_browser.py), [`services.py`](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/services.py) | Current HA-to-MA player, queue, search and action mapping; not proof about every MA provider or physical player. This is the authoritative Core implementation; older `hass-music-assistant` custom-component code is not substituted for it. | +| Home Assistant Core, generic media player | [`media_player/__init__.py`](https://github.com/home-assistant/core/blob/dev/homeassistant/components/media_player/__init__.py) | WebSocket browse/search checks and entity-service feature gates. | +| Music Assistant server, YouTube Music | [`providers/ytmusic/__init__.py`](https://github.com/music-assistant/server/blob/dev/music_assistant/providers/ytmusic/__init__.py), [provider guide](https://www.music-assistant.io/music-providers/youtube-music/) | Unofficial MA music-source implementation and its prerequisites; **not** a YouTube Music remote-control integration for Google's own clients. | +| Community `ytube_music_player` | [`media_player.py`](https://github.com/KoljaWindeler/ytube_music_player/blob/main/custom_components/ytube_music_player/media_player.py), [README](https://github.com/KoljaWindeler/ytube_music_player/blob/main/README.md) | A separate, optional HA media-player bridge with its own queue and remote-player dependency; not an official Google/HA integration. | + +## The three control models + +| Model | State and control owner | What “where” means | What Symphonia must not infer | +| --- | --- | --- | --- | +| HA Spotify | One Spotify-account controller entity; Spotify Connect holds actual playback | A Spotify device/source associated with that account, **not** another HA `media_player` selected by name | That a Symphonia library account equals this Spotify account, that a speaker name is a stable ID, or that an accepted service call reached the chosen output. | +| HA Music Assistant + provider | MA player entity and MA server queue; YT Music may supply content | One exact MA player/group; its active source can be its queue or an external input | That YT Music content provider owns an existing native YT Music app session, or that an MA queue is present whenever the player reports media. | +| Community `ytube_music_player` | Custom entity buffers track list; a second HA player renders stream URLs | The configured remote HA player used by that entity | That its locally maintained queue/skip represents Google's native client queue or that all its advertised features work on every remote player. | + +### Operation feasibility (source-code evidence, not release support) + +| User operation | Stock HA Spotify | HA Music Assistant | Community `ytube_music_player` | +| --- | --- | --- | --- | +| Now playing | Current Spotify-account session when fetched; playlist title may be absent | Selected MA player can report MA queue or external input; queue details only for active MA queue | Custom entity's locally tracked state and selected remote HA player; coverage of Google's native clients not shown | +| Pause/resume | Advertised on active, unrestricted Premium session; no `STOP` | Player command; MA may emulate pause via stop/resume; effect must be observed | Forwards play/pause to remote HA player and updates its own state; remote capability/effect must be observed | +| Next/previous | Advertised on active, unrestricted session | Player command; external-source semantics require device-profile proof | Advances its own buffered track list, then starts the chosen stream | +| Stop | Not advertised | Advertised; player command, subject to effective source/player behavior | Forwards stop to remote HA player | +| Seek, shuffle, repeat | Advertised on active, unrestricted session | Seek command; shuffle/repeat are no-ops without active MA queue | Seek forwarded to remote; shuffle/repeat are custom-list semantics; source contains a suspect seek-feature check | +| Browse/search/select playlist | Entity WebSocket browse active only; no search flag; normal `play_media` active only; separate browse service needs proof | Browse/search available; browser list can truncate; exact URI play routes to MA queue | Custom browse/search and playlist buffering; separate auth and target support review required | +| Change output | `select_source` by device **name**, first match; exact ID unavailable through entity state | Select exact MA player entity; `select_source` means its input, not physical target; queue transfer is a separate action | Changes configured remote HA player, can stop previous output and choose a default speaker | + +The first two columns are independent routes: a YouTube Music **MA content provider** does not make HA's Spotify controller manage a YT Music session. The last column is an optional community implementation, not a fallback when the first two fail. Source links and caveats for each cell follow below. + +### Home Assistant Core transport surface + +The generic HA media-player component registers WebSocket `media_player/browse_media` and `media_player/search_media` with an exact `entity_id`; each rejects a player without the corresponding feature flag. Browse accepts an optional *pair* of `media_content_type`/`media_content_id`; search requires `search_query` and can filter classes. Ordinary transport actions are entity services gated by their feature bits. Therefore, an adapter needs typed read/browse/search/command paths and cannot assume that implementing `async_browse_media` alone makes browse callable. [Core browse/search and service registration](https://github.com/home-assistant/core/blob/dev/homeassistant/components/media_player/__init__.py#L256-L320), [WebSocket gates](https://github.com/home-assistant/core/blob/dev/homeassistant/components/media_player/__init__.py#L1230-L1365). + +[ADR 0006](../decisions/0006-narrow-home-assistant-playback-broker.md) subsequently selected a narrow companion integration instead of the broad App-to-Core path. These source files establish operations, **not** that any broker call preserves the Ingress user's identity or narrows Core authority without its own authorization checks. No browser token or generic service proxy follows from this review; the broker's transport/pairing contract remains `OQ-011`. + +### Spotify: real controls and hard output/browse limits + +- When Premium and active on a non-restricted device, the entity advertises browse, play/pause, next/previous, seek, source selection, shuffle/repeat and volume. It does **not** advertise `STOP` or `SEARCH_MEDIA`. On no playback or a restricted device it advertises only `SELECT_SOURCE`; on non-Premium it advertises nothing. The generic WebSocket gate consequently blocks this entity's browse while idle, and the normal HA `play_media` entity service is feature-gated as well. HA also registers a separate response-producing `browse_media` service without a feature gate in this source, but its suitability as an approved idle browser still needs a live permission/identity test. The product cannot promise universal idle start, browse or search through this entity. [Feature calculation](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/media_player.py#L44-L56), [dynamic flags](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/media_player.py#L126-L143), [generic play/browse services](https://github.com/home-assistant/core/blob/dev/homeassistant/components/media_player/__init__.py#L396-L434), [WebSocket browse gate](https://github.com/home-assistant/core/blob/dev/homeassistant/components/media_player/__init__.py#L1230-L1272). +- The state exposes a current item's URI, title, artist, artwork, duration/position, playlist *title when retrievable*, and active `source` name. A playlist lookup failure is deliberately tolerated; `media_playlist` can be absent while music plays. Polling defaults to 30 seconds, commands sleep one second and refresh, and device discovery defaults to five minutes. Observation freshness and device-list freshness are distinct. [Entity state](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/media_player.py#L137-L266), [coordinator](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/coordinator.py#L42-L43), [playlist failure](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/coordinator.py#L123-L166), [device refresh](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/coordinator.py#L169-L193). +- `source_list` exposes sorted device **names**, not IDs. `select_source` transfers to the first matching name and silently returns if none matches. The `async_play_media` method contains a no-current-playback branch that chooses the *first cached device ID*, although the ordinary feature-gated `play_media` service should not reach it while the entity advertises only `SELECT_SOURCE`; this branch must not be treated as an approved idle-start workaround. Duplicate names, old discovery data, idle starts and a concurrent external controller therefore defeat an exact-output promise through the stock HA entity alone. Symphonia must block ambiguous selection, re-observe after any source selection, and not claim deterministic “play here” until a live-supported path proves it; a direct device-ID adapter or a narrower broker would be separate, permission-reviewed options. [Source list/selection](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/media_player.py#L255-L266), [play/selection code](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/media_player.py#L317-L371), [service feature gate](https://github.com/home-assistant/core/blob/dev/homeassistant/components/media_player/__init__.py#L396-L410), [device polling](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/coordinator.py#L169-L193). +- `play_media` dispatches track/episode URIs versus playlist/album/artist contexts; queue `ADD` is only for track/episode/music. Unsupported kinds log and return, so request completion is not effect confirmation. Its browser lists playlists, followed artists, saved albums/tracks/shows and recent/top content, with account-specific roots keyed by the HA config-entry ID; it has no search handler in this source. Browse output supplies exact Spotify URIs but the entity browser and a separate media-source path must not be conflated. Playlist children are built from the returned playlist page; complete large-playlist traversal is **not** demonstrated here. [Play dispatch](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/media_player.py#L317-L360), [browser roots](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/browse_media.py#L101-L120), [account routing](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/browse_media.py#L174-L244), [playlist children](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/browse_media.py#L274-L343). +- The entity's `unique_id` is the Spotify user ID. That suggests a possible account-proof route via the HA entity/config registries and Symphonia's own account ID, but access to those registry details through the chosen bridge and correct consent semantics are **unproven**; a matching provider label or display name is not proof. [Entity identity](https://github.com/home-assistant/core/blob/dev/homeassistant/components/spotify/media_player.py#L106-L120). + +### Music Assistant: queue, source and user context + +- The Core entity registers a broad base feature set, including browse, search, stop, seek, next/previous, clear playlist, enqueue and pause. Some features vary with player configuration, but shuffle/repeat methods return without effect when there is no active MA queue. `active_queue` is resolved from `player.active_source`; the entity still maps `player.current_media` to state/metadata for external sources, with queue-only shuffle/repeat cleared. Effective controls must therefore intersect HA bits, source/queue state and a tested player/provider profile. [Base flags](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/media_player.py#L67-L86), [active queue/state](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/media_player.py#L175-L216), [queue-only actions](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/media_player.py#L327-L351), [metadata](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/media_player.py#L649-L691). +- MA maps source display names to source IDs internally. Duplicate names collapse in that mapping, and a missing name raises an error; the HA state exposes names, not the stable source ID. This is different from choosing the exact MA player entity or choosing a YT Music *content provider*. [Source mapping](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/media_player.py#L203-L216), [source action](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/media_player.py#L405-L412). +- HA `play_media` forwards the media ID to MA. Depending on MA server schema it verifies URI, accepts library IDs/local paths, or finally searches by name. Symphonia must send only a validated exact MA/provider URI, never an arbitrary URL/path or title fallback; the stock integration's permissive input formats are not Symphonia authorization. It plays on the active queue if one exists, else the player's own queue ID. [Resolution and queue routing](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/media_player.py#L424-L520). +- For `media_player.play_media`, Core passes the HA calling user's ID to MA for a linked-user lookup, with a default-account fallback when no link exists. A Symphonia companion-broker call may therefore be attributed to the broker's HA context rather than the listener; exact broker caller context must be verified before claiming per-user provider filtering or private-library access. The advanced `music_assistant.play_media` action exposes an explicit `username`, but that would itself need a validated, authorized user binding. [User resolution](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/media_player.py#L434-L446), [advanced action schema](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/services.py#L128-L144), [HA guide](https://www.home-assistant.io/integrations/music_assistant/). +- MA exposes `music_assistant.get_queue` with queue item/index/current/next and `transfer_queue` as separate actions, but no queue exists on an external active source; transfer without `source_player` chooses the first playing queue. Keep queue editing/transfer outside the first slice; if later offered, require an exact source and target. [Queue and transfer handlers](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/media_player.py#L557-L606), [action registration](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/services.py#L168-L186). +- The HA MA browser has browse and search; its playlist/artist/album listings request at most the first 500 library items because HA's browser does not paginate, and search defaults to five results per media type. The separate `music_assistant.get_library` action has `offset`/`limit` and can support an explicitly approved paginated surface. Do not label a truncated HA browser as the user's whole library. [Browser listing](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/media_browser.py#L192-L212), [search](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/media_browser.py#L634-L671), [library action](https://github.com/home-assistant/core/blob/dev/homeassistant/components/music_assistant/services.py#L108-L127). + +### YouTube Music implementations: feasible MA playback, not native-client control + +The MA YT Music provider declares library, playlists, browse, search and recommendations. Its startup needs a cookie and reachable PO-token service and rejects accounts without YT Music Premium; stream extraction depends on `yt-dlp`/token machinery and emits expiring HTTP stream URLs to **MA**, not to Symphonia. This supports browsing and MA-managed playback, not evidence that an already-playing native YT Music phone/web session can be inspected or controlled. Its playlist read returns the full list rather than pages, can serve expired cache, skips unavailable tracks, and caps dynamic personal mixes at 300: this cannot be reused as a proof of complete or fresh playlist equivalence. [Features/setup](https://github.com/music-assistant/server/blob/dev/music_assistant/providers/ytmusic/__init__.py#L130-L217), [playlist behavior](https://github.com/music-assistant/server/blob/dev/music_assistant/providers/ytmusic/__init__.py#L420-L491), [streams](https://github.com/music-assistant/server/blob/dev/music_assistant/providers/ytmusic/__init__.py#L660-L691), [MA guide](https://www.music-assistant.io/music-providers/youtube-music/). + +The separate `ytube_music_player` custom entity stores playlist/track position and forwards pause/stop/seek to a selected remote HA player; next/previous select tracks from its own list. Its source change can stop the old player and try to resume on another, and setup may choose the first speaker if none is configured. Its code also uses bitwise **OR** where a seek-capability check appears intended; this is a concrete reason not to adopt its seek/transfer semantics as a product guarantee without a dedicated versioned test/security review. It is only an optional compatibility profile, never a default dependency or source of native-client session truth. [Local state](https://github.com/KoljaWindeler/ytube_music_player/blob/main/custom_components/ytube_music_player/media_player.py#L109-L145), [source transfer/default](https://github.com/KoljaWindeler/ytube_music_player/blob/main/custom_components/ytube_music_player/media_player.py#L923-L1045), [transport](https://github.com/KoljaWindeler/ytube_music_player/blob/main/custom_components/ytube_music_player/media_player.py#L1649-L1724). + +## Contract consequences for Symphonia + +1. Discovery returns separate `library connection`, `playback source/account`, `HA entity/MA player`, `active source/queue`, and `output`. A UI picker never collapses these into one same-named “device”. +2. Each adapter profile maps **operation × entity state × source/queue × media kind × output identity** to `supported`, `unavailable`, or `unverified`; static `supported_features` is an upper bound. `STOP` is not offered for HA Spotify; Spotify idle `play_media` and WebSocket browse are feature-gated; MA queue-only actions are not implied by a generic `media_player`. +3. Exact media references are accepted only from an approved browser/resolver or a separately proven binding. Symphonia never passes free text, arbitrary URLs, filesystem paths, or unchecked provider IDs to a permissive integration resolver. +4. “Play on this output” is a blocked claim for the stock Spotify entity until the device-ID/name/idle/race matrix is resolved. Safer partial value is now-playing and eligible transport on a selected account/entity, with output displayed as reported rather than promised. +5. A service response means `accepted` at most. Confirm content, state and, where contractually required, output using a fresh observation; missing playlist title, external MA source and 30-second polling can yield `uncertain`, not success. +6. Account attribution for HA/MA browse, search and play is security-relevant. Resolve who Core/MA regards as the caller before exposing private content; an App token must not be assumed to act as each Ingress listener. +7. The MA YT Music provider and community player are different integrations with different session coverage and support promises. Neither provides demonstrated universal remote control of Google's native clients. + +## `RG-007` proof matrix before release + +Use a disposable HA installation, dedicated Premium accounts, harmless test playlists and two distinguishable devices; record HA/MA/integration versions, source revision, entity/config identifiers (redacted), account IDs (hashed for evidence), command payload category, result, timestamps and cleanup. Never archive cookies, PO tokens, raw titles or listening histories. + +| Probe | Expected decision/evidence | +| --- | --- | +| Spotify playing/paused/idle/restricted and non-Premium; `play_media`, browse service and WebSocket browse/search errors | Which browse/start routes are exposed in each state; whether an alternative approved idle-start/browser adapter is required. | +| Spotify two devices with duplicate names, stale cached device, no current playback, and another controller switching source | Whether stock HA can meet exact-output semantics; if not, block “play here” and decide direct ID adapter/broker or narrower UX. | +| Spotify imported public/private playlist under same/different HA account; unavailable track; large playlist | Account-proof route, exact reference compatibility and whether browse/completeness claims are true. | +| MA queue active vs external source, two MA players, duplicate source names, linked vs default MA user | Effective control/queue matrix and caller attribution through the actual App-to-Core route. | +| MA YT Music expired cookie, missing PO service, dynamic mix, large playlist, simultaneous stream | Opt-in setup/error/partial-list behavior without asserting native-client control. | +| Community player with two remote HA targets and unsupported seek/stop | Whether its advertised features reflect target effects; decide whether to support at all. | +| Accepted call with no state change; external-controller race; HA restart/disconnect; inaccessible artwork | Confirmation budget, stale-state UX, no replay, secret/metadata sanitation, operator repair path. | + +**Remaining decision:** ADR 0006 settles the narrow Core authority boundary; `OQ-011` still needs broker transport/pairing and the initial integration/operation matrix. `RG-007` must establish actual output, account and caller identity behavior and timing. This review reduces the unknowns to concrete probes; it does not close either gate or authorize production implementation. diff --git a/docs/providers/provider-research.md b/docs/providers/provider-research.md index 792b57d..044dc24 100644 --- a/docs/providers/provider-research.md +++ b/docs/providers/provider-research.md @@ -1,7 +1,7 @@ # Provider and platform research **Status:** research snapshot, not an architectural decision -**Reviewed:** 2026-09-27 +**Reviewed:** 2026-09-28 **Source policy:** official documentation only; revalidate before implementation and every release ## How to read this document @@ -10,6 +10,20 @@ This snapshot separates what Symphonia needs from what an official API documents Existing Home Assistant and community implementations are reviewed separately in [Home Assistant music ecosystem review](home-assistant-ecosystem-review.md). They provide valuable implementation evidence but do not replace an official provider contract. +## 2026-09-28 playback and Home Assistant API update + +This is a dated official-documentation check, not a live account/device feasibility result. Listening is a separate capability from library import and provider playlist writes. + +| Official source | Documented playback fact | Limit for Symphonia | +| --- | --- | --- | +| [Home Assistant Spotify integration](https://www.home-assistant.io/integrations/spotify/) | Exposes account playback and library browsing; documents `media_player.select_source` and `media_player.play_media` with a Spotify playlist URI/link. Requires Premium and a Spotify-compatible known output device. State is polled at least every 30 seconds. | A Symphonia Spotify grant does not install/authorize this HA integration. An entity may be delayed, inactive, or lack a usable output; starting content is not a universal device-discovery contract. | +| [Home Assistant media-player entity](https://developers.home-assistant.io/docs/core/entity/media-player/) and [actions](https://www.home-assistant.io/integrations/media_player/) | Standard states, optional title/artist/artwork/position/playlist attributes, and per-entity supported features include play/pause/stop/next/previous/play-media/browse/select-source. | Metadata and controls are optional and integration-specific; a feature flag alone does not validate an arbitrary provider playlist reference. | +| [Home Assistant Music Assistant integration](https://www.home-assistant.io/integrations/music_assistant/) | Exposes MA players as HA `media_player` entities with current media and controls; provides richer browse/play/queue actions when the MA server and integration are configured. | It is a separate server/integration lifecycle; MA source credentials, account identity, and queue are not Symphonia's imported library or durable playlist. | +| [App communication](https://developers.home-assistant.io/docs/apps/communication/) and [configuration](https://developers.home-assistant.io/docs/apps/configuration/) | An App can request `homeassistant_api: true` and use `SUPERVISOR_TOKEN` with the Supervisor Core REST/WebSocket proxy. [HA WebSocket](https://developers.home-assistant.io/docs/api/websocket/) can stream state changes and call services; [REST](https://developers.home-assistant.io/docs/api/rest/) distinguishes state representation from service actions. | This is broader Core authority than a music-specific permission. It requires a threat review, server-only token, exact entity/action allowlist, and an accepted decision versus a narrower companion broker. | +| [Spotify currently playing](https://developer.spotify.com/documentation/web-api/reference/get-the-users-currently-playing-track) and [start/resume](https://developer.spotify.com/documentation/web-api/reference/start-a-users-playback) | Direct Web API offers current item/context and playlist context start, with separate read/control scopes and Premium playback requirements. | A direct fallback would require new consent, policy, account/device handling, and tests; it is not silently inherited from the library adapter or selected for the first listening SDD. | + +The reviewed [official YouTube Data API reference](https://developers.google.com/youtube/v3/docs) is about video/channel/playlist resources, not a documented YouTube Music now-playing or remote-control API. Absence from reviewed documentation is an inference, not a claim that no private integration exists. Community playback examples and their risks are kept in the [ecosystem review](home-assistant-ecosystem-review.md#playback-source-evidence-2026-09-28). + ## 2026-09-27 write-outcome and reconciliation update This is a dated review of official API documentation only, not a live provider feasibility spike. The official pages describe mutation requests and successful responses, but do not document a general client idempotency-key contract or a provider-independent way to prove whether a timed-out request took effect. diff --git a/docs/providers/provider-specification.md b/docs/providers/provider-specification.md index 0d00964..1a4206b 100644 --- a/docs/providers/provider-specification.md +++ b/docs/providers/provider-specification.md @@ -1,12 +1,14 @@ # Provider specification **Status:** proposed contract; specific support is a dated research fact -**Last reviewed:** 2026-09-27 +**Last reviewed:** 2026-09-28 ## Purpose Provider adapters translate external authentication, catalog, library, and playlist behavior into Symphonia concepts. The core asks for semantic capabilities and operations; it never branches on `spotify` or `youtube` to decide domain policy. +Listening uses a distinct playback-source/player adapter contract. A library adapter may contribute an exact playback reference for one of its objects, but a library grant is not a playback grant, a provider playlist is not a player queue, and the presence of a provider adapter does not imply that Home Assistant has a corresponding `media_player` entity. + This document states what Symphonia needs. [Provider research](provider-research.md) separately records what current official APIs appear to support, while the [Home Assistant music ecosystem review](home-assistant-ecosystem-review.md) records reusable implementation patterns and cautions from existing projects. ## Provider, connection, and adapter @@ -101,6 +103,25 @@ The initial catalog is intentionally granular: - `metadata.artist_credits.read` - `metadata.version_markers.read` +### Playback source and player (separate from library/write capabilities) + +- `playback.source.browse` +- `playback.source.search` +- `playback.media.start` +- `playback.state.observe` +- `playback.transport.play` +- `playback.transport.pause` +- `playback.transport.stop` +- `playback.transport.next` +- `playback.transport.previous` +- `playback.output.select` +- `playback.position.seek` +- `playback.shuffle.set` +- `playback.repeat.set` +- `playback.queue.read` + +These are **effective capabilities of a configured playback source and selected player at a point in time**, not promises made by a library connection. A Home Assistant adapter may derive them from `supported_features`, entity state, integration-specific media types, account/permission evidence, and live availability; a static feature bit alone does not establish that a particular playlist URI or remote device will work. The adapter must distinguish an observed capability, a tested source-specific mapping, and an unknown mapping. Music Assistant or community integrations retain their own access-basis/support labels. + Adapters MAY add namespaced experimental capabilities, but product workflows use only cataloged stable capabilities until the catalog is revised. ## Capability requirements by workflow @@ -113,9 +134,13 @@ Adapters MAY add namespaced experimental capabilities, but product workflows use | Copy to new playlist | target search/get, playlist create, entries add | metadata update, duplicates preserve, revision read | | Strict mirror sync | read/revision on both, target add/remove/reorder or replace | push changes, conditional writes | | Add-only sync | read/revision on source, target search/get/add | push changes | +| Listen to selected media | configured source, explicit player/output, exact compatible media reference or supported source browser, `playback.media.start` | queue/seek/shuffle/repeat when supported | +| Show/control now playing | `playback.state.observe` and the selected transport actions | playlist/context/position/artwork only when observed | A workflow MUST fail during planning with a typed capability explanation if a required capability is absent. It must not discover this after creating a partial target where a preflight probe was possible. +Playback start is an interactive command, not a durable playlist mutation. Its response may be accepted before the target changes state, or may be lost after the target acts; the listening SDD owns pending/reconciliation behavior. It must not inherit the copy job's retry semantics. + ## Normalized provider ports Exact language signatures are deferred, but an adapter must cover these behaviors: @@ -176,6 +201,8 @@ For reconciliation, adapters and application ports MUST distinguish an effect-co - **SYM-PROV-018:** An external object identity MUST include its object type and MUST include a provider-instance or connection namespace whenever upstream IDs are not proven globally unique. - **SYM-PROV-019:** Planning MUST calculate effective capabilities from adapter, connection, object, and live-health constraints; a connection-wide capability MUST NOT override an object-level denial such as a read-only playlist. - **SYM-PROV-020:** Unofficial or `best_effort` status MUST be visible before authorization and again in any plan that depends on that adapter. +- **SYM-PROV-021:** A provider adapter that offers an imported item for playback MUST return a typed, exact reference compatible with the selected playback source, or `unsupported`/`unknown` with a reason; it MUST NOT construct a playback request from title similarity or unchecked URLs. +- **SYM-PROV-022:** Library and playback authorization/capabilities MUST be probed and disclosed independently; connecting a provider for import or playlist writes MUST NOT imply a Home Assistant integration, player, matching account, or playback permission. ## Normalized error taxonomy diff --git a/specs/CATALOG.md b/specs/CATALOG.md index c355e8b..0022c84 100644 --- a/specs/CATALOG.md +++ b/specs/CATALOG.md @@ -8,6 +8,7 @@ This is the human-readable view of [`catalog.json`](./catalog.json). Until gener | --- | --- | --- | --- | | `home-assistant-app-runtime` | Draft | [Home Assistant App runtime and Ingress](home-assistant-app-runtime-and-ingress.md) | Blocked by storage/recovery, supported platform matrix, and secret-key design | | `home-assistant-native-ui` | Ready for review | [Home Assistant-native UI foundation](home-assistant-native-ui.md) | Lit/TypeScript/Vite is accepted; blocked by supported matrix, public host-context/fallback proof, build/browser evidence, and visual-reference review | +| `listening-and-playback-control` | Draft | [Listening and playback control](listening-and-playback-control.md) | Narrow companion-broker authority accepted; blocked by broker protocol/security, source/player/output and account evidence, command confirmation, and YouTube Music support policy | | `provider-connections-and-authorization` | Draft | [Provider connections and authorization](provider-connections-and-authorization.md) | Blocked by direct callback reachability, secret/backup design, and provider feasibility spikes | | `library-import-and-provider-projections` | Draft | [Library import and provider projections](library-import-and-provider-projections.md) | Blocked by provider completeness, retention, and representative scale | | `recording-identity-resolution` | Draft | [Recording identity resolution](recording-identity-resolution.md) | Blocked by the labeled corpus and accepted automatic-link policy | diff --git a/specs/README.md b/specs/README.md index 731fdca..9811ae2 100644 --- a/specs/README.md +++ b/specs/README.md @@ -17,6 +17,7 @@ The existing `docs/` documents remain the horizontal sources of truth: | [`docs/providers/provider-specification.md`](../docs/providers/provider-specification.md) | Provider port and capability contract | | [`docs/providers/provider-research.md`](../docs/providers/provider-research.md) | Dated official API evidence | | [`docs/providers/home-assistant-ecosystem-review.md`](../docs/providers/home-assistant-ecosystem-review.md) | Dated implementation patterns and cautions | +| [`docs/providers/playback-integration-source-review.md`](../docs/providers/playback-integration-source-review.md) | Dated HA/MA/YT Music source-code behavior, operation limits and live proof matrix | | [`docs/decisions/README.md`](../docs/decisions/README.md) | Accepted architectural decisions | | [`docs/open-questions.md`](../docs/open-questions.md) | Unresolved owner choices and research gates | @@ -107,6 +108,7 @@ Readiness is necessary but not authorization to implement. The owner must still | --- | --- | --- | | Home Assistant App runtime | [App runtime and Ingress](home-assistant-app-runtime-and-ingress.md) | Packaging, lifecycle, authentication boundary, persistence, backup | | Home Assistant-native UI | [UI foundation](home-assistant-native-ui.md) | Shared component families, host context, accessibility, responsive behavior, catalog, visual compatibility | +| Listening and playback control | [Listening and playback control](listening-and-playback-control.md) | Explicit source/player/output selection, now playing, supported controls, and command confirmation without an audio engine | | Provider connections | [Provider connections and authorization](provider-connections-and-authorization.md) | OAuth/cookies, secrets, callback boundary, effective capabilities | | Provider imports | [Library import and provider projections](library-import-and-provider-projections.md) | Completeness, provenance, unavailable items, retention | | Recording identity | [Recording identity resolution](recording-identity-resolution.md) | Conservative matching, evidence, manual decisions | diff --git a/specs/catalog.json b/specs/catalog.json index c4a5eb7..07a84a3 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -30,6 +30,7 @@ "SYM-HA-007", "SYM-HA-008", "SYM-HA-009", + "SYM-HA-011", "SYM-TEST-009", "SYM-TEST-012", "SYM-TEST-013", @@ -46,6 +47,7 @@ ], "evidence": [ "docs/decisions/0003-home-assistant-app-primary.md", + "docs/decisions/0006-narrow-home-assistant-playback-broker.md", "docs/architecture/system-architecture.md", "docs/providers/provider-research.md", "docs/providers/home-assistant-ecosystem-review.md" @@ -109,6 +111,7 @@ "SYM-UI-013", "SYM-UI-014", "SYM-UI-015", + "SYM-UI-016", "SYM-JOB-007", "SYM-SEC-004", "SYM-SEC-008", @@ -141,6 +144,57 @@ "Accepted dated visual-reference capture/update procedure and reviewed visuals" ] }, + { + "id": "listening-and-playback-control", + "title": "Listening and playback control", + "status": "draft", + "scope": "Browse configured playback sources, choose an exact player/output, observe now playing, and control supported playback without owning audio streams", + "owner": "Symphonia maintainers", + "specification": "specs/listening-and-playback-control.md", + "requirements": [ + "SYM-PLAY-001", + "SYM-PLAY-002", + "SYM-PLAY-003", + "SYM-PLAY-004", + "SYM-PLAY-005", + "SYM-PLAY-006", + "SYM-PLAY-007", + "SYM-PLAY-008", + "SYM-PLAY-009", + "SYM-PROV-021", + "SYM-PROV-022", + "SYM-UI-016", + "SYM-HA-011", + "SYM-ARCH-016", + "SYM-ACC-005", + "SYM-SEC-004", + "SYM-SEC-008" + ], + "evidence": [ + "docs/decisions/0006-narrow-home-assistant-playback-broker.md", + "docs/product/product-specification.md", + "docs/product/home-assistant-ui-specification.md", + "docs/domain/domain-model.md", + "docs/providers/provider-specification.md", + "docs/providers/provider-research.md", + "docs/providers/home-assistant-ecosystem-review.md", + "docs/providers/playback-integration-source-review.md", + "docs/architecture/system-architecture.md", + "docs/open-questions.md" + ], + "implementationEvidence": { + "code": [], + "tests": [], + "documentation": [] + }, + "blockers": [ + "OQ-011 broker transport/pairing/identity and first playback source/output matrix; narrow-broker authority decision accepted", + "RG-007 Spotify idle-start/exact-output, MA caller/queue, account/source/player and exact playlist-reference feasibility", + "Threat review of broker pairing, versioning, replay, caller identity, and action/entity allowlist", + "Accepted observation freshness and command-reconciliation budgets", + "YouTube Music community-source support and risk decision" + ] + }, { "id": "provider-connections-and-authorization", "title": "Provider connections and authorization", diff --git a/specs/home-assistant-app-runtime-and-ingress.md b/specs/home-assistant-app-runtime-and-ingress.md index f1652f9..20e3b0f 100644 --- a/specs/home-assistant-app-runtime-and-ingress.md +++ b/specs/home-assistant-app-runtime-and-ingress.md @@ -1,12 +1,12 @@ # Home Assistant App runtime and Ingress - Status: Draft -- Date: 2026-09-20 +- Date: 2026-09-28 - Catalog capability ID: `home-assistant-app-runtime` - Owners: Symphonia maintainers - Scope: define the install, lifecycle, authentication, persistence, recovery, upgrade, backup, and diagnostics contract for the primary Supervisor-managed App. -- Related requirements: `SYM-PROD-001`, `SYM-ACC-005`, `SYM-HA-001`–`SYM-HA-009`, `SYM-DEP-001`–`SYM-DEP-010`, `SYM-SEC-008`–`SYM-SEC-011` -- Related decisions/research: [ADR 0003](../docs/decisions/0003-home-assistant-app-primary.md), [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [system architecture](../docs/architecture/system-architecture.md), [Home Assistant platform research](../docs/providers/provider-research.md#home-assistant-platform), [UI foundation](home-assistant-native-ui.md) +- Related requirements: `SYM-PROD-001`, `SYM-ACC-005`, `SYM-HA-001`–`SYM-HA-009`, `SYM-HA-011`, `SYM-DEP-001`–`SYM-DEP-010`, `SYM-SEC-008`–`SYM-SEC-011` +- Related decisions/research: [ADR 0003](../docs/decisions/0003-home-assistant-app-primary.md), [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [ADR 0006](../docs/decisions/0006-narrow-home-assistant-playback-broker.md), [system architecture](../docs/architecture/system-architecture.md), [Home Assistant platform research](../docs/providers/provider-research.md#home-assistant-platform), [UI foundation](home-assistant-native-ui.md) - Required review gates: product UX, architecture, Home Assistant platform, testing, documentation, security/operations - Open decisions blocking readiness: storage/recovery result from `RG-004`; supported Home Assistant versions and CPU architectures; encryption-key/backup contract; standalone release timing from `OQ-005` @@ -53,7 +53,7 @@ The current foundation is intentionally limited to reversible composition, persi | Home Assistant administrator | Install, configure, operate, upgrade, back up, and restore Symphonia | App repository and App page | App configuration, logs, Ingress panel, backup UI | | Symphonia user | Manage provider connections and music operations | Ingress panel | Symphonia UI and operation history | | Operator/contributor | Diagnose lifecycle or packaging faults | App logs/diagnostics and repository | Health/readiness, sanitized bundle, release notes | -| Companion integration | Expose future native HA surfaces | Versioned local API | Entities/actions/events, availability state | +| Companion integration | Broker HA playback when separately implemented; potentially expose later native surfaces | Versioned authenticated local contract | Listening availability, later entities/actions/events | **Health** means the process can answer a liveness probe. **Readiness** means configuration, storage, migrations, and recovery are safe enough to serve requests without claiming that every provider is online. **Ingress listener** is the trusted proxied management surface. **Direct listener** is any separately exposed callback or standalone endpoint and never inherits Ingress identity. @@ -71,7 +71,7 @@ The current foundation is intentionally limited to reversible composition, persi 1. Choosing the backend, frontend, database, init system, or image-build stack. 2. Defining provider authorization; that belongs to the provider-connections SDD. 3. Shipping standalone mode in the first release. -4. Exposing playback/media-player entities. +4. Defining listening behavior or creating Symphonia-owned playback/media-player entities; the separate [listening SDD](listening-and-playback-control.md) owns read/control of explicitly selected existing Home Assistant player entities through the [ADR 0006](../docs/decisions/0006-narrow-home-assistant-playback-broker.md) companion broker. Broker pairing and operation contracts are not part of this runtime foundation and remain gated there. ### 4.3 Fixed invariants @@ -127,7 +127,7 @@ Text equivalent: startup validates configuration, migrates, and recovers before ### 6.2 Alternative and boundary paths - A provider outage degrades only that connection/capability; it does not make local history unavailable. -- Home Assistant Core or the optional integration may be unavailable while the App continues safe provider work. +- Home Assistant Core or the playback companion may be unavailable while the App continues safe provider work; only listening is disabled. - A migration or restore incompatibility keeps the service `blocked`; no fresh empty database replaces user-authored state. - An unsupported downgrade is detected before workers start. - A callback listener, if selected by the authorization SDD, exposes only its bounded callback routes. @@ -169,7 +169,7 @@ The current foundation profile applies this boundary before opening stores or cr | HTTP/Ingress presentation | Base-path routing, trusted identity adaptation, status views | Provider/domain decisions | | Persistence/job adapters | Schema, transactions, leases, recovery | Presentation or HA identity | | Composition | Concrete profile and startup/shutdown order | New business rules | -| Companion integration | Versioned local client and native projections | Database/token-store access or duplicate orchestration | +| Companion integration | Versioned authenticated playback broker; optional native projections | Database/token-store access, broad App-to-Core authority, or duplicate orchestration | ### 8.2 Contracts, durable state, and trust boundaries @@ -289,7 +289,7 @@ The eight feature-specific UI cases above supplement rather than replace the 84- 4. Given a migration failure, the prior database remains recoverable and no fresh rescan replaces user-authored state. 5. Given a backup request with active work, the App produces a transactionally consistent backup or refuses it explicitly. 6. Given any direct callback port, management routes and Ingress identity headers are unusable on that listener. -7. Given Home Assistant integration unavailability, local App history remains readable and eligible provider jobs are not corrupted. +7. Given Home Assistant playback-broker unavailability or version mismatch, listening is unavailable without fallback to broad Core API access; local App history remains readable and eligible provider jobs are not corrupted. 8. Given an unsupported downgrade, the App blocks before writes and points to compatible restore/upgrade guidance. 9. Given hostile restored/config/diagnostic values, no path escape, arbitrary URL, secret output, or code execution occurs. 10. Given Supervisor stop, new admissions stop and shutdown leaves every lease recoverable within the bounded time. @@ -301,6 +301,7 @@ The eight feature-specific UI cases above supplement rather than replace the 84- | --- | --- | --- | --- | | `SYM-HA-001`–`SYM-HA-003` | App composition/platform adapter | artifact + disposable HA smoke | install/upgrade/backup guides | | `SYM-HA-004` | dependency rule | static architecture test | contributor architecture | +| `SYM-HA-011` | optional broker lifecycle isolation | unavailable/incompatible broker and no-Core-proxy fixture; scenario 7 | setup/operator guide | | `SYM-SEC-008`–`SYM-SEC-011` | listeners/image composition | route/port/privilege abuse tests | security/setup guide | | `SYM-ARCH-003`, `SYM-ARCH-006`, `SYM-ARCH-007` | migration/backup use cases | release-fixture migration + restore tests | recovery guide | | `SYM-DEP-001`–`SYM-DEP-010` | release artifact | metadata/package/smoke matrix | release support policy | @@ -330,8 +331,8 @@ The eight feature-specific UI cases above supplement rather than replace the 84- ## 20. References and decisions - Primary sources: Home Assistant App/Ingress/security documentation linked from [provider research](../docs/providers/provider-research.md#home-assistant-platform). -- Related SDDs: [provider authorization](provider-connections-and-authorization.md), [durable operations](durable-operations-and-recovery.md). +- Related SDDs: [provider authorization](provider-connections-and-authorization.md), [durable operations](durable-operations-and-recovery.md), [listening and playback control](listening-and-playback-control.md). - Related UI contract: [Home Assistant-native UI foundation](home-assistant-native-ui.md) and [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md). - Accepted: App is the primary deployment boundary; core remains HA-independent; UI follows the owned Home Assistant-native compatibility layer. - Rejected: all logic in a custom integration; unauthenticated management port; fresh database fallback after migration failure. -- Follow-up: standalone release and companion integration native surface. +- Follow-up: standalone release and native projections beyond the selected playback broker; the broker's own contract is specified by the listening SDD. diff --git a/specs/home-assistant-native-ui.md b/specs/home-assistant-native-ui.md index d7a810c..1571c03 100644 --- a/specs/home-assistant-native-ui.md +++ b/specs/home-assistant-native-ui.md @@ -1,11 +1,11 @@ # Home Assistant-native UI foundation - Status: Ready for review -- Date: 2026-09-27 +- Date: 2026-09-28 - Catalog capability ID: `home-assistant-native-ui` - Owners: Symphonia maintainers - Scope: establish the shared shell, component families, semantic tokens, host-context adaptation, accessibility, responsive behavior, component catalog, and visual compatibility evidence for every Symphonia web view. -- Related requirements: `SYM-UI-001`–`SYM-UI-015`, `SYM-JOB-007`, `SYM-SEC-004`, `SYM-SEC-008`, `SYM-TEST-005`, `SYM-TEST-009`, `SYM-TEST-014`, `SYM-TEST-015` +- Related requirements: `SYM-UI-001`–`SYM-UI-016`, `SYM-JOB-007`, `SYM-SEC-004`, `SYM-SEC-008`, `SYM-TEST-005`, `SYM-TEST-009`, `SYM-TEST-014`, `SYM-TEST-015` - Related decisions/research: [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md), [ADR 0005](../docs/decisions/0005-lit-typescript-vite-ui.md), [UI specification](../docs/product/home-assistant-ui-specification.md), [official platform research](../docs/providers/provider-research.md#home-assistant-platform), [Gateway/HA UI evidence](../docs/providers/home-assistant-ecosystem-review.md#home-assistant-native-ui-lessons-from-homeassistant-gateway), [RG-006 host-context evidence](../docs/development/ui-spike/host-context-evidence.md) - Required review gates: product UX, Home Assistant platform, frontend architecture, accessibility, localization, testing/visual QA, documentation, security/privacy - Open decisions blocking implementation readiness: supported Home Assistant/browser matrix; verified public host-context/Ingress contract and deterministic fallbacks across that matrix; Lit/Vite compatibility and bundle evidence; accepted visual-reference capture/update procedure and visual/accessibility review @@ -49,7 +49,7 @@ No complete Symphonia UI or UI package exists. The repository has an experimenta | Actor | Goal | Entry point | Visible surfaces | | --- | --- | --- | --- | -| Home Assistant administrator/user | Use Symphonia without learning an alien UI | Ingress panel | shell, navigation, connections, library, matching, copy, operations, diagnostics | +| Home Assistant administrator/user | Use Symphonia without learning an alien UI | Ingress panel | shell, navigation, listening/now playing, connections, library, matching, copy, operations, diagnostics | | Standalone user, if released | Use the same product semantics outside HA | authenticated standalone URL | same feature views with fallback host framing | | Product/accessibility reviewer | Inspect every state before feature integration | component catalog and reference manifest | variants, themes, focus, viewports, long text, failures | | Contributor | Compose consistent views without duplicating presentation behavior | public UI package entry point | tokens, components, layouts, stories/fixtures | @@ -190,6 +190,7 @@ Users cannot configure away visible focus, semantic errors, required confirmatio - Route tests enumerate non-root base paths, refresh, deep links, assets, HTTP, and push transports. - Context tests validate allowed origin/message/schema and fallback behavior; unsupported fields remain inert. - Component semantics are exercised by interaction/accessibility tests, not source-string assertions alone. +- The shared now-playing/transport family is presentation-only: tests reject service dispatch, capability inference, or player-selection policy inside the component package. Feature fixtures cover stale, missing-media, multi-player, pending, unsupported, and uncertain states. ## 9. UI/UX and content contract @@ -234,6 +235,12 @@ Copy completed with omissions Primary action: Review result ``` +```text +Now playing · Kitchen speaker +Playing: Track A — Artist B. Last reported 12 seconds ago. +Pause and Next are available; unsupported controls explain why they are disabled. +``` + ### 9.3 Accessibility and localization - Locale/direction/timezone use validated public HA context with region-to-language-to-English fallback. @@ -281,16 +288,16 @@ Primary action: Review result ## 14. Testing strategy and numeric budget -Minimum **84 distinct cases**: +Minimum **92 distinct cases**: | Area | Minimum distinct cases | Behaviors/risks covered | | --- | ---: | --- | | Tokens/context/configuration/pure policy | 14 | host/fallback theme, locale, RTL, safe area, invalid context, token mapping, preferences | -| Component interaction and accessibility | 24 | every required family; keyboard/focus/ARIA; loading/disabled/error/destructive; announcements | +| Component interaction and accessibility | 28 | every required family including now-playing/transport; keyboard/focus/ARIA; loading/disabled/error/destructive; announcements | | Responsive/theme/locale geometry | 16 | light/dark/contrast/reduced motion, phone/tablet/desktop, zoom, long/RTL text, overflow | -| Shell/feature/Ingress/security contracts | 18 | arbitrary base path, deep link/refresh/assets/push, stale/cancelled requests, hostile content, secret canaries | +| Shell/feature/Ingress/security contracts | 22 | arbitrary base path, deep link/refresh/assets/push, stale/cancelled requests, listening-state freshness/target fencing, hostile content, secret canaries | | Visual/reference/compatibility/release | 12 | dated HA comparisons, intentional divergence, snapshot review, supported host/browser upgrade/rollback | -| **Total** | **84** | No double counting | +| **Total** | **92** | No double counting | Pure tests use fixed context messages, tokens, locales, translations, routes, and view models. Component/browser fixtures use no network and synthetic public-safe data. Browser behavior runs across the supported engine matrix; canonical pixel baselines may use one declared engine, while every engine must pass semantics and geometry contracts. @@ -322,6 +329,7 @@ Required human evidence reviews official Home Assistant references versus the ca 12. Given a visual snapshot change, release evidence identifies the dated Home Assistant reference and records intended parity, accessibility/product divergence, or regression correction before approval. 13. Given a Home Assistant frontend/component migration, compatibility tests prove independent fallback and the supported matrix/reference manifest are updated before release. 14. Given a browser action whose response is lost, the UI reloads authoritative server state and never infers success or blindly repeats an irreversible operation. +15. Given a selected player with rapidly changing progress or a track change, the shared now-playing family preserves focus, avoids per-tick announcements, displays observation freshness and current capability reasons, and never dispatches Home Assistant calls itself. ## 17. Requirements traceability @@ -334,6 +342,7 @@ Required human evidence reviews official Home Assistant references versus the ca | `SYM-UI-009` | presentation/output boundaries | hostile-content and secret-canary suite | security/content guide | | `SYM-UI-010`–`SYM-UI-011`, `SYM-UI-015` | catalog/reference/release tooling | catalog completeness, visual review, HA/browser compatibility matrix | reference manifest/runbook | | `SYM-UI-013` | composition profiles | Ingress versus standalone fixture parity | deployment guides | +| `SYM-UI-016` | now-playing/transport family and listening feature composition | component catalog plus stale/multi-player/announcement fixtures; scenario 15 | listener/accessibility guide | | `SYM-TEST-005`, `SYM-TEST-009`, `SYM-TEST-014`, `SYM-TEST-015` | verification tooling | release gates | contributor testing guide | ## 18. Implementation sequence @@ -350,7 +359,7 @@ Required human evidence reviews official Home Assistant references versus the ca - [ ] Status is `Ready for implementation` and explicit owner approval to implement exists. - [ ] Supported HA/browser matrix, Lit/Vite compatibility/build evidence, context/safe-area/fallbacks, and reference procedures are accepted. - [ ] Every `SYM-UI-*` requirement maps to acceptance and deterministic or explicit human evidence. -- [ ] At least 84 distinct cases and architecture/lint/secret gates pass. +- [ ] At least 92 distinct cases and architecture/lint/secret gates pass. - [ ] Every required component family is available only through the public package and complete in the catalog. - [ ] Representative feature states pass light/dark, contrast, reduced-motion, keyboard, screen-reader, narrow/wide, long/RTL text, and sanitization review. - [ ] Ingress prefix, deep links, assets, reconnect/push, safe area, and standalone fallback are proven. @@ -362,7 +371,7 @@ Required human evidence reviews official Home Assistant references versus the ca - Primary sources: official Home Assistant links in the [UI specification](../docs/product/home-assistant-ui-specification.md) and [platform research](../docs/providers/provider-research.md#home-assistant-platform). - Community evidence: pinned `vypdev/homeassistant-gateway` sources listed in the UI specification and ecosystem review. -- Related SDDs: [App runtime/Ingress](home-assistant-app-runtime-and-ingress.md), [provider connections](provider-connections-and-authorization.md), [imports](library-import-and-provider-projections.md), [identity resolution](recording-identity-resolution.md), [playlist copy](one-time-playlist-copy.md), [durable operations](durable-operations-and-recovery.md). +- Related SDDs: [App runtime/Ingress](home-assistant-app-runtime-and-ingress.md), [provider connections](provider-connections-and-authorization.md), [listening](listening-and-playback-control.md), [imports](library-import-and-provider-projections.md), [identity resolution](recording-identity-resolution.md), [playlist copy](one-time-playlist-copy.md), [durable operations](durable-operations-and-recovery.md). - Accepted: Home Assistant-native-adjacent direction, owned compatibility layer, no private HA frontend runtime dependency, shared Ingress/standalone semantics, Lit + TypeScript + Vite direction (ADR 0005), dated catalog/reference evidence. - Open: supported matrix, verified public App context and fallbacks, Lit/Vite bundle/Ingress evidence, reference capture/update automation and visual review. - Rejected: generic SaaS dashboard, private HA component imports by default, frozen pixel copy, decorative ambient UI, feature-local duplicate primitives. diff --git a/specs/listening-and-playback-control.md b/specs/listening-and-playback-control.md new file mode 100644 index 0000000..08dbd79 --- /dev/null +++ b/specs/listening-and-playback-control.md @@ -0,0 +1,372 @@ +# Listening and playback control + +- Status: Draft +- Date: 2026-09-28 +- Catalog capability ID: `listening-and-playback-control` +- Owners: Symphonia maintainers +- Scope: browse playable content, choose an explicit playback source and output, observe now-playing state, and control a configured external player from the Home Assistant-integrated Symphonia UI. +- Related requirements: `SYM-PLAY-001`–`SYM-PLAY-009`, `SYM-PROV-021`–`SYM-PROV-022`, `SYM-UI-016`, `SYM-HA-011`, `SYM-ARCH-016`, `SYM-ACC-005`, `SYM-UI-001`–`SYM-UI-015`, `SYM-SEC-004`, `SYM-SEC-008` +- Related decisions/research: [product specification](../docs/product/product-specification.md), [domain model](../docs/domain/domain-model.md), [provider specification](../docs/providers/provider-specification.md), [system architecture](../docs/architecture/system-architecture.md), [ADR 0006](../docs/decisions/0006-narrow-home-assistant-playback-broker.md), [official playback research](../docs/providers/provider-research.md#2026-09-28-playback-and-home-assistant-api-update), [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md), [integration source review](../docs/providers/playback-integration-source-review.md), [UI foundation](home-assistant-native-ui.md), [App runtime](home-assistant-app-runtime-and-ingress.md), `OQ-011`, `RG-007` +- Required review gates: product UX, Home Assistant platform/API, provider feasibility, architecture, accessibility, testing, documentation, security/privacy +- Open decisions blocking readiness: `OQ-011` supported playback-source matrix and broker transport/pairing/identity contract (the narrow-broker authority choice is accepted); `RG-007` Spotify idle-start/exact-device feasibility, account/source/player binding, MA caller identity and playlist-reference evidence; broker threat/version/upgrade review; live-state freshness and command-reconciliation thresholds; first-release Music Assistant/community policy + +## 1. Executive summary + +Symphonia should feel like a focused Home Assistant music surface: after setting up a library provider and a separate compatible playback source, a user can choose what to hear and where, see what that selected player reports now, and operate only controls it supports. The same App retains Symphonia's higher-assurance playlist import, identity resolution, and copy workflows. [ADR 0006](../docs/decisions/0006-narrow-home-assistant-playback-broker.md) selects a narrow companion integration that alone reads/commands explicitly allowed Home Assistant `media_player` entities; the App uses a typed broker port and does not stream or decode audio. + +The primary correctness rule is **no inferred authority**: an imported Spotify playlist, a Home Assistant Spotify entity, a Music Assistant source, and a speaker may refer to different accounts or devices. A title match, shared provider name, or successful action request never proves the intended media is playing on the intended output. + +```text +Configure library connection (optional for listening) + configure HA playback source +-> explicitly link source/account and select target/output +-> browse supported media or exact compatible imported playlist +-> confirm replacement target -> send one bounded command +-> observe player state -> show confirmed, pending, or uncertain result +``` + +## 2. Problem, current behavior, and evidence + +### 2.1 Problem + +A user may connect providers and manage playlists but still need to leave Symphonia to listen. Treating library connection as playback authorization would leak authority and promise controls or account coverage that the underlying Home Assistant integration does not expose. Polling and remote device behavior also make accepted commands distinct from confirmed playback. + +### 2.2 Current behavior + +No Symphonia listening route, playback port, companion playback broker, player discovery, or now-playing implementation exists. The prior product specification explicitly excluded playback and the App-runtime SDD excluded exposing playback entities. This SDD changes the proposed product boundary; it is **Draft** and creates no production implementation permission. Existing import/copy foundations and the synthetic UI spike are not playback evidence. + +### 2.3 Evidence and unknowns + +- Repository decisions: App + Ingress, independent Home Assistant-native UI, and provider-neutral core remain accepted; see [ADRs](../docs/decisions/README.md). Provider research keeps official API claims separate from [community implementation evidence](../docs/providers/home-assistant-ecosystem-review.md). +- Official Home Assistant sources reviewed 2026-09-28: [Spotify integration](https://www.home-assistant.io/integrations/spotify/) exposes playback/browsing, source selection and playlist `play_media`, requires Premium and a known compatible device, and polls at least every 30 seconds; [Music Assistant integration](https://www.home-assistant.io/integrations/music_assistant/) exposes player entities/current media and controls; [media-player entity contract](https://developers.home-assistant.io/docs/core/entity/media-player/) defines optional state fields and feature flags; [App communication](https://developers.home-assistant.io/docs/apps/communication/) documents the Core API proxy and `SUPERVISOR_TOKEN`; [WebSocket API](https://developers.home-assistant.io/docs/api/websocket/) documents state-change events and service calls. +- Official Spotify source: [currently playing](https://developer.spotify.com/documentation/web-api/reference/get-the-users-currently-playing-track) and [start/resume playback](https://developer.spotify.com/documentation/web-api/reference/start-a-users-playback) demonstrate an alternative direct adapter, but require separate scopes/account/device/policy review; no direct Spotify adapter is selected here. +- Community/project evidence: [Music Assistant's YouTube Music source](https://www.music-assistant.io/music-providers/youtube-music/) uses unofficial access, cookies and a PO-token service; [`ytube_music_player`](https://github.com/KoljaWindeler/ytube_music_player/blob/main/README.md) is another community playback path. Neither establishes official Google support or universal observation of the native YouTube Music app's external sessions. +- The dated [integration source review](../docs/providers/playback-integration-source-review.md) identifies specific stock-HA constraints: Spotify's idle/restricted state removes `PLAY_MEDIA` and WebSocket browse; device selection is name-based and can be ambiguous; MA's active source is not always an MA queue, its broad feature flags do not prove effect, its name fallback is unsafe for exact playback, and its per-user behavior depends on the HA caller context. These findings narrow the `RG-007` live probes; none is a release guarantee on unpinned versions. +- Accepted authority boundary: the companion broker, not the broad Supervisor Core proxy, under ADR 0006. Unknowns: broker transport/pairing/version/caller contract and threat review; how HA entity identity/account is verified; exact imported-playlist-to-source mapping; target/source behavior across Spotify, Music Assistant and community players; private/session metadata behavior; timing, action error, and coexistence with other HA clients on target hardware. + +## 3. Actors, surfaces, and terminology + +| Actor | Goal | Entry point | Visible surfaces | +| --- | --- | --- | --- | +| Home Assistant administrator | Configure/authorize playback bridge and link sources/targets | Ingress setup | permissions, integrations, bindings, supported/unsupported reasons | +| Listener | Choose music/output and control the selected session | Ingress Listen route | browse/search when exposed, now playing, source/output picker, transport | +| Home Assistant companion broker | Observe allowed Core entities and dispatch typed, bounded actions | authenticated versioned broker channel | setup/health, never raw Core API in browser | +| Home Assistant Core/player integration | Report entity state and accept supported actions | broker only | none directly in browser | +| Operator | Diagnose absent/stale/ambiguous playback without exposing accounts | diagnostics | sanitized integration health and command category | + +**Playback source** identifies an approved configured service/account, not necessarily an imported provider connection. **Player target** is one exact entity or endpoint; **output** is an integration-specific destination. **Observation** is an ephemeral, timestamped report from the player, not a canonical library fact. **Binding** is an explicit relation between a source/player and an optional library connection, with verified or visibly unverified account evidence. **Command result** is `pending`, `confirmed`, `rejected`, or `uncertain` based on observation, not merely transport acceptance. + +## 4. Goals, non-goals, and fixed invariants + +### 4.1 Goals + +1. Make selecting media and output, now-playing display, and available controls usable in a familiar Home Assistant-like view. +2. Preserve Symphonia's playlist-management experience alongside listening, without conflating playback queue changes with durable playlist edits. +3. Support configured Spotify HA and, subject to independent evidence/policy, Music Assistant or community sources such as YouTube Music through the same capability-driven UI. +4. Degrade playback independently of the library/import/copy system. + +### 4.2 Non-goals + +1. Streaming, downloading, decoding, transcoding, or mixing audio in Symphonia. +2. Universal observation/control of arbitrary provider apps, browsers, devices, or accounts outside the selected integration. +3. Building a second Music Assistant server, a general Home Assistant API browser, or a generic service-call console. +4. Queue editing, multi-room synchronization, cross-service handoff, and playback history/scrobbling in the first vertical slice. Queue read and richer controls require their own verified capability and scope decision. +5. Changing the playlist-copy safety/confirmation contract; starting a playlist changes playback only, not the provider playlist. + +### 4.3 Fixed invariants + +1. No player/output is selected solely from a provider type, mutable display name, last active device, or imported playlist. +2. `unknown`, stale, unsupported, or missing effective capability never enables a command. A feature bit is necessary but not sufficient for a source-specific media reference. +3. A selectable output is not a verified output identity: ambiguous names, idle first-device fallback, stale device lists, or an unobserved transfer block an exact-output claim. A profile that cannot meet “play here” remains read-only/limited or unavailable for that action until a separately approved path exists. +4. A transport-accepted command is not shown as effect-confirmed. Ambiguous outcomes are read/reconciled, never blindly retried. +5. No browser-side Core/Supervisor token, arbitrary Home Assistant service/domain, arbitrary media URL, local path, or provider refresh grant is exposed. MA's title-search fallback is not an exact reference. +6. Library data and copy state remain usable when the playback bridge is offline, and listening observations never silently update recording identity or imported snapshots. +7. Unofficial YouTube Music paths remain opt-in and visibly unofficial/best-effort; an official YouTube Data connection is not relabeled YouTube Music playback. +8. A server-side App action is not assumed to carry the Ingress listener's HA/MA identity. Private browsing or per-user playback is blocked until caller attribution and account authorization are proven. + +## 5. Product journey + +| Stage | Proposed behavior | User/operator effect | +| --- | --- | --- | +| Set up | Show library connection and playback-source setup separately; explain extra HA/MA integration and permission | No false “connected means playable” promise | +| Bind | Choose an allowed source/account and exact HA entity; show verified or unverified association | Wrong account/player cannot be hidden behind a provider logo | +| Select | Pick content from supported source browser or an exact compatible imported item; pick explicit player/output | User sees target and media before replacing playback | +| Start | Issue one command and show pending, then observe target state | Accepted request is not mislabeled success | +| Control | Show current metadata/freshness and controls the player currently supports | Pause, stop, skip, seek or output changes have accurate meanings | +| Recover | Explain stale/offline/unsupported/ambiguous states and safe next action | Library/playlist work remains available | + +```mermaid +flowchart LR + A[Configured playback source] --> B[Explicit player/output selection] + B --> C[Browse or exact media reference] + C --> D[Confirm replacement and send command] + D --> E[Observe target state] + E --> F[Confirmed, rejected or uncertain] +``` + +Text equivalent: a separately configured source and explicitly chosen target precede content selection; a bounded command is sent only after target/content confirmation, then observation determines its visible result. + +## 6. Functional behavior and state model + +### 6.1 Happy path + +1. Administrator installs/pairs the narrow broker and selects allowed `media_player` entities; Symphonia discovers their state and effective features through the broker without holding a broad Core credential or putting broker credentials in the browser. +2. Listener opens Listen, selects an approved source and one target/output. If several sessions are active, the UI lists them instead of silently electing one. +3. The source presents playable tracks, albums, and playlists through supported browse/search capabilities, plus compatible imported playlists where an exact reference exists. Exact playback references are adapter-validated. A missing reference offers the source's own browser or a clear unsupported explanation. A browser route must pass its own HA feature/permission check; a method existing in an integration is insufficient. +4. User selects content, sees the target/current-playback replacement warning, and starts it **only if** that source/state/output profile is proven eligible. Command status stays pending until a bounded observation confirms the target's state, media and required output or becomes uncertain. +5. Now playing shows only reported fields: state, title, artist, artwork, position, context/playlist, target/source, and last observation. Transport/output actions depend on current effective capability and selected player. + +### 6.2 Alternative and boundary paths + +- A Symphonia provider connection exists but no HA playback source: library works; Listen shows setup guidance, not a fake player. +- HA Spotify is configured but has no known Spotify Connect device: now-playing may be idle; start/output explains the missing device. While idle or on a restricted device, stock HA Spotify advertises only source selection, not normal `play_media` or WebSocket browse. A separately approved browse-service route is unproven; no idle-start workaround is assumed. +- HA Spotify lists duplicate device names or an output selected by name cannot be freshly verified: keep “play here” disabled, even if `SELECT_SOURCE` is advertised. Source selection/transfer can alter an existing session and requires an explicit warning and effect observation. +- MA reports an external active source with current media but no active queue: display that media without fabricating queue controls, next item or playlist context. A base HA feature flag alone does not enable queue-only actions. +- MA is invoked via a server-side Core bridge: verify the actual HA/MA caller mapping before showing private source results or attributing a play request to the listener; absent evidence, constrain to a configured single-admin/default-account profile or keep the path unavailable. +- Music Assistant has a YouTube Music source but no playable target or expired cookie/PO-token service: show source unavailable with its unofficial status; do not fall back to Google/YouTube Data or ask Symphonia for cookies. +- A community entity may report only sessions it manages: show its verified coverage limitation; do not claim visibility into playback begun in the official YouTube Music mobile app without specific evidence. +- An imported private playlist belongs to an unverified/different account: no automatic `play_media` from that import. Use the configured source browser or explicit user-approved compatible reference only after evidence. +- Two active players: show both, with one explicitly selected. Commands never target an area, device group, wildcard, or every `media_player` implicitly. +- A player reports `playing` but omits title/playlist/artwork: preserve `playing` and label missing metadata; never substitute last imported or last commanded item. +- A source becomes unavailable or HA disconnects: last observation is labeled stale, controls disable, bounded reconnect/read is attempted; no command queue is replayed when it returns. +- A request is accepted but observation does not match before the confirmation budget: mark `uncertain` and offer refresh/inspect in HA, not an automatic repeat. + +### 6.3 State machine + +| State | Entered when | User-visible meaning | Allowed next states | Recovery/owner | +| --- | --- | --- | --- | --- | +| `unconfigured` | no approved source/bridge | Listening needs setup; library unaffected | `idle`, `unavailable` | administrator | +| `no_target` | source exists, no explicit eligible player/output | Choose where to listen | `idle`, `unavailable` | listener | +| `idle` | fresh player report says idle/off or no active media | No current song on selected player | `pending`, `playing`, `paused`, `unavailable` | listener/external controller | +| `pending` | one authorized command awaits observed effect | Requested action not yet confirmed | `playing`, `paused`, `idle`, `uncertain`, `rejected`, `unavailable` | bounded observation | +| `playing` | fresh selected-player report says playing | Current reported media/controls available | `pending`, `paused`, `idle`, `stale`, `unavailable` | listener/external controller | +| `paused` | fresh selected-player report says paused | Reported media retained, if provided | `pending`, `playing`, `idle`, `stale`, `unavailable` | listener/external controller | +| `stale` | prior observation exceeds accepted freshness or stream lost | Last known state, not current fact; controls disabled | `idle`, `playing`, `paused`, `unavailable` | bounded refresh | +| `unavailable` | HA/source/player unavailable or permission denied | Playback unavailable; library continues | `idle`, `playing`, `paused`, `unconfigured` | reconnect/setup/operator | +| `rejected` | HA/integration explicitly refuses command | Nothing claimed to have changed | `idle`, `playing`, `paused`, `stale` | listener fixes target/permission | +| `uncertain` | command outcome or target effect cannot be confirmed | Do not repeat automatically | `idle`, `playing`, `paused`, `stale` after authoritative read | listener/refresh | + +State reports and command results are separate dimensions: a later external action may change `playing` to `paused` without a Symphonia command. Out-of-order observations and old command responses are fenced by target, observation time/version, and command correlation. Player selection changes cancel the prior UI pending view but do not cancel an already accepted external effect. + +## 7. Configuration contract + +| Input | Type | Recommended default | Allowed values/range | Scope/persistence | +| --- | --- | --- | --- | --- | +| Playback broker | versioned, paired companion integration | disabled until pairing/threat review pass | one accepted authenticated broker profile; no Core proxy fallback | App/integration configuration | +| Allowed players | exact entity identifiers | none | discovered, validated `media_player` entities only; explicit selection | App configuration, backed up without HA token | +| Source/connection binding | typed identity and evidence | unbound | verified account match or visibly unverified user assertion | local configuration; cleared/reviewed on disconnect | +| Preferred target | exact allowed player/output reference | none | one existing eligible target; no wildcard/area default | user preference; invalidated when entity/source disappears | +| Observation/command budget | bounded policy | conservative fixed defaults after `RG-007` | documented freshness/timeout ranges, never unbounded rapid polling | deployment policy; no per-request override | + +Migration from an earlier install adds no automatic binding or allowed target. If an HA entity is renamed/recreated, the stored selection becomes action-required rather than silently targeting a different entity. Security authorization, allowlisted actions, and secret exposure are not configurable downward. + +## 8. Clean Architecture design + +### 8.1 Responsibilities and dependency direction + +| Boundary | Owns | Must not own/import | +| --- | --- | --- | +| Domain/pure policy | target/source identity, freshness, capability intersection, command eligibility | HA constants/SDKs, provider DTOs, audio streams | +| Application | list targets/sources, bind, browse, observe, command, reconcile | direct HA/Spotify calls, UI layout | +| App playback adapter | broker DTO mapping, source-specific state/feature/media validation | provider library/copy policy, Core credentials, generic Core proxy | +| Companion broker | HA entity observation and typed, exact-target Core action dispatch | Symphonia database/grants, arbitrary HA service proxy, music-domain policy | +| Infrastructure/composition | broker pairing, versioning, allowlist synchronization, bounded transport/subscriptions | implicit user account equivalence or broad App-to-Core authority | +| Presentation | listening view model, confirmation, pending/stale/uncertain feedback | token access, raw service dispatch, authority decisions | + +### 8.2 Contracts, durable state, and trust boundaries + +- Pure decisions: eligible target and source, playable reference validation, current action intersection, freshness, observed-effect matching, safe recovery choice. +- Effective feature policy is integration-specific: HA Spotify active versus idle/restricted, device-name uniqueness and freshness; MA queue versus external source, player capability and caller identity; optional community target-player capability. No generic HA bitmask alone proves the media/output effect. +- Application contracts: discover/select/bind source and target; browse supported media; read observation; issue one typed command with correlation; reconcile once/boundedly. Broker protocol/auth/identity and reconnect semantics remain a design gate, not a guessed WebSocket/MQTT choice. +- Semantic ports: player catalog, playback-source browser, exact-reference resolver, state observer, command dispatcher, clock, policy/permissions, sanitized audit. +- Durable state/schema ownership: only approved player allowlist, explicit binding/preference, and minimal command audit/correlation where required; now-playing media/title/artwork/position and full listening history are ephemeral by default. No operation-job checkpoint is created for ordinary transport control. +- Concurrency/idempotency: at most one unresolved command per selected player from Symphonia; external HA controllers can race; lost responses trigger observation, not automatic replay. A newly selected player fences old events/UI requests. Command audit distinguishes requested, accepted, observed, rejected, and inconclusive. +- Trusted/untrusted inputs: entity IDs, HA state/attributes, media-browser references, artwork URLs, remote source names, imported playlist IDs, and service errors are untrusted. Only adapter-owned validated mappings cross the Core boundary; MA's generic `play_media` acceptance of search text, URLs or local paths cannot broaden it. +- Error mapping: source absent, target absent, account unverified, permission denied, unsupported capability, no compatible device, stale state, integration unavailable, rate limit, and uncertain effect have distinct safe categories. + +### 8.3 Executable architecture constraints + +- Static tests reject HA/Supervisor and playback SDK imports in domain/application and Core-token/API imports in browser/presentation. +- Contract tests enumerate every broker read/action and reject arbitrary HA service domains, wildcard targets, unapproved entities, unchecked media URLs, stale feature claims, missing/rotated credentials, incompatible versions, and reconnect replay. +- Binding tests prove names/provider kinds cannot establish account or target identity. +- Command tests prove accepted response is not success, lost response is not replayed, and stale/out-of-order state cannot confirm the wrong target/content. + +## 9. UI/UX and content contract + +The Listen route composes the [Home Assistant-native UI foundation](home-assistant-native-ui.md): existing shell/navigation, cards, settings rows, selectors, dialogs, status/alert, loading/empty, and the shared now-playing/transport family from the [UI specification](../docs/product/home-assistant-ui-specification.md). Provider artwork identifies content; Home Assistant interaction patterns remain primary. This SDD owns content/state/actions, not a parallel design system. + +### 9.1 Information hierarchy + +1. Selected source/account and player/output, with binding verification and integration support label. +2. Current reported player state, track/artist/artwork/progress when available, and observation age. +3. Playable browse/search/library content with exact compatibility/availability explanation. +4. Only currently eligible controls; disabled reasons and pending/uncertain command result. +5. Setup/recovery action and retained library/playlist-management availability. + +### 9.2 Representative states + +```text +Ready to listen +Spotify library is connected. Playback needs a Home Assistant Spotify source and an output. +Action: Set up playback. Your playlists remain available to manage. +``` + +```text +Now playing · Living room · Spotify +Playing: Track A — Artist B. Reported by Home Assistant 12 seconds ago. +Pause and Next are available. Stop and output transfer are not available on this player. +``` + +```text +Starting playlist · Kitchen speaker +Request accepted; waiting for the player to report the new playlist. +Do not press Play again while we confirm the result. +``` + +```text +Playback could not be confirmed +The command may have reached the selected player, but Home Assistant has not confirmed the effect. +Action: Refresh player state or inspect it in Home Assistant. Your playlist was not edited. +``` + +```text +YouTube Music source unavailable · unofficial/best-effort +Music Assistant cannot access this source right now. Its login or token service may need attention. +Action: Check Music Assistant. Imported playlists and Symphonia history remain available. +``` + +### 9.3 Accessibility and localization + +- Locale/terminology use the UI foundation's validated host/browser fallback; `source`, `player`, and `output` receive distinct translated labels. +- Source/target selectors, browse results, transport icon buttons, confirmation, and error recovery have keyboard support and accessible names/states. +- Passive position ticks are not live-announced; track changes, command outcomes, target changes, and important failures have bounded announcements without focus theft. Dialog focus returns to the invoking control. +- Narrow/mobile and embedded Ingress heights preserve selected target, current track/state, and controls before secondary metadata; no hover-only commands. +- All provider names, artwork URLs, track metadata, playlist names, and HA attributes are bounded, escaped, bidi-safe, and URL-scheme-checked. + +## 10. Failure, recovery, and cleanup + +| Failure/partial state | User impact | Retained facts | Automatic retry | Required action | Cleanup | +| --- | --- | --- | --- | --- | --- | +| HA/companion broker unavailable or incompatible | stale/unavailable player | library and last labeled observation | bounded authenticated read reconnect only | inspect HA/broker if persistent | discard pending transport request; no replay | +| Source/account mismatch | imported item not startable | library item and binding evidence | none | verify/bind account or use source browser | no media command | +| No compatible output/device | start unavailable | content selection and library state | bounded rediscovery | activate/select supported device | no fallback target | +| Feature disappeared | control disabled or command rejected | last safe observation | re-probe once | choose another supported action | invalidate capability cache | +| Media start accepted but unobserved | possible external effect | correlation and last state, not a success claim | bounded reads; no command replay | refresh/inspect in HA | expire pending marker safely | +| Target removed/renamed | stored selection invalid | library and old preference label | none | choose an exact new target | remove stale binding after explicit confirmation | +| Unofficial YT Music cookie/token fault | playback source unavailable | Symphonia library/history | no credential workaround | repair external integration | no cookie stored in Symphonia | +| Provider/HA sends hostile media metadata | affected display sanitized | safe state category | none | report diagnostic if broken | drop unsafe field/URL | + +Errors use impact → cause → next action → retained state. A third-party outage is never described as Symphonia library loss. + +## 11. Security, permissions, and privacy + +1. The Ingress management surface remains admin-only initially. ADR 0006 selects the narrow companion broker; the App does not enable `homeassistant_api: true` or use `SUPERVISOR_TOKEN` for listening. The broker's pairing, mutual authentication, credential rotation/revocation, versioning, replay defense, and user attribution need a threat-reviewed contract before production implementation. +2. Only the broker holds Core entity/service authority. It accepts exact allowlisted `media_player` entity IDs and typed media-player reads/actions; no caller-selected service domain, arbitrary URL, HA template, area/floor/label broadcast, or browser token relay. +3. Provider authorization for import/copy does not grant playback control. HA/Music Assistant/community sources keep separate credentials and disclose unofficial support/risks. Unverified account bindings are visible and cannot authorize automatic private-playlist mapping. + The HA Core caller and MA linked/default user must be identified before claiming per-user private-library access; a server-side token may not represent the Ingress listener. +4. Media IDs, album art, browser output names, entity state and HA errors are untrusted; resolve through adapter-owned allowlists and safe URL schemes. No raw external media reference becomes filesystem/module/egress authority. +5. Now-playing observations are ephemeral by default. Avoid recording listening history or track names in ordinary logs, metrics labels, diagnostics, backups, screenshots, or support bundles; user-approved persistence would need separate policy. +6. One command targets one explicit player; mutation is attributable, rate-bounded, CSRF-protected through the authenticated App surface, and fenced against stale UI requests. Unknown outcomes are not replayed. + +## 12. Observability and operational UX + +- The user sees bridge/source/player health, capability age, observation time/staleness, binding verification, command pending/confirmed/rejected/uncertain category, and repair action. +- Sanitized logs/metrics cover discovery success, observations and lag, commands by action/category, time to confirmation, unsupported/permission faults, and reconnects; never track title, private playlist, token, cookie, or raw HA state by default. +- Command correlation joins UI request to HA service attempt and follow-up observation without treating an HA request ID as provider effect proof. +- Stable healthy playback emits no periodic notifications. Track/position polling is visually quiet; only meaningful state changes or user-required failures announce. + +## 13. Compatibility, migration, rollout, and rollback + +- Initial schema adds optional source/target binding and allowlist only after permission choice; existing provider connections, imports, plans, and jobs are unchanged. No implicit migration from provider account to HA entity. +- Phase 1 is broker protocol/security and source-matrix research plus deterministic fake-broker fixtures. Phase 2 proves one official Spotify HA end-to-end path through the broker. Phase 3 may add Music Assistant and optional unofficial community sources after separate feasibility/support decisions. +- A supported HA version/entity feature change disables only affected playback capabilities, not library/copy. Compatibility is rechecked on HA, Music Assistant, and community integration upgrades. +- Rollback removes listening UI/bridge while retaining library/copy state. Any external playback already started cannot be undone by rolling back Symphonia; no compensating stop is automatic. +- Standalone mode shares the same listening contract but shows playback setup/unavailable until an explicitly authenticated supported playback adapter exists; it never assumes a hidden Home Assistant instance. + +## 14. Testing strategy and numeric budget + +Minimum **76 distinct cases**: + +| Area | Minimum distinct cases | Behaviors/risks covered | +| --- | ---: | --- | +| Domain/configuration/capability policy | 16 | binding verification, exact references, missing scopes/targets, feature intersections, stale/unknown | +| State/application/concurrency | 18 | pending/confirmed/uncertain, external player races, duplicate click, out-of-order events, restart/target change | +| HA/MA/provider adapter contracts | 18 | Spotify idle/active/restricted and duplicate-device fixtures, WS versus service browse, MA queue/external and caller identity, exact refs, errors/timeouts | +| HTTP/UI/accessibility/sanitization | 14 | all visible states, keyboard/focus/announcements, mobile, hostile metadata/artwork, unsupported reasons | +| Integration/security/migration | 10 | broker pairing/version/allowlist/secret canaries, Ingress isolation, reconnect/no replay, rollback/backup, opt-in live smoke | +| **Total** | **76** | No double counting | + +The ordinary suite uses synthetic HA entity states, feature flags, service responses, deterministic clocks, command IDs, source browsers, and no network/account. Each supported integration gets its own versioned fixture matrix; Spotify and Music Assistant are not treated as the same protocol mapping. Security tests require exact entity/action allowlisting, no MA name/URL/path fallback, caller-context checks, and secret canaries. Opt-in live smoke uses disposable/dedicated accounts and devices; it tests Spotify idle start/browse and duplicate-name outputs, MA queue/external source and actual user attribution, the supported source/output matrix, accepted-but-unobserved commands, independent external controller races, and cleanup. Human evidence covers Home Assistant-native visual parity, phone/wide, light/dark, keyboard/screen reader, long metadata, multiple players, and all blocked/uncertain states. Live tests supplement, not replace, deterministic contracts. + +## 15. Documentation and discoverability + +| Audience | Artifact | Required content | Validation/navigation | +| --- | --- | --- | --- | +| Listener | Listen/now-playing guide | source vs library, choose output, supported controls, stale/uncertain meaning | linked from Listen and playlist views | +| Setup owner | Playback integration guide | Spotify HA/Music Assistant prerequisites, account binding, permissions, unofficial labels, no-device recovery | setup checklist/fixture | +| Operator | Playback troubleshooting | broker pairing/version, source outage, auth/token repair in owning integration, state lag and safe retry | sanitized decision tree | +| Contributor | Playback adapter/port contract | feature mapping, exact media refs, confirmation model, security allowlist and fixtures | architecture/contract gates | + +## 16. Acceptance scenarios + +1. Given a library provider but no playback source, Listen explains separate setup and library/copy remain usable. +2. Given an approved HA Spotify entity with a proven active, uniquely identifiable compatible output and eligible `PLAY_MEDIA`, an exact Spotify playlist reference is dispatched only for that selected account/target. Success requires observed media **and output**; otherwise the result is uncertain. If output identity or idle-start eligibility is unproven, start is disabled with a reason, even if a playlist URI is known. +3. Given a configured Music Assistant source/player, the user can browse/select only content and controls its integration exposes; unofficial YouTube Music status and prerequisites stay visible. +4. Given a private imported playlist with an unverified/mismatched playback account, Symphonia does not send a guessed URI to the player and offers a supported source-browser path. +5. Given two active players or a changed output, one explicit target is shown and only that target receives the command; a stale selector cannot redirect it. +6. Given missing title/artwork/context, the UI reports the observed state and unknown fields rather than fabricating metadata from library snapshots. +7. Given unsupported `STOP`, `NEXT_TRACK`, `PLAY_MEDIA`, or source selection, the corresponding action is disabled with an accessible reason; it is not inferred from other supported controls. +8. Given a service request accepted but no matching player observation, the command becomes uncertain after the bounded budget; no automatic replay or false success occurs. +9. Given an external controller action during a pending Symphonia command, out-of-order events cannot confirm the wrong media or overwrite the newer player state. +10. Given HA or a YT Music community integration outage, playback is marked stale/unavailable while imported library, copy plans, and history remain intact. +11. Given a forged entity ID, service domain, wildcard target, media URL, or malicious metadata, the broker/server rejects unsafe authority and the UI remains inert/secret-free. +12. Given mobile/desktop, light/dark, keyboard/screen reader and an arbitrary Ingress prefix, source/target/current track/controls and pending/error recovery remain accessible through the shared UI families. +13. Given a restart, upgrade, or rollback, no old pending transport request is replayed and no provider playlist is modified by playback reconciliation. +14. Given a source with browse/search capabilities, the user sees its playable media kinds and can select an exact item; given no search or a missing media kind, the UI explains that limitation without substituting imported-library guesses. +15. Given HA Spotify idle/restricted or a non-Premium account, `play_media` and WebSocket browse are not inferred from an existing browse method; `STOP` and search are not offered. Any alternative browse-service route requires its own approved, tested permission path. +16. Given duplicate Spotify or MA source names, a stale Spotify device list, or a concurrent output switch, source/output selection does not silently choose the first matching device; an unverified output cannot be shown as confirmed. +17. Given an MA player on an external source without an active queue, now-playing may show the reported item, but shuffle/repeat/clear-queue/next-item claims are suppressed unless the effective source/player profile proves them. +18. Given a server-side HA request whose MA user identity is missing, defaulted or mismatched, private library/search/play is not attributed to the listener; unchecked URL/path/name inputs never reach MA's permissive resolver. +19. Given no broker, a revoked pairing, an incompatible protocol version, or a lost broker response, listening is unavailable/uncertain without fallback to a broad Core API token or replay; library/copy remains usable. + +## 17. Requirements traceability + +| Requirement | Policy/use case/adapter/presentation | Test or evidence | Documentation | +| --- | --- | --- | --- | +| `SYM-PLAY-001`, `SYM-PROV-022` | explicit binding/setup policy | account/target identity fixtures; scenarios 1, 4, 5 | setup guide | +| `SYM-PLAY-002`, `SYM-PLAY-008` | observation adapter/ephemeral view | missing/stale/privacy fixtures; scenarios 6, 10 | listener + privacy guide | +| `SYM-PLAY-003`, `SYM-PLAY-007` | capability intersection/typed dispatcher | feature/action/target matrix; scenarios 5, 7 | control reference | +| `SYM-PLAY-004`, `SYM-PROV-021` | exact-reference resolver and confirm view | Spotify/MA/unknown ref fixtures; scenarios 2–4 | browse/play guide | +| `SYM-PLAY-005` | command reconciliation | accepted/lost/race tests; scenarios 8, 9, 13 | troubleshooting | +| `SYM-PLAY-006`, `SYM-ARCH-016` | composition/isolation | HA outage and architecture tests; scenarios 1, 10, 13 | operator guide | +| `SYM-PLAY-009` | source browser/search and playable-kind mapping | Spotify/MA/unsupported kind fixtures; scenario 14 | listener browse guide | +| `SYM-HA-011`, `SYM-SEC-004`, `SYM-SEC-008` | broker auth/version/allowlist | forged-target/service, pairing/replay and token canaries; scenarios 11, 19 | security/setup guide | +| `SYM-UI-001`–`SYM-UI-016` | shared component family + route | catalog, responsive/a11y/Ingress fixtures; scenario 12 | UI/content guide | + +## 18. Implementation sequence + +1. Complete `OQ-011` and `RG-007`: the broker authority choice is accepted, but its transport/pairing/threat model, supported source/output matrix, identity rules, and bounded state/command thresholds remain open. +2. Add architecture, broker protocol/allowlist, capability, binding, and command-confirmation contract tests with fake HA/Spotify/MA observations. +3. Implement pure listening policy and application ports/use cases with ephemeral state and minimal persistent binding only. +4. Implement the approved, versioned companion broker and App adapter only after this SDD is ready, then prove the Spotify HA vertical; add Music Assistant/community mappings only after their separate evidence/support decision. +5. Add shared now-playing/transport UI family, Listen route, accessibility, documentation, and Ingress/standalone degraded-state behavior. +6. Run opt-in supported-matrix smoke and security/visual review; update source evidence and catalog before readiness/implementation claims. + +## 19. Definition of Done + +- [ ] SDD is `Ready for implementation`, explicit owner approval to implement exists, and broker protocol/security plus source-matrix blockers are resolved. +- [ ] Account/source/player/output distinctions, exact-reference rules, and unavailable/unverified states pass acceptance and documentation review. +- [ ] At least 76 distinct deterministic cases, architecture checks, permission/allowlist tests, and secret-canary gates pass. +- [ ] Spotify HA and any advertised MA/community path have dedicated opt-in integration evidence on supported versions/accounts/targets. +- [ ] Command acceptance versus observed effect, timeout/uncertain/no-replay, external races, and restart behavior are proven. +- [ ] Listening UI passes the shared component catalog, HA visual-reference, responsive, localization, accessibility, and Ingress/standalone gates. +- [ ] No audio pipeline, implicit account binding, broad App-to-Core access/generic Core proxy, plaintext listening history, or playlist-write retry leakage exists. +- [ ] User/setup/operator/contributor guides, privacy terms, compatibility matrix, and catalog implementation evidence are current. + +## 20. References and decisions + +- Primary sources: dated official Home Assistant and Spotify links in [provider research](../docs/providers/provider-research.md#2026-09-28-playback-and-home-assistant-api-update). +- Integration implementation evidence and version-sensitive risk matrix: [source review](../docs/providers/playback-integration-source-review.md); broader community patterns: [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md#playback-source-evidence-2026-09-28). +- Related SDDs: [App runtime](home-assistant-app-runtime-and-ingress.md), [UI foundation](home-assistant-native-ui.md), [provider connections](provider-connections-and-authorization.md), [library import](library-import-and-provider-projections.md), [playlist copy](one-time-playlist-copy.md). +- Accepted authority decision: [ADR 0006](../docs/decisions/0006-narrow-home-assistant-playback-broker.md) requires a narrow Home Assistant companion broker for listening; App-to-Core Supervisor proxy is not the initial playback path. The broker transport/pairing protocol and supported source matrix are **not accepted** yet. +- Rejected: automatic provider-to-player linking, guessing playlist URIs from titles, command-success from HTTP acceptance, silent YT Music unofficial fallback, and treating Symphonia as an audio streamer. +- Follow-up: queue editing, cross-player handoff/multi-room, direct provider playback, and listening history require separate scope/evidence decisions. diff --git a/specs/provider-connections-and-authorization.md b/specs/provider-connections-and-authorization.md index 8d29425..f4bd405 100644 --- a/specs/provider-connections-and-authorization.md +++ b/specs/provider-connections-and-authorization.md @@ -1,7 +1,7 @@ # Provider connections and authorization - Status: Draft -- Date: 2026-09-27 +- Date: 2026-09-28 - Catalog capability ID: `provider-connections-and-authorization` - Owners: Symphonia maintainers - Scope: disclose provider risk, authorize one external account, protect and refresh its grant, probe effective capabilities, reauthorize, and disconnect safely. @@ -88,6 +88,7 @@ not a claim that the connection capability is implemented. 2. Selecting a shared hosted OAuth application for every future deployment. 3. Treating browser-cookie export as an ordinary OAuth equivalent. 4. Sharing one provider grant across different Symphonia installations without an explicit provider contract. +5. Claiming a provider library grant also configures a Home Assistant playback source, proves the same account, or grants player control; see the [listening SDD](listening-and-playback-control.md). ### 4.3 Fixed invariants @@ -145,6 +146,7 @@ Text equivalent: an unconfigured adapter starts one authorization attempt; verif - Consent denial returns to `not_configured` with no connection or stored user grant. - Missing optional scopes create a connected but reduced capability set only when the user explicitly accepted that mode. +- A provider connection may be healthy for library/copy while no playback source or player exists. Connection status names these capabilities separately; playback setup never borrows the stored provider grant implicitly. - Expired/revoked credentials become `action_required`; pending writes are not blindly retried. - Provider outage can be `degraded` without claiming reauthorization is required. - Multiple accounts of one provider retain separate connection IDs, secret references, account identity, capability probes, and rate budgets. @@ -249,6 +251,12 @@ Action: Reconnect Spotify. Retained state: mappings, plans, and audit history are unchanged. ``` +```text +Spotify connected for library and playlists +Listening: not configured. Home Assistant's Spotify integration and an output need separate setup. +Action: Set up listening. Your imports and playlist-copy capabilities are unaffected. +``` + ```text Disconnected from Spotify Provider access and local credential material were removed. Provider-derived cached data is scheduled for policy cleanup; manual decisions and operation summaries remain. @@ -372,7 +380,7 @@ The twelve feature-specific UI cases supplement the UI-foundation budget and inh ## 20. References and decisions - Primary sources: official provider and Home Assistant sources in [provider research](../docs/providers/provider-research.md). -- Related SDDs: [App runtime](home-assistant-app-runtime-and-ingress.md), [imports](library-import-and-provider-projections.md), [durable operations](durable-operations-and-recovery.md). +- Related SDDs: [App runtime](home-assistant-app-runtime-and-ingress.md), [imports](library-import-and-provider-projections.md), [durable operations](durable-operations-and-recovery.md), [listening](listening-and-playback-control.md). - Related UI contract: [Home Assistant-native UI foundation](home-assistant-native-ui.md) and [ADR 0004](../docs/decisions/0004-home-assistant-native-ui.md). - Accepted: capability and risk disclosure before authorization; opaque secret references; distinct unofficial adapters. - MVP direction: direct App-owned callback-only OAuth; a companion broker is deferred. From e0f1291da13573eca6ca5c20957ef3496d2b38aa Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 28 Sep 2026 09:38:47 +0200 Subject: [PATCH 164/167] test: measure branch coverage and harden runtime HTTP contracts --- .github/workflows/verify.yml | 16 ++++++ Makefile | 8 ++- docs/development/quality-audit.md | 12 ++++- pyproject.toml | 10 ++++ tests/test_runtime_http.py | 87 ++++++++++++++++++++++++++++++- 5 files changed, 130 insertions(+), 3 deletions(-) diff --git a/.github/workflows/verify.yml b/.github/workflows/verify.yml index 8dc6a16..6501979 100644 --- a/.github/workflows/verify.yml +++ b/.github/workflows/verify.yml @@ -71,3 +71,19 @@ jobs: run: npm run build - name: Inspect relative assets run: npm run verify-output + + branch-coverage: + runs-on: ubuntu-latest + steps: + - name: Check out repository + uses: actions/checkout@v4 + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.13" + - name: Install pinned quality tool + run: python -m pip install coverage==7.16.1 + - name: Measure line and branch coverage + run: | + PYTHONPATH=src:. python -m coverage run -m unittest discover -s tests -q + python -m coverage report --sort=cover diff --git a/Makefile b/Makefile index f118bdb..d836dbc 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,6 @@ -.PHONY: verify specs test spike-storage +.PHONY: verify specs test coverage spike-storage + +COVERAGE_PYTHON ?= python3 specs: PYTHONPATH=. python3 tools/validate_specs.py @@ -6,6 +8,10 @@ specs: test: PYTHONPATH=src:. python3 -m unittest discover -s tests -v +coverage: + PYTHONPATH=src:. $(COVERAGE_PYTHON) -m coverage run -m unittest discover -s tests -q + $(COVERAGE_PYTHON) -m coverage report --sort=cover + spike-storage: PYTHONPATH=src:. python3 tools/storage_recovery_spike.py --operations 1000 diff --git a/docs/development/quality-audit.md b/docs/development/quality-audit.md index 404cd23..8109f0c 100644 --- a/docs/development/quality-audit.md +++ b/docs/development/quality-audit.md @@ -42,4 +42,14 @@ Re-ran the documented offline commands against the local `develop` working tree Graphify's `--code-only --no-cluster` extraction produced **1,181 nodes and 3,060 edges**. `OperationRepository` remains the most connected symbol (57 edges); its one-hop dependents include runtime HTTP/resources, copy and import execution, the operation runner, and their tests. This reinforces the existing plan to define repository ports and preserve restart/reconciliation characterization before changing operation persistence. The broker design is kept out of domain/application imports and is not wired into this graph while its SDD is Draft. -The local `make verify` run passed **227 `unittest` cases (2 skipped)** plus specification/whitespace checks. The isolated Lit spike also passed `npm ci --ignore-scripts`, TypeScript, presentation-boundary checks, three pure fallback tests, build, and relative-asset verification. Neither result proves Home Assistant integration, live playback, visual parity, or CI on GitHub. RepoWise still has no measured branch-coverage input; that remains a separate tooling decision rather than a fabricated coverage percentage. +The local `make verify` run passed **232 `unittest` cases (2 skipped)** plus specification/whitespace checks. The isolated Lit spike also passed `npm ci --ignore-scripts`, TypeScript, presentation-boundary checks, three pure fallback tests, build, and relative-asset verification. Neither result proves Home Assistant integration, live playback, visual parity, or CI on GitHub. + +The previously absent coverage measurement is now reproducible with pinned `coverage.py` 7.16.1 (development-only, not a runtime dependency): + +```text +python3 -m venv .venv +.venv/bin/python -m pip install -e '.[quality]' +make coverage COVERAGE_PYTHON=.venv/bin/python +``` + +The 2026-09-28 local Python 3.14 run measured **81% combined line/branch coverage** across 3,112 statements and 934 branch opportunities; it is not an 81% branch-only claim. Five new runtime HTTP contract tests brought `runtime/http.py` from 59% to 73%. Remaining examples are `apple_music.py` at 63%, `copy_execution.py` at 75%, `sqlite_operations.py` at 79%, and `__main__.py` at 0% under this unit suite. Those gaps guide targeted tests; no global minimum or release pass claim is inferred from a single number. CI now measures the same report on Python 3.13 as an additional non-threshold check. RepoWise's own coverage field may remain `null` until its ingestion is configured; do not present its heuristic scores as measured coverage. diff --git a/pyproject.toml b/pyproject.toml index ff8077e..f7e2336 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -11,6 +11,9 @@ license = { text = "TBD" } authors = [{ name = "Symphonia maintainers" }] dependencies = [] +[project.optional-dependencies] +quality = ["coverage==7.16.1"] + [project.scripts] symphonia = "symphonia.__main__:main" @@ -19,3 +22,10 @@ where = ["src"] [tool.pytest.ini_options] testpaths = ["tests"] + +[tool.coverage.run] +branch = true +source = ["symphonia"] + +[tool.coverage.report] +skip_covered = true diff --git a/tests/test_runtime_http.py b/tests/test_runtime_http.py index 00499d7..e3750c0 100644 --- a/tests/test_runtime_http.py +++ b/tests/test_runtime_http.py @@ -9,7 +9,12 @@ import unittest from symphonia.infrastructure import OperationRepository -from symphonia.runtime.http import SymphoniaRequestHandler, create_server, route_get +from symphonia.runtime.http import ( + SymphoniaHTTPServer, + SymphoniaRequestHandler, + create_server, + route_get, +) class RuntimeHTTPTests(unittest.TestCase): @@ -65,6 +70,43 @@ def test_readiness_can_use_the_composed_runtime_healthcheck(self) -> None: self.assertEqual(status, 503) self.assertEqual(payload["status"], "not_ready") + def test_readiness_exception_fails_closed_without_exposing_detail(self) -> None: + def broken_readiness() -> bool: + raise RuntimeError("secret database detail") + + status, payload = route_get( + "/ready", self.repository, readiness_check=broken_readiness + ) + + self.assertEqual(status, 503) + self.assertEqual(payload, {"service": "symphonia", "status": "not_ready"}) + self.assertNotIn("secret", str(payload)) + + def test_non_origin_and_cross_ingress_paths_never_route_to_health(self) -> None: + for path in ( + "health", + "//other-host/health", + "https://other-host/health", + "/local_symphonia/health#fragment", + "/local_symphonia-extra/health", + "/other/health", + ): + with self.subTest(path=path): + status, payload = route_get( + path, self.repository, ingress_path="/local_symphonia" + ) + self.assertEqual((status, payload), (404, {"error": "not_found"})) + + def test_server_configuration_rejects_missing_or_conflicting_store(self) -> None: + with self.assertRaisesRegex(ValueError, "must be supplied"): + SymphoniaHTTPServer(("127.0.0.1", 0)) + with self.assertRaisesRegex(ValueError, "mutually exclusive"): + SymphoniaHTTPServer( + ("127.0.0.1", 0), + repository=self.repository, + resources=object(), # type: ignore[arg-type] + ) + def test_json_surface_sets_no_cache_and_content_sniffing_headers(self) -> None: class FakeHandler: def __init__(self) -> None: @@ -110,6 +152,49 @@ def end_headers(self) -> None: {"value": float("nan")}, ) + def test_json_surface_closes_connection_on_disconnected_client(self) -> None: + class DisconnectedWriter: + def write(self, body: bytes) -> None: + raise ConnectionError("client disconnected") + + class FakeHandler: + close_connection = False + wfile = DisconnectedWriter() + + def send_response(self, status: int) -> None: + return + + def send_header(self, name: str, value: str) -> None: + return + + def end_headers(self) -> None: + return + + handler = FakeHandler() + SymphoniaRequestHandler._json(handler, 200, {"status": "ok"}) # type: ignore[arg-type] + self.assertTrue(handler.close_connection) + + def test_method_errors_have_bounded_json_categories(self) -> None: + class FakeHandler: + def __init__(self) -> None: + self.result: tuple[int, dict[str, str], bool] | None = None + + def _json( + self, status: int, payload: dict[str, str], *, close_connection: bool + ) -> None: + self.result = (status, payload, close_connection) + + for code, category in ((400, "bad_request"), (501, "not_implemented"), (503, "server_error")): + with self.subTest(code=code): + handler = FakeHandler() + SymphoniaRequestHandler.send_error( # type: ignore[arg-type] + handler, + code, + message="secret request detail", + explain="internal stack", + ) + self.assertEqual(handler.result, (code, {"error": category}, True)) + def test_composed_runtime_http_smoke_exposes_health_and_readiness(self) -> None: with tempfile.TemporaryDirectory() as directory: try: From 05c5b15fea79b6faaf5a7f24565e6740d523959c Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 28 Sep 2026 10:52:08 +0200 Subject: [PATCH 165/167] docs: define full listening release evidence and gates --- docs/README.md | 1 + docs/development/playback-release-gates.md | 47 ++++++++++++++++++++++ docs/open-questions.md | 8 ++-- docs/product/product-specification.md | 10 ++--- docs/providers/provider-research.md | 4 +- specs/CATALOG.md | 2 +- specs/catalog.json | 5 ++- specs/listening-and-playback-control.md | 37 +++++++++-------- 8 files changed, 85 insertions(+), 29 deletions(-) create mode 100644 docs/development/playback-release-gates.md diff --git a/docs/README.md b/docs/README.md index f07dc71..1dd82c6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -18,6 +18,7 @@ This documentation is the horizontal implementation contract for Symphonia. It d | [Provider research](providers/provider-research.md) | Dated, sourced facts about provider APIs | Product policy or permanent architecture | | [Home Assistant music ecosystem review](providers/home-assistant-ecosystem-review.md) | Reusable patterns and cautions from existing HA music projects | Dependency selection or provider guarantees | | [Playback integration source review](providers/playback-integration-source-review.md) | Dated source-code findings, operation limits, risk and live proof matrix for HA Spotify, Music Assistant, and YT Music projects | Official API guarantees, release support, or SDD readiness | +| [Complete listening proof plan](development/playback-release-gates.md) | Desired Spotify/MA/YT Music release claims, evidence matrix, broker threat gate, and disposable-system probes | Permission to implement or a guarantee that upstream services expose every action | | [Development specification](development/development-specification.md) | Specification workflow, testing and delivery gates | Product scope | | [Local quality audit](development/quality-audit.md) | Reproducible offline RepoWise/Graphify review and current architecture/test debt | A release or SDD readiness claim | | [ADRs](decisions/README.md) | Decisions that have actually been accepted | Proposals and guesses | diff --git a/docs/development/playback-release-gates.md b/docs/development/playback-release-gates.md new file mode 100644 index 0000000..b12c6bd --- /dev/null +++ b/docs/development/playback-release-gates.md @@ -0,0 +1,47 @@ +# Complete listening: release claims and proof gates + +**Status:** research/acceptance plan, not an implementation approval + +**Reviewed:** 2026-09-28 + +**Owner direction:** do not ship an active-session-only Spotify slice as the promised complete listening experience. Internal increments may be smaller; the outward claim must wait for the supported-profile evidence below. + +This plan refines [OQ-011](../open-questions.md#oq-011--which-complete-playback-profiles-can-be-promised) and the [listening SDD](../../specs/listening-and-playback-control.md). It does not change the accepted [narrow Home Assistant broker authority](../decisions/0006-narrow-home-assistant-playback-broker.md). A connection used to import playlists does not authorize playback, and a media-player entity does not prove that an arbitrary provider account or output is the same one. + +## What “complete” means here + +For every **advertised supported profile**, setup must lead to a usable source and exact output, browse/selectable tracks and playlists, an eligible start path (including idle when claimed), current playback, the controls actually supported by that profile, switching to another playlist, and comprehensible recovery. One profile may expose Pause while another exposes Stop; the UI must use those words accurately. A playlist switch is a playback command, not an edit to the provider playlist. Unknown or unsupported capabilities stay visible as limitations, never become fabricated controls. + +The desired profiles are stock Home Assistant Spotify, Home Assistant Music Assistant with Spotify or YouTube Music as a source, and any separately accepted community YouTube Music player. Their control models differ; the requirement is a complete and honest end-to-end experience for each **accepted** profile, not identical feature bits. Native Google YouTube Music app sessions, arbitrary speakers, and offline devices are not implicitly supported. If evidence shows that a desired operation is impossible or unacceptably risky, the owner must approve an explicit product exception or a different, separately authorized route before a complete-release claim. + +## Source/output evidence matrix + +| Profile | Current evidence | Proof still needed before a supported claim | +| --- | --- | --- | +| HA Spotify through the broker | [HA's guide](https://www.home-assistant.io/integrations/spotify/) requires Premium and a known compatible Spotify device. The [source review](../providers/playback-integration-source-review.md#spotify-real-controls-and-hard-outputbrowse-limits) shows idle `PLAY_MEDIA`/browse gating and name-based, potentially ambiguous source selection. | Freshly identify account and device, start a playlist from idle on the exact requested output, distinguish duplicate names and restricted devices, switch playlist, observe actual item/output, and establish action/freshness limits on pinned HA versions. A stock entity may fail this gate. | +| Explicit Spotify Connect enhancement, **not yet authorized** | [Available Devices](https://developer.spotify.com/documentation/web-api/reference/get-a-users-available-devices) returns IDs that may change and can omit devices; [Start/Resume Playback](https://developer.spotify.com/documentation/web-api/reference/start-a-users-playback) targets a device ID and playlist context; [Transfer Playback](https://developer.spotify.com/documentation/web-api/reference/transfer-a-users-playback) accepts one device ID. These are official API contracts, not proof that every device is startable. | Owner authorization for a separate grant; current endpoint availability in a new developer app; OAuth scopes, policy/terms and account-equivalence review; fresh non-restricted device ID, idle start and output confirmation, ordering/race/429 recovery, grant revocation and backup behavior. No implicit reuse of a library or HA credential. | +| HA Music Assistant with a configured source/player | [MA's HA integration guide](https://www.music-assistant.io/integration/installation/) documents distinct HA player entities and richer actions, and warns that some user attribution depends on an admin integration account. The [source review](../providers/playback-integration-source-review.md#music-assistant-queue-source-and-user-context) shows that an external active source need not have an MA queue. | Exact source/provider and player identity, effective caller/linked-user policy through the actual broker, source browse completeness, exact playable references, idle start and playlist switch, queue versus external-source controls, and state/effect confirmation. | +| Music Assistant YouTube Music source | [MA's provider guide](https://www.music-assistant.io/music-providers/youtube-music/) documents unofficial cookie + PO-token setup and best-effort operation; free accounts are not supported. | Explicit opt-in/support/security policy, dedicated-account setup and expiry recovery, content/playlist identity and completeness, playable target, session scope, source loss and cleanup. MA-managed playback must not be described as observation of Google's native app. | +| Community `ytube_music_player` | The [source review](../providers/playback-integration-source-review.md#youtube-music-implementations-feasible-ma-playback-not-native-client-control) describes a separate HA entity and remote-player dependency, not an official or universal YouTube Music session. | Owner acceptance of a version-pinned optional profile; exact remote target, feature/effect mapping, credential isolation, upstream maintenance and threat review. Reject silent default-player selection. | + +The official [YouTube Data API](https://developers.google.com/youtube/v3/docs) models YouTube video playlists; it is not evidence of an official YouTube Music playback/session API. The provider import/copy choice is separate from the MA listening-source choice. + +## Broker protocol/security gate before production code + +The authority boundary is decided, but the wire contract is not. The protocol RFC must select and test deployment discovery, connection direction, transport protection, pairing/secret rotation/revocation, broker and App identity, version negotiation, authenticated listener attribution, and upgrade/rollback. It must specify one exact entity per action, a closed operation/media-kind allowlist, bounded request/response/event sizes, timeout and rate budgets, monotonic observation ordering, command correlation, and no automatic command replay. The broker must never accept a generic HA domain/service, template, URL/path, area, wildcard, or arbitrary MA search string as an action target. The App must not receive a broad Core credential; the browser must not receive broker or provider credentials. A threat review must include other apps on the internal network, a compromised browser/session, forged entity IDs, stolen pairing material, replay after restart, and accidental private metadata in logs/backups. + +## Required disposable-system proof + +Use dedicated accounts and disposable Home Assistant/Music Assistant installations with pinned versions; never personal libraries or copied browser cookies in test artifacts. Record prerequisites, account tier, scope, integration version, player model, output ID, supported flags, request/response category, observation latency, and cleanup for each case. Ordinary CI uses deterministic synthetic fixtures; these live probes are opt-in and add evidence rather than replacing offline tests. + +1. **Spotify:** no session, active, paused, restricted/private, duplicate device names, stale device cache, unavailable and reappearing device, multiple accounts; start and switch exact playlists on an exact output or record the failure. Compare HA-only and any separately authorized direct path without silently crossing credentials. +2. **Music Assistant:** idle and active MA queue, external input with no queue, two players, duplicate source names, linked versus default/unknown caller, long/partially visible library, exact Spotify and YouTube Music playlist references, race with external controller. +3. **YouTube Music:** MA-managed playback, expired cookie/PO service, unavailable track, library-versus-playlist visibility, and an independently started native Google-client session to establish whether it is visible (do not assume it is). +4. **Bridge and UX:** Ingress user versus broker caller, denied/rotated pairing, forged target and media reference, version mismatch, network loss, accepted-but-unobserved command, delayed/out-of-order state, HA and App restart, no replay, mobile/desktop keyboard and screen-reader recovery. +5. **Operations:** backup/restore, uninstall/reinstall, account disconnect, broker-only outage, secure data deletion, audit redaction, and support diagnostics without track names, tokens, cookies, or raw HA state. + +For each advertised action, record `supported`, `supported with stated preconditions`, or `not supported` with a test and user-facing explanation. An accepted service/API response is never a playback-effect proof. Observation and command-confirmation budgets must be measured per profile before setting thresholds; Spotify's documented polling interval alone does not define a safe universal deadline. + +## Release decision + +Do not label listening complete, mark its SDD `Ready for implementation`, or begin production playback code until owner choices, broker protocol/threat review, source/output/account proof, and the SDD readiness checklist are satisfied. Internal spikes may reduce uncertainty, but the release claim requires the agreed profile matrix plus offline, live, security, accessibility, documentation, migration, and rollback evidence. This is a quality gate, not a promise that upstream services expose every requested operation. diff --git a/docs/open-questions.md b/docs/open-questions.md index f180a7c..ddf975c 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -78,11 +78,11 @@ The repository currently has no license. The license should be selected before a This foundation deliberately does not guess at provider reconciliation or expose an unauthenticated resolution endpoint. A future provider-aware reconciliation handler may resolve the outcome automatically only after its provider contract is proven. The repository primitive for explicit resolution is not itself an authorization boundary; any caller must enforce the approved operator identity and action policy. The durable-operation SDD remains blocked on that provider/UI vertical and its other readiness gates, but OQ-010 no longer represents an unresolved policy choice. -### OQ-011 — Which playback sources and Home Assistant authority ship first? +### OQ-011 — Which complete playback profiles can be promised? -The owner wants a Home Assistant-integrated listening surface alongside playlist management; see the [listening SDD](../specs/listening-and-playback-control.md). **Authority boundary resolved (2026-09-28):** the owner chose a narrow companion playback broker, recorded in [ADR 0006](decisions/0006-narrow-home-assistant-playback-broker.md), instead of granting the App broad Supervisor Core API authority. The first supported source/output matrix and broker transport, pairing, versioning, user attribution, and threat model remain open. This is independent of `OQ-004`'s provider OAuth callback decision: provider import credentials do not authorize HA player control. +The owner wants a Home Assistant-integrated listening surface alongside playlist management; see the [listening SDD](../specs/listening-and-playback-control.md). **Authority boundary resolved (2026-09-28):** the owner chose a narrow companion playback broker, recorded in [ADR 0006](decisions/0006-narrow-home-assistant-playback-broker.md), instead of granting the App broad Supervisor Core API authority. **Scope direction (2026-09-28):** the owner rejected an active-session-only Spotify first release and requested the complete listening experience across the desired connected providers, including starting media, choosing where it plays, current playback, transport controls, and changing playlists. Internal implementation increments are allowed, but an active-session-only prototype is not a complete-release claim. The supported source/output matrix and broker transport, pairing, versioning, user attribution, and threat model remain open. This is independent of `OQ-004`'s provider OAuth callback decision: provider import credentials do not authorize HA player control. -**Proposed staged default, conditional on proof:** first evaluate the official Home Assistant Spotify entity via the broker on an already active, uniquely identifiable compatible output. The [source review](providers/playback-integration-source-review.md) shows that stock HA Spotify drops `PLAY_MEDIA` while idle, selects devices by name and may not support exact-output promises. Decide whether this deliberately limited path is useful or a separately authorized device-ID-capable profile is needed before promising start-from-idle or “play here.” Then consider Music Assistant players and its unofficial YouTube Music source with independent account/caller/queue and support-risk review. `ytube_music_player` is optional community evidence, not an automatic dependency. Multiple active players are selected explicitly. There is no fallback to a generic Core API relay. +**Unresolved feasibility and authority:** the [source review](providers/playback-integration-source-review.md) shows that stock HA Spotify drops `PLAY_MEDIA` while idle and selects devices by name, so it alone cannot substantiate the requested idle-start/exact-output experience. A separate, explicitly consented Spotify Connect adapter with its own read/control scopes is a possible complement, **not** an accepted fallback; it requires owner approval, a separate threat/policy review, account binding and live proof. Music Assistant players and its unofficial YouTube Music source need independent account/caller/queue and support-risk review. `ytube_music_player` is optional community evidence, not an automatic dependency. Native YouTube Music app sessions are not proven observable. Multiple active players are selected explicitly. There is no fallback to a generic Core API relay. The [full-scope proof plan](development/playback-release-gates.md) makes the release claims and required evidence explicit. ## Research/design gates (not owner preference alone) @@ -152,7 +152,7 @@ The tooling choice is accepted; the result must still establish a supported matr ### RG-007 — Listening source, target, and command feasibility -Use a disposable Home Assistant installation and dedicated accounts/devices after the broker protocol/security decision. Record the exact versions and account identities for HA Spotify, Music Assistant, and any proposed community player. Run the [dated source-review proof matrix](providers/playback-integration-source-review.md#rg-007-proof-matrix-before-release): verify Spotify idle/active/restricted and duplicate-name device behavior, WebSocket versus service browsing and service feature gates, exact-output/start feasibility, MA queue versus external source and per-user/default-user attribution through the **actual companion-broker route**, playlist reference mapping, private/unavailable/large playlist behavior, state-update/device-discovery lag, accepted-but-unobserved commands, external-controller races, HA restart, and credential recovery. Prove whether account identity can be machine-verified; if not, specify a visible user-confirmed but unverified binding and forbid automatic private-playlist mapping. Test hostile entity/media IDs, MA free-text/URL/path fallback rejection, broker pairing/version/replay/secret isolation, and ensure no broker credential reaches Ingress/browser state. Decide freshness/command-confirmation budgets from these observations. Keep community implementation findings separate from official platform evidence and do not infer control of native YouTube Music app sessions. +Use a disposable Home Assistant installation and dedicated accounts/devices after the broker protocol/security decision. Record the exact versions and account identities for HA Spotify, Music Assistant, and any proposed community player. Run the [dated source-review proof matrix](providers/playback-integration-source-review.md#rg-007-proof-matrix-before-release) and the [complete-release plan](development/playback-release-gates.md): verify Spotify idle/active/restricted and duplicate-name device behavior, WebSocket versus service browsing and service feature gates, exact-output/start feasibility, MA queue versus external source and per-user/default-user attribution through the **actual companion-broker route**, playlist reference mapping, private/unavailable/large playlist behavior, state-update/device-discovery lag, accepted-but-unobserved commands, external-controller races, HA restart, and credential recovery. If a separate Spotify Connect path is approved, also prove current endpoint access, fresh device IDs, separate read/control grants, account identity, idle-start/output confirmation and revocation without reusing the HA broker credential. Prove whether account identity can be machine-verified; if not, specify a visible user-confirmed but unverified binding and forbid automatic private-playlist mapping. Test hostile entity/media IDs, MA free-text/URL/path fallback rejection, broker pairing/version/replay/secret isolation, and ensure no broker credential reaches Ingress/browser state. Decide freshness/command-confirmation budgets from these observations. Keep community implementation findings separate from official platform evidence and do not infer control of native YouTube Music app sessions. ## Future synchronization questions diff --git a/docs/product/product-specification.md b/docs/product/product-specification.md index c31bd2d..a9fc4d2 100644 --- a/docs/product/product-specification.md +++ b/docs/product/product-specification.md @@ -7,7 +7,7 @@ Symphonia is a self-hosted personal music hub through which a person can see and manage their music across providers. Symphonia owns the user's provider-independent view of music; Spotify, YouTube, Apple Music, Plex, Navidrome, and future services are replaceable representations and execution targets. -The product combines a Home Assistant-integrated listening surface with trustworthy library interoperability. A user can choose music and an output, see what is playing, and control an existing playback session where a configured integration supports it; Symphonia also imports, explains, matches, reviews, and copies playlists. Symphonia is a playback **controller**, not a replacement audio-streaming engine. Library access and playback access are separate capabilities even when they refer to the same service. +The product combines a Home Assistant-integrated listening surface with trustworthy library interoperability. A user can choose music and an exact compatible output, start or switch a playlist, see what is playing, and use the controls that a configured source actually supports; Symphonia also imports, explains, matches, reviews, and copies playlists. Symphonia is a playback **controller**, not a replacement audio-streaming engine. Library access and playback access are separate capabilities even when they refer to the same service. An active-session-only Spotify controller is an internal increment, not the complete listening promise. ## Product principles @@ -29,7 +29,7 @@ Multi-user authorization, sharing between Symphonia users, and hosted SaaS opera ## Goals - Provide one inventory of connected provider playlists and library items with clear provenance. -- Provide a familiar listening view for configured playback sources: browse/select available music, choose a supported output, see current playback, and use only supported controls. +- Provide a familiar listening view for supported Spotify and Music Assistant/YouTube Music profiles: browse/select available music, start or switch playlists on an exact compatible output (including from idle where advertised), see current playback, and use only supported controls. The [full-scope proof plan](../development/playback-release-gates.md) defines the evidence required before making that release claim. - Recognize when provider-specific items likely represent the same recording. - Let the user resolve ambiguity and preserve that decision. - Copy a playlist between supported providers with a complete preview and result. @@ -178,7 +178,7 @@ The first useful release is complete when one local user can: - authenticate with supported provider flows; - import supported library collections and owned/followed playlists; - browse provider playlists and their freshness; -- discover at least one explicitly linked Home Assistant playback source/player, browse its playable content, start a supported track or playlist on a chosen output, observe its current session, and use the controls it supports; starting an imported playlist requires a verified compatible reference; +- set up explicitly linked Spotify and Music Assistant/YouTube Music playback profiles that pass the agreed source/output matrix, browse playable content, start and switch supported tracks or playlists on the selected exact output (including idle start where claimed), observe current sessions, and use each profile's genuine controls; starting an imported playlist requires a verified compatible reference; - resolve track identities automatically where safe and manually where needed; - preview and execute a copy in each direction only where the target adapter declares all required capabilities; - inspect basic operation history; and @@ -186,7 +186,7 @@ The first useful release is complete when one local user can: The wording “validated Google/YouTube connection” is deliberate. Symmetric **YouTube Music** library access is not yet proven through an official API; see [provider research](../providers/provider-research.md). If official feasibility fails, the owner must revise the MVP rather than silently adopting a reverse-engineered API. -This listening goal does **not** promise a YouTube Music session outside a configured Home Assistant/Music Assistant/community player, or that connecting Symphonia to Spotify automatically configures Home Assistant's Spotify integration. If the first-release playback source/output pairing or account-binding contract cannot be proven, the release boundary requires an explicit owner revision; a screenshot-only listening view does not satisfy it. +This listening goal does **not** promise a YouTube Music session outside a configured Home Assistant/Music Assistant/community player, or that connecting Symphonia to Spotify automatically configures Home Assistant's Spotify integration. The owner rejected treating an active-session-only Spotify path as a complete first release. Stock Home Assistant Spotify may not meet idle-start/exact-output requirements; a separately consented direct Spotify Connect complement is under consideration, not yet approved. If the source/output pairing, account binding, idle start or provider contract cannot be proven, the release boundary requires an explicit owner revision; a screenshot-only listening view does not satisfy it. ## Future scope @@ -195,7 +195,7 @@ This listening goal does **not** promise a YouTube Music session outside a confi - Apple Music, Plex/Plexamp, Navidrome, and other adapters. - Multiple accounts per provider in the UI. - Home Assistant entities, actions, and events over a stable Symphonia API. -- Direct provider playback adapters where a supported contract materially improves coverage beyond Home Assistant players, subject to separate scopes, permissions, and policy review. +- Direct provider playback adapters beyond any separately accepted Spotify Connect complement, subject to separate scopes, permissions, and policy review. - More complete album, artist, release, and musical-work modeling. - Export/import of user-authored mappings and operation history. diff --git a/docs/providers/provider-research.md b/docs/providers/provider-research.md index 044dc24..98dfd87 100644 --- a/docs/providers/provider-research.md +++ b/docs/providers/provider-research.md @@ -20,10 +20,12 @@ This is a dated official-documentation check, not a live account/device feasibil | [Home Assistant media-player entity](https://developers.home-assistant.io/docs/core/entity/media-player/) and [actions](https://www.home-assistant.io/integrations/media_player/) | Standard states, optional title/artist/artwork/position/playlist attributes, and per-entity supported features include play/pause/stop/next/previous/play-media/browse/select-source. | Metadata and controls are optional and integration-specific; a feature flag alone does not validate an arbitrary provider playlist reference. | | [Home Assistant Music Assistant integration](https://www.home-assistant.io/integrations/music_assistant/) | Exposes MA players as HA `media_player` entities with current media and controls; provides richer browse/play/queue actions when the MA server and integration are configured. | It is a separate server/integration lifecycle; MA source credentials, account identity, and queue are not Symphonia's imported library or durable playlist. | | [App communication](https://developers.home-assistant.io/docs/apps/communication/) and [configuration](https://developers.home-assistant.io/docs/apps/configuration/) | An App can request `homeassistant_api: true` and use `SUPERVISOR_TOKEN` with the Supervisor Core REST/WebSocket proxy. [HA WebSocket](https://developers.home-assistant.io/docs/api/websocket/) can stream state changes and call services; [REST](https://developers.home-assistant.io/docs/api/rest/) distinguishes state representation from service actions. | This is broader Core authority than a music-specific permission. It requires a threat review, server-only token, exact entity/action allowlist, and an accepted decision versus a narrower companion broker. | -| [Spotify currently playing](https://developer.spotify.com/documentation/web-api/reference/get-the-users-currently-playing-track) and [start/resume](https://developer.spotify.com/documentation/web-api/reference/start-a-users-playback) | Direct Web API offers current item/context and playlist context start, with separate read/control scopes and Premium playback requirements. | A direct fallback would require new consent, policy, account/device handling, and tests; it is not silently inherited from the library adapter or selected for the first listening SDD. | +| [Spotify currently playing](https://developer.spotify.com/documentation/web-api/reference/get-the-users-currently-playing-track), [available devices](https://developer.spotify.com/documentation/web-api/reference/get-a-users-available-devices), [start/resume](https://developer.spotify.com/documentation/web-api/reference/start-a-users-playback), and [transfer](https://developer.spotify.com/documentation/web-api/reference/transfer-a-users-playback) | Direct Web API offers current item/context, device IDs, playlist-context start on a selected device, and a one-device transfer. Playback requires Premium and distinct read/control scopes. Device IDs may change, be null or omit some devices; a restricted device cannot be controlled. | This is a possible **explicitly consented complement**, not a silent fallback or an accepted implementation path. It needs new owner authorization, endpoint/policy review, account/device proof, fresh-ID and effect reconciliation, and tests; a library adapter or HA source does not automatically grant control. | The reviewed [official YouTube Data API reference](https://developers.google.com/youtube/v3/docs) is about video/channel/playlist resources, not a documented YouTube Music now-playing or remote-control API. Absence from reviewed documentation is an inference, not a claim that no private integration exists. Community playback examples and their risks are kept in the [ecosystem review](home-assistant-ecosystem-review.md#playback-source-evidence-2026-09-28). +The owner rejected an active-session-only Spotify first release as the complete listening claim. [The full-scope proof plan](../development/playback-release-gates.md) therefore requires idle start, exact output, playlist switch and observed effect on each accepted profile. The official Spotify device-ID endpoints suggest a possible route around the stock HA entity's name/idle limits, but documentation alone does not prove device availability, account equivalence, current Development Mode access or a safe deployment contract. + ## 2026-09-27 write-outcome and reconciliation update This is a dated review of official API documentation only, not a live provider feasibility spike. The official pages describe mutation requests and successful responses, but do not document a general client idempotency-key contract or a provider-independent way to prove whether a timed-out request took effect. diff --git a/specs/CATALOG.md b/specs/CATALOG.md index 0022c84..f8ca88d 100644 --- a/specs/CATALOG.md +++ b/specs/CATALOG.md @@ -8,7 +8,7 @@ This is the human-readable view of [`catalog.json`](./catalog.json). Until gener | --- | --- | --- | --- | | `home-assistant-app-runtime` | Draft | [Home Assistant App runtime and Ingress](home-assistant-app-runtime-and-ingress.md) | Blocked by storage/recovery, supported platform matrix, and secret-key design | | `home-assistant-native-ui` | Ready for review | [Home Assistant-native UI foundation](home-assistant-native-ui.md) | Lit/TypeScript/Vite is accepted; blocked by supported matrix, public host-context/fallback proof, build/browser evidence, and visual-reference review | -| `listening-and-playback-control` | Draft | [Listening and playback control](listening-and-playback-control.md) | Narrow companion-broker authority accepted; blocked by broker protocol/security, source/player/output and account evidence, command confirmation, and YouTube Music support policy | +| `listening-and-playback-control` | Draft | [Listening and playback control](listening-and-playback-control.md) | Narrow broker accepted and active-session-only release rejected; blocked by complete Spotify/MA/YT Music profile proof, broker protocol/security, exact account/output and command confirmation, and any direct Spotify complement approval | | `provider-connections-and-authorization` | Draft | [Provider connections and authorization](provider-connections-and-authorization.md) | Blocked by direct callback reachability, secret/backup design, and provider feasibility spikes | | `library-import-and-provider-projections` | Draft | [Library import and provider projections](library-import-and-provider-projections.md) | Blocked by provider completeness, retention, and representative scale | | `recording-identity-resolution` | Draft | [Recording identity resolution](recording-identity-resolution.md) | Blocked by the labeled corpus and accepted automatic-link policy | diff --git a/specs/catalog.json b/specs/catalog.json index 07a84a3..e5b68e0 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -179,6 +179,7 @@ "docs/providers/provider-research.md", "docs/providers/home-assistant-ecosystem-review.md", "docs/providers/playback-integration-source-review.md", + "docs/development/playback-release-gates.md", "docs/architecture/system-architecture.md", "docs/open-questions.md" ], @@ -188,8 +189,8 @@ "documentation": [] }, "blockers": [ - "OQ-011 broker transport/pairing/identity and first playback source/output matrix; narrow-broker authority decision accepted", - "RG-007 Spotify idle-start/exact-output, MA caller/queue, account/source/player and exact playlist-reference feasibility", + "OQ-011 broker transport/pairing/identity and complete playback profile matrix; narrow-broker authority accepted, active-session-only release rejected", + "RG-007 Spotify idle-start/exact-output, MA caller/queue, account/source/player and exact playlist-reference feasibility; direct Spotify Connect complement needs separate approval", "Threat review of broker pairing, versioning, replay, caller identity, and action/entity allowlist", "Accepted observation freshness and command-reconciliation budgets", "YouTube Music community-source support and risk decision" diff --git a/specs/listening-and-playback-control.md b/specs/listening-and-playback-control.md index 08dbd79..f04ec50 100644 --- a/specs/listening-and-playback-control.md +++ b/specs/listening-and-playback-control.md @@ -6,14 +6,16 @@ - Owners: Symphonia maintainers - Scope: browse playable content, choose an explicit playback source and output, observe now-playing state, and control a configured external player from the Home Assistant-integrated Symphonia UI. - Related requirements: `SYM-PLAY-001`–`SYM-PLAY-009`, `SYM-PROV-021`–`SYM-PROV-022`, `SYM-UI-016`, `SYM-HA-011`, `SYM-ARCH-016`, `SYM-ACC-005`, `SYM-UI-001`–`SYM-UI-015`, `SYM-SEC-004`, `SYM-SEC-008` -- Related decisions/research: [product specification](../docs/product/product-specification.md), [domain model](../docs/domain/domain-model.md), [provider specification](../docs/providers/provider-specification.md), [system architecture](../docs/architecture/system-architecture.md), [ADR 0006](../docs/decisions/0006-narrow-home-assistant-playback-broker.md), [official playback research](../docs/providers/provider-research.md#2026-09-28-playback-and-home-assistant-api-update), [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md), [integration source review](../docs/providers/playback-integration-source-review.md), [UI foundation](home-assistant-native-ui.md), [App runtime](home-assistant-app-runtime-and-ingress.md), `OQ-011`, `RG-007` +- Related decisions/research: [product specification](../docs/product/product-specification.md), [domain model](../docs/domain/domain-model.md), [provider specification](../docs/providers/provider-specification.md), [system architecture](../docs/architecture/system-architecture.md), [ADR 0006](../docs/decisions/0006-narrow-home-assistant-playback-broker.md), [official playback research](../docs/providers/provider-research.md#2026-09-28-playback-and-home-assistant-api-update), [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md), [integration source review](../docs/providers/playback-integration-source-review.md), [full-scope proof plan](../docs/development/playback-release-gates.md), [UI foundation](home-assistant-native-ui.md), [App runtime](home-assistant-app-runtime-and-ingress.md), `OQ-011`, `RG-007` - Required review gates: product UX, Home Assistant platform/API, provider feasibility, architecture, accessibility, testing, documentation, security/privacy -- Open decisions blocking readiness: `OQ-011` supported playback-source matrix and broker transport/pairing/identity contract (the narrow-broker authority choice is accepted); `RG-007` Spotify idle-start/exact-device feasibility, account/source/player binding, MA caller identity and playlist-reference evidence; broker threat/version/upgrade review; live-state freshness and command-reconciliation thresholds; first-release Music Assistant/community policy +- Open decisions blocking readiness: `OQ-011` complete supported playback-profile matrix and broker transport/pairing/identity contract (the narrow-broker authority choice is accepted); `RG-007` Spotify idle-start/exact-device feasibility, account/source/player binding, MA caller identity and playlist-reference evidence; whether an explicitly consented direct Spotify Connect complement is authorized; broker threat/version/upgrade review; live-state freshness and command-reconciliation thresholds; Music Assistant/community YouTube Music support policy ## 1. Executive summary Symphonia should feel like a focused Home Assistant music surface: after setting up a library provider and a separate compatible playback source, a user can choose what to hear and where, see what that selected player reports now, and operate only controls it supports. The same App retains Symphonia's higher-assurance playlist import, identity resolution, and copy workflows. [ADR 0006](../docs/decisions/0006-narrow-home-assistant-playback-broker.md) selects a narrow companion integration that alone reads/commands explicitly allowed Home Assistant `media_player` entities; the App uses a typed broker port and does not stream or decode audio. +The owner rejected an active-session-only Spotify release as the complete product. The intended acceptance covers starting/switching playlists, exact output selection, now-playing and supported transport across the accepted Spotify and Music Assistant/YouTube Music profiles. Development may proceed in internal increments, but no narrower increment may be advertised as complete. The [release proof plan](../docs/development/playback-release-gates.md) separates that target from currently demonstrated provider behavior; impossible or unsafe upstream operations require an explicit exception or a separately approved route, not a fabricated control. + The primary correctness rule is **no inferred authority**: an imported Spotify playlist, a Home Assistant Spotify entity, a Music Assistant source, and a speaker may refer to different accounts or devices. A title match, shared provider name, or successful action request never proves the intended media is playing on the intended output. ```text @@ -61,7 +63,7 @@ No Symphonia listening route, playback port, companion playback broker, player d 1. Make selecting media and output, now-playing display, and available controls usable in a familiar Home Assistant-like view. 2. Preserve Symphonia's playlist-management experience alongside listening, without conflating playback queue changes with durable playlist edits. -3. Support configured Spotify HA and, subject to independent evidence/policy, Music Assistant or community sources such as YouTube Music through the same capability-driven UI. +3. Complete an evidence-backed Spotify and Music Assistant/YouTube Music listening profile, including starting and switching playable playlists on an exact compatible output. Each profile uses the same capability-driven UI but may have genuinely different supported controls and setup requirements. 4. Degrade playback independently of the library/import/copy system. ### 4.2 Non-goals @@ -69,7 +71,7 @@ No Symphonia listening route, playback port, companion playback broker, player d 1. Streaming, downloading, decoding, transcoding, or mixing audio in Symphonia. 2. Universal observation/control of arbitrary provider apps, browsers, devices, or accounts outside the selected integration. 3. Building a second Music Assistant server, a general Home Assistant API browser, or a generic service-call console. -4. Queue editing, multi-room synchronization, cross-service handoff, and playback history/scrobbling in the first vertical slice. Queue read and richer controls require their own verified capability and scope decision. +4. Queue editing, multi-room synchronization, cross-service handoff, and playback history/scrobbling in this contract. Starting a different playlist is in scope; editing the active queue or a saved playlist is not. Queue read and richer controls require their own verified capability and scope decision. 5. Changing the playlist-copy safety/confirmation contract; starting a playlist changes playback only, not the provider playlist. ### 4.3 Fixed invariants @@ -120,6 +122,7 @@ Text equivalent: a separately configured source and explicitly chosen target pre - A Symphonia provider connection exists but no HA playback source: library works; Listen shows setup guidance, not a fake player. - HA Spotify is configured but has no known Spotify Connect device: now-playing may be idle; start/output explains the missing device. While idle or on a restricted device, stock HA Spotify advertises only source selection, not normal `play_media` or WebSocket browse. A separately approved browse-service route is unproven; no idle-start workaround is assumed. - HA Spotify lists duplicate device names or an output selected by name cannot be freshly verified: keep “play here” disabled, even if `SELECT_SOURCE` is advertised. Source selection/transfer can alter an existing session and requires an explicit warning and effect observation. +- If a pinned HA-only Spotify route cannot meet the requested idle-start/exact-output contract in live tests, do not re-label an active-session-only prototype as complete. Evaluate an explicitly consented, device-ID-capable Spotify Connect complement only if the owner authorizes its separate grant, policy review and live proof; until then, this release gate remains open. - MA reports an external active source with current media but no active queue: display that media without fabricating queue controls, next item or playlist context. A base HA feature flag alone does not enable queue-only actions. - MA is invoked via a server-side Core bridge: verify the actual HA/MA caller mapping before showing private source results or attributing a play request to the listener; absent evidence, constrain to a configured single-admin/default-account profile or keep the path unavailable. - Music Assistant has a YouTube Music source but no playable target or expired cookie/PO-token service: show source unavailable with its unofficial status; do not fall back to Google/YouTube Data or ask Symphonia for cookies. @@ -168,6 +171,7 @@ Migration from an earlier install adds no automatic binding or allowed target. I | Domain/pure policy | target/source identity, freshness, capability intersection, command eligibility | HA constants/SDKs, provider DTOs, audio streams | | Application | list targets/sources, bind, browse, observe, command, reconcile | direct HA/Spotify calls, UI layout | | App playback adapter | broker DTO mapping, source-specific state/feature/media validation | provider library/copy policy, Core credentials, generic Core proxy | +| Optional direct Spotify Connect adapter, **only if separately accepted** | fresh device-ID and account-bound read/control mapping behind the same semantic ports | implicit reuse of HA/library grants, exposing tokens to browser/broker, fallback without user consent | | Companion broker | HA entity observation and typed, exact-target Core action dispatch | Symphonia database/grants, arbitrary HA service proxy, music-domain policy | | Infrastructure/composition | broker pairing, versioning, allowlist synchronization, bounded transport/subscriptions | implicit user account equivalence or broad App-to-Core authority | | Presentation | listening view model, confirmation, pending/stale/uncertain feedback | token access, raw service dispatch, authority decisions | @@ -277,23 +281,23 @@ Errors use impact → cause → next action → retained state. A third-party ou ## 13. Compatibility, migration, rollout, and rollback - Initial schema adds optional source/target binding and allowlist only after permission choice; existing provider connections, imports, plans, and jobs are unchanged. No implicit migration from provider account to HA entity. -- Phase 1 is broker protocol/security and source-matrix research plus deterministic fake-broker fixtures. Phase 2 proves one official Spotify HA end-to-end path through the broker. Phase 3 may add Music Assistant and optional unofficial community sources after separate feasibility/support decisions. +- Internal increments may prove the broker, Spotify HA, Music Assistant and optional community profiles in sequence, but the owner has not accepted an active-session-only Spotify increment as the complete listening release. The outward claim waits for the agreed full profile matrix, including a proven idle-start/exact-output solution or an explicitly accepted exception. A direct Spotify Connect complement, if approved, needs its own grant, account-binding, policy, recovery and test contract; it is not a silent broker fallback. - A supported HA version/entity feature change disables only affected playback capabilities, not library/copy. Compatibility is rechecked on HA, Music Assistant, and community integration upgrades. - Rollback removes listening UI/bridge while retaining library/copy state. Any external playback already started cannot be undone by rolling back Symphonia; no compensating stop is automatic. - Standalone mode shares the same listening contract but shows playback setup/unavailable until an explicitly authenticated supported playback adapter exists; it never assumes a hidden Home Assistant instance. ## 14. Testing strategy and numeric budget -Minimum **76 distinct cases**: +Minimum **112 distinct cases** for the requested full-profile target: | Area | Minimum distinct cases | Behaviors/risks covered | | --- | ---: | --- | -| Domain/configuration/capability policy | 16 | binding verification, exact references, missing scopes/targets, feature intersections, stale/unknown | -| State/application/concurrency | 18 | pending/confirmed/uncertain, external player races, duplicate click, out-of-order events, restart/target change | -| HA/MA/provider adapter contracts | 18 | Spotify idle/active/restricted and duplicate-device fixtures, WS versus service browse, MA queue/external and caller identity, exact refs, errors/timeouts | -| HTTP/UI/accessibility/sanitization | 14 | all visible states, keyboard/focus/announcements, mobile, hostile metadata/artwork, unsupported reasons | -| Integration/security/migration | 10 | broker pairing/version/allowlist/secret canaries, Ingress isolation, reconnect/no replay, rollback/backup, opt-in live smoke | -| **Total** | **76** | No double counting | +| Domain/configuration/capability policy | 20 | binding verification, exact references, missing scopes/targets, feature intersections, stale/unknown, profile-specific eligibility | +| State/application/concurrency | 24 | pending/confirmed/uncertain, external player races, duplicate click, out-of-order events, restart/target change, playlist switch | +| HA/MA/provider adapter contracts | 28 | Spotify idle/active/restricted and duplicate-device fixtures, device-ID route if approved, WS versus service browse, MA queue/external and caller identity, YouTube Music expiry, exact refs, errors/timeouts | +| HTTP/UI/accessibility/sanitization | 18 | all visible states, keyboard/focus/announcements, mobile, hostile metadata/artwork, unsupported reasons, source/output switching | +| Integration/security/migration | 22 | broker pairing/version/allowlist/secret canaries, Ingress isolation, separate direct grant if approved, account identity, reconnect/no replay, rollback/backup, opt-in live smoke | +| **Total** | **112** | No double counting | The ordinary suite uses synthetic HA entity states, feature flags, service responses, deterministic clocks, command IDs, source browsers, and no network/account. Each supported integration gets its own versioned fixture matrix; Spotify and Music Assistant are not treated as the same protocol mapping. Security tests require exact entity/action allowlisting, no MA name/URL/path fallback, caller-context checks, and secret canaries. Opt-in live smoke uses disposable/dedicated accounts and devices; it tests Spotify idle start/browse and duplicate-name outputs, MA queue/external source and actual user attribution, the supported source/output matrix, accepted-but-unobserved commands, independent external controller races, and cleanup. Human evidence covers Home Assistant-native visual parity, phone/wide, light/dark, keyboard/screen reader, long metadata, multiple players, and all blocked/uncertain states. Live tests supplement, not replace, deterministic contracts. @@ -327,6 +331,7 @@ The ordinary suite uses synthetic HA entity states, feature flags, service respo 17. Given an MA player on an external source without an active queue, now-playing may show the reported item, but shuffle/repeat/clear-queue/next-item claims are suppressed unless the effective source/player profile proves them. 18. Given a server-side HA request whose MA user identity is missing, defaulted or mismatched, private library/search/play is not attributed to the listener; unchecked URL/path/name inputs never reach MA's permissive resolver. 19. Given no broker, a revoked pairing, an incompatible protocol version, or a lost broker response, listening is unavailable/uncertain without fallback to a broad Core API token or replay; library/copy remains usable. +20. Given the complete-release claim, a dedicated-account fixture demonstrates idle start, exact output, playlist switch, current media and applicable transport for every accepted Spotify and Music Assistant/YouTube Music profile. A failed or unsupported operation produces an explicit owner-reviewed exception or keeps the profile outside that claim; a working active-session-only prototype alone cannot satisfy this scenario. ## 17. Requirements traceability @@ -347,7 +352,7 @@ The ordinary suite uses synthetic HA entity states, feature flags, service respo 1. Complete `OQ-011` and `RG-007`: the broker authority choice is accepted, but its transport/pairing/threat model, supported source/output matrix, identity rules, and bounded state/command thresholds remain open. 2. Add architecture, broker protocol/allowlist, capability, binding, and command-confirmation contract tests with fake HA/Spotify/MA observations. 3. Implement pure listening policy and application ports/use cases with ephemeral state and minimal persistent binding only. -4. Implement the approved, versioned companion broker and App adapter only after this SDD is ready, then prove the Spotify HA vertical; add Music Assistant/community mappings only after their separate evidence/support decision. +4. Implement the approved, versioned companion broker and App adapter only after this SDD is ready, then prove Spotify HA, Music Assistant and each accepted community mapping separately. If the owner approves a direct Spotify Connect complement, specify and review its separate grant and adapter before implementation. 5. Add shared now-playing/transport UI family, Listen route, accessibility, documentation, and Ingress/standalone degraded-state behavior. 6. Run opt-in supported-matrix smoke and security/visual review; update source evidence and catalog before readiness/implementation claims. @@ -355,8 +360,8 @@ The ordinary suite uses synthetic HA entity states, feature flags, service respo - [ ] SDD is `Ready for implementation`, explicit owner approval to implement exists, and broker protocol/security plus source-matrix blockers are resolved. - [ ] Account/source/player/output distinctions, exact-reference rules, and unavailable/unverified states pass acceptance and documentation review. -- [ ] At least 76 distinct deterministic cases, architecture checks, permission/allowlist tests, and secret-canary gates pass. -- [ ] Spotify HA and any advertised MA/community path have dedicated opt-in integration evidence on supported versions/accounts/targets. +- [ ] At least 112 distinct deterministic cases, architecture checks, permission/allowlist tests, and secret-canary gates pass. +- [ ] Every advertised Spotify and MA/YouTube Music profile has dedicated opt-in idle/start/output/playlist-switch/current-control evidence on supported versions/accounts/targets, or an explicit owner-reviewed exception; an active-session-only Spotify path is not a complete-release claim. - [ ] Command acceptance versus observed effect, timeout/uncertain/no-replay, external races, and restart behavior are proven. - [ ] Listening UI passes the shared component catalog, HA visual-reference, responsive, localization, accessibility, and Ingress/standalone gates. - [ ] No audio pipeline, implicit account binding, broad App-to-Core access/generic Core proxy, plaintext listening history, or playlist-write retry leakage exists. @@ -369,4 +374,4 @@ The ordinary suite uses synthetic HA entity states, feature flags, service respo - Related SDDs: [App runtime](home-assistant-app-runtime-and-ingress.md), [UI foundation](home-assistant-native-ui.md), [provider connections](provider-connections-and-authorization.md), [library import](library-import-and-provider-projections.md), [playlist copy](one-time-playlist-copy.md). - Accepted authority decision: [ADR 0006](../docs/decisions/0006-narrow-home-assistant-playback-broker.md) requires a narrow Home Assistant companion broker for listening; App-to-Core Supervisor proxy is not the initial playback path. The broker transport/pairing protocol and supported source matrix are **not accepted** yet. - Rejected: automatic provider-to-player linking, guessing playlist URIs from titles, command-success from HTTP acceptance, silent YT Music unofficial fallback, and treating Symphonia as an audio streamer. -- Follow-up: queue editing, cross-player handoff/multi-room, direct provider playback, and listening history require separate scope/evidence decisions. +- Follow-up: queue editing, cross-player handoff/multi-room, other direct-provider playback, and listening history require separate scope/evidence decisions. The proposed Spotify Connect complement remains unaccepted until its own approval and review. From a4cc8e9d6423c2f9c72c9cc4b9c8fff7a83441ad Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 28 Sep 2026 10:57:04 +0200 Subject: [PATCH 166/167] test: enforce SDD readiness evidence in catalog validator --- docs/development/quality-audit.md | 2 +- specs/README.md | 11 ++- specs/catalog.schema.json | 49 +++++++++---- tests/test_spec_validator.py | 112 +++++++++++++++++++++++++++++- tools/validate_specs.py | 47 +++++++++---- 5 files changed, 190 insertions(+), 31 deletions(-) diff --git a/docs/development/quality-audit.md b/docs/development/quality-audit.md index 8109f0c..f0eba6e 100644 --- a/docs/development/quality-audit.md +++ b/docs/development/quality-audit.md @@ -42,7 +42,7 @@ Re-ran the documented offline commands against the local `develop` working tree Graphify's `--code-only --no-cluster` extraction produced **1,181 nodes and 3,060 edges**. `OperationRepository` remains the most connected symbol (57 edges); its one-hop dependents include runtime HTTP/resources, copy and import execution, the operation runner, and their tests. This reinforces the existing plan to define repository ports and preserve restart/reconciliation characterization before changing operation persistence. The broker design is kept out of domain/application imports and is not wired into this graph while its SDD is Draft. -The local `make verify` run passed **232 `unittest` cases (2 skipped)** plus specification/whitespace checks. The isolated Lit spike also passed `npm ci --ignore-scripts`, TypeScript, presentation-boundary checks, three pure fallback tests, build, and relative-asset verification. Neither result proves Home Assistant integration, live playback, visual parity, or CI on GitHub. +The latest local `make verify` run passed **239 `unittest` cases (2 skipped)** plus specification/whitespace checks. Seven new validator cases exercise readiness blockers, implemented evidence, valid status transitions, malformed arrays, and missing SDD status. The isolated Lit spike also passed `npm ci --ignore-scripts`, TypeScript, presentation-boundary checks, three pure fallback tests, build, and relative-asset verification. Neither result proves Home Assistant integration, live playback, or visual parity. The new validator checks are structural guards, not substitutes for human review or live evidence. The previously absent coverage measurement is now reproducible with pinned `coverage.py` 7.16.1 (development-only, not a runtime dependency): diff --git a/specs/README.md b/specs/README.md index 9811ae2..e1b265a 100644 --- a/specs/README.md +++ b/specs/README.md @@ -117,7 +117,7 @@ Readiness is necessary but not authorization to implement. The owner must still Persistent synchronization intentionally has no implementation SDD yet. It remains future scope until playlist ownership, conflict semantics, provider revision evidence, and the copy contract are resolved. -## Manual validation while the repository has no toolchain +## Automated and manual validation Run after every SDD or catalog change: @@ -129,7 +129,12 @@ PYTHONPATH=. python3 tools/validate_specs.py The validator checks that `catalog.json` parses, referenced files exist, each SDD has one known catalog capability ID, catalog statuses and paths agree with `CATALOG.md`, requirement IDs are present in the horizontal specifications, -and local Markdown links resolve. Generated/vendor directories such as +and local Markdown links resolve. It also rejects a `Ready for implementation` +or `Implemented` catalog entry with remaining blockers, requires an implemented +entry to list code, test, and documentation evidence, and requires exactly one +SDD status declaration. These are structural guards, **not** a substitute for +owner approval, threat review, live provider proof, or the readiness checklist. +Generated/vendor directories such as `node_modules`, `dist`, and virtual environments are excluded from Markdown scanning; owned documentation is still checked even when dependencies are installed locally. @@ -143,4 +148,4 @@ Also verify: 5. Requirement IDs referenced by SDDs exist in the horizontal specifications. 6. `CATALOG.md` agrees with `catalog.json`. -A repository-native validator and generated catalog may be added after the implementation toolchain is selected; that tooling choice must not drive the application stack. +The repository-native validator uses only the Python standard library. A generated catalog and richer semantic checks may be added later; tooling must not drive the application stack. diff --git a/specs/catalog.schema.json b/specs/catalog.schema.json index dc79df9..3bfe2d6 100644 --- a/specs/catalog.schema.json +++ b/specs/catalog.schema.json @@ -18,6 +18,36 @@ "items": { "type": "object", "additionalProperties": false, + "allOf": [ + { + "if": { + "properties": { + "status": { "enum": ["ready-for-implementation", "implemented"] } + }, + "required": ["status"] + }, + "then": { + "properties": { "blockers": { "maxItems": 0 } } + } + }, + { + "if": { + "properties": { "status": { "const": "implemented" } }, + "required": ["status"] + }, + "then": { + "properties": { + "implementationEvidence": { + "properties": { + "code": { "minItems": 1 }, + "tests": { "minItems": 1 }, + "documentation": { "minItems": 1 } + } + } + } + } + } + ], "required": [ "id", "title", @@ -62,17 +92,12 @@ }, "requirements": { "type": "array", - "items": { - "type": "string", - "pattern": "^SYM-[A-Z]+-[0-9]{3}$" - }, + "items": { "type": "string", "pattern": "^SYM-[A-Z]+-[0-9]{3}$", "minLength": 1 }, "uniqueItems": true }, "evidence": { "type": "array", - "items": { - "type": "string" - }, + "items": { "type": "string", "minLength": 1 }, "uniqueItems": true }, "implementationEvidence": { @@ -82,26 +107,24 @@ "properties": { "code": { "type": "array", - "items": { "type": "string" }, + "items": { "type": "string", "minLength": 1 }, "uniqueItems": true }, "tests": { "type": "array", - "items": { "type": "string" }, + "items": { "type": "string", "minLength": 1 }, "uniqueItems": true }, "documentation": { "type": "array", - "items": { "type": "string" }, + "items": { "type": "string", "minLength": 1 }, "uniqueItems": true } } }, "blockers": { "type": "array", - "items": { - "type": "string" - }, + "items": { "type": "string", "minLength": 1 }, "uniqueItems": true } } diff --git a/tests/test_spec_validator.py b/tests/test_spec_validator.py index 2162446..f5bc444 100644 --- a/tests/test_spec_validator.py +++ b/tests/test_spec_validator.py @@ -2,14 +2,124 @@ from tempfile import TemporaryDirectory import unittest -from tools.validate_specs import _validate_markdown_links, _validate_requirements, validate +from tools.validate_specs import ( + _validate_catalog_shape, + _validate_markdown_links, + _validate_requirements, + _validate_sdds, + validate, +) class SpecValidatorTests(unittest.TestCase): + @staticmethod + def _capability(status: str, blockers: list[str]) -> dict[str, object]: + return { + "id": "example", + "title": "Example", + "status": status, + "scope": "Example scope", + "owner": "Owner", + "specification": "specs/example.md", + "requirements": ["SYM-PLAY-001"], + "evidence": [], + "implementationEvidence": {"code": [], "tests": [], "documentation": []}, + "blockers": blockers, + } + def test_repository_specifications_are_consistent(self) -> None: root = Path(__file__).resolve().parents[1] self.assertEqual(validate(root), []) + def test_ready_sdd_cannot_keep_catalog_blockers(self) -> None: + capability = self._capability( + "ready-for-implementation", ["unproven broker pairing"] + ) + errors: list[str] = [] + _validate_catalog_shape( + Path("/unused"), {"version": 1, "capabilities": [capability]}, errors + ) + self.assertIn( + "catalog capability 0 cannot be ready-for-implementation while blockers remain", + errors, + ) + + def test_implemented_sdd_needs_code_tests_and_documentation_evidence(self) -> None: + capability = self._capability("implemented", []) + errors: list[str] = [] + _validate_catalog_shape( + Path("/unused"), {"version": 1, "capabilities": [capability]}, errors + ) + for field in ("code", "tests", "documentation"): + self.assertIn( + f"catalog capability 0 implemented status requires implementationEvidence.{field}", + errors, + ) + + def test_draft_can_retain_blockers_and_foundation_evidence(self) -> None: + capability = self._capability("draft", ["live provider proof"]) + errors: list[str] = [] + _validate_catalog_shape( + Path("/unused"), {"version": 1, "capabilities": [capability]}, errors + ) + self.assertEqual(errors, []) + + def test_ready_sdd_with_no_blockers_is_structurally_valid(self) -> None: + capability = self._capability("ready-for-implementation", []) + errors: list[str] = [] + _validate_catalog_shape( + Path("/unused"), {"version": 1, "capabilities": [capability]}, errors + ) + self.assertEqual(errors, []) + + def test_implemented_sdd_with_complete_evidence_is_structurally_valid(self) -> None: + capability = self._capability("implemented", []) + capability["implementationEvidence"] = { + "code": ["src/example.py"], + "tests": ["tests/test_example.py"], + "documentation": ["docs/example.md"], + } + errors: list[str] = [] + _validate_catalog_shape( + Path("/unused"), {"version": 1, "capabilities": [capability]}, errors + ) + self.assertEqual(errors, []) + + def test_malformed_catalog_arrays_are_reported_without_crashing(self) -> None: + capability = self._capability("draft", []) + capability["requirements"] = [{"not": "a requirement"}] + capability["blockers"] = [{"not": "a blocker"}] + errors: list[str] = [] + _validate_catalog_shape( + Path("/unused"), {"version": 1, "capabilities": [capability]}, errors + ) + self.assertIn( + "catalog capability 0 requirements must be an array of unique non-empty strings", + errors, + ) + self.assertIn( + "catalog capability 0 blockers must be an array of unique non-empty strings", + errors, + ) + + with TemporaryDirectory() as directory: + root = Path(directory) + (root / "docs").mkdir() + _validate_requirements(root, [capability], errors) + self.assertEqual(len(errors), 2) + + def test_primary_sdd_must_declare_exactly_one_status(self) -> None: + with TemporaryDirectory() as directory: + root = Path(directory) + specs = root / "specs" + specs.mkdir() + (specs / "example.md").write_text( + "- Catalog capability ID: `example`\n", encoding="utf-8" + ) + errors: list[str] = [] + _validate_sdds(root, [self._capability("draft", [])], errors) + self.assertEqual(errors, ["specs/example.md must declare exactly one status"]) + def test_generated_dependency_markdown_is_not_treated_as_ours(self) -> None: with TemporaryDirectory() as directory: root = Path(directory) diff --git a/tools/validate_specs.py b/tools/validate_specs.py index 3b3660f..5ce8617 100644 --- a/tools/validate_specs.py +++ b/tools/validate_specs.py @@ -63,6 +63,14 @@ def _local_path(root: Path, raw: str) -> Path: return root / raw +def _unique_text_array(value: Any) -> bool: + return ( + isinstance(value, list) + and all(isinstance(item, str) and bool(item.strip()) for item in value) + and len(value) == len(set(value)) + ) + + def _validate_catalog_shape(root: Path, catalog: Any, errors: list[str]) -> list[dict[str, Any]]: if not isinstance(catalog, dict): _add(errors, "catalog.json must contain an object") @@ -103,7 +111,7 @@ def _validate_catalog_shape(root: Path, catalog: Any, errors: list[str]) -> list else: ids.add(capability_id) status = capability.get("status") - if status not in CATALOG_STATUSES: + if not isinstance(status, str) or status not in CATALOG_STATUSES: _add(errors, f"{prefix} has invalid status: {status!r}") specification = capability.get("specification") if not isinstance(specification, str) or not specification: @@ -113,19 +121,25 @@ def _validate_catalog_shape(root: Path, catalog: Any, errors: list[str]) -> list else: specifications.add(specification) requirements = capability.get("requirements") - if not isinstance(requirements, list) or len(requirements) != len(set(requirements)): - _add(errors, f"{prefix} requirements must be a unique array") + if not _unique_text_array(requirements): + _add(errors, f"{prefix} requirements must be an array of unique non-empty strings") for field in ("evidence", "blockers"): - if not isinstance(capability.get(field), list): - _add(errors, f"{prefix} {field} must be an array") + if not _unique_text_array(capability.get(field)): + _add(errors, f"{prefix} {field} must be an array of unique non-empty strings") + blockers = capability.get("blockers") + if status in ("ready-for-implementation", "implemented") and isinstance(blockers, list) and blockers: + _add(errors, f"{prefix} cannot be {status} while blockers remain") implementation = capability.get("implementationEvidence") if not isinstance(implementation, dict): _add(errors, f"{prefix} implementationEvidence must be an object") else: for field in ("code", "tests", "documentation"): - if not isinstance(implementation.get(field), list): - _add(errors, f"{prefix} implementationEvidence.{field} must be an array") + values = implementation.get(field) + if not _unique_text_array(values): + _add(errors, f"{prefix} implementationEvidence.{field} must be an array of unique non-empty strings") + elif status == "implemented" and not values: + _add(errors, f"{prefix} implemented status requires implementationEvidence.{field}") return [capability for capability in capabilities if isinstance(capability, dict)] @@ -164,10 +178,13 @@ def _validate_sdds(root: Path, capabilities: list[dict[str, Any]], errors: list[ matches = CAPABILITY_ID_RE.findall(text) if matches != [capability_id]: _add(errors, f"{specification} must declare exactly catalog capability ID {capability_id!r}") - status_match = re.search(r"^\s*-\s+Status:\s+(.+?)\s*$", text, re.MULTILINE) - expected_status = CATALOG_STATUSES.get(capability.get("status")) - if status_match and expected_status and status_match.group(1) != expected_status: - _add(errors, f"{specification} status {status_match.group(1)!r} disagrees with catalog {expected_status!r}") + status_matches = re.findall(r"^\s*-\s+Status:\s+(.+?)\s*$", text, re.MULTILINE) + status = capability.get("status") + expected_status = CATALOG_STATUSES.get(status) if isinstance(status, str) else None + if len(status_matches) != 1: + _add(errors, f"{specification} must declare exactly one status") + elif expected_status and status_matches[0] != expected_status: + _add(errors, f"{specification} status {status_matches[0]!r} disagrees with catalog {expected_status!r}") for path in sorted((root / "specs").glob("*.md")): if path.name in EXCLUDED_SPEC_MARKDOWN: @@ -185,7 +202,10 @@ def _validate_requirements(root: Path, capabilities: list[dict[str, Any]], error for path in _repository_markdown(root / "docs"): known.update(REQUIREMENT_RE.findall(path.read_text(encoding="utf-8"))) for capability in capabilities: - for requirement in capability.get("requirements", []): + requirements = capability.get("requirements") + for requirement in requirements if isinstance(requirements, list) else []: + if not isinstance(requirement, str): + continue # The catalog-shape error already identifies malformed values. if requirement not in known: _add(errors, f"{capability.get('id', '')} references unknown requirement {requirement}") @@ -203,7 +223,8 @@ def _validate_catalog_markdown(root: Path, capabilities: list[dict[str, Any]], e row = rows.get(capability_id) if row is None: continue - expected_status = CATALOG_STATUSES.get(capability.get("status")) + status = capability.get("status") + expected_status = CATALOG_STATUSES.get(status) if isinstance(status, str) else None expected_path = Path(capability.get("specification", "")).name if row != (expected_status, expected_path): _add(errors, f"specs/CATALOG.md row for {capability_id} disagrees with catalog.json") From 052ec32e0453376436124899d7f8aa14e4070cc4 Mon Sep 17 00:00:00 2001 From: Efra Espada Date: Mon, 28 Sep 2026 11:05:10 +0200 Subject: [PATCH 167/167] docs: prepare playback broker RFC and disposable HA-MA lab --- .github/workflows/verify.yml | 11 +++ .gitignore | 1 + docs/README.md | 2 + .../playback-broker-protocol-rfc.md | 80 +++++++++++++++++++ docs/architecture/system-architecture.md | 2 +- docs/development/playback-lab/.env.example | 4 + docs/development/playback-lab/README.md | 40 ++++++++++ docs/development/playback-lab/compose.yaml | 28 +++++++ docs/development/playback-release-gates.md | 4 +- docs/development/quality-audit.md | 2 + docs/open-questions.md | 4 +- specs/catalog.json | 4 +- specs/listening-and-playback-control.md | 4 +- 13 files changed, 179 insertions(+), 7 deletions(-) create mode 100644 docs/architecture/playback-broker-protocol-rfc.md create mode 100644 docs/development/playback-lab/.env.example create mode 100644 docs/development/playback-lab/README.md create mode 100644 docs/development/playback-lab/compose.yaml diff --git a/.github/workflows/verify.yml b/.github/workflows/verify.yml index 6501979..98afbaf 100644 --- a/.github/workflows/verify.yml +++ b/.github/workflows/verify.yml @@ -87,3 +87,14 @@ jobs: run: | PYTHONPATH=src:. python -m coverage run -m unittest discover -s tests -q python -m coverage report --sort=cover + + playback-lab-config: + runs-on: ubuntu-24.04 + defaults: + run: + working-directory: docs/development/playback-lab + steps: + - name: Check out repository + uses: actions/checkout@v4 + - name: Validate disposable lab configuration without starting services + run: docker compose --env-file .env.example -f compose.yaml config --quiet diff --git a/.gitignore b/.gitignore index cf09668..a9ef423 100644 --- a/.gitignore +++ b/.gitignore @@ -12,3 +12,4 @@ node_modules/ *.db .repowise/ graphify-out/ +docs/development/playback-lab/.env diff --git a/docs/README.md b/docs/README.md index 1dd82c6..a237b79 100644 --- a/docs/README.md +++ b/docs/README.md @@ -14,11 +14,13 @@ This documentation is the horizontal implementation contract for Symphonia. It d | [Home Assistant-native UI specification](product/home-assistant-ui-specification.md) | Cross-cutting UI direction, component families, host context, accessibility, responsive and visual compatibility requirements | Feature-specific content/state or frontend framework selection | | [Domain model](domain/domain-model.md) | Ubiquitous language, invariants, identity, copy and sync semantics | Provider API facts | | [System architecture](architecture/system-architecture.md) | Boundaries, execution model, security and operations | Final implementation stack | +| [Playback broker protocol RFC](architecture/playback-broker-protocol-rfc.md) | Proposed channel, pairing, command semantics, threat model and topology proof | Accepted transport or implementation permission | | [Provider specification](providers/provider-specification.md) | Provider port, capabilities, normalized errors | Claims about a specific API | | [Provider research](providers/provider-research.md) | Dated, sourced facts about provider APIs | Product policy or permanent architecture | | [Home Assistant music ecosystem review](providers/home-assistant-ecosystem-review.md) | Reusable patterns and cautions from existing HA music projects | Dependency selection or provider guarantees | | [Playback integration source review](providers/playback-integration-source-review.md) | Dated source-code findings, operation limits, risk and live proof matrix for HA Spotify, Music Assistant, and YT Music projects | Official API guarantees, release support, or SDD readiness | | [Complete listening proof plan](development/playback-release-gates.md) | Desired Spotify/MA/YT Music release claims, evidence matrix, broker threat gate, and disposable-system probes | Permission to implement or a guarantee that upstream services expose every action | +| [Disposable playback lab](development/playback-lab/README.md) | Isolated Home Assistant Core and Music Assistant test environment, launch limits and evidence procedure | Supervisor/Ingress proof or a live provider/device pass | | [Development specification](development/development-specification.md) | Specification workflow, testing and delivery gates | Product scope | | [Local quality audit](development/quality-audit.md) | Reproducible offline RepoWise/Graphify review and current architecture/test debt | A release or SDD readiness claim | | [ADRs](decisions/README.md) | Decisions that have actually been accepted | Proposals and guesses | diff --git a/docs/architecture/playback-broker-protocol-rfc.md b/docs/architecture/playback-broker-protocol-rfc.md new file mode 100644 index 0000000..41df515 --- /dev/null +++ b/docs/architecture/playback-broker-protocol-rfc.md @@ -0,0 +1,80 @@ +# RFC: narrow Home Assistant playback broker protocol + +**Status:** proposed design for review; no production implementation permission + +**Reviewed:** 2026-09-28 + +**Decision already accepted:** [ADR 0006](../decisions/0006-narrow-home-assistant-playback-broker.md) places Home Assistant Core playback authority in a companion integration, not in the App or browser. This RFC proposes a transport and security contract; it has not been proven in a disposable Home Assistant installation or accepted as the final protocol. + +## Deployment and trust model + +Home Assistant Core loads a `symphonia` custom integration using a [config flow](https://developers.home-assistant.io/docs/core/integration/config_flow/) and the normal custom-integration [file layout](https://developers.home-assistant.io/docs/creating_integration_file_structure/). The Symphonia App owns its own music data, Ingress UI and internal broker endpoint. Home Assistant's [App communication documentation](https://developers.home-assistant.io/docs/apps/communication/) describes internal App naming/networking; that network helps discovery but is **not** an authorization boundary. A separate internal listener or route must not become an unauthenticated public management API. + +```text +authenticated Ingress admin -> App listening use case -> typed broker port + <-> private, authenticated channel +HA companion integration -> exact allowed HA media_player -> HA player integration +``` + +Text equivalent: the browser calls only the App's authenticated use case. The paired integration connects to the App over a dedicated channel, reports selected player observations, and executes a closed set of typed operations against one allowlisted Core entity. Library imports, provider grants, playlist copies and audio streams do not cross this channel. + +The primary product currently has one local Symphonia user and an admin-only Ingress management surface. This does **not** prove that an App-side action carries the same Home Assistant or Music Assistant user identity. The protocol therefore treats listener identity and playback-source identity as separate evidence. Until the actual MA caller mapping is proven, private per-user MA browsing/playback is denied or explicitly bound to one configured admin/default account without claiming per-listener attribution. + +## Candidate transport, not yet selected + +**Preferred candidate for the topology spike:** the HA integration initiates one outbound, versioned WebSocket connection to an App listener address discovered from a configured internal App alias. The App never requests `homeassistant_api: true` or holds `SUPERVISOR_TOKEN`; the integration holds Core authority. The channel needs TLS with explicit App certificate/public-key pinning plus a separately generated integration credential, or an equivalently reviewed encrypted and mutually authenticated construction. App alias/DNS, local networking and Ingress cookies are not authenticators. + +Why this direction is preferred: the App can send bounded commands and receive observations over one channel without a general Core token, a public HA endpoint, MQTT dependency, or polling every player from the browser. This is a hypothesis: the HA/App network reachability, certificate provisioning, upgrade behavior, and configuration UX must be demonstrated before selection. + +| Alternative | Rejection/remaining condition | +| --- | --- | +| App calls HA REST/WebSocket with `SUPERVISOR_TOKEN` | Broad Core credential contradicts ADR 0006 even if application code allowlists calls. | +| Browser calls a custom HA WebSocket command | Browser token/host-context coupling and direct action authority contradict the App-owned use-case boundary. | +| Custom HA HTTP endpoint polled by App | Requires a separately proven narrow authentication/authorization surface and may expose an endpoint beyond the internal network; not automatically safer than a reverse channel. | +| MQTT/general event bus | Adds another service and still needs pairing, exact-target authorization, command correlation and no-replay semantics. | + +## Pairing and lifecycle proposal + +1. The App creates its installation identity and channel certificate/key inside its protected persistent state. The admin views a short-lived, single-use pairing challenge and the App certificate fingerprint in Ingress. The challenge is **not** the long-term broker credential. No Core, broker or provider bearer token is returned to ordinary browser state. +2. The HA integration config flow asks for the App alias/address and pairing challenge. Before any secret exchange, the admin compares the displayed certificate fingerprint with the one observed by the integration; a mismatch aborts. A successful flow exchanges a new random long-term channel credential over the pinned encrypted channel. The HA-side credential location, export/backup exposure and revocation behavior require proof; a config entry must **not** be presumed encrypted. The App needs an approved protected secret store. Actual certificate/key generation, encrypted key source and backup behavior are release blockers; a self-signed certificate without pin verification is not accepted. +3. A connection starts with mutual challenge/response, installation identities, a fresh session nonce, and major/minor protocol negotiation. An incompatible major version disables listening with a repair message; library/copy remain available. A reconnect gets a new epoch and never drains old commands. +4. An administrator can revoke and re-pair. Rotation overlaps old/new credentials only for one bounded window, fences old sessions, and emits a sanitized audit category. Backup restoration to another HA instance cannot silently reuse the old pairing; the secret/key restore policy must explicitly decide whether to re-pair. +5. On unload/uninstall, the integration cancels subscriptions and closes the channel. The App invalidates observations and pending commands on disconnect; it never queues control actions for replay. + +The one-time challenge, certificate fingerprint, long-term secret and Ingress admin identity each have different roles. No authentication material is embedded in an App option, URL, log, diagnostic bundle, artwork URL, playlist reference or copied browser state. Secret storage is blocked by the App runtime SDD's encryption-key/backup decision. + +## Minimal version-1 semantic contract + +The wire format is not accepted until the topology/threat spike, but every operation must map to a semantic port. Proposed message families: + +| Message | Direction | Required checks | +| --- | --- | --- | +| `hello` / `health` | both | installation identity, protocol version, paired channel, feature set; no account names/secrets | +| `players.list` | integration → App | only explicitly allowed `media_player` entities; stable entity ID plus integration/source evidence, never name as authority | +| `player.observe` | integration → App | exact entity, session epoch + monotonic sequence, observation time, bounded safe attributes and effective features; absence stays unknown | +| `media.browse` / `media.search` | App → integration | exact entity, source profile, validated typed cursor/query and current feature/permission check; bounded result page; no raw URL/path | +| `player.command` | App → integration | one exact allowed entity, closed action/media-kind enum, typed exact reference, current feature and account checks, correlation ID, no wildcard/area/broadcast | +| `command.accepted` / `command.rejected` | integration → App | transport/HA dispatch category only; never asserts playback effect | + +The action set is initially restricted to read/browse/search where proven and the applicable play, pause, resume, stop, next, previous, seek, volume, source/output and exact-media-start operations. Each is separately feature-gated; the broker may reject an advertised HA feature when the concrete profile is unsafe. MA's permissive name/URL/path search fallback is not an exact media reference. Spotify's name-based source selection is not a stable output ID. A directly consented Spotify Connect adapter, if later approved, uses a **separate** provider port and grant; it does not extend this HA broker's Core authority. + +One unresolved command per selected player is allowed from Symphonia. The broker validates operation, entity, profile and reference independently from App-side checks. A successful command dispatch becomes `accepted`; the App confirms `playing`, `paused` or another effect only after a fresh observation matches the requested media/output. A timeout, disconnect or race is `uncertain`; no automatic command retry occurs. Broker session epoch and observation sequence fence stale events; command correlation cannot turn a delayed response into success. External HA controllers may change the same player at any time. + +The topology spike must select concrete JSON schemas, maximum request/result sizes, browse pagination, nonce lifetime/replay cache, per-player rate limit, freshness and confirmation budgets. They are **not** left to caller-provided values. Source-specific versioned fixtures must prove every field and rejection path. + +## Threat review and negative contracts + +| Threat | Required proof | +| --- | --- | +| Another App on the internal network connects or impersonates the broker | Correctly pinned encrypted channel, independent broker credential, pairing expiry/single use, failed-auth rate limit | +| Ingress client forges entity, action, media ID or prior selection | App authorization plus broker-side exact entity/action/media-kind allowlists and stale-selector fence | +| Stolen or restored pairing secret replays an old command | Rotation/revocation, session nonce/epoch, no queued replay, bounded replay defense | +| HA/MA current user differs from Symphonia listener | Distinct caller/source identities; private content denied absent proved binding; no name-only inference | +| Hostile metadata, artwork or diagnostics leak secrets/track history | Bounded sanitized DTOs, safe URL schemes, no raw HA state or listening history in ordinary logs/backups | +| Version skew or broker outage harms library/copy | Playback-only unavailable state and explicit repair path; durable music workflows remain independent | + +Every negative contract must have deterministic fake-broker tests and an opt-in disposable-HA test. Security review must include whether internal-network peers can observe/manipulate transport, HA custom integration permission semantics, denial of service from rapid browse/command calls, compromised admin session, upgrade/rollback, and restore-to-new-instance behavior. + +## Decision gate + +This RFC is ready for a topology/security spike, **not** for production implementation. Select the transport only after a disposable HA installation proves internal alias reachability, config-flow pairing, certificate pinning and storage, mutually authenticated reconnect, exact entity/action enforcement, version skew, admin/MA attribution, and secret-safe backup/restore. Record the selected protocol in a new accepted ADR or an explicit ADR 0006 successor, then update the listening/App-runtime SDDs, catalog and conformance tests together. An unverified design sketch does not clear `OQ-011` or `RG-007`. diff --git a/docs/architecture/system-architecture.md b/docs/architecture/system-architecture.md index 78a173f..ef8642d 100644 --- a/docs/architecture/system-architecture.md +++ b/docs/architecture/system-architecture.md @@ -143,7 +143,7 @@ Whether standalone packaging ships in the first public release or immediately af ### Companion Home Assistant integration -A companion custom integration is required for the selected HA playback path under [ADR 0006](../decisions/0006-narrow-home-assistant-playback-broker.md); it MAY later expose other native entities, actions, events, and configuration discovery. It must use a stable, authenticated, versioned Symphonia contract and MUST NOT duplicate matching, sync, provider-credential, or durable-write retry logic. The exact transport and pairing protocol remain unselected. +A companion custom integration is required for the selected HA playback path under [ADR 0006](../decisions/0006-narrow-home-assistant-playback-broker.md); it MAY later expose other native entities, actions, events, and configuration discovery. It must use a stable, authenticated, versioned Symphonia contract and MUST NOT duplicate matching, sync, provider-credential, or durable-write retry logic. The exact transport and pairing protocol remain unselected. The [broker protocol RFC](playback-broker-protocol-rfc.md) proposes a reverse, mutually authenticated channel and enumerates the topology, secret-storage, caller-identity, replay, and version proofs needed before selection. The MVP does not require the companion integration to broker provider authorization. Provider authorization is owned by the App's adapters following diff --git a/docs/development/playback-lab/.env.example b/docs/development/playback-lab/.env.example new file mode 100644 index 0000000..0de912a --- /dev/null +++ b/docs/development/playback-lab/.env.example @@ -0,0 +1,4 @@ +# Bootstrap only. Replace mutable tags with recorded digests before live evidence. +HA_IMAGE=ghcr.io/home-assistant/home-assistant:stable +MA_IMAGE=ghcr.io/music-assistant/server:latest +LAB_TZ=Europe/Madrid diff --git a/docs/development/playback-lab/README.md b/docs/development/playback-lab/README.md new file mode 100644 index 0000000..2ac5f91 --- /dev/null +++ b/docs/development/playback-lab/README.md @@ -0,0 +1,40 @@ +# Disposable playback evidence lab + +**Status:** prepared configuration, not a completed live feasibility test + +**Reviewed:** 2026-09-28 + +This lab prepares a dedicated [Home Assistant Container](https://www.home-assistant.io/installation/linux) and [Music Assistant server](https://www.music-assistant.io/installation/) for the [RG-007 matrix](../../open-questions.md#rg-007--listening-source-target-and-command-feasibility). It contains no provider credentials, player addresses, broker implementation or sample personal library. The owner will provide disposable accounts/devices later. Do **not** copy a production Home Assistant or Music Assistant database into these volumes. + +CI runs `docker compose config` against `.env.example` without pulling images, starting services, or claiming a live test pass. + +## Safety and limits + +- Run only in a **disposable Linux VM or dedicated Linux test host** with Docker Engine and enough memory for both services. The official Home Assistant Container guide does not support Docker Desktop, and Music Assistant requires host/macvlan networking on the same flat LAN as physical players. A macOS/OrbStack dry run is not evidence of multicast discovery or audio delivery. +- This Compose file uses `network_mode: host`; Home Assistant follows its official container example with `privileged: true`. Thus ports 8123 and 8095 become reachable on the test host, not just inside the repository. Isolate the VM/host with a firewall and disposable admin identity, and inspect port conflicts before starting. Music Assistant receives no extra mount capabilities. +- Home Assistant Container has **no Supervisor, Apps or Ingress**. It can test Core Spotify/Music Assistant integration behavior, source/player observations and commands. It cannot close the Symphonia App packaging, Ingress-user attribution, App-to-Core network alias, pairing, or backup/restore gates. Use a separate Home Assistant OS/Supervisor-compatible environment for those; the official [local App testing guide](https://developers.home-assistant.io/docs/apps/testing/) describes a Supervisor devcontainer or real HAOS host. +- The image tags in `.env.example` are mutable bootstrap defaults, **not pinned evidence**. Before a dated live run, record image digest/version, HA integration version, MA server version, source provider version, account tier, hardware/player IDs, timezone, network topology and cleanup result in a private test log without secrets. + +## Prepare and start on the disposable Linux host + +Copy `.env.example` to `.env` in this directory and check or replace image references. Never put OAuth secrets or YouTube Music cookies in `.env`. Then, from this directory: + +```text +docker compose --env-file .env config +docker compose --env-file .env up -d +docker compose --env-file .env ps +``` + +Open Home Assistant at `http://TEST_HOST:8123` and Music Assistant at `http://TEST_HOST:8095` through the isolated test network. Complete empty, disposable onboarding. Add the HA Music Assistant integration only after the MA server is running; then add test sources/players and Spotify after dedicated credentials/devices are provided. Do not install an unofficial YouTube Music source or export browser cookies until its risk/owner decision is accepted. The official MA [HA integration guide](https://www.music-assistant.io/integration/installation/) explains account/caller prerequisites. + +Stop without deleting evidence volumes: + +```text +docker compose --env-file .env down +``` + +Only on a confirmed disposable host, after saving the sanitized probe results and verifying the Compose project/volumes, remove this lab's volumes if desired. `down` alone intentionally retains them; no cleanup script or broad recursive delete is supplied. + +## Evidence to capture later + +Use [the full release matrix](../playback-release-gates.md) and [integration-source proof matrix](../../providers/playback-integration-source-review.md#rg-007-proof-matrix-before-release). For each profile, record: exact action, account/source/player/output identities (hashed or synthetic in shared reports), allowed capabilities, accepted/rejected response, independently observed effect, elapsed time, duplicate-name and idle behavior, restart/recovery, and cleanup. Record `unknown` when the test lacks a real account/device; do not convert an empty lab into a pass. diff --git a/docs/development/playback-lab/compose.yaml b/docs/development/playback-lab/compose.yaml new file mode 100644 index 0000000..26fb06d --- /dev/null +++ b/docs/development/playback-lab/compose.yaml @@ -0,0 +1,28 @@ +name: symphonia-playback-lab + +# Disposable Core/MA evidence lab. Run only on a dedicated Linux VM/host. +# Home Assistant Container has no Supervisor, Apps or Ingress. +services: + home-assistant: + image: ${HA_IMAGE:?Set HA_IMAGE in the lab env file} + network_mode: host + privileged: true + restart: "no" + stop_grace_period: 60s + environment: + TZ: ${LAB_TZ:-Europe/Madrid} + volumes: + - ha_config:/config + + music-assistant: + image: ${MA_IMAGE:?Set MA_IMAGE in the lab env file} + network_mode: host + restart: "no" + environment: + LOG_LEVEL: info + volumes: + - ma_data:/data + +volumes: + ha_config: + ma_data: diff --git a/docs/development/playback-release-gates.md b/docs/development/playback-release-gates.md index b12c6bd..b9a7a94 100644 --- a/docs/development/playback-release-gates.md +++ b/docs/development/playback-release-gates.md @@ -28,11 +28,11 @@ The official [YouTube Data API](https://developers.google.com/youtube/v3/docs) m ## Broker protocol/security gate before production code -The authority boundary is decided, but the wire contract is not. The protocol RFC must select and test deployment discovery, connection direction, transport protection, pairing/secret rotation/revocation, broker and App identity, version negotiation, authenticated listener attribution, and upgrade/rollback. It must specify one exact entity per action, a closed operation/media-kind allowlist, bounded request/response/event sizes, timeout and rate budgets, monotonic observation ordering, command correlation, and no automatic command replay. The broker must never accept a generic HA domain/service, template, URL/path, area, wildcard, or arbitrary MA search string as an action target. The App must not receive a broad Core credential; the browser must not receive broker or provider credentials. A threat review must include other apps on the internal network, a compromised browser/session, forged entity IDs, stolen pairing material, replay after restart, and accidental private metadata in logs/backups. +The authority boundary is decided, but the wire contract is not. The [broker protocol RFC](../architecture/playback-broker-protocol-rfc.md) proposes a reverse authenticated channel and must select and test deployment discovery, connection direction, transport protection, pairing/secret rotation/revocation, broker and App identity, version negotiation, authenticated listener attribution, and upgrade/rollback. It must specify one exact entity per action, a closed operation/media-kind allowlist, bounded request/response/event sizes, timeout and rate budgets, monotonic observation ordering, command correlation, and no automatic command replay. The broker must never accept a generic HA domain/service, template, URL/path, area, wildcard, or arbitrary MA search string as an action target. The App must not receive a broad Core credential; the browser must not receive broker or provider credentials. A threat review must include other apps on the internal network, a compromised browser/session, forged entity IDs, stolen pairing material, replay after restart, and accidental private metadata in logs/backups. ## Required disposable-system proof -Use dedicated accounts and disposable Home Assistant/Music Assistant installations with pinned versions; never personal libraries or copied browser cookies in test artifacts. Record prerequisites, account tier, scope, integration version, player model, output ID, supported flags, request/response category, observation latency, and cleanup for each case. Ordinary CI uses deterministic synthetic fixtures; these live probes are opt-in and add evidence rather than replacing offline tests. +Use dedicated accounts and disposable Home Assistant/Music Assistant installations with pinned versions; never personal libraries or copied browser cookies in test artifacts. The [prepared Core/MA lab](playback-lab/README.md) can exercise provider/player behavior after accounts and devices are supplied, but its Container installation cannot test Supervisor, Apps or Ingress; that requires a separate HAOS/Supervisor-compatible environment. Record prerequisites, account tier, scope, integration version, player model, output ID, supported flags, request/response category, observation latency, and cleanup for each case. Ordinary CI uses deterministic synthetic fixtures; these live probes are opt-in and add evidence rather than replacing offline tests. 1. **Spotify:** no session, active, paused, restricted/private, duplicate device names, stale device cache, unavailable and reappearing device, multiple accounts; start and switch exact playlists on an exact output or record the failure. Compare HA-only and any separately authorized direct path without silently crossing credentials. 2. **Music Assistant:** idle and active MA queue, external input with no queue, two players, duplicate source names, linked versus default/unknown caller, long/partially visible library, exact Spotify and YouTube Music playlist references, race with external controller. diff --git a/docs/development/quality-audit.md b/docs/development/quality-audit.md index f0eba6e..078d079 100644 --- a/docs/development/quality-audit.md +++ b/docs/development/quality-audit.md @@ -42,6 +42,8 @@ Re-ran the documented offline commands against the local `develop` working tree Graphify's `--code-only --no-cluster` extraction produced **1,181 nodes and 3,060 edges**. `OperationRepository` remains the most connected symbol (57 edges); its one-hop dependents include runtime HTTP/resources, copy and import execution, the operation runner, and their tests. This reinforces the existing plan to define repository ports and preserve restart/reconciliation characterization before changing operation persistence. The broker design is kept out of domain/application imports and is not wired into this graph while its SDD is Draft. +After the SDD-validator increment, a fresh Graphify code-only extraction produced **1,197 nodes and 3,103 edges**; `OperationRepository` remained first at 57 edges. RepoWise safe dead-code analysis still reported **0 findings**. These are architecture/navigation snapshots, not a claim that the high-coupling persistence or copy hotspots are fixed. No production broker code was introduced while the listening SDD remains Draft. + The latest local `make verify` run passed **239 `unittest` cases (2 skipped)** plus specification/whitespace checks. Seven new validator cases exercise readiness blockers, implemented evidence, valid status transitions, malformed arrays, and missing SDD status. The isolated Lit spike also passed `npm ci --ignore-scripts`, TypeScript, presentation-boundary checks, three pure fallback tests, build, and relative-asset verification. Neither result proves Home Assistant integration, live playback, or visual parity. The new validator checks are structural guards, not substitutes for human review or live evidence. The previously absent coverage measurement is now reproducible with pinned `coverage.py` 7.16.1 (development-only, not a runtime dependency): diff --git a/docs/open-questions.md b/docs/open-questions.md index ddf975c..782ba87 100644 --- a/docs/open-questions.md +++ b/docs/open-questions.md @@ -82,7 +82,7 @@ This foundation deliberately does not guess at provider reconciliation or expose The owner wants a Home Assistant-integrated listening surface alongside playlist management; see the [listening SDD](../specs/listening-and-playback-control.md). **Authority boundary resolved (2026-09-28):** the owner chose a narrow companion playback broker, recorded in [ADR 0006](decisions/0006-narrow-home-assistant-playback-broker.md), instead of granting the App broad Supervisor Core API authority. **Scope direction (2026-09-28):** the owner rejected an active-session-only Spotify first release and requested the complete listening experience across the desired connected providers, including starting media, choosing where it plays, current playback, transport controls, and changing playlists. Internal implementation increments are allowed, but an active-session-only prototype is not a complete-release claim. The supported source/output matrix and broker transport, pairing, versioning, user attribution, and threat model remain open. This is independent of `OQ-004`'s provider OAuth callback decision: provider import credentials do not authorize HA player control. -**Unresolved feasibility and authority:** the [source review](providers/playback-integration-source-review.md) shows that stock HA Spotify drops `PLAY_MEDIA` while idle and selects devices by name, so it alone cannot substantiate the requested idle-start/exact-output experience. A separate, explicitly consented Spotify Connect adapter with its own read/control scopes is a possible complement, **not** an accepted fallback; it requires owner approval, a separate threat/policy review, account binding and live proof. Music Assistant players and its unofficial YouTube Music source need independent account/caller/queue and support-risk review. `ytube_music_player` is optional community evidence, not an automatic dependency. Native YouTube Music app sessions are not proven observable. Multiple active players are selected explicitly. There is no fallback to a generic Core API relay. The [full-scope proof plan](development/playback-release-gates.md) makes the release claims and required evidence explicit. +**Unresolved feasibility and authority:** the [source review](providers/playback-integration-source-review.md) shows that stock HA Spotify drops `PLAY_MEDIA` while idle and selects devices by name, so it alone cannot substantiate the requested idle-start/exact-output experience. A separate, explicitly consented Spotify Connect adapter with its own read/control scopes is a possible complement, **not** an accepted fallback; it requires owner approval, a separate threat/policy review, account binding and live proof. Music Assistant players and its unofficial YouTube Music source need independent account/caller/queue and support-risk review. `ytube_music_player` is optional community evidence, not an automatic dependency. Native YouTube Music app sessions are not proven observable. Multiple active players are selected explicitly. There is no fallback to a generic Core API relay. The [full-scope proof plan](development/playback-release-gates.md) makes the release claims and required evidence explicit. The [broker protocol RFC](architecture/playback-broker-protocol-rfc.md) proposes a reverse authenticated channel but is **not accepted** until topology, pairing, secret/backup and caller-identity proof closes its gates. ## Research/design gates (not owner preference alone) @@ -154,6 +154,8 @@ The tooling choice is accepted; the result must still establish a supported matr Use a disposable Home Assistant installation and dedicated accounts/devices after the broker protocol/security decision. Record the exact versions and account identities for HA Spotify, Music Assistant, and any proposed community player. Run the [dated source-review proof matrix](providers/playback-integration-source-review.md#rg-007-proof-matrix-before-release) and the [complete-release plan](development/playback-release-gates.md): verify Spotify idle/active/restricted and duplicate-name device behavior, WebSocket versus service browsing and service feature gates, exact-output/start feasibility, MA queue versus external source and per-user/default-user attribution through the **actual companion-broker route**, playlist reference mapping, private/unavailable/large playlist behavior, state-update/device-discovery lag, accepted-but-unobserved commands, external-controller races, HA restart, and credential recovery. If a separate Spotify Connect path is approved, also prove current endpoint access, fresh device IDs, separate read/control grants, account identity, idle-start/output confirmation and revocation without reusing the HA broker credential. Prove whether account identity can be machine-verified; if not, specify a visible user-confirmed but unverified binding and forbid automatic private-playlist mapping. Test hostile entity/media IDs, MA free-text/URL/path fallback rejection, broker pairing/version/replay/secret isolation, and ensure no broker credential reaches Ingress/browser state. Decide freshness/command-confirmation budgets from these observations. Keep community implementation findings separate from official platform evidence and do not infer control of native YouTube Music app sessions. +A [disposable Core/MA Compose lab](development/playback-lab/README.md) is now prepared and its Compose configuration validates locally. It has not been started or used with test accounts/devices; it provides no live pass. Because Home Assistant Container lacks Supervisor/Apps/Ingress, a separate HAOS/Supervisor-compatible environment is still required for broker topology and App authentication proof. + ## Future synchronization questions These do not block the copy MVP but block sync implementation: diff --git a/specs/catalog.json b/specs/catalog.json index e5b68e0..cfef973 100644 --- a/specs/catalog.json +++ b/specs/catalog.json @@ -172,6 +172,7 @@ ], "evidence": [ "docs/decisions/0006-narrow-home-assistant-playback-broker.md", + "docs/architecture/playback-broker-protocol-rfc.md", "docs/product/product-specification.md", "docs/product/home-assistant-ui-specification.md", "docs/domain/domain-model.md", @@ -180,6 +181,7 @@ "docs/providers/home-assistant-ecosystem-review.md", "docs/providers/playback-integration-source-review.md", "docs/development/playback-release-gates.md", + "docs/development/playback-lab/README.md", "docs/architecture/system-architecture.md", "docs/open-questions.md" ], @@ -189,7 +191,7 @@ "documentation": [] }, "blockers": [ - "OQ-011 broker transport/pairing/identity and complete playback profile matrix; narrow-broker authority accepted, active-session-only release rejected", + "OQ-011 broker RFC needs topology/security/secret/caller proof and complete playback profile matrix; narrow-broker authority accepted, active-session-only release rejected", "RG-007 Spotify idle-start/exact-output, MA caller/queue, account/source/player and exact playlist-reference feasibility; direct Spotify Connect complement needs separate approval", "Threat review of broker pairing, versioning, replay, caller identity, and action/entity allowlist", "Accepted observation freshness and command-reconciliation budgets", diff --git a/specs/listening-and-playback-control.md b/specs/listening-and-playback-control.md index f04ec50..0ac19a3 100644 --- a/specs/listening-and-playback-control.md +++ b/specs/listening-and-playback-control.md @@ -6,7 +6,7 @@ - Owners: Symphonia maintainers - Scope: browse playable content, choose an explicit playback source and output, observe now-playing state, and control a configured external player from the Home Assistant-integrated Symphonia UI. - Related requirements: `SYM-PLAY-001`–`SYM-PLAY-009`, `SYM-PROV-021`–`SYM-PROV-022`, `SYM-UI-016`, `SYM-HA-011`, `SYM-ARCH-016`, `SYM-ACC-005`, `SYM-UI-001`–`SYM-UI-015`, `SYM-SEC-004`, `SYM-SEC-008` -- Related decisions/research: [product specification](../docs/product/product-specification.md), [domain model](../docs/domain/domain-model.md), [provider specification](../docs/providers/provider-specification.md), [system architecture](../docs/architecture/system-architecture.md), [ADR 0006](../docs/decisions/0006-narrow-home-assistant-playback-broker.md), [official playback research](../docs/providers/provider-research.md#2026-09-28-playback-and-home-assistant-api-update), [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md), [integration source review](../docs/providers/playback-integration-source-review.md), [full-scope proof plan](../docs/development/playback-release-gates.md), [UI foundation](home-assistant-native-ui.md), [App runtime](home-assistant-app-runtime-and-ingress.md), `OQ-011`, `RG-007` +- Related decisions/research: [product specification](../docs/product/product-specification.md), [domain model](../docs/domain/domain-model.md), [provider specification](../docs/providers/provider-specification.md), [system architecture](../docs/architecture/system-architecture.md), [ADR 0006](../docs/decisions/0006-narrow-home-assistant-playback-broker.md), [broker protocol RFC](../docs/architecture/playback-broker-protocol-rfc.md), [official playback research](../docs/providers/provider-research.md#2026-09-28-playback-and-home-assistant-api-update), [ecosystem review](../docs/providers/home-assistant-ecosystem-review.md), [integration source review](../docs/providers/playback-integration-source-review.md), [full-scope proof plan](../docs/development/playback-release-gates.md), [UI foundation](home-assistant-native-ui.md), [App runtime](home-assistant-app-runtime-and-ingress.md), `OQ-011`, `RG-007` - Required review gates: product UX, Home Assistant platform/API, provider feasibility, architecture, accessibility, testing, documentation, security/privacy - Open decisions blocking readiness: `OQ-011` complete supported playback-profile matrix and broker transport/pairing/identity contract (the narrow-broker authority choice is accepted); `RG-007` Spotify idle-start/exact-device feasibility, account/source/player binding, MA caller identity and playlist-reference evidence; whether an explicitly consented direct Spotify Connect complement is authorized; broker threat/version/upgrade review; live-state freshness and command-reconciliation thresholds; Music Assistant/community YouTube Music support policy @@ -299,7 +299,7 @@ Minimum **112 distinct cases** for the requested full-profile target: | Integration/security/migration | 22 | broker pairing/version/allowlist/secret canaries, Ingress isolation, separate direct grant if approved, account identity, reconnect/no replay, rollback/backup, opt-in live smoke | | **Total** | **112** | No double counting | -The ordinary suite uses synthetic HA entity states, feature flags, service responses, deterministic clocks, command IDs, source browsers, and no network/account. Each supported integration gets its own versioned fixture matrix; Spotify and Music Assistant are not treated as the same protocol mapping. Security tests require exact entity/action allowlisting, no MA name/URL/path fallback, caller-context checks, and secret canaries. Opt-in live smoke uses disposable/dedicated accounts and devices; it tests Spotify idle start/browse and duplicate-name outputs, MA queue/external source and actual user attribution, the supported source/output matrix, accepted-but-unobserved commands, independent external controller races, and cleanup. Human evidence covers Home Assistant-native visual parity, phone/wide, light/dark, keyboard/screen reader, long metadata, multiple players, and all blocked/uncertain states. Live tests supplement, not replace, deterministic contracts. +The ordinary suite uses synthetic HA entity states, feature flags, service responses, deterministic clocks, command IDs, source browsers, and no network/account. Each supported integration gets its own versioned fixture matrix; Spotify and Music Assistant are not treated as the same protocol mapping. Security tests require exact entity/action allowlisting, no MA name/URL/path fallback, caller-context checks, and secret canaries. Opt-in live smoke uses disposable/dedicated accounts and devices; the [prepared Core/MA lab](../docs/development/playback-lab/README.md) is only one part of that evidence, not a Supervisor/Ingress substitute. Live tests cover Spotify idle start/browse and duplicate-name outputs, MA queue/external source and actual user attribution, the supported source/output matrix, accepted-but-unobserved commands, independent external controller races, and cleanup. Human evidence covers Home Assistant-native visual parity, phone/wide, light/dark, keyboard/screen reader, long metadata, multiple players, and all blocked/uncertain states. Live tests supplement, not replace, deterministic contracts. ## 15. Documentation and discoverability