V4.36: Onboarding a New Data Source in an Hour, Not a Sprint

[deeprnd]

The institutional valuation gap isn't a math problem — it's an onboarding problem. Every new financial data source (ERP extract, SEC EDGAR filing, XBRL feed) has demanded core-runtime code changes: code review, deployment, risk assessment. That's not how you build a system investors trust. They trust what they can verify, and they can't verify a system where adding a data source means touching the core. V4.36 eliminates that bottleneck entirely. Onboarding a new source becomes writing a single plugin file. The core runtime stays untouched, auditable, and replay-stable.

Adding a new financial data source is a single plugin file, not a code change.

Product Intent

Financial institutions need valuation data from dozens of sources — Excel spreadsheets, SEC EDGAR filings, XBRL data, ERP extracts, third-party data vendors. Until now, every new source required modifying the core runtime. Hardcoded adapters created a bottleneck: data onboarding took weeks, code reviews, and deployment cycles.

V4.36 creates an extensible plugin infrastructure where new data sources are added by writing a plugin module — not by modifying the core runtime. The plugin framework owns input plugins (data source adapters), output plugins (canonical fact producers, validation report producers, audit trail producers), and a deterministic discovery/registration system. V4.34 (Valuation Data Layer) consumes this infrastructure transparently.

The constraint that makes this a trust signal: plugins register and load in deterministic, replay-stable order. The same plugin version and configuration always produce the same output. Audit trail captures version, configuration hash, and source attribution for every canonical fact. No surprises, no version drift, no "it worked on my machine."

Users And Jobs

• Data engineer: onboard a new ERP or EDGAR data source by writing a plugin, without touching the core runtime.
• Quant researcher: load financial data from multiple sources through a uniform plugin interface, without learning each source's internals.
• Risk reviewer: audit which plugin version, configuration, and source produced each canonical fact, with full traceability.
• Engine builder: compose plugin-based data flows on top of tknorm and tkdedu without re-implementing source adapters.

Success Metrics

• New data sources can be added by writing a plugin module without modifying the core plugin framework.
• Input plugins for ERP (Excel/FreeXL) and EDGAR (Edgartools) register and load without code changes.
• Output plugins for canonical facts, validation reports, and audit trails are pluggable.
• Plugin discovery and registration is deterministic and replay-stable.
• All plugin activity is audit-traced with source attribution, version, and configuration hash.
• Replay-stable plugin execution: same plugin version and configuration produce same output.
• The plugin framework defines a clear interface with lifecycle hooks (init, ingest, validate, output, shutdown).

Demo Moment

just plugin-demo loads an AAPL FY2023 Excel fixture through the ERP plugin, retrieves an EDGAR XBRL fixture through the EDGAR plugin, runs the canonical fact output plugin, and outputs validated canonical facts ready for tkval.

A new data source flows through the system end-to-end with full audit trail, source attribution, and plugin version hashes — all from plugin configuration, not core code changes.

Scope

In Scope

• Plugin interface and lifecycle definition — init, ingest, validate, output, shutdown hooks.
• Plugin registry and discovery system — deterministic loading order, version tracking, configuration hash.
• Input plugins: ERP (Excel/FreeXL), EDGAR (Edgartools) integration points with plugin wrappers.
• Output plugins: canonical fact producer, validation report producer, audit trail producer.
• Plugin configuration schema — YAML/JSON config for plugin parameters, source paths, and options.
• Plugin audit trail — source attribution, version, configuration hash, lifecycle events.
• Replay-stable plugin execution — same plugin version and config produce same output.
• Plugin test harness — fixtures, replay tests, and validation for plugin modules.

Out Of Scope

• DCF computation (V4.32).
• Actual EDGAR API rate limit handling — handled by V4.34 consuming the plugin.
• Market data collection plugins — handled by V4.34.S4 as a standalone plugin story.
• Plugin signing or sandboxing — not a runtime isolation boundary.
• Plugin hot-reload or live update — plugins are versioned and deterministic.
• Plugin marketplace or distribution — deployment is internal.

Conditional Boundary Checklist

Boundary	Status	Notes / linked story
Financial capability and policy	N/A	Plugin framework is infrastructure; policy is enforced by V4.34 stories.
Audit records and evidence	Applies	V4.36.S5 owns plugin audit trail with source attribution and version hash.
Replay and divergence behavior	Applies	V4.36.S5 requires deterministic plugin output under replay.
Runtime topology, tile ownership, or links	Applies	V4.36.S1 adds plugin framework; V4.34 consumes it.
Model gateway governance (tkmodl)	N/A	Plugin framework is data source infrastructure; no model calls.
Tool broker or adapter dispatch (tktool / tkadpt)	N/A	Plugin interface is distinct from MCP-style tool calls.
Approved execution (tkexec)	N/A	Analysis-only; no execution.
CaseOps API/UI (tkapi)	N/A	Plugin configuration surfaces through V4.34 stories; no direct API.
Storage role: Memory, Analytics, Ledger	N/A	Plugin configs and audit records are captured data; not storage role changes.
Metrics, diagnostics, and operations	Applies	V4.36.S5 ties plugin lifecycle to tkmetr counters.
Security and fail-closed behavior	Applies	Plugin config validation, input bounds checking, no unvalidated source paths.

Story Breakdown

How Many Stories

Mid-range epic — 6 stories covering plugin framework, input plugins, output plugins, discovery/registration, testing, and closure.

Decomposition Rules

1. Independent delivery: each story delivers a testable plugin slice.
2. One capability per story: core framework, input plugins, output plugins, discovery, testing.
3. Split by plugin lifecycle: framework, input, output, registration, tests.
4. Closure via T10 child tasks in the final story.
5. Domain boundaries: framework, input, output, registration are separate stories.

Bad Splits (Avoid)

• "Plugin interface" and "plugin lifecycle" as two stories — they are the same framework story.
• "ERP plugin" and "EDGAR plugin" as two stories — they are both input plugins, one story with two implementations.
• "Plugin discovery" and "plugin registration" as two stories — discovery is part of registration, same story.

Good Split Example

V4.36.S1: Plugin framework — interface, lifecycle hooks (init, ingest, validate, output, shutdown), registry, version tracking.
V4.36.S2: Input plugins — ERP (Excel/FreeXL) and EDGAR (Edgartools) plugin wrappers with source adapters.
V4.36.S3: Output plugins — canonical fact producer, validation report producer, audit trail producer.
V4.36.S4: Plugin discovery and registration — deterministic loading, configuration schema, version and config hash.
V4.36.S5: Plugin test harness — fixtures, replay tests, audit trail validation, metrics integration.
V4.36.S6: Evidence, demo, and release closure — T10 expansion covering demo, quality gates, documentation.

Story Line Format

• V4.36.S1: Plugin framework — interface, lifecycle hooks, registry, version tracking — the extensible foundation that makes every new data source possible — #[github-story-issue]
• V4.36.S2: Input plugins — ERP (Excel/FreeXL) and EDGAR (Edgartools) plugin wrappers — turn real-world data formats into plugin entries — #[github-story-issue]
• V4.36.S3: Output plugins — canonical fact producer, validation report, audit trail producer — produce standardized downstream facts from any plugin source — #[github-story-issue]
• V4.36.S4: Plugin discovery and registration — deterministic loading, config schema, version/hash — register new sources without code changes — #[github-story-issue]
• V4.36.S5: Plugin test harness — fixtures, replay, audit trail validation, metrics — prove plugins are deterministic and auditable — #[github-story-issue]
• V4.36.S6: Evidence, demo, and release closure — T10 expansion covering demo, quality gates, docs — close the epic with proof — #[github-story-issue]

Epic Acceptance

• All required story issues are done or explicitly deferred with rationale.
• The demo moment succeeds using deterministic local inputs.
• Relevant policy, audit, replay, adapter, API/UI, metrics, diagnostics, and documentation artifacts are linked from this epic.
• Non-goals and deferred work are visible in the roadmap.
• No live model, broker, payment, trading, crypto, approved execution ledger, or execution side effects occur unless this epic explicitly enables an approved sandbox/live execution path.

Release / Evidence Gate

• What can the user or operator do now that they could not before this epic?
  Add new financial data source plugins without code changes to the core runtime.

• What changed from the previous roadmap increment?
  Data sources were hardcoded into the runtime. Now they are pluggable — onboarding a new source is a file, not a code change.

• What is this epic's wow-effect: the visible moment in this epic's demo or workflow that makes Tickoni feel unlike a generic agent harness?
  A new ERP or EDGAR data source added as a single plugin file — no core runtime changes. Onboarding goes from weeks to hours.

• How does this epic progress Tickoni's unique value proposition from doc/strategy/positioning.md: speed, isolation/control, spend governance, policy-gated action, and forensic replay?
  Brings institutional-grade data source extensibility into the governed runtime with deterministic replay and audit trail.

• Which demo command or CaseOps flow closes the epic?
  just plugin-demo with ERP and EDGAR plugin fixtures.

• Which account, beneficiary, IBAN, wallet, rail, currency, market, venue, asset class, instrument, notional, amount, exposure, and frequency checks are enforced, if any?
  N/A — plugin infrastructure; data validation is enforced by V4.34 stories.

• What happens when the requested action exceeds policy, scope, or evidence?
  N/A — plugin config validation rejects malformed source paths or options.

• Is execution paper-only, draft-only, sandbox, live, or disabled?
  Analysis-only; no execution.

• Which fixtures, samples, audit records, replay capsules, screenshots, or API examples prove the behavior?
  AAPL FY2023 Excel fixture, EDGAR XBRL fixture, plugin audit trail JSONL, replay byte-identical comparison.

• Can replay run without external side effects when replay applies?
  Yes — deterministic plugin execution; no external calls on replay.

• What intentional blocked-flow or divergence example proves fail-closed behavior?
  Plugin config with missing source path or invalid plugin version triggers validation error; no plugin loads.

Dependencies And Decisions

• An ADR is needed to evaluate the best approach for how plugins work (language bindings, plugin loading model, sandboxing, interface definition) before any story work can commence.
• Plugin interface must be stable before V4.34 stories begin implementation.
• Decision: plugin configuration format — YAML preferred for readability; JSON fallback.
• Decision: plugin loading order — deterministic by registration order with explicit dependency graph.
• Depends on V4.32.S1 (tkval tile) for canonical fact struct shape that output plugins produce.
• V4.34 (Valuation Data Layer) depends on this epic for extensible data source support.