diff --git a/README.md b/README.md index 4ee61e9..aa5f93d 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,26 @@ Manual account aggregation is implemented. Bank connections, brokerage execution | --- | --- | | ![Fictional XDEMO report with result explanation, calculation checks, chronological segments, and an equity chart.](docs/evidence/consumer-report.jpg) | ![Narrow-screen backtest dialog with stock selection, starting cash, collapsed advanced settings, and input confirmation.](docs/evidence/consumer-narrow.jpg) | +## Stock data pipelines + +Three ingestion paths converge on the same private price-snapshot model: + +| Input path | What crosses the boundary | Implementation | +| --- | --- | --- | +| **Alpha Vantage through the server** | A bounded daily-close response, fetched with the IBM demo key or a configured server secret. Provider notices stop the import instead of becoming prices. | [Provider route](app/api/provider/route.ts) · [Response normalizer](lib/finance/provider.ts) | +| **Alpha Vantage from the browser** | A one-time key goes directly to the provider. Only normalized prices and provider metadata return to MarketLab, where the server validates them again. | [Browser client](lib/finance/browser-provider.ts) · [Reservation and save route](app/api/provider/browser/route.ts) | +| **Historical CSV** | One symbol's dated USD prices, an explicit source, adjustment basis, and historical/synthetic declaration. Invalid dates, duplicate observations, and malformed quoting fail validation. | [CSV validator](lib/finance/market-data.ts) · [Dataset route](app/api/datasets/route.ts) | + +![MarketLab stock data paths: server and browser Alpha Vantage plus CSV converge on validation, immutable owner-scoped snapshots, exact backtests, and saved reports; background native checks, independent replay, and public sharing are separate owner-triggered branches.](docs/media/stock-data-pipeline.png) + +Validated observations become **immutable, owner-scoped D1 snapshots**. Their SHA-256 identity includes precision-preserving values and declared metadata; provider snapshots additionally bind origin, refresh date, and timezone. Saving identical inputs reuses the existing snapshot. `csv`, `alphavantage`, and `alphavantage-browser` remain distinct in storage and exports. [Snapshot persistence](lib/datasets.ts). + +Research binds those snapshots to complete declared split/dividend records, computes exact accounting and next-close backtests, then checks the preview fingerprint before saving frozen inputs and outputs. Separately, imported portfolio ledgers bind to saved prices for exact cash/share accounting and cash-flow-matched comparisons. [Saved research](app/api/research/route.ts) · [Portfolio accounting](lib/finance/historical-portfolio.ts). + +**Verification and sharing are separate branches.** A saved backtest can be copied into bounded, signed Node/MongoDB jobs for fenced recomputation and C++ risk checks, or sent to independent Python replay for a receipt bound to the exact report. Reviewed sharing publishes only an allowlisted, expiring, revocable summary. It does not require either verification action; the redacted summary cannot replay withheld raw prices. [Background jobs](app/api/research/jobs/route.ts) · [Replay receipts](lib/research-replay.ts) · [Redaction](lib/finance/shared-research.ts). + +These are user-triggered historical-data paths, not a streaming feed. Provider paths share one atomic per-account cooldown and do not retry provider requests automatically. A validated browser import is not server-authenticated market data; source coverage and adjustment assumptions stay visible. [Cooldown](lib/provider-runs.ts) · [Editable diagram](docs/media/stock-data-pipeline.svg). + ## Why the results are reproducible ### Exact money, explicit execution diff --git a/docs/media/stock-data-pipeline.png b/docs/media/stock-data-pipeline.png new file mode 100644 index 0000000..634a400 Binary files /dev/null and b/docs/media/stock-data-pipeline.png differ diff --git a/docs/media/stock-data-pipeline.svg b/docs/media/stock-data-pipeline.svg new file mode 100644 index 0000000..b01cf85 --- /dev/null +++ b/docs/media/stock-data-pipeline.svg @@ -0,0 +1,80 @@ + + MarketLab stock data and research pipeline + Server Alpha Vantage requests, direct browser Alpha Vantage requests, and historical CSV imports converge on validation and immutable owner-scoped D1 price snapshots. Explicit corporate-action records feed exact accounting and backtests; the same saved price snapshots also support a separate portfolio-ledger accounting path. Saved private research reports support separate owner-triggered background recomputation with MongoDB leases and C++ checks, independent Python replay with report-bound receipts, or reviewed redacted public sharing. + + + + + + From stock prices to an inspectable result + Three input paths. Frozen private records. Separate actions for verification and sharing. + + SERVER REQUEST + Alpha Vantage daily closes + IBM demo or configured server key + Bounded fetch; no automatic retry + + BROWSER REQUEST + Direct provider connection + One-time key stays with the browser + Only normalized prices reach the app + + FILE IMPORT + Historical price CSV + Declared source and adjustment basis + Historical or explicitly synthetic + + + + + Validate before saving + Symbol, USD, dates, positive prices, duplicates, size + Adjustment basis retained; browser data revalidated + Provider paths share one atomic owner cooldown + + Freeze price snapshots in D1 + Owner + SHA-256 identity; identical saves reuse it + Source, basis, origin, coverage and provider metadata + CSV and browser/provider provenance stay distinct + + + + + Bind inputs; calculate explicitly + Complete declared split/dividend records are required + Exact cash, fees, shares and split fractions + Backtests: next-close signals, costs, matched dates + Missing or incompatible inputs stop the calculation + + Save the private research report + Worker computes the canonical TypeScript result + D1 retains frozen settings, inputs and outputs + Preview fingerprint is checked again before save + Private JSON / CSV exports preserve the full report + + + + + + + SEPARATE OWNER-TRIGGERED ACTIONS + + Background recomputation + Signed snapshot copy → Node service + Bounded MongoDB jobs; fenced leases + Workers; C++ in native-check mode + Result remains in the research service + + Independent Python replay + Full saved report → bounded verifier + Accounting, trades, signals and risk + Receipt bound to exact report bytes + Matching receipt stored against D1 run + + Reviewed public summary + Explicit consent; allowlisted redaction + Expiring, replaceable, revocable link + Raw prices and private labels withheld + Sharing does not require either verifier + Daily historical data, not a live feed. Validation does not authenticate browser-supplied prices. +