diff --git a/README.md b/README.md index 289ecfd..4ee61e9 100644 --- a/README.md +++ b/README.md @@ -1,271 +1,150 @@ # MarketLab -A portfolio planning and analysis app with saved goals, a combined view of accounts, transparent fee projections and independently checkable backtests. +Portfolio planning and reproducible research, with the inputs and accounting behind every result kept inspectable. -**Engineering direction:** follow the adopted [engineering roadmap](docs/engineering-roadmap.md), based on Michael's supplied standards. It includes the directory map, ordered milestones and observed status. The first reliability work now has [reproducible clock-race and process-crash evidence](docs/reliability-evidence.md). +**TypeScript · React · Workers / D1 · Node.js / MongoDB · C++ · Python** -**Reliability extension:** bounded background calculations and per-owner limits -are active on the hosted staging worker, with friendly retry messages and -reproducible fixed-arrival overload/crash evidence. The approved database migration, -signed HTTPS/native/Python/crash checks and ordinary-role restart passed; temporary -administrator references were removed. Real-user pilot acceptance remains pending. -See [mechanism, measurements and limits](docs/overload-handling.md). +[Open the app](https://marketlab-portfolio.michaelbaffour240306.chatgpt.site) · [Try a fictional report](https://marketlab-portfolio.michaelbaffour240306.chatgpt.site/example) · [Engineering notes](docs/engineering-roadmap.md) · [Test guide](tests/README.md) -**Current release:** personal saved workspaces with email or ChatGPT sign-in, a zero-setup practice portfolio, and public read-only report sharing. Each account opens on My plan and can return to its own saved accounts, goals and reports. It includes on-demand Alpha Vantage daily-price fetching, historical CSV imports, split/dividend research, user-imported historical portfolios with cash-flow-matched benchmarks, saved strategy experiments with benchmark/risk comparisons, and a separate synthetic portfolio demo. The separate demo’s six price series are explicitly synthetic, covering 127 weekday observations from April 1 to September 24, 2026. They are not real market history, exchange calendars, forecasts, or a backtest. +![MarketLab's fictional $10,000 example compares the same rule and prices with and without trading costs.](docs/evidence/consumer-welcome.jpg) -See [personal workspaces and acceptance evidence](docs/personal-workspaces.md) for account separation, first-use flow, and the remaining live pilot gates. +The welcome screen gives you a result before asking for an account: change the costs on the same fictional investment and see what changes. From there, create a private workspace, review an imported portfolio, or run a saved research experiment. -## What works +The screenshots here are captures of local browser fixtures using fictional data. They show implemented screens; they are not real customer portfolios or evidence of production adoption. -- **My plan:** saved manual accounts and debts, goal progress under editable assumptions, - cost-of-fees projections, allocation drift alerts, contribution and stress scenarios, - plus a transparent planning checklist. Historical backtests can supply a goal stress - scenario without being treated as future forecasts. -- **Strategy library:** tested trend templates and explicitly published community reports - with report-bound independent replay receipts. Reuse rule settings on your own data. -- **Optional peer comparison:** fixed-quarter cohorts, at least 20 opt-in accounts, - rounded aggregate results, frozen snapshots and withdrawal controls. -- **Trust and support:** in-app support requests and owner replies, privacy/deletion - disclosures, educational-use framing and an on-demand service status check. - No bank-sync, bank-grade certification, completed accessibility audit or uptime - guarantee is claimed. [Release details and acceptance evidence](docs/ongoing-use-release.md). +## A short tour -- Invite-only report discussions with ChatGPT or verified email sign-in: - single-use invitations bind to the chosen account, owners revoke access and - moderate comments, and database triggers retain an append-only audit trail. - The ChatGPT two-person comment/revocation check passed as user-reported evidence. - Email sign-in through WorkOS Staging is confirmed by the user and live callback - log; the user also confirmed the email-account comment/revocation checklist. - [Discussion model](docs/research-collaboration.md) · [Email sign-in and evidence](docs/discussion-email-sign-in.md). - -- Selectable buy, sell, split and dividend markers on Research lab equity curves, - with exact cash/fee explanations and event dates retained across missing closes. - [Timeline behavior and evidence](docs/research-sharing-and-events.md). - -- Reviewed, redacted read-only experiment summaries with expiring links, - replacement and revocation. Public access is activated while private workspace - routes remain private to each signed-in account. A [fictional public example](https://marketlab-portfolio.michaelbaffour240306.chatgpt.site/example) - contains only built-in teaching data. Live browser verification remains outstanding. - [Sharing boundary and activation](docs/research-sharing-and-events.md). - -- Confidence certificates and plain-language takeaways on analytical reports, with source/coverage details, actual per-result consistency checks, assumptions and explicit independent-replay status. JSON/CSV reports carry the certificate. [Scope and evidence](docs/confidence-certificates.md). - -- A one-click **Quick experiment**: compare the same fictional rule and prices with three cost settings, inspect exact values and download a reproducible report. No upload or market-data key is required. [Demo inputs and acceptance evidence](docs/quick-demo.md). - -- Side-by-side saved-experiment comparisons in Research lab, with exact-cent differences and explicit checks for matching frozen prices, event records, starting cash and observation dates. - -- Cash-flow-matched portfolio benchmarks with frozen price/event inputs, exact fractional holdings, observed return comparisons and JSON/CSV exports. -- A supporting Python verifier that independently replays Research lab reports and checks exact accounting, signals, trades, risk metrics and input fingerprints. - -- A Node/MongoDB research job service with signed ownership, concurrency-safe quotas, crash recovery, bounded worker threads and C++ risk parity checks. Railway free-trial staging passed remote crash recovery; the owner's September 27 saved-experiment browser flow passed all nine native comparisons. [Evidence and limits](docs/research-staging.md). - -- A reproducible TypeScript service load runner with raw timings and correctness checks. Three local 1,000-job runs achieved **39.96 verified jobs/s** and **319.77 ms combined p99**, using two workers and eight concurrent clients with 500 observations per instrument. [Workload, environment and limits](docs/service-performance.md). - -- A Research lab with next-close SMA backtests, costs, buy-and-hold and selected benchmark comparisons, observed-interval risk, separate chronological evaluation and immutable JSON/CSV reports. - -- An Alpha Vantage daily-price connector with IBM public demo access, request-only API keys, bounded fetching, persisted request history, and source-aware snapshots. -- A historical-data workspace with CSV validation, preview, persistent immutable snapshots, price charts, observed declines, coverage details, and CSV export. -- Saved, source-declared split/dividend event records with exact fractional-share arithmetic, cash-inclusive performance comparison and an audit export. -- An empty historical portfolio workspace with full-ledger CSV imports, exact-date price bindings, frozen data/event inputs, dividend receivable/payment accounting, reconciliation and export. -- A sample portfolio dashboard with value history, contribution overlays, time-weighted returns, allocation, and holdings. -- Search and price charts for six sample assets. -- Server-persisted buy, sell, deposit, withdrawal, and dividend entries, including trade fees and fractional shares. -- Historical cash and position validation: an entry cannot create negative cash or a short position, including on earlier dates. -- An append-only manual ledger with request idempotency and an atomic revision check for concurrent writes. -- A dataset ingestion endpoint with validation, immutable dataset identity, unique symbol/date keys, resumable batches, and persisted run outcomes. -- A downloadable CSV report containing dataset provenance, valuations, costs, and the full ledger. -- Per-user ledger and run isolation using the hosting platform's authenticated user ID. - -The first portfolio is intentionally a paper account. The built-in ledger is immutable; up to 500 additional manual events are supported. There is no brokerage connection, execution, or money transfer. - -## Technology - -| Area | Implemented | +| Start with | What you can do | | --- | --- | -| Interface | React 19, TypeScript, custom SVG charts, Shadcn primitives | -| Application server | TypeScript route handlers on a Cloudflare-compatible Worker | -| Persistence | Managed D1 / SQLite, schema migrations generated with Drizzle | -| Accounting | Pure TypeScript; integer cents and millionths of shares using BigInt | -| Validation | Zod request validation plus historical ledger replay | -| Verification | Node finance/integration tests, independent Python replay, TypeScript checking, ESLint and browser workflow checks | - -Node runs local tooling and tests. A separate signed Node research service persists jobs and results in MongoDB, computes through worker threads, and verifies risk metrics with a compiled C++ Node-API module. It runs on Railway free-trial staging, with remote crash-recovery evidence and an owner-confirmed saved-experiment verification flow. The live website continues to use Workers + D1. See [staging evidence](docs/research-staging.md) and `docs/research-service.md` for the deployment boundary and setup. - -## Run and check - -Use Node 24 for the complete engineering gate and the checked-in pnpm lockfile. - -```sh -pnpm install --frozen-lockfile -pnpm test -pnpm typecheck -pnpm lint -pnpm dev -``` - -`pnpm test` runs the unit layer. With a C++17 compiler, Python 3.11+ and `mongod` -installed, `pnpm test:service:local --ci` runs the complete verification gate -against a disposable database. Set `MONGOD_BIN` when needed. The gate includes -real HTTP/worker/crash tests and independent Python/C++ comparisons. See the -[test layers and focused commands](tests/README.md). The -[GitHub workflow](https://github.com/michaelbawuah/MarketLab/actions/workflows/verify.yml) -runs this same gate on pushes and pull requests; inspect its run result for remote -verification evidence. - -For a standalone local environment, the starter's execution-profile helper configures the development server. This repository was created in the managed Sites environment; managed preview uses `sites-preview start` instead of launching an additional server. Generate database migrations with `pnpm db:generate`, build to generate the local Worker config, and apply pending SQL migrations to the local D1 database using Wrangler. See `docs/runtime.md` for the runtime setup and migration command. - -## Accounting conventions - -Amounts crossing the API and stored in the ledger are integer strings. No financial amount is parsed with JavaScript's floating-point `parseFloat`. A share has six decimal places; a currency amount has two. Share-price products are rounded half up to the nearest cent. Final display values and dimensionless performance ratios use Number. - -Cost basis is weighted-average acquisition cost, including buy fees. Partial sales allocate cost proportionally; a final sale takes the exact remaining basis. Realized gain deducts sell fees. This is a research convention, not tax-lot accounting. - -Valuation uses only the latest quote dated on or before the observation. Missing initial prices stop valuation. Daily close prices mark existing holdings first; that date's ledger events then execute in stable order. Return chains each market movement and each internal ledger event. Deposits and withdrawals change value without changing return. Fees, execution prices, and dividends affect return. Same-day event order and the close-price timing assumption matter; this is not intraday-accurate performance measurement. +| **My plan** | Combine manually entered accounts and debts, track a goal, inspect fee projections, and compare contribution or stress scenarios. | +| **Your data** | Validate historical CSVs or fetch Alpha Vantage daily closes, then save source-labeled, immutable price snapshots. A Schwab transaction adapter retains a conversion and reconciliation receipt. | +| **Research lab** | Compare a next-close trend rule with buying and holding, model fees and slippage, inspect earlier/later periods, and save the full inputs and results. | +| **Reports** | Read plain-language takeaways and calculation checks, download JSON/CSV, request independent replay, and explicitly publish a redacted summary. | +| **Discussions** | Invite an account to comment on a shared report, then revoke access without exposing the private workspace. | -The demo's weekday calendar intentionally includes weekdays that may be exchange holidays. No holiday-calendar accuracy is claimed. The synthetic portfolio ledger does not apply corporate actions. Separate historical research supports the explicit model below. Foreign exchange, shorting, leverage, and multi-account support are not implemented. Historical portfolios support the separate cash-flow benchmark model documented below. +Manual account aggregation is implemented. Bank connections, brokerage execution, and automatic account refresh are not. -## Pipeline behavior - -`POST /api/pipeline` validates all 762 records, then stores batches of 50 with unique `(dataset, symbol, date)` keys. Running it again leaves the same number of rows. Completed and failed runs retain actual counts and elapsed times. Interrupted batches can be resumed by replay; there is no automatic scheduler or retry queue yet. Dataset version IDs must change if fixture values change. +| A report you can inspect | A backtest with sensible defaults | +| --- | --- | +| ![Fictional XDEMO report with result explanation, calculation checks, chronological segments, and an equity chart.](docs/evidence/consumer-report.jpg) | ![Narrow-screen backtest dialog with stock selection, starting cash, collapsed advanced settings, and input confirmation.](docs/evidence/consumer-narrow.jpg) | -Before ingestion, the UI uses the explicitly labeled fixture preview. After ingestion, calculations read the stored dataset. Partial persisted coverage fails closed; the Data pipeline view remains available to repair it with another ingestion. Prices are shared immutable fixtures; manual transactions and run history are scoped to the signed-in user. +## Why the results are reproducible -## Historical CSV research +### Exact money, explicit execution -Open **Historical data** to upload or paste a daily USD price series. The importer accepts `date` (or `timestamp`) with `close`, or an explicitly selected `adjusted_close` / `Adj Close` column. An optional `symbol` column must match the selected symbol; an optional `currency` column must be USD. Choose the source name, historical versus synthetic classification, and adjustment basis explicitly. Unknown adjustments are supported and labeled. +The [accounting engine](lib/finance/core.ts) uses `BigInt` cents and millionths of shares. Split fractions remain exact in the research engine. Monetary rounding is explicit; floating-point numbers are reserved for display and dimensionless statistics. Historical validation rejects overdrafts and short positions. -Imports are limited to 256 KiB, 2–2,500 observations, one symbol, and 30 saved datasets per user. Prices use integer millionths of a dollar, preserving up to six decimal places. Invalid dates, future observations, duplicate dates, mixed symbols, non-USD currency and invalid or over-precision prices fail validation before saving. Missing exchange sessions are not invented or declared complete. +Research signals use information available at an observation and execute at the **next supplied close**. Costs, dividends, splits, starting cash, missing prices, and evaluation dates have defined rules. Deposits and withdrawals are excluded from the portfolio's time-weighted return. -Each canonical dataset includes the metadata and sorted exact observations in its SHA-256 identity. Equivalent replays preserve the original snapshot; changed prices or metadata produce another snapshot. Each bounded snapshot is stored as normalized metadata plus a canonical JSON observation vector in one owner-scoped D1 row. A single SQL statement enforces the workspace limit and atomically inserts the whole snapshot, so partial imports are never visible. The raw uploaded file and unused columns are not retained. Original canonical observations are available in the export. +[Accounting conventions](docs/reference-guide.md#accounting-conventions) · [Runnable money examples](docs/money-arithmetic.md) · [Research methodology](docs/reference-guide.md#research-lab) -Source, symbol and adjustment claims come from the uploader and are not independently authenticated. Charts show only supplied observations, with lines connecting them across gaps. Series change and maximum observed peak-to-trough decline use the declared price basis. They are not portfolio returns, intraday risk measures, or strategy results. Imports never replace synthetic portfolio marks or use the demo's fictional trade ledger. The Alpha Vantage connector below can fetch directly from the provider; CSV uploads keep their user-supplied classification. +### Frozen inputs, separate evidence -## Daily-price provider +Saved experiments contain the settings, method version, complete price/event inputs, outputs, and a SHA-256 identity. Changed inputs produce a different identity. A report's internal checks are distinct from its independent verification status. -**Stock explorer** is the default view. **Fetch IBM · public access** requests the documented public IBM endpoint. Select another card and use **Fetch [symbol] with a key** for AAPL, AMZN, GOOGL, IBM, MSFT, NVDA or SPY. The same connector remains in Historical data. This intentionally small US/USD universe avoids inferring currency from arbitrary tickers: the daily endpoint has no currency field. +The [Python verifier](verification/python/verify_research.py) independently replays accounting, signals, trades, risk metrics, and fingerprints. The service validates replay receipts against the **exact report bytes**. C++ separately checks nine risk comparison groups during native-verifying research jobs. -Stock explorer reads the existing owner-scoped dataset APIs. It admits only server-stamped Alpha Vantage historical unadjusted-close snapshots, choosing the newest observation date for each supported symbol, then save time and fingerprint to break ties. Reload saved prices makes no provider request. Missing symbols show no price; there is no synthetic fallback. A failed reload retains previously loaded snapshots and their dates. The card and chart link to the exact dataset and provenance; request-history links open their exact snapshot even when a newer one exists. Latest change compares the two last supplied observations, displays both dates, and excludes dividends. +A passing consistency check does not authenticate imported prices or predict future performance. Browser-imported Alpha Vantage data retains separate provenance because the server cannot authenticate that transfer directly. -Demo portfolio, transactions and pipeline now sit under **Demo · Fictional data**. Their dialog and transaction controls explicitly identify simulated prices and demo records. Real provider prices never revalue the demo ledger. +[Report checks](docs/confidence-certificates.md) · [Independent replay](docs/independent-replay.md) · [Verifier](verification/python/README.md) -The connector calls Alpha Vantage `TIME_SERIES_DAILY` with its default compact output (up to 100 recent observations), preserves exact closing prices, provider refresh date and timezone, and labels all observations unadjusted. This is on-demand daily historical data, not a streaming feed. It does not infer exchange-calendar completeness, intraday freshness, total return or split corrections. Provider demo availability and account entitlements remain controlled by Alpha Vantage. +### Recovery with a bounded queue -A one-time key goes directly from the user's browser to the fixed Alpha Vantage HTTPS endpoint, with cookies and referrers omitted. MarketLab first reserves the request under the owner's shared cooldown, then validates the returned price transfer again before saving it. The key is cleared from the UI after submission and is not sent to the MarketLab server or written to browser storage. Only validated price fields and dates return to the app; provider notices remain redacted and ephemeral in the browser. An optional `ALPHA_VANTAGE_API_KEY` server secret supports a configured connection. Production secrets belong in Sites; `.env.example` contains only the empty local setting. +Research submissions have signed ownership, nonce replay protection, and database-enforced global/per-owner active limits. MongoDB-clock leases and lease tokens fence heartbeats and finalization. A crashed calculation may run again; a stale worker cannot replace the current worker's durable result. -Requests have a 20-second timeout, a 128 KiB response cap, redirect rejection and strict response validation. Provider notices are mapped to fixed errors instead of returning messages that could echo keys. A database reservation atomically enforces a one-minute per-user cooldown. There is no automatic polling or retry. Already saved data is read locally without another provider call; repeating an unchanged response reuses the same snapshot. Changes create a new immutable snapshot. +The crash fixture kills a real runner after calculation and before saving, waits for natural lease expiry, and starts competing replacements. Fixed-arrival measurements separately exercise overload, rejection, and recovery. -The server records server-fetched (`alphavantage`) and browser-transferred (`alphavantage-browser`) origins separately from CSV claims. Browser data is validated but its provider origin is not independently authenticated; the interface and confidence certificate preserve that distinction. Provider identities include the connection provenance while all existing CSV identities remain stable. Completed and failed requests retain timestamps, symbols, connection method (never the key), counts and safe messages. Browser reservations expire after five minutes and accept one completion; concurrent replays cannot save additional datasets. If execution ends before finalization, an old running or saving entry is shown as unfinished; a run-history failure does not falsely report successfully saved data as lost. The workspace retains the 30-dataset limit. +[Lease-race and crash evidence](docs/reliability-evidence.md) · [Admission and overload](docs/overload-handling.md) · [Store implementation](services/research/store.ts) -Provider contract references: [Alpha Vantage daily API documentation](https://www.alphavantage.co/documentation/#daily), [public IBM example](https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol=IBM&apikey=demo), [provider access and rate limits](https://www.alphavantage.co/support/). +## Runtime boundaries -## Splits and cash dividends +![MarketLab architecture: the React interface talks to a TypeScript Worker and D1; signed copied snapshots go to a separate Node/MongoDB research service with C++ and Python checks.](docs/media/architecture.png) -In **Historical data → Splits & dividends**, add an event record to an unadjusted close dataset. Give the event source, split effective dates with new/old whole-share ratios, and cash-dividend ex-dates with USD per post-split share. Confirm complete coverage of the period after the first close through the last close, including an explicit empty list if there were no events. Corporate actions are manually supplied; the daily-price connector does not fetch them. Records cannot be applied to adjusted or unknown-price datasets. +The hosted app uses **Workers + D1**. Its saved workspaces and reports remain authoritative in D1. A separate **Node / MongoDB research service**, verified on Railway staging, owns copied research inputs, background jobs, and their results. C++ is a Node-API addon; Python is an independent, bounded replay process. The website has not been migrated to MongoDB. -The model starts with one share at the first supplied close. A split multiplies the holding by new/old while original acquisition cost stays constant. Shares and dividend amounts use reduced BigInt fractions, retaining reverse-split fractional shares exactly. A dividend adds the then-held share count times the declared per-share amount to a cash receivable on ex-date. That receivable earns no interest and is not reinvested. Same-date splits precede dividends, which must be specified per post-split share. First-date events are excluded because the modeled holding begins at that close. +[Diagram source](docs/media/architecture.svg) · [Architecture notes](docs/architecture-roadmap.md) · [Service setup](docs/research-service.md) · [Staging evidence](docs/research-staging.md) -The chart compares raw price, split-aware holding value, and holding value plus accumulated dividends, each rebased to 100. It values only supplied observations. Events between observations accrue chronologically before the next supplied valuation; no event-day close or reinvestment price is invented. Observed wealth decline uses only those dates and is not an intraday drawdown. Number is used only for bounded presentation ratios after exact arithmetic. +## Measured service workload -This is a hypothetical gross buy-and-hold comparison, not broker payment-date cash accounting, reinvested total return, a tax calculation, or the synthetic portfolio's return. Taxes, fees, cash in lieu, spin-offs, rights, mergers, return-of-capital cost adjustments and other non-cash distributions are outside this model. Users must confirm there are no unsupported distributions; completeness and source authenticity are not independently verified. +Three local runs completed **3,000 verified jobs** with zero failures. The combined measurements were: -Each dataset has one current owner-scoped event record (maximum 100 events). Edits replace that record with an incremented revision; an atomic revision check rejects stale concurrent saves. Prior event revisions are not retained, so export an analysis before replacing it if needed. The immutable original price dataset and its hash never change. The JSON analysis export includes exact observations, source, current events and revision, modeling assumptions, and all normalized comparison values. +| Metric | Result | Scope | +| --- | ---: | --- | +| Verified throughput | **39.96 jobs/s** | 3 × 1,000 jobs; warm-up excluded | +| Client-observed p99 | **319.77 ms** | Combined raw samples; signing through validated result retrieval | +| Calculation workers / clients | **2 / 8** | Closed-loop clients, one job in flight per client | +| Inputs per job | **500 observations per instrument** | Two fictional instruments; native risk checks enabled | -Financial conventions reference [Investor.gov: stock splits](https://www.investor.gov/introduction-investing/investing-basics/glossary/stock-split) and [Investor.gov: ex-dividend dates](https://www.investor.gov/introduction-investing/investing-basics/glossary/ex-dividend-dates-when-are-you-entitled-stock-and). The ex-date receivable treatment is an explicit research convention, separate from payment-date cash settlement. +The service, load driver, and standalone MongoDB ran on the same Linux machine with Node 24.19.0 and MongoDB 8.0.17. Timings include HTTP, queueing, computation, storage, polling, and client checks. They exclude process startup and the Workers/D1 gateway. These are **local workload measurements**, not production capacity, live-site latency, or an open-loop SLO. -## Historical portfolios +[Full methodology and environment](docs/service-performance.md) · [Raw report](docs/evidence/service-load-2026-09-26/report.json) · [Separate fixed-arrival results](docs/overload-handling.md) -**My portfolio** starts empty. Upload a supported Charles Schwab transaction-history CSV directly, or use MarketLab's normalized ledger format. The Schwab adapter handles its native headers, title row, dates, quoted USD values and optional total without manual cleanup. Review exact net cash, explicit fees, same-day ordering and symbol mappings before previewing and saving. Unsupported activity stops the entire import. See [supported actions, limitations and acceptance evidence](docs/brokerage-import.md). +## Run locally -For the normalized ledger, import complete USD history from cash funding using this exact header: +Use **Node 24** and **pnpm 11.25.0** for the full engineering gate. Core unit checks do not require an API key or a paid provider account. -```csv -id,date,type,symbol,shares,amount,fee,reference -fund-1,2026-09-14,deposit,,0,1000,0, +```sh +pnpm install --frozen-lockfile +pnpm test +pnpm typecheck +pnpm lint ``` -Types are `deposit`, `withdrawal`, `buy`, `sell`, and `dividend_payment`. `amount` is gross USD consideration, not unit price; the fee is separate. Currency amounts support two decimals and shares six decimals. Cash flows use no symbol/reference; dividend payments use a symbol and the original ex-date in `reference`, zero shares and zero fee. Dates are processed chronologically and same-date ledger rows retain file order. IDs must be unique. The import replaces the complete current ledger; repeated identical saves do not create duplicates. - -Map each symbol to one owner-held unadjusted close dataset with a saved complete corporate-action record. Instrument identity (including any ticker reuse), currency and event completeness are explicitly user-declared. Historical and fictional data retain their classification; a fictional broker file or synthetic price input prominently marks the portfolio as containing fictional inputs. The sample portfolio's seed ledger is never imported into this workspace. - -The `historical-close-v1` engine applies splits to carried-in holdings, then accrues dividends on ex-date before that day's trades. Fractional split shares use exact reduced BigInt fractions. Aggregate share count times dividend per post-split share is rounded half up once to USD cents for each entitlement. Entitlements remain tied to those original shares, even if the shares are subsequently sold. A matching `dividend_payment` transfers receivable to cash without additional income. Unpaid dividends cannot fund purchases or withdrawals. The full entitlement must be paid at once; partial payments and withholding are not represented. - -All ledger events use the daily-close timing convention. Every positive holding needs an exact-date raw close on every valued date (the union of supplied dates, ledger dates, corporate-action dates and the chosen final date). Trades also require their own symbol's exact-date close. No stale/future marks or implicit calendar filling are allowed. Consequently, a cash movement on an unquoted date while stocks are held stops valuation in this first version. Each bound price/event record must cover the chosen final date. Dates before the initial holding may be cash-only. - -Cost is weighted average including purchase fees. Sale fees reduce proceeds; remaining basis is allocated proportionally in cents. TWR chains market changes and internal events while excluding external flows. Reconciliation requires `wealth - net contributions = realized gain + unrealized gain + earned dividends`. Prices preserve six decimals and currency/share arithmetic is exact until documented cent rounding and dimensionless display ratios. - -One current portfolio per owner supports 1–500 rows, up to 10 symbols and a combined frozen payload of 1 MiB. It starts in cash: opening stock balances and incomplete histories are unsupported. No shorting, leverage, FX, tax-lot accounting, spin-offs, cash in lieu or other unsupported distributions. Preview errors identify missing coverage or invalid cash/holdings before save. This is research accounting, not a brokerage integration. - -Saving requires the exact preview fingerprint and an atomic revision check. The saved payload pins complete price observations, corporate-action contents and metadata, transactions, as-of date and calculation version. Later edits in Historical data do not rewrite the saved result. A new import uses the current chosen records and must be previewed again. The JSON export includes the pinned payload, assumptions, daily values, holdings, event journal and dividend entitlements. Previous portfolio revisions are replaced, so export before updating if an archive is required. - -## Portfolio benchmark comparison - -In **My portfolio → Edit portfolio**, select an optional saved USD benchmark. The model invests the same deposits and sells shares for the same withdrawals at exact closes, preserving same-date order. Splits and non-reinvested dividend receivables are included. It assumes zero benchmark trading costs; your actual ledger retains recorded fees. Insufficient benchmark funds or missing closes stop the comparison. The screen shows matched value paths, dollar gain differences and time-weighted return differences. Full inputs and calculation version are frozen with the saved portfolio; old portfolios remain compatible. +A clean clone selects the portable development profile. Run: -See [portfolio-benchmark.md](docs/portfolio-benchmark.md) for timing, limits, hand-calculated fixtures and export semantics. This is a hypothetical chosen-instrument comparison, not an official total-return index. +```sh +pnpm build +# Apply each pending drizzle/*.sql migration in filename order, once. +node --import ./scripts/sites-env.mjs ./node_modules/wrangler/bin/wrangler.js \ + d1 execute DB --local --config dist/server/wrangler.json \ + --persist-to .wrangler/state --file drizzle/0000_demonic_nightmare.sql +# Repeat the migration command with each subsequent pending SQL filename. +pnpm dev +``` -## Independent Python verification +Open `http://localhost:5173`. Portable development includes a loopback-only mock sign-in for fictional local workspaces; this is not the hosted identity provider. Managed Sites previews use their supervised preview command instead. See [runtime setup](docs/runtime.md) and [the local sign-in implementation](build/sites-vite-plugin.ts). -Saved Research lab experiments now offer **Run independent replay**. Successful -checks attach a receipt bound to both the frozen inputs and results; the -confidence certificate and downloads show its timestamp and scope. Existing -public summaries change only after a new sharing review. See -[receipt binding and acceptance boundaries](docs/independent-replay.md). +The first-run example and authored practice portfolio need no market-data key. Alpha Vantage access is optional and subject to its provider limits. Keep keys in server secrets or the one-time provider flow, never in committed files. -Python 3.11+ independently replays exported Research lab experiments using only its standard library. Exact cash/share/trade results and input hashes must agree; floating risk/return metrics use documented tolerances. It does not call TypeScript or verify market-data authenticity. +For the complete gate, also install **MongoDB 8.0.17, a C++17 compiler, and Python 3.11+**: ```sh -pnpm verify:python exported-research.json +# Starts an isolated loopback database and removes its test data afterward. +MONGOD_BIN=/path/to/mongod pnpm test:service:local --ci + +# Independent Python rejection tests and a fresh cross-language replay. pnpm test:python pnpm test:parity ``` -The verifier is supporting tooling; TypeScript remains the application core. See [verification/python/README.md](verification/python/README.md) for supported report formats and checks. Historical portfolio reports use a separate format and are not accepted by this verifier. - -## API - -| Endpoint | Purpose | -| --- | --- | -| `GET /api/workspace` | Current analytics, quotes, ledger, revision, and run history | -| `POST /api/transactions` | Validate and append an idempotent transaction | -| `POST /api/pipeline` | Validate and ingest/replay the fixture dataset | -| `GET /api/report` | Download the synthetic portfolio CSV report | -| `GET /api/datasets` | List the signed-in user’s imported datasets | -| `GET /api/datasets?id=…` | Read an owned dataset; `download=1` exports it | -| `POST /api/datasets` | Validate and atomically save an immutable CSV snapshot | -| `GET /api/provider` | Read connection readiness and owned provider-run history | -| `POST /api/provider` | Fetch, validate and save supported daily price history | -| `GET /api/actions?id=…` | Read the owned event record; `download=1` exports analysis JSON | -| `POST /api/actions` | Validate and save events with revision conflict protection | -| `GET /api/portfolio` | Read the owned historical portfolio/comparison; `download=1` exports frozen JSON, `download=csv` exports observations | -| `POST /api/portfolio` | Preview or save a complete historical ledger and its bound inputs | -| `GET /api/research` | List owned runs; `id` opens one; `download=json` or `csv` exports | -| `POST /api/research` | Preview or save an immutable experiment, with preview fingerprint verification | -| `POST /api/research/replay` | Owner-only Python replay of a saved report, with a bound persisted receipt | - -Production requires platform authentication. A development-only identity supports the internal preview. User identity is never accepted from a transaction request body. +`pnpm test` runs the unit layer. The complete gate also runs database/HTTP/worker/crash integration, native comparisons, Python replay, and checks against the production-built Worker with isolated D1. Test records use fictional inputs. [See the layers and prerequisites](tests/README.md). -## Bowers connection +To reproduce the recorded local service workload: -MarketLab is an independent project. The reporting and data-quality components could later support a specific Bowers reporting or teaching need, if agreed with the team. This repository makes no claim of Bowers sponsorship, deployment, investment activity, or access to institutional financial data. - -## Next engineering milestones - -The [adopted roadmap](docs/engineering-roadmap.md) replaces the earlier provider-expansion-first order. The client-clock lease bug is fixed with failing-before/passing-after evidence; a SIGKILL recovery test proves one durable result after a retry. The project is published to [GitHub](https://github.com/michaelbawuah/MarketLab) with an automated unit, integration and cross-check gate. See [verification records](docs/verification.md) for observed local and remote results. - -The one-click synthetic demo, report confidence certificates and Schwab CSV import are implemented and browser-checked using fictional fixtures. The service now has a measured local throughput/p99 baseline with raw evidence. The interactive exact-money explanation and runnable floating-point failure examples are also complete; see [representation, rounding and reproduction](docs/money-arithmetic.md). Shareable reports, event overlays and Railway free-trial staging are now implemented. Invite-only report discussion is accepted within its approved scope: automated authorization checks passed, and the user reported completing the separate-account comment and revocation flow. Additional strategies and asset classes remain deferred. See the engineering roadmap for the latest acceptance evidence and remaining checks. - -Service performance numbers apply only to their documented local workload. They establish no production SLA, investment return or hiring outcome. See `docs/verification.md` for the checks performed on this release. +```sh +pnpm native:build +MONGOD_BIN=/path/to/mongod pnpm bench:service \ + --output /tmp/marketlab-service-run +``` -## Research lab +The output directory must not already exist. The runner creates its own database and retains raw timing and correctness evidence. [Options and reproduction details](docs/service-performance.md#reproduce). -**Compare saved experiments** lets you inspect a second saved run without recalculating or changing either snapshot. It follows the full/earlier/later period selected above. Return, observed drawdown, ending wealth, executed trades and fees appear side by side. Differences are comparison minus current and are shown only when methods, frozen price/event inputs, initial cash, evaluation boundaries and exact observation grids match. Window, fee and slippage changes are listed explicitly; multiple changes do not isolate a single cause. A positive drawdown difference means a smaller observed decline. Comparisons are descriptive and do not select an optimal strategy or certify predictive performance. +## Browse the code -The `observed-close-sma-v1` engine uses raw USD close snapshots plus complete user-declared event records. Long means current close strictly above its trailing SMA; otherwise cash. Windows count observations. The previous observation signal executes at the next supplied close, after split and dividend entitlement processing. New purchases use millionth shares, split fractions remain exact, fills use price millionths, cash/gross/fees round half up to cents. Purchases conservatively cap sizing at exact pre-rounding affordability and then enforce rounded cash affordability. No shorting, borrowing, interest, final liquidation or dividend reinvestment is modeled; dividends remain nonspendable receivables. +| Path | Responsibility | +| --- | --- | +| [`app/`](app/) | React screens and hosted API routes | +| [`lib/finance/`](lib/finance/) | Exact accounting, validation, portfolio analysis, backtests, and report fingerprints | +| [`services/research/`](services/research/) | Signed Node API, MongoDB store, bounded worker pool, crash recovery, and Python replay | +| [`native/`](native/) | C++ risk kernel and Node-API binding | +| [`verification/python/`](verification/python/) | Independent replay and verifier rejection tests | +| [`tests/`](tests/) | Unit, real integration, native parity, and fictional fixtures | +| [`docs/evidence/`](docs/evidence/) | Retained execution records, workload definitions, and raw samples | -The asset and selected benchmark must have identical full observation grids in the evaluation range. Each uses the same starting cash, fee and adverse slippage rates. Buy-and-hold is a modeled investment in the selected instrument, not an official total-return index. The earlier and later periods independently restart with initial cash; only historical observations warm up signals. Repeated parameter trials can overfit the later period. +## Current limits -Risk statistics use consecutive observed wealth returns: sample standard deviation, Sharpe with zero cash return, covariance beta and Pearson correlation. They are not annualized and do not certify exchange-calendar completeness. Costs at the first evaluation close affect total return/drawdown but are outside the observed risk intervals; later costs enter those intervals. Undefined statistics are null, not zero. All three comparisons use the same dates. +The app is for educational planning and research. Assumptions and historical comparisons are inspectable; they are not forecasts or investment recommendations. Read-only sharing requires an explicit reviewed summary, and invited discussions do not grant workspace access. -Up to 30 immutable experiments per owner save full price/event inputs, all settings, method version, computed results and trades. Combined input/result JSON is capped at 1,900,000 bytes with room below D1's row limit. Replays of identical snapshots reuse the same SHA-256 ID; changed event payloads/settings create distinct runs. Preview/save compare fingerprints to detect intervening input edits. JSON exports preserve reproducibility; CSV contains exact-cent wealth, cash and receivables for every observation and segment. This module evaluates hypothetical single-asset rules; it does not benchmark a user's cash-flow ledger or claim predictive profitability. +Automated fixtures, local browser previews, hosted staging checks, and user-reported acceptance are recorded separately. The new-account production pilot with three real participants remains open. No production throughput, database failover, completed accessibility audit, or uptime guarantee is claimed. -Method references: [Sharpe's original ratio and time-scaling discussion](https://web.stanford.edu/~wfsharpe/art/sr/sr.htm), [NIST sample variance](https://www.itl.nist.gov/div898/handbook/prc/section3/prc32.htm), and [QuantConnect bar availability](https://www.quantconnect.com/docs/v2/writing-algorithms/key-concepts/time-modeling/periods). Execution at the next observed close is this project's explicit conservative convention. +[Observed verification](docs/verification.md) · [Personal workspace acceptance](docs/personal-workspaces.md) · [Full reference guide](docs/reference-guide.md) diff --git a/docs/media/architecture.png b/docs/media/architecture.png new file mode 100644 index 0000000..b640bbc Binary files /dev/null and b/docs/media/architecture.png differ diff --git a/docs/media/architecture.svg b/docs/media/architecture.svg new file mode 100644 index 0000000..01eb647 --- /dev/null +++ b/docs/media/architecture.svg @@ -0,0 +1,50 @@ + + MarketLab runtime boundaries + A React interface calls the Worker application, which stores private workspaces and reports in D1. The Worker separately submits signed frozen research snapshots to a Node service with a MongoDB job store, worker threads, C++ risk comparisons, and on-demand independent Python replay. + + + + + + One interface. Two distinct runtimes. + D1 owns the app's saved records. MongoDB owns copied research jobs and their results. + + HOSTED APPLICATION · WORKERS + D1 + + React interface + Plan · import · compare · share + Fictional demo without sign-in + + TypeScript Worker + Verified identity · owner isolation + Validation · canonical finance engine + + D1 / SQLite + Private workspaces · frozen reports + Reviewed public-summary links + + + + + Signed request · immutable input copy + + SEPARATE RESEARCH SERVICE · NODE + MONGODB + + MongoDB job store + Bounded global / owner admission + Database-clock leases + Fenced durable finalization + + Node API + worker pool + HMAC authentication · replay checks + TypeScript accounting / backtests + Bounded threads · retry recovery + + Independent checks + C++ · nine risk comparison groups + Python · on-demand report replay + Receipts bound to exact report bytes + + + A retried calculation may run again; only the current lease holder can finalize its result. + diff --git a/docs/reference-guide.md b/docs/reference-guide.md new file mode 100644 index 0000000..b627905 --- /dev/null +++ b/docs/reference-guide.md @@ -0,0 +1,274 @@ +# MarketLab reference guide + +This is the detailed implementation reference preserved from the former root README. +For the current overview and first-run commands, see the [project README](../README.md). + +A portfolio planning and analysis app with saved goals, a combined view of accounts, transparent fee projections and independently checkable backtests. + +**Engineering direction:** follow the adopted [engineering roadmap](engineering-roadmap.md), based on Michael's supplied standards. It includes the directory map, ordered milestones and observed status. The first reliability work now has [reproducible clock-race and process-crash evidence](reliability-evidence.md). + +**Reliability extension:** bounded background calculations and per-owner limits +are active on the hosted staging worker, with friendly retry messages and +reproducible fixed-arrival overload/crash evidence. The approved database migration, +signed HTTPS/native/Python/crash checks and ordinary-role restart passed; temporary +administrator references were removed. Real-user pilot acceptance remains pending. +See [mechanism, measurements and limits](overload-handling.md). + +**Current release:** personal saved workspaces with email or ChatGPT sign-in, a zero-setup practice portfolio, and public read-only report sharing. Each account opens on My plan and can return to its own saved accounts, goals and reports. It includes on-demand Alpha Vantage daily-price fetching, historical CSV imports, split/dividend research, user-imported historical portfolios with cash-flow-matched benchmarks, saved strategy experiments with benchmark/risk comparisons, and a separate synthetic portfolio demo. The separate demo’s six price series are explicitly synthetic, covering 127 weekday observations from April 1 to September 24, 2026. They are not real market history, exchange calendars, forecasts, or a backtest. + +See [personal workspaces and acceptance evidence](personal-workspaces.md) for account separation, first-use flow, and the remaining live pilot gates. + +## What works + +- **My plan:** saved manual accounts and debts, goal progress under editable assumptions, + cost-of-fees projections, allocation drift alerts, contribution and stress scenarios, + plus a transparent planning checklist. Historical backtests can supply a goal stress + scenario without being treated as future forecasts. +- **Strategy library:** tested trend templates and explicitly published community reports + with report-bound independent replay receipts. Reuse rule settings on your own data. +- **Optional peer comparison:** fixed-quarter cohorts, at least 20 opt-in accounts, + rounded aggregate results, frozen snapshots and withdrawal controls. +- **Trust and support:** in-app support requests and owner replies, privacy/deletion + disclosures, educational-use framing and an on-demand service status check. + No bank-sync, bank-grade certification, completed accessibility audit or uptime + guarantee is claimed. [Release details and acceptance evidence](ongoing-use-release.md). + +- Invite-only report discussions with ChatGPT or verified email sign-in: + single-use invitations bind to the chosen account, owners revoke access and + moderate comments, and database triggers retain an append-only audit trail. + The ChatGPT two-person comment/revocation check passed as user-reported evidence. + Email sign-in through WorkOS Staging is confirmed by the user and live callback + log; the user also confirmed the email-account comment/revocation checklist. + [Discussion model](research-collaboration.md) · [Email sign-in and evidence](discussion-email-sign-in.md). + +- Selectable buy, sell, split and dividend markers on Research lab equity curves, + with exact cash/fee explanations and event dates retained across missing closes. + [Timeline behavior and evidence](research-sharing-and-events.md). + +- Reviewed, redacted read-only experiment summaries with expiring links, + replacement and revocation. Public access is activated while private workspace + routes remain private to each signed-in account. A [fictional public example](https://marketlab-portfolio.michaelbaffour240306.chatgpt.site/example) + contains only built-in teaching data. Live browser verification remains outstanding. + [Sharing boundary and activation](research-sharing-and-events.md). + +- Confidence certificates and plain-language takeaways on analytical reports, with source/coverage details, actual per-result consistency checks, assumptions and explicit independent-replay status. JSON/CSV reports carry the certificate. [Scope and evidence](confidence-certificates.md). + +- A one-click **Quick experiment**: compare the same fictional rule and prices with three cost settings, inspect exact values and download a reproducible report. No upload or market-data key is required. [Demo inputs and acceptance evidence](quick-demo.md). + +- Side-by-side saved-experiment comparisons in Research lab, with exact-cent differences and explicit checks for matching frozen prices, event records, starting cash and observation dates. + +- Cash-flow-matched portfolio benchmarks with frozen price/event inputs, exact fractional holdings, observed return comparisons and JSON/CSV exports. +- A supporting Python verifier that independently replays Research lab reports and checks exact accounting, signals, trades, risk metrics and input fingerprints. + +- A Node/MongoDB research job service with signed ownership, concurrency-safe quotas, crash recovery, bounded worker threads and C++ risk parity checks. Railway free-trial staging passed remote crash recovery; the owner's September 27 saved-experiment browser flow passed all nine native comparisons. [Evidence and limits](research-staging.md). + +- A reproducible TypeScript service load runner with raw timings and correctness checks. Three local 1,000-job runs achieved **39.96 verified jobs/s** and **319.77 ms combined p99**, using two workers and eight concurrent clients with 500 observations per instrument. [Workload, environment and limits](service-performance.md). + +- A Research lab with next-close SMA backtests, costs, buy-and-hold and selected benchmark comparisons, observed-interval risk, separate chronological evaluation and immutable JSON/CSV reports. + +- An Alpha Vantage daily-price connector with IBM public demo access, request-only API keys, bounded fetching, persisted request history, and source-aware snapshots. +- A historical-data workspace with CSV validation, preview, persistent immutable snapshots, price charts, observed declines, coverage details, and CSV export. +- Saved, source-declared split/dividend event records with exact fractional-share arithmetic, cash-inclusive performance comparison and an audit export. +- An empty historical portfolio workspace with full-ledger CSV imports, exact-date price bindings, frozen data/event inputs, dividend receivable/payment accounting, reconciliation and export. +- A sample portfolio dashboard with value history, contribution overlays, time-weighted returns, allocation, and holdings. +- Search and price charts for six sample assets. +- Server-persisted buy, sell, deposit, withdrawal, and dividend entries, including trade fees and fractional shares. +- Historical cash and position validation: an entry cannot create negative cash or a short position, including on earlier dates. +- An append-only manual ledger with request idempotency and an atomic revision check for concurrent writes. +- A dataset ingestion endpoint with validation, immutable dataset identity, unique symbol/date keys, resumable batches, and persisted run outcomes. +- A downloadable CSV report containing dataset provenance, valuations, costs, and the full ledger. +- Per-user ledger and run isolation using the hosting platform's authenticated user ID. + +The first portfolio is intentionally a paper account. The built-in ledger is immutable; up to 500 additional manual events are supported. There is no brokerage connection, execution, or money transfer. + +## Technology + +| Area | Implemented | +| --- | --- | +| Interface | React 19, TypeScript, custom SVG charts, Shadcn primitives | +| Application server | TypeScript route handlers on a Cloudflare-compatible Worker | +| Persistence | Managed D1 / SQLite, schema migrations generated with Drizzle | +| Accounting | Pure TypeScript; integer cents and millionths of shares using BigInt | +| Validation | Zod request validation plus historical ledger replay | +| Verification | Node finance/integration tests, independent Python replay, TypeScript checking, ESLint and browser workflow checks | + +Node runs local tooling and tests. A separate signed Node research service persists jobs and results in MongoDB, computes through worker threads, and verifies risk metrics with a compiled C++ Node-API module. It runs on Railway free-trial staging, with remote crash-recovery evidence and an owner-confirmed saved-experiment verification flow. The live website continues to use Workers + D1. See [staging evidence](research-staging.md) and `docs/research-service.md` for the deployment boundary and setup. + +## Run and check + +Use Node 24 for the complete engineering gate and the checked-in pnpm lockfile. + +```sh +pnpm install --frozen-lockfile +pnpm test +pnpm typecheck +pnpm lint +pnpm dev +``` + +`pnpm test` runs the unit layer. With a C++17 compiler, Python 3.11+ and `mongod` +installed, `pnpm test:service:local --ci` runs the complete verification gate +against a disposable database. Set `MONGOD_BIN` when needed. The gate includes +real HTTP/worker/crash tests and independent Python/C++ comparisons. See the +[test layers and focused commands](../tests/README.md). The +[GitHub workflow](https://github.com/michaelbawuah/MarketLab/actions/workflows/verify.yml) +runs this same gate on pushes and pull requests; inspect its run result for remote +verification evidence. + +For a standalone local environment, the starter's execution-profile helper configures the development server. This repository was created in the managed Sites environment; managed preview uses `sites-preview start` instead of launching an additional server. Generate database migrations with `pnpm db:generate`, build to generate the local Worker config, and apply pending SQL migrations to the local D1 database using Wrangler. See `docs/runtime.md` for the runtime setup and migration command. + +## Accounting conventions + +Amounts crossing the API and stored in the ledger are integer strings. No financial amount is parsed with JavaScript's floating-point `parseFloat`. A share has six decimal places; a currency amount has two. Share-price products are rounded half up to the nearest cent. Final display values and dimensionless performance ratios use Number. + +Cost basis is weighted-average acquisition cost, including buy fees. Partial sales allocate cost proportionally; a final sale takes the exact remaining basis. Realized gain deducts sell fees. This is a research convention, not tax-lot accounting. + +Valuation uses only the latest quote dated on or before the observation. Missing initial prices stop valuation. Daily close prices mark existing holdings first; that date's ledger events then execute in stable order. Return chains each market movement and each internal ledger event. Deposits and withdrawals change value without changing return. Fees, execution prices, and dividends affect return. Same-day event order and the close-price timing assumption matter; this is not intraday-accurate performance measurement. + +The demo's weekday calendar intentionally includes weekdays that may be exchange holidays. No holiday-calendar accuracy is claimed. The synthetic portfolio ledger does not apply corporate actions. Separate historical research supports the explicit model below. Foreign exchange, shorting, leverage, and multi-account support are not implemented. Historical portfolios support the separate cash-flow benchmark model documented below. + +## Pipeline behavior + +`POST /api/pipeline` validates all 762 records, then stores batches of 50 with unique `(dataset, symbol, date)` keys. Running it again leaves the same number of rows. Completed and failed runs retain actual counts and elapsed times. Interrupted batches can be resumed by replay; there is no automatic scheduler or retry queue yet. Dataset version IDs must change if fixture values change. + +Before ingestion, the UI uses the explicitly labeled fixture preview. After ingestion, calculations read the stored dataset. Partial persisted coverage fails closed; the Data pipeline view remains available to repair it with another ingestion. Prices are shared immutable fixtures; manual transactions and run history are scoped to the signed-in user. + +## Historical CSV research + +Open **Historical data** to upload or paste a daily USD price series. The importer accepts `date` (or `timestamp`) with `close`, or an explicitly selected `adjusted_close` / `Adj Close` column. An optional `symbol` column must match the selected symbol; an optional `currency` column must be USD. Choose the source name, historical versus synthetic classification, and adjustment basis explicitly. Unknown adjustments are supported and labeled. + +Imports are limited to 256 KiB, 2–2,500 observations, one symbol, and 30 saved datasets per user. Prices use integer millionths of a dollar, preserving up to six decimal places. Invalid dates, future observations, duplicate dates, mixed symbols, non-USD currency and invalid or over-precision prices fail validation before saving. Missing exchange sessions are not invented or declared complete. + +Each canonical dataset includes the metadata and sorted exact observations in its SHA-256 identity. Equivalent replays preserve the original snapshot; changed prices or metadata produce another snapshot. Each bounded snapshot is stored as normalized metadata plus a canonical JSON observation vector in one owner-scoped D1 row. A single SQL statement enforces the workspace limit and atomically inserts the whole snapshot, so partial imports are never visible. The raw uploaded file and unused columns are not retained. Original canonical observations are available in the export. + +Source, symbol and adjustment claims come from the uploader and are not independently authenticated. Charts show only supplied observations, with lines connecting them across gaps. Series change and maximum observed peak-to-trough decline use the declared price basis. They are not portfolio returns, intraday risk measures, or strategy results. Imports never replace synthetic portfolio marks or use the demo's fictional trade ledger. The Alpha Vantage connector below can fetch directly from the provider; CSV uploads keep their user-supplied classification. + +## Daily-price provider + +**Stock explorer** is the default view. **Fetch IBM · public access** requests the documented public IBM endpoint. Select another card and use **Fetch [symbol] with a key** for AAPL, AMZN, GOOGL, IBM, MSFT, NVDA or SPY. The same connector remains in Historical data. This intentionally small US/USD universe avoids inferring currency from arbitrary tickers: the daily endpoint has no currency field. + +Stock explorer reads the existing owner-scoped dataset APIs. It admits only server-stamped Alpha Vantage historical unadjusted-close snapshots, choosing the newest observation date for each supported symbol, then save time and fingerprint to break ties. Reload saved prices makes no provider request. Missing symbols show no price; there is no synthetic fallback. A failed reload retains previously loaded snapshots and their dates. The card and chart link to the exact dataset and provenance; request-history links open their exact snapshot even when a newer one exists. Latest change compares the two last supplied observations, displays both dates, and excludes dividends. + +Demo portfolio, transactions and pipeline now sit under **Demo · Fictional data**. Their dialog and transaction controls explicitly identify simulated prices and demo records. Real provider prices never revalue the demo ledger. + +The connector calls Alpha Vantage `TIME_SERIES_DAILY` with its default compact output (up to 100 recent observations), preserves exact closing prices, provider refresh date and timezone, and labels all observations unadjusted. This is on-demand daily historical data, not a streaming feed. It does not infer exchange-calendar completeness, intraday freshness, total return or split corrections. Provider demo availability and account entitlements remain controlled by Alpha Vantage. + +A one-time key goes directly from the user's browser to the fixed Alpha Vantage HTTPS endpoint, with cookies and referrers omitted. MarketLab first reserves the request under the owner's shared cooldown, then validates the returned price transfer again before saving it. The key is cleared from the UI after submission and is not sent to the MarketLab server or written to browser storage. Only validated price fields and dates return to the app; provider notices remain redacted and ephemeral in the browser. An optional `ALPHA_VANTAGE_API_KEY` server secret supports a configured connection. Production secrets belong in Sites; `.env.example` contains only the empty local setting. + +Requests have a 20-second timeout, a 128 KiB response cap, redirect rejection and strict response validation. Provider notices are mapped to fixed errors instead of returning messages that could echo keys. A database reservation atomically enforces a one-minute per-user cooldown. There is no automatic polling or retry. Already saved data is read locally without another provider call; repeating an unchanged response reuses the same snapshot. Changes create a new immutable snapshot. + +The server records server-fetched (`alphavantage`) and browser-transferred (`alphavantage-browser`) origins separately from CSV claims. Browser data is validated but its provider origin is not independently authenticated; the interface and confidence certificate preserve that distinction. Provider identities include the connection provenance while all existing CSV identities remain stable. Completed and failed requests retain timestamps, symbols, connection method (never the key), counts and safe messages. Browser reservations expire after five minutes and accept one completion; concurrent replays cannot save additional datasets. If execution ends before finalization, an old running or saving entry is shown as unfinished; a run-history failure does not falsely report successfully saved data as lost. The workspace retains the 30-dataset limit. + +Provider contract references: [Alpha Vantage daily API documentation](https://www.alphavantage.co/documentation/#daily), [public IBM example](https://www.alphavantage.co/query?function=TIME_SERIES_DAILY&symbol=IBM&apikey=demo), [provider access and rate limits](https://www.alphavantage.co/support/). + +## Splits and cash dividends + +In **Historical data → Splits & dividends**, add an event record to an unadjusted close dataset. Give the event source, split effective dates with new/old whole-share ratios, and cash-dividend ex-dates with USD per post-split share. Confirm complete coverage of the period after the first close through the last close, including an explicit empty list if there were no events. Corporate actions are manually supplied; the daily-price connector does not fetch them. Records cannot be applied to adjusted or unknown-price datasets. + +The model starts with one share at the first supplied close. A split multiplies the holding by new/old while original acquisition cost stays constant. Shares and dividend amounts use reduced BigInt fractions, retaining reverse-split fractional shares exactly. A dividend adds the then-held share count times the declared per-share amount to a cash receivable on ex-date. That receivable earns no interest and is not reinvested. Same-date splits precede dividends, which must be specified per post-split share. First-date events are excluded because the modeled holding begins at that close. + +The chart compares raw price, split-aware holding value, and holding value plus accumulated dividends, each rebased to 100. It values only supplied observations. Events between observations accrue chronologically before the next supplied valuation; no event-day close or reinvestment price is invented. Observed wealth decline uses only those dates and is not an intraday drawdown. Number is used only for bounded presentation ratios after exact arithmetic. + +This is a hypothetical gross buy-and-hold comparison, not broker payment-date cash accounting, reinvested total return, a tax calculation, or the synthetic portfolio's return. Taxes, fees, cash in lieu, spin-offs, rights, mergers, return-of-capital cost adjustments and other non-cash distributions are outside this model. Users must confirm there are no unsupported distributions; completeness and source authenticity are not independently verified. + +Each dataset has one current owner-scoped event record (maximum 100 events). Edits replace that record with an incremented revision; an atomic revision check rejects stale concurrent saves. Prior event revisions are not retained, so export an analysis before replacing it if needed. The immutable original price dataset and its hash never change. The JSON analysis export includes exact observations, source, current events and revision, modeling assumptions, and all normalized comparison values. + +Financial conventions reference [Investor.gov: stock splits](https://www.investor.gov/introduction-investing/investing-basics/glossary/stock-split) and [Investor.gov: ex-dividend dates](https://www.investor.gov/introduction-investing/investing-basics/glossary/ex-dividend-dates-when-are-you-entitled-stock-and). The ex-date receivable treatment is an explicit research convention, separate from payment-date cash settlement. + +## Historical portfolios + +**My portfolio** starts empty. Upload a supported Charles Schwab transaction-history CSV directly, or use MarketLab's normalized ledger format. The Schwab adapter handles its native headers, title row, dates, quoted USD values and optional total without manual cleanup. Review exact net cash, explicit fees, same-day ordering and symbol mappings before previewing and saving. Unsupported activity stops the entire import. See [supported actions, limitations and acceptance evidence](brokerage-import.md). + +For the normalized ledger, import complete USD history from cash funding using this exact header: + +```csv +id,date,type,symbol,shares,amount,fee,reference +fund-1,2026-09-14,deposit,,0,1000,0, +``` + +Types are `deposit`, `withdrawal`, `buy`, `sell`, and `dividend_payment`. `amount` is gross USD consideration, not unit price; the fee is separate. Currency amounts support two decimals and shares six decimals. Cash flows use no symbol/reference; dividend payments use a symbol and the original ex-date in `reference`, zero shares and zero fee. Dates are processed chronologically and same-date ledger rows retain file order. IDs must be unique. The import replaces the complete current ledger; repeated identical saves do not create duplicates. + +Map each symbol to one owner-held unadjusted close dataset with a saved complete corporate-action record. Instrument identity (including any ticker reuse), currency and event completeness are explicitly user-declared. Historical and fictional data retain their classification; a fictional broker file or synthetic price input prominently marks the portfolio as containing fictional inputs. The sample portfolio's seed ledger is never imported into this workspace. + +The `historical-close-v1` engine applies splits to carried-in holdings, then accrues dividends on ex-date before that day's trades. Fractional split shares use exact reduced BigInt fractions. Aggregate share count times dividend per post-split share is rounded half up once to USD cents for each entitlement. Entitlements remain tied to those original shares, even if the shares are subsequently sold. A matching `dividend_payment` transfers receivable to cash without additional income. Unpaid dividends cannot fund purchases or withdrawals. The full entitlement must be paid at once; partial payments and withholding are not represented. + +All ledger events use the daily-close timing convention. Every positive holding needs an exact-date raw close on every valued date (the union of supplied dates, ledger dates, corporate-action dates and the chosen final date). Trades also require their own symbol's exact-date close. No stale/future marks or implicit calendar filling are allowed. Consequently, a cash movement on an unquoted date while stocks are held stops valuation in this first version. Each bound price/event record must cover the chosen final date. Dates before the initial holding may be cash-only. + +Cost is weighted average including purchase fees. Sale fees reduce proceeds; remaining basis is allocated proportionally in cents. TWR chains market changes and internal events while excluding external flows. Reconciliation requires `wealth - net contributions = realized gain + unrealized gain + earned dividends`. Prices preserve six decimals and currency/share arithmetic is exact until documented cent rounding and dimensionless display ratios. + +One current portfolio per owner supports 1–500 rows, up to 10 symbols and a combined frozen payload of 1 MiB. It starts in cash: opening stock balances and incomplete histories are unsupported. No shorting, leverage, FX, tax-lot accounting, spin-offs, cash in lieu or other unsupported distributions. Preview errors identify missing coverage or invalid cash/holdings before save. This is research accounting, not a brokerage integration. + +Saving requires the exact preview fingerprint and an atomic revision check. The saved payload pins complete price observations, corporate-action contents and metadata, transactions, as-of date and calculation version. Later edits in Historical data do not rewrite the saved result. A new import uses the current chosen records and must be previewed again. The JSON export includes the pinned payload, assumptions, daily values, holdings, event journal and dividend entitlements. Previous portfolio revisions are replaced, so export before updating if an archive is required. + +## Portfolio benchmark comparison + +In **My portfolio → Edit portfolio**, select an optional saved USD benchmark. The model invests the same deposits and sells shares for the same withdrawals at exact closes, preserving same-date order. Splits and non-reinvested dividend receivables are included. It assumes zero benchmark trading costs; your actual ledger retains recorded fees. Insufficient benchmark funds or missing closes stop the comparison. The screen shows matched value paths, dollar gain differences and time-weighted return differences. Full inputs and calculation version are frozen with the saved portfolio; old portfolios remain compatible. + +See [portfolio-benchmark.md](portfolio-benchmark.md) for timing, limits, hand-calculated fixtures and export semantics. This is a hypothetical chosen-instrument comparison, not an official total-return index. + +## Independent Python verification + +Saved Research lab experiments now offer **Run independent replay**. Successful +checks attach a receipt bound to both the frozen inputs and results; the +confidence certificate and downloads show its timestamp and scope. Existing +public summaries change only after a new sharing review. See +[receipt binding and acceptance boundaries](independent-replay.md). + +Python 3.11+ independently replays exported Research lab experiments using only its standard library. Exact cash/share/trade results and input hashes must agree; floating risk/return metrics use documented tolerances. It does not call TypeScript or verify market-data authenticity. + +```sh +pnpm verify:python exported-research.json +pnpm test:python +pnpm test:parity +``` + +The verifier is supporting tooling; TypeScript remains the application core. See [verification/python/README.md](../verification/python/README.md) for supported report formats and checks. Historical portfolio reports use a separate format and are not accepted by this verifier. + +## API + +| Endpoint | Purpose | +| --- | --- | +| `GET /api/workspace` | Current analytics, quotes, ledger, revision, and run history | +| `POST /api/transactions` | Validate and append an idempotent transaction | +| `POST /api/pipeline` | Validate and ingest/replay the fixture dataset | +| `GET /api/report` | Download the synthetic portfolio CSV report | +| `GET /api/datasets` | List the signed-in user’s imported datasets | +| `GET /api/datasets?id=…` | Read an owned dataset; `download=1` exports it | +| `POST /api/datasets` | Validate and atomically save an immutable CSV snapshot | +| `GET /api/provider` | Read connection readiness and owned provider-run history | +| `POST /api/provider` | Fetch, validate and save supported daily price history | +| `GET /api/actions?id=…` | Read the owned event record; `download=1` exports analysis JSON | +| `POST /api/actions` | Validate and save events with revision conflict protection | +| `GET /api/portfolio` | Read the owned historical portfolio/comparison; `download=1` exports frozen JSON, `download=csv` exports observations | +| `POST /api/portfolio` | Preview or save a complete historical ledger and its bound inputs | +| `GET /api/research` | List owned runs; `id` opens one; `download=json` or `csv` exports | +| `POST /api/research` | Preview or save an immutable experiment, with preview fingerprint verification | +| `POST /api/research/replay` | Owner-only Python replay of a saved report, with a bound persisted receipt | + +Production requires platform authentication. A development-only identity supports the internal preview. User identity is never accepted from a transaction request body. + +## Bowers connection + +MarketLab is an independent project. The reporting and data-quality components could later support a specific Bowers reporting or teaching need, if agreed with the team. This repository makes no claim of Bowers sponsorship, deployment, investment activity, or access to institutional financial data. + +## Next engineering milestones + +The [adopted roadmap](engineering-roadmap.md) replaces the earlier provider-expansion-first order. The client-clock lease bug is fixed with failing-before/passing-after evidence; a SIGKILL recovery test proves one durable result after a retry. The project is published to [GitHub](https://github.com/michaelbawuah/MarketLab) with an automated unit, integration and cross-check gate. See [verification records](verification.md) for observed local and remote results. + +The one-click synthetic demo, report confidence certificates and Schwab CSV import are implemented and browser-checked using fictional fixtures. The service now has a measured local throughput/p99 baseline with raw evidence. The interactive exact-money explanation and runnable floating-point failure examples are also complete; see [representation, rounding and reproduction](money-arithmetic.md). Shareable reports, event overlays and Railway free-trial staging are now implemented. Invite-only report discussion is accepted within its approved scope: automated authorization checks passed, and the user reported completing the separate-account comment and revocation flow. Additional strategies and asset classes remain deferred. See the engineering roadmap for the latest acceptance evidence and remaining checks. + +Service performance numbers apply only to their documented local workload. They establish no production SLA, investment return or hiring outcome. See `docs/verification.md` for the checks performed on this release. + +## Research lab + +**Compare saved experiments** lets you inspect a second saved run without recalculating or changing either snapshot. It follows the full/earlier/later period selected above. Return, observed drawdown, ending wealth, executed trades and fees appear side by side. Differences are comparison minus current and are shown only when methods, frozen price/event inputs, initial cash, evaluation boundaries and exact observation grids match. Window, fee and slippage changes are listed explicitly; multiple changes do not isolate a single cause. A positive drawdown difference means a smaller observed decline. Comparisons are descriptive and do not select an optimal strategy or certify predictive performance. + +The `observed-close-sma-v1` engine uses raw USD close snapshots plus complete user-declared event records. Long means current close strictly above its trailing SMA; otherwise cash. Windows count observations. The previous observation signal executes at the next supplied close, after split and dividend entitlement processing. New purchases use millionth shares, split fractions remain exact, fills use price millionths, cash/gross/fees round half up to cents. Purchases conservatively cap sizing at exact pre-rounding affordability and then enforce rounded cash affordability. No shorting, borrowing, interest, final liquidation or dividend reinvestment is modeled; dividends remain nonspendable receivables. + +The asset and selected benchmark must have identical full observation grids in the evaluation range. Each uses the same starting cash, fee and adverse slippage rates. Buy-and-hold is a modeled investment in the selected instrument, not an official total-return index. The earlier and later periods independently restart with initial cash; only historical observations warm up signals. Repeated parameter trials can overfit the later period. + +Risk statistics use consecutive observed wealth returns: sample standard deviation, Sharpe with zero cash return, covariance beta and Pearson correlation. They are not annualized and do not certify exchange-calendar completeness. Costs at the first evaluation close affect total return/drawdown but are outside the observed risk intervals; later costs enter those intervals. Undefined statistics are null, not zero. All three comparisons use the same dates. + +Up to 30 immutable experiments per owner save full price/event inputs, all settings, method version, computed results and trades. Combined input/result JSON is capped at 1,900,000 bytes with room below D1's row limit. Replays of identical snapshots reuse the same SHA-256 ID; changed event payloads/settings create distinct runs. Preview/save compare fingerprints to detect intervening input edits. JSON exports preserve reproducibility; CSV contains exact-cent wealth, cash and receivables for every observation and segment. This module evaluates hypothetical single-asset rules; it does not benchmark a user's cash-flow ledger or claim predictive profitability. + +Method references: [Sharpe's original ratio and time-scaling discussion](https://web.stanford.edu/~wfsharpe/art/sr/sr.htm), [NIST sample variance](https://www.itl.nist.gov/div898/handbook/prc/section3/prc32.htm), and [QuantConnect bar availability](https://www.quantconnect.com/docs/v2/writing-algorithms/key-concepts/time-modeling/periods). Execution at the next observed close is this project's explicit conservative convention.