Feat/focus ledger schema

CHANGELOG.md

@@ -16,13 +16,64 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Budget tracking
 
 ### Added
+- **FOCUS billing ledger** (roadmap P0/PR2)
+  - New `billing.duckdb` with `fct_charge`, `ingest_batch`,
+    `fct_balance_snapshot` and `dim_fx_rate`, named after
+    [FOCUS](https://focus.finops.org/) columns
+  - Transactional whole-period replacement keyed by
+    (source, account, billing period), with deterministic charge ids so a
+    repeated ingest of an unchanged bill is a no-op
+  - Amounts stored in the currency they were billed in; conversion is left
+    to a view (PR6)
+- **Raw payload store** (roadmap P0/PR3)
+  - `fetch` persists provider responses unchanged as Hive-partitioned
+    Parquet under `raw/provider=…/account=…/billing_period=…/batch=…/`,
+    the same layout a bill export bucket uses
+  - `normalize` is a pure function from a stored batch to FOCUS rows, so
+    billing logic is testable from a recorded response and a mapping fix
+    replays payloads on disk instead of paying for another fetch
+- **AWS charges land as FOCUS rows** (roadmap P0/PR4)
+  - One Cost Explorer call now carries `UnblendedCost`, `AmortizedCost` and
+    `UsageQuantity`, grouped by service and record type
+  - `charge_category` comes from the record type, so credits, refunds,
+    taxes and support fees are each labelled as themselves instead of all
+    reading as usage; amounts keep their sign
+- **Alibaba Cloud and DeepSeek land in the ledger** (roadmap P0/PR5)
+  - Each Alibaba Cloud voucher, coupon and discount becomes its own
+    `Credit` row beside a gross usage charge, so a product's rows sum to
+    what was actually charged; an unexplained gap becomes one `Adjustment`
+    row instead of vanishing
+  - DeepSeek balances are recorded as snapshots, and a rise in the
+    topped-up balance between observations is derived as a `Purchase`
+- **The dashboard reads the ledger** (roadmap P0/PR6)
+  - Totals come from `v_charge_normalized`, which converts each charge at a
+    rate dated no later than the charge itself, so cross-cloud figures are
+    in one currency instead of adding dollars to yuan
+  - Reporting currency is a setting; switching it rebuilds a view and
+    rewrites nothing
+  - Charges in a currency no rate covers are reported on the dashboard
+    rather than being counted at par
 - **DeepSeek Integration**
   - DeepSeek API integration for balance queries
   - Display account balance instead of cost for DeepSeek accounts
   - Balance breakdown showing granted and topped-up balances
   - Support for multiple currencies (CNY, USD)
 
 ### Changed
+- The response cache tables are gone (application schema v2). A refresh
+  checks when a period was last ingested, which the ledger already records
+- A billing source only fetches and normalizes now; `get_cost_summary` and
+  `get_cost_trend` are gone, along with the per-call trend fetch — the
+  trend chart reads rows the refresh already stored
+- `CloudService` is now `BillingSource`, with `fetch` and `normalize` split
+  apart: the first touches the network and interprets nothing, the second
+  interprets and touches nothing
+- Billing source registry replaces the `CloudProvider` enum; an unknown
+  source id is skipped with a warning instead of being read as AWS
+- Application database is versioned and rebuilt at schema v1: the dead
+  `cost_data` table and the credential columns are gone, `provider` is now
+  `source_id`, and accounts and budgets are carried across. Credentials
+  live in the OS keyring only.
 
 ### Fixed
 

docs/roadmap.md

@@ -31,22 +31,38 @@ carry their weight for a personal ledger:
 
 ## Where we are
 
-The DuckDB file today is a cache, not a ledger. `cost_data` is dead code;
-the live path is two per-account cache tables holding JSON blobs. There is
-no fact table, so Sankey, attribution, anomaly detection and month-end
-freezing have nothing to build on.
-
-Three structural problems block everything downstream:
-
-1. **`CloudProvider` is a compile-time enum** with 48 references across 7
-   files. Adding a source means editing five `match` arms. `db.rs` also
-   silently coerces an unknown provider string to `AWS`.
-2. **Amounts are summed across currencies.** The dashboard total adds AWS
-   USD to Alibaba Cloud CNY and shows the result as one number.
-3. **`fetch` and `normalize` are fused.** `get_cost_summary()` returns a
-   display-shaped struct straight from the API. Cost Explorer charges per
-   request, so any schema change means paying to re-fetch, and there is no
-   way to unit-test the billing logic.
+**P0 is done.** A source is a registry row rather than an enum variant.
+`billing.duckdb` holds `fct_charge`, `ingest_batch`,
+`fct_balance_snapshot` and `dim_fx_rate` behind a transactional
+whole-period write. Every source fetches raw payloads to Parquet and
+normalizes them through a pure function, tested against a recorded
+response. The dashboard reads `v_charge_normalized`, so every figure on it
+is in one currency, converted per charge at a rate dated no later than the
+charge itself.
+
+The acceptance test holds: all three sources land in one `fct_charge`
+table, `SELECT sum(billed_cost_base) FROM v_charge_normalized WHERE
+billing_period = ?` is the cross-cloud total, and a repeated ingest of an
+unchanged bill produces identical rows.
+
+All three structural problems are closed:
+
+1. ~~**`CloudProvider` is a compile-time enum**~~ — replaced by the source
+   registry in PR1. An unrecognized source id is skipped with a warning
+   instead of being silently read as AWS.
+2. ~~**Amounts are summed across currencies.**~~ — fixed in PR6. Charges
+   are stored in the currency they were billed in and converted in a view,
+   so a rate correction or a change of reporting currency costs nothing.
+   A charge in a currency no rate covers is counted nowhere and reported
+   on the dashboard rather than quietly folded in at par.
+3. ~~**`fetch` and `normalize` are fused.**~~ — split in PR3. `fetch`
+   persists what the provider returned and interprets nothing;
+   `normalize` interprets and touches nothing, so a mapping fix replays
+   payloads already on disk instead of paying Cost Explorer again.
+
+What P0 did *not* do, and P1 owns: instance-level detail. Alibaba Cloud's
+bill overview is one row per product per month, so its trend chart is as
+coarse as its source data.
 
 ## P0 — FOCUS normalization
 
@@ -58,7 +74,7 @@ The one-sentence acceptance test for the whole phase:
 
 The six changes are a dependency chain — land them in order.
 
-### PR1 · Source registry
+### PR1 · Source registry — landed
 
 Replace the `CloudProvider` enum with a `SourceId` plus a descriptor table
 carrying a `Capabilities` struct. The unknown-provider fallback becomes a
@@ -69,12 +85,19 @@ its granularity is `SnapshotOnly`, not because it is called DeepSeek.
 Pure refactor, no behavior change. It comes first because every later PR
 would otherwise have to edit the same 48 sites.
 
-### PR2 · New database, `fct_charge`, batch table
+### PR2 · New database, `fct_charge`, batch table — landed
 
 A fresh `billing.duckdb` with a `schema_version` table; credentials move to
 their own store. Four tables: `fct_charge`, `ingest_batch`,
-`fct_balance_snapshot`, `dim_fx_rate`. The three cache tables are dropped —
-the data is re-fetchable, so no migration is written.
+`fct_balance_snapshot`, `dim_fx_rate`.
+
+As landed, the two cache tables stay behind in `cloudbridge.duckdb` rather
+than being dropped here: they are the only thing feeding the dashboard
+until PR4 and PR5 normalize into `fct_charge`, and dropping them early
+would mean paying Cost Explorer for a fetch on every launch in between.
+`cost_data` — dead code — is gone, and so are the credential columns: the
+application database is versioned and rebuilt at v1, keeping accounts and
+budgets, with secrets in the OS keyring only.
 
 Writes are transactional whole-period replacement keyed by
 `(provider, account_id, billing_period)`. Providers re-issue a bill in full
@@ -98,7 +121,7 @@ deferred:
 - `pricing_unit` is not restricted to cloud units. Today it holds `GB-Mo`
   and `Hrs`; tomorrow it holds `Tokens`.
 
-### PR3 · Split fetch from normalize, land raw Parquet
+### PR3 · Split fetch from normalize, land raw Parquet — landed
 
 `BillingSource` replaces `CloudService`. `fetch` retrieves and persists raw
 payloads unchanged; `normalize` is a pure function from raw to FOCUS rows.
@@ -116,26 +139,69 @@ so P1's S3/OSS export channel only replaces the `fetch` implementation —
 A pure `normalize` is also the first time billing logic becomes testable:
 record one API response per provider as a fixture and assert on the rows.
 
-### PR4 · AWS to FOCUS
+As landed, `CloudService` became `BillingSource` and all three sources
+implement both halves, so the pipeline is whole end to end
+(`ingest::ingest_period`) and re-normalizing without fetching is a
+supported operation (`ingest::renormalize_period`). What the normalizers
+do *not* do yet is the mapping detail PR4 and PR5 own: AWS asks only for
+`UnblendedCost` and files everything as `Usage`, Alibaba Cloud records the
+discount as the gap between `billed_cost` and `list_cost` rather than as
+`Credit` rows, and DeepSeek writes balance snapshots without deriving
+top-ups.
+
+### PR4 · AWS to FOCUS — landed
 
 Cost Explorer currently requests only `UnblendedCost`, grouped by `SERVICE`.
 Request `UnblendedCost`, `AmortizedCost` and `UsageQuantity` in a single
 call — each call is billed, so do not split it — and add `RECORD_TYPE` to
 the grouping to populate `charge_category`. `cost_basis` is `authoritative`.
 
-### PR5 · Alibaba Cloud and DeepSeek
+Three decisions worth recording:
+
+- Amounts keep the sign Cost Explorer gives them, so credits and refunds
+  stay negative and a period total is a plain sum.
+- A record type this build does not recognize is an `Adjustment`, with a
+  warning naming it. Money moved; calling it `Usage` would quietly inflate
+  what reads as consumption.
+- A row is only dropped when it is zero on *both* cost metrics. Usage
+  covered by a commitment is zero unblended and non-zero amortized, and
+  dropping it would lose what the commitment actually bought. Grouping by
+  service also mixes usage types, and Cost Explorer says so by returning
+  the unit `N/A`: a quantity like that is not stored, because it cannot be
+  added to anything.
+
+### PR5 · Alibaba Cloud and DeepSeek — landed
 
 Alibaba Cloud `QueryBillOverview`: `PretaxAmount` to `billed_cost`,
 `PretaxGrossAmount` to `list_cost`, each voucher/deduction as its own
 `Credit` row. Currency CNY.
 
+As landed, the usage row carries the **gross** amount and each deduction
+is a negative `Credit` beside it. Putting the net amount on the usage row
+*and* the deductions next to it would count them twice — Alibaba Cloud
+reports both figures on the same line, unlike AWS, which bills the
+discount as a line of its own. Decomposed this way a product's rows sum to
+`PretaxAmount`, which is what was actually charged, and a total stays a
+plain sum. Where the named deductions do not close the gap between gross
+and net, the remainder becomes one `Adjustment` row rather than
+disappearing.
+
 DeepSeek reports a balance, which is state, not a charge. It moves to
 `fct_balance_snapshot`; only top-ups become `fct_charge` rows with
 `charge_category = Purchase`. The current code stuffs the balance into
 `current_month_cost`, which is semantically wrong and blocks any correct
 total.
 
-### PR6 · Read through views, fix cross-currency
+Top-ups are *derived*, not stored: a rise in the topped-up balance between
+two consecutive observations is the only evidence of a purchase such a
+source gives, and re-ingesting a period recomputes them, since replacing a
+period clears what was there. The first observation of an account yields
+nothing — a balance that was simply there the first time it was looked at
+was not witnessed being paid for. The display path still reports the
+balance as `current_month_cost`; that is PR6's to fix, along with
+everything else the dashboard reads.
+
+### PR6 · Read through views, fix cross-currency — landed
 
 Amounts are stored in their original currency. Conversion happens in a
 view, never at write time, because rates get corrected and the user may
@@ -154,6 +220,19 @@ ASOF LEFT JOIN dim_fx_rate f
 Ships with a built-in rate table and a reporting-currency setting. This is
 where the cross-currency total is actually fixed.
 
+As landed, the view also carries `effective_cost_base` and the `fx_rate` it
+used, and a charge already in the reporting currency converts at 1.0
+without needing a row in the rate table. A charge whose currency no rate
+covers keeps a NULL `billed_cost_base`: it is left out of every converted
+total and counted separately, so the dashboard can say how many charges it
+is not showing rather than under-reporting silently.
+
+The freshness window moved into the ledger with the same change:
+`ingest_batch` records when each period was last written, which is what a
+refresh checks. The two response cache tables are gone, and so are
+`get_cost_summary` and `get_cost_trend` — a source now fetches and
+normalizes, and nothing else.
+
 ## P1
 
 - **Bill file export channel (S3 / OSS + Parquet).** Replaces per-request