diff --git a/LMLRoadmaps/Generalization/README.md b/LMLRoadmaps/Generalization/README.md new file mode 100644 index 0000000..6a4b174 --- /dev/null +++ b/LMLRoadmaps/Generalization/README.md @@ -0,0 +1,696 @@ +# Roadmap: Refactors toward the generality of the literature + +This roadmap restates material that LMLExplore already contains, in the generality that the +[Identification](../Identification/README.md), [Bandits](../Bandits/README.md) and +[ReinforcementLearning](../ReinforcementLearning/README.md) roadmaps specify. Those roadmaps ask +for the forms of the literature: tasks on arbitrary models, stopping-time tools for every +environment, thresholds that read the history, one algorithm where the literature has one, +adaptive adversaries and anytime learning rates, suprema that need not be attained, a discount in +every finite-horizon quantity. LMLExplore contains narrower versions of many of these items. Each +target here is a refactor of one merged development: it restates the development in the general +form and keeps every result it replaces. + +Each target comes before the items it unblocks, which it lists. A worker choosing between a target +and an item it unblocks takes the target first; Layer 0 comes first because new work builds on it. +A target whose prerequisites include an item of another roadmap that is not merged makes that item +its next step. + +## Scope and dependencies + +In scope: + +- identification: the tasks of `LMLExplore/Identification/Task/`, the GLR stopping rules and + Track-and-Stop (`Identification/GLR.lean`, `TrackAndStop.lean`, `ForcedExploration.lean`), the + elimination and LUCB algorithms; +- the sequential-learning tools of `LMLExplore/SequentialLearning/`: Wald's identity, the + transfer from runs to the reward stream, the divergence of stopped histories; +- adversarial bandits and online learning: `LMLExplore/Bandits/AdversarialRegret.lean`, + `ImportanceWeighted.lean`, `Exp3/`, `Exp4.lean`, `LaiRobbins.lean`, and + `LMLExplore/OnlineLearning/`; +- design operators and self-normalized bounds: `LMLExplore/Analysis/InnerProductSpace/` and + `LMLExplore/Probability/Martingale/SelfNormalized.lean`; +- dynamic programming and Markov decision processes: `LMLExplore/Analysis/FixedPoint/`, + `LMLExplore/Probability/Kernel/Resolvent.lean` and `LMLExplore/MarkovDecisionProcess/`; +- declarations of LMLExplore that Tau Ceti provides at the pinned version (Layer 2). LMLExplore's + scan for overlaps covers Mathlib only. + +**Relation to the other roadmaps.** The Identification, Bandits and ReinforcementLearning roadmaps +specify the mathematics, and each target here points to the item that specifies its general form. +This roadmap restates merged code in that generality and adds no item of those roadmaps. Where a +target needs a general statement that is not merged, it lists that item as a prerequisite. + +**Restatements that the other roadmaps own.** Some items of the other roadmaps specify a general +statement that LMLExplore does not have yet, and say which merged declarations become its +corollaries. The pull request that builds such an item derives those corollaries, so they are not +targets here: + +- Identification 1.2: the exponential laws and the Gaussian laws of known mean + (`ExponentialFamily/Exponential.lean`, `expFamilyDiv_sub_sq_gaussianReal`) as cases of the gamma + laws, and `iIndepFun.measureReal_sum_ge_le_exp_expFamilyDiv` as a case of the Cramér–Chernoff + bound with the rate function; Identification 1.3: `klDiv_gaussianReal_gaussianReal`; +- Identification 2.1: the equality case of the chain rule at a stopping time; Identification 3.3: + `continuousOn_altCost_baiTask_restrict`; Identification 3.4: best arm identification with + distinct per-arm families; +- Identification 4.1: observations in any measurable space, from the real case of target 0.4; + Identification 4.2: `supermartingale_exp_sum_mul_of_hasCondSubgaussianMGF` and + `unionBoundThreshold`; Identification 4.4: the almost-sure reduction; Identification 5.2: the + `δ`-PAC and limit theorems of Track-and-Stop for exponential families; +- Bandits 3.1–3.4: `klucb` and `ucbF` as instances of KL-UCB for exponential families; Bandits + 5.1: `gaussianTS` as an instance of Thompson sampling for exponential families; Bandits 8.3: + `ridgeEstimate` as the case `λ > 0` of least squares with the Moore–Penrose inverse; +- ReinforcementLearning 8.2 and 8.4: backward induction, and `EpisodicMDP.value_le_optValue` from + it. + +## Conventions + +- **A refactor keeps every result.** Each declaration it replaces is either derived from the new + general statement (when it is the form users call) or deleted with every use migrated to the + general statement. Deleted names leave no alias, wrapper or deprecated shim, as everywhere in + LMLExplore. +- **A target is met only when the narrow development is restated.** Adding a general version + beside the declarations a target names does not meet it: they must be derived from the general + version or deleted. A merged pull request that implements the narrow version of an item of + another roadmap does not cover the target here that restates it. +- **One target, one pull request**, attributed `Roadmap: Generalization` and citing the target + number, for example "Generalization 0.6". A target listed as "After" another one comes after it, + because it needs its statements or edits the same files. +- **Prerequisites in other roadmaps.** An item of another roadmap listed as a prerequisite is built + first, in its own pull request attributed to that roadmap. The refactor then adds no mathematics + beyond what its general statement needs. +- **The general forms are those of the other roadmaps,** with their conventions: `ℝ≥0∞`-valued + thresholds and costs in identification, rewards maximized in bandits with finitely many arms and + losses minimized in online learning, learning rates as nonincreasing sequences, operator + statements on `boundedMeasurable 𝓢`. +- **Names.** A generalized declaration keeps the name of the declaration it generalizes when the + name still describes it. LML's names are those of LMLExplore's pin (`Environment.bandit`, + `Environment.evalSeq`). + +## Layers and dependencies + +| Layer | Content | Prerequisites | Unblocks | +| --- | --- | --- | --- | +| 0 | Generalizations that new work builds on | merged LMLExplore code | Identification 1.1, 1.4, 2.1, 4.1–4.5, 5, 6, 7, 9.1; Bandits 6.3–6.6, 7, 8.2–8.5, 9, 10; ReinforcementLearning 6, 8–12, 23 | +| 1 | Consolidation of finished developments | Layer 0 targets as listed; Identification 0.5, 1.1, 3.6; ReinforcementLearning 0.6, 8.1 | Identification 3.6, 4.4, 5.2–5.3, 7, 8.1–8.2; Bandits 4.5, 6.4, 7.2–7.4, 8.4; ReinforcementLearning 1.4–1.5, 9.4, 10, 12.2 | +| 2 | Material Tau Ceti provides | merged LMLExplore code | none | + +## Layer 0: Generalizations that new work builds on + +### Identification and sequential learning + +1. **Best arm identification and ε-best arm identification with a mean functional.** + - In `Identification/Task/MeanVector.lean`, restate `baiTask ν` and `epsBaiTask ν ε` for a type + `Θ` of models, bandits `ν : Θ → Kernel α 𝓨` and a mean functional `mean : Θ → α → ℝ`, as + `baiTask ν mean` and `epsBaiTask ν mean ε` (Identification 1.1). The tasks on vectors of + means are the case `Θ = α → ℝ`, `mean = id`. Restate every lemma about them with `mean θ` in + place of the vector of means. + - Migrate every use: `Task/BestArm.lean`, `Task/TopTwo/BestArm.lean`, `Task/EpsBestArm.lean`, + `EpsBestArmLowerBound.lean`, `SuccessiveElimination.lean` and `MedianElimination.lean`. + - If `IdentTask.comap` (Identification 1.1) is merged, prove + `baiTask (fun θ ↦ ν' (mean θ)) mean = (baiTask ν' id).comap mean` and its analogue for + `epsBaiTask`; otherwise the pull request that adds `comap` proves them. + - Why: best arm identification with unknown variances is + `baiTask gaussianMeanVarBandit (fun θ a ↦ (θ a).1)`, the task whose costs + `Task/GaussianUnknownVariance.lean` computes; linear models and the bandits with real + observations are models of the same kind. + - Unblocks: Identification 1.1 (the model of all bandits with real observations), 4.5, 5.2 (the + two-parameter Gaussian model), 7, 9.1; targets 0.2, 1.1. +2. **The other classical tasks with a mean functional.** After 0.1. + - Restate in the same way `topMTask`, `epsTopMTask`, `thresholdingTask`, `minThresholdTask`, + `anyLowArmTask` and `halfSpaceTask` (`Task/MeanVector.lean`) and `sphereTask` + (`Task/Sphere.lean`), with every lemma about them, and the identities with `comap` under the + same condition as in target 0.1. + - Migrate every use: `Task/TopM.lean`, `TopMLowerBound.lean`, `LUCB.lean`, `KLLUCB.lean`, + `Task/Threshold.lean`, `Task/AnyLowArm.lean`, `Task/HalfSpace.lean` and `Task/Sphere.lean`. + - Rename `Task/MeanVector.lean` to `Task/Classical.lean`, and update its importers. + - Unblocks: Identification 3.6, 7.3, 9.1; targets 1.1, 1.6–1.9. +3. **Wald's identity for every environment.** + - In `SequentialLearning/Wald.lean`, prove Wald's identity for the feedback of + Identification 0.3: for a run of any algorithm in any environment with real feedback, a + stopping time of `IsAlgEnvSeq.filtrationAction` with finite expectation, and conditional + absolute means of the feedback bounded by a constant, + $\mathbb E[\sum_{t<\tau}Y_t]=\mathbb E[\sum_{t<\tau}m_t(A_t)]$ with LML's conditional means + `Environment.means`. The current proof replaces $Y_t$ by its conditional mean on + $\{t<\tau\}$ through LML's `IsAlgEnvSeq.condExp_feedback`, which holds for every environment; + only its integrability bounds are specific to bandits, and the bound on the conditional + absolute means replaces them. + - Derive `IsAlgEnvSeq.integral_sum_feedback_eq_sum_integral_pullCount_mul` (stationary bandits + with integrable arms) from it with LML's `means_bandit`, and restate its helpers + `IsAlgEnvSeq.setLIntegral_comp_feedback_bandit`, `setIntegral_feedback_bandit` and + `integrable_feedback_bandit` for every environment. + `IdentAlg.integral_sum_feedback_eq_sum_integral_pullCount_mul` + (`Identification/SampleComplexity.lean`) keeps its statement. + - Unblocks: ReinforcementLearning 6.2 (Wald's identity by keys), 14.6. +4. **The event transfer for every event about the observations of each arm.** + - In `SequentialLearning/SumRewards.lean`, define the observations of arm `a` before round `t`, + `armObs A Y a t ω : Fin (pullCount A a t ω) → ℝ` in the order of the pulls, related to LML's + `rewardByCount`. Prove the event transfer of Identification 4.1 for real observations: for + every predicate on the pull counts `s : α → ℕ` and the observations + `(a : α) → Fin (s a) → ℝ`, its joint form, its form up to a horizon, its single-arm form and + its counting forms, for predicates that need not be measurable (outer measures). The proof is + the current one: the pathwise inclusion on the product of the run's law with + `streamMeasure ν`, and `Measure.le_map_apply`. + - Derive with their current statements the transfers for the sums of rewards: + `measure_exists_pullCount_sumRewards_pi_le`, `measure_exists_lt_pullCount_sumRewards_pi_le`, + `measure_exists_pullCount_sumRewards_le`, `measure_exists_lt_pullCount_sumRewards_le`, + `sum_setLIntegral_action_eq_pullCount_sumRewards_le`, + `sum_measure_action_eq_and_pullCount_sumRewards_le`, + `sum_measureReal_action_eq_and_pullCount_sumRewards_le` and + `measureReal_pullCount_sumRewards_le_sum`. Their users do not change. + - Observations in any measurable space `𝓨` are Identification 4.1, which then generalizes the + statements of this target: LML's `hasLaw_rewardByCount_infinitePi`, on which the transfer + rests, is stated for real rewards. + - Unblocks: Identification 4.1, 4.5 (events about the sums of $Y$ and $Y^2$); Bandits 1.3; + ReinforcementLearning 6.1. +5. **The chain rule at a stopping time for any two algorithm–environment pairs.** + - In `SequentialLearning/DivergenceDecomposition.lean`, prove the inequality of + Identification 2.1 for any two pairs `(alg, env)` and `(alg', env')`, by the current proof: + both pairs are changed after the stopping time (`stoppedHistMeasure_congr`). This needs the + algorithm that follows one algorithm before the stopping rule and another after it: define + `Algorithm.piecewise` beside LMLExplore's `Environment.piecewise` + (`SequentialLearning/Algorithm.lean`), with its step-kernel lemma. The per-round divergences + with a restricted first marginal need no countably generated observation space (LML's + `klDiv_restrict_add_restrict_compl`); the integral form needs it (LML's + `klDiv_compProd_right_eq_lintegral`). + - Derive `klDiv_stoppedHistMeasure_le` (one algorithm against `Environment.bandit κ` and + `Environment.bandit κ'`) as its corollary, and delete the private `swapEnv`, + `klDiv_compProd_stepKernel_swapEnv` and `klDiv_trajMeasure_swapEnv`. + `Identification.klDiv_histOutputMeasure_le` (`Identification/ChangeOfMeasure.lean`) and its + users keep their statements. + - Unblocks: Identification 2.1 (the equality case); ReinforcementLearning 6.2, 14.6, 19. +6. **GLR stopping rules with thresholds that read the history.** + - In `Identification/GLR.lean`, the thresholds become `β : (n : ℕ) → Hist Unit α 𝓨 n → ℝ≥0∞`, + depending on the round and on the history, indexed by the confidence level in the asymptotic + section (Identification Conventions and 4.3). Restate `IdentTask.glrt` and its lemmas + (`glrt_def`, `alg_glrt`, `mem_stopSet_glrt`, `output_glrt`, `stoppingTime_glrt_le`, + `measureReal_histOutputMeasure_glrt_le`, + `measureReal_stoppingTime_ne_top_and_notMem_le_of_isRun_glrt`), `isDeltaPAC_glrt`, whose + deviation event becomes $\exists n,\ \beta_n(H_n)0$ and $F\ge0$, and the strongly convex + anytime bound [Ora, Corollary 7.6]. + - Derive `ftrl_regret_le` and `ftrl_regret_le_of_strongConvexOn` as their constant cases, with + their current statements. + - Unblocks: Bandits 6.3 (local norms), 6.4, 10.2; target 1.17. +12. **Design operators with a regularizer.** + - In `Analysis/InnerProductSpace/DesignOperator.lean`, define + `designOp (V₀ : E →L[ℝ] E) x t` $=V_0+\sum_{s 0` a decision `d` with `B x - ε • e ≤ T d x`. + - Prove: `B` satisfies Blackwell's conditions (restating `IsBlackwell.of_isGreatest_range`); + the fixed point of every `T d` is below the fixed point $x^*$ of `B`; a decision `ε`-greedy + for $x^*$ has its fixed point above $x^*-\frac{\varepsilon}{1-\gamma}e$; $x^*$ is the least + upper bound (`IsLUB`) of the fixed points of the `T d`, restating + `isGreatest_iUnion_fixedPoints`, which becomes the case where a decision is greedy for + $x^*$; with finitely many decisions, every `x` has a greedy decision. + - Unblocks: ReinforcementLearning 9.3–9.4, 10; targets 0.14, 1.23. +14. **Policy iteration and modified policy iteration with the greedy decisions they use.** After + 0.13. + - Restate under the hypotheses of target 0.13 the lemmas of + `Analysis/FixedPoint/DynamicProgramming.lean` that assume attainment: `le_map_of_isFixedPt`, + `IsPolicyIteration.le_map`, `map_le`, `monotone`, `iterate_le`, `le_of_isFixedPt`, + `norm_sub_le` and `exists_le_card_eq`, and `IsModifiedPolicyIteration.le_map`, `map_le`, + `iterate_le`, `le_of_isFixedPt` and `norm_sub_le`. The structures `IsPolicyIteration` and + `IsModifiedPolicyIteration` already record the greedy decisions their runs use; no other + greedy decision is assumed. + - Migrate `MarkovDecisionProcess/PolicyIteration.lean` (`exists_le_card_value_eq_optValue`) and + `MarkovDecisionProcess/ModifiedPolicyIteration.lean`. The lemmas that provide the attainment + hypothesis, `MDP.isGreatest_bellmanEval` (`Bellman.lean`) and `MDP.isGreatest_bellmanEvalOp` + (`ValueIteration/Basic.lean`), become lemmas that provide the two hypotheses of target 0.13: + the evaluation operators of deterministic policies are below the Bellman operator, and greedy + policies attain it. `Monotone.of_isGreatest_range` (`Order/Bounds/Basic.lean`) then has no + user and is deleted. + - Unblocks: ReinforcementLearning 10.2–10.3; targets 0.15, 1.22. +15. **Bellman operators for every action set.** After 0.14. + - Restate for every action set the statements that use only that `M.bellman γ v` is a + supremum: `lookahead_le_bellman`, `bellman_le_iff`, `bellman_sub_add_le_bellman_sub`, + `const_le_bellman_const` and `bellmanEval_le_bellman` (`MarkovDecisionProcess/Bellman.lean`; + ReinforcementLearning 8.1), `value_le_bellman` (`Value.lean`), `optValue_le_of_bellman_le` + (`Comparison.lean`) and `optValue_le_bellman_iterate_add` (`ValueIteration/Bounds.lean`). + They assume `[Finite 𝓐]` and hold for every `v`; their general forms assume that the + lookahead is bounded above and integrable, which finitely many actions give. + - Replace `MDP.IsGreedy` by the predicate `M.IsApproxGreedy γ ε v d` of + ReinforcementLearning 8.1; greediness is the case `ε = 0`. Migrate its users in + `Bellman.lean` (`isGreedy_greedy`, `isGreedy_iff_bellmanEval_eq`), `PolicyIteration.lean` + (`KeepsGreedy`, `exists_isPolicyIteration`) and `ModifiedPolicyIteration.lean`. + - Unblocks: ReinforcementLearning 8.1–8.3, 9.3–9.4; targets 1.21, 1.23. +16. **Discounted returns and one definition of the state marginals.** + - In `MarkovDecisionProcess/RunValue.lean`, give `MDP.runReturn` the discount as an argument, + `M.runReturn γ N u alg μ` (ReinforcementLearning 7.4), and restate its lemmas + (`runReturn_def`, `abs_runReturn_le` with the bound + $(\sum_{n 0` some decision is `ε`-greedy for `x`. It replaces the hypothesis +`∀ x, IsGreatest (Set.range fun d ↦ T d x) (B x)` of `Analysis/FixedPoint/DynamicProgramming.lean`, +which is the case where every `x` has a greedy decision. -/ +structure IsUniformSup (e : E) (T : D → E → E) (B : E → E) : Prop where + le (d : D) (x : E) : T d x ≤ B x + exists_sub_smul_le (x : E) (ε : ℝ) (hε : 0 < ε) : ∃ d, B x - ε • e ≤ T d x + +/-- The restated `IsBlackwell.of_isGreatest_range`: the uniform supremum of maps satisfying +Blackwell's conditions satisfies them. -/ +theorem IsBlackwell.of_isUniformSup (hT : ∀ d, IsBlackwell e γ (T d)) (hB : IsUniformSup e T B) : + IsBlackwell e γ B := + sorry + +/-- The restated `isGreatest_iUnion_fixedPoints`: the fixed point of `B` is the least upper bound +of the fixed points of the maps `T d`, which need not be attained. -/ +theorem isLUB_iUnion_fixedPoints [CompleteSpace E] [ClosedIicTopology E] [ClosedIciTopology E] + (hnorm : ∀ x, orderUnitNorm e x = ‖x‖) (hT : ∀ d, IsBlackwell e γ (T d)) (hγ : γ < 1) + (hB : IsUniformSup e T B) {x : E} (hx : IsFixedPt B x) : + IsLUB (⋃ d, fixedPoints (T d)) x := + sorry + +/-- LMLExplore's runs of policy iteration (`IsPolicyIteration`), unchanged: the field `map_eq` +records the greedy decisions that the run uses. -/ +structure IsPolicyIteration (T : D → E → E) (B : E → E) (d : ℕ → D) (v : ℕ → E) : Prop where + isFixedPt (k : ℕ) : IsFixedPt (T (d k)) (v k) + map_eq (k : ℕ) : T (d (k + 1)) (v k) = B (v k) + +/-- The restated `IsPolicyIteration.norm_sub_le`: policy iteration converges geometrically, +assuming the greedy decisions it uses and no others. -/ +theorem IsPolicyIteration.norm_sub_le [ClosedIicTopology E] [ClosedIciTopology E] + (hnorm : ∀ x, orderUnitNorm e x = ‖x‖) (hT : ∀ d, IsBlackwell e γ (T d)) (hγ : γ < 1) + (hB : IsUniformSup e T B) {d : ℕ → D} {v : ℕ → E} (h : IsPolicyIteration T B d v) {x : E} + (hx : IsFixedPt B x) (k : ℕ) : ‖v k - x‖ ≤ γ ^ k * ‖v 0 - x‖ := + sorry + +end DynamicProgramming + +/-! ## Exp4-IX as the single algorithm -/ + +section Exp4IX + +variable {α : Type*} [Fintype α] [DecidableEq α] [MeasurableSpace α] [MeasurableSingletonClass α] + +/-- LMLExplore's probability vectors on `α` (`simplex`). -/ +def simplex (α : Type*) [Fintype α] : Set (α → ℝ) := {w | (∀ a, 0 ≤ w a) ∧ ∑ a, w a = 1} + +theorem single_mem_simplex (b : α) : Pi.single b 1 ∈ simplex α := + sorry + +/-- Exp4-IX [Neu15] (Bandits 7.2), with the advice map `adv` from the observation of a round to +the advice of the experts `β`, learning rates `η` and implicit-exploration parameters `γ`. -/ +def exp4IX {𝓞 β : Type*} [MeasurableSpace 𝓞] [Fintype β] (adv : 𝓞 → β → simplex α) + (hadv : Measurable adv) (η γ : ℕ → ℝ) : Algorithm 𝓞 α ℝ := + sorry + +/-- The restated `exp4`: Exp4-IX without implicit exploration, the advice being the observation. -/ +def exp4 {β : Type*} [Fintype β] (η : ℕ → ℝ) : Algorithm (β → simplex α) α ℝ := + exp4IX id measurable_id η 0 + +/-- The restated `exp3IX`: Exp4-IX with the experts `α`, expert `b` advising the arm `b`. -/ +def exp3IX {𝓞 : Type*} [MeasurableSpace 𝓞] (η γ : ℕ → ℝ) : Algorithm 𝓞 α ℝ := + exp4IX (fun _ b ↦ ⟨Pi.single b 1, single_mem_simplex b⟩) measurable_const η γ + +/-- The restated `exp3`: Exp3-IX without implicit exploration. -/ +def exp3 {𝓞 : Type*} [MeasurableSpace 𝓞] (η : ℕ → ℝ) : Algorithm 𝓞 α ℝ := + exp3IX η 0 + +end Exp4IX + +end LMLRoadmaps.Generalization diff --git a/LMLRoadmaps/Generalization/metadata.toml b/LMLRoadmaps/Generalization/metadata.toml new file mode 100644 index 0000000..151aa3a --- /dev/null +++ b/LMLRoadmaps/Generalization/metadata.toml @@ -0,0 +1 @@ +topic = "math.ST"