Repository navigation
feat: Exp3 for adversarial linear bandits with the Kiefer–Wolfowitz exploration, for features that need not span the space - #459
Conversation
…xploration, for features that need not span the space Co-Authored-By: Claude Code <noreply@github.com>
There was a problem hiding this comment.
🟡 reuse — request_changes claude/claude-opus-5-5
The Moore–Penrose bridge inner_ringInverse_sum_smul_rankOne_spanFeatures and its helpers span_le_range_sum_smul_rankOne and coe_sum_smul_rankOne_spanFeatures_apply re-derive the existing inner_moorePenroseInverse_designOperator_map_subtype and designOperator_map_subtype_apply in OptimalDesign.lean with the same proof steps. The rest of the file correctly reuses the existing Kiefer–Wolfowitz and Exp3 results.
LMLExplore/Bandits/LinearBandit/Exp3/KieferWolfowitz.lean:152—inner_ringInverse_sum_smul_rankOne_spanFeaturesproves the same fact asinner_moorePenroseInverse_designOperator_map_subtype(OptimalDesign.lean:611): ⟪a, V⁺ c⟫ on E equals ⟪a, V_K⁻¹ c⟫ on a subspace K where V_K is invertible. The proof follows the same steps: write c as V (V_K⁻¹ c), show K lies in the range of V, then applyIsSymmetric.moorePenroseInverse_apply_apply_of_mem_range. The new lemma differs only in allowing two different vectors and arbitrary weightsp. Fix: Generaliseinner_moorePenroseInverse_designOperator_map_subtypein OptimalDesign.lean to two vectorsa c : K, at operator level (a symmetric T on E that maps K into K, with an invertible restriction T_K, gives ⟪a, T⁺ c⟫ = ⟪a, T_K⁻¹ c⟫). Derive both the existing lemma and the Exp3 statement from it.LMLExplore/Bandits/LinearBandit/Exp3/KieferWolfowitz.lean:129—coe_sum_smul_rankOne_spanFeatures_applyrestatesdesignOperator_map_subtype_apply(OptimalDesign.lean:325) for the sum form, andspan_le_range_sum_smul_rankOne(line 137) restates thehmemstep insideinner_moorePenroseInverse_designOperator_map_subtype. They are the same duplicate as the previous finding. Fix: Put the generalised subspace-restriction lemma, with its range-membership step, in OptimalDesign.lean and use it here instead of these two public helpers. Remove them if nothing else uses them.
Reply in this thread to contest a finding; that re-runs only this rubric and posts an answer here. (To fix it, just push a commit — that re-reviews on its own. To contest again after an answer, post a NEW reply rather than editing an old one.)
claude/claude-opus-5-5 · 50s · 182.7k in (140.1k cache read, 42.6k cache write) / 4.4k out tokens · reviewing this diff · rubric ·
There was a problem hiding this comment.
🟡 naming — request_changes claude/claude-opus-5-5
Two new Moore–Penrose restriction theorems take hsym : LinearMap.IsSymmetric as their first explicit argument but are placed in ContinuousLinearMap, which breaks the IsSymmetric. dot-notation precedent in the same file. All other names and namespaces are fine.
LMLExplore/Analysis/InnerProductSpace/MoorePenrose.lean:179—ContinuousLinearMap.moorePenroseInverse_apply_coe_of_apply_coe_eqtakeshsym : (T : E →ₗ[𝕜] E).IsSymmetricas its first explicit argument but is not in theLinearMap.IsSymmetricnamespace, sohsym.…dot notation does not work. Fix: Rename it toLinearMap.IsSymmetric.moorePenroseInverse_apply_coe_of_apply_coe_eq, next to the existingIsSymmetriclemmas, and update its callers.LMLExplore/Analysis/InnerProductSpace/MoorePenrose.lean:190— Same problem forContinuousLinearMap.inner_moorePenroseInverse_of_apply_coe_eq: its first explicit argument ishsym : LinearMap.IsSymmetric. Fix: Rename it toLinearMap.IsSymmetric.inner_moorePenroseInverse_of_apply_coe_eqand update its uses inOptimalDesign.lean(inner_moorePenroseInverse_designOperator_map_subtype,inner_ringInverse_sum_smul_rankOne_spanFeatures) tohsym.inner_moorePenroseInverse_of_apply_coe_eq ….
Reply in this thread to contest a finding; that re-runs only this rubric and posts an answer here. (To fix it, just push a commit — that re-reviews on its own. To contest again after an answer, post a NEW reply rather than editing an old one.)
claude/claude-opus-5-5 · 28s · 83.5k in (66.5k cache read, 17.1k cache write) / 2.9k out tokens · reviewing this diff · rubric ·
There was a problem hiding this comment.
Fixed in 1efdf60: both theorems now live in namespace LinearMap.IsSymmetric, right after the ContinuousLinearMap section of MoorePenrose.lean. They are LinearMap.IsSymmetric.moorePenroseInverse_apply_coe_of_apply_coe_eq and LinearMap.IsSymmetric.inner_moorePenroseInverse_of_apply_coe_eq, and their statements are unchanged. The callers use dot notation on hsym: the inner-product form calls hsym.moorePenroseInverse_apply_coe_of_apply_coe_eq, and OptimalDesign.lean calls (isPositive_designOperator _).toLinearMap.isSymmetric.inner_moorePenroseInverse_of_apply_coe_eq … and (isSymmetric_sum_smul_rankOne _ φ p).inner_moorePenroseInverse_of_apply_coe_eq …. apply_coe_ringInverse_of_apply_coe_eq and le_range_of_apply_coe_eq stay in ContinuousLinearMap, because they take no symmetry hypothesis. I updated the module docstring and the PR description. Local checks pass: whole-library lake build --iofail (9081 jobs), and local-axioms: checked 3 changed module(s), 135 declaration(s); 0 violation(s).
There was a problem hiding this comment.
🟡 proof-quality — request_changes claude/claude-opus-5-5
The proofs are short and readable, but one inline symmetry proof re-derives a structural fact by unfolding inner products instead of using existing Mathlib lemmas, and one rewrite chain is duplicated across two proofs instead of being factored into a lemma.
LMLExplore/Bandits/LinearBandit/Exp3/KieferWolfowitz.lean:158—hsymproves symmetry of∑ b, p b • rankOne ℝ (φ b) (φ b)by unfolding into a longsimp only [coe_coe, sum_apply, smul_apply, rankOne_apply, sum_inner, inner_sum, real_inner_smul_left, real_inner_smul_right]plussum_congr/real_inner_comm/ring. This chain depends on many lemma names, and it hides a reusable fact: the design operator is symmetric for arbitrary weights. Fix: Factor out a lemmaisSymmetric_sum_smul_rankOne(next toisPositive_sum_smul_rankOnein OptimalDesign.lean). Prove it structurally withcoe_sum+LinearMap.isSymmetric_sum+IsSymmetric.smul+isSymmetric_rankOne_self, and use it here.LMLExplore/Bandits/LinearBandit/Exp3/KieferWolfowitz.lean:167— The rewrite chainrw [coe_coe, ← coe_sum_smul_rankOne_spanFeatures_apply, apply_ringInverse_apply hu, …]appears twice: inspan_le_range_sum_smul_rankOne(lines 143–144) and inhc(lines 167–168). Both prove the same fact, thatV(p) ↑(V_W(p)⁻¹ w) = ↑w. Fix: Add a lemmasum_smul_rankOne_apply_coe_ringInverse (hu) (w) : V(p) ↑(Ring.inverse V_W(p) w) = ↑w. Use it in both places:span_le_range_sum_smul_rankOnebecomesfun v hv ↦ ⟨_, lemma hu ⟨v, hv⟩⟩, andhcbecomes(lemma hu _).symm.
Reply in this thread to contest a finding; that re-runs only this rubric and posts an answer here. (To fix it, just push a commit — that re-reviews on its own. To contest again after an answer, post a NEW reply rather than editing an old one.)
claude/claude-opus-5-5 · 42s · 143.3k in (129.2k cache read, 14.1k cache write) / 3.3k out tokens · reviewing this diff · rubric ·
AI review — changes requestedEach rubric is judged independently by multiple review agents; the PR merges only once every rubric is green — any rubric that is not green (changes requested, blocked, errored, stale, or not yet run) blocks the merge. See the rubrics.
♻️ = approved on an earlier commit, re-run before merge. Reviewing this diff at head |
…Design Move `spanFeatures` and its lemmas, and the Kiefer–Wolfowitz design on the span of the features (now `kieferWolfowitzDesign`, with the existential `exists_mem_simplex_inner_ringInverse_spanFeatures_le_finrank`), to `Analysis/InnerProductSpace/OptimalDesign.lean`. Factor the Moore–Penrose restriction argument into the operator-level `inner_moorePenroseInverse_of_apply_coe_eq` (with `apply_coe_ringInverse_of_apply_coe_eq` and `le_range_of_apply_coe_eq`), derive `inner_moorePenroseInverse_designOperator_map_subtype` and `inner_ringInverse_sum_smul_rankOne_spanFeatures` from it, and add `isSymmetric_sum_smul_rankOne`. Rename the regret theorems after their bound. Co-Authored-By: Claude Code <noreply@github.com>
`apply_coe_ringInverse_of_apply_coe_eq`, `le_range_of_apply_coe_eq` and `inner_moorePenroseInverse_of_apply_coe_eq` are generic facts about an operator acting on a subspace and its Moore–Penrose inverse, with no design content. Move them from OptimalDesign.lean to MoorePenrose.lean, in the `ContinuousLinearMap` namespace next to `moorePenroseInverse_eq_ringInverse`, over a general `RCLike` field. Co-Authored-By: Claude Code <noreply@github.com>
Add ContinuousLinearMap.moorePenroseInverse_apply_coe_of_apply_coe_eq, T⁺ c = T_K⁻¹ c for c ∈ K, and derive the inner-product form from it. Co-Authored-By: Claude Code <noreply@github.com>
…rMap.IsSymmetric Co-Authored-By: Claude Code <noreply@github.com>
This PR completes Bandits Layer 10.1, Exp3 with exploration: "prove [LS, Theorem 27.1] … hence$R_n\le2\sqrt{3dn\log k}$ with the Kiefer–Wolfowitz design of Layer 8.6". It also covers the case of the completion check "The Kiefer–Wolfowitz theorem applies to the finite action sets of Layers 9.3 and 10.1, including sets that do not span $E$ ".
On
main,LinearExp3.integral_regret_le_two_mul_sqrt_threetakes the Kiefer–Wolfowitz properties of the exploration as hypotheses:hπu : IsUnit V(π)andhg : ⟪φ b, V(π)⁻¹ φ b⟫ ≤ d. No caller supplies them, and they cannot hold when the features do not spanE. This PR supplies the design and removes both hypotheses. The features may be arbitrary, spanningEor not. After this PR, nothing of Layer 10.1 remains. The rest of Layer 10 (item 10.2) is already onmain.The construction runs Exp3 for linear bandits on the span
Wof the features, so it never needsV(π)to be invertible onE. The learner only observes⟪φ c, y⟫, which equals⟪φ c, P_W y⟫, so a run against an adaptive adversary inEis a run against the projected adversary inW. The bound then holds withm = dim W ≤ d.Generic Moore–Penrose material, in
LMLExplore/Analysis/InnerProductSpace/MoorePenrose.lean(namespaceContinuousLinearMap, andLinearMap.IsSymmetricfor the two lemmas with a symmetry hypothesis, over anyRCLikefield): operator-level restriction. ForT : E →L[𝕜] Eacting on a subspaceKas an invertibleT_K : K →L[𝕜] K(hT : ∀ v : K, T v = T_K v):apply_coe_ringInverse_of_apply_coe_eq:T ↑(T_K⁻¹ v) = v;le_range_of_apply_coe_eq:K ≤ range T;LinearMap.IsSymmetric.moorePenroseInverse_apply_coe_of_apply_coe_eq(dot notation onhsym): for symmetricTon finite-dimensionalE,T⁺ c = T_K⁻¹ cforc : K;LinearMap.IsSymmetric.inner_moorePenroseInverse_of_apply_coe_eq: its inner-product form,⟪a, T⁺ c⟫ = ⟪a, T_K⁻¹ c⟫fora c : K.The existing
inner_moorePenroseInverse_designOperator_map_subtypeinOptimalDesign.leanis now a one-line corollary of the last one (statement unchanged).Design material, in
LMLExplore/Analysis/InnerProductSpace/OptimalDesign.lean(namespaceLMLExplore):isSymmetric_sum_smul_rankOne:∑ i ∈ s, w i • φ i ⊗ φ iis symmetric for arbitrary real weights (structural proof from Mathlib'sLinearMap.isSymmetric_sum,IsSymmetric.smul,isSymmetric_rankOne_self).spanFeatures φ: the features, as elements ofSubmodule.span ℝ (Set.range φ), withcoe_spanFeatures(simp),span_range_spanFeatures(via Mathlib'sSubmodule.span_span_coe_preimage) andinner_spanFeatures_orthogonalProjectionOnto.exists_mem_simplex_inner_ringInverse_spanFeatures_le_finrank: Kiefer–Wolfowitz for finitely many arms on the span of features that need not spanE[LS, Exercise 21.6], from the existingexists_mem_simplex_inner_ringInverse_le_finrankapplied on the span.kieferWolfowitzDesign φ, its choice, withkieferWolfowitzDesign_mem_simplex,isUnit_sum_smul_rankOne_kieferWolfowitzDesignandinner_ringInverse_sum_smul_rankOne_kieferWolfowitzDesign_le_finrank(G-value at mostm).inner_ringInverse_sum_smul_rankOne_spanFeatures: whenV_W(p)is invertible,⟪φ a, V_W(p)⁻¹ φ c⟫ = ⟪φ a, V(p)⁺ φ c⟫, with the Moore–Penrose inverse onE(a corollary ofLinearMap.IsSymmetric.inner_moorePenroseInverse_of_apply_coe_eq). So the algorithm on the span is Algorithm 15 withV⁺in place ofV⁻¹.Bandit material, new file
LMLExplore/Bandits/LinearBandit/Exp3/KieferWolfowitz.lean(namespaceLMLExplore.Bandits):linearExp3Estimate_spanFeatures: for spanning features, the loss estimates on the span are those onE.linearExp3KieferWolfowitz φ η γ hγ, which islinearExp3 (spanFeatures φ) (kieferWolfowitzDesign φ) η γ _ hγ, withlinearExp3KieferWolfowitz_def.linearExp3KieferWolfowitz_eq_linearExp3proves that, for spanning features andγ > 0, it equalslinearExp3 φ (kieferWolfowitzDesign φ) η γ _ hγonE.LinearExp3.integral_regret_linearExp3KieferWolfowitz_le_two_mul_sqrt_three: against an adaptive adversary, withη = √(log k / (3 m n))andγ = η m, the expected regret against every arm is at most2 √(3 m n log k). There is no hypothesis on the features or the design. When all features vanish (m = 0), the regret is0.LinearExp3.integral_regret_linearExp3KieferWolfowitz_le_two_mul_sqrt_three_finrank: the same regret is at most2 √(3 d n log k), which is the roadmap's statement.LinearExp3.integral_regret_detFeedback_linearExp3KieferWolfowitz_le_two_mul_sqrt_three: the first bound for every run againstEnvironment.detFeedback obs (fun t x ↦ ⟪φ x.2, y t x.1⟫) _.No public declaration of
mainis removed or renamed.Reuse: I grepped Mathlib, Tau Ceti, LML, Statlib and
LMLExplore/for a restriction of a family to its span (codRestrict,rangeFactorization, maps intoSubmodule.span ℝ (Set.range _)) and found none.DesignSelection.kieferWolfowitzis the phased-elimination design selection on finsets with the Moore–Penrose inverse inE, a different interface. The proofs consume existing results:exists_mem_simplex_inner_ringInverse_le_finrank.integral_regret_le_two_mul_sqrt_three,isUnit_sum_smul_rankOne_linearExp3Weights.IsSymmetric.moorePenroseInverse_apply_apply_of_mem_range,moorePenroseInverse_eq_ringInverse.Submodule.inner_orthogonalProjectionOnto_eq_of_mem_left,LinearMap.isUnit_iff_range_eq_top,LinearMap.isSymmetric_sum.The source is cited in the files: Lattimore and Szepesvári, Bandit Algorithms, Theorem 27.1, Theorem 21.1 and Exercise 21.6.
Hypotheses:
[Nonempty α]is needed to choose the design.0 < γinlinearExp3KieferWolfowitz_eq_linearExp3keeps the exploration weights invertible at every round. The tuned rates haveγ > 0.integral_regret_le_two_mul_sqrt_three.Local checks only (CI verifies the pushed PR):
scope: scoped: against 20105e47ae0239a41ff62aef39ff39df15cda49clocal-axioms: checked 3 changed module(s), 135 declaration(s); 0 violation(s)lake build --iofail(9081 jobs), the module-system audit, the header check and the environment lint (135 declarations, 14 environment linters) passed, and the verify script exited 0.Roadmap: Bandits
🤖 Generated with Claude Code