diff --git a/LMLExplore/Analysis/InnerProductSpace/MoorePenrose.lean b/LMLExplore/Analysis/InnerProductSpace/MoorePenrose.lean index da3cf271..7c0b2ca3 100644 --- a/LMLExplore/Analysis/InnerProductSpace/MoorePenrose.lean +++ b/LMLExplore/Analysis/InnerProductSpace/MoorePenrose.lean @@ -18,6 +18,8 @@ singular, as for the Gram operator of unregularized least squares or of a design * for every Moore–Penrose pair `(A, B)`, the composition `A B` is the orthogonal projection onto the range of `A`; * the Moore–Penrose inverse of an invertible operator is its inverse; +* if an operator acts on a subspace as an invertible operator of that subspace, its Moore–Penrose + inverse acts there as the inverse of that restriction; * the Moore–Penrose inverse of a symmetric operator `V` is symmetric, has the same range as `V`, and inverts `V` on that range; that of a positive operator is positive. @@ -27,6 +29,11 @@ singular, as for the Gram operator of unregularized least squares or of a design inverse of `A`, then `A B` is the orthogonal projection onto the range of `A`. * `ContinuousLinearMap.moorePenroseInverse_eq_ringInverse`: `V⁺ = V⁻¹` for an invertible operator `V`. +* `LinearMap.IsSymmetric.moorePenroseInverse_apply_coe_of_apply_coe_eq`: if a symmetric operator + `T` on `E` acts on a subspace `K` as an invertible operator `T_K` on `K`, then `T⁺ c = T_K⁻¹ c` + for `c ∈ K`, and `LinearMap.IsSymmetric.inner_moorePenroseInverse_of_apply_coe_eq`: + `⟪a, T⁺ c⟫ = ⟪a, T_K⁻¹ c⟫` for `a c ∈ K`; `ContinuousLinearMap.le_range_of_apply_coe_eq`: then + `K` lies in the range of `T`. * `LinearMap.IsSymmetric.moorePenroseInverse`, `LinearMap.IsPositive.moorePenroseInverse`: the Moore–Penrose inverse of a symmetric (positive) operator is symmetric (positive). * `LinearMap.IsSymmetric.range_moorePenroseInverse`: `V⁺` and a symmetric `V` have the same @@ -138,7 +145,26 @@ end LinearMap namespace ContinuousLinearMap variable {𝕜 E : Type*} [RCLike 𝕜] [NormedAddCommGroup E] [InnerProductSpace 𝕜 E] - [FiniteDimensional 𝕜 E] + +section Restrict + +variable {K : Submodule 𝕜 E} {T : E →L[𝕜] E} {T_K : K →L[𝕜] K} + +/-- If an operator `T` on `E` acts on a subspace `K` as an invertible operator `T_K` on `K`, then +`T` maps `T_K⁻¹ v` to `v` for `v ∈ K`. -/ +theorem apply_coe_ringInverse_of_apply_coe_eq (hT : ∀ v : K, T v = T_K v) (hu : IsUnit T_K) + (v : K) : T ↑(Ring.inverse T_K v) = v := by + rw [hT, apply_ringInverse_apply hu] + +/-- If an operator `T` on `E` acts on a subspace `K` as an invertible operator `T_K` on `K`, then +`K` lies in the range of `T`. -/ +theorem le_range_of_apply_coe_eq (hT : ∀ v : K, T v = T_K v) (hu : IsUnit T_K) : + K ≤ LinearMap.range (T : E →ₗ[𝕜] E) := fun v hv ↦ + ⟨_, apply_coe_ringInverse_of_apply_coe_eq hT hu ⟨v, hv⟩⟩ + +end Restrict + +variable [FiniteDimensional 𝕜 E] /-- The Moore–Penrose inverse of an invertible operator is its inverse: `V⁺ = V⁻¹`. -/ theorem moorePenroseInverse_eq_ringInverse {V : E →L[𝕜] E} (hV : IsUnit V) : @@ -148,3 +174,31 @@ theorem moorePenroseInverse_eq_ringInverse {V : E →L[𝕜] E} (hV : IsUnit V) (LinearMap.ext (apply_ringInverse_apply hV)))).symm end ContinuousLinearMap + +namespace LinearMap.IsSymmetric + +variable {𝕜 E : Type*} [RCLike 𝕜] [NormedAddCommGroup E] [InnerProductSpace 𝕜 E] + [FiniteDimensional 𝕜 E] + +/-- If a symmetric operator `T` on a finite-dimensional space `E` acts on a subspace `K` as an +invertible operator `T_K` on `K`, then the Moore–Penrose inverse of `T` acts on `K` as the inverse +of `T_K`: `T⁺ c = T_K⁻¹ c` for `c ∈ K`. -/ +theorem moorePenroseInverse_apply_coe_of_apply_coe_eq {K : Submodule 𝕜 E} {T : E →L[𝕜] E} + {T_K : K →L[𝕜] K} (hsym : (T : E →ₗ[𝕜] E).IsSymmetric) (hT : ∀ v : K, T v = T_K v) + (hu : IsUnit T_K) (c : K) : + (T : E →ₗ[𝕜] E).moorePenroseInverse c = Ring.inverse T_K c := by + have h := hsym.moorePenroseInverse_apply_apply_of_mem_range + (ContinuousLinearMap.le_range_of_apply_coe_eq hT hu (Ring.inverse T_K c).2) + rwa [ContinuousLinearMap.coe_coe, + ContinuousLinearMap.apply_coe_ringInverse_of_apply_coe_eq hT hu] at h + +/-- If a symmetric operator `T` on a finite-dimensional space `E` acts on a subspace `K` as an +invertible operator `T_K` on `K`, then `⟪a, T⁺ c⟫ = ⟪a, T_K⁻¹ c⟫` for `a c ∈ K`, the inner products +on the right being those of `K`. -/ +theorem inner_moorePenroseInverse_of_apply_coe_eq {K : Submodule 𝕜 E} {T : E →L[𝕜] E} + {T_K : K →L[𝕜] K} (hsym : (T : E →ₗ[𝕜] E).IsSymmetric) (hT : ∀ v : K, T v = T_K v) + (hu : IsUnit T_K) (a c : K) : + ⟪(a : E), (T : E →ₗ[𝕜] E).moorePenroseInverse c⟫_𝕜 = ⟪a, Ring.inverse T_K c⟫_𝕜 := by + rw [hsym.moorePenroseInverse_apply_coe_of_apply_coe_eq hT hu, Submodule.coe_inner] + +end LinearMap.IsSymmetric diff --git a/LMLExplore/Analysis/InnerProductSpace/OptimalDesign.lean b/LMLExplore/Analysis/InnerProductSpace/OptimalDesign.lean index 9f761eef..45e532e5 100644 --- a/LMLExplore/Analysis/InnerProductSpace/OptimalDesign.lean +++ b/LMLExplore/Analysis/InnerProductSpace/OptimalDesign.lean @@ -67,6 +67,10 @@ point `a` for each of them gives a design with the same design operator, hence t * `LMLExplore.designOperator w`: the design operator `V(w) = ∑_a w a • a ⊗ a` of `w`. * `LMLExplore.gValue 𝒜 w`: the G-value `g(w) = sup_{a ∈ 𝒜} ⟪a, V(w)⁺ a⟫` in `ℝ≥0∞`, where a point of `𝒜` outside the range of `V(w)` contributes `∞`. +* `LMLExplore.spanFeatures φ`: the features `φ i`, as elements of the span of the features. +* `LMLExplore.kieferWolfowitzDesign φ`: a Kiefer–Wolfowitz design on finitely many arms with + features `φ`, which need not span `E`: weights in the simplex whose design operator on the span + of the features is invertible, with G-value at most the dimension of the span. ## Main statements @@ -75,7 +79,8 @@ point `a` for each of them gives a design with the same design operator, hence t * `LMLExplore.isPositive_sum_smul_rankOne`, `LMLExplore.smul_sum_smul_rankOne_le`, `LMLExplore.isUnit_sum_smul_rankOne_of_mul_le`: for a feature map `φ`, the weighted sum `∑ i ∈ s, w i • φ i ⊗ φ i` is positive for nonnegative weights (on a finite type, monotone in - the weights in the Loewner order, and invertible above an invertible one). + the weights in the Loewner order, and invertible above an invertible one), and + `LMLExplore.isSymmetric_sum_smul_rankOne`: it is symmetric for arbitrary real weights. * `LMLExplore.inner_sum_smul_rankOne_apply_self`: its quadratic form is `⟪v, (∑ i ∈ s, w i • φ i ⊗ φ i) v⟫ = ∑ i ∈ s, w i ⟪φ i, v⟫²`, and `LMLExplore.sum_smul_rankOne_apply_of_orthonormal`: for an orthonormal family `φ`, it maps `φ i` @@ -114,8 +119,14 @@ point `a` for each of them gives a design with the same design operator, hence t * `LMLExplore.exists_mem_simplex_inner_ringInverse_le_finrank`: **Kiefer–Wolfowitz for finitely many arms**: if the features `φ i` of finitely many arms span `E`, some weights `w` in the simplex have `⟪φ i, V⁻¹ φ i⟫ ≤ d` for every arm, with `V = ∑ i, w i • φ i ⊗ φ i` invertible. +* `LMLExplore.exists_mem_simplex_inner_ringInverse_spanFeatures_le_finrank`: **Kiefer–Wolfowitz + for finitely many arms on the span of the features** [LS, Exercise 21.6]: the same on the span `W` + of features that need not span `E`, with the dimension `m` of `W` in place of `d`. * `LMLExplore.inner_moorePenroseInverse_designOperator_map_subtype`: for a design on a subspace - `K` with invertible design operator `V_K(w)` on `K`, `⟪a, V(w)⁺ a⟫ = ⟪a, V_K(w)⁻¹ a⟫` on `K`. + `K` with invertible design operator `V_K(w)` on `K`, `⟪a, V(w)⁺ a⟫ = ⟪a, V_K(w)⁻¹ a⟫` on `K`; + `LMLExplore.inner_ringInverse_sum_smul_rankOne_spanFeatures`: the same for weights `p` on arms + with features `φ`, whose operator `V_W(p) = ∑ i, p i • φ i ⊗ φ i` on the span `W` of the features + is invertible: `⟪φ a, V_W(p)⁻¹ φ c⟫ = ⟪φ a, V(p)⁺ φ c⟫`. * `LMLExplore.exists_designOperator_eq_card_support_le`: a design with invertible design operator and `⟪a, V(w)⁻¹ a⟫ ≤ d` on its support has the design operator of a design supported in its support with at most `d (d + 1) / 2` points [LS, Theorem 21.1]. @@ -246,6 +257,14 @@ theorem isPositive_sum_smul_rankOne {ι : Type*} (s : Finset ι) (φ : ι → E) (hw : ∀ i ∈ s, 0 ≤ w i) : (∑ i ∈ s, w i • rankOne ℝ (φ i) (φ i)).IsPositive := isPositive_sum _ fun i hi ↦ (isPositive_rankOne_self (φ i)).smul_of_nonneg (hw i hi) +/-- For arbitrary real weights `w`, `∑ i ∈ s, w i • φ i ⊗ φ i` is symmetric. -/ +theorem isSymmetric_sum_smul_rankOne {ι : Type*} (s : Finset ι) (φ : ι → E) (w : ι → ℝ) : + ((∑ i ∈ s, w i • rankOne ℝ (φ i) (φ i) : E →L[ℝ] E) : E →ₗ[ℝ] E).IsSymmetric := by + rw [toLinearMap_sum] + exact LinearMap.isSymmetric_sum _ fun i _ ↦ by + rw [toLinearMap_smul] + exact (isSymmetric_rankOne_self (φ i)).smul (by simp) + /-- The quadratic form of `∑ i ∈ s, w i • φ i ⊗ φ i`: `⟪v, (∑ i ∈ s, w i • φ i ⊗ φ i) v⟫ = ∑ i ∈ s, w i * ⟪φ i, v⟫²`. -/ theorem inner_sum_smul_rankOne_apply_self {ι : Type*} (s : Finset ι) (φ : ι → E) (w : ι → ℝ) @@ -320,6 +339,37 @@ theorem isUnit_sum_smul_rankOne_of_mul_le [FiniteDimensional ℝ E] {ι : Type*} have hvp := isPositive_sum_smul_rankOne Finset.univ φ fun i _ ↦ hv i exact (hvp.smul_of_nonneg hc.le).isUnit_of_le hcu (smul_sum_smul_rankOne_le h) +/-! ### Features in their span -/ + +section SpanFeatures + +variable {ι : Type*} (φ : ι → E) + +/-- The features `φ i`, as elements of the span `Submodule.span ℝ (Set.range φ)` of the features. +They span it (`span_range_spanFeatures`). -/ +def spanFeatures (i : ι) : Submodule.span ℝ (Set.range φ) := + ⟨φ i, Submodule.subset_span (Set.mem_range_self i)⟩ + +@[simp] +theorem coe_spanFeatures (i : ι) : (spanFeatures φ i : E) = φ i := (rfl) + +/-- The features span their span. -/ +theorem span_range_spanFeatures : Submodule.span ℝ (Set.range (spanFeatures φ)) = ⊤ := by + have : Set.range (spanFeatures φ) = (↑) ⁻¹' Set.range φ := by + ext x + simp [Subtype.ext_iff] + rw [this, Submodule.span_span_coe_preimage] + +/-- The inner product of a feature with the orthogonal projection of a vector onto the span of the +features is its inner product with the vector. -/ +theorem inner_spanFeatures_orthogonalProjectionOnto + [(Submodule.span ℝ (Set.range φ)).HasOrthogonalProjection] (i : ι) (v : E) : + ⟪spanFeatures φ i, (Submodule.span ℝ (Set.range φ)).orthogonalProjectionOnto v⟫_ℝ = + ⟪φ i, v⟫_ℝ := by + rw [Submodule.inner_orthogonalProjectionOnto_eq_of_mem_left, coe_spanFeatures] + +end SpanFeatures + /-- The design operator on `E` of a design on a subspace `K` restricts on `K` to the design operator on `K`: `V(w) v = V_K(w) v` for `v ∈ K`. -/ theorem designOperator_map_subtype_apply {K : Submodule ℝ E} (w : StdSimplex ℝ K) (v : K) : @@ -605,25 +655,73 @@ theorem exists_mem_simplex_inner_ringInverse_le_finrank {ι : Type*} [Fintype ι rw [designOperator_map] at hu hle exact ⟨v.weights, v.weights_mem_simplex, hu, fun i ↦ hle _ ⟨i, rfl⟩⟩ +/-- **The Kiefer–Wolfowitz theorem for finitely many arms on the span of the features** +[LS, Theorem 21.1, Exercise 21.6]. For features `φ i` of the elements of a nonempty finite type, +which need not span `E`, some weights `w` in the simplex have an invertible design operator +`V_W = ∑ i, w i • φ i ⊗ φ i` on the span `W` of the features, with `⟪φ i, V_W⁻¹ φ i⟫ ≤ m` for every +`i`, where `m` is the dimension of `W`. -/ +theorem exists_mem_simplex_inner_ringInverse_spanFeatures_le_finrank {ι : Type*} [Fintype ι] + [Nonempty ι] (φ : ι → E) : + ∃ w ∈ LMLExplore.simplex ι, + IsUnit (∑ i, w i • rankOne ℝ (spanFeatures φ i) (spanFeatures φ i)) ∧ + ∀ i, ⟪spanFeatures φ i, Ring.inverse (∑ i, w i • rankOne ℝ (spanFeatures φ i) + (spanFeatures φ i)) (spanFeatures φ i)⟫_ℝ ≤ finrank ℝ (Submodule.span ℝ (Set.range φ)) := + exists_mem_simplex_inner_ringInverse_le_finrank (span_range_spanFeatures φ) + +section KieferWolfowitzDesign + +variable {ι : Type*} [Fintype ι] [Nonempty ι] (φ : ι → E) + +/-- The **Kiefer–Wolfowitz design** of the features `φ` on finitely many arms: weights `w` in the +simplex whose design operator `V_W(w) = ∑ i, w i • φ i ⊗ φ i` on the span `W` of the features is +invertible, with `⟪φ i, V_W(w)⁻¹ φ i⟫ ≤ m` for every arm `i`, where `m` is the dimension of `W`. +They are chosen by `exists_mem_simplex_inner_ringInverse_spanFeatures_le_finrank`. -/ +noncomputable def kieferWolfowitzDesign : ι → ℝ := + (exists_mem_simplex_inner_ringInverse_spanFeatures_le_finrank φ).choose + +/-- The Kiefer–Wolfowitz design is a probability vector. -/ +theorem kieferWolfowitzDesign_mem_simplex : kieferWolfowitzDesign φ ∈ LMLExplore.simplex ι := + (exists_mem_simplex_inner_ringInverse_spanFeatures_le_finrank φ).choose_spec.1 + +/-- The design operator of the Kiefer–Wolfowitz design is invertible on the span of the +features. -/ +theorem isUnit_sum_smul_rankOne_kieferWolfowitzDesign : + IsUnit (∑ i, kieferWolfowitzDesign φ i • rankOne ℝ (spanFeatures φ i) (spanFeatures φ i)) := + (exists_mem_simplex_inner_ringInverse_spanFeatures_le_finrank φ).choose_spec.2.1 + +/-- The G-value of the Kiefer–Wolfowitz design is at most the dimension of the span of the +features. -/ +theorem inner_ringInverse_sum_smul_rankOne_kieferWolfowitzDesign_le_finrank (i : ι) : + ⟪spanFeatures φ i, Ring.inverse (∑ i, kieferWolfowitzDesign φ i • + rankOne ℝ (spanFeatures φ i) (spanFeatures φ i)) (spanFeatures φ i)⟫_ℝ ≤ + finrank ℝ (Submodule.span ℝ (Set.range φ)) := + (exists_mem_simplex_inner_ringInverse_spanFeatures_le_finrank φ).choose_spec.2.2 i + +end KieferWolfowitzDesign + /-- Let `w` be a design on a subspace `K` whose design operator `V_K(w)` on `K` is invertible. For `a ∈ K`, the weighted norm of `a` for the Moore–Penrose inverse of the design operator `V(w)` on `E` is its weighted norm for the inverse of `V_K(w)`: `⟪a, V(w)⁺ a⟫ = ⟪a, V_K(w)⁻¹ a⟫`. -/ theorem inner_moorePenroseInverse_designOperator_map_subtype {K : Submodule ℝ E} {w : StdSimplex ℝ K} (hu : IsUnit (designOperator w)) (a : K) : ⟪(a : E), (designOperator (w.map ((↑) : K → E)) : E →ₗ[ℝ] E).moorePenroseInverse a⟫_ℝ = - ⟪a, Ring.inverse (designOperator w) a⟫_ℝ := by - set V := designOperator (w.map ((↑) : K → E)) - -- `V` maps `V_K(w)⁻¹ v` to `v` for `v ∈ K`, so `K` lies in the range of `V` - have hV : ∀ v : K, V ↑(Ring.inverse (designOperator w) v) = v := fun v ↦ by - rw [designOperator_map_subtype_apply, apply_ringInverse_apply hu] - have hmem : ∀ v : K, (v : E) ∈ LinearMap.range (V : E →ₗ[ℝ] E) := fun v ↦ - ⟨_, by rw [coe_coe, hV]⟩ - calc ⟪(a : E), (V : E →ₗ[ℝ] E).moorePenroseInverse a⟫_ℝ - = ⟪(a : E), (V : E →ₗ[ℝ] E).moorePenroseInverse - ((V : E →ₗ[ℝ] E) ↑(Ring.inverse (designOperator w) a))⟫_ℝ := by rw [coe_coe, hV] - _ = ⟪a, Ring.inverse (designOperator w) a⟫_ℝ := by - rw [(isPositive_designOperator _).toLinearMap.isSymmetric - |>.moorePenroseInverse_apply_apply_of_mem_range (hmem _), Submodule.coe_inner] + ⟪a, Ring.inverse (designOperator w) a⟫_ℝ := + (isPositive_designOperator _).toLinearMap.isSymmetric.inner_moorePenroseInverse_of_apply_coe_eq + (designOperator_map_subtype_apply w) hu a a + +/-- **The inverse on the span is the Moore–Penrose inverse.** If the operator +`V_W(p) = ∑ i, p i • φ i ⊗ φ i` on the span `W` of the features is invertible, then the weighted +inner products of the features for its inverse are those for the Moore–Penrose inverse `V(p)⁺` of +the operator `V(p) = ∑ i, p i • φ i ⊗ φ i` on `E`: `⟪φ a, V_W(p)⁻¹ φ c⟫ = ⟪φ a, V(p)⁺ φ c⟫`. -/ +theorem inner_ringInverse_sum_smul_rankOne_spanFeatures {ι : Type*} [Fintype ι] {φ : ι → E} + {p : ι → ℝ} (hu : IsUnit (∑ i, p i • rankOne ℝ (spanFeatures φ i) (spanFeatures φ i))) + (a c : ι) : + ⟪spanFeatures φ a, Ring.inverse (∑ i, p i • rankOne ℝ (spanFeatures φ i) (spanFeatures φ i)) + (spanFeatures φ c)⟫_ℝ = + ⟪φ a, ((∑ i, p i • rankOne ℝ (φ i) (φ i) : E →L[ℝ] E) : E →ₗ[ℝ] E).moorePenroseInverse + (φ c)⟫_ℝ := + ((isSymmetric_sum_smul_rankOne _ φ p).inner_moorePenroseInverse_of_apply_coe_eq + (fun v ↦ by simp) hu _ _).symm /-- **Support reduction of G-optimal designs** [LS, Theorem 21.1]. Let `w` be a design with invertible design operator `V(w)` on a nontrivial real inner product space of dimension `d`, with diff --git a/LMLExplore/Bandits/LinearBandit/Exp3/KieferWolfowitz.lean b/LMLExplore/Bandits/LinearBandit/Exp3/KieferWolfowitz.lean new file mode 100644 index 00000000..b6eaa87c --- /dev/null +++ b/LMLExplore/Bandits/LinearBandit/Exp3/KieferWolfowitz.lean @@ -0,0 +1,232 @@ +/- +Copyright (c) 2026 LMLExplore contributors. All rights reserved. +Released under Apache 2.0 license as described in the file LICENSE. +Authors: LMLExplore contributors +-/ +module + +public import LMLExplore.Analysis.InnerProductSpace.OptimalDesign +public import LMLExplore.Bandits.LinearBandit.Exp3.Algorithm + +/-! +# Exp3 for adversarial linear bandits with the Kiefer–Wolfowitz exploration + +The regret bound `2 √(3 d n log k)` of Exp3 for linear bandits [LS, Theorem 27.1] needs an +exploration distribution `π` whose design operator `V(π) = ∑ b, π b • φ b ⊗ φ b` is invertible +and whose G-value `max_b ⟪φ b, V(π)⁻¹ φ b⟫` is at most the dimension. The Kiefer–Wolfowitz +theorem provides one when the features `φ` span the space. Both hypotheses are discharged here. + +The features need not span `E`. Exp3 for linear bandits is run on the span `W` of the features, +with the features `LMLExplore.spanFeatures φ` seen as elements of `W` and the Kiefer–Wolfowitz +design `LMLExplore.kieferWolfowitzDesign φ` of `W` as exploration. The learner observes only the +losses `⟪φ c, y⟫` of the arms, which are the losses `⟪φ c, P y⟫` of the orthogonal projection `P y` +of the loss vector onto `W`. So the regret against an adaptive adversary in `E` is the regret +against the projected adversary in `W`. The bound then holds with the dimension `m` of the span of +the features in place of `d`, and `m ≤ d`, with no assumption on the features. This is +[LS, Theorem 27.1] together with the Kiefer–Wolfowitz theorem for sets that need not span the space +[LS, Theorem 21.1, Exercise 21.6]. + +## Main definitions + +* `LMLExplore.Bandits.linearExp3KieferWolfowitz φ η γ hγ`: Exp3 for linear bandits on the span of + the features, with the Kiefer–Wolfowitz exploration. + +## Main statements + +* `linearExp3Estimate_spanFeatures`: when the features span `E`, the loss estimates of Exp3 for + linear bandits with the features in their span are those with the features in `E`, and + `linearExp3KieferWolfowitz_eq_linearExp3`: then the algorithm is Exp3 for linear bandits on `E` + with the Kiefer–Wolfowitz exploration. In general, by + `LMLExplore.inner_ringInverse_sum_smul_rankOne_spanFeatures`, the loss estimates are those of + [LS, Algorithm 15] in `E` with the Moore–Penrose inverse `V⁺` in place of `V⁻¹`. +* `LinearExp3.integral_regret_linearExp3KieferWolfowitz_le_two_mul_sqrt_three`: against an + adaptive adversary, with the tuned rates, the expected regret of `linearExp3KieferWolfowitz` + against every arm is at most `2 √(3 m n log k)`, where `m` is the dimension of the span of the + features. +* `LinearExp3.integral_regret_linearExp3KieferWolfowitz_le_two_mul_sqrt_three_finrank`: the same + expected regret is at most `2 √(3 d n log k)`, where `d` is the dimension of `E`. +* `LinearExp3.integral_regret_detFeedback_linearExp3KieferWolfowitz_le_two_mul_sqrt_three`: the + first bound, for every run against the environment of an adaptive adversary. + +## References + +* [LS] T. Lattimore and Cs. Szepesvári, *Bandit Algorithms*, Cambridge University Press, 2020, + Chapters 21 and 27. +-/ + +public section + +open MeasureTheory ProbabilityTheory Finset Learning ContinuousLinearMap InnerProductSpace Module + LMLExplore.OnlineLearning +open scoped InnerProductSpace + +namespace LMLExplore.Bandits + +variable {α 𝓞 E : Type*} [NormedAddCommGroup E] [InnerProductSpace ℝ E] + +section Estimate + +variable [Fintype α] [FiniteDimensional ℝ E] (φ : α → E) {p : α → ℝ} + +/-- If the features span `E` and the operator `V_W(p) = ∑ b, p b • φ b ⊗ φ b` on their span is +invertible, then the loss estimates of Exp3 for linear bandits with the features in their span are +those with the features in `E`. -/ +theorem linearExp3Estimate_spanFeatures (hspan : Submodule.span ℝ (Set.range φ) = ⊤) + (hu : IsUnit (∑ b, p b • rankOne ℝ (spanFeatures φ b) (spanFeatures φ b))) : + linearExp3Estimate (spanFeatures φ) p = linearExp3Estimate φ p := by + -- `V(p)` is onto, hence invertible, and its Moore–Penrose inverse is its inverse + have hVu : IsUnit (∑ b, p b • rankOne ℝ (φ b) (φ b)) := by + rw [isUnit_iff_isUnit_toLinearMap, LinearMap.isUnit_iff_range_eq_top, eq_top_iff, ← hspan] + exact le_range_of_apply_coe_eq (fun v ↦ by simp) hu + funext c ℓ a + rw [linearExp3Estimate_apply, linearExp3Estimate_apply, + inner_ringInverse_sum_smul_rankOne_spanFeatures hu, moorePenroseInverse_eq_ringInverse hVu, + coe_coe] + +end Estimate + +section Algorithm + +variable [Fintype α] [FiniteDimensional ℝ E] [Nonempty α] [MeasurableSpace α] + [MeasurableSingletonClass α] [MeasurableSpace 𝓞] (φ : α → E) (η γ : ℝ) + +/-- **Exp3 for adversarial linear bandits with the Kiefer–Wolfowitz exploration**: Exp3 for linear +bandits [LS, Algorithm 15] with learning rate `η` and exploration rate `γ ∈ [0, 1]`, run on the +span of the features `φ`, with the features `spanFeatures φ` and the exploration +`kieferWolfowitzDesign φ`. The features need not span `E`. -/ +noncomputable def linearExp3KieferWolfowitz (hγ : γ ∈ Set.Icc 0 1) : Algorithm 𝓞 α ℝ := + linearExp3 (spanFeatures φ) (kieferWolfowitzDesign φ) η γ + (kieferWolfowitzDesign_mem_simplex φ) hγ + +lemma linearExp3KieferWolfowitz_def (hγ : γ ∈ Set.Icc 0 1) : + linearExp3KieferWolfowitz (𝓞 := 𝓞) φ η γ hγ = + linearExp3 (spanFeatures φ) (kieferWolfowitzDesign φ) η γ + (kieferWolfowitzDesign_mem_simplex φ) hγ := (rfl) + +/-- When the features span `E` and the exploration rate is positive, Exp3 for linear bandits with +the Kiefer–Wolfowitz exploration is Exp3 for linear bandits [LS, Algorithm 15] with the features +`φ` in `E` and the exploration `kieferWolfowitzDesign φ`. -/ +theorem linearExp3KieferWolfowitz_eq_linearExp3 (hspan : Submodule.span ℝ (Set.range φ) = ⊤) + (hγ : γ ∈ Set.Icc 0 1) (hγ0 : 0 < γ) : + linearExp3KieferWolfowitz (𝓞 := 𝓞) φ η γ hγ = + linearExp3 φ (kieferWolfowitzDesign φ) η γ + (kieferWolfowitzDesign_mem_simplex φ) hγ := by + have hsum (n : ℕ) (h : Hist 𝓞 α ℝ n) : + linearExp3HistSum (spanFeatures φ) (kieferWolfowitzDesign φ) η γ n h = + linearExp3HistSum φ (kieferWolfowitzDesign φ) η γ n h := by + induction n with + | zero => simp + | succ n ih => + rw [linearExp3HistSum_succ, linearExp3HistSum_succ, ih, + linearExp3Estimate_spanFeatures φ hspan (isUnit_sum_smul_rankOne_linearExp3Weights + (mem_simplex_iff.1 (kieferWolfowitzDesign_mem_simplex φ)).1 + (isUnit_sum_smul_rankOne_kieferWolfowitzDesign φ) hγ0 hγ.2 _)] + rw [linearExp3KieferWolfowitz_def, linearExp3_def, linearExp3_def] + congr 1 + funext n x + rw [hsum] + +end Algorithm + +namespace LinearExp3 + +variable [Fintype α] [FiniteDimensional ℝ E] [MeasurableSpace E] [BorelSpace E] [Nonempty α] + [MeasurableSpace α] [MeasurableSingletonClass α] [MeasurableSpace 𝓞] {φ : α → E} {η γ : ℝ} + {hγ : γ ∈ Set.Icc 0 1} + {Ω : Type*} {mΩ : MeasurableSpace Ω} {P : Measure Ω} {O : ℕ → Ω → 𝓞} {A : ℕ → Ω → α} + {Y : ℕ → Ω → ℝ} {env : Environment 𝓞 α ℝ} {y : (n : ℕ) → Hist 𝓞 α ℝ n × 𝓞 → E} {n : ℕ} + +/-- **Regret of Exp3 for adversarial linear bandits with the Kiefer–Wolfowitz exploration against +an adaptive adversary** [LS, Theorem 27.1, Exercise 21.6]. Let `m` be the dimension of the span of +the features `φ`, which need not span `E`, and `k ≥ 2` the number of arms. Let an adaptive +adversary choose at round `t` the loss vector `y t (h, o) ∈ E`, a measurable function of the +history `h` and the observation `o` of round `t` with `|⟪φ c, y t (h, o)⟫| ≤ 1` for every arm `c`, +and let the feedback `Y t` of a run of `linearExp3KieferWolfowitz φ η γ` be the loss +`⟪φ (A t), y t (h, o)⟫` of the arm played. With the learning rate `η = √(log k / (3 m n))` and the +exploration rate `γ = η m`, the expected regret after `n ≥ 1` rounds against every arm is at most +`2 √(3 m n log k)`. -/ +theorem integral_regret_linearExp3KieferWolfowitz_le_two_mul_sqrt_three + [IsProbabilityMeasure P] + (h : IsAlgEnvSeq O A Y (linearExp3KieferWolfowitz φ η γ hγ) env P) + (hk : 1 < Fintype.card α) (hn : 0 < n) + (hη : η = √(Real.log (Fintype.card α) / + (3 * finrank ℝ (Submodule.span ℝ (Set.range φ)) * n))) + (hγm : γ = η * finrank ℝ (Submodule.span ℝ (Set.range φ))) (hy : ∀ t < n, Measurable (y t)) + (hy1 : ∀ t < n, ∀ x c, |⟪φ c, y t x⟫_ℝ| ≤ 1) + (hY : ∀ t < n, Y t =ᵐ[P] fun ω ↦ ⟪φ (A t ω), y t (history O A Y t ω, O t ω)⟫_ℝ) (b : α) : + ∫ ω, ∑ t ∈ range n, (⟪φ (A t ω), y t (history O A Y t ω, O t ω)⟫_ℝ - + ⟪φ b, y t (history O A Y t ω, O t ω)⟫_ℝ) ∂P ≤ + 2 * √(3 * finrank ℝ (Submodule.span ℝ (Set.range φ)) * n * + Real.log (Fintype.card α)) := by + rcases subsingleton_or_nontrivial (Submodule.span ℝ (Set.range φ)) with hW | hW + · -- all the features vanish, so does the regret + have hφ (a : α) : φ a = 0 := by + simpa using congrArg Subtype.val (Subsingleton.elim (spanFeatures φ a) 0) + simp only [hφ, inner_zero_left, sub_self, sum_const_zero, integral_zero] + positivity + -- the run is a run of Exp3 for linear bandits on `W` against the projected adversary + rw [linearExp3KieferWolfowitz_def] at h + have key := integral_regret_le_two_mul_sqrt_three + (y := fun t x ↦ (Submodule.span ℝ (Set.range φ)).orthogonalProjectionOnto (y t x)) h + (isUnit_sum_smul_rankOne_kieferWolfowitzDesign φ) + (inner_ringInverse_sum_smul_rankOne_kieferWolfowitzDesign_le_finrank φ) hk hn hη hγm + (fun t ht ↦ (Submodule.span ℝ _).orthogonalProjectionOnto.continuous.measurable.comp + (hy t ht)) + (fun t ht x c ↦ by rw [inner_spanFeatures_orthogonalProjectionOnto]; exact hy1 t ht x c) + (fun t ht ↦ by simp only [inner_spanFeatures_orthogonalProjectionOnto]; exact hY t ht) b + simp only [inner_spanFeatures_orthogonalProjectionOnto] at key + exact key + +/-- **Regret of Exp3 for adversarial linear bandits with the Kiefer–Wolfowitz exploration against +an adaptive adversary, in terms of the dimension of the space** [LS, Theorem 27.1]. Under the +hypotheses of `LinearExp3.integral_regret_linearExp3KieferWolfowitz_le_two_mul_sqrt_three`, on a +space of dimension `d`, the expected regret against every arm is at most `2 √(3 d n log k)`. -/ +theorem integral_regret_linearExp3KieferWolfowitz_le_two_mul_sqrt_three_finrank + [IsProbabilityMeasure P] + (h : IsAlgEnvSeq O A Y (linearExp3KieferWolfowitz φ η γ hγ) env P) + (hk : 1 < Fintype.card α) (hn : 0 < n) + (hη : η = √(Real.log (Fintype.card α) / + (3 * finrank ℝ (Submodule.span ℝ (Set.range φ)) * n))) + (hγm : γ = η * finrank ℝ (Submodule.span ℝ (Set.range φ))) (hy : ∀ t < n, Measurable (y t)) + (hy1 : ∀ t < n, ∀ x c, |⟪φ c, y t x⟫_ℝ| ≤ 1) + (hY : ∀ t < n, Y t =ᵐ[P] fun ω ↦ ⟪φ (A t ω), y t (history O A Y t ω, O t ω)⟫_ℝ) (b : α) : + ∫ ω, ∑ t ∈ range n, (⟪φ (A t ω), y t (history O A Y t ω, O t ω)⟫_ℝ - + ⟪φ b, y t (history O A Y t ω, O t ω)⟫_ℝ) ∂P ≤ + 2 * √(3 * finrank ℝ E * n * Real.log (Fintype.card α)) := by + refine (integral_regret_linearExp3KieferWolfowitz_le_two_mul_sqrt_three h hk hn hη hγm hy hy1 hY + b).trans ?_ + have hm : (finrank ℝ (Submodule.span ℝ (Set.range φ)) : ℝ) ≤ finrank ℝ E := + Nat.cast_le.2 (Submodule.finrank_le _) + have hlog : 0 ≤ Real.log (Fintype.card α) := Real.log_nonneg (by exact_mod_cast hk.le) + gcongr + +/-- **Regret of Exp3 for adversarial linear bandits with the Kiefer–Wolfowitz exploration, for +every run against an adaptive adversary** [LS, Theorem 27.1, Exercise 21.6]. Let an adaptive +adversary choose at round `t` the loss vector `y t (h, o) ∈ E`, a measurable function of the +history `h` and of the observation `o` of round `t`, drawn from arbitrary observation kernels `obs`, +with `|⟪φ c, y t (h, o)⟫| ≤ 1` for every arm `c`, and let its environment +`Environment.detFeedback obs (fun t x ↦ ⟪φ x.2, y t x.1⟫) _` return the loss of the arm played. +With the rates of `LinearExp3.integral_regret_linearExp3KieferWolfowitz_le_two_mul_sqrt_three`, +every run of `linearExp3KieferWolfowitz φ η γ` against it has expected regret at most +`2 √(3 m n log k)` against every arm, where `m` is the dimension of the span of the features. -/ +theorem integral_regret_detFeedback_linearExp3KieferWolfowitz_le_two_mul_sqrt_three + [IsProbabilityMeasure P] + {obs : (n : ℕ) → Kernel (Hist 𝓞 α ℝ n) 𝓞} [∀ n, IsMarkovKernel (obs n)] + {hf : ∀ n, Measurable fun x : (Hist 𝓞 α ℝ n × 𝓞) × α ↦ ⟪φ x.2, y n x.1⟫_ℝ} + (h : IsAlgEnvSeq O A Y (linearExp3KieferWolfowitz φ η γ hγ) + (Environment.detFeedback obs (fun n x ↦ ⟪φ x.2, y n x.1⟫_ℝ) hf) P) + (hk : 1 < Fintype.card α) (hn : 0 < n) + (hη : η = √(Real.log (Fintype.card α) / + (3 * finrank ℝ (Submodule.span ℝ (Set.range φ)) * n))) + (hγm : γ = η * finrank ℝ (Submodule.span ℝ (Set.range φ))) (hy : ∀ t < n, Measurable (y t)) + (hy1 : ∀ t < n, ∀ x c, |⟪φ c, y t x⟫_ℝ| ≤ 1) (b : α) : + ∫ ω, ∑ t ∈ range n, (⟪φ (A t ω), y t (history O A Y t ω, O t ω)⟫_ℝ - + ⟪φ b, y t (history O A Y t ω, O t ω)⟫_ℝ) ∂P ≤ + 2 * √(3 * finrank ℝ (Submodule.span ℝ (Set.range φ)) * n * + Real.log (Fintype.card α)) := + integral_regret_linearExp3KieferWolfowitz_le_two_mul_sqrt_three h hk hn hη hγm hy hy1 + (fun t _ ↦ h.feedback_detFeedback_ae_eq t) b + +end LinearExp3 + +end LMLExplore.Bandits