[RemyDegenne]

Amends the ReinforcementLearning roadmap so that its planning layers ask for the theory at the generality of [P] and [Ber22], and its learning layers take the reward model of each source. It also cites the Identification roadmap for the general change of measure, event transfer, MGF domination and Wald's identity, instead of specifying bandit-only versions here. The MDP model keeps two separate kernels P and R; the Conventions now state this explicitly. Companion to #10, the Identification roadmap amendment, whose Layers 0.3, 2.1–2.2, 4.1 and 4.2 it cites.

Why.

• Attainment. Layer 0.3 assumed that the supremum T x = sup_d T_d x is attained. As a result, LMLExplore's optimality equation, its uniqueness, policy iteration and modified policy iteration all need [Finite 𝓐], even though the bounded measurable fixed point needs only [Countable 𝓐]. Suprema with ε-greedy decisions are enough for the fixed point, the optimality equation and ε-optimal policies [P, Theorems 6.2.2, 6.2.5, 6.2.11], [Ber22, Chapter 2]. IsLUB alone is not enough: a two-state example with two decisions is now a completion check.
• Finite horizon. The finite horizon was specified for EpisodicMDP only. runReturn had no discount, and episodic optimality was compared only against Markov policies, without a terminal value. Backward induction gives the supremum over all history-dependent policies, with a terminal value and any discount [P, Theorems 4.3.2–4.3.4, 4.4.2], [Ber12, Chapter 1]. The episodic case is the instance M.augmented.
• Operators. The convention "value functions are pointwise" kept the operator theory off the Banach space that Layers 2 and 13 need. LMLExplore has boundedMeasurable, Kernel.integralCLM and Kernel.resolvent, but the resolvent is never identified with (1 - γP)⁻¹, there is no action-value Bellman operator, and the abstract PI/MPI rates have no users.
• Rewards. The simulation lemma assumed equal mean rewards, and every learning layer assumed known deterministic rewards. UCRL2 [JOA10], MVP [ZJD21] and [ZCLD24] learn unknown stochastic rewards, Kearns–Singh's simulation lemma has a reward-error term, and Tsitsiklis's Q-learning allows random rewards.
• Ownership. Layer 6.2 told contributors to use a bandit-only LMLExplore lemma for key-determined environments. The chain rule at a stopping time for arbitrary environments [GMS19, §2] belongs to the Identification roadmap.

Changes (only LMLRoadmaps/ReinforcementLearning/):

• Conventions.

  • The two-kernel model, with reward and next state conditionally independent, is made explicit.
  • The reward model of each learning item is fixed layer by layer.
  • The finite horizon allows every γ : ℝ≥0.
  • Operator statements live on boundedMeasurable 𝓢, and pointwise statements are corollaries.
  • EpisodicMDP keeps real rewards and entropy-mdp's curried meanReward h s a.
  • Unbounded rewards with weighted sup norms, and admissible action sets A(s), are listed outside the roadmap.

• Dependency table. New rows cite the Identification roadmap:

  • Layers 2.1–2.2: the chain rule at a stopping time and data processing;
  • Layer 4.1: the event transfer from runs to reward streams;
  • Layer 4.2: conditional MGF domination;
  • Layer 0.3: Wald's identity.

  A new row lists the existing operator modules.

• Layer 0.

  • 0.1: boundedMeasurable is the space of operator statements.
  • 0.2: Blackwell's argument is proved once.
  • 0.3: abstract dynamic programming under two hypotheses, T_d ≤ T and ε-greedy decisions, with IsLUB of the fixed points and the bound ε/(1-γ). Greedy decisions are required only by PI and MPI. The two-state counterexample is a completion check.
  • 0.4: RVI uses the topical oscillation bound.

• Layer 1.4–1.5. Kernel.integralCLM, and R_γ = (1 - γκ)⁻¹ as a Neumann series. The simulation identity takes two rewards.

• Layer 3.1. Freedman's supermartingale is the one-step bound E[exp(λX - ψ_b(λ)E[X²|𝓕])|𝓕] ≤ 1, fed into the Identification domination theorem [HRMS20]. There is no separate supermartingale argument.

• Layer 6. The keyed transfer is stated in the shape of Identification Layer 4.1. The keyed divergence decomposition is derived from the Identification chain rule, the step divergence of key-determined environments and the keyed Wald regrouping.

• Layer 7.

  • 7.3: one state-marginal definition for kernels ℕ → Kernel (𝓢 × 𝓐) 𝓢, replacing the two LMLExplore copies.
  • 7.4: runReturn γ N u takes a discount.
  • 7.6: episodic values as runReturn of M.augmented.
  • 7.7: a new episodeEnv, whose episodes also return the rewards.

• Layer 8 (rewritten).

  • Bellman operators for every γ, with the predicate IsApproxGreedy.
  • Backward induction for any MDP: comparison of every algorithm, ε-optimal and greedy Markov policies, and policy evaluation.
  • Compact action sets.
  • Episodic MDPs through M.augmented, with optimality against all algorithms.

• Layer 9. Countable actions throughout, as an instance of Layer 0.3.

  • 9.1: operators on boundedMeasurable built from integralCLM, including qBellmanEval and qBellman on boundedMeasurable (𝓢 × 𝓐).
  • 9.2: v^π = (1 - γP_π)⁻¹ r_π.
  • 9.4: optimality equation, sup_d v^d and ε-optimal policies for countable actions.
  • 9.5: finite action sets.

• Layers 10–13.

  • 10: rates on boundedMeasurable, derived from ContractingWith and Layer 0.3.
  • 11.3–11.4: cross-references updated.
  • 12.2: simulation lemma and componentwise bounds with a reward error [KS02].
  • 13.1: Laurent expansion through Layer 9.2.

• Learning layers.

  • 14.1–14.2: empirical rewards and reward streams in M.env μ and episodeEnv.
  • 14.6: the MDP change of measure Σ_{s,a} E[N_{sa}(τ)](KL(R) + KL(P)) [OPT18, AGP21], plus a reward term in episodeEnv.
  • 16.1–16.2: asynchronous Q-learning with observed stochastic rewards, through qBellman.
  • 17: any reward kernel with rewards in [0, 1].
  • 18.5–18.6: MVP and [ZCLD24] with unknown stochastic rewards in episodeEnv (c₂ = 2√2, [ZCLD24, Assumption 1]).
  • 18.4: also for episodeEnv.
  • 21.1: UCRL2 with unknown rewards and realized-reward regret.
  • 21.2: uses Layer 14.6.

• References. [Ber12], [AGP21], [GMS19], [HRMS20], [KS02], [OPT18].

• Suggested.lean.

  • Added: abstract DP without attainment (isLUB_setOf_isFixedPt, abs_sub_le_of_isFixedPt_of_sub_le), Freedman's one-step bound, saKernel, stateMarginal, runReturn with a discount, bellmanEval, IsApproxGreedy, qBellman, the finite-horizon theorems iSup_runReturn_eq_bellman_iterate and runReturn_markov_greedy, exists_optValue_sub_le_value, qBellman_optQValue, and the simulation lemma with a reward error.
  • bellman_optValue, eq_optValue_of_bellman_eq and value iteration take [Countable 𝓐].
  • value_npg_ge drops the deterministic-reward hypothesis.

Points to check.

• 0.3. Abstract dynamic programming assumes $T_d\le T$ and ε-greedy decisions. The hypothesis that $Tx$ is the least upper bound of the $T_dx$ is not enough: the completion check gives two decisions on Fin 2 → ℝ where the fixed point of $T$ is $(1-\gamma)^{-1}(1,1)$, while the least upper bound of the fixed points is $(1-\gamma^2)^{-1}(1,1)$.
• Unknown rewards. 18.5–18.6 follow [ZJD21] and [ZCLD24] for unknown stochastic rewards (reward bonus $c_2=2\sqrt2$, and [ZCLD24, Assumption 1] for 18.6). 16.2 relies on the random rewards of Tsitsiklis 1994, which were not checked against the paper.
• 9.4–9.5. Puterman's theorems are split as [P, 6.2.11] for countable actions and [P, 6.2.7, 6.2.9, 6.2.10 (a)] for finite actions. This split is unverified.
• New names. IsApproxGreedy, episodeEnv, qBellman, qBellmanEval, MDP.stateMarginal P π μ for a sequence of kernels, and runReturn γ N u.
• Unchanged. The model keeps two kernels P and R, as the code owner decided. Layer 7.1 still points to LML's mdp branch; an LML pull request for MDPs is being prepared and can replace that reference.
• Outside. Weighted sup norms and admissible action sets $A(s)$ move to the list of material outside the roadmap.

Suggested.lean builds.

Drafted with Claude Opus 5.5 at the request of the code owner.

🤖 Generated with Claude Code