Principal lecturer: Prof Neil Lawrence
Taken by: MPhil ACS, Part III
Code: L172
Term: Michaelmas
Hours: 16 (8 × 2hr lectures)
When: Tuesdays 10:00–12:00, 13 October – 1 December 2026
Where: FW26, William Gates Building
Class limit: max. 10 students
Prerequisites: Undergraduate-level probability and statistics (distributions, Bayes theorem), linear algebra (matrix operations, eigenvalues), and basic multivariate calculus. No prior physics or thermodynamics assumed.
Develop the mathematical connections between thermodynamics, information theory, and Bayesian inference, and understand how they provide a toolkit for reasoning about the foundations of intelligent systems.
Entropy appears in three apparently separate traditions — thermodynamics (Boltzmann, Gibbs), information theory (Shannon), and Bayesian inference (Jaynes) — and turns out to be the same mathematical object viewed from different operational assumptions. The operational split is: entropy forbids, probability prescribes. The course covers:
- probability and entropy review; Socratic LLM method; that theme via perpetual motion and the human–machine bandwidth gap of The Atomic Human; then the Boltzmann distribution and free energy,
- Shannon entropy and its formal equivalence to thermodynamic entropy; mutual information and the data-processing inequality (statement); MaxEnt and the exponential family,
- Maxwell's demon and Landauer's principle: the thermodynamic cost of decision-making,
- information geometry: the Fisher metric, Crooks' thermodynamic length, dually flat geometry, MaxEnt as projection, and natural gradient descent,
- multi-information,
$I+H=C$ , DPI and the information bottleneck, von Neumann entropy, - probability transport, Schrödinger bridges, and information-theoretic limits on intelligent agency.
Four ten-minute in-class Moodle quizzes sit at the start of lectures 2, 5, 7 and 8.
Worksheets use LLMs under a Socratic protocol introduced in lecture 1: students act as Socrates (curiosity, then skepticism). Worksheet 1 is a ten-turn dialogue; later worksheets mix code with shorter LLM probes. Submissions are anonymous (candidate number, not CRSid or name).
Equip students to reason rigorously about entropy across thermodynamics, information theory, and Bayesian inference, and to evaluate claims about intelligent systems using information-theoretic constraints.
Four take-home worksheets (15% each, 60% total). Worksheet 1 is a Socratic LLM dialogue plus reflection; later worksheets combine a short Python notebook with written reflection. Four short in-class Moodle quizzes (10% each, 40% total).
Shannon (1948), Bell System Technical Journal; Jaynes (1957), Physical Review; Landauer (1961), IBM Journal; Amari & Nagaoka (2000), Methods of Information Geometry (Chapters 1–3); Crooks (2007), Physical Review Letters. Cover & Thomas (2006) and MacKay (2003) as background. Welling, Lu and Holdijk (2026), Generative AI and Stochastic Thermodynamics (GAIST), is a supplementary monograph: Chapter 3 for free energy, Chapter 5 for the ELBO, Chapters 14 and 22 for the Wasserstein and Schrödinger-bridge geometries. It is not a substitute for Crooks (2007) or Amari.