Vorlesung: Information, Inference and inverse Problems in Science

This course develops the analytical foundations of information theory and statistical inference from the perspective of theoretical physics. Its central theme is the structural equivalence between the mathematical frameworks underlying statistical mechanics and those underlying data inference: the same variational principles that yield equilibrium ensembles also govern optimal inference from incomplete observations.

The course covers Shannon's information measures, the maximum entropy principle and its role in deriving statistical-mechanical ensembles, Bayesian inference and model selection, the Fisher information metric and its geometric interpretation, and inverse problems in which interaction parameters are reconstructed from measured correlations. Throughout, the emphasis is on exact analytical methods rather than numerical implementation or algorithmic recipes.

There likely is no single book that covers the selection of topics of this lecture and specific literature will be provided throughout.

If you want to get a taste that is rather easy to read: von Baeyer, Information: The New Language of Science

Monographs that cover a lot of ground and most of it in more detail than this lecture

Jaynes, Probability Theory: The Logic of Science 

MacKay, Information Theory, Inference and Learning Algorithms (A usable version of this is freely available via the author. There are also youtube recordings of corresponding lectures)

lectures in English or German, up to students' taste. There might be optional problems, particularly if there is interest for it

none

My approach to this comes from statistical mechanics and being familiar with that will likely be helpful, but I will try to be self-contained or at the least provide literature to fill any gaps.

no exams intended, do reach out if you require one

• Information-theoretic foundations: Shannon entropy, Kullback-Leibler divergence, mutual information, data processing inequality
• Maximum entropy principle: constrained optimization, derivation of canonical and grand-canonical ensembles as MaxEnt distributions, connection to thermodynamic potentials
• Bayesian inference: Bayes' theorem as a learning rule, prior selection, conjugate priors, posterior computation
• Model comparison and complexity: marginal likelihood, Bayesian evidence, Occam factors, overfitting from an information-theoretic perspective
• Fisher information: definition, Cramér-Rao bound, Fisher information as a Riemannian metric on probability space
• Information geometry near phase transitions: divergence of the Fisher metric at critical points, connection to susceptibilities and fluctuations
• Inverse problems in statistical physics: inferring couplings from correlation data, the inverse Ising problem, mean-field and pseudo-likelihood methods
• Stochastic inference: trajectory-level information, inference on dynamical systems driven by noise

Open to all interested students. The course is formally offered as a Wahlvorlesung im Schwerpunkt B (M.Sc. Physics). If you are enrolled in a different programme and would like to use this course as an elective or minor component, please contact me in advance so that we can clarify the crediting and examination requirements.