Statistical Mechanics and Entropy

Neueste Paper

• Parameter Estimation for Unnormalized Discrete Models via Empirically Localized Deformed Bregman Divergence

  Takashi Takenouchi · 28. September 2026

  Estimation of parameter of probabilistic models is an important task in the field of machine learning.For models of discrete variables, calculation of the normalization constant of model is sometimes difficult and a lot of researches have been done to avoid the calculation of the normalization const…

• A Generalization of Amari's Bayesian Duality

  Mohammad Emtiyaz Khan, Thomas M\"ollenhoff · 9. September 2026

  Amari's contributions to information geometry and machine learning are well known. Here, we revisit Amari's work on Bayesian duality which has not received as much attention. We connect Amari's Bayesian duality to a convex duality of Bayes' rule. Using this connection, we present a generalization of…

• The Dually Flat Geometry of Planning as Inference

  Nikola Milosevic, Asaki Kataoka, Nicolas Hinrichs, Kenji Doya, Nico Scherf · 4. September 2026

  We present an alternative characterization of the occupancy measure of reinforcement learning, obtained by embedding the planning criterion into the dynamics through a resetting planning process. Its stationary measure, which we term visitation measure, is the object on which the information geometr…

• Information on trajectories: martingales and random times

  Akshay Balsubramani · 21. August 2026

  Accounting for information flow on the path space of trajectories of a nonnegative martingale yields exact variational identities for it, even at arbitrary random times. This recovers the widely used classical concentration inequalities, from Ville to PAC-Bayes, and measures what each one discards. …

• K\"ahler landscapes for complex neural network descents and guarantees including a search and destroy of the Calabi-Yau manifold

  Andrew Gracyk · 21. August 2026

  We study landscapes for complex-parameterized networks. Our approach is motivated with an information-theoretic manifold perspective of the parameter and via classical optimization guarantees although of complex geometric variety such as through Dolbeault asymptotics. The descent path admits a K\"ah…

• Density-Reweighted Entropic Optimal Transport: Decoupling Geometry from Sampling Density

  Keyi Li, Yuval Kluger, Boris Landa · 18. August 2026

  Dataset alignment is a central step in data analysis across science and engineering, where the goal is to match observations between datasets. Entropic Optimal Transport (EOT) offers a computationally tractable framework for this task by encoding cross-dataset affinities in a transport plan. However…

• Variational Bounds for Perceptron Learning from Structured Data

  Francesco Camilli, Pierluigi Contucci, Federica Gerace, Emanuele Mingione · 6. August 2026

  We introduce a variational approach to a finite-temperature continuous-spin perceptron trained on a Gaussian mixture. The model allows for a broad class of concave utilities and log-concave separable prior measures on the spins. By combining the interpolation method with log-concavity and concentrat…

• Perspectives on Tsallis Statistics for Artificial Intelligence

  Kleyton da Costa, Bernardo Modenesi · 4. August 2026

  Tsallis statistics generalizes Boltzmann-Gibbs statistical mechanics through a single real parameter $q$ that controls the weight assigned to rare and frequent events. Originally proposed to describe physical systems with long-range correlations, multifractal geometry, and heavy-tailed fluctuations,…

• On a joint simultaneous learning of relevant feature subsets and subspaces in regression-like problems

  Illia Horenko · 3. August 2026

  We extend a recently introduced Entropy-Optimal Manifold Clustering (EOMC) to allow for a joint simultaneous identification of subsets and subspaces of relevant features in nonstationary and nonlinear regression problems. It is shown that the proposed extension - that we coin as Entropy-Optimal Mani…

• Learning Ergodic Dynamical Systems from a Finite Trajectory

  Oleksii Kachaiev, Silvia Villa, Lorenzo Rosasco · 27. Juli 2026

  We consider the problem of learning from a single finite trajectory of an ergodic stochastic dynamical system. More precisely, we study discrete-time autonomous stochastic systems defining time-homogeneous Markov processes. We first focus on estimating the optimal one-step prediction function by non…

• Fisher Widths: Local Learning Geometry and Anisotropic Recovery

  Vu Khac Ky · 24. Juli 2026

  We study Gaussian-width complexity on statistical manifolds through a pair of functionals: the primal Fisher width $w_G(T) = w(G^{1/2}T)$, induced by the Fisher metric, and the inverse-Fisher width $w_{G^{-1}}(T) = w(G^{-1/2}T)$, induced by the inverse Fisher metric. The two widths play complementar…

• The Curvature Shadow: An Apparent Failure of Maximum-Entropy Equilibrium Selection is a Removable Artifact

  Luis Leal · 21. Juli 2026

  In two-player zero-sum games whose Nash equilibria form a convex set, regularized solvers such as Regularized Nash Dynamics (R-NaD) empirically select the maximum-entropy member: the information projection (I-projection) of a uniform reference onto the Nash set. On a panel of small games this match …

• A Transport-Based Geometry of Belief-Cost

  Laurent Caraffa · 30. Juni 2026

  A finite agent, a machine's digital twin or any bounded reasoner, infers a fixed and noisy world through finite sensors, so its coherent output is a belief: a probability density over states (the Bayes posterior). Such an agent stops short of certainty, and revising a belief carries a cost. We propo…

• Efficient and Stable Multi-Dimensional Kolmogorov-Smirnov Distance

  Peter Matthew Jacobs, Foad Namjoo, Jeff M. Phillips · 29. Juni 2026

  We revisit extending the Kolmogorov-Smirnov distance between probability distributions to the multi-dimensional setting, and make new arguments about the proper way to approach this generalization. Our proposed formulation maximizes the difference over orthogonal dominating rectangular ranges (d-sid…

• Beyond Global Divergences: A Local-Mass Perspective on Bayesian Inference

  Hanli Xu, Fengxiang He, Sarat Moka · 26. Juni 2026

  Global objectives, such as KL divergence and ELBO, are widely used in Bayesian inference for measuring distributional discrepancy. This paper studies their local-mass behaviour that is not directly captured by such objectives. We introduce and use two mathematical tools: (1) Mass Index for recording…

• A Bregman Perspective on Classification and Regression Trees

  Mathias Bourel · 25. Juni 2026

  Classification and Regression Trees (CART) constitute one of the most influential paradigms in statistical learning. Although a variety of impurity measures have been proposed for different statistical models, these criteria are typically introduced on a case-by-case basis and analyzed separately. I…

• The Cost Geometry of Belief: finite-resource inference under noisy observation

  Laurent Caraffa · 23. Juni 2026

  We equip the space of beliefs with a cost geometry (what it costs to pass from one belief to another): optimal transport in Wasserstein space, reweighted conformally by Fisher information (the price of the precision at stake), distinct from the Fisher-Rao metric. In the setting we consider, a finite…

• Doeblin Curves

  Dongmin Lee, William Lu, Anuran Makur, Japneet Singh · 19. Juni 2026

  Recent research on Doeblin coefficients has shed light on their usefulness as a multi-way generalization of the Dobrushin contraction coefficient for TV distance, in a separate vein from their classic role in the theory of Markov chain ergodicity. However, strong conditions, such as being bounded aw…

• Generalized Kullback-Leibler Divergence Loss

  Jiequan Cui, Beier Zhu, Qingshan Xu, Zhuotao Tian, Xiaojuan Qi, Bei Yu, Hanwang Zhang, Richang Hong · 18. Juni 2026

  In this paper, we delve deeper into the Kullback-Leibler (KL) Divergence loss and mathematically prove that it is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss that consists of (1) a weighted Mean Square Error (wMSE) loss and (2) a Cross-Entropy loss incorporating soft labels. T…

• On the Entropy Formula for Real, Complex, and Quaternionic Deep Linear Networks

  Luis Contreras, Marco Nahas, Tejas Kotwal · 16. Juni 2026

  We extend the entropy formula of Menon and Yu for the real Deep Linear Network (DLN) to its complex and quaternionic analogues, obtaining a unified formula for DLNs over $\mathbb{R}$, $\mathbb{C}$, and $\mathbb{H}$.…

• A General Framework for Decision Trees via Bregman Divergences

  Mathias Bourel · 15. Juni 2026

  Decision trees are one of the fundamental tools in statistical learning due to their interpretability, flexibility, and their ability to adapt to nonlinear structures. Among them, the Classification and Regression Trees, introduced by Breiman, Friedman, Olshen, and Stone in 1984, became one of the m…

• Calibeating for general proper losses: A Bregman divergence approach

  Maximilian Fichtl, Crist\'obal Guzm\'an, Nishant A. Mehta · 19. Mai 2026

  This work introduces a general framework for calibeating based on regret minimization. As compared to Foster and Hart's seminal calibeating work which had specialized treatments of Brier score (squared loss) and log loss, we consider a large family of proper losses that includes $\alpha$-Tsallis los…

• Change of measure through the Legendre transform

  Antoine Picard-Weibel, Benjamin Guedj · 15. Mai 2026

  PAC-Bayes generalisation bounds are derived via change-of-measure inequalities that transfer concentration properties from a reference measure to all posterior measures. The specific choice of change of measure determines the assumptions required on the empirical risk; in particular, the classical D…

• Information-theoretic Limits of Learning and Estimation

  Abbas El Gamal, Maxim Raginsky · 11. Mai 2026

  Information theory plays a central role in establishing fundamental limits on what any learning or estimation algorithm can -- and cannot -- achieve, regardless of computational power. In this chapter, we provide an introduction to these connections. End-of-chapter exercises makes the material suita…

• A Closed-Form Upper Bound for Admissible Learning-Rate Steps in Belief-Space Dynamics

  Zixi Li, Youzhen Li · 11. Mai 2026

  Learning-rate steps are usually treated as hyperparameters. This paper isolates a local beliefspace calculation: when an update is modeled as a projected forward step on the probability simplex, admissibility means contractivity in the natural KL/Bregman geometry. Under this model, the upper bound o…