Differential entropy

The differential entropy of a continuous random variable $X$ with probability density function (pdf) $p(x)$ is

It is the continuous analogue of Shannon entropy for discrete distributions,

Characteristics:

• Scale dependence:
  Shannon entropy $H(X)$ is always nonnegative and invariant under relabeling of outcomes. By contrast, $h(X)$ can be negative and depends on the units of $x$. If we rescale $x\mapsto ax$, then

• Discretization link:
  If a continuous random variable $X$ is discretized into bins of size $\mathrm{\Delta }$, then the Shannon entropy of the discretized distribution satisfies

Thus, differential entropy is not absolute: it only gains physical meaning when a fundamental resolution scale is specified.

Planck Length as a Natural Cutoff

In physics, the Planck length

is often regarded as the minimal meaningful length scale. When interpreting entropy in continuous systems (fields, spacetime degrees of freedom, black holes), the bin size $\mathrm{\Delta }$ can be taken to be on the order of ${\ell }_P$.
This gives a bridge between differential entropy and a physically grounded discrete Shannon entropy:

• Without such a cutoff, $h(X)$ remains ambiguous.
• With ${\ell }_P$ as the "ultimate resolution," entropy becomes a dimensionless, absolute quantity.
• This perspective is important in quantum gravity and black hole thermodynamics, where entropy counts microscopic states per Planck-scale cell of phase space or spacetime.

Gibbs entropy

Consider a system in Classical Statistical Mechanics with ensemble given by $\rho$.
The statistical definition of entropy, known as the Gibbs entropy, is

Important caution:

• In the continuous case, the numerical value of $S/k$ depends on the coordinates used and on the units of $q,p$.
• Unlike the discrete case, $S=0$ does not imply determinism. For example, a uniform distribution on $[0,1]$ has $S/k=0$ (in natural log units), yet it represents maximum uncertainty given that support.
• A completely deterministic macrostate (Dirac delta distribution) has $S=-\mathrm{\infty }$ in this formula, not zero.

In some sense, $S$ counts the number of microstates compatible with the macrostate. For a macrostate with ensemble a Dirac delta, there is only 1 compatible microstate, and the entropy diverges to $-\mathrm{\infty }$.

Important relation

Consider a system in thermal equilibrium with a bath at temperature $T$ described by the canonical ensemble, where the probability density is ${\rho }_S(q,p)=\frac{1}{Z}{e}^{-\beta H_S(q,p)}$ with $\beta =1/(kT)$.

1. Express Entropy using the partition function ($Z$)
   Substituting ${\rho }_S$ into the entropy definition:$$\ln {\rho }_S=\ln \left(\frac{e^{-\beta H_S}}{Z}\right)=-\beta H_S-\ln Z$$Taking the average:$$⟨\ln {\rho }_S⟩=-\beta ⟨H_S⟩-\ln Z$$Plugging into $S=-k⟨\ln {\rho }_S⟩$:$$S=k\beta ⟨E⟩+k\ln Z$$With $\beta =1/(kT)$:$$S=\frac{⟨E⟩}{T}+k\ln Z$$
2. Take the Differential
   For fixed Hamiltonian parameters:$$\frac{dS}{d\beta }=k\beta \frac{d⟨E⟩}{d\beta }$$Hence:$$dS=k\beta d⟨E⟩$$Substituting $\beta =1/(kT)$:$$\boxed{{\displaystyle d⟨E⟩=TdS}}$$

Boltzmann entropy

See harmonic oscillator in CSM#5) Entropy

Maximum Entropy Distributions

• Constraint: only mean fixed.
  • On $[0,\mathrm{\infty })$, the maximum entropy distribution is the exponential distribution (Boltzmann distribution):

• On $\mathbb{R}$, there is no maximum entropy distribution with only the mean fixed (entropy can be made arbitrarily large).

• Constraint: mean and variance fixed.

  • On $\mathbb{R}$, the maximum entropy distribution is the Gaussian distribution:

Derivation sketch (Lagrange multipliers)

Maximize Shannon entropy

subject to normalization and the moment constraints. The variational problem

yields (after variation)

• If only the mean is constrained: ${\lambda }_2=0$, giving an exponential distribution (normalizable only on $[0,\mathrm{\infty })$).
• If mean and variance are constrained: ${\lambda }_2>0$, giving a Gaussian.

In general, maximum entropy solutions belong to exponential families, with form determined by the active constraints.