Two modeling targets

• Discriminative: \(p(y|\mathbf{x})\) - label given input, input distribution never modeled
• Generative: \(p(\mathbf{x})\), or jointly \(p(\mathbf{x}, y)\) - the data distribution itself

What each must keep

• Discriminative: only what separates the classes - the boundary, everything else discarded
• Generative: everything - every direction of structure belongs to the model
• One-way street: \(p(\mathbf{x}, y)\) recovers \(p(y|\mathbf{x})\) by Bayes - the reverse is impossible

Why it is harder

• Output space: all of \(\mathbb{R}^d\), not \(\{1, \dots, C\}\)
• No labels supervise the structure
• Normalization over the whole data space - the obstruction every method confronts

The output of a trained model: samples

• \(\mathbf{x} \sim p_{\boldsymbol{\theta}}\) - new data, not labels or scores
• No per-output ground truth - a generated face matches no training face
• Individually ungradable - only the collection can be judged

The target is a distribution, seen only through samples

• \(p_{\text{data}}\): unknown - available as i.i.d. draws
• Success: \(p_{\boldsymbol{\theta}} \approx p_{\text{data}}\) - the model’s samples collectively indistinguishable from data

The consequence for training

• Objectives become distribution comparisons: forward KL, JS, Wasserstein
• Evaluation becomes distribution comparison too - a sample set against a sample set

Shannon’s link

• An outcome of probability \(p\): codable in \(-\log_2 p\) bits - arithmetic coding achieves it, up to rounding
• Code length under a model: \(-\log_2 p_{\boldsymbol{\theta}}(\mathbf{x})\) bits
• Maximizing log-likelihood ⇔ minimizing the bits to encode the data - the training objective is a compression rate

Worked exactly (the figure)

• 8 × 8 binary bar images: raw code = 64 bits
• Per-pixel model (rates fitted, pixels independent): 57.8 bits - marginal rates capture almost nothing
• True distribution: 8.1 bits - orientation + which bars, nothing else to say
• The compression lives entirely in the between-pixel structure - and noise never compresses

Bits per dimension

• Bits per dimension: \(-\log_2 p_{\boldsymbol{\theta}}(\mathbf{x}) / d\) - how image likelihoods are reported
• Learned image densities: ~3 bits/dim on CIFAR-10, against 8 raw

Energy scores configurations

• \(E: \mathbb{R}^d \to \mathbb{R}\), low energy → plausible

The form follows from maximum entropy

• Maximize \(S = -\sum_i p_i \log p_i\) subject to:
  • fixed mean energy: \(\langle E \rangle = U\) (\(\langle \cdot \rangle\) = expectation, used throughout)
  • normalization: \(\sum_i p_i = 1\)
• Lagrange multipliers force \(p_i \propto e^{-\beta E_i}\), \(\beta = 1/T\)

No generality lost

• Any positive density: set \(E(\mathbf{x}) = -\log \tilde{p}(\mathbf{x})\)
• Energy determines the distribution completely

\(T\) rescales energy differences - minima unmoved

• \(T \to 0\): mass collapses onto the global minimum - sampling becomes optimization
• \(T \to \infty\): flattens toward uniform
• Intermediate \(T\): sets the mass kept by shallower basins
• Decreasing \(T\) during search: simulated annealing

The score needs no normalization

• \(\nabla_{\mathbf{x}} \log p(\mathbf{x}) = -\frac{1}{T}\nabla_{\mathbf{x}} E(\mathbf{x})\) - differentiating in \(\mathbf{x}\) removes \(\log Z\)
• Log-probability slope: available without \(Z\)

What requires \(Z\)

• Likelihood values: \(\log p(\mathbf{x}) = -E(\mathbf{x}) - \log Z\)
• Model comparison and evaluation
• Training: \(\nabla_{\boldsymbol{\theta}} \log Z\) in every likelihood gradient

What survives without \(Z\)

• Ratios: \(p(\mathbf{x}_1)/p(\mathbf{x}_2) = e^{-(E(\mathbf{x}_1) - E(\mathbf{x}_2))}\)
• The score: \(\nabla_{\mathbf{x}} \log p(\mathbf{x}) = -\nabla_{\mathbf{x}} E(\mathbf{x})\)
• Enough to drive samplers - and score-based training

Counting the terms (binary image, 224 × 224 × 3)

• \(d = 150{,}528\) binary variables: \(Z = \sum_{\mathbf{x} \in \{0,1\}^d} e^{-E(\mathbf{x})}\)
• \(2^{150{,}528} \approx 10^{45{,}313}\) terms
• Atoms in the observable universe: \(\sim 10^{80}\)

Differentiate once

• One scalar’s derivative = an average under the model
• Physics name: Helmholtz free energy, \(F = U - TS\), minimized at equilibrium

What the identity costs

• Left side: one derivative. Right side: an integral against \(p(\mathbf{x}|\boldsymbol{\theta})\)
• Exact evaluation: as hard as \(Z\)
• Estimation: requires samples from the model

Data \(\{\mathbf{x}_1, ..., \mathbf{x}_N\} \sim p_{\text{data}}\):

The same objective as a divergence

• \(\text{KL}(p \| q) = \mathbb{E}_{p}[\log(p/q)] \geq 0\), zero iff \(p = q\)
• \(N \to \infty\): \(\frac{1}{N}\sum_i \log p_{\boldsymbol{\theta}}(\mathbf{x}_i) \to \mathbb{E}_{p_{\text{data}}}[\log p_{\boldsymbol{\theta}}(\mathbf{x})]\)
• \(\mathbb{E}_{p_{\text{data}}}[\log p_{\boldsymbol{\theta}}] = -\text{KL}(p_{\text{data}} \| p_{\boldsymbol{\theta}}) - H(p_{\text{data}})\), with \(H(p_{\text{data}})\) constant in \(\boldsymbol{\theta}\)
• Max likelihood ⇔ min \(\text{KL}(p_{\text{data}} \| p_{\boldsymbol{\theta}})\) - forward KL, data in the first argument

What forward KL demands

• \(p_{\text{data}} > 0\), \(p_{\boldsymbol{\theta}} \to 0\): integrand \(\to \infty\) - a missed data mode is unboundedly expensive
• \(p_{\text{data}} = 0\): extra model mass costs nothing directly
• Result: cover every mode - spread thin if the family cannot fit all

The objective is a gap

• \(\log p = -E + F\), averaged: \(\mathcal{L}(\boldsymbol{\theta}) = F(\boldsymbol{\theta}) - \langle E \rangle_{\text{data}}\)
• Differentiate, using \(\nabla_{\boldsymbol{\theta}} F = \mathbb{E}_{p}[\nabla_{\boldsymbol{\theta}} E]\):

Positive phase - push down at data

• \(\nabla_{\boldsymbol{\theta}} E\) at observed examples - one backward pass each

Negative phase - push up where the model puts mass

• Raises energy where the model concentrates - data or not
• Needs samples from \(p(\mathbf{x}|\boldsymbol{\theta})\) - which changes at every update

Why the \(F\) term must be there

• Without \(F\): \(\max_{\boldsymbol{\theta}} \sum_i -E(\mathbf{x}_i)\) is degenerate - sink all energies, commit mass nowhere
• \(F\) is the counterweight - energy at data counts only relative to everywhere else
• Negative phase = \(\nabla_{\boldsymbol{\theta}} F\), the free-energy gradient - the single intractable term

At any maximum of the likelihood the two phases balance:

Moment matching

• Feature energy \(E(\mathbf{x}; \boldsymbol{\theta}) = -\boldsymbol{\theta}^\top T(\mathbf{x})\) (exponential family, \(T\) = sufficient statistics): condition becomes \(\langle T(\mathbf{x}) \rangle_{\text{data}} = \langle T(\mathbf{x}) \rangle_{\text{model}}\)
• Matches exactly the statistics the energy exposes - blind to everything outside \(T\)
• The fit: the max-entropy distribution with those statistics - the same principle behind the Boltzmann form

Richer statistics need richer energies

• Hidden variables: extend \(T(\mathbf{x})\) implicitly
• Network energies: learn \(T(\mathbf{x})\)

Two different asks of the same landscape

• Optimization: \(\arg\max_{\mathbf{x}}\, p(\mathbf{x})\) - one point, the mode (MAP)
• Sampling: visit regions in proportion to their mass - the whole distribution
• The negative phase is an expectation - it needs mass, not modes

The mode is not where the mass is

• Standard Gaussian, \(d\) dimensions: density peaks at \(\mathbf{0}\) - mass concentrates on the shell \(\|\mathbf{x}\| \approx \sqrt{d}\)
• \(d = 100\): no sample ever lands near the mode - the most probable point is not typical
• High dimension: typical set ≠ neighborhood of the mode

One walk, two algorithms

• Deterministic ascent on \(\log p\): parks at a mode - an optimizer
• Matched noise added: visits by mass - a sampler

Monte Carlo

• \(\mathbb{E}_{p}[f(\mathbf{x})] \approx \frac{1}{K}\sum_{k=1}^K f(\mathbf{x}^{(k)})\), with \(\mathbf{x}^{(k)} \sim p\)
• Error \(O(1/\sqrt{K})\), dimension-free - for independent samples
• Negative phase: \(f = \nabla_{\boldsymbol{\theta}} E\), \(\;p = p(\mathbf{x}|\boldsymbol{\theta})\)

Producing \(\mathbf{x}^{(k)} \sim p\)

• Direct samplers need more than \(e^{-E}\):
  • inverse-CDF: needs the normalized CDF - needs \(Z\)
  • rejection sampling: acceptance collapses exponentially in \(d\)
• High-dimensional \(p \propto e^{-E}\): no direct method exists

Build the chain around the target

• Known: irreducible + aperiodic → the chain converges to its stationary \(\pi\), from any start
• MCMC reverses it: given target \(p\), build \(T\) with \(\pi = p\)
• Run the chain - its states become the samples

Detailed balance

• \(p(\mathbf{x})\, T(\mathbf{x} \to \mathbf{x}') = p(\mathbf{x}')\, T(\mathbf{x}' \to \mathbf{x})\) \(\Rightarrow\) \(p\) stationary
• Local, checkable - no global computation

Metropolis-Hastings

• Propose \(\mathbf{x}' \sim q(\mathbf{x}'|\mathbf{x})\), accept with \(\min\!\left(1,\; \frac{e^{-E(\mathbf{x}')}\, q(\mathbf{x}|\mathbf{x}')}{e^{-E(\mathbf{x})}\, q(\mathbf{x}'|\mathbf{x})}\right)\)
• Only energy differences - \(Z\) cancels in the ratio

Named kernels

• Metropolis: random-walk proposal + accept/reject
• Gibbs: one block from its exact conditional - always accepted
• Langevin: gradient-informed proposal

Burn-in

• Early states: reflect the start, not \(p\) - the transient toward stationarity
• Standard practice: discard them

Mixing time

• Steps until the chain forgets its start
• Sets the burn-in length - and the spacing between independent-looking states

Autocorrelation

• Successive states dependent - each step adds less than one sample of information
• Effective sample size: \(K_{\text{eff}} = K \,/\, (1 + 2\sum_{t \geq 1} \rho_t)\), \(\;\rho_t\) = lag-\(t\) autocorrelation
• Monte Carlo error: \(O(1/\sqrt{K_{\text{eff}}})\), not \(O(1/\sqrt{K})\)

In practice

• Multiple chains, dispersed starts - agreement diagnoses convergence (\(\hat{R}\))
• No finite test proves stationarity - diagnostics only falsify it

The kernel deep EBMs use in practice - proposals follow the energy gradient

Langevin dynamics

• Drift = the score: \(\frac{\epsilon}{2}\,\mathbf{s}(\mathbf{x})\), \(\;\mathbf{s} = -\nabla_{\mathbf{x}} E\) - a small step uphill on \(\log p\)
• Noise on, scales matched (\(\epsilon/2\) drift, \(\sqrt{\epsilon}\) noise): stationary distribution exactly \(p \propto e^{-E}\) as \(\epsilon \to 0\)
• Noise is not a nuisance - it is what makes the walk a sampler rather than an optimizer
• Finite \(\epsilon\): discretization bias - a Metropolis correction fixes it (MALA)

Hamiltonian Monte Carlo

• Auxiliary momentum \(\mathbf{p} \sim \mathcal{N}(0, \mathbf{M})\), simulate \(H(\mathbf{x},\mathbf{p}) = E(\mathbf{x}) + \frac{1}{2}\mathbf{p}^\top\mathbf{M}^{-1}\mathbf{p}\)
• Long coherent moves + Metropolis correction - fewer, longer steps

Escaping a basin is a rare event

• Barrier of height \(\Delta E\): crossing fluctuation has probability \(\propto e^{-\Delta E / T}\) per attempt
• Expected escape time (Kramers): \(\tau \sim e^{\Delta E / T}\) - exponential in barrier height

High dimension compounds it

• Image modes: far apart, separated by wide near-zero-probability regions
• Stable \(\epsilon\) capped by the steepest curvature - higher \(d\) → smaller steps, more of them

Spectral view

• Kernel eigenvalues \(1 = \lambda_1 > |\lambda_2| \geq \dots\): mixing time \(\sim 1/(1 - |\lambda_2|)\)
• Deep barrier → \(|\lambda_2| \to 1\): two near-invariant sets (metastability)
• Kramers’ \(e^{\Delta E/T}\) = the spectral gap closing - a property of the landscape, not the budget

Truncate the chain - contrastive divergence

• A few MCMC steps from the data, not equilibrium
• Gives up: unbiasedness
• The training rule of restricted Boltzmann machines

Match the score - score matching, diffusion models

• Fit \(\nabla_{\mathbf{x}} \log p\) - \(Z\) never appears
• Gives up: likelihood values

Bound the likelihood - variational autoencoders

• Latent variables + a tractable lower bound on \(\log p\)
• Gives up: exactness of the objective

Drop the density - generative adversarial networks

• A sampler trained against a learned discriminator - no explicit density
• Gives up: likelihoods entirely

Boltzmann machine (Ackley, Hinton, Sejnowski 1985)

• Binary units, every pair connected: \(E(\mathbf{x}) = -\mathbf{x}^\top \mathbf{J}\, \mathbf{x} - \mathbf{b}^\top \mathbf{x}\)
• Undirected graphical model: edge = direct pairwise interaction
• Gibbs sampling: one unit at a time, each conditioned on all its neighbors

The restriction

• Split the units: visible \(\mathbf{v}\) (the data) and hidden \(\mathbf{h}\) (latent)
• Delete every within-layer edge - “restricted” names exactly this deletion
• Bipartite graph: interactions between layers only

What a deleted edge means

• No edge → no direct interaction → conditional independence given the other layer
• Graph separation as the Markov property - on a graph instead of a chain
• Visible-visible structure: not lost - routed through shared hidden units

What the units are

• \(\mathbf{v}\) = the data vector, componentwise - binary MNIST: 784 visible units, \(v_i\) = pixel \(i\)
• \(\mathbf{h}\) = auxiliary variables - no pixel meaning, only induced dependencies among \(\mathbf{v}\)
• Trained: each hidden unit → a feature detector, a learned pattern over pixels

The data model is the marginal

• The same Boltzmann machinery, instantiated on pixels, hidden units integrated out
• Training: the two-phase gradient, on this graph
• Sampling an image: block Gibbs, on this graph

Role

• The first EBM trained at useful scale: MNIST features, deep belief networks (Hinton 2002-2006)
• Surviving in 2026: the concepts, not the model - CD → deep energy models, hidden units → latent variables

Bipartite energy

• Visible \(\mathbf{v} \in \{0,1\}^D\), hidden \(\mathbf{h} \in \{0,1\}^H\)
• Connections between layers only - none within

Fix \(\mathbf{v}\) - the energy regroups

• One term per \(h_j\), no cross terms - the deleted edges are the missing cross terms
• \(e^{-E}\) factorizes over \(j\) → each binary factor normalizes to a sigmoid

Conditionals

• \(p(\mathbf{h}|\mathbf{v}) = \prod_j p(h_j|\mathbf{v})\) - and symmetrically for \(\mathbf{v}\) given \(\mathbf{h}\)
• \(p(\mathbf{h}|\mathbf{v})\): exactly a sigmoid layer - a familiar network object

Block Gibbs sampling

• All of \(\mathbf{h}\) in one parallel draw, then all of \(\mathbf{v}\) - one matrix multiply each
• Unrestricted machine: \(h\)-\(h\) edges couple the hiddens - \(p(\mathbf{h}|\mathbf{v})\) does not factorize, Gibbs goes one unit at a time

CD-k

1. Start at data: \(\mathbf{v}^{(0)} = \mathbf{x}_{\text{data}}\)
2. Alternate: \(\mathbf{h}^{(t)} \sim p(\mathbf{h}|\mathbf{v}^{(t)})\), then \(\mathbf{v}^{(t+1)} \sim p(\mathbf{v}|\mathbf{h}^{(t)})\)
3. After \(k\) rounds: \((\mathbf{v}^{(k)}, \mathbf{h}^{(k)})\) = the stand-in model sample

Gradient estimate

• Visible-hidden correlation matching, truncated at \(k\) steps
• Bias: \(\langle \cdot \rangle_k \neq \langle \cdot \rangle_{\text{model}}\) for every finite \(k\)
• CD-1: the standard choice in practice (Hinton 2002)
• No burn-in, no equilibrium wait - \(k\) steps, total

Objective (Hinton 2002)

• \(p_k\): distribution after \(k\) Gibbs steps from the data
• Not maximum likelihood - the second term is the truncation

Why the gradient is tractable

• The \(\nabla_{\boldsymbol{\theta}} \log Z\) terms cancel between the two KLs - no equilibrium sampling anywhere
• Third term (\(\partial p_k / \partial \boldsymbol{\theta}\)): dropped - empirically negligible (Hinton 2002)

Consequences

• \(p_k\) starts at the data → data manifold learned quickly
• Between-mode regions: poorly modeled - spurious modes possible
• Persistent CD: chains continue across minibatches - less bias, more computation
• CD-1 ≈ score matching at the data (Hyvärinen 2007)

Fit the score, not the density

• Score: \(\mathbf{s}(\mathbf{x}; \boldsymbol{\theta}) = \nabla_{\mathbf{x}} \log p(\mathbf{x}; \boldsymbol{\theta})\)
• For an EBM: \(\mathbf{s}(\mathbf{x}) = -\nabla_{\mathbf{x}} E(\mathbf{x}; \boldsymbol{\theta})\) - no \(Z\) anywhere

\(\nabla_{\mathbf{x}} \log p_{\text{data}}\) is unknown

• Integration by parts removes it (Hyvärinen 2005):

• Data samples only - no model samples, no MCMC

What the trace term costs

• \(\text{tr}(\nabla_{\mathbf{x}} \mathbf{s})\): \(d\) Hessian-diagonal entries - one backward pass per dimension
• Images (\(d \sim 10^5\)): the objective itself becomes the bottleneck

Perturb, then regress

• Noisy copies: \(\tilde{\mathbf{x}} = \mathbf{x} + \boldsymbol{\epsilon}\), \(\;\boldsymbol{\epsilon} \sim \mathcal{N}(0, \sigma^2 \mathbf{I})\)
• Known conditional score: \(\nabla_{\tilde{\mathbf{x}}} \log p(\tilde{\mathbf{x}}|\mathbf{x}) = -\frac{\tilde{\mathbf{x}} - \mathbf{x}}{\sigma^2}\)

Equivalence (Vincent 2011)

• A regression target replaces the trace term - one forward pass per example

Denoising autoencoder connection

• Predicting \(\mathbf{x} - \tilde{\mathbf{x}}\) = learning the score: network output \(= \sigma^2\, \mathbf{s}(\tilde{\mathbf{x}})\)
• Denoising is score estimation

Basis of diffusion models

• Multiple noise levels → annealed score matching
• Learn \(\mathbf{s}(\mathbf{x}, t)\) across noise scales \(\sigma_t\) - the basis of DDPM and score-based models
• A learned score dropped into the Langevin walk = a sampler with no energy anywhere - how score-based models generate

Energy parameterizations

• Direct: \(E(\mathbf{x}; \boldsymbol{\theta}) = f_{\boldsymbol{\theta}}(\mathbf{x})\), a network with scalar output
• From a classifier: \(E(\mathbf{x}) = -\log \sum_y e^{f_{\boldsymbol{\theta}}(\mathbf{x}, y)}\) - logits define an energy

Training loop (sketch)

# inner loop: short-run Langevin from the replay buffer
x = buffer.sample()
for t in range(60):
    x = x - eps/2 * grad_x(E(x)) + sqrt(eps) * randn()

# outer loop: two-phase update
g = grad_E(x_data).mean() - grad_E(x).mean()
theta = theta - lr * g

Budget (Du and Mordatch 2019)

• 60 Langevin steps per negative sample, replay buffer of 10,000 images
• Memory: buffer + per-step sampling gradients - well above a discriminative model’s
• Chains nowhere near equilibrium - the CD trade at network scale

Persistent problems

• Mode coverage poor - short chains inherit the mixing limits
• Training unstable at high capacity
• Sampling stays expensive: 60+ gradient evaluations per image
• Evaluation without likelihoods: unresolved

Fit the field - denoising score matching

• Objective changed: likelihood abandoned, the score regressed
• No sampling anywhere in training
• Learns \(\nabla_{\mathbf{x}} \log p_\sigma\) - the noised marginals, at every scale \(\sigma\)
• Sampling deferred to generation time - the Langevin walk on the learned field

Follow the field - short-run Langevin, network energy

• Objective kept: the two-phase ML gradient, unchanged
• Samples on every training step - Langevin, truncated
• Replay buffer = persistent CD - short chains = the CD-\(k\) trade at network scale
• Sampling cost paid throughout training

Cost per update, at the published truncation

• 60 Langevin steps per negative sample (Du and Mordatch 2019)
• Each step: one forward + one backward → ~120 network evaluations per sample
• Discriminative update: 2 - a factor of ~60, before any bias correction
• Unbiased training: no finite budget - escape times scale as \(e^{\Delta E}\)

Memory

• Replay buffer: 10,000 persistent sample images
• Sampling gradients at every Langevin step, on top of the parameter gradients

Beyond computation

• High-dimensional spaces: almost entirely empty
• Modes separated by wide near-zero-probability regions
• Local moves cannot cross them - the landscape, not the hardware

Score matching → diffusion models

• Denoising score matching across noise scales
• DDPM and score-based models - the dominant image generators

Contrastive ideas → self-supervised learning

• InfoNCE ← noise-contrastive estimation, an EBM training method
• SimCLR, MoCo: positive pairs pushed down, negatives pushed up
• CLIP: a contrastive objective over image-text pairs

Around the intractability → the other families

• GANs: drop the density, keep the sampler
• VAEs: bound the likelihood
• Normalizing flows: architectures with \(Z = 1\) by construction

EBM: model \(p(\mathbf{x})\) explicitly

• Density up to \(Z\) - sampling by MCMC
• Dominant cost: the chain, every update

GAN: model the sampler directly

• A network \(G(\mathbf{z})\) = the sampler - one forward pass per sample
• No density, no \(Z\), no chain

What replaces the negative phase

• Discriminator: estimates model mass against data - a learned density ratio
• Push-up at model samples: survives, via the discriminator’s gradients

The trade

• Partition function: gone
• Sample generation: one pass
• New dominant cost: training stability
• Mode collapse replaces slow mixing

thispersondoesnotexist.com

• One StyleGAN2 face per reload - none of the people exist
• A family of single-model demo sites followed

{width=90%}

Peak numbers (FFHQ faces, 1024 × 1024)

• FID 2.84 (StyleGAN2)
• Generator: ~30M parameters
• One sample = one forward pass

{width=100%}

In 2026

• Frontier image synthesis: diffusion models
• GANs persist where one fast pass matters: real-time synthesis, super-resolution, distillation targets
• Adversarial losses: still inside modern generation pipelines

Architecture (Goodfellow et al. 2014)

• Fully connected throughout - ReLU and sigmoid in G, maxout in D
• Predates batch norm
• Trained with the original minimax objective

Reported results

• Parzen-window log-likelihood: 225 ± 2 (MNIST)
• Parzen evaluation: itself unreliable

What was new

• Implicit generation: no Markov chains, no inference network
• Adversarial training: a learned objective in place of a likelihood
• Evolution: DCGAN → Progressive GAN → StyleGAN

{width=100%}

Direction matters

• \(\text{KL}(p\|q) \neq \text{KL}(q\|p)\) - nonnegative, not symmetric, not a metric

Forward: \(\text{KL}(p_{\text{data}} \| q)\) - the maximum-likelihood direction

• Penalty where \(p_{\text{data}} > 0\), \(q \to 0\): unbounded - a missed mode
• Behavior: cover everything, spread thin if needed

Reverse: \(\text{KL}(q \| p_{\text{data}})\)

• Penalty where \(q > 0\), \(p_{\text{data}} \to 0\): unbounded - mass off the data
• Missing a data mode: cheap
• Behavior: concentrate on modes it keeps, drop the rest

No density → no likelihood objective - training becomes a game against a learned opponent

Zero-sum game

• Two players, one value \(V\): one maximizes it, the other minimizes it
• One side’s gain = the other side’s loss

Best response

• Optimal play against a fixed opponent strategy

Minimax

• \(\min_G \max_D V(D, G)\): \(G\) optimizes against \(D\)’s best response

Nash equilibrium

• Mutual best response - neither player improves by deviating alone
• At a saddle point of \(V\): minimum along \(G\)’s directions, maximum along \(D\)’s

For GANs

• Strategies = network weights \(\boldsymbol{\theta}_G\), \(\boldsymbol{\theta}_D\) - a continuous, non-convex game
• Clean statements hold in function space - networks approximate them

\(D\): an ordinary binary classifier

• Class 1: real data \(\mathbf{x} \sim p_{\text{data}}\) - class 0: samples \(G(\mathbf{z})\), \(\mathbf{z} \sim p(\mathbf{z})\)
• \(D(\mathbf{x})\) = predicted probability of “real”
• \(V\) = negative binary cross-entropy - a familiar training loss, maximized

\(G\): trained through the classifier

• No direct view of the data - gradient arrives only through \(D\)
• \(\mathbf{z} \sim \mathcal{N}(0, \mathbf{I})\), samples \(G(\mathbf{z})\) - the sampler-first model from the EBM handoff

Alternation

• \(D\) step: improve the classifier at fixed \(G\)
• \(G\) step: degrade the classifier’s job at fixed \(D\)

Fix \(G\), maximize over \(D\) pointwise

• Per point: maximize \(f(y) = a \log y + b \log(1-y)\), with \(a = p_{\text{data}}(\mathbf{x})\), \(b = p_g(\mathbf{x})\)

• Maximum confirmed: \(f''(y) = -a/y^2 - b/(1-y)^2 < 0\)

Reading the result

• \(D^*\) = a density-ratio estimate - likelihood information, extracted with no likelihood computed
• \(p_g = p_{\text{data}}\) → \(D^* = \tfrac{1}{2}\) everywhere: an undecidable classifier
• The contrastive push-down/push-up idea, as a classifier

Substitute \(D^*\) into the value

Jensen-Shannon divergence

• Symmetric, and \(0 \leq \text{JS} \leq \log 2\)
• Zero iff \(p = q\) - the minimax optimum is exactly \(p_g = p_{\text{data}}\)
• \(\log 2\) = the ceiling, reached when \(p\) and \(q\) share no mass

Saturation

• Two distributions that barely overlap: JS pinned near \(\log 2\), nearly flat in \(p_g\)

With separated modes, each KL reduces to a KL between the mixture weights

Removing a data mode - \(p_{\boldsymbol{\theta}}\)'s weight on one data mode \(\to 0\)

• Forward KL: grows without bound
• Reverse KL: \(\to \log 2\) - the weight cost \(\mathrm{KL}\!\left((1{-}w, w) \,\|\, (\tfrac{1}{2}, \tfrac{1}{2})\right)\) alone - zero from the missing region
• JS: \(\to 0.22\) - bounded

Adding a spurious mode - \(p_{\boldsymbol{\theta}}\) mass where \(p_{\text{data}}\) has none

• Reverse KL: explodes
• Forward KL: \(-\log(1 - v)\) - the true mode’s dilution alone - zero from the spurious region
• JS: bounded again

Objective vs behavior

• JS is the objective at optimal \(D\)
• The generator’s loss in use determines which divergence training follows

Early training: \(G\) poor → \(D\) separates easily → \(D(G(\mathbf{z})) \to 0\)

Differentiate in logit space - \(D = \sigma(a)\), gradients reach \(G\) through \(a\)

• Saturating loss (the minimax original):

• Non-saturating alternative, \(\max_G \log D\):

Same fixed points, opposite failure

• \(D(G(\mathbf{z})) \approx 0\): saturating gradient \(\approx 0\) - non-saturating \(\approx 1\)
• The loss-level view of the JS plateau: barely-overlapping distributions, flat objective
• Non-saturating: standard practice

Reverse KL, applied

• Spurious mass off the data: unbounded penalty - stay on modes
• A dropped mode: \(\log(1/w)\) nats - affordable
• Limited capacity → match a subset of modes exactly

Two named failures

• Mode dropping: a subset of modes missing, the rest sharp
• Mode collapse: outputs concentrate on one mode or point

The cycle in practice

1. \(G\) concentrates on one mode
2. \(D\) learns to reject that region
3. \(G\) jumps to another mode
4. Repeat - oscillation, not convergence

Equilibrium exists - in function space

• \(p_g = p_{\text{data}}\), \(D = \tfrac{1}{2}\) everywhere: a Nash point of the game
• Convergence proof (Goodfellow et al. 2014): convexity in function space, \(D\) optimal at every step
• Networks train in parameter space - non-convex, \(D\) never optimal

Simultaneous updates on the simplest saddle - \(V(g, d) = g \cdot d\)

• \(G\) descends, \(D\) ascends, simultaneously: \(\dot{g} = -d\), \(\;\dot{d} = g\)
• Continuous time: circles around the equilibrium - never approaching it
• Discrete steps: each update multiplies the radius by \(\sqrt{1 + \eta^2}\) - an outward spiral

Observed GAN training

• Losses oscillate - no reliable stopping signal in the loss values
• Stopping: sample-quality metrics, not loss
• Averaged generator weights (EMA): smooths the orbit’s iterates

Update ratios

• \(D\) steps per \(G\) step: a design choice - more \(D\) steps → closer to the optimal-\(D\) assumption

Targets (Salimans et al. 2016)

real_targets = 0.9   # smoothed
fake_targets = 0.0   # never smoothed

• Real side softened → D stops at 0.9 confidence - weaker saturating pressure

Why the fake side stays at zero

• Smoothed targets \(\alpha\) (real), \(\beta\) (fake) → optimal discriminator:

• \(\beta > 0\): where \(p_g \gg p_{\text{data}}\), \(D^* \approx \beta\) - spurious samples keep a nonzero score
• \(G\)’s current mistakes get rewarded exactly where they live
• \(\beta = 0\): spurious regions still driven to zero

One line per layer (Miyato et al. 2018)

• \(\sigma\) by one power-iteration step per update - negligible cost
• Every layer 1-Lipschitz + 1-Lipschitz activations → a network-wide slope cap

Placement

• All \(D\) layers (Miyato et al. 2018)
• \(G\) too, at scale (BigGAN)

Measured (CIFAR-10 unconditional, Inception Score)

• Weight clipping 6.41 → WGAN-GP 6.68 → layer norm 7.19 → spectral norm 7.42

Support - where a distribution puts its mass

Image data is thin in pixel space

• Pixel space: \(d \sim 10^5\) dimensions - valid images: a vanishing, structured subset
• Locally few directions of variation (pose, lighting, identity) - a low-dimensional manifold, embedded in \(\mathbb{R}^d\)

Generated data is thin by construction

• \(p_g\) = the image of \(\mathbf{z} \in \mathbb{R}^{100}\) under \(G\) - dimension ≤ 100, whatever \(G\) does

Two thin sets rarely meet (Arjovsky and Bottou 2017)

• Low-dimensional sets in high dimension: generically zero shared mass
• Disjoint supports → a perfect discriminator exists at every step
• JS pinned at \(\log 2\) → the saturation plateau is the generic case, not early-training bad luck

Reading the definition

• \(\Pi(p, q)\): joint distributions with marginals \(p\) and \(q\) - the transport plans
• \(\gamma(\mathbf{x}, \mathbf{y})\): how much mass moves from \(\mathbf{x}\) to \(\mathbf{y}\)
• \(W\) = cheapest total mass × distance - defined with or without overlap

Two point masses, separation \(\alpha\)

• \(\text{JS}(\delta_0 \| \delta_\alpha) = \log 2\) for every \(\alpha \neq 0\) - blind to how far
• \(W(\delta_0, \delta_\alpha) = \alpha\) - the distance itself

What that buys the generator - a slope toward the data even from far away

Kantorovich-Rubinstein duality

Lipschitz constraint - \(\|f\|_L \leq 1\):

• Slope bounded by 1, everywhere

What duality changes

• Primal: optimize over transport plans \(\gamma\) - intractable
• Dual: optimize over functions \(f\) - a neural network’s job
• \(f\) = the critic: real-valued scores, not probabilities

Why the cap is essential

• Unconstrained \(f\): send \(f \to +\infty\) on \(p\), \(-\infty\) on \(q\) - supremum diverges
• The bound ties the score difference to actual distance

Changes from the classifier game

• \(D\) outputs real values - no sigmoid, no saturation
• Critic, not discriminator - it scores, it does not classify
• The achieved value ≈ an estimate of \(W(p_{\text{data}}, p_g)\)

Training loop

for step in range(iters):
    for _ in range(n_critic):        # typically 5
        d_loss = -D(x_real).mean() + D(G(z)).mean()
        update(D, d_loss)
        enforce_lipschitz(D)          # clip or penalty
    g_loss = -D(G(z)).mean()
    update(G, g_loss)

• \(n_{\text{critic}} = 5\): run \(D\) near its supremum - the optimal-critic assumption, by iteration count

What the loss now means

• Critic loss ≈ \(-\hat{W}\) - it estimates a distance
• Correlates with sample quality during training (Arjovsky et al. 2017)
• A training curve worth reading - absent from the classifier game

What stays

• Alternating updates - still a game
• Mode behavior: improved coverage reported, not guaranteed
• The generator still learns only through the critic

WGAN-GP objective (Gulrajani et al. 2017)

• \(\hat{\mathbf{x}} = \epsilon\, \mathbf{x}_{\text{real}} + (1 - \epsilon)\, \mathbf{x}_{\text{fake}}\), \(\;\epsilon \sim U[0, 1]\) - the segment between the supports

Why target \(\|\nabla D\| = 1\)

• Optimal critics have unit gradient norm almost everywhere on the transport paths (Gulrajani et al. 2017, Prop. 1)
• A soft constraint at sampled points - the function class stays intact

Cost and settings

• One extra backward pass - the penalty differentiates a gradient
• \(\lambda = 10\) default (Gulrajani et al. 2017)
• Alternative enforcement: spectral normalization - the layerwise cap from stabilization

eps = torch.rand(b, 1, 1, 1)
x_hat = eps * x_real + (1 - eps) * x_fake
x_hat.requires_grad_(True)

d_hat = D(x_hat)
grad = autograd.grad(d_hat.sum(), x_hat,
                     create_graph=True)[0]
gnorm = grad.view(b, -1).norm(2, dim=1)
gp = lam * ((gnorm - 1) ** 2).mean()

Against clipping

• No boundary pileup - weights free
• Critic surfaces keep their structure
• Stable across tested architectures (Gulrajani et al. 2017)