Sampling · Optimization · Geometry

Sampling from Manifolds of Minimizers

From loss and gradient evaluations to sampling an entire solution set.

Saeed Masiha18 October 2026Interactive explainer

Suppose a smooth loss g(x) has a curved set of minimizers. We can evaluate the loss and its gradient, but cannot parametrize the set or project onto it. How can we sample a distribution that explores the whole solution set?

Sampling settings that work for one loss may deteriorate as the loss changes. Can one sampling prescription work across an entire family?

We start with Gibbs concentration and Langevin sampling. Our Superquantile–Gibbs (SQ-G) paper [1] establishes uniform guarantees: under its structural, regularity and tail assumptions, common choices of temperature, step size and run length control concentration and sampling errors across the family.

The minimizer set and local growth.

S = \underset{x \in R^{d}}{arg min} g (x), g^{⋆} = \min_{x} g (x)

Every point in S has the same loss. Local quadratic growth describes how loss rises away from the set:

g (x) - g^{⋆} \geq \frac{m}{2} {dist}^{2} (x, S) near S.

Here m > 0; dist(x, S) is the distance to the nearest minimizer. Nearby, small excess loss means proximity to S. It does not explain how to explore the set.

Where local PL enters

The local Polyak–Łojasiewicz (PL) condition relates the gradient to excess loss:

\frac{1}{2} {‖ \nabla g (x) ‖}^{2} \geq α (g (x) - g^{⋆})

For α > 0 near S, local PL prevents the gradient from being too small relative to excess loss. Together with the paper’s smoothness, tail and global structural assumptions, it yields a compact, connected C² minimizer manifold with positive-definite loss Hessian in normal directions.

This geometry supports a Gibbs limit and concentration near S, alongside low-temperature Poincaré control for Langevin mixing. ULA additionally needs the step-size and initialization conditions discussed below.

Cool the distribution toward the valley.

At temperature λ > 0, the Gibbs law assigns more density to lower loss:

μ^{λ} (dx) = \frac{1}{Z_{λ}} e^{- (g (x) - g^{⋆}) / λ} dx

Z_λ normalizes the density. Decrease λ below: probability gathers in a thinner band around the minimizers.

Watch Gibbs approach its limiting law

A two-dimensional Gibbs density: as λ decreases, probability lies in a thinner band around the minimizer set.

Dashed curve: exact minimizers S

Probability density per unit area

λ = 0.300. The color scale stays fixed as temperature decreases.

What the colors and limit mean

At positive temperature, the color shows the normalized density $ρ^{λ} (x) = e^{- g (x) / λ} / Z_{λ}$ . Its integral over the whole plane is one. The fixed nonlinear color scale reveals both the broad warm distribution and the narrow cold band; each frame uses the same scale.

At λ = 0, the limiting law lives on S and has no density per unit area. The plot switches to color along the curve, with a separate square-root color scale for density per unit curve length, keeping low-density stretches visible. The nonlinear bend changes how quickly the curve is traced, so some parts receive more probability per unit length than others.

The toy loss is $g = \frac{{(q - 1)}^{2}}{1 + q}$ , where $q = {(\frac{x_{1}}{a})}^{2} + {(\frac{x_{2} - s (x_{1})}{b})}^{2}$ . Here a = 1.45 and b = 0.7. The wavy loop uses s(t) = 0.5 sin(2t); the bent loop uses s(t) = 0.55[1 − cos(1.7t)] − 0.4. The solution set is {q = 1}.

This smooth coordinate change has a constant Jacobian, so a one-dimensional radial integral gives the whole-plane normalization. The fixed window contains over 99.9% of the probability throughout the displayed temperatures; the image is not renormalized inside it.

These examples illustrate the Gibbs limit directly. The sampling experiments below use their separately specified losses and assumptions.

Peak area density: 0.223
Upper bound on W₁ to the limit: 0.408

The bound multiplies a reference same-angle transport cost by the bend’s Lipschitz factor. It uses numerical radial quadrature and need not be sharp; it is not an exact W₁ calculation.

Static Gibbs density at λ = 0.300. Enable JavaScript to decrease λ or view the limiting law.

As λ decreases, the band narrows around S. In the limit, all probability lies on the curve; it does not collapse to a single minimizer.

A quantitative guarantee.

For fixed losses satisfying SQ-G’s geometric, smoothness and tail conditions, the Gibbs law approaches a law μ⁰ supported on S, with

W_{1} (μ^{λ}, μ^{0}) \leq K \sqrt{λ}

For sufficiently small λ. W₁ is the minimum average distance needed to transport one probability law into another.

Conditions and references for this bound

SQ-G, Proposition 2.10, uses the paper’s PL^∘ condition, a bounded second fundamental form of S, and specified smoothness and tail bounds.

The rate uses Assumption 2.6 and Assumption 2.7, together with four clauses of Assumption 2.8: a C² extension, a globally Lipschitz lower-level gradient, a continuous Hessian, and the uniform tail gradient error bound.

The full PL^∘ condition in Definition 2.2 also includes C⁴ regularity near local minima, boundedness of the set of local minima, and a negative-definite Hessian at critical points outside the local PL neighborhoods.

The Laplace/Wasserstein reference used in SQ-G is Hasenpflug, Rudolf and Sprungk (2024), Theorem 3.6 [2]. SQ-G adapts their argument to its geometric assumptions: Proposition 2.9 identifies the limit and Proposition 2.10 gives the fixed-loss W₁ bound.

Which distribution lives on the minimizer set?

An ellipse makes the next comparison easy to see: move the same distance away from two minimizers and compare the loss.

The limiting law need not be uniform along the set. Where the loss rises more gently as we leave the set, more nearby points have low loss. These regions receive more limiting probability per unit of manifold volume.

Same distance away. Different rise in loss.

Start at two minimizers, A and B. Move both points the same distance perpendicular to the curve.

Curve = solution set S · every point on it has loss 0.

Both points: 0.40 units from S.

Increase in loss · same scale, 0–0.50

A · right tip 0.161

B · top tip 0.481

The same step raises the loss more at B. This steeper rise perpendicular to the solution set is what “greater normal stiffness” means.

“Normal stiffness” means how sharply loss rises perpendicular to the set. A gentler rise leaves more room for low-loss points nearby, giving more limiting probability per unit of arc length. Values come from the actual toy loss.

\frac{d μ^{0}}{d {vol}_{S}} (x) \propto {\det (H_{N} g (x))}^{- 1 / 2}

H_Ng is the Hessian restricted to normal directions. It measures how sharply the loss rises perpendicular to the set. This differs from the curvature of the set itself. In this toy ellipse, the limit is uniform in the ellipse’s parametric angle, not in arc length.

Now sample the Gibbs law.

Hold λ fixed. The unadjusted Langevin algorithm (ULA) combines a gradient step with Gaussian noise:

X_{k + 1} = X_{k} - h \nabla g (X_{k}) + \sqrt{2 h λ} ξ_{k}

h is the step size; ξ_k are independent standard Gaussian vectors. The gradient pulls toward low loss, while noise lets samples explore. The update needs no parametrization of S, projection or normalizing constant.

Reaching the solution set is only the first step.

Samples can get close to S while remaining clustered in one region. To approximate the Gibbs law, they must also explore other regions in the right proportions. A Poincaré inequality helps control how quickly the underlying Langevin diffusion approaches this distribution:

{Var}_{μ^{λ}} (φ) \leq C_{PI} \int {‖ \nabla φ ‖}^{2} d μ^{λ}

For a smooth test function φ, this compares variance with average squared gradient. At fixed λ, a smaller C_PI gives a stronger relaxation bound for the diffusion.

Yun Gong, Niao He and Zebang Shen [3] establish a temperature-independent bound on C_PI for sufficiently small λ in their local PL setting, with additional regularity, tail and geometric assumptions. Their result relates this bound to the minimizer manifold’s spectral gap.

ULA also needs smoothness, controlled initialization and jointly chosen sampling parameters; a fixed step leaves discretization error. The experiments use ellipse/circle losses with globally Lipschitz gradients, distinct from the opening Gibbs examples.

Watch ULA approach a fixed Gibbs target

The density field shows where the Gibbs target places probability. Watch actual ULA chains approach and explore this region.

λ = 0.06 · h = 0.010 · 24,000 steps per run · 256 chains

Static endpoint

Samples against their Gibbs target

ULA samplesMinimizer set6 recent paths

Target density

Static result after 24,000 ULA steps at λ = 0.06. Enable JavaScript to restart and watch the run.

Sampling diagnostics and reference cloud

Coarse discrepancy from Gibbs0.129Independent Gibbs cloud (finite-sample fluctuations): 0.141

Mean distance to the solution set0.102Proximity and distributional agreement are different.

Coarse discrepancy over ULA steps

Independent Gibbs reference cloud

Direct reference draws from the numerical Gibbs law, used for the finite-sample comparison below. These points are not ULA output.

The shading is probability density per unit area, on a fixed nonlinear scale shared across θ and schedules. The density is normalized on the whole plane, not rescaled to the visible window.

The binned discrepancy is half the sum of absolute differences between empirical frequencies and Gibbs probabilities in 32 regions: four radial bins and eight angular sectors.

Independent Gibbs draws also have a nonzero score because the cloud is finite. A sampler can score below this particular reference cloud without establishing better sampling or smaller full-distribution error. The reference score is neither an error floor nor an acceptance threshold.

Mean distance to the solution set measures proximity. The binned score measures agreement across these regions; it does not certify convergence of the full distribution.

Dots follow gradient-plus-noise updates. Six short paths connect recorded positions every ten updates; they are not smoothed trajectories. The blue field is the independently normalized Gibbs density, not a density estimate from these particles. A cooler schedule uses smaller steps and more iterations to bring samples closer to the set. These illustrative schedules do not certify a target accuracy.

The sufficient conditions and the cooling tradeoff

ULA needs gradients. Function values alone require an additional gradient estimator, which is outside this illustration.

SQ-G, Proposition 5.6, uses a Lipschitz gradient for G = g/λ, a Poincaré bound, and finite order-three Rényi divergence of the initial law relative to the Gibbs target. It jointly chooses a horizon and step to control order-two Rényi error, which yields a W₁ bound using Poincaré control and finite second moments.

The theorem’s step for G is λh in our convention. It does not justify arbitrary initialization or indefinitely increasing the iteration count at any fixed step.

For the corresponding continuous Langevin diffusion, finite initial χ² divergence gives decay by a factor exp(−2λt/C_PI). With temperature-independent Poincaré control, the bound gives a timescale proportional to 1/λ, apart from initialization and accuracy factors. Cooling sharpens the target but slows exploration; this exact diffusion rate is not a finite-step ULA rate.

Tune two errors, not one.

Let ν_n,h,λ be the law after n ULA steps. The triangle inequality separates the computational and temperature errors:

W_{1} (ν_{n, h, λ}, μ^{0}) \leq W_{1} (ν_{n, h, λ}, μ^{λ}) + K \sqrt{λ}

Temperature: choose λ ≤ (ε/2K)² within the theorem’s low-temperature range, so the second term is at most ε/2.
Sampling: choose an admissible initialization, step size h and count n together, so the first term is at most ε/2.

The endpoint law is then within ε of μ⁰ in W₁; in particular, its expected distance to S is at most ε. With the paper’s controlled Gaussian initialization, the bound is polynomial in inverse target accuracy, with dimension- and problem-dependent constants. The example schedules above illustrate the tradeoff; they are not certified ε-budgets or guarantees for arbitrary nonconvex losses.

One sampling prescription for every θ.

Return to the opening question: settings that work for one loss may deteriorate for another. The constants can grow, or the temperature range where a bound holds can shrink.

Uniform control means common bounds and parameter choices work across the whole parameter set. This concerns the guarantees; the limiting law can still give different weights to different minimizers.

Let θ range over a compact parameter set Θ, and define:

S (θ) = \underset{x}{arg min} g (θ, x)

For each θ, μ_θ^λ is the Gibbs law and μ_θ⁰ its limit on S(θ). SQ-G establishes the common controls below under parameter-uniform assumptions.

What must hold uniformly?

The local PL condition now reads ½‖∇_xg(θ, x)‖² ≥ α(g(θ, x) − g^⋆(θ)), with a common positive α in the controlled neighborhoods of S(θ). Here g^⋆(θ) is the minimum at that parameter.

SQ-G assumes this parameter-uniform local PL structure, smoothness and tail control over compact Θ. Its uniform Poincaré result concerns positive-dimensional, compact connected minimizer manifolds. The proof establishes common geometric neighborhoods and spectral bounds; a pointwise PL or Poincaré statement alone is insufficient. Assumptions 2.6–2.8 and the uniform Poincaré proof.

Sampling also requires controlled initialization and a jointly chosen step size and run length, as specified in the sampling conditions. The example schedules illustrate the mechanism; they are not theorem-calibrated accuracy guarantees.

First: uniform Gibbs concentration.

SQ-G, Proposition 3.10, gives a common constant K and a common positive temperature threshold:

sup_{θ \in Θ} W_{1} (μ_{θ}^{λ}, μ_{θ}^{0}) \leq K \sqrt{λ}

One temperature choice controls the entire family. Within the common temperature range, the same K√λ bound applies to every θ.

Second: uniform sampling control.

Proposition 3.11 gives one finite inverse Poincaré bound and one low-temperature window valid across Θ:

C_{PI} (θ, λ) \leq C_{PI}^{⋆} \forall θ \in Θ

Together with uniform smoothness and initialization bounds, this allows a common choice of ULA step size and run length at a given temperature and accuracy. SQ-G’s sampling construction makes these choices explicit.

A family of targets, shared sampling settings

Try two values of θ. Choose θ = 0, restart, and run ULA to the displayed step count. Repeat at θ = 1 with the same schedule. Watch the samples explore each target while the sampling settings stay fixed.

Ellipse · shared across θ · λ = 0.06 · h = 0.010 · 24,000 steps per run · 256 chains

Static endpoint

Current target · θ = 0.50

One slice of the family

Each curve is S(θ) at one parameter value. The blue slice is the target shown in the sampling view.

ULA samplesMinimizer set6 recent paths

Target density

Reference cloud and sampling diagnostics

Independent Gibbs reference cloud

Direct reference draws from the numerical Gibbs law, used for the finite-sample comparison below. These points are not ULA output.

Coarse discrepancy from Gibbs0.129Independent Gibbs cloud (finite-sample fluctuations): 0.141

Mean distance to the solution set0.102Proximity and distributional agreement are different.

Coarse discrepancy over ULA steps

These are the same proximity and binned-discrepancy diagnostics as in the fixed-loss example. How to interpret the scores and finite-sample fluctuations.

Static result after 24,000 ULA steps at λ = 0.06. Enable JavaScript to restart and watch the run.

Keep the displayed temperature, step size and run length fixed when comparing θ values. The density field and the family portrait are computed from the same loss. Changing θ preserves the particles and noise stream; short paths restart at that change. These illustrative schedules do not certify an accuracy budget or guarantee tracking of a time-varying target.

The takeaway is uniform control of both errors. Choose λ to control concentration for every θ, then choose h and n to control sampling for every θ:

sup_{θ \in Θ} W_{1} (ν_{n, h, λ}^{θ}, μ_{θ}^{0}) \leq ε

The guarantee concerns runs at each fixed θ with admissible initialization and sampling parameters. Tracking a continuously changing target is a different problem.

Sampling supplies the candidates.

In bilevel learning, θ can represent training-data weights and x a model. Sampling lets us examine different training minimizers using validation loss. Select-then-differentiate [4] pursues optimistic selection; SQ-G uses an upper-tail average to approximate pessimistic selection.

The previous article explains the data-reweighting application and LLM experiments. Those use practical optimizer branches; this article’s simulations isolate the mathematical sampling problem.

The toy losses and what the figures compute

The opening Gibbs animation uses wavy and bent loops. A smooth shear of an ellipse defines each loss; its density and limiting law are computed directly. These examples illustrate Gibbs concentration and are not used for the ULA simulations.

The normal-rise illustration and ULA experiments use g_θ(x) = (q_θ(x) − r_θ²)²/(1 + q_θ(x)), where q_θ is a positive quadratic form around a translated center. Their minimizers form ellipses or circles; θ changes the center and radius. The fixed-loss examples hold θ = 0.5. This separate family has globally Lipschitz gradients; the sheared examples need not.

Numerical quadrature supplies the Gibbs densities and reference clouds. ULA uses analytic gradients and seeded Gaussian noise, without projection or a normalizing constant. Known geometry is used only for drawing and diagnostics. These illustrative computations do not certify a theorem’s full-distribution error bound.

References

Saeed Masiha, Zebang Shen, Negar Kiyavash, and Niao He. Superquantile-Gibbs Relaxation for Minima-selection in Bilevel Optimization. Preprint, arXiv:2505.05991, 2025 (v5, 2026).
Mareike Hasenpflug, Daniel Rudolf, and Björn Sprungk. Wasserstein convergence rates of increasingly concentrating probability measures. The Annals of Applied Probability 34(3):3320–3347, 2024. Open preprint.
Yun Gong, Niao He, and Zebang Shen. Poincare Inequality for Local Log-Polyak-Łojasiewicz Measures: Non-asymptotic Analysis in Low-temperature Regime. Preprint, arXiv:2501.00429v2, 2025.
Saeed Masiha, Zebang Shen, Negar Kiyavash, and Niao He. Select-then-differentiate: Solving Bilevel Optimization with Manifold Lower-level Solution Sets. NeurIPS, 2026.

Cite this post

@misc{masiha2026samplingminimizers,
  author = {Masiha, Saeed},
  title = {{Sampling from Manifolds of Minimizers}},
  year = {2026},
  month = oct,
  howpublished = {Saeed Masiha's Research Blog},
  url = {https://smasiha.github.io/writing/sampling-hidden-solutions/},
  note = {Published 18 October 2026}
}

← Research Blog