Optimisation finds the best estimate. Inference asks how much to trust it.
In the optimisation section, likelihood plus data defined a surface over parameter space, and algorithms — or your own sliders — climbed it to find the peak. The coordinates of that summit are the maximum likelihood estimates: the single best configuration of parameter values.
But a point estimate on its own is an answer without a confidence attached. Two studies can agree exactly on where the summit is and still deserve very different levels of trust: one climbed a needle-sharp pinnacle, the other wandered onto a broad, flat plateau where hundreds of nearby parameter values explain the data almost as well.
The new question. Optimisation asked “where is the top?” Inference asks “how quickly does the surface fall away from the top?” The location of the peak gives the estimate; the shape of the peak gives the uncertainty.
Below is the log-likelihood curve for a single parameter — the mean resting heart rate $\mu$ in a synthetic sample (the same dataset used throughout this section). Drag the slider to change how many observations the model sees. Watch what happens to the curve, and to the interval beneath it.
The dashed horizontal line sits 1.92 log-likelihood units below the peak. The stretch of curve above that line — projected down to the axis as the purple bar — is the 95% interval. A sharper curve crosses the line sooner, so the bar shrinks.
The quantity doing the work here is the second derivative of the log-likelihood at its peak — the instantaneous curvature. At a maximum it is negative (the surface bends downwards), and its negation is called the observed information:
$$I(\hat\theta) = -\,\ell''(\hat\theta) \qquad\qquad \text{Var}(\hat\theta) \approx \frac{1}{I(\hat\theta)} \qquad\qquad \text{SE}(\hat\theta) = \frac{1}{\sqrt{I(\hat\theta)}}$$
Uncertainty is inversely related to information. A flat peak — small curvature, little information — means many parameter values fit the data nearly as well, so intervals are wide. A sharp peak — large curvature, lots of information — means the data strongly favour one value, so intervals are narrow.
And because each independent observation adds its own contribution to the log-likelihood, curvatures accumulate: information grows in proportion to $n$, so the standard error shrinks in proportion to $1/\sqrt{n}$. That is exactly the sharpening you watched above — for this model, $\ell''(\hat\mu) = -n/\hat\sigma^2$.
Everything on this page was one-dimensional. Real models have many parameters, and their likelihood surface can fall away at different rates in different directions — and the directions can interact. The next two pages develop two complementary ways of turning the shape of a multi-parameter peak into uncertainty statements.
Measure the sharpness of the summit in every direction at once with the
Hessian matrix, then invert it to get standard errors and a confidence
ellipse. Fast, and exactly what R and Python's summary() tables do —
but it approximates the peak by an idealised quadratic shape.
Don't assume a shape — walk the surface and record where you spend your time, as the MCMC chains did on Arthur's Seat. The visit record captures joint uncertainty of any shape, at the price of more computation.
Bayesian uncertainty →Once standard errors exist, hypothesis tests follow: Wald tests, likelihood-ratio tests and model comparison (AIC/BIC), plus assumption-free resampling methods (the bootstrap). Planned future additions to this section.
You have — twice, implicitly. Every tutorial's summary() output contains a
Std. Error column sitting right next to the coefficients: those numbers are
computed from the curvature described here. And in
Multi-Optima, an unexplained ellipse was
stamped around the MLE in the “representativeness” comparison. Both get their
explanations on the next page.