22Bias–Variance & Regularization

Chapter 21 left us holding a thermometer and no medicine. We can measure the gap between fit and prediction, and we watched the test error fall, turn and climb. What we cannot do is say why the right arm rises, in parts we could attack one at a time. This chapter builds those parts, and the sentence it turns on is one nobody says out loud. Your fitted model is not a thing you built. It is a draw you got. Sample another year from the same world, run the identical code, and a different function comes out. Once you can see that spray of functions, the error at a point you have never visited splits into exactly three pieces: how far the average of those functions sits from the truth, how widely they scatter around that average, and the noise nobody can touch. Only the first two are yours, and they move in opposite directions as you hand the fit more freedom. So the whole craft is a purchase. You spend a little bias to buy back a lot of variance. Regularization is that purchase made deliberately, on a dial rather than a switch.

Look at what this page stands on, because almost all of it is already yours. Chapter 21 gave us the law that fitting is searching, and the toll that comes with it: the optimism you are charged is set by the size of the search. It gave us the U-curve, watched rather than explained. It gave us the p ≫ n regime and named three exits from it. Chapter 20 gave us Var(β̂) = σ²(XᵀX)⁻¹, the collinearity canyon and the bootstrap. Chapter 16 gave us the sampling distribution, the identity MSE = Var + bias², and the add-and-subtract trick behind Bessel's correction. Chapter 8 gave us tangency, the Lagrange multiplier and its reading as a shadow price. Chapter 7 gave us the SVD and the condition number. That is the whole toolkit, and this chapter is mostly a matter of pointing it in one direction.

So the seam is short. Chapter 21 ended by saying, in its own words, that it had shown the U-curve as a phenomenon without ever taking it apart. It then named the door: constrain, reduce or select. Walking through a door the previous chapter labelled costs us nothing new.

01A diagnosis with no treatment

Two things are missing from what Chapter 21 handed over, and both of them are load-bearing.

The first is a mechanism in parts. Show me a model that is predicting badly, and I cannot yet tell you whether it is failing because it is too rigid to hold the shape of the truth, or too free to ignore the noise. That distinction is not academic, because the two diseases need opposite treatments. Do the wrong one and you walk further from the floor rather than toward it.

Here are two patients, and they are as different as two models can be. The first is a straight line fitted to sixty rows whose truth is genuinely curved. The second is a degree-nine polynomial fitted to twelve rows whose truth is genuinely a straight line. Score both on fresh data and they come back at 0.174 and 0.175. The same number, to two decimals.

Run each one four thousand times and the long-run split is not remotely the same. The line's error is 0.1011 of one kind plus 0.0020 of the other, sitting on a noise floor of 0.0625. The polynomial's is 0.0000 and 0.0793 on the identical floor. Strip the floor away and the line's remaining error is 98% one thing, while the polynomial's is essentially all the other. We have no names for those two piles yet. That is the point.

The one reading Chapter 21 does give us is the gap between training and test error, and it is worth being precise about what that buys. The polynomial trains at 0.002 and tests at 0.175, which is a scream. The line trains at 0.213 and tests at 0.174, which is silence. So the gap does flag the second patient. What it cannot tell you is how much of the line's error is fixable at all, or by how much, or which knob to turn on either one.

The second missing thing is the lever. Everything Chapter 21 left us is a row of on/off switches: which features go in. That sounds like the sober option, and the sober-sounding version of it is best-subset selection. Try every subset, cross-validate each one, keep the winner.

Count the candidates before you agree to that. With ten features there are 1,024 subsets. With twenty there are 1,048,576. With thirty there are over a billion, and with five hundred the count has 151 digits. Now set Chapter 21's own law beside those numbers: the optimism you are charged is set by the size of the search, and Chapter 19 measured what the best of m noisy tries looks like. The careful-sounding remedy is the largest winner's curse in the entire course.

Which hands us the chapter's programme in one sentence. If the toll is the size of the search, then shrink the search. Not the feature list. The search.

02Aiming at a row that does not exist yet

Before we can break the error into parts we have to define it, and we should define it where it actually pays. Two moves get us there, one cosmetic and one substantive.

The cosmetic one is vocabulary, and it is worth doing quickly so it stops being a barrier. What Chapter 20 called predictors are features. The target y is the label. Fitting is training, the β̂ are the learned parameters, and RSS is the loss. Ordinary least squares, entirely unchanged, is already a supervised learning algorithm. The costume is new and the machine underneath is the one you built.

The substantive move is that the question changes. Chapter 20 asked what β is. We are now asking what number to write on a row that is not in the dataset and has not happened yet. So write the world's law without assuming a straight line, because the decomposition does not need one: y = f(x) + ε, where ε has mean zero, variance σ² and no relationship with anything else. Then the quantity we want small is the prediction error at that new row, Err(x₀) = E[(y₀ − f̂(x₀))²].

Now notice what we are aiming at, because it is not y₀. That value carries a fresh ε that no model, no data and no amount of cleverness can reach. The best prediction that could possibly exist is f(x₀) = E[y|x₀], the conditional mean. Everything beyond it is a coin nobody can see. So even a perfect model eats σ², and that is not a footnote. It is the ceiling on the whole enterprise.

Price that ceiling with a number Chapter 20 already made you accept. A daily-return R² near 0.01 is normal in this business. Read it literally: one per cent of the variance of tomorrow's return is explainable and ninety-nine per cent is the floor. Put it in standard deviations and it is starker still. A model with an R² of 0.01 cuts the spread of what you are predicting from 1.0000 to 0.9950. That half of one per cent is the entire industry.

One bookkeeping note before we move, because the word error now means three different things in three different chapters. There is the residual e, which you compute and which sits at a right angle to the columns by construction. There is the population error ε, which you never see, and which Chapter 20 spent a whole section separating from the residual. And now there is prediction error at a new point, which is an expectation over things that have not happened yet. Keep them apart and the rest of this chapter is easy.

03★★ Your model is a draw, not a decision

Here is the rung the whole chapter pivots on, and it is the one nearly every treatment sprints past.

Chapter 16 proved that a statistic computed from a random sample is itself a random variable, and that your estimate is one dot in a cloud you will never observe. Now read that one level up. β̂ is a statistic. The fitted function f̂ = Xβ̂ is built out of that statistic by arithmetic. So the entire fitted curve is a random object, and f̂(x₀) — one prediction, at one fixed point, from your one model — is a random variable with a distribution.

Say out loud what that distribution is over, because this is the exact inch where readers are lost. It is not over the rows of your dataset. You have spent five chapters averaging over rows, and the notation E[f̂(x₀)] arrives with no subscript to warn you. The randomness being averaged over is which dataset you got.

Before the picture, commit to a number, because the guess is what makes this stick. Twelve data points sit on a gentle curve with noise of size σ = 0.10. That is small, and it looks small: the dots hug the curve. Fit a degree-eleven polynomial, which has twelve coefficients and therefore threads all twelve points exactly. Now I draw nineteen more datasets from the same truth with the same noise, refit the identical model to each, and read all twenty predictions at the single point x₀ = 0.93. How far apart will those twenty numbers be?

Almost everyone answers in the neighbourhood of the noise. The data only wobbles by a tenth, so the answers should wobble by a tenth or two. The twenty predictions run from −2.380 to +1.593. That is a span of 3.974, on a truth whose value there is −0.2129. A wobble of 0.10 in the data came out as a wobble of nearly 4 in the answer, and the model performed that amplification entirely on its own. The reason is mechanical: a curve forced through every point must swing hard between them, so a small vertical nudge at one point becomes a large excursion at the next. Over twenty thousand refits the standard deviation settles at 1.343, which is 13.4 times the noise it was fitted to.

Overlay twenty straight-line fits of the same twenty datasets and you get the opposite object. They span 0.262, so tight you can barely tell them apart, and every one of them misses the same way. The polynomial's average is right and its spread is enormous. The line's spread is nearly nothing and its average is wrong by 0.148. Those two facts have names. The offset of the centre is the bias. The width around that centre is the variance. Both are errors you pay.

And now the stroke, which is one sentence. Neither of those pictures is available to you. You hold exactly one curve. From the inside, a single wild polynomial and a single straight line look equally reasonable, because both pass near the data, both plot fine, and both come with a coefficient table.

That is the invisibility at the centre of this subject. The width of the spray is an error you are paying and cannot see from inside your own dataset, and it is the entire reason the right arm of the U-curve goes up. It is also, I think, the honest reason backtests feel so persuasive. You are looking at one draw and reading it as a decision.

04Three lines and two dead cross-terms

Now the algebra, and it uses a move your own hands have already performed.

Start from Err(x₀) = E[(y₀ − f̂(x₀))²] and substitute y₀ = f(x₀) + ε. That ε belongs to the new observation, so it is independent of the training set that produced f̂, and it has mean zero. Its cross-term dies and it contributes exactly σ². What is left is E[(f(x₀) − f̂(x₀))²].

Here we run Chapter 16's trick a second time. Add and subtract E[f̂(x₀)] inside the square, then expand. This is the same add-and-subtract that produced Bessel's correction, and it behaves the same way.

The cross-term is where every textbook waves, so let us not. Write m = E[f̂(x₀)] as a single letter first. That m is a plain number: the expectation has already been taken, and there is no randomness left inside it. The cross-term is then (m − f(x₀)) × E[f̂(x₀) − m]. The second factor is zero by the definition of m, and the first is a constant, so the product is zero and no assumption was needed anywhere.

Three terms survive, and they are the chapter: Err(x₀) = σ² + (E[f̂(x₀)] − f(x₀))² + E[(f̂(x₀) − E[f̂(x₀)])²]. In words, noise plus bias squared plus variance. Look back at Chapter 16 and you will see the same identity one level down. MSE = Var + bias² had no σ² term because there was no fresh observation to predict, only a fixed θ to estimate.

Check it against the spray rather than trusting it. At x₀ = 0.93 the degree-eleven fit has bias 0.0000 and variance 1.8046, on a floor of 0.0100. So the formula predicts a prediction error of 1.8146 at that point. Draw twenty thousand fresh observations at that point and the measured mean squared error is 1.7885, against 1.7883 from the formula on the same draws. The identity is not a story about the error. It is the error.

Two things to say plainly before moving on. The noise term is not spendable: test error can never go below σ², so a model reporting error under the floor is reporting a bug. And the decomposition is pointwise. It holds at x₀, and both bias and variance change as x₀ moves, so the single test number you report is this decomposition averaged over the rows you expect to see.

05Taking the U apart

Re-run Chapter 21's complexity sweep, and this time plot the three parts instead of the total.

Bias squared falls as freedom rises, because a more flexible family can on average get closer to the true f. Variance rises, because every extra direction the fit is allowed to use is one more thing that can chase this particular sample's noise, and Chapter 21 already priced that at k/n. The noise floor is flat, because nothing you do to a model touches it.

On our running curve the numbers are these. At degree 0 the bias squared is 0.1351 and the variance is 0.0008. At degree 3 they are 0.0011 and 0.0027, which is the floor of the total at 0.0138. By degree 11 the bias squared is 0.0001 and the variance has climbed to 0.2812. So the U is not mysterious at all. It is a falling curve plus a rising curve sitting on a constant floor, and its minimum is simply where the two slopes cancel.

Both arms now have mechanisms rather than labels. Underfitting is bias-dominated: the model is too rigid to hold the structure, so it is wrong the same way every time. Overfitting is variance-dominated: the model is free enough to fit this sample's noise, so it is wrong a different way every time. Our two patients from the opening are exactly one of each, and now they are separable.

Which means we have to deal with the word bias before it detonates, because you have met it twice and both times it was a defect.

Chapter 16's estimator bias was a flaw to be corrected, and Bessel's correction was the fix. Chapter 20's omitted-variable bias was the unforgivable one: the failure that aims your coefficient at a different quantity altogether, that no computational repair reaches, and that does not shrink with n. Later in this chapter we are going to add bias on purpose. Unless the two are separated by name, right here, you will either refuse the trade as obviously wrong or decide Chapter 20 was being dramatic.

The distinction is sharp. Chapter 20's bias is an identification failure: it buys you nothing, you cannot dial it, and more data does not touch it. This chapter's bias is a chosen, quantified, reversible shift, purchased in exchange for variance, and you can set it back to zero whenever you like. Same word, opposite moral standing.

Two smaller corrections ride along, and they point in opposite directions. Chapter 21 may have left you with "simpler is safer", and the left arm of this curve is a picture of rigidity losing money. And you will now go looking for a function that returns bias and variance for your own model. It does not exist. Both quantities are defined over the datasets you did not get, so the decomposition is a thinking tool rather than a measurement.

06Where the variance actually lives

"More freedom, more variance" is true and far too coarse to act on. We need the exact address, because a remedy aimed at the wrong place is indistinguishable from superstition.

Chapter 20 gave the variance of the coefficients as Var(β̂) = σ²(XᵀX)⁻¹. Chapter 7 gave the coordinates in which that matrix is diagonal. Write the SVD as X = UΣVᵀ, so that XᵀX = V D² Vᵀ with the singular values d_j on the diagonal. In the direction v_j, the coefficient's variance is exactly σ²/d_j², and the total variance the fit carries is σ²·Σ 1/d_j².

Read that sum, because it is the whole point. Variance is not spread evenly across a model. It is concentrated in the directions where the data barely moved, since those have the smallest d_j and therefore the largest 1/d_j². A small singular value is not an abstraction: it says that along that particular combination of columns, your rows hardly varied at all, so there is almost no evidence about what happens when it changes.

Take the smallest forcing example: two features at correlation 0.9917, a hundred rows, columns standardized. The singular values come back 14.1126 and 0.9135. The per-direction variances are then 0.00502 and 1.19832. One of two directions carries 99.6% of the total. If you could fix one number in this model, it is not "a bit of each". It is almost entirely that one direction, and the other one is close to free.

And those directions are not features. The V columns here are (−0.707, −0.707) and (+0.707, −0.707), which is to say the sum of the two features and their difference. The bad direction is the difference between two near-twins, and no single column can be blamed for it. Bootstrap the fit two hundred times and you can watch this directly: each coefficient has a standard deviation of 0.818, their difference has 1.633, and their sum has 0.091. The sum is pinned, the split is free, and their correlation is −0.994.

Now weld three chapters together, because this is one object that you have already met under three names. Chapter 7 warned that inverting divides by d, so a small d amplifies error enormously, and gave the condition number to measure it. Chapter 20 drew the RSS canyon, where near-dependent columns turn the bowl into a long flat valley whose floor lets noise slide the estimate a long way. This chapter calls it the variance term. The canyon's floor direction is the small-singular-value direction, and the bootstrap streak you watched in Chapter 20 is σ²/d_min² made visible.

Push the correlation and watch the same three readouts move together. At ρ = 0.5 the condition number is 1.7 and the smaller direction holds 73% of the variance. At ρ = 0.99 it is 14.7 and 99.5%. At ρ = 0.999 it is 47.1, the total variance has grown to 11.10, and the small direction holds essentially all of it.

Which tells us what any fix has to do, and it is more specific than "use fewer features". We need to stop the fit from spending freely in the directions where the data has nothing to say, while leaving it alone where the data spoke loudly.

07Shrink the search, do not cut the list

Chapter 21's law again: the optimism you are charged is set by the size of the search. So make the search smaller, which is the one instruction this whole chapter is built on. Not by deleting columns, which is the row of switches we already rejected, but by putting a fence around the candidates.

Instead of minimising RSS over all of ℝᵖ, minimise it subject to ‖β‖₂² ≤ t. The fit can still use every feature. It just cannot use them all loudly. That is the budget form, and it is worth pausing on how literal it is. A smaller t is a smaller region of coefficient space, which is a smaller candidate set, which is exactly the quantity Chapter 21 charged the toll on.

Now Chapter 8 pays out in full. The objective is a convex bowl and the fence is a convex set, so the constrained minimum sits where an RSS contour is tangent to the fence. The tangency condition is ∇RSS = −λ∇g, and λ is the shadow price: how much RSS one extra unit of budget would buy back. Turn that sentence around and you have the form everyone writes, minimise RSS(β) + λ‖β‖₂².

These are not two methods. They are one problem seen from two sides, joined by a multiplier you already own, with λ up meaning t down. And KKT closes it honestly. If the unconstrained OLS solution already sits inside the fence, the constraint is slack, λ = 0, and you get OLS back unchanged. A knob that does nothing at zero is exactly what a knob should do.

This also kills the framing that ruins the idea for most people. A penalty is not a moral preference for small coefficients, and "discourage large weights" is a sentence that explains nothing. The penalty is a fence, and the fence is a smaller search. Regularization is any term added to the loss whose effect is to shrink the search space.

One warning before we go on, because it will otherwise cause real damage. This λ is Chapter 8's Lagrange multiplier, not Chapter 7's eigenvalue. Two chapters, one letter, and shortly we are going to add this λ to those eigenvalues in the same formula.

08The units become a modelling decision

Before that penalty can mean anything, something has to be true that is not yet. The term ‖β‖₂² adds coefficients from different features together. A coefficient on market capitalisation in dollars, one on a price-to-earnings ratio, one on a percentage: adding those three numbers up means comparing their units.

Here is the fact that makes this load-bearing rather than tidy, and it is worth predicting before you read it. Multiply one feature column by a thousand. Do the OLS predictions change?

They do not. The coefficient obediently divides by a thousand, going from 0.921760 to 0.00092176, and every fitted value is identical to within 2.6e−13. OLS is scale-equivariant: it does not care what units you recorded your data in. Now the second question, and this is where nearly everyone gets caught. Having just watched OLS shrug, do the ridge predictions change?

They change by up to 0.1712. The reason is arithmetic rather than mysterious. That rescaled coefficient is a thousand times smaller, so its square is a million times smaller, so the penalty barely notices it. Feature one's share of the total penalty collapses from 56.4% to 0.000145%. It has been quietly exempted, and read back in the original units its coefficient has moved from 0.8354 to 0.9044: barely shrunk at all.

So the moment you add a penalty, the units your data happens to be recorded in become a modelling decision. The fix is standardization: centre each column and divide by its standard deviation, so that a coefficient means "change in y per one standard deviation of this feature". That is a comparable quantity across features, which is precisely what the penalty needs in order to be adding like to like.

Two riders travel with it, and both of them bite in practice. The intercept is never penalised, because it is a level rather than a direction. Shrink it and your fit starts depending on where you happened to put the zero of y. Convert returns from percent to basis points and a penalised intercept will move your predictions.

The second rider is the one that ships to production. The standardiser is a fitted transform: it estimates a mean and a standard deviation from data. By Chapter 21's rule it therefore belongs inside the fold, computed on training rows only. This is the exact leak Chapter 21 flagged as the one that survives a perfect split, and it is doubly seductive here, because standardizing does not feel like fitting anything. It feels like cleaning.

09Ridge, solved and then read

The penalised loss RSS(β) + λ‖β‖₂² is a convex quadratic in β, so Chapter 8's recipe applies with no ceremony. Set the gradient to zero, and you get (XᵀX + λI)β = Xᵀy, hence β̂_ridge = (XᵀX + λI)⁻¹Xᵀy. One line, no new theory.

But do not read that inverse yet, because there is a version of ridge with no matrix algebra in it at all, and it is the one that makes the penalty something you can point at.

Stack √λ·I underneath X, and p zeros underneath y. You have added p fake rows. Each one says "this coefficient is zero", and each carries weight √λ. Now run plain ordinary least squares on that augmented dataset. On our three-feature example with λ = 4, augmented OLS returns (0.885070, 0.736897, 0.284760) and the ridge formula returns the same three numbers, agreeing to 2.8e−15. The algebra is one line, since X̃ᵀX̃ = XᵀX + λI.

So the penalty is not punishment and not discouragement. It is prior evidence, written as rows, mixed into your data on the same footing as your observations, with λ setting how loudly it speaks. It also explains in advance why the fit is unique when p > n. You literally added p rows.

Now read the machine in Chapter 7's coordinates, and three consequences fall out at once. Since XᵀX = VD²Vᵀ, adding λI keeps the same eigenvectors and lifts every eigenvalue from d_j² to d_j² + λ.

First, the coefficient along direction j is multiplied by the shrinkage factor d_j²/(d_j² + λ). That is near one where d_j is large and near zero where it is small. On the collinear design, λ = 5 multiplies the strong direction by 0.9755 and the weak one by 0.1430. That is the surgical shrink we asked for, and nobody had to hand-tune it. The geometry does it.

Second, the smallest eigenvalue is lifted off the floor. The eigenvalue ratio of XᵀX on that design is 238.7, and at λ = 5 it is 35.0, at λ = 20 it is 10.5. Chapter 7's warning, answered. And since d² + λ > 0 always, XᵀX + λI is invertible even when XᵀX is singular. Ridge dissolves Chapter 21's p ≫ n non-uniqueness outright. On forty rows and a hundred columns, two OLS solutions with identical predictions had norms of 1.455 and 7.150; ridge at λ = 1 returns one answer, for a stated reason, instead of the silent pseudo-inverse.

Third, add up the shrinkage factors and you get the effective degrees of freedom, df(λ) = Σ d_j²/(d_j² + λ). On our two-feature design it slides from 2.000 at λ = 0 through 1.450, then 1.119, then 0.674.

A value of 1.119 deserves a stop, because Chapter 20 defined degrees of freedom as a dimension count and counts are integers. What df(λ) counts is fractional usage. The fit uses a well-determined direction almost fully and a poorly-determined one a seventh of the way, and adding those fractions gives a real number that behaves exactly like a count in every formula that used one. At λ = 0 it returns p exactly, and it falls smoothly toward zero as the fence tightens, which is what a dial should do. That is the continuous lever the opening asked for, and it drops straight into Chapter 21's toll as df(λ)/n.

One last framing note, because this formula lives a double life in the literature. Numerical-analysis courses present adding λI as a fix for an ill-conditioned matrix, and statistics courses present it as a fix for overfitting. Those are the same story, and the weld we built is why.

10The first inch of shrinkage is free

Everything so far says the trade is possible. Now ask whether it is ever actually worth making, and get a real answer rather than a habit.

Write both terms as explicit functions of λ, using the shrinkage factors we just derived. The variance is σ²·Σ d_j²/(d_j² + λ)², and the bias squared is Σ λ²β_j²/(d_j² + λ)², with the β_j read in the singular coordinates. At λ = 0 the variance is the number we already computed, and the bias squared is zero.

Now the question that decides whether any of this is worth doing. Which of the two wins the first inch of shrinkage? Most readers say it depends on the data, which is the sophisticated answer and it is wrong. Bias squared is quadratic in λ, and Chapter 4 told us a quantity that leaves a point quadratically leaves it flat. The variance leaves steeply. On our design the slope of the variance at zero is −2.8720 and the slope of the bias squared is 7.2e−07, which is zero to numerical precision.

So the derivative of the total at λ = 0 is strictly negative. Not usually. Always, for any dataset with noise in it. There is always some λ > 0 whose total error is strictly lower than OLS's. Watch it happen on the numbers: a step to λ = 0.5 drops the variance by 0.7298 and raises the bias squared by 0.0702, a trade of better than ten to one. The best setting is λ = 2.00, where the total is 0.3579 against OLS's 1.2033. That is 70% of the error removed by a knob.

A careful reader is now holding an objection, and it is the right one. Chapter 20 proved Gauss–Markov: OLS is the best linear unbiased estimator. So how can anything beat it?

Point at the adjective. The theorem's promise is about variance, and only among estimators whose centre is already pinned to the truth. Ridge is biased, so it is not in the competition the theorem was judging. And the quantity that costs you money is MSE, which Chapter 16 already told us is variance plus bias squared. Chapter 16 also said, in as many words, that a biased estimator can beat an unbiased one. This is that sentence, cashed.

Say the honest half too, because this is where the result is usually oversold. The result guarantees that a good λ exists. It does not tell you what it is. Too much λ walks you into the bias-dominated arm and loses money there instead, and the only way to locate the right one is a data-driven search we will build shortly.

11Same fence logic, different fence

Keep the entire machine and change one thing: the shape of the fence. Replace ‖β‖₂² ≤ t with ‖β‖₁ = Σ|β_j| ≤ t. Same budget logic, same bridge to a penalised form RSS + λ‖β‖₁, and one dramatic consequence.

In two dimensions the L2 fence is a disc. The L1 fence is a diamond, and its extreme points sit on the axes. A point on an axis has a coordinate that is exactly zero. So the lasso does not merely shrink coefficients. It sets some of them to exactly zero, which means it selects features.

That much is the famous picture, reproduced in every textbook on the subject. The question it never answers is why the tangency lands on a corner rather than merely passing close to one.

Here is the reason, and it is pure geometry. At a smooth point of a fence, Chapter 8's condition needs the gradient to line up with the single outward normal at that point. That is one direction out of a continuum, a knife-edge coincidence. At a corner there is no single normal. There is a whole fan of them, so an entire open set of gradient directions gets caught by that one point. Corners capture a positive share of problems, and smooth boundary points capture a vanishing one, so the zeros are not a coincidence you got lucky with. That, and nothing more mystical, is why L1 produces exact zeros and L2 never does.

The same fan explains the thing readers experience as an arbitrary software difference. Lasso has no closed form, because |β| has no derivative at zero, so "set the gradient to zero" is simply unavailable. The object that replaces the gradient there is the subgradient, which is the same fan of directions under another name. Coordinate descent is the practical consequence of a corner, not a quirk of a library.

Now say it quantitatively, in the one case where both have closed forms: orthogonal columns.

Ridge multiplies each OLS coefficient by 1/(1 + λ). Lasso subtracts λ from its magnitude and clips at zero, giving sign(β̂)(|β̂| − λ)₊, which is called the soft threshold. Draw the two side by side and read what each one does to a coefficient. Ridge is a straight line through the origin with a gentler slope. Lasso is a dead band, then a 45-degree line offset by λ. That dead band is feature selection, drawn.

And the picture corrects a slogan everybody repeats. At λ = 0.5 in each formula, an OLS coefficient of 0.20 becomes 0.1333 under ridge and 0.0000 under lasso. But an OLS coefficient of 3.00 becomes 2.0000 under ridge and 2.5000 under lasso. Lasso removed 0.50 and ridge removed 1.00. So the lasso is harsher on small coefficients, which it kills outright, and gentler on large ones, which it only nudges. One subtracts a constant amount, the other removes a constant fraction. The two λ's are different objects and the shape is the point.

12Which fence, and the trap inside the answer

Which one, then? The choice is not a matter of taste. It is a statement of belief about the world.

If you think many features each contribute a little and they move together, which is the normal condition in factor research, ridge is right. It keeps correlated twins and splits the credit between them, which is stable precisely because the quantity it is splitting is the one the data actually determined. If you think a few things genuinely matter among hundreds of candidates, lasso is right, and it is the honest continuous version of Chapter 21's third exit.

But there is a trap sitting inside that second answer. Give the lasso two nearly identical predictors and it has to pick one. Chapter 20 already told us it has no basis for choosing, because the sum is pinned and the split is free.

Watch it rather than take the warning. Two twins at correlation 0.9791, both genuinely carrying a coefficient of 1, plus six columns of pure noise, on two hundred rows. In this particular sample the first twin correlates 0.718 with the label and the second correlates 0.708. A gap of one hundredth is the entire basis on which the credit gets handed out.

Bootstrap the fit two hundred times and read which features come back. Twin one survives in 99% of the resamples and twin two in 54%. Both are kept 53% of the time and exactly one is kept the other 47%. The coefficients tell the same story with more precision: β̂₁ = 1.251 ± 0.461 and β̂₂ = 0.332 ± 0.439, correlated at −0.945, while their sum is 1.583 ± 0.151. The sum barely moves and the split swings wildly. Run ridge on the identical data and both twins are present every single time, at 1.211 and 0.830.

So the list of selected features is not a discovery. It is a random object with a distribution. What you report is its stability, meaning how often each feature survives resampling, and not the set your one run happened to produce.

Elastic net splits the difference by adding a little L2 back, minimising with α‖β‖₁ + (1−α)‖β‖₂². That rounds the corners just enough to keep correlated groups together while still zeroing the dead ones.

One more consequence, and it is the one that ends up in research notes. The bias you deliberately bought is still sitting on the coefficients that survived, so their magnitudes are shrunk toward zero. That is why people refit plain OLS on the selected subset, which is called the relaxed or debiased lasso, and it works. It comes with one standing warning: the t-statistics from that refit are lies. The features were chosen by looking at the data, and the refit computes its inference as though the choice had been made in advance. That is Chapter 19's winner's curse, wearing a lab coat.

13Choosing lambda without cheating

λ is a hyper-parameter, which means it is chosen by search rather than by the optimiser, so Chapter 21 already wrote every rule we need. It is chosen on validation folds and never on the test set. In finance those folds are walk-forward with a purge sized to the label horizon. And every fitted transform, including the standardiser, is fitted inside the fold.

Now plot the cross-validation error against λ, with the standard-error bars Chapter 16 taught us to attach.

On a run with two hundred rows and thirty features, the CV curve dips from 9.626 at λ = 0.1 to 9.178 at λ = 30, then climbs back to 9.943 at 300. The argmin looks clean until you draw the error bars. One standard error there is 0.933, and the whole visible dip between λ = 10 and λ = 300 is 0.766. The entire feature you are reading is smaller than the noise on any one point of it.

So the location of the argmin is being decided by noise, and you can watch that too. Reshuffle the folds forty times and the argmin lands on λ = 100 thirty-six times and on λ = 30 four times. Scored against fifty thousand fresh rows, the truly best setting on this data is λ = 100, with a test error of 9.996 against OLS's 10.367. The noise floor is 9.000, so most of what is left is unreachable.

There is a second problem, and it is Chapter 21's own law turning around on the tool we are using. Taking the minimum of a search over noisy scores is itself a search. So the CV error at the chosen λ is optimistic. Here it says 9.178, while that model's true test error is 10.087. If you must report a number, put an outer loop around the whole λ-choosing procedure. That is nested cross-validation: the inner folds choose λ, the outer folds score the whole procedure that chose it, and no row does both jobs at once.

The one-standard-error rule follows from the flatness. Choose the largest λ, meaning the simplest model, whose CV score is within one standard error of the best. That is not a fudge. It is shrinkage applied to the hyper-parameter itself, justified by the two facts we just watched.

Be honest about what it buys, though, because it is usually oversold. On this data the rule picks λ = 300, whose true test error is 10.410, slightly worse than the argmin's 10.087. It bought a simpler model and a more stable choice, not a better score. That is the trade it always makes, and it is worth making when the basin is flat and you would rather not re-tune every month.

One practical detail carries a real reason behind it, so it is worth a line. Search λ on a logarithmic grid rather than an evenly spaced one. The shrinkage factor d²/(d² + λ) depends on λ relative to d², and singular values span orders of magnitude, so a linear grid spends nearly all of its points in a region where nothing changes.

14Three exits closed, and the shape of the idea

Go back to the door Chapter 21 labelled and close all three exits.

Constrain is ridge and lasso: fence the search. Select is lasso, or an economic argument you can defend out loud. Reduce is projecting onto the leading directions before fitting, and Chapter 7's coordinates show it is the same act with a harsher weighting. Principal-components regression uses the first m singular directions fully and the rest not at all, which is a step function of d. Ridge uses every direction by the smooth amount d²/(d² + λ). Same axis, two weighting functions: one a cliff, one a slope. The algorithm behind that projection arrives properly in Chapter 24.

Now the reading that turns the penalty from a device into a statement. Minimising RSS + λ‖β‖₂² is exactly finding the most probable β under a Gaussian prior centred at zero with variance τ², and the exchange rate is λ = σ²/τ². That is the noise in your data divided by how large you believe the coefficients are. Chapter 19 said priors are shrinkage toward zero. This is that sentence with the number filled in, and it means λ is a belief you can argue about with a colleague rather than a constant you tune in silence. The lasso is the same statement with a Laplace prior, whose sharp peak at zero is exactly why it produces exact zeros.

So regularization is not a technique. It is a shape. Whenever the variance of an unconstrained search costs more than the data can pay for, buy it back with structure. Every later chapter is another purchase of the same kind, and you will now recognise them: Chapter 23's bagging averages many fits to attack the variance term directly, boosting fits residuals to attack the bias term, and Chapter 31's covariance shrinkage is this identical move performed on a matrix.

Then the limit, said flatly, because a chapter that ends on a rescue is a dangerous chapter. Regularization does not repair overfitting. It prices it. Every warning from Chapter 21 still stands, unchanged. A leaked pipeline with a beautifully tuned λ is still a leaked pipeline. A strategy selected out of two hundred is still cursed no matter how carefully the survivor was shrunk. The penalty only prices the part of the search that ran inside the optimiser, and the lookbacks you tried, the thresholds you nudged and the reruns after a bad month are all outside it.

Which leaves the door open in a nice place. We now hold two named terms rather than one vague hazard, and the obvious question is what a method would look like that went straight at each one. What attacks the variance term head-on? What attacks the bias term head-on? Chapter 23 is those two answers, and they are called bagging and boosting.