20OLS Regression: Mechanics & Diagnostics

Chapter 19 left us screening candidates one at a time and paying a bill for every look. There is an obvious improvement available, and we are about to take it. Put every candidate driver into one model and ask what each one is worth once the others are accounted for. That model is ordinary least squares, and it is the most-run calculation in quantitative finance. Here is the thing almost nobody is told about it. The whole first half of this chapter needs no probability at all. Not one distribution, not one assumption, not one random variable. A regression is a perpendicular, dropped from a point onto a flat plane, and that act is available on any table of numbers you will ever hold. The famous fine print arrives later and it is not a precondition for the arithmetic. It is admission money for a different question, about whether the answer means anything. And of the five assumptions, exactly one protects the number. The rest protect only the error bar. That single split organises everything in the back half of this chapter, and it is the sentence to carry out the door.

Look at what this page stands on, because almost all of it is already yours. Chapter 6 gave us the column space, rank, the overdetermined system, and the orthogonal projection that it deliberately set up and left uncompleted. Chapter 7 gave us the condition number and the pseudo-inverse. Chapter 8 gave us contour maps and the flat spot at the bottom of a convex bowl. Chapter 16 gave us the estimand, the sampling distribution, and Bessel's correction with a promise attached to it. Chapter 17 gave us the whole testing template. Chapter 18 gave us the identification ρ = cos θ, and Chapter 19 gave us the family and its bill.

That is the seam. Testing candidates one at a time was never the best available move, and putting them all in one model is a genuine improvement. But a model with twelve columns prints twelve t-statistics, so hold your enthusiasm. Chapter 19 is waiting inside a single function call.

01The number a correlation cannot give you

Start with a trade you actually have to place, because the gap opens the moment you try.

You are long £10m of a single stock and you want to hedge it with index futures. Chapter 18 hands you the relationship between the two return series: ρ = 0.80. Good. Now write the ticket. How many contracts?

You cannot answer. Correlation is unitless by construction, because we divided by both standard deviations, and that was the entire point of doing it. A trade needs a slope, a number in pounds of index per pound of stock. So the first honest statement of this chapter is that we need a different object from the one Chapter 18 gave us.

Bring the two volatilities back and the slope appears. The stock runs at 28% a year and the index at 16%, so β = ρ·σ_y/σ_x = 0.80 × 28/16 = 1.400. Your stock moves 1.4 pounds for every pound the index moves, so the hedge notional is £14.0m. With the index at 8,200 and £10 a point, one contract carries £82,000 of exposure, which makes the trade 171 contracts. Notice what happened there. The correlation alone was not enough, and the thing we actually wanted was a slope the whole time.

Now make it harder in the direction real work goes. You rarely have one candidate driver. You have the market, a sector ETF, and a currency, and the question is how much of each.

Chapter 6 already has a word for a weighted mix of columns. It is a linear combination, and the mix you are hunting for is a vector of weights. So write it down properly. Stack 500 days of predictor observations into a matrix X, one row per day and one column per predictor, put the 500 target returns into a vector y, and the model is Xβ = y.

Now count what you just wrote. Five hundred equations. Three unknowns. Chapter 6 named this exactly: an overdetermined system, with the target sitting outside the column space and no exact solution.

Do not read that as failure, because it is not a symptom of bad data and no amount of cleaning will fix it. Five hundred arbitrary numbers cannot be produced by three knobs, and that is the permanent situation for every regression anyone has ever run. What it does is convert the whole problem into one open question. Since we cannot hit y, what does closest mean?

One more object before we answer that, because it is the thing the rest of the chapter is about. For any candidate β, the residual vector e = y − Xβ holds one number per observation. It is the whole of the model's failure, carried as a single arrow.

02What does closest actually mean?

We have to choose what closest means, and the choice is not cosmetic. It is the definition of the answer.

Take the smallest experiment that settles it. Here are three numbers: 1, 2 and 12. Fit a single constant c to them, which is the tiniest regression imaginable, one column of all ones.

First, notice you cannot just minimise the plain sum of residuals. Push c upward and Σ(yᵢ − c) runs away to minus infinity, so there is no minimum to find. The criterion has to be non-negative before it can mean anything at all.

There are two obvious ways to do that, and both of them sound perfectly sensible. You can take absolute values, or you can take squares. So run both and see what each one picks.

Minimising Σ|yᵢ − c| gives c = 2, which is the median. Minimising Σ(yᵢ − c)² gives c = 5, which is the mean, exactly as Chapter 16 proved in one line of calculus. Sit with that for a second, because it settles the question we opened with. Same three numbers, two perfectly reasonable criteria, two different answers, and neither is wrong.

So why does the whole field build on squares? The usual answers are that squaring makes residuals positive and punishes large errors more heavily. Both are true, and both are cover stories for the reason that actually matters.

Write the squared-error criterion out as a sum: Σ(yᵢ − ŷᵢ)². Chapter 6 and Chapter 18 both told us that a list of n numbers is a vector in ℝⁿ, and that the length of a vector is the square root of the sum of its squared entries. So that sum is nothing other than ‖y − ŷ‖², the squared distance from the point y to the point ŷ.

At c = 5 the sum of squares reads 74.00, and the residual arrow has length 8.6023, whose square is 74.00. Same number, twice.

So least squares does not mean a loss function that happens to square things. It means NEAREST POINT. That is the whole vocabulary shift of this chapter, and everything from here is a consequence of it.

Be honest about what we just discarded, because respecting the alternative is what makes the choice credible. Absolute deviation gives you a median, which shrugs off a crazy outlier in a way squares never will, and that is a genuine advantage and the reason quantile regression exists. It also has kinks where a residual crosses zero, it is not differentiable there, its answer can be non-unique, and it has no closed form. Squares are smooth, convex, uniquely minimised, solvable in one line of algebra, and they are a distance. That last property is the one we are about to spend the entire chapter cashing in.

03★★ Least squares is a perpendicular

This is the move the whole chapter turns on, and it is worth going slowly, because there are two different pictures in play and mixing them up is the most common permanent confusion in this subject.

Picture one is the row picture, and you already have it. Put n dots on two axes, x across and y up, and run a candidate line through them. Each residual is a vertical gap between a dot and the line.

Picture two is the column picture, which belongs to Chapter 6 and Chapter 18, and in it each data column becomes one arrow in ℝⁿ. So the predictor x is a single arrow, the target y is a single arrow, and a candidate fit ŷ = βx is a point sliding along the line spanned by x. Same numbers, transposed.

Name the trap now rather than later, because it catches everybody. In the row picture the residuals are vertical, and they are not perpendicular to the fitted line. In the column picture the residual arrow is about to become exactly perpendicular to the predictor arrow. Both statements are true, because they are statements about two different drawings of the same three numbers.

Now do the experiment yourself. Take three observations and one predictor, slide β across its range, and watch three things at once: the vertical gaps, the running total Σeᵢ², and the dot product x·e. Before you touch anything, commit to a guess about what happens to that dot product at the best fit.

At the β that minimises the sum of squares, and at no other β, the dot product hits exactly zero.

Confirm it in one line of Chapter 3 calculus. Differentiating gives d/dβ Σ(yᵢ − βxᵢ)² = −2Σxᵢ(yᵢ − βxᵢ) = −2(x·e). So "set the derivative to zero" and "set the dot product to zero" are literally the same equation, and Chapter 18 already told us that a zero dot product means a right angle. The calculus route and the geometry route are one sentence read in two languages.

Solving it gives β̂ = (x·y)/(x·x), which is worth reading rather than memorising. It is how much of y lies along x, measured per unit of x's own squared length.

Now generalise, and notice that generalising costs nothing at all. With k predictors the reachable set is the column space of X, which is a k-dimensional plane. The nearest point on it is the orthogonal projection ŷ = Xβ̂. And perpendicular to the plane means perpendicular to every column, which written out is Xᵀe = 0. Substitute e = y − Xβ̂ and you have the normal equations:

XᵀXβ̂ = Xᵀy, and if XᵀX can be inverted, β̂ = (XᵀX)⁻¹Xᵀy.

Read the normal equations out loud, because that is what they are. The part of y you could not explain has to be perpendicular to everything you used to explain it.

And it has to be, by an argument with almost no algebra in it. Suppose some of the leftover still lay along one of your columns. Step a distance δ along that column and the sum of squares changes by exactly −2δ(x·e) + δ²(x·x). Choose δ = (x·e)/(x·x) and the error falls by (x·e)²/(x·x), which is strictly positive whenever the dot product is not zero. A free improvement was available, so you were not at the minimum.

That is why the apparatus is inevitable rather than conventional. Any leftover still lying along a predictor is unexploited signal, and at the minimum there cannot be any.

Two last things before we start collecting the interest. First, the dimension is not a problem. The three vectors y, ŷ and e span at most a three-dimensional world, whether n is 3 or 3 million, so the picture on the page is not an analogy. It is the actual geometry, seen from the only angle that matters.

Second, and this will organise the entire back half of the chapter, count the probabilistic assumptions we just used. Zero. Not one. The fit is deterministic geometry, and it is available on any data anyone will ever hand you.

04Can you even name the answer?

Here is a question that sounds pedantic and is about to become the most expensive practical issue in the chapter. Does β̂ exist, and is it the only answer?

Separate the two objects the projection quietly produced. ŷ is a point, the nearest point of the plane to y, and a point on a plane is unique however you choose to describe it. β̂ is a coordinate vector, the recipe that says how much of column one and how much of column two. Coordinates need an independent basis.

Make that concrete, because this failure is a rite of passage. Build a design matrix with an intercept, a column that is 1 on up-days, and a column that is 1 on down-days. Those two dummies add up to the ones column, exactly.

Three completely different coefficient vectors, and every one of them predicts 1.100 on up days and −0.575 on down days, with an identical RSS of 0.8475. The predictions agree to the last bit. Nothing is wrong with the data and nothing is wrong with the fit. The question how much is due to the intercept versus the up-dummy simply has no answer, because you asked for coordinates in a basis that is not a basis.

Now name it with Chapter 6's chain, all of which fires at once. The columns are dependent, so X is rank-deficient, so XᵀX is singular, so it cannot be inverted, so β̂ = (XᵀX)⁻¹Xᵀy has no meaning. The run confirms it flatly: rank 2 out of 3 columns, and det(XᵀX) = 0. Chapter 6 already told us why in one sentence, which is that a collapse destroys information, and destroyed information cannot be recovered.

Two practical notes are worth having now, and the first one costs people real money. Software rarely refuses this fit. It typically reaches for Chapter 7's pseudo-inverse and hands back one of the infinitely many answers, or it silently drops a column, so a coefficient you are reading may be an arbitrary member of a family.

Second, the far more common case in real work is not exact dependence. It is near dependence, where columns are almost but not quite mixes of each other. Then XᵀX is technically invertible and practically dreadful, Chapter 7's condition number blows up, and the inverse amplifies noise ferociously. That is multicollinearity. We now know precisely what it is, and we will come back for it properly once we have error bars to read it with.

05Fitting spends dimensions

Here is a small structural fact that pays for three later ideas, so it earns its own beat. The intercept is not a special object. It is a column of ones.

Write it into X like any other predictor, and "the model has an intercept" becomes "one of my columns is constant". Now run the previous sentence through it. The residual must be perpendicular to every column, so it is perpendicular to the ones column. A dot product with a vector of ones is just a sum, so Σeᵢ = 0, exactly and always, whenever an intercept is included.

That is not a modelling assumption and it is not a coincidence. It is orthogonality to one particular direction. Two consequences fall out of that immediately. The fitted line passes through the point (x̄, ȳ), because the residuals average to zero. And the mean of y is inside the model's reach, which is what makes it legitimate to measure everything relative to ȳ.

Chapter 16 said exactly this once already. Its Bessel argument was that the residual vector is perpendicular to the all-ones direction, for the case where the ones column is the only column.

Now generalise the accounting, because it answers a question that everybody gets wrong. Each column you fit imposes one perpendicularity constraint on the residual. With k independent columns, e is forced perpendicular to k independent directions, so it cannot roam freely in ℝⁿ. It is confined to a subspace of dimension n − k.

That is what degrees of freedom means. Not a fudge factor. A dimension count. And it is why the honest noise estimate is σ̂² = RSS/(n−k) rather than RSS/n. Chapter 16 proved the one-column case and openly owed us the rest, and this is the rest, the same theorem with more columns. A quick check makes the size of the correction concrete: over 200,000 samples of five draws from a standard Normal, dividing by n−1 gives a mean of 0.9985 against a truth of 1.0, while dividing by n gives 0.7988.

The tedious exam question dissolves too. Is it n−k or n−k−1? Count columns, and the intercept is a column.

Now watch the extreme, because it will matter enormously later. With three observations and one column the residual lives in a plane. Add a second column and it is confined to a line. Add a third and it is pinned to the origin. Fit as many columns as you have observations and the residual is forced to zero, the fit is perfect, σ̂² is 0/0, and you have learned precisely nothing. Hold that picture, because it is the geometry of overfitting and we meet it twice more.

06R² is a squared cosine

Now collect the interest, because the most quoted number in applied statistics falls out with no work at all.

Centre everything about ȳ, which is legitimate because we just showed the mean lives inside the model. You now hold three vectors: the centred target y − ȳ1, the centred fit ŷ − ȳ1, and the residual e. The residual is perpendicular to everything in the column space, which includes both the fit and the ones column, so it is perpendicular to the centred fit.

So those three vectors form a right-angled triangle, with the centred target as the hypotenuse.

Pythagoras, and nothing else: ‖y − ȳ1‖² = ‖ŷ − ȳ1‖² + ‖e‖². Those three squared lengths have names. TSS, ESS and RSS, the total, explained and residual sums of squares. On the five-point example the numbers read 14.800 = 12.100 + 2.700, and the cross-term that textbooks make vanish through half a page of algebra is a dot product between perpendicular vectors, which computes to 0.0 exactly. That is the whole proof.

Now define R² = ESS/TSS, the fraction of the squared length the plane accounted for, and read what it actually is. In a right triangle, the ratio of the adjacent leg squared to the hypotenuse squared is cos²θ, where θ is the angle between the centred target and the column space.

R² is a squared cosine. And now Chapter 18's keystone comes back with interest. There, ρ was the cosine of the angle between two centred data vectors. Here R² is the cosine squared of the angle between one centred data vector and an entire subspace. With a single predictor those are the same angle, so R² = ρ² exactly. On the five-point example, ρ = 0.9042 and R² = 0.8176, and 0.9042 squared is 0.8176.

Which makes a very cheap and very useful test possible. A stock has a correlation of 0.7 with the index. What fraction of its variance does the index account for? The instinct says 70%. The answer is 49%, because R² is a cosine squared, and the missing 51% is the length of the perpendicular leg you were not picturing.

One honesty note to carry. All of this assumed the model contains an intercept, which is what puts ȳ inside the column space. Drop the intercept and the decomposition no longer holds, which is why a regression without one can report a negative R² and surprise everybody.

07What R² is not

Having built R² honestly, spend one section dismantling what people do with it. Notice as we go that every criticism below is geometry, not statistics.

Start with the criticism that matters most, and it takes one sentence. Add a column to X and the column space gets bigger. The old fit is still available inside that new, larger space, because you can always set the new coefficient to zero. So the nearest point of the bigger plane cannot be further from y than the nearest point of the smaller one.

Which means RSS cannot rise and R² cannot fall. Ever. For any column, including a column of numbers you generated with a random number generator this morning.

Sixty days of returns, and columns of pure noise added one batch at a time. One noise column reads 0.0505. Five read 0.0911. Twenty read 0.3450. Fifty read 0.9284. And fifty-nine noise columns beside the intercept read 1.0000, with the residual crushed to 1.3e−31, which is zero wearing floating-point clothes.

Nothing broke here. The geometry is behaving exactly as specified, and what you are watching is the collapse we drew when the residual ran out of room to move.

That is precisely why an unadjusted in-sample R² cannot compare models of different sizes, and it is why adjusted R² exists. Adjusted R² is not a fudge factor. It is the same ratio recomputed with the honest degrees of freedom: 1 − (RSS/(n−k))/(TSS/(n−1)). A column now has to earn more than the dimension it costs, and if it does not, the number falls.

Watch it work and then watch it fail, because both are honest. At twenty noise columns, R² reads 0.3450 while adjusted R² reads 0.0091, which is the rent doing its job. At fifty noise columns on sixty rows, adjusted R² still reads 0.5308 on pure noise. With nine degrees of freedom left there is nothing honest to compare against, so treat adjusted R² as rent charged, never as a defence.

The rest of the list is one sentence each. R² says nothing about causality, and Chapter 18's four explanations are all still live. It says nothing about whether the functional form is right. It says nothing about out-of-sample performance, which is Chapter 21's entire subject.

And it says nothing about economic size, which is where finance readers most need recalibrating. A regression of daily stock returns on any honest signal produces an R² of about 0.01, and practitioners routinely read that as failure. It is not. It is the correct order of magnitude for a market that is mostly unpredictable. A signal with an information coefficient near 0.05, which is an R² of 0.0025, applied across a thousand names and rebalanced for years, is a serious business. The money is not in explaining the variance. It is in the small predictable part, repeated.

08The line: geometry above, inference below

Stop and draw a line across the page, because everything changes here and almost nobody says so.

Look back at what we have built. A projection, a residual, degrees of freedom, R². Now count the probability in it. There is none. Not one distribution, not one expectation, not one random variable. Everything so far is linear algebra performed on a table of numbers. It is always computable, it never fails, and it means exactly one thing, which is that ŷ is the closest point reachable from your columns.

If all you want is to fit a curve through data you already hold, you can stop reading. The chapter is over and it needed nothing.

But nobody stops there. You want to say your hedge ratio is 0.83, that a plausible range for it is 0.83 ± 0.06, and that the value coefficient is significant. Every one of those sentences is a claim about a truth you never see, a number the world would have if you could sample forever. To reach it you must first assert that such a truth exists and describe how your data came from it. That is what an assumption is for.

So write the model of the world: y = Xβ + ε, where β is a fixed unknown vector, which is Chapter 16's estimand, and ε is an unobservable random shock, one per observation.

And now mark the most expensive notation collision in this subject, loudly, because it is the reason one assumption gets misread as trivially true. The residual e is what your fit left over. You can compute it, plot it, and it is orthogonal to your columns by construction, always, on any data, even if the model is nonsense. The error ε is what the world added. You can never observe it, and whether it is orthogonal to X is a claim about reality. Same word in casual speech, two different objects, and every assumption below is about ε.

With that separation in hand, β̂ becomes what Chapter 16 trained us to see. It is a function of a random sample, therefore a random variable, therefore one dot in a sampling distribution.

Now the assumptions. The move that makes this chapter usable is refusing to list them flat.

Sort them by what each one buys. A0, no perfect collinearity, buys existence and uniqueness of β̂, and that is a fact about your matrix rather than about the world. A1, linearity in the parameters, buys that the truth is reachable at all. Note that it does not forbid curves, because x², log x and interactions are all perfectly legal columns. What is forbidden is β entering non-linearly.

A2, exogeneity, written E[ε|X] = 0, buys unbiasedness, meaning E[β̂] = β. A3, homoscedasticity, and A4, no autocorrelation, buy the correctness of the standard-error formula, and together with A1 and A2 they buy Gauss–Markov, that OLS is the best linear unbiased estimator. A5, normal errors or a large sample and the CLT, buys that the t-statistic really follows a t-distribution, so the p-value is exact.

Now read down the unbiasedness column, because this is the sentence to carry out of the chapter. Two assumptions sit there, and they do different jobs. A1 says the target exists. A2 says your estimate is aimed at it. Every other assumption on the table sits in an error-bar column.

So of the things people actually diagnose after a fit, exactly one protects the coefficient. Everything in the rest of this chapter is a consequence of that single asymmetry.

09The error bar, and the lever arm

Now build the error bar, and refuse to leave it as a matrix formula.

Under A0 to A4, the covariance matrix of the estimate is Var(β̂) = σ²(XᵀX)⁻¹. That is one object holding every coefficient's variance on its diagonal and every pair's covariance off it. As written, it teaches nothing. So collapse it to one predictor and read it as a sentence:

SE(β̂) = σ_e / (σ_x √n)

Three levers, each with a physical meaning. Noisier residuals mean a worse-measured slope, which is obvious. More data shrinks the error like 1/√n, which is Chapter 16's law arriving unchanged. And then the one people never internalise: a bigger σ_x, more spread in the predictor, improves the estimate.

That is the lever arm. If x barely moves across your sample you are measuring a slope over a two-inch baseline, so any wobble in y swings the fitted angle wildly. Spread x out over a long baseline and the same wobble barely tilts it. Put numbers on it. With a daily residual of 1.2%, a predictor spread of 1.0% and 500 days, the standard error is 0.0537, so the hedge ratio reads 1.40 ± 0.11. Squeeze the predictor's spread to 0.4% and change nothing else, and the standard error becomes 0.1342, exactly 2.5 times wider.

Which tells you what (XᵀX)⁻¹ is doing in general. XᵀX is a variance-like summary of your predictors, so its inverse punishes predictors that do not move, and predictors that move together. Hold that second half, because it is the seed of the next problem.

The inference is then Chapter 17 with nothing new whatsoever. The t-statistic for a coefficient is t = (β̂_j − 0)/SE(β̂_j), the same universal template, and the null value is usually zero because the natural question is whether this predictor does anything at all. Compare it against a t-distribution on n − k degrees of freedom, which is the same n − k we counted as a dimension, so Chapter 17's mysterious parameter is now something you can point at.

There is a joint version worth naming too. The F-test asks whether the columns together explain more than nothing, which is a question about the whole subspace rather than any one coordinate. It becomes important the moment individual coefficients start disagreeing with the group.

And now the sting, which is the bridge we walked in on turning round and biting.

A regression printout with twelve predictors displays twelve t-statistics and twelve p-values, and everybody reads them one at a time, hunting stars. That is Chapter 19's crime committed inside a single function call. Twelve tests at 5% carry a 46.0% chance of at least one false star even if every coefficient is truly zero. The regression table is a multiple-testing machine, and running it fifty times with different column combinations is the garden of forking paths with a nicer interface.

10What a coefficient actually means

Before we can talk about what goes wrong, we have to say precisely what a coefficient means in a multiple regression. The standard phrase is that it gives the effect of x_j holding the other variables constant, and that phrase is a spell rather than a mechanism.

Nothing was held constant, because your data was observed rather than controlled. The market and the sector ETF moved together on every single day of the sample, and nobody froze either of them.

So what does the software actually do? Three things, in order. It strips x_j of everything the other columns can explain, then strips y of everything the other columns can explain, then regresses the first leftover on the second.

That is the whole meaning of controlling for something, and the astonishing part is that it is exactly, numerically true.

The full three-predictor regression gives β̂₁ = 1.484167662730. The two residual regressions, run separately and then combined, give a slope of 1.484167662730. The difference is 1.55e−15, which is the floating-point noise floor and nothing else. It has a name, Frisch–Waugh–Lovell, and you should watch it happen rather than take it on faith.

Nothing was frozen. Something was SUBTRACTED.

Read what that means and three separate mysteries dissolve at once. First, β_j is the price of the part of x_j that no other column could account for. In this run the predictor has a standard deviation of 0.951, and after the other columns take their share the leftover has a standard deviation of 0.676. Only that leftover identifies the coefficient.

Second, a coefficient can flip sign when you add a column, and that is not a bug and not a betrayal. Regressing y on this predictor alone gives 1.8838, while the partial coefficient is 1.4842. The leftover is a different vector from the original column, so of course its slope against y is a different number. A raw correlation and a partial coefficient are answers to different questions.

Third, this one picture already tells you how regression fails, in both directions. If the other columns explain nearly all of x_j, the leftover is a tiny noisy stub, and you are estimating a slope on almost no variation, so the standard error explodes. And if something that matters was never put into X, it never got subtracted, so it is still sitting inside the leftover, contaminating the slope.

One picture, both diseases. We take them in that order, worst first.

11The only fatal violation

This is the violation that poisons the number itself, and it deserves to be met first because everything else is comparatively cosmetic.

Exogeneity says that knowing your predictors tells you nothing about the shock. Break it and β̂ is no longer an estimate of β. It becomes a precise estimate of a different quantity.

Run it with a generator you own, so the truth is not in doubt. Build data where y = 1.0·x + 2.0·z + noise, and let x and z be correlated at 0.6. Now fit y on x alone, because z is something you did not measure. An unobserved regime, a liquidity factor, a fund flow.

Predict before you look. What will the coefficient on x be? Almost everyone says about 1, a bit noisy.

The answer is 2.0926, with a standard error of 0.0609 and a t-statistic of 34.4. Not noisy. Not uncertain. That estimate is confidently, precisely and beautifully wrong.

And here is the sentence that makes endogeneity a different species from every other problem in this chapter. Bias does not shrink with n. Variance does, and that is the whole content of 1/√n. Bias does not. Collect ten times the data and the estimate goes to 2.2228 with a standard error of 0.0190 and a t-statistic of 116.9. Push to a hundred thousand rows and the t reaches 367. More data makes an endogenous estimate more confidently wrong.

Where did 2.2 come from? The leftover picture already told us. The coefficient is a slope on the part of x that the other columns did not account for, and z was never a column, so it was never subtracted. The arithmetic is β̂₁ → β₁ + β₂·δ, where δ is the slope of z on x. That is the omitted effect times how well x stands in for it: 1.0 + 2.0 × 0.6 = 2.2. Chapter 18's confounder, drawn as two arrows pulled toward a third, now has a number on it. Put z back into the regression and the coefficient snaps to 0.9611 with z's own coefficient at 2.0001.

Endogeneity is not only omission, and it wears several coats in finance. Simultaneity: you regress your returns on order flow, but your own trading moved the price, so the arrow runs both ways. Measurement error in a regressor: noise in x pulls its coefficient toward zero, which is called attenuation, and the honest reading is that you measured a proxy's slope rather than the thing's. Selection: you regressed on the funds that survived.

There is a trap in the opposite direction too, and it needs saying because the instinct is so strong. Faced with omitted-variable bias, everyone concludes control for everything. That is wrong in a specific way. Conditioning on a variable that is caused by x, which is a mediator, subtracts part of the very effect you are trying to measure. Conditioning on a collider, something that both x and y influence, manufactures a correlation that was never there. Adding columns is not automatically safer, and it is a modelling claim like any other.

Finally the verdict, and it is why this violation comes first. None of the repairs in the rest of this chapter touch it. Robust standard errors, HAC standard errors, a bootstrap: every one of them is a better description of the spread of a wrong number. Only a better model, a natural experiment, or an instrument helps.

12Stable plane, unstable coordinates

Now the violation everyone has heard of and almost nobody can state precisely. Multicollinearity is near-dependence among the columns, which is the case we named and deferred.

Ask the diagnostic question the tier table trained us to ask, which is the only one that decides what you do next. What does it damage?

Slide the correlation between two predictors from 0 up toward 0.99, with the true coefficients both fixed at 1.0, and watch what moves.

Between ρ = 0.90 and ρ = 0.99, R² goes from 0.8083 to 0.8116. The predictions barely move. Meanwhile the standard error on each coefficient goes from 0.178 to 0.544, and bootstrapped across resamples the cloud of (β̂₁, β̂₂) pairs stretches into a long thin diagonal streak that runs from −0.91 to 2.52 on one axis.

Now look at what is pinned and what is free. The bootstrap spread of the sum of the two coefficients is 0.070 at ρ = 0.90, and 0.070 at ρ = 0.99. Identical. The spread of each one alone triples over the same move, and their correlation across resamples reaches −0.992.

That is the entire disease in one sentence. A stable projection with unstable coordinates. It is the dummy trap with the word "nearly" inserted. The plane is fine and the point on it is fine, but the two columns naming it point in nearly the same direction, so many different recipes reach nearly the same place.

Chapter 8 gives the same fact as a landscape. The RSS surface, which for independent columns is a clean bowl, becomes a long flat canyon running along the direction where the two coefficients trade off. A canyon floor is exactly where small changes in the data slide you a long way. Chapter 7 gives it as a number: XᵀX is ill-conditioned, the condition number is huge, and inverting it magnifies noise.

Now measure it, using something you already own. The coefficient is priced on the leftover of x_j after the other columns are subtracted. If the other columns explain most of x_j, that leftover is a stub. How much of x_j do they explain? That is an R², the R² of regressing predictor j on all the other predictors.

Call it R²_j, and the variance inflation factor is VIF_j = 1/(1 − R²_j), the factor by which the coefficient's variance is multiplied. So R²_j = 0.9 gives a VIF of 10 and a standard error 3.16 times larger. VIF is not a new formula. It is R² pointed sideways at your own design matrix.

Recognise the signature in the wild, because it looks like a contradiction and is not. A strongly significant joint F-test, a high R², and not one individually significant t-statistic. Together the columns explain plenty. Individually no single one can be credited, because each one's unique contribution is a stub.

And notice that multicollinearity is not a violation of A1 to A4 at all. β̂ remains unbiased and still best linear unbiased. The data is simply mute about the split. So sort the fixes by what you actually want out of the model. If you only want to predict, then do nothing at all, because the projection was never harmed. If you must attribute, get data where the columns separate, or combine them into one factor with PCA in Chapter 24, or drop one on economic grounds and say so out loud. Or accept a little bias to buy a lot of stability, which is ridge regression in Chapter 22, and geometrically it is the operation that rounds the canyon floor back into a bowl.

13Two liars about the error bar

Two violations left, and the tier table tells us in advance what to expect from both. They sit in the standard-error row, so the coefficient survives and only the error bar lies. That is genuinely good news and genuinely dangerous news, because a wrong error bar is invisible.

First, heteroscedasticity. The spread of the errors depends on X. The picture is a fan, which you get by plotting the residuals against the fitted values and watching the cloud widen as you move right. In finance this is not an exotic failure. It is the default, because Chapter 18's leverage effect and volatility clustering mean the scale of returns changes through time.

Here is what breaks and what does not. Across 400 independent repeats the coefficient averages 0.9929 against a truth of 1.0, so β̂ is still unbiased. What fails is the arithmetic of the error bar. The classical formula averaged one σ² across observations that do not share one, and it reports 0.0472 when the true spread of the estimate is 0.1008. The t-statistic prints 20.65 and has no right to.

The fix is elegant and cheap. Keep β̂, and recompute its variance with each observation's own squared residual in the middle. Those are White, or heteroscedasticity-consistent, standard errors, and the shape is a sandwich: (XᵀX)⁻¹ Xᵀ diag(eᵢ²) X (XᵀX)⁻¹. On this run the robust standard error reads 0.0836, which is 1.77 times the classical one and much closer to the truth. An honest note goes with it. Robust closes most of the gap and not all of it, since 0.0836 still sits under the true 0.1008 in a finite sample. It is a repair, not a guarantee.

Second, serial correlation, and this one is the quiet killer in finance because it does not look like a mistake. The residuals predict their own neighbours, so today's leftover tells you something about tomorrow's. Read it two ways: the ACF of the residuals, which is Chapter 18's correlation applied to a series against its own lag, or the Durbin–Watson statistic, which is about 2(1 − ρ̂₁), so 2 means clean and near 0 means badly positively autocorrelated.

What breaks is the same thing. β̂ is unbiased and the error bar is wrong. The magnitude is what people miss, and it comes down to one idea. Effective sample size. If consecutive rows carry overlapping information, then n rows are not n independent facts, and a formula that divides by √n is dividing by the wrong number.

Now the specific finance trap, which is manufactured by bookkeeping rather than by markets. Sample the 12-month forward return monthly, and consecutive observations share eleven of their twelve months by construction. You did nothing wrong. You built the dataset the natural way, and adjacent rows are nearly the same measurement.

Predict before the arithmetic, because the gap between the guess and the answer is the whole lesson. You have 240 such monthly observations, so how many independent ones do you actually hold?

People say 240. The honest count is 20 non-overlapping windows. Across 4,000 runs of a zero-edge series built this way, the t-statistic comes out with a spread of 3.655 when an honest t-statistic has a spread of 1.000. That is close to the bookkeeping factor √12 = 3.464. So a nominal 5% test fires in 58.3% of runs, and the t-statistic of 4.0 that ended the meeting is really about 1.09.

Hand that straight back to Chapter 19. Understated standard errors are a false-discovery factory, and a research group with this bug in its pipeline generates significant strategies indefinitely.

The fixes are known. Newey–West HAC standard errors widen the bar to account for dependence out to a chosen lag. Or model the dependence explicitly. Or simply stop overlapping. And the honest caveat stands: HAC is a patch, it needs a lag choice, and it is not absolution.

14The diagnostic kit

Every violation above announced itself in a picture, and not one of them announced itself in the numbers. So here is the argument for looking, and then the kit.

The argument is Anscombe's quartet. Four datasets with the same intercept of 3.00, the same slope of 0.500, the same standard error of 0.118, the same t of 4.24, and the same R² of 0.667. Before you see them, guess how similar the four scatter plots are.

They are a plausible linear cloud, a perfect parabola, a perfect line with one outlier dragging it, and a vertical stack of points at one x-value plus a single distant point. In that last one the far point carries a leverage of 1.000, which means it sets the entire slope by itself. Identical summaries, four completely different worlds, one of which is not even a relationship.

Now the kit, four instruments and what each is for.

First, residual-versus-fitted, the general-purpose instrument. It should look like a shapeless cloud, and any structure is a message. Systematic curvature means the functional form is wrong. A widening fan means heteroscedasticity. And any pattern at all means there is signal left in the leftover, which by the exploitation argument means a column you have not included.

Second, the Q-Q plot, and we build it rather than show it, because it is the plot people pattern-match without understanding. Sort your residuals. Ask what the k-th smallest of n standard Normal draws should be, which is Chapter 13's order statistics sitting at a predictable expected position. Plot observed against expected. If the residuals are Normal, the points lie on the diagonal.

Notice what that is. It is exactly the picture from Chapter 19, where sorted p-values were laid against the diagonal pure noise would have produced, and the departure was the finding. Same move, different sorted quantity. So the reading is one sentence: ends peeling away above and below the line mean your tails are fatter than the Normal's. That is the default in finance, which is why A5 is the assumption to worry about least and the CLT is the thing carrying your inference.

Third, the ACF of the residuals and the Durbin–Watson statistic, which between them read the dependence from one row to the next. Fourth, the VIF table, which reads how much of each predictor its neighbours already account for.

And then one more instrument that gets skipped and should not. Influence. The projection can be written ŷ = Hy with H = X(XᵀX)⁻¹Xᵀ, called the hat matrix, which is nothing other than the projection operator itself. It is the machine that turns y into ŷ. Its diagonal entry hᵢᵢ is how much observation i determines its own fitted value, and that is called leverage.

The trace of H is k, so the leverages sum to k and the average is k/n. Put one far-out point in a six-point sample and you can watch it happen: the leverages read 0.2598, 0.2252, 0.1987, 0.1801, 0.1694 and 0.9668, summing to exactly 2.000, which is the column count. Anything past about 2k/n is a point with a loud vote. Combine leverage with the size of a residual and you have Cook's distance, which asks how much the whole coefficient vector moves if you delete that one observation.

Then the finance sting, which is a genuine dilemma rather than a rule. In a beta regression the single most influential day is almost always a crisis day. Delete it and beta looks stable, and your risk model has been taught to be blind to precisely the event it exists to measure. The diagnostic tells you where the number came from. It does not tell you to throw the day away.

15The bootstrap, checked before it is trusted

One tool left, and it is the one to reach for when you would rather not sign the fine print at all.

Every error bar so far came from a formula that assumed something. Constant variance, independent errors, sometimes normality. Suppose you want an error bar for something with no formula, like a Sharpe ratio, a maximum drawdown, a median, or a ratio of two coefficients. Or suppose you simply do not believe A3 and A5.

Go back to Chapter 16's definition. The sampling distribution is the law of your statistic over all the samples you could have drawn from the population. The reason it is unreachable is that you have exactly one sample and no access to the population.

Here is the whole idea of the bootstrap, in one sentence. You do have an estimate of the population, because your sample is one: the empirical distribution that puts mass 1/n on each row you observed. So sample from that.

Draw n rows with replacement, refit the model, and record β̂. Do that a few thousand times and you are holding a cloud of estimates you can actually plot. Its standard deviation is a standard error, and its 2.5th and 97.5th percentiles are a confidence interval, with no formula and no distributional assumption anywhere.

Now, everyone's first reaction is that this is cheating. You invented data and learned something from nothing. It deserves a straight answer rather than a reassurance. You are not inventing data. You are sampling from your best available estimate of the population, which is the same plug-in principle you used every time you put s in place of σ.

And you can check it rather than trust it, which is the move that should make it credible.

Bootstrap the sample mean, where Chapter 16 already tells us the right answer. The formula says s/√n = 0.1096. Ten thousand resamples say 0.1102. The intervals agree to three decimals as well. Validate the new instrument against a formula you already believe, on a case where the formula is known to be right, and then take it where formulas run out.

So take it there, and point the same procedure at the heteroscedastic regression from the fan, where the classical formula reported 0.0472 against a true spread of 0.1008. Resampling whole rows gives 0.0848, which lands right beside the robust answer of 0.0836. The bootstrap found the correction without being told that anything was wrong. And on a statistic with no clean formula, the median of that same sample, it returns a standard error of 0.1744 with no extra theory required.

Two flavours are worth distinguishing. The pairs bootstrap resamples whole rows together, which makes no assumption about the error variance and therefore survives heteroscedasticity. The residual bootstrap refits, then reshuffles the residuals onto the fitted values, which assumes the errors are interchangeable across observations and therefore quietly assumes homoscedasticity back in.

Then the honest limits, because the bootstrap is sold as assumption-free and is not. It assumes the rows are iid, which is precisely what serially correlated financial data is not, so the naive bootstrap on a time series is as over-confident as the classical standard error. The repair is the block bootstrap, resampling contiguous chunks so the local dependence rides along inside the block.

And the harder limit. The bootstrap estimates spread, never bias. Bootstrap the endogenous regression and it returns a beautifully tight 95% interval of [1.972, 2.209] around a truth of 1.000. Every resample inherits the same contamination, so the tightness is real and the aim is still wrong.

One loan forward. This is the tool that gives a Sharpe ratio a standard error in Chapter 30, where no clean formula exists at all.

16The audit, in order

Put it back together, because after fifteen sections the risk is that the lesson reads as "regression is fragile, check five things". That is demoralising and it is also wrong.

Here is the accurate summary, in two sentences. A regression is an orthogonal projection, always valid, always computable, never in question, and it tells you the closest point reachable from the columns you supplied. Everything else is a stack of claims about the world, each one auditable separately.

And the audit has a fixed order, because the tier table sorted it for us.

One. Do the coordinates exist, and are they identified? Check rank, check VIF, and remember that a stable projection can carry wildly unstable coordinates. If you only need predictions, this question may not even matter.

Two. Is the number aimed at the right target? What is omitted, what runs in both directions, what is measured with noise, who was selected out of the sample. This is the only question in the whole chapter whose wrong answer poisons the coefficient, and no computational repair reaches it.

Three. Is the error bar honest? Robust standard errors as the default, HAC when the data is a time series, the bootstrap when the statistic has no formula. And always ask what the effective sample size really is.

Four. Is the shape right? Plot the residuals, because four datasets can agree on every printed number and disagree about everything that matters.

Notice what that ordering does. It puts the fatal question second, above the two that most practitioners spend their time on. And it puts the number everybody quotes, R², nowhere on the list at all. In-sample R² cannot fall when you add columns, it reaches 1.0000 for a model that has memorised its data, and it is silent about causation, functional form and the future.

Which is precisely the shape of the door into the next chapter, and it is worth naming the shape exactly.

Every diagnostic in this one was computed on the same data the fit was chosen on. The residuals were made orthogonal to the columns by construction. The R² was maximised on those rows and no others. And Chapter 19 already taught us that a search leaves its fingerprints on the number it selects.

So there is exactly one question none of this machinery can answer. Does the relationship hold on data the fit has never seen? Answering it needs no formula, no assumption and no correction factor. It needs data the search never touched, and that is Chapter 21.