21In-Sample vs Out-of-Sample

Chapter 20 closed with a confession, and this chapter is what that confession costs. Every number the machine printed was computed on the rows that chose the coefficients. The R², the t-statistics, the residual plots: all of them scored the fit on its own homework. There is a second question hiding behind that apparatus, and it is the only one that ever pays you. Not how well does this fit the data I have. How well does it predict data I have never seen. Those two questions have different answers. The gap between them is not bad luck, it is manufactured by the act of fitting, and you can compute its size before you run anything. The instrument that measures it is embarrassingly simple. Hold some rows back, fit on the rest, and score on the rows that had no vote. Everything else in this chapter is about protecting those rows from voting, because information leaks into them through channels you cannot see in the code. Fit is what you got. Predict is what you get paid for.

Look at what this page stands on, because nearly all of it is already yours. Chapter 20 gave us least squares as a projection, the fact that ŷ is the nearest point of the column space, and the confession that every diagnostic it built was computed on the fitting rows. It also gave us the dimension count behind n − k, and the warning that R² can only rise when you add a column. Chapter 19 gave us the winner's curse, the family of tests you never counted, and the handoff line that matters most here: selection luck belongs to the sample it was selected in. Chapter 16 gave us bias as a separate disease from noise. That is the whole toolkit.

So the seam is a single sentence. Chapter 20 built an apparatus and admitted it could not answer whether the fit holds on rows it has never seen. There is exactly one honest next move available, and it is to ask that question out loud and go and get the number.

01The question Chapter 20 could not ask

Here is a research note of a kind you will read a hundred times. A moving-average crossover on one equity index, tested over ten years of daily data. Buy when the fast average crosses above the slow one, sell when it crosses back. The backtest returns +18% a year.

The desk funds it. Twelve months later the strategy has returned −2%.

Before you read another line, commit to an answer, because the whole chapter is one long reply to whatever you pick. Is that gap bad luck, or is something about the +18% not what it appears to be?

Name the object properly first. A backtest is a simulated history of decisions — the strategy's rules replayed over past data, trade by trade, as if you had been there. It is not a track record, and nobody ever made or lost a penny inside one. It is a measurement, and like any measurement it can be biased by the way it was taken.

And there are two different questions tangled up inside that measurement. One is how well does this model fit the data I have. The other is how well does it predict data I have never seen. The rows the model was fitted on are called in-sample. Rows it never touched are called out-of-sample. Chapter 20 answered the first question in exhausting detail and could not touch the second.

Now here is the ugly empirical fact that makes this a chapter rather than a footnote. Across the whole industry, live results come in below their backtests, and they do it systematically. Not sometimes above and sometimes below. Below.

02This is not the noise you already know

Most readers have already filed that gap in the wrong drawer, and it is worth stopping to unfile it. Part 2 spent five chapters teaching you that a sample can be trusted once you attach the right error bar. So the natural reaction to +18% and −2% is: small sample, wide interval, nothing new here.

That reaction is wrong, and Chapter 16 already gave us the tool that shows why. It taught two distinct diseases, and you have been using only one of them.

The first is sampling noise, and it is scatter — your estimate lands around the truth, too high about as often as too low, with a spread that shrinks like σ/√n. The second is bias, defined as bias(θ̂) = E[θ̂] − θ, and it is a number that comes out wrong in the same direction every single time you run the experiment. Only one of those two is cured by collecting more rows.

So ask the diagnostic question directly of the backtest itself. If I re-ran this entire research process on a different five years of data, would the reported backtest return come out too high about as often as too low? The answer is no. And that single no is what separates this chapter from a footnote about error bars.

The dartboard is the picture to keep. Scatter is darts spread around the bullseye, and the cure is to throw more darts and take the average, which is exactly what a larger sample does for you. Bias is a board bolted to the wall six inches off-centre. Throw ten thousand darts, average them with beautiful precision, and you land six inches off.

Ground that back on the spot, because an analogy left floating is worth nothing. The backtest number is the average of the darts. The offset of the board is a quantity we will compute exactly, and it turns out to be k/n, the amount of freedom you had divided by the amount of data you had.

Which gives the sentence to carry through the rest of the chapter. More data cures variance. More data does not cure optimism. In practice a bigger dataset usually makes things worse, because it tempts you to spend the extra rows on extra freedom, and the ratio that governs the damage stays exactly where it was.

03The only instrument there is

You cannot detect a one-sided bias without a yardstick that does not share it. So the design of the instrument is forced rather than chosen. We need a score computed on rows the fit never touched.

The mechanics take one line: take the data, hold a block of it back, fit the model on what remains, and then score that fitted model on the block. Two numbers now exist where there was one. The in-sample score, computed on the fitting rows, and the out-of-sample score, computed on the held-back block. The difference between them has a name — the generalization gap — and measuring it honestly is what the rest of this chapter is about.

Before pointing that at anything real, point it at a world where we already know the answer. Chapter 19 built the noise factory: candidate strategies with genuinely zero edge, by construction, so that any result the machine reports is manufactured and there is nothing to argue about. Dress it in Chapter 20's clothes. Two hundred and fifty rows of coin-flip returns as the target, and fifty columns of pure random noise as the predictors, drawn from a generator that has never heard of the target.

The true relationship is exactly zero, so every coefficient in that regression should come back as nothing. Run it.

The in-sample R² comes back 0.2000. We built a world with nothing whatsoever in it, and the machine reports that a fifth of the variation is explained.

Two honest asides before we chase that number down. The first is the question everyone asks about the split, which is why 80/20 and not something else. There is no rule, and anyone who gives you one is inventing it. The split size is a trade between how well you can fit and how precisely you can measure, and we will improve on the whole arrangement later in the chapter. Right now the split is not a convention. It is the only way to get an untainted yardstick at all.

The second aside is a warning about a shrug. Chapter 20 already told us that R² rises when you add columns, so it is tempting to file 0.2000 under "yes, R² is inflated, use adjusted R² instead" and move on. Please do not. That shrug treats the disease as cosmetic and already patched. By the end of this chapter you will be able to predict that 0.2000 before running anything, from two integers, and you will see exactly what adjusted R² was doing all along.

04★★ Fitting is searching

Why does a fit to nothing report something? The answer is sitting in Chapter 20's own words, and almost everybody reads straight past it.

Chapter 20 said that ŷ is the nearest point of the column space. Read that literally. Nearest means best of every candidate point. Least squares does not find β̂ the way you find a key in a drawer. It holds a competition among every possible coefficient vector, judges them all on the same rows, and reports the champion. The closed form β̂ = (XᵀX)⁻¹Xᵀy is just the shortcut that skips the contest and hands you the winner.

The objection arrives immediately, and it is an honest one. I did not run a thousand tests. I ran one regression. That objection is exactly why this goes unnoticed. Chapter 19 built the winner's curse on a countable m, and here the family is a continuum with no m to count. Nobody sees a search because a formula arrives looking like an answer.

So let us draw the competition the formula hides. One predictor, one coefficient, twelve training rows and two hundred held-out rows. Sweep β across a range and plot the squared error twice: once on the training rows, once on the rows held back.

Two bowls, and they do not sit on top of each other — that offset is the entire disease, drawn at the smallest size it can exist. The training bowl's floor sits at 1.0588, which is by definition β̂, the answer the formula would hand you. The held-out bowl's floor sits at 0.5032, and the coefficient that actually generated the data was 0.500. We did not find the best β. We found the bottom of the wrong bowl, and standing there costs 25.9% extra error on the rows that were held back.

Now scale it up in one line. With k parameters you are locating the floor of a k-dimensional bowl, and every extra dimension is one more direction in which the search can get lucky on these particular rows.

Which brings us back to the noise regression, where in-sample R² is already showing 0.2000 on data built to contain nothing. Before anything runs, write down one number. What will the out-of-sample R² be? Nearly everyone writes zero. Some write slightly above zero. Commit, then look.

It is −0.2424. Negative. The model did not merely fail to help. It did actively worse than a flat line drawn at the mean of the held-out rows.

And the right objection arrives instantly. Chapter 20 proved that R² = cos²θ, and a squared cosine cannot be negative. So which chapter is lying?

Neither. R² is two different objects wearing one name, and here is the single stroke that separates them. In-sample, TSS = ESS + RSS is Pythagoras on a right triangle. The triangle is right-angled only because the residual is perpendicular to the fit, and that perpendicularity is the normal equations, which hold only on the rows the fit minimised over. Carry β̂ to rows it has never seen and the residual is perpendicular to nothing. The triangle collapses, Pythagoras goes with it, and 1 − RSS/TSS is free to fall straight through zero.

You can watch it in the arithmetic. On the fitting rows the dot product of the fit with the residual is −2.8 × 10⁻¹⁴, which is zero to machine precision. On the held-out rows the same dot product is −58.253. And the residual sum of squares out of sample is 309.63 against a total sum of squares of 249.22, so the model's errors are 24.2% larger than simply guessing the average every time.

Now put the two scores side by side, because the shape of the pair is the point. +0.2000 in, −0.2424 out. The amount it appeared to help is almost exactly the amount it hurts. That mirror is the winner's curse being paid back with interest. The search borrowed this sample's noise in order to win the contest, and on any other rows that borrowed noise is simply a wrong answer carried forward.

Which forces us to re-label two things you thought you understood. Overfitting is not a model being too wiggly, and that mislabel is why people believe a linear model cannot overfit. It is a property of a procedure that had too much freedom. And k, the number of fitted parameters, is not a modelling detail. It is the size of the search.

05The exact price of freedom

So put a number on the freedom. On pure noise the expected in-sample R² is exactly k/n, and the derivation is short enough to hold in your head. Two integers, and you can predict the optimism before you run anything.

Chapter 20 established that fitting spends dimensions. The residual is forced perpendicular to k independent directions, so it lives in the remaining n − k. Now add the one extra ingredient. If the target is pure noise, no direction is special, so by symmetry each of the n directions carries on average 1/n of the total variance. Hand the search k of them and it captures k/n. That is the whole argument.

Check it against the number we already watched appear. Fifty columns, two hundred and fifty rows, k/n = 0.2000, and the observed in-sample R² was 0.2000. Averaged over four hundred repeats the match holds at every rung of the ladder: 0.0196 against 0.020 at k = 5, 0.1019 against 0.100 at k = 25, 0.4023 against 0.400 at k = 100.

Reflect the same argument onto fresh rows and the expected out-of-sample R² is about −k/n. The exact version is −k/(n−k−1), which for our regression predicts −0.2513 against an observed −0.2424. So the generalization gap on noise is roughly 2k/n, and every one of those quantities is a ratio. The parameter count on its own says nothing. Fifty columns is a catastrophe on two hundred rows and barely noticeable on fifty thousand.

Now collect the payment that has been sitting in Chapter 20 unclaimed. Adjusted R² was defined as 1 − (RSS/(n−k))/(TSS/(n−1)), and σ̂² = RSS/(n−k) divided the residual by its honest dimension count. Both were taught as bookkeeping. They are this exact toll, charged in advance and in miniature. On the pure-noise ladder, adjusted R² averages −0.0005, +0.0006, +0.0016, −0.0010, +0.0011 as k runs from 5 to 100. It reports zero at every level of freedom, which is the correct answer, while raw R² climbs to 0.40.

That is degrees of freedom finally cashed. Not an incantation you subtract to keep software happy. A literal count of the dimensions the residual is still allowed to occupy, and the price of the ones you took away.

One warning before we move on. It is tempting to file k/n as a fact about linear regression. It is not. It is a statement about search freedom, and it will come back untouched around trees, around neural networks, and around a hand-tuned trading rule with no parameters in it at all.

06The U, watched rather than asserted

Everything so far ran on data with nothing in it. So put something real in and run the identical sweep.

Here is the design, and it is deliberately shaped like real quant data. One hundred and twenty candidate factors, each carrying a genuine but weak relationship to the target, with the j-th coefficient set to 0.30/√j so that they fade as you go down the list. Two hundred training rows, and eight thousand fresh rows held back for scoring. The true R², the ceiling that even a model handed the exact answer could not beat, is 0.3258.

Now fit models of increasing complexity, taking the strongest k factors each time, and score each one twice.

The in-sample error falls at every single step, from 1.2482 down to 0.3317, and it has no choice about it. Chapter 20's geometry guarantees that a bigger column space cannot be further from the target.

The out-of-sample error does something completely different. It falls from 1.4456 to a floor of 1.2182 at k = 12, then it turns and climbs all the way to 2.8020 by k = 120. Past that turn every extra column buys more fake fit than real signal, and the k/n toll quietly eats the difference.

Only now are the names worth handing over, because you have watched both failures happen with your own eyes. The left arm is underfitting — a model too rigid to hold the structure that is genuinely sitting there in the data. The right arm is overfitting. And the floor between them is not a property of the model family at all: it is the right complexity for this much data.

Which kills the over-correction you may already be forming. Everything since the noise regression makes simpler is always safer sound like wisdom, and the left arm of that curve is a picture of it losing money. At k = 1 the model captures 0.0538 of the available variation when 0.1548 was on the table.

Run the same sweep at four sample sizes and the point sharpens. With 50 rows the best choice is k = 1. With 150 rows it is k = 12. With 600 rows it is k = 45, and with 2400 rows it is k = 100. Same signal, same model family, same everything. Only n changed, and the floor walked right.

Now the fact that makes the entire rest of this chapter necessary rather than procedural. You cannot locate that floor from in-sample numbers alone. There is no formula that reads a training fit and returns the right k. The minimum is only visible if you have an out-of-sample curve, which means the rest of the chapter is about how to get one honestly.

And one thing we are deliberately not doing yet. We have watched the U-curve, and we have not taken it apart. Splitting that error into bias squared, variance and irreducible noise is Chapter 22's entire job, and it is what turns this diagnosis into a dial you can turn.

07The freedom nobody counts

k counts what the optimiser searched. It does not count what you searched.

The 60-day lookback you settled on after trying 20 and 120. The entry threshold you nudged from 1.5 sigma to 2 sigma. The universe you narrowed to liquid names because the small caps looked messy. The three signals you dropped and the two you kept. The rerun after a disappointing result. Every one of those is a parameter fitted to the data by a human being, and by Chapter 19's accounting each is another member of the family.

None of them appears anywhere in the regression output. The information travelled through your head rather than through the code, and that is precisely why careful, honest people ship overfit strategies while believing they were rigorous. Nothing on screen changes when it happens. The code is identical, the diagnostics are identical, and the model has no idea.

Which forces an upgrade to the definition we have been using, and it is the most important sentence in the chapter after the keystone. Out-of-sample does not mean later in the file. It means data that had no opportunity to influence any decision. Information flow, not chronology.

The corollary is brutal. The test set is a consumable. The first time you read its score and change anything in response, its information has entered your model through you, and no re-split of the same file can restore it.

People accept that as etiquette and then peek anyway, because one look feels harmless. So price it instead of moralising about it. Thirty variants tried against a holdout is m = 30, and Chapter 19 already told us what that costs. On data with no edge whatsoever, the chance that at least one of thirty tries clears a t of 1.96 is 78.5%, and the expected best t of the thirty is 2.04. That is the number that gets a strategy funded.

Give the whole phenomenon its name. Backtest overfitting is the winner's curse charged on a research process rather than on a single fit, and it is invisible in every printout the process produces.

08Three jobs, then rotation

If the test set can only be read once, and you still have choices to make, then the choosing has to happen somewhere else. So split by job rather than by name.

Train fits the parameters. Validation chooses among models and settings, which are the things you would otherwise have tuned against the test set. Test is read exactly once, at the end, and reported. Define them by the question each one answers rather than by its label, because that framing survives a colleague using the words backwards. A setting chosen by search rather than by the optimiser has a name too: it is a hyper-parameter.

Now improve the middle one, because a single validation block is a small sample, and Chapter 16 says a small sample gives a wide standard error. It also wastes rows you could have been fitting on.

The picture is the definition. Cut the training data into k blocks and go round k times, fitting on k−1 of them and scoring on the single block left out. Average the k scores. Every row gets exactly one turn as a scoring row and k−1 turns as a training row, so nothing is wasted. That is k-fold cross-validation, and you could have invented it yourself from the grid.

The reason to bother is variance, and it is worth measuring rather than asserting. Repeat the whole exercise on three hundred fresh datasets. A single 40-row holdout gives an estimate with a standard deviation of 0.3079. The five-fold average gives 0.1465. The wobble is more than halved.

Notice it is not divided by √5 = 2.24, which is what Chapter 13's law of large numbers would give for five independent estimates. The observed ratio is 2.10, because the five fold-models share training rows and are therefore not independent. Push that to the extreme and you get leave-one-out cross-validation, where k = n. Its folds differ by a single row each, so they are almost the same model, and averaging them settles the number far less than the count suggests.

And here is the caveat almost nobody states out loud. Those k fits are k different models. On our own run the coefficient on the leading factor comes back as 0.3611, 0.2688, 0.2339, 0.2607 and 0.2850 across the five folds, while the model you would actually deploy, refitted on all two hundred rows, has 0.2809.

So a cross-validation score estimates the out-of-sample performance of a procedure. Fit this recipe on this much data. It does not estimate the performance of the single model you are about to ship, which was fitted on all n rows rather than on n(k−1)/k of them. The mismatch is small when k is large and it is not small when k is 2, and nobody tells you.

09The assumption inside the shuffle

k-fold has an assumption inside it that nobody says out loud. Shuffling the rows before you cut them into blocks assumes the rows are exchangeable, meaning a row is a row and it does not matter which ones land in the scoring block.

Financial rows violate that assumption twice over. First, they carry an arrow of time, and a shuffled fold will cheerfully train on next year in order to predict last year. Second, and this is the one that does the real damage, they are dependent — today's row shares information with yesterday's, so a row is not a row.

Before any definitions, take a number. Here is a random walk, a running sum of independent coin flips. Its level is enormously persistent, and its increments are pure noise by construction, so there is provably nothing to predict. Set up a shuffled 5-fold that predicts today's level from yesterday's level. What R² do you expect?

The shuffled folds report 0.9865, and they agree with each other beautifully: 0.9853, 0.9864, 0.9871, 0.9819, 0.9920. Five independent-looking confirmations of a model that knows nothing.

Look at what it actually learned. The fitted slope is 0.9988 with an intercept of 0.0626, which is the model saying tomorrow equals today. Score the only quantity you could ever trade, the day-to-day change, and the R² is +0.0001. The correlation between predicted change and actual change is +0.0106.

Now define the thing that made it possible, because the definition lands properly once you have felt the phenomenon. The autocorrelation at lag h is ρ(h) = Corr(x_t, x_{t+h}), which is Chapter 18's correlation with the series shifted against itself. Read across all the lags and you have the ACF. For our walk, ρ(1) of the level is +0.9871 and ρ(1) of the increments is −0.0721, which is what zero looks like when you measure it on 500 rows.

When ρ(1) is high, every held-out row has a near-twin sitting in the training set, one day away. The model does not have to predict anything at all. It only has to recognise a neighbour it has already memorised, and interpolate between the two rows on either side.

The second half of that run shows the same leak in the form quant work actually meets it. Chapter 20 flagged the mechanism: a 20-day forward return computed every day makes neighbouring labels 95% the same data. So take a feature built from an independent noise source and smoothed over 20 days, and a 20-day overlapping forward return as the label, with a true relationship of exactly zero. The sample correlation over the whole file comes out at −0.4091. Shuffled 5-fold reports +0.1455, positive in all five folds. Walk it forward in time instead and the average is −0.0801.

And the blocks of that walk-forward run swing from −0.6429 to +0.2201, which is the honest signal here. Five hundred rows, but the effective sample size is about 500/20 = 25 independent observations. There was never enough information in that file for a stable number, and both the split and every error bar you compute from it are lying about how much you know.

One naming trap to close, because it catches good people. Chapter 20 met autocorrelation as a residual pathology, fixed by HAC standard errors. Here it is a property of the series that invalidates an entire resampling scheme. Same word, different job, and the HAC fix does not cover this one.

10Walk forward, and cut a hole

Two violations were named, so fix exactly two things.

For the arrow of time: fit on everything up to time t, predict the block that runs from t to t+h, then roll forward and repeat. That is walk-forward validation, also called rolling-origin cross-validation, and it is the only scheme in which every single prediction was made using information that already existed when it was made.

For the dependence: the rows on either side of the boundary are still near-twins, so remove them. Purge the training rows whose labels overlap the test window, and embargo a buffer of test rows immediately after the boundary, in case the feature construction reaches backwards as well.

The width of that hole is not a taste parameter, and this is where purging stops feeling like a hack bolted on to rescue a method. The purge width IS the label horizon. If your target is a 20-day forward return, the overlap you measured is 20 days, so the hole is 20 days wide. You are removing exactly the rows whose answers were partly written by the test period.

There are two flavours of the training window, and they are two different bets about the world. An expanding window keeps all history and bets that the relationship is stable. A rolling window keeps only the last N rows and bets that the world changes, paying for that adaptivity with a smaller n and therefore a larger k/n toll. Neither is correct in general. They are hypotheses, and you should be able to say which one you are making.

Two practical warnings. Stitch the fold predictions end to end and you get an out-of-sample equity curve, which is the closest thing to an honest simulated track record you will ever produce. But the earliest folds are fitted on very little data, so their scores are dominated by k/n rather than by the strategy, and averaging them in tells you about your sample size rather than about your idea.

Now the temptation, said out loud because it is the thing that actually happens. Walk-forward returns worse numbers and fewer of them, and there is a k-fold script sitting right there that gives lovelier results in one line. The answer is not discipline, it is arithmetic. We watched the prettier number report +0.1455 on a relationship that is exactly zero. It is not a better score. It is a different measurement.

The advanced version of this same idea has a name worth knowing: combinatorial purged cross-validation, which runs many purged train-test arrangements rather than one path, so you get a distribution of outcomes instead of a single number. Same two fixes, applied many times over.

11Information that did not exist yet

Everything so far assumed the data itself was honest. It usually is not, and one law covers all of the ways it fails. You may use only information that existed, in the form it existed, at the instant of the decision.

Most readers will now think of using future prices, decide they would obviously never do that, and read the rest of this section as a list of other people's mistakes. That is the exact inch where the section fails, so let me be blunt about it. Real leaks are indistinguishable from competent code, and every one of them is a line you have written without a second thought.

Start with the one that catches good engineers, because it survives a perfect walk-forward split with a correctly sized purge. It happens on the line above it. Standardise your features with the full-sample mean and standard deviation, impute missing values with the full-sample median, winsorize on full-sample percentiles, or fit a PCA on everything, and the test set's information has already walked into the training pipeline before the split ever ran.

The rule that fixes it is short. Every transform that estimates a number from data must be fitted inside the fold, on training rows only, and then applied unchanged to the rows you held out. A scaler is a model. It has parameters — a mean and a standard deviation — and they have to be learned from training data like any others.

Then the catalogue, each with its mechanism. The overnight-to-open gap: you compute a signal from today's close and the backtest fills you at today's close, which is impossible, because the close is the last price of the day and you only learned it after trading stopped. The earliest you could act is tomorrow's open, and a large share of the move lives in that jump. Survivorship bias: the universe file you downloaded today contains only the names that survived, and the ones that vanished were the bad ones. Restatement: fundamentals get revised, so the value in the database now is not the value published then, and you need point-in-time data. The off-by-one label: align the forward return one bar early and you have built a money machine out of a typo.

None of these throws an error. None of them looks like cheating. So carry the tell instead. A spectacular result is a bug report. Chapter 16's arithmetic makes it concrete: t = SR·√T, so a Sharpe ratio of 8 measured over two years is a t-statistic of 11.3. Nothing real produces a t of 11 on two years of data. Go and find the leak.

12When the columns outnumber the rows

Now push the ratio to its limit, because that is where quant data actually lives. Five hundred candidate factors against twenty years of monthly returns is 500 columns and 240 rows. This is not an exotic corner case from genomics. It is the default shape of a modern factor research problem.

Chapter 6 and Chapter 20 answer immediately. When k reaches n the columns span all of ℝⁿ, the target lies inside the column space, the projection is the target itself, the residual is exactly zero, and R²_in = 1. Push past it and β̂ is not even unique: infinitely many coefficient vectors reproduce the data exactly.

So walk the ladder past the wall on data that contains nothing, and watch what the software does at each rung.

At k = 239 the design matrix has 240 columns for 240 rows, in-sample R² reads 1.000000, and the largest residual in the whole file is 8.2 × 10⁻¹⁴. A perfect fit, on pure noise. Out of sample the same model scores −302.58.

Past the wall it gets stranger and quieter. At k = 500 the rank is stuck at 240 with 261 free directions, so the answer is genuinely non-unique, and the software returns one anyway. Here are two solutions to that same problem. One has a coefficient vector of length 0.8794, the other of length 37.0104, and a single coefficient differs between them by 22.82. Their predictions agree to 9.7 × 10⁻¹⁴, and both have a residual sum of squares of essentially zero.

Nothing on the screen warns you that any of this happened. That is the minimum-norm pseudo-inverse Chapter 20 named, quietly picking one member of an infinite family and printing it with a full table of coefficients, standard errors and p-values beside it.

Which gives the sentence for the door. A perfect fit is the loudest possible failure signal. Every scrap of variance is explained, and not one bit of it is information.

There are exactly three exits, and they are the same move wearing different clothes. Constrain the fit so the search cannot use all k directions freely. Reduce the dimensions before fitting. Or select fewer features, honestly, inside the fold. All three are Chapter 22's door.

13The protocol, and what it cannot reach

Write the chapter down as something you could hand a colleague on their first day, with each rule carrying the failure that forced it.

State the question and the horizon before you look at anything. Split by time, with a purge sized to the label horizon. Build every transform inside the fold. Tune only on validation folds. Touch the test set once, and write down what you did to earn the right to touch it. Keep an honest tally of everything you tried, and expect Chapter 19's haircut on whatever survives. And treat a spectacular result as a bug report until you have found the bug or proved there is not one.

Now the limit, stated plainly, because a protocol that promises safety is the most dangerous object in this chapter.

This does not make you safe. The final decision, to deploy this strategy out of everything you looked at, is itself a selection made after seeing all the evidence. No amount of internal hygiene can make that decision out-of-sample, because there is no data left that it did not see. The only truly clean sample is the money.

Which reframes the industry fact we opened with. Live results decaying relative to backtests is not a sign that something went wrong. It is the expected behaviour of an honest process, and the right expectation for a well-built strategy is somewhat below its backtest, not at it.

So be precise about what this chapter buys. It does not buy immunity. It converts an unbounded, invisible, one-directional bias into a bounded and stated one. That is all it does, and that is everything.

And it leaves one door standing open. We watched the out-of-sample curve fall, turn and climb, and we never took the turn apart. Why does the left arm lose? Why does the right arm lose? The moment you decompose that error into bias squared, variance and irreducible noise, overfitting stops being a hazard you avoid and becomes a dial with a knob on it. That is Chapter 22, and the knob is called regularization.