23Trees, Ensembles & Boosting

Every model in this course so far has answered with a formula in x. You hand it a row, it multiplies and adds, and a number comes out. A tree does something else entirely. It cuts the input space into boxes, finds the box your row landed in, and reports the average of the training labels inside it. That sounds almost too simple to be interesting. It is the first model here that discovers where the structure sits instead of being told, and that is why it can hold a regime, a threshold and an interaction that nobody named in advance. Grown all the way down it is nearly unbiased and violently unstable. So it sits at the far right end of Chapter 22's U-curve, which means there are exactly two ways to make it useful. Average many deliberately overfit trees and the variance falls. Or start deliberately underfit and keep adding small trees fitted to what is still left over. Those are Chapter 22's two terms, attacked from opposite sides. And the second one hides the sentence this whole chapter turns on, because the leftover you keep fitting is not a heuristic at all. It is a gradient.

Look at what this page stands on, because nearly all of it is already yours. Chapter 22 gave us the decomposition Err(x₀) = σ² + bias² + variance, and the spray picture that makes it concrete. It gave us the purchase: spend a little bias to buy back a lot of variance. It closed by naming two doors and labelling them both: bagging attacks the variance term, and boosting attacks the bias term. Chapter 21 gave us the U-curve, the hyper-parameter chosen on validation folds, and walk-forward cross-validation with a purge. Chapter 20 gave us RSS, the residual and the bootstrap, and Chapter 12 gave us the Bernoulli variance p(1−p). Chapter 11 gave us the expand-a-square move, Chapter 18 gave us covariance, and Chapter 8 gave us gradient descent. Chapter 2 gave us the limit that turns (1 − 1/n)ⁿ into 1/e, and every one of those gets spent somewhere on this page.

So the seam is short, and it has a shape worth noticing. Both of Chapter 22's doors are things you do to a model you already have. That makes the first honest question a different one: which model? And the honest first observation is that everything we have built so far is a weighted sum of the features. Before we can attack either term, we have to ask what lies outside that class.

01A shrunken line is still a line

Chapter 22 handed us a dial and a theory of what the dial buys. It did not hand us a bigger world to dial inside. Ridge and lasso shrink the coefficients of a weighted sum of the features. A shrunken weighted sum is still a weighted sum. Turn λ as far as you like and the fitted surface stays a plane, just a flatter one.

Now ask what a market actually does, starting from a sentence you could hear on any trading desk. When VIX is below 20, momentum continues. Above 30, it reverses. That is not a plane, and no amount of λ will bend one into it, because it is a regime, which means the rule itself changes past a threshold.

Try to write it linearly and you get stuck twice, in two different places. You would need an interaction column, something like momentum × 1{VIX > 30}. So first you must already know that momentum and VIX interact, and then you must already know the threshold sits at 30. Both of those have to be true before you fit anything at all.

Basis expansion is the standard answer to the first problem, and it genuinely works. Add squared terms, add every pairwise product, and let ridge sort out which ones matter. Count what that costs before you agree to it. With 50 features there are 1,225 pairwise products and 19,600 triples. You also have to decide in advance how far up to go, and Chapter 21 already priced that decision for us. A bigger candidate set is a bigger search, and the optimism you are charged is set by the size of the search.

But the cost is the easier half of the objection, and almost everyone stops there. Here is the sharper failure: a polynomial can bend, but it cannot put a corner at a location the data has to tell you about. Crank the degree as high as you like and you get a smooth curve that wiggles near 30 without ever committing to a break there. The regime is a discontinuity in the rule, and a smooth basis is structurally the wrong object for it.

So the demand this chapter has to meet is specific, and it is worth stating before we build anything. Not "a non-linear model." We need a model that discovers which variables interact, and where the thresholds sit, from the data, without being handed a list.

02Condition on a box, not on a point

Here is the way I have come to think about what a tree really is, and it is not the flowchart. Chapter 22 told us the best prediction that could possibly exist is f(x₀) = E[y|x₀], the conditional mean. Chapter 10 told us what conditioning does. It shrinks the world down to the conditioning event and re-measures inside it.

So go and actually compute E[y | VIX = 24.37]. You cannot, because no other row in your dataset has VIX equal to 24.37. You have a sample of one, or of none, and an average over one row is not an estimate of anything.

That leaves exactly one move available. Widen the condition until rows appear. Condition on a box around that x, say VIX between 20 and 30 with momentum above zero, and average the y values of the rows that landed inside. The width of that box is not a detail. Make it too narrow and you are averaging two rows, so the answer jumps around wildly. Make it too wide and you have swallowed both regimes, so the answer is a compromise that is wrong everywhere. That is Chapter 22's purchase again, wearing a new costume.

That is a tree, and everything else in this chapter is bookkeeping about where to put the boxes.

The bookkeeping has a name: recursive binary splitting. Pick one feature and one threshold, cut the space in two, then repeat the same move inside each half. Because every cut uses a single feature, the boxes it makes are axis-aligned rectangles. And the answer inside a box is the mean, which we can derive in one line rather than assume. The constant c minimising Σ(y − c)² has derivative −2Σ(y − c), and setting that to zero gives c = ȳ. Chapter 20's own loss picks the mean for us.

Now draw the same fitted tree twice and watch them turn out to be one object. On one side sits the flowchart everyone shows, and on the other sits the surface it implies, which is a staircase: flat on each box, jumping at each cut. The flowchart is the staircase written as instructions. That staircase is the hypothesis class, and it is worth saying exactly what that means. A tree can be any piecewise-constant function on axis-aligned boxes, and it can be nothing else.

Which tells you the honest cost up front. A boundary that runs diagonally through feature space has to be approximated by a staircase of many cuts, and every cut spends data. Trees are cheap at thresholds and expensive at slopes. That is the exact opposite of the linear model, which is precisely why they are worth having.

03Where to cut, and the dullest algorithm here

So where do you cut? The answer is the least sophisticated thing in this chapter, and it is worth saying plainly, because most readers assume something clever is going on. Nothing clever is going on.

Sort each feature, then walk down every possible threshold between consecutive distinct values. For each candidate cut, split the rows into left and right, take each side's mean, and add up the squared deviations inside each side. That total is RSS, Chapter 20's own quantity, computed on this node's rows. The score of the cut is the parent's RSS minus the two children's RSS combined, which in plain words is how much sum-of-squares this cut removed.

Take the winner, then do the identical thing again inside the left child and inside the right child, and keep going until a stopping rule fires. The usual rules are a minimum number of rows in a leaf, a maximum depth, and a minimum gain worth having. That is the entire training algorithm. Once the columns are sorted it costs O(n·p) per node, which is why a tree on a million rows is not a heroic computation.

Two honest limits belong right here, not in a footnote at the end.

The first is that the scan is greedy. The split at the root is chosen to look best right now, with no idea at all what it makes possible two levels down. So the tree you get is not the best tree, and finding the best tree is NP-hard, which is why nobody looks for it.

That limit matters more than it sounds, and it pays off later in the chapter. A reader who believes the tree is optimal cannot understand why deliberately handicapping the split choice could ever help. You do not improve an optimal procedure by tying its hands. You can certainly improve a greedy one, and that is exactly what a random forest will do.

The second limit is about the stopping rules, and it is a relabelling rather than a caveat. Those rules are not safety features. Minimum leaf size, maximum depth and minimum gain are Chapter 21's complexity knob wearing work clothes, and we will treat them that way shortly.

04What plays the role of variance for a class

Everything so far assumed y is a number. Make it a class instead, up day or down day, default or no default, and the split score has to change. "The mean of the rows in this box" and "their sum of squared deviations" both need a number to work on.

So ask the question that generates the answer rather than reaching for a formula. What plays the role of variance when the label is a class? A node is pure when every row inside shares a label, and messy when the labels are evenly mixed. We want a quantity that is zero at pure and largest at even odds.

Before we write one down, deal with the instinct you almost certainly have, because it is a good instinct and it is wrong. Why not just split on accuracy, so that you get the most rows right? Here is the smallest example that kills that idea. A node holds 800 rows, 400 of each class. Split A gives children of (300, 100) and (100, 300), while split B gives (200, 400) and (200, 0).

Count the rows each split classifies correctly. Split A gets 300 right on the left and 300 on the right, so 600. Split B gets 400 right on the left and 200 on the right, so 600 again. Accuracy is completely indifferent between them. Now look at split B's right child, which is pure: two hundred rows, no ambiguity, permanently finished, and nothing below it will ever need another cut. Accuracy could not see that, because it only counts majorities and neither split flipped a majority anywhere.

The reason lives in the geometry. Plot the misclassification rate against p and you get two straight lines meeting in a corner. Plot a concave impurity and you get a smooth arch. A straight-line criterion is flat exactly where a curved one is still falling, so it cannot reward a split that purifies without yet flipping a decision.

Now write the curved one down, and watch it turn out to be something we already own. Let p be the fraction of class 1 in a node, and draw a row at random from it. That draw is a Bernoulli(p), and Chapter 12 told us its variance is p(1−p). Gini impurity, defined as 1 − Σ p_k², is algebraically Σ p_k(1 − p_k), and for two classes that is exactly 2p(1−p).

So Gini is not a new invention with an intimidating name. It is the variance of the label, computed the only way a categorical label allows. The classification criterion and the regression criterion are the same object, which is why one algorithm handles both.

You will also meet entropy, written −Σ p_k log p_k, and I want to be straight about where it comes from. It is Shannon's, and Shannon's theory is not something this course has built. So we are not going to pretend the word information has been earned here. We use entropy for the three properties we actually need: zero when pure, maximal at even odds, strictly concave. In practice it and Gini pick the same split almost every time. And "information gain" is just parent impurity minus the weighted average of the children's impurity, which is the identical shape as the RSS reduction we already built.

05The variance is structural, and pruning will not fix it

Put the tree on Chapter 22's axis and read it. Depth is the complexity dial, and at depth 1 you have two boxes and one threshold. By the time every leaf holds a single row, the training error is exactly zero and the staircase passes through every point.

So a fully grown tree sits at the far right end of the U, with variance enormous and bias near zero, because with enough cuts it can approximate essentially any function. But "enormous variance" is a slogan, and slogans do not transfer. Look at what kind of variance this is, because it is a different animal from anything in Chapter 22.

In a linear model, nudge one data point and every coefficient moves a little, so the fitted plane tilts slightly. The change is small and it is smooth. In a tree, nudge one data point across a threshold near the root and the root split can flip to a different feature entirely. Every box below it, in both subtrees, is then rebuilt from scratch. The prediction changes at points nowhere near the point you moved.

The mechanism has a name worth holding onto. A split location is chosen by an argmax over candidates, and an argmax is a step function of the data. So an arbitrarily small change in the data can produce a discontinuous change in the model. Nothing in Chapter 20 or Chapter 22 had that property. That is what makes tree variance structural rather than merely large.

Chapter 22 taught us to picture the spray of refits. Run that experiment again with trees and the picture is not a fuzzy band around a curve. It is a collection of staircases that disagree about where the steps even are.

Pruning is the honest response, and it is a shape we already own. Grow the tree fully, then charge α for every leaf and prune back the subtree that pays for itself least. That is cost-complexity pruning, and it is Chapter 22's penalty with the number of leaves in place of ‖β‖² and α in place of λ. It is tuned exactly the same way, on validation folds.

And here is the verdict, said flatly, because pretending otherwise wastes your time. Pruning walks you along the U, and it does not lower it. A well-pruned single tree is a mediocre model. Too rigid when it is small, too unstable when it is large, and beaten by a ridge regression on most smooth problems.

So stop trying to fix the tree, and use many of them instead.

06Average the draws

If the problem is that the answer is a wild draw, the oldest instrument in this course applies: average the draws. Chapter 13's law of large numbers and Chapter 16's SE = σ/√n both say the same thing in different words. Averaging independent copies shrinks the spread.

The obstacle is that you do not have many datasets. You have one, and Chapter 20 already solved that exact obstacle. The bootstrap manufactures pseudo-datasets by resampling your rows with replacement.

So draw B bootstrap samples, grow a tree on each, and average the B predictions. That is bagging, short for bootstrap aggregating, and it is about four lines of code.

Now the question that decides how you should build the ingredient, and it is settled by one line of algebra rather than by taste. What is the bias of the average? The B trees are identically distributed, because they came from the same recipe and the same kind of draw. So each of them has the same expectation, and expectation is linear, so the average carries that same expectation too. The bag's bias is exactly one tree's bias, and nothing was paid for it.

Read that again with Chapter 22's dial in mind, because it inverts the instinct you have been building for two chapters. Averaging costs no bias and buys variance. So you want the ingredient to have as little bias as possible, and you genuinely do not care how much variance it has. That means you grow the trees deep and you do not prune them, because inside a bag, pruning is an actual mistake.

The picture is worth watching rather than describing. One tree is a jagged staircase. Average twenty trees whose steps sit in different places and the jumps smooth out into something that looks like a curve, because the discontinuities are somewhere else in every member.

And watch the variance band as B rises. It collapses fast at first, then less, then it flattens, and it does not flatten at zero. That gap between the measured curve and the 1/B line you expected is exactly what the correlation floor explains.

07The floor more trees cannot reach

The 1/B line assumed the trees were independent. They are not, and it is worth being concrete about why before doing any algebra.

Two bootstrap samples drawn from the same n rows overlap heavily. Each one contains about 63% of the distinct original rows, so any two of them share most of their data. And there is a second reason that does more damage. If one feature is genuinely the strongest predictor, then nearly every tree, on nearly every resample, picks it at the root. The trees are not twenty different opinions. They are twenty variations on the same opinion.

So redo the algebra honestly. Let each tree's prediction at a fixed x₀ have variance σ², and let any two of them have correlation ρ. Then expand the square, exactly the way Chapter 11 taught us to.

The variance of the average is (1/B²)[B σ² + B(B−1) ρσ²], and tidying that up gives ρσ² + (1 − ρ)σ²/B. Now read it slowly, because this single line is the most useful thing in the first half of the chapter.

The second term is the one everybody expects. It falls toward zero as B grows, and it is why more trees never hurt. The first term does not contain B at all, which makes it a floor.

Put numbers on it, because the abstraction hides how brutal this is. At ρ = 0.6, fifty trees give you 0.608σ² and five thousand trees give you 0.600σ². A hundredfold increase in compute bought you about 1.3% of one tree's variance. Sixty per cent of the original variance is still sitting there after infinitely many trees.

So the question "how many trees?" has a boring answer. Enough, then stop, because more trees are never worse and never enough. The interesting question is the one the formula hands you, and it is the only lever left. How do you lower ρ? Everything a random forest does that looks strange is an answer to that one question.

08Hide the best feature on purpose

Decorrelate them, and there is only one honest way to do it. If every tree keeps choosing the same feature at the root, stop letting them all see it.

At every split, not once per tree but at every single node, draw a random subset of m features out of p and choose the best cut among only those. Trees forced to build around the second-best or third-best predictor look genuinely different from trees that used the strongest one. So ρ falls. Bagging plus per-split feature subsampling is the random forest, and that is the entire difference.

Notice that this is per split and not per tree. The per-tree version is a real method and it is a weaker one, and people routinely mix the two up. The per-split version is also why the forest still finds the strong feature at all. With p = 100 and m = 10, that feature appears in about a tenth of the candidate sets, so it wins somewhere inside almost every tree. Just not everywhere.

Now read the tradeoff straight off the floor formula, because it is already there and nothing new is needed. Shrinking m lowers ρ, which lowers the floor. Good. It also makes each individual tree worse, because the tree is sometimes forbidden from using the best available cut, which raises σ² and adds a little bias. Bad. The total is a falling factor multiplied by a rising one, so m has its own U-curve.

That makes m a hyper-parameter in Chapter 21's exact sense, chosen on validation folds. It also makes the famous defaults, √p for classification and p/3 for regression, starting points rather than truths. On a market dataset with a hundred weakly-informative features and one strong one, the right m is an empirical question.

One more thing falls out for free, and it is a genuinely elegant consequence of the bootstrap. Every bootstrap sample leaves some rows out. Which fraction? Row i is missed on a single draw with probability 1 − 1/n, and there are n draws, so it is left out entirely with probability (1 − 1/n)ⁿ. Chapter 2 told us that limit, and it is 1/e, or 36.8%.

So about a third of the rows are out-of-bag for each tree, and every row can be predicted using only the trees that never saw it. That is a held-out estimate with no folds and no extra fitting, which is a lovely thing to get for nothing.

Then the caveat that saves real money: out-of-bag scoring draws rows uniformly at random. On a time series that means the trees scoring today were trained on tomorrow. It is Chapter 21's lookahead leak wearing a friendly name and printed by default. Out-of-bag error is a fine model-selection tool on cross-sectional data, and it is not a substitute for a purged walk-forward split on anything with a date column.

09The other direction: start rigid and step

The forest attacked one of Chapter 22's two terms. Now go at the other one, and notice that every design choice has to invert.

If you want to reduce bias, averaging is useless, and we proved that a moment ago. The average of B identically distributed fits has exactly the bias of one of them, so no amount of averaging can move it. Something has to change between one fit and the next.

So try the crudest possible version of changing. Start with the dumbest model in existence: predict the overall mean of y for every row. That has enormous bias and essentially no variance, which is the far left end of the U, the mirror image of where the forest started.

Now compute what is left over, r = y − F, for every row. Fit a small tree, a stump or depth 3, to those leftovers. Notice what that tree is doing, because it is not trying to predict y. It is trying to predict what the current model is getting wrong, which is a different and much smaller job. Add a fraction of it to the model, F ← F + ν·h, with ν something like 0.05. Recompute the leftovers, fit another small tree, and repeat the whole move M times.

That is boosting, and three features of it are worth pinning down straight away, because all three are the opposite of the forest.

The trees are grown sequentially, because each one is fitted to a target defined by all the previous ones, so this cannot be parallelised the way bagging trivially can. The pieces are summed with weight ν, not averaged with weight 1/B. The model is F₀ + ν·h₁ + ν·h₂ + … and it grows rather than staying in place.

And the ingredient is deliberately weak, which is not a stylistic preference. The residual is the fuel. If your first tree is deep and fits everything, the leftovers are nearly zero, and every subsequent tree has nothing to work with except noise. Try it and the sequence stalls in a single round. Boosting has to be handed a job that is not finished.

10★★ The leftover is the gradient

Everything in the last section was correct and none of it was explained. Why the residual and not something else? Why does adding a fraction of a correction converge instead of oscillating? What do you do when your loss is not squared error, when you want to be robust to outliers, or when the label is a class and you are scoring a probability?

Every one of those questions has the same answer, and it is one line of Chapter 3 calculus.

Forget trees for a moment, and forget models entirely. Suppose I let you set the N training predictions to anything you liked. A lookup table, F = (F(x₁), …, F(x_N)), holding N free numbers with no structure at all. And I score you with L(F) = ½ Σ (yᵢ − Fᵢ)².

That is an ordinary function of N variables, and it is exactly the object Chapter 8 built gradient descent for. So take the gradient. ∂L/∂Fᵢ = −(yᵢ − Fᵢ), which means the negative gradient in coordinate i is (yᵢ − Fᵢ). The residual. Not something analogous to the residual, and not a relative of it, but the same number we have been computing since Chapter 20.

So "fit what's left over" was never a heuristic. It was steepest descent, and almost nobody writes the line that says so. Textbooks put "fit the residuals" in the algorithm box, then put "boosting can be viewed as gradient descent in function space" in a remark a page later, and never write the derivative that makes those the same sentence.

Now the second half, which is where the phrase in function space stops being decoration. Gradient descent has just handed you a step at the N training rows and nowhere else. You would be updating a lookup table, and a lookup table is silent at a new x. That is the only kind of row you will ever be paid for.

So we fit a small tree to those N step values. Not to y, but to the step. The tree is the interpolator. It takes a direction that exists only on your data and extends it to the whole input space, snapping the ideal step onto the closest thing our model class can actually express. That constraint is the only difference from ordinary gradient descent, and it is why each step is a function rather than a vector.

Everything else transfers verbatim: ν is Chapter 8's learning rate η, and M is how many steps you take.

And now the payoff, because the loss has become a dial you can turn. Squared loss gives a negative gradient of (y − F). Absolute loss gives sign(y − F), so the trees chase direction rather than magnitude and a single outlier stops dominating. That is a robust booster, for free, with no new algorithm. A loss that scores probabilities gives its own negative gradient, and that is the door into Chapter 24. This is what the word gradient is doing in "gradient boosting". The algorithm is fixed, and the loss chooses the target.

11B is safe, M is a dial

Now the asymmetry, and it is the thing to walk out of this chapter holding, because it is where real money gets lost.

In a forest, B is not a tuning parameter. More trees shrink the second term of ρσ² + (1 − ρ)σ²/B and touch nothing else. You stop when the curve flattens, and running longer is a waste of electricity rather than a mistake.

In a booster, M is a complexity dial in Chapter 21's exact sense, because every round adds another ν-sized step of capacity. So the model becomes monotonically less biased and monotonically more able to chase this particular sample's noise, and the test curve is the U all over again.

The reason nothing inside the algorithm will warn you is worth stating as a mechanism, not as a warning. Boosting descends on the training loss, which falls at every round by construction. So the algorithm's own scoreboard says "better" forever, and there is no internal signal anywhere inside it to stop.

Which makes M a hyper-parameter, chosen exactly as Chapter 21 prescribed. On a validation fold, walk-forward, purged to your label horizon, and never on the test set. That is early stopping, and it is not a library convenience. It is the only honest way to choose a distance to travel when the thing you are descending is not the thing you care about.

Two consequences follow, and the first is that ν and M are one knob wearing two labels. What has been spent is roughly ν × M, so halving ν means roughly doubling M to arrive at the same place. Small ν is better, and we can now name why precisely. Many tiny steps take a smoother path through function space and give any single tree less opportunity to latch onto a quirk. That is Chapter 22's shrinkage, moved off the coefficients and onto the steps.

Second, since we know how to buy variance reduction cheaply, we can smuggle the forest's trick into the booster. Fit each round on a random subsample of rows and a random subset of columns. That is stochastic gradient boosting, and it is why every serious implementation exposes subsample and colsample parameters.

Finally, the folklore, killed cleanly. You will read that boosting resists overfitting. It does not. That belief comes from AdaBoost on clean, high-signal classification data, where test error kept falling long after training error hit zero. Chapter 20 told us a daily-return R² near 0.01 is normal. In a dataset that thin, a booster will find spurious structure quickly and confidently, and it will do it while its training curve looks magnificent.

12XGBoost, taken apart

Every serious tabular result you will read about is XGBoost or LightGBM. So let us take the brands apart and see how much of the difference is a real idea. It is not much, and the real idea is one you already own.

Go back to the step, where we are minimising L(F) and we want the best move. Boosting as we built it uses the first derivative only, which is plain gradient descent. But Chapter 4 says a smooth function near a point is its value, plus slope times step, plus half its curvature times step squared. So expand the loss around the current prediction to second order and keep both terms. For each row, gᵢ = ∂L/∂Fᵢ and hᵢ = ∂²L/∂Fᵢ². If you are doing descent anyway, do Newton.

Then add the regularizer explicitly into the objective instead of leaving it to the stopping rules. Charge γ per leaf, and add ½λ times the sum of squared leaf values. That second piece is Chapter 22's ridge penalty applied not to coefficients but to the numbers a leaf is allowed to output.

Inside one leaf j, the objective is Σ(gᵢw + ½hᵢw²) + ½λw², which is a plain quadratic in the single number w. Chapter 8's condition ∇f = 0 gives it in one line: w* = −G/(H + λ), with G = Σg and H = Σh. The most famous expression in applied machine learning, and no new mathematics entered the room.

Look at what it is. The numerator is the accumulated gradient in that leaf, and the denominator is its curvature plus λ. So it is a Newton step with a ridge in the bottom, which means a leaf holding few rows or low-curvature rows is automatically damped. That is precisely the surgical shrinkage Chapter 22 described, shrinking hardest where the data spoke least.

Substitute w* back and the leaf contributes −½G²/(H + λ). So the gain of a proposed split is the two children's scores minus the parent's, minus γ. That last term is what makes this different in kind from the split rule we built earlier. The complexity charge is inside the criterion, so a split that does not pay for itself is never made, and pruning stops being a separate pass afterwards.

The famous dials stop being superstition at that point. λ shrinks leaf values, and γ charges per leaf. And min_child_weight is a threshold on H, the summed curvature in a leaf. For squared loss hᵢ = 1, so H is just the row count. Under a probability loss hᵢ = p(1−p), so the same parameter means something completely different, and almost nobody says so.

LightGBM's differences are engineering rather than statistics, and they are worth knowing exactly. It bins each continuous feature into a few hundred buckets, so the split scan costs O(bins) per node instead of O(n). And it grows leaf-wise, always splitting the single most promising leaf anywhere in the tree, rather than level by level. That gets more gain per leaf and it overfits harder on small data. So the honest summary is one statistical upgrade, one explicit regularizer, and a great deal of very good engineering.

13The cliff, the gift, and the lie in the importance chart

Now the practical truth, and it starts with the single fact about trees that costs quants the most money. A tree cannot extrapolate.

Its prediction is the mean of a box, and outside the range it was trained on there are no new boxes. The boundary leaf simply extends forever, flat. Feed a tree trained on prices between 100 and 140 a price of 200 and it returns whatever the 130-to-140 box said. A linear model would at least keep going in the right direction, often wrongly, but it moves.

This is not a curiosity to file away. It is why you model returns, spreads, ranks and z-scores rather than raw levels. That is the same instinct Chapter 5 gave you when it made log-returns additive across time. A tree fitted on a trending series will confidently flatline the moment the series leaves its historical range, and it will do it without any warning in the output.

The mirror-image fact is a gift, and it is the exact opposite of Chapter 22's hardest-won hygiene rule. A tree only ever asks whether a feature is above or below a threshold, so it uses only the order of a feature and nothing else. Take logs, standardize, winsorize, cube it, and any strictly increasing transform hands you back the identical tree, bit for bit. In Chapter 22 the moment we added a penalty, the units of the data became a modelling decision. Under a tree, units are irrelevant.

Two more properties fall straight out of the hypothesis class. Depth is interaction order, because a leaf at depth d is defined by d conditions ANDed together. So a booster of stumps is a purely additive model that can hold no interactions at all, and depth 3 buys you three-way interactions and no more. That is a far more meaningful way to choose depth than a grid search. And missing values need no imputation, because a split can learn a default direction for rows whose value is absent.

Then the honest warning about the output everyone trusts most. Feature importance by split count is biased toward high-cardinality and continuous features, simply because they offer more thresholds to try and therefore more chances to win a scan. And when two features are near-identical, the credit between them is split arbitrarily by whichever one happened to win at each node.

You have now met that disease three times. Chapter 20 called it a pinned sum with a free split, and Chapter 22 showed it as the lasso's unstable selected set. Here it is the importance ranking reshuffling under a bootstrap. So bootstrap the whole fit and watch the ranking move before you write any of it into a research note. Permutation importance on held-out data is the honest replacement.

And finally, when do trees lose? When the truth is smooth and near-linear, because a staircase spends many splits to approximate what one coefficient says exactly. When you must extrapolate, or when the structure is genuinely a low-dimensional linear factor model. And in the thinnest-signal regimes, where a heavily shrunk linear model with an economic argument behind it will often beat a booster that has enough freedom to find beautiful things that are not there.

14Two directions, one minimum

Stand back and look at what this chapter actually was, because it was not three algorithms.

Chapter 22 left us a decomposition with exactly two movable terms and a door. A fully grown tree is the most extreme object we can put on that axis, with bias near nothing and variance enormous, which puts it at the far right end. Everything since has been an approach to the minimum from one side or the other.

Bagging and the random forest approach from the right. Keep the near-zero bias, average the variance away, and when averaging hits the floor ρσ², decorrelate to lower the floor itself. Boosting approaches from the left. Start deliberately rigid where variance is nothing and bias is everything, take small steps downhill against the bias, and stop before you have travelled far enough to start collecting variance.

Two directions, one minimum, one decomposition. And both are Chapter 22's shape in a new currency, which is to buy the variance back with structure. The structure here is not a fence around the coefficients, but "many fits averaged" in one case and "small steps, stopped early" in the other.

Two things travel with you out of this chapter, and the first is the ledger from Chapter 22, entirely unchanged. An ensemble prices overfitting. It does not repair a leak, and it does not undo a selection. A gradient booster fitted on a pipeline that standardized before splitting is still leaking. A strategy chosen as the best of two hundred backtests is still cursed, no matter how carefully it was early-stopped. These tools make the bias-variance purchase cheaper, and they buy nothing else. That warning matters more here than it did last chapter, because these methods are strong enough to be seductive.

The second is a door, because everything on this page predicted a number. But the negative gradient is defined by the loss, and we watched the target change when we swapped the loss. So hand the machine a loss that scores a probability instead of a squared error, and the identical loop trains a classifier. That is exactly where Chapter 24 starts. And it starts with the honest follow-up question that comes with it: once a model outputs a class, how do you score it without lying to yourself, especially when one class is rare?