24Classification, Imbalance & PCA

Chapter 23 closed on a door it did not walk through. The boosting loop never cared what the loss was. It took a derivative, fitted a small tree to the negative gradient, and stepped. So hand it a loss that scores a probability and the identical machine trains a classifier. That is the door, and this chapter walks through it. But it opens onto a question the last chapter could not answer, because once a model says fraud or not fraud you have to score it without lying to yourself. Everything hard on this page is one quantity being squeezed through something. Squeeze a probability through a linear model and it will not fit, so you move it to a coordinate where it does. Squeeze it through a threshold and you have stopped modelling and started deciding. Squeeze it through two hundred correlated features and the weights stop meaning anything at all. Three squeezes, one instinct underneath them: stop working in the coordinates you were handed.

Look at what this page stands on, because almost all of it is already yours. Chapter 20 gave us least squares and the fact that a fitted value estimates the conditional mean E[y|x]. It also gave us multicollinearity, where near-duplicate columns make the coefficients wild while the predictions sit still. Chapter 12 gave us the Bernoulli variable and the small sentence this chapter turns on, E[y] = p. Chapter 5 gave us the logarithm, including the move we lean on constantly, which is that ln turns a product into a sum. Chapter 10 gave us conditional probability and Bayes. Chapter 11 gave us expectation, Chapter 17 gave us Type I and Type II errors, and Chapter 8 gave us gradient descent and Lagrange multipliers. Chapter 7 gave us eigenvectors, and Chapter 18 gave us the covariance matrix. Every one of those is spent somewhere below.

So the seam from last chapter is short. Boosting's target is whatever the loss says it is, so a loss built for probabilities turns that loop into a classifier without changing a line of it. But that raises the question the previous chapter left standing. We now need a loss that scores a probability, and a way to judge the result when one class almost never happens.

01A yes/no column, and the tool you already own

Start with the smallest honest thing we can do, which is to use what we have. Here are twelve borrowers. Each one has a single feature, debt-to-income, and a column y that is 1 if they defaulted and 0 if they did not. Nothing about that column is special, because it is just twelve numbers, and Chapter 20 fits a line to numbers.

So fit it, and least squares does not complain — it hands back ŷ = −0.3382 + 2.3949·DTI. The line slopes upward, which is what you would expect, because higher debt-to-income does go with more default in this book. Then read a prediction off the right-hand end.

At a debt-to-income of 0.66 the model says 1.2424, and at 0.10 it says −0.0987. Say those out loud as answers to the question we asked. A 124% chance of default is nothing at all, and a negative chance of default is worse than nothing.

Now the diagnostic question, and it is the one that turns a broken toy into the foundation of the chapter. Is the idea wrong, or is the range wrong? Put two sentences you already own next to each other. Chapter 20 said a fitted value estimates the conditional mean E[y|x], and Chapter 12 said that for a Bernoulli variable, E[y] = p.

A 0/1 column is a Bernoulli variable, so its conditional mean is a conditional probability. That means least squares on a yes/no target was estimating P(y=1|x) the whole time, without anybody saying so. It was never solving the wrong problem. It was solving the right problem with an output range that cannot hold the answer.

This is worth pausing on, because it settles what a classifier is actually for. Ask what "will this loan default" means for one loan. The loan defaults once, or it does not, and there is no repeated experiment hiding behind it. So the only answer with any content is how likely. The honest deliverable of a classifier is a probability, not a label, and everything later in this chapter depends on that sentence being installed now.

There is a second crack in the same picture, and it is quieter. Take one defaulting borrower and drag them far to the right, out to a debt-to-income of 1.60. The whole line tilts. The point where it crosses 0.5 slides from 0.3500 to 0.4567, so the boundary moved because of a single row. Squared error takes distant points extremely seriously, since it squares the distance.

That gives us a specification, and it is worth writing down because the next three sections are just filling it in. We want the same linear combination of features, and we want an output trapped inside (0,1). And we want a loss that cares how confident you were rather than how far away you sat. That list is not something I handed you — it came out of watching the tool you already had break.

02A coordinate a probability can live in

Here is the move, and it is the reverse of the obvious one. Do not go hunting for a function that squashes the real line into (0,1). Instead, take the probability and push it outward until it covers the whole line. Building a map forwards from a bounded quantity is elementary algebra, and we can run any map backwards later.

Half-step one, and you have been reading this number your whole life. Instead of the probability, take the odds, p/(1−p). A 75% chance gives 0.75/0.25 = 3, which is what a bookmaker calls three to one on. A 50% chance gives exactly 1, even money, and a 10% chance gives 1/9, about 0.1111.

One warning before it costs us later. In ordinary speech people say "the odds are 30%", and they mean the probability. In this chapter odds means the ratio and nothing else. Probability 0.75 and odds 3 are the same belief written as two different numbers, and they are never interchangeable here.

Now watch the range. As p runs from 0 to 1, the odds run from 0 to +∞. The ceiling is gone, which is real progress. But the floor is still there, since odds can never drop below zero, and a linear model living in odds-space could still predict something impossible.

So we need a function that takes (0, ∞) and spreads it across all of ℝ, and Chapter 5 handed us exactly one. That is the whole stated job of the natural logarithm. Take z = ln(p/(1−p)) and the floor disappears too.

That number is the log-odds, also called the logit, and it is worth reading rather than just computing. At p = 0.5 the odds are 1 and ln 1 = 0, so even money sits exactly at the origin. Positive z means more likely than not, negative means less likely than not, so the sign carries which side you are betting on. And the map is antisymmetric: p = 0.9 gives ln 9 = 2.1972 while p = 0.1 gives −2.1972 exactly. Default and no-default are mirror images, and neither one is privileged.

The strategy now has a name worth having. We are not going to model p directly, but a transform of p instead, and that transform is called the link function. The linear model is not being abandoned — it is being moved into a room where it fits.

03Run the map backwards

Put the linear model in the new room, and declare that the log-odds is linear in the features: ln(p/(1−p)) = w·x + b.

That is the entire model statement, and notice what we did not do. We did not invent a squashing function. We asserted linearity in the one coordinate where linearity is harmless, which is the whole reason we built that coordinate. Everything left is bookkeeping, because we want a probability back and the map runs both ways.

Before the algebra, try it yourself. Invent a function that takes any real number and returns something in (0,1). Most people sketch something S-shaped. Some propose a scaled arctangent, or a straight line clipped at both ends, and every one of those genuinely works. That is exactly the problem with starting from this end. There are infinitely many squashers and no reason on earth to prefer one.

So we do not choose. Write z = w·x + b, then solve the equation we already built. Exponentiate both sides of z = ln(p/(1−p)) to get e^z = p/(1−p). Multiply out to get p = e^z − p·e^z, collect to get p(1 + e^z) = e^z, and divide. That gives p = e^z/(1 + e^z), and dividing top and bottom by e^z gives the form everyone writes, p = σ(z) = 1/(1 + e^(−z)).

That is the logistic function, the sigmoid. And here is the sentence that should change how this model feels: we did not choose it. Its S-shape, its flat tails, its exact steepness at the origin, all of it is a consequence of demanding an unbounded symmetric coordinate and then coming home. The sigmoid is the log-odds map read backwards. Nothing about it is a matter of taste.

Two readings arrive free, and both matter more than they look. The first is about the boundary. Wherever σ(z) = 0.5 we have z = 0, and z = 0 means w·x + b = 0. That is a straight line in x, a flat plane in higher dimensions. So logistic regression is a linear classifier, despite the curve on your screen. The curvature lives entirely in how confidence grows as you walk away from the boundary, not in the shape of the boundary itself.

The second reading is about the coefficients, and this is where nearly everyone gets hurt. Because the model is linear in log-odds, adding one unit of x_j adds w_j to the log-odds, which multiplies the odds by e^(w_j). That number is the odds ratio, and it is what credit-risk teams actually report.

Watch what that does to the naive reading. Take w = 0.4 and apply the identical one-unit bump to two borrowers. Start one at p = 0.50 and it moves to 0.5987, a jump of about 0.099. Start the other at p = 0.99 and it moves to 0.9933, a jump of 0.0033. Same coefficient, same change in x, and the probability moves thirty times further in one place than the other. Meanwhile the odds were multiplied by e^0.4 = 1.4918 in both cases, identically.

So "a one-unit increase raises the default probability by w_j" is not merely imprecise. It is dimensionally impossible for any borrower already above p = 0.6. The coefficient is linear in log-odds and emphatically non-linear in probability, which is why e^(w_j) is the honest thing to quote.

04What probability did you assign to what happened?

We have a model and no way to choose w and b. The obvious answer, reuse squared error, is wrong in a way that tells us the right answer. So take the smallest example that forces it.

Two candidate models look at the same loan, and that loan defaulted. Model A said p̂ = 0.6. Model B said p̂ = 0.99. Any rule that only asks "did you get the label right" scores them identically, which already tells us accuracy is useless as a training signal. Squared error can separate them, but it has no story about why, and it treats a confidently wrong answer as merely somewhat bad.

So stop picking a loss and ask a question about the data instead. What probability did my model assign to what actually happened? For one row that is easy, because the outcome is Bernoulli. The model said p̂ for a 1 and (1 − p̂) for a 0, and both cases fit in one expression, p̂^y · (1 − p̂)^(1−y).

That looks exotic and it is nothing but a switch. Set y = 1 and the second factor has exponent 0, so it collapses to 1, leaving p̂. Set y = 0 and the first factor collapses to 1, leaving (1 − p̂). Anything to the power zero is one, and that is the entire trick.

Across n independent rows, probabilities of independent things multiply, so the likelihood of the whole dataset is the product. Three rows that all defaulted, predicted at 0.9, 0.8 and 0.7, give 0.9 × 0.8 × 0.7 = 0.504. Now your Chapter 5 reflex should fire, because a product of hundreds of numbers below one is unusable. In IEEE double precision, 0.5^1000 is 9.33e−302 and still a number, while 0.5^1100 is exactly 0.0.

Take the logarithm and the product becomes a sum, which is the same move Chapter 5 used on log returns. Because ln is strictly increasing, it does not move where the maximum sits. Negate it so we are minimising, divide by n so the number does not depend on dataset size, and read off what fell out: L = −(1/n) Σ [ y·ln p̂ + (1−y)·ln(1−p̂) ].

That is log-loss, also called binary cross-entropy. We derived it, and nobody decreed it. On those same three rows it gives 0.2284, and e^(−0.2284) = 0.504, which is the likelihood we started from.

Now look at what this loss does, because the rest of the chapter leans on it. If the truth is y = 1 and you said p̂ = 0.01, your contribution is −ln(0.01) = 4.6052. If you said p̂ = 0, it is −ln(0) = ∞. Confident wrongness costs infinity, without limit and without appeal.

That is the mechanism, and it is the thing almost nobody says out loud. The sigmoid gives you a number inside (0,1), which is not the same as a probability. The loss is what makes it one. The only way to minimise log-loss is to output numbers you would genuinely bet at, so p̂ = 0.03 has to correspond to things that really do happen about three times in a hundred. That property is called calibration, and it is the licence that lets us read the output as a probability at all.

Then the mechanics, and one correction to an expectation you may have picked up. Setting the gradient to zero here gives equations with p̂ tangled inside a sigmoid, and no algebraic solution exists. There is no closed form. Chapter 20's normal equations were the exception in this course, not the rule.

It does not matter, because the loss is convex in w. Chapter 4 told us a convex function has one minimum and nowhere else to be, so Chapter 8's gradient descent walks straight to it. And the gradient is worth staring at, because it is ∇L = (1/n)·Xᵀ(p̂ − y). That is the same shape as the OLS gradient, with the residual sitting in the middle. Same machine, same descent, different link.

05The model's job ends at p̂

The model is finished, so hand it a borrower and it returns 0.031, which is a complete and honest answer. Notice what has not happened, because nothing has been classified.

To get a decision you have to compare p̂ against a cutoff t and flag everything above it. The moment you do that you have left statistics and entered economics, because t is not estimated from data. It is priced.

Everybody defaults to t = 0.5 and hardly anybody says why, so let us find out when 0.5 is right. You are screening loan applications. If you approve a loan that defaults you lose the principal, call it $50,000. If you refuse a loan that would have been fine you lose the interest, call it $500. Both are mistakes and they are not the same size of mistake.

Before the arithmetic, commit yourself and write down the cutoff you would actually use. Most people write something between 0.3 and 0.5, and a few write 0.2 and feel daring about it.

Now compute, using nothing but Chapter 11's expectation. Approving costs $50,000 with probability p̂, so its expected cost is p̂ × 50,000. Refusing costs $500 with probability (1 − p̂), so its expected cost is (1 − p̂) × 500. Refusing is the better action exactly when p̂ × 50,000 > (1 − p̂) × 500, and solving that gives p̂ > 500/50,500.

The cost-optimal threshold on this problem is 0.0099, which is about one percent and not fifty.

In general t* = C_FP/(C_FP + C_FN), so setting the two costs equal returns exactly 0.5. So the universal default is not neutral at all — it is a specific claim that your two mistakes cost the same, and in finance that is almost never true.

The gut resists this, and it is worth naming why. Flagging a hundred good borrowers to catch one bad one feels like generating false alarms on purpose. It is simply correct when the bad one costs a hundred times more, and $50,000 divided by $500 is exactly one hundred.

This gives us the separation the whole second half of the chapter needs. The score is what the model produces, and it is the model's business. The decision is what you do with the score, and it is the desk's business. They are computed by different people from different information. A great many broken production classifiers are the two of them welded together at 0.5 by nobody in particular.

06Four boxes, and nowhere else to go

Fix a threshold and every row now carries two bits: what was actually true, and what you said. Two bits give four combinations, and there is nowhere else for anything to go.

Do it by hand before any vocabulary shows up. Take the twelve loans, four of which really defaulted, flag everything scoring above 0.30, and sort them into four piles. Really defaulted and I flagged it. Really defaulted and I missed it. Was fine and I flagged it anyway. Was fine and I left it alone. The piles come out at 3, 1, 2 and 6, and they add to twelve, because they have to.

Arrange those four counts in a grid with the truth down the side and your call across the top. That grid is the confusion matrix, and it is the complete accounting of a thresholded classifier. Nothing is thrown away and nothing is added.

Now the names, with a rule that makes them permanent. The second word is what you said, and the first word says whether you were right. A false positive means you said positive and you were wrong. A false negative means you said negative and you were wrong. True positive means you said positive and you were right. There is no fifth case and no ambiguity, and you never have to look this up again.

Test it on the case that trips everyone. The loan was fine and you flagged it, so which box? You said positive, so the second word is positive, and you were wrong, so the first word is false. False positive. The reason this is hard without the rule is that "false" feels like it should describe the truth, when it actually describes your claim.

Then weld this to something you already own, because it is the same idea in different clothes. Chapter 17 gave us Type I error, the false alarm, rejecting a null that was true. That is the false positive. It gave us Type II error, the miss, failing to detect a real effect. That is the false negative. One concept, two vocabularies, taught in two different rooms by two different traditions. Statistics says Type I, machine learning says false positive, medicine says false alarm, and they are the same box in the same table.

One last observation, and it drives the next two sections. Every number in this table depends on t. Drop the cutoff to 0.15 and the same model on the same rows gives 4, 0, 4 and 4 instead. The table is not a property of the model — it is a property of the model and a decision.

07★★ Two directions through the same four numbers

Before anything else, a prediction, and please write it down. Your fraud detector has been evaluated on a million transactions, a hundred of which were fraud. The model caught ninety of the hundred, so its recall is 90%. It is also stingy, and flags only 1% of the perfectly good transactions.

Of everything this model flags, what fraction is actually fraud?

Most people write something between 80% and 95%. Ninety percent caught and one percent false alarms sounds like a strong model from both sides.

Now count, rather than reaching for a formula, because counting cannot be argued with. There were 999,900 good transactions, and 1% of those is 9,999 false alarms. Set that against 90 real catches, and of 10,089 flags exactly 90 are fraud.

That is 0.89%, which means this model is wrong about ninety-nine times out of a hundred every single time it opens its mouth. Its accuracy, meanwhile, is 989,991 out of a million, or 99.0%. And a model consisting of the single word "no" scores 99.99% and beats it.

Nothing went wrong, and there is no error anywhere in that table. What changed is the direction you read it in.

Read along a row of the truth, taking the transactions that really were fraud, all hundred of them, and asking what fraction you flagged. That is TP/(TP+FN) = 90/100, and the denominator is a truth row. So the question is "given that it was bad, did I catch it". It has three names because three fields invented it separately: recall, sensitivity, and the true positive rate. Written honestly it is P(flag | bad).

Now read down a column of your own predictions, taking everything you flagged, all 10,089, and asking what fraction really was bad. That is TP/(TP+FP) = 90/10,089, and the denominator is a prediction column. The question is "given that I raised the alarm, was I right". It is called precision, and honestly written it is P(bad | flag).

Stare at those two, P(flag | bad) and P(bad | flag): the same two events, conditioned in opposite directions. These are Bayes-inverses, and Chapter 10 already told you exactly what joins them: P(bad | flag) = P(flag | bad) · P(bad) / P(flag).

The base rate P(bad) is the exchange rate between the two directions. That is why they can come apart so violently. Hold recall at 0.90 and the false-alarm rate at 1%, then slide the base rate down. Precision reads 0.9890 when half the population is bad, 0.4762 at a base rate of 1%, and 0.0089 at one in ten thousand. Recall never moves at all, because its denominator never contained the base rate, and precision's denominator is made of almost nothing else.

You have solved this before, and you got it right. Chapter 10's disease test was this table with different words on it, and the reason you got it right there is that somebody told you to look at the denominator.

Now look at accuracy, (TP+TN) over everything, and notice what is missing. It has no conditioning direction at all, because it is a raw tally over the whole population, which means it is owned by whichever class is bigger. On a rare-event problem, accuracy is a measurement of the base rate wearing a model's clothes. The accuracy paradox is not a warning to remember — it is Chapter 10's base-rate neglect with a job title.

Two more definitions fall out of the same reading, so take them while the table is in front of you. Specificity is TN/(TN+FP) = P(no flag | good), which is the other truth row. And the false positive rate is FP/(FP+TN), which is that same row read the other way, so FPR = 1 − specificity.

Finally F1, the harmonic mean 2PR/(P+R), and there is an honest reason for the harmonic mean rather than the ordinary one. Take a model with precision 0.01 and recall 1.00, which is a machine that flags literally everything. The arithmetic mean says 0.505, a respectable-looking number for something that has learned nothing. The harmonic mean says 0.0198.

That is not a convention, because the harmonic mean cannot be rescued by the healthy member of a pair, and it always sits near the smaller one. That single property is the entire reason to use it.

08Stop choosing a threshold — sweep it

Everything in the last section was computed at one threshold, and we already know the threshold was a choice. So stop choosing, and instead sweep t from 1 down to 0 and watch the table evolve.

At t = 1 you flag nothing, so recall is 0 and there are no false alarms. At t = 0 you flag everything, so recall is 1 and precision has collapsed to the base rate. Every cutoff in between is one point, and the points trace a curve. Plot FPR against TPR and you have the ROC curve. Plot recall against precision and you have the precision–recall curve.

Now the part that turns folklore into mechanism, and it is one sentence long. Look at the denominators of the ROC's two axes. TPR = TP/(TP+FN) is a truth row, and FPR = FP/(FP+TN) is the other truth row. Both coordinates are conditioned on the truth, so the count of positives and the count of negatives each divide out of their own axis.

That means the base rate cancels, and the ROC curve barely notices when a problem becomes a hundred times rarer. This is not a defect being hidden. It is genuinely what you want when you need a prevalence-independent view of a scorer. But it is exactly why ROC flatters a model on a rare-event problem, because it is structurally incapable of seeing that your flags are now drowning in false alarms.

Precision's denominator is TP+FP, a prediction column, and prediction columns fill up with false alarms as the problem gets rarer. So the PR curve collapses and reports the damage. That is the whole of "use PR for imbalanced problems", derived instead of remembered.

Then the areas. AUC-ROC has an exact interpretation, and it is worth reading twice: it is the probability that a randomly chosen positive receives a higher score than a randomly chosen negative.

So AUC measures ranking, and nothing else. A model whose probabilities all sit squashed between 0.01 and 0.03, but in perfect order, scores an AUC of 1.0 and is useless for the threshold arithmetic we just built. A perfectly calibrated model that ranks badly scores poorly on AUC while its numbers are entirely honest. Ranking quality and calibration quality are two different virtues, and the phrase "a good model" hides which one you meant.

Last, the baselines, without which no AUC number can be read at all. A coin-flip model traces the diagonal and scores 0.5 on ROC. That same coin-flip model scores an AUC-PR equal to the base rate. So AUC-PR of 0.10 on a problem with a 0.01% base rate is a thousand times better than random and might be excellent, while AUC-ROC of 0.75 on the same problem may be worth nothing. Always quote the baseline beside the number.

09Imbalance, in order of honesty

Now we can say something useful about imbalance, and the order matters more than the list. Most readers meet a rare-event problem and reach straight for SMOTE, because resampling is what tutorials teach and because "the data is imbalanced" sounds like a data defect. Usually it is not.

Remedy zero, which solves a large share of real cases: change nothing about the data and fix the scoring instead. Stop reporting accuracy. Report precision and recall at a stated threshold, plus AUC-PR against its base-rate baseline. Set the threshold from costs rather than from habit. A great deal of "the model can't handle imbalance" turns out to mean "we scored it with accuracy and cut it at 0.5", and both of those are now fixed.

Remedy one is class weights. Multiply the minority terms in the log-loss by a factor, so the loss becomes −(1/n) Σ [ w₁·y·ln p̂ + w₀·(1−y)·ln(1−p̂) ], and a missed positive hurts more during training. Two things are worth noticing. It is not a new idea, because it is the cost asymmetry from the threshold section moved out of the decision and into the objective. And it is mechanically the same as duplicating each minority row w₁ times, which demystifies both at once.

Remedy two, and last, is resampling. Undersample the majority, which is cheap and throws information away, or oversample the minority, which invites overfitting to the exact rows you duplicated. Or use SMOTE, which manufactures synthetic minority rows by interpolating between a minority point and its near neighbours. All three change the class ratio the model sees.

Now the two warnings, and they are the actual content of this section, because the remedies are in every tutorial and the warnings are in almost none.

First: reweighting and resampling break calibration, and they break it by design. You deliberately changed the prior the model was trained under, so p̂ no longer means what it says. A model trained on a rebalanced 50/50 sample reports default probabilities near 0.5 for a book whose true default rate is 1%.

If your deliverable is a ranking, that may be fine. If the deliverable is the probability itself, you have just destroyed the product to improve a metric. In credit risk the probability is the product, because PD feeds straight into expected loss, and Chapter 32 will spend that number. The fix is to not resample, or to recalibrate afterwards by correcting for the prior shift.

What makes this so dangerous is that it is invisible. Every metric you were taught to check is either threshold-based or rank-based, so precision, recall, F1 and AUC are all completely blind to a global shift in the probabilities. The one-line check is to print the mean predicted probability and compare it to the observed base rate.

Second: resampling must happen inside the cross-validation fold, after the split, never before. Oversample first and copies of the same row land in both training and validation. Run SMOTE first and a synthetic point interpolated from two training rows sits in your validation set. Your score goes up and it is a lie. That is Chapter 21's data leakage in a new costume, and it is one of the most common ways a rare-event model ships broken.

So the correct pipeline is an ordered list, and it is worth copying down. Split first. Resample the training part only. Fit. Then score the untouched validation part.

10Two hundred features that are secretly three

The first half of this chapter fixed what the model outputs, and this half fixes what goes in.

Hand our logistic model a realistic feature set: twenty points along a government yield curve, say, or two hundred variables from a credit bureau. Something breaks that has nothing to do with probabilities or thresholds.

Fit the model on the first half of the data and read the coefficients, then fit it on the second half and read them again. The two-year weight is strongly positive in one fit and strongly negative in the other. The three-year weight moves the opposite way by almost exactly as much. And the fitted probabilities barely change at all.

That last part is the tell, and Chapter 20 already named the cause. This is multicollinearity. When two features are near-duplicates the model cannot tell which one is doing the work, because only their sum is determined by the data, and any split of that sum fits equally well. Chapter 6 said a near-singular matrix collapses space, so undoing it amplifies noise. Chapter 16 said a statistic is itself a random variable, and this one has an enormous spread.

The result is a model that is simultaneously predictive and uninterpretable, which is fatal anywhere you must justify a decision. Add Chapter 21's p ≫ n on top, two hundred features against three hundred observations, and there is not enough data to pin anything down at all.

Chapter 22 already gave us one answer, which is to shrink: ridge stops the coefficients running to ±∞ in opposite directions, and lasso zeroes most of them outright. Both are real answers and neither is wrong.

But there is a completely different answer available, and it starts by noticing something about the data rather than about the model. Look at how a yield curve actually moves from one day to the next. Overwhelmingly, the whole curve shifts up or down together, and every tenor moves the same way. Sometimes the short end goes one way while the long end goes the other, so the curve steepens or flattens. Occasionally the belly bulges relative to both ends.

Those three motions have names on every rates desk: level, slope and curvature. Between them they account for almost everything that ever happens to the curve. So the twenty numbers you were handed were never twenty independent facts. They are about three real degrees of freedom wearing twenty costumes, and the correlation matrix is the shadow that redundancy casts.

Which gives us the question the rest of the chapter answers. If the data really only moves in three ways, what are those three directions, and how would we find them from the data itself rather than from a story about bond markets?

11The direction that carries the movement

"Find the directions the data actually moves in" is a wish, not a definition. So make it precise, and watch it turn into something we already own.

A direction is a unit vector u. Project every centred observation onto it and you get one number per row, u·x, which is how much of that direction is present in that row. Now ask how much those numbers vary. Chapter 18's covariance matrix gives the answer in one expression, Var(u·x) = uᵀΣu.

A matrix sandwiched between two copies of a vector is a genuine shock the first time, so ground it before we use it. Take u = (1,0), which is the first feature's own axis. Then uᵀΣu picks out Σ₁₁, the variance of feature one. So this is not a new object at all. It is the variance in a direction, and the direction happens to be one you can point at.

Now "the direction that carries the most movement" has an exact meaning. Maximise uᵀΣu, but not without a constraint, and the reason for the constraint matters. Double the length of u and uᵀΣu quadruples, so without a constraint you would just make u longer forever and learn nothing about direction. Fix the length: uᵀu = 1.

That is a constrained optimisation, which is Chapter 8's Lagrange multipliers, and it is four lines. Form the Lagrangian uᵀΣu − λ(uᵀu − 1). Differentiate with respect to u to get 2Σu − 2λu = 0, then rearrange, and out comes Σu = λu.

Stop and look at what just happened. We asked a purely statistical question, which direction has the biggest spread, and out fell the eigenvector equation from Chapter 7. We met that as pure linear algebra with no statistics anywhere near it. Nobody chose eigenvectors here — they were forced by asking for maximum variance at unit length.

There is a second gift in the same four lines. Substitute the solution back into the objective: uᵀΣu = uᵀ(λu) = λ(uᵀu) = λ. The value of the thing we were maximising, at the optimum, is exactly the eigenvalue.

So λ is not an abstract stretch factor here. The eigenvalue IS the variance of the data along its own eigenvector, measured in the units of the data. That one identity is why every sentence ever written about PCA mentions eigenvalues, and why "this component explains 62% of the variance" is a literal statement rather than a figure of speech. That direction has a name: the principal component.

12The rotation that makes the covariance diagonal

One direction is not enough, so keep going. The second principal component is the max-variance direction among those orthogonal to the first, the third is orthogonal to both, and so on down.

That sounds like a fresh constrained problem every time, and Chapter 7 already spared us the work. Σ is symmetric, so its eigenvectors are automatically orthogonal, which means the eigenvectors sorted by descending eigenvalue are that sequence. And because Σ is positive-semidefinite, every eigenvalue is at least zero, which is the only sane outcome given each one is a variance.

So here is the whole algorithm, in seven moves. Centre the data, then standardize it, form Σ, and eigendecompose. Sort λ₁ ≥ λ₂ ≥ …, keep the top k, and project.

Three parts of that deserve more than a mention, and the first is a word that names two different objects. A loading is a direction: an eigenvector, a fixed recipe of length p over the original features. A score is a number per observation: the projection of one row onto that direction, of which there are n. People call both of them "the component", which is why nobody can say whether PC1 is a vector of length p or of length n. It is both, in two different senses, and keeping the words apart fixes it.

The second is standardization, which is not optional. PCA maximises variance, and variance has units. A feature recorded in basis points has a variance ten thousand times larger than the same feature in percent, purely because of the scale. Skip Chapter 22's z-scoring and PC1 is simply whichever column happened to be written in the smallest units. Nothing errors, the scree plot looks fine, and the answer is wrong.

The third is the payoff, and it is the real reason to do any of this. In the new coordinates the covariance matrix is diagonal. Projections onto different eigenvectors have zero covariance with each other, by construction, because Σ diagonalises in its own eigenbasis. The components are uncorrelated, which cures exactly the disease that sent us here. One line describes the whole method: PCA is the rotation that diagonalizes the covariance matrix.

That also settles how much to keep. Total variance is Σλᵢ, which equals trace(Σ), which equals the sum of the individual feature variances. So λ_k/Σλ is a genuine fraction of the total variability rather than a metaphor. Plot the sorted eigenvalues and you get the scree plot, and on real yield-curve data the first three bars typically account for something like 99% of daily variation. Level, slope and curvature reappear as a number the data produced by itself.

One practical note from Chapter 7 that matters more than it looks. In production you never form Σ at all. You take the SVD of the centred, standardized data matrix, whose right singular vectors are exactly these eigenvectors and whose singular values squared over (n−1) are exactly these eigenvalues. Forming Σ squares the data, which squares the condition number, and Chapter 7 told us what that does to accuracy.

13What PCA never looked at

Now the honesty, because PCA has one blind spot and it is enormous. It never looked at y.

Go back through the derivation and check. Every line used Σ, the covariance of the features. The labels were not in the room at any point, which is what it means to say PCA is unsupervised. "The direction with the most variance" and "the direction that separates the classes" are two different requests, and they coincide often enough that people stop distinguishing them.

Here is the smallest picture that makes the failure permanent. Two features, two classes. The cloud is enormously spread along one axis, with both classes fully mixed along it. Along the other axis, the narrow one carrying a few percent of the variance, the two classes sit in clean separated bands.

PCA sorts by variance, so PC1 is the useless long axis and PC2 carries the entire signal. Keep "the first component, explaining 96% of the variance" and you have thrown away everything you needed. Your explained-variance number looks excellent while your classifier goes to chance.

So "reducing dimensions improves the model" is not a rule. It is sometimes true and sometimes catastrophic, and the deciding question is whether the signal happens to live in the high-variance directions. Nothing in the algorithm checks that for you.

Three more limits, stated plainly. The components are dense blends of every original feature, so interpretability is gone. You can no longer say "debt-to-income was the driver", only "component 2 was the driver", and component 2 is a recipe with two hundred non-zero ingredients. That is the direct trade against Chapter 22's lasso, which shrinks worse but hands you names. Choose lasso when you must justify a decision to a regulator. Choose PCA when the features are a redundant basis like a curve and what you need is stability.

Next, PCA is linear and it only measures variance. Structure living on a curved manifold, or in a higher moment, is invisible to it, and Chapter 18 already warned that covariance only sees linear co-movement.

Last, and operationally the one that bites: the rotation is fitted from data. Fit it on the training fold only and apply that same rotation to validation and test. Fitting PCA on the full dataset before splitting leaks the test set's covariance structure into the transform. Preprocessing does not feel like fitting, which is exactly why this one slips past people who would never dream of fitting a model before splitting. It is a fit, and it is Chapter 21's sin.

14One machine, both halves

Close the loop the chapter opened. Take the twenty correlated curve features, rotate them onto three principal components, and feed those three into the logistic regression we built at the start.

Look at what that pipeline has. The collinearity is gone by construction, because the components are uncorrelated. The coefficients are stable, and the p ≫ n pressure is gone. And the output is still a calibrated probability you can threshold by cost.

It has a name, principal component logistic regression, and it is exactly the shape Chapter 32 uses when it turns borrower features into a probability of default and multiplies by loss-given-default to get an expected loss. That is this entire chapter, both halves, doing one job for money.

Two things travel with you out of here. The first is a habit of asking two questions of any classifier somebody hands you. What threshold is this running at? And what do the two mistakes cost? If the answer to the first is 0.5 and nobody can answer the second, the model has not been evaluated yet.

The second is the instinct underneath all three squeezes, and it is the same instinct every time. A probability would not live on the real line, so we moved to the log-odds. A pile of correlated features would not give stable weights, so we rotated to the axes where the covariance is diagonal. When a quantity will not behave, do not fight it in the coordinates you were handed. Find the coordinates where it is simple, and go there.

That closes Part 3. Everything since Chapter 20 has been about estimation, which is the business of saying what returns and defaults are, from data, honestly. Chapter 25 asks a different question and it needs almost none of this machinery. Given that a claim pays off in some future state, what is it worth today? The surprise is where the answer starts. Not with probability, and not with stochastic calculus, but with a single sentence about two portfolios that pay the same.