19Multiple Testing & the Bayesian Split

Chapter 18 closed on a warning. A single honest correlation can wander by 0.13 through luck alone, so screening a thousand pairs for the largest one is not research. This chapter is the bill for that sentence, and the bill is larger than it looks. Test twenty things at the 5% level and one discovery arrives free, made of nothing at all. The two standard repairs are Bonferroni and Benjamini–Hochberg, and both are usually taught as hygiene. They are not hygiene. They are prices, and the thing they price is a number you never wrote down: how many hypotheses you walked in with, and that number has another name. Chapter 17 told you a p-value can never be P(H₀ | data), because the posterior needs a prior the test deliberately never asks for. The prior was in the room the whole time. It is your search. Once you see that, the multiple-testing correction and the Bayesian prior stop being two subjects. They are one bill in two currencies, and it is charged twice — once on the verdict, and once on the size of the effect.

Look at what this page stands on, because all of it is already yours. Chapter 9 gave us the complement rule and inclusion–exclusion, and Chapter 10 gave us independence, Bayes' theorem, and the habit of counting a crowd instead of multiplying percentages. Chapter 12 gave us the Beta distribution and its conjugate update. Chapter 13 gave us order statistics, and the fact that the maximum of a sample is a random variable with a life of its own. Chapter 16 gave us SE = σ/√n and the decomposition MSE = Var + bias². And Chapter 17 gave us the whole testing machine: the null distribution, the p-value as a tail area, α as a line drawn before the data, and one warning it could not cash.

That is the seam. Chapter 17 did not withhold P(H₀ | data) to be pedantic. That quantity needs an ingredient the procedure never asks for, and the whole job of this chapter is to find out where the ingredient has been hiding.

01Twenty strategies made of nothing

Let us start by building nothing, twenty times, and watching it look like something.

Here is the generator, and you own it completely. Each day, each strategy returns a draw from a Normal distribution with mean exactly zero and a daily standard deviation of 1%. There is no edge. There is no signal. There is no subtle structure hiding in the data that a clever reader might defend, because we wrote the line of code that produced every number.

Run twenty of them over 1,260 days, which is five years of daily data. Then run Chapter 17's machine on each one. Compute t = mean / (sd/√n), look up the tail area, and read off twenty p-values.

Look at the whole set rather than any single one. The p-values are spread roughly evenly across the interval from 0 to 1. Some sit near 0.9, some near 0.4, and one of them, strategy #4, comes in at 0.0354. That result is significant at the 5% level. It is also, by construction, made of absolutely nothing.

Press it again and the pattern holds. Across twelve batches the run produces 13 false alarms in 240 tests, which is 5.4%. The expected count per batch of twenty is exactly 1.00, and the run delivers 1.08.

Chapter 17 stated this fact in its closing breath: under H₀ the p-value is uniform on the interval from 0 to 1. It arrived there as a curiosity, and it is nothing of the kind. It is a manufacturing specification, and it deserves one line of proof using nothing we have not already built.

The p-value is defined as the fraction of the null cloud lying beyond you. Now suppose you are a draw from the null cloud. Then asking what fraction of the cloud lies beyond you is asking where you landed in your own ranking. And the bottom 5% of a ranking contains 5% of the draws, by the meaning of the word fraction.

That is the entire argument, and nothing else is being claimed. The 5% smallest p-values and the 5% most extreme null draws are the same set of draws called by two different names.

So read α correctly, once and for all. It is not a rate of being right. It is not a rate of being wrong about things that are real. It is a manufacturing rate, the number of false alarms a procedure produces per hundred boring worlds you point it at. Point it at twenty boring worlds and it manufactures, on average, one.

Now say the objection out loud, in your own words, because it is coming anyway. The data does not know how many other tests I ran.

Exactly right. It does not. Strategy #4's 1,260 returns are the same 1,260 numbers whether it was the only thing you tried or one of a thousand. Nothing in that column of data changes when you run something else in another window.

Hold on to that objection, because it is not a defence of the winner. It is the whole diagnosis, and everything that follows is the consequence of taking it seriously.

02The family, and the number fourteen

Every guarantee Chapter 17 handed you was about one test. Draw the line before you look, and you accuse an innocent null 5% of the time. Nobody in this business runs one test.

So change the unit of accounting. Define the family as the set of tests whose errors you have decided to control together, and then ask the only question that matters about a family. What is the probability that at least one of them produces a false alarm? That quantity is the family-wise error rate, or FWER.

Before we compute it, commit to a guess. At the 5% level, how many honest tests does it take before at least one false alarm is more likely than not?

Almost everyone answers twenty, and the reasoning behind twenty is sound as far as it goes: you want one expected false positive, and one expected false positive takes 1/α = 20 tests. The trouble is that this computes E[V] = mα, the expected count of false alarms, and the question asked for P(V ≥ 1). Two different quantities, one intuition, and they separate fast.

The real calculation takes two facts you already hold. Each test stays quiet with probability 1 − α, and Chapter 10's product rule lets us multiply across independent tests, so P(all quiet) = (1−α)^m, and Chapter 9's complement rule finishes it.

FWER = 1 − (1−α)^m

Now put numbers on it, because the numbers are the shock. At α = 0.05, five tests give 22.6%, ten give 40.1%, and fourteen give 51.2%. Twenty give 64.2%, and a hundred give 99.4%.

Fourteen. Fourteen impeccably conducted tests, each one at the 5% level, and it is already more likely than not that you are holding a discovery made of noise.

The shape behind those numbers is worth seeing too. Chapter 2 gave us the approximation (1−x)^m ≈ e^(−xm) for small x, so FWER ≈ 1 − e^(−αm). At m = 14 that reads 0.5034 against the exact 0.5123, which is close enough to trust the shape.

Read what that approximation says, because it is the sentence worth carrying. The thing governing your exposure is not α and it is not m, but their product. Cut α by a factor of ten, run ten times as many tests, and you are standing in exactly the same place.

Which means the fix, if there is one, is already visible. If αm is what you are exposed to, then the only way to hold your exposure still while m grows is to divide α by m. We will derive that properly in a moment, but notice that the picture handed it to you first.

03You do not report a random one

There is a second crime at this scene, and it is the one that actually costs money.

Nobody runs a thousand backtests and reports a randomly chosen one. You sort by Sharpe ratio and you report the top of the list. So the statistic you hand to your risk committee is not a t-stat at all. It is the maximum of m t-stats, and Chapter 13 already told us that the maximum of a sample is a random variable in its own right, with a distribution of its own.

Building that distribution takes one honest line. The maximum is below x exactly when every one of the m tests is below x. Those are the same event written two ways, so independence gives us the answer immediately.

P(max ≤ x) = Φ(x)^m

Set that equal to 0.5 and solve for x, and you get the typical best t-statistic that pure noise produces. It is arithmetic, not simulation, and the numbers are unpleasant.

Twenty tests give a best t of 1.82, a hundred give 2.46, and a thousand give 3.20. And a million-combination parameter sweep, which is an afternoon on a laptop, gives 4.83.

Sit with that last number for a moment. A t-statistic of 4.8 is the kind of number that ends arguments in a meeting. It is also the ordinary, expected output of a machine searching noise.

You will also meet the rule of thumb √(2 ln m), which gives 2.45, 3.04, 3.72 and 5.26 at those same four sizes. It runs a shade generous at every m on the table, so treat it as an upper sketch rather than a value. Its real lesson is the one Chapter 5 taught: ln m crawls. A million tests buy only about 2.6 times the best t-stat that twenty tests buy, so every extra decimal place of t costs you an order of magnitude more searching.

Now the deeper damage, and this is the one the corrections will never touch.

Suppose one of your candidates is genuinely real, with a true Sharpe of 0.80. What got it to the top of your sorted list? Its edge, plus a good run of luck. Among a thousand candidates the winner is almost never the one with the best truth. It is the one with the best truth plus luck.

And that luck is now sitting inside the number you reported, positively, by construction, because you selected on it. This is the winner's curse, and it survives the effect being completely real.

So multiplicity does not damage you once. It damages you twice. There is a false verdict, where you believe something is there, and there is an inflated estimate, where you believe it is this big. Hold on to the second one, because half of this chapter exists to pay for it.

04Divide by m, and what it costs

Now fix the verdict, with the bluntest instrument in statistics.

You want the probability of at least one false alarm across the whole family to stay under α. Start from Chapter 9's inclusion–exclusion for two events, P(A∪B) = P(A) + P(B) − P(A∩B), and notice one small thing. The subtracted term is a probability, so it cannot be negative, so throwing it away can only make the right-hand side bigger.

Extend that to m events and you have the union bound, P(A₁∪…∪A_m) ≤ ΣP(Aᵢ). It is inclusion–exclusion with the overlaps thrown in the bin, which makes it a deliberate over-count. Point it at the family: if each test runs at level α', then FWER ≤ mα', and setting α' = α/m caps the whole thing at α.

That is Bonferroni, and the most important thing about it is what the derivation never used. Nothing was assumed about how the tests relate to one another. Your thousand signals can be tangled together in any way you like and the guarantee still holds, which is exactly why a crude nineteenth-century inequality is what a compliance department reaches for.

Now count the cost, on Chapter 17's own instrument. At m = 100 the per-test level is 0.0005, and the two-sided critical value moves from 1.96 out to 3.48. Slide that line to the right on the two-curve picture and watch the area under the alternative collapse behind it.

Hold the effect size and the sample size fixed, and place the alternative where you have 80% power at the ordinary bar. The power ladder then reads 80% at m = 1, 50% at m = 10, 25% at m = 100 and 10% at m = 1,000.

So Bonferroni does not make you careful. It makes you deaf. Real and modest edges go undetected, and for a fund that is not safety. It is bankruptcy in a different font. The word conservative is doing quiet damage here, because in ordinary English it means safe, and what it actually means is that Type I risk has been converted into Type II risk. Chapter 17 was clear that those two costs are not equal and not interchangeable.

The over-count has one more expensive shape, and it is specific to quant work. Suppose your thousand strategies are really one idea with a thousand parameter settings. They are near-duplicates, so the overlap terms the union bound discarded were enormous, the effective number of independent tests might be a dozen, and Bonferroni charges you for a thousand.

05Two error rates wearing one word

Stop and ask a question nobody asked you. What are you actually trying to guarantee?

Bonferroni's objective is a strong one: across this entire family, be near-certain you have not made even one false claim. For a drug going into humans that is obviously right. For a quant screening five hundred candidate signals it is close to mad, because you are not making a claim. You are building a book, and a book of forty signals of which three are junk is an excellent book.

So write down the tally of an m-test campaign honestly. Of the m hypotheses, some are truly null. You reject R of them in total. Of those R, V were nulls you falsely accused and S were real effects you correctly caught, so R = V + S.

Bonferroni controls P(V ≥ 1), while the quantity a signal book actually cares about is V/R, the fraction of the things you are about to trade that are noise. Its expectation is the false discovery rate, FDR = E[V/R].

Now look at where those two numbers live in the same table, because this is the sentence to carry out of the chapter. Lay the m tests out as a two-by-two grid. The rows are what the world is, null or real, and the columns are what you declared, significant or not.

Take 500 candidates of which 60 are genuinely real, test at 5%, and give yourself 50% power. The 440 nulls throw up 22 false alarms, and the 60 real effects yield 30 catches. So R = 52 discoveries and V = 22 of them are noise.

Read along the row and you get 22/440 = 5%, which is α, which is P(significant | null). Read down the column and you get 22/52 = 42%, which is the FDR, which is P(null | significant). Same table, same cells, two perpendicular sweeps, and a factor of eight between the answers.

You have met this pair before, wearing a medical coat. Chapter 10's test with a 99% detection rate produced a set of positives that was mostly false. The false positive rate and the false discovery rate differ by one flip of the conditioning bar, and confusing them is base-rate neglect with a job in finance.

Which objective you want is a business question, not a statistical one. Ask which sentence you would rather be able to say to a risk committee. Either I am 95% sure not one of these forty signals is fake, or I expect about four of these forty to be fake. The first is unbuyable at any workable power, and the second is what you wanted anyway. Call the FDR level you are willing to run a book at q.

Before we build the procedure, notice one tell. To compute an FDR you must know how many of the m candidates were real. That is a base rate. That is a prior. The frequentist objective a quant actually wants already has a prior sitting inside it.

06The sag, and the line drawn through it

Now build the procedure that controls that quantity, and build it out of a picture rather than a recipe.

Sort your m p-values from smallest to largest, write them p₍₁₎ ≤ p₍₂₎ ≤ … ≤ p₍m₎, and plot each one against its rank k. Then ask what that plot looks like if nothing in your pool is real.

We already know the answer. The null p-values are uniform, and m uniform draws cut the interval into m+1 gaps of equal expected width, so the k-th smallest sits on average at k/(m+1). The all-null picture is a straight diagonal from bottom-left to top-right, drawn by pure noise, free of charge.

Now plot a real campaign on top of it. If some of your hypotheses genuinely carry an effect, their p-values pile up near zero, and the left end of your curve sags below the diagonal.

That sag is not decoration. It is a measurement. It is the visible excess of small p-values over what noise alone would have produced, and how far it extends tells you roughly how many real effects are sitting in the pool.

Benjamini and Hochberg turn exactly that sag into a licence. Draw a second line through the origin with slope q/m. Find the largest rank k whose p-value still falls under that line. Reject everything up to and including rank k.

Two things about that rule deserve slow attention. The first is why we take the largest passing rank rather than judging each point on its own. This is a step-up procedure, so the crossing sets one cutoff for the whole family. In the ten-value set on screen, rank 4 sits at 0.041 against its own line of 0.040 and fails. It is rejected anyway, because rank 5 came in at 0.048 against a line of 0.050 and passed. That is one decision about where to cut, not ten separate verdicts.

The second is why it beats Bonferroni, and the answer is that the bar is adaptive. The more small p-values you actually observe, the further right the crossing happens and the looser the final bar becomes. BH lets the data tell it how many real effects are around, and then spends that information as power. On the same ten values, Bonferroni at 5% has a bar of 0.005 and rejects one. BH at q = 0.10 rejects five.

And look at where the BH ladder starts. At k = 1 the threshold is q/m, which is Bonferroni run at level q, character for character. So these are not two rival recipes. Bonferroni is BH's first rung, the bar you must clear if you found nothing else, and every additional discovery loosens the bar for the rest.

07The correction that never touched the number

Now collect on the debt from the winner's curse, because it is about to reorganize the chapter.

Run the honest experiment on two hundred candidate strategies, with ten years of daily data each. Twenty of them are genuinely real with a true annualised Sharpe of 0.80, and the other 180 are dead. You know exactly which is which, because you built them that way. Sort by measured Sharpe, apply both corrections to the whole family, and check both things at once.

The verdict machinery works, and it works well. BH at q = 0.10 makes 12 discoveries, of which 11 are real and 1 is noise, so the realised false-discovery proportion is 8.3% against a promised 10%. Bonferroni makes 5 discoveries with no false ones at all, and pays for that purity by missing six real strategies that BH caught. The whole of the last two sections is vindicated in one run.

Then look at the number attached to the winner. Strategy #184 has a true Sharpe of 0.80. Its measured Sharpe is 1.496, and its t-statistic of 4.73 clears Bonferroni's bar of 3.66 with room to spare.

BH said yes. BH was right to say yes, because the effect is real. And the reported number is nearly double the truth, and neither correction touched it, because neither of them is in that business.

Read what a correction actually does, because it is narrower than it sounds. It moves the line separating reject from fail to reject. It is a verdict machine, so it changes who gets through. It has no opinion whatsoever about how big the thing that got through is.

That gap has a price, and it is not academic. Chapter 15 established that the size of the bet is set by the size of the edge, because Kelly stakes a fraction of your bankroll proportional to your estimated advantage. Feed a Sharpe of 1.496 into a sizing rule when the truth is 0.80 and you are 1.87 times too levered, on a position that is genuinely real. That is the precise recipe for a drawdown that is survivable in theory and terminal in practice.

So we need a second machine, one that fixes the number. Ask what such a machine would have to know. It would have to know something you have never once written down: how big effects tend to be among the kinds of things you test. Not this effect. The population of effects your search draws from.

If most of your candidates are duds, and even the good ones earn a Sharpe of 0.3, then a measured 1.5 is mostly luck and should be pulled hard toward zero. If you work somewhere real effects are usually huge, a measured 1.5 is unremarkable and should be left alone. That distribution over the truth is the missing ingredient, and it has a name we have been circling since Chapter 10.

08★★ The search IS the prior

Here is the identification the whole chapter turns on.

Two research notes land on your desk, and every number on them is identical. Five years of daily data. A mean daily return, a standard error, t = 2.5 and p = 0.012. Note A comes from a quant who tested one signal she had an economic reason to expect, with a specific mechanism she can argue for. Note B comes from a quant who swept a thousand mechanical parameter combinations and printed the best one.

Commit to an answer before the arithmetic. Is one of these more likely to be a real edge than the other, and by how much?

The p-values are the same, and they are the same for a structural reason rather than by oversight. Chapter 17 built the p-value entirely inside H₀'s world, and H₀'s world is a generator with numbers in it. It can tell you how often a boring market coughs up a t of 2.5. It has no channel through which to learn that you walked in carrying a thousand hypotheses instead of one.

So the objection you raised in the first section was correct all along. The data does not know. The p-value cannot know.

And here is the resolution, which agrees with you rather than overruling you. The number you actually wanted was never the p-value. You wanted P(this edge is real | it came out significant), which is Chapter 10's Bayes read in the direction that matters, and whose denominator contains the base rate of real effects in your candidate pool.

So put the p-value away and count candidates. Quant B swept a thousand variants, so suppose 2% of mechanical variants carry any real edge, which leaves 20 real and 980 dead. The 980 dead ones, tested at the 1.2% bar his winner cleared, throw up about 12 false winners. His 20 real ones, at realistic power, yield about 10 true winners. His headline result is one of 22 winners, and 12 of those 22 are noise, so P(real | significant) ≈ 45%. A coin flip.

Quant A tested one hypothesis with an economic argument behind it, so call her prior 50-50. Half the time it is real and half of those get caught, which puts 0.25 of the mass on a true winner. Half the time it is dead and 1.2% of those fire, which is 0.006. So P(real | significant) = 0.25/0.256 ≈ 98%.

Ninety-eight per cent against forty-five, from the same t-statistic. Nothing in the data differed. Nothing in the p-value could have differed. The only thing that differed between these two quants was how they searched, and the only number that moved was the prior.

That is the identification. The search is the prior. Sweeping a thousand mechanical variants is not a neutral act of diligence. It is a statement that you had no particular reason for any one of them, which is exactly a claim that the base rate of your pool is low. The prior did not arrive from philosophy. It arrived through your workflow, and it has been in the room the entire time, unstated.

Once you see that, the three procedures on this page collapse into one object. Bonferroni multiplies the bar by m because a large m means a low prior, and it is crude precisely because it uses only the head-count of your search. Benjamini–Hochberg does better, because the sag in the sorted p-value plot measures how many real effects are in the pool, so it estimates the prior from your own data and spends it as power. And the Bayesian route writes the prior down as a distribution and pays the bill on the estimate instead of the threshold.

One bill, three currencies. And notice which of the three can also fix the inflated number the winner's curse left behind. Only the one that puts a distribution on the truth.

One last thing before we name the split. The prior is never optional. It is only ever unstated. Quoting a raw p-value is acting on a flat prior, and in a world where most backtests are junk, flat is the most aggressive assumption in the room.

09What the two camps disagree about

Name the split, now that it has already happened underneath us.

Everything in this chapter has been a fight over one question. Is θ, the true edge, a fixed unknown constant, or is it a thing you can put a distribution on? Both answers are coherent, and they define the two camps.

The frequentist says θ is a fixed number, unknown but not random. The randomness lives entirely in the data, because the data is what you re-draw when you imagine the experiment repeated. That is why every frequentist guarantee is a statement about a procedure over hypothetical repeats. It is also why Chapter 17 insisted that a 95% confidence interval's 95% belongs to the interval-making machine and not to θ.

The Bayesian says the data is not random at all, because you already saw it and it is sitting on your desk. What is uncertain is θ. So put a distribution on θ that encodes what you know, and let the data update it by Chapter 10's rule. Every Bayesian statement is then about this dataset, conditional on a prior you must write down and defend.

Now cash the mirror, because it retro-justifies the pedantry you resented in Chapter 17. When you first met a confidence interval you wanted to read it as there is a 95% chance θ is in here, and you were told firmly that this is not what it says.

That reading was never stupid. It was Bayesian. It is the credible interval, P(θ ∈ [a,b] | data) = 0.95, and it is completely legitimate. It costs exactly one thing, which is a stated prior. Your instinct was right about the question and wrong about which machine answers it.

Two closing honesties. The first is that neither camp is correct, because they answer different questions. How weird is this data if nothing is going on, and what should I now believe about θ, are not rival answers. Swapping one for the other is precisely the inversion error Chapter 17 warned about.

The second is the practical stance, said plainly. A working quant is Bayesian in the gut and frequentist in the report. You do not truly believe that a signal found by grid search and a signal derived from a market mechanism deserve the same credence. That belief is a prior, and it is doing work whether or not you write it down. All anyone is asking is that you write it down, so that it can be argued with.

And if the word subjective is being used as a slur, take an inventory first. Choosing α = 0.05 is unargued. Choosing which tests belong to the family is unargued. Choosing to report a raw p-value is unargued. The difference between the two camps is not which one makes assumptions. It is which assumptions get written down.

10Four imaginary coin flips

Build the machine that fixes the number, out of something you already own, so that it costs almost nothing.

Go back to Chapter 12's conjugate pair. You put a Beta(a,b) prior on a coin's bias, saw h heads and t tails, and the posterior was Beta(a+h, b+t) with mean (a+h)/(a+b+h+t). Stare at that fraction, because it is hiding a message.

It is exactly the weighted average of the prior mean a/(a+b) and the sample mean h/(h+t), with weights (a+b) and (h+t). In other words the prior behaves like a batch of pseudo-observations. A Beta(2,2) prior is worth four imaginary flips that came up even.

Watch it work on real counts. Ten real flips with 7 heads against those four pseudo-flips gives a posterior mean of 9/14 = 0.643, which sits between the prior's 0.5 and the sample's 0.7 and leans toward the data. Three real flips that all came up heads gives 5/7 = 0.714, nowhere near the sample's 1.0, because four imaginary flips outweigh three real ones.

That is all a posterior mean ever is. It is the data dragged toward the prior by an amount set by how much data you brought. It has a name, shrinkage, and you have been doing it since Chapter 12 without calling it that.

Now transplant the shape onto a continuous effect like a mean return or a Sharpe ratio, where the conjugate story is Normal on Normal and the role once played by the count is played by precision, which is one over variance.

Your estimate θ̂ arrives with a standard error SE, so it carries precision 1/SE². Your prior says effects around here are centred on zero with spread τ, so it carries precision 1/τ². Take the same precision-weighted average of the prior mean and the data, and the posterior mean falls out.

posterior mean = θ̂ · τ²/(τ² + SE²)

With the prior centred at zero, that factor always sits between 0 and 1, so the posterior mean is the estimate pulled toward zero. Never past it, and never beyond θ̂.

Read the two limits and the formula stops being a formula. When the data is precise, so SE ≪ τ, the factor goes to 1 and you keep your estimate, because the data is telling you something the prior did not know. When the data is noisy, so SE ≫ τ, the factor goes to 0 and you keep almost nothing, because a measurement whose wobble dwarfs the entire spread of real effects is mostly wobble.

Notice what you have just agreed to do. You are returning an estimate you know is biased low, on purpose. That feels wrong, and Chapter 16 wrote you the permission slip three chapters ago: MSE = Var + bias², and a biased estimator beats an unbiased one whenever the variance it buys back is worth more than the bias it pays. Shrinkage is not a philosophical concession. It is that trade, executed.

And if the flat prior still feels like the neutral choice, draw it. A flat prior over Sharpe ratios from −3 to +3 says a true Sharpe of 2.8 is exactly as plausible as a true Sharpe of 0.1. Nobody believes that. Refusing to choose a prior is not neutrality. It is choosing a bad one quietly.

11The Sharpe haircut

Point the machine at the number a quant is actually judged on.

You have a backtest reporting a Sharpe ratio of 1.5 over T years. Chapters 16 and 17 already handed you its wobble for free. The template t = SR·√T puts SE(SR̂) ≈ 1/√T in the denominator, so five years of daily data leaves your Sharpe with a standard error of about ±0.45. That is an enormous error bar, and almost nobody quotes it.

One honesty note before we spend that number. The 1/√T form ignores the error in estimating the volatility itself. The fuller expression for iid normal returns is √((1 + SR²/2)/T), which at a Sharpe of 0.8 over ten years gives 0.363 rather than 0.316. So 1/√T is a floor on your uncertainty, and the honest haircut is slightly larger than the one we are about to compute.

Now the prior, which asks a question you have never been made to answer. What is the spread of true Sharpe ratios among the strategies people actually test? Not among the ones that get published, but among the ones that get tried. Most are exactly zero, and the survivors, after costs, mostly sit under 0.5. So a prior centred at zero with spread τ ≈ 0.3 is not pessimism. It is an inventory of the industry, and it is a modelling choice you should be ready to defend.

Run the shrinkage factor. With τ² = 0.09 and SE² = 1/5 = 0.2, you keep 0.09/0.29 = 31%, so a reported 1.5 becomes 0.47. That is the Sharpe haircut, and it is the single most useful number in this chapter.

Be candid about how much of that answer the prior is carrying. At τ = 0.2 the reported 1.5 becomes 0.25. At τ = 0.3 it becomes 0.47. At τ = 0.5 it becomes 0.83. The shape of the answer is stable and the size of it is yours to argue for, which is exactly what writing a prior down is for.

Now run the case that settles the verdict-versus-estimate argument beyond appeal, using the same 200-candidate pool. Strategy #184 has a t-statistic of 4.73 and clears Bonferroni's bar of 3.66. The verdict machine says yes, and says it convincingly. Then shrink it. With ten years of data and τ = 0.3, the factor is 0.474, and the reported 1.496 becomes 0.709 against a truth of 0.80.

Look at what that just did. The raw report missed the truth by 0.70, and the shrunk estimate missed it by 0.09. Across all eleven real strategies the run discovered, the mean squared error falls from 0.1522 to 0.0766, which is Chapter 16's cheque cashed in full: a deliberately biased estimator, measurably better.

The frequentist route reaches the same neighbourhood by a different road. The published factor literature contains several hundred candidates, so a family that large has a multiplicity-adjusted hurdle nearer t = 3.0 than the familiar 2.0, and the believable part of any reported result scales down accordingly. The two routes do not agree to two decimal places, and they should not, because one is priced by m and the other by τ. What they agree on is that a headline 1.5 does not survive contact with either.

Read the two levers before leaving. The haircut grows with m, which is how hard you searched, and shrinks with T, which is how much data you brought. Those are the only two dials, and one of them costs nothing but patience.

12The number nobody can audit

Close on the thing that makes all of this harder than it looks.

Every correction in this chapter needs m. Bonferroni divides by it, BH ranks against it, and the prior is set by it. So what is your m?

Count honestly. The eight lookback windows in your final grid. The two ways you handled the overnight gap. The four universes you screened before settling on this one. The dozen variants you tried on Monday and abandoned. And the signal you shelved in March and quietly retried in July.

Tallied properly, one ordinary research week reaches 81 trials, for a project you would describe to a colleague as I tested one idea.

And then there is the part nobody counts at all. More than three hundred factors have been documented in the published finance literature, and a large share of them fail to replicate. You are choosing your novel idea from among the survivors of that search. So your m inherits a search you did not run, cannot audit, and were never shown.

Say the consequence plainly, because pretending otherwise is how corrections become theatre. The family is not a fact about the world. It is a judgement about which tests belong together, two honest people will draw the boundary in different places, and they will get different verdicts from the same data. Multiple-testing corrections are a negotiation, not a calculation, and their honesty rests on a number only you know.

So what survives? Two things, and only two.

The first is pre-registration, which Chapter 17 already named. Commit to the test before you look, and m is one because you made it one.

The second is fresh data, a stretch of history the search never touched. That one is the escape hatch, and the mechanism is worth stating exactly. The luck that pushed your winner to the top of the sorted list was a property of that sample. It is not a property of the strategy, so it does not travel. Point the winner at data the search never saw and the luck does not come along. Only the edge does, if there was one.

You do not need to know m. You do not need to justify τ. You need data your search never touched.

Look back at what you are holding, because it is one idea rather than seven. Twenty tests at 5% manufacture one discovery out of nothing, because α was always a rate. Fourteen tests pass a coin flip, because at least one is not the same question as one on average. The winner of a thousand-run sweep carries a t near 3.2 out of pure noise, and its measured edge is inflated even when the edge is real. Bonferroni raises the bar by m, BH measures how far the sag extends and raises it adaptively, and the Bayesian writes the prior down and pays on the estimate instead. Three procedures, one quantity, one bill.

Chapter 20 does not make this easier. It stops testing correlations one at a time and regresses the target on every predictor at once, which multiplies the number of coefficients you can go fishing in and hands you an R² that always rises when you add another. Then Chapter 21 gives the escape hatch its proper name and its discipline, because the only correction that charges the right price without counting the trials is data the search never touched.