17Hypothesis Testing, Honestly

Chapter 16 handed you one sentence and then walked away from its consequence. Your estimate is a single dot in a cloud you will never observe. That is a beautiful thing to know and a useless thing to hold, because a dot on its own means nothing at all. A dot can only be judged against a cloud. So the whole of this chapter grows out of one forced question: which cloud am I judging it against, and how far out on that cloud did my dot land? Everything you have heard about this subject is bookkeeping on that single picture. The p-value is a tail of it. α is a line drawn on it before you looked. The two errors are areas either side of that line, power is the same line measured on a second cloud, and a confidence interval is the same picture read backwards. You will build every one of those from a coin and 1,024 counted outcomes. You will also learn the one thing this machine can never hand you, and why that is a fact about its wiring rather than a warning about your manners. And near the end, twenty strategies made of pure noise will hand us a discovery with a Sharpe of 2.07 — honestly, with no fraud anywhere.

Look at what this page stands on. Chapter 9 gave us the sample space and favourable-over-total. Chapter 10 gave us Bayes, and the hard lesson that P(A|B) is not P(B|A). Chapter 12 gave us the Binomial pmf and the Normal curve. Chapter 13 gave us the √n law and the central limit theorem. And Chapter 16 gave us the object this chapter is about to point at something: the sampling distribution, plus a debt it flagged and did not pay.

That is the gap. Chapter 16 built the cloud and left it hanging in space with nothing to compare against, and this chapter finally gives it a job.

01The verdict you cannot give

Here is a coin. I flip it ten times and it comes up heads eight times. Now tell me whether that coin is loaded.

Most people answer immediately, and the answer is some version of "probably, yes — eight out of ten feels like a lot." Sit with that sentence for a second, because it is not reasoning. It is a feeling about a number, and you have no benchmark to attach the feeling to. A lot compared to what?

Now the same problem in the clothes you will actually meet it in. A signal averages +0.08% a day over 60 trading days. Compounded, that is roughly +20% a year, which is a career. Chapter 16 already told you that this number is one dot in a cloud. So the honest question is not "is +0.08% good?" It is whether a world with nothing going on at all could have thrown a dot out this far.

Notice what you just did with the slider. You could not deliver a verdict, and you could feel precisely which number was missing. You wanted to know how often a fair coin gives eight or more heads in ten flips. Nobody has told you that number yet, and until somebody does, the eight is just an eight.

This reframes what a test is for, and the reframing is the part most courses skip. A test is not machinery for confirming your claim. It is machinery for measuring the one rival explanation that is always sitting there, for free, in every dataset anyone has ever collected. That rival is luck. A result is not evidence until luck has been measured and found insufficient.

So the first job is to say which cloud we are judging the dot against. That requirement, and no convention anywhere, is what forces everything that follows.

02The hypothesis that computes

To draw a cloud you need a world with actual numbers in it. Not a story, not a direction, not a hunch. A fully specified model that a formula can eat.

We have two candidate worlds in front of us. One is interesting: the coin is biased. One is boring: the coin is fair. Every instinct says to assume the interesting one, because that is the thing we care about. So let us genuinely try it, and let us try it with the tool we already own.

Chapter 12 gave us the Binomial pmf, C(n,k)·pk(1−p)n−k. Feed it the interesting hypothesis and watch what the machine does.

The machine stalls. Not because assuming your own hypothesis is forbidden by some rule of statistical etiquette, but because the pmf is a function of a number p, and "biased" does not supply one. Biased to 0.8, or biased to 0.9? Those give different answers, and nothing in the hypothesis chooses between them. H₁, the alternative, is a whole family of worlds wearing one name.

Now feed the same formula the boring hypothesis instead. p = 0.5 is a single number, so the pmf runs straight through and returns everything: every outcome, with its exact probability. In particular P(8 heads | p = 0.5) = 45/1024 = 0.0439.

That is the whole derivation, and it is worth saying flatly. H₀, the null hypothesis, is not the thing you believe. It is the thing you can compute with. You test the boring hypothesis because it is the only statement precise enough to hand you a complete distribution of results, and the interesting one is not a statement at all yet. It is a family.

Everything lopsided about this subject comes out of that one asymmetry. You can reject H₀ but never accept it. The logic runs backwards, ruining the rival explanation instead of proving your own. And there is a third consequence that most readers never get told, so look at it directly now, before it can hurt you.

Look at where H₀ sits in P(8 heads | p = 0.5). It is on the right of the conditioning bar, and we are the ones who put it there. Everything downstream is computed inside the world where H₀ holds, which means the thing on the left of that bar can only ever be the data. So every number this machine will ever emit has the shape P(something | H₀).

Which tells you something sharper than the usual warning. P(H₀ | data) is not a misreading of the p-value. It is a quantity that was never inside the machine. To get it you would have to condition on the data instead, and the moment you try that, Chapter 10 tells you a prior is required. The test never asked you for one. We will price that gap exactly, a little later on this page.

03Drawing the cloud

One probability is not a cloud, so do the same computation for every outcome rather than only the one we happened to see.

There are ten flips, each with two faces, so there are 210 = 1024 possible sequences. Under p = 0.5 they are all equally likely, which means Chapter 9's favourable-over-total applies directly. Collapse each sequence to its head count and the 1,024 sequences pile into eleven bars.

The counts are 1, 10, 45, 120, 210, 252, 210, 120, 45, 10, 1, and they sum to 1,024 exactly. Nothing was approximated and nothing was simulated. We counted. That is why C(10,8) = 45 can be pointed at on screen as a pile of actual sequences rather than a symbol you take on trust.

Keep two levels straight here, because one word is doing two jobs. The strip underneath the bars is the data: individual flips, HTHHTH and so on. The bars above are the test statistic, which is the one number the whole sample gets collapsed to. Here that number is the head count. The histogram describes the statistic, never the flips.

That histogram has a name, and it is the object the rest of the chapter reads off. It is the null distribution — the sampling distribution of the test statistic in the world where H₀ is true. Chapter 16 built exactly this kind of object for the sample mean. The only new thing is that a specific claim now owns it.

Mark 8 on its axis, and the number we collected finally has a cloud to be judged against. It is a cloud we built by counting rather than by trusting anybody's formula.

04How far out is far?

Now read the surprise off the picture. The obvious move is to quote the height of the bar we landed on: P(exactly 8) = 0.0439. Under 5%. Case closed?

No, and the reason is worth meeting head-on rather than being told. Look at the tallest bar in the whole picture. P(exactly 5 heads) = 252/1024 = 0.2461. That is the most likely single outcome there is, and it still only happens about a quarter of the time. On the point-probability rule, five heads is "somewhat surprising" too. Every outcome is individually unlikely, and for a continuous statistic like a mean return it is worse than unlikely — the probability of landing on exactly your value is zero, always.

So point probabilities cannot measure surprise at all. What can measure it is area: how much of the null's probability sits at or beyond where you landed. That is Chapter 3's definite integral doing inference work, and it is the only quantity on this picture that grows as your result gets more extreme.

Before the shading appears, commit to a guess. How often does a genuinely fair coin give eight or more heads in ten flips? Almost everyone says something like "rare — maybe one in fifty."

It is (45 + 10 + 1)/1024 = 56/1024 = 5.5%. Counting the other tail as well, because a coin biased toward tails would be just as interesting, it is 112/1024 = 10.9%. So the result that opened this chapter, the one that felt obviously loaded, is a thing a fair coin does about one time in nine.

Your gut was out by roughly an order of magnitude, and mine was too the first time, which is exactly the useful part. Everything after this you will believe a little less quickly.

That shaded area is the p-value, and here is the sentence to memorise exactly: it is the probability, in the world where H₀ holds, of a test statistic at least as extreme as the one you actually observed. The awkward clause is not bookkeeping, and it is the thing that makes the whole quantity mean anything at all.

There is one more decision hiding in the word "extreme", and it costs people real money. Extreme in which direction?

Take a different result: twelve heads in sixteen flips. Counted the same way, the upper tail alone gives p = 0.0384, which clears a 5% line. Both tails together give p = 0.0768, which does not. Same coin, same data, opposite verdicts, and the only difference is which question you asked.

So the tail is fixed by the question, before the data arrives. If you genuinely only care about one direction — a strategy is only tradable if it makes money — say so first and take the one-tailed test. If you would have been interested either way, you owe both tails. Choosing after you have seen which way the data went is not a judgement call. It is a free extra shot that you did not pay for.

05The bar points the other way

Let me ask you the question every practitioner gets asked in a meeting. Your backtest returns p = 0.03. What is the probability that this strategy has no edge?

The instinctive answer is 3%, and it is wrong in a way that is worth being precise about. The p-value answers "how weird is this data, if nothing is going on." The question asked "how likely is it that nothing is going on." Those are the two sides of Chapter 10's conditioning bar, and we already saw which side H₀ was nailed to.

The gut resists this, and it resists for a good reason. The wrong argument looks like clean logic: if H₀ then this data is unlikely, the data happened, so H₀ is unlikely. With plain true-or-false statements that argument is valid. With probabilities it smuggles something in, and the gut cannot see the smuggling. So let us not argue. Let us count.

Here is your strategy pipeline for the year. You test 1,000 candidates, and suppose 100 of them genuinely have an edge. That is your prior, expressed as a count instead of a probability. You test every one of them at the 5% line, and your test catches 80% of the real ones.

Now count the two piles that come out of that year. The real edges give you 80 true finds. The 900 duds give you 0.05 × 900 = 45 false alarms. So you end the year holding 125 things you called discoveries, and 45/125 = 36% of them are noise — even though every single one had a p-value under 0.05.

Now drop the catch rate to 30%, which is a realistic number for a one-year backtest. You get 30 true finds against the same 45 false alarms, and now 60% of your discoveries are noise. The p-value never moved, because it sat at 0.03 the whole time.

This is exactly Chapter 10's medical-test table wearing a trading firm's clothes, and it is the same trap: the base rate is invisible because nobody counts the strategies they tested and threw away. The 900 duds never appear in anyone's mental picture, and they are nonetheless sitting there in the arithmetic.

And read what that table actually says about the p-value. Bayes gives P(H₀ | data) = P(data | H₀) · P(H₀) / P(data). The p-value is built from the first factor only. The other two are the prior and the total, and the test refuses to know either of them. So the machine supplied one of three ingredients. It could not possibly have returned the answer.

One phrase is doing damage here too, so let us kill it plainly. "Significant at 5%" does not mean "a 5% chance I am wrong". It is not the false-discovery rate either. That third thing, the one everyone thinks they are saying, does not exist as a number the test produces.

06Where to draw the line

A p-value is a continuous measure of surprise. But at some point you have to either allocate capital to this strategy or shelve it, and a decision needs a line.

So draw one on the null curve. Say the outermost 5% of it, and commit to that before you look at your data. That line is α, the significance level, and the value of the statistic where it falls is the critical value. Cross it and the verdict is reject H₀. Fall short and the verdict is fail to reject H₀. Those are the only two verdicts there are, and the second one is deliberately not called "accept".

Now the part that turns α from an arbitrary ritual into a machine you are holding. Take a thousand strategies that are genuinely worthless — an edge of exactly zero, built that way on purpose — and run the test on every one.

Fifty-six of them cleared the 5% line. That is what α is. It is not a property of your data and it is not the chance you are wrong. It is the fraction of innocent nulls you agreed, in advance, to accuse. Tighten to α = 0.01 and the count falls to 13, which is the same dial turned down.

Then look at the loudest one in that run, because it is the most useful number on this page. The best of those thousand worthless strategies averaged +0.25% a day over its 60 days, which annualises to a Sharpe ratio of 7.03, with a p-value of 0.0011. Nothing was rigged and there is no bug. We looked a thousand times, and looking a thousand times is what produced it.

The famous objection now answers itself without any help from us. Is p = 0.049 really different from p = 0.051? Obviously not, and the cliff is an artefact of forcing a continuous measure through a binary decision. The fix is not to abandon the line. It is to report the p-value itself, and treat α as where you have decided to act rather than where truth changes.

One honesty note, because our coin cannot actually do what those strategies did. With ten flips the statistic is discrete, so a 5% line has nowhere clean to land. The reject region k ≤ 1 or k ≥ 9 costs 2.1%, the next region out costs 10.9%, and there is nothing available at 5%. A discrete test is conservative: it spends less α than it advertises. That is why the thousand-strategy run above used a continuous statistic instead of a coin.

07The two ways to be wrong

Once a line exists, there are exactly two ways to be wrong. They live in different worlds, and that is why they need different names.

Start with a smoke alarm, because the world already named these for us. The alarm goes off while you are making toast. Nothing is burning, the alarm fired anyway, and that is a false alarm. Now the other one: a fire starts in the night and the alarm stays silent. Something is burning, the alarm said nothing, and that is a miss.

Both errors have price tags in finance, and they are not the same size. A false alarm means you allocate capital to a strategy with no edge, then bleed it away in fees and spread. It hurts, it is measurable, and it is survivable. A miss means you shelve the one real edge you found all year, and nobody ever writes that loss down anywhere.

Swap the roles and the asymmetry flips completely. A risk model's job is to see the fire. Its false alarm costs a little unnecessary hedging. Its miss is the one that ends the firm. So the two errors are not symmetric, and the ratio of their costs is what should choose α for you. Five per cent is a convention, not a law of nature.

Only now do the numerals earn their place. The false alarm is a Type I error, and it lives on the null curve, past the line. The miss is a Type II error, and its rate gets the letter β.

Here is the piece that is usually handed over as a fact to memorise. The Type I error rate is α, by construction. It is not a second fact and it is not a coincidence. α was defined as the outermost 5% of the null curve, and a false alarm is exactly the event of landing out there when H₀ is true. That is one line described twice.

Which raises the obvious question about the other one. Where on this picture is β?

08The second curve

Go and look for β on the null curve. It is not there, and that absence is the tell.

A miss is failing to cross the line when something real is going on. The null curve is the world where nothing is going on, so it cannot possibly contain that event. β belongs to a different world, which means it is an area under a second curve — the sampling distribution of the same statistic when a specific alternative is true.

This is the rung almost every course skips, and skipping it is why so many people can recite the definition of β and cannot locate it on any picture. So let us draw the thing.

Two curves, one critical line running through both of them, and suddenly all four cells of the world-by-verdict table are visible at once as four shaded regions. The area past the line under the alternative curve is the catch rate we were already using without a name. It gets one now: power, written 1 − β.

Notice what drawing that second curve just cost you. To place it on the axis at all, you had to name a specific effect size — not "the strategy works", but "the strategy makes 0.05% a day". H₁ was allowed to stay vague while you were testing. It has to become a number the moment you want to know whether your test could have seen anything.

That looks like the rules just changed, and it is worth saying out loud that they did not. Vague is fine for testing, because testing only ever computes inside H₀'s world. Vague is fatal for planning, because planning asks a question about H₁'s world and you cannot draw a curve without a centre for it.

Now take the three levers, and watch each one with your eyes rather than reading a rule. Pull the effect size up and the curves separate, so the overlap shrinks and power climbs. Raise n and both curves narrow at once, which is Chapter 16's SE = σ/√n doing visible work on two objects simultaneously. Slide α and something less comfortable happens: the two shaded errors trade against each other. You can shrink either one on demand. You cannot shrink both without paying in n.

Then the number that should change how you read every result you ever see. Take a genuinely real strategy with an annualised Sharpe of 0.5, and test it on one year of daily data at the usual 5% line. Its power is 7.9%.

Read that again, because it is not a rounding issue. The edge is real, and the test will fail to find it about 92 times in 100. At a Sharpe of 1.0 the power is still only 17%. So a non-significant result from a test like that carries almost no information, and writing it up as "no edge" is reporting a fact about your sample size as though it were a fact about the world.

09Measuring in wobbles

The coin was countable, and a mean return is not. Nobody is going to enumerate sixty days of prices into a grid of equally likely cases.

We do not need to. Chapters 13 and 16 already handed over the cloud for an average: it is Normal, centred on the true mean, with width SE = σ/√n. So put H₀'s claim into that cloud. The claim is no edge, which pins the centre at μ₀ = 0. That number gets a name, the null value, because in general it is whatever value H₀ nails the parameter to.

Now say our result out loud, twice. First: the signal averaged +0.08% a day. That sentence means nothing yet. Second: the day-to-day wobble of that 60-day average is 0.6/√60 = 0.0775%. Now the number says itself.

The signal is one wobble. That is it, and that is the whole measurement.

The count of wobbles is the z-score, z = (x̄ − μ₀)/SE, and here it is 0.08/0.0775 = 1.03. Standardizing is a change of units and nothing more exotic than that. Subtract the centre, divide by the width, and you have converted per-cent-a-day into typical-wobbles. It is the same species of act as converting inches to metres.

What you get for it is enormous, though. Every test of this shape now lands on one curve, the standard Normal, so the α-lines sit at fixed places forever: 1.645 for a one-tailed 5% test, 1.96 for two-tailed 5%, 2.576 for two-tailed 1%. A "table lookup" is just reading an area off the picture you have been staring at since the coin.

One warning, and it is the most expensive routine error in applied work. The denominator is SE, the spread of the statistic. It is not σ, the spread of the data. Chapter 16 spent a whole section separating those two rulers because they are printed in identical units. Divide 0.08 by 0.6 instead of by 0.0775 and you get 0.13 instead of 1.03, which turns a discovery into a nothing or a nothing into a discovery depending on which way you slipped.

And now cash out what z = 1.03 means for our strategy, because the answer is genuinely uncomfortable. Two-sided, that is p = 0.30. Meanwhile the same numbers give a daily Sharpe of 0.08/0.6 = 0.133, which annualises to 2.12. A strategy with an annualised Sharpe above 2, and sixty days still cannot distinguish it from zero.

There is a lie in all of that, and Chapter 16 told us it was coming. SE = σ/√n contains σ, and nobody ever had σ. What we actually have is Bessel's s, computed from the same sample. Substitute it and the quantity changes character in a way that matters.

Before we look at the answer, commit to a guess. At n = 5, how far does using s instead of σ move the 5% line? Most people say a few per cent.

The numerator was already random, and now the denominator is random too. A ratio with a wobbly denominator occasionally comes out enormous purely because the denominator happened to land small. That is a fatter-tailed object, and it has a name: Student's t-distribution, with n − 1 degrees of freedom.

That n − 1 is not a new coincidence. It is Chapter 16's residual freedom, the same object doing the same job — the residuals must sum to zero, so only n − 1 of them can move.

Now read the measured numbers off that run. At n = 5 the two-sided 5% line is not 1.96 but 2.7764, which is 42% further out. Run the test at 1.96 anyway and you reject a true null 12.2% of the time instead of 5%. Then let n climb and watch the two histograms merge: at n = 30 the line is 2.045, at n = 100 it is 1.984.

So "use t when n is under 30" is a consequence, not a definition, and you can now retire it. The cause is that you estimated the denominator. Also note what t is not: it is not a distribution for a different kind of data, and our data here was perfectly Normal. t is the law of a ratio.

10One fraction, every test

Step back and look at what we actually built. It is one fraction.

t = (estimate − null value) / SE(estimate). Read it as signal over noise: how many typical wobbles separate what you measured from what the boring world claims. That is the entire apparatus.

Every named test you will ever meet is that one row with a different SE dropped into the denominator. One mean, a difference of two means, a regression slope in Chapter 20, a Sharpe ratio in Chapter 30. The estimate column changes, the null-value column is almost always zero, and the SE column is where all the actual work lives.

Point it at a track record and Chapter 16's arithmetic falls straight out: t = SR·√T. Check it against our own signal, where an annualised Sharpe of 2.117 over T = 60/252 = 0.238 years gives 2.117 × 0.488 = 1.033, which is the same z we computed from wobbles, to three decimals. Two completely different routes, one number.

That reading also prices a career. Reaching |t| ≥ 2 takes 4 years at a Sharpe of 1.0, 16 years at a Sharpe of 0.5, and forty-four years at a Sharpe of 0.3.

So when you meet a test you have never seen, do not go looking for its name. Ask three questions: what is being estimated, what number does the null pin it to, and how wide is that estimate's cloud. Answer those and you have the test, whatever it is called.

Now use the fraction backwards, which is where confidence intervals actually come from. So far we have interrogated one candidate value at a time: is μ₀ = 0 survivable? Nothing stops us from asking it of every value on the axis.

Sweep the null value across, and for each one ask whether the test would reject. The values that survive form an interval. That interval is the confidence interval, so it is not a new formula at all. Solve |θ̂ − θ₀| ≤ t*·SE for θ₀ and you get θ̂ ± t*·SE directly.

Now the coverage statement, which is the single most misread sentence in statistics. People say "there is a 95% chance the true value lies in this interval", and that is not carelessness. It is a genuine confusion about which object is random, and both readings sound identical in English.

The picture settles it. θ is a fixed number and it never moves — it is the vertical line that holds still through all hundred rows. The interval is the random thing, redrawn with every sample. About 95 of the hundred bars cross the line and about five do not, and those five are not defective, because they were simply unlucky draws. Once your interval is on screen, nothing in it is random any more. The randomness was spent when you drew the sample.

And the interval and the p-value were never competitors. They are two readings of one picture.

Drag the hypothesized value across the axis and watch the p-value rise through α at exactly the moment the value crosses the interval's edge. That is what duality means here. The interval is the set of null values that would not be rejected, which is why it carries more information than a p-value: it reports the size of the effect, not only the surprise.

Let us cash the whole template out on the test a quant actually runs most weeks. Two order-routing algorithms, 1,000 orders each: algorithm A fills 540 of them and algorithm B fills 500.

Fill in the template's row. The estimate is p̂₁ − p̂₂ = 0.04 and the null value is 0. The SE comes from Chapter 12's Bernoulli variance p(1−p), divided by n by Chapter 13's √n law, and added across two independent arms.

One detail in that SE is stated in every textbook and justified in almost none. You compute it with a pooled fill rate, p̂ = 1040/2000 = 0.52, rather than with the two separate rates. Here is why, and it comes straight from the keystone. We are drawing the cloud that H₀ generates, and in H₀'s world there is exactly one true fill rate. Estimating two of them would be estimating something that world says does not exist.

That gives SE = 0.0223, so z = 0.04/0.0223 = 1.79 and the two-sided p = 0.073. Not significant. Most memos stop there and write "the algorithms are the same," which is a claim the data cannot support.

So make the move almost nobody makes, and compute the power. Against a true four-point gap, this test had a 43% chance of detecting it. It was a coin flip whether a real difference would show up at all. To get to 80% power you needed about 2,450 orders per arm, not 1,000.

Notice what just happened. The chapter's own machinery convicted the experiment, not the algorithms. That distinction is roughly the whole difference between a practitioner and a person quoting a number.

11What the machine can never do

Three closing honesties, and each one is a habit rather than a formula.

The first honesty: you never accept H₀. Failing to reject is failing to see, which is not the same as seeing nothing. We just watched a real four-point gap go undetected by an experiment with 43% power, and we watched a real Sharpe-0.5 edge survive a whole year of testing unnoticed 92 times in 100. Absence of evidence is not evidence of absence, and now you can put a number on the difference.

The second: significance is not size. These are two different words and they must stay two different words.

Here is an edge of 0.001% a trade — a tenth of a basis point — measured across a million trades. The standard error collapses to 0.0002%, so t = 5.00 and p = 0.00000057. That is about as significant as anything in finance ever gets. And the 95% interval runs from 0.00061% to 0.00139% a trade, while the round-trip cost is 0.002%. The entire interval sits below the cost.

So the edge is real, overwhelmingly significant, and loses money on every single trade. Large n manufactures significance without manufacturing anything worth having.

The third honesty is the one that opens the next chapter, and it is arithmetic rather than a warning. When H₀ is genuinely true, the p-value is uniform on [0,1]. Every value is equally likely, by construction, because p is the tail area at your own randomly-placed statistic.

Ten thousand true nulls, and the histogram of their p-values is a flat rectangle — decile counts 950, 1007, 1026, 982, 1005, 992, 1018, 1039, 958, 1023. Read the consequence straight off that rectangle. Five per cent of a flat rectangle is five per cent, no matter how many times you draw from it.

So twenty honest tests on pure noise produce, on average, one result under 0.05. The chance of at least one is 1 − 0.9520 = 64%. We ran exactly that: twenty strategies, every one of them made of noise with an edge of exactly zero. One of the twenty came back at p = 0.0393. Traded in the direction the test pointed, it returns +20.2% for the year, with an annualised Sharpe of 2.07 and a maximum drawdown of 4.2%.

There is no fraud in that, no bug, and nothing anyone would call p-hacking, because it is simply what twenty looks does. Which means a p-value is a property of your result and of the search that produced it, and reporting one without the other is reporting half a number.

The fixes that are actually in reach are small and boring. Pre-register the hypothesis, the tail and α before you look. Report intervals and effect sizes rather than verdicts. Count and disclose the looks. Chapter 19 is where the looks get priced properly, with Bonferroni and false-discovery-rate control, and where the prior this chapter kept refusing to supply finally gets handed over.

Look back at what you are holding. It is one picture: the cloud a boring world would generate. The p-value is its tail. α is a line drawn on it before the data. Type I error is the area past that line, power is the same line measured on a second cloud, and a confidence interval is the set of boring worlds that would have tolerated your data. Whether it is z or t is only the question of which cloud exactly.

That is one object with six readings, in place of six rules with no object. And it has a permanently missing seventh reading, which is the probability that the boring world was true. Chapter 18 takes the same machinery and points it at two variables instead of one, where a new and much more seductive error is waiting.