Skip to content
QDA2 Lab

MODULE 04

p-values

The p-value measures how unusual the observed result would be if the were true. Nothing more, and nothing less.
  1. Research question
  2. Variables
  3. Variable roles
  4. Measurement levels
  5. Method choice
  6. Statistical logic (trained here)
  7. SPSS workflow
  8. Output interpretation (trained here)
  9. Defensible conclusion (trained here)

Understand

Level 1: intuition

Imagine H₀ is true and we drew thousands of samples. The curve shows which test statistics we would typically get. The p-value is the share of those imaginary results that are at least as extreme as ours.

H₀: μ = 60. Sample of n = 25, sample mean 64. Where does the result fall if H₀ were true?

-4-3-2-101234t statistic under H₀ (t distribution, df = 24)α = .05 cut-offα = .05 cut-offobserved t = 2.20−2.20
2.20

Move the observed result further from what H₀ predicts (0) and watch the shaded tail area shrink.

Shaded area = p-value

p = .038

p < .05: reject H₀ at α = .05

If μ really were 60, results at least this far from 60 would be rare. The data are hard to reconcile with H₀.

Two-tailed: extreme in either direction counts, so both tails are shaded. The dashed line marks the boundary.

The p-value is

the probability of a result at least as extreme as the observed one, if the null hypothesis were true.

The p-value is not

  • The probability that H₀ is truep is calculated assuming H₀ is true, so it cannot also measure how likely H₀ is.
  • The probability that the result is due to chancep describes how unusual the data would be under H₀, not the probability that chance ‘caused’ them.
  • A measure of effect sizeA tiny effect can give a tiny p in a huge sample. Use Cramer's V, r or Cohen's d for magnitude.
  • Proof that H₀ is true when p > .05Not rejecting H₀ means the evidence is insufficient, not that there is no effect.

Try

Same effect, different sample size

The p-value depends on two things: how big the difference is and how precisely it is measured. Precision grows with N.
The same substantive effect every time: sample mean 64 vs H₀ μ = 60, standard deviation 20. Only the sample size changes.
45505560657075Sample means we would expect if H₀ were true (sampling distribution)H₀: μ = 60observed mean 64
N = 20

Larger N → narrower sampling distribution → the same 4-point difference sits further out in the tail.

Difference
4.0 points
unchanged
4.47
s / √N
t
0.89
difference / SE
p (two-tailed)
= .382
not significant

Statistical significance ≠ substantive importance

Significance tells you whether a result is distinguishable from H₀ given your sample size. Whether the difference matters is a separate, substantive question answered by effect sizes and subject knowledge.

Study A

N = 50,000 · p < .001

tiny effect, huge N

0.4 points

Mean difference on a 0–100 scale (bar spans 0–10 points)

Study B

N = 40 · p = .08

larger effect, small N

6 points

Mean difference on a 0–100 scale (bar spans 0–10 points)

Interpret p

Study A: mean difference 0.4 points on a 0–100 scale, N = 50,000, p < .001. Study B: mean difference 6 points, N = 40, p = .08. Which study found the larger effect?

Understand

Level 2: show me the math

Optional, but it makes the sample-size effect obvious.

SPSS

Where the p-value appears in SPSS

SPSS never prints the letter p. It prints ‘Sig.’, and it rounds.

Before you open SPSS

  1. 1What is the null hypothesis?State it before reading any output, e.g. μ = 60 or ‘no association’.
  2. 2What is α?Usually .05, decided in advance.
  3. 3One- or two-tailed?SPSS reports two-tailed p by default: Sig. (2-tailed).
One-Sample Test (Test Value = 60)
tdfSig. (2-tailed)Mean Difference
Interview length (min)2.0624.0504.00
  1. 1The test statistic: the difference measured in standard errors.
  2. 2The p-value. ‘.000’ in SPSS means p < .001; never report p = 0.
  3. 3The effect in original units (minutes). This, not Sig., tells you how big the difference is.

Reporting

“The mean interview length (M = 64.0, SD = 9.7) did not differ significantly from 60 minutes, t(24) = 2.06, p = .050.” SPSS rounds here: for t = 2.06 with df = 24, the exact two-sided p is about .0504, so it is not below α = .05. Report the value and avoid treating .049 and .051 as different worlds.

Test yourself

Mastery check

Mastery check

0/0

Interpret p

A one-sample t-test gives p = .03. Which statement is correct?

1 / 4