Hypothesis Testing

Connect the research idea with a population quantity

Hypothesis testing uses sample evidence to examine a specified claim about a population or statistical model. It asks how the observed result compares with the variation expected under that claim.

A research hypothesis expresses an expectation about the world. A statistical hypothesis translates that expectation into a statement about a quantity such as a population mean, a difference between means, a proportion, a correlation, or a regression coefficient.

Define the population, measurement, comparison, and period before selecting a test. “Media use” might mean minutes per day, frequency categories, or a multi-item score. Each definition points to a different analysis and interpretation.

A parameter describes a population or model: μ for a population mean, for example. A statistic is calculated from sample data: x̄ for a sample mean. Hypotheses usually concern the parameter; the statistic supplies evidence about it.

Distinguish the spread of scores from the spread of estimates

Individual observations vary, and estimates calculated from samples vary too. Hypothesis testing uses a model of the second kind of variation to assess a sample result.

Standard deviation describes observations

The standard deviation (SD) describes how widely scores spread around their mean. For a normal population, about 68.27% of values fall within one SD of the mean, 95.45% within two, and 99.73% within three. The central 95% lies approximately between μ − 1.96σ and μ + 1.96σ. Here σ denotes the population SD. See the NIST normal-distribution guide.

These percentages describe the normal model. A finite sample will show some variation around them. Examine the distribution relevant to your analysis rather than applying the rule automatically to every dataset.

Standard error describes an estimator

Imagine repeatedly drawing samples of the same size using the same design and calculating a mean from each. Those means form a sampling distribution. The standard deviation of that distribution is the standard error of the mean.

For independent observations with a finite population variance, averaging more observations generally reduces the sampling variation of the mean. Under suitable conditions, the central limit theorem explains why sampling distributions of means can become approximately normal as sample size grows. The approximation depends on the population’s shape and other conditions.

A standardised individual score uses z = (x − μ) ÷ σ. A test of a mean instead compares the sample mean with a null value and scales that difference by its standard error. This distinction explains why a small difference can become more precisely estimated with a larger sample.

Write the null and alternative hypotheses

The null hypothesis, H₀, specifies the reference claim. The alternative, H₁ or H_a, states the departure the test is designed to detect.

A null hypothesis often states equality or zero association. It can also specify a non-zero benchmark or a boundary. Write hypotheses in terms of population quantities and give those quantities a clear meaning.

Choose a direction that matches the research question

On a small screen, swipe across the table.

For these usual one-sided tests, the reference calculation uses the null boundary, μ = 50. A two-sided test of a signed statistic divides its rejection probability across two tails; a one-sided test places it in the specified direction.

Under a standard normal reference distribution and α = .05, the two-sided critical values are approximately −1.96 and +1.96. An upper one-sided test uses +1.645; a lower one-sided test uses −1.645. A t-test uses values from the relevant t distribution, with its degrees of freedom. The NIST critical-value guide shows how to read these cut-offs.

The shape of the reference test also matters. For example, a chi-square test of association usually rejects for large positive statistics even though its research question concerns any departure from independence.

Follow a planned testing process

A hypothesis test connects a research question, an analysis model, a sample result, and an interpretation. Decide how these parts will fit together before using the result to draw a conclusion.

1. Specify the question and parameter.Define the population, measurements, and comparison.
2. State H₀ and H₁.Choose the null value and test direction.
3. Choose the analysis and decision rule.Set α, the sampling or model assumptions, and any multiple-testing strategy.
4. Examine the data and design.Check coding, missingness, dependence, distributions, and influential observations.
5. Calculate the result.Obtain the estimate, SE, test statistic, p-value, and a suitable interval.
6. Interpret and report.State the decision, effect size, uncertainty, and scope of the evidence.

Understand the test statistic

Many tests of a mean, difference, or coefficient have a common form:

If the sample mean is three estimated standard errors above the null mean, a one-sample t statistic is +3. The reference distribution tells us how frequently a statistic that extreme would occur under H₀ and the model assumptions.

The reference distribution depends on the test. A z-test uses a standard normal distribution under appropriate conditions. A t-test estimates the population SD and uses a t distribution. Its degrees of freedom affect the tails and critical values.

Interpret the p-value and the significance level

A p-value measures how unusual the observed test statistic is under the specified null model. Its calculation begins by assuming H₀ and the other conditions of that model.

More precisely, it is the probability, under that model, of a test statistic at least as extreme as the one observed, with “extreme” defined by the alternative and the test procedure.

Alpha sets the decision threshold

The significance level, α, specifies the intended Type I error control for a testing procedure. A common choice is .05. Select the level through the research context and consequences of error, then use it consistently.

For a valid level-.05 procedure, repeated studies under a true H₀ would reject it in about 5% of cases, or at most 5% for some procedures. This is a property of the rule under stated conditions.

A critical-value calculation expresses the same decision through a rejection region. For the illustrative two-sided z-test, |z| = 2 exceeds 1.96, and p ≈ .0455 is below .05. Both routes agree when the model, direction, α, and treatment of boundaries match.

Read p-values with the study’s design, measurement quality, estimate, and uncertainty. Results just either side of .05 may describe very similar evidence; explain the numerical result instead of letting a label carry the argument.

Understand errors and statistical power

A testing decision is made with incomplete information. Type I and Type II errors describe two ways the rule can disagree with the underlying state.

Power describes a test’s ability to detect a specified departure

Power = 1 − β. It is the probability of rejecting H₀ when a particular alternative is true. A power statement therefore needs an effect size and a design. A study may have good power for a large difference and poor power for a small one.

For a fixed model and effect, increasing sample size or improving measurement precision generally increases power. With the sample size and other conditions fixed, lowering α usually reduces Type I errors while increasing the chance of Type II errors.

Plan sample size around the smallest effect that matters, the intended analysis, α, desired power, and expected data loss. If the design is already fixed, a sensitivity analysis can show which effects it is reasonably equipped to detect. Penn State’s hypothesis-testing lesson introduces these connections.

Match the analysis to the data structure

Choose a test through the parameter, measurement level, number of groups, dependence structure, and model conditions. A familiar test becomes useful when those features fit.

Parametric and nonparametric approaches

Parametric methods specify a family of probability models and work with its parameters. Nonparametric methods often use ranks or other features to compare distributions under different conditions. Choose between approaches through the research question and the assumptions each requires; explain precisely which feature of the population the analysis addresses.

Check what the assumptions actually require

Independence concerns relationships among observations. Repeated scores from the same student, students nested in a class, or linked pairs can carry dependence that the analysis must handle. Equal selection probabilities describe a sampling feature; they do not establish statistical independence.

Normality is considered at the level relevant to the model. In a paired t-test, examine the differences. In a two-group model, examine the within-group distributions or errors. Use plots and subject knowledge alongside diagnostics, especially for small samples and influential outliers.

Variance assumptions depend on the test. A pooled two-sample t-test assumes equal population variances. Welch’s test estimates the comparison without that equality assumption. A one-sample t-test has a different structure.

Measurement shapes the parameter’s meaning. Categories, ordered responses, counts, and numerical scores offer different information. A single ordered rating item needs careful treatment; justify any score construction and the analysis used.

The NIST two-sample t-test guide explains independent and paired structures and unequal-variance calculations. Use a fuller method-specific guide before applying an analysis to a real project.

A one-sample t-test from question to conclusion

This fictional example shows how the estimate, standard error, reference distribution, and decision fit together. Work through each calculation before opening the answer.

1. State the hypotheses

H₀: μ = 50   ·   H₁: μ ≠ 50

The observed mean is 3 points above the benchmark. The test asks how that difference compares with its sampling uncertainty.

2. Calculate SE and t

3. Obtain p or use the critical value

With 35 degrees of freedom, the two-sided p-value for t = 3.00 is approximately .00495. The .05 two-sided critical values are approximately −2.030 and +2.030.

The observed |t| exceeds 2.030, and p is below .05. We therefore reject H₀ under the chosen rule. The NIST one-sample guide explains this calculation.

4. Estimate the difference and its interval

Subtracting the fixed benchmark of 50 gives an estimated difference of 3.00 points, with a 95% CI of approximately [0.97, 5.03].

5. Write the conclusion

The reported p = .005 is rounded from approximately .0049488. Decisions use the unrounded result and the prespecified α.

Interpret effect sizes and confidence intervals

The estimate describes the size and direction of a result. Its interval describes uncertainty under the analysis. Together, they give the reader more to assess than a threshold decision alone.

Start with the original units

In the worked example, the estimated mean difference is 3 points on the quiz. Those units connect the statistical result with the educational question. Explain the scale and what a difference of that size could mean in practice.

A standardised effect size expresses the result relative to a measure of variation. For this one-sample example, Cohen’s d = (53 − 50) ÷ 6 = 0.50. State which definition you use, because standardisation differs across one-sample, independent-group, and paired designs. Interpret the value through the field and measurement context.

Read the interval through its procedure

A 95% confidence-interval method is designed so that, across repeated samples under its conditions, about 95% of the intervals cover the fixed population parameter. The observed interval shows the estimates compatible with the data at that confidence level under the model.

For a matching two-sided .05 test and 95% interval, excluding the null value corresponds to rejecting that value, apart from boundary and rounding details. The worked example’s mean interval excludes 50; its difference interval excludes zero. The NIST confidence-interval example shows this connection.

Make sense of a result that fails to reject H₀

Consider a second fictional sample summary with n = 36, SD = 6, and mean = 51, using the same benchmark and procedure. Now t = 1.00, df = 35, and p ≈ .324. The estimated difference is 1 point, with a 95% CI of approximately [−1.03, 3.03].

At α = .05, we fail to reject H₀. The interval includes zero and a range of positive values. A useful conclusion describes that uncertainty and considers which of those values would matter to the research question.

Plan tests and record analytical choices

A defensible analysis keeps the question, decisions, and data handling visible. Planning helps you distinguish the evidence sought in advance from patterns discovered while exploring.

Record primary outcomes, hypotheses, direction, sample-size reasoning, exclusions, missing-data handling, α, and the intended analysis before examining the relevant results. A preregistration can make that record inspectable. Document necessary changes and explain their reasons.

Account for multiple testing

Testing many claims creates more opportunities for false positives. If 20 tests were independent, every null were true, and each test had a .05 Type I error probability, the probability of at least one false rejection would be:

Define the family of claims and choose a suitable multiplicity strategy. One simple option is Bonferroni: for a family of m valid tests, compare each p-value with α ÷ m to control the familywise Type I error rate at most α. For five tests and family α = .05, the per-test threshold is .01. Bonferroni can reduce power; other approaches suit other purposes. See the NIST guide.

Keep exploration informative

Exploratory plots, alternative models, and subgroup patterns can reveal useful questions. Label them clearly, disclose the analyses considered, and treat promising findings as candidates for further examination.

Use an appropriate sequential design when results will be examined repeatedly during data collection. For an ordinary fixed-sample test, specify the stopping rule in advance and follow it. Repeated opportunities to stop after a small p-value can change the error rate.

Check the chain from data to conclusion

Verify coding and score construction, preserve an analysis record, and examine whether clustering or repeated measurements affect uncertainty. Assess the effect of justified alternative specifications. Report results that complicate the hypothesis alongside those that fit it.

Report the test as part of the research argument

A useful report lets readers reconstruct the question, inspect the evidence, and assess the interpretation. Present the test together with the estimate and the conditions that give it meaning.

• Question and design: define the population, measurements, comparison, and recruitment or assignment.
• Analysis: name the test, hypotheses, direction, α, and any multiplicity adjustment.
• Data: report sample sizes, descriptive statistics, exclusions, and missing-data handling.
• Evidence: include the statistic, degrees of freedom where relevant, p-value, effect estimate, and suitable interval.
• Interpretation: state the decision, substantive meaning, practical uncertainty, and scope.

Report a very small p-value using a suitable bound when necessary, such as p < .001. If software displays .000, obtain more precision or use an appropriate bound. A test result supports a research argument when its measurement, sampling, model, and context are explained.

The ASA guidance places p-values within a wider practice of study design, contextual interpretation, and complete reporting. Use that wider view when deciding what the research has established.

Use the lecture as a guided learning session

Watch Dr. USP’s lecture with the worked example beside you. Pause at the points below and explain each concept in your own words before continuing.

Open the lecture on YouTube. The links mark approximate starting points from the supplied transcript. Use the definitions and worked calculations in this chapter alongside the lecture.

Check your understanding

Write a response first. Then open the explanation.

Glossary and further reading

Use these terms to explain the logic without relying on software labels. The linked sources develop the definitions, calculations, and interpretation.

Null hypothesis

The reference claim about a population quantity or model used by a test.

Alternative hypothesis

The specified departure from the null that the procedure is designed to detect.

Sampling distribution

The distribution of a statistic across repeated samples under a specified design or model.

Standard error

The standard deviation of an estimator’s sampling distribution, usually estimated in practice.

Test statistic

A quantity that summarises the data for comparison with a reference distribution.

p-value

The null-model probability of a test statistic at least as extreme as observed, according to the test.

Alpha

The chosen significance level controlling Type I error under the test conditions.

Degrees of freedom

A quantity reflecting independent information relevant to a reference distribution; n − 1 in the worked one-sample t-test.

Power

The probability of rejecting H₀ at a specified true alternative, equal to 1 − β.

Effect size

A measure of the magnitude of a difference or relationship, in original or standardised units.

Confidence interval

An interval produced by a method with stated repeated-sampling coverage under its conditions.

Equivalence bounds

Prespecified limits defining effects small enough for a particular practical or theoretical purpose.

1. Pandey, U. S. Hypothesis testing by Dr. Uma Shankar Pandey [Video]. YouTube.
   https://www.youtube.com/watch?v=1yDYhMTlJ9cThe lecture and supplied transcript introduce the chapter’s core topics.
2. NIST/SEMATECH. What do we mean by “Normal” data? e-Handbook of Statistical Methods.
   https://itl.nist.gov/div898/handbook/pmc/section5/pmc51.htmThe normal model, standardisation, and empirical-rule probabilities.
3. Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133.
   https://doi.org/10.1080/00031305.2016.1154108
   ASA’s freely accessible summary of the six principlesDefinitions and guidance for interpreting p-values within a wider research argument.
4. NIST/SEMATECH. Are the data consistent with the assumed process mean?
   https://www.itl.nist.gov/div898/handbook/prc/section2/prc22.htmOne-sample hypotheses, z and t statistics, and rejection regions.
5. NIST/SEMATECH. Critical values of the Student’s t distribution.
   https://www.itl.nist.gov/div898/handbook/eda/section3/eda3672.htmUsing degrees of freedom, α, and tail direction to select critical values.
6. NIST/SEMATECH. Two-sample t-test for equal means.
   https://itl.nist.gov/div898/handbook/eda/section3/eda353.htmIndependent and paired designs, variance assumptions, and Welch–Satterthwaite calculations.
7. NIST/SEMATECH. Confidence limits for the mean—Example using R.
   https://www.itl.nist.gov/div898/handbook/eda/section3/eda3521.htmReading a confidence interval beside a matching one-sample test.
8. Penn State Department of Statistics. Lesson 6: Hypothesis testing, Part 2. STAT 200.
   https://online.stat.psu.edu/stat200/Lesson06Further practice with testing, Type I and II errors, and power.
9. Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355–362.
   https://doi.org/10.1177/1948550617697177
   Read the open full textAn introduction to defining equivalence bounds and testing effects against them.
10. NIST/SEMATECH. Bonferroni’s method.
    https://itl.nist.gov/div898/handbook/prc/section4/prc473.htmFamilywise error control and simultaneous comparisons.

Return to the top ↑