Experimental Research
Change a condition. Measure the response. Build a fair comparison that helps you explain whether, how much, and under what circumstances an intervention makes a difference.
What you will learn
- Connect manipulation, comparison, and random assignment with causal inference.
- Read experimental notation and compare major designs.
- Plan stimuli, measures, allocation, and fieldwork for a media experiment.
- Examine validity and interpret a worked treatment comparison.
- Prepare a responsible protocol and a transparent research report.
Understand: Read Sections 1–4 to define the causal question and comparison.
Choose: Use Sections 5–8 to compare designs and examine their implications.
Build: Complete Sections 9–12 with a draft protocol. Return to Section 13 for further reading.
Begin with a precise causal question
Experimental research studies the consequences of deliberately varying a condition. In a randomized experiment, a chance procedure assigns participants or other units to conditions, allowing a planned comparison of their outcomes.
Media and communication researchers can vary a news frame, source cue, advertisement, teaching activity, interface feature, or message format. They then measure an outcome such as recall, accuracy, attitude, behaviour, or response time. The question concerns the difference made by the assigned condition.
Think through the counterfactual
A causal effect compares what would happen under one condition with what would happen under another. The counterfactual is the relevant alternative outcome for the same unit. A well-designed comparison group helps estimate that missing alternative at the group level. J-PAL’s introduction develops this logic.
Order
The intervention comes before the outcome being studied.
Comparison
The analysis estimates how outcomes differ across conditions.
Alternative explanations
The design addresses other reasons the outcomes might differ.
Question: Does a short verification prompt improve immediate news-judgement accuracy compared with a matched formatting prompt?
Participants receive one of two instructions before completing the same news-judgement task. The verification instruction asks them to consider the original source and supporting evidence. The comparison instruction asks them to attend to layout and presentation.
Intended contrast: the average accuracy score under assignment to the verification prompt minus the average score under assignment to the formatting prompt.
This is a fictional teaching example. Its materials, participants, and numerical results are invented.
Experiments provide a strong route to causal inference when the assignment, implementation, measurement, and analysis support it. Quasi-experimental approaches can also investigate causal questions through other defensible comparisons and explicit assumptions.
Define the treatment, comparison, and outcome
The treatment specifies what changes between conditions. The outcome specifies what you will measure after that change. Define both closely enough for someone else to understand the experiment.
The manipulated independent variable, or factor, has levels such as verification prompt and formatting prompt. The dependent variable is the outcome: in the running example, a score on a news-judgement task.
Give the comparison condition a clear purpose
A control or comparison condition may involve no intervention, usual practice, an alternative intervention, or an active task matched on selected features. The choice determines the question the experiment answers.
Comparing a verification prompt with a formatting prompt estimates the effect of that particular contrast. Comparing it with no instruction would ask a different question, because attention, reading time, and expectations might also differ.
Operationalise the treatment
Specify the wording, duration, delivery device, instructions, exposure sequence, and features held constant. Save the final versions.
Operationalise the outcome
Specify the items or task, scoring rule, timing, and interpretation. Explain why the measure captures the intended construct.
Self-reported willingness to verify, factual accuracy, time spent opening a source, and actual clicking behaviour provide different evidence. Choose the outcome that answers the question and identify primary and secondary outcomes before analysis.
Separate a manipulation check from an outcome
A manipulation check examines whether participants perceived or experienced the intended difference. An outcome measures the response the experiment seeks to explain. For example, recognising the instruction’s focus checks its reception; accuracy on the task measures performance.
Build a fair comparison through random assignment
Random assignment uses a chance mechanism to allocate enrolled units to conditions. It makes treatment assignment independent of their pre-existing characteristics under the allocation procedure.
Across possible assignments, randomization balances observed and unobserved characteristics in expectation. A particular allocation can still show chance differences. Record the procedure and preserve the allocation rather than rearranging participants to improve an observed result.
Random sampling
Concerns how units enter the study from a population or frame. It shapes the basis for population representation.
Random assignment
Concerns how enrolled units enter experimental conditions. It supports the causal comparison within the study.
A study can recruit volunteers and still randomly assign them. Its allocation supports one part of the causal argument; the recruitment process and setting shape how far the finding may extend.
Read the verification prompt.
Complete the news-judgement task.
Read the matched formatting prompt.
Complete the same task.
Choose the unit of assignment
You may assign individuals, classes, schools, work teams, or other units. When whole classes receive a training activity, assignment may occur at the class level while measurement occurs at the student level. Account for that clustering in sample-size planning and analysis.
Consider spillovers: one person’s assignment may influence another person’s outcome. Students may share the verification prompt with friends in the comparison arm. A suitable cluster design, separate sessions, or a design that measures spillovers can help address the problem. J-PAL’s randomization resource explains these choices.
Match the setting and structure to the question
Experiments can take place in laboratories, classrooms, workplaces, field settings, or online environments. Choose the setting through the experience you need to study and the control you need to preserve.
| Setting | What it can offer | What to plan |
|---|---|---|
| Laboratory | Consistent presentation, equipment, and observation. | How the task and participants connect with the communication setting of interest. |
| Field or classroom | Interventions within an existing activity or institution. | Access, implementation, assignment units, spillovers, and changes outside the study. |
| Online | Randomized messages, interface features, or tasks delivered remotely. | Device display, exposure, duplicate participation, timing, privacy, and attrition. |
Between-participants and within-participants designs
In a between-participants design, different people experience different conditions. The running example uses this structure. Random assignment helps make the comparison defensible, and each participant encounters only one prompt.
In a within-participants design, the same person experiences several conditions. This can improve precision by using each person as part of the comparison. Plan the order, counterbalancing, fatigue, practice, and carryover effects, and use an analysis that handles the linked observations.
Ask a broader question with a factorial design
A factorial design combines the levels of two or more manipulated factors. A 2 × 2 design has four combinations. For example, cross message frame with the presence of a source cue:
| Message frame | Source cue present | Source cue absent |
|---|---|---|
| Gain frame | Gain message + cue | Gain message without cue |
| Loss frame | Loss message + cue | Loss message without cue |
A main effect compares levels of one factor, averaged over levels of the other in a balanced design. An interaction asks whether the effect of one factor changes across levels of the other. For instance, the frame difference might depend on whether a source cue appears. The NIST factorial-design guide explains the combinations.
Compare randomized experimental designs
Design notation records allocation, exposure, and measurement in time. It helps you see which comparisons a study can support.
C is added here to name the comparison explicitly. “None” means no assessment at that point. Subscripts identify measurement occasions within each design. Read time from left to right in the wide view, or down each group on a phone.
Randomized pretest–posttest design
Measure the outcome before exposure and again afterwards in both randomly assigned groups. The baseline helps describe starting values and can improve precision when included in a suitable planned analysis.
Randomized posttest-only design
Randomly assign participants, deliver the conditions, and assess the outcome afterwards. This is a randomized experimental design. It can support a causal comparison without a pretest when implementation and the other assumptions support the inference.
Solomon four-group design
Combine treatment and comparison conditions with the presence or absence of a pretest. Random assignment creates four groups. All four receive a posttest; two receive a pretest.
Plan the treatment comparison, pretest effect, and treatment-by-pretest interaction. The added groups require resources and suitable sample-size planning. Del Siegle’s design guide provides standard notation for these randomized designs.
Understand pre-experimental and quasi-experimental approaches
When random assignment is unavailable, researchers can still examine interventions through carefully structured comparisons. The design must explain why its comparison represents a credible alternative outcome.
One-shot case study
Observe an outcome after an exposure: X followed by O. For example, assess recall after a campaign message. This describes a response following exposure but supplies little information about the response under another condition.
One-group pretest–posttest
Assess one group before and after an intervention: O₁, then X, then O₂. The observed change may inform a pilot or programme description. Its causal interpretation needs to address changes that might have occurred during the same period anyway.
Non-equivalent comparison groups
Measure an intervention group and a non-randomly formed comparison group. For example, compare students enrolled in a public-speaking class with students taking another course, before and after the semester.
Describe how the groups formed, their baseline characteristics, and the reasons the comparison is plausible. Baseline adjustment or matching can address specified observed differences. The argument must also consider unmeasured differences and changes that affect the groups differently.
Interrupted time series
Measure an outcome repeatedly before and after an intervention introduced at a known point. Examine whether the outcome’s level or trend changes beyond the prior pattern. For instance, track a newsroom’s correction rate before and after a verification protocol.
The notation is schematic. A practical time-series study usually needs many observations, a stable measurement process, and analysis of seasonality, autocorrelation, and other changes near the intervention date. A suitable comparison series can strengthen the design.
Public speaking classes and writing abilities
A published study can show how researchers connect an intervention, outcome, and practical setting. Read its design alongside the claim it seeks to support.
In the lecture, Dr. USP discusses Yun, Costantini, and Billingsley (2012). Their study examined writing among 678 participants, approximately half enrolled in a public-speaking course. Students produced papers at the semester’s start and about 12 weeks later. Assessment covered five dimensions of writing.
The published abstract reports improvements associated with the course in organisation and syntactic control, while the other assessed dimensions showed no reported change. The lecture describes course-based groups rather than random assignment. On that account, the example is useful for examining a non-equivalent-group comparison.
- How did students enter the courses?
- How were the writing tasks and assessment standards made comparable?
- Which features of the assessors’ procedure reduced expectation effects?
- What other learning or selection processes could affect writing over the semester?
Use the example to recognise the components of a design: course exposure, writing outcomes, repeated assessment, and a comparison group. Then examine the full methods before deciding how strongly the study supports a causal interpretation.
Use validity questions to improve the design
Validity concerns the support for an interpretation. Examine the causal comparison, the meaning of the measures, the statistical reasoning, and the settings to which the result may extend.
Internal validity
How well does the comparison support attributing the outcome difference to the assigned condition?
Construct validity
How well do the manipulation and measure represent the concepts in the question?
Statistical conclusion validity
How well do the analysis and uncertainty estimates fit the data, assignment, and planned comparison?
External validity
Which people, messages, settings, and time periods might the finding inform?
Turn possible threats into design decisions
The lecture identifies changes over time, group differences, participant responses, and contamination as important sources of uncertainty. The table shows concrete ways to address them.
On a small screen, swipe across the table.
| Issue | Media-research example | A useful response |
|---|---|---|
| History | A major news event changes attitudes between assessments. | Use concurrent comparison groups; record relevant events. |
| Maturation | Students develop skills during a semester. | Compare changes over the same period in an appropriate group. |
| Testing | A baseline quiz draws attention to the later task. | Consider posttest-only or Solomon designs; pilot the assessment sequence. |
| Instrumentation | The rubric, item wording, device, or coding procedure changes. | Stabilise and document measurement; examine comparability when changes are necessary. |
| Attrition | Different kinds of participants leave the two arms. | Record flow and reasons by arm; assess missing-outcome assumptions. |
| Regression to the mean | Students selected for unusually low baseline scores score less extremely later. | Use a similarly selected comparison group and a suitable analysis of repeated measures. |
| Selection | Students choose courses through different interests or prior skills. | Randomize where feasible; otherwise examine how the comparison addresses selection. |
| Compensatory responses | A comparison group works harder or disengages after learning about the other arm. | Manage information and expectations; make the comparison experience appropriate. |
| Demand characteristics | Participants infer the desired result and tailor their responses. | Use neutral instructions, suitable masking, and less reactive measures where possible. |
| Researcher influence | Instructions or ratings vary with knowledge of condition. | Standardise delivery and mask assessors where feasible. |
| Reactivity or novelty | Being observed or using a new interface changes engagement. | Match relevant features across arms and assess whether the effect persists. |
| Diffusion or spillovers | Participants share instructions across conditions. | Plan assignment units, sessions, and measurement around likely contact. |
Reactivity can take different forms and directions. Examine how participation or observation might affect the particular task. A design earns confidence through the threats it addresses and the evidence it provides.
Build, pilot, and record the protocol
A protocol turns the research idea into a sequence that participants, researchers, and analysts can follow. Specify the decisions before launch and pilot the whole experience.
- Define the question and contrast: name the population, conditions, outcome, and time horizon.
- Develop the materials: create and save the stimuli, comparison materials, instructions, and assessment.
- Plan assignment: choose the unit, chance procedure, allocation ratio, and how the sequence will be concealed until assignment.
- Plan measurement: state outcome scoring, check timing, reliability procedures, and any follow-up.
- Plan sample size: use a meaningful effect, desired precision or power, assignment structure, and expected attrition.
- Specify analysis: record primary comparisons, exclusions, missingness, multiplicity, and any baseline adjustment.
- Pilot the procedure: examine consent, display, understanding, duration, routing, data export, and debriefing.
- Freeze and document: preserve versions and approved changes; train staff and monitor delivery.
Make stimulus construction part of the method
When manipulating a source cue, keep other relevant message features aligned. A longer text, a different photograph, or a different topic can change the interpretation of the treatment. State what the contrast is intended to capture and how the materials achieve it.
Use several message examples where the research question concerns a class of messages. A finding from one headline or advertisement may depend on its particular content. Plan stimulus sampling and analysis through the intended generalisation.
Preserve the comparison during delivery
Check that the assigned content reaches participants. Keep records of technical failures, interruptions, exposure, departures, and outcome availability. Explain who knew the assigned condition during recruitment, delivery, assessment, and analysis.
An intention-to-treat approach estimates the effect of assignment by retaining participants in their assigned groups. Missing outcomes still require a justified strategy and sensitivity assessment. A different question about treatment received needs its own assumptions. J-PAL’s analysis guidance discusses these issues.
A worked randomized message experiment
This fictional example uses a randomized posttest-only design. It shows why the treatment comparison, its uncertainty, and the scope of the conclusion belong together.
Eighty eligible volunteers are randomly assigned to two arms of 40. One arm receives a short verification prompt; the other receives a matched formatting prompt. Both complete the same ten-item accuracy task immediately afterwards.
The comparison keeps timing, device format, task, and scoring aligned. In this invented dataset, all 80 complete the outcome, each person contributes one score, and there is no cross-arm sharing. The planned analysis compares the two group means using a two-sided Welch t-test at α = .05.
These numbers illustrate analysis rather than provide a sample-size recommendation. Calculations use the displayed summary values. A real protocol needs justified measures, ethics arrangements, recruitment details, and sample-size planning.
| Assigned condition | n | Mean | SD |
|---|---|---|---|
| Verification prompt | 40 | 7.20 | 1.60 |
| Formatting prompt | 40 | 6.40 | 1.60 |
Estimate the treatment contrast
Reveal the contrast and uncertainty calculation
Estimated difference = 7.20 − 6.40 = 0.80 points.
For the illustrative independent-group model, SE = √[(1.60² ÷ 40) + (1.60² ÷ 40)] ≈ 0.358 points.
Welch t ≈ 0.80 ÷ 0.358 = 2.24; df = 78 for these equal-size, equal-SD summaries. The two-sided p-value is approximately .028.
The 95% confidence interval for the mean difference is approximately [0.09, 1.51] points.
Write a result that fits the design
Participants assigned to the verification prompt scored higher on the immediate accuracy task (M = 7.20, SD = 1.60, n = 40) than those assigned to the formatting prompt (M = 6.40, SD = 1.60, n = 40). The estimated mean difference was 0.80 points out of 10, 95% CI [0.09, 1.51], Welch t(78) = 2.24, p = .028. Under the study’s assignment and implementation conditions, this comparison supports an effect of prompt assignment on immediate task performance among the participating volunteers.
A standardised mean difference using the pooled SD is d = 0.80 ÷ 1.60 = 0.50. Report the definition and interpret the size through the task and context. The interval describes uncertainty under the analysis; practical importance needs a meaningful criterion.
Analyse pretest–posttest data through the group comparison
When a baseline outcome is collected, a planned ANCOVA can compare posttest outcomes while accounting for baseline values. Change-score comparisons or longitudinal models may suit other questions and structures. Choose the model through the design and assumptions before inspecting which analysis produces the smaller p-value. Vickers and Altman (2001) explain baseline adjustment.
Compare the treatment arms directly. Separate tests of improvement within each arm address different questions from the between-arm treatment contrast. For clustered or repeated data, carry the dependence structure into the analysis.
Conduct and report the experiment responsibly
A responsible experiment gives participants a clear basis for participation and gives readers a clear basis for judging the study. Build both into the protocol.
Obtain the required institutional ethics approval or determination. Explain participation, procedures, foreseeable risks, data use, and withdrawal arrangements in accessible language. In student research, make declining participation a realistic choice and provide an equitable alternative where participation is tied to course credit.
Plan masking and debriefing
Masking can limit knowledge of condition or hypothesis among participants, assessors, or analysts where feasible. State exactly who was masked and to what. In media research, participants may recognise the message they see even when they do not know the predicted result.
Deception or incomplete disclosure requires a specific ethical justification and prior approval under the applicable review process. Consider effective alternatives and plan an appropriate debriefing. The APA research ethics standards discuss consent, student participation, deception, and debriefing.
For a misinformation-related task, explain the material appropriately at debriefing and correct misconceptions created by the procedure. Limit identifiers, protect assignment and response records, and obtain suitable permission for audio or video recording.
Make the experiment assessable
- Participants and allocation: recruitment, eligibility, numbers, assignment unit, procedure, and allocation ratio.
- Conditions: materials, comparison, exposure, timing, delivery, and relevant masking.
- Measures: primary and secondary outcomes, scoring, checks, and assessment occasions.
- Participant flow: assigned, exposed, followed up, excluded, missing, and analysed, by condition.
- Analysis and evidence: prespecified and exploratory comparisons, estimates, intervals, tests, and sensitivity analyses.
- Interpretation: the treatment contrast, validity, practical meaning, and scope.
The APA JARS–Quant resources include reporting guidance for experimental manipulation and for random and nonrandom assignment. Use them alongside the target journal’s instructions.
Use the lecture as an experimental-design workshop
Watch Dr. USP’s lecture with a protocol draft beside you. Pause at the points below and connect each concept with a decision in your own study.
Open the lecture on YouTube. These links mark approximate starting points from the supplied transcript. Use the chapter’s design distinctions alongside the lecture.
Check your understanding
Write a response before opening the suggested answer.
1. Can volunteers take part in a randomized experiment?
Yes. Volunteer recruitment describes entry into the study. Random assignment describes allocation to conditions after entry. The first shapes the population reach; the second supports the study’s treatment comparison.
2. Can a randomized posttest-only study support a causal comparison?
Yes, when assignment, implementation, outcome availability, and analysis support the inference. Randomization supplies the basis for comparing the arms. A pretest can serve useful purposes, including improved precision, but it is one design choice.
3. Why might a comparison group receive another message?
An active comparison can align time, attention, format, and expectations. The resulting estimate concerns the contrast between the particular messages or tasks. Choose the comparison through the question you want to answer.
4. What does a Solomon design add?
It crosses treatment with whether participants receive a pretest. Its four groups help examine a pretest effect and whether the treatment response differs after pretesting. It still requires careful measurement, implementation, and follow-up.
5. Why does course enrolment create a selection question?
Students may choose courses through interests, earlier skills, scheduling, or other characteristics related to the outcome. A course-based comparison must explain how those processes affect the causal argument, even when both groups receive pretests and posttests.
6. What is the problem with analysing only people who passed a post-exposure check?
Passing the check can be affected by the assigned condition and participant characteristics. Selecting on it may change the groups formed by randomization. Prespecify checks and report the assigned-group analysis, with any additional analysis clearly justified.
7. Does improvement in the treatment group identify the treatment contrast?
The treatment contrast requires comparison with the relevant alternative. A group may improve over time through several processes. Analyse the between-arm difference or another design-appropriate contrast directly.
Your workshop output: a one-page experimental protocol
- State the causal question and the treatment contrast.
- Define the units, recruitment, assignment, and setting.
- Describe both conditions and the features held constant.
- Specify the primary outcome, timing, scoring, and checks.
- Choose a design and explain its notation.
- Address the three most consequential validity issues.
- Plan sample size, participant flow, analysis, ethics, and reporting.
Ask a peer to reconstruct what a participant experiences and what comparison the analysis estimates. Revise the protocol where the two accounts fail to connect.
Glossary and further reading
Return to these terms and sources as you build a protocol or assess an experimental paper.
- Treatment
- The intervention or condition whose consequences are being studied.
- Comparison condition
- The specified alternative against which treatment outcomes are assessed.
- Counterfactual
- The relevant outcome that would occur under an alternative condition.
- Random assignment
- Allocation of enrolled units to conditions through a chance procedure.
- Allocation concealment
- Keeping upcoming assignments unavailable to those whose enrolment decisions could be influenced by them.
- Masking
- Limiting knowledge of condition or hypothesis for specified people or stages.
- Manipulation check
- A measure of whether the intended treatment distinction was perceived or experienced.
- Factorial design
- A design combining levels of multiple factors so their separate and joint effects can be examined.
- Interaction
- A pattern in which the effect of one factor depends on the level of another.
- Attrition
- Loss of participants or outcome information during a study.
- Spillover
- An effect of one unit’s treatment on another unit’s outcome.
- Intention-to-treat
- An analysis principle that retains units in their assigned conditions to estimate the effect of assignment.
- Pandey, U. S. Experimental research in media and communication by USP [Video]. YouTube.
https://www.youtube.com/watch?v=W3yT3rz0Y34The lecture and supplied transcript provide the teaching base. - Siegle, D. Experimental research designs. University of Connecticut, Neag School of Education.
https://researchbasics.education.uconn.edu/experimental_research_designs/Notation for pre-experimental, randomized pretest–posttest, posttest-only, and Solomon designs. - Abdul Latif Jameel Poverty Action Lab. Introduction to randomized evaluations.
https://www.povertyactionlab.org/resource/introduction-randomized-evaluationsThe comparison and counterfactual logic of randomized evaluation. - Abdul Latif Jameel Poverty Action Lab. Randomization.
https://www.povertyactionlab.org/resource/randomizationAssignment units, allocation procedures, and spillovers. - Abdul Latif Jameel Poverty Action Lab. Data analysis.
https://www.povertyactionlab.org/resource/data-analysisAnalysis choices, treatment assignment, attrition, and uncertainty. - Yun, K. A., Costantini, C., & Billingsley, S. (2012). The effect of taking a public speaking class on one’s writing abilities. Communication Research Reports, 29(4), 285–291.
https://doi.org/10.1080/08824096.2012.723270The published study discussed in the lecture. The chapter’s summary uses the publisher’s abstract alongside the lecture’s account of allocation. - NIST/SEMATECH. Full factorial designs. e-Handbook of Statistical Methods.
https://www.itl.nist.gov/div898/handbook/pri/section3/pri333.htmCombining the levels of experimental factors. - Vickers, A. J., & Altman, D. G. (2001). Analysing controlled trials with baseline and follow up measurements. BMJ, 323, 1123–1124.
https://doi.org/10.1136/bmj.323.7321.1123Planned comparisons using baseline and follow-up scores. - American Psychological Association. Quantitative research design (JARS–Quant).
https://apastyle.apa.org/jars/quantitativeReporting resources for experimental manipulation and random or nonrandom assignment. - American Psychological Association. (2017). Ethical principles of psychologists and code of conduct.
https://www.apa.org/ethics/code/Research consent, student participation, deception, and debriefing. Apply the requirements of your own institution and study setting.
Carry the design into the analysis
The contrast defined in your protocol determines what the statistical test should examine. Draft the result in original units, add its confidence interval, and explain how the assignment and implementation support the conclusion. Use Research Methodology to connect these decisions with the wider research plan.