Choosing Research Methods
Begin with the question. Choose the evidence that can answer it, build a design that supports the intended claim, and select an analysis that respects how the data were produced.
What you will learn
- Translate a research problem into an answerable question and evidence plan.
- Compare survey, longitudinal, experimental, qualitative, and mixed-method designs.
- Identify measurement levels, outcomes, predictors, and dependent observations.
- Choose starting analyses for comparisons, associations, and prediction.
- Check assumptions and write a defensible methods justification.
Design the inquiry: Use Sections 1–5 to choose the evidence and study structure.
Plan the analysis: Work through Sections 6–10 with a variable map.
Apply and explain: Complete the worked example and activities in Sections 11–12. Extend your reading in Section 13.
Start with a precise research question
A research method earns its place by helping answer a question. Begin by specifying the phenomenon, the people or materials involved, the setting, the period, and the kind of claim you want to make.
“AI and journalism” names a field of interest. “What proportion of journalists in a defined region can identify an unsupported AI-generated claim?” specifies a population and an outcome. “How do reporters decide whether to trust an AI summary during a deadline?” asks about a situated practice. These questions call for different evidence.
Connect four levels of decision
Methodology supplies the reasoning and assumptions behind the inquiry. Design organises the study: for example, a panel survey, experiment, or bounded case study. Methods collect or generate evidence through questionnaires, interviews, observation, documents, or tasks. Analysis develops findings from that evidence.
- Question and intended claimEstimate a pattern, understand a practice, explain change, or evaluate an intervention.
- Design and selectionChoose the comparison, cases, population, sampling route, and timing.
- Measures and evidenceDefine concepts and decide what observations would represent them.
- Data structureIdentify outcomes, units, groups, repeated measures, and clusters.
- Analysis and interpretationEstimate or interpret the relevant relationship, examine alternatives, and state the claim’s scope.
Your assumptions about knowledge also matter. A study estimating a measurable skill and a study interpreting how journalists define “responsible use” ask different questions about the world. Explain those assumptions and adopt quality criteria suited to the inquiry.
Match the question with a plausible design
Research purposes include description, comparison, explanation, interpretation, reconstruction, and prediction. Several designs may serve a purpose; compare them through the evidence they can provide and the assumptions they require.
On a narrow screen, scroll the table sideways.
| What you want to know | A plausible route | What the design needs |
|---|---|---|
| How common is a practice? | A cross-sectional survey or suitable existing dataset. | A defined population, credible sampling, and valid measures. |
| How is a population changing? | Repeated cross-sectional surveys. | Comparable population definitions, sampling, and measures across waves. |
| How do individuals change? | A panel or cohort follow-up design. | Linked observations, retention, and analysis of within-person dependence. |
| What difference does an intervention make? | An experiment, suitable quasi-experiment, or justified theory-based evaluation. | A causal argument, comparison or process evidence, and attention to rival explanations. |
| How do people understand and enact a practice? | Interviews, observation, ethnography, or an appropriate case study. | Relevant experience, contextual depth, and an explicit interpretive approach. |
| How are issues represented? | Content, discourse, narrative, or multimodal analysis, chosen through the question. | A bounded corpus, clear units, and an analytical framework. |
| How did a communication practice develop? | Historical inquiry using archives and other relevant sources. | Source provenance, authenticity, context, chronology, and critical interpretation. |
| Can an outcome be predicted? | A predictive modelling study. | A defined outcome and intended use, suitable predictors, and validation on new data. |
Use specialised surveys for specific purposes
A knowledge, attitudes, and practices (KAP) survey organises questions around what people know, how they evaluate an issue, and what they report doing. In the AI example, factual items can assess knowledge, evaluative items can assess attitudes, and behaviour questions can assess reported practices. Observed tasks or records can supply additional evidence about actual practice.
Choose qualitative analysis through its focus
Thematic analysis can investigate patterned meanings across accounts. Discourse analysis examines communication in context and what it accomplishes. Narrative analysis attends to how stories organise events and experience. Ethnography studies practices and relationships through sustained engagement with a setting. A case study brings evidence together around a bounded case.
Choose quantitative, qualitative, or mixed methods
Quantitative approaches estimate quantities and model relationships. Qualitative approaches investigate meanings, practices, processes, and contexts. A mixed-method study deliberately integrates quantitative and qualitative strands to answer connected questions.
Give each strand a job
- Explanatory sequential: analyse quantitative findings, then use qualitative follow-up to explore their interpretation. Interview journalists showing contrasting patterns of measured skill and confidence.
- Exploratory sequential: begin with qualitative inquiry, then develop and investigate a quantitative measure or pattern. Use newsroom interviews to inform survey items, followed by piloting and validation.
- Convergent: develop both strands in a coordinated period, then compare and integrate the findings. Examine task performance alongside accounts of newsroom constraints.
The NIH mixed-methods guide develops these design options. Their usefulness depends on the integration: how one strand informs the next, how evidence is brought together, and what the combined interpretation adds.
Use triangulation with a clear purpose
Triangulation compares relevant sources, methods, theories, or investigators. It can be used within a qualitative study, a quantitative programme, or a mixed-method design. Disagreement between sources may reveal different settings, meanings, or recording processes.
Choose a time design
The time design determines whether the study describes a period, compares populations across periods, or follows change within the same units.
A cross-sectional study observes a sample during one defined period. It can describe current awareness or examine associations among measured characteristics. A trend study repeats comparable surveys of the same target population, usually with newly selected samples.
A panel study follows the same units over time. A cohort is a group defined by a shared characteristic or entry event, such as beginning journalism work in the same year. A cohort follow-up often tracks those individuals repeatedly; “cohort” describes membership, while “panel” highlights repeated observation of units.
Trend · Population change
New samples from the same defined population.
Panel · Within-person change
The same participants followed across rounds.
Preserve comparability
Track changes in question wording, mode, eligibility, recruitment, and the meaning of key concepts. In a panel, maintain secure linkage and examine who drops out. In a trend study, changing sample composition can contribute to an observed population difference.
UK Data Service guidance explains these survey structures. Use the time design to decide what sort of change the data can show.
Choose a design for interventions and impact
An intervention study asks what difference a programme, message, policy, or treatment makes. The design must establish a credible account of what would have happened under a relevant alternative condition.
Use random assignment where it fits
In an experiment, researchers deliberately vary a condition. Random assignment allocates units to conditions through chance, supporting a planned comparison. For example, journalists could receive a verification prompt or a comparison instruction before completing the same task.
Specify the treatment, comparison, outcome, assignment unit, exposure, and timing. If whole newsrooms receive training, newsroom-level assignment and within-newsroom similarity belong in the sample-size and analysis plan.
Evaluate alternatives when assignment is constrained
Quasi-experimental designs can use features such as eligibility thresholds, comparison trends, or repeated observations around an intervention. Their causal argument depends on assumptions specific to the design. A before-and-after study of one participating group directly estimates observed change; attribution requires further reasoning and evidence.
Match the evaluation question
Process evaluation investigates how delivery unfolds, who participates, and what helps or hinders implementation. Impact evaluation investigates changes attributable to the intervention. Prospective impact assessment considers likely consequences of a proposed action. These purposes may use complementary methods.
The Magenta Book develops experimental, quasi-experimental, and theory-based evaluation approaches. Case evidence and process tracing can investigate mechanisms and rival explanations, including settings where a single numerical effect estimate is difficult to establish.
Map the variables and measurement levels
Statistical analysis begins with what the variables represent and how they were measured. Identify the outcome, the predictors or group factors, and the quantity the analysis should estimate.
A dependent or outcome variable is what the analysis seeks to explain or predict. Predictors may be observed characteristics or assigned conditions. In an experiment, a manipulated factor specifies the treatment contrast. In an observational study, its interpretation follows the study design and assumptions.
On a narrow screen, scroll the table sideways.
| Level | Meaning and example | Analytical implication |
|---|---|---|
| Nominal | Categories without an intrinsic order: newsroom type or yes/no verification. | Use counts, proportions, and categorical-outcome methods where appropriate. |
| Ordinal | Ordered categories with unspecified spacing: a single five-category confidence item. | Consider ordered-category models or suitable rank-based analyses. |
| Interval | Equal differences have meaning; zero is a reference point, as in calendar year. | Differences are interpretable; ratios need a meaningful absolute zero. |
| Ratio | Equal differences and a meaningful zero: time spent verifying or number of corrections. | Choose a model through the outcome’s distribution, range, and structure. |
Nominal and ordinal variables are both categorical. Numerical variables can be continuous or discrete. Counts are numerical but may call for count models. Software often groups interval and ratio variables under “scale”; inspect the actual measurement rather than relying on that label.
Choose the quantity of interest
Do you want a proportion, a mean difference, a change score, an association, an odds ratio, or a prediction for a new case? This quantity, sometimes called an estimand, helps determine the analysis and reporting.
A single confidence item and a validated multi-item score present different measurement questions. Justify any decision to treat a composite score numerically through its construction, evidence of validity, and use in the analysis.
Recognise independence, pairing, and clustering
The same variable can require different analyses depending on how observations are related. Establish the unit of analysis and the dependence structure before choosing a test.
Independent groups
Different participants supply scores in two groups. A journalist appears in one group only, with no relevant pairing or clustering left unmodelled.
Paired or matched observations
The same journalist supplies a pre-score and post-score, or a design creates meaningful matched pairs. Analyse the linkage.
Repeated observations
Each journalist supplies several measurements. The analysis must represent within-person dependence across occasions.
Clustered observations
Journalists belong to newsrooms, or articles belong to outlets. Shared conditions can create within-cluster similarity.
Three hundred articles from ten outlets provide 300 article records, with an outlet structure that may matter to the analysis. Likewise, repeated survey waves do not create a new independent person at every wave. Preserve the IDs needed to model those relationships.
Carry sampling into analysis
Population estimation may require survey weights, strata, and cluster information. A simple test selected from a chart may need a survey-adjusted version or another suitable procedure. Plan recruitment, size, retention, and the smallest subgroup or number of clusters needed for the main question.
Compare groups or measurement occasions
For a comparison, specify the outcome and contrast first. Then identify the number of groups or occasions, their relationship, and the assumptions supporting the analysis.
Starting points for numerical mean comparisons
On a narrow screen, scroll the table sideways.
| Question and structure | Starting analysis | What to check or report |
|---|---|---|
| One sample mean versus a benchmark | One-sample t-test. | Define the benchmark and report the mean difference and interval. |
| Two independent group means | Welch’s independent-samples t-test. | Independent units, outliers, distributional adequacy, and the mean difference. Welch allows unequal variances. |
| Two linked measurements | Paired t-test on the within-pair differences. | Meaningful pairing, independent pairs, and the distribution of differences. |
| Three or more independent group means | One-way ANOVA under appropriate variance assumptions, or Welch ANOVA. | The omnibus comparison, planned contrasts, and any adjusted follow-up comparisons. |
| Three or more occasions on the same units | A suitable repeated-measures model or mixed-effects model. | The time contrast, within-unit covariance, missing observations, and model assumptions. |
Welch’s test addresses unequal population variances in an independent two-group mean comparison. The paired t-test instead uses linked differences. Keeping those structures distinct is a central analytical decision.
Consider rank-based routes deliberately
On a narrow screen, scroll the table sideways.
| Structure | Possible analysis | Interpretive focus |
|---|---|---|
| Two independent groups | Mann–Whitney U / Wilcoxon rank-sum. | Compare distributions or a specified stochastic ordering. A location or median interpretation needs additional assumptions. |
| Three or more independent groups | Kruskal–Wallis. | An omnibus rank-based comparison; specify the interpretation and follow-up procedure. |
| Two linked measurements with meaningfully rankable differences | Wilcoxon signed-rank, with an appropriate symmetry assumption for the differences. | A signed-rank hypothesis about paired differences. A paired sign test can address a directional-change question under different assumptions. |
| Three or more linked occasions or conditions | Friedman test in a suitable complete-block design, with each unit observed in every condition. | Compare within-unit ranks across conditions; account for ties and plan follow-up comparisons. |
Add covariates through a justified model
ANCOVA compares a numerical outcome across groups while modelling covariates, such as a baseline score. It belongs to the linear-model family and can involve two or more groups. Specify group contrasts, covariate relationships, and interactions where warranted.
Investigate associations and prediction
Association asks how variables relate. Prediction asks how well information can estimate an outcome for new observations. Begin with the outcome’s form and the relationship you intend to study.
On a narrow screen, scroll the table sideways.
| Variables and structure | Starting analysis | Key decision |
|---|---|---|
| Two numerical variables | Pearson correlation for a linear association. | Inspect the scatterplot, influential observations, and conditions for the intended inference. |
| Ordered or numerical variables | Spearman rank correlation for a monotonic association. | Ask whether a generally increasing or decreasing relationship is the target; inspect ties and shape. |
| Two categorical variables from independent units | Chi-square test of independence; Fisher’s exact test for an appropriate sparse 2 × 2 table. | Use counts with mutually exclusive cells. Assess expected counts and the approximation or exact procedure. |
| The same binary outcome measured twice | McNemar’s test. | Use the paired cross-tabulation and discordant pairs; consider an exact version with sparse discordance. |
For a single categorical variable against a specified distribution, consider a goodness-of-fit procedure. A one-sample binary proportion can use a binomial or other appropriate proportion procedure. Those questions differ from an independence test between two variables.
Choose the regression family through the outcome
On a narrow screen, scroll the table sideways.
| Outcome | Model to consider | Example and planning requirement |
|---|---|---|
| A numerical score with a suitable linear-model interpretation | Simple or multiple linear regression. | Model verification score using experience and training; examine functional form, residuals, and influential observations. |
| A binary outcome | Binary logistic regression. | Model whether a journalist verifies a claim. Define outcome coding and relevant predictors. |
| An ordered categorical outcome | An ordinal regression model. | Model an ordered confidence response; assess assumptions such as proportional odds where applicable. |
| A count | Poisson or a suitable alternative such as negative-binomial regression. | Model correction counts, examining dispersion and any exposure term for different observation periods or publication volumes. |
Multiple regression uses more than one predictor. Predictors may be numerical or appropriately encoded categorical variables. The model need not contain exactly two predictors or require one predictor to be numerical. Official model documentation provides further detail on binary, ordinal, nominal, and count outcomes.
Explain coefficients in useful units. For logistic regression, distinguish odds ratios from predicted probabilities. For prediction, evaluate performance on new or appropriately held-out data, preserving any person or newsroom grouping in the validation design.
Check assumptions, uncertainty, and feasibility
A defensible analysis fits the question, measurement, dependence structure, and model conditions. Use diagnostics and subject knowledge to assess that fit before interpreting the result.
Examine what the model assumes
- Independence: assess how observations were sampled, assigned, repeated, or clustered.
- Distribution and form: inspect relevant plots, model residuals, outliers, and functional relationships. For a paired t-test, examine the paired differences.
- Variance and covariance: check the assumptions of the chosen procedure. Equal variance is relevant to a pooled t-test; Welch’s test allows unequal variances.
- Data adequacy: inspect sparse categories, ties, missing values, attrition, and whether the sample supports the planned complexity.
Normality assumptions for many linear-model inference procedures concern errors or residuals rather than every measured variable. A normality-test p-value alone supplies too little information for selecting an analysis. Consider the estimand, sample size, shape, influential values, and robustness of the procedure together.
Plan what you will report
Lead with estimates in meaningful units and suitable confidence intervals. Add effect sizes and statistical tests when they serve the question. A p-value evaluates how extreme the data are under the specified null model and procedure. Interpret it alongside magnitude, precision, design, and practical meaning.
Specify the primary outcome, main comparisons, missing-data strategy, and treatment of multiple comparisons before examining the results where possible. Power or precision planning should follow those targets. Document exploratory analyses as such.
Check the fieldwork can deliver the evidence
Assess access, budget, language, time, skills, consent, and data protection. Pilot the measures and recruitment. If the available design cannot support the intended claim, revise the question or obtain the missing evidence. Qualitative studies likewise need justified selection, contextual adequacy, reflexivity, and a transparent interpretive process.
Work through a before-and-after study
A numerical example shows why design and analysis need to be planned together. Here the intended quantity is the average within-person change in a verification score.
Sixty volunteer journalists complete a verification assessment before a workshop and again afterwards. Scores range from 0 to 20. All sixty provide both scores. The mean pre-score is 11.4 and the mean post-score is 12.6.
The mean paired change is 1.20 points, with an SD of the individual changes of 2.40. Assume independent participants and differences adequately represented by the paired t-test model. The illustration has no comparison group.
Choose the analysis through the structure
The same people contribute the two measurements, so the observations are paired. A paired t-test examines whether the population mean change represented by the model differs from zero. Calculate change as post-score minus pre-score.
Follow the paired calculation
SE of mean change = SD of changes ÷ √n= 2.40 ÷ √60 ≈ 0.310t = (1.20 − 0) ÷ 0.310 ≈ 3.87df = 60 − 1 = 5995% CI ≈ 1.20 ± 2.001 × 0.310≈ [0.58, 1.82]The SD needed here is the SD of the within-person changes. Separate pre-score and post-score SDs alone do not supply it without information about their covariance.
Write a result that fits the design
“Among the sixty participating journalists, verification scores increased by an estimated mean of 1.20 points out of 20, 95% CI [0.58, 1.82], paired t(59) = 3.87, p < .001. The estimate describes observed change across the two assessments.”
See how a changed question changes the route
On a narrow screen, scroll the table sideways.
| Revised question | Design implication | Analysis implication |
|---|---|---|
| What difference does assignment to the workshop make? | Randomly assign an appropriate comparison condition where feasible. | Compare the assigned groups directly, for example using a planned baseline-adjusted model for the score. |
| Did the same participants change a yes/no verification behaviour? | Retain linked binary outcomes. | Consider McNemar’s test or a suitable repeated binary-outcome model. |
| Why did some journalists report difficulty applying the skills? | Recruit relevant experiences for qualitative follow-up. | Develop a justified qualitative analysis linked to context and participant accounts. |
“The study asks [question] and seeks to estimate or interpret [target]. We chose [design] because it provides [required evidence]. The units are [units], selected through [sampling or case rationale]. Data comprise [measures or materials], collected [timing]. We will use [analysis], accounting for [pairing, clusters, weights, covariates, or interpretive context]. We will assess [assumptions and quality criteria]. The resulting claim will concern [scope].”
Learn with both lectures
Use the first lecture to compare research designs and the second to connect measurement and data structure with statistical analysis. Keep a question sheet and variable map beside you.
Lecture 1 · Which research method to use
Open the research-methods lecture on YouTube. Timestamp links below give approximate starting points from the supplied transcript.
Lecture 2 · Which statistical tests to use
Open the statistical-tests lecture on YouTube. Use the chapter’s detailed assumptions and interpretation alongside the overview.
Attempt each question before opening the response
1. You survey different journalists each year. What change can you study?
A repeated cross-sectional or trend design can examine change in the defined population or subgroups. Comparable sampling and measures matter. Individual trajectories require linked observations of the same people.
2. What makes a survey-plus-interview study mixed methods?
It deliberately integrates quantitative and qualitative evidence. For example, the survey can guide the selection of follow-up interviewees, and the interview analysis can inform the interpretation of a measured pattern. Explain where and how that integration occurs.
3. The same people provide numerical scores before and after training. Where do you begin?
Identify the within-person change and inspect its measurement and distribution. A paired t-test can address a mean-change question under its assumptions. Retain the pairs and use the design to judge the reach of any intervention claim.
4. The same people answer yes/no twice. Would an ordinary independence chi-square test fit?
The observations are paired. Use the paired cross-tabulation and consider McNemar’s test for a two-occasion binary comparison, or an appropriate model for a broader repeated-outcome question.
5. Does a Mann–Whitney result automatically test a difference in medians?
Its general interpretation concerns distributions or a specified stochastic ordering. A median or location interpretation requires additional conditions about the distributions. State the hypothesis and assumptions that justify the interpretation.
6. Must multiple regression contain exactly two predictors?
Multiple regression uses more than one predictor, selected through the question and an adequately supported model. Predictors can include numerical variables and appropriately encoded categories. Plan complexity, interactions, and sample size deliberately.
7. What analysis might suit the number of corrections published by each outlet?
A count model such as Poisson or negative-binomial regression may be appropriate. Examine dispersion, differences in publication volume or observation time, repeated outlet observations, and the substantive meaning of the outcome.
8. What does the workshop example establish directly?
It estimates an observed mean increase in scores across paired assessments among the participating volunteers. A causal workshop-effect claim requires a design and evidence addressing what would have happened over the same period under a relevant alternative condition.
Your independent learning project
Choose one communication problem and prepare a one-page design-and-analysis justification.
- Write the question and intended claim.
- Compare two plausible designs and explain your choice.
- Specify units, sampling, timing, measures, and evidence.
- Map outcomes, predictors, and any pairing or clustering.
- Select the analysis and define the quantity or meaning it will address.
- State assumptions, quality checks, missing-data decisions, and practical constraints.
- Write the strongest claim the design can support and identify a useful next study.
Completion check: A reader should be able to follow the route from the question to the evidence, analysis, and conclusion.
Glossary and further learning
Use these terms to sharpen a methods justification. The resources below develop the design logic and the statistical procedures introduced in the chapter.
- Research design
- The structure connecting the question, selection, evidence, comparisons, and analysis.
- Estimand
- The quantity a study seeks to estimate, specified for a population, comparison, and time where relevant.
- Outcome
- The variable the analysis seeks to explain, compare, or predict.
- Predictor
- A variable used to model an outcome; its causal interpretation depends on design and assumptions.
- Repeated cross-section
- Comparable observations of a population at multiple times, usually through newly selected samples.
- Panel
- A design that follows the same units across measurement occasions.
- Cohort
- A group defined by a shared characteristic or entry event.
- Triangulation
- Comparison of relevant sources, methods, perspectives, or investigators to examine an account.
- Integration
- The planned connection of quantitative and qualitative strands in a mixed-method study.
- Residual
- The difference between an observed value and its model-predicted value.
- Covariate
- A variable included in a model alongside the focal comparison or predictor.
- Confounding
- A distortion of a causal comparison arising from the relationship of other causes with the exposure or treatment.
- Omnibus test
- A test addressing an overall multi-group or multi-parameter hypothesis.
- Validation
- Assessment of a measure or model against evidence appropriate to its intended interpretation or use.
Lectures and design resources
- Pandey, U. S. Which research method to use and Which statistical tests to use in social science research. Dr. USP [Videos].
Research-methods lecture · Statistical-tests lectureThe supplied lectures and transcripts provide the chapter’s starting framework. - Creswell, J. W., Klassen, A. C., Plano Clark, V. L., & Smith, K. C. (2011). Best practices for mixed methods research in the health sciences. National Institutes of Health, Office of Behavioral and Social Sciences Research.
Read the mixed-methods design guide [PDF]Explains design options and integration. This chapter applies the general design ideas to communication examples. - UK Data Service. Using survey data.
Explore survey structures and change over timeIntroduces cross-sectional, repeated cross-sectional, and longitudinal data. - HM Treasury & Evaluation Task Force. (2026). Magenta Book: Central government guidance on evaluation.
Read the evaluation guidanceDevelops process, impact, theory-based, experimental, and quasi-experimental evaluation choices. - UCLA Office of Advanced Research Computing. Choosing the correct statistical test in SAS, Stata, SPSS and R.
Consult the statistical-analysis overviewA broader starting guide organised by variable and study structure. Use alongside method-specific assumptions. - SciPy. Independent-samples t-test documentation.
Read the independent-sample and Welch test definitionsDistinguishes equal-variance and unequal-variance mean comparisons. - SciPy. Related-samples t-test documentation.
Read the paired-test documentationDescribes the related-samples mean comparison and confidence interval. - SciPy. Mann–Whitney U and Wilcoxon signed-rank documentation.
Independent rank comparison · Paired signed-rank comparisonMethod-specific hypotheses, assumptions, and treatment of ties or zero differences. - statsmodels. Regression with discrete dependent variable.
Explore binary, nominal, ordinal, and count modelsOfficial documentation linking outcome types with model families.
Turn the choice into a protocol
Take the justification you have written and specify the actual materials, participants, records, and procedures. Use Research Methodology to connect the decisions across the study. For work with communication texts, Content Analysis and Ethnography open different routes into representations and social practice.