Scale Development

Understand what a scale measures

Scale development creates and evaluates a set of items whose responses support an interpretation of a construct. A construct is a theoretically defined attribute, such as confidence, perceived belonging, or satisfaction, that researchers investigate through observable indicators.

Think of a scale as a carefully marked measuring instrument. Before interpreting its readings, you need to know what it measures, how its markings were established, and the conditions under which those readings remain useful. In social research, theory and evidence perform that work.

A questionnaire can contain background questions, factual questions, and several scales. An individual rating item produces one response. A multi-item scale combines responses according to a defensible scoring model. The development task is to establish what that combination means.

Decide how the items relate to the construct

In a reflective model, a latent attribute helps explain the pattern of item responses. Several items about confidence in checking news may share variation because they reflect related confidence judgments. Factor analysis examines the proposed structure of that shared variation.

A composite index combines components that define an overall measure, such as access to devices, connectivity, and service availability. Its components can differ substantially without indicating a defective index. The design and evaluation should follow the meaning of the components and the intended combination.

Define the construct

A construct definition identifies the attribute, its proposed components, and the conditions in which it will be measured. A strong definition guides item selection and explains what a higher score should signify.

Read existing instruments before creating another

Search for scales that already address the construct. Examine their theoretical definitions, populations, languages, response formats, scoring rules, and validation evidence. An existing instrument may answer the question well. If you adapt it, document the changes, retain attribution, and check permissions or licensing.

Write a short justification for developing a new scale. Perhaps the available measures concern general digital confidence while your question concerns specific verification tasks. Explain the gap in content and intended use, then show how the proposed items address it.

Map the proposed domain

For this teaching example, source checking, context checking, and evidence comparison are provisional facets. The blueprint below helps generate items across the domain. Subsequent qualitative and quantitative work must examine whether these facets are meaningful and how responses are structured.

On a small screen, swipe the table horizontally.

Separate neighbouring constructs

Confidence, factual knowledge, actual verification skill, and the frequency of checking news are related research topics. Give each a distinct role in the study. The proposed scale concerns a belief about capability. Performance tasks can supply separate evidence about demonstrated skill.

Generate the item pool

Item generation translates a construct blueprint into questions or statements that target users can answer. A broad initial pool creates room to revise wording, remove weak items, and retain coverage of important facets.

Combine theoretical and experiential sources

Use literature and existing instruments to identify established content. Use interviews, focus groups, or other suitable qualitative work with target users to discover language, situations, and experiences that published measures may miss. Seek expert input on the construct and the measurement task.

Keep an item record with an ID, wording, proposed facet, source, rationale, and revision history. This record will later help explain why an item was retained, revised, or removed. Preserve the original pool alongside every subsequent version.

Write one clear judgment per item

Choose a consistent task and point of reference. The running example can use the common stem, “How confident are you that you can perform each task when you encounter a doubtful news claim online?” The individual items then name the tasks.

Match the response options to the question

For a confidence stem, a possible five-category format is: 1 = not at all confident; 2 = slightly confident; 3 = moderately confident; 4 = very confident; 5 = extremely confident. Explain the options and examine whether users can distinguish them.

Frequency, agreement, confidence, and difficulty scales invite different judgments. Select the format that matches the construct. A recall period is useful when the question concerns experiences over time; justify it and make it consistent across relevant items.

Generate breadth before reducing length

The lecture recommends beginning with substantially more items than the intended final scale. Treat the suggested three-to-fourfold pool as a planning heuristic. The required breadth depends on domain complexity, available items, respondent burden, and the need to retain adequate indicators for each proposed dimension.

Review and pretest

Content review and cognitive pretesting investigate whether the items represent the intended domain and whether respondents understand the task as intended. Together they help uncover problems that response correlations alone can miss.

Review relevance, coverage, and comprehension

Ask construct specialists, measurement specialists, and target users what each item contributes. Examine whether important content is missing, whether any item strays beyond the definition, and whether the instructions and options fit the target population.

The COSMIN content-validity methodology, developed for patient-reported measures, distinguishes relevance, comprehensiveness, and comprehensibility. These questions also provide useful prompts for discussing the content of a social-research scale, with the population and context specified.

Expert ratings can organise this review. For example, a content-validity index may summarise the proportion of reviewers judging an item relevant under a defined rating rule. Retain reviewers’ explanations and specify panel composition and decision rules. A numerical summary should remain connected to what the panel actually examined.

Ask users to reveal their response process

In a cognitive interview, a participant can paraphrase an item, explain how they selected a response, and identify unfamiliar words. Use think-aloud techniques and focused probes. Examine comprehension, retrieval of relevant experiences, judgment, and mapping that judgment to an option.

Work in small, purposeful rounds

The lecture suggests groups of roughly 5–15 people across iterative rounds. Use small rounds to discover and investigate problems; recruit across relevant experiences, language use, and levels of familiarity. Continue or refine recruitment according to the questions still unresolved.

A pilot administration then checks the wider process: recruitment, instructions, survey flow, accessibility, timing, missing responses, and data export. Preserve consent and confidentiality arrangements, especially when items ask about sensitive experiences.

Plan the development samples

A development sample should represent the people and conditions for which the scores are intended. Its size must also support the proposed analyses, including item evaluation, factor exploration, and confirmation.

Define who the scale is for

Specify the target population, language, age range where relevant, recruitment setting, and expected use. For the running example, a sample drawn entirely from journalism students may offer limited evidence for an instrument intended for adult social-media users across varied educational backgrounds.

Seek coverage of relevant experiences and capability levels. Restricted variation can obscure item relationships and make a scale appear less informative. Document who was invited, who responded, and which groups were difficult to reach.

Justify size beyond a participants-per-item ratio

Required sample size depends on the number of factors and indicators, loading strength, communalities, response-category frequencies, estimator, missingness, and the precision or power required. MacCallum et al. (1999) show why simple fixed ratios cannot capture these conditions.

The lecture’s suggestions of 5–10 participants per item and around 300 participants are useful prompts for initial planning. Develop a study-specific justification. When feasible, use simulation or an appropriate model-based sample-size analysis and make assumptions explicit.

Protect the confirmatory stage

For a newly explored structure, plan an independent sample to test the proposed model. A new recruitment wave can also help examine replication across settings or time. A planned split of a sufficiently large dataset can support exploratory and confirmatory subsets, provided each subset has adequate information.

Describe the design accurately. Repeatedly revising a model on the same cases answers an exploratory question. Testing that final model on new cases gives a stronger assessment of whether it travels beyond the data that shaped it.

Plan the data record

Keep participant IDs separate from identifying details. Preserve raw item responses, missing-value codes, version numbers, language, administration mode, and the analysis sample used at each stage. Predefine exclusions and document justified changes.

Examine items and responses

Item analysis investigates how people use each item and how responses relate across the proposed scale. It helps identify scoring errors, ambiguity, redundancy, weak coverage, and limited discrimination among respondents.

Begin with distributions and data quality

Inspect missing responses, category frequencies, unusual patterns, and possible floor or ceiling effects. If nearly everyone selects the highest confidence category, consider whether the task is too easy for this population, whether the wording invites a socially desirable answer, or whether the response options need attention.

Check data entry and reverse scoring before interpreting correlations. An accidentally unreversed item can correlate negatively with its intended subscale. Record the correction and rerun the relevant analysis.

Use correlations as diagnostic evidence

Inter-item correlations show relationships among responses. Very weak relationships may signal different content, misunderstanding, or limited variation. Extremely similar items can produce strong relationships while adding little breadth.

A corrected item–total correlation relates an item to the score formed from the other relevant items. Excluding the item from that total avoids part–whole inflation. For a multidimensional scale, examine relationships with the intended subscale and the proposed factor structure.

Keep statistical flags beside the content blueprint. An item about a rarely represented but important task may need rewriting or better sampling. Removing it immediately can narrow the construct while improving a coefficient.

Distinguish rating scales from knowledge tests

In a right-or-wrong knowledge test, classical item difficulty is commonly the proportion answering correctly; a higher proportion indicates an easier item. Distractor analysis examines how incorrect options function. Point-biserial relationships can help assess discrimination for dichotomously scored items.

A confidence rating has ordered response categories and no objectively correct answer. Examine endorsement patterns and relationships appropriate to that format. Use the measurement task to choose the item-analysis method.

Locate item response theory in the wider toolkit

IRT models the probability of an item response conditional on a latent trait and item parameters. Suitable models can handle binary responses or ordered ratings and describe where items provide information along the trait.

Explore the factor structure

Exploratory factor analysis investigates how many latent dimensions may account for shared variation among items and how the items relate to those dimensions. Theory and item meaning guide the interpretation of the resulting structure.

Check the correlation structure

Inspect whether responses contain meaningful relationships, near-duplicate items, and patterns consistent with the proposed domain. For ordered ratings, consider whether polychoric correlations and suitable estimation methods are appropriate. Sparse response categories can make those correlations unstable.

Kaiser–Meyer–Olkin measures assess sampling adequacy for the correlation structure and range from 0 to 1. A value around .60 is often used as an initial screening guide. Bartlett’s test of sphericity examines whether the population correlation matrix is an identity matrix; a small p value indicates evidence against that hypothesis.

Choose common-factor methods for a latent-factor question

Use an appropriate extraction method, such as minimum residual, principal axis factoring, or maximum likelihood, with assumptions suited to the data. Principal components analysis summarises total observed variance through components; common-factor EFA targets shared variation through latent factors.

Decide how many factors to retain

Compare theoretical expectations, parallel analysis, a scree plot, and other appropriate evidence, such as a minimum average partial procedure. Examine whether candidate solutions yield interpretable factors with adequate indicators. Record the alternatives considered and why one solution was preferred.

For confidence facets likely to relate, an oblique rotation, such as oblimin or promax, allows factor correlations. Interpret the pattern matrix alongside the factor correlations and, where useful, the structure matrix. These matrices answer different questions when factors correlate.

Make item decisions with content in view

A loading indicates an item’s relationship with a factor under the model. A cross-loading indicates a relationship with more than one factor. Screening values, such as .40 for a loading, can organise inspection; their usefulness depends on the model, sample, item content, and uncertainty.

On a small screen, swipe the table horizontally.

Return to wording and pretest evidence before revising. Keep a decision log that records the statistical signal, content judgment, action, and effect on domain coverage. Boateng et al. (2018) provide a wider development framework connecting these stages.

Test the proposed model

Confirmatory factor analysis evaluates a specified measurement model: which items indicate which factors, how factors relate, and which additional parameters are justified. For a newly developed scale, an independent sample offers a stronger test of the structure suggested during exploration.

Specify the model and estimator

Explain the theoretical rationale, item–factor assignments, factor correlations, identification choices, and estimator. Use methods suited to the response format and distribution. Ordered-category CFA can use estimators such as WLSMV; continuous-variable approaches require their own justification.

The official lavaan categorical-data tutorial shows how declaring indicators as ordered changes estimation. Whatever software you use, retain the model specification, item ordering, estimator settings, missing-data method, and relevant output.

Read global fit and local problems

Inspect loadings, uncertainty, factor correlations, residuals, improper estimates, and overall fit. Look for convergence problems or negative residual variances. Compare theoretically defensible alternatives rather than searching blindly for the best-fitting arrangement.

On a small screen, swipe the table horizontally.

Common reference values include CFI/TLI near .95, RMSEA near .06, and SRMR near .08. Hu and Bentler (1999) studied combinations of fit indices under specified simulation conditions. Interpret reference values in relation to your model and estimator.

Estimate reliability

Reliability concerns the consistency or precision of scores under specified measurement conditions. Select an estimate that matches the score, measurement model, population, and source of variation you need to examine.

Examine internal consistency after structure

Internal consistency addresses relationships among items contributing to a score. Report it for a defensible scale or subscale, with evidence about dimensionality. For the candidate three-factor model, begin with the three proposed subscales. An overall total requires an additional argument about what it represents.

Cronbach’s alpha summarises a function of item variances and covariances. Its interpretation as reliability depends on assumptions, including an appropriate measurement structure and error relationships. Alpha can increase with the number of items and with redundant wording.

Choose the appropriate omega

McDonald’s omega uses a factor model to estimate reliability for a specified score. Different omega coefficients address different interpretations, especially in hierarchical or bifactor models. State which coefficient you report, its model and estimator, and any uncertainty estimate. Flora (2020) explains why this choice needs care.

A value around .70 is often quoted for early research, but adequacy depends on purpose and required precision. Avoid treating one threshold as sufficient for every use. Extremely high internal consistency can prompt an examination of item duplication and narrow coverage.

Examine stability when the purpose requires it

Test–retest work studies scores across a justified interval under conditions in which the construct is expected to remain reasonably stable. State the interval, intervening events, attrition, and the type of consistency or agreement evaluated. An appropriate intraclass correlation can address agreement or consistency, depending on its specification.

A simple correlation mainly describes rank ordering and can remain high when all scores shift. Training may deliberately change verification confidence, so evaluating change requires a design that distinguishes real change from measurement error and altered interpretation.

Build the validity argument

Validation evaluates whether theory and evidence support the proposed interpretation and use of scores. It grows across studies, populations, settings, and decisions, with each source of evidence addressing a particular part of the argument.

The Standards for Educational and Psychological Testing frame validity around proposed score interpretations and uses. For a research scale, state what a score means and then identify the evidence needed to support that meaning.

Specify predictions before testing relationships

For news-verification confidence, predict a positive relationship with a suitable task-specific confidence measure. Predict how the scale should relate to demonstrated verification performance, general digital confidence, and unrelated attributes. Explain expected directions and plausible strengths from theory.

Convergent evidence concerns relationships expected for the same or closely related constructs. Discriminant evidence examines whether the scale remains distinguishable from constructs that theory treats as different. Related constructs may reasonably correlate; distinctiveness requires a reasoned pattern rather than a demand for every correlation to approach zero.

Use model-based diagnostics in context

In appropriate reflective measurement models, average variance extracted, or AVE, summarises variance captured by a factor relative to indicator error. A value of .50 is a common reference. The Fornell–Larcker approach compares the square root of each factor’s AVE with its correlations with other factors.

HTMT, the heterotrait–monotrait ratio, compares relationships across constructs with those within constructs. Henseler et al. (2015) proposed it for discriminant-validity assessment in variance-based SEM. Values such as .85 or .90 are commonly used references; justify the criterion and examine uncertainty and model suitability.

Define criterion relationships carefully

Criterion-related evidence examines relationships with a defensible external criterion. A concurrent study measures it at roughly the same time; a predictive study assesses a future outcome. Choose the statistical model according to the outcome and design.

A similar self-report scale administered at the same time primarily offers convergent evidence unless it genuinely serves as an appropriate criterion. For confidence, actual verification performance is a separate attribute whose relationship needs a theoretical rationale; perfect agreement between confidence and skill would be an unrealistic requirement.

Examine adaptation and group comparisons

For another language or setting, review conceptual relevance, translate and reconcile wording, test comprehension, and examine measurement properties in the target population. Keep version changes visible.

Measurement invariance examines whether specified measurement relationships are sufficiently comparable across groups or occasions. Configural, metric, and scalar or threshold models address progressively different comparison questions, with details depending on the item model. Putnick and Bornstein (2016) provide guidance on conventions and reporting.

Use evidence appropriate to the comparison: latent mean comparisons require suitable intercept or threshold comparability. Differential item functioning asks whether item responses differ across groups conditional on the latent attribute; a raw item-mean difference alone cannot establish it.

Score and interpret

A scoring system states which items contribute, how they are coded, how missing responses are handled, and what the resulting number supports. The measurement evidence should justify both the score’s construction and its interpretation.

Choose the score that matches the structure

A sum or mean can be practical for a sufficiently supported scale or subscale. Document the item set, direction, range, and any weighting. Factor scores are model-based estimates with their own estimation method and uncertainty; preserve the model and scoring procedure used.

For a multidimensional instrument, report meaningful subscale scores. Combine them into a total only when theory and evidence support that total’s interpretation. Item deletion or altered response options creates a version that may need new evaluation.

Apply reverse coding before combining items

For a reverse-keyed item coded from 1 to 5, the recoded response is 6 − original response. The following four-item example is separate from the running item pool and illustrates scoring only.

State a missing-response rule

For teaching, suppose a four-item score permits a mean of available responses when at least three items are answered. Scored responses 4, 3, 5, and missing yield 4.00 under that rule. An actual instrument needs a justified, evaluated rule and disclosure of missingness; the rule should match its manual and intended use.

Distinguish administration, standard scores, and norms

Standardised administration uses consistent instructions, delivery, and scoring. Standard scores express a result on a stated transformed metric. Norms locate scores within a relevant reference population.

In a hypothetical reference group with M = 12 and SD = 4, a score of 16 gives z = 1 and T = 60. Calculate empirical percentiles from the reference distribution. State the norm sample, population, date, version, and scoring method.

A z transformation changes the metric while preserving the distribution’s shape. Norming can use the observed reference distribution; transformations such as Box–Cox require a separate methodological rationale. Research scales do not automatically need norms or individual cutoffs.

Prepare the manual and reporting record

Provide the definition, final items and response labels, target users, administration, scoring, missing-data rules, structure, reliability estimates, validity evidence, permissions, and limits. Report the item-selection history, sample design, estimator, alternative models, and changes made after analysis.

Watch and practise

Use the lecture to follow the complete development route. Pause at each stage and write down the evidence you would need before moving forward with your own item pool.

Master Scale Development: From Theory to Validation

Dr. USP · 14:00 · Open the lecture on YouTube. The chapter links below follow the video’s published chapter markers.

Glossary and reading

Use the glossary to keep the measurement argument coherent. The readings provide development guidance, testing standards, and specialist explanations of the main statistical decisions.

Construct

A theoretically defined attribute investigated through indicators.

Item pool

The candidate questions or statements considered during development.

Content blueprint

A map linking proposed facets to the content represented by items.

Cognitive pretesting

Investigation of how target users interpret items and form responses.

Latent factor

An unobserved variable used to explain shared response variation.

Loading

A model-based relationship between an item and a factor.

Cross-loading

An item’s modelled relationship with more than one factor.

Reliability

Consistency or precision of scores under specified conditions.

Validation

Evaluation of evidence supporting score interpretations and uses.

Measurement invariance

Comparability of specified measurement relationships across groups or occasions.

Norms

A reference distribution used to interpret scores relative to a defined population.

References and primary resources

1. Dr. USP. (2025, April 15). Master scale development: From theory to validation [Video]. YouTube. Watch the lecture.
2. Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, Article 149. https://doi.org/10.3389/fpubh.2018.00149
3. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association. Official open-access editions.
4. COSMIN. Methodology for assessing the content validity of patient-reported outcome measures: User manual. Read the methodology and criteria.Use its content-review questions with attention to its original health-measurement scope.
5. MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84–99. https://doi.org/10.1037/1082-989X.4.1.84
6. Flora, D. B. (2020). Your coefficient alpha is probably wrong, but which coefficient omega is right? A tutorial on using R to obtain better reliability estimates. Advances in Methods and Practices in Psychological Science, 3(4), 484–501. https://doi.org/10.1177/2515245920951747
7. Hu, L.-t., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
8. Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50. https://doi.org/10.1177/002224378101800104
9. Henseler, J., Ringle, C. M., & Sarstedt, M. (2015). A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science, 43(1), 115–135. https://doi.org/10.1007/s11747-014-0403-8
10. Putnick, D. L., & Bornstein, M. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review, 41, 71–90. https://doi.org/10.1016/j.dr.2016.06.004 · Accessible author manuscript.
11. The lavaan project. Categorical data. Official ordinal-CFA tutorial.Estimator settings for ordered indicators; retain the software version used in an actual study.

Return to the top ↑