Unit 3: Questionnaire design, reliability and validity
Marketing Research notes · PTU syllabus (MBA 302-18)
On this page
Unit summary
A questionnaire is only as good as its questions, and a scale is useful only if it is reliable and valid. This unit covers the principles of writing questionnaires, the true score model, measurement errors, nominal, ordinal, interval and ratio scales, coefficient alpha and internal consistency, types of reliability and validity, and generalisability.
After this unit you can
- Apply the principles of questionnaire design
- Explain the true score model and measurement errors
- Explain levels of measurement and scaling techniques
- Assess reliability, validity and generalisability
PTU syllabus topics
- Principles of writing questionnaires
- true score model
- measurement errors
- nominal/ordinal/interval/ratio scales
- coefficient alpha and internal consistency
- types of reliability and validity
- generalizability
Question
Is the measure consistent?
Does it measure the right thing?
Types
Test-retest, split-half, internal consistency
Content, construct, criterion
Statistic
Cronbach's alpha (≥ 0.7 is acceptable)
Correlations with criteria, factor loadings
Relationship
Needed for validity
Implies reliability
Topic 1
Principles of writing questionnaires
- Structured (fixed questions and options) vs unstructured (open questions).
- Disguised (purpose hidden) vs non-disguised (purpose clear).
- Administered personally, by mail, by telephone or online (Google Forms).
- Question types: open-ended, closed (dichotomous yes/no, multiple choice) and scale-based.
Topic 2
Question wording and sequence
- 1
Decide the information needed
- 2
Choose the method of administration
- 3
Decide question content and type
- 4
Word questions carefully
- 5
Sequence them logically
Easy, general first; sensitive last
- 6
Design the layout
- 7
Pre-test (pilot) and revise
Avoid: leading questions ("Don't you agree our product is the best?"), double-barrelled questions (two questions in one), jargon, ambiguous words, and questions that rely on memory or are too personal.
Topic 3
The true score model and measurement errors
Classical test theory
Observed score (X) = True score (T) + Error (E)
Error components
E = Systematic error + Random error
- Systematic error (bias): affects measurement consistently in one direction — leading questions, poorly calibrated scales, social desirability. It threatens validity.
- Random error: varies unpredictably — mood, fatigue, ambiguous wording, recording mistakes. It threatens reliability.
Respondent
Fatigue, mood, guessing, social desirability
Situation
Interviewer presence, distractions, privacy
Interviewer
Wording changes, recording errors, cues
Instrument
Ambiguous, double-barrelled or leading questions; poor scale
- A measure can be reliable without being valid, but cannot be valid unless it is reliable.
Topic 4
Nominal, ordinal, interval and ratio scales
Nominal
Labels or categories only
Gender, city, brand used
Ordinal
Order or rank
Rank of brands, education level
Interval
Equal intervals, no true zero
Temperature in °C, rating scales
Ratio
Equal intervals and a true zero
Income, age, sales
Higher levels allow more statistical analysis: mode for nominal, median for ordinal, mean for interval and ratio.
Topic 5
Scaling techniques
Comparative: paired comparison
Choose between two at a time
Comparative: rank order
Rank several items
Comparative: constant sum
Divide 100 points among items
Non-comparative: Likert
Agreement on a 5- or 7-point scale
Non-comparative: semantic differential
Between bipolar adjectives (cheap–expensive)
Non-comparative: Stapel and graphic rating
−5 to +5, or a marked line
Example
Likert item: "The staff were helpful." Strongly disagree (1) — Disagree (2) — Neutral (3) — Agree (4) — Strongly agree (5).
Uni-dimensional vs multidimensional scaling
- Uni-dimensional scales measure one attribute (satisfaction with service speed) — Likert, Thurstone, Guttman.
- Multidimensional scaling (MDS) represents perceptions of objects on several dimensions simultaneously as a perceptual map (brands positioned by price and quality).
Topic 6
Reliability and coefficient alpha
Reliability is the extent to which a scale gives consistent results on repeated measurement.
Test–retest
Same respondents, same scale, two points in time — correlation between scores
Alternative (parallel) forms
Two equivalent versions of the scale
Internal consistency
Items of a scale measuring the same construct agree — split-half and coefficient alpha
Inter-rater
Agreement between different observers or coders
Alpha
α = (k ÷ (k − 1)) × (1 − Σ item variances ÷ variance of total score)
Interpretation
0.7 or above acceptable; 0.8 good; 0.9 excellent; below 0.6 poor
- Split-half reliability: correlate scores on two halves of the items; adjust with the Spearman–Brown formula.
Topic 7
Validity
Validity is the extent to which a scale measures what it is intended to measure.
Content (face) validity
Experts judge that items cover the construct
Criterion validity
Correlates with a criterion — concurrent (same time) or predictive (future)
Construct validity
Measures the theoretical construct — convergent (correlates with related measures) and discriminant (does not correlate with unrelated ones)
Nomological validity
Relates to other constructs as theory predicts
- Construct validity is the most sophisticated and is assessed through factor analysis, average variance extracted (AVE above 0.5) and correlation patterns.
Topic 8
Generalisability
- Generalisability theory: the extent to which results from a sample of items, respondents, occasions or settings can be generalised to the universe of interest.
- Improving generalisability: probability sampling, adequate sample size, multiple items and occasions, replication across settings.
Key terms
- True score model
- Observed score equals true score plus error
- Systematic error
- Constant bias affecting measurement in one direction
- Cronbach's alpha
- Measure of internal consistency reliability
- Construct validity
- Extent to which a scale measures the intended construct
- Generalisability
- Extent to which findings extend beyond the study conditions
Quick revision
- Questionnaire: objectives, question types, wording, sequence, layout, pre-test.
- X = T + E; systematic (validity) and random (reliability) error.
- NOIR scales; comparative and non-comparative scaling (Likert, semantic differential, Stapel).
- Reliability: test–retest, alternative forms, internal consistency (alpha ≥ 0.7), inter-rater.
- Validity: content, criterion, construct (convergent, discriminant); generalisability.
Important exam questions
Practice questions written to the PTU exam pattern for this unit's syllabus: short answers (Section A style) and long answers (Sections B and C style).
Short-answer questions
- Q1.State four principles of question wording.
- Q2.What is the true score model?
- Q3.Distinguish systematic and random error.
- Q4.What is Cronbach's alpha and its acceptable value?
- Q5.Distinguish convergent and discriminant validity.
- Q6.What is generalisability?
Long-answer questions
- Q1.Explain the principles of questionnaire design.
- Q2.Explain the true score model and sources of measurement error.
- Q3.Explain the types of reliability and the computation of coefficient alpha.
- Q4.Discuss the types of validity and generalisability of measurement.
Stuck on this unit?
Message SBS on WhatsApp for help with Marketing Research, or to ask about studying MBA at Synetic.