Unit 3: Questionnaire design, reliability and validity
Marketing Research notes · PTU syllabus (MBA 302-18)
On this page
Unit summary
A questionnaire is only as good as its questions, and a scale is useful only if it is reliable and valid. This unit covers the principles of writing questionnaires, the true score model, measurement errors, nominal, ordinal, interval and ratio scales, coefficient alpha and internal consistency, types of reliability and validity, and generalisability.
After this unit you can
- Apply the principles of questionnaire design
- Explain the true score model and measurement errors
- Explain levels of measurement and scaling techniques
- Assess reliability, validity and generalisability
PTU syllabus topics
- Principles of writing questionnaires
- true score model
- measurement errors
- nominal/ordinal/interval/ratio scales
- coefficient alpha and internal consistency
- types of reliability and validity
- generalizability
Topic 1
Principles of writing questionnaires
A good questionnaire is short, clear, unbiased and easy for both the respondent and the analyst.
- Structured (fixed questions and options) vs unstructured (open questions).
- Disguised (purpose hidden) vs non-disguised (purpose clear).
- Administered personally, by mail, by telephone or online (Google Forms).
- Question types: open-ended, closed (dichotomous yes/no, multiple choice) and scale-based.
Example
Fixing a poor question: "Don't you agree that our fast service is excellent?" is leading. A neutral version is: "How would you rate the speed of our service?" with a five-point scale from very poor to very good.
Exam tip
Learn the common errors: leading, double-barrelled, ambiguous and loaded questions.
Topic 2
Question wording and sequence
Sequencing rules: start with simple, interesting questions; follow a funnel approach (general to specific); put sensitive and classification questions (income, age) at the end; and use branching (skip) questions clearly.
- 1
Decide the information needed
- 2
Choose the method of administration
- 3
Decide question content and type
- 4
Word questions carefully
- 5
Sequence them logically
Easy, general first; sensitive last
- 6
Design the layout
- 7
Pre-test (pilot) and revise
Avoid: leading questions ("Don't you agree our product is the best?"), double-barrelled questions (two questions in one), jargon, ambiguous words, and questions that rely on memory or are too personal.
Example
A double-barrelled question: "How satisfied are you with the price and quality of our product?" asks two things at once. Split it into two questions, one on price and one on quality.
Exam tip
Always pre-test the questionnaire on a small sample before the full survey.
Topic 3
The true score model and measurement errors
The true score model explains why measurements are never perfect: every observed score contains error, which is either systematic (consistent bias, affects validity) or random (varies by chance, affects reliability).
Classical test theory
Observed score (X) = True score (T) + Error (E)
Error components
E = Systematic error + Random error
- Systematic error (bias): affects measurement consistently in one direction — leading questions, poorly calibrated scales, social desirability. It threatens validity.
- Random error: varies unpredictably — mood, fatigue, ambiguous wording, recording mistakes. It threatens reliability.
Respondent
Fatigue, mood, guessing, social desirability
Situation
Interviewer presence, distractions, privacy
Interviewer
Wording changes, recording errors, cues
Instrument
Ambiguous, double-barrelled or leading questions; poor scale
- A measure can be reliable without being valid, but cannot be valid unless it is reliable.
Example
Two kinds of error: a weighing scale that always shows 1 kg extra has systematic error — results are consistent but wrong. A survey answered carelessly by a tired respondent has random error — repeating it would give a different score.
Exam tip
Link: random error → reliability; systematic error → validity.
Topic 4
Nominal, ordinal, interval and ratio scales
The level of measurement decides which statistics are allowed: mode for nominal, median for ordinal, mean for interval, and all statistics including ratios for ratio data.
Nominal
Labels or categories only
Gender, city, brand used
Ordinal
Order or rank
Rank of brands, education level
Interval
Equal intervals, no true zero
Temperature in °C, rating scales
Ratio
Equal intervals and a true zero
Income, age, sales
Higher levels allow more statistical analysis: mode for nominal, median for ordinal, mean for interval and ratio.
Example
One survey, four scales: "Which city do you live in?" (nominal); "Rank these five brands" (ordinal); "Rate the service from 1 to 7" (treated as interval); "How many rupees did you spend last month?" (ratio — it has a true zero).
Exam tip
Remember NOIR: Nominal, Ordinal, Interval, Ratio — each adds a property to the one before.
Topic 5
Scaling techniques
Likert scales are ordinal; many researchers treat summed scores as interval data for analysis
Comparative: paired comparison
Choose between two at a time
Comparative: rank order
Rank several items
Comparative: constant sum
Divide 100 points among items
Non-comparative: Likert
Agreement on a 5- or 7-point scale
Non-comparative: semantic differential
Between bipolar adjectives (cheap–expensive)
Non-comparative: Stapel and graphic rating
−5 to +5, or a marked line
Example
Likert item: "The staff were helpful." Strongly disagree (1) — Disagree (2) — Neutral (3) — Agree (4) — Strongly agree (5).
Uni-dimensional vs multidimensional scaling
- Uni-dimensional scales measure one attribute (satisfaction with service speed) — Likert, Thurstone, Guttman.
- Multidimensional scaling (MDS) represents perceptions of objects on several dimensions simultaneously as a perceptual map (brands positioned by price and quality).
Topic 6
Reliability and coefficient alpha
Cronbach's alpha measures internal consistency — how well the items of a scale measure the same thing. It ranges from 0 to 1; a value of 0.7 or above is generally acceptable. Reliability is the extent to which a scale gives consistent results on repeated measurement.
Test–retest
Same respondents, same scale, two points in time — correlation between scores
Alternative (parallel) forms
Two equivalent versions of the scale
Internal consistency
Items of a scale measuring the same construct agree — split-half and coefficient alpha
Inter-rater
Agreement between different observers or coders
Alpha
α = (k ÷ (k − 1)) × (1 − Σ item variances ÷ variance of total score)
Interpretation
0.7 or above acceptable; 0.8 good; 0.9 excellent; below 0.6 poor
- Split-half reliability: correlate scores on two halves of the items; adjust with the Spearman–Brown formula.
Example
Checking a scale: a five-item customer satisfaction scale gives α = 0.82, so the items are consistent. Another scale gives α = 0.55; deleting one confusing item raises it to 0.74.
Exam tip
Test–retest, alternative forms and internal consistency are the three types of reliability.
Topic 7
Validity
A scale can be reliable without being valid, but it cannot be valid without being reliable. Validity is the extent to which a scale measures what it is intended to measure.
Content (face) validity
Experts judge that items cover the construct
Criterion validity
Correlates with a criterion — concurrent (same time) or predictive (future)
Construct validity
Measures the theoretical construct — convergent (correlates with related measures) and discriminant (does not correlate with unrelated ones)
Nomological validity
Relates to other constructs as theory predicts
- Construct validity is the most sophisticated and is assessed through factor analysis, average variance extracted (AVE above 0.5) and correlation patterns.
Example
Reliable but not valid: a questionnaire meant to measure brand loyalty actually measures price sensitivity. It gives the same answers every time (reliable) but does not measure loyalty (not valid).
Exam tip
Name the types: content, criterion (concurrent, predictive) and construct (convergent, discriminant) validity.
Topic 8
Generalisability
Generalisability matters because a study is useful only if its findings hold beyond the specific sample, place and time studied.
- Generalisability theory: the extent to which results from a sample of items, respondents, occasions or settings can be generalised to the universe of interest.
- Improving generalisability: probability sampling, adequate sample size, multiple items and occasions, replication across settings.
Example
Limited generalisability: a survey of only engineering students in one college finds strong interest in a new app. The result may not generalise to working professionals or other cities, so the sample must be widened before a national decision.
Exam tip
Larger, more representative samples and repeated studies across settings improve generalisability.
Key terms
- True score model
- Observed score equals true score plus error
- Systematic error
- Constant bias affecting measurement in one direction
- Cronbach's alpha
- Measure of internal consistency reliability
- Construct validity
- Extent to which a scale measures the intended construct
- Generalisability
- Extent to which findings extend beyond the study conditions
Quick revision
- Questionnaire: objectives, question types, wording, sequence, layout, pre-test.
- X = T + E; systematic (validity) and random (reliability) error.
- NOIR scales; comparative and non-comparative scaling (Likert, semantic differential, Stapel).
- Reliability: test–retest, alternative forms, internal consistency (alpha ≥ 0.7), inter-rater.
- Validity: content, criterion, construct (convergent, discriminant); generalisability.
Important exam questions
Practice questions written to the PTU exam pattern for this unit's syllabus: short answers (Section A style) and long answers (Sections B and C style).
Short-answer questions
- Q1.State four principles of question wording.
- Q2.What is the true score model?
- Q3.Distinguish systematic and random error.
- Q4.What is Cronbach's alpha and its acceptable value?
- Q5.Distinguish convergent and discriminant validity.
- Q6.What is generalisability?
Long-answer questions
- Q1.Explain the principles of questionnaire design.
- Q2.Explain the true score model and sources of measurement error.
- Q3.Explain the types of reliability and the computation of coefficient alpha.
- Q4.Discuss the types of validity and generalisability of measurement.
Stuck on this unit?
Message SBS on WhatsApp for help with Marketing Research, or to ask about studying MBA at Synetic.
