Unit 3 of 4 · MBA Sem 3

Unit 3: Questionnaire design, reliability and validity

Marketing Research notes · PTU syllabus (MBA 302-18)

4 min read8 topics10 exam questions
On this page
  1. Unit summary
  2. Principles of writing questionnaires
  3. Question wording and sequence
  4. The true score model and measurement errors
  5. Nominal, ordinal, interval and ratio scales
  6. Scaling techniques
  7. Reliability and coefficient alpha
  8. Validity
  9. Generalisability
  10. Key terms
  11. Quick revision
  12. Important questions

Unit summary

A questionnaire is only as good as its questions, and a scale is useful only if it is reliable and valid. This unit covers the principles of writing questionnaires, the true score model, measurement errors, nominal, ordinal, interval and ratio scales, coefficient alpha and internal consistency, types of reliability and validity, and generalisability.

After this unit you can

  • Apply the principles of questionnaire design
  • Explain the true score model and measurement errors
  • Explain levels of measurement and scaling techniques
  • Assess reliability, validity and generalisability

PTU syllabus topics

  • Principles of writing questionnaires
  • true score model
  • measurement errors
  • nominal/ordinal/interval/ratio scales
  • coefficient alpha and internal consistency
  • types of reliability and validity
  • generalizability
1

Topic 1

Principles of writing questionnaires

A good questionnaire is short, clear, unbiased and easy for both the respondent and the analyst.

  • Structured (fixed questions and options) vs unstructured (open questions).
  • Disguised (purpose hidden) vs non-disguised (purpose clear).
  • Administered personally, by mail, by telephone or online (Google Forms).
  • Question types: open-ended, closed (dichotomous yes/no, multiple choice) and scale-based.

Example

Fixing a poor question: "Don't you agree that our fast service is excellent?" is leading. A neutral version is: "How would you rate the speed of our service?" with a five-point scale from very poor to very good.

Exam tip

Learn the common errors: leading, double-barrelled, ambiguous and loaded questions.

2

Topic 2

Question wording and sequence

Sequencing rules: start with simple, interesting questions; follow a funnel approach (general to specific); put sensitive and classification questions (income, age) at the end; and use branching (skip) questions clearly.

ProcessDesigning a questionnaire
  1. 1

    Decide the information needed

  2. 2

    Choose the method of administration

  3. 3

    Decide question content and type

  4. 4

    Word questions carefully

  5. 5

    Sequence them logically

    Easy, general first; sensitive last

  6. 6

    Design the layout

  7. 7

    Pre-test (pilot) and revise

Avoid: leading questions ("Don't you agree our product is the best?"), double-barrelled questions (two questions in one), jargon, ambiguous words, and questions that rely on memory or are too personal.

Example

A double-barrelled question: "How satisfied are you with the price and quality of our product?" asks two things at once. Split it into two questions, one on price and one on quality.

Exam tip

Always pre-test the questionnaire on a small sample before the full survey.

3

Topic 3

The true score model and measurement errors

The true score model explains why measurements are never perfect: every observed score contains error, which is either systematic (consistent bias, affects validity) or random (varies by chance, affects reliability).

Key formulasTrue score model
  • Classical test theory

    Observed score (X) = True score (T) + Error (E)

  • Error components

    E = Systematic error + Random error

  • Systematic error (bias): affects measurement consistently in one direction — leading questions, poorly calibrated scales, social desirability. It threatens validity.
  • Random error: varies unpredictably — mood, fatigue, ambiguous wording, recording mistakes. It threatens reliability.
ClassificationSources of measurement error
Measurement error
  • Respondent

    Fatigue, mood, guessing, social desirability

  • Situation

    Interviewer presence, distractions, privacy

  • Interviewer

    Wording changes, recording errors, cues

  • Instrument

    Ambiguous, double-barrelled or leading questions; poor scale

  • A measure can be reliable without being valid, but cannot be valid unless it is reliable.

Example

Two kinds of error: a weighing scale that always shows 1 kg extra has systematic error — results are consistent but wrong. A survey answered carelessly by a tired respondent has random error — repeating it would give a different score.

Exam tip

Link: random error → reliability; systematic error → validity.

4

Topic 4

Nominal, ordinal, interval and ratio scales

The level of measurement decides which statistics are allowed: mode for nominal, median for ordinal, mean for interval, and all statistics including ratios for ratio data.

ComparisonLevels of measurement
Property
Example

Nominal

Labels or categories only

Gender, city, brand used

Ordinal

Order or rank

Rank of brands, education level

Interval

Equal intervals, no true zero

Temperature in °C, rating scales

Ratio

Equal intervals and a true zero

Income, age, sales

Higher levels allow more statistical analysis: mode for nominal, median for ordinal, mean for interval and ratio.

Example

One survey, four scales: "Which city do you live in?" (nominal); "Rank these five brands" (ordinal); "Rate the service from 1 to 7" (treated as interval); "How many rupees did you spend last month?" (ratio — it has a true zero).

Exam tip

Remember NOIR: Nominal, Ordinal, Interval, Ratio — each adds a property to the one before.

5

Topic 5

Scaling techniques

GraphFive-point Likert scale
StronglydisagreeDisagreeNeutralAgreeStronglyagree12345

Likert scales are ordinal; many researchers treat summed scores as interval data for analysis

ClassificationScaling techniques
Scales
  • Comparative: paired comparison

    Choose between two at a time

  • Comparative: rank order

    Rank several items

  • Comparative: constant sum

    Divide 100 points among items

  • Non-comparative: Likert

    Agreement on a 5- or 7-point scale

  • Non-comparative: semantic differential

    Between bipolar adjectives (cheap–expensive)

  • Non-comparative: Stapel and graphic rating

    −5 to +5, or a marked line

Example

Likert item: "The staff were helpful." Strongly disagree (1) — Disagree (2) — Neutral (3) — Agree (4) — Strongly agree (5).

Uni-dimensional vs multidimensional scaling

  • Uni-dimensional scales measure one attribute (satisfaction with service speed) — Likert, Thurstone, Guttman.
  • Multidimensional scaling (MDS) represents perceptions of objects on several dimensions simultaneously as a perceptual map (brands positioned by price and quality).
6

Topic 6

Reliability and coefficient alpha

Cronbach's alpha measures internal consistency — how well the items of a scale measure the same thing. It ranges from 0 to 1; a value of 0.7 or above is generally acceptable. Reliability is the extent to which a scale gives consistent results on repeated measurement.

ClassificationTypes of reliability
Reliability
  • Test–retest

    Same respondents, same scale, two points in time — correlation between scores

  • Alternative (parallel) forms

    Two equivalent versions of the scale

  • Internal consistency

    Items of a scale measuring the same construct agree — split-half and coefficient alpha

  • Inter-rater

    Agreement between different observers or coders

Key formulasCronbach's coefficient alpha
  • Alpha

    α = (k ÷ (k − 1)) × (1 − Σ item variances ÷ variance of total score)

  • Interpretation

    0.7 or above acceptable; 0.8 good; 0.9 excellent; below 0.6 poor

  • Split-half reliability: correlate scores on two halves of the items; adjust with the Spearman–Brown formula.

Example

Checking a scale: a five-item customer satisfaction scale gives α = 0.82, so the items are consistent. Another scale gives α = 0.55; deleting one confusing item raises it to 0.74.

Exam tip

Test–retest, alternative forms and internal consistency are the three types of reliability.

7

Topic 7

Validity

A scale can be reliable without being valid, but it cannot be valid without being reliable. Validity is the extent to which a scale measures what it is intended to measure.

ClassificationTypes of validity
Validity
  • Content (face) validity

    Experts judge that items cover the construct

  • Criterion validity

    Correlates with a criterion — concurrent (same time) or predictive (future)

  • Construct validity

    Measures the theoretical construct — convergent (correlates with related measures) and discriminant (does not correlate with unrelated ones)

  • Nomological validity

    Relates to other constructs as theory predicts

  • Construct validity is the most sophisticated and is assessed through factor analysis, average variance extracted (AVE above 0.5) and correlation patterns.

Example

Reliable but not valid: a questionnaire meant to measure brand loyalty actually measures price sensitivity. It gives the same answers every time (reliable) but does not measure loyalty (not valid).

Exam tip

Name the types: content, criterion (concurrent, predictive) and construct (convergent, discriminant) validity.

8

Topic 8

Generalisability

Generalisability matters because a study is useful only if its findings hold beyond the specific sample, place and time studied.

  • Generalisability theory: the extent to which results from a sample of items, respondents, occasions or settings can be generalised to the universe of interest.
  • Improving generalisability: probability sampling, adequate sample size, multiple items and occasions, replication across settings.

Example

Limited generalisability: a survey of only engineering students in one college finds strong interest in a new app. The result may not generalise to working professionals or other cities, so the sample must be widened before a national decision.

Exam tip

Larger, more representative samples and repeated studies across settings improve generalisability.

Key terms

True score model
Observed score equals true score plus error
Systematic error
Constant bias affecting measurement in one direction
Cronbach's alpha
Measure of internal consistency reliability
Construct validity
Extent to which a scale measures the intended construct
Generalisability
Extent to which findings extend beyond the study conditions

Quick revision

  • Questionnaire: objectives, question types, wording, sequence, layout, pre-test.
  • X = T + E; systematic (validity) and random (reliability) error.
  • NOIR scales; comparative and non-comparative scaling (Likert, semantic differential, Stapel).
  • Reliability: test–retest, alternative forms, internal consistency (alpha ≥ 0.7), inter-rater.
  • Validity: content, criterion, construct (convergent, discriminant); generalisability.

Important exam questions

Practice questions written to the PTU exam pattern for this unit's syllabus: short answers (Section A style) and long answers (Sections B and C style).

Short-answer questions

  1. Q1.State four principles of question wording.
  2. Q2.What is the true score model?
  3. Q3.Distinguish systematic and random error.
  4. Q4.What is Cronbach's alpha and its acceptable value?
  5. Q5.Distinguish convergent and discriminant validity.
  6. Q6.What is generalisability?

Long-answer questions

  1. Q1.Explain the principles of questionnaire design.
  2. Q2.Explain the true score model and sources of measurement error.
  3. Q3.Explain the types of reliability and the computation of coefficient alpha.
  4. Q4.Discuss the types of validity and generalisability of measurement.

Stuck on this unit?

Message SBS on WhatsApp for help with Marketing Research, or to ask about studying MBA at Synetic.

Planning your MBA in Ludhiana? See the MBA at SBS →

WhatsApp us