Unit 4 of 4 · MBA Sem 3

Unit 4: Evaluation and validation

Data Sciences Using R notes · PTU syllabus (MBA 962-18)

4 min read6 topics10 exam questions
On this page
  1. Unit summary
  2. Holdout method
  3. Cross-validation
  4. Bootstrap method
  5. Confusion matrix
  6. Statistical significance of data mining results
  7. Advanced topics
  8. Key terms
  9. Quick revision
  10. Important questions

Unit summary

A model is only useful if its performance on new data is known and significant. This unit covers methods for estimating classifier performance — cross-validation, the holdout method, the bootstrap method and the confusion matrix — assessing the statistical significance of data mining results, and advanced topics: scalable machine learning, big data techniques, stream data mining and social network analysis.

After this unit you can

  • Estimate classifier performance with holdout, cross-validation and bootstrap
  • Interpret the confusion matrix and related metrics
  • Assess the statistical significance of data mining results
  • Explain scalable ML, big data techniques, stream mining and social networks

PTU syllabus topics

  • Methods for estimating classifier performance — cross-validation
  • holdout method
  • bootstrap method
  • confusion matrix
  • assessing statistical significance of data mining results
  • advanced topics (scalable ML, big data techniques, stream data mining, social networks)
1

Topic 1

Holdout method

In the holdout method, data is split into a training set (often 70–80%) to build the model and a test set (20–30%) to measure performance on unseen data.

GraphHoldout split and 5-fold cross-validation
Training (70%)Test (30%)Fold 1Fold 2Fold 3Fold 4Fold 5

Cross-validation rotates the validation fold, so every record is used for both training and testing

  • Split data into training (e.g., 70%) and test (30%) sets; build the model on training data and evaluate on test data.
  • Stratified sampling keeps class proportions similar in both sets.
  • Limitation: results depend on one random split; wasteful with small data.
  • In R: caret::createDataPartition(y, p = 0.7).

Example

A holdout split in R: idx <- sample(nrow(df), 0.7 * nrow(df)); train <- df[idx, ]; test <- df[-idx, ]. A model with 95% accuracy on training data but 70% on test data is overfitting.

Exam tip

Always evaluate on data the model has not seen; set a seed (set.seed()) so the split can be repeated.

2

Topic 2

Cross-validation

k-fold cross-validation splits data into k parts, trains on k − 1 and tests on the remaining one, repeating k times so every record is used for testing once. The average performance is more reliable than a single holdout split.

Processk-fold cross-validation
  1. 1Split data into k equal folds
  2. 2Train on k − 1 folds
  3. 3Test on the remaining fold
  4. 4Repeat k times so each fold is tested once
  5. 5Average the performance
  • Common choices: 5-fold or 10-fold; leave-one-out (k = n) for very small data; repeated cross-validation for stability.
  • In R: caret::trainControl(method = "cv", number = 10).

Example

5-fold cross-validation: a model's accuracy on five folds is 82%, 80%, 85%, 79% and 84%. The cross-validated accuracy is the average, 82%, and the narrow range shows the model is stable.

Exam tip

Cross-validation is especially useful when the data set is small.

3

Topic 3

Bootstrap method

  • Draw samples with replacement of the same size as the data; train on the bootstrap sample and test on the left-out (out-of-bag) observations — about 36.8% of cases are left out each time.
  • .632 bootstrap: combines training and out-of-bag error to reduce bias.

Example

Bootstrap estimate: to estimate how uncertain an average customer spend of ₹1,200 is, an analyst draws 1,000 bootstrap samples, computes the mean of each, and uses the middle 95% of those means (say ₹1,120–₹1,290) as a confidence interval.

Exam tip

On average, about 63.2% of the original records appear in each bootstrap sample; the rest can be used for testing (out-of-bag).

4

Topic 4

Confusion matrix

ComparisonConfusion matrix
Predicted positive
Predicted negative

Actual positive

True positive (TP)

False negative (FN)

Actual negative

False positive (FP)

True negative (TN)

Predicted positivePredicted negative
Actual positiveTrue positive (TP)False negative (FN)
Actual negativeFalse positive (FP)True negative (TN)
Key formulasClassifier metrics
  • Accuracy

    (TP + TN) ÷ total

  • Precision

    TP ÷ (TP + FP)

  • Recall (sensitivity)

    TP ÷ (TP + FN)

  • Specificity

    TN ÷ (TN + FP)

  • F1 score

    2 × precision × recall ÷ (precision + recall)

Example

Of 1,000 customers, 100 churn. The model flags 120: 80 actual churners (TP), 40 non-churners (FP). Precision = 80 ÷ 120 = 67%; recall = 80 ÷ 100 = 80%; accuracy = (80 + 860) ÷ 1,000 = 94%.

  • ROC curve and AUC: trade-off between sensitivity and false positive rate across cut-offs (pROC package).
  • Imbalanced data: accuracy misleads — use precision, recall, F1, AUC; rebalance with oversampling (SMOTE) or class weights.
5

Topic 5

Statistical significance of data mining results

  • Why: with many models and variables, some patterns appear by chance (data snooping).
  • Comparing two classifiers: paired t-test on cross-validation folds, McNemar's test on the same test set.
  • Confidence intervals for accuracy; permutation tests (shuffle labels to see whether performance beats chance).
  • Multiple comparisons: adjust significance levels (Bonferroni) when testing many patterns or rules.

Example

A chance pattern: an analyst tests 100 customer variables against sales and finds 5 'significant' at the 5% level. That is exactly what chance alone would produce, so the findings must be checked on new data or adjusted for multiple testing.

Exam tip

Validate patterns on holdout data and use corrections such as Bonferroni when testing many hypotheses.

6

Topic 6

Advanced topics

  • Scalable machine learning: distributed training (Spark MLlib, sparklyr in R), stochastic gradient descent, online learning, GPUs.
  • Big data techniques: MapReduce, Hadoop and Spark, NoSQL stores, data lakes, sampling and dimensionality reduction for very large data.
  • Stream data mining: analysing continuous data in real time (sensor feeds, clickstreams, transactions) with one-pass algorithms, sliding windows and concept-drift detection — e.g., real-time fraud alerts.
  • Social network analysis: nodes and edges; centrality measures (degree, betweenness, closeness, eigenvector), community detection, influence and diffusion — igraph package in R.

Example

Scaling up: a company's transaction data is too large for one computer. Using sparklyr, the analyst runs the same R code on a Spark cluster, training a model on billions of records in parallel.

Exam tip

Mention scalable ML, deep learning, explainable AI and AutoML as current directions.

Key terms

Holdout method
Single train–test split for evaluation
k-fold cross-validation
Rotating training and test folds
Recall
Share of actual positives correctly identified
Permutation test
Significance test by shuffling labels
Concept drift
Change over time in the patterns a model learned

Quick revision

  • Holdout (stratified split); k-fold and leave-one-out CV; bootstrap and .632.
  • Confusion matrix: TP, FP, FN, TN; accuracy, precision, recall, specificity, F1; ROC–AUC.
  • Imbalanced data: SMOTE, class weights.
  • Significance: paired t-test, McNemar, permutation, Bonferroni.
  • Scalable ML, Spark, stream mining, social network centrality.

Important exam questions

Practice questions written to the PTU exam pattern for this unit's syllabus: short answers (Section A style) and long answers (Sections B and C style).

Short-answer questions

  1. Q1.What is the holdout method?
  2. Q2.Explain 10-fold cross-validation.
  3. Q3.What fraction of data is out-of-bag in a bootstrap sample?
  4. Q4.Define precision and recall.
  5. Q5.Why is accuracy misleading for imbalanced data?
  6. Q6.What is stream data mining?

Long-answer questions

  1. Q1.Explain holdout, cross-validation and bootstrap methods of estimating classifier performance.
  2. Q2.Explain the confusion matrix and the metrics derived from it with an example.
  3. Q3.Discuss how the statistical significance of data mining results is assessed.
  4. Q4.Discuss scalable machine learning, big data techniques, stream mining and social network analysis.

Stuck on this unit?

Message SBS on WhatsApp for help with Data Sciences Using R, or to ask about studying MBA at Synetic.

Planning your MBA in Ludhiana? See the MBA at SBS →

WhatsApp us