Unit 3: Ensemble methods and clustering
Data Sciences Using R notes · PTU syllabus (MBA 962-18)
On this page
Unit summary
Combining many models often beats any single model, and clustering reveals hidden groups. This unit covers ensemble methods — bagging, random forests and boosting — and clustering methods — K-means, K-medoids, agglomerative and hierarchical clustering, X-means and DBSCAN.
After this unit you can
- Explain bagging and random forests
- Explain boosting
- Apply K-means, K-medoids and X-means clustering
- Apply hierarchical clustering and DBSCAN
PTU syllabus topics
- Bagging
- random forests
- boosting
- K-means clustering
- K-medoids
- agglomerative and hierarchical clustering
- X-means
- DBSCAN
Topic 1
Ensemble methods
Ensemble learning combines several models to improve accuracy and stability.
Training
Models trained in parallel on bootstrap samples
Models trained sequentially, each correcting previous errors
Combination
Average or majority vote
Weighted vote
Main effect
Reduces variance (overfitting)
Reduces bias
Examples
Bagged trees, random forest
AdaBoost, gradient boosting, XGBoost
Example
Why ensembles work: three different models each predict loan default with 80% accuracy but make different mistakes. Combining their votes gives a more accurate and stable prediction than any one of them.
Exam tip
Main ensemble types: bagging (reduces variance), boosting (reduces bias) and stacking.
Topic 2
Bagging and random forests
Bagging (bootstrap aggregating) trains many models on bootstrap samples and averages their predictions. A random forest is bagging of decision trees with an extra twist: each split considers only a random subset of variables, making the trees more different and the forest more accurate.
Each tree sees a bootstrap sample and a random feature subset; averaging reduces variance
- Bagging (bootstrap aggregating, Breiman 1996): draw many bootstrap samples, fit a model (usually a tree) to each, and combine predictions.
- Random forest: bagging plus a random subset of variables at each split, making trees less correlated.
- In R: randomForest(default ~ ., data = train, ntree = 500, mtry = 3); importance(model) and varImpPlot(model) show variable importance.
- Out-of-bag (OOB) error: estimate of test error using observations left out of each bootstrap sample.
Example
A random forest in R: randomForest(default ~ ., data = train, ntree = 500) builds 500 trees. Each tree votes, and the majority decides. importance() shows which variables — such as EMI ratio — matter most.
Exam tip
Random forests reduce overfitting compared with a single tree.
Topic 3
Boosting
Boosting builds models one after another, with each new model focusing on the errors of the previous ones. Popular methods are AdaBoost, gradient boosting and XGBoost.
- AdaBoost: increases the weights of misclassified cases so later models focus on them.
- Gradient boosting: each new tree fits the residual errors of the current ensemble; XGBoost adds regularisation and speed — popular in competitions and credit scoring.
- In R: gbm, xgboost packages; tune learning rate, number of trees and depth to avoid overfitting.
Example
Boosting in practice: an insurer uses gradient boosting to predict claim fraud. The first tree catches obvious cases; later trees focus on the cases it missed, steadily improving accuracy.
Exam tip
Boosting can overfit if too many rounds are used — tune the number of trees and the learning rate.
Topic 4
K-means and K-medoids
- K-means: kmeans(scale(df), centers = 4, nstart = 25) — assigns points to the nearest of k centroids and updates centroids until stable.
- Choosing k: elbow method (within-cluster sum of squares), silhouette width (cluster::silhouette), gap statistic.
- K-medoids (PAM): cluster::pam(df, k = 4) — uses actual observations as centres; more robust to outliers.
Example
Choosing k: a retailer clusters customers on recency, frequency and spending. The elbow plot of within-cluster variation flattens at k = 4, giving four segments — loyal high spenders, occasional big buyers, frequent small buyers and lapsed customers.
Exam tip
K-means uses averages and is sensitive to outliers; K-medoids uses actual data points and is more robust.
Topic 5
X-means
- X-means (Pelleg and Moore, 2000): an extension of K-means that chooses the number of clusters automatically by repeatedly splitting clusters and keeping splits that improve the Bayesian Information Criterion (BIC).
- Useful when k is unknown and data are large.
Example
Letting the data choose: instead of testing k = 2 to 10 manually, X-means starts with a small number of clusters and splits a cluster only if a statistical criterion (BIC) improves — automatically settling on, say, five customer segments.
Exam tip
X-means solves K-means' main weakness: having to choose k in advance.
Topic 6
Agglomerative and hierarchical clustering
- Agglomerative: start with each observation as a cluster and merge the closest pairs; divisive splits from one cluster.
- In R: d <- dist(scale(df)); hc <- hclust(d, method = "ward.D2"); plot(hc); cutree(hc, k = 4).
- Linkage: single, complete, average, Ward; the dendrogram shows merge order and heights.
Example
Hierarchical clustering in R: hc <- hclust(dist(scale(df)), method = "ward.D2") builds the tree; plot(hc) shows the dendrogram, and cutree(hc, k = 3) assigns each store to one of three clusters.
Exam tip
No need to fix the number of clusters in advance — cut the dendrogram where the gaps are largest.
Topic 7
DBSCAN
- Density-based clustering: groups points in dense regions; points in sparse regions are noise.
- Parameters: eps (neighbourhood radius) and minPts; choose eps from the k-nearest-neighbour distance plot (dbscan::kNNdistplot).
- In R: dbscan::dbscan(df, eps = 0.5, minPts = 5).
- Strengths: arbitrary-shaped clusters and outlier detection without fixing k.
K-means
Fixed in advance
Large data, roughly spherical clusters
K-medoids
Fixed in advance
Data with outliers
X-means
Chosen by BIC
Unknown k, large data
Hierarchical
Chosen from dendrogram
Small data, nested structure
DBSCAN
Found from density
Irregular shapes, noise
Example
DBSCAN in R: using the dbscan package, dbscan(scale(df), eps = 0.5, minPts = 5) finds dense groups of transactions and labels isolated ones as noise (cluster 0) — useful for spotting unusual, possibly fraudulent, transactions.
Exam tip
Set eps (neighbourhood radius) and minPts (minimum points) carefully — results depend heavily on them.
Key terms
- Ensemble
- Combination of multiple models
- Bootstrap sample
- Sample drawn with replacement
- Random forest
- Bagged trees with random variable selection at splits
- Gradient boosting
- Sequential trees fitting residual errors
- X-means
- K-means variant choosing k by BIC
Quick revision
- Bagging (variance) vs boosting (bias).
- Random forest: ntree, mtry, OOB error, variable importance.
- AdaBoost, gradient boosting, XGBoost.
- K-means (elbow, silhouette), K-medoids (PAM), X-means (BIC).
- Hierarchical (hclust, linkage, dendrogram); DBSCAN (eps, minPts).
Important exam questions
Practice questions written to the PTU exam pattern for this unit's syllabus: short answers (Section A style) and long answers (Sections B and C style).
Short-answer questions
- Q1.What is ensemble learning?
- Q2.Distinguish bagging and boosting.
- Q3.What is out-of-bag error?
- Q4.How does X-means choose the number of clusters?
- Q5.Write the R command for hierarchical clustering.
- Q6.What are the parameters of DBSCAN?
Long-answer questions
- Q1.Explain bagging and random forests.
- Q2.Explain boosting methods and their advantages.
- Q3.Explain K-means, K-medoids and X-means clustering.
- Q4.Explain hierarchical clustering and DBSCAN with R implementation.
Stuck on this unit?
Message SBS on WhatsApp for help with Data Sciences Using R, or to ask about studying MBA at Synetic.
