Data Warehousing and Data Mining Notes
Subject Overview
The third Elective-I option, covering data warehouse architecture and OLAP, data preprocessing and multidimensional modeling (star/
Subject at a glance
4
Units
33
Topics
4
Credits
4
Infographics
Unit-wise Syllabus
4 units — open any unit for detailed topic-wise notes, diagrams and important exam questions
- Data warehouse components and architecture
- mapping to multiprocessor architecture
- need for data warehousing
- operational vs informational data stores
- OLAP vs OLTP and their differences
- OLAP operations
Purpose
Run daily operations
Analyse for decisions
Data
Current, detailed
Historical, summarised
Operations
Insert, update, delete
Read, aggregate, slice
Schema
Normalised
Star or snowflake
Study guide
This unit covers warehouse components and architecture, the need for warehousing, operational versus informational stores, OLAP versus OLTP and OLAP operations. Learn roll-up, drill-down, slice and dice.
- Design
- technical and implementation considerations
- data preprocessing (summarization, cleaning, transformation)
- ETL process
- multidimensional data model
- star/snowflake/fact-constellation schemas
- data warehouse architecture and design
- OLAP three-tier architecture
- cube computation
- data marts
- 1Extract
From source systems
- 2Clean
Fix errors, remove duplicates
- 3Transform
Standardise, aggregate
- 4Load
Into the data warehouse
- 5Refresh
On a schedule
Study guide
You study design and implementation, preprocessing, ETL, the multidimensional model, star, snowflake and fact-constellation schemas, architecture, OLAP tiers, cube computation and data marts. Design a star schema for a retail business.
- Data mining functionalities
- mining different kinds of data
- classification vs clustering
- predictive vs descriptive mining
- association rule mining
- market basket analysis
- the Apriori algorithm
Support
Count(A and B) / total transactions
Confidence
Support(A and B) / Support(A)
Lift
Confidence(A ⇒ B) / Support(B)
Apriori principle
Subsets of frequent itemsets are frequent
Study guide
You study data mining functionalities, kinds of data, classification versus clustering, predictive versus descriptive mining, association rules, market basket analysis and the Apriori algorithm. Work an Apriori example by hand.
- Bayesian classification (Bayes theorem, Bayesian belief networks, Naive Bayes)
- classification by back-propagation
- k-Nearest Neighbour
- Support Vector Machine
- linear and multiple regression prediction
- clustering data types
- partitioning methods (K-Means, K-Medoids)
- hierarchical agglomerative methods
- DBSCAN
- using Weka/Python for implementation
Naive Bayes
Classification
Bayes' theorem with independent features
k-NN
Classification
Vote of the k nearest neighbours
SVM
Classification
Maximum-margin boundary
K-means
Clustering
Assign to nearest centroid, update
DBSCAN
Clustering
Dense regions; finds noise
Study guide
You study Bayesian classification and Naive Bayes, backpropagation classification, k-NN, SVM, regression, clustering (K-Means, K-Medoids, hierarchical, DBSCAN) and implementation with Weka or Python. Try each algorithm on one data set.
Why Data Warehousing and Data Mining Matters
Warehousing and mining form the base of business intelligence and data science, a high-demand career area.
How to Prepare for Data Warehousing and Data Mining
- Implement classification and clustering on a real data set.
- Learn the difference between OLTP and OLAP.
- Practise Apriori and decision tree problems.
Data Warehousing and Data Mining: Frequently Asked Questions
What is ETL?
ETL stands for Extract, Transform, Load: the process of collecting data from sources, cleaning and converting it and loading it into a data warehouse.