Big Data Analytics Notes
Subject Overview
The fourth Elective-II option, covering big data fundamentals and high-performance architecture (HDFS, MapReduce, YARN), advanced analytical methods (clustering, decision trees, Naive Bayes, association rules, recommendation systems), stream computing and real-time analytics, and Hadoop implementation and deployment. A 4-credit elective theory paper.
Subject at a glance
4
Units
27
Topics
4
Credits
4
Infographics
Unit-wise Syllabus
4 units — open any unit for detailed topic-wise notes, diagrams and important exam questions
- Evolution and characteristics of big data
- best practices
- big data use cases
- understanding big data storage
- high-performance architecture overview — HDFS
- MapReduce and YARN
- MapReduce programming model
Volume
Huge amounts of data
Velocity
Arrives fast
Variety
Structured to unstructured
Veracity
Uncertain quality
Value
Useful insight
Study guide
This unit covers the evolution and characteristics of big data, best practices, use cases, storage, high-performance architecture, HDFS, MapReduce, YARN and the MapReduce programming model. Understand the idea of moving computation to the data.
- K-means clustering use cases and diagnostics
- decision tree algorithms and evaluation
- Naive Bayes classification
- association rules and the Apriori algorithm
- evaluation of candidate rules
- finding association and similarity
- collaborative/content-based/knowledge-based/hybrid recommendation systems
Collaborative filtering
Similar users' behaviour
People who bought this also bought
Content-based
Item features
Similar movies by genre
Knowledge-based
Explicit rules and needs
Configure a laptop by requirements
Hybrid
A mix
Netflix-style recommendations
Study guide
You study K-means use cases and diagnostics, decision trees and evaluation, Naive Bayes, association rules and Apriori, candidate rule evaluation, similarity and recommendation systems (collaborative, content-based, knowledge-based, hybrid). Build a simple recommender.
- Stream data model and architecture
- sampling and filtering streams
- counting distinct elements
- estimating moments
- decaying windows
- real-time analytics platform applications
- real-time sentiment analysis
- stock market prediction
- graph analytics for big data
- 1Data streams in
Tweets, ticks, sensors
- 2Sample or filter
- 3Windowed computation
Counts, averages, decaying windows
- 4Detect patterns
Sentiment, anomalies
- 5Act or alert in real time
Study guide
You study the stream data model and architecture, sampling and filtering, counting distinct elements, estimating moments, decaying windows, real-time analytics, sentiment analysis, stock prediction and graph analytics. Understand why streams need approximate algorithms.
- Hadoop cluster components and architecture
- Hadoop ecosystem
- evaluation criteria for distributed MapReduce runtimes
- enterprise-grade Hadoop deployment and implementation
- 1Input split
HDFS blocks
- 2Map
Emit key-value pairs
- 3Shuffle and sort
Group by key
- 4Reduce
Aggregate each key
- 5Output
Written to HDFS
Study guide
You study Hadoop cluster components and architecture, the ecosystem, evaluation criteria for MapReduce runtimes and enterprise deployment. Set up a small local Hadoop cluster.
Why Big Data Analytics Matters
Big data skills support data engineer and analyst roles in companies that handle large volumes of data, and Hadoop and Spark knowledge is widely requested.
How to Prepare for Big Data Analytics
- Practise MapReduce with a word-count and a log-analysis example.
- Learn the Hadoop ecosystem components and what each does.
- Explore Spark as a modern alternative.
Big Data Analytics: Frequently Asked Questions
What are the characteristics of big data?
Commonly described as volume, velocity and variety, with veracity and value often added.