Synetic College Logo
Unit 3 of 4 · M.Sc IT Sem 4

Unit 3: Stream computing and real-time analytics

Big Data Analytics notes · PTU syllabus (PGCA1947)

3 min read6 topics10 exam questions
On this page
  1. Unit summary
  2. Stream data model and architecture
  3. Sampling and filtering streams
  4. Counting distinct elements
  5. Estimating moments and decaying windows
  6. Real-time analytics applications
  7. Graph analytics for big data
  8. Key terms
  9. Quick revision
  10. Important questions

Unit summary

Some data must be analysed as it arrives. This unit covers the stream data model and architecture, sampling and filtering streams, counting distinct elements, estimating moments, decaying windows, real-time analytics platforms, sentiment analysis, stock market prediction and graph analytics.

After this unit you can

  • Explain the stream data model
  • Apply sampling, Bloom filters and Flajolet–Martin
  • Explain moments and decaying windows
  • Describe real-time and graph analytics applications

PTU syllabus topics

  • Stream data model and architecture
  • sampling and filtering streams
  • counting distinct elements
  • estimating moments
  • decaying windows
  • real-time analytics platform applications
  • real-time sentiment analysis
  • stock market prediction
  • graph analytics for big data
ProcessReal-time stream analytics
  1. 1Data streams in

    Tweets, ticks, sensors

  2. 2Sample or filter
  3. 3Windowed computation

    Counts, averages, decaying windows

  4. 4Detect patterns

    Sentiment, anomalies

  5. 5Act or alert in real time
1

Topic 1

Stream data model and architecture

ProcessStream architecture
  1. 1Sources

    Sensors, clicks, tweets, transactions

  2. 2Ingestion

    Apache Kafka

  3. 3Stream processor

    Spark Streaming, Flink, Storm with limited working memory

  4. 4Standing and ad hoc queries
  5. 5Output

    Dashboards, alerts, storage

  • Data arrives continuously and can be seen only once; algorithms use small memory and give approximate answers.
2

Topic 2

Sampling and filtering streams

Key termsTechniques
Fixed-proportion sampling
Hash a key (e.g., user ID) to keep a representative 1/10 of users
Reservoir sampling
Keep a uniform sample of size s from an unknown-length stream
Bloom filter
Bit array with k hash functions to test membership; no false negatives, some false positives

Example

Bloom filter for spam senders: an e-mail address hashes to bits 3, 7 and 12; if all three are set the address is possibly on the list, if any is 0 it is definitely not.

3

Topic 3

Counting distinct elements

  • Flajolet–Martin: hash each element; track R = maximum number of trailing zeros seen; estimate distinct count ≈ 2^R. Combine many hash functions (averages of medians) for accuracy. HyperLogLog is the modern version.
4

Topic 4

Estimating moments and decaying windows

Key formulasMoments

  • k-th moment = Σ (mi)^k

    mi = count of element i

  • 0th moment

    Number of distinct elements

  • 1st moment

    Stream length

  • 2nd moment (surprise number)

    Measures skew — AMS algorithm estimates it

  • Decaying window: weight recent items more — score = Σ a(t−i) × (1 − c)^i with small c; used for "trending" items.
5

Topic 5

Real-time analytics applications

ComparisonApplications
Data
Approach

Sentiment analysis

Tweets and reviews

Tokenise, classify positive or negative (Naive Bayes, deep models) in streams; track brand mood

Stock market prediction

Price ticks, news

Moving averages, time-series and ML models in real time; event alerts

Fraud detection

Card transactions

Score each transaction in milliseconds

6

Topic 6

Graph analytics for big data

Key termsGraph analytics
PageRank
Importance of nodes from links
Community detection
Groups of tightly connected nodes
Shortest path
Routing, logistics
Centrality
Influencers in social networks
Tools
Neo4j, Spark GraphX, Apache Giraph

Key terms

Data stream
Continuous, unbounded sequence of data
Reservoir sampling
Uniform sample from a stream of unknown length
Bloom filter
Probabilistic set-membership structure
Flajolet–Martin
Algorithm estimating distinct elements
Decaying window
Weighting recent elements more

Quick revision

  • Stream model; Kafka, Spark, Flink.
  • Sampling; Bloom filter.
  • Flajolet–Martin; HyperLogLog.
  • Moments; AMS; decaying windows.
  • Sentiment, stock, fraud analytics; PageRank and graph tools.

Important exam questions

Practice questions written to the PTU exam pattern for this unit's syllabus: short answers (Section A style) and long answers (Sections B and C style).

Short-answer questions

  1. Q1.Why are stream algorithms approximate?
  2. Q2.What is reservoir sampling?
  3. Q3.Can a Bloom filter give false negatives?
  4. Q4.How does Flajolet–Martin estimate distinct counts?
  5. Q5.What is the second moment?
  6. Q6.What is PageRank?

Long-answer questions

  1. Q1.Explain the stream data model and architecture.
  2. Q2.Explain sampling and filtering of streams.
  3. Q3.Explain counting distinct elements and estimating moments.
  4. Q4.Explain real-time analytics applications.

Stuck on this unit?

Message SBS on WhatsApp for help with Big Data Analytics, or to ask about studying M.Sc IT at Synetic.

WhatsApp us