Unit 3: Data sourcing and analytics techniques

• Types: structured, semi-structured, unstructured; internal and external; first-, second- and third-party; transactional, behavioural, sensor, text, image, audio.
• Sources: ERP and CRM, websites and apps, social media, IoT, public data (data.gov.in, RBI, MOSPI), commercial data providers, partner data, account aggregators.

• Data warehouse: structured, integrated data for reporting and BI.
• Data lake: raw data of all types stored cheaply for exploration and machine learning; lakehouse combines both.
• Cloud computing: scalable storage and processing (AWS, Azure, Google Cloud); distributed processing (Spark); streaming platforms (Kafka).

• Bias: sampling bias, historical bias (past discrimination in data), measurement bias, survivorship bias, confirmation bias in analysis — can produce unfair or wrong decisions.
• Remedies: data profiling, cleaning, representative sampling, bias testing of models, documentation (data sheets).

• Text analytics: sentiment analysis, topic modelling, entity extraction, classification of emails and complaints — natural language processing.
• Speech analytics: converting call-centre audio to text to detect sentiment, compliance and reasons for calls.
• Image and video analytics: quality inspection, facial and object recognition, retail shelf monitoring, medical imaging.

• Machine learning: algorithms learn patterns from data — supervised (classification, regression), unsupervised (clustering), reinforcement learning.
• Deep learning: neural networks with many layers for images, speech and language.
• Generative AI: large language models that create text, code and images — used for summarisation, customer service, content and analysis assistance.

Data storytelling combines data, visuals and narrative to explain what happened, why it matters and what to do.

• Other frameworks: Minto's pyramid principle (answer first, then supporting arguments), SCQA (situation, complication, question, answer), "What? So what? Now what?".