Big Data and Hadoop

Module 1 - Introduction To Big Data

Introduction to Big Data

Introduction

What is Big data?

Evolution of Data

5Vs pf Big Data

Different Kinds of Data

Big Data Sources

Processing Big Data

Big data Analytics

Big Data Insight

Applications of Big Data Analytics

Benefits of Big Data

How big Data Impacts IT

Introduction To Big Data Quiz

Hadoop and Its Architecture

what is hadoop?

About Hadoop

Problems in Distributed Computing

Famouse Hadoop Users

Why Hadoop?

Features of Hadoop

Simple Architecture of Hadoop

Core Components of Hadoop

What is HDFS ?

What is Map Reduce?

Hadoop Versions

Types of Nodes in Hadoop

Hadoop System

Hadoop 1.x Architecture

Hadoop 1.x Cluster Administration

Hadoop 1.x Meta Data Management

Hadoop 1.x Architecture Disadvantages

Hadoop 2.x Architecture

Hadoop 2.x Description

Hadoop Spt QUIZ

Map Reduce and YARN

Hadoop Core Components

Design of HDFS

Concept of HDFS

Map Reduce - Data Processing

Map Reduce : Parallel Processing of Data

Map Reduce Programming Model

Map Reduce Example : Vote Count (Traditional)

Map Reduce Example : Vote Count (Map Reduce)

Map Reduce in Detail through Word Count

YARN

YARN FLOW

Scheduling in YARN

Scheduling in YARN - Capacity Scheduler

Map Reduce Vs YARN

Yarn Take AWAY

Map Reduce and Yarn QUIZ

Cloudera Installation

Basic Commands in Hadoop

Basic Commands in Hadoop Description

Basic Commands in Hadoop Description - version

Basic Commands in Hadoop Description - jps

Basic Commands in Hadoop Description - ls

Basic Commands in Hadoop Description -mkdir

Basic Commands in Hadoop Description -put

Basic Commands in Hadoop Description -cat

Basic Commands in Hadoop Description - touchz

Basic Commands in Hadoop Description - get

Basic Commands in Hadoop Description -cp

Basic Commands in Hadoop Description -mv

Basic Commands in Hadoop Description -rm-r

Basic Commands in Hadoop Description -du

Basic Commands in Hadoop Description -stat

Basic Commands in Hadoop Description -report

Basic Hadoop Commands Quiz

Tasks (Hadoop Commands)

Hadoop Distribution Systems

Popular Hadoop Distributions

Popular Hadoop Distributions - Cloudera

Popular Hadoop Distributions - Horton Works

Popular Hadoop Distributions - MapR

Choosing a Hadoop Distribution

Hadoop Distribution Systems Quiz

Module 2 – Sqoop

Sqoop

Sqoop Introduction

Why Sqoop

Sqoop Architecture

Sqoop Features

Sqoop Import

SQOOP Import Internal Process

Sqoop Import Important Parameters

Sqoop import Sample Execution

Sqoop Import Commands

Sqoop Import Command using WHERE condition

Sqoop Import for importing only Specific column

Sqoop Handson in CLOUDERA

Sqoop Import Assignments

Sqoop Import Important Commands

Sqoop EVAL

SQOOP EVAL SELECT

Sqoop EVAL Insert

Sqoop EVAL Update

Sqoop EVAL DELETE

Sqoop Import Split by

Sqoop Import Split by Handson

Sqoop Import Direct Mode

Sqoop Import Direct Mode Handson

Sqoop Import EVAL,Split by ,direct Mode Assignments

Module 3 – Hive

Hive Introduction

Why another Warehousing system

Hive Introduction

what is hive?

Architectural Overview

Hive Execution plan

Quiz

Apache Hive Tables & Data types

Hive Data types

Data Abstraction in Hive

Hive Data Model

Hive Default Warehouse

Hive Default Warehouse - Sample Execution

Hive Tables

Create database in Hive

Internal Table

Drop Internal Table

Drop External Table

When to use Internal and External Table?

Internal Table Vs External Table

Industry Usage

Module 4 - Hbase

No SQL Database Introduction

Summary of Early Database systems

RDBMS

Issues with RDBMS - Scalability

Why RDBMS is not suitable for Big data

What is NoSQL?

Need of NoSQL

Types of NoSQL Databases

Characteristics of NoSQL Database

Key Value Pair Based

Column Based

Document Based

Graph Based

CAP Theorem

Advantages of NoSQL Database

What are not provided by NoSQL

Where to use NoSQL

Conclusion

Module 5 - Scala                  

Basics of Scala

What is Scala?

What is Functional Programming?

Pure Functions Example

Impure Functions

Variable declaration and Initialization

VAR

VAL

Type Interference

Lazy Evaluation

String Interpolation

Different String Interpolation Methods

String -s Interpolator

String -f Interpolator

String -raw Interpolator

Module 6 – PYSPARK

Spark Introduction

Big Data Processing

Why Spark?

What is Spark?

HADOOP Vs Spark

 Hadoop MapReduce Vs Apache Spark

Spark Components

Spark Ecosystem

 Spark Core

Spark SQL

Spark Streaming

Mlib

Module 7 - SPARK STREAMING

Spark Streaming

What is Streaming Data?

What is Spark Streaming

Simple Architecture of Spark Streaming

Spark Streaming Work Flow

Key Concepts in Spark Streaming

What is Dstream?

Streaming Context

Dstream Interface

Mapped Dstream

Windowed Dstream

Network Input Dstream

Transformations on Dstreams

stateful Transformations

Windowed Transformations

UpdatestateBykey Transformation

Action/output Operations

Network Receiver

Components of Spark Streaming

Execution Model - Receiving Data

Execution Model - Job Scheduling

Job Scheduling

Dstream Persistance

What is RDD Checkpointing ?

Why is RDD Checkpointing necessary ?

RDD Checkpointing

Performance Tunning

Performance Tunning - Step - 1

Step - 2 : Optimize for Lower Latency

Code for Spark Streaming - Word Count

Code for Spark Streaming - Word Count flow

Spark Streaming Use Cases

Module 8 – KAFKA

Kafka Introduction & Topics

Why kafka is needed?

Kafka Use Cases

Who uses Kafka?

 What is Kafka ?

 Kafka Core Concepts

 Kafka API's

 Kafka API Representation

 Kafka Fundamental Concepts

 Kafka Topic

Module 9 – Projects

Project 1 - Hive AVRO Data Standandization

Project 2 - Banking Data processing using Hive

Project 3 - SPARK HBASE HIVE Project