Hadoop

Introduction to Hadoop

• High Availability
• Scaling
• Advantages and Challenges 

• What is Big data
• Big Data opportunities,Challenges
• Characteristics of Big data 

• Hadoop Distributed File System
• Comparing Hadoop & SQL
• Industries using Hadoop
• Data Locality
• Hadoop Architecture
• Map Reduce & HDFS
• Using the Hadoop single node image (Clone)

• HDFS Design & Concepts
• Blocks, Name nodes and Data nodes
• HDFS High-Availability and HDFS Federation
• Hadoop DFS The Command-Line Interface
• Basic File System Operations
• Anatomy of File Read,File Write
• Block Placement Policy and Modes
• More detailed explanation about Configuration files
• Metadata, FS image, Edit log, Secondary Name Node and Safe Mode
• How to add New Data Node dynamically,decommission a Data Node dynamically (Without stopping cluster)
• FSCK Utility. (Block report)
• How to override default configuration at system level and Programming level
• HDFS Federation
• ZOOKEEPER Leader Election Algorithm
• Exercise and small use case on HDFS

• Map Reduce Functional Programming Basics
• Map and Reduce Basics
• How Map Reduce Works
• Anatomy of a Map Reduce Job Run
• Legacy Architecture ->Job Submission, Job Initialization, Task Assignment, Task Execution, Progress and Status Updates
• Job Completion, Failures
• Shuffling and Sorting
• Splits, Record reader, Partition, Types of partitions & Combiner
• Optimization Techniques -> Speculative Execution, JVM Reuse and No. Slots
• Types of Schedulers and Counters
• Comparisons between Old and New API at code and Architecture Level
• Getting the data from RDBMS into HDFS using Custom data types
• Distributed Cache and Hadoop Streaming (Python, Ruby and R)
• YARN
• Sequential Files and Map Files
• Enabling Compression Codec’s
• Map side Join with distributed Cache
• Types of I/O Formats: Multiple outputs, NLINEinputformat
• Handling small files using CombineFileInputFormat

• Hands on “Word Count” in Map Reduce in standalone and Pseudo distribution Mode
• Sorting files using Hadoop Configuration API discussion
• Emulating “grep” for searching inside a file in Hadoop
• DBInput Format
• Job Dependency API discussion
• Input Format API discussion,Split API discussion
• Custom Data type creation in Hadoop

• ACID in RDBMS and BASE in NoSQL
• CAP Theorem and Types of Consistency
• Types of NoSQL Databases in detail
• Columnar Databases in Detail (HBASE and CASSANDRA)
• TTL, Bloom Filters and Compensation

• HBase Installation, Concepts
• HBase Data Model and Comparison between RDBMS and NOSQL
• Master  & Region Servers
• HBase Operations (DDL and DML) through Shell and Programming and HBase Architecture
• Catalog Tables
• Block Cache and sharding
• SPLITS
• DATA Modeling (Sequential, Salted, Promoted and Random Keys)
• JAVA API’s and Rest Interface
• Client Side Buffering and Process 1 million records using Client side Buffering
• HBase Counters
• Enabling Replication and HBase RAW Scans
• HBase Filters
• Bulk Loading and Co processors (Endpoints and Observers with programs)
• Real world use case consisting of HDFS,MR and HBASE

• Hive Installation, Introduction and Architecture
• Hive Services, Hive Shell, Hive Server and Hive Web Interface (HWI)
• Meta store, Hive QL
• OLTP vs. OLAP
• Working with Tables
• Primitive data types and complex data types
• Working with Partitions
• User Defined Functions
• Hive Bucketed Tables and Sampling
• External partitioned tables, Map the data to the partition in the table, Writing the output of one query to another table, Multiple inserts
• Dynamic Partition
• Differences between ORDER BY, DISTRIBUTE BY and SORT BY
• Bucketing and Sorted Bucketing with Dynamic partition
• RC File
• INDEXES and VIEWS
• MAPSIDE JOINS
• Compression on hive tables and Migrating Hive tables
• Dynamic substation of Hive and Different ways of running Hive
• How to enable Update in HIVE
• Log Analysis on Hive
• Access HBASE tables using Hive
• Hands on Exercises

• Pig Installation
• Execution Types
• Grunt Shell
• Pig Latin
• Data Processing
• Schema on read
• Primitive data types and complex data types
• Tuple schema, BAG Schema and MAP Schema
• Loading and Storing
• Filtering, Grouping and Joining
• Debugging commands (Illustrate and Explain)
• Validations,Type casting in PIG
• Working with Functions
• User Defined Functions
• Types of JOINS in pig and Replicated Join in detail
• SPLITS and Multiquery execution
• Error Handling, FLATTEN and ORDER BY
• Parameter Substitution
• Nested For Each
• User Defined Functions, Dynamic Invokers and Macros
• How to access HBASE using PIG, Load and Write JSON DATA using PIG
• Piggy Bank
• Hands on Exercises

• Sqoop Installation

• Import Data.(Full table, Only Subset, Target Directory, protecting Password, file format other than CSV, Compressing, Control Parallelism,  All tables Import)
• Incremental  Import(Import only New data, Last Imported data, storing Password in Metastore, Sharing Metastore between Sqoop Clients)
• Free Form Query Import
• Export data to RDBMS,HIVE and HBASE
• Hands on Exercises

• HCatalog Installation
• Introduction to HCatalog
• About Hcatalog with PIG,HIVE and MR
• Hands on Exercises

• Flume Installation
• Introduction to Flume
• Flume Agents: Sources, Channels and Sinks
• Log User information using Java program in to HDFS using LOG4J and Avro Source, Tail Source
• Log User information using Java program in to HBASE using LOG4J and Avro Source, Tail Source
• Flume Commands
• Use case of Flume: Flume the data from twitter in to HDFS and HBASE. Do some analysis using HIVE and PIG

• HUE.(Hortonworks and Cloudera)

• Workflow (Action, Start, Action, End, Kill, Join and Fork), Schedulers, Coordinators and Bundles.,to show how to schedule Sqoop Job, Hive, MR and PIG
• Real world Use case which will find the top websites used by users of certain ages and will be scheduled to run for every one hour
• Zoo Keeper
• HBASE Integration with HIVE and PIG
• Phoenix
• Proof of concept (POC)

• Spark Overview
• Linking with Spark, Initializing Spark
• Using the Shell
• Resilient Distributed Datasets (RDDs)
• Parallelized Collections
• External Datasets
• RDD Operations
• Basics, Passing Functions to Spark
• Working with Key-Value Pairs
• Transformations
• Actions
• RDD Persistence
• Which Storage Level to Choose?
• Removing Data
• Shared Variables
• Broadcast Variables
• Accumulators
• Deploying to a Cluster
• Unit Testing
• Migrating from pre-1.0 Versions of Spark
• Where to Go from Here