Data Science and Big Data Analytics

Outline

Chapter 1 - Data Science Algorithms and Analytical Methods

• Supervised vs Unsupervised Machine Learning
• Supervised Machine Learning Algorithms
• Unsupervised Machine Learning Algorithms
• Choose the Right Algorithm
• Life-cycles of Machine Learning Development
• Classifying with k-Nearest Neighbors (SL)
• k-Nearest Neighbors Algorithm
• k-Nearest Neighbors Algorithm
• The Error Rate
• Decision Trees (SL)
• Using Decision Trees
• Random Forests
• Naive Bayes Classifier (SL)
• Classification of Documents with Naive Bayes
• Unsupervised Learning Type: Clustering
• K-Means Clustering (UL)
• K-Means Clustering in a Nutshell
• K-Means Clustering in a Nutshell
• Regression Analysis
• Types of Regression
• Simple Linear Regression Model
• Linear Regression Illustration
• Least-Squares Method (LSM)
• LSM Assumptions
• Fitting Linear Regression Models in R
• Example of Using R's lm() Function
• Example of Using lm() with a Data Frame
• Regression Models in Excel
• Logistic Regression
• Regression vs Classification
• Time-Series Analysis
• Decomposing Time-Series
• Decomposing Time-Series
• Summary

Chapter 2 - Getting Started with R

• Introduction
• Positioning of R in the Data Science Arena
• R Integrated Development Environments
• Running R
• Running RStudio
• Ending the Current R Session
• Getting Help
• Getting System Information
• General Notes on R Commands and Statements
• R Data Structures
• R Objects and Workspace
• Assignment Operators
• Assignment Example
• Arithmetic Operators
• Logical Operators
• System Date and Time
• Operations
• User-defined Functions
• User-defined Function Example
• R Code Example
• Type Conversion (Coercion)
• Control Statements
• Conditional Execution
• Repetitive Execution
• Repetitive execution
• Built-in Functions
• Reading Data from Files into Vectors
• Example of Reading Data from a File
• Writing Data to a File
• Example of Writing Data to a File
• Logical Vectors
• Character Vectors
• Matrix Data Structure
• Creating Matrices
• Working with Data Frames
• Matrices vs Data Frames
• A Data Frame Sample
• Accessing Data Cells
• Getting Info About a Data Frame
• Selecting Columns in Data Frames
• Selecting Rows in Data Frames
• Getting a Subset of a Data Frame
• Sorting (ordering) Data in Data Frames by Attribute(s)
• Applying Functions to Matrices and Data Frames
• Using the apply() Function
• Example of Using apply()
• Executing External R commands
• Loading External Scripts in RStudio
• Listing Objects in Workspace
• Removing Objects in Workspace
• Saving Your Workspace in R
• Saving Your Workspace in RStudio
• Saving Your Workspace in R GUI
• Loading Your Workspace
• Loading Your Workspace
• Hands-on Exercises
• Getting and Setting the Working Directory
• Getting the List of Files in a Directory
• Diverting Output to a File
• Batch (Unattended) Processing
• Importing Data into R
• Exporting Data from R
• Hands-on Exercise
• Standard R Packages
• Extending R
• Extending R in R GUI
• Extending R in RStudio
• CRAN Page
• Summary

Chapter 3 - Text Mining

• What is Text Mining?
• The Common Text Mining Tasks
• What is Natural Language Processing (NLP)?
• Some of the NLP Use Cases
• Machine Learning in Text Mining and NLP
• Machine Learning in NLP
• TF-IDF
• The Feature Hashing Trick
• Stemming
• Example of Stemming
• Stop Words
• Popular Text Mining and NLP Libraries and Packages
• Summary

Chapter 4 - Introduction to Functional Programming

• What is Functional Programming (FP)?
• Terminology: Higher-Order Functions
• Terminology: Lambda vs Closure
• A Short List of Languages that Support FP
• FP with Java
• FP With JavaScript
• Imperative Programming in JavaScript
• The JavaScript map (FP) Example
• The JavaScript reduce (FP) Example
• Using reduce to Flatten an Array of Arrays (FP) Example
• The JavaScript filter (FP) Example
• Common High-Order Functions in Python
• Common High-Order Functions in Scala
• Elements of FP in R
• Summary

Chapter 5 - What is NoSQL?

• Limitations of Relational Databases
• Limitations of Relational Databases (Cont'd)
• Defining NoSQL
• What are NoSQL (Not Only SQL) Databases?
• The Past and Present of the NoSQL World
• NoSQL Database Properties
• NoSQL Benefits
• NoSQL Benefits
• NoSQL Database Storage Types
• NoSQL Database Storage Types
• The CAP Theorem
• NoSQL Systems CAP Triangle
• Mechanisms to Guarantee a Single CAP Property
• Limitations of NoSQL Databases
• Big Data Sharding
• Sharding Example
• Quiz
• Quiz Answers
• Summary

Chapter 6 - MapReduce Overview

• The Client – Server Processing Pattern
• Distributed Computing Challenges
• MapReduce Defined
• Google's MapReduce
• MapReduce Phases
• The Map Phase
• The Reduce Phase
• MapReduce Word Count Job
• MapReduce Shared-Nothing Architecture
• Similarity with SQL Aggregation Operations
• Example of Map & Reduce Operations using JavaScript
• Example of Map & Reduce Operations using JavaScript
• Problems Suitable for Solving with MapReduce
• Typical MapReduce Jobs
• Fault-tolerance of MapReduce
• Distributed Computing Economics
• MapReduce Systems
• Summary

Chapter 7 - Hadoop Overview

• Apache Hadoop
• Apache Hadoop Logo
• Typical Hadoop Applications
• Hadoop Clusters
• Hadoop Design Principles
• Hadoop Versions
• Hadoop's Main Components
• Hadoop Simple Definition
• Side-by-Side Comparison: Hadoop 1 and Hadoop 2
• Hadoop-based Systems for Data Analysis
• Other Hadoop Ecosystem Projects
• Hadoop Caveats
• Hadoop Distributions
• Cloudera Distribution of Hadoop (CDH)
• Cloudera Distributions
• Hortonworks Data Platform (HDP)
• MapR
• Summary

Chapter 8 - Hadoop Distributed File System Overview

• Hadoop Distributed File System (HDFS)
• HDFS Considerations
• HDFS High Availability
• Storing Raw Data in HDFS
• HDFS Security
• HDFS Rack-awareness
• Data Blocks
• Data Block Replication Example
• HDFS NameNode Directory Diagram
• File Metadata Records (Conceptual View)
• NameNode Meta Information Size
• HDFS Balancing
• Accessing HDFS
• Examples of HDFS Commands
• Other Supported File Systems
• WebHDFS
• Examples of WebHDFS Calls
• HDFS Daemon Web UI Ports
• Viewing Replica Factor and Block Size in NameNode Web UI
• HDFS Write Operation
• HDFS Read Operation
• Read Operation Sequence Diagram
• Communication inside HDFS
• Summary

Chapter 9 - MapReduce with Hadoop

• Hadoop's MapReduce
• MapReduce 1 and MapReduce 2
• Why do I need Discussion of the Old MapReduce?
• MapReduce v1 ("Classic MapReduce")
• JobTracker and TaskTracker (the "Classic MapReduce")
• YARN (MapReduce v2)
• YARN vs MR1
• YARN As Data Operating System
• MapReduce Programming Options
• Hadoop's Streaming MapReduce
• Python Word Count Mapper Program Example
• Python Word Count Reducer Program Example
• Setting up Java Classpath for Streaming Support
• Streaming Use Cases
• The Streaming API vs Java MapReduce API
• Amazon Elastic MapReduce
• Amazon Elastic MapReduce
• Apache Tez
• Summary

Chapter 10 - Apache Pig Scripting Platform

• What is Pig?
• Pig Latin
• Apache Pig Logo
• Pig Execution Modes
• Local Execution Mode
• MapReduce Execution Mode
• Running Pig
• Running Pig in Batch Mode
• What is Grunt?
• Pig Latin Statements
• Pig Programs
• Pig Latin Script Example
• SQL Equivalent
• Differences between Pig and SQL
• Statement Processing in Pig
• Comments in Pig
• Supported Simple Data Types
• Supported Complex Data Types
• Arrays
• Defining Relation's Schema
• Not Matching the Defined Schema
• The bytearray Generic Type
• Using Field Delimiters
• Loading Data with TextLoader()
• Referencing Fields in Relations
• Summary

Chapter 11 - Apache Pig Relational and Eval Operators

• Pig Relational Operators
• Example of Using the JOIN Operator
• Example of Using the JOIN Operator
• Example of Using the Order By Operator
• Caveats of Using Relational Operators
• Pig Eval Functions
• Caveats of Using Eval Functions (Operators)
• Example of Using Single-column Eval Operations
• Example of Using Eval Operators For Global Operations
• Summary

Chapter 12 - Hive

• What is Hive?
• Apache Hive Logo
• Hive's Value Proposition
• Who uses Hive?
• What Hive Does Not Have
• Hive's Main Sub-Systems
• Hive Features
• The "Classic" Hive Architecture
• The New Hive Architecture
• HiveQL
• Where are the Hive Tables Located?
• Hive Command-line Interface (CLI)
• The Beeline Command Shell
• Summary

Chapter 13 - Hive Command-line Interface

• Hive Command-line Interface (CLI)
• The Hive Interactive Shell
• Running Host OS Commands from the Hive Shell
• Interfacing with HDFS from the Hive Shell
• The Hive in Unattended Mode
• The Hive CLI Integration with the OS Shell
• Executing HiveQL Scripts
• Comments in Hive Scripts
• Variables and Properties in Hive CLI
• Setting Properties in CLI
• Example of Setting Properties in CLI
• Hive Namespaces
• Using the SET Command
• Setting Properties in the Shell
• Setting Properties for the New Shell Session
• Setting Alternative Hive Execution Engines
• The Beeline Shell
• Connecting to the Hive Server in Beeline
• Beeline Command Switches
• Beeline Internal Commands
• Summary

Chapter 14 - Hive Data Definition Language

• Hive Data Definition Language
• Creating Databases in Hive
• Using Databases
• Creating Tables in Hive
• Supported Data Type Categories
• Common Numeric Types
• String and Date / Time Types
• Miscellaneous Types
• Example of the CREATE TABLE Statement
• Working with Complex Types
• Working with Complex Types
• Table Partitioning
• Table Partitioning
• Table Partitioning on Multiple Columns
• Viewing Table Partitions
• Row Format
• Data Serializers / Deserializers
• File Format Storage
• File Compression
• More on File Formats
• The ORC Data Format
• Converting Text to ORC Data Format
• The EXTERNAL DDL Parameter
• Example of Using EXTERNAL
• Creating an Empty Table
• Dropping a Table
• Table / Partition(s) Truncation
• Alter Table/Partition/Column
• Views
• Create View Statement
• Why Use Views?
• Restricting Amount of Viewable Data
• Examples of Restricting Amount of Viewable Data
• Creating and Dropping Indexes
• Describing Data
• Summary

Chapter 15 - Apache Sqoop

• What is Sqoop?
• Apache Sqoop Logo
• Sqoop Import / Export
• Sqoop Help
• Examples of Using Sqoop Commands
• Data Import Example
• Fine-tuning Data Import
• Controlling the Number of Import Processes
• Data Splitting
• Helping Sqoop Out
• Example of Executing Sqoop Load in Parallel
• A Word of Caution: Avoid Complex Free-Form Queries
• Using Direct Export from Databases
• Example of Using Direct Export from MySQL
• More on Direct Mode Import
• Data Export from HDFS
• Export Tool Common Arguments
• Data Export Control Arguments
• Data Export Example
• INSERT and UPDATE Statements
• INSERT Operations
• UPDATE Operations
• Example of the Update Operation
• Failed Exports
• Sqoop2
• Summary

Chapter 16 - Introduction to Apache Spark

• What is Apache Spark
• A Short History of Spark
• Where to Get Spark?
• The Spark Platform
• Spark Logo
• Common Spark Use Cases
• Languages Supported by Spark
• Running Spark on a Cluster
• The Driver Process
• Spark Applications
• Spark Shell
• The spark-submit Tool
• The spark-submit Tool Configuration
• The Executor and Worker Processes
• The Spark Application Architecture
• Interfaces with Data Storage Systems
• Limitations of Hadoop's MapReduce
• Spark vs MapReduce
• Spark as an Alternative to Apache Tez
• The Resilient Distributed Dataset (RDD)
• Spark Streaming (Micro-batching)
• Spark SQL
• Example of Spark SQL
• Spark Machine Learning Library
• GraphX
• Spark vs R
• Summary

Chapter 17 - The Spark Shell

• The Spark Shell
• The Spark Shell
• The Spark Shell UI
• Spark Shell Options
• Getting Help
• The Spark Context (sc) and SQL Context (sqlContext)
• The Shell Spark Context
• Loading Files
• Saving Files
• Basic Spark ETL Operations
• Summary

Chapter 18 - Spark RDDs

• The Resilient Distributed Dataset (RDD)
• Ways to Create an RDD
• Custom RDDs
• Supported Data Types
• RDD Operations
• RDDs are Immutable
• Spark Actions
• RDD Transformations
• RDD Transformations
• Other RDD Operations
• Chaining RDD Operations
• RDD Lineage
• The Big Picture
• What May Go Wrong
• Checkpointing RDDs
• Local Checkpointing
• Parallelized Collections
• More on parallelize() Method
• The Pair RDD
• Where do I use Pair RDDs?
• Example of Creating a Pair RDD with Map
• Example of Creating a Pair RDD with keyBy
• Miscellaneous Pair RDD Operations
• RDD Caching
• RDD Persistence
• The Tachyon Storage
• Summary

Chapter 19 - Parallel Data Processing with Spark

• Running Spark on a Cluster
• Spark Stand-alone Option
• The High-Level Execution Flow in Stand-alone Spark Cluster
• Data Partitioning
• Data Partitioning Diagram
• Single Local File System RDD Partitioning
• Multiple File RDD Partitioning
• Special Cases for Small-sized Files
• Parallel Data Processing of Partitions
• Parallel Data Processing of Partitions
• Spark Application, Jobs, and Tasks
• Stages and Shuffles
• The "Big Picture"
• Summary

Chapter 20 - Shared Variables in Spark

• Shared Variables in Spark
• Broadcast Variables
• Creating and Using Broadcast Variables
• Example of Using Broadcast Variables
• Accumulators
• Creating and Using Accumulators
• Example of Using Accumulators
• Custom Accumulators
• Summary

Chapter 21 - Introduction to Spark SQL

• What is Spark SQL?
• What is Spark SQL?
• Uniform Data Access with Spark SQL
• Hive Integration
• Hive Interface
• Integration with BI Tools
• Spark SQL is No Longer Experimental Developer API!
• What is a DataFrame?
• The SQLContext Object
• The SQLContext API
• Changes Between Spark SQL 1.3 to 1.4
• Example of Spark SQL (Scala Example)
• Example of Working with a JSON File
• Example of Working with a Parquet File
• Using JDBC Sources
• JDBC Connection Example
• Performance & Scalability of Spark SQL
• Summary
• Chapter 22 - Graph Processing with GraphX
• What is GraphX?
• Supported Languages
• Vertices and Edges
• Graph Terminology
• Example of Property Graph
• The GraphX API
• The GraphX Views
• The Triplet View
• Graph Algorithms
• Graphs and RDDs
• Constructing Graphs
• Graph Operators
• Example of Using GraphX Operators
• GraphX Performance Optimization
• The PageRank Algorithm
• GraphX Support for PageRank
• Summary

Chapter 23 - The Spark Machine Learning Library

• What is MLlib?
• Supported Languages
• MLlib Packages
• Dense and Sparse Vectors
• Labeled Point
• Python Example of Using the LabeledPoint Class
• LIBSVM format
• An Example of a LIBSVM File
• Loading LIBSVM Files
• Local Matrices
• Example of Creating Matrices in MLlib
• Distributed Matrices
• Example of Using a Distributed Matrix
• Classification and Regression Algorithm
• Clustering
• Summary

Chapter 24 - Machine Learning with BigML

• What is BigML?
• How BigML Service Works
• Data Files
• Data Sets
• Data Sets Example
• Models
• Predictions
• The Prediction UI Form
• Text Analysis in BigML
• REST API
• Summary