Confluent Kafka for System Administrators

Outline

Chapter 1. Introduction to Confluent Kafka

• Messaging Architectures – What is Messaging?
• Messaging Architectures – Steps to Messaging
• Messaging Architectures – Messaging Models
• What is Kafka?
• What is Kafka? (Contd.)
• Kafka Overview
• Need for Kafka
• When to Use Kafka?
• Kafka Architecture
• Core concepts in Kafka
• Kafka Topic
• Architecting Topics
• Kafka Partitions
• Kafka Producer
• Kafka Consumer
• Kafka Broker
• Kafka Cluster
• Why Kafka Cluster?
• Sample Multi-Broker Cluster
• Overview of ZooKeeper
• Kafka Cluster & ZooKeeper
• Who Uses Kafka?
• Summary

Chapter 2. Installing and Configuring Confluent Kafka

• Planning for Kafka - Platform
• Planning for Kafka - OS
• Planning for Kafka - Java
• Planning for Kafka - System Requirements
• Installing Confluent Platform (Kafka)
• Downloading Confluent Platform
• Setting up for Development and Testing
• Setting up for Production
• Running in Docker Containers
• Configuration Files
• Starting Kafka
• Using Kafka Command Line Client Tools
• Setting up a Multi-Broker Cluster
• Using Multi-Broker Cluster
• Kafka Cluster Planning
• Kafka Cluster Planning – Producer/Consumer Throughput
• Kafka Cluster Planning – Number of Brokers (and ZooKeepers)
• Kafka Cluster Planning – Sizing for Topics and Partitions
• Kafka Cluster Planning – Sizing for Storage
• Kafka Connect
• Kafka Connect – Configuration Files
• Using Kafka Connect to Import/Export Data
• Summary

Chapter 3. Data Pipelines Overview

• Building Data Pipelines
• Best Practices for Designing Pipelines
• Considerations When Building Data Pipelines
• Timeliness
• Reliability
• High and Varying Throughput
• High and Varying Throughput (Contd.)
• Data Formats
• Data Formats (Contd.)
• Transformations
• Transformations (Contd.)
• Security
• Security Best Practices
• Failure Handling
• Coupling and Agility
• Ad-hoc Pipelines
• Loss of Metadata
• Extreme Processing
• Kafka Connect Versus Producer and Consumer
• Kafka Connect Versus Producer and Consumer (Contd.)
• Summary

Chapter 4. Integrating Kafka with Other Systems

• Introduction to Kafka Integration
• Kafka Connect
• Kafka Connect (Contd.)
• Running Kafka Connect
• Key Configurations for Connect workers:
• Kafka Connect Example – File Source
• Kafka Connect Example – File Sink
• Kafka Connector Example – PostgresSQL to Elasticsearch
• PostgresSQL to Elasticsearch Process
• Write Data to Elasticsearch
• Kafka Connector Scenario – SOAP Web Service to REST
• Building Custom Connectors
• Kafka Connect – Connectors
• Kafka Connect - Tasks
• Kafka Connect - Workers
• Kafka Connect – Workers (Contd.)
• Kafka Connect - Converters and Connect’s data model
• Kafka Connect - Offset management
• Alternatives to Kafka Connect
• Alternatives to Kafka Connect (Contd.)
• Introduction to Hadoop
• Hadoop Components
• Integrating Hadoop with Kafka
• Hadoop Consumers
• Hadoop Consumers (Contd.)
• Hadoop Consumers (Contd.)
• Hadoop Consumers – Produce Topic
• Hadoop Consumers – Fetch Generated Topic
• Kafka at Uber
• Kafka at Uber (Contd.)
• Kafka at LinkedIn
• Kafka at LinkedIn – Core Kafka Services
• Kafka at LinkedIn – Core Kafka Services (Contd.)
• Kafka at LinkedIn – Libraries
• Kafka at LinkedIn – Monitoring and Stream Processing
• Conclusion
• Summary

Chapter 5. Schema Management, Streams and KSQL

• Evolving Schema
• Protobuf (Protocol Buffers) Overview
• Avro Overview
• Managing Data Evolution Using Schemas
• Confluent Platform
• Confluent Schema Registry
• Schema Change and Backward Compatibility
• Collaborating over Schema Change
• Handling Unreadable Messages
• Deleting Data
• Segregating Public and Private Topics
• What is KSQL?
• What is KSQL? (Contd.)
• Starting KSQL
• Using the KSQL CLI
• KSQL Data Types
• Summary
• Conclusion

Chapter 6. Managing Kafka

• Monitoring Kafka
• Confluent Control Center
• Control Center - Screen Shot
• Control Center - Running
• Control Center - Monitoring
• Control Center - Management
• Control Center - Alerts
• Control Center - Development
• Troubleshooting basics
• Zookeeper CLI Shell
• kafka-topics Command Line Script
• console-consumer Command Line Script
• Viewing logs
• Inspecting Message Queue
• Viewing Messages with Control Center
• Viewing messages with REST Proxy
• Maintaining Kafka Cluster
• Updating Kafka version
• Backup and Restore
• Disaster recovery
• Summary

Lab Exercises

• Lab 1. Confluent Kafka Basics
• Lab 2. Confluent Kafka Multiple Brokers and Import/Export Messages
• Lab 3. Securing Confluent Kafka
• Lab 4. Integrating Confluent Kafka with PostgreSQL
• Lab 5. Confluent Kafka Monitoring using Graphite
• Lab 6. KSQL Basics
• Lab 7. Troubleshoot a Kafka Application