Corporate Databricks Training Course

1. What is Apache Spark?

2. Defining big data

3. Spark languages

• Scala
• Python
• R
• Java
• SQL

4. Introduction to Databricks community edition

5. Databricks architecture

6. Understanding data analytics

7. Understanding machine learning

8. Azure implementation of Databricks

9. AWS implementation of Databricks

1. Creating a Databricks workspace on Azure
   • Account setup
   • Workspace creation and configuration
2. Creating and configuring your Databricks cluster
   • Cluster types and specifications
   • Instance sizing and scaling options
   • Cluster configuration settings
3. Creating and attaching your first notebook in Databricks
   • Notebook creation and organization
   • Notebook settings and configurations
4. Testing and running your Notebook in Databricks
   • Executing code cells
   • Debugging and troubleshooting
   • Monitoring and logging

1. Creating a table in Databricks
   • Table creation options
   • Schema definition and data types
2. Connecting to a Spark data source
   • Importing data from various sources (e.g., CSV, JSON, databases)
   • Configuring connection settings
3. Previewing data in your Table
   • Sampling data
   • Exploring data structure and content
4. Basics of columns and datatypes in Databricks
   • Working with columns and column operations
   • Understanding data types and conversions

1. Writing SQL queries to import data into Databricks notebook
   • SELECT, INSERT, and other SQL operations
   • Filtering and sorting data
2. Viewing aggregates in Databricks
   • Aggregation functions (e.g., COUNT, SUM, AVG)
   • Grouping and aggregating data
3. Performing joins in Databricks
   • Different types of joins (e.g., inner, outer, left, right)
   • Joining multiple tables
   • Handling join conditions and duplicates

1. Understanding data types in Databricks
   • Numeric, string, boolean, and complex data types
   • Type conversions and casting
2. Working with DataFrames in Databricks
   • DataFrame creation and manipulation
   • Filtering, transforming, and aggregating data
3. Handling Images in Databricks
   • Image data formats and processing
   • Image manipulation and analysis
4. Exploring structured streaming DataFrames in Databricks
   • Real-time data processing and analysis
   • Windowing and time-based operations
5. Creating plots in Databricks
   • Data visualization libraries and tools in Databricks
   • Plotting charts, graphs, and histograms
6. Choosing chart types and using the chart toolbar
   • Customizing chart appearance and styling
   • Selecting appropriate chart types for different data scenarios
7. Layout and styling considerations for visualizations
   • Arranging multiple charts and visuals
   • Adding titles, labels, and annotations
8. Visualizations for machine learning in Databricks
   • Visualizing model outputs and predictions
   • Interpreting model performance metrics

1. Creating a Job in Databricks
   • Job configuration settings
   • Defining job tasks and workflows
2. Viewing Jobs and Job details in Databricks
   • Monitoring job status and progress
   • Accessing job logs and history
3. Running your first Job in Databricks
   • Executing job tasks
   • Handling job dependencies and inputs
4. Scheduling Jobs in Databricks
   • Setting up recurring and automated job runs
   • Configuring scheduling options
5. Setting parameters for Jobs in Databricks
   • Parameterizing job tasks and inputs
   • Passing parameters at runtime
6. Viewing completed jobs in Databricks
   • Analyzing job results and outputs
   • Reviewing job performance metrics
7. Managing dependencies in Databricks
   • Handling interdependent tasks and workflows
   • Managing job dependencies and execution order
8. Setting up alerts for Databricks jobs
   • Configuring email notifications and alerts
   • Monitoring job failures and critical events

1. Getting data into Delta Lake in Databricks
   • Creating Delta tables
   • Importing data into Delta format
2. Performing delete, update, and merge operations in Delta tables
   • Modifying data in Delta Tables
   • Handling updates and deletes efficiently
3. Working with constraints in Delta Tables
   • Defining constraints (e.g., uniqueness, referential integrity)
   • Enforcing data integrity in Delta Tables
4. Versioning in Delta Tables
   • Tracking and managing table versions
   • Rollbacks and time travel capabilities
5. Concurrency considerations in Delta Tables
   • Handling concurrent read and write operations
   • Managing conflicts and consistency
6. Integrations with other systems in Databricks
   • Integrating Delta tables with other data systems
   • Data sharing and interoperability
7. Overview of Delta engine in Databricks
   • Accelerating query performance with Delta engine
   • Benefits and use cases of Delta engine

1. Collaboration features in Databricks
   • Notebooks collaboration
   • Workspace sharing and permissions
   • Version control and Git integration
2. Scaling capabilities of Databricks
   • Auto-scaling clusters
   • High-performance distributed computing
3. Integrating Databricks into data pipelines
   • Data ingestion and ETL processes
   • Integration with data storage systems
   • Streamlining data workflows