Overview

What is Kubeflow Spark Operator?

The Kubernetes Operator for Apache Spark aims to make specifying and running Spark applications as easy and idiomatic as running other workloads on Kubernetes. It uses Kubernetes custom resources for specifying, running, and surfacing status of Spark applications.

Benefits

• Declarative Workload Management: Specify and manage Spark applications declaratively using standard Kubernetes YAML files and Custom Resource Definitions (SparkApplication) (User Guide: Writing a SparkApplication and API Reference).
• Automated Execution: Automatically handles spark-submit on behalf of users whenever a valid application manifest is created or updated (Overview: Architecture).
• Native Cron Scheduling: Provides built-in cron support for running recurring Spark jobs on scheduled intervals (User Guide: Running Spark Applications on a Schedule).
• Advanced Pod Customization: Utilizes a mutating admission webhook to perform pod customizations beyond Spark’s native capabilities, such as mounting ConfigMaps and volumes or configuring pod affinity/anti-affinity (Overview: Mutating Admission Webhook and User Guide: Customizing Spark Operator).
• Failure Handling & Auto-Restart: Supports configurable restart policies, automatic re-submissions upon spec updates, and submission retries with configurable linear back-off (Overview: Handling Application Restart And Failures).
• Prometheus Observability: Native support for collecting and exporting application-level, driver, and executor metrics directly to Prometheus (User Guide: Monitoring Spark Applications).

Architecture

Next Steps

1. Follow the Getting Started Guide to deploy the Spark Operator into your Kubernetes cluster.
2. Read the User Guide (including Writing a SparkApplication) to learn how to write, configure, and submit your first SparkApplication manifest.