This document explains the key concepts, fundamental building blocks, core features, and benefits of Managed Service for Apache Spark. Understanding these fundamentals help you effectively use Managed Service for Apache Spark for your data processing tasks.
The cluster-based model
Managed Service for Apache Spark on clusters is the standard, infrastructure-centric way of using Managed Service for Apache Spark. It gives you full control over a dedicated set of virtual machines for your data processing tasks.
- Clusters: A cluster is your personal data processing engine, made up of Cloud de Confiance by S3NS virtual machines. You create a cluster to run open-source frameworks such as Apache Spark and Apache Hadoop. You have full control over cluster size, machine types, and configuration.
- Jobs: A job is a specific task, such as a PySpark script or Hadoop query. Instead of running a job directly on a cluster, you submit the job to Managed Service for Apache Spark, which manages job execution for you. You can submit multiple jobs to the cluster.
- Workflow Templates: A workflow template is a reusable definition that orchestrates a series of jobs (a workflow). It can define dependencies between jobs, for example to run a machine learning job only after a data cleaning job successfully completes. The templated workflow can run on an existing cluster or on a temporary (ephemeral) cluster that is created to run the workflow, and then deleted after the workflow completes. You can use the template to run the defined workflow whenever needed.
- Autoscaling policies: An autoscaling policy contains rules that you define to add or remove worker machines from a cluster based upon cluster workload in order to dynamically optimize cluster cost and performance.
Environment customization
Managed Service for Apache Spark on clusters offers cluster features and components you can use to customize your application environment.
Notebook and development environments
Managed Service for Apache Spark serverless notebooks and IDEs link to integrated development environments where you can write and execute your code.
- BigQuery Studio & Workbench: These are unified analytics and notebook environments. They allow you to write code (for example in a Jupyter notebook) and use a Managed Service for Apache Spark cluster or serverless session as the powerful backend engine to execute your code on large datasets.
- Managed Service for Apache Spark JupyterLab Plugin: This official
JupyterLabextensionacts as a control panel for Managed Service for Apache Spark serverless inside your notebook environment. It simplifies your workflow by allowing you to browse, create, and manage clusters and submit jobs without having to leave the Jupyter interface. - Google Cloud Data Agent Kit: Data Agent Kit lets data scientists, engineers, and developers manage their entire data workload lifecycle within their IDE. Data Agent Kit provides support for Managed Service for Apache Spark, enabling you to develop code, create interactive sessions, and build pipelines directly from VS Code or a supported agentic CLI.
- Managed Service for Apache Spark Connect Python Connector: This Python library streamlines the process of using Spark Connect with Managed Service for Apache Spark. It handles authentication and endpoint configuration, making it much simpler to connect your local Python environment, such as a notebook or IDE, to a remote Managed Service for Apache Spark cluster for interactive development.