Source
Serverless Apache Spark on Google Cloud: Architecture Choices & AI Troubleshooting
This technical guide covers architectural choices for deploying Apache Spark on Google Cloud and techniques for cost optimization. It specifically highlights the Managed Service for Apache Spark and Gemini Cloud Assist for troubleshooting.
Why it matters
Data engineers can reduce time spent managing infrastructure and sifting through logs, allowing them to resolve pipeline failures faster and optimize compute costs.
The details
- Serverless batches eliminate idle compute costs by provisioning resources only during execution.
- History-based autotuning optimizes recurring batch workloads using telemetry from previous runs.
- Gemini Cloud Assist enables natural language diagnosis of pipeline failures and suggests fixes.
Show entities and relationshipsHide entities and relationships
In this article
Key connections
Google owns Managed Service for Apache Airflow
Google provides Managed Service for Apache Airflow as an orchestrator for Spark batch pipelines.
Google owns Cloud Scheduler
Cloud Scheduler is a Google Cloud service used to trigger serverless batch jobs on schedule.
Managed Service for Apache Spark uses PySpark
Serverless batch jobs on Managed Service for Apache Spark are authored as PySpark scripts.
Gemini Cloud Assist is related to Managed Service for Apache Spark
Gemini Cloud Assist is natively integrated into Managed Service for Apache Spark for AI-powered troubleshooting of batch pipeline failures.
Managed Service for Apache Airflow uses Managed Service for Apache Spark
Managed Service for Apache Airflow orchestrates serverless batch jobs on Managed Service for Apache Spark in production.
Cloud Scheduler uses Managed Service for Apache Spark
Cloud Scheduler can trigger serverless batch jobs on Managed Service for Apache Spark on a defined schedule.
Show 2 more connectionsShow fewer connections
Managed Service for Apache Spark uses Google Cloud Storage
Spark ETL pipelines read customer transaction data from Google Cloud Storage buckets.
Managed Service for Apache Spark is built with Apache Spark
Google Cloud Managed Service for Apache Spark is built on open-source Apache Spark.
Get the weekly recap
The stories like this one, picked and explained — once a week, straight to your inbox.