Dataproc

Run demanding Spark workloads with the control of a managed Dataproc cluster, now supercharged with 3.6x* query speed by the Lightning Engine,** in Preview. Experience significant performance gains for Spark SQL and DataFrame operations. Configure Spark environments precisely to your needs, choosing versions and libraries.

*The queries are derived from the TPC-DS standard and TPC-H standard and as such are not comparable to published TPC-DS standard and TPC-H standard results, as these runs do not comply with all requirements of the TPC-DS standard and TPC-H standard specification.

**Available for Dataproc on Compute Engine premium tier.

Customize Dataproc clusters with a wide range of machine types (including GPUs), preemptible VMs, disk options, autoscaling policies, initialization actions, container/images, and optional components. Use features like Workflow Templates for orchestrating complex jobs and manage clusters via the console, gcloud, API, or client libraries. Gain deep visibility into cluster performance and health through integration with Cloud Monitoring, providing comprehensive metrics, dashboards, and alerting capabilities.

Dataproc clusters integrate natively with BigLake Metastore, allowing you to process data stored in open formats like Apache Iceberg on Cloud Storage. For traditional Hive-based metadata needs, there’s seamless integration with the managed Dataproc Metastore service. Leverage Dataplex Universal Catalog for unified discovery, lineage, and governance across your lakehouse assets. Extend your data applications by connecting Dataproc with BigQuery, Vertex AI, Spanner, Pub/Sub, and Data Fusion, creating powerful, end-to-end solutions.

Benefit from Google Cloud's robust security. Configure Kerberos, manage access with IAM, enforce network policies with VPC Service Controls, and use CMEK. Integrate with Dataplex Universal Catalog for centralized policy management and enable fine grained access control with BigLake.