What is Cloud Dataflow?
Answer
Cloud Dataflow is a fully managed, serverless stream and batch data processing service based on Apache Beam. You write pipelines using the Apache Beam SDK (Java or Python), and Dataflow manages the distributed execution, auto-scaling, and fault tolerance. Key concepts: Pipeline: the entire data processing graph. PCollection: a distributed dataset. Transforms: operations on PCollections (ParDo, GroupByKey, Combine, Flatten). Windowing: process infinite streams in time windows (tumbling, sliding, session). Watermarks: handle late-arriving data. Common patterns: ETL from Cloud Storage to BigQuery, real-time fraud detection from Pub/Sub, log processing. The same Beam pipeline runs in batch on historical data or streaming on real-time data without code changes.