r/dataflow • • Feb 01 '18

Apache Beam lowers barriers to entry for big data processing technologies

Thumbnail
oreilly.com
3 Upvotes

r/dataflow • • Apr 20 '25

Dataflow - Positive experience

0 Upvotes

I’ve seen quite a few negative posts about DataFlow, and I’d like to share my own experience — which has actually been quite positive.

I submitted my documents for verification on April 10, 2025. I uploaded everything I believed was relevant to my application — even my previous WES certification, just in case it helped.

On April 11, they requested my birth certificate, which I promptly uploaded.
On April 12, I received another request for documents related to my personal status, so I uploaded my marriage certificate (from a previous marriage) and my divorce decree.

Then on April 13, I received this status update:

On April 16, I got another update:

And finally, on April 18, I received the email I was waiting for:

All in all, it took just 8 days from submission to final verification.

In my opinion, the process works efficiently if your documents are complete and clear. From what I’ve experienced, the delays might often come from missing or unclear submissions — not from DataFlow itself.

Just to clarify: I have no affiliation with DataFlow. This is simply based on my own personal experience, and I hope it helps anyone currently going through the process.


r/dataflow • • Jun 10 '15

JDA generates replenishment orders in real-time for over 50,000 point-of-sale transactions per minute with Google Cloud Dataflow

Thumbnail
googlecloudplatform.blogspot.com
1 Upvotes

r/dataflow • • Nov 30 '17

How to read multiline CSV in dataflow (java sdk)?

1 Upvotes

r/dataflow • • May 29 '19

[gif] Getting started with Dataflow/Beam, best explanation yet

Thumbnail
twitter.com
1 Upvotes

r/dataflow • • Aug 07 '18

Simple backup and replay of streaming events using Cloud Pub/Sub, Cloud Storage, and Cloud Dataflow

Thumbnail
cloud.google.com
0 Upvotes

r/dataflow • • Jun 16 '20

Reading NUMERIC fields with BigQueryIO in Apache Beam

Thumbnail
medium.com
1 Upvotes

r/dataflow • • Sep 12 '18

How Distributed Shuffle improves scalability and performance in Cloud Dataflow pipelines

Thumbnail
cloud.google.com
2 Upvotes

r/dataflow • • Feb 24 '19

An approach to community building from Apache Beam

Thumbnail blogs.apache.org
3 Upvotes

r/dataflow • • Jun 26 '20

Building production-ready data pipelines using Dataflow

Thumbnail
cloud.google.com
4 Upvotes

r/dataflow • • Oct 16 '18

Apache Beam 2.7.0: KuduIO, Amazon SNS + SqsIO, updates, fixes

Thumbnail
beam.apache.org
4 Upvotes

r/dataflow • • Jul 14 '21

/r/dataflow hit 1k subscribers yesterday

Thumbnail
frontpagemetrics.com
6 Upvotes

r/dataflow • • Jul 19 '21

What does <project-id>/instanceGroupManagers/<dataflow_job_name> not found mean?

1 Upvotes

One of my dataflow jobs failed because of this error:

Workflow failed. Causes: Error: Message: The resource 'projects/<project-id>/zones/<region-name>/instanceGroupManagers/<job-name>-07170345-ay75-harness' was not found HTTP Code: 404

Do you know what it means?


r/dataflow • • Jul 06 '21

How do you handle deadlock when using Dataflow/Beam for ETL jobs into relational DBs?

6 Upvotes

Hi all,

I'm running into deadlocks when using a job to load data into a CloudSQL (MySQL) database. Currently, we're writing to the database in batches, and we've written some logic to get these batched writes to retry later if they run into a deadlock (within a certain number of retries). The database's isolation level is already set to repeatable read (the least strict isolation).

I have two questions:

  1. Where exactly should deadlocks be handled? I thought databases were supposed to handle deadlocks (e.g. aborting and queueing deadlocked transactions and running them after locked resources get released), but it seems like we don't have a choice but to handle deadlocks with our Beam pipeline. Should I consider this more of a database issue or a pipeline issue? There's also the connector between Beam and CloudSQL- is there anything in there that might help us handle deadlocks?

  2. Are there any best practices around dealing with deadlocks when using Beam for ETL? We're not really working with a huge data set or anything. While it's not surprising that writing to tables is a bottleneck, the deadlocks are rather unexpected.

Thanks in advance!


r/dataflow • • Jul 26 '21

Profiling Python Dataflow jobs

2 Upvotes

How can we profile dataflow jobs written using apache beam python sdk? I know about cloud profiler but I am not sure how it will be used for dataflow jobs? If there is any other service or product or framework I can work with to profile the dataflow job


r/dataflow • • Jun 19 '20

Industrialization of a ML model using Airflow and Apache BEAM

Thumbnail
medium.com
4 Upvotes

r/dataflow • • Jun 19 '20

Decoupling Dataflow with Cloud Tasks and Cloud Functions

Thumbnail
medium.com
1 Upvotes

r/dataflow • • May 21 '20

Predicting the cost of a Dataflow job

Thumbnail
cloud.google.com
7 Upvotes

r/dataflow • • Jun 26 '20

differential-privacy/privacy-on-beam: Privacy on Beam is an end-to-end differential privacy solution built on Apache Beam

Thumbnail
github.com
3 Upvotes

r/dataflow • • Dec 05 '17

Fun with Serializable Functions and Dynamic Destinations in Cloud Dataflow

Thumbnail
shinesolutions.com
1 Upvotes

r/dataflow • • Feb 11 '17

New in Dataflow: Auto scaling visualization

Thumbnail
cloud.google.com
1 Upvotes

r/dataflow • • Nov 28 '17

Google Cloud Dataprep: Spreadsheet-Style Data Wrangling Powered by Google Cloud Dataflow

Thumbnail
medium.com
6 Upvotes

r/dataflow • • Jul 18 '17

[Podcast] What's Next for Apache Beam? Featuring Frances Perry of Google

Thumbnail
talend.com
2 Upvotes

r/dataflow • • Aug 26 '16

Cloud Dataflow can autoscale R programs for massively parallel data processing

Thumbnail
medium.com
3 Upvotes

r/dataflow • • Feb 11 '17

Release Notes: Dataflow SDK 2.0.0-beta2 for Java

Thumbnail
cloud.google.com
1 Upvotes