r/dataflow • u/fhoffa • Feb 01 '18
r/dataflow • u/Feeling_Net2821 • Apr 20 '25
Dataflow - Positive experience
I’ve seen quite a few negative posts about DataFlow, and I’d like to share my own experience — which has actually been quite positive.
I submitted my documents for verification on April 10, 2025. I uploaded everything I believed was relevant to my application — even my previous WES certification, just in case it helped.
On April 11, they requested my birth certificate, which I promptly uploaded.
On April 12, I received another request for documents related to my personal status, so I uploaded my marriage certificate (from a previous marriage) and my divorce decree.
Then on April 13, I received this status update:
On April 16, I got another update:
And finally, on April 18, I received the email I was waiting for:
All in all, it took just 8 days from submission to final verification.
In my opinion, the process works efficiently if your documents are complete and clear. From what I’ve experienced, the delays might often come from missing or unclear submissions — not from DataFlow itself.
Just to clarify: I have no affiliation with DataFlow. This is simply based on my own personal experience, and I hope it helps anyone currently going through the process.
r/dataflow • u/fhoffa • Jun 10 '15
JDA generates replenishment orders in real-time for over 50,000 point-of-sale transactions per minute with Google Cloud Dataflow
r/dataflow • u/fhoffa • May 29 '19
[gif] Getting started with Dataflow/Beam, best explanation yet
r/dataflow • u/fhoffa • Aug 07 '18
Simple backup and replay of streaming events using Cloud Pub/Sub, Cloud Storage, and Cloud Dataflow
r/dataflow • u/fhoffa • Jun 16 '20
Reading NUMERIC fields with BigQueryIO in Apache Beam
r/dataflow • u/fhoffa • Sep 12 '18
How Distributed Shuffle improves scalability and performance in Cloud Dataflow pipelines
r/dataflow • u/fhoffa • Feb 24 '19
An approach to community building from Apache Beam
blogs.apache.orgr/dataflow • u/fhoffa • Jun 26 '20
Building production-ready data pipelines using Dataflow
r/dataflow • u/fhoffa • Oct 16 '18
Apache Beam 2.7.0: KuduIO, Amazon SNS + SqsIO, updates, fixes
r/dataflow • u/TrendingB0T • Jul 14 '21
/r/dataflow hit 1k subscribers yesterday
r/dataflow • u/AntiqueMidnight4 • Jul 19 '21
What does <project-id>/instanceGroupManagers/<dataflow_job_name> not found mean?
One of my dataflow jobs failed because of this error:
Workflow failed. Causes: Error: Message: The resource 'projects/<project-id>/zones/<region-name>/instanceGroupManagers/<job-name>-07170345-ay75-harness' was not found HTTP Code: 404
Do you know what it means?
r/dataflow • u/fluffycatsinabox • Jul 06 '21
How do you handle deadlock when using Dataflow/Beam for ETL jobs into relational DBs?
Hi all,
I'm running into deadlocks when using a job to load data into a CloudSQL (MySQL) database. Currently, we're writing to the database in batches, and we've written some logic to get these batched writes to retry later if they run into a deadlock (within a certain number of retries). The database's isolation level is already set to repeatable read (the least strict isolation).
I have two questions:
Where exactly should deadlocks be handled? I thought databases were supposed to handle deadlocks (e.g. aborting and queueing deadlocked transactions and running them after locked resources get released), but it seems like we don't have a choice but to handle deadlocks with our Beam pipeline. Should I consider this more of a database issue or a pipeline issue? There's also the connector between Beam and CloudSQL- is there anything in there that might help us handle deadlocks?
Are there any best practices around dealing with deadlocks when using Beam for ETL? We're not really working with a huge data set or anything. While it's not surprising that writing to tables is a bottleneck, the deadlocks are rather unexpected.
Thanks in advance!
r/dataflow • u/OrdinaryGanache • Jul 26 '21
Profiling Python Dataflow jobs
How can we profile dataflow jobs written using apache beam python sdk? I know about cloud profiler but I am not sure how it will be used for dataflow jobs? If there is any other service or product or framework I can work with to profile the dataflow job
r/dataflow • u/Snoo_47594 • Jun 19 '20
Industrialization of a ML model using Airflow and Apache BEAM
r/dataflow • u/fhoffa • Jun 19 '20
Decoupling Dataflow with Cloud Tasks and Cloud Functions
r/dataflow • u/fhoffa • Jun 26 '20
differential-privacy/privacy-on-beam: Privacy on Beam is an end-to-end differential privacy solution built on Apache Beam
r/dataflow • u/fhoffa • Dec 05 '17
Fun with Serializable Functions and Dynamic Destinations in Cloud Dataflow
r/dataflow • u/fhoffa • Feb 11 '17
New in Dataflow: Auto scaling visualization
r/dataflow • u/fhoffa • Nov 28 '17
Google Cloud Dataprep: Spreadsheet-Style Data Wrangling Powered by Google Cloud Dataflow
r/dataflow • u/fhoffa • Jul 18 '17
[Podcast] What's Next for Apache Beam? Featuring Frances Perry of Google
r/dataflow • u/fhoffa • Aug 26 '16
Cloud Dataflow can autoscale R programs for massively parallel data processing
r/dataflow • u/fhoffa • Feb 11 '17