r/dataflow • u/Mayor18 • May 26 '21
Dataflow Prime
r/dataflow • u/fhoffa • Apr 15 '20
r/dataflow • u/fhoffa • Apr 18 '20
r/dataflow • u/fhoffa • Aug 07 '19
r/dataflow • u/TrendingB0T • Jul 14 '21
r/dataflow • u/fluffycatsinabox • Jul 06 '21
Hi all,
I'm running into deadlocks when using a job to load data into a CloudSQL (MySQL) database. Currently, we're writing to the database in batches, and we've written some logic to get these batched writes to retry later if they run into a deadlock (within a certain number of retries). The database's isolation level is already set to repeatable read (the least strict isolation).
I have two questions:
Where exactly should deadlocks be handled? I thought databases were supposed to handle deadlocks (e.g. aborting and queueing deadlocked transactions and running them after locked resources get released), but it seems like we don't have a choice but to handle deadlocks with our Beam pipeline. Should I consider this more of a database issue or a pipeline issue? There's also the connector between Beam and CloudSQL- is there anything in there that might help us handle deadlocks?
Are there any best practices around dealing with deadlocks when using Beam for ETL? We're not really working with a huge data set or anything. While it's not surprising that writing to tables is a bottleneck, the deadlocks are rather unexpected.
Thanks in advance!
r/dataflow • u/shravan_rcb • Nov 13 '20
“DataPiepeline using Apache Beam and Google Cloud DataFlow as Runner and BigQuery as DataSink” by Shravan C https://link.medium.com/RuCCuVANmbb
r/dataflow • u/fhoffa • Nov 03 '20
r/dataflow • u/fhoffa • Apr 11 '20
r/dataflow • u/fhoffa • Oct 10 '19
r/dataflow • u/fhoffa • Nov 27 '18
r/dataflow • u/fhoffa • Nov 28 '17
r/dataflow • u/fhoffa • Jan 18 '17
r/dataflow • u/fhoffa • Jan 13 '17
r/dataflow • u/fhoffa • Jan 20 '16
r/dataflow • u/fhoffa • Feb 18 '20
r/dataflow • u/fhoffa • Feb 12 '20
r/dataflow • u/fhoffa • Jan 29 '20
r/dataflow • u/fhoffa • May 17 '19
r/dataflow • u/fhoffa • Apr 13 '19
r/dataflow • u/fhoffa • Aug 13 '18