r/spark • u/Bhima • Aug 10 '12
r/spark • u/Bhima • Aug 10 '12
What is the relationship between SPARK and Ada?
r/spark • u/Bhima • Mar 29 '18
FOSDEM 2017 - Prove with SPARK: No Math, Just Code -- How to prove key properties of Tetris and run it on ARM Cortex M
r/spark • u/Bhima • May 09 '18
Taking on a Challenge in SPARK -- Formal methods in SPARK
r/spark • u/Bhima • Mar 22 '18
SPARK article: "You think your software works? Prove it!"
r/spark • u/Bhima • Mar 29 '18
SPARKZumo Part 1: Ada and SPARK on Any Platform
r/spark • u/Bhima • Dec 20 '17
Peter Chapin recognized with ACM SIGAda Robert Dewar Award
r/spark • u/Bhima • Nov 09 '22
SPARK as good as Rust for safer coding? AdaCore cites Nvidia case study.
r/spark • u/dragon_spirit_wtp • Jun 06 '25
Article: NVIDIA drives Ada and SPARK into driverless cars - General
r/spark • u/Dirk042 • Sep 20 '25
Minutes of Ada Monthly Meetup, 13th of September
reddit.comr/spark • u/dragon_spirit_wtp • May 31 '25
New 0.7.0 Release of Ironclad kernel
r/spark • u/lostengineer7 • May 30 '25
Best books for spark
Anyone pls recommend latest relevant books for learning spark :)
r/spark • u/nidalap24 • Jan 21 '25
Need Help Optimizing MongoDB and PySpark for Large-Scale Document Processing (300M Documents)
Hi,
I’m facing significant challenges while working on a big data pipeline that involves MongoDB and PySpark. Here’s the scenario:
Setup
Data volume: 300 million documents in MongoDB.
MongoDB cluster: M40 with 3 shards.
Spark cluster: Using 50+ executors, each with 8GB RAM and 4 cores.
Tasks:
Read 300M documents from MongoDB into Spark and save to GCS.
Delete 30M documents from MongoDB using PySpark.
Challenges
Reading with PySpark crashes MongoDB
Using 50+ executors leads to MongoDB nodes going down.
I receive errors like
Prematurely reached end of stream, causing connection failures and slowing down the process.I'm using normal code to load with pyspark
Deleting documents is extremely slow
Deleting 30M documents using PySpark and PyMongo takes 16+ hours.
The MongoDB connection is initialized for each partition, and documents are deleted one by one using
delete_oneBelow is the code snippet for the delete
def delete_documents(to_delete_df: DataFrame):
to_delete_df.foreachPartition(delete_one_documents_partition)
def delete_one_documents_partition(iterator: Iterator[Row]):
dst = config["sources"]["lg_dst"]
client = MongoClient(secrets_manager.get("mongodb").get("connection.uri"))
db = client[dst["database"]]
collection = db[dst["collection"]]
for row in iterator:
collection.delete_one({"_id": ObjectId(row["_id"])})
client.close()I will try soon to change to :
def delete_many_documents_partition(iterator: Iterator[Row]):
dst = config["sources"]["lg_dst"]
client = MongoClient(secrets_manager.get("mongodb").get("connection.uri"))
db = client[dst["database"]]
collection = db[dst["collection"]]
deleted_ids = [ObjectId(row["_id"]) for row in iterator]
result = collection.delete_many({"_id": {"$in": deleted_ids}})
client.close()Questions
Reading optimization:
How can I optimize the reading of 300M documents into PySpark without overloading MongoDB?
I’m currently using the
MongoPaginateBySizePartitionerwith apartitionSizeMBof 64MB, but it still causes crashes.
Deletion optimization:
How can I improve the performance of the deletion process?
Is there a better way to batch deletes or parallelize them while avoiding MongoDB overhead?
Additional Info
Network and storage resources appear sufficient, but I suspect there’s room for improvement in configuration or design.
Any suggestions on improving MongoDB settings, Spark configurations, or even alternative approaches would be greatly appreciated.
Thanks for your help! Let me know if you need more details.
r/spark • u/BreadfruitClear7098 • May 15 '25
Spark Loops
i am confusing about what is meant by spark loops and i do not found any example about it
can any one help?
r/spark • u/Fabien_C • Jun 28 '21
New Competition: Ada/SPARK Crate Of The Year Award
r/spark • u/Wootery • Feb 22 '26
Ada/SPARK Crate Of The Year 2025 Winners Announced
r/spark • u/Wootery • Jul 05 '26
Announcing the 2026 Ada/SPARK Crate of the Year Award
r/spark • u/BewitchedHare • Dec 07 '22
How to apply different code to different blocks from XML files?
I am working with xml files that can have seven different types of blocks. What is the most efficient way to correctly identify each block and apply code to it based on its identity?
Are iterators the solution?
r/spark • u/Wootery • Nov 26 '22
NVIDIA Security Team: “What if we just stopped using C?"
r/spark • u/Bhima • Nov 05 '17
Rust and SPARK: Software Reliability for Everyone
r/spark • u/Bhima • Nov 05 '17