r/spark • u/Wootery • Aug 06 '26
SPARK Doesn't Comply With MISRA C. It Makes Most of It Moot. [AdaCore blog post]
r/spark • u/Pleeb • Oct 25 '23
The moderation team for r/SPARK hasn't been around for a while, and the subreddit has been flooded with questions related to Apache Spark, PySpark, etc. and I've claimed ownership of the subreddit.
I've went through and removed the last several posts involving those, but if your post got caught up in the crossfire while actually being related to SPARK (that is, the subset of Ada) then please write to the mods and let us know.
Hoping to help bring this subreddit back to prosperity once again š
r/spark • u/Wootery • Aug 06 '26
r/spark • u/Wootery • Jul 05 '26
r/spark • u/Wootery • Feb 22 '26
r/spark • u/Dirk042 • Nov 23 '25
r/spark • u/Dirk042 • Sep 20 '25
r/spark • u/dragon_spirit_wtp • Jun 06 '25
r/spark • u/dragon_spirit_wtp • Jun 04 '25
r/spark • u/adacore1 • Jun 04 '25
We will be holding a free webinar on theĀ 18th of June introducing SPARK, the industrial-strength language and toolset for formal verification.
Discover how SPARK enables:
Modular, scalable proof for embedded and high-integrity systems
Automatic error detection through deductive verification
Real-world assurance ā including an example from a M.A.R.S. Rover safety monitor
Future-ready development with applications in Generative AI
Whether youāre new to SPARK or exploring formal methods for safety-critical software, this session will give you practical insights into how formal verification works ā and why it matters.
Sign up - https://bit.ly/4jvqC7F
r/spark • u/dragon_spirit_wtp • May 31 '25
r/spark • u/lostengineer7 • May 30 '25
Anyone pls recommend latest relevant books for learning spark :)
r/spark • u/BreadfruitClear7098 • May 15 '25
i am confusing about what is meant by spark loops and i do not found any example about it
can any one help?
r/spark • u/Dirk042 • Feb 13 '25
I didnāt see this mentioned anywhere yet. Would appreciate comments/review by āSPARK knowledgeableā people.
r/spark • u/nidalap24 • Jan 21 '25
Hi,
Iām facing significant challenges while working on a big data pipeline that involves MongoDB and PySpark. Hereās the scenario:
Data volume: 300 million documents in MongoDB.
MongoDB cluster: M40 with 3 shards.
Spark cluster: Using 50+ executors, each with 8GB RAM and 4 cores.
Tasks:
Read 300M documents from MongoDB into Spark and save to GCS.
Delete 30M documents from MongoDB using PySpark.
Reading with PySpark crashes MongoDB
Using 50+ executors leads to MongoDB nodes going down.
I receive errors like Prematurely reached end of stream, causing connection failures and slowing down the process.
I'm using normal code to load with pyspark
Deleting documents is extremely slow
Deleting 30M documents using PySpark and PyMongo takes 16+ hours.
The MongoDB connection is initialized for each partition, and documents are deleted one by one using delete_one
Below is the code snippet for the delete
def delete_documents(to_delete_df: DataFrame):
to_delete_df.foreachPartition(delete_one_documents_partition)
def delete_one_documents_partition(iterator: Iterator[Row]):
dst = config["sources"]["lg_dst"]
client = MongoClient(secrets_manager.get("mongodb").get("connection.uri"))
db = client[dst["database"]]
collection = db[dst["collection"]]
for row in iterator:
collection.delete_one({"_id": ObjectId(row["_id"])})
client.close()I will try soon to change to :
def delete_many_documents_partition(iterator: Iterator[Row]):
dst = config["sources"]["lg_dst"]
client = MongoClient(secrets_manager.get("mongodb").get("connection.uri"))
db = client[dst["database"]]
collection = db[dst["collection"]]
deleted_ids = [ObjectId(row["_id"]) for row in iterator]
result = collection.delete_many({"_id": {"$in": deleted_ids}})
client.close()Reading optimization:
How can I optimize the reading of 300M documents into PySpark without overloading MongoDB?
Iām currently using the MongoPaginateBySizePartitioner with a partitionSizeMB of 64MB, but it still causes crashes.
Deletion optimization:
How can I improve the performance of the deletion process?
Is there a better way to batch deletes or parallelize them while avoiding MongoDB overhead?
Network and storage resources appear sufficient, but I suspect thereās room for improvement in configuration or design.
Any suggestions on improving MongoDB settings, Spark configurations, or even alternative approaches would be greatly appreciated.
Thanks for your help! Let me know if you need more details.
r/spark • u/Wootery • Nov 28 '24
r/spark • u/Wootery • May 07 '24
r/spark • u/Wootery • May 03 '24
r/spark • u/Wootery • Mar 02 '24