r/spark • • Aug 10 '12

AdaCore Responds to Knight Capital Group’s High-frequency Trading Software Error; The Real Lesson from the $440M Bug.

Thumbnail
us.generation-nt.com
2 Upvotes

r/spark • • Aug 10 '12

What is the relationship between SPARK and Ada?

Thumbnail
youtube.com
3 Upvotes

r/spark • • Mar 29 '18

FOSDEM 2017 - Prove with SPARK: No Math, Just Code -- How to prove key properties of Tetris and run it on ARM Cortex M

Thumbnail
archive.fosdem.org
4 Upvotes

r/spark • • May 09 '18

Taking on a Challenge in SPARK -- Formal methods in SPARK

Thumbnail
blog.adacore.com
4 Upvotes

r/spark • • Mar 22 '18

SPARK article: "You think your software works? Prove it!"

Thumbnail
embedded.com
4 Upvotes

r/spark • • Mar 29 '18

SPARKZumo Part 1: Ada and SPARK on Any Platform

Thumbnail
blog.adacore.com
5 Upvotes

r/spark • • Dec 20 '17

Peter Chapin recognized with ACM SIGAda Robert Dewar Award

Thumbnail
eurekalert.org
5 Upvotes

r/spark • • Dec 13 '25

The Undisputed Queen of Safe Programming

Thumbnail medium.com
6 Upvotes

r/spark • • Nov 09 '22

SPARK as good as Rust for safer coding? AdaCore cites Nvidia case study.

Thumbnail
devclass.com
6 Upvotes

r/spark • • Dec 07 '17

Safe Pointers in SPARK 2014

Thumbnail
arxiv.org
7 Upvotes

r/spark • • Jun 06 '25

Article: NVIDIA drives Ada and SPARK into driverless cars - General

Thumbnail
forum.ada-lang.io
7 Upvotes

r/spark • • Sep 20 '25

Minutes of Ada Monthly Meetup, 13th of September

Thumbnail reddit.com
5 Upvotes

r/spark • • May 31 '25

New 0.7.0 Release of Ironclad kernel

Thumbnail
codeberg.org
5 Upvotes

r/spark • • May 30 '25

Best books for spark

7 Upvotes

Anyone pls recommend latest relevant books for learning spark :)


r/spark • • Jan 21 '25

Need Help Optimizing MongoDB and PySpark for Large-Scale Document Processing (300M Documents)

5 Upvotes

Hi,

I’m facing significant challenges while working on a big data pipeline that involves MongoDB and PySpark. Here’s the scenario:

Setup

  • Data volume: 300 million documents in MongoDB.

  • MongoDB cluster: M40 with 3 shards.

  • Spark cluster: Using 50+ executors, each with 8GB RAM and 4 cores.

  • Tasks:

    1. Read 300M documents from MongoDB into Spark and save to GCS.

    2. Delete 30M documents from MongoDB using PySpark.

Challenges

  1. Reading with PySpark crashes MongoDB

    • Using 50+ executors leads to MongoDB nodes going down.

    • I receive errors like Prematurely reached end of stream, causing connection failures and slowing down the process.

    • I'm using normal code to load with pyspark

  2. Deleting documents is extremely slow

    • Deleting 30M documents using PySpark and PyMongo takes 16+ hours.

    • The MongoDB connection is initialized for each partition, and documents are deleted one by one using delete_one

    • Below is the code snippet for the delete

def delete_documents(to_delete_df: DataFrame):
    to_delete_df.foreachPartition(delete_one_documents_partition)

def delete_one_documents_partition(iterator: Iterator[Row]):
    dst = config["sources"]["lg_dst"]
    client = MongoClient(secrets_manager.get("mongodb").get("connection.uri"))
    db = client[dst["database"]]
    collection = db[dst["collection"]]
    for row in iterator:
        collection.delete_one({"_id": ObjectId(row["_id"])})
        client.close()

I will try soon to change to :

def delete_many_documents_partition(iterator: Iterator[Row]):
    dst = config["sources"]["lg_dst"]
    client = MongoClient(secrets_manager.get("mongodb").get("connection.uri"))
    db = client[dst["database"]]
    collection = db[dst["collection"]]
    deleted_ids = [ObjectId(row["_id"]) for row in iterator]
    result = collection.delete_many({"_id": {"$in": deleted_ids}})
    client.close()

Questions

  1. Reading optimization:

    • How can I optimize the reading of 300M documents into PySpark without overloading MongoDB?

    • I’m currently using the MongoPaginateBySizePartitioner with a partitionSizeMB of 64MB, but it still causes crashes.

  2. Deletion optimization:

    • How can I improve the performance of the deletion process?

    • Is there a better way to batch deletes or parallelize them while avoiding MongoDB overhead?

Additional Info

  • Network and storage resources appear sufficient, but I suspect there’s room for improvement in configuration or design.

  • Any suggestions on improving MongoDB settings, Spark configurations, or even alternative approaches would be greatly appreciated.

Thanks for your help! Let me know if you need more details.


r/spark • • May 15 '25

Spark Loops

1 Upvotes

i am confusing about what is meant by spark loops and i do not found any example about it
can any one help?


r/spark • • May 09 '25

Formal Methods for an Insecure World

Thumbnail
blog.adacore.com
6 Upvotes

r/spark • • Jun 28 '21

New Competition: Ada/SPARK Crate Of The Year Award

Thumbnail
blog.adacore.com
17 Upvotes

r/spark • • Feb 22 '26

Ada/SPARK Crate Of The Year 2025 Winners Announced

Thumbnail
adacore.com
5 Upvotes

r/spark • • Jul 05 '26

Announcing the 2026 Ada/SPARK Crate of the Year Award

Thumbnail
adacore.com
3 Upvotes

r/spark • • Dec 07 '22

How to apply different code to different blocks from XML files?

4 Upvotes

I am working with xml files that can have seven different types of blocks. What is the most efficient way to correctly identify each block and apply code to it based on its identity?

Are iterators the solution?


r/spark • • Nov 26 '22

NVIDIA Security Team: “What if we just stopped using C?"

Thumbnail
blog.adacore.com
1 Upvotes

r/spark • • Nov 05 '17

Rust and SPARK: Software Reliability for Everyone

Thumbnail
electronicdesign.com
4 Upvotes

r/spark • • Nov 05 '17

GNAT GPL 2017, including support for SPARK GPL, is out!

Thumbnail
blog.adacore.com
4 Upvotes