r/apachespark • • Aug 05 '23

Spark now lets you easily query a DataFrame with SQL

Post image
103 Upvotes

r/apachespark • • Feb 05 '21

PSA: The Spark maintainers are intentionally avoiding exposing certain Spark SQL functions via the Scala / PySpark APIs

94 Upvotes

They're adding some great new methods to the Spark SQL API but these methods won't necessarily be added to the Scala / PySpark APIs.

regexp_extract_all was actually removed from the Scala API before the 3.1 release and it's uncertain if regexp_extract_all will be added to PySpark.

I asked the mailing list about making the APIs consistent and the maintainers made arguments for having this functionality in SQL, but not the other APIs. They'd like determination on a case-by-case basis. I'd prefer to plug the functionality gaps in all the APIs, but the maintainers are smarter than me and I respect their decision.

I started a separate project called bebe to add the functions that won't be implemented in Spark codebase. I'll also be implementing some other developer friendly functions as Spark native functions, so there is a performant solution for the community and folks don't need to use UDFs.

Feel free to send me function requests (e.g. beginning_of_month, jaro_winkler, etc) and I'll try to get them implemented as performant native Spark functions.


r/apachespark • • May 29 '25

Spark 4.0.0 released!

Thumbnail spark.apache.org
81 Upvotes

r/apachespark • • Aug 27 '20

Not sure if this has been done here, but wanted to share.

Post image
81 Upvotes

r/apachespark • • Jun 19 '20

Spark Release 3.0.0

Thumbnail
spark.apache.org
73 Upvotes

r/apachespark • • May 09 '25

Waiting for Scala 3 native support be like

Post image
70 Upvotes

r/apachespark • • Mar 07 '21

Spark 3.1.1 released

Thumbnail
spark.apache.org
68 Upvotes

r/apachespark • • Apr 19 '21

We're publicly releasing our optimized Docker Images for Spark - they contain Spark, Scala, Java, Hadoop, Python, and connectors to S3, GCS, Azure Data Lake, Snowflake and Delta Lake.

Thumbnail
datamechanics.co
57 Upvotes

r/apachespark • • Feb 21 '26

Spark Theory for Data Engineers

51 Upvotes

Hi everyone, I'm building Spark Playground and have added a Spark Theory section with 9 in-depth tutorials covering these concepts:

  1. Introduction to Apache Spark

  2. Spark Architecture

  3. Transformations & Actions

  4. Resilient Distributed Dataset (RDD)

  5. DataFrames & Datasets

  6. Lazy Evaluation

  7. Catalyst Optimizer

  8. Jobs, Stages, and Tasks

  9. Adaptive Query Execution (AQE)

Disclaimer - content is created with the help of AI, reviewed, checked and edited by me.

Each tutorial breaks down Spark topics with practical examples, configuration snippets, comparison tables, and performance trade-offs. Written from a data engineering perspective.

Ongoing WIP: planning to add more topics like join strategies, partitioning strategies, caching & persistence, memory management etc.

If you'd like to help write tutorials, improve existing content, or suggest topics, the tutorials are open-source:

GitHub: https://github.com/rizal-rovins/learn-pyspark

Let me know what Spark topics would you find most valuable to see covered next


r/apachespark • • Jul 27 '23

Spark 3.4+ supports lateral column aliases

Post image
50 Upvotes

r/apachespark • • Apr 14 '23

Spark 3.4 released

Thumbnail spark.apache.org
50 Upvotes

r/apachespark • • Feb 23 '21

My experience contributing to the Spark codebase

49 Upvotes

I recently got two pull requests merged into the Spark codebase:

I closed the add_hours() function in favor or make_interval.

Contributing to the codebase was a positive experience. There are a bunch of great developers that build an maintain the Spark codebase. Many of them have a good understanding of the entire codebase and collaborating with them teaches you a lot.

I even started working with one of the Spark contributors on a side project called itachi for functions that won't get merged into Spark, He's also helping me out with code reviews on the bebe project I'm working on. Spark folks are friendly & willing to help!

Some tips for contributing to Spark:

  • Try to write code "the Spark way" and fit your contributions into the existing way of doing things

  • Listen closely to feedback from the maintainers and if you want to push back, do so in a respectful manner

  • Thank the maintainers for their excellent work on their project. Many of them have been working on Spark for years. They deserve thanks and it's a great way to start the relationship on the right foot.

If you want to work on open source Spark development, check out itachi, spark-daria, spark-fast-tests, quinn, or chispa. Those are easier projects to get started on than the Spark codebase itself.


r/apachespark • • Jun 15 '20

Spark Summit is fully remote and free this year. Here's the registration link.

Thumbnail
databricks.com
49 Upvotes

r/apachespark • • Apr 21 '23

How custom PySpark DataFrame transformations got a lot better in the 3.3 release

Post image
47 Upvotes

r/apachespark • • Jul 25 '25

Parquet has been there for years but no one thought of deduplicating the data

46 Upvotes

Here is the idea: chunk and deduplicate your data and you will speed up uploads and downloads.

This is now possible for Parquet. Krisztian Szucs (Arrow PMC member) just announced that Parquet is more efficient thanks to a recent feature in Apache Arrow: Content Defined Chunking.

Instead of defining pages boundaries based on an arbitrary size, Content Defined chunking chunks the Parquet pages in a way that we can detect duplicate data. Hugging Face uses this to speed up data workflows on their platform (they use a dedupe-based storage called Xet).

Here is Krisztian's blog post: https://huggingface.co/blog/parquet-cdc

I'm pretty excited about this new paradigm and what it can bring to Spark, What do you think ?


r/apachespark • • Aug 28 '24

I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake

45 Upvotes

I made Zillacode Open Source. Here it is on GitHub. You can practice Spark and PySpark LeetCode like problems by spinning it up locally:

https://github.com/davidzajac1/zillacode 

I left all of the Terraform/config files for anyone interested on how it can be deployed in AWS.


r/apachespark • • Jul 30 '23

Spark SQL now supports default values

Post image
42 Upvotes

r/apachespark • • Apr 13 '23

How the Delta Lake MERGE statement allows for complex upsert logic with PySpark

Post image
41 Upvotes

r/apachespark • • Dec 08 '22

How to unit test PySpark code with the chispa library

Post image
41 Upvotes

r/apachespark • • Nov 24 '25

Anyone using Apache Gravitino for managing metadata across multiple Spark clusters?

40 Upvotes

Hey r/apachespark, wanted to get thoughts from folks running Spark at scale about catalog federation.

TL;DR: We run Spark across multiple environments with different catalogs (Hive, Iceberg, etc.) and metadata management is a mess. Started exploring Apache Gravitino for unified metadata access. Curious if anyone else is using it with Spark.

Our Problem

We have Spark jobs running in a few different places:

  • Main production cluster on EMR with Hive metastore

  • Newer lakehouse setup with Iceberg tables on Databricks

  • Some batch jobs still hitting legacy Hive tables

  • Data science team spun up their own Spark env with separate catalogs

The issue is our Spark jobs that need data from multiple sources turn into a nightmare of catalog configs and connection strings. Engineers waste time figuring out which catalog has what, and cross catalog queries are painful to set up every time.

Found Apache Gravitino

Started looking at options and found Apache Gravitino. Its an Apache Top Level Project (graduated May 2025) that does metadata federation. Basically acts as a unified catalog layer that can federate across Hive, Iceberg, JDBC sources, even Kafka schema registry.

GitHub: https://github.com/apache/gravitino (2.3k stars)

What caught my attention for Spark specifically:

  • Native Iceberg REST catalog support so your existing Spark Iceberg configs just work

  • Can federate across multiple Hive metastores which is exactly our problem

  • Handles both structured tables and what they call filesets for unstructured data

  • REST API so you can query catalog metadata programmatically

  • Vendor neutral, backed by companies like Uber, Apple, Pinterest

Quick Test I Ran

Set up a POC connecting our main Hive metastore and our Iceberg catalog. Took maybe 2 hours to get running. Then pointed a Spark job at Gravitino and could query tables from both catalogs without changing my Spark code beyond the catalog config.

The metadata discovery part was immediate. Could see all tables, schemas, and ownership info in one place instead of jumping between different UIs and configs.

My Questions for the Community

  1. Anyone here actually using Gravitino with Spark in production? Curious about real world experiences beyond my small POC.

  2. How does it handle Spark's catalog API? I know Spark 3.x has the unified catalog interface but wondering how well Gravitino integrates.

  3. Performance concerns with adding another layer? In my POC the metadata lookups were fast but production workloads are different.

  4. We use Delta Lake in some places. Documentation says it supports Delta but anyone actually tested this?

Why Not Just Consolidate

The obvious answer is "just move everything to one catalog" but anyone who's worked at a company with multiple teams knows that's a multi year project at best. Federation feels more pragmatic for our situation.

Also we're multi cloud (AWS + some GCP) so vendor specific solutions create their own problems.

What I Like So Far

  • Actually solves the federated metadata problem instead of requiring migration

  • Open source Apache project so no vendor lock in worries

  • Community seems active, good response times on GitHub issues

  • The metalake concept makes it easy to organize catalogs logically

Potential Concerns

  • Self hosted adds operational overhead

  • Still newer than established solutions like Unity Catalog or AWS Glue

  • Some advanced features like full lineage tracking are still maturing


Anyway wanted to share what I found and see if anyone has experience with this. The project seems solid but always good to hear from people running things in production.

Links:


r/apachespark • • Oct 06 '25

Detect and Redact Signatures in documents using ScaleDP powered by Apache Spark

Post image
40 Upvotes

I’ve been working on ScaleDP, an open-source library for document processing in Apache Spark, and it now supports automatic signature detection + redaction in PDFs.

🚀 Why it matters:

Handle massive PDF collections (millions of docs) in parallel Detect signatures with ML models and redact them automatically.

https://stabrise.com/scaledp/

Install via PyPI: pip install scaledp

💬 I’d love feedback from the community:

Do you see a use case for signature redaction at scale in your work? What other document processing challenges (tables, stamps, forms?) should an open-source Spark library tackle next?

Would be great to hear your thoughts.


r/apachespark • • Nov 01 '24

Introducing Spark Playground: Your Go-To Resource for Practicing PySpark!

39 Upvotes

Hey everyone!

I’m excited to share my latest project, Spark Playground, a website designed for anyone looking to practice and learn PySpark! 🎉

I created this site primarily for my own learning journey, and it features a playground where users can experiment with sample data and practice using the PySpark API. It removes the hassle of setting up local environment to practice.Whether you're preparing for data engineering interviews or just want to sharpen your skills, this platform is here to help!

🔍 Key Features:

Hands-On Practice: Solve practical PySpark problems to build your skills. Currently there are 3 practice problems, I plan to add more.

Sample Data Playground: Play around with pre-loaded datasets to get familiar with the PySpark API.

Future Enhancements: I plan to add tutorials and learning materials to further assist your learning journey.

I also want to give a huge shoutout to u/dmage5000 for open sourcing their site ZillaCode, which allowed me to further tweak the backend API for this project.

If you're interested in leveling up your PySpark skills, I invite you to check out Spark Playground here: https://www.sparkplayground.com/

The site currently requires login using Google Account. I plan to add login using email in the future.

Looking forward to your feedback and any suggestions for improvement! Happy coding! 🚀


r/apachespark • • Jan 11 '23

How to significantly speed up your Spark test suite runtime

Post image
39 Upvotes

r/apachespark • • Jul 13 '26

Best resources to learn Apache Spark in depth

38 Upvotes

Hi everyone!
I want to learn Apache Spark in depth, not just the DataFrame API.
I started with the official documentation, but I find it difficult to learn from because it isn't very interactive.
What resources would you recommend for someone who wants to really understand Spark?
I'm looking for recommendations on books, courses, YouTube channels, blogs, or hands-on projects. I'd especially like to understand Spark internals, architecture, optimization, and best practices.
If you were starting over today, how would you learn Spark?
Thanks!