r/apachespark • u/MrPowersAAHHH • Aug 05 '23
r/apachespark • u/MrPowersAAHHH • Feb 05 '21
PSA: The Spark maintainers are intentionally avoiding exposing certain Spark SQL functions via the Scala / PySpark APIs
They're adding some great new methods to the Spark SQL API but these methods won't necessarily be added to the Scala / PySpark APIs.
regexp_extract_all was actually removed from the Scala API before the 3.1 release and it's uncertain if regexp_extract_all will be added to PySpark.
I asked the mailing list about making the APIs consistent and the maintainers made arguments for having this functionality in SQL, but not the other APIs. They'd like determination on a case-by-case basis. I'd prefer to plug the functionality gaps in all the APIs, but the maintainers are smarter than me and I respect their decision.
I started a separate project called bebe to add the functions that won't be implemented in Spark codebase. I'll also be implementing some other developer friendly functions as Spark native functions, so there is a performant solution for the community and folks don't need to use UDFs.
Feel free to send me function requests (e.g. beginning_of_month, jaro_winkler, etc) and I'll try to get them implemented as performant native Spark functions.
r/apachespark • u/brayellison • Aug 27 '20
Not sure if this has been done here, but wanted to share.
r/apachespark • u/JY-DataMechanics • Apr 19 '21
We're publicly releasing our optimized Docker Images for Spark - they contain Spark, Scala, Java, Hadoop, Python, and connectors to S3, GCS, Azure Data Lake, Snowflake and Delta Lake.
r/apachespark • u/guardian_apex • Feb 21 '26
Spark Theory for Data Engineers
Hi everyone, I'm building Spark Playground and have added a Spark Theory section with 9 in-depth tutorials covering these concepts:
Disclaimer - content is created with the help of AI, reviewed, checked and edited by me.
Each tutorial breaks down Spark topics with practical examples, configuration snippets, comparison tables, and performance trade-offs. Written from a data engineering perspective.
Ongoing WIP: planning to add more topics like join strategies, partitioning strategies, caching & persistence, memory management etc.
If you'd like to help write tutorials, improve existing content, or suggest topics, the tutorials are open-source:
GitHub: https://github.com/rizal-rovins/learn-pyspark
Let me know what Spark topics would you find most valuable to see covered next
r/apachespark • u/MrPowersAAHHH • Feb 23 '21
My experience contributing to the Spark codebase
I recently got two pull requests merged into the Spark codebase:
I closed the add_hours() function in favor or make_interval.
Contributing to the codebase was a positive experience. There are a bunch of great developers that build an maintain the Spark codebase. Many of them have a good understanding of the entire codebase and collaborating with them teaches you a lot.
I even started working with one of the Spark contributors on a side project called itachi for functions that won't get merged into Spark, He's also helping me out with code reviews on the bebe project I'm working on. Spark folks are friendly & willing to help!
Some tips for contributing to Spark:
Try to write code "the Spark way" and fit your contributions into the existing way of doing things
Listen closely to feedback from the maintainers and if you want to push back, do so in a respectful manner
Thank the maintainers for their excellent work on their project. Many of them have been working on Spark for years. They deserve thanks and it's a great way to start the relationship on the right foot.
If you want to work on open source Spark development, check out itachi, spark-daria, spark-fast-tests, quinn, or chispa. Those are easier projects to get started on than the Spark codebase itself.
r/apachespark • u/MrPowersAAHHH • Jun 15 '20
Spark Summit is fully remote and free this year. Here's the registration link.
r/apachespark • u/MrPowersAAHHH • Apr 21 '23
How custom PySpark DataFrame transformations got a lot better in the 3.3 release
r/apachespark • u/qlhoest • Jul 25 '25
Parquet has been there for years but no one thought of deduplicating the data
Here is the idea: chunk and deduplicate your data and you will speed up uploads and downloads.
This is now possible for Parquet. Krisztian Szucs (Arrow PMC member) just announced that Parquet is more efficient thanks to a recent feature in Apache Arrow: Content Defined Chunking.
Instead of defining pages boundaries based on an arbitrary size, Content Defined chunking chunks the Parquet pages in a way that we can detect duplicate data. Hugging Face uses this to speed up data workflows on their platform (they use a dedupe-based storage called Xet).
Here is Krisztian's blog post: https://huggingface.co/blog/parquet-cdc
I'm pretty excited about this new paradigm and what it can bring to Spark, What do you think ?
r/apachespark • u/dmage5000 • Aug 28 '24
I made Zillacode.com Open Source - LeetCode for PySpark, Spark, Pandas and DBT/Snowflake
I made Zillacode Open Source. Here it is on GitHub. You can practice Spark and PySpark LeetCode like problems by spinning it up locally:
https://github.com/davidzajac1/zillacode
I left all of the Terraform/config files for anyone interested on how it can be deployed in AWS.

r/apachespark • u/MrPowersAAHHH • Apr 13 '23
How the Delta Lake MERGE statement allows for complex upsert logic with PySpark
r/apachespark • u/MrPowersAAHHH • Dec 08 '22
How to unit test PySpark code with the chispa library
r/apachespark • u/Q-U-A-N • Nov 24 '25
Anyone using Apache Gravitino for managing metadata across multiple Spark clusters?
Hey r/apachespark, wanted to get thoughts from folks running Spark at scale about catalog federation.
TL;DR: We run Spark across multiple environments with different catalogs (Hive, Iceberg, etc.) and metadata management is a mess. Started exploring Apache Gravitino for unified metadata access. Curious if anyone else is using it with Spark.
Our Problem
We have Spark jobs running in a few different places:
Main production cluster on EMR with Hive metastore
Newer lakehouse setup with Iceberg tables on Databricks
Some batch jobs still hitting legacy Hive tables
Data science team spun up their own Spark env with separate catalogs
The issue is our Spark jobs that need data from multiple sources turn into a nightmare of catalog configs and connection strings. Engineers waste time figuring out which catalog has what, and cross catalog queries are painful to set up every time.
Found Apache Gravitino
Started looking at options and found Apache Gravitino. Its an Apache Top Level Project (graduated May 2025) that does metadata federation. Basically acts as a unified catalog layer that can federate across Hive, Iceberg, JDBC sources, even Kafka schema registry.
GitHub: https://github.com/apache/gravitino (2.3k stars)
What caught my attention for Spark specifically:
Native Iceberg REST catalog support so your existing Spark Iceberg configs just work
Can federate across multiple Hive metastores which is exactly our problem
Handles both structured tables and what they call filesets for unstructured data
REST API so you can query catalog metadata programmatically
Vendor neutral, backed by companies like Uber, Apple, Pinterest
Quick Test I Ran
Set up a POC connecting our main Hive metastore and our Iceberg catalog. Took maybe 2 hours to get running. Then pointed a Spark job at Gravitino and could query tables from both catalogs without changing my Spark code beyond the catalog config.
The metadata discovery part was immediate. Could see all tables, schemas, and ownership info in one place instead of jumping between different UIs and configs.
My Questions for the Community
Anyone here actually using Gravitino with Spark in production? Curious about real world experiences beyond my small POC.
How does it handle Spark's catalog API? I know Spark 3.x has the unified catalog interface but wondering how well Gravitino integrates.
Performance concerns with adding another layer? In my POC the metadata lookups were fast but production workloads are different.
We use Delta Lake in some places. Documentation says it supports Delta but anyone actually tested this?
Why Not Just Consolidate
The obvious answer is "just move everything to one catalog" but anyone who's worked at a company with multiple teams knows that's a multi year project at best. Federation feels more pragmatic for our situation.
Also we're multi cloud (AWS + some GCP) so vendor specific solutions create their own problems.
What I Like So Far
Actually solves the federated metadata problem instead of requiring migration
Open source Apache project so no vendor lock in worries
Community seems active, good response times on GitHub issues
The metalake concept makes it easy to organize catalogs logically
Potential Concerns
Self hosted adds operational overhead
Still newer than established solutions like Unity Catalog or AWS Glue
Some advanced features like full lineage tracking are still maturing
Anyway wanted to share what I found and see if anyone has experience with this. The project seems solid but always good to hear from people running things in production.
Links:
Datastrato (commercial support if needed): https://datastrato.com
r/apachespark • u/Mykola_Melnyk_ML • Oct 06 '25
Detect and Redact Signatures in documents using ScaleDP powered by Apache Spark
I’ve been working on ScaleDP, an open-source library for document processing in Apache Spark, and it now supports automatic signature detection + redaction in PDFs.
🚀 Why it matters:
Handle massive PDF collections (millions of docs) in parallel Detect signatures with ML models and redact them automatically.
Install via PyPI: pip install scaledp
💬 I’d love feedback from the community:
Do you see a use case for signature redaction at scale in your work? What other document processing challenges (tables, stamps, forms?) should an open-source Spark library tackle next?
Would be great to hear your thoughts.
r/apachespark • u/guardian_apex • Nov 01 '24
Introducing Spark Playground: Your Go-To Resource for Practicing PySpark!
Hey everyone!
I’m excited to share my latest project, Spark Playground, a website designed for anyone looking to practice and learn PySpark! 🎉
I created this site primarily for my own learning journey, and it features a playground where users can experiment with sample data and practice using the PySpark API. It removes the hassle of setting up local environment to practice.Whether you're preparing for data engineering interviews or just want to sharpen your skills, this platform is here to help!
🔍 Key Features:
Hands-On Practice: Solve practical PySpark problems to build your skills. Currently there are 3 practice problems, I plan to add more.
Sample Data Playground: Play around with pre-loaded datasets to get familiar with the PySpark API.
Future Enhancements: I plan to add tutorials and learning materials to further assist your learning journey.
I also want to give a huge shoutout to u/dmage5000 for open sourcing their site ZillaCode, which allowed me to further tweak the backend API for this project.
If you're interested in leveling up your PySpark skills, I invite you to check out Spark Playground here: https://www.sparkplayground.com/
The site currently requires login using Google Account. I plan to add login using email in the future.
Looking forward to your feedback and any suggestions for improvement! Happy coding! 🚀
r/apachespark • u/MrPowersAAHHH • Jan 11 '23
How to significantly speed up your Spark test suite runtime
r/apachespark • u/roksolana_shendiukh • Jul 13 '26
Best resources to learn Apache Spark in depth
Hi everyone!
I want to learn Apache Spark in depth, not just the DataFrame API.
I started with the official documentation, but I find it difficult to learn from because it isn't very interactive.
What resources would you recommend for someone who wants to really understand Spark?
I'm looking for recommendations on books, courses, YouTube channels, blogs, or hands-on projects. I'd especially like to understand Spark internals, architecture, optimization, and best practices.
If you were starting over today, how would you learn Spark?
Thanks!