r/apachespark • • Aug 25 '26

Apache Spark ships with 28 known CVEs in production. In 2026. And nobody thinks this is a problem worth talking about publicly.

8 Upvotes

This is not acceptable any longer in a modern developer world and community.

I run security scans on our Spark deployment and came back with 28 vulnerabilities — high severity, public CVEs, all sitting in transitive dependencies like Netty and Apache Thrift. Nothing exotic. Netty published fixes for 22 of them in a single batch in June. Thrift fixed their issues in 0.23.0. The fixes exist. Spark just didn’t include them.

What bothers me more than the CVEs themselves is the process — or the lack of one from Apaches side.

In 2026, any serious DevSecOps pipeline is expected to have mandatory quality gates that block releases on high/critical CVEs in dependencies. This isn’t cutting-edge practice — it’s baseline hygiene. The full toolchain is free and mature

There is no automated dependency CVE gate in Spark’s release pipeline. No Trivy. No Dependabot. No OWASP Dependency-Check. Nothing that would block a release because a bundled library has a known high-severity vulnerability. These are free tools. Adding one to a CI pipeline is an afternoon of work. It hasn’t been done.

ProblemHonest Assessment
No automated dependency scanningInexcusable in 2026. Free tools exist. One CI step.
Decoupled release calendarsReal coordination challenge, but solvable with Dependabot PRs that can be reviewed and merged quickly
Volunteer PMCDoesn’t excuse Databricks, Google, Apple, and Amazon — all Spark committers with paid engineers — from contributing a security gate
“Good enough” cultureActively harmful when Spark is used in AI/ML pipelines processing personal data under GDPR

The July 2026 maintenance releases — 4.2.0, 4.1.3, 4.0.4, 3.5.9 — all shipped after Netty 4.1.135.Final was available. They didn’t include the bump. There was no public statement that the team was aware of the issue and working on it. It just shipped, vulnerable, into production systems everywhere.

The “volunteer PMC” argument doesn’t land anymore. Databricks is worth somewhere around $62 billion. Google, Apple, Amazon, and Microsoft all have paid Spark committers. The resources to fix this governance gap before lunch exist. The will apparently doesn’t.

The EU Cyber Resilience Act (CRA) — which came into force in 2024 and has mandatory compliance deadlines rolling in through 2027 — specifically targets software supply chain security, including transitive dependencies and SBOM (Software Bill of Materials) requirements. The ASF has acknowledged this directly, stating they need to prepare projects for “CRA and U.S. CISA guidance”.

Spark ships into commercial products. This will become a compliance problem for a lot of companies very soon, and the fix is genuinely trivial from an engineering standpoint.

The ASF did make progress in 2025 — launching “Apache Trusted Releases (ATR)” for distribution security and ratifying CycloneDX 1.7 for SBOM standards — but none of this yet translates to a blocking CVE gate on the release pipeline for projects like Spark.

In the meantime: if you’re running Spark and using JFrog Xray or Trivy on your deployment, you can force-override the affected Netty and Thrift versions in your own build. It’s not clean but it works until the next maintenance release, expected sometime in Q4.

Is anyone else tracking this or pushing upstream to get a CVE gate added to the build?


r/apachespark • • Jul 09 '26

Pyspark Interview : Normalize Comma-Separated Values

1 Upvotes

data = {"1":"945,545","3":"2345,3456,45678"}

from pyspark.sql import SparkSession

from pyspark.sql.functions import col, split, explode

spark = SparkSession.builder.appName("ExplodeExample").getOrCreate()

df = spark.createDataFrame(

[(k, v) for k, v in data.items()],

["id", "value"]

)

print("Original DataFrame:")

df.show()

# Split the value column based on comma so it will create a array of values and use explode it will create multiple rows

result_df = df.select(

col("id"),

explode(split(col("value"), ",")).alias("value")

)

print("Result DataFrame:")

result_df.show()


r/apachespark • • Dec 29 '21

DataFrames vs SparkSQL - Which One Should You Choose?

Thumbnail
confessionsofadataguy.com
1 Upvotes

r/apachespark • • Nov 29 '24

Looking for consultancy work for apache spark's performance issues

0 Upvotes

Hi,

I can guarantee extra ordinary performance in spark's querying capabilities if the queries being executed are complex and if compilation itself is running into > 3 min. I can also help improve runtime performances by pushing down non partititioning equi join preds to data source levels.

I can send interested people my resume. I have +26 years of development experience and 9 years of apache spark internals. Restricted on LinkedIn due to political position.

The solutions are 100% logically correct and with the margin of human errors, will not result in any new bug other than what may be present in spark master branch. Some of the solutions are in production since past 4 years in my previous company.

Apart from the PRs mentioned, I have some more work which I have not made open through PRs.

None of the solution will involve patch work like disabling the existing rules.

I do not expect any of the opened PRs to get into upstream spark, because I do feel spark committers work like a cartel , controlled by few companies. Have heard things like, not having credibility with spark committers.

The attitude of committers it seems, is that spark is their fiefdom, forgetting that it is an open source product and any person can get deep into its workings. The attitude is that any solution to complex problems can only come from the committers, and if it at all comes from a non committer , then either it is to be ignored, or since it is not easily comprehensible to them , it must be wrong. And as I see it , unfortunately, the committers have not been able to solve those problems. since 2015.


r/apachespark • • Sep 17 '17

Top mistakes to avoid when writing Spark Application

Thumbnail
ravindranaik.com
0 Upvotes

r/apachespark • • Jun 02 '24

Question?

0 Upvotes
  • What is an RDD, and why is Spark's RDD preferred over other data structures?

  • What are the benefits of using an RDD?

  • How did the concept of RDDs come about?

  • Are there any resources regarding RDDs?

  • how rdd use physical and logical plan??


r/apachespark • • Dec 05 '22

Is there any tool to convert spark SQL into dataframe?

0 Upvotes

r/apachespark • • Aug 04 '26

I benchmarked Spark vs DuckDB on the same Parquet files

0 Upvotes

Since you seemed to enjoy my last Spark post, I ran another controlled benchmark.

This time I wanted to understand when Spark’s distributed execution overhead actually becomes worth paying for.

For each workload, both engines ran the same SQL against the same Snappy Parquet files, on the same 20 logical-core machine, with Spark using local[20] and DuckDB using threads=20.

I discarded one warm-up, alternated the engines across 7 timed runs, reported the median, and verified the results row by row. I also ran the entire benchmark twice.

For an aggregation with 8 output groups:

RowsSparkDuckDB
1,000154 ms3 ms
1M183 ms16 ms
100M925 ms167 ms

For a high-cardinality aggregation with roughly 500,000 groups:

RowsSparkDuckDB
1M2.29 s67 ms
10M3.31 s431 ms
100M8.83 s2.99 s

DuckDB was faster at every measured point.

The interesting part was how much the gap narrowed as the workload grew.

On the high-cardinality query, increasing the input from 1M to 100M rows increased Spark’s runtime by about 3.9 times and DuckDB’s by about 44.8 times.

That does not prove Spark eventually becomes faster. The curves had not crossed, and this benchmark does not establish where, or whether, they ever would.

Query shape also mattered enormously.

On the same 100M rows, the eight-group endpoint aggregation generated around 17 kB of Spark shuffle, while the high-cardinality user aggregation generated around 131 MB.

So I do not think rules such as “use DuckDB below X GB” are very useful. Query shape, data location, concurrency, hardware, fault tolerance and operational requirements can matter as much as the size of the dataset.

This was one machine running one query at a time, so it measures the cost of distribution much better than the capabilities Spark is actually selected for, such as distributed data, recovery and concurrent workloads.

I made a visual breakdown of the benchmark and what both engines are doing under the hood:

https://youtu.be/bXyXvxeHSrw?si=vMsEQkv3QqzLcTf9

What usually makes Spark the right choice for your production workloads?


r/apachespark • • May 29 '21

How to read/write CSV file/data in Apache Spark

Thumbnail
youtube.com
0 Upvotes

r/apachespark • • Jan 14 '21

Array Sorting in Spark 3.0

0 Upvotes

r/apachespark • • Sep 05 '24

Spark to Polars (contd.)

0 Upvotes

How would I change this code into Polars

monthly_credits_window_latest=Window.partitionBy([COLS_CONSTS.TENANT_IDENTIFIER_COL, COLS_CONSTS.MASKED_ACCOUNT_NUMBER_COL])
.orderBy(COLS_CONSTS.TRANSACTION_MONTH_COL).rowsBetween( Window.currentRow - 2, Window.currentRow)


r/apachespark • • Apr 18 '21

How to Install Spark Databricks for Free - In 3 Min! | https://youtu.be/SYi8yLEOgfk

Thumbnail
youtu.be
1 Upvotes

r/apachespark • • Jul 03 '24

AWS to Azure migration

1 Upvotes

Hai,

In few days we are starting a migration project from AWS to azure

Below are list of AWS services 1.Etcd 2.Neo4j 3.Elastis search 4. Elastic cache 5. postgresql 6. S3

Any idea how to do this job at very efficient way.I wanted to provide more details but customer not yet shared


r/apachespark • • Sep 23 '26

Spark 4.2.0??

3 Upvotes

r/apachespark • • Oct 11 '18

30x speedup: A comparison of OmniSci GPU database vs. Apache Spark

Thumbnail
omnisci.com
2 Upvotes

r/apachespark • • Apr 07 '22

What is the future of Apache Spark?

Thumbnail
futureentech.com
2 Upvotes

r/apachespark • • Oct 19 '23

Spark is a Top 5 language for Data Science / ML jobs

3 Upvotes

Source: https://jobs-in-data.com/blog/machine-learning-vs-data-scientist

About the dataset: 9,261 jobs crawled from 1605 companies worldwide in June-Sep 2023


r/apachespark • • Jul 11 '19

Spark Streaming question

0 Upvotes

Hi everyone, new to spark streaming.

So my requirement is that to read a stream of files to be read from a directory. I am able to do that and also able to do minor spark sql operations.

I have two problems here. In my first stage, with the data that is coming (example) :

NAME, SALARY

I need to create various KPI's from the two fields. Eg :

NAME, SALARY_GOOD, SALARY_BAD, SALARY_AVERAGE

and use this spark streaming dataframe to further calculate values in a different stage.

(these are example data, the real data is more complicated)

But basically this is the requirement. Is this possible?


r/apachespark • • Jun 13 '17

Spark and Scala Online Training | Spark Certification Course - Mentor Global

Thumbnail
mentor-global.com
0 Upvotes

r/apachespark • • Dec 03 '15

One year on: Apache Spark continues a winning battle against data management misery

Thumbnail
comparethecloud.net
0 Upvotes

r/apachespark • • May 28 '23

How spark session is created

0 Upvotes

Hi,this could be a very basic question,What exactly is spark session and do we need spark to be installed locally to run the spark scala code in local mode

And what spark submit command will do and how is it different from running the code from IDE


r/apachespark • • May 13 '22

Apache CXF Library redirect http to https

0 Upvotes

We are having some issues where we noticed that CXF is having issues with redirect from HTTP to HTTPS.

Caused by: javax.wsdl.WSDLException: WSDLException (at /wsdl:definitions/wsdl:import/wsdl:definitions/wsdl:types/xs:schema/xs:schema): faultCode=PARSER_ERROR: Problem parsing 'http://schemas.xmlsoap.org/ws/2004/09/policy/ws-policy.xsd'.: org.xml.sax.SAXParseException: Premature end of file. ( if you try to browse to http://schemas.xmlsoap.org/ws/2004/09/policy/ws-policy.xsd it will redirect to https://schemas.xmlsoap.org/ws/2004/09/policy/ws-policy.xsd

{http://gpki-adapters.zenprise.com/symantec}GpkiAdapterService#{http://gpki-adapters.zenprise.com/symantec}signCertificate has thrown exception, unwinding now: com.zenprise.zdm.gpki.GpkiAdapterFault: Could not get a cert from the server, error: signCertificate() got javax.xml.ws.WebServiceException error: org.apache.cxf.service.factory.ServiceConstructionException: Failed to create service.'

soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><soap:Fault><faultcode>soap:Server</faultcode><faultstring>Could not get a cert from the server, error: 'signCertificate() got javax.xml.ws.WebServiceException error: 'org.apache.cxf.service.factory.ServiceConstructionException: Failed to create service.''</faultstring><detail><ns1:GpkiAdapterFault xmlns:ns1="http://zdm.zenprise.com/gpki/sample"></ns1:GpkiAdapterFault></detail></soap:Fault></soap:Body></soap:Envelope>

Is there a way to get CXF to be able to redirect from HTTP to HTTPS?


r/apachespark • • May 09 '22

What proves that Apache Spark is better than Hadoop?

Thumbnail dataarchitect.cloud
0 Upvotes