r/databricks • • Jul 07 '26

Discussion Extremely unreliable AI features

0 Upvotes

Really disappointed at the AI features databricks has.. the genie works horribly sometimes and cant even make basic changes to a file without the web page crashing. Don't know what happened. Was working unreliable earlier as well.. bit now it's just insane. Can't seem to do any work


r/databricks • • Sep 21 '26

General What does being a Databricks MVP look like?

0 Upvotes

It's not a client project. It is a privilege. It's also a journey. Below, I have put down my first year's journey and see what it looks like.

https://medium.com/@jasonyip_77999/what-does-being-a-databricks-mvp-look-like-707b9dcdcb28


r/databricks • • Feb 07 '26

General Databricks Data Engineer Exam

0 Upvotes

Why risk it? Practice with our free tests first, build your confidence, identify weak areas, and save your money. Only take the real exam when you're truly ready. https://testlogichub.web.app/


r/databricks • • Jul 02 '26

General Data Engineering Is Moving From Pipelines to Intelligent Decisions

1 Upvotes

I recently created a short YouTube video sharing my thoughts on where data engineering is heading in the AI era.

The main idea is simple: data engineering is no longer only about building pipelines, tables, validations, and dashboards. That foundation is still important, but the next chapter feels bigger.

I think we are moving toward intelligent decision systems where data platforms do more than show numbers. They help explain what changed, why it changed, where the issue happened, who is impacted, and what action should be taken next.

In real projects, the hard part is often not just moving data. It is understanding the context behind the data. A count may drop, a field may go missing, or a join may filter thousands of records. The business question sounds simple, but the investigation can go deep.

That is where I believe AI can help, not as a replacement for data engineers, but as a teammate that helps with investigation, metadata, quality checks, root-cause analysis, and clearer decision-making.

Here is the video: https://youtu.be/q6Xz7RcFp4w

Curious to hear from this community: do you think AI will mainly help data engineers write code faster, or will it change how business users interact with data entirely?


r/databricks • • 26d ago

Discussion LTAP without lakebase?

1 Upvotes

Is there any world where an LTAP architecture can be created that does NOT include lakebase, or databricks for that matter? It seems to me that it isn't much more than a marketing term for selling lakebase.

Every public doc about LTAP has references to lakebase sprinkled all over it. Lakebase is mentioned from the beginning to the very end. As I understand LTAP is essentially defined as a shared storage layer for OLTP and analytics workloads. And that sort of storage is an implementation detail that is ONLY found in the deepest and most proprietary parts of lakebase.

(IMO, Microsoft has had their own flavor of LTAP for many years, in the form of Fabric DW. That engine looks and acts similar to a TSQL database, yet emits lakehouse files in storage. Yet I suppose it lacks some very particular features, or performs badly on certain disqualifying OLTP workloads. The disqualifications will always allow a databricks salesman to refuse to call it LTAP. If these disqualifications aren't purely subjective, then maybe we also need to define LTAP in terms of a certain TPS threshold and a certain set of minimal DBMS capabilities. I haven't seen anything like that.)


r/databricks • • Aug 03 '26

Tutorial 9 Apache Spark Alternatives You Should Know in 2026

Thumbnail
itnext.io
0 Upvotes

r/databricks • • Oct 02 '25

Help Error while reading a json file in databricks

Post image
0 Upvotes

I am trying to read this json file which I have uploaded in the workspace.default location. But I am getting this error. How to fix this. I have simply uploaded the json file after going to the workspace and then create table and then added the file..

Help!!!


r/databricks • • Mar 18 '25

Help Looking for someone who can mentor me on databricks and Pyspark

2 Upvotes

Hello engineers,

I am a data engineer, who has no experience in coding and currently my team migrating from legacy to unity catalog which needs lots of Pyspark code. I need to start but question is where to start from and also what are the key concepts ?


r/databricks • • Mar 17 '24

Discussion Does anyone find it funny that databricks doesn't warn the user about what's happening on AWS

0 Upvotes

Maybe it's just me because I'm a noob, but I find it a little strange that databricks doesn't warn the user it's creating whole bunch of EC2 instances, like t3.2xlarge or something (and there's no option to choose smaller instances).

And then when you delete the workspace and tables, it doesn't delete the associated EC2 instances. If someone were to assume databricks would automatically clean up after itself and terminate all those instances, they would have a huge AWS bill at the end of the month. Is it just me or is this weird?


r/databricks • • 11d ago

News AI_DECIDE benchmark

Post image
1 Upvotes

I benchmarked AI_DECIDE against 20 LLMs. It was definitely the fastest. I was using a moderation test to approve or reject posts.

https://medium.com/databrickscommunity/i-tested-databricks-ai-decide-against-20-llms-d37694fbf792

https://www.sunnydata.ai/blog/databricks-ai-decide-benchmark


r/databricks • • Feb 26 '25

Discussion is it worth databricks

0 Upvotes

hi
I am learning data bricks (Azure and AWS). I noticed that creating delta live tables using a pipeline is annoying. The issue is getting the proper resources to run the pipeline.

I have been using ADF, and I never had an issue.

What do you think the Databricks pipeline is worth


r/databricks • • Aug 07 '26

Tutorial Databricks Lakemeter Explained

Thumbnail
youtu.be
0 Upvotes

r/databricks • • Sep 09 '26

Discussion Unity Catalog Open Source in Name Only (UCOSINO)

Post image
0 Upvotes

Consider a callstack where something bad is happening in Spark or Unity Catalog (image above).

Any software engineer will google for the message, and then for the Exception class, and then for the call frames shown on the stack (starting at the top or bottom). For any commonly encountered Exceptions from UC (something like com.databricks.sql.managedcatalog.acl.UnauthorizedAccessException), we will find dozens of results from a search engine. Others on the internet have already shared their experiences, and the search results are normally actionable. The users tell us what they had done to avoid or fix the error.

But software engineers have heard for two years that "unity catalog is open source". So a software engineer will proceed to look for the source repo where they might find the full definition of "UnauthorizedAccessException", along with all the related references. No such thing exists. (Admittedly there is a public-facing github, called "unitycatalog", but it is virtually worthless and there is no overlap with the real-world UC in databricks, as we experience it.)

It only takes one or two repeats of this, before a software engineer will realize that none of this stuff is actually open source. UC doesn't compare to a REAL open source software like Apach Spark. If we search for spark references in the call stack (eg. "org.apache.spark.sql.DataFrameReader"), then we are immediately taken to the source repo at github!

I do give Databricks a lot of credit for open-sourcing spark. But nowadays they take too much liberty with the word "open source", to the point where it lost all of its meaning. UC is not opensource in any substantial way. Maybe there is an API spec that is open, but that is the extent of it. Another example is lakebase which the CEO claimed to be open source at the recent summit. There has never been any software as proprietary as neon/lakebase.

It doesn't actually bother me if a CEO forgets how to use the term "open souce" correctly in English. What makes me more upset is when I expect to be able to use google to find the source code for "UnauthorizedAccessException", and come up with absolutely bupkis. Can anyone tell me a definition of "open source" which would potentially include either Unity Catalog or Lakebase? I'm assuming that when these words are used by the CEO, he does NOT intend to imply that the actual source is open to the public.


r/databricks • • May 13 '26

Discussion Since joining Databricks, I've stopped running init commands for new web apps

2 Upvotes

Ever since joining Databricks, I've stopped running init commands manually when starting new web app projects.

If you're a web developer, starting a new app usually starts with running an "init" command in your CLI. For example, if you were using Next.js, you would run:

npx create-next-app@latest my-app --yes
cd my-app
npm run dev

Building apps on Databricks with AppKit is similar:

databricks apps init --name my-app
cd my-app
npm run dev

As a developer, that workflow is pretty much muscle memory:

  • run the init command

  • cd into the project

  • start the dev server

But lately I've been doing something different since it is recommended by the AppKit docs to just prompt your AI assistant instead.

So now, what I do is:

mkdir my-app
cd my-app
cursor . (or sub with your favorite coding agent)

Once Cursor is open, I type:

"Create a new Databricks app in this folder and start the dev server"

The agent scaffolds the app using the same init command behind the scenes and starts everything for me.

Honestly, doing it this way feels super weird...after years of manually running setup commands, this takes a lot to get used to 😅

But the more I do this, the more I see the benefits.

Firstly, I no longer need to remember what the init command is, or look up what CLI flags it accepts.

Secondly, this pattern is framework-agnostic. I no longer need to remember framework-specific setup commands or look up docs.

The same prompt pattern works across frameworks:

  • "Create a new Next.js app..."

  • "Create a new Nuxt app..."

  • "Create a new SvelteKit app..."

  • "Create a new TanStack Start app..."

  • "Create a new Databricks app..."

You get the picture... Same intent. Different tooling.

Thirdly, and most importantly, I can infinitely expand the prompt.

I can easily add any additional instructions to the end of my prompt for the agent to follow.

For example, I can say:

"Create a new Databricks app in this folder and start the dev server and commit everything to git and deploy my app"

This will not only create my app, but also create a git repo and deploy the app without me needing to remember specific commands or look up any docs!

I'm curious, have you also started shifting your workflow this way, or are you still mostly scaffolding projects manually?

---

One important note, the reason the agent can reliably handle prompts like:

"Create a new Databricks app in this folder and start the dev server"

...is because of agent skills. Specifically, if you're building Databricks apps, you should check out and install databricks-agent-skills.

Without the proper skills/context, the agent may not know the correct setup flow or commands to use.


r/databricks • • Apr 17 '26

General Terrible notebooks

0 Upvotes

Im convinced databricks notebooks is among the worst notebooks softwares out there. Maybe learn a little from aws.. and disable the annoying ai edit features.. they are doing more harm then good.


r/databricks • • May 28 '25

Discussion Why Does Databricks Certification Portal Only Accept Credit Cards & USD Pricing for Indian Candidates?

0 Upvotes

Hi all,

I'm from India and I'm registering for a Databricks certification for the first time. I was surprised to see that the payment portal only accepts credit cards in USD, with no options for debit cards, UPI, or net banking—which are widely used and standard on other exam platforms.

While I understand USD pricing from a global consistency perspective (and I truly appreciate how platforms like Azure localize pricing to INR), it's the lack of basic payment flexibility that’s surprising.

Is there a specific reason Databricks has not enabled alternative modes of payment for markets like India, where credit card penetration is relatively low?

Would love to hear from Databricks team members or anyone who’s navigated this differently. Thanks!

#databricks, #certification, #IndiaTech


r/databricks • • Jul 09 '26 Popcorn

General Databricks AI Agent Genie Code is no longer Free. Now you have to pay as you go

Thumbnail
gallery
0 Upvotes

On July 8, 2026, Databricks introduced one of the biggest changes to its AI experience.

Databricks AI Agent Genie products (Genie Code, Genie Spaces, and Genie One) are no longer free, as I wrote earlier. Instead, they now use a pay-as-you-go pricing model based on AI token consumption, with a free monthly allowance for each user before paid usage begins.

I wanted to understand how this works in practice. Not by reading the documentation. By using it.

Also, be careful! Currently, only the Alert feature works*, but the* documentation states that there should also be a block after reaching the limit*!*
I think you need to notify users so they can use AI more carefully and wisely.

Full text - Medium


r/databricks • • Sep 29 '26

Discussion Can a Delta table be “correct” even when the source and target row counts match?

0 Upvotes

Suppose the source and target both contain 100M rows.

The row counts match, but the target could still have:

  • Missing records

  • Duplicate records

  • Incorrectly transformed values

  • Changed or truncated values

For a large Delta table, what checks would you use beyond row counts to prove that the source and target data actually match?

Would you use hashes, key-level reconciliation, aggregates, sampling, or something else?


r/databricks • • Feb 12 '26

General Scaling Databricks Pipelines with Templates & ADF Orchestration

0 Upvotes

In a Databricks project integrating multiple legacy systems, one recurring challenge was maintaining development consistency as pipelines and team size grew.

Pipeline divergence tends to emerge quickly:

• Different ingestion approaches
• Inconsistent transformation patterns
• Orchestration logic spread across workflows
• Increasing operational complexity

Standardization Approach

We introduced templates at two critical layers:

1️⃣ Databricks Pipeline Templates

Focused on processing consistency:

✅ Standard Bronze → Silver → Gold structure
✅ Parameterized ingestion logic
✅ Reusable validation patterns
✅ Consistent naming conventions

Example:

def transform_layer(source_table, target_table):
    df = spark.table(source_table)
    
    (df.write
       .mode("overwrite")
       .saveAsTable(target_table))

Simple by design. Predictable by architecture.

2️⃣ Azure Data Factory (ADF) Templates

Focused on orchestration consistency:

✅ Reusable pipeline skeletons
✅ Standard activity sequencing
✅ Parameterized notebook execution
✅ Centralized retry/error handling

Example pattern:

Databricks Notebook Activity → Parameter Injection → Logging → Conditional Flow

Instead of rebuilding orchestration logic, new pipelines inherited stable behavior.

Observed Impact

• Faster onboarding of new developers
• Reduced pipeline design fragmentation
• More predictable execution flows
• Easier monitoring & troubleshooting
• Lower long-term maintenance overhead

Most importantly:

Developers focused on data logic, not pipeline plumbing.


r/databricks • • Jun 14 '25

Tutorial Top 5 Pyspark job optimization techniques used by senior data engineers.

0 Upvotes

Optimizing PySpark jobs is a crucial responsibility for senior data engineers, especially in large-scale distributed environments like Databricks or AWS EMR. Poorly optimized jobs can lead to slow performance, high resource usage, and even job failures. Below are 5 of the most used PySpark job optimization techniques, explained in a way that's easy for junior data engineers to understand, along with illustrative diagrams where applicable.

✅ 1. Partitioning and Repartitioning.

❓ What is it?

Partitioning determines how data is distributed across Spark worker/executor nodes. If data isn't partitioned efficiently, it leads to data shuffling and uneven workloads which can incur cost and time.

💡 When to use?

  • When you have wide transformations like groupBy(), join(), or distinct().

  • When the default partitioning (like 200 partitions) doesn’t match the data size.

🔧 Techniques:

  • Use repartition() to increase partitions (for parallelism).

  • Use coalesce() to reduce partitions (for output writing).

  • Use custom partitioning keys for joins or aggregations.

📊 Visual:

Before Partitioning:
+--------------+
| Huge DataSet |
+--------------+
      |
      v
 All data in few partitions
      |
  Causes data skew

After Repartitioning:
+--------------+
| Huge DataSet |
+--------------+
      |
      v
Partitioned by column (e.g. 'state')
  |
  +--> Node 1: data for 'CA'
  +--> Node 2: data for 'NY'
  +--> Node 3: data for 'TX' 

✅ 2. Broadcast Join

❓ What is it?

Broadcast join is a way to optimize joins when one of the datasets is small enough to fit into memory. This is one of the most commonly used way to optimize the query.

💡 Why use it?

Regular joins involve shuffling large amounts of data across nodes. Broadcasting avoids this by sending a small dataset to all workers.

🔧 Techniques:

  • Use broadcast() from pyspark.sql.functions.from pyspark.sql.functions import broadcast df_large.join(broadcast(df_small), "id")

📊 Visual:

Normal Join:
[DF1 big] --> shuffle --> JOIN --> Result
[DF2 big] --> shuffle -->

Broadcast Join:
[DF1 big] --> join with --> [DF2 small sent to all workers]
            (no shuffle) 

✅ 3. Caching and Persistence

❓ What is it?

When a DataFrame is reused multiple times, Spark recalculates it by default. Caching stores it in memory (or disk) to avoid recomputation.

💡 Use when:

  • A transformed dataset is reused in multiple stages.

  • Expensive computations (like joins or aggregations) are repeated.

🔧 Techniques:

  • Use .cache() to store in memory.

  • Use .persist(storageLevel) for advanced control (like MEMORY_AND_DISK).df.cache() df.count() # Triggers the cache

📊 Visual:

Without Cache:
DF --> transform1 --> Output1
DF --> transform1 --> Output2 (recomputed!)

With Cache:
DF --> transform1 --> [Cached]
               |--> Output1
               |--> Output2 (fast!) 

✅ 4. Avoiding Wide Transformations

❓ What is it?

Transformations in Spark can be classified as narrow (no shuffle) and wide (shuffle involved).

💡 Why care?

Wide transformations like groupBy(), join(), distinct() are expensive and involve data movement across nodes.

🔧 Best Practices:

  • Replace groupBy().agg() with reduceByKey() in RDD if possible.

  • Use window functions instead of groupBy where applicable.

  • Pre-aggregate data before full join.

📊 Visual:

Wide Transformation (shuffle):
[Data Partition A] --> SHUFFLE --> Grouped Result
[Data Partition B] --> SHUFFLE --> Grouped Result

Narrow Transformation (no shuffle):
[Data Partition A] --> Map --> Result A
[Data Partition B] --> Map --> Result B 

✅ 5. Column Pruning and Predicate Pushdown

❓ What is it?

These are techniques where Spark tries to read only necessary columns and rows from the source (like Parquet or ORC).

💡 Why use it?

It reduces the amount of data read from disk, improving I/O performance.

🔧 Tips:

  • Use .select() to project only required columns.

  • Use .filter() before expensive joins or aggregations.

  • Ensure file format supports pushdown (Parquet, ORC > CSV, JSON).df.select("name", "salary").filter(df["salary"] > 100000)df.filter(df["salary"] > 100000) # if applied after joinEfficient Inefficient

📊 Visual:

Full Table:
+----+--------+---------+
| ID | Name   | Salary  |
+----+--------+---------+

Required:
-> SELECT Name, Salary WHERE Salary > 100K

=> Reads only relevant columns and rows 

Conclusion:

By mastering these five core optimization techniques, you’ll significantly improve PySpark job performance and become more confident working in distributed environments.


r/databricks • • 7d ago

Discussion Delta Tables: Is there any possibility on Delta tables Data file formats other than Parquet

0 Upvotes

Hi Team,

Since Parquet files are immutable and Delta tables framework writes data changes into new data files, Is there any option to specify any other file format and try to use same data file for minor data changes.(Below 10 Record transaction changes on Million/Billions of records).

Additionally I saw somewhere that Performance optimized on Delta tables by restricting Parquet files usage. I am wondering Is this really possible or not? because according to delta framework only writes data into Parquet files only. 

Regards,

Krishna


r/databricks • • Jun 16 '26

Help Can anyone help with this databricks?

0 Upvotes

I recently joined an MNC and have been assigned to the Databricks stream. I'm completely new to Databricks and related technologies, with almost zero knowledge in this area. Could anyone guide me on how to start preparing for it? What topics should I focus on first, and what learning path would you recommend for a beginner? Also, I would appreciate it if anyone could share their experience working with Databricks. Does it have good career scope and growth opportunities in the future? Thanks in advance for your suggestions and guidance!


r/databricks • • Dec 11 '24

Help Memory issues in databricks

2 Upvotes

I am so frustrated right now because of Databricks. My organization has moved to Databricks, and now I am stuck with this, and very close to letting them know I can't work with this. Unless I am misunderstanding something.

When I do analysis on my 16GB laptop, I can read a dataset of 1GB/12M rows into an R-session, and work with this data here without any issues. I use the data.table package. I have some pipelines that I am now trying to move to Databricks. It is a nightmare.

I have put the 12M rows dataset into a hive metastore table, and of course, if I want to work with this data I have to use spark. Because that I what we are forced to do:

  library(SparkR)
  sparkR.session(enableHiveSupport = TRUE)
  data <- tableToDF(path)
  data <- collect(data)
  data.table::setDT(data)

I have a 32GB one-node cluster, which should be plenty to work with my data, but of course the collect() function above crashes the whole session:

The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.

I don't want to work with spark, I want to use data.table, because all of our internal packages use data.table. So I need to convert the spark dataframe into a data.table. No.way.around.it.

It is so frustrating that everything works on my shitty laptop, but moving to Databricks everything is so hard to do with just a tiny bit of fluency.

Or, what am I not seeing?


r/databricks • • Oct 16 '26

Discussion Databricks vs. Snowflake: Are we comparing features instead of foundations?

Post image
3 Upvotes