r/databricks • • 6h ago

Discussion What’s something you believed about Databricks for way too long before finding out you were wrong?

18 Upvotes

I’ll go first with two.

For an embarrassingly long time I ran scheduled jobs on an all-purpose cluster because it was already up and I figured compute is compute. Nobody told me job clusters are billed at a lower rate. Found out when someone asked why our bill looked the way it did.

Second one: I assumed Auto Loader schema evolution just quietly picked up new columns. It does, but only after the stream fails first and you restart it. Learned that from a 2am alert, not from the docs.

Neither is complicated. I just never had a reason to question them until something broke.

What’s yours? Could be Spark, Delta, Unity Catalog, billing, whatever. Bonus points if you found out in production.


r/databricks • • 6h ago

News 50% OFF any Databricks Certification

9 Upvotes

Join us for a four-week event dedicated to learning, upskilling, and advancing your career in data engineering, analytics, machine learning, and generative AI. Whether you are new to the field or aiming to deepen your knowledge, this is a great opportunity to invest in your professional development alongside a global community.

🎟️ Finish all modules in at least one learning pathway during the event window to receive:

50% OFF any Databricks Certification

Learn more/ https://community.databricks.com/t5/learning-events/databricks-advanced-learning-festival-november-10-december-08/ev-p/172328


r/databricks • • 7h ago

Discussion Do you cache DataFrames in Spark as often as you can, or only when necessary?

9 Upvotes

I've seen people use cache() and persist() quite frequently in Databricks notebooks, especially when working with large DataFrames.

But I've also read that caching everything can sometimes consume memory unnecessarily and even slow things down.

Curious about how people handle this in real projects. Do you cache intermediate DataFrames when they're reused multiple times, or do you prefer letting Spark optimize the execution plan?

Have you ever had caching make a job slower instead of faster?


r/databricks • • 13h ago

News Subsidized tokens came to Databricks

Post image
18 Upvotes

Now, in Unity Gateway, you can connect a ChatGPT or Claude subscription and use cheaper tokens from there.

https://medium.com/databrickscommunity/databricks-news-genie-cli-app-builder-agent-memory-dabs-41820427fcc2


r/databricks • • 2h ago

Discussion Genie vs DBT Wizard

2 Upvotes

Any thoughts coming from people that use both ?


r/databricks • • 7h ago

Help Databricks Streaming table to Fabric for Gold Lakehouse

2 Upvotes

Has anyone built an external reference from Lakeflow databricks table to Fabric? The Lakeflow databricks table is sourced from On-prem SQL server and uses change tracking to identify rows that are changed. Databricks writes this out to ADLS Gen 2, which is then shortcut (external referenced) to Fabric. We need the data in near real time.

The streaming delta tables in ADLS Gen 2 created by Databricks seem to have additional rows that keep track of deleted rows etc, is there a way to get only the final data after deletes and updates, and secondly, is there a way to get table names without mirroring the databricks unity catalogue? The unity catalogues don't support streaming tables which is what change tracking SQL tables are.

Appreciate any help on this or alternative approaches to getting streaming data from Databricks to Fabric.


r/databricks • • 1d ago

Help Lakeflow Declarative Pipelines with external: managed tables, or publish step?

11 Upvotes

I'm working on an Azure Databricks project where we must expose Delta tables through stable ADLS paths so external engines like Snowflake, Trino, and delta-rs apps can read directly from storage. We'd like to use Lakeflow, but managed streaming tables and materialized views live in Unity Catalog managed storage.

Can we safely expose their physical paths like regular external Delta tables, or is path-based access unsupported? The alternative seems to be external Delta tables as sinks, but sinks look limited: Python/streaming only, no expectations or Auto CDC directly, and full refresh doesn't clear prior data.

For this requirement, what architecture would you recommend: external sinks, a publish step, or something else?


r/databricks • • 1d ago

Tutorial Governed Spreadsheets on Databricks: Row Zero Acquisition & What It Means for Business Users on Databricks (Demo + Chat)

Thumbnail
youtu.be
18 Upvotes

Hey everyone!

Had the wonderful opportunity of recording a demo + chatting with Breck, the founder & former CEO of Row Zero, a governed spreadsheets platform Databricks just acquired. For business users, this will pair up nicely alongside the Genie One / Genie Agents experience.

Some quick answers to common questions the video answers (used AI + transcript for them):

When did Databricks acquire Row Zero?
=The acquisition was announced in September 24, 2026.

Can Databricks customers use Row Zero today?
=Yes, with a Row Zero account. A built-in Databricks experience doesn't have a date yet.

How will it be priced?
=Not final. Expected to follow the rest of Databricks (pay for what you use), not per seat.

Does it respect Unity Catalog permissions?
=Yes. Row Zero queries Databricks as the signed-in user (OAuth recommended, service principals supported), so row-level security carries through.

Can you write back to Databricks?
=Yes, to tables with a primary key. You review a diff before changes apply.

Can you export back to Excel?
=Yes, to Excel or CSV. Exports over about 1 million rows get truncated because of Excel's row limit, and admins can turn export off entirely.

All roads in data begin and end with a spreadsheet after all!

PS: Always love to hear your feedback on the video.


r/databricks • • 22h ago

Help Spark Declarative Pipelines - possible to write UC Managed Iceberg Tables instead of materialized views?

5 Upvotes

Hi All,

Based on my experience using SDP, the tables are typically rendered as "Materialized Views". My company is making a decision to move towards Managed Iceberg for tables and I was wondering if creating Managed Iceberg tables is supported by Databricks's native SDP capabilities?


r/databricks • • 1d ago

News Databricks new Amsterdam office

Thumbnail
medium.com
20 Upvotes

Last week, I was invited to the opening of Databricks’ new Amsterdam office as a representative of the MVP community.

I took quite a few photos, and I thought some of you might be interested in seeing what it’s like inside.

The new office spans 11 floors in "The Rock" building, which offers stunning views of Amsterdam. Inside, there’s a barista, a pool table, foosball, a piano, and a dedicated game room with a PS5. I suspect the latter will become especially popular once GTA VI is released.


r/databricks • • 1d ago

News Event-Driven Triggers

Post image
15 Upvotes

One of the most important parts of FinOps is to run jobs only when needed. With event-driven triggers like job completion or table triggers, we can run a job as soon as data is available.

https://www.sunnydata.ai/blog/lakeflow-jobs-triggers-events-conditions

https://medium.com/databrickscommunity/lakeflow-jobs-triggers-06af6feae80c


r/databricks • • 1d ago

Help Deploy models pattern via MLflow deployment jobs + multiple workspaces

2 Upvotes

Hi,

I'm quite new to MLOps and need some hints for setting up what Databricks calls "Deploy model" pattern (https://docs.databricks.com/aws/en/machine-learning/mlops/deployment-patterns).

Our Data Science team currently trains ML models in dev environment, in our dev Databricks account/workspace. They have direct access to production data, and the trainingㅤ is quite expensive. We'd like to train the model once and then promote it to production, which lives in a separate Databricks account/workspace.

My idea to handle the deployment was to use MLflow deployment jobs (https://docs.databricks.com/aws/en/mlflow/deployment-job) and just allow the Data Science team to register the model in the production Unity Catalog. After the registration, the deployment job would kick in, do whatever else is needed, and then stop at an approval step. However, it turned out they can register the model version in the Unity Catalog from the dev workspace, but the deployment job doesn't run automatically. That makes the whole process suboptimal, as they now have to ping me to run the deployment job manually each time they register a new model version.

I did some testing and the sole reason of that behavior seems to be the fact that the model comes from a different workspace, which doesn't "see" the attached deployment job that lives in the production workspace. Am I missing something here, or is it designed to work like that? What are the alternatives?


r/databricks • • 1d ago

Discussion Does throwing more resources at a slow Spark job actually solve the problem?

9 Upvotes

I've noticed that whenever a Spark job starts running slowly, the first thought is often to increase the cluster size or add more workers.

But sometimes the actual problem is something else, like data skew, unnecessary shuffles, poor partitioning, or inefficient joins.

I'm curious how experienced Databricks users approach this. Do you usually start with the Spark UI and execution plan, or do you try increasing resources first?

Have you ever had a job that became significantly faster just by changing the code instead of upgrading the cluster?


r/databricks • • 2d ago

Discussion Are you using Databricks Materialized Views for your Gold layers yet? Thoughts?

26 Upvotes

Hey everyone,

I’ve been implementing Databricks Materialized Views for executive reporting (Gold layer aggregates like daily revenue summaries, customer 360 metrics, etc.) and wanted to see how others are using them in production.

Historically, keeping summary tables up to date meant writing custom MERGE INTO scripts, managing orchestration dependencies, or dealing with sluggish standard views that re-scanned massive Delta tables on every query.

Having the platform automatically handle incremental updates under the hood via serverless compute has been a massive time-saver.

Curious to hear your experiences:

  • Are you fully embracing Materialized Views for Gold layer reporting, or sticking to traditional Delta tables with custom upsert logic?

  • Have you run into any gotchas regarding compute costs or refresh frequencies with larger datasets?

  • How do they hold up when dealing with complex multi-table joins and apportioned aggregations?


r/databricks • • 2d ago

Help How to make skarf file completed?

Post image
7 Upvotes

I'm completing few learning modules on academy but there's this 'SORM' kind file which shows 'in progress' only although I've completed the quiz. Any tips??


r/databricks • • 2d ago

Discussion How do you manage table schema changes in Databricks across dev/test/prod?

11 Upvotes

We run Databricks with Unity Catalog, and I'm trying to understand how other teams handle the lifecycle of managed tables (not pipelines or jobs, just the tables themselves).

What I keep seeing:

  • CREATE TABLE IF NOT EXISTS at the top of Job

  • A DDL script in the repo that someone runs by hand

  • Terraform for some tables and a different approach for the rest

A few genuine questions:

  1. Where does the table definition live in your setup: in the pipeline code, in a separate repo, or only in the catalog?

  2. How do you review a schema change before it reaches prod?

  3. How do you roll the same change through dev, test, and prod?


r/databricks • • 2d ago

Discussion Genie ignored my Unity Catalog wiki, so I tested what it actually reads

Thumbnail
gallery
9 Upvotes

Last week I tried the new Unity Catalog wikis (Beta). I wrote a business rule into a table's wiki ("revenue = total_price of orders with status F only") and asked a Genie agent for revenue. It summed every order. When I asked who owns the table, it suggested checking a "wiki page outside this environment".

I asked about it on the Databricks Community, and someone from Databricks replied: Genie agents don't use wikis as context today (no timeline yet). They read table and column comments. For a rule like this, use a metric view or a SQL expression, and keep text instructions for clarifications.

So I tested that. Setup, in my own Azure workspace with the TPC-H sample data:

- `samples.tpch.orders` copied into a table, all years

- I asked about 1995 because it has F, O and P orders. The right answer (F only) is 32.69B. All orders is 171.63B

- 5 Genie agents, each with ONE kind of context

- 3 benchmark questions per agent ("What was our revenue in 1995?", "How much revenue did we make in 1995?", "What was our revenue in Q2 1995?"), run once in Agent mode and once in Chat mode

Results:

ContextAgent modeChat mode
Nothing0/30/3
Table + column comments3/33/3
One text instruction3/33/3
SQL expression (measure) + one example query3/33/3
Metric view (filter inside the measure)3/33/3

Things I noticed:

- With no context, Agent mode wrote "This includes orders of every status." It states the assumption, but it doesn't ask.

- The comments show up as the table and column descriptions under Configure > Sources. Two ALTER statements and it was fixed.

- The measure editor wants table-qualified columns. `SUM(CASE WHEN status = 'F' THEN total_price END)` fails with "Table name or alias is required for column 'status'". `orders.status` works.

- In Chat mode, Genie reused my example query word for word, `:year` parameter included. In Agent mode it didn't use the measure and wrote `WHERE status = 'F'` itself (still right).

- The metric view agent used `MEASURE(revenue)` every time. The measure:

```yaml
- name: revenue
  expr: SUM(total_price) FILTER (WHERE status = 'F')
```

Bonus: I also published a Unity Catalog Page called "Revenue". Genie One found it through the ontology ("Read page: Revenue"), answered 32.69B, said the definition "isn't certified yet" and listed the orders it left out. Best answer of the lot.

Caveats: one run per mode, a simple rule, one table. Genie One also looked at an older table and agent in my workspace, but its reasoning says the Page decided the answer. And for a rule this simple even plain text worked, so this doesn't settle "structured context beats text" either way.

Questions for you:

- Has anyone seen text instructions break where a metric view or SQL expression held up? What kind of rule was it?

- Where do you keep business rules for Genie today?

Links:

- The Databricks answer to my question: https://community.databricks.com/t5/data-governance/do-genie-agents-use-unity-catalog-wikis-as-context/td-p/175921

- Genie benchmarks: https://docs.databricks.com/aws/en/genie/benchmarks

- Unity Catalog Pages: https://docs.databricks.com/aws/en/uc-semantics/pages


r/databricks • • 1d ago

General Databricks AI Dev round — anyone taken this recently?

0 Upvotes

Has anyone here done the Databricks AI dev on CoderPad recently?

Trying to figure out how to prep for it since there isn't much info out there yet. Are you debugging a medium-sized repo, building a feature on top of existing code, or refactoring? Also curious what interviewers seem to grade heaviest on.

Any pointers on how to prepare would be super helpful. Thanks!


r/databricks • • 2d ago

Discussion Using the Apache DataFusion Comet accelerator to save cost

6 Upvotes

I wonder if anybody has utilized the comet accelerator to speed up things and therefore save cost? Given the photon tax this seems like obvious choice but I'm not sure if the project is already mature enough.


r/databricks • • 2d ago

Discussion How do you handle dirty data and data quality failures in your pipelines? (Quarantine vs. Fail Fast)

7 Upvotes

Hey everyone,

I wanted to start a discussion on how different teams handle data quality failures in production pipelines—specifically dealing with bad records (e.g., null foreign keys, negative prices, schema mismatches).

The two extremes are:

  1. Fail Fast: The pipeline crashes the moment a single bad row appears (very safe, but brittle and blocks all good data).

  2. Silent Drop: The pipeline quietly ignores bad rows (keeps data flowing, but causes massive data discrepancies later when numbers don't match source systems).

Lately, I’ve been leaning heavily into the Quarantine Pattern (tagging bad rows, splitting the stream into Valid vs. Quarantine tables, and building monitoring alerts on top of the quarantine logs without blocking valid data).

Curious how you all approach this:

  • Do you use quarantine/dead-letter tables in your architecture, or do you prefer failing hard on errors?

  • How do you handle operational alerts when quarantine tables spike?

  • What tools or frameworks are you using to enforce and monitor data quality rules?


r/databricks • • 2d ago

Discussion How are you handling enterprise CDC and file ingestion in Databricks lately? (Lakeflow Connect vs. Custom Spark)

7 Upvotes

Hey everyone,

I’ve been leaning heavily into Databricks Lakeflow Connect and Auto Loader (cloudFiles) recently for managing multi-source ingestion (pulling relational CDC streams like SQL Server alongside high-volume S3 file drops), and I wanted to see how the community is approaching this.

Historically, setting up reliable ingestion meant writing custom PySpark streaming jobs, managing checkpoints manually, and dealing with schema evolution headaches when upstream source systems changed unexpectedly.

Using Lakeflow Connect combined with Unity Catalog Volumes has noticeably streamlined how raw data hits the Bronze layer, especially for incremental file processing and maintaining clean lineage.

I’m curious:

  • Are you guys relying heavily on native Lakeflow/Auto Loader connectors for ingestion, or still building custom Spark/Delta streaming pipelines for specific edge cases?

  • How are you handling schema drift when upstream source tables alter their structure?

  • Any major pain points you've run into with CDC replication in Databricks so far?


r/databricks • • 2d ago

News Job SQL condition

Post image
27 Upvotes

SQL condition can check whether a trigger should fire. This way, we can avoid unnecessary job runs.

https://medium.com/databrickscommunity/lakeflow-jobs-triggers-06af6feae80c

https://www.sunnydata.ai/blog/lakeflow-jobs-triggers-events-conditions


r/databricks • • 2d ago

General Inside Apache Iceberg V3: Deletion Vectors and Puffin Files

Post image
15 Upvotes

r/databricks • • 2d ago

Help What do you use for Databricks cost optimization beyond system tables?

1 Upvotes

We’re spending roughly $900 a day in DBUs and a bit over $2k a day on the AWS side for the same workspaces. For a long time, I only had a dashboard for the first number, it’s the usual one on system.billing.usage joined to list_prices and it looked fine until finance put it next to the AWS invoice.

Tags get me part of the way because cluster tags land on the EC2 instances and cost explorer can group by them. Most of our scheduled jobs moved onto instance pools last year to cut startup time though, and on pool clusters our custom tags only show up on the DBU side. Idle pool instances don’t appear in system tables at all because they don’t burn DBUs while they sit there but AWS still bills every hour of them.

On top of that, about 40 scheduled jobs run on one shared all-purpose cluster and that’s a single line on the EC2 side that I have no fair way to split back to teams.

Which tool ended up tying the Databricks and AWS sides together for you, ofcourse past the built-in system tables?