r/DataBuildTool • • Jul 17 '24

Join the DataBuildTool (dbt) Slack Community

Thumbnail
getdbt.com
3 Upvotes

r/DataBuildTool • • 18h ago

Show and tell dbt + DuckDB beginner project (one commit per lesson): from raw Parquet to sources, models, Jinja, docs, tests, CI with GitHub Actions and a public warehouse, built on a year of NYC taxi data

Thumbnail
youtu.be
3 Upvotes

I made a beginner dbt + DuckDB project, recorded step by step on video, and wanted to share it here. Disclosure: I work on Rosetta DBT Studio, the app I use in the videos.

Everything in it is open source and free: DuckDB, dbt Core, dbt-duckdb, Rosetta DBT Studio, and the course code itself. The data is public too. No cloud account, no warehouse bill, no paid tier needed. Everything also works with plain dbt from the command line if you prefer your own editor.

Video : https://youtu.be/6nt5JRKIS0A

Code (one commit per lesson): https://github.com/Nuri1977/nyc-taxi-dbt

The data is the NYC TLC green taxi trips for 2025: 12 monthly Parquet files (~14 MB, 591,375 trips) plus the 265-row taxi zone lookup as a seed. It runs on a laptop in seconds.

What the course covers:

- Creating a dbt project on DuckDB and loading the raw Parquet with read_parquet(union_by_name)

- Sources and seeds

- First models, then linking them with ref() and Jinja

- Properties: describing models and columns in YAML

- Docs and the DAG (lineage graph)

- Materializations: keeping a thin view and making the marts tables, with a project default plus a per-model config override

- Data tests: unique, not_null, accepted_values, relationships, and dbt build running everything in DAG order

- CI with GitHub Actions: a fresh build on every push, with the prod DuckDB file published as a GitHub release

- Docs on GitHub Pages, a README with a build badge, and a warehouse anyone can query by attaching the .duckdb file straight from the release URL

Feedback welcome, especially on anything that could be done more idiomatically in dbt.


r/DataBuildTool • • 8d ago

Show and tell MUST WATCH - 45-min dbt Power User walkthrough (Jaffle Shop | Core, Cloud & Fusion)

11 Upvotes

The walkthrough turns VS Code or Cursor into a full dbt IDE. It covers the features data engineers and analysts use every day.

  1. Run a model and see results, compiled SQL and run history in one panel.

  2. Profile Query finds aggregation before a join, SELECT * and hard-coded values. It changes nothing until you approve.

  3. Explain and optimize any model in one click.

  4. Execute a single CTE, or profile every CTE for runtime and row count.

Video: https://youtu.be/1yWejM5YW1o

Disclosure first: I work at Altimate, which builds dbt Power User - AI Agents for data teams.


r/DataBuildTool • • 9d ago

dbt news and updates [Virtual Event] What's shipping in dbt + Fivetran

Thumbnail
getdbt.com
3 Upvotes

r/DataBuildTool • • 13d ago

Question When an agent writes a dbt model for you, what do you check before it runs?

5 Upvotes

I find the SQL is usually fine, at first glance. But what about the stuff around it, like the grain, backfills, incremental logic, etc.

Curious what other people look at first - or do you just leave it to the agent written tests to decide? 🫣🧐


r/DataBuildTool • • 16d ago

Show and tell Loom-dbt: an author-time CLI to define models once, generate them at scale, and search for what already exists before building something new.

1 Upvotes

I've been a big fan of DBT since picking it up a few years ago, but there's always been a few nuances at Enterprise scale that I felt detracted from the wizardry it unlocks. Simple things like discovery (does this already exist), duplication for similarly-shaped models, remembering to include appropriate tests (e.g. primary key uniqueness), MSP-angosticity, and rebuilding standard warehouse paradigms are things that I always found myself thinking "surely someone has already solved this". When I looked, I found some great point solutions but none that I felt had a big-picture lens.

Loom-dbt (PyPI, GitHub) is my approach to solving these bugbears. It's a standalone CLI utility that:

  • Enables teams to declare model shapes (patterns) once using approachable metadata and generate models (instances) repeatably, supplying only what differs between them

  • Embeds features (data tests, keys, load-timestamping) for standard warehouse shapes (paradigms), with multi-paradigm support

  • Registers complex and hand-built models outside the generatable patterns

  • Provides detailed searching capabilities for project-wide discovery

  • Proves what's generated matches what's been declared (or identifies discrepancies)

  • and more!

Critically, loom-dbt outputs plain DBT with no long-lived dependency. The models you generate today will survive if you delete loom-dbt tomorrow; adoption is reversible by design.

And of course, no modern project would be complete without appropriate AI-augmentation guardrails: 6 Skills covering core functionality, maintenance, authoring, adoption, bootstrapping, and searching - standardised, npx installable, and ready for your favourite harness.

If you give it a try or are keen to learn more, let me know!


r/DataBuildTool • • 18d ago

Show and tell We pointed an AI agent at 146M NYC taxi rows and asked it to build a Data Vault mart. Here's what broke.

Thumbnail reddit.com
1 Upvotes

r/DataBuildTool • • 19d ago

Show and tell Trusting your data agent

4 Upvotes

Now that we are all rushing to use agentic analytics, something that personally concerns me is whether agents are accurate.

For this reason, I built an open source tool called the honest agent. You can hook it to your mcp and run your tests (evals) and check whether your agent is, well, telling you the truth. You can check the project here: https://thehonestagent.dev/ and here's the repo https://github.com/jlcoto/the_honest_agent. If you already work with dbt, the setup and workflow will seem very familiar 😉

I'm also writing a series of posts on evals in and why they should be a part of your data strategy if you are using data agents: https://thehonestagent.dev/blog/

Your feedback is really appreciated. And if you want to discuss agentic analytics, evals, etc, let's connect!


r/DataBuildTool • • 19d ago

Show and tell Reprocesamiento de BigQuery a gran escala: los modos de fallo ocultos de las canalizaciones incrementales de dbt

Thumbnail
medium.com
1 Upvotes

r/DataBuildTool • • 20d ago

Show and tell Medallion Layer Enforcement

Thumbnail
gallery
1 Upvotes

As dbt projects grow and more people contribute, keeping standards consistent gets harder. Someone could make a silver model (cleansed data) read from another silver model. Someone could make a gold model (business-ready data) read straight from bronze (raw data).

 

That’s why I think enforcing medallion layer dependencies matters: silver should read from bronze, and gold should read from silver.

 

This isn’t just tidiness. If a gold table skips silver and reads raw data directly, a report on billable surgical procedures could be built on data that was never cleaned. Duplicates, bad values and missing fields flow straight into the numbers people make decisions on.

 

There’s an engineering cost too. When models read from each other within a layer, two models can end up depending on each other. dbt can’t build either one due to cycle errors, and fixing it means time lost re-engineering models.

 

dbt checks (new in dbt 2.0) can enforce this. You apply it by writing a SQL query in a checks folder, plus a small YAML file for settings, that looks at every model and what it depends on and returns anything that breaks the rule. Per the dbt docs, checks are enforced “before any warehouse work runs.” If the project violates a rule, the build stops before a single model is compiled or built.

 

I tested it by pointing a gold model straight at a raw table, and a silver model at another silver model. The build failed and showed exactly which connections broke the rule (see the last image).

Rules people must remember get broken. Rules the build enforces don’t.

#dbt #AnalyticsEngineering #DataEngineering


r/DataBuildTool • • 21d ago

Show and tell Are you using Claude Code or Codex for data engineering? Would you let it near prod?

6 Upvotes

Quick question for data engineers: are you using Claude Code or Codex in your daily work?

I'm asking because we've been going back and forth on this while building Rosetta AI, the agent inside dbt Studio.

Claude Code and Codex give the agent a real shell and keep it in check with an OS sandbox. It's on by default in Codex and opt-in in Claude Code. For coding, that works really well. The agent can run any CLI, write a quick script, or install whatever it needs.

Data work feels different to me, though. A sandbox knows which files and hosts a command can reach, not what the command actually does. Once your warehouse is reachable, a SELECT and a DROP TABLE look exactly the same to it. And by default, Claude Code's sandbox still lets commands read most of your machine, credential files included, unless you lock it down.

So we went the other way. Rosetta AI has no shell at all. It can run dbt, but only a short list of commands, and only after you click approve. Any SQL that changes data asks first too. Credentials stay in the OS keychain through keytar, so the model only ever sees env_var() references, never the actual passwords.

That has a cost. Every capability has to be built as a tool, so the agent can't improvise the way it can with a shell. We haven't fully settled which trade-off is right.

So I'd love to hear from you. Are you using Claude Code or Codex for data engineering? And would you let it anywhere near a production warehouse?

dbt Studio is open source if you want to look under the hood: github.com/rosettadb/dbt-studio


r/DataBuildTool • • Oct 20 '26

Show and tell DBT OPTIMIZATION

0 Upvotes

r/DataBuildTool • • Oct 16 '26

Show and tell 5 short dbt Power User demos: profiling, column lineage, docs and tests, CTE debugging, macro hover

16 Upvotes

Hey all, disclosure first: I work at Altimate, which builds dbt Power User. We recorded one short video per workflow, 4 to 8 minutes each.

  1. Profile Query: review a model's SQL for bugs and cost in one click. https://youtu.be/uMptd4t_mGI

  2. Column-level lineage, with AI chat about the model. https://youtu.be/A-z_R0QJig0

  3. Docs, data tests and unit tests in one editor. https://youtu.be/NrEYqEUcpHY

  4. Explain, optimize and profile each CTE from inside the model. https://youtu.be/MDN2PSgVAmA

  5. Hover over a ref() or macro to see what it does. https://youtu.be/kbRZ-hrENyc

Which of these would you like us to go deeper on?


r/DataBuildTool • • Oct 15 '26

Show and tell I built a tool that generates seeds with intact foreign keys from a schema, plus a dbt project whose tests pass on them

4 Upvotes

Disclosure: I built this.

The situation I kept hitting: a new client, a model to design, and no source access for weeks. Hand-writing seeds that survive relationships tests is the boring part, and faker scripts give you orders for customers that don’t exist.

The video is the browser version, model2data studio. You write or paste a schema (DBML, SQL DDL or an existing schema.yml), it generates parents before children so every FK resolves, and scaffolds the project: seeds, sources, staging models, and not_null / unique / relationships / accepted_values tests that pass on the generated data. Modelling needs no account; generating needs a free one.

https://studio.jbanalytica.com

The engine underneath is open source (MIT), if you’d rather stay in the terminal:

pip install model2data
model2data --file schema.dbml --rows 200 --seed 42
cd dbt_schema && dbt deps && dbt build

https://github.com/JB-Analytica/model2data

What I’d most like to know: do the generated tests match what you’d write by hand, and what’s missing?


r/DataBuildTool • • Oct 15 '26

Show and tell Optimization in DBT

0 Upvotes

r/DataBuildTool • • Oct 14 '26

Show and tell Webinar | Data, trust, and autonomy: An executive conversation on enterprise AI

Thumbnail
go.fivetran.com
1 Upvotes

r/DataBuildTool • • Oct 10 '26

Question Anyone can share a real dbt project for learning??

3 Upvotes

I'm not talking about a small project with 10-20 models.


r/DataBuildTool • • Oct 07 '26

Question Snapshot redesign

1 Upvotes

I’m fairly new to DBT. Please help me and be kind enough to answer my question.

I need to redesign the existing flow, creating a snapshot model(which is not done right).

We have existing flow like this

Raw1–> snapshot1–>dim1–✒️
dim
Raw2–>snapshot2–>dim2-🖊️

From the above flow we are missing direct history tracking and goal is to create common snapshot for final dimension by eliminating dim1 and dim 2

Also, currently airflow run dependency had set like after snapshot dag run completion dim dag will start.

And project folder is as follows
-model
|_dags
|_bronze
|_silver
|_gold
|_consumer
-snapshots

If anybody has any idea about how to proceed in this case please help. TIA


r/DataBuildTool • • Oct 07 '26

Question How to implement idempotent incremental loads after an initial full snapshot without breaking on full refresh?

0 Upvotes

Hello guys,

I have a question regarding some data I am working with. I have a source system where the initial dataset arrives as a one-time full load, followed by periodic incremental (delta) loads.

I wanted to achieve a design the ingestion/merge strategy so running a full refresh (or reprocessing the initial snapshot) doesn't duplicate records, violate constraints, or break the pipeline.

What is the recommended design pattern and checklist for this scenario in dbt?

Thank you!


r/DataBuildTool • • Sep 29 '26

Show and tell We built a free, open-source desktop app for dbt Core, with SQL, notebooks, dashboards, DuckLake/Iceberg and AI agents in one place

21 Upvotes

One dbt project. Too many tools. So we built one app for all of it 👇

Rosetta DBT Studio is a free, open-source desktop studio for dbt Core™: connections, modeling, runs, SQL, dashboards, data lakes and AI agents in one place.

⭐ If you like it, star us on GitHub. It really helps an open-source project grow:

https://github.com/rosettadb/dbt-studio

#dbt #DuckDB #DuckLake #ApacheIceberg #Iceberg #DataEngineering #AnalyticsEngineering #OpenSource


r/DataBuildTool • • Sep 18 '26

Question Is it possible to use dbt VS Code extension for my dbt-core personal project?

3 Upvotes

r/DataBuildTool • • Sep 16 '26

Question dbt Core + Databricks Job

5 Upvotes

Hi everyone, I just inherited a tricky pipeline at my job, and I need some guidance on orchestrating dbt models using Databricks Jobs (Workflows).

The situation is that the job we have is not reliable; it's a complete mess to reprocess after it breaks, and we don't have idempotency guaranteed. Given all this, I need help dealing with retries: how to repair the job without re-running models that already finished successfully, and how to retry tasks whose error was caused by an internal error, like a Databricks server error (it happened already during the weekend)


r/DataBuildTool • • Sep 16 '26

Show and tell Local dbt vs Rosetta DBT Studio - same project, two ways in

Thumbnail
reddit.com
3 Upvotes

r/DataBuildTool • • Sep 16 '26

Question s there a way to use dbt State without dbt Cloud?

1 Upvotes

our client wants to use dbt-state, but we want to check if there is any official or community way to use dbt-state without app state or dbt cloud?

i played with dbt-state + app state and it's actually a really amazing feature,

so the question is around what the community is doing. is there an option to use dbt-state without a dbt account, either official or community?