r/analyticsengineering • • Apr 17 '25

How dirty is your data?

1 Upvotes

While I find these Buzzfeed-style quizzes somewhat… gimmicky, they do make it easy to reflect on how your team handles core parts of your analytics stack. How does your team stack up in these areas?

Semantic Layer Documentation:

Data Testing:

  • ✅ Automated tests run prior to merging anything into main. Failed tests block the commit.

  • 🟡 We do some manual testing.

  • 🚩 We rely on users to tell us when something is wrong.

Data Lineage:

  • ✅ We know where our data comes from.

  • 🟡 We can trace data back a few steps, but then it gets fuzzy.

  • 🚩 Data lineage? What's that?

Handling Data Errors:

  • ✅ We feel confident our errors are reasonably limited by our tests. When errors come up, we are able to correct them and implement new tests as we see fit.

  • 🟡 We fix errors as they come up, but don't track them.

  • 🚩 We hope the errors go away on their own.

Warehouse / RB Access Control:

  • ✅ Our roles are defined in code (Terraform, Pulumi, etc...) and are git controlled, allowing us to reconstruct who had access to what and when.

  • 🟡 We have basic access controls, but could be better.

  • 🚩 Everyone has access to everything.

Communication with Data Consumers:

  • ✅ We communicate changes, but sometimes users are surprised.

  • 🟡 We communicate major changes only.

  • 🚩 We let users figure it out themselves.

Scoring:

Each ✅ - 0 points, Each 🟡 - 1 point, Each 🚩 - 2 points.

0-4: Your data practices are in good shape.

5-7: Some areas could use improvement.

8+: You might want to prioritize a data quality initiative.


r/analyticsengineering • • May 11 '26

Bruin’s AI Data Agent

0 Upvotes

There’s a lot of database/warehouse specific AI tools as well as legacy BI tools that now have AI features. There’s a few problems with those tools that we’ve addressed by building a platform that seamlessly connects to any data source, imports context from your data pipelines and knowledge base, and integrates with your Slack, Teams, WhatsApp, etc. so that you can analyze data, build reports, and get alerts right inside your existing conversations.

Bruin Cloud has been around for a few years and trusted dozens of companies, but we're excited to announce that it is now generally available.

Feel free to give it a try - no payment method is required and the free credits (~$100 + 50 free questions per month) will get you started.

Note that the platform is SOC 2 Type 2 certified and GDPR compliant.

Disclaimer: I am one of the founders of Bruin

https://cloud.getbruin.com/register


r/analyticsengineering • • Mar 20 '26

Visitran — Open-source AI-powered data transformation tool (think Cursor, but for data pipelines)

0 Upvotes

Visitran: An open-source data transformation platform that lets you build ETL pipelines using natural language, a no-code visual interface, or Python.

How it works:
Describe a transformation in plain English → the AI plans it, generates a model, and materializes it to your warehouse
Everything compiles to clean, readable SQL — no black boxes
The AI only processes your schema (not your data), preserving privacy

What you can do:
Joins, aggregations, filters, window functions, pivots, unions — all via drag-and-drop or a chat prompt
The AI generates modular, reusable data models (not just one-off queries)
Fine-tune anything the AI generates manually — it doesn't force an all-or-nothing approach

Integrations:
BigQuery, Snowflake, Databricks, DuckDB, Trino, Starburst

Stack:
Python/Django backend, React frontend, Ibis for SQL generation, Docker for self-hosting. The AI supports Claude, GPT-4o, and Gemini.

Licensed under AGPL-3.0. You can self-host it or use their managed cloud.

GitHub:
https://github.com/Zipstack/visitran

Docs:
https://docs.visitran.com

Website:
https://www.visitran.com


r/analyticsengineering • • Jul 11 '25

Dev Setup - dbt Core 1.9.0 with Airflow 3.0 Orchestration

1 Upvotes

Hello Data Engineers 👋

I've been scouting on the internet for the best and easiest way to setup dbt Core 1.9.0 with Airflow 3.0 orchestration. I've followed through many tutorials, and most of them don't work out of the box, require fixes or version downgrades, and are broken with recent updates to Airflow and dbt.

I'm here on a mission to find and document the best and easiest way for Data Engineers to run their dbt Core jobs using Airflow, that will simply work out of the box.

Disclaimer: This tutorial is designed with a Postgres backend to work out of the box. But you can change the backend to any supported backend of your choice with little effort.

So let's get started.

Prerequisites

Video Tutorial

https://www.youtube.com/watch?v=bUfYuMjHQCc&ab_channel=DbtEngineer

Setup

  1. Clone the repo in prerequisites.

  2. Create a data folder in the root folder on your local.

  3. Rename .env-example to .env and create new values for all missing values. Instructions to create the fernet key at the end of this Readme.

  4. Rename airflow_settings-example.yaml to airflow_settings.yaml and use the values you created in .env to fill missing values in airflow_settings.yaml.

  5. Rename servers-example.json to servers.json and update the host and username values to the values you set above.

Running Airflow Locally

  1. Run docker compose up and wait for containers to spin up. This could take a while.

  2. Access pgAdmin web interface at localhost:16543. Create a public database under the postgres server.

  3. Access Airflow web interface at localhost:8080. Trigger the dag.

Running dbt Core Locally

Create a virtual env for installing dbt core

python3 -m venv dbt_venv
source dbt_venv/bin/activate

Optional, to create an alias

alias env_dbt='source dbt_venv/bin/activate'

Install dbt Core

python -m pip install dbt-core dbt-postgres

Verify Installation

dbt --version

Create a profile.yml file in your /Users/<yourusernamehere>/.dbt directory and add the following content.

default:
  target: dev
  outputs:
    dev:
      type: postgres
      host: localhost
      port: 5432
      user: your-postgres-username-here
      password: your-postgres-password-here
      dbname: public
      schema: public

You can now run dbt commands from the dbt directory inside the repo.

cd dbt/hello_world
dbt compile

Cleanup

Run Ctrl + C or Cmd + C to stop containers, and then docker compose down.

FAQs

Generating fernet key

python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

I hope this tutorial was useful. Let me know your thoughts and questions in the comments section.

Happy Coding!


r/analyticsengineering • • Jun 28 '24

Alteryx Snack newsletter

0 Upvotes

Hello all,

I wanted to introduce to the community a new newsletter, called the Alteryx Snack!
Twice a month a new article is posted. Join now to help grow the community, and also suggest new themes!

https://alteryx-snack.beehiiv.com/subscribe


r/analyticsengineering • • Dec 07 '23

I've definitely never received a snapchat from a girl, but I can auto-format my SQL queries to TitleCase!

0 Upvotes

r/analyticsengineering • • Apr 29 '26

open-source dashboard-as-code tool - the free & open answer to AI BI services

0 Upvotes

I’ve built an open source CLI tool to build dashboards, but the key point is that it is based on “dashboard as code” principles so that every dashboard’s properties, queries, and semantic layer lives inside yaml or tsx files, which makes it agent-friendly out of the box.

This is my answer to the whole AI dashboard and BI tools out there, but focusing more on the framework and semantic layer so that it works better with AI agents.

Today's the first day of releasing this publicly, so please share your honest feedback, skepticism, and even roast it - and if you want, give the repo a star:

https://github.com/bruin-data/dac


r/analyticsengineering • • Oct 27 '24

Need a mentor

0 Upvotes

Hi guys! Urgent need a mentor who can give me tasks from Data cleaning to visualization. I never studied data analytics formely, just studied from YouTube. Need help, I am counting on this reddit community.


r/analyticsengineering • • Jun 01 '26

Stop hardcoding business logic in your BI tool’s proprietary layer?

5 Upvotes

My small media production company  recently dealt with many-to-many data chaos" where Finance was in Power BI and Product was in Tableau. This then resulted in different numbers for the same ARR metric. The fix was pulling the modeling out of the BI tools entirely.
By using a universal semantic layer like what Cube Core has, we defined joins, dimensions, and measures once in code (Git-versioned). And because it exposes a Postgres-compliant SQL API, both Tableau and Power BI connect to the same governed model. This way, RLS and metric definitions are uniform across every surface. It’s essentially Gen 3 architecture (warehouse-native) now evolving into Gen 4 (AI-ready) because that same model can then ground an LLM without it hallucinating your schema too much


r/analyticsengineering • • May 05 '26

end-to-end NBA data app using Claude Code

0 Upvotes

r/analyticsengineering • • Sep 10 '26

Data Engineer w/ 2+ YOE seeking resume feedback

Post image
1 Upvotes

r/analyticsengineering • • Aug 28 '25

What are you vibe coding on dbt with?

0 Upvotes

r/analyticsengineering • • Jun 06 '24

Web3 for Analytics Engineers

1 Upvotes

I'm thrilled to announce the launch of my first official newsletter: "Web3 for Analytics Engineers"! 🚀

As someone passionate about both data and blockchain technology, I created this newsletter to help bridge the gap between these two exciting fields. Each issue will dive into innovative techniques, tools, and insights to help you master blockchain data analytics. Subscribe now and stay ahead of the game! https://web3foranalyticsengineers.substack.com/p/decentralize-your-data-journey-introducing


r/analyticsengineering • • Sep 17 '26

Why does an analytics project spend most of its budget before the first dashboard exists?

1 Upvotes

I wrote about the part of analytics projects nobody puts in the kickoff deck.
A client asked us to consolidate three years of data. The Excel format changed every year. We wrote a pipeline per year just to read the files. Then the validation rules started rejecting rows, because a spreadsheet will accept a typo in a supplier ID and a pipeline will not.
Months of that, before the first dashboard existed. That was not a delay. That was the actual work.
In the post: how Medallion Architecture on Databricks let us keep changing business rules without breaking production reports, and the three questions clients always ask about messy data.
Full post https://www.rubicon-world.com/blog/analytics-project-budget-before-first-dashboard


r/analyticsengineering • • Sep 16 '26

Instead of another BI Tool Selection cycle, we put a Semantic Layer in front of our warehouse

5 Upvotes

Lately, I’ve been focusing on consolidating our semantic layer. Here is the exact scenario that made my manager push for this:

We realized "net sales" was defined in 6 different ways across departments. Finance subtracted returns and markdowns; Merchandising subtracted returns but ignored markdowns; my weekly report did something in between because I inherited a query from a guy who left two years ago. Every executive meeting had that awkward moment where two dashboards showed different numbers and nobody could explain why beyond "different SQL."
During our BI tool selection review, we realized the core problem wasn't our dashboarding UI - it was the lack of a centralized data modeling layer
So instead of switching visualization platforms, we put Cube in front of our warehouse as a headless semantic layer You define your "cubes" mapped to fact tables, list your dimensions (store, region, category, week), and write the math for your measures once in YAML.
Now net_sales lives in one place. Whether an analyst queries it via SQL, a custom app calls it via REST/GraphQL, or a BI dashboard pulls it, everyone inherits the exact same logic.

Oddly enough, the technical setup took less than a day. The hardest part was getting Finance, Merchandising, and Ops in a room for three meetings just to agree on what "Net Sales" actually means. But once you survive that battle, you lock it into code and never have to revisit it. Unexpected win- BI for Agents: We’ve recently started testing AI agents on top of our data. Previously, LLMs gave inconsistent answers because our underlying logic was fragmented. With a structured semantic layer acting as BI for Agents, the AI just requests pre-defined metrics instead of trying to write complex raw SQL from scratch

How are you all handling metric consolidation across teams? Are you using dbt, built-in BI modeling, or dedicated headless tools?


r/analyticsengineering • • Feb 21 '23

Data modeling patterns every analyst should know

2 Upvotes

A few months ago, Ergest Xheblati taught over 3,000 data professionals how to dramatically improve one of their core competencies - writing SQL.

In a nutshell, he taught how to make your every-day SQL queries more robust, maintainable, performant, and easier to write (If you haven’t watched it yet, check it out! It’s a fantastic resource for all SQL practitioners, regardless of your skill level).

Next month, Ergest will teach best practices about another core competency - data modeling!

If you’ve built data models before, you’ve probably realized how much maintenance and rebuilding is required. It can be a time-sucking black hole.

Luckily, Ergest has been data modeling for 15+ years, and in his upcoming workshop he’ll teach you (hands-on) how to make all your data models sustainable, starting from inception.

More specifically, you’ll learn:

  • How to build robust and maintainable ER data models.

  • How to use said models for multiple use cases (BI, analysis, reverse ETL, etc.)

If you’re interested in attending with me, sign up here!

0 CommentsShareSave


r/analyticsengineering • • Mar 30 '23

Twitch Steam - Learning + applying dbt for the first time

2 Upvotes

My friend Matthew Brandt (Matty_TwoShoes on twitch) is doing a Twitch steam next Thursday where he'll try dbt for the first time! (note: that's "dbt".....not "dmt".) Matthew brandt is not trying dmt over a live stream. That'd be incredibly inappropriate.

If you're interested in seeing what it's like to learn and implement dbt, check it out with me!

Matthew will uncover the all the easiest + hardest aspects of learning dbt, and you'll get to enjoy watching all his successes and failures from the comfort of your home 🤣

If you'd like to add the event to your calendar, sign up here:https://www.operationalanalytics.club/events/unboxing-dbt-a-twitch-livestream-w-matthew-brandt-1


r/analyticsengineering • • 26d ago

The fix that worked for me was boring: every number in the status report gets one plain sentence saying what it counts, and one line saying what it left out.

0 Upvotes

The fix that worked for me was boring: every number in the status report gets one plain sentence saying what it counts, and one line saying what it left out.

For example:

- Past due: open projects (no done date, not archived) whose due date is before the data date

- Left out: title row, totals row, archived projects (rows 25 and 26)

- Data through: Oct 18

Then when the sponsor counts 26 rows and you say 4 are late, the answer is already on the page: 26, minus the title row, the totals row, 2 archived and 18 that aren't late, is 4.

Also write down who decides "Red". A status color without a written rule is an opinion, and that's where most of the Monday arguments come from.


r/analyticsengineering • • Apr 08 '26

We just launched a semantic layer for agentic analytics

0 Upvotes

After a lot of debate internally, we just relaunched Mitzu.

We kept getting the same feedback: people tried ChatGPT or Claude on their data and got burned by hallucinated metrics. The problem isn't the AI — it's that general models have no idea what "active user" or "churned MRR" means in your company.

Our answer was to build the agent on top of the warehouse semantic layer instead. It only answers using metric definitions your data team has already signed off on. The SQL is visible, the data never moves.

Honestly not sure if this is the right architecture long-term — curious what others think. Is the semantic layer the right trust mechanism, or are you solving this differently?

mitzu.io


r/analyticsengineering • • Aug 28 '24

Analytics Engineers: $6000 Social Media Data Modeling Challenge (12 Days Left!)

8 Upvotes

Hey all! There's still time to jump into our Social Media Data Modeling Challenge (Think hack-a-thon) and compete for $6000 in prizes! Don't worry about being late to the party – most participants are just getting started, so you've got plenty of time to craft a winning submission! Even with just a few hours of focused work, you could create a competitive entry!

What's the Challenge?

Your mission, should you choose to accept it, is to analyze real social media data, uncover fascinating insights, and showcase your SQL, dbt™, and data analytics skills. This challenge is open to all experience levels, from seasoned data pros to eager beginners.

Some exciting topics you could explore include:

  • Tracking COVID-19 sentiment changes on Reddit

  • Analyzing Donald Trump's popularity trends on Twitter/Reddit

  • Identifying and explaining who the biggest YouTube creators are

  • Measuring the impact of NFL Superbowl commercials on social media

  • Uncovering trending topics and popular websites on Hacker News

But don't let these limit you – the possibilities for discovery are endless!

What You'll Get

Participants will receive:

  • Free access to professional data tools (Paradime, MotherDuck, Hex)

  • Hands-on experience with large, relevant datasets (great for your portfolio)

  • Opportunity to learn from and connect with other data professionals

  • A shot at winning: $3000 (1st), $2000 (2nd), or $1000 (3rd)

How to Join

To ensure high-quality participation (and keep my compute costs in check 😅), here are the requirements:

  • You must be a current or former data professional

  • Solo participation only

  • Hands-on experience with SQL, dbt™, and Git

  • Provide a work email (if employed) and one valid social media profile (LinkedIn, Twitter, etc.) during registration

Ready to dive in? Register here and start your data adventure today! With 12 days left, you've got more than enough time to make your mark. Good luck!


r/analyticsengineering • • Feb 25 '26

Agentic Ai cohort

Thumbnail
reddit.com
0 Upvotes

r/analyticsengineering • • Sep 05 '25

Seeking Advice: Business Analyst Internship in San Antonio with No Direct Experience

0 Upvotes

Hi everyone,

I’m currently hunting for a Business Analyst internship in San Antonio. I graduated last year with a B.Com Honours, but I don’t have any direct work experience in Business Analytics.

So far, I’ve applied to a few companies but have faced rejections and I’m not sure what I’m missing. I’d really like some guidance on:

  1. Resume tips: What are the must-have elements on a resume for someone without direct BA experience to get shortlisted?

  2. Strategies: What steps should I follow to increase my chances of landing a BA internship? Are there certifications, skills, or types of projects that help?

  3. Application approach: Should I focus on certain types of companies, or ways to connect with hiring managers/HR in San Antonio?

I’d greatly appreciate any advice, tips, or personal experiences you could share.

Thank you in advance!


r/analyticsengineering • • Jul 29 '25

New playbook for Data Product Managers

Post image
0 Upvotes

r/analyticsengineering • • Jul 17 '26

[Academic] Handmade Product Preferences & Buying Habits (Everyone)

0 Upvotes

Hi everyone! I’m conducting a short anonymous market survey to understand people’s preferences toward handmade products and small businesses. It only takes around 2–3 minutes to fill out and would really help with my research.

Form link:
https://forms.gle/GwjfSW7jQiZNaWfQ6

Thank you!


r/analyticsengineering • • 19d ago

I'm building an AI data engineer that runs as a Databricks App. Here's what broke along the way

Thumbnail reddit.com
0 Upvotes