r/analyticsengineering • • 1d ago

Transition from data analytics

6 Upvotes

Hi! I’ve been in analytics for 6-7 years, working in sql/sas/snowflake/dbt and a few different BI tools. For a large portion of that time I was a “senior data analyst” but I built our analytic models, dbt tests, and also owned all downstream reporting and dashboarding. I didn’t own the data pipelines but understood them.

I always loved the data exploration and modeling pieces the most. I recently got a new job and looked for the same title. However, in this role it turns out I’m mostly doing BI, which is mind-numbing. At my company we have analytics engineers (which honestly didn’t even know was a job title) and that seems much more up my alley. Do you think this would be an easy transition to make? At my current company it would make total sense, as they own the analytics models and warehouse but not the data pipelines, so they mostly work in dbt. But curious if that’s the case most places. Is this a logical move that hiring managers would understand?


r/analyticsengineering • • 2d ago

Purview Data Governance is being retired. If you’re on it, are you rethinking the program or just the tool?

Thumbnail
reddit.com
2 Upvotes

r/analyticsengineering • • 2d ago

How do you actually find out about data quality problems? Before or after a stakeholder does?

Thumbnail reddit.com
1 Upvotes

r/analyticsengineering • • 2d ago

Vivanti Consulting Final Round for Senior Data Engineer

Thumbnail reddit.com
0 Upvotes

r/analyticsengineering • • 3d ago

Data science interview prep

Thumbnail reddit.com
1 Upvotes

r/analyticsengineering • • 5d ago

How do you catch dbt descriptions that no longer match the SQL?

5 Upvotes

Someone added a filter to fct_orders that drops cancelled orders. The description still says "all orders". Nothing fails, the docs site looks fine and anyone reading the description would assume cancellations are in there. Right now the only check we have is someone noticing in review.


r/analyticsengineering • • 7d ago

power bi alternative: testing 5 open-source and code-first tools after getting tired of desktop .pbix files and Fabric license creep

13 Upvotes

For the past four years, our reporting ran almost entirely on Power BI. It did the job while our data team was small and stakeholders were happy receiving scheduled PDF exports and opening dashboard links inside Microsoft Teams.

Over the last 18 months, our engineering and analytics setup moved to Linux and Mac, our warehouse matured on Snowflake, and three specific friction points made Power BI painful to live with:

  1. The desktop-client bottleneck. Half our analytics engineers and product managers work on macOS or Linux. Power BI Desktop remains Windows-only. Having engineers spin up Windows VMs or cloud desktop instances simply to tweak a report or verify a measure felt absurd for a modern stack.

  2. Logic trapped in binary .pbix files. DAX measures, table relationships, and row-level security filters lived inside individual report files or workspace models. We could not run standard git pull request reviews on metric changes, track lineage across versions, or prevent two analysts from writing slightly different versions of the same retention calculation.

  3. License sprawl and the Fabric push. What started as $10/user/month Pro licenses steadily escalated with Premium-per-user upgrades, external viewer licensing constraints, and heavy pressure to commit to Microsoft Fabric capacity (F64+) just to access basic direct lake and copilot features.

and thing is that we spent the past month evaluating five open-source and code-first tools to replace Power BI. We wanted an architecture where metric logic lives in version-controlled code, dashboards run cross-platform in the browser, and numbers can be queried both by BI tools and by downstream applications or AI agents; here is the breakdown of our test setup, the comparison matrix, and where each option landed.

The Comparison Matrix

Evaluation DimensionPower BI (Baseline)Apache SupersetMetabase (OSS)LightdashEvidence.devCube (Cube Core OSS)
Primary Mental ModelDesktop-first drag-and-drop report builderOpen-source web-based dashboard platformFast self-serve query builder and dashboardsdbt-native BI built directly on dbt modelsCode-based reporting in Markdown + SQLHeadless semantic layer + open data APIs and web UI
Metric & Logic StorageProprietary DAX in .pbix files or Fabric semantic modelsVirtual datasets and SQL Lab queries (stored in metadata DB)Questions and Model objects in application databaseDefined as metrics in dbt schema YAML filesSQL queries embedded in Markdown filesCode-first YAML/JS models in Git repositories
Git / PR WorkflowDifficult; Git integration requires Fabric / PBIP formatRudimentary; requires export/import scripts or API syncSerialization via YAML requires paid Enterprise planNative; metric definitions live inside your dbt repoNative; entire project is markdown and SQL in GitNative; repository-first with standard branch & PR workflows
Cross-Platform AccessWindows-only desktop client for authoring100% browser-based authoring and consumption100% browser-based authoring and consumption100% browser-based authoringBrowser-based, CLI local server on Mac/Linux/WinBrowser-based, CLI local server on Mac/Linux/Win
Downstream & API AccessClosed; XMLA endpoints require Premium capacityREST API for metadata; chart data exportableREST API (returns card query results)Query API through dbt Cloud / Lightdash CLIStatic site export; not an API serviceUniversal SQL API (Postgres protocol) + REST/GraphQL APIs
AI / Agent ReadinessCopilot locked behind Fabric capacity commitmentsCommunity AI plugins; mostly text-to-SQL over raw schemasMetabot assistant; generates queries over Metabase modelsAI chat grounded in dbt metric definitionsLLMs can generate Markdown/SQL filesNative remote MCP server and semantic layer context
Biggest Trade-off / GotchaWindows lock-in, proprietary DAX, per-seat licensingHeavy Docker setup, permission model can get complexMetric governance is light; joins duplicate easilyRequires dbt for everything; not independentNo ad-hoc drag-and-drop for non-technical usersYou have to model data in YAML before building charts

What We Learned Testing Each Option

  1. Apache Superset: Great for rich charting, heavy on operational maintenance

If your primary goal is finding an open-source tool that matches Power BI's extensive chart variety, Superset is the closest visual equivalent. It handles complex dashboards, geospatial visualizations, and slice-and-dice exploration without per-seat licensing.

* Where it worked: Solid out-of-the-box dashboards, zero Windows dependencies, and strong community support.

* Where it pinched: The permission system is intricate and requires careful role configuration. While Superset has virtual datasets, business logic still ends up scattered across ad-hoc SQL snippets in different charts rather than sitting in a centralized semantic layer.

2. Metabase: Unmatched for rapid self-serve, but lighter on modeling governance

Metabase remains the fastest tool for getting non-technical team members exploring data. The graphical query builder allows sales and product managers to ask questions without writing SQL.

* Where it worked: Onboarding takes less than an afternoon. Anyone comfortable with Excel filters can build their own cards immediately.

* Where it pinched: Metric logic is stored in Metabase's application database rather than in code. Version control via YAML requires higher-tier licensing, and managing row-level security across multi-tenant departments required duplicating questions once our role hierarchies became complex.

3. Lightdash: Ideal if your team is already 100% committed to dbt

Lightdash takes a dbt-first stance: your dbt project is your single source of truth, and Lightdash auto-generates your BI layer from your dbt schema YAML files.

* Where it worked: All metric definitions live in Git pull requests alongside transformation models. Clean versioning and developer workflow.

* Where it pinched: If an analysis doesn't fit your pre-compiled dbt DAG, you cannot easily explore it on the fly. You also cannot point other non-Lightdash tools or customer-facing apps at its metric engine without going through their specific query layers.

4. Evidence.dev: Excellent for data engineers, foreign to drag-and-drop business users

Evidence replaces traditional BI dashboards with code-generated analytical markdown documents. You write SQL queries in fenced blocks and render charts with declarative tags.

* Where it worked: Code-driven, fast static rendering, and versioned cleanly in Git. Our analytics engineers liked the developer experience.

* Where it pinched: It is fundamentally a publishing tool for analysts, not an ad-hoc exploration tool for business stakeholders. If a VP wants to drag a dimension into a pivot table on Friday afternoon, Evidence won't serve that workflow.

5. Cube dev and decoupling the metric layer from the visualization

We tested Cube Core as an open-source semantic layer sitting between Snowflake and our visualization tools. Instead of defining metrics inside each dashboard tool, we defined cubes, dimensions, joins, and measures in YAML files stored in our git repository.

* Where it worked:

- Universal consumption: We can connect Superset or Metabase to Cube's SQL API (which speaks the Postgres protocol). Both tools query the exact same governed metric definitions, so nobody argues over differing numbers.

- Performance and caching: Cube's pre-aggregation engine stores partitioned rollups, serving repetitive dashboard filter changes in sub-second response times without triggering live warehouse compute on every query.

- Multi-surface readiness: Because Cube provides REST, GraphQL, and remote MCP endpoints on top of the same model, our engineers can build embedded customer portal widgets and point LLM agents at governed measures without re-implementing business logic.

* Where it pinched:

- Upfront data modeling investment: Unlike Power BI where you can drag raw tables and click auto-detect relationships, Cube requires explicit modeling of join paths and primary keys in code. If your underlying schema has messy foreign keys, you must resolve them before metrics compile.

- Dashboard building requires picking a front-end: While Cube provides built-in workbooks and web exploration, teams wanting traditional heavy dashboards typically pair it with an open-source visualizer like Superset or custom React components.

---

so how we decided to structure our stack

Rather than trading one monolithic vendor for another single closed tool, we settled on a two-tier decoupled architecture:

  1. Semantic & Governance Tier: We use Cube Core as our open-source metric layer. All calculations (such as gross retention, ARR, and active seats), join topologies, and role-based access rules are written in code, reviewed in pull requests, and versioned in Git.

  2. Consumption & Visualization Tier: For internal exploratory dashboards and operations teams, we point open-source Superset at Cube's SQL API. For our engineering team's customer-facing portal and agentic workflows, we query Cube's data APIs directly.

as for questions! for analytics engineers who migrated off Power BI:

- Did you replace it with an all-in-one platform like Superset or Metabase, or did you decouple the semantic layer from visualization?

- How did your analysts adapt to modeling metrics in Git and YAML instead of writing DAX formulas in desktop files?


r/analyticsengineering • • 8d ago

AI agent/skill hidden gems

3 Upvotes

Hey everyone,
What’s an AI agent or skill you built or found that helped you doing you AE work? Trying to find new creative ways to help me.
Even if it’s a silly thing i would love to hear it and how you’re using it


r/analyticsengineering • • 9d ago

If every bit of code is not build by AI, how much importance should be given how things are build, in data side of things?

7 Upvotes

I agree that scalable processes depend on good engineering practices. But LLMs can now write clean, efficient code from plain-English instructions. You can tell them to follow best practices, and they can use what they know about each tool to do it.

If that holds, the whole workflow can be generated and controlled through code written from simple prompts. That includes building pipelines, curating data, and creating reports, even app-based ones.

So I’d like to hear from people doing this work every day:

  • Is it still worth learning how things happen under the hood

  • Should we understand the code an LLM generates for a pipeline or an app-based report, or is it enough to review the output?


r/analyticsengineering • • 9d ago

Analytics Engineering Intern Rounds?

Thumbnail reddit.com
1 Upvotes

r/analyticsengineering • • 9d ago

Decision on creating reports via app vs BI tool

Thumbnail reddit.com
2 Upvotes

r/analyticsengineering • • 9d ago

Open source entity resolution platform on Databricks marketplace

2 Upvotes

Hello All,

Zingg lead dev here! Excited to have Zingg available as an app on Databricks marketplace. Took a lot of effort from our tiny team, and we are glad we finally made it! Hoping it helps folks here build massive scale identity resolution systems, master data management, composable CDPs...

Please do check it out and let us know your feedback. Always eager to know how the product is getting used and what we could do better.

https://marketplace.databricks.com/details/6e0e1411-65c8-44ae-bdfa-ab706770ac11/Zingg-Labs-Inc_Zingg-Entity-Resolution-and-MDM


r/analyticsengineering • • 10d ago

GraveTrail Lineage tool

0 Upvotes

Hi All

As a data engineer i spent a lot of annoying time understanding the table Lineage in a prehistoric legacy system.

Based on my hardships and understanding I've built this Lineage tool

https://sahasilak.github.io/GraveTrail/

I would appreciate it if you can spend some time trying it out by sharing feedback like

Good, bad, need improvements , something like this already exists


r/analyticsengineering • • 10d ago

Marketing BA hired as a Technical BA on a cloud data project: what should I learn about Dataproc, Spark, BigQuery and GCP?

Thumbnail reddit.com
2 Upvotes

r/analyticsengineering • • 10d ago

Business Analyst Career in 2027 – What Skills Should I Learn in the AI Era?

Thumbnail reddit.com
1 Upvotes

r/analyticsengineering • • 11d ago

Anyone else spending way too much time cleaning messy data?

Thumbnail reddit.com
1 Upvotes

^ Dm us for AI that solves your analytics engineering needs.


r/analyticsengineering • • 12d ago

I feel like am not gaining any skills

9 Upvotes

So am jr data analyst/engineer idk what to name myself. My job role is ‘developer’ and my team is a backend dev team and am the only data person helping data processes to migrate from legacy to cloud. So basically
\\- Am just rewriting sql scripts from legacy to databricks
\\- Built an adf pipeline for this process
\\- Worked few 2-4 Java dev stories when there was no data work
\\- Duplicate transactions issue
\\- designed an excel dashboard for our new feature
\\- basically am leading the migration process but am the only one doing and this is my first job and no other teammate doesn’t involve in my works at all so am my tickets are being carry over (3-4) from last 2 sprints
But I feel like my TL never feels like am doing important work and I don’t have communication with other dev teammates as they don’t give a shit about what I work.
I feel like am under valued..am always available for all the works.
My PM encourages me always positive and recognize and appreciate my works in sprint reviews

But I really want to know…are these real data jobs and responsibilities or am I not learning anything? Am I not important at my work?


r/analyticsengineering • • 16d ago

User study on a data automation tool

Thumbnail reddit.com
1 Upvotes

r/analyticsengineering • • 17d ago

Sales Targets vs Actuals for Individual Quota Attainment

1 Upvotes

I’m struggling to model my hubspot & google sheets data to report on quota attainment for our sales team members.

For example, we enter their targets in a google sheet that looks like:

employee_id | metric_key | target_value | period_start | period_end

And we match the targets to actuals using metric_key, that part is easy.

The part where it starts to break down is that they want to be able to filter by “team” ie. Retention vs New Business.

We attach a user to a hubspot deal on the deal_owner field, regardless if they are on the Retention team or the New Business team.

So I started assigning a “plan” to each target row in the google sheet, and created a dbt seed that maps plans to valid metrics. So the Retention plan might have renewed_arr as a valid metric, for example.

That mostly works, but someone could have closed a deal when they were on the Retention team, and later moved to the New Business team, and if that closed deal was cohorted during the period that their target on the New Business plan was active, the deal would count toward that plan.

Looking for ideas from anyway who has worked on a similar problem.


r/analyticsengineering • • 17d ago

Migrating snowflake worksheet/notebook

Thumbnail reddit.com
1 Upvotes

We are in the process of migrating from one client Snowflake account to another. The client has a large number of worksheets and notebooks across different users that need to be migrated. Could someone advise whether this is possible and, if so, what the best approach would be?


r/analyticsengineering • • 17d ago

Mapping datasets by reach and promises

Thumbnail
steffenmoll.github.io
1 Upvotes

The Open Data Contract Standard, Apache Ossie and Medallion structure all lacks a central component, who the data is for and how it hits the organization! Join me as I elaborate on some thoughts while working on data platform in a broader context


r/analyticsengineering • • 18d ago

Worked six months unpaid at a startup, trusted promises of a paid role, then was let go. Looking for advice and job leads.

1 Upvotes

I spent six months working unpaid at an early-stage real estate startup founded by a fellow UMass Amherst alum. I joined while on OPT after a couple of interview rounds. The company was building an AI/CRM product, and I came on as a data engineering intern.

About two months in, the founder told me he liked my work and wanted to hire me full-time. When my internship ended and I asked about the transition, he said he needed another two months. He assured me we’d be able to move forward then, so I stayed and continued working unpaid.

My colleague and I believed in the company’s goal and the future the founder described. He told us that once we became full-time employees and the company scaled, we would grow into bigger roles because we were laying its foundation. We took that seriously and approached the work with the care and responsibility we would have given our own company. From data sourcing and building pipelines to system design, we put in long days and worked weekends because we wanted it to succeed.

By late October, the timeline shifted again. He said full-time roles might have to wait until next year, without a clear date. My colleague and I explained that we couldn’t keep working without income. We asked whether we could at least move to paid part-time roles so we could cover basic expenses while continuing to contribute.

He told us he understood our situation, recognized how many hours we had been working, and would definitely arrange part-time pay. He described a plan to increase that pay each month over the next four months, with the possibility of full-time roles around the fifth month. He asked for a few days to work things out. Roughly ten days later, both of us were let go.

I understand that startups face uncertainty and plans can change. What hurts is that I made decisions based on repeated assurances that a paid role was coming. After six months of unpaid work, asking for some financial support was followed by losing the role entirely.

The timing has made this especially difficult. It’s now November, and I have limited time remaining on my OPT unemployment clock. I’m worried hiring will slow over the next few weeks and stay limited through the holidays and into January. If interviews don’t materialize until February, a potential start date could end up being April of next year. That isn’t certain, but with my immigration timeline, the possibility is frightening.

For the founders here: please be honest about what you can commit to. People may be planning their finances, careers, and immigration timelines around those conversations.

For anyone considering a similar arrangement: get the terms in writing, set a limit on how long you can afford to work unpaid, and keep your job search active. I wish I had done that sooner.

I have three years of experience across analytics engineering and data engineering, working with complex datasets, building data pipelines, and transforming raw data into reliable datasets for analysis and reporting. My work bridges the engineering and analytical sides of data: integrating sources, structuring data, and making it useful for answering business questions and supporting decisions. I’m looking for opportunities where I can bring that mix of technical skills and analytical thinking.

I’d also really appreciate hearing from anyone who has been through something similar. Honestly, I’m struggling right now. After putting six months into something I believed would become a stable opportunity, I’m suddenly trying to figure out how to support myself and find another role while my OPT clock keeps running. It’s overwhelming, and I’m scared about what comes next. If you have a job lead, know someone who’s hiring, can offer a referral, or have advice that could help me take the next step, please reach out. Even a small lead or an introduction would mean a lot to me right now.


r/analyticsengineering • • 18d ago

Free Microsoft DP-600 vouchers during Data Days

4 Upvotes

Hello from the Microsoft team -- wanted to let you all know we're offering free vouchers for the Microsoft Fabric Analytics Engineer exam during Data Days. https://aka.ms/datadays