Hello,

In some industries, we have to deal with sensitive data fields, whether it's PII (personally identifiable information) or other types of confidential data (e.g. investment orders, tax classes, medical records, etc.).

So far, in the European companies I've worked for, I've seen different approaches to this problem, despite all being subject to the same regulations. I've also tried reading some online content to get different points of view, but the approaches seem to vary quite a bit. These are the main options I've found or come across:

1. Import everything into the datalake/bronze layer, then only provide access to the datamart/silver/gold layers, which contain only anonymized or aggregated data.

2. Skip ingesting sensitive data fields unless they are strictly needed for downstream transformations.

3. Mask or hash sensitive data values at ingestion.

My take on these options is:

- Option 1 feels like the most natural approach because it fully replicates the source data. However, it also brings the highest compliance risks, as sensitive data is stored in multiple locations, increasing the attack surface and the number of places where access needs to be controlled.

- Option 2 is the strictest and potentially the most compliant approach, but it means our datalake/bronze layer is no longer a full replication of its data sources. If some of the excluded fields are needed later, we might have lost access to historical data. The same applies to option 3 when masking is used.

- With option 3, hashing preserves the ability to perform joins and unique/distinct operations. However, apparently, some regulators still consider hashed PII to be PII if the hashing function is injective (i.e. each output value maps to a single input value). Even though the original PII values are no longer available, we can still link an individual across multiple records.

Overall, I think the right approach also depends on whether the people working with the data have a way to temporarily access the source systems directly. If they do, there may be no need to store sensitive data in the datalake/bronze layer. If they don't, keeping it there provides a way to access it when needed without having to configure privileged access to the source systems.

At my current company, there is quite a bit of disagreement on this topic. My manager is in favor of option 1, based on what he has seen in his previous experience, while the security/risk/data protection teams lean more towards option 2, arguing that sensitive data should only be stored where there is a valid business purpose and with strict access permissions.

If you've worked in an industry with regulations around sensitive data, what approach did your company take? I guess it is very different from one company and country to another but I'm curious about what approach you use(d) and if there are other alternatives to these 3 options.

Thanks!

tl;dr: I am currently working on building my personal data lake and medallion + platinum pipeline and need help deciding whether to have an Landing zone to Raw to Bronze steps for ingesting data or go directly to Bronze from the Landing zone.

Context:

I am working on a quantfied_self project where i am collecting a bunch of data on myself. Things like (everything is timestamped):
- Location at all points in the day,
- Every single workout set
- Major actions i do during a day like brushing teeth, reading a book, meals, work, socialising etc etc etc
- Financial transactions
- Hourly health metrics like heart rate, Hrv, steps, energy spent, calorie intake etc etc etc
- And a lot of other data. Again everything is timestamped so can be watermarked easily.

The plan:

The plan is to build my own local data platform where i upload / send the data and it runs PySpark transformations on the data through a medallion architecture and i can use the gold/platinum datasets for visualisations and getting insights into my behaviour patterns.

What I currently have:
- An old laptop that i have converted to a ubuntu server (with GUI) to be used for storage and compute.
- Samba is setup and I have automated data transfer to the exposed “incoming” folder using automations that can run automatically or on demand.
- The repo is set up and a cron job inside the repo runs the pipelines on schedule on the server.
- Current architecture: Data collection (csv or json) -> auto save via automations to “/incoming/health” or “/incoming/financial” or “/incoming/gps” etc etc folders. It works and I am happy to proceed with the next step which is ingestion.

What i want to ask is:
- Should the next step be “/incoming/*” -> “/raw/health”, “raw/workout”, “/raw/financial” folders, which would be an exact replica of the incoming folder for redundancy purposes or
- Go straight to “/bronze/workouts” “/bronze/transactions” or “bronze/heart_rate” and skip raw since that will be one less step to manage??

For anyone thinking this is overkill for my requirements, i am doing it purely for learning purposes. I have been a data engineer for about a year now (but have 6ish years experience working with all sorts of data, plus personal projects) and wanted to learn something new with something that I own while avoiding generic off the shelf project ideas.

Any ideas appreciated. Thanks

Posted this in discord but trying here too, any help is greatly appreciated.

Is anyone out there using Starrocks shared-data in production? I would really appreciate picking your brain because it's been a STRUGGLE for me and my team.

My suspicion is we are trying to use starrocks for too much. Right now we are using both pipes and stream loads to populate raw (bronze) tables then we are using temporal to orchestrate transforms/enrichments into a silver layer. So far this has been nothing but pain with transactions getting piled up until we manually drop partitions. I should mention that most of our silver tables are pk tables which is probably a big drain on performance.

I suspect most starrocks users are pointing starrocks at an already manicured open table format catalog and using it to serve up queries instead of a do it all solution like I think snowflake is? (never used snowflake).

Palantir Foundry... two words that just might get you crucified around here. At least... they seem to bring out the worst in people on this sub and I kind of see why and also kind of don't.
I spent the last 11 months or so working in Foundry and just recently made the jump back into the Snowflake ecosystem... which, for the record, I'm very happy about.

That said, I think people around here often misunderstand what Foundry is and is trying to do. Yes, it's a platform or ecosystem and yes there are a lot of "gotchas" and weird/odd things about the platform but it's not truly a DE tool. It's a full-stack tool... that's abstracted. Yes, there are DE related features (python data pipelines, code repositories, git integration) but at its core Foundry is about letting the business make decisions based on ingested data and then write back those decisions to a dataset of some sort.
Honestly, it's a unique platform. There's nothing out there, to my knowledge, that does exactly what it does (and yes, its costs and arm and a leg too but... again... nothing does quite what it does).

Why did I go back to Snowflake you ask? Well... most of my career (thus far) has been spent in some aspect of data warehousing or data modeling so Snowflake feels like home to me.
I think if I had spent more time writing python or working in the full-stack realm Foundry might feel more familiar to me (or maybe not), but ultimately I like working on the backend.

Realistically though, I don't see Palantir and Snowflake as direct competitors. Foundry is about decision making, Snowflake, while it's moving in that direction (all the big players are from what I can tell), is ultimately still a data warehouse with a lot of different capabilities that stem from being a high powered OLAP DB.

That's probably too simplistic of a synopsis but it's the way I currently see these tools. Anyways, I thought I'd poke the bear and see if others had thoughts (or questions) as well.

I was looking for blog posts about row vs column stores in database systems, trying to find something that I could link to in documentation. I was pretty disappointed with what I saw, so I decided to write my own 🤣

If you work with analytical database systems and have wondered why they are distinct from transactional systems, you might find this post interesting. I give a high-level overview of the many different consequences of "simply" choosing to represent data in columns instead of rows.

Spoiler: this is a very significant decision!

Given that this is an active area of research, I unfortunately couldn't write about everything, but I think I give a good overview of the general techniques that analytical DBMSs (OLAP) utilize. I also link to a lot of other resources in the footnotes, so check them out!

Hi all, wanted to get some extra opinions;

We have a Power BI dashboard that tracks usage of our main app. It’s mostly used by product managers and has a fair number of KPIs. The data sits in a separate DB instance: a scheduled job copies it over from the app DB and recomputes the aggregations, then Power BI refreshes its model from there.

It mostly works, but two things annoy me:

• Power BI feels rigid. We had to make a few compromises on layout, and it’s not easy to get more fluid or interesting visuals out of it. I may be biased though. Drilldown and DAX measures with cross-filtering are nice features, and we don’t have any dashboard changes planned for the next few months anyway.

• Refreshes on Power BI cloud failed a lot. Our DB is behind an SSH tunnel, so the infra team set up a gateway that wasn’t very stable. On top of that, Power BI silently disables scheduled refresh after repeated failures. We now monitor refresh status through the API and the gateway is mostly fine, so it’s less of an issue today.

We also have a separate React app for business administration stuff around the main app, and the KPI data is basically already available to it. So the idea would be to build the KPIs into that app with a charting library instead.

I’m struggling to justify it though, i'm curious if anyone has had an experience where they switched from powerbi to a custom app or decided against it; was it worth it, what was the most complex part about it etc.

thanks

Hello there!

I don't know where this is going, but I thought I might share the idea here. :) I am working on an on-premise equivalent to the big hyperscaler-only SaaS lakehouse platforms out there.

At first, I started this project to teach myself some agentic engineering concepts. But now I am thinking about where this is actually going ... e.g. being an AI first & native on-premise lakehouse platform easily deployable with a single operator in your k8s cluster on your bare metal.

https://stardelt.io - if you like, I would appreciate some feedback or leaving a star on the project :) Cheers, Martin

Hi there,

I've recently worked with the Etleap engineering team on a 3-part technical blog series about how to handle Iceberg deletes in streaming / CDC. This is a known pain point as deletes tend to get expensive, slow, or create tables that are unreadable in query engines like Snowflake.

We put a ton of work into these - it's not the usual vendor spam, promise 😅

First 2 parts are purely educational, and we introduce our own solution in the 3rd. We cover position deletes, equality deletes, GET request costs, compaction conflicts, and a lot of other juicy stuff.

Part 1: Why Streaming Updates Break Iceberg Compaction

Why you're seeing issues like compaction jobs failing with Cannot commit, found new delete for replaced data file**,** ingestion commits being rejected with Missing required files to delete**,** or degraded query performance. Link

Part 2: Understanding Iceberg CDC costs on S3: requests, not bytes

The mechanism behind a rapid increase in S3 GET requests when moving from batch to streaming ingestion. Link

Part 3: Using Iceberg Branching to Stream Updates Within 2 Minutes

How we convert equality deletes into deletion vectors to maintain low-latency data, avoid compaction conflicts, and deliver Iceberg tables that are readable in any engine. Link

Would love to understand if there's anything we missed here, or if this kind of content is helpful for the community! I'm around for any questions, and will loop in any of the engineers to answer your questions :)

Best,

Eran

Hi all, this is a general qns on levelling up our dbt set up as a junior analytics engineer. This includes more test coverage, architectural and modeling improvements, adherence to best practices, and so on.

For context, I am working at a small organization of about 50-100 employees and have set up the dbt repo myself in the past year. We do not have the budget for a senior and so I have been trying to learn using web resources and books (like star schema by adamson). The problem is that I feel we are missing an ‘expert’ that can look through what we are doing and advising on what is suboptimal or not. Even though there are lots of resources out there, every dbt implementation is different and many 'best practices' may not be relevant for a small set up like ours. Ultimately the goal is to improve our set up wherever we can and in the process also improve my skills.

We are looking at hiring external consultants as a possible solution. I am wondering if people here have gone through something similar, either with consultants or self learning through public resources/on the job, and what your experience was like?

I'm on the job-hunt 75% through the process for a DE Manager position. I'm a staff at a scale up (but I feel some title inflation and impostor syndrome).

They're working on bringing data in-house from a vendor and it's basically Fabric + DBT -> Power BI so deliverables are a bunch of dashboards with future state being all dashboards, then warehouse + ML as a product down the road.

The catch of all this is it's in Fabric. I see all the hate but I've never used the platform, and I've been all over - Oracle, Postgres, Airflow, Pre-Databricks Spark platform, Scala, python, pandas, databricks, some DBT. Pretty much exclusively in AWS as far as cloud providers go.

The role on the ground is a 75% IC + 25% Mgr role.

I'll have a tech round with their matrixed Senior/Staff level in the next week that's been setting up the infra and environments.

What should I be asking to make sure this isn't awful badly done? What should I be thinking about or looking out for?

I’ve been working for around 5–6 years across different roles and companies, and one thing I’ve consistently struggled with is taking ownership. I tend to be reactive rather than proactive. I hesitate to make decisions, rarely take initiative, and often rely on others for direction. I’ve noticed this pattern across multiple jobs, and I genuinely want to change it.

For those who have developed a strong sense of ownership, how did you do it?

Hi all,

Me and some other people are responsible for all data and analytics throughout the company. Analytics is enabled company wide via a standard “medallion architecture” inside Databricks. This means that we use a standard pipeline job with notebooks to ingest the raw data in the bronze layer, clean/govern/format the data in the silver layer, and in gold we enforce a kimball schema and business logic.

Up until recenty, it was fine to append-only in bronze and just dedup those tables in silver so no information is ever lost. However, with the increase of our data streams, appending all into bronze is quickly turning into a massive data lake. Many of these tables may only have 5% of them changing throughout consecutive ingestions, but we cannot miss those changes either.

My question is; how do you guys deal with keeping high visibility on changes (our system does not allow for direct extraction of CDC tables), whilst also keeping the size of your data lake in check? (I heard that lakeflow declarative pipelines might be useful). Any suggestion is welcome!

Hello everyone!

Around Februrary this year I've started working on my side project, which is now known as Braidplane. It's an open source platform for data integration; it's self-hostable and the core is fully open (Apache 2.0). At this moment it's in Alpha.

The goal of this project is to provide safe and reviewable data integrations for engineering teams. From the beginning this project was built upon data contracts (inline and ODCS), e2e Braidplane Proof engine for reviewing data pipeline updates and declarative pipelines (YAML, JSON, cue).

As an example, a pipeline can read data from a database and a file, transform and compare the records, and write the results to another system. The pipeline definition stays in Git, so changes can be reviewed before deployment.

In the current form Braidplane core includes backend, frontend, CLI and executor. MCP support is coming soon.

About me: my name is Lukas. I've been working as backend engineer for the last 10 years. I've spent a lot of time around different products and APIs. I've built the Braidplane's framework myself - based on my experience and good practices I've found around the other OSS projects. Later on I've added AI assisted programming around the hardest/most time-consuming parts, like Connectors integrations, tests, documentation, frontend, issues management.

The Braidplane Core repository is available here: https://github.com/Braidplane/braidplane-core

Examples repo: https://github.com/Braidplane/examples/blob/main/recipes/opaque-file-archive/opaque-file-archive.pipeline.json

Cheers

Working on a personal project and asked copilot if my old laptop can handle the dataset and has enough compute to comfortably run spark locally because it is my own workout dataset.

I only wanted to know whether my machine is good enough, wasn’t expecting a full on roasting.

One of the senior engineers who is responsible of reviewing PRs tried to optimize a job's parts of code with AI, it was a 2000 lines job, the AI removed a small code snippet, he didn't notice that, pushed to production

We spent weeks handling business complaints, incidents, investigating till we found that AI removed a tiny code snippet

With technology evolving every f@¢k1ng day, how do y'all manage to keep learning and be motivated to keep learning everyday? Is there some kind of passion in learning that you have or do u push yourself to keep learning even if it gets hard sometimes?

EDIT - The title should say a 'Polyglot fork' (Polyglot is a SQLGlot inspired transpiler). Can't edit it now unfortunately.

Hey guys! I'm the creator of SQLBuild, and for a while I've wanted a more reliable way to run warehouse SQL locally on DuckDB.

SQLGlot and Polyglot are both SQL transpilers that have been a major blessing to the community, and some of my projects (this obvs included) wouldn't have been possible without them. However, at least as of today, these libraries both tend to lean toward generating SQL, even if the SQL may not be runnable or may produce and incorrect result in the target dialect.

I had the idea for SQLQuack about six months ago, but only recently got around to building it. The idea was to build a fail-closed transpiler that would tell you first of all if it was parseable, second if it would violate a rule (of the souce dialect), and lastly translate it to DuckDB, only if each statement had some sort of supporting evidence for it from a real warehouse recording (e.g. real snowflake recorded result matching DuckDB).

Below is an example of what an error might look like in SQLQuack:

SELECT customer_id, SUM(amount) AS total
FROM orders
GROUP BY customer_id
QUALIFY total > 100

$ sqlquack check qualify.sql
error[SQ0203]: QUALIFY needs a window function
 --> qualify.sql:4:1
  |
4 | QUALIFY total > 100
  | ^^^^^^^^^^^^^^^^^^^ QUALIFY is used without a window function in the SELECT list or QUALIFY
  |
  = help: move the condition to WHERE or HAVING, or filter on a window function's result

The basic goal is to basically create a dataset of all the warehouse / databases and their true behaviours, such that we a have a truly accurate local compiler for all dialects that can be verified with anyone willing to run the recordings.

In the near term, my immediate focus will actually be improving / simplifying the generated SQL though (some if it is too verbose) and getting snowflake to be 100% finished (it's maybe 60/70% of the way there atm depending on how you measure it). Other dialects are in the works but not ready yet.

Credit (to Community / Maintainers)

A big thank you to Tobias (@tobilg), who maintains Polyglot and fixed many an issue I raised while working on SQLBuild (~6 months).

I ended up maintaining a fork, but over time realised I wanted a different approach to how conversions are checked and when they should be refused. SQLQuack's parser started from code derived from Polyglot, so I definitely don't want to present this as something built without that foundation.

The same goes for SQLGlot. A lot of this work wouldn't have been possible without those projects and the people who built them. This is a different set of trade-offs, not me saying their approach has no value.

Playground: https://sqlquack.io/playground/

GitHub: https://github.com/sqlstack-io/sqlquack (Apache 2.0)

NOTE - I will see if SQLGlot or Polyglot would like this evidence to be recontributed upstream, as then much more of the community will benefit.

Hi,

I'm close to releasing our Dataform branching and release process, but I'm unsure about the hotfix approach.

Current setup: Dataform's default branch is main, so every workspace is cut from main and developers PR back to main. Each sprint, we cut a release branch from main, run final tests, and create a GitHub release, which updates the Dataform release config with the new tag.

Current hotfix flow: Devs PR to main with a "hotfix" label. On merge, automation cuts a branch from the latest release branch, cherry-picks the hotfix commits, and opens a PR to the release branch. Someone then cuts a new release.

The problem: The hotfix is made against main but cherry-picked onto the release branch, so merge conflicts are possible. When that happens, the automation fails and the dev has to resolve it locally.

What I'd prefer: Avoid cherry-picking by branching from the release branch, applying the hotfix, and merging into both the release branch and main (merge issues could then be handled in the UI). The catch is that Dataform workspaces always cut from main, and I can't change that without disrupting other developers, so I don't know how devs would develop and test a fix on a release-based branch.

Has anyone solved this, or is there a better pattern for hotfixes in Dataform that I am unaware of. It’s possible I am thinking about this the wrong way.

I have considered having an entirely separate Dataform repo devoted to hotfixes with the same GitHub connection - where devs are expected to change the default branch on the fly before creating a workspace. But this seems clunky and inefficient.

Hey all, so, I have been doing this for about 6 years now on a legacy on-prem stack. Every time I try to transition to something more modern - dlt/dbt I get shot down. Stick to .NET, that's what the company wants. I try to introduce metadata importing, and no time for that, just stick to what you have.

Well, I finally got what I want. We're moving to the cloud. Only it's been decided we don't have enough python/pyspark background, so someone else is writing an engine for us.

I have been told I am going from owning the entire pipeline to only owning bronze/silver and the BAs will own gold.

And Bronze/Silver will be automated. The best I get is to tweak some yaml files. Anything related to actual code will be handled by the "developers".

I can see this maybe lasting 6 months to a year, until all of our sources are in yaml form and there's nothing left for my team of 3 people to do.

To be honest, most of this is exactly what I wanted - except I was hoping I would get to retain ownership of the gold, and possibly have some idea of what my job looked like still.

Any ideas from anyone who has faced similar, being automated out and any advice on how you keep seeing yourself having value and a job?

Current setup
Azure SQL Database (Standard S2, DTU model). No linked servers or cross-database queries.
Power BI Pro for dashboards (40 users, 3 authors).
We have no tool for scheduled or paginated reports, and no SQL Server licenses or Fabric capacity.
What we need
Automated Excel, PDF and delimited-text (CSV/pipe/tab) reports
Delivery by email and to a network share drive (SFTP would be a bonus, but we can move files with external processes)
Parameterized subscriptions, with a large list of them managed from a table, like SSRS data-driven subscriptions
About [N] reports and [N] recipients, mostly monthly
Background
At past jobs we used SSRS or Crystal Reports, and SSRS would fit. But SQL Server 2025 shipped without it (2022 is the last version), and Power BI Report Server replaces it. We don't own SQL Server, so a license would be a new $8k spend just to host reporting. I can only fund one path.
Options I'm weighing
Fabric F2 (1-year reserved, about $1.9k/yr), scaling to F4 on report days. Power BI paginated reports plus subscriptions. As I understand it, delivery goes to email and OneDrive/SharePoint, and per-recipient dynamic subscriptions are email only. For file shares and bulk parameterized runs I'd need to build something with Power Automate, Logic Apps or the Export API.
SQL Server 2025 license (about $8k) to run Power BI Report Server. On-prem, with data-driven subscriptions and file share delivery. Needs a server to host it.
A third-party scheduler (PBRS or similar) on top of Power BI.
Questions
Has anyone built SSRS-style data-driven subscriptions on Fabric paginated reports plus Power Automate or the Export API? How did it hold up (throttling, timeouts, failures)?
Can an F2/F4 handle paginated report exports from an S2 Azure SQL database without throttling?
When scaling a reserved F2 up to F4 for a few days each month, how is the extra billed?
Can the CSV delimiter be controlled in paginated report exports?
For a small team with this budget, which option would you pick, and what would you avoid?
Thanks in advance.

I've been a big fan of DBT since picking it up a few years ago, but there's always been a few nuances at Enterprise scale that I felt detracted from the wizardry it unlocks. Simple things like discovery (does this already exist), duplication for similarly-shaped models, remembering to include appropriate tests (e.g. primary key uniqueness), MSP-angosticity, and rebuilding standard warehouse paradigms are things that I always found myself thinking "surely someone has already solved this". When I looked, I found some great point solutions but none that I felt had a big-picture lens.

Loom-dbt (PyPI, GitHub) is my approach to solving these bugbears. It's a standalone CLI utility that:

• Enables teams to declare model shapes (patterns) once using approachable metadata and generate models (instances) repeatably, supplying only what differs between them

• Embeds features (data tests, keys, load-timestamping) for standard warehouse shapes (paradigms), with multi-paradigm support

• Registers complex and hand-built models outside the generatable patterns

• Provides detailed searching capabilities for project-wide discovery

• Proves what is generated matches what's been declared (or identifies discrepancies)

• and more!

Critically, loom-dbt outputs plain DBT with no long-lived dependency. The models you generate today will survive if you delete loom-dbt tomorrow; adoption is reversible by design.

And of course, no modern project would be complete without appropriate AI-augmentation guardrails: 6 Skills covering core functionality, maintenance, authoring, adoption, bootstrapping, and searching - standardised, npx installable, and ready for your favourite harness.

If you give it a try or are keen to learn more, let me know!

So at work I’m using Azure Pipelines as an orchestrator to materialize several tables in parallel. Each one is running a Create Table As Select sql statement and uses a DBT merge strategy so that each of them are incremental. My concern is that I can’t control for these jobs to run at the exact same time. Table X might start its job at 10:01. And Table Y might start at 10:03 and even a difference of a few seconds could mean that the source (which is Postgres) is not the exact same for both tables.

My fellow data wranglers,

Has anyone built a real production solution using DuckLake? I mean something with an ELT pipeline and actual users, that would have sid ducklake as a component?

I have been offered a position as a Junior Data Engineer at QB.

The problem is that I don't really know too much about their reputation and if I would gain experiences and skills that would transfer well to BigTech/FAANG (which is my end goal).

I worked at a couple of non-technical consultancies prior to my career change and they all involved boring work like making decks, communicating with clients, drafting proposals etc.

The pay is good but the hours make me second guess myself.

I think the typical advice is to avoid consultancies and aim for product tech companies where you are involved with systems throughout its lifecycle.

I think the image in my head when I hear 'tech consultancy' is implementing low complexity pipelines into the code bases of companies that are ancient and having to deal with old tech stacks and corporate politics rather than working on meaningful projects that have impact and use the latest technologies.

Any thoughts on their prestige/reputation/depth of work?