Hey everybody, I wanted to talk a bit about my journey as a data engineer and candidly answer any questions you might have.

I started as a data analyst back in 2013 and learned about this yellow elephant named Hadoop. I became obsessed with learning it because big data was so hot back then.

Late 2014, I landed a job at Teradata doing big data and Hadoop work making around $80,000 in Utah. This job was exciting but I realized if I wanted to make any real money I needed to get to New York, Seattle, SF or DC.

In 2016 I picked a defense startup in DC which paid $95k. After adjusting for cost of living, it was a worse compensation than $80k in Utah.

I worked there for 7 months before Facebook reached out in August 2016.

I fly from DC to Silicon Valley for my chance. It was the most intense 8 hour experience of my life. I get low balled and offered an L3 position (it was $185k and since it was so much more I didn’t realize I was lowballed until later).

I worked at Facebook for 9 months and get promoted to L4 after grinding out some projects that saved hundreds of terabytes of space and thousands of compute hours.

I got impatient at Facebook because when I got L4 I realized I was actually an L5. I tried to get promoted from L4 to L5 in six months and it didn’t happen and I was kind of furious.

So I looked outside and ended up landing a senior DE role at Netflix in 2018 making $365k. (Again, lowballed but I didn’t realize it since it was almost double Facebook). About six months into my time at Netflix I realized my lowest paid team mate was making $500k. This made me furious and I worked really hard to get the bump I deserved. In 2019 I built a graph database for Netflix that mapped their entire microservice architecture and landed 2 cybersecurity patents. This effort got me bumped from $365k to $550k.

Netflix culture was kind of overwhelming for me so I quit in the middle of 2020 and took six months off. I learned many painful lessons from this experience.

After six months of depression and COVID, I decided to check out working at Airbnb and I landed a staff offer there for $600k. I worked there for the next 2ish years and got great performance reviews each year to get a bump (the stock did horrible so the compensation bump just evened out with the stock price fall).

After 2023, I quit to be a full time entrepreneur which I’ve been doing for the last 3 years.

I’m here to answer anybody’s questions for the next few hours. Let me know what you got!

I keep seeing people discuss having a gold layer in their data warehouse here. Then, they decide between one-big-table (OBT) versus star schemas with facts and dimensions.

I genuinely believe that these concepts are outdated now due to semantic layers that eliminate the need to make that choice. They allow the simplicity of OBT for the consumer while providing the flexibility of a rich relational model that fully describes business activities for the data engineer.

Gold layers inevitably involve some loss of information depending on the grain you choose, and they often result in data engineering teams chasing their tails, adding and removing elements from the gold layer tables, creating more and so on. Honestly, it’s so tedious and unnecessary.

I wrote a blog post on this that explains it in more detail:

https://davidsj.substack.com/p/you-can-take-your-gold-and-shove?r=125hnz

Hey folks,

you know that special kind of pain when you open yet another REST API doc and it's terrible? We felt it too, so we did something a bit unhinged? - we systematically went through 1000+ API docs and turned them into LLM-native context (we call them scaffolds for lack of a better word). By compressing and standardising the information in these contexts, LLM-native development becomes much more accurate.

Our vision: We're building dltHub, an LLM-native data engineering platform. Not "AI-powered" marketing stuff - but a platform designed from the ground up for how developers actually work with LLMs today. Where code generation, human validation, and deployment flow together naturally. Where any Python developer can build, run, and maintain production data pipelines without needing a data team.

What we're releasing today: The first piece - those 1000+ LLM-native scaffolds that work with the open source dlt library. "LLM-native" doesn't mean "trust the machine blindly." It means building tools that assume AI assistance is part of the workflow, not an afterthought.

We're not trying to replace anyone or revolutionise anything. Just trying to fast-forward the parts of data engineering that are tedious and repetitive.

These scaffolds are not perfect, they are a first step, so feel free to abuse them and give us feedback.

Read the Practitioner guide + FAQs

Check the 1000+ LLM-native scaffolds.

Announcement + vision post

Thank you as usual!

With the decline of low interest rates (see Fed Chair Jerome Powell signals rate cuts aren’t imminent) and recent bemoanings of the failure of the modern data stack (see DBT's take), it'd seem that at least in the VC world data-centric startups are more risky than ones with a clearer, more direct profit motive for the time being.

Given the culture of the subreddit and the oft-mentioned adage that the best data engineers apply software engineering skills, my bet is a lot of folks here could wear multiple hats. I'm in that boat.

If some data jobs are on the way out, wearing the software engineer hat would seem the safer move.

How are you, or would you, sell yourself in this market - as a data engineer (think ETL, dashboards, MDS, etc.) or as a software engineer (REST APIs, web apps, etc.)?

Edit: to join two thoughts

TL;DR: My wife is a staff data analyst from Middle East trying to move into data engineering at tier 1 or 2 companies in the SF Bay Area. I stretch my skills a lot to get calls, but she refuses to lie and wants an honest path. How can she make the switch? Any advice from people who’ve done it? Stretching on the is good?

Hi everyone,

My wife and I live in the SF Bay Area. She grew up in Middle East and moved here after our marriage. Back there, she worked for one of the top employers, but here she’s struggling to get calls from tier 1 (FAANG) or tier 2 (just below FAANG) companies — or sometimes even tier 3.

She has a BS in Computer Science and currently a senior data analyst. She’s genuinely smart and extremely good at what she does — honestly, she’s smarter than me in most ways!

She wants to move from data analytics into data engineering, but as you all know, the required skill sets are different — pipelines, cloud, distributed systems, ETL frameworks, etc.

Here’s where we differ: I came to the US for my master’s and I’m more of a hustler. I stretch my skills a lot to match the job description — for example, I highlight skills I may not be an expert in but know enough to talk about, just to get calls and clear interviews. Then I prep like crazy to bridge any gaps. She, on the other hand, doesn’t want to do that — she feels it’s dishonest and wants to break in purely on what she’s actually done.

So my question is: what’s the best honest path for her to break into a tier 1 or tier 2 company as a data engineer? Can anyone tell her it’s good to stretch the skill sets

If anyone here has made this switch (without a US degree), we’d love to hear how you did it. Any advice, resources, or realistic tips to help her get her foot in the door in tier1 or tier2 companies with or without lying on her would mean a lot.

Thanks so much in advance!

Are people still expected to do these LeetCode style interviews? It’s 2024, we have co-pilot.. why the heck would anyone spend time grinding nonsense coding questions. As a hiring manager, if I asked someone to code something live I fully expect, and hope, they’d explain the concept and then tell me they’d run it thru some AI coding. I don’t want someone wasting their time and my money.

Edit - this is not to say someone shouldn’t understand everything they’re doing. I simply see no value in making someone code in a google doc off the top of their brain.. it’s like asking someone to do calculations without a calculator. Anyone who tries is wasting time.. using the tools available is far more valuable to me than someone who can grind nonsense coding questions. Anyone here who codes knows that most of your time is spent googling and bashing into errors to fix what you need. Why would I hire someone that doesn’t know how to do that?

I'm building a data warehouse for a startup and I've gotten source data into a Snowflake bronze layer, flattened JSONs, orchestrated a nightly build cycle.

I'm ready to start building the dim/fact tables. Based on what I've researched online, dbt is the industry standard tool to do this with. However management (which doesn't get DE) is wary of spending money on another license, so I'm planning to go with dbt-core.

The problem I'm running into: there don't appear to be any docs. The dbt website reads like a giant ad for their cloud tools and the new dbt-fusion, but I just want to understand how to get started with core. They offer a bunch of paid tutorials, which again seem focused on their cloud offering. I don't see anything on there that teaches dbt-core beyond how to install it. And when I asked ChatGPT to help me find the docs, it sent me a bunch of broken links.

In short: is there a good free resource to read up on how to get started with dbt-core?

Tasked with leading a BI/BW team for a mid sized organization. Plenty of $$$, moderate amount of data. Many data sources (100+). I am not an experienced data engineer, but I did stay at a Holiday Inn Express last night.

Doing my research I have read up on Kimball and dimensional modeling and data normalization. Sounds fucking painful.

Me and my team are Pretty good with Power BI. Fucking love the Transform data within Power BI. Using Azure Synapse/ Data Lake for central repository.

The dimensional modeling and normalization seems to be a massive lift. Feels like we could spend years working on that - meanwhile the rest of the organization is wondering what the fuck we are doing?

I can take two dimensional tables from various sources and just model them in PowerBI. Do it 1 project at a time. Can turn a project around in just a few weeks. People can see we are producing some dashboards in the near future.

With MPP processing power and columnar data bases - am I crazy to say fuck Kimball and fuck OLAP cubes? I just want to store denormalized tables in the Datalake and model them in PowerBi as needed for individual projects.

Part of me thinks this might be short sighted, but I don’t feel like I have years available to work through fucking Kimball methods for 100+ data sources.

Feedback / suggestions are appreciated.

Preamble here

Hi there!

Orchestration has been coming up in a lot of conversations lately, mostly because everyone's trying to figure out how to actually get AI workloads into production without it turning into a mess.

Airflow is one of the most significant open source projects (80k+ organizations use it), and it's also been about a year since Airflow 3 landed, which was a pretty big deal for the project. Some of the stuff we've been excited about: Dag versioning, human-in-the-loop, event-driven scheduling, the UI refresh, and backfills.

As an introduction, we are:

• Marc u/marclamberti (Educational Content Lead)

• Pete u/petedejoy (CEO)

• Carter u/CarterAtAstronomer (EVP, R&D)

• Julian u/julian-astronomer (CTO)

• Tamara u/TJanif (Senior Developer Advocate) (proof)

• Kaxil u/kaxil_naik (Sr. Director of Engineering) (proof)

Here are some questions you might have for us:

• Can you share more about Otto, your new data engineering agent for Airflow?

• What do the open source Airflow plans and roadmap look like?

• What kind of internal AI projects are you working on?

• How the heck did you come up with the name Astronomer? Do you have astronomy nerds on staff or something?

• I’ve got some feedback on Astro and/or Airflow. How do I make a suggestion?

I am pretty weak at programming. But have proficiency in SQL and PL/SQL. Can i pursue DE as a career?

Why is the standard for data engineering to use python? all of our orchestration tools are python, libraries are python, even dbt and frontend stuff are python.

why would we not use lower level languages like C or Rust? especially when it comes to orchestration tools which need to be precise on execution. or dataframe tools which need to be as memory efficient as possible (thank you duckdb and polars for making waves here).

it seems almost counterintuitive python became the standard. i imagine its because theres so much overlap with data science and machine learning so the conversion was easier?

edit: every response is just parroting the same thing that python is easy for noobs to pick up and understand. this doesnt really explain why our orchestrations tools and everything else need to use python. a good example here would be neovim, which is written in C but then easily extended via lua so people can rapidly iterate on it. why not have airflow written in c or rust and have dags written python for easy development? everyone seems to take this argumentative when i combat the idea that a lot of DE tools are unnecessarily written in python.

The reason I don't like SQL over something like Pandas is because in Pandas/Python I can break down the complex transformations in small simpler components and they are easy to interact and debug with. In SQL my only option is to write stored procedures and yes in that I can also do the same but I think the ecosystem of interacting and debugging with data step by step is not as good as python. And it is something that we should have. Recently I was exploring snowpark and I felt it is a step in a right direction. It allows to me to write my query in multiple python statements and then it will transpile the whole thing into an SQL query.

Generally in my mind I assume the tables/dataframes as 2 dimensional structures and Pandas API give much more intuitive and wide variety of constructs to explore that structure rather than SQL. What do you guys think?

I am completely new to this. I just switched from mathematics to data engineering and had my first job. I am wondering whether the job market of this particular profession is tough or not? The US and Europe are both of interest to me.

I have been using Boto3 to orchestrate data pipelines, interacting with s3, glue, redshift etc. Have absolutely loved how robust it is. I was thinking of expanding/depending my knowledge of boto3 however it seems as though the industry has other priorities. Is boto3 going to be relevant 2023 and onwards or should I move on to other tools. If so, which ones would you recommend?

Some principles I have been thinking about productive modern data engineering culture, sharing this here to see different perspectives about my outlook.

First, I want to begin by making an assertion that in this AI age, code production is a very cheap commodity. The expensive part is in reviewing & testing the code. But, as long as the pipelines are batch, the processing is not in a regulated environment, and the output is not directly affecting the core business, cost of mistakes are REALLY low. In most cases you can simply rerun the pipeline and replace the bad data, and if you design the pipeline well, processing cost should be very low.

So, here are my principles:

• ⁠Unit tests and component-specific tests are worthless. It slows down development, and it doesn’t really check the true output (product of complex interactions of functions and input data). It adds friction when expanding/optimizing the pipeline. It’s better to do WAP (Write-Audit-Publish) patterns to catch issues in production and block the pipeline if the output is not within expectations rather than trying to catch them locally with tests. (edit: write your e2e tests, DQ checks, and schema contracts. Unit test coverage shouldn’t give you any excuse to not have the other three, and if having the other three nullifies the value of unit tests, then the unit tests are worthless)

• ⁠Dependencies has to be explicit. If table A is dependent on table B, this dependency has to be explicitly defined in orchestration layer to ensure that issue in table A blocks the pipeline and doesn’t propagate to table B. It might be alluring to separate the DAGs to avoid alerts or other human conveniences, but it’s not a reliable design.

• ⁠With defensive pipelines (comprehensive data quality check suites, defensive DAGs, etc), teams can churn out codes faster and ship features faster rather than wasting time adjusting unit tests/waiting for human reviews. Really, nowadays you can build something in 1 hour and wait 2-3 days for review.

• ⁠the biggest bottleneck in data engineering is not the labor of producing code, but the frictions of design/convention disagreements, arguments in code reviews, bad data modeling, and inefficient use of tables/pipelines. This phenomenon is inevitable when you have a big team, hence I argue in most cases, it’s more sensible to have a very lean data engineering team. I would even go further to the point that it makes more sense to have a single REALLY GOOD data engineer (that can communicate well with business, solid data modeling skills, deep technical expertise to design efficient storage/compute, etc) rather than hiring 5 “okay” data engineers. Even if this really good one costs 5x than the average one, it’s more worth the money: allowing faster shipping volume and better ROI.

So, what do you think? Are these principles BS?

I have 10 years of experience in web, JavaScript, Python, and some Go. I recently learned my new roll will require me to implement and maintain ETLs. I understand what the acronym means, but what I don’t know is HOW it’s done, or if there are specific best practices, workflows, frameworks etc. can someone point me at resources so I can get a crash course on doing it correctly?

Assume it’s from 1 db to another like Postgres and sql server.

I’m really not sure where to start here.

Databricks is very popular right now (along with snowflake and big query)

It’s more popular as Azure databricks as compared to aws databricks.

I feel like fabric is going to be popular soon and in demand. I am getting consultant roles asking for fabric like Capgemini, Avanade etc

Given Microsoft gives lot of discount on using its ecosystem like teams, outlook, Microsoft tools etc i think it will given heavy discount to databricks users for using fabric.

This is my thought and speculation only - prove me wrong. (Please don’t mention fabric has lot of issues because MS will still sell you out)

DoubleDelta on latitude/longitude and timestamp columns is black magic

https://monimiller.com/blog/should-my-data-engineering-title-be-revoked-if-i-dont-know-python

“We need a data lake!”
“Let’s switch to a lakehouse!”
“Our warehouse can’t scale anymore.”

Fine. But what do any of those words mean, and when do they actually make sense?

This week in Cloud Warehouse Weekly, I talked clearly about:

What each one really is,
Where each works best

Here’s the post

https://open.substack.com/pub/cloudwarehouseweekly/p/cloud-warehouse-weekly-5-data-warehouses

What’s your team using today, and is it working?

As we’re starting to use DBT more, and are using it for codifying our business rules, I’m wondering if anyone’s had success training GPT on a DBT project. My initial attempts were lack luster. GPT seemed to know a fair amount about DBT, and of SQL, but struggled to be of much use when I trained it on a DBT project (zipped, uploaded to OpenAI GPT4). My next step is to create a prep doc with some basic support/prompt engineering. For example, our mart “Legal First Name” and “Preferred First Name” fields come from different source tables, have different staging models. I asked GPT to tell me the difference between the two, as well as derive what SQL each would be off of source tables. Another approach I’m going to try is to do a DBT run materializing as much as I can ephemerally, and see if that helps get useful responses. If this marriage of well-designed DBT project/models and GPT/LLM can be cracked, I think (at least our org) will enter a new level of accessible data literacy.