r/dataengineeringjobs • • Oct 10 '26 Bravo

Blog I’ve mentored a lot of people trying to break into Data Engineering. Here’s the mistake I see most often.

349 Upvotes

I’ve spent the last few years working in data and mentoring people who are trying to get their first Data Analyst / Data Engineer job.

And honestly, I think a lot of people are preparing the wrong way.

I regularly see people learning:

SQL → Python → PySpark → Kafka → Airflow → AWS → Databricks → Snowflake → Kubernetes...

…and after 6–12 months, they still don't feel ready for an interview.

The problem usually isn't lack of effort.

It's that they're learning tools instead of learning how data systems actually work.

For example, someone may know how to write a Spark transformation but struggle when I ask:

Why did this Spark job suddenly become 5x slower?

What actually happens when you join two 500 GB datasets?

When should you broadcast a table?

What happens when one partition becomes much larger than the others?

Why did an Airflow pipeline fail even though the code itself is correct?

How would you design a pipeline that can safely process the same data twice?

What would you monitor in production?

What happens when a downstream system is unavailable?

These are the questions that separate “I completed a Data Engineering course” from “I can actually work as a Data Engineer.”

If you're learning Data Engineering right now, my advice would be:

Don't try to memorize 20 tools.

Understand these fundamentals deeply:

Data → Storage → Compute → Transformation → Orchestration → Reliability → Observability

Once you understand those, learning another tool becomes much easier.

I'm going to start spending some time here sharing the practical things I've learned from working with data systems and mentoring people — including things I wish someone had explained to me when I started.

No course pitch. No sales link.

If you're currently preparing for Data Engineering, tell me where you're stuck.

I'll try to help where I can.


r/dataengineeringjobs • • May 05 '26

Career Built My First AWS End-to-End Data Engineering Project

Post image
316 Upvotes

Hello guys,

I took a small CSV dataset from Kaggle and uploaded it to Amazon S3 as the raw data source. From there, I started building an ELT workflow to practice real data engineering concepts.

I organized the data using a medallion architecture approach:

  • Bronze layer: raw data stored directly in S3

  • Silver layer: cleaned and transformed data using Athena SQL

  • Gold layer: aggregated business insights such as revenue by category, region, and top customers

I used AWS Athena to query and transform the data without moving it out of S3.

To automate schema detection, I used AWS Glue Crawler, which scanned the S3 bucket and created catalog tables for easier querying.

I also explored compute concepts by launching an EC2 instance, installing Docker, and running a PostgreSQL container to understand containerization and data persistence using volumes. Claude gave me the idea.


r/dataengineeringjobs • • 20d ago Bravo Heartwarming

If I had to become a Production-Ready Data Engineer from scratch, this is the exact roadmap I would follow.

299 Upvotes

I’ve seen too many aspiring Data Engineers spend months learning random tools, collecting certificates, and building tutorial projects, only to struggle when asked to design or debug an actual production pipeline.
So here’s the roadmap I’d follow if I already knew Python and SQL and wanted to become a Data Engineer who could actually own pipelines in production.

Phase 1: Master the fundamentals before chasing tools.
I’d start with advanced SQL: window functions, CTEs, query execution plans, deduplication, incremental loads, MERGE operations, and query optimization. Alongside that, I’d learn data modeling, dimensional modeling, fact and dimension tables, SCD Type 1 and Type 2, and how to define the grain of a dataset.

Phase 2: Learn to write Python that survives production.
Not just scripts that work on my laptop. I’d learn modular code, exception handling, structured logging, configuration management, API pagination, rate limits, unit testing with pytest, mocking, type hints, and Git workflows. Every pipeline component should be testable, reusable, and understandable by another engineer.

Phase 3: Understand how reliable data pipelines actually work.
I’d build ETL/ELT pipelines involving APIs, databases, and files. Then I’d learn full loads versus incremental loads, CDC, watermarks, idempotency, deduplication, schema drift, late-arriving data, data reconciliation, and backfills.
One concept I’d pay special attention to is idempotency: rerunning a failed job should not corrupt the data or create duplicates. This is the kind of detail that separates a working demo from a reliable pipeline.

Phase 4: Learn orchestration properly.
I’d pick Apache Airflow and learn DAG design, scheduling, task dependencies, retries, timeouts, sensors, backfills, catchup, and failure handling. I’d also understand why tasks should be designed to recover safely instead of assuming every run will succeed.

Phase 5: Go deep into one cloud platform.
I’d choose AWS, Azure, or GCP rather than trying to learn all three simultaneously.
I’d learn object storage, IAM and service identities, networking basics, secrets management, compute, cloud monitoring, and a warehouse such as Redshift, Synapse, or BigQuery.

The objective wouldn’t be to memorize service names. It would be to understand how data moves securely and reliably through the platform.

Phase 6: Learn Spark and lakehouse engineering.
I’d focus on PySpark DataFrames, lazy evaluation, partitions, shuffles, join strategies, data skew, caching, and execution plans. Then I’d learn Delta Lake, ACID transactions, schema enforcement, schema evolution, time travel, compaction, and medallion architecture.

I’d also learn when Spark is unnecessary. Distributed computing is useful, but using a cluster for a tiny dataset is an expensive way to demonstrate enthusiasm.

Phase 7: Learn the part most roadmaps ignore: production operations.
I’d learn Docker, CI/CD, environment-specific configuration, infrastructure as code, automated testing, data quality checks, freshness SLAs, monitoring, alerting, incident response, rollback strategies, and disaster recovery.
I’d deliberately break my own pipelines: introduce duplicate records, change a source schema, interrupt a job, send malformed data, and simulate a failed deployment. Then I’d practice detecting, diagnosing, and recovering from each failure.

Phase 8: Build one project that proves everything.
I’d build an end-to-end data platform that ingests data from an API, lands raw files in cloud storage, transforms them with PySpark, stores curated data in Delta Lake, loads analytical tables into a warehouse, and orchestrates everything with Airflow.
I’d add automated tests, data quality checks, CI/CD, logging, monitoring, documentation, and a recovery procedure.
Most importantly, I’d demonstrate that the pipeline can be rerun safely, handle schema changes, recover from failures, and reconcile source counts against destination counts.
That’s the project I’d want to discuss in an interview, because it demonstrates engineering decisions rather than simply showing that I followed a tutorial.

The advice I’d give anyone starting out
Don’t try to learn 20 technologies at once.
Learn SQL deeply. Write good Python. Understand data modeling. Build reliable pipelines. Pick one cloud. Learn Spark when your use case requires distributed processing. Learn how to deploy, monitor, debug, and recover what you build.
A production-ready Data Engineer isn’t someone who knows the most tools. It’s someone who understands what can go wrong, designs for failure, protects data integrity, and can explain the trade-offs behind their decisions.

If you can build a pipeline, deploy it, monitor it, break it, recover it, and explain every design decision, you’re moving in the right direction.

I’m sharing this because I wish more learning resources focused on the actual work rather than endless tool lists.

For anyone following this roadmap, my recommendation is simple: spend less time collecting technologies and more time building systems you can confidently own.


r/dataengineeringjobs • • Jul 16 '25

Is the market really this bad?

Post image
254 Upvotes

Disclaimer: throwaway account so this doesn't get linked with main.

About a year ago, I was let go from a role with a consulting firm where I was working in the healthcare space. I took the first job I could get, which claimed to be a data engineering position, but has turned out to not be that **at all**. I needed to wait 6 months to look for work due to getting a sign-on bonus, but I've been looking for work for a couple of months now and I'm not getting any hits. I assume it's because every job post is flooded with literally hundreds of applicants within an hour of being posted, but, if I'm being honest, I kind of expected the fact that I have experience at Facebook and with healthcare data that I'd have a leg up.

Am I just completely failing at providing relevant examples in my work experience? Am I being punished for not working in a place that uses Snowflake, dbt, and/or Databricks? Or is the market really just so bad that having that experience doesn't make me special anymore?


r/dataengineeringjobs • • Sep 21 '26

Blog Is data engineering slowly dying or am I just paranoid?

236 Upvotes

4 years in DE. Stuff that used to eat my whole afternoon (broken DAGs, pipeline boilerplate) now takes 30 seconds with an LLM.

Thought that was great at first. Then I noticed we never backfilled the two people who left, and the junior roles on our careers page quietly disappeared.

I still think someone has to know why the data is wrong and argue with stakeholders who say "real time" but mean "by Monday". AI is bad at that. But the number of hands needed feels like it's shrinking, and entry level is going first.

Or maybe this is just SQL will kill DBAs again.

Is your team growing, flat, or quietly shrinking?


r/dataengineeringjobs • • Sep 24 '26 Heartwarming

A Data engineer story

225 Upvotes

A year ago, around this same time, I was struggling in a service-based company earning around 4 LPA, even after having close to 3.8 years of experience.

I had been trying to switch for almost 1.5 years.

Almost no interview calls.

A few interviews came, but I got rejected.

At one point, I genuinely started wondering whether I was doing something wrong or whether I would ever be able to move into a better role.

Then in December, I finally received an offer from a product-based company.

The package wasn't exactly what I wanted, but I accepted it because I felt I needed that first breakthrough. I resigned and started serving my notice period.

And somehow, after that first offer, everything started changing.

By February, I had received 12+ offers/interview conversions, and some of the offers were around 6x my previous salary.

I finally joined a new company in March.

Then, just a few months later, another opportunity came my way.

I left that company in September and joined one of the world's biggest pharmaceutical companies.

So within roughly 8 months, I made two switches, and compared with where I started, my compensation became roughly 8x.

But salary is not actually the biggest thing I learned from this journey.

What actually changed for me?

I realized that although my title was related to Data Engineering, I wasn't really working on core Data Engineering work.

A lot of my experience was more on the support/backend side.

So instead of waiting for my company to give me the perfect project, I started creating my own projects.

I watched YouTube videos, studied GitHub repositories, understood existing architectures, and then tried building things myself.

That made a huge difference.

Once you understand how an end-to-end data engineering architecture works, interviews become much easier because you're no longer just memorizing answers.

You understand why things are being done.

For anyone trying to become a Data Engineer, especially someone coming from a support role, I would suggest focusing on:

SQL – Make this very strong.

PySpark/Spark – Understand transformations, joins, partitions, shuffling, optimization, etc.

Databricks / Microsoft Fabric / Snowflake – You don't need to master all three initially. Pick one and understand it properly.

dbt – Don't just watch tutorials. Build a small project.

Airflow – Again, create a project and understand how orchestration actually works.

Don't keep jumping between 20 different YouTube playlists.

Pick one good resource.

Learn the basics.

Practice.

Build something.

Then slowly move toward advanced topics.

Most importantly, master SQL and Spark. Once these two become strong, many Data Engineering concepts start becoming much easier.

My suggestion would be:

Learn SQL + PySpark + one cloud data platform + dbt + Airflow.

Build at least one proper end-to-end project using them.

Then start giving interviews.

Don't wait until you feel like you know everything. You never will.

Go for interviews, get rejected, understand where you went wrong, learn that topic, and try again.

Coming from a support background, I used to think that people working in core Data Engineering or product companies were far ahead of me.

They weren't.

They simply had knowledge and exposure that I hadn't built yet.

That can be learned.

So if you're currently stuck in a support role, earning less than you expected, getting rejected, or not receiving calls keep working on your skills and keep applying.

One good opportunity can completely change the direction of your career.

If I could make this transition from a support/backend role, you can do it too.

Good luck. Keep learning and keep going💓


r/dataengineeringjobs • • Aug 14 '26

Data Engineering Project

Post image
217 Upvotes

r/dataengineeringjobs • • Jul 29 '25

Companies looking to hire Data people have become delusional

216 Upvotes

I have latest certifications in nearly all five of the tools I regularly use or have experience with. You’d think that would count for something, but it hasn’t made the slightest difference. If certifications really opened doors and made it easy to get hired then I wouldn’t still be unemployed after nearly a year and sending out over 1,500 applications. On top of that I have 6 years of work experience in my field.

The truth is, certifications have become more of a money-making scheme for these tech companies and a way for professionals to indirectly market these tools, nothing more. Most hiring managers don’t actually care. They’re not looking for certified professionals; they’re looking for unicorns.

Every job description seems to expect one person to be a business analyst, a data engineer, and a machine learning expert all at same time, and who can take that job for as little salary as possible. And the requirements? Every single listing reads like the same tired script: "Must be proficient in Python and SQL, can write complex SQL quires, must have experience with AWS, Microsoft, Databricks, Pyspark, Scala, Knowledge of xyz domain, Airflow, Dbt, xyz and etc cloud platforms, Excellent communication and interpersonal skills, with the ability to present complex data and insights." The list goes on.

Sometimes, these expectations are so unrealistic, it’s like reading a fantasy wishlist, kind of like when someone describes their ideal partner as someone who doesn’t actually exist and when you ask why are you single then they will say "I haven't found the right one", exact same mindset got these companies when they are trying to hire.


r/dataengineeringjobs • • Feb 21 '26

Career Data engineer (20+ yrs) — sharing real-world experience and lessons learned

200 Upvotes

I’ve been working as a data engineer and database specialist for a little over 20 years.

Most of my days are not about fancy tools, but about routine work that slowly builds reliable systems:

• data modeling (mostly Kimball-style thinking)
• writing and maintaining SQL-heavy transformations and batch loads
• using Python for validation, API calls, logging, and small pieces of glue code
• spending a lot of time translating business questions into data that actually helps decisions

For me, growth never felt like a jump. It was more a slow accumulation of context — things breaking, models aging, requirements changing, and gradually learning how to handle that better.

There aren’t many shortcuts — and the scars tend to fade with time.

If you’re early or mid-career and want to talk about:

• what day-to-day data engineering work really looks like
• how to focus your learning without chasing every new tool
• how to think about data work from a business point of view

Feel free to ask in the comments. I’m happy to share what worked (and what didn’t) for me.

P.S.
Remember: sources change, events are redefined, fields disappear, and backfills happen.


r/dataengineeringjobs • • Jan 22 '26

Guys, I finally did it. Got a job within a month after being laid off on 15th December 2025

175 Upvotes

I just wanted to share this here because this community helped me more than I can explain.

I was laid off on 15th December 2025. It came out of nowhere and honestly put me in a very bad mental space. I did get some severance, which helped financially, but mentally it was still very tough. A lot of anxiety, self doubt, overthinking, and constant fear about the future.

I gave more than 15 interviews in the last one month. I failed multiple online assessments, got rejected in several first rounds, and even failed a couple of second rounds. There were days when it felt like nothing was working. This particular role had 4 rounds, and somehow I managed to clear all of them.

Today, within just one month, I finally got a Data Engineer role, even before using my severance (I got 5 month salary as severance after I was laid off ). I was honestly just hoping to get a job, but I ended up with almost a 50 percent hike. I was at 8 LPA earlier and now I’m joining at around 12.5 LPA. This isn’t my dream package or a planned switch, this was a survival move after a layoff, and I’m taking it as a big win.

The last one month of upskilling was intense (Was studying 10-12hrs daily). I failed, learned, fixed my basics, and kept going.

Looking back, this layoff phase turned out to be one of the most important phases of my life. It forced me to grow, face my gaps, and stop being comfortable.

Thank you to everyone here who shared advice, experiences, and motivation. Reading posts and comments here really helped me stay sane during this phase. If anyone is going through layoffs or struggling right now, please know that things can turn around quicker than you think.

If anyone needs help or just wants to talk, feel free to DM me. I’d be happy to help however I can.


r/dataengineeringjobs • • May 28 '25

Career 2025 layoffs are reshaping the workforce... - My Advice

163 Upvotes

Hey all, if you've seen the news recently and noticed all the layoffs, it might come as a surprise to you that tech jobs are under fire - especially in data engineering. They're harder to land interviews for, it's harder to get into big tech... It's a disaster. It's times like these that I like to give back to the community and list resources to help those in trying times.

We all know that places like LinkedIn are not working for job applications, so I've decided to list a few niche job board sites that have personally helped either a) me or b) a friend of mine find a job in big tech. Don't be application #1,000,000 - apply to jobs on niche communities to have greater success. Make sure your skills align with your job search site. Here goes my list:

• HappyTechies.com - Great if you want a Microsoft-tech job (not an official Microsoft Careers site, but good for Azure, Fabric, and Dynamics jobs with mid-size companies). High-quality resource, high-quality postings, definitely recommend.

• Python.org/jobs - Who knew? Python has a job board... very helpful if you want to work in Python exclusively and can't find any jobs for data science focused on Python.

• Rustjobs.dev/ - SUPER clean UI, made exclusively for Rust jobs. The market for Rust is always expanding, and many data engineering jobs demand Rust to increase speed and efficiency. Give this a go!

• Hirement.com - This is a good resource to find aggregated niche job board sites. You can sort and visit sites from here if none on my list work for you.

• Levels.fyi - Not used to land a job, but rather to know the salary of the job you want. Most people aren't demanding enough compensation for the tough work they're doing - know your salaries!

I hope this list helped somebody :) For anyone struggling with the job search, you got this 🔥


r/dataengineeringjobs • • Mar 12 '26

Do this to get most out of LinkedIn job search and get rid of Ghost jobs

Thumbnail
gallery
146 Upvotes

I got overwhelmed in my recent job search, finding a remote job while being a full time employee is a challenge considering the limited time we get and amount of Ghost jobs, non responsive jobs in linkedin. So I got couple of tips and challenges from many helpful peers, these are

  1. Apply past 4 hrs jobs first and so on

  2. Avoid easy apply as conversion is low, first clear non easy apply jobs

  3. Many times job titles are simple software engineer and it becomes difficult to judge whether it's backend heavy or frontend etc

  4. Ignore companies posting many similar jobs

  5. Linkedin often shows jobs which you already applied or viewed

  6. Try to apply for companies whose review time is available to linkedin and many more...

To productize these I have built a chrome extension that provides all these features, filters and search options which linkedin lacks today. I will keep adding more features if they proved to be effective. Try it today and let me know what do you think. Planning to keep it free for lifetime for first 500 users. It's fully local and private and no sign ups required. Link: https://chromewebstore.google.com/detail/glkllakoboeiaefcfcbbdgfkkmkoggga?utm_source=item-share-post


r/dataengineeringjobs • • Oct 08 '26

Looking for a Data Engineering Learning & Project Partner

135 Upvotes

I’m looking for 1–2 motivated people who want to join me in building strong skills in Data Engineering and work together toward one common goal:

🎯 Become job-ready and target Data Engineering opportunities by January 2027.

Rather than learning everything individually, I’m looking to build a small group where we can:

• Learn concepts together
• Discuss and solve technical problems
• Work on real-world/live projects whenever opportunities are available
• Build projects that we can showcase on our resumes/GitHub
• Conduct mock interviews and technical discussions
• Keep each other accountable and motivated

👨‍💻 Who I’m Looking For

It would be great if you already have some experience with:

  • Coding / Programming

  • SQL

  • Any Cloud Platform (AWS / Azure / GCP)

  • Basic understanding of software/data concepts

You don't need to be an expert. The most important things are consistency, willingness to learn, and commitment to working together.

📚 Technologies & Topics We Plan to Cover

  • Data Engineering Fundamentals

  • SQL & Python

  • Data Modeling

  • Distributed Systems

  • Data Engineering System Design

  • Databricks / Apache Spark

  • Snowflake

  • Cloud Data Architecture

  • ETL / ELT & Data Pipelines

  • Real-world Data Engineering Projects

The idea is not just to complete courses, but to learn → build → discuss → solve → prepare for interviews → get a Data Engineering job.

📅 Target: January 2027

If you're also serious about making the transition into Data Engineering and would like to collaborate, feel free to:

📩 DM me directly

or

💬 Telegram: @ abhikvk

Let's build together, stay consistent, and help each other become job-ready Data Engineers. 🚀

#DataEngineering #DataEngineer #DataScience #SQL #Python #AWS #Azure #GCP #Snowflake #Databricks #ApacheSpark #ETL #ELT #DataPipelines #DataModeling #CareerTransition #TechJobs


r/dataengineeringjobs • • Oct 16 '26

I’ve reviewed a lot of mid-level DE job searches lately. The biggest mistake isn’t skills, it’s targeting.

130 Upvotes

I spent 26 years in data engineering, including running a data org at AT&T, and these days I spend a lot of time with people trying to land mid-level DE roles. The same patterns keep showing up, so here they are in one place.

1. You’re creating two gaps at once.
If you don’t have a DE title yet and you’re also applying to an industry you’ve never worked in, you’re asking a hiring manager to bet on two unknowns. Apply where your background is rare instead. A chemist moving into data should go after pharma and life sciences first. Someone from healthcare ops should go after payers and providers. Your domain knowledge is your way in, so use it. Mass applying is dumping feathers in front of a fan when you need to be shooting arrows.

2. Your project works but you can’t explain it.
A GitHub repo with a dashboard at the end doesn’t tell me much. I want a README that covers ingestion, cleaning, transformation, architecture, the tradeoffs you made, and what you’d do next. In the interview, walk me through the business problem, the architecture, the cost-aware choices you made, where it breaks, and what you learned. If AI wrote part of the code, you need to be able to explain every line in your own words. That gets tested.

3. Your answers are too long and too deep too early.
Lead with impact, not tenure. Stay high level until they probe. Prepare three STAR stories (a win, a conflict, a failure) and you can cover most of a one-hour behavioral round. On live coding, ask the interviewer about assumptions and edge cases instead of reasoning through them silently. Sometimes you only have five minutes to code.

4. The gaps that actually matter at mid-level.
These matter more than another cert:

  • SQL beyond the basics: window functions, CTEs, self-joins, date logic

  • Real data modeling: grain, facts and dimensions, SCDs. dbt is a tool, not data modeling, and saying otherwise in an interview is a red flag.

  • Production habits: Git and PRs, testing, CI/CD, promoting code across environments

  • Visible cloud platform experience. Strong SQL and Python with no Snowflake, Databricks, or cloud on your resume gets filtered out. dbt and Airflow show up in most postings regardless of platform.

5. Your LinkedIn and resume don’t match.
Put your keywords in the experience bullets, not only the skills section. Headline, About section, and resume should all tell the same story.

6. Hiring managers are hiring for communication too.
Most teams want to know whether you can explain a pipeline to an exec and leave documentation the next engineer can actually use. I’ll take an enthusiastic, teachable engineer over a brilliant one with an ego every time.

Happy to answer questions in the comments. If you want a hiring manager’s read on your resume bullets, drop them below.


r/dataengineeringjobs • • Jul 01 '26

Career Looking for feedback on an end-to-end data engineering project

Thumbnail
gallery
116 Upvotes

Hi everyone,

I've been working as a data engineer for about a year, and over the past month I wanted to push myself by building something significantly larger than the usual tutorial project.

The result is an end-to-end analytics platform built around the Wikimedia Pageviews dataset.

The original objective was simply to improve my dbt skills, but the project gradually expanded into something much larger.

One of the biggest challenges turned out not to be the scale (roughly 2 TB of public data), but the data model itself. Wikimedia publishes pageviews at the page level, while I wanted to analyze attention at the entity level. Solving that required reconciling pages across more than 300 language editions using Wikidata before building the analytical warehouse.

The project now includes:

  • BigQuery as the analytical warehouse

  • dbt for semantic modeling and testing

  • Terraform for Infrastructure as Code

  • AWS Fargate for orchestration

  • GitHub Actions CI/CD with path-based deployments

  • CloudWatch dashboards for operational monitoring and BigQuery cost attribution

  • Optional Power BI deployment through the REST API

  • Fully deployed dbt documentation

  • End-to-end reproducible infrastructure

Everything is documented, including the modeling decisions, infrastructure architecture, dashboard methodology, and deployment process.

Repository: Github Link

Live dashboard: Dashboard Link

dbt documentation: Dbt docs link

I'm posting here because I'd genuinely appreciate feedback from other data engineers.

If this project landed in front of you during a hiring process:

  • What would stand out positively?

  • What would you improve?

  • Are there parts that feel over-engineered or under-engineered?

  • Is there anything missing that you'd expect from an end-to-end data engineering project?

I'm especially interested in feedback on the architecture, modeling choices, CI/CD pipeline, documentation, and overall project structure.

Thanks in advance!


r/dataengineeringjobs • • Aug 30 '26

Data Engineers: What do you actually do at work?

114 Upvotes

I’m trying to understand what data engineering actually looks like in a real job, rather than relying only on job descriptions.

When I look at Data Engineering roles on Naukri, LinkedIn, etc., the list of required skills seems endless — SQL, Python, ETL/ELT, Airflow, Spark, Kafka, AWS/Azure/GCP, Databricks, Snowflake, dbt, Docker, Kubernetes, CI/CD, Terraform, APIs, data warehousing, data lakes, and many more. It’s difficult to tell which of these are genuinely important and which are simply added to job descriptions.

So I’d love to hear from people who are actually working as Data Engineers. What does your day-to-day work really look like? What kind of tasks do you work on, what technologies do you use regularly, and how much of your time is spent coding, debugging, monitoring pipelines, working with databases, or attending meetings?

I’m particularly interested in understanding what skills and tools you use every day, occasionally, or barely at all, despite seeing them frequently in job postings. I’d also like to know what a typical Data Engineering project looks like from start to finish and what you would realistically expect from someone entering the field.

I’m not looking for another generic “learn SQL → Python → Spark → AWS” roadmap. I’m trying to understand the actual work behind the job title and what I should focus on learning to become job-ready.

If you’re a Data Engineer, especially with around 1–5 years of experience, I’d really appreciate hearing about your experience. If possible, share an example of a real task or project you’ve worked on (without revealing anything confidential).

Thanks!


r/dataengineeringjobs • • Aug 10 '25

The AI Scam: How Decision-Makers Are Wrecking Tech Careers and the Economy

114 Upvotes

I'm a senior level engineer with over 11 years of experience from Europe, and I still can't get hired after almost one year. We're toast, and it's not at all because of the AI itself. It's because of the recruiters and CEOs who think that AI is good enough to replace us. But don't worry, everything will come crashing down like a house of cards! Including the economy. Just give it one year more or less than you will see how this will development will lead to civil war.

Don’t tell me it’s about my résumé or “soft skills”, I know all that. The CEOs and hiring managers across these companies are the real problem. They should be fired immediately. Since the post-pandemic era, they’ve been the root cause of this toxic decline.


r/dataengineeringjobs • • May 31 '26

Book Idea: The Core Concepts Every Data Engineer Must Master

110 Upvotes

I am considering writing a book focused on the timeless foundations of data engineering.

The goal is not to teach a specific cloud platform, framework, or tool. Instead, the book would focus on the core concepts that every data engineer should understand deeply, regardless of technology trends.

Part 1: Relational Data Modeling

Understanding how data is structured and represented:

  • Entities, attributes, and relationships

  • Cardinality and relationship types

  • Primary and foreign keys

  • Data normalization

    • First Normal Form (1NF)

    • Second Normal Form (2NF)

    • Third Normal Form (3NF)

  • Practical modeling patterns and common mistakes

Part 2: Dimensional Modeling

Building analytical data models that support business decisions:

  • Choosing the business process

  • Defining the grain (the most atomic level of detail)

  • Identifying dimensions and facts

  • Surrogate keys

  • Shared and conformed dimensions

  • Star schemas and practical warehouse design

Part 3: SQL Engineering

Writing SQL that is correct, efficient, and maintainable:

  • Query design principles

  • Table access and join strategies

  • Subqueries

  • Common Table Expressions (CTEs)

  • Window functions

  • Data types and type conversions

  • NULL handling

  • EXISTS vs IN

  • Common SQL anti-patterns

Part 4: Functional Programming with Python

Applying functional programming concepts to data engineering:

  • Pure functions

  • Higher-order functions

  • Function composition

  • itertools

  • functools

  • operator

  • Building reusable and testable data transformations

Part 5: Data Quality Foundations

Practical concepts every data engineer must know:

  • Data quality dimensions

  • Data validation techniques

  • Data contracts

  • Data lineage

  • Reconciliation processes

  • Monitoring and observability

  • Real-world examples from production environments

Part 6: ETL and Pipeline Engineering

The most common challenges faced in production systems:

Basic Problems

  • Five issues every junior engineer encounters

Intermediate Problems

  • Five issues commonly faced in growing data platforms

Advanced Problems

  • Five complex production scenarios that senior engineers must solve

Each case study would include:

  • The problem

  • Root cause analysis

  • Solution approach

  • Lessons learned

Part 7: Stakeholder Communication

Technical skills alone are not enough.

Topics include:

  • Requirement gathering

  • Managing expectations

  • Communicating technical concepts to non-technical audiences

  • Writing effective documentation

  • Presenting findings and recommendations

  • Handling ambiguity and conflicting priorities

The Vision

The objective is not to reinvent the wheel.

The book should be practical, timeless, and experience-driven. It should be the kind of book that engineers revisit throughout their careers—not because it teaches a specific technology, but because it reinforces the fundamental principles that make great data engineers.

The book website: https://olive-crocodile-564190.hostingersite.com/index.html


r/dataengineeringjobs • • May 28 '26

What I'd learn for 2027

107 Upvotes

Disclaimer, i'm cofounder and data engineer at dlthub and we push what's possible. I work in the industry for 15 years so i am trying to not fall for hype or fear and navigate with clarity.

We recently took an old Airflow / dlt / dbt setup and rebuilt it as a clean, machine-generated stack with ontology based data access chat bi on top. Most of the human pipeline work disappeared. So we started asking: what's actually left for a data engineer to do in 2027 (supposing teams and companies move forward too)

Turns out more than I expected and possibly more than we did before. Here's how I'd split it:

Worth learning (newly relevant stuff):

  • Modeling theory. Logical data models, canonical models, how they map into ontologies and knowledge graphs. This is the foundation for "virtual knowledge graphs" / ontology-based data access — giving agents structured access to your data without copying it around.

  • Agentic automations. Once you have a virtual knowledge graph plus MCP, you can hand an agent data, rules, and actions. New automations open up here that weren't practical before — processing support tickets, email orders, anything that used to be a human message queue.

Worth learning (still resilient):

  • Streaming. Hard to specify in natural language, still needs real engineers.

  • Identity resolution. LLMs implementing canonical models won't catch this unless you declare it. Someone still has to go out into the world and figure out how the data maps to reality.

  • High-precision code. Anywhere language is too loose for the spec.

Worth deprioritizing:

  • Deep SQL and Python fluency for pipeline-building. LLMs handle imperative pipelines with dependencies fine now. What's left is reading a join and knowing if it's right — because language is semantic and code is precise, and precision still has to be expressed precisely.

  • Dashboarding tools. If humans don't sit in them, they fade. We're already seeing teams ask "do we need a dashboard, or do we just need one report and a chat?"

Are you seeing similar or different?


r/dataengineeringjobs • • Apr 02 '26

Interview Please help with first DE interview with a screener exercise

Thumbnail
gallery
106 Upvotes

Hi all, I got my first interview for a DE role after a 30 minute initial call with the hiring manager. Attached is the exercise. I think I have an idea about where to start and can do some research on how to go about this, but do you have any advice on how to go about this and what kind of questions they will ask me in the interview?


r/dataengineeringjobs • • Jul 26 '26

Interview After 17 years in data engineering and a lot of time on the interviewer side of the table, I wrote down the two frameworks I kept wishing candidates had

102 Upvotes

Hi everyone,

I've spent about 17 years in data engineering, the last 10 at Amazon, building large-scale data platforms and sitting on the interviewer side of a lot of loops. The pattern I kept seeing: strong engineers failing interviews they were qualified for. Not for lack of knowledge, but because they had a hundred facts and no order to deploy them in. They'd freeze, or ramble, and the interviewer couldn't follow the thread.

I ended up writing down the two "spines" I coach people to use, and turned them into books. Sharing the core of both here because it's useful even if you never buy anything:

For the system design round (CAMEOS): Clarify the scope → Assess the price (size it with real numbers) → Model the grain → Engineer the pipes → Optimize the pacer → Stress-test it (break your own design before the interviewer does) → then Guard it for production. You carry six words into the room; each unpacks into a short checklist only when you get there. The last two steps, failure modes and production operations, are where senior candidates actually separate themselves, and they're what most frameworks skip.

For the behavioral round (COMPASS): Context → Ownership → Moves → Pressure → Achievements → Shift → Sound-bite close. The two beats that senior loops actually score, and most answers skip, are Pressure (the trade-off or risk you navigated, what made it hard) and Shift (how it changed the way you lead). If your stories are all activity and no judgment, this is usually why.

--------------

The books, if you want the full treatment:

Win the Data Engineering System Design Interview - the framework phase by phase, plus 15 complete worked designs (streaming metrics, IoT lakehouse, CDC, sessionization, billing reconciliation, feature stores, GDPR deletion, fraud scoring, diagnosing a slow join, and more), each with real numbers and named failure modes. A trade-offs catalog, 30 practice prompts, and a 30-minute pre-interview revision guide. https://www.amazon.com/dp/B0H87TJ8V6

Win the Leadership Behavioral Interview - the framework across every family of question (vision, ambiguity, conflict, underperformance, hiring, retention, coaching), 25+ worked examples, in-the-room transcripts, a map from your answer to the interviewer's rubric, and a 30-day practice plan plus a 5-day crash plan. Aimed at senior DEs, tech leads, and EMs. https://www.amazon.com/dp/B0H8LDF5WF

Honest caveats: these aren't beginner books, they target senior and staff-level loops. The worked examples are drawn from data and analytics, so that's where they'll feel most native. And CAMEOS/COMPASS are teaching scaffolds I built, not industry standards; their job is to give you an order to think in under pressure.

I'd genuinely value feedback for future editions.🚀


r/dataengineeringjobs • • Jun 21 '25

Resume Review 1 year of applying and no calls.

Post image
97 Upvotes

Hey guys, I am in a desperate need of a resume review. By now I’ve applied to way too many companies but not received a single call back. Ik once I get in the interviews I can crack it or atleast work on it making myself better but reaching that stage seems like a long shot. Some advice would be really helpful as I am in a stage of life rn that I need a switch for a better pay to support my family. Ik I’m underpaid but my current company won’t match the market value.


r/dataengineeringjobs • • Sep 16 '26

Career Is Data Engineering really the best IT role right now?

96 Upvotes

I’ve heard that Data Engineering is a high-demand job with less AI impact compared to other IT roles, requires less DSA, and needs less continuous upskilling compared to AI roles. I’ve also heard that the working hours are better and the pay is still good. Is Data Engineering really that much of a chill job compared to other IT roles? I’m a 2026 graduate and currently working as an SDE. Should I consider switching to Data Engineering?


r/dataengineeringjobs • • 6d ago

Stop applying to “data engineer” jobs. Apply to data engineer jobs in one industry.

95 Upvotes

I’ve been doing data engineering for 26 years, including running a data and analytics team at AT&T. Now I coach people trying to break into mid-level DE roles. The biggest mistake I see is people treating every DE posting like it’s the same job. They’re not.

A DE role at a hospital system looks nothing like one at a fintech or a manufacturer. The tools are different, the data is different, and the problems they’re hiring someone to fix are different. When your resume and projects are generic, you end up competing with everyone. When they’re aimed at one industry, the pool you’re competing in gets a lot smaller.

Here’s what I have people do:

  1. Pick one industry you have some connection to, even a loose one. Prior job, degree, or just real interest.

  2. Pull 10 real job postings from that industry and tally every tool mentioned. You’ll usually see 4 or 5 that keep coming up. That’s your stack. Ignore the rest for now.

  3. Look up 3 problems companies in that industry are dealing with right now. Regulatory reporting, claims data quality, supply chain visibility, whatever it is. You can find these in earnings calls, industry news, and the postings themselves.

  4. Build your projects around those problems with that stack. Not a Netflix recommendation engine. A claims data pipeline that handles late-arriving records, for example.

  5. In interviews, talk about the business problem first and the tools second.

Most hiring managers never open your GitHub. They do notice when you understand their world better than the other candidates do.

Full disclosure: this is the method I teach in my group cohort, From Stuck to Hired. We meet live every Wednesday at 7pm ET and work through it together over the program. The launch price ends Nov 11. Details are here if it’s useful to you: https://learn.gambilldata.com/offers/V7JNsoTG

Either way, try the 10-postings exercise this week. It takes about an hour and it’ll change how you look at your job search.