r/dataengineer • • Jun 02 '25 Bravo

General Please Stop Using AI During Interviews

266 Upvotes

My team has interviewed 45 candidates in the last several weeks, and at least half of them have been just reading AI prompt output to respond to interview questions. You're not slick. It's obvious when you're reading from a prompt. It sounds canned, no human beings talk like that. It's a clear tell when you're waffling/repeating the question; you're stalling waiting for the prompt to generate a reply.

Please just stop. You're wasting my time, my team's time, and your time.

Others in the field, how have you combatted this when interviewing prospective members for your team?


r/dataengineer • • Jun 27 '26

Datasets for data engineering projects

23 Upvotes

I want to find datasets for a data engineering project where i work with pyspark and sql in databricks. I want a dataset that challenges my data modelling skills and my pipeline creation skills. I tried kaggle, but i only keep getting a single csv file as a dataset. is there a dataset that has multiple csv files as data sources or something? i want to be able to perform all the data architecture creation by myself... Recommend any datasets that you know as well!


r/dataengineer • • Sep 04 '25

Data Engineering Academy - reverse engineering because i wont spend 20K

14 Upvotes

I came across these guys on TikTok called Data Engineering Academy and decided to hop on a call with them. Honestly, it felt like a high-pressure sales pitch, which was a red flag for me. They kept repeating that $20K in debt “isn’t that much” compared to the return on investment. In the back of my head, I was thinking: if you’re that confident in my success, why not let me pay once I land the job you’re promising? My gut told me to bail, so I ended the call and probably won’t take another.

That’s why I’m here. I got a copy of their curriculum, and when you break it down, all the topics they teach are already out there for free. Since I’m on paternity leave for the next 70 days, I had ChatGPT put together a study plan where I put in 2–3 hours each night. The plan actually looks pretty solid.

But I’d like to hear from people who’ve been through programs like this (or even that one specifically). What are the key skills I should focus on? What kinds of projects are “must-haves” for building a strong portfolio? I want to cover the same ground without dropping 20K.

Any advice would be hugely appreciated.


r/dataengineer • • Jun 24 '26

Discussion Data Engineering series

12 Upvotes

Started practicing Data engineering using this repository link https://github.com/danielbeach/data-engineering-practice/tree/main/Exercises/Exercise-1. I have completed the described exercise one requirements and now decided to extend it and create Divvy Rides ETL Pipeline designed to answer concrete business questions that map
directly to decisions that operations, marketing, and infrastructure teams at a
bike-share company would make. Looking forward to post final solutions for reviews and advice


r/dataengineer • • Jul 16 '25

Discussion My First Self-Driven SQL Data Warehouse Project – Would Love Your Honest Feedback!

12 Upvotes

Hey everyone!

I just completed my first self-driven SQL data warehouse project, and I’d really appreciate your honest feedback. I'm currently learning data engineering and trying to build a solid portfolio.

🔗 GitHub Repo:
👉 Retail Data Warehouse (SQL Server + Power BI)


r/dataengineer • • Jun 06 '25

Discussion Review for Data Engineering Academy - Disappointing

11 Upvotes

Took a bronze plan for DEAcademy, and sharing my experience.

Pros

  • Few quality coaches, who help you clear your doubts and concepts. Can schedule 1:1 with the coaches.

  • Group sessions to cover common Data Engineering related concepts.

Cons

  • They have multiple courses related to DE, but the bronze plan does not have access to it. This is not mentioned anywhere in the contract, and you get to know only after joining and paying the amount. When I asked why can’t I access and why is this not menioned in the contract, their response was, it is written in the contract what we offer, which is misleading. In the initial calls before joining, they emphasized more on these courses as an highlight.

  • Had to ping multiple times to get a basic review on CV.

  • 1:1 session can only be scheduled twice with a coach. There are many students enrolled now, and very few coaches are available. Sometimes, the availability of the coaches is more than 2 weeks away.

  • Coaches and their teams response time is quite slow. Sometimes the coaches don’t even respond. Only 1:1 was a good experience.

  • Sometimes the group sessions gets cancelled with no prior information, and they provide no platform to check if the session will begin or not.

  • Job application process and their follow ups are below average. They did not follow the job location preference and where just randomly appling to any DE role irrespective of which level you belong to.

  • For the job applications, they initially showed a list of referrals supported, but were not using that during the application process. Had to intervene multiple times, and then only a few of those companies from the referral list were used.

  • Had to start applying on my own, as their job search process was not that reliable.

———————————————————————— Overall, except the 1:1 with the coaches, I felt there was no benefit. They take a hughe amount, instead taking multiple online DE courses would have been a better option.


r/dataengineer • • Aug 11 '26

Help GCP made my life hell

10 Upvotes

I live in Hyderabad, India.

I have 1.5 years of professional experience as a GCP data engineer.

Our company had a layoff after which I started looking for other opportunities but to my surprise nobody requires GCP data engineer with less than 4 years of experience.

Meanwhile other platforms like azure, AWS, databricks even snowflake has plenty of opportunities for 1~ year of experience.

I tried to apply to them too quoting equivalent experience but no results.

And add salt to the wound we never used pyspark which is one of the biggest requirements out there.

I am looking for fresher level opportunities where the platform requirment is not a must but it's a very thin spectrum.

I have literally worked on an airflow dag which goes through a beautiful hot to cold storage flow with observation, logging, maintainance etc.

Worked on dataform , wrote scd type 2 scripts.

Worked on views, stored procedures.

I have plenty experience in data warehousing but it all looks simply useless now.

Had I survived for few years maybe I wouldn't be going through this.

I am literally applying for any state or city to grab an opportunity


r/dataengineer • • May 14 '26

Discussion Need some serious help

Post image
10 Upvotes

What is wrong with my resume? I have applied for 200+ job positions from roles data engineer to data analyst. Not a single response back.
Please help


r/dataengineer • • Oct 01 '25

Advice for switching- DE

10 Upvotes

So, I do not have a tech background, but I am from an IIT college, and I ended up working in an MNC. But it is a very specific industry, the mining industry. And I am working here as a data engineer, but they work on a legacy system. So, not much of the advanced tech is used. Like, we only work sometimes on SQL, PL-SQL, stuff like that. Python is also very rarely used, with no cloud technology, because clients do not want to go on cloud. So, my skills could not be very well developed.

And since it's an MNC, there is a lot of work. So, if I want to switch currently with 2 plus years of experience, what should I be starting with? So, my first guess is Python. So, what is the best way from where and best resource that I should start learning?

Can you please, tag some resources that will actually help me to switch. Because I want to learn Python for switching, and also to have a very good understanding. So, for the data engineer role, if someone can suggest. And also, what are the other skills that I need to work upon, so that in the coming 6 months, I will end up very, I want to switch and end up with a job.

Thanks!


r/dataengineer • • Feb 13 '26

Help Tearing apart my resume before recruiters do

Post image
9 Upvotes

Hello fellow engineers,

I am a data engineer with around 4 years of experience and preparing for a switch. I would really appreciate your feedback on my resume. Also, I tried to check ATS score and saw that different websites are giving different scores..not sure if my resume really passes these scans. What are some websites you have used?

Looking forward to brutally honest feedbacks here. Thanks in advance!


r/dataengineer • • Oct 25 '24

This is something I have been missing!!

Post image
9 Upvotes

I have been trying my hands on llms for quite sometime and came across one of the best resources available out there the "LLM Engineer's Handbook", what intrigued me the most was the attention to detail that the authors provides here from fundamentals to deploying the most advanced applications using llmops best practices.

What I liked the most about this book is the way the book reads through its course and explains all the Fundamental concepts using a practical example project throughout the book. I believe this is the best resource out there to dwell into as no book out there has the these kinds of descriptive theoretical flow as mentioned above.

Ps: Not sponsored by Packt


r/dataengineer • • Jul 29 '26

Help 10+ years as a solo Data/Infra generalist in agtech — how should I position myself for the job market?

Post image
8 Upvotes

Hi everyone,

Quick context on my background:

  • 10+ years at a small agricultural-sector company, as the only IT/data person

  • Started in Data Analysis (background in biology), grew into full-stack data ownership

  • Current stack: Python, FastAPI, PostgreSQL, MongoDB, Airflow, GitLab CI/CD, all on an on-premise RHEL8 server

  • Recently completed an intensive Data Engineering certification to formalize my skills

  • No production experience with cloud platforms, Kafka, or Kubernetes — only training-level exposure (certifications passed, not deployed in production)

  • No experience with some common tools like Snowflake, Databricks, or Terraform

My situation:
I'm looking for a new job — I feel like I've hit a ceiling where I am, and I'd thrive best in a small company needing versatility, though I'm open to other setups. So far I've had very few callbacks searching with tech keywords ("python", "fastapi") rather than "data engineer".

My main question:
Given this profile — heavy on infra/backend/data ownership, light on cloud-native tools — what job title would you use to describe someone like me? Data Platform Engineer? Backend Engineer? Something else? I've gotten conflicting suggestions and none of them quite fit.

Secondary questions, if you have thoughts:

  1. Does it make sense to shift from applying to active postings toward networking / volunteering (e.g. Data For Good) to build experience and connections instead?

  2. Any glaring red flags on how I'm presenting this that might explain the low response rate?

Thanks in advance for any input!


r/dataengineer • • Jul 28 '26

Looking for a Complete Data Engineering Roadmap (2026) – End-to-End Resources, Learning Order & Tips

8 Upvotes

Hi everyone,

I'm a beginner and I want to become a Data Engineer. There are so many roadmaps, courses, and YouTube channels that I'm feeling overwhelmed and confused about what to follow.

I'm looking for a complete end-to-end roadmap that reflects what companies actually expect from freshers in 2026.

I'd really appreciate your guidance on the following:

  1. What is the correct learning order from beginner to job-ready?

  2. Which topics are must-learn and which can be skipped initially?

  3. What are the best free and paid resources for each topic?

  4. Which YouTube channels, courses, books, or documentation do you recommend?

  5. How much depth should I learn for each technology before moving to the next?

  6. At what stage should I start building projects?

  7. Which projects helped you land your first Data Engineering job?

  8. What mistakes do beginners commonly make, and how can I avoid them?

  9. If you were starting from scratch today, what roadmap would you personally follow?

The stack I'm considering includes:

Python

SQL

Linux

Git

PostgreSQL

Spark / PySpark

Airflow

Docker

AWS

Snowflake / Redshift

dbt

Kafka

Delta Lake

Great Expectations

MongoDB

If you have a roadmap that worked for you or resources that you genuinely found useful, I'd be grateful if you could share them.

Thanks in advance! 🙏


r/dataengineer • • May 23 '26

Help Suggest good platform to learn pyspark

8 Upvotes

Hi,

I want to change my domain from vb.net to data engineer. Can anyone suggest where I need to learn and which platform is good.

I have 4+ years of experience.


r/dataengineer • • Jul 14 '25

Review my resume - Aspiring DE

Post image
8 Upvotes

I am working as a software engineer (data related) for 1 yr. I don't have much experience on spark, airflow, EMR since I am a beginner, hope will get some in the future. Attached my resume, kindly provide your suggestion. I am desperate to get a data engineer role for career growth, also my college days dream. I am currently upskilling since I am not having any hands-on experience on PySpark like big data tools, also suggest any projects and certifications that will be helpful.

Thank you.


r/dataengineer • • May 10 '25

Data Engineer | Open to Opportunities | Recently Laid Off

8 Upvotes

Hey everyone,

I’m Kshitij Patil, a data professional with a strong background in data engineering, analytics automation, and ETL pipeline development. I was recently laid off and am now actively seeking new opportunities in the data engineering space to continue growing my career.

Over the past 2+ years, I’ve:

  • Built scalable data pipelines using Apache Airflow, PySpark, and Pandas.

  • Streamlined complex MIS systems for large-scale reporting (522+ clients).

  • Automated workflows using AWS services (Glue, Lambda, Athena).

  • Worked on real-time analytics and reduced manual data ops by 50–80%.

  • Created unified data platforms and dashboards using SQL, Mixpanel, and Redash.

I’m passionate about making data accessible, reliable, and impactful. Open to remote or on-site roles in data engineering or analytics engineering.

LinkedIn: https://www.linkedin.com/in/kshitij-patil-1512aaa174/
GitHub: https://github.com/kshi-glitch

If you know of any openings, referrals, or contract gigs — I’d be extremely grateful. Feel free to DM me!

Thanks for the support!


r/dataengineer • • Sep 23 '26

The data engineering problem hiding inside every AI agent project: nobody's applied pipeline discipline to agent writes yet

7 Upvotes

Most data engineers I talk to have already solved this problem for their pipelines. Idempotent writes, schema validation, lineage tracking, retry logic that doesn't create duplicates. Then the same team builds an AI agent and none of that discipline carries over. The agent writes directly into production with no idempotency key, no lineage on what it touched or why, and no reconciliation step if its internal state drifts from the actual data.

It's the same class of problem we've already solved for ETL and streaming pipelines, just showing up in a new place because agents write more often and with less human review than a scheduled job does.

There's a workshop on Sept 26 built around exactly this, treating agent write paths, state management, and provenance the way a data engineer would treat any other production pipeline. Run by Sandipan Bhaumik, a Data & AI Technical Lead at Databricks.

Details here if it's relevant to anyone else here dealing with this handoff


r/dataengineer • • Aug 08 '26

Discussion Is there something still left in the data and platform layer with real depth behind it?

7 Upvotes

This is just an example, but Apache Spark expertise used to be a real differentiator 10 years ago or so. If you knew it well, you were swimming in job offers.

Then Databricks simplified it and made Spark silly-proof with built-in optimizations left and right. Now the exact same thing is happening to Databricks consultants and FTEs because Databricks know-how just isn't as premium as before anymore as they simplified the platform and genie-d the S out of it.

Abstraction eventually seems to eat all expertise and each cycle seems shorter than the last.

Is there anything left in the data and platform layer with real depth behind it?

And I'm not looking for "just learn AI, bro.


r/dataengineer • • Apr 10 '26

4.5 years of gap in IT

7 Upvotes

Hi everyone,

I’ve been on a career break for the past 4.5 years to take care of my kids, and I’m now looking to return to work. I have a background in testing and Python, and recently I’ve been upskilling in PySpark, Databricks, and a bit of ADF. I’ve also just started exploring Generative AI.

I wanted to understand if it’s possible to re-enter the industry after this gap, and if so, could you please recommend any good project-based courses that focus on the latest industry tech stack?


r/dataengineer • • Apr 09 '26

Infosys Snowflake Data Engineer L1 Done – What to Expect in L2 (F2F Round)?

7 Upvotes

Hi everyone,

I recently completed my L1 (technical) interview for a Snowflake Data Engineer role at Infosys, and I have my L2 round coming up next week.

I wanted to understand what kind of questions I can expect in the next round.

In L1, most of the discussion was focused on Snowflake fundamentals and practical concepts. I was asked:

  • How I receive and ingest source data into Snowflake

  • Different types of tables in Snowflake

  • Tasks and their usage

  • Types of SCD (Slowly Changing Dimensions)

  • General architecture-related questions

The round was more concept-driven rather than coding-heavy, and there were no questions on dbt or other tools.

For those who have attended Infosys or similar Snowflake interviews:

  • How deep does the L2 round go compared to this?

  • Is it more project discussion or scenario-based problem solving?

  • Should I expect more hands-on SQL/coding in L2?

  • Any specific Snowflake topics I should focus on?

Would really appreciate your insights. Thanks!


r/dataengineer • • Apr 04 '26

Help Transitioning from IoT to Finance DE (Databricks): How to handle the shift toward "Audit-Ready" pipelines?

7 Upvotes

Hello everyone,

I’ve spent the last 2 years working as a Data Engineer in the IoT space (high-frequency streaming, sensor data, etc.). Starting this fiscal year, I’m moving into a Finance Data Engineering role.

The primary goal is building a Databricks-based Datalake from scratch. The stakes are much higher than my previous role: the focus is on audit-ready pipelines, strict data lineage, and financial compliance.

The Challenge: I have zero background in finance. I’m currently "alphabet souping" my way through acronyms like GL (General Ledger) and LC (Letter of Credit), but I’m finding the domain knowledge gap a bit daunting in meetings.

My Questions for the Community:

Technical: For those using Databricks for finance, what are your "must-haves" for auditability? (e.g., Unity Catalog for lineage, Delta Lake versioning strategies, or specific testing frameworks?)

Domain: Which finance concepts are non-negotiable for a DE to understand? I’m struggling with the jargon—are there specific "Finance for Engineers" resources you recommend?

Process: What are the common pitfalls when moving from "noisy" data (IoT) to "precise" data (Finance) where reconciliation is king?

I’d love to hear from anyone who has made a similar jump or works in FinTech/Banking. Thanks!


r/dataengineer • • Feb 23 '26

Resume - Feedback needed

Post image
7 Upvotes

r/dataengineer • • Jan 31 '26

Trying to switch to Data Engineering – can’t find a clear roadmap

7 Upvotes

I’m currently working in an operations role at a MNC and trying to move into Data Engineering through self-study.

I’ve got a Bachelor’s in Computer Science, but my current job isn’t data-related, so I’m kind of starting from the outside. The biggest problem I’m facing is that I can’t find a clear learning roadmap.

Everywhere I look:

One roadmap jumps straight to Spark and Big Data

Another assumes years of backend experience

Some feel outdated or all over the place

I’m trying to figure out things like:

What should I actually learn first?

How strong do SQL, Python, and databases need to be before moving on?

When does cloud (AWS/GCP/Azure) come in?

What kind of projects really help for entry-level DE roles?

Not looking for shortcuts or “learn DE in 90 days” stuff. Just want a sane, realistic path that works for self-study and career switching.

If you’ve made a similar switch or work as a data engineer, I’d really appreciate any advice, roadmaps, or resources that worked for you.

Thanks!