r/ETL • • Jun 30 '26

I've used 6 million claude code tokens in 3 months

3 Upvotes

I burned through 6 million tokens on Claude Code last quarter. It started as a productivity boost, I was shipping faster than ever, crushing dbt models and ingestion scripts in hours instead of days.

Then Claude went down for about half an hour during a pretty routine ETL fix. I just sat there staring at the terminal. Couldn't remember how to structure a simple window function without it suggesting the whole thing. That's when it hit me: I've outsourced my working memory to an autocomplete tool.

Coding assistants are great for finishing your sentences, but they don't manage the full pipeline. They don't understand your schema drift, your monitoring, or your deployment steps. You still have to hold all that context yourself and stitch the pieces together. That's the real bottleneck, not writing the code but knowing what to write and where it fits.

I've been looking at platforms that try to automate more of the end-to-end data engineering workflow, Databricks, AI workflows, Genesis Data Agents, and Snowflake Cortex AI, but they all introduce their own failure modes. Debugging a multi-step agent decision is even harder than debugging a bad SQL query.

The tool doesn't fix the fundamental need to understand your data and your architecture. I'm now forcing myself to do one raw coding session per week without any AI help, just to keep the muscle memory alive. Feels like studying for a test I already passed, but the alternative is being useless when the API goes down.


r/ETL • • Nov 21 '22

A meme about ETL

1 Upvotes

So I had some free time...

I felt the same, so I created Jeeves, a no-code easy-to-automate and collaborative space to get data-driven answers quick without the pain of manual ETL.

Check it out here and let me know what you think!


r/ETL • • Aug 29 '23

dbt vs Airflow: Which data tool is best for your organization?

Thumbnail
datacoves.com
0 Upvotes

r/ETL • • Oct 07 '24

Found a Great Resource Choosing ETL Tools!

1 Upvotes

Hello everyone,

Recently, I found myself in a crunch situation - deciding on the best ETL tool for my data integration needs. It was after much research and I finally found this amazing blog- it went quite a long way in helping me make a final choice.

The articles do not only outline different ETL tools, their features, and use cases but also will make the decision-making process of choosing which one is the best suitable to the requirements easier. If you experience something similar or just curious to know what your options are when it comes to ETL, then you should certainly have a look!

Best ETL Tools

Hope you find it as helpful as I did!


r/ETL • • Jul 30 '24

How Are You Handling Blockchain Data Challenges? Join Our Webinar to Learn from QuickNode and Bitcoin.com

2 Upvotes

Hey r/ETL,

Are you grappling with the complexities of blockchain data in your ETL processes? We’re hosting a webinar on August 8th at 12 PM EDT that dives into Blockchain ETL & Data Pipelines Best Practices, and we'd love for you to join us.

In this webinar, you'll learn about:

  • The unique difficulties blockchain data presents compared to traditional ETL.

  • Hear directly from Andrei Terentiev, CTO of Bitcoin.com, and Seb Melendez, ETL Software Engineer at Artemis, on overcoming these challenges.

  • Watch live demos of real-time data synchronization and indexing.

This session is perfect for Data Scientists, ETL Engineers, and CTOs who are looking to enhance their strategies for managing blockchain data or anyone curious about the future of data processing in blockchain technology.

What you’ll gain:

  • Firsthand insights from leaders in blockchain data management.

  • Answers to your pressing questions in a live Q&A session.

  • A deeper understanding of blockchain ETL tools and practices.

Interested? Register for free here and secure your spot: Webinar Registration Link

Hope to see you there and engage in some great discussions!


r/ETL • • Nov 28 '22

Comparing 7 ETL solutions with benchmarks

0 Upvotes

We did a comparison of 7 on-premise ETL solutions: R, Pandas, Knime, Power Query, Tableau Prep, Alteryx and Easy Data Transform, with benchmarks where possible.

https://www.easydatatransform.com/data_wrangling_etl_tools.html

As the test includes our own product, we can't claim it be independent, but we have tried to be fair to all products.


r/ETL • • Mar 24 '22

Kestra - Open-Source Airflow Alternative

Thumbnail
github.com
0 Upvotes

r/ETL • • Jan 17 '22

Hiring trend of popular ETL tools

0 Upvotes

Hi Guys,

If you are based out of India, this piece of information is for you.

Check the hiring trend for ETL tools in India and take informed decision about which tool you should learn to grow your IT career.

Hiring Trends - ETL Tools


r/ETL • • May 10 '26

A tool to catch schema drift and API changes before they break your ETL pipelines. Looking for feedback!

0 Upvotes

Most pipelines break because an upstream source changed without warning. I built a platform to catch these issues before they crash your ETL.

What it does:

  • Schema Monitoring: Detects renamed columns, dropped fields, or type changes in real-time.

  • Uptime Checks: Verifies your APIs and Databases are online before the pipeline runs.

  • Instant Alerts: Notifies you the moment drift is detected or any problem with the source.

  • Simple Setup: Connect your SQL DBs or REST APIs in under 2 minutes.

Would you use it and what features would make this a "must-have" for your workflow? Thanks!


r/ETL • • Dec 06 '19

I find ETL developers to have the worst coding practices, true or false?

4 Upvotes

I do some etl development tasks from time to time. In my experience, when etl devs are forced to abandon a restrictive drag an drop or simple scripted snippets within the drag and drop etl apps, and they have to create a full application to deal with some custom logic, they come up with the most convoluted mess of code I have ever seen in my life. What has been your experiences? I'm I generalizing? The few times i have proposed to organize etl logic in logical abstractions using oop then just compose the etl using these classes chaining them up as needed, I usually get a " we dont do that here" response. So we end up with crappy verbose code allover the place


r/ETL • • Dec 09 '22

Why We Normalize The Input Data

0 Upvotes

Hi guys,

I have made a video on YouTube here where I explain why we normalize the input data when training machine learning models.

I hope it may be of use to some of you out there. As always, feedback is more than welcomed! :)


r/ETL • • Jul 14 '20

Spark Vs. Snowflake: The Cloud Data Engineering (ETL) Debate!

0 Upvotes

For Enterprises, navigating the cloud transition can be tricky. With both Spark and Snowflake available for Cloud DataEngineering, how do you choose?

We've penned down our thoughts here. I'm keen to know what you think!


r/ETL • • Jun 03 '21

ETL vs. ELT in simple words!

Thumbnail
medium.com
0 Upvotes

r/ETL • • May 02 '25

Why generating EL pipelines works so well explained

0 Upvotes

Hi folks I'm a co-founder at dlt, the open source pip install self maintaining EL library.

Recent LLM models got so good that it's possible to write better than commercial grade pipelines in minutes

In this blog post I explain why it works so well and offer you the recipe to do it yourself (no coding needed, just vibes)

https://dlthub.com/blog/vibe-llm

Feedback welcome


r/ETL • • May 07 '21

‘Reverse ETL’ Can Help Companies Operationalize Data Warehouses

Thumbnail
thenewstack.io
1 Upvotes

r/ETL • • Jul 07 '26

[FOR HIRE] Senior Data Engineer – ETL/ELT Pipelines, Python, Spark, Databricks, AWS | Bangalore & Remote | $25-$50/hr

0 Upvotes

About Me

Senior Data Engineer with 5+ years of experience specializing in ETL/ELT pipeline design, Data Engineering, and Applied AI. Based in Bangalore, India. Available for remote work globally.

Rate: $25 - $50/hr depending on project scope and complexity.

ETL/ELT & Data Engineering Stack

Python, SQL, PySpark, Spark

Apache Airflow (DAG design, orchestration)

Databricks (Delta Lake, Unity Catalog)

AWS (S3, Glue, Athena, Lambda, EMR)

Snowflake, Redshift, BigQuery

dbt (data build tool)

Data Quality & Testing Frameworks

FastAPI, REST APIs

What I Can Help With

Design and build scalable ETL/ELT pipelines (batch & streaming)

Migrate on-prem pipelines to cloud

Orchestrate workflows with Airflow

Optimize slow pipelines and reduce costs

Data quality testing and validation frameworks

Build AI applications using LLMs, RAG, and agent-based workflows

Training & Mentorship

ETL/ELT Design & Data Engineering fundamentals

Apache Airflow & Orchestration

PySpark & Databricks

Snowflake & Cloud Data Warehousing

AWS for Data Engineering

In-person weekend sessions available in Bangalore. Remote sessions available globally.

Availability

Freelance projects & consulting

Part-time remote roles

Weekend training & mentorship

DM me with a brief description of your requirements!


r/ETL • • Oct 14 '26

ETL pricing is one thing. Getting help when something breaks is another

Thumbnail hevodata.com
1 Upvotes

One thing I've noticed in ETL discussions is how often people talk about rising bills, unexpected usage costs, and the time spent troubleshooting pipelines when something goes wrong.

The pricing can look fine when you're setting things up, but once the data volume grows, you need to understand what you're actually being charged for. And when a pipeline breaks in production, getting useful help matters a lot more than it did during setup.

Full disclosure, I work at Hevo, so I'm obviously biased. But I don't see it mentioned much in these discussions, and I thought it might be useful for people comparing tools.

A couple of things worth knowing about Hevo:

  • Pricing: It's consumption-based, using credits tied to data events. Historical data loading is free, and the pricing page has a calculator to estimate costs based on your usage.

  • Support: live chat on the Free plan to priority 24x7 support on small companies or big enterprise.

  • Pipeline troubleshooting: You can monitor pipelines, inspect failed events, and replay events after fixing the underlying issue.

It's not automatically the cheapest option for every workload, and usage still matters. I'd recommend checking the actual numbers for your data volume and looking at the support coverage you'd get before comparing it with another tool.

Here's the pricing page for anyone who wants to look at the details:

Hevo pricing and plans

Curious what matters more when you're evaluating an ETL tool: keeping the bill under control, or knowing you can get help when a production pipeline breaks?
Also would love to understand what other problems you guys face, maybe Hevo has a solution?


r/ETL • • Aug 14 '26

We compared 30+ ETL tools

5 Upvotes

We put together side-by-side comparisons of 30+ ETL and data integration tools, including features, pricing, and where each tool fits best.

https://etlworks.com/etl-tools-comparison.html


r/ETL • • May 13 '21

Moving past Airflow: Why Dagster is the next-generation data orchestrator

Thumbnail
dagster.io
7 Upvotes

r/ETL • • Jul 15 '26

Building and Troubleshooting an ETL Pipeline with an AI Agent

0 Upvotes

I work at Estuary, and next week we’re doing a live technical demo of our new Agent Skills.

We’ll show an AI coding agent:

  • Creating a real-time capture and materialization

  • Working from the same pipeline specifications an engineer would edit

  • Checking task health and pipeline status

  • Inspecting logs and troubleshooting failures

  • Handling routine configuration without hiding what it changed

The interesting question for us is where agents genuinely reduce data engineering work today, versus where an engineer still needs to step in.

We’ll build the pipeline live and leave time for technical questions.

Registration: https://zoom.us/webinar/register/4517840417866/WN_1oZszaPUQ_Sz0uEGM_p-zA


r/ETL • • 20d ago

What’s the most frustrating part of managing your data pipelines right now?

3 Upvotes

Hey Guys!,

I work at Hevo, where we build a platform for moving data from databases, SaaS apps, and other sources into data warehouses.

A lot of the problems we work around are things like pipelines failing, source schemas changing unexpectedly, data arriving late, troubleshooting sync issues, and keeping track of what's happening across different pipelines. We also have built-in transformations and monitoring to help teams manage these workflows.

I wanted to open up a discussion here because I'd like to hear what people are actually dealing with in their day-to-day work.

Are you spending too much time maintaining connectors? Running into problems with CDC? Dealing with unexpected costs, slow syncs, or support issues when something breaks?

Feel free to ask me anything about Hevo, how certain things work, or where it might fit into your setup. And if you've used Hevo before and had a frustrating experience, I'd be interested in hearing about that too.

I'll answer what I can here, and I'll be honest if something isn't a good fit for Hevo.

What are you struggling with most in your current data setup?


r/ETL • • Apr 15 '24

Why is ETL still a thing

10 Upvotes

I see there are no posts here, so let me be the first.

When I first got into Data Fivetran had barely done a Series A but I kinda already felt like ELT was solved ( know this subreddit is ETL but whatever).

That's because I pressed a button and data (in this case, Salesforce) simply landed in my destination. Schema updates were handled, stuff didn't really break, life was good.

Years on there are a million vendors building cloud saas elt. There are open-source servers like Airbyte. There are open source frameworks for ingesting data where you would run it yourself.

The ELT market also suffers from intense competition, and (rightly) a scornful eye from many data engineers. People don't want to be paying hundreds of thousands of dollars for connectors they could run cheaply, but no-one can be bothered to build them (fair) so we buy them anyway. There's lots of demand and also a race to the bottom, in terms of price.

So the question is - why hasn't the ELT market reached a perfect equilibrium? Why are Salesforce buying Informatica? Why are GCP and Snowflake investing millions in this area of Data? Why are there smart people still thinking about novel ways to move data if we know what good looks like? Prices are going down, competition is heating up, everything should become similar, but it's never looked more different. Why?


r/ETL • • Sep 11 '26

Data Engineers, What Would You Actually Want AI to Automate?

5 Upvotes

I'm working on a personal BI/ETL project and I'm exploring how AI could be genuinely useful in data engineering. Before I build anything, I wanted to ask people who actually work with data what problems are worth solving.

For context, I'm currently using **Power BI, Power Query (M), DAX, SQL, and Python** for the project. I'm also building an ETL/data-quality layer with things like validation rules, exception handling, and quarantining bad records.

I'm considering adding an AI component that could help with things like:

* Building ETL pipelines from natural-language requirements
* Automatically profiling new datasets
* Suggesting Power Query/SQL/Python transformations
* Suggesting data-quality rules
* Detecting and investigating data-quality issues
* Tracing bad data back to its source
* Estimating the business impact of data-quality problems
* Generating documentation and lineage
* Helping troubleshoot failed pipelines

One area I'm particularly interested in is **security**. Ideally, the AI wouldn't need direct access to sensitive company data. It could work from schema, metadata, statistics, and other safe information while the actual transformations happen inside the company's controlled environment.

I'm aware that there are already AI tools for ETL, data integration, coding, and BI, so I'm not trying to reinvent something that already exists. I'm mainly trying to identify a problem where an AI system could provide **real practical value**.

For data engineers, analytics engineers, BI developers, or data platform teams:

**What would you actually want AI to automate?**

What repetitive or frustrating part of your job takes way more time than it should?

Are ETL development, data-quality investigation, debugging, documentation, or pipeline maintenance actually painful enough that you'd use an AI tool for them?

And what would you **never trust AI to automate**?

I'm trying to validate the problem before building the project, so honest criticism and recommendations for existing tools are very welcome. 🙏


r/ETL • • 24d ago

Your data pipeline can be running successfully while quietly producing wrong data.

7 Upvotes

A pipeline finishes on schedule.

No errors. No failed jobs. Everything is green.

But what if a source system changed the meaning of a field?

What if duplicate events are being counted twice?

What if records are arriving late and yesterday's dashboard is missing 15% of the transactions?

Technically, the pipeline succeeded.

From the business's perspective, it failed.

I think this is why data engineering needs more than monitoring whether jobs complete.

You also need checks around freshness, completeness, uniqueness, schema changes, and important business rules.

And when something fails, the team should know which downstream reports or systems are affected.

For data engineers, which data quality checks have caught the most serious issues in your pipelines?

And which problems were hardest to detect because everything appeared to be running normally?