r/ETL • • Jul 30 '26

I built an open source data platform in go called Renart

108 Upvotes

Hi all, I'm a software engineer who made his way into data engineering. Coming into the field, I was really surprised by the lack of developer experience provided by tools in the data space. When we're writing SQL pipelines, there's no intellisense or static typechecking - features that any normal software developer would expect from their IDE. This is why I built Renart, an open source data platform and IDE. It is written in golang and it tries to combine all basic features of a data platform:

  • sql, python, api and load assets, all supporting jinja and chained together into pipelines

  • live typechecking and intellisense for all asset types

  • supporting multiple data warehouses (even in one pipeline) and setting up different environments

  • snapshotting the workspace into deployments that can be schedules using cron espressions

  • notebooks with sql and python cells working on a local duckdb

  • an interactive pipeline canvas, with sql previews, ad-hoc queries

  • staleness tracking based on canonicalized sql fingerprints, so you only rerun what changed

  • all pipelines definitions are 100% git tracked and diffable

Source: https://github.com/renart-data/renart Website: https://getrenart.com Disclaimer: I am the founder of Renart. I'm a solo dev doing this as my side project. This is currently still in an alpha state, so treat it accordingly.


r/ETL • • May 25 '26

Duckle - The local-first AI ETL/ELT data studio.

Post image
47 Upvotes

I have been building Open Source -
Duckle where you can simply drag a pipeline onto the canvas, describe their requirements in plain English to Duckie, the on-device AI assistant, and execute tasks at native speed using DuckDB.

It currently has:
- 290+ connectors
- 50+ transforms
- A built-in scheduler
- A chat assistant that operates entirely on your CPU

Repo link: https://github.com/SouravRoy-ETL/duckle


r/ETL • • Jul 06 '23

Hard Truth!

Post image
44 Upvotes

r/ETL • • Jun 24 '26

Duckle and DuckDB ecosystem just got stronger!

Post image
34 Upvotes

Duckle is the local-first, open-source visual ETL/ELT studio built on DuckDB. What's new:

GizmoData - GizmoSQL integration - read and write GizmoSQL over a clean-room Arrow Flight SQL client, right from the canvas.

Browser-based, dockerized editor - run the full drag-and-drop Duckle editor in your browser. One docker compose up, open localhost, and build + run
pipelines with live per-node progress. Self-hosted, no cloud, no account.

Qlik QVD read + write - native for Qlik Sense, no Qlik runtime required.

Bring-your-own AI - point the built-in assistant at any OpenAI-compatible endpoint.

Plus bulk SQL Server writes, run-to-here, and a stack of fixes.

100% free, yours and open source.
👉 https://github.com/slothflowlabs/duckle


r/ETL • • Sep 23 '26

ETL vs ELT — when do you actually still choose traditional ETL over ELT?

33 Upvotes

With modern cloud data warehouses, ELT seems to be the common approach because you can load the raw data first and transform it inside the warehouse.

So in what situations would you still choose traditional ETL, where the data is transformed before loading?

Are there specific cases involving security, data volume, performance, legacy systems, or compliance where ETL makes more sense?


r/ETL • • Aug 22 '26

Top 10 ETL/ELT Tools Ranked by Pricing (2026)

31 Upvotes

Just to be clear up front: this isn't a promotional post. I've been down the rabbit hole of comparing ETL/ELT tools for a while now, and honestly most "best tool" lists online are just ads in disguise. So I put together my own list, ranked purely by pricing, so you can quickly see what fits your budget and pick the right tool for your own use case. Hope this saves someone a weekend of research.

1. Meltano

Price: Free

Open source, Singer based, CLI first. Great if your team is comfortable in the terminal and wants version controlled pipelines.

Pros

  • Completely free

  • Version controlled, code first pipelines

  • Good for CI/CD driven workflows

Cons

  • No polished UI

  • Really a developer's tool, not something you hand to a non-technical analyst

Best for

Engineering heavy teams that live in Git and don't mind a learning curve.

2. Airbyte

Price: Free self-hosted, or credit based cloud starting around a couple dollars per credit

Huge open source connector library with 600+ integrations. You can self-host for free if you've got the engineering time, or pay for their cloud version if you don't.

Pros

  • Massive connector catalog

  • Free to self-host

  • Open source, so no vendor lock-in

Cons

  • Self-hosting needs real engineering effort

  • Community built connectors can be inconsistent in quality

Best for

Technical teams that want flexibility and control without paying licensing fees.

3. AWS Glue

Price: Pay per use

Makes sense if you're already deep in the AWS ecosystem. Serverless, so you're not paying for idle infrastructure.

Pros

  • Serverless, scales automatically

  • Tight integration with the rest of AWS

Cons

  • Locks you into AWS

  • Steeper technical setup than most tools on this list

Best for

Teams already committed to AWS who want ETL without managing servers.

4. Matillion

Price: Credit based, roughly $2 per credit

A visual, drag and drop transformation builder that runs natively inside your warehouse.

Pros

  • Strong visual interface for complex transformations

  • Runs natively in the warehouse, so it's fast

Cons

  • Seat based pricing can add up as your team grows

Best for

Teams who want warehouse native transformations without writing tons of code.

5. Windsor.ai

Price: $19 to $499/month, flat pricing

Worth calling out separately because it's built specifically for marketing data like Google Ads, Meta, TikTok, and Shopify, rather than being a general purpose ETL tool.

Pros

  • Genuinely cheap for what it offers

  • Transparent, flat pricing with no surprise per connector fees

  • Built in multi touch attribution

  • Can feed data directly into AI assistants

Cons

  • Not a general purpose ETL tool

Best for

Marketers and agencies who need affordable, no-code pipelines for ad and marketing data specifically.

6. Stitch

Price: From $100/month

Simple, no frills, gets data from point A to point B fast.

Pros

  • Fast to set up

  • Predictable pricing for straightforward needs

Cons

  • Limited transformation capabilities

  • Usually needs to be paired with something like dbt

Best for

Teams with basic replication needs who don't want complexity.

7. Hevo Data

Price: From $239/month

No code platform that's easy for non-technical teams to pick up, with a decent free tier to test things out first.

Pros

  • No code, easy onboarding

  • Good free tier for testing

Cons

  • Transformations are fairly basic compared to bigger players

Best for

Small to mid sized teams that want fast setup with minimal engineering effort.

8. Fivetran

Price: From roughly $1,200/month

The industry standard, and for good reason. Massive connector library and basically maintenance free once it's running.

Pros

  • Huge, well maintained connector library

  • Very low maintenance overhead

Cons

  • Monthly Active Rows pricing model can balloon quickly and is hard to predict

Best for

Teams that want a hands-off, enterprise-grade tool and can absorb the cost.

9. Integrate.io

Price: From $1,999/month

Covers ETL, ELT, CDC, and reverse ETL in one platform with fixed fee, unlimited volume pricing.

Pros

  • All-in-one platform

  • Predictable, fixed fee pricing regardless of volume

Cons

  • Entry price puts it out of reach for smaller teams

Best for

Fast scaling teams that want predictable costs as data volume grows.

10. Talend (Qlik)

Price: Custom, enterprise pricing

Built for large, regulated industries that need serious governance and compliance tooling.

Pros

  • Deep governance and compliance features

  • Enterprise grade support

Cons

  • Expensive and often overkill outside regulated industries

Best for

Large enterprises in regulated sectors that need robust data governance.

Bottom Line

There's no single "best" tool here, it really depends on your budget and use case. If you're small and scrappy, start free with Airbyte or Meltano. If you're marketing focused, Windsor.ai is hard to beat on price. If you need something that just works and you can afford it, Fivetran is still the safe bet.

Happy to answer questions if anyone's trying to decide between a couple of these.


r/ETL • • Jun 03 '26

Break boundaries with Duckle - a local-first data ETL/ELT Tool that runs on DuckDB

Thumbnail
gallery
31 Upvotes

8 million rows in. 600,000 out. 5.7 seconds. On a 16GB RAM laptop.

Duckle joined 4 sources at 2M rows each - an ADBC (Arrow) source, a CSV file, a MySQL table, and a second ADBC source - through one visual mapper: a 3-way join, 9 expressions, and a filter, straight to Parquet.

No cloud. No servers. Just Duckle on your laptop/desktop.
This is what local-first data engineering looks like now. 🦆

Repository: https://github.com/SouravRoy-ETL/duckle


r/ETL • • Jul 22 '26

Duckle is now on PyPI 🚀 pip install duckle

Post image
30 Upvotes

Duckle is a local-first ETL/ELT framework powered by DuckDB.
You can now define pipelines in Python. DuckDB executes them as optimized, vectorized SQL. Your data stays on your machine from start to finish.

Why Duckle?

✅ No Python bottleneck
Pipelines are compiled into SQL before execution.
No rows flow through the Python interpreter.
No hidden to_pandas() conversions.

✅ Minimal setup
~20 MB install
Bundles the DuckDB CLI
No JVM
No Docker
No server
No account required

✅ Python-first API
import duckle
from duckle import col
(duckle.read_csv("orders.csv")
.where(col.amount >= 20)
.derive(total="round(amount * 1.2, 2)")
.write_parquet("out.parquet")
.run())

Write familiar Python expressions while Duckle translates them into efficient DuckDB SQL.

More than file transformations
Duckle includes 359 built-in components:
-104 Sources
-66 Sinks
-138 Transforms

Supporting databases and services including PostgreSQL, MySQL, SQL Server, Oracle, Snowflake, Databricks, Kafka, Salesforce, SAP OData, S3, SFTP, WebSocket, IMAP, LanceDB, dbt and many more.

Built for automation
Validate pipelines without connecting to data sources or requiring credentials.
duckle validate
duckle validate --json
duckle --pipeline my.json

Perfect for CI/CD, containers, cron jobs, and local development.

Code ↔ Visual Studio
The same pipeline can be authored in Python or opened directly in the Duckle visual studio because both use the same JSON format.

Open source. Local first. Built on DuckDB.
⭐ GitHub: https://github.com/slothflowlabs/duckle/
📦 PyPI: https://pypi.org/project/duckle/
🔗 Links: https://github.com/slothflowlabs/duckle#quick-links


r/ETL • • Aug 30 '26

Are we overusing the Medallion architecture?

28 Upvotes

Bronze -> Silver -> Gold seems to have become the default architecture for almost every data pipeline.

Has anyone deliberately simplified this -- for example, skipping a layer -- and actually gotten better results in production ?

When do you think Medallion is genuinely useful, and when does it just add unnecessary complexity?


r/ETL • • Aug 29 '19

The expert way of structuring a project for Python ETL.

28 Upvotes

Hi,
I have recently moved from Informatica based ETL project to Python/Pyspark based ETL.
I want to know the best way to structure the projects & modules.


r/ETL • • Sep 03 '20

Modern Data Engineer Roadmap 2020

27 Upvotes

Hey all — in the last couple of weeks I've put a lot of effort into creating a modern & high quality roadmap for data engineers. I tried to gather the most important open-source and cloud based tools that a data engineer should know about. I published it to GitHub today.

https://github.com/datastacktv/data-engineer-roadmap

Let me know what you think and what you would add/improve!

Thanks in advance — Hope you will find it useful!


r/ETL • • Mar 06 '23

Why Pandas 2.0 is up to 3000% faster with Arrow (plus: all the Pandas alternatives explained)

Thumbnail
airbyte.com
26 Upvotes

r/ETL • • Feb 20 '22

Apache Airflow for Beginners Tutorial Series

26 Upvotes

Hey there,

I have been using Airflow for a couple of years in my work. I think it is a great tool for data pipeline or ETL management. Therefore, I have created this tutorial series to help folks like you want to learn Apache Airflow. So far, there are 12 episodes uploaded, and more will come.

If you are interested, you can watch the whole playlist on YouTube. If you think it is helpful, consider subscribing to my youtube channel and star my GitHub repository. Comment what topics you want to see or discuss about Airflow in the next episode.

Latest episode: Airflow Hooks S3 PostgreSQL

Updated Tutorial Episode 16.05.2022

  1. Introduction and Local Installation

  2. Get Airflow running in Docker

  3. Airflow Core Concepts in 5 mins

  4. Airflow Task Lifecycle and Basic Architecture

  5. Airflow DAG with BashOperator

  6. Airflow DAG with PythonOperator and XComs

  7. Airflow TaskFlow API

  8. Airflow Catchup and Backfill

  9. Schedule Airflow DAG with Cron Expression

  10. Airflow Connection and PostgresOperator

  11. Add Python Dependencies via Airflow Docker Image Extending and Customizing

  12. AWS S3 Key Sensor Operator

  13. Airflow Hooks S3 PostgreSQL


r/ETL • • Jan 18 '21

Trying to break into data engineering? Offering some help as a hiring manager

26 Upvotes

Hi everyone!

I am a Senior Data Engineering Manager (10+ years in the field and have hired 50+ data folks) and I want to help people who are trying to break into data engineering. I'm offering a 30 minute call to give advice on building skills, tweaking your resume to get noticed, prepping for job interviews, or negotiating job offers.

If you are interested, please send me a message and we can set up some time to talk.

LinkedIn - http://linkedin.com/in/sam-johnson-37026415


r/ETL • • Sep 20 '26

Should an ETL pipeline fail because of a data quality issue?

24 Upvotes

I was thinking about this while working with ETL and data validation. Suppose a pipeline is processing 10 million records and around 500 records have some data quality issues.

Should the whole pipeline fail, or should it continue loading and just raise an alert? or

should the bad records be separated and the good records continue?

Iam curious how people usually handle this in real production ETL pipelines, especially when a small data issue can block the entire downstream process.


r/ETL • • Mar 21 '26

Data integration tools - what are people actually happy with long term?

24 Upvotes

I’ve been comparing different data integration tools lately, and a lot of them look similar on the surface until you get into setup, maintenance, connector quality, and how much manual fixing they need later.

I’m less interested in feature-list marketing and more in what has held up well in real use. Especially for teams that need recurring data movement between apps, databases, and files without turning every new workflow into a mini engineering project.

For people here who’ve worked with a few options, which data integration tools have actually been reliable over time, and which ones ended up creating more overhead than expected?


r/ETL • • May 13 '20

Why GitLab is building Meltano, an open source platform for ELT pipelines

Thumbnail
meltano.com
24 Upvotes

r/ETL • • Apr 15 '20

Anyone else feel ETL work is treated like a 2nd class citizen?

24 Upvotes

Not sure where I'm going with this (maybe I just need to rant a bit), but I'm curious if others have experienced this over their careers. During my time, it seems like ETL/data work is treated like a second class citizen compared to more traditional software development like web development/services/application programming in a few ways.

Note that when I'm talking about ETL , I'm generally talking about the more traditional ETL tool-based software like Informatica, Talend, SSIS, DataStage, etc.

  1. I've noticed that the tooling around ETL code is immature/nonexistent/not used. For example, version control and automated testing. I know ETL is a niche, but wow.

  2. The compiled code from ETL tools does not work well at all with versioning systems like Git in my experience. I know this is because the output from these ETL tools tends to be proprietary or binary files that make identifying changes by comparing lines in the code difficult/impossible, but is there a solution? I feel like there has to be. People pumping out Java can easily tell the differences between versions of code and I'm jealous.

  3. Automated testing - I have never encountered any automated testing for ETL code in my time. Perhaps that's on me, but I haven't seen any programs or really any paradigms for this. It's always been people verifying things manually by querying the database or looking at files on Linux. This leads to lots of issues and slows down new development significantly.

  4. ETL work seems far less appreciated than services/front end work - by the business and other developers. I understand its easier for a business user to appreciate some new flashy feature on the website or application, but usually those features have involved data work in some way. Maybe it's only a problem in the orgs I've been in, but it feels like non-ETL teams get more of the glory and data/ETL teams usually only get the spotlight when things go awry - like bad data or missed SLAs.

  5. In my experience, it seems like companies use more contractors than employees for ETL work when compared to other software development. For example, at my current gig, I'm pretty sure I'm the only full-timer that is doing ETL - the rest are contractors. Which is definitely not the case for our services and front end teams.

Has anybody else experienced this? Am I missing something on the versioning and automated testing?

Look, I get it - using these GUI-based ETL tools isn't as glamorous as banging out a bunch of Python; however, I feel like the work involved is just as challenging as the more traditional software work. Just less appreciated.


r/ETL • • Sep 16 '26

What ETL mistake caused the biggest problem in a real project?

23 Upvotes

For people working with ETL/data pipelines, what is one mistake or design decision that caused a major problem in a real project?

Some examples I'm thinking about:

- Poor incremental-load strategy

- Duplicate records

- Missing data-quality checks

- Inefficient transformations

- Bad error handling/retries

- Schema changes breaking pipelines

- Full loads instead of incremental loads

- Lack of proper logging and monitoring

I'd be interested in hearing real-world lessons, especially things that aren't obvious when you're first building ETL pipelines.

NOTE : please don't post from Claude or chatgpt


r/ETL • • Jul 10 '26

We benchmarked Duckle against most used ELT & ETL Tools

Post image
23 Upvotes

So we benchmarked Duckle against the ETL tools everyone already
uses, with every tool tuned to its BEST configuration:

The task is deliberately boring: read one CSV(TPC-H Width and style), land it as a table.
It is the single most common job in ETL.

Three things we want to be upfront about, because benchmarks are
easy to rig 🫣:

  1. Duckle sits right on raw DuckDB
    own load (CREATE TABLE lineitem AS SELECT * FROM read_csv('out/lineitem_20m.csv') floor (~16s to fully parse and write 20M typed rows to disk). Duckle wraps the engine with pipelines, connectors and
    then gets out of its way. That is the whole design goal.

  2. Talend and Informatica used their bulk output connectors at max
    config, not the slow default row-by-row sink. On defaults they
    5-7x slower. We did not want to strawman them.

  3. Airbyte's number is derived from real 2M and 5M runs, and it needs
    an always-on 8 GB platform just to start.

Duckle is free and open source.
Try it: https://github.com/slothflowlabs/duckle


r/ETL • • Sep 26 '26

Your ETL job says "SUCCESS", but the actual business data is wrong. What checks do you use to catch those issues?

22 Upvotes

I usually start with source vs target row counts and data reconciliation. Then I check for nulls, duplicates, data type issues, and important business rules. I also think basic validation is not always enough, because the ETL can complete successfully even when the data is logically wrong.

How do you guys handle this in your projects?


r/ETL • • Sep 24 '26

ETL Job Description

Thumbnail
gallery
22 Upvotes

Hello!

5ish years ago I was an ETL programmer for a small homeowners insurance company. They had a small Data warehouse (SQL server) and some tools like SSIS and PDI I didn’t get to do to to much since I was there only 2 years before they went insolvent, but really sharpened my SQL skills. I’m currently a learning full stack development from being a database engineer, but I was reached out to the company who I have ties with about applying. Am I crazy for thinking this is a huge wish list for an ETL developer? When I spoke to them, the only thing they had set up was a database in AWS where the application rights to, but that’s it. Everything else would have to be created from the ground up.

I have another meeting scheduled with them tomorrow any major questions I should ask them?
Appreciate any in all advice thanks!


r/ETL • • Jun 06 '26

Duckle just got a lot more powerful - CDC, incremental loads, parallel pipelines, a visual joiner - and it still finishes in a blink.

Thumbnail
gallery
22 Upvotes

Duckle is a free, open-source, local-first Data Studio: build pipelines on a visual canvas, run them on DuckDB, ship them as a single binary. No cloud, no account, no telemetry. Your data never leaves your machine.

What's new in v0.2.0:
- Visual Map: join a main input to lookups across CSV, Parquet, DuckDB, SQLite and warehouses, with per-output expressions and no SQL.
- Parallelize: independent branches run concurrently, auto-scaled to your CPU cores.
- Universal upsert + CDC delete propagation across every relational family plus MongoDB.
- DuckLake CDC change-feed and watermark incremental loads.

Every number in the screenshots ran on a plain 16 GB laptop, nothing fancy:
- 16-node monolithic pipeline (5M-row 3-way Map join + parallel branches + 4 sinks): ~3.0s
- 100k-row DuckLake CDC mirror with upsert + deletes: ~1.7s
- 5,000,000-row watermark incremental load: ~1.8s

Heavy workloads finish before you can blink. And both dark and light themes are tuned to feel native to DuckDB.

Single binary. Engines download on first launch. 60 UI languages.

Repository: https://github.com/SouravRoy-ETL/duckle

Download + changelog: https://github.com/SouravRoy-ETL/duckle/releases/tag/v0.2.0


r/ETL • • Jan 15 '21

Modern Data Engineer Roadmap 2021

Thumbnail
github.com
22 Upvotes