r/ETL • • 6h ago

Looking for real-world Saga examples to implement advanced transactions in HQL

2 Upvotes

TL;DR: I'm looking for good real-world examples of Saga implementations, especially how they handle compensating actions, retries, idempotency, and recovery across multiple systems. I'm developing HQL, a query language in Rust, and considering adding Saga support. The HQL examples below are mostly there to provide context.

Over the past few months I've been developing HQL, a strongly typed functional query language written in Rust.

The idea is something like SQL, but more of a Swiss Army knife for data, connecting PostgreSQL, Kafka, Spark, Delta Lake, and other systems through the same programming model.

I now have something that works relatively well as version 0.7.1 for basic OLAP workloads, including transformations, aggregations, streams, and data contracts.

For example, here's a real Kafka -> PostgreSQL pipeline that validates incoming events and routes invalid records into quarantine:

tap("order_events_raw")
| validate(EventData)
| fork {
    clean =
        oks()
        | map(e => Event(
            id = e.id,
            kind = e.kind,
            customer = e.customer,
            amount = e.amount,
            at = e.at))
        | sink("order_clean"),

    quarantine =
        errs()
        | map(e => Rejected(
            id = match e.value.id { some(Int n) => n, _ => 0 },
            message = e.value,
            rules = e.violations | map(v => v.rule)))
        | sink("order_quarantine"),
}

Complete example: https://github.com/ewiger/hql-demo/blob/main/kafka-postgres/quarantine.hql

For OLTP, I recently had to introduce a transaction mechanism into the language. Here's a snippet from a working PostgreSQL ledger example:

with transaction(attempts = 3, isolation = Serializable) tx on ledger {
    sender = source(accounts) | filter(a => a.id == 1) | one
    tx.rollback("insufficient balance") given sender.balance < amount

    debited =
        update accounts
        | filter(a => a.id == 1)
        | set(a => {balance: a.balance - amount})

    credited =
        update accounts
        | filter(a => a.id == 2)
        | set(a => {balance: a.balance + amount})

    tx.rollback("no such account") given debited != 1 or credited != 1
}

Complete example: https://github.com/ewiger/hql-demo/blob/main/ledger/transfer.hql

The transaction scope makes the block atomic over its participants. isolation = Serializable only selects the isolation level.

A transaction scope, with transaction(…) tx on {…}, makes its block atomic over its participants: tables of one PostgreSQL database or one SQLite file, or one Delta table. They are read with source and changed with insert, update and delete. isolation = Serializable sets the strongest isolation: concurrent scopes that would conflict are refused, and attempts retries them. Writes outside the participants, such as a sink to a topic or file, are not part of the transaction.

In general, HQL transaction scopes rely on the ACID guarantees of the underlying storage engine (PostgreSQL, SQLite, or Delta Lake). isolation = Serializable specifies the isolation level, not the entire ACID contract. Currently, a transaction scope is limited to a single transactional participant/target; it does not provide atomic commits across independent databases, Kafka, or external services. That's the problem I'm exploring with Sagas.

My next idea is to add a Saga-style scope alongside with transaction.

So now I'm looking into how Saga patterns could fit into HQL, coordinating operations across multiple databases, Kafka, and external services.

I'm particularly interested in good real-world examples of Saga implementations, especially how to model compensating actions, retries, idempotency, and recovery. Hence, if you've implemented Sagas in production or seen an elegant approach in another language, I'd be interested in hearing your thoughts.

PS

Disclaimer: This is all still very much WIP. I also use Claude Code extensively for tests and the Rust backend of the language. Even if you're allergic to AI-assisted projects but know a lot about Sagas, I'd be super grateful for your input.

PPS

This blog post looks great:

https://hosseinnejati.medium.com/the-saga-pattern-managing-distributed-transactions-ae8236ccd4c4

However, I've also read a lot of literature warning against Sagas. They're hard to implement, a lot of things can break, and distributed systems are already extremely complex in general.

But I feel super tempted to dive into this topic. If you really, really know a good real-world example, or an attempt to build such a design, I'd be super grateful if you could share it!


r/ETL • • 7h ago

Company job history and hiring momentum

Thumbnail
foorilla.com
1 Upvotes

foorilla.com v1.14.24 | foorilla API v1.12

Every company now comes with its job history: how many jobs it has open, how many it published and how many expired, week by week and month by month, across every job source we track, not only the jobs listed on foorilla. A company's Insight tab shows its open jobs against a quarter earlier, the latest four weeks of new and expired jobs, the median stated annual pay of its open jobs, and how its open jobs split by experience, work mode and country. You can download the whole history as CSV.

Company lists show each company's open jobs and their 13-week change, and sort by open jobs or hiring momentum. In Insight, the Hiring board adds open jobs across the market week by week, the industries gaining or losing open jobs, and the companies hiring fastest; Movers shows the companies rising and cooling.

API v1.12 adds this without breaking changes. /api/v1/hiring/company/st/ includes each company's history_open, history_open_before and history_open_change; company lists filter with open_min and sort with order. The new /api/v1/hiring/company/history/?company=ID returns a company's weekly and monthly history. See the API documentation for details.


r/ETL • • 8h ago

Data Warehousing: Talking Book Spoiler

Thumbnail gallery
1 Upvotes

Fresh off the Good Strat presses!


r/ETL • • 1d ago

Suggestions for my open-source lakehouse

6 Upvotes

I was tasked to design and setup an enterprise grade open-source Data Lakehouse on on-prem RedHat OpenShift.

Reason for on-prem: Sensitive data

Reason for open-source: Cost effective & feasible to have every component on-prem without data ever leaving the closed LAN

These are the components I choose:

RustFS

Apache NiFi

Apache Airflow

Apache Spark

Trino

PostgreSQL

Planning to use OAuth2, OIDC or Kerberos for user and component to component authentication

The project is still in design and POC stage. So I am open for suggestions and improvements.

Looking forward for valuable suggestions. Thanks in advance!


r/ETL • • 1d ago

I built a browser-local tool that turns a Snowflake catalogue into review-gated governance SQL, looking for feedback!!

Thumbnail
reddit.com
1 Upvotes

r/ETL • • 3d ago

Braidplane - Open Source (Apache 2.0) and self-hostable platform for declarative and contract-based data integration

3 Upvotes

Hello everyone!

Around Februrary this year I've started working on my side project, which is now known as Braidplane. It's an open source platform for data integration; it's self-hostable and the core is fully open (Apache 2.0). At this moment it's in Alpha.

The goal of this project is to provide safe and reviewable data integrations for engineering teams. From the beginning this project was built upon data contracts (inline and ODCS), e2e Braidplane Proof engine for reviewing data pipeline updates and declarative pipelines (YAML, JSON, cue).

As an example, a pipeline can read data from a database and a file, transform and compare the records, and write the results to another system. The pipeline definition stays in Git, so changes can be reviewed before deployment.

In the current form Braidplane core includes backend, frontend, CLI and executor. MCP support is coming soon.

About me: my name is Lukas. I've been working as backend engineer for the last 10 years. I've spent a lot of time around different products and APIs. I've built the Braidplane's framework myself - based on my experience and good practices I've found around the other OSS projects. Later on I've added AI assisted programming around the hardest/most time-consuming parts, like Connectors integrations, tests, documentation, frontend, issues management.

The Braidplane Core repository is available here: https://github.com/Braidplane/braidplane-core

Examples repo: https://github.com/Braidplane/examples/blob/main/recipes/opaque-file-archive/opaque-file-archive.pipeline.json

Cheers


r/ETL • • 4d ago

Advice for our ingestion layer

7 Upvotes

Hi all,

Me and some other people are responsible for all data and analytics throughout the company. Analytics is enabled company wide via a standard “medallion architecture” inside Databricks. This means that we use a standard pipeline job with notebooks to ingest the raw data in the bronze layer, clean/govern/format the data in the silver layer, and in gold we enforce a kimball schema and business logic.

Up until recenty, it was fine to append-only in bronze and just dedup those tables in silver so no information is ever lost. However, with the increase of our data streams, appending all into bronze is quickly turning into a massive data lake. Many of these tables may only have 5% of them changing throughout consecutive ingestions, but we cannot miss those changes either.

My question is; how do you guys deal with keeping high visibility on changes (our system does not allow for direct extraction of CDC tables), whilst also keeping the size of your data lake in check? (I heard that lakeflow declarative pipelines might be useful). Any suggestion is welcome!


r/ETL • • 4d ago

Streamhouse and Lakehouse with Apache Iceberg

Thumbnail
lakeops.dev
3 Upvotes

r/ETL • • 4d ago

What should instantly fail an Airflow DAG’s grade? Here’s our draft rubric, please tear it apart

Thumbnail
reddit.com
4 Upvotes

r/ETL • • 5d ago

Skipping the CDC pipeline: what I learned making an app's change log double as an Iceberg table

6 Upvotes

I've been building an open source embedded database (NodusDB, I'm the author) and one part of it might be interesting here, because it's basically an attempt to not need an ETL job for one kind of data.

The data is permissions: who belongs to which group, who can see which document. The usual setup is a permission service with its own database, and if anyone wants history ("who could see this file last Tuesday?") you build a CDC or batch pipeline into the warehouse and hope the copy doesn't drift.

What I tried instead: the database writes every change into one ordered log, and that log is shipped to S3 as signed, hash-linked objects. A projector reads the same log and writes an Iceberg v2 table from it, one row per change (commit time, LSN, event, object, relation, subject), partitioned by day, with names resolved instead of internal ids. So the table isn't a copy that gets synced, it's the log itself in another format. DuckDB and pyiceberg read it directly, which is what the tests use.

Some numbers from a run against real S3 (c5.xlarge, about 5.5k relationships, writes at 10/s):

- A write showed up in the Iceberg table after 31 s at the median and 61 s at worst, with the default 60 s commit interval. So freshness is basically "one commit interval", nothing clever.

- "Who could view this document at time T?" gave the same answer two ways: rebuilding the graph from S3 (214 to 248 ms, then 1,002 checks in 6 ms), and a recursive DuckDB query over the Iceberg table (122 to 245 ms). Having the two agree was the test I cared about most.

- Deleting a user (GDPR style) was the hardest part, because Iceberg files are immutable. The projector has to rewrite the affected data files and expire the older snapshots. In the run it took 6.9 s for everything to be scrubbed, and a byte scan found the name in 7 places before and 0 after (with bucket versioning off, otherwise the old objects stick around).

Things that are still rough:

- It doesn't merge small files or manifests yet, so a busy table will accumulate them.

- The freshness floor is the commit interval. Lower it and you get more, smaller files.

- Only one writer process at a time, and the projector is fenced with a conditional write on the metadata file so two can't commit over each other.

- The Iceberg and chain formats aren't covered by a compatibility promise yet.

The other half of the project is fast permission checks inside the app process (about 14 µs at the median vs 3.7 to 4.6 ms for SpiceDB and OpenFGA on the same data), but that's less relevant here.

Code and the raw benchmark results, if anyone wants to dig in:

https://github.com/Pragadeesh-19/nodusdb

https://github.com/Pragadeesh-19/authzbench

I'm curious how people here handle deletes in Iceberg tables in practice, and whether you'd trust a table written by the application itself instead of by a pipeline you control.


r/ETL • • 5d ago

Superstore Dataset

Thumbnail reddit.com
0 Upvotes

r/ETL • • 6d ago

Has anyone used Drogon framework to build a Data Pipeline????

Thumbnail reddit.com
1 Upvotes

r/ETL • • 6d ago

I built an open-source tool for migrating data between PostgreSQL and Neo4j (RtoGtoR)

Thumbnail
gallery
1 Upvotes

Hey everyone!

I've been working on an open-source project called RtoGtoR, and I'd love to get feedback from the developer community.

The idea behind RtoGtoR is to make it easier to migrate data between relational databases and graph databases.

Currently, it supports PostgreSQL ↔ Neo4j migration.

Some of the features I've implemented:

  • Explicit schema and data mapping

  • Migration planning before execution

  • Resumable migration runs with checkpoints

  • Batch-based migration execution

  • Migration verification

  • CLI and Go API

  • A shared engine that can be accessed through SDKs

The project is still in its early stages, so I'm especially interested in feedback from people who work with databases, data engineering, or graph technologies.

I'd love to know:

  • Would a tool like this be useful in your projects?

  • What features would you expect from a relational-to-graph migration tool?

  • What potential issues or limitations do you see with this approach?

I'm open to suggestions, contributions, and constructive criticism.

GitHub: https://github.com/TuanFaied/rtogtor

PyPI: https://pypi.org/project/rtogtor/
NuGet: https://www.nuget.org/packages/RtoGtoR
npm: https://www.npmjs.com/search?q=rtogtor

Thanks for taking a look! 🙌


r/ETL • • 7d ago

Added weekly and monthly company job history

Thumbnail
jobdataapi.com
3 Upvotes

jobdataapi.com v4.39 / API version 1.37

We’ve added a new /api/companyjobhistory/ endpoint with weekly and monthly job count history for every company. Each row covers one completed period and includes jobs open at the end of the period, newly published and expired jobs, the cumulative total, and breakdowns by work mode, experience level, salary disclosure, top countries and job types. Filter by company_id, period (week or month, monthly by default), period_start_since and period_start_until.

The history is derived from published and expiration dates and recalculated daily for recent periods, so late expirations are picked up. The coverage_start field shows when tracking began for each company. The endpoint is available to API access pro+ customers and higher.

API access ultra customers also get the full history as company_job_history.csv.gz with the weekly downloads, and a lightweight sample is now part of the 5K sample files and demo SQLite database.


r/ETL • • 7d ago

Added API endpoints for entities @ foo🦍

Thumbnail
foorilla.com
3 Upvotes

foorilla.com v1.14.18 | foorilla API v1.11 November 9, 2026

The organizations and people behind jobs, stories, papers, code and events are now in the API. Each entity comes with what identifies it: ROR and ORCID IDs, LinkedIn pages and Wikidata items. It also lists where it's based, and the companies, sources, organizers and institutions linked to it. The /st/ variant adds how many records name it in each Context, so you can rank the most-covered organizations with order=-records.

/api/v1/entity/evidence/?entity=… lists the records behind an entity, each with how it's connected: as employer, publisher, author, organizer, affiliation or funder, or as a text mention. Jobs, media items, papers, repositories and events also accept a new entity filter. You can go the other way too: find the entity for a company, source, organizer or institution by passing its ID.

API v1.11 adds all of this without breaking changes. Entity descriptions taken from Wikipedia are CC BY-SA 4.0; credit the article in description_source. See the API documentation for details.


r/ETL • • 8d ago

Snowflake into PostgreSQL using Hydra ETL

Thumbnail
gallery
9 Upvotes

Moving data out of Snowflake into PostgreSQL looks easy until the types come out wrong!

That's the kind of silent change I wanted Hydra ETL's new Snowflake source to rule out. Tested on a real Snowflake account, 50,000 orders into PostgreSQL:
→ 38-digit IDs and decimal amounts stay exact
→ DATE stays a date, TIMESTAMP_TZ keeps its time zone
→ VARIANT lands as JSONB
→ Snowflake's uppercase column names can be lowercased for PostgreSQL
Then I broke it on purpose: an ID too large for PostgreSQL's bigint, in the middle of 50,000 updates. The run stopped, and PostgreSQL still held exactly the previous data, to the cent. Nothing half-loaded.
Coming in Hydra ETL 0.20. The example, with a local Snowflake emulator if you don't have an account, is in the first comment.

Example: https://github.com/bejaouibechir/Hydra/tree/main/examples/snowflake-to-postgres


r/ETL • • 7d ago

Suggestions for streamlining data processing

4 Upvotes

Let me explain what is happening with experiments, data processing, and data visualisation at where I am working. I need advice on if there is a more streamlined way of doing this.

We are currently testing lots of samples on many identical instruments. The instruments can run different types of analysis but the data is mostly the same coming from it. It does need some "cleaning" after collection.

We get the files from the instrument (CSV or DTA) and import them into Microsoft Excel using macros that have been setup by god knows who. These macros "clean" the data (it knows where to look because of how standard the CSV files are) and drops it into the relevant places into the Excel sheet or column. We are doing this multiple times per experiment as the data becomes available. Data is visualised by preconfigured plots as it become available.

Once the experiment finishes, the Excel worksheet is then passed around and opened by researchers and they view the data there.

This works when the experiment is very standard because the macros don't need to change. When slight changes are made to the experiment, the macro does not function as the raw data (CSV or DTA) has changed in format slightly (extra columns).

Is there a better way of doing this that is more flexible and can be more automated?

My intuition and basic research is telling me that Python is the way to go here, watch the folder for new files, somehow get the sample ID and the data needed into pandas and clean it there. Not sure where I'd go after this.

Would love some suggestions!


r/ETL • • 10d ago

Built an AI extension that generates complete Python data pipelines from data samples

Thumbnail
gallery
10 Upvotes

Hey,

I'm Kevin, I run product at PrettyWhale.ai, a small team based in France. We've spent the last few months building a VS Code extension that writes the tedious part of ingestion code from a data sample, and we just opened it to everyone.

https://marketplace.visualstudio.com/items?itemName=prettywhale-ai.prettywhale

How It Works

You give it a CSV or JSON sample. It profiles the fields, you pick the ones to keep, then it suggests transformations you can accept, edit or drop. At the end you get a Python project in your workspace:

  • Pipeline & transformation code

  • Unit tests

  • Config files

  • Documentation

Under the hood

It relies on our own small model fine-tuned for data engineering, driven by a deterministic workflow rather than an endless chat window (no prompting required!). You can choose your target framework (Pandas, Polars).

Before the code reaches your editor, we run the pipeline and its unit tests on your sample to make sure it executes out of the box. What you get is plain Python, no proprietary dependencies, no lock-in, and no connection to our service.

Right now, the tool handles CSV and JSON samples, and upcoming versions will feature advanced transformations as well as multi-source and multi-destination support: https://docs.prettywhale.ai/roadmap

Privacy

Your sample goes to our servers, hosted in France. It's only used to generate your project, never for training, and it's deleted after.

Access

Access is free with limits (5 pipelines the first month, then 2 per month), with paid plans available via commercial licenses if you need higher limits.

VS Code only for now but we plan to expand.

Curious to hear your thoughts or what you'd like to see next in terms of features. Happy to answer any questions!


r/ETL • • 10d ago

How do you stop an agent from making unnecessary tool calls?

3 Upvotes

I’m building a company research agent that can query external data through MCP, and I’m using Coresignal’s MCP for the company and employee lookups. Letting the model decide what to fetch works better than stuffing everything into the prompt. What I’m still struggling with is the stopping condition. On some accounts it makes two calls and writes the brief. On others it keeps going, checking one more thing each time.

I’ve tried a hard cap on tool calls in the system prompt, plus “stop once you can answer the three questions below,” and neither is reliable.

How do you word stopping conditions for agents like this? Do you set a call budget in code and tell the model about it, or does that make the output worse?


r/ETL • • 10d ago

Automating Data Preprocessing: A Guide to Python AI Data Cleaning

Thumbnail reddit.com
3 Upvotes

r/ETL • • 10d ago

How does your team deal with source schema changes breaking downstream pipelines?

6 Upvotes

I've been thinking about how much time data teams spend dealing with unexpected schema changes.

A source column gets renamed, a new field appears, a data type changes, or an API response gets updated. Suddenly, something downstream starts failing, and someone has to figure out where the change happened and what needs fixing.

I've seen teams handle this through custom scripts, schema validation checks, alerts and manual fixes. Others seem to rely on features built into their data integration platforms.

I'm curious how different teams approach this:

  • Do you automatically detect schema changes, or catch them when a pipeline fails?

  • Does your setup handle new columns automatically, or does someone need to update the pipeline?

  • How much time does your team spend investigating these issues?

I work at Hevo Data, so I see these pipeline problems from the data integration side too. I'd genuinely like to hear how other engineers are handling them, especially if they're working with multiple sources and destinations.


r/ETL • • 11d ago

Managed versus self-hosted ingestion

9 Upvotes

Posting this here. For a small data team when do i need to run my own ingestion stack versus paying for a managed service? I’m interested in the maintenance and reliability tradeoffs not just the monthly price.


r/ETL • • 11d ago

Built an End to End Azure Lakehouse (ADF, Databricks, Delta lake, dbt, Airflow) - Key Learnings & Architecture Patterns

Thumbnail
github.com
15 Upvotes

Hi Everyone!

I recently completed a hands on data engineering project called WeatherFlow - Azure based Weather Data Lakehouse. I built to gain practical experience with tools and workflows.
The project Uses Weather source from Open-Meteo REST API and Process it through Medallion Architecture.

I wanted to move beyond basic tutorial pipelines and tackle real production engineering challenges—like dynamic metadata control, incremental ingestion tracking, and automated retries.

Architecture & tech stack:

1. REST API: Open-Meteo weather data ingestion as a primary source.

2. Azure Data factory: Parametrized pipeline for forecast and historical data ingestion. SQL control table and Procedures handle locations metadata

3. Azure Data Lake storage: Bronze layer for storing raw JSON data.

4. Databricks & PySpark: Data Parsing, transformation, standardization and deduplicate.

5. Delta lake: Silver Layer storage and incremental data updates.

6. dbt: Gold Layer transformation, dimensional modeling, incremental loads and quality tests.

7. Apache Airflow: End to End Orchestration.

Pipeline flow:

  1. Fetches weather data for Chennai, Bengaluru and Hyderabad.

  2. Supports both historical data ingestion and 7 days forecast ingestion.

  3. Using stored procedures and SQL tables to manage location ingestion and automated retries of failed location data specific to ingestion mode.

  4. Transforms and cleans the data using PySpark in Databricks.

  5. Loads the Processed data into the Silver managed Delta table.

  6. Silver layer only process the new files from bronze container which is managed by process control log table in silver layer.

  7. Uses dbt to build Gold models:

  • dim_date

  • dim_location

  • fct_weather_hourly

  • agg_weather_daily

  1. uses Airflow to orchestrate ADF, Databricks and dbt.

What I learned:

This Project gave me practical exposure to:

  • Building Parameterized data ingestion pipelines.

  • Working with REST API and cloud infrastructure.

  • Implementing Medallion Architecture.

  • Handling Duplicate records and incremental data process.

  • Using Delta lake MERGE operations.

  • Building models with dbt.

  • Implementing data quality tests for incremental data.

  • Orchestrating multi-stage pipelines using Airflow.

  • Troubleshooting cloud authentication, pipeline failures and integration issues.

Thanks for reading! Any feedback or suggestions are welcome.


r/ETL • • 11d ago

Not-so-beginner wanting to independently build ETL infrastructure but don't know where to look for material (100-1000 sources) to use.

5 Upvotes

The title.

I want to work on things like schema drift across sources, entity resolution, warehousing, fetching, idempotence, checkpointing and transaction management. I have already had practice with most of these ideas, but they have been primitive things. I know how to log, I know how to store to Postgres, but I dont have experience with a stack.

I have considered finding my own relationships between sources, but I have yet to discover a concrete avenue for handling volume. Other than that, I don't know what else to use. I want to make something real without paying, as much as I can.

Thanks <3