r/dataengineeringvault • u/sspaeti • 3d ago
AI Two Perspectives on *Finding* Context: Agents vs. Human
r/dataengineeringvault • u/sspaeti • Jun 17 '26
Hey everyone! I'm u/sspaeti, a founding moderator of r/dataengineeringvault.
This community was created due to the 5 years of my existing data engineering vault and the value it provided (as illustrated in r/dataengineering see here).
I also created a Daily Dev Community, a year or more ago, focused on data engineering that people like. That's why I created this community, to share useful content I wrote and publish almost daily, so others like you can profit from it too.
I hope it will be useful to you. Let me know what you think, and let's see how it goes.
Happy to get your posts in, as long as they are not AI-generated. Most interested in this community is open-source data engineering, and related data stuff. Also SQL editors, spirit and data management, business intelligence (where I come from), and anything else related to day-to-day data work.
How to Get Started
Introduce yourself in the comments below.
Post something today that you found interesting or worthwhile to read. Or ask a simple question that prompts some conversation for us to discuss.
If you know someone who would love this community, invite them to join.
Thanks for being part of this. It's just starting, but I'm sure we can grow and learn together.
This community actively encourages links and blog posts, unlike other communities that block them. Please share your writing or blog posts.
PS: Please let me know if I should change anything on the sub-reddit settings, happy to make it a more pleasant place.
r/dataengineeringvault • u/sspaeti • 3d ago
r/dataengineeringvault • u/sspaeti • 8d ago
A non-technical domain expert builds a full data pipeline, map and live dashboard using Claude connected to MotherDuck, without writing SQL. The example pulls three public Swiss sources (MeteoSwiss weather stations, Swissgrid energy Excel files, and a cantons GeoJSON) to analyze how temperature drives electricity demand. The agent built a Python ETL pipeline, found the heating threshold sits at 13.5°C in Bern rather than the assumed 12°C, and caught a bug where a 2025 Excel file stored dates as text and dropped 61% of rows on first read. The final result is a dashboard with a live map that auto-refreshes on weekdays, built through iterative prompting and verification, with business-domain knowledge required to catch errors and interpret results correctly.
r/dataengineeringvault • u/sspaeti • 9d ago
I saw the hype on X about Herdr and Rex (before it was Zellij, and many more), but that got me thinking: why not manage my agents in Neovim, where I like to type, read and navigate anyway - so I prompted away with the new Fable 5.2 model, burnt a lot of tokens and built my dream Agents-IDE.nvim plugin.
It works with a Claude Code subscription, as this is the one I have been using for many years, but there's a PR for more agents, not sure if I have time to integrate them swiftly, but I wanted to share here to get an idea if any of you are doing similar things, or if you run 100 agents on different servers.
This plugin is mostly to handle local agents on your machine, and has a nice UI, rendering the feedback of Claude in Markdown, and having a drawer for git diffs, commit graphs, a fuzzy search for all your recent chats, saved prompts, and libraries in one place. The idea or design choice is heavily inspired by the vim-dadbod-ui that I use as my SQL-IDE, and I thought that design might work well for agents, too. You are the judge, it's very early, but it's fun to use - currently I tend to switch between project and everything scope (all projects and current git-repo I'm in, which is also works nice if you have many agent folder where you prompt).
Disclaimer: Is this plugin open source? Yes. Is this plugin completely free? Yes. Is this plugin vibe-coded beyond the author's ability to comprehend how it works? No.
r/dataengineeringvault • u/sspaeti • 11d ago
Fable 5.1 is way faster and seems to fix my requests as intended, whereas the new Fable 5.2 took way longer for everything and wasn’t always what I wanted. I switched back to Fable 5.1 now, and it just solves and fixes things.
One advantage of Fable 5.2, the tokens burn much slower, way slower than Fable 5.1. I guess it’s still the most capable model of Claude.
r/dataengineeringvault • u/sspaeti • 12d ago
Typed graphs add declared data types and constraints to graph data models, so every node and edge write is checked before it's committed. This matters because agents now write data faster than humans can review it. Fritz Laux's Typed Graph Model (TGM) paper underlies the idea, and Omnigraph implements it with schema files (.pg), checked queries (.gq), and policy files (.yaml) versioned in git. Checked queries validate traversals offline against the schema before execution, catching errors like a filter on the wrong type. Types only guarantee data is valid, not true; humans still decide what's true through branch review and merges, with Omnigraph v0.14.0 and later edge builds adding branch review, merge dry-runs, and u/card cardinality enforcement
r/dataengineeringvault • u/sspaeti • 13d ago
Shifting left means moving data quality checks, business logic, and transformations earlier in the data lifecycle, closer to the source, instead of handling them downstream in BI tools. The term originated as 'Shift-Left Testing', coined by Larry Smith in a 2001 Dr. Dobb's article, then spread to security (DevSecOps) and now data engineering via tools like SDF, dbt, and SQLMesh. Declarative, code-first platforms let teams move logic between config files without big rewrites. SQLGlot, the transpiler underlying SQLMesh, supports 24 SQL dialects, enabling local testing with DuckDB against production dialects like Spark SQL.
r/dataengineeringvault • u/sspaeti • 16d ago
r/dataengineeringvault • u/sspaeti • 17d ago
r/dataengineeringvault • u/sspaeti • 17d ago
Writing is a lonely sacrament. You sit alone in your room writing, being with yourself and not knowing what anybody thinks about it. You are with your ideas, with your doubts, you try to write something interesting. You read, you talk to people, but at some point, there's no going around it, you need to bring it into one single article or form of writing.
Most of the time, the writing feels small and insignificant, you don't know if any of it resonates, if any of the ideas work, if it's just a bad write-up. You never know until you share it with the world. It's a rollercoaster of emotions, ups and downs. Writing is hard.
r/dataengineeringvault • u/sspaeti • 18d ago
The post explores transaction logs in open table formats like Delta Lake, Apache Hudi, and Apache Iceberg. These logs are essential for ACID transactions, metadata management, and concurrent operations in data lakes. It outlines how each format handles concurrency, metadata, and schema evolution while also providing time travel capabilities.
r/dataengineeringvault • u/sspaeti • 18d ago
As AI agents write more SQL and dashboards, verification becomes the real bottleneck in business intelligence work. A Metabase survey found 74% of teams were building with AI but only 2% were scaling it, and the bottleneck has moved from creating analysis to checking it. The piece proposes five testing hierarchy levels: linting, compiler-style whole-tree checks, unit tests, integration tests, and end-to-end tests. It argues for moving all metrics, dashboards, and ETL logic into one declarative place with a unified lineage DAG, so agents can generate test suites against it. Metabase's CEO Sameer Al-Sakran says business people are the best data contracts, and that mutable state undermines trust in the output.
r/dataengineeringvault • u/sspaeti • 19d ago
r/dataengineeringvault • u/sspaeti • 19d ago
Reading Karpathy’s HackerTube, and then seeing a bunch of other apps on top of Hacker News like HackerBooks and Wikihacks got me thinking.
We can just run HackerTube, HackerBooks & Wikihack directly in your terminal with DuckDB and Hugging Face.
r/dataengineeringvault • u/sspaeti • 23d ago
If you want to get up to speed on data engineering, start with what the community voted for. These are my articles that hit the Hacker News front page or the top of r/dataengineering:
101 Concepts Every Data Engineer Should Know (915+ upvotes on r/dataengineering) https://www.ssp.sh/brain/data-engineering-concepts
Data Engineering Vault: the map of the entire field (HN front page) https://www.ssp.sh/brain/data-engineering
The Data Engineering Toolkit: Essential Tools for Your Machine (221+ upvotes) https://www.ssp.sh/blog/data-engineering-toolkit
Building a Data Engineering Project in 20 Minutes (211+ upvotes) https://www.ssp.sh/blog/data-engineering-project-in-twenty-minutes
A Diary of a Data Engineer (55+ upvotes) https://www.ssp.sh/blog/diary-of-a-data-engineer
Data Modeling, The Unsung Hero of Data Engineering: Architecture Pattern, Tools and the Future (HN front page) https://www.ssp.sh/blog/data-modeling-for-data-engineering-architecture-pattern-tools-future
Data Modeling Guide for Real-Time Analytics with ClickHouse (HN front page) https://www.ssp.sh/blog/practical-data-modeling-clickhouse
Why Semantic Layers Matter, and How to Build One with DuckDB (HN front page) https://www.ssp.sh/blog/semantic-layer-duckdb
Why Pivot Tables Never Die (HN front page) https://www.ssp.sh/blog/why-pivot-tables-never-die
A Beginner's Guide to Geospatial with DuckDB https://www.ssp.sh/blog/duckdb-geospatial
All free, no signup. If you liked this post, repost & help others find it.
r/dataengineeringvault • u/sspaeti • 24d ago
r/dataengineeringvault • u/sspaeti • 24d ago
r/dataengineeringvault • u/sspaeti • 25d ago
Inspired by my Neovim workflow.
r/dataengineeringvault • u/sspaeti • 26d ago
As analysts write less SQL by hand and prompt agents instead, verification becomes the new bottleneck in BI work. A Metabase survey found 74% of teams were building with AI but only 2% were scaling it, with the bottleneck having moved from creating analysis to checking it. Metabase CEO Sameer Al-Sakran argues verification needs five testing layers: linting, a compiler-like full-tree check, unit tests, integration tests, and end-to-end tests. He also says mutable state undermines trust in outputs, and that business people's intuition still catches errors math alone can't. The piece points to Metabase's upcoming Transform tests (v65) and its Data Studio product as early steps toward a unified lineage graph for verification.
r/dataengineeringvault • u/sspaeti • 26d ago
r/dataengineeringvault • u/sspaeti • 27d ago
Latest DuckDB Ecosystem Newsletter with the usual 10 updates, starting with Jev, the new "System One" model and how to use it with DuckDB and MotherDuck, the new PEG parser and faster S3 reads in DuckDB 2.0, a maintenance strategy for DuckLake, querying OneLake in the browser, and a native Mac app. Lots of great articles from the community as always.
r/dataengineeringvault • u/sspaeti • Oct 22 '26