r/DuckDB • • Sep 21 '20

r/DuckDB Lounge

3 Upvotes

A place for members of r/DuckDB to chat with each other


r/DuckDB • • 1d ago

Built vastgrid (MIT) – Open 100M-row Parquet, CSV & DuckDB tables like a 60fps spreadsheet (VS Code, CLI, MCP, Jupyter, React)

11 Upvotes

Hi r/duckdb! I built vastgrid, an MIT-licensed HTML5 canvas data grid backed by DuckDB (vastgrid-duckdb on Node.js) and a matching Web Worker engine (vastgrid-js) that both pass the same 300-plan conformance suite.

What it does with DuckDB:

  • npx vastgrid-cli sales.parquet (or the VS Code/Cursor extension yhay81.vastgrid-vscode) queries Parquet, CSV, TSV, JSONL, Excel, and .duckdb databases in place — sorting a 100M-row (1.38 GB) Parquet file to first visible rows in 953 ms on an M2 Pro.

  • Full-table multi-key grouping, cross-tab pivots, live column totals, exact decimal formulas (ROUND(0.285, 2) = 0.29), and 60 fps canvas scrolling with an ARIA keyboard grid.

  • Surgical CSV/TSV saves: editing a cell rewrites only the changed field's byte span, preserving untouched quotes ("0019"), formatting, and CRLF/LF so git diff stays clean.

  • Also runs inside Claude Desktop / ChatGPT as an interactive MCP App (npx -y vastgrid-mcp ./data) and in Jupyter (pip install vastgrid).

Would love feedback from DuckDB folks on tough Parquet/DuckDB schemas or features you'd want next!


r/DuckDB • • 1d ago

Building a bounded, read-only file-transformation CLI around embedded DuckDB v2.0.0 (sqrail)

3 Upvotes

Wanted to share an open-source project built on top of embedded DuckDB v2.0.0: sqrail (https://github.com/yhay81/sqrail, MIT).

First, a clear disclaimer (I am the author): sqrail does not try to replace the DuckDB CLI or claim faster query execution. Our benchmark harness builds both sqrail and duckdb from the exact same DuckDB commit (d54cbbe3) to verify output equivalence and track startup/RSS overhead (docs/BASELINE.md).

Why build a separate frontend? When automated callers or coding agents run local data transformations, a general-purpose shell exposes more surface area than a single step needs (arbitrary path reads inside SQL, COPY overwrites, ATTACH/INSTALL, unbounded spill/output sizes, or environment-heuristic output switching).

sqrail wraps DuckDB in a single-operation contract:

  • Explicit -t name=path bindings (files, globs, .gz/.zst text, Hive-partitioned Parquet directories, --strict-schema, and --all-varchar to preserve leading-zero CSV/TSV codes without touching Parquet/JSON types)

  • DuckDB allowed_paths locked to bound inputs + isolated per-process --spill workspace, external access disabled

  • Atomic no-overwrite -o commits across CSV, TSV, JSON, JSONL, and Parquet

  • Explicit CLI budgets (--memory, --threads, --timeout, --max-rows, --max-output-bytes, --max-spill)—e.g., sorting 10M Parquet rows under a 128 MiB DuckDB memory cap in 1.86s

Would love thoughts from heavy DuckDB users on any CSV/Parquet edge cases where --strict-schema or --all-varchar could be sharper.


r/DuckDB • • 2d ago

DuckLake walkthrough: snapshots, change feed and dbt incremental models on 3 months of Ethereum data

Thumbnail
youtube.com
16 Upvotes

I made a beginner course on DuckLake and wanted to share it here. Disclosure: I work on Rosetta DBT Studio, the app I use in the videos.

Everything in it is open source: DuckDB, DuckLake, dbt Core, Rosetta DBT Studio, and the course code itself. The data is public too. No cloud account, no server, no paid tier needed. Everything also works with plain DuckDB SQL if you prefer the CLI.

Code (one commit per lesson): https://github.com/Nuri1977/ethereum-ducklake-dbt

The data is public Ethereum Parquet on S3: 92 days of blocks (~660k rows) and one week of transactions (12.7M rows). It needs about 3 GB of disk and runs on a laptop.

What the course covers:

- Creating a DuckLake with a DuckDB catalog and loading it from S3- Snapshots and time travel (AT VERSION / AT TIMESTAMP)

- Updates and deletes without rewriting Parquet (delete files)

- Schema evolution, and the change feed with ducklake_table_changes

- Partitioning, file statistics, data inlining, compaction and cleanup

- Registering existing Parquet with ducklake_add_data_files (and why the lake then owns the file)

- Reading the catalog tables by hand

- dbt on top: staging views, plus incremental marts with merge and commit messages via a post-hook

Feedback welcome, especially on anything I got wrong about DuckLake.


r/DuckDB • • 3d ago

Querying live sensor readings and their Parquet history in one DuckDB view via NATS JetStream and ZeBridge

Thumbnail
github.com
6 Upvotes

I’m building ZeBridge, and one of its examples started with a dashboard question: if a truck sends a temperature reading now and I flush Parquet files once a minute, why should “latest temperature” be up to a minute behind?

Sensors publish over MQTT or NATS into a JetStream stream. An archiver reads the stream in chunks, uses DuckDB to write Parquet, and uploads the files to S3-compatible storage. A separate query process follows the same stream and keeps recent readings in DuckDB. One view combines those readings with the archived files, so a query for the last ten minutes can span both.

The interesting bit was deciding when DuckDB could drop a recent reading. Each archive manifest records the last stream sequence covered by the uploaded files. The archiver publishes that manifest only after verifying the upload. The query process loads the file, then retires the recent rows it covers. Using stream position for that handoff means a delayed reading can have yesterday’s sensor timestamp and still show up in the right place. If a sensor resends a value for the same timestamp, the view counts it once, including when the two copies fall on opposite sides of the archive boundary.

On a 4-vCPU Linux VM with local NATS and SeaweedFS, the native archiver processed 500,000 readings in 2.33 seconds, best of three runs. That measures archive throughput. I haven’t yet measured the delay from a sensor publishing a reading to a dashboard query seeing it.

For anyone querying a live feed alongside Parquet in DuckDB: how do you decide when to flush a file? I’d be interested in how you balance small files against the amount of recent data you keep in the live table.

Runnable example, architecture diagram and benchmark


r/DuckDB • • 4d ago

I made Duck: DuckDB for Swift with typed SQL, Codable, async/await and SwiftUI reactivity

3 Upvotes

Hey DuckDB folks!

I've been working on something I really wanted to show here because, well, it wouldn't exist without DuckDB :)

Meet SwiftStream/Duck, my Swift layer around DuckDB. I just shipped the first stable 1.0.0 release.

I originally found DuckDB while looking for a much more powerful database for my iOS and macOS apps. I already love working with SQL on the server side, so when I saw how much DuckDB gives you in one embedded engine, I immediately wanted the same thing to feel really good from Swift.

DuckDB already has an official Swift client, but I wanted to go quite a bit further on the application side: models, Codable, async/await, typed SQL, migrations, pooling, observation for SwiftUI/UIKit, and I was inspired to bring Bridges swiftly way to deal with all of it.

So now you can start with something simple:

import Duck

struct Todo: Table {
    @Column("id") var id: UUID
    @Column("name") var name: String
    @Column("created_at") var createdAt: Date
    @Column("done_at") var doneAt: Date?
}

let todo = Todo(
    id: UUID(),
    name: "Ship Duck",
    createdAt: Date()
)

try await todo.insertNonReturning(on: .app)

let todos = try await Todo
    .query
    .where(\Todo.$doneAt == nil)
    .all(on: .app)

But the part I'm especially happy with is that Duck re-exports my SwiftStream/SQL library, so when the query gets more interesting you don't have to drop into raw strings.

For example:

let query = SQL {
    Select {
        \Todo.$id
        \Todo.$name
        \Todo.$createdAt
        \Todo.$doneAt
    }

    From(Todo.table)

    Where(\Todo.$doneAt == nil)

    OrderBy(.asc(\Todo.$createdAt))
}

let todos = try await query.all(
    decoding: Todo.self,
    on: .app
)

So you're still writing actual SQL structure, just with typed Swift references instead of stringly-typed table and column names.

And the same query can also become reactive:

for try await result in try Todo
    .select
    .where(\Todo.$doneAt == nil)
    .orderBy(.asc(\Todo.$createdAt))
    .observe(on: .app)
{
    let todos = try result.all(decoding: Todo.self)
}

That plugs directly into SwiftUI or UIKit without adding another persistence abstraction on top.

The first stable release also has:

  • migrations

  • transactions

  • reusable prepared statements

  • connection pooling

  • bulk ingestion through DuckDB's Appender

  • native LIST/STRUCT values

  • Codable decoding/encoding

  • async/await throughout the API

Underneath all of this is the real DuckDB 2.0.0 engine.

Right now Duck targets iOS 15+ and macOS 12+. I've also tested it on Linux. DuckDB obviously goes much further platform-wise, but I haven't validated my Swift layer everywhere yet.

There's still a looooooot of DuckDB surface area I want to bring over, so 1.0.0 is really just the first solid foundation.

I'm especially curious what DuckDB people think about the API and the SQL layer, because my goal was basically: keep all the power of DuckDB underneath, but make the Swift side something I'd actually enjoy using every day.

There are also complete SwiftUI and UIKit example apps in the repo if you want to play with it.

https://github.com/swiftstream/Duck


r/DuckDB • • 4d ago

Ruddy is now running DuckDB 2.0!

0 Upvotes

Hey everyone,

I’m an independent developer who started building Ruddy in my free time: a fast, modern macOS IDE for data management, analysis, and data science powered 100% by DuckDB.

With this new release, the core engine has officially been upgraded to DuckDB 2.0!

Ruddy runs on a true Pay-What-You-Want (PWYW) model—support independent development with whatever you feel it’s worth, or start exploring for free.

Check it out: ruddy.pro

I’d love your thoughts, benchmark requests, or features you'd like to see next!

What’s inside Ruddy:

  • DuckDB 2.0 Core Engine: Blazing-fast columnar processing over local files, Parquet, Arrow, CSV, and Iceberg/Delta Lake tables without spinning up heavy infra.

  • Multi-Cloud Direct Querying: Federate and query across AWS S3, Cloudflare R2, Google Cloud Storage, Azure Blob, and MotherDuck seamlessly in a single SQL statement.

  • Reactive Notebooks & Marimo: Build reproducible, reactive Python data science notebooks that sync directly with your in-memory DuckDB tables.

  • First-Class Geospatial Analytics: Full DuckDB spatial extension support—render GeoParquet, Shapefiles, and spatial geometries interactively on high-performance maps.

  • Auto-Ontology & Modular Semantic Layer: Automatically infers FACT/DIMENSION roles, deduces canonical SQL metrics, verifies joins via physical data overlap, and generates Open Semantic Interchange (OSI v1.0 / Apache Ossie) YAML specifications per dataset domain.

  • Context-Aware AI Assistant (Ruddynie): Text-to-SQL agent grounded in your workspace ontology and self-improving query memory—zero phantom table or join hallucinations.

  • Model Context Protocol (MCP) Integration: Expose your DuckDB catalogs, query tools, and semantic context to any external MCP-compatible AI agent or client.

  • Built-in Git Version Control: Track schema migrations, SQL queries, notebooks, and semantic models with standard Git repositories right from the workbench.

  • Force-Directed Semantic Graphs & Visual SQL Builder: Explore domain schemas visually or drill down with interactive force-directed relationship maps.


r/DuckDB • • 4d ago

Streamhouse and Lakehouse with Apache Iceberg

Thumbnail
lakeops.dev
3 Upvotes

r/DuckDB • • 5d ago

NLQ: plain text to DuckDB SQL, read-only and local-first

Thumbnail
reddit.com
2 Upvotes

r/DuckDB • • 7d ago

DuckLocal – A local-first data workspace built on DuckDB

13 Upvotes

DuckLocal is a local-first data workspace built natively on DuckDB. Open CSV, Parquet, JSON, Excel, or DuckDB files directly without setting up a database, configuring connections, or uploading your data.

It combines a native SQL editor, data explorer, interactive dashboards, and a headless CLI for AI agents. Human users and AI agents can work with the same local data, share analysis context, and review proposed changes.


r/DuckDB • • 7d ago

RLS for Quack: vcatbridge filters rows before they leave your DuckDB

7 Upvotes

Quack lets clients query a DuckDB database over the network, but it can't limit which tables or rows they see.

I wrote vcatbridge, a DuckDB extension that can. Your data stays in one DuckDB, SourceDB. You serve a second one, SharedDB, which reaches SourceDB's tables only through a policy.

Follow along here (Python, DuckDB 2.0.0): https://github.com/n6k-io/duckdb-virtual-catalog/blob/main/docs/quack-row-policies.md

The policy, set on SourceDB

SELECT vcatbridge_set_context('b1', 'tenant', 42);
SELECT vcatbridge_set_policy('b1', 'main.users',  'select', 'tenant_id = $tenant');
SELECT vcatbridge_set_policy('b1', 'main.users',  'insert', 'tenant_id = $tenant');
SELECT vcatbridge_set_policy('b1', 'main.users',  'update', 'tenant_id = $tenant');
SELECT vcatbridge_set_policy('b1', 'main.orders', 'select', 'tenant_id = $tenant');

secrets gets no policy, and nothing gets delete. Only SourceDB can change these.

What SharedDB sees

SELECT * FROM users;
-- (1, 42, 'ada'), (2, 42, 'grace')

INSERT INTO users VALUES (5, 7, 'mallory');
-- Invalid Input Error: vcatbridge: row violates the check predicate on this table

DELETE FROM users WHERE id = 2;
-- Permission Error: crossing: 'users' does not have 'delete' permission

FROM secrets;
-- Catalog Error: Table with name secrets does not exist!

secrets isn't hidden behind a permission error. It isn't in SharedDB's catalog, so there's nothing to name.

The query crosses as a plan, not SQL text

EXPLAIN SELECT name, sum(amount) FROM users JOIN orders ON id = user_id GROUP BY name;

Crossing Remote                        <- everything below runs on SourceDB
└─ Projection       name, sum(amount)
   └─ Aggregate     group by name
      └─ Join       INNER, id = user_id
         ├─ Projection  id, name
         │  └─ Crossing Floor   users
         └─ Projection  user_id, amount
            └─ Crossing Floor   orders

Each Crossing Floor is a table SharedDB doesn't scan itself. SourceDB swaps it for a scan with the row filter applied, so the filter never appears in SharedDB's plan. A function crosses only if it's on the bridge's whitelist.

Serve it over Quack, locked down

SharedDB runs quack_serve, then vcatbridge_lockdown, which takes files, network, extensions and settings away from it. From any DuckDB:

ATTACH 'quack:localhost:9494' AS app;

SELECT * FROM app.users;
-- (1, 42, 'ada'), (2, 42, 'grace')

FROM quack_query('quack:localhost:9494', 'SELECT * FROM read_csv(''/etc/passwd'')');
-- Permission Error: File system LocalFileSystem has been disabled by configuration

The threat model, including the leaks it accepts: https://github.com/n6k-io/duckdb-virtual-catalog/blob/main/SECURITY.md

It's submitted as a community extension (https://github.com/duckdb/community-extensions/pull/3570), so it's unsigned until that merges.

n6kd does all of this in one binary, with policies from a file: n6kd quack -f schema.sql --perms perms.yml. https://github.com/n6k-io/n6kd


r/DuckDB • • 7d ago

DuckDB vs Spark for larger ETL workloads, when is moving to distributed processing actually worth it?

23 Upvotes

For background, I'm trying to understand where DuckDB starts becoming less practical for larger data processing workloads, so feel free to point out any incorrect assumptions here.

DuckDB seems like a good fit for a lot of ETL work, especially when dealing with Parquet files, SQL transformations, joins, and aggregations without needing to manage a whole Spark cluster.

The part I'm less sure about is what happens when those workloads grow. Not necessarily because the SQL becomes more complicated, but because of the amount of data being processed, memory requirements, execution time, and how much infrastructure is needed to keep everything running.

I've also been looking into Spark performance and options like databricks photon and dualbird for acceleration. But I'm not sure at what point optimizing a distributed Spark workload makes more sense than continuing with DuckDB on a larger machine.

Some things I'm particularly curious about:

  • At what dataset size or workload complexity does DuckDB start becoming difficult to manage?

  • Are joins, aggregations, and sorting usually the first performance bottlenecks, or does it depend more on memory and storage?

  • If a workload already runs on Spark, is it worth looking at acceleration before considering whether DuckDB could handle the same transformations?

  • How much does operational complexity influence the decision compared to execution time and compute costs?

The goal isn't necessarily to replace one with the other. I'm more interested in understanding where the tradeoff changes, especially for batch ETL pipelines rather than interactive analytics.

Curious if anyone has worked with both in production and what made you stick with DuckDB or move to Spark.

Cheers! THANKS!


r/DuckDB • • 7d ago

DuckDB + Cloudflare + Your Agent = 100x+ cheaper analytics/BI than Amplitude/Sentry/Openpanel etc

Post image
11 Upvotes

Hey all, a little while ago I set up Openpanel in some of my projects but very quickly hit the limits and was starting to have to pay $50 to keep the events flowing in. I thought this was crazy, as I really wasnt sending that many events.

I was thinking, surely it can't be that expensive to store events. I looked into self hosting Openpanel, and it's even crazier. You need redis, clickhouse, postgres, beefy machine. I realized I am paying a tonne of money for features that are completely irrelevent now. I don't need instant charts. In fact, I would prefer to never even go to your ugly dashboard and figure out how to use it. So i'm paying this extreme premium so these services can compute all these indexes and what not so they can deliver me any chart I want in a second. It's crazy.

I thought about what I really needed as a lot of my work is done with claude/codex now and realized all I really wanted was to get the raw events and have my agent analyze them with duckdb, let me ask questions, and make charts.

I looked at a few ways of doing this but the simplest and cheapest was cloudflare. Literally, just store the events in R2, and then the agent can pull them down when it wants.

What I do instead is I have a 1tb SSD, I pull down my events from r2 when I want and load it into duckdb. Now my agent can do whatever it wants. It's not limited by the capabilities gifted to me by amplitude etc. Heck, it can even write python scripts or anything. You could create Jupyter notebooks to view it, so many options.

I have been using this in prod, but thought i'd open it up if anyone wants to try: https://slurp.run/

It's fully open source, and you can one click to install it. Lmk if you try it and run into any issues.


r/DuckDB • • 8d ago

DuckDB Extension - PQL : train predictive models in SQL

13 Upvotes

Feature engineering has been a huge pain for some of our customers.

A question can sound simple: “Will this customer place another order in the next 30 days?” But getting to a useful prediction means pulling together customer and order history, building features, creating labels and evaluating the model. That’s a lot of work before you even get to the interesting part.

That frustration led us to build PQL, an open-source DuckDB extension for training and running predictive models in SQL.

The idea is to describe what you want to predict and let PQL do some of the repetitive work. For the reorder example, it can use relationships between customer and order tables to build features such as historical counts, recency and event spacing. It then trains a model and makes predictions available inside DuckDB.

For forecasts, features are built from history up to the prediction point, and outcomes are measured over the following horizon. You can also backtest a model and inspect what it learned.

It’s still early, and there are real limitations. Models currently live only for the DuckDB session. Training is single-threaded and runs outside DuckDB’s buffer manager, so it doesn’t respect DuckDB’s memory limit. Attribute prediction also doesn’t have the same time-based leakage guarantee as forecasting but the benchmarks are really promising !

We’d love feedback from people who work with DuckDB: would something like this help you get to a useful first model faster?

Link : https://duckdb.org/community_extensions/extensions/pql


r/DuckDB • • 8d ago

I added experimental DuckDB support to Ptah, an open-source schema migration tool

6 Upvotes

I maintain Ptah, an open-source schema migration tool, and recently added experimental DuckDB support.

You describe your desired schema in SQL, and Ptah compares it with your database to generate migrations you can review and commit to Git.

If you'd like to see how it works before installing anything, you can try Ptah with DuckDB directly in your browser (note, on contrast with cli ptah duckdb plugin, here it uses DuckDB-Wasm v1.5.4, not 2.0.0, due to missing package for 2.0.0).

There is a small walkthrough using a persistent DuckDB database: create a table, then add a column and an index. For the second change, the generated SQL looks like this:

ALTER TABLE "main"."events" ADD COLUMN "observed_at" TIMESTAMP;
CREATE INDEX "events_kind_idx" ON "main"."events" ("kind");

Ptah generates paired up/down migration files and tracks which migrations have been applied. You can also inspect the resulting schema and check for pending migrations.

DuckDB support comes as a separate plugin:

ptah providers install duckdb

The plugin embeds DuckDB, so no separate database setup is needed. The walkthrough uses Ptah 0.25.0 and plugin 0.2.0, which bundles DuckDB 2.0.0.

The integration currently supports local files and in-memory databases only. MotherDuck and DuckLake aren’t supported (yet). Schema inspection currently rejects unsupported objects such as macros and named types, and there are other documented limitations around schema changes, external file access and loading additional extensions.

Happy to answer your questions in the comments. How are you currently handling schema changes in persistent DuckDB databases?

P.S. DuckDB support is still evolving. If there's a feature you'd like to see supported, feel free to open an issue or contribute a pull request. Contributions are very welcome!


r/DuckDB • • 9d ago

Ducklake in prod

Thumbnail reddit.com
1 Upvotes

r/DuckDB • • 10d ago

DuckDB - Extension in VS Code for doing MSSQL T-SQL queries against a Duckdb

12 Upvotes

My first VS Code Extension. demonstrating how I'm using DuckDB in some very non-standard ways. with microsoft clients inside VS-Code.

This extension does alot more, but I focused only on the DuckDB features for this one. ie. making your "PC as a data warehouse" portion using DuckDB for the engine

The trick/novel thing here, is you can connect from SQL Server Clients like PowerBI, ADO, SQL ODBC, SQL Server JDBC, SSMS directly using the native MSSQL connector.. but it's actually duckdb it's hitting. Of course this can of course also hit SQL Server (from DuckDB). Anyway, sounds confusing, but the demo walks you thru it all. I'm Pretty happy with the result even if nobody ends up using it.. learned a ton, just looking for feedback from the duckdb community.

Core Benefits

  • Using DuckDB - Mount postgres, Mount SQL Server, Mount Parquet, Excel sheets, CSV, json, MYSQL, SQLite - shows up as a single database in SSMS/MSSQL

  • Run native complex T-SQL (on Duckdb)

  • Run #temp tables (on duck)

  • Use MS t-SQL Declare statements. (on duck)

The demo:

https://www.youtube.com/watch?v=a6r0YkkNHA4

Thanks,

Scott

If you've ever heard of babelfish for postgres, that does a similar thing to what I'm doing here. But mine connects to duckdb... which can ALSO connect to postgres... ;)


r/DuckDB • • 13d ago

Using duckdb with baked-in knowledge base for agentic systems

Post image
16 Upvotes

For a recent project we had to educate a conversational system on carbon accounting. Looking for a frugal building block capable of semantic search and of keyword search, we developed an open-source project called duckdb-vectors. It happens that an embedded knowledge base powered by duckdb and extensions can answer hybrid queries in tens of milliseconds and restart in about a second. Such a level performance has been a good surprise. This is my first message on reddit, so thank you for your guidance.

More at https://boringsteps.org/duckdb-vectors


r/DuckDB • • 13d ago

Shareable DuckDb in the browser (yes, another one)

Thumbnail tinylake.io
4 Upvotes

As I post this, i see a handful of similar projects in this subreddit - seems like there's good momentum in this space, but would be great to hear the community's feedback regardless.

I built this over a few months to scratch an itch in my personal toolchain for data analysis. I wanted a quick way to upload, explore and share data with my peers, without the friction of onerous application installations or sign-ups (a bit like pastebin for dataa).

In the spirit of simplicity, the platform (which I would definitely consider 'alpha') and its respective sandboxes are open by design, so use at your own risk. Keen to hear any thoughts / feedback / suggestions!


r/DuckDB • • 14d ago

Querying HackerTube, HackerBooks and Wikihack directly with DuckDB through Hugging Face

Thumbnail
ssp.sh
4 Upvotes

r/DuckDB • • 14d ago

Duckstring - Pull-orchestrated data pipelines on DuckDB

16 Upvotes

https://duckstring.com

This is Duckstring, an open source data engineering platform, using DuckDB as the engine. I've built it to address some of the main frustrations I've had in my years as a data engineer: modularising logic so that it can grow in complexity over time, methods for upgrading components without breaking everything, and implying the DAG through dependencies so that I don't have to manage it.

I posted recently about DuckDB performance vs some of the big engines (write up here). I'm convinced DuckDB is the only engine needed to cover most real-world data pipelines.

Some specific features of Duckstring:

- Data transformations are treated like software packages, allowing the DAG to be implied by dependencies

- A pull-based orchestration system, throttling every process both upstream and downstream of the bottleneck so that nothing runs if it can't be consumed

- An incremental framework based on DBSP that handles updates and deletes efficiently

- CLI-first, so that you can point your own AI agents at the platform rather than relying on a janky integration

I've made a Playground app to help get a feel for the orchestration method:

Playground

You can also just go straight to the project on GitHub:

Duckstring GitHub

It's pre-1.0, though has been in active development since January, and is ready to try on real pipelines. I'd love to hear your feedback!


r/DuckDB • • 15d ago

Snowflake to Iceberg Lakehouse Migration Guide

Thumbnail
lakeops.dev
9 Upvotes

r/DuckDB • • 16d ago

Ducklake catalog migration from DuckDb to PostgreSQL

14 Upvotes

For background, I am not a data engineer but rather someone who happens to be responsible for data engineering, so feel free to (gently) point out any incorrect assumptions or core design issues. My background is in Bioinformatics and Data Science.

I have a created a datalake ducklake with a duckdb catalog for an initial prototype of the org data. For now this is mostly public data which I structured to make it fit for purpose. With it we have a data exploration dashboard, a react website, where users can view data and do some filtering - multiple users but read-only interactions with the data.

So for this purpose, a DuckDB catalog was fine. However, we will soon start implementing other functionality and creating new apps where users can write data back, for example scoring data points (rows) or adding new data themselves. This means it's time to move to PostgreSQL for concurrent writes (I think).

The question is if it is possible to convert / migrate the DuckDB catalog to PostgreSQL directly, or if a new catalog needs to be created? I am not too worried about losing the schema versioning. It's a nice to have but it's not critical.

Some additional background:

  • The number of users for the website(s) will be low at the start, less than 20, but we expect it will increase but not more than a few thousand in a few years' time.

  • The current parquet files are well structured and I don't expect them to grow by much, but in the near future we will include logs from instruments which will be a large number of small files (low thousands per year), and thus the datalake decision - flexibility.

  • very soon we will include some form of vector store, tbd if it will be separate store such as LanceDB or using pgvector inside the PostgreSQL or whatever (this is where I don't know if I am thinking this right). The vectors come from documents which will link to some central entities in our data.

  • This is a non-tech startup, so the data is domain specific.

  • Deploying the PostgreSQL is not an issue. When I started it was just me and thus a DuckDB catalog was perfect, but now there is web developer with Backend experience which will deal it.

  • Data is stored in GCS

  • The data and requirements are fairly basic and the team is small, so we want to keep it the infrastructure simple and flexible, but I am also aware that we will have to increase complexity as the requirements change.

Cheers!

Edit: to clarify, the goal is not to migrate the entire lake into PostgreSQL, only the catalog.


r/DuckDB • • 16d ago

Free text search via Parquet and DuckDB

Post image
13 Upvotes