r/DuckDB • • Dec 18 '25

Goated Duck

Post image
0 Upvotes

r/DuckDB • • Aug 26 '26

Build Streaming Lakehouses with RisingWave + Apache Iceberg + DuckDB

Post image
0 Upvotes

You can build your streaming lakehouse as a single system in which streaming ingestion, streaming analytics, open table storage, and external batch query engines all work together.

Kafka → RisingWave → Iceberg → DuckDB

*I work at RisingWave.*


r/DuckDB • • Jun 09 '26

You can now connect Claude directly to Duckle : AI-built pipelines that never leave your machine.

Thumbnail
gallery
6 Upvotes

You can now connect Claude directly to Duckle.

Duckle ships its own MCP server, so Claude (or any MCP client - Claude Desktop, Claude Code, Cursor) can build your data pipelines for you, right inside your local workspace.

Ask in any language, and Claude can:

🦆 Generate a pipeline (simple or complex) into your working directory

🦆 Validate it against 328 connectors (307 available out of the box)

🦆 Run it on DuckDB at native speed

🦆 Package it into a single standalone executable you can schedule anywhere

One click in Duckle ("Connect to Claude") wires it up. No cloud, no servers, no data leaving your machine - the engine and the MCP server both run locally.

Open source, local-first.

https://github.com/SouravRoy-ETL/duckle


r/DuckDB • • Jan 29 '26

Aprenda Duckdb Como se fosse uma criança de 12 anos

2 Upvotes

Decidi criar um blog e cursos para quem quer aprender Duckdb avançado de forma bem explicadinha.

Segue : https://duckdb.learn-apps.com/blog


r/DuckDB • • Dec 10 '23

Trending on GitHub top 10 for the 4th day in a row: Open-source framework for integrating AI with DuckDB

0 Upvotes

It is for building AI (into your) apps easily without needing to move your data into complex pipelines and specialized vector databases, by integrating AI at the data's source.

Not another database, but rather making your existing favorite database intelligent/super-duper (funny name for serious tech); think: db = superduper(duckdb)

Definitely check it out: https://github.com/SuperDuperDB/superduperdb


r/DuckDB • • Aug 20 '26

Benchmarking different data ingestion tools (to/from duckdb)

0 Upvotes

r/DuckDB • • Jul 28 '26

I built a cross-source SQL desktop app on DuckDB — one query across Postgres, MySQL and CSVs (feedback welcome)

1 Upvotes

Disclosure up front: this is my own project (Varan). Sharing it here because it's

built entirely on DuckDB and I'd genuinely like feedback from people who know

DuckDB well.

The idea: use DuckDB as a local federation engine so you can run one SQL query —

including JOINs — across a Postgres database, a MySQL host, a DuckDB file, and

CSV/Excel files at the same time. No ETL, nothing copied to a server.

How DuckDB does the heavy lifting:

- Each source is registered as a table — Postgres/MySQL via DuckDB's scanner

extensions, files via read_csv / read_parquet, plus native DuckDB files.

- Sources are "stubbed" on connect (schema only) and hydrate lazily on first

query, pulling only what's needed, then stay warm.

- Mutations round-trip back to the origin: an UPDATE on a Postgres-backed table

writes back to Postgres; a DELETE on a CSV-backed view re-serialises the file.

- Git-style version history — commit metadata + snapshots live in the DuckDB

file, so you can roll back a change like reverting a commit.

- Anomaly detection (duplicate keys, nulls, orphaned FKs) runs on a separate

DuckDB connection so scans don't compete with your query. Orphaned-FK detection

is a containment / inclusion-dependency check rather than name matching.

30-second demo (a Postgres orders table joined to two CSVs): https://varan.cloud

Free beta, macOS/Windows and soon Linux

Where I'd love this sub's take:

- Anyone running cross-DB federation with the postgres/mysql scanners in anger —

where does it fall over (type mapping, predicate pushdown, large scans)?

- Better patterns for the write-back / mutation round-trip than what I'm doing?

- Gotchas keeping version snapshots inside the DuckDB file as it grows?

Happy to go deep on any of it.


r/DuckDB • • Jan 21 '26

Analyze files with SQL or Your language — Powered by DuckDB

1 Upvotes

I’m excited to share rdsai-cli — a fast, lightweight CLI tool that brings the power of DuckDB and AI-driven natural language together for instant local file analysis.

Built on DuckDB, Designed for Speed

Load and analyze CSV and Excel files instantly — no database setup needed.

$ rdsai
> /connect sales.csv
✓ Loaded 'sales.csv' → table `sales` (15k rows, schema inferred)

Once connected, you can use full native SQL just like in DuckDB:

SELECT product, SUM(amount) 
FROM sales 
WHERE status = 'paid'
GROUP BY product 
ORDER BY 2 DESC LIMIT 5;

→ Runs directly on DuckDB: fast aggregations, joins, window functions — all supported.

But here’s the twist: Data Analysis in your language

No need to write SQL. Just ask:

> what are the top 5 customers by lifetime spending?
  • Automatically translated into efficient DuckDB SQL

  • Executed securely in-process — no data leaves your machine

  • Results formatted and ready to read

This is AI-Agent analytics, not a chat wrapper:
The model understands your schema, generates correct SQL, and leverages DuckDB’s engine for real execution.

Mix SQL and Natural Language Freely

  • Use natural language for quick exploration

  • Drop into raw SQL when you need precision

  • Press Ctrl+E after any query to get a plain explanation of results

GitHub: https://github.com/aliyun/rdsai-cli
Install: curl -LsSf https://raw.githubusercontent.com/aliyun/rdsai-cli/main/install.sh | sh

Is this the future of accessible data analysis? Let’s discuss!


r/DuckDB • • Mar 17 '26

Capire meglio DuckDb

2 Upvotes

Ciao r/duckdb vi scrivo perché in azienda ci hanno chiedo di valutare duckdb come sostituto di altre soluzioni. Abbiamo iniziato con alcuni benchmark (usiamo un approccio documentale ai dati, quindi abbiamo inserito duckdb nella nostra pipeline di benchmark). I primi risultati non sono stati entusiasmanti e ho avuto l’impressione che stessi sbagliando a verificare le performance di questo database.

Qualcuno che lo usa e lo conosce bene mi può dire quali sono i contesti di utilizzo e i punti di forza su cui vale la pena misurarlo rispetto ad altre soluzioni?

Grazie in anticipo a chi mi può dare una mano a capirci un po’ di più 🤗


r/DuckDB • • Oct 01 '26

I fed DuckDB 706,896 small JSON files on a 24 GB Mac Mini M4 Pro — here's what happened

2 Upvotes

I fed DuckDB 706,896 small JSON files on a local Mac Mini M4 Pro with 24 GB of RAM and 500 GB of storage.

The task was pretty simple on paper:

  • read 706,896 small .json files

  • extract the required fields

  • convert everything to .parquet

The .json files are responses from a public API that I collected while scraping a website over the course of a month.

The main problem turned out to be the number of small files.

On my first attempt, DuckDB hit _duckdb.OutOfMemoryException twice.

DuckDB supports larger-than-memory workloads through out-of-core processing: when intermediate data doesn't fit into RAM, it can spill that data to disk.

I tried that approach as well, but in my case it didn't solve the problem. DuckDB eventually exhausted the available disk space while spilling intermediate data to disk.

So I had to change the processing strategy and process the data day by day.

Fortunately, I had designed the raw data layout with this possibility in mind. Each file name contains its timestamp:

20260715080454689.json
20260827080648986.json

That made it possible to process a single day using a simple path pattern:

SELECT *
FROM 'data/20260715*.json';

After about 3 hours of processing, DuckDB had successfully converted:

706,896 JSON files → 64,557,511 rows

The final data was stored as .parquet files.

No cluster, no Spark, no distributed processing — just DuckDB running locally on a 24 GB Mac Mini.

For me, the interesting part wasn't just the final number of rows. It was seeing where the limits actually appeared: the workload was technically larger than memory, DuckDB could spill to disk, but eventually the storage itself became the bottleneck.

I'm continuing to experiment with the resulting dataset, so there are a few more interesting problems to solve from here.


r/DuckDB • • Sep 10 '24

Best LLM for duckdb?

0 Upvotes

In my experience with gpt 4o and Claude 3.5 they are both not super proficient at it.

Got 4o has tried several times to use a specific inexistent function and doesn't use many native functions, instead preferring to do some processing outside of duckdb.

Claude 3.5 also isn't super good at it but at least it doesnt repeat the same error insistently.

They both have trouble instantiating duckdb Wasm, they work 100x better if using duckdb for python.

Anyway, what has been your experience? Any recommendation?

I was hoping to use the Wasm more, leveraging the LLMs because I'm not a front end person, but im not getting a lot of help from it in the end.


r/DuckDB • • Sep 30 '26

Duckle an open-source ETL tool where every pipeline compiles to plain DuckDB SQL

12 Upvotes

- Author on a canvas, in Python, or in SQL. Every pipeline is one JSON file in git.

- duckle-runner serve runs it headless on a schedule, in Docker or on a box you own, with a web console, roles and an audit trail.

- Quality checks send bad rows to a reject port, so they land in a quarantine table instead of your warehouse.

- CDC and SCD Type 2 built in. Runs on DuckDB and uses every core you give it.

-Many more.

Github Repository - https://github.com/slothflowlabs/duckle


r/DuckDB • • Aug 10 '26

GitHub - sjjian/openhare: AI-powered desktop SQL client. Cross-platform. Built with Flutter. [Support DuckDB now]

Thumbnail
github.com
2 Upvotes

r/DuckDB • • Mar 01 '25

Is there a chatbot that can Connect to DuckDB? ChatGPT, Claude…

2 Upvotes

r/DuckDB • • Sep 06 '26

Automating Apache Iceberg Table Maintenance

Thumbnail
youtube.com
0 Upvotes

r/DuckDB • • 19d ago

I built an open-source ETL tool that uses AI to build DuckDB data pipelines

9 Upvotes

I’ve been building an open-source ETL tool for over a year, and I wanted to share it with the DuckDB community first. It’s called Ternlux. It’s an Electron desktop app that runs locally and uses DuckDB as its engine for transformations.

It includes a database management tool that lets you join Excel, CSV, Parquet, JSON, SQLite, DuckDB, and Avro files together. You don’t have to write any DuckDB read_* functions. You just write regular DuckDB SQL to select, insert, update, or delete, and the app handles the rest. Under the hood, when you run a DML statement, Ternlux creates temp tables, executes the code, and writes the results back to the original file format.

The second thing I wanted to share, and really the main point of the app, is that you can build pipelines with the AI assistant. It creates workflows that use DuckDB SQL for local transformations and joins. You can pull data from databases, vector databases, cloud storage, and other sources (we’re adding more all the time) down to your local machine. Then DuckDB runs the transformations, and the results are pushed to any destination you like.

GitHub: https://github.com/Ternlux/releases

I’d love any feedback. If you’d like to try it or contribute, check out the links above!


r/DuckDB • • Aug 18 '25

A DuckDB Server with Postgres interface

7 Upvotes

You can run boilstream, a DuckDB Server, and connect with Postgres interface.

Also, through FlightRPC with DuckDB Airport extension. There is also FlightSQL interface.

Disclaimer: I'm the author


r/DuckDB • • 4d ago

I made Duck: DuckDB for Swift with typed SQL, Codable, async/await and SwiftUI reactivity

3 Upvotes

Hey DuckDB folks!

I've been working on something I really wanted to show here because, well, it wouldn't exist without DuckDB :)

Meet SwiftStream/Duck, my Swift layer around DuckDB. I just shipped the first stable 1.0.0 release.

I originally found DuckDB while looking for a much more powerful database for my iOS and macOS apps. I already love working with SQL on the server side, so when I saw how much DuckDB gives you in one embedded engine, I immediately wanted the same thing to feel really good from Swift.

DuckDB already has an official Swift client, but I wanted to go quite a bit further on the application side: models, Codable, async/await, typed SQL, migrations, pooling, observation for SwiftUI/UIKit, and I was inspired to bring Bridges swiftly way to deal with all of it.

So now you can start with something simple:

import Duck

struct Todo: Table {
    @Column("id") var id: UUID
    @Column("name") var name: String
    @Column("created_at") var createdAt: Date
    @Column("done_at") var doneAt: Date?
}

let todo = Todo(
    id: UUID(),
    name: "Ship Duck",
    createdAt: Date()
)

try await todo.insertNonReturning(on: .app)

let todos = try await Todo
    .query
    .where(\Todo.$doneAt == nil)
    .all(on: .app)

But the part I'm especially happy with is that Duck re-exports my SwiftStream/SQL library, so when the query gets more interesting you don't have to drop into raw strings.

For example:

let query = SQL {
    Select {
        \Todo.$id
        \Todo.$name
        \Todo.$createdAt
        \Todo.$doneAt
    }

    From(Todo.table)

    Where(\Todo.$doneAt == nil)

    OrderBy(.asc(\Todo.$createdAt))
}

let todos = try await query.all(
    decoding: Todo.self,
    on: .app
)

So you're still writing actual SQL structure, just with typed Swift references instead of stringly-typed table and column names.

And the same query can also become reactive:

for try await result in try Todo
    .select
    .where(\Todo.$doneAt == nil)
    .orderBy(.asc(\Todo.$createdAt))
    .observe(on: .app)
{
    let todos = try result.all(decoding: Todo.self)
}

That plugs directly into SwiftUI or UIKit without adding another persistence abstraction on top.

The first stable release also has:

  • migrations

  • transactions

  • reusable prepared statements

  • connection pooling

  • bulk ingestion through DuckDB's Appender

  • native LIST/STRUCT values

  • Codable decoding/encoding

  • async/await throughout the API

Underneath all of this is the real DuckDB 2.0.0 engine.

Right now Duck targets iOS 15+ and macOS 12+. I've also tested it on Linux. DuckDB obviously goes much further platform-wise, but I haven't validated my Swift layer everywhere yet.

There's still a looooooot of DuckDB surface area I want to bring over, so 1.0.0 is really just the first solid foundation.

I'm especially curious what DuckDB people think about the API and the SQL layer, because my goal was basically: keep all the power of DuckDB underneath, but make the Swift side something I'd actually enjoy using every day.

There are also complete SwiftUI and UIKit example apps in the repo if you want to play with it.

https://github.com/swiftstream/Duck


r/DuckDB • • Mar 05 '25

Not reliables queries in DuckDB

0 Upvotes

When I do: .mode box COPY (SELECT * FROM read_csv_auto('*.csv', delim=';', ignore_errors=true) WHERE column05 = 2 AND column11 LIKE '6202%' AND column19 = 'DF';) TO './result.parquet';

works fine, but If I do SELECT DISTINCT column19 FROM './result.parquet';

It returns lots of columns I explicity said that I don't want

what did I miss here


r/DuckDB • • Jul 01 '26

Gartner Magic Quadrant Report

1 Upvotes

The time is counting to start seeing DuckLabs heading upwards in the quadrant.


r/DuckDB • • May 18 '26

Duckdb-WASM precompiled 1.5 version

1 Upvotes

Hi all,

currently my application uses Duckdb-wasm precompiled using version 1.32.0. Does anyone know when they will create a precomiled version for 1.5.0. i am pulling via NPM.


r/DuckDB • • May 04 '26

You drop a CSV/JSON/Xlsx/XML/log files. It becomes a dashboard. Then a report. Then a slide deck. Then an SDK inside your own app. None of it ever leaves the browser.

16 Upvotes

Four months ago I got tired of every "just look at this CSV" task turning into a 45-minute detour. So I built DuckViz, and it kept growing into something I didn't quite plan.

Here's what happens when you drop a CSV (or Parquet, or JSON) into app.duckviz.com:

Second 1 — it's a dashboard. DuckDB compiled to WebAssembly is doing the querying right there in your browser tab. Your file never uploads. Open devtools, check the network panel, you'll see nothing leave. Charts, filters, drilldowns, all of it — instant.

Second 30 — you ask it a question in plain English. "Show me revenue by region for customers older than 6 months." It generates the chart. The AI doesn't see your data either — it only sees your schema and writes the SQL.

Minute 2 — it's a report. Switch to Report mode and you get a rich-text doc you can write narrative in, with live charts embedded inline. Like Notion, but the charts actually query your data.

Minute 5 — it's a slide deck. Same data, presentation layout. Drop charts on slides, present from the browser. Your quarterly review just wrote itself.

Minute 6 — it looks like your brand. 10 built-in themes plus a full custom theme builder. Pick a vibe or paste your brand colors.

Minute 10 — it's inside your own product. the same thing renders inside your SaaS. Your customers' data never touches your servers either. That's the part that closes deals with security teams.

It's free to try, no signup for the demo, no email gate. The full thing — saved projects, sharing, AI — is behind a free account.

duckviz.com

Checkout the sample dashboards created using DuckViz for mock API's

https://duckviz-examples.vercel.app/


r/DuckDB • • Aug 13 '25

Adding duckdb to existing analytics stack

2 Upvotes

I am building a vertical AI analytics platform for product usage analytics. I want it to be browser only without any backend processing.

The data is uploaded using csv or in future connected. I currently have nextjs frontend running a pyodide worker to generate analysis. The queries are generated using LLm calls.

I found that as the file row count increases beyond 100,000 this fails miserably.

I modified it and added another worker for duckdb and so far it reads and uploads 1,000,000 easily. Now the pandas based processing engine is the bottleneck.

The processing is a mix of transformation, calculations, and sometimes statistical. In future it will also have complex ML / probabilistic modelling.

Looking for advice to structure the stack and best use of duckdb .

Also, this premise of no backend, is it feasible?


r/DuckDB • • Jul 25 '26

Did anyone tried the DuckDB Firestore community extension?

2 Upvotes

i came across a duckdb extension which accepts firestore as the source and then we can run all duckdb ops in sql.

want to check if anyone have tried it.