r/DuckDB • • Sep 11 '26

Why DuckDB 2.0 is faster

Thumbnail
motherduck.com
132 Upvotes

When there is new major release, it's not always obvious how to get the speed bump as they often (not always) depends on how your data is shaped.
This breaks down the 3 main features of 2.0 - async i/o, recurcive cte and VARIANT type, anything missing ?


r/DuckDB • • 26d ago

Announcing DuckDB 2.0.0 LTS “Cinnamon Teal”

Thumbnail
duckdb.org
131 Upvotes

r/DuckDB • • Oct 09 '26

I'm now convinced single-node DuckDB is the best engine at any realistic scale.

119 Upvotes

I use Databricks (Spark) a lot for work, and wondered whether I could write a distributed compute engine on top of DuckDB. Long story short: I didn't need to.

It's clear DuckDB wins at small to medium scale, but I figured Spark and Trino might win out at larger scales, like 1TB+. And whatever approach I tried for splitting up queries and shuffling between nodes, Trino did win, at first by a lot (4-5x at 100GB on 16 nodes).

However, then I discovered that every engine spent most of its time waiting on S3, with CPUs around 90% idle. Compute speed barely mattered (and DuckDB turned out to be faster when not waiting on read). Trino was winning almost entirely on read rate from S3: it keeps far more bytes in flight per node, even running single-node.

DuckDB through httpfs uses the same threads to decode Parquet and to issue S3 reads, one per core by default. Each read waits ~30ms, so a 16-core box only has a few MB in flight. I wrote a custom reader to deal with that, but the big win was simply running 4x as many threads as cores: about 3x the read rate. After that, distributed DuckDB was roughly even with Trino at 10TB for the queries I tried (TPC-DS, q14, q33, q55). The distribution overhead was eating DuckDB's natural advantage.

Then I realised there was no need for distribution at all. I ran plain single-node DuckDB (with the custom reader) on one r7gd.4xlarge (r7gd.8xlarge for q14). On the same machine it beat Trino on all three 10TB queries I tried, 1.7-2.1x faster. Against a 5-13 node Trino cluster it was 2-3x slower but about half the cost. At 100GB, one DuckDB machine was about 2x cheaper than 17-node Spark or Trino clusters.

With the upcoming v2.0 (in alpha) S3 reads are expected to be 2-3x faster, so I think single-node DuckDB on a big machine is about to be the cheapest engine available for anything up to around 10TB. Possibly if someone figures out distribution too, it could go further.

I'm a convert!

Edit: I ran a 1TB benchmark on DuckDB 2.0 now that it's out, and it's even better: 3x faster than StarRocks and 10x faster than Spark. See the post: https://www.reddit.com/r/DuckDB/s/Crriqmse1a


r/DuckDB • • Sep 09 '26

I put DuckDB behind the file viewer in my IDE: a 1B row, 29 GB Parquet opens in about a second

Post image
97 Upvotes

The official Big Data File Viewer prints my string column as {114,111,119,45,57,57,50,49,57,53} instead of row-992195, and lists and maps come out the same way, buried in element and key/value noise, even though the schema says string.

That's what pushed me to write my own viewer for JetBrains IDEs: double-click a Parquet, Avro or ORC file and you get a table in the editor with DuckDB doing the reading underneath.

Opening times, measured with a small harness that fires the same three queries the plugin does, on files that have a list and a struct column in them:

RowsFileOpen
10M291MB28ms
100M2.9GB112ms
1B29GB1039ms

Paging to row 999,999,800 in the 1B file costs 800ms, less than opening it.

It's 100% free: Parquet, Avro & ORC Viewer

Editing a cell works but rewrites the file, 17 minutes on that 29 GB one. I have it down to 1.8s in a branch by having DuckDB rebuild only the row group and splicing it back in. Not shipped yet.

P.S. honestly I built this for the way I work with these files, and that is probably not the way you do, so if there is something you keep doing by hand and you want it in there, just say it and I will add it.


r/DuckDB • • Mar 12 '25

Duckdb just launched a UI !

93 Upvotes

Any new version of duckdb always come with an unexpected treat. Today they released a local UI that can be launched with one line of call !

Blog post here : https://duckdb.org/2025/03/12/duckdb-ui.html

Gonna try it after my current meeting 😁


r/DuckDB • • Jun 04 '26

Duckle just got a lot more powerful - CDC, incremental loads, parallel pipelines, a visual joiner - and it still finishes in a blink.

Thumbnail
gallery
81 Upvotes

Duckle is a free, open-source, local-first Data Studio: build pipelines on a visual canvas, run them on DuckDB, ship them as a single binary. No cloud, no account, no telemetry. Your data never leaves your machine.

What's new in v0.2.0:
- Visual Map: join a main input to lookups across CSV, Parquet, DuckDB, SQLite and warehouses, with per-output expressions and no SQL.
- Parallelize: independent branches run concurrently, auto-scaled to your CPU cores.
- Universal upsert + CDC delete propagation across every relational family plus MongoDB.
- DuckLake CDC change-feed and watermark incremental loads.

Every number in the screenshots ran on a plain 16 GB laptop, nothing fancy:
- 16-node monolithic pipeline (5M-row 3-way Map join + parallel branches + 4 sinks): ~3.0s
- 100k-row DuckLake CDC mirror with upsert + deletes: ~1.7s
- 5,000,000-row watermark incremental load: ~1.8s

Heavy workloads finish before you can blink. And both dark and light themes are tuned to feel native to DuckDB.

Single binary. Engines download on first launch. 60 UI languages.

Repository: https://github.com/SouravRoy-ETL/duckle

Download + changelog: https://github.com/SouravRoy-ETL/duckle/releases/tag/v0.2.0


r/DuckDB • • Jun 03 '26

Break boundaries with Duckle - a local-first data ETL/ELT Tool that runs on DuckDB

Thumbnail
gallery
69 Upvotes

8 million rows in. 600,000 out. 5.7 seconds. On a 16GB RAM laptop.

Duckle joined 4 sources at 2M rows each - an ADBC (Arrow) source, a CSV file, a MySQL table, and a second ADBC source - through one visual mapper: a 3-way join, 9 expressions, and a filter, straight to Parquet.

No cloud. No servers. Just Duckle on your laptop/desktop.
This is what local-first data engineering looks like now. 🦆

Repository: https://github.com/SouravRoy-ETL/duckle


r/DuckDB • • 27d ago

I built Obelisk: A lightweight, self-hosted alternative to Microsoft Fabric for sub-TB workloads (DuckDB, Parquet, dbt, Marimo)

Thumbnail
gallery
66 Upvotes

Hey guys,
I'm a data engineer, and I have been building pipelines for small to mid sized clients (with less than 1 terabytes of data) and most of the time the workload isn't really worth paying the premium for Saas products like Snowflake, Databricks .. etc, and can be done on a single beefy machine but stitching open sources tools usually comes with maintenance, networking and configuration headache, also these tools aren't unified under one control plane and don't know that other tools exists.
so I built a data platform that can be deployed on a single machine or server, and since I'm Egyptian I called it Obelisk 😄
Under the hood Obelisk uses:

  1. DuckDB, so you can do queries that joins multiple different sources in the same query (local or remote lakehouse, dbs, erp systems, mongodb .. etc)

  2. the platform also act as a daemon manager, so you can deploy tools you need with a single click and they will integrate directly into the platform (or you can connect remote instances)

  3. for variables and connection strings, you just add them to the platform once and every tool can access them.

  4. built-in support for cronjobs and advanced orchasteration tools like dagster or airflow

  5. a browser based dbt studio

  6. reactive notebooks

  7. I also added file manager and terminal access to the machine, so you don't need to SSH directly.

GitHub Repo: https://github.com/amrelsawalhi/obelisk

this isn't a finished product, I still have a lot of ideas to implement like auth and access control for teams
I’d love to get your feedback on the platform and contributions are most welcome.


r/DuckDB • • Oct 16 '26

DuckDB 2.0 and S3 File Read Performance

Thumbnail
substack.com
59 Upvotes

Not my article, but a very nice write up about how DuckDB's upcoming 2.0 release is going to make reading files on S3 no longer a bottleneck. I don't know what they've pulled off at DuckLabs but it is simply amazing.


r/DuckDB • • Aug 15 '25

Made an SQL learning app that runs DuckDB in the browser

58 Upvotes

Just launched https://dbquacks.com - a free interactive SQL learning app!

Retro arcade-style tutorial to learn SQL and explore DuckDB features. Progressive tutorial with 38 levels using DuckDB WASM, runs entirely in your browser, works on mobile.

Perfect for beginners who want to learn SQL in a fun way.


r/DuckDB • • Jan 15 '26

Data Platform built with DuckDB

Post image
57 Upvotes

Hi! I've been working with DuckDB for many years now.

I've used all sorts of the APIs, from Python, JS, Swift and most recently the C++ API.

Currently I'm building a full fledged data platform for cleaning, EDA, visualization, analysis, ad-hoc querying, etc. A general purpose tool to work with datasets. Think Tableau + Alteryx had a baby, and that baby turns out to be Usain Bolt. The core data execution is run using DuckDB, or our variants of it. It is a gift from god.

It's called Coco Alemana

Anyway...

One of the things I've used DuckDB for was creating a transpiler. Basically converting DuckDB SQL into a variety of other dialects. Goal being that you can query data against any database with full predicate pushdown without re-writing anything.

It's been a lot of work, but DuckDB's C++ APIs are so insanely well structured that it takes away a lot of the headache. They provide access to the AST, and the Binder. These two things alone take care of 70% of the work. The rest of the transpiler work is custom, and yes, is painstakingly boring.

I'm pretty well versed on the DuckDB internals and ecosystem, so if you have questions, I love talking all things DuckDB!


r/DuckDB • • May 04 '26 Bravo

Wanted to learn more about how DuckDB worked, so wrote a blog on the internals. lmk what you think

Thumbnail
greybeam.ai
56 Upvotes

r/DuckDB • • 26d ago

Just here to say I love DuckDB.

55 Upvotes

Using it along with R thanks to the duckdb package have made wrangling and reporting on large csv and parquet files a breeze. Thank you from the bottom of my heart.

--A grateful Data Analyst from Colombia.


r/DuckDB • • May 12 '26

Quack Remote Protocol

Thumbnail
duckdb.org
55 Upvotes

r/DuckDB • • Jul 18 '26

DuckDB crunches gigabytes of CSV and Parquet with no server, faster than Pandas

52 Upvotes

I got curious about the "it's fast" claims and benchmarked it myself on 41 million rows of real NYC taxi data (median of 5 runs, each engine's data loaded in memory first). The full script is on GitHub if you want to run it:

Operation (41M rows)PandasDuckDBPolars
Filter127ms23ms157ms
Group-by aggregation893ms20ms272ms
Join (two tables)6795ms20ms1316ms

Reproducible benchmark scripts (one run.sh, downloads the data and runs every test): https://github.com/botmonster/benchmarks/tree/main/duckdb-csv-parquet-analytics

On a 24-thread machine DuckDB was fastest on all three, roughly 45x faster than Pandas on the group-by and over 300x on the join (Pandas materializes the whole merged frame). Polars leads plenty of pure in-memory DataFrame benchmarks and stayed well ahead of Pandas, but DuckDB's multithreaded columnar engine took the top spot in my run. DuckDB's other edge is that it speaks SQL and queries files on disk, on S3, or over HTTP without loading them into a DataFrame first. If you already know SQL the learning curve is about zero.

The reason it beats SQLite at this is the storage model. SQLite is row-based and reads every column of every row; DuckDB is columnar, reads only the columns your query touches, and skips row groups that do not match your WHERE clause. The same idea shows up on file formats: in my tests a selective query ran 14x faster against Parquet than against the same data as CSV.

The sweet spot is datasets from 100MB to hundreds of GB, too big for Pandas to be pleasant and too small to justify Postgres or Spark. It handles larger-than-memory data by spilling to disk on its own. To check that, I capped memory_limit at 2GB and fully sorted all 41M rows; it spilled 8.8GB to disk and still finished in about 6 seconds.

Zero-copy Arrow interop is the other win worth knowing. Calling .df() or .pl() shares memory buffers with Pandas and Polars directly, so a 500MB query result lands in a DataFrame in milliseconds with no serialization round-trip.

Install is pip install duckdb, and it runs in-process in Python, Node, Rust, Go, and the CLI. Current stable is v1.5.4 (June 2026) with a v1.4.x LTS line beside it, MIT-licensed, with a stable on-disk format since v1.0. If you have ever spun up a Postgres instance just to aggregate a few log files, this removes that whole step.

Full writeup with the S3 and Postgres-scanner query recipes: https://botmonster.com/coding/duckdb-developers-analyze-csv-parquet-no-server


r/DuckDB • • May 25 '26

Duckle - The local-first AI ETL/ELT data studio that runs on DuckDB.

52 Upvotes

I have been building Open Source -
Duckle where you can simply drag a pipeline onto the canvas, describe their requirements in plain English to Duckie, the on-device AI assistant, and execute tasks at native speed using DuckDB.

It currently has:
- 290+ connectors
- 50+ transforms
- A built-in scheduler
- A chat assistant that operates entirely on your CPU

Repo link: https://github.com/SouravRoy-ETL/duckle


r/DuckDB • • Nov 15 '25

New Book Alert: Spatial Data Management with DuckDB

Post image
52 Upvotes

I’m thrilled to share that my new book (Spatial Data Management with DuckDB) is now published!

At 430 pages, this book provides a practical, hands-on guide to scalable geospatial analytics and visualization using DuckDB. All code examples are open-source and freely available on GitHub so you can follow along, adapt, and extend them.

GitHub repo: https://github.com/giswqs/duckdb-spatial

The PDF edition of the book is available on Leanpub.

Full-color print edition will be available on Amazon soon. Stay tuned.


r/DuckDB • • May 28 '25

DuckLake: SQL as a Lakehouse Format

Thumbnail
duckdb.org
52 Upvotes

Huge launch for DuckDB


r/DuckDB • • Oct 01 '26

Amazon Aurora PostgreSQL embeds DuckDB; supports direct querying of Apache Iceberg and Parquet data

Thumbnail
aws.amazon.com
50 Upvotes

r/DuckDB • • Aug 26 '26

I put DuckDB-WASM in a Google Sheets sidebar

50 Upvotes

I've been working on SQL Cell, a Google Sheets add-on that runs DuckDB-WASM entirely in the sidebar.

The basic idea is to use Sheets as the destination, not the data engine. You can attach a Parquet, Avro, SQLite, GeoJSON, Excel, etc. file, query it with DuckDB, preview the result, then write it into a range in the workbook.

Sheets themselves are queryable too, so you can do things like join a Parquet file from S3 against a tab someone maintains by hand and put the result back into the spreadsheet.

Since it is actually DuckDB-WASM rather than a SQL-ish layer over the Sheets API, httpfs/S3, DuckLake, Iceberg, and the usual DuckDB file formats work as well.

The architecture is intentionally backendless. It's an Apps Script add-on, with the UI and DuckDB bundle running inside an HtmlService iframe. There is no server of mine involved, and spreadsheet/file contents never get sent anywhere.

A couple of consequences of that:

  • S3 credentials only live for the browser session, so you have to enter them again next time.

  • Query history is derived from the workbook itself. When a result is inserted, SQL Cell writes a small LAMBDA formula alongside it containing the original SQL. That gives each result its own provenance, and the sidebar can scan those formulas later to rebuild history, jump to old results, or rerun them.

  • An upstream duckdb-sqlite issue currently blocks SQLite scanning.

It went live on the Workspace Marketplace yesterday, and it's free. I haven't put feature gates around DuckDB itself.

Still early, so I'm mostly interested in finding the cases where this falls over.

One thing I'm particularly curious about: if you're using DuckDB-WASM against private cloud storage in the browser, how are you handling credentials? Session-only credentials are safe enough for what I'm doing, but re-entering S3 keys is probably the part of the UX I'm least happy with.

Details and limitations: https://sqlcell.com
Install: https://workspace.google.com/marketplace/app/sql_cell/178915154116


r/DuckDB • • Aug 26 '26

🚨 DuckLabs to be acquired by AWS

Thumbnail
reddit.com
47 Upvotes

r/DuckDB • • Dec 15 '25

Everytime...

Post image
47 Upvotes

10 months later, just add "WASM" 😁


r/DuckDB • • Aug 31 '26

DuckLake (DuckDB as Catalog) is 41x faster than Iceberg for our Postgres CDC workload

45 Upvotes

I've been building an open-source (Apache 2.0) "analytics for Postgres" project called Streambed, and the piece this subreddit might find interesting is the DuckDB/DuckLake layer.

Streambed streams Postgres WAL changes into lake storage and serves queries through DuckDB as the query engine.

I started Iceberg as the default target but one of the community members recently mentioned about DuckLake. It caught my attention immediately (since it uses Catalog as the DB). So I added the DuckLake support and did the benchmark.

Ducklake is 41 times faster than Iceberg in my benchmark.

Benchmark slice: 1M rows, 100k updates, flush=1,000

  Write path:
  Iceberg COW:        269s
  Iceberg MoR:        4.7s
  DuckLake + DuckDB:  6.6s
  Full aggregate median:
  Iceberg MoR: 87s
  DuckLake:    16ms

  TopN median:
  Iceberg MoR: 85s
  DuckLake:    21ms

Takeaway: for a DuckDB-centered CDC path, DuckLake looks like the cleaner default. Iceberg is still great when broad multi-engine interoperability matters.

DuckDB/DuckLake is a magical primitive. So much to build on top of it.

Blog: https://streambed.dev/blog/ducklake-target-support/

Benchmark: https://github.com/viggy28/streambed/blob/main/docs/benchmarks/ducklake-vs-iceberg.md


r/DuckDB • • Dec 12 '25

Built a browser-native SQL workbench on DuckDB WASM, handles 100M+ rows, no install

45 Upvotes

Been experimenting with how far DuckDB WASM can go as a daily-driver SQL tool.

The result is dbxlite - a full SQL workbench that runs entirely in the browser. No backend, nothing to install.

What it does:

  • Query local files (CSV, Parquet, Excel) via File System Access API

  • Attach .db files with persistent handles across sessions

  • Monaco editor, schema explorer for nested data, keyboard-navigable results grid

  • Share executable SQL via URL

  • BigQuery connector (Snowflake coming)

Tested with 100M+ rows and 50GB+ local files. DuckDB WASM handles it surprisingly well.

Live demo: https://sql.dbxlite.com
GitHub (MIT): https://github.com/hfmsio/dbxlite

Share your SQL: https://sql.dbxlite.com/share/