Building a bounded, read-only file-transformation CLI around embedded DuckDB v2.0.0 (sqrail)
Wanted to share an open-source project built on top of embedded DuckDB v2.0.0: sqrail (https://github.com/yhay81/sqrail, MIT).
First, a clear disclaimer (I am the author): sqrail does not try to replace the DuckDB CLI or claim faster query execution. Our benchmark harness builds both sqrail and duckdb from the exact same DuckDB commit (d54cbbe3) to verify output equivalence and track startup/RSS overhead (docs/BASELINE.md).
Why build a separate frontend? When automated callers or coding agents run local data transformations, a general-purpose shell exposes more surface area than a single step needs (arbitrary path reads inside SQL, COPY overwrites, ATTACH/INSTALL, unbounded spill/output sizes, or environment-heuristic output switching).
sqrail wraps DuckDB in a single-operation contract:
Explicit
-t name=pathbindings (files, globs,.gz/.zsttext, Hive-partitioned Parquet directories,--strict-schema, and--all-varcharto preserve leading-zero CSV/TSV codes without touching Parquet/JSON types)DuckDB
allowed_pathslocked to bound inputs + isolated per-process--spillworkspace, external access disabledAtomic no-overwrite
-ocommits across CSV, TSV, JSON, JSONL, and ParquetExplicit CLI budgets (
--memory,--threads,--timeout,--max-rows,--max-output-bytes,--max-spill)—e.g., sorting 10M Parquet rows under a 128 MiB DuckDB memory cap in 1.86s
Would love thoughts from heavy DuckDB users on any CSV/Parquet edge cases where --strict-schema or --all-varchar could be sharper.
