r/databasedevelopment • • May 11 '22

Getting started with database development

411 Upvotes

This entire sub is a guide to getting started with database development. But if you want a succinct collection of a few materials, here you go. :)

If you feel anything is missing, leave a link in comments! We can all make this better over time.

Books

Designing Data Intensive Applications

Database Internals

Readings in Database Systems (The Red Book)

The Internals of PostgreSQL

Courses

The Databaseology Lectures (CMU)

Database Systems (CMU)

Introduction to Database Systems (Berkeley) (See the assignments)

Build Your Own Guides

chidb

Let's Build a Simple Database

Build your own disk based KV store

Let's build a database in Rust

Let's build a distributed Postgres proof of concept

(Index) Storage Layer

LSM Tree: Data structure powering write heavy storage engines

MemTable, WAL, SSTable, Log Structured Merge(LSM) Trees

Btree vs LSM

WiscKey: Separating Keys from Values in SSD-conscious Storage

Modern B-Tree Techniques

Original papers

These are not necessarily relevant today but may have interesting historical context.

Organization and maintenance of large ordered indices (Original paper)

The Log-Structured Merge Tree (Original paper)

Misc

Architecture of a Database System

Awesome Database Development (Not your average awesome X page, genuinely good)

The Third Manifesto Recommends

The Design and Implementation of Modern Column-Oriented Database Systems

Videos/Streams

CMU Database Group Interviews

Database Programming Stream (CockroachDB)

Blogs

Murat Demirbas

Ayende (CEO of RavenDB)

CockroachDB Engineering Blog

Justin Jaffray

Mark Callaghan

Tanel Poder

Redpanda Engineering Blog

Andy Grove

Jamie Brandon

Distributed Computing Musings

Companies who build databases (alphabetical)

Obviously companies as big AWS/Microsoft/Oracle/Google/Azure/Baidu/Alibaba/etc likely have public and private database projects but let's skip those obvious ones.

This is definitely an incomplete list. Miss one you know? DM me.

Credits: https://twitter.com/iavins, https://twitter.com/largedatabank


r/databasedevelopment • • Jul 10 '25

Why do devs treat SQL as sacred when the rest of the stack changes every 6 months?

146 Upvotes

I’ve noticed this recurring pattern: every part of the web/app stack is up for debate. Frameworks come and go. Frontends are rewritten in the flavor of the month. People switch from REST to GraphQL to RPC and back again. Everyone’s fine throwing out tools, languages, or even entire architectures in favor of better DX, productivity, or performance.

But the moment someone suggests replacing SQL with a different query language — even one purpose-built for a specific use case — there's enormous pushback. Not just skepticism, but often outright dismissal. As if SQL is the one layer that must never change.

Why? Is it just because it’s been around for decades? Because there’s too much muscle memory built into it? Because the ecosystem is too tied to ORMs and existing infra?

Genuinely curious what others think. Why is SQL off-limits when everything else changes constantly?


r/databasedevelopment • • Jan 10 '25

My very own toy database

137 Upvotes

About 7 months ago, I started taking CMU 15-445 Database Systems. Halfway through the lectures, I decided to full send it and write my own DB from scratch in Rust (24,000 lines so far).

Maybe someone will find it interesting/helpful (features and some implementation details are in the README).

Would love to hear your thoughts and questions.

www.github.com/MohamedAbdeen21/niwid-db

Edit: Resources used to build this:


r/databasedevelopment • • May 11 '25

toyDB rewritten: a distributed SQL database in Rust, for education

88 Upvotes

toyDB is a distributed SQL database in Rust, built from scratch for education. It features Raft consensus, MVCC transactions, BitCask storage, SQL execution, heuristic optimization, and more.

I originally wrote toyDB in 2020 to learn more about database internals. Since then, I've spent several years building real distributed SQL databases at CockroachDB and Neon. Based on this experience, I've rewritten toyDB as a simple illustration of the architecture and concepts behind distributed SQL databases.

The architecture guide has a comprehensive walkthrough of the code and architecture.


r/databasedevelopment • • Jan 31 '24

Samsung NVMe developers AMA

78 Upvotes

Hey folks! I am very excited that Klaus Jensen (/u/KlausSamsung) and Simon Lund (/u/safl-os) from Samsung, have agreed to join /r/databasedevelopment for an hour-long AMA here and now on all things NVMe.

This is a unique chance to ask a group of NVMe experts all your disk/NVMe questions.

To pique your interest, take another look at these two papers:

  1. What Modern NVMe Storage Can Do, And How To Exploit It: High-Performance I/O for High-Performance Storage Engines

  2. I/O Interface Independence with xNVMe

One suggestion: to even the playing field if you are comfortable, when you leave a question please share your name and company since you otherwise have the advantage over Simon and Klaus who have publicly come before us. 😁


r/databasedevelopment • • Jul 09 '25

I'm writing a free book on query engines

Thumbnail book.laplab.me
74 Upvotes

Hey folks, I recently started writing a book on query engines. Previously, I worked on a bunch of databases, including YDB, ClickHouse and MongoDB. This book is a way for me to share what I learned while working on various parts of query execution, optimization and parsing.

It's work-in-progress, but you can subscribe to be notified about new chapters, if you want to. All released and future chapters will be freely available on the website.

Constructive feedback is welcome!


r/databasedevelopment • • Jan 01 '25

Databases in 2024: A Year in Review

Thumbnail
cs.cmu.edu
69 Upvotes

r/databasedevelopment • • May 19 '25

Lessons learned building a database from scratch in Rust

66 Upvotes

Hey r/databasedevelopment,

TL;DR Built an embedded key/value DB in Rust (like BoltDB/LMDB), using memory-mapped files, Copy-on-Write B+ Tree, and MVCC. Implemented concurrency features not covered in the free guide. Learned a ton about DB internals, Rust's real-world performance characteristics, and why over-optimizing early can be a pitfall. Benchmark vs BoltDB included. Code links at the bottom.

I wanted to share a personal project I've been working on to dive deep into database internals and get more familiar with Rust (as it was a new language for me): five-vee/byodb-rust. My goal was to follow the build-your-own.org/database/ guide (which originally uses Go) but implement it using Rust.

The guide is partly free, with the latter part pay-walled behind a book purchase. I didn't buy it, so I didn't have access to the reader/writer concurrency part. But I decided to take the challenge and try to implement that myself anyways.

The database implements a Copy-on-Write (COW) B+ Tree stored within a memory-mapped file. Some core design aspects:

  1. Memory-Mapped File: The entire database resides in a single file, memory-mapped to leverage the OS's virtual memory management and minimize explicit I/O calls. It starts with a meta page.

  2. COW B+ Tree: All modifications (inserts, updates, deletes) create copies of affected nodes (and their parents up to the root). This is key for snapshot isolation and simplifying concurrent access.

  3. Durability via Meta Page: A meta page at the file's start stores a pointer to the B+ Tree's current root and free list state. Commits involve writing data pages, then atomically updating this meta page. The page is small enough that torn writes shouldn't be an issue: meta page writes are atomic.

  4. MVCC: Readers get consistent snapshots and don't block writers (and vice-versa). This is achieved by allowing readers to access older versions of memory-mapped data, managed with the arc_swap crate, while writers have exclusive access for modifications.

  5. Free List and Garbage Collection: Unused B+ Tree pages are marked for garbage collection and managed by an on-disk free list, allowing for space reclamation once no active transactions reference them (using the seize crate).

You can interact with it via DB and Txn structs for read-only or read-write transactions, with automatic rollback if commit() isn't called on a read-write transaction. See the rust docs for more detail.

Comparison with BoltDB

boltdb/bolt is a battle-tested embedded DB written in Go.

Both byodb-rust and boltdb share similarities, thus making it a great comparison point for my learning:

  • Both are embedded key/value stores inspired by LMDB.

  • Both support ACID transactions and MVCC.

  • Both use a Copy-on-Write B+ Tree, backed by a memory-mapped file, and a page free list for reuse.

Benchmark Results

I ran a simple benchmark with 4 parallel readers and 1 writer on a DB seeded with 40,000 random key-values where the readers traverse the tree in-order:

  • byodb-rust: Avg latency to read each key-value: 0.024µs

  • boltdb-go: Avg latency to read each key-value: 0.017µs

(The benchmark setup and code are in the five-vee/db-cmp repo)

Honestly, I was a bit surprised my Rust version wasn't faster for this specific workload, given Rust's capabilities. My best guess is that the bottleneck here was primarily memory access speed (ignoring disk IO since the entire DB mmap fit into memory). Since BoltDB also uses memory-mapping, Go's GC might not have been a significant factor. I also think the B+ tree page memory representation I used (following the guide) might not be the most optimal. It was a learning project, and perhaps I focused too heavily on micro-optimizations from the get-go while still learning Rust and DB fundamentals simultaneously.

Limitations

This project was primarily for learning, so byodb-rust is definitely not production-ready. Key limitations include:

  • No SQL/table support (just a key-value embedded DB).

  • No checksums in pages.

  • No advanced disaster/corruption recovery mechanisms beyond the meta page integrity.

  • No network replication, CDC, or a journaling mode (like WAL).

  • No built-in profiling/monitoring or an explicit buffer cache (relies on OS mmap).

  • Testing is basic and lacks comprehensive stress/fuzz testing.

Learnings & Reflections

If I were to embark on a similar project again, I'd spend more upfront time researching optimal B+ tree node formats from established databases like LMDB, SQLite/Turso, or CedarDB. I'd also probably look into a university course on DB development, as build-your-own.org/database/ felt a bit lacking for the deeper dive I wanted.

I've also learned a massive amount about Rust, but crucially, that writing in Rust doesn't automatically guarantee performance improvements with its "zero cost abstractions". Performance depends heavily on the actual bottleneck – whether it's truly CPU bound, involves significant heap allocation pressure, or something else entirely (like mmap memory access in this case). IMO, my experience highlights why, despite criticisms as a "systems programming language", Go performed very well here; the DB was ultimately bottlenecked on non-heap memory access. It also showed that reaching for specialized crates like arc_swap or seize didn't offer significant improvements for this particular concurrency level, where a simpler mutex might have sufficed. As such, I could have avoided a lot of complexity in Rust and stuck out with Go, one of my other favorite languages.

Check it out

I'd love to hear any feedback, suggestions, or insights from you guys!


r/databasedevelopment • • Jun 18 '24

LSM tree "popular" implementations

64 Upvotes

Looking for implementations of LSM tree that are used in well-known projects either in Go or Rust. C++ or Zig is ok too but prefer any from the first 2. Please comment the link/s below. It may not be separate package, can be an internal one but at least has well defined interface. Thanks!


r/databasedevelopment • • Jan 05 '26

Databases in 2025: A Year in Review

62 Upvotes

r/databasedevelopment • • Dec 09 '24

Building a Database From Scratch - SimpleDB

60 Upvotes

Hello everybody, I started a learning project, to build a simple relational database from scratch and document everything on Youtube so folks can follow along.

As part one I implemented a simple file manager, you can check it out here: https://youtu.be/kj4ABYRI_NA

Here is an intro video to the whole series: https://youtu.be/pWeY93KhF4Q

In the next part, I'm implementing a log manager.


r/databasedevelopment • • Feb 24 '25

Built a database from scratch in Go

57 Upvotes

Recently i created a minimal persistent relational database in Go. Main focus was on implementing & understanding working the of database, storage management & transaction handling. Use of B+ Tree for storage engine(support for indexing), managing a Free List (for reusing nodes), Supoort for transactions, Concurrent Reads.
Still have many things to add & fix like query processing being one of the main & fixing some bugs

Repo link - https://github.com/Sahilb315/AtomixDB

Would love to hear your thoughts


r/databasedevelopment • • Sep 14 '25

Cachey, a read-through cache for S3

Thumbnail
github.com
54 Upvotes

Cachey is an open source read-through cache for S3-compatible object storage.

It is written in Rust with a hybrid memory+disk cache powered by foyer, accessed over a simple HTTP API. It runs as a self-contained single-node binary – the idea is to distribute yourself and lean on client-side logic for key affinity and load balancing.

If you are building something heavily reliant on object storage, the need for something like this is likely to come up! A bunch of companies have talked about their approaches to distributed caching atop S3 (such as Clickhouse, Turbopuffer, WarpStream, RisingWave, Chroma).

Why we built it

Recent records in s2.dev are owned by a designated process for each stream, and we could return them for reads with minimal latency overhead once they were durable. However this limited our scalability in terms of concurrent readers and throughput, as well as implied cross-zone network costs when the zones of the gateway and stream-owning process did not align.

The source of durability was S3, so there was a path to slurping recently-written data straight from there (older data would already be read directly), and take advantage of free bandwidth. But even S3 has RPS limits, and avoiding the latency overhead as much as possible is desirable.

Caching helps reduce S3 operation costs, improves the latency profile, and lifts the scalability ceiling. Now, regardless of whether records are recent or old, our reads always flow through Cachey.

Cachey internals

  • It borrows an idea from OS page caches by mapping every request into a page-aligned range read. This did call for requiring the typically-optional Range header, with an exact byte range.

    • Standard tradeoffs around picking page sizes apply, and we went with fixing it at the high end of S3's recommendation (16 MB).

    • If multiple pages are accessed, some limited intra-request concurrency is used.

    • The sliced data is sent as a streaming response.

  • It will coalesce concurrent requests to the same page (another thing an OS page cache will do). This was easy since foyer provides a native fetch API that takes a key and thunk.

  • It mitigates the high tail latency of object storage by maintaining latency statistics and making a duplicate request when a configurable quantile is exceeded, picking whichever response becomes available first. Jeff Dean discussed this technique in The Tail at Scale, and S3 docs also suggest such an approach.

A more niche thing Cachey lets you do is specify more than 1 bucket an object may live on, and attempt up to 2, prioritizing the client's preference blended with its own knowledge of recent operational stats. This is actually something we rely on since we offer regional durability with low latency by ensuring a quorum of zonal S3 express buckets for recently-written data, so the desired range may not exist on an arbitrary one. This capability may end up making sense to reuse for multi-region durability in future, too.

I'd love to hear your feedback and suggestions! Hopefully other projects will also find Cachey to be a useful part of their stack.


r/databasedevelopment • • Feb 15 '25

What is your favorite podcast on tech, databases, or distributed systems?

51 Upvotes

We all love databases, and tech in this sub. So I guess many people share the same area of interests, and we can share our favorite podcasts on these topics.

Personally, I could name a few tech podcasts, which I listen on regular basis:

  1. DeveloperVoices - https://www.youtube.com/@DeveloperVoices - general tech podcast (not just about databases, or distributed systems), but many episodes related somehow to it.

  2. TheGeekNarrator - https://www.youtube.com/@TheGeekNarrator/podcasts - interviews with people (mostly, startup founders) about their database related projects/products.

  3. Disseminate - https://disseminatepodcast.podcastpage.io/ - interviews with people from academia, who is working on database related research


r/databasedevelopment • • Dec 14 '25

Lessons from implementing a crash-safe Write-Ahead Log

Thumbnail
unisondb.io
48 Upvotes

I wrote this post to document why WAL correctness requires multiple layers (alignment, trailer canary, CRC, directory fsync), based on failures I ran into while building one.


r/databasedevelopment • • Jan 20 '25

Building a Database from Scratch (part 03) - Log Manager

46 Upvotes

Hello folks, here is part 3 of my Building a Database from the Scratch series.

In this part, I implemented the log manager, a component that is used to do write-ahead logging. The component just provides the mechanism to log records safely and durably and the ability to go over the records.

If you're interested in checking all the details, here is the link to the video: https://youtu.be/NXafQ-jFCN0

Hope you find it interesting and useful.


r/databasedevelopment • • Apr 21 '25

Career advice for database developer

43 Upvotes

Hi everyone,
I am a postgres c extenstion developer and I've been feeling stuck for a while now. I started my career two years ago on this. I like my job and I am good at it but not sure how to switch out of current company. My reason to switch is to get more salary and build more technical expertise in a bigger company. Currently I feel whatever I could learn at my current work place is done and knowledge growth is now very slow.

My doubts:

  1. What is my job even called? what ever database role I search, I only get sql queries related stuff

  2. How should I prepare for my interview? should I focus on DSA? postgres internals? OS stuff?

  3. How relevant will DB development be in AI world?

  4. How do I target companies? How does my resume need to look like?

I appreciate all of your answers and any kind of suggestions regarding career growth. Thank you


r/databasedevelopment • • Dec 22 '24

How bloom filters made SQLite 10x faster

Thumbnail avi.im
43 Upvotes

r/databasedevelopment • • Nov 30 '23

Write throughput differences in B-tree vs LSM-tree based databases?

43 Upvotes

Hey folks, I am reading about the differences between B-Trees and LSM trees. What I understand is the following (please correct me if I'm wrong):

  • B-trees are read-optimized and not so write-optimized. Writes are slower because updating a B-Tree means random IO. Reads are fast though because you just have to find the node containing the data.

  • LSM trees are write-optimized and not so read-optimized. Writes are faster because data is written to immutable segments on disk in a sequential manner. Reads then become slower because that means reconciliation between multiple segments on disk.

All this makes sense, but where I'm stuck is that both B-tree based databases and LSM tree based databases would have a write-ahead log(which is sequential IO). Agreed that writing to a B-tree would be slower compared to an LSM tree , but the actual writes to disk happen asynchronously (right?). From the client's perspective, once the write is written to the write-ahead log, the commit is complete. So my question is - why would you observe a higher write throughput(from the client's perspective) with LSM-based databases?


r/databasedevelopment • • Jan 05 '25

Looking for suggestions on how to slowly get into publishing papers (industry background)

42 Upvotes

I joined a FAANG company immediately after completing my graduate studies and have accumulated nearly 10 years of industry experience, primarily working with distributed systems and databases. Recently, I've realized that despite my technical background, I have limited published work to showcase. I'm interested in hearing from others who began their publishing journey from an industry rather than academic background - what was your approach to getting started?


r/databasedevelopment • • Jan 24 '25

Database development is not for the faint of heart

41 Upvotes

Ever time I see an article like this, it's from a database developer! No other software product pushes the boundary of hardware, drivers, programming languages, compilers, and os.

https://www.edgedb.com/blog/c-stdlib-isn-t-threadsafe-and-even-safe-rust-didn-t-save-us


r/databasedevelopment • • Apr 22 '23

The “Build Your Own Database” book is finished | Blog | build-your-own.org

Thumbnail
build-your-own.org
41 Upvotes

r/databasedevelopment • • May 27 '25

My minimalist home-made C++ database

40 Upvotes

Hi,

After 10 years of development, I am releasing a stable version of Joedb, the Journal-Only Embedded Database:

I am a C++ programmer who wanted to write data to files with proper ACID transactions, but was not so enthusiastic about using SQL from C++. I said to myself it should be possible to implement ACID transaction in a lower-level library that would be orders of magnitude less complex than a SQL database, and still convenient to use. I developed this library for my personal use, and I am glad to share it.

While being smaller than popular json libraries, joedb provides powerful features such as real-time synchronous or asynchronous remote-backup (you can see demo videos at the bottom of the intro page linked above). I am working in the field of machine learning, and am using joedb to synchronize machines for large distributed calculations. From a 200Gb image database to very small configuration files, I am in fact using joedb whenever I have to write anything to a file, and appreciate its ability to cleanly handle concurrency, durability, and automatic schema upgrades.

I discovered this forum recently, and I fixed my MacOS fsync thanks to information I found here. So thanks for sharing such valuable information. I would be glad to talk about my database with you.


r/databasedevelopment • • Dec 27 '24

Building a Database from Scratch (part 02) - Memory Management Principles

40 Upvotes

Hello folks, I published part 2 of my Building a DB from scratch series and this video is a bit theoretical.

I try to explain the main principles of database memory management and how they drive the design and the implementation of more-or-less the entire database engine, and the two principles I cover are:

- Minimize Disk Access

- Don't Rely on OS Virtual Memory

In case you're interested in all the details, here is the link to the video: https://youtu.be/TYBwOLlMLnI

I will appreciate all the feedback. Thanks