r/databasedevelopment • u/eatonphil • 13h ago
r/databasedevelopment • u/eatonphil • May 11 '22
Getting started with database development
This entire sub is a guide to getting started with database development. But if you want a succinct collection of a few materials, here you go. :)
If you feel anything is missing, leave a link in comments! We can all make this better over time.
Books
Designing Data Intensive Applications
Readings in Database Systems (The Red Book)
Courses
The Databaseology Lectures (CMU)
Introduction to Database Systems (Berkeley) (See the assignments)
Build Your Own Guides
Build your own disk based KV store
Let's build a database in Rust
Let's build a distributed Postgres proof of concept
(Index) Storage Layer
LSM Tree: Data structure powering write heavy storage engines
MemTable, WAL, SSTable, Log Structured Merge(LSM) Trees
WiscKey: Separating Keys from Values in SSD-conscious Storage
Original papers
These are not necessarily relevant today but may have interesting historical context.
Organization and maintenance of large ordered indices (Original paper)
The Log-Structured Merge Tree (Original paper)
Misc
Architecture of a Database System
Awesome Database Development (Not your average awesome X page, genuinely good)
The Third Manifesto Recommends
The Design and Implementation of Modern Column-Oriented Database Systems
Videos/Streams
Database Programming Stream (CockroachDB)
Blogs
Companies who build databases (alphabetical)
Obviously companies as big AWS/Microsoft/Oracle/Google/Azure/Baidu/Alibaba/etc likely have public and private database projects but let's skip those obvious ones.
This is definitely an incomplete list. Miss one you know? DM me.
Credits: https://twitter.com/iavins, https://twitter.com/largedatabank
r/databasedevelopment • u/drdrxp • 1d ago
Relaxed reads in Raft: wait for the leader's no-op entry instead of the commit index
In standard Raft, a linearizable read waits until the state machine has applied the log up to the commit index. A relaxed read waits only for a lower bound: the no-op entry the leader wrote when it was elected. That entry is almost always applied already, so a relaxed read almost never waits, and it is still linearizable.
The standard ReadIndex makes a read see every log committed before the read arrives, including writes that have not replied to the client yet. Linearizability does not require the read to see such writes. A write that has replied is already readable from the leader's state machine, and every entry from earlier leaders comes before the no-op entry.
p50 read latency from the OpenRaft benchmark:
| Apply delay per batch | Standard read | Relaxed read |
|---|---|---|
| none | ~1,100–1,300 µs | ~1,100–1,300 µs |
| 500 µs (range over two runs) | 1,212–2,008 µs | 925–956 µs |
| 2 ms (apply can't keep up) | 2.3 s | 1,213 µs |
Limitation: it works only for reads on the leader. A read on a follower falls back to a standard read, and standard follower reads can get slower with relaxed reads enabled. Leader-only reads also need a term check on the last applied log; the design doc has a counterexample showing why.
The approach was proposed by neog24. It ships in OpenRaft 0.10.1 as an opt-in read mode.
r/databasedevelopment • u/Reasonable-Tour-8246 • 2d ago
Is it worth learning more about database internals right now? (9 months in it)
For the past 9 months, I have been learning about databases purely about them. I started off focusing on database administration concept, then switched over to database internals. I began by understanding how hard disks and RAM actually work, and since then I tried to built my own page manager, buffer pool manager, and heap manager.
I want to keep learning, so I would love to get some answers on a few things from indrustry experts:
- Is it worth learning database internals right now, especially in today's AI era?
- Are there actual jobs for this kind of role?
- What is the typical salary range for remote jobs in this niche?
- What should I focus on learning next to get really good at this?
Thanks in advance for any advice!
r/databasedevelopment • u/eatonphil • 6d ago
Next-Level Benchmarking Overfitting
databasearchitects.blogspot.comr/databasedevelopment • u/linearizable • 12d ago
Building the Gravity Query Engine (pgrust)
r/databasedevelopment • u/linearizable • 13d ago
MySQL large-transaction binary log optimization
r/databasedevelopment • u/AutoModerator • 15d ago
Monthly Release and Update Thread
This subreddit is primarily for discussing the implementation of databases, and not about sharing release announcements (either for the first time or your updates).
This thread is the exception!
Please tell us about the new database you (or your agent) built. Tell us about all the cool new features you added. Tell us about anything else you learned or worked on that you haven't gotten around to blogging about yet.
r/databasedevelopment • u/Much-Illustrator-650 • 16d ago
RamCourt Design Decisions
Hello everyone,
I'm the RamCourt author, and I decided to make a discussion about where RamCourt matches/differs from RAMCloud, the key-value storage solution it was inspired by. You can find that here:
https://github.com/OferMania/ramcourt-draft/discussions/13
Additional info: RamCourt grew out of my years working on RAMCloud at Stateless. RamCourt is an early-stage Rust key-value store: single node so far, with mandatory version-checks and expiry implemented, and replication and recovery not built yet.
r/databasedevelopment • u/IndicationAntique667 • 29d ago
I Tried to Find How ClickHouse Implements String Hash Tables
Hi everyone,
I'm learning and exploring topics around database systems. This is a continuation from my last post here: https://www.reddit.com/r/Clickhouse/comments/1x0hu4s/i_tried_to_find_how_clickhouse_keeper_takes/
I think in one of the lectures in CMU's Database Systems course, Prof. Andy had mentioned that ClickHouse has different implementations of hash tables for different datatypes. I wanted to see what these are. So I tried to explore how String Hash Tables are implemented in ClickHouse.
I recorded a concept and code walkthrough here: https://youtu.be/wFCXI9P6Trg
A short summary of the points in the video:
Why String keys can hamper performance of hash tables
How ClickHouse uses String Adaptive Hash Tables to improve performance
The code walk through the files: StringHashTable.h
Do give it a watch and consider supporting by liking the video or subscribing :)
This is partly for my own future reference and partly to share with others who might be interested in it. Would love feedback and corrections from people who know this stuff deeply.
r/databasedevelopment • u/Much-Illustrator-650 • Oct 16 '26
Why I replaced RAMCloud-style version counters with random UUIDs (and what that doesn’t solve)
I am the RamCourt author. RamCourt grew out of my years working on RAMCloud at Stateless. RamCourt is an early-stage Rust key-value store: single node so far, with mandatory version-checks and expiry implemented, and replication and recovery not built yet. The part I most want feedback on is the last section: how to order a recreated key against its tombstone when the client-visible version is random, and whether the internal sequence should be per key or table-wide. Happy to be told where I’m wrong. Discussion thread here: https://github.com/OferMania/ramcourt-draft/discussions/7
r/databasedevelopment • u/louischoi • Oct 17 '26
Tuple key (primary key) really sould be mutable? I dont think so,
Hi, Im UkJin CHOI from South Korea, Im really passinate for building database engine. Claude code makes it possible. and I found this channel. I worked as DBA and Data engineer for years.
While building many services with PostgreSQL, SQLite, Azure SQL. I found that there are rarely cases in which primary key were update in buisiness logic.
Mutable keys expand undo logs, deepen WAL record and plus complecity.
So Im building a database with immutable tuple key specialized in OLTP database for 2month with Claude code.
Minimum feature were finished like SQLs, Catalog, Transaction (ReadRepeatable, Read Committed), MVCC, WAL, Indexes, Forein Key. Dropping them keeps the engine simpler and faster.
What do you think?
Am I on the proper way? do you think that it will works and company adopt it rather than PostgreSQL if it is mature?
I really appreciate any advice and opinion!
Thank you.
r/databasedevelopment • u/vinaykakade • Oct 15 '26
What changes when your search index lives on S3?
r/databasedevelopment • u/eatonphil • Oct 14 '26
Building a Linearizable Log on READ COMMITTED Isolation
r/databasedevelopment • u/Low_Brilliant_2597 • Oct 13 '26
Interview with Deepthi Sigireddi, Head of Databases at Supabase
Read the story of Deepthi Sigireddi and how she came into the world of databases and has spent more than 20 years scaling databases through Vitess at PlanetScale and now Multigres at Supabase. She's among the people who have worked a lot on scaling databases and are still working on some really cool database challenges.
Also, if you love learning about databases, The Consensus is really an amazing resource and has really great blogs about various topics of database systems.
https://theconsensus.dev/p/2026/10/08/deepthi-sigireddi-from-supabase.html?focus=1
r/databasedevelopment • u/rcron • Oct 10 '26
Fencing an LSM on object storage: how SlateDB works and what breaks on "S3-compatible" stores
I'm using SlateDB myself to save few bucks on RAM, here's some information about how it works internally. It's like a duckdb/leveldb, so it's embeded, but has persistent storage on S3, and I feel like we going into a "Storage Storage everything" era but not all S3 are the same, so there's some bugs I got as well when running under DigitalOcean Spaces
r/databasedevelopment • u/diagraphic • Oct 09 '26
Keys and Values Don't Always Belong Together
r/databasedevelopment • u/Mindless-Piece-47 • Oct 08 '26
MariaDB Foundation Adds PostgreSQL to Its Engine‑Agnostic Testing Framework (TAF).
MariaDB Foundation continues expanding TAF with PostgreSQL support
More engines, more reproducibility, more value for contributors and the ecosystem.
https://mariadb.org/mariadb-foundation-adds-postgresql-to-its-engine-agnostic-testing-framework-taf/
Have fun benchmarking!
r/databasedevelopment • u/IndicationAntique667 • Oct 08 '26
I Tried to Find How ClickHouse Keeper Takes Non-Blocking Snapshots
Hi everyone,
I'm learning and exploring topics around database systems. I wanted to see how ClickHouse Keeper takes a non-blocking snapshot of its storage.
I recorded a concept and code walkthrough here: https://www.youtube.com/watch?v=7J93WKEFinA&t
A short summary of the points in the video:
Brute force approaches and why they don't work
How ClickHouse Keeper uses HashMap + Doubly Linked List to solve this problem
The code walk through the files: SnapshotableHashTable.h, KeeperStateMachine.cpp and KeeperSnapshotManager.cpp to see how it's actually implemented.
Do give it a watch and consider supporting by liking the video or subscribing :)
This is partly for my own future reference and partly to share with others who might be interested in it. Would love feedback and corrections from people who know this stuff deeply.
Thank you so much!
r/databasedevelopment • u/no-bugs • Oct 07 '26
MECHLOVE Blueprint - 2/7. Storage (ZFS-style) and WAL (template-based logical with physical hints)
2nd part of the 7-part MECHLOVE Blueprints. All the criticism is extremely welcome (hey, that's the point we're publishing it - to see if it stands).
r/databasedevelopment • u/no-bugs • Oct 06 '26
Radical MVCC and Replay-Based Rebasing OCC (Re2OCC)
As a follow-up to previous MECHLOVE Part 1/7: https://6it.dev/blog/radical-mvcc-and-replay-based-rebasing-occ-re2occ-80742
r/databasedevelopment • u/eatonphil • Oct 05 '26
Thread Pool in Percona Server and MySQL (Part 1)