r/databasedevelopment • • Aug 16 '24

Database Startups

Thumbnail transactional.blog
29 Upvotes

r/databasedevelopment • • May 11 '22

Getting started with database development

411 Upvotes

This entire sub is a guide to getting started with database development. But if you want a succinct collection of a few materials, here you go. :)

If you feel anything is missing, leave a link in comments! We can all make this better over time.

Books

Designing Data Intensive Applications

Database Internals

Readings in Database Systems (The Red Book)

The Internals of PostgreSQL

Courses

The Databaseology Lectures (CMU)

Database Systems (CMU)

Introduction to Database Systems (Berkeley) (See the assignments)

Build Your Own Guides

chidb

Let's Build a Simple Database

Build your own disk based KV store

Let's build a database in Rust

Let's build a distributed Postgres proof of concept

(Index) Storage Layer

LSM Tree: Data structure powering write heavy storage engines

MemTable, WAL, SSTable, Log Structured Merge(LSM) Trees

Btree vs LSM

WiscKey: Separating Keys from Values in SSD-conscious Storage

Modern B-Tree Techniques

Original papers

These are not necessarily relevant today but may have interesting historical context.

Organization and maintenance of large ordered indices (Original paper)

The Log-Structured Merge Tree (Original paper)

Misc

Architecture of a Database System

Awesome Database Development (Not your average awesome X page, genuinely good)

The Third Manifesto Recommends

The Design and Implementation of Modern Column-Oriented Database Systems

Videos/Streams

CMU Database Group Interviews

Database Programming Stream (CockroachDB)

Blogs

Murat Demirbas

Ayende (CEO of RavenDB)

CockroachDB Engineering Blog

Justin Jaffray

Mark Callaghan

Tanel Poder

Redpanda Engineering Blog

Andy Grove

Jamie Brandon

Distributed Computing Musings

Companies who build databases (alphabetical)

Obviously companies as big AWS/Microsoft/Oracle/Google/Azure/Baidu/Alibaba/etc likely have public and private database projects but let's skip those obvious ones.

This is definitely an incomplete list. Miss one you know? DM me.

Credits: https://twitter.com/iavins, https://twitter.com/largedatabank


r/databasedevelopment • • 13h ago

Please Implement A Quorum Database

Thumbnail transactional.blog
10 Upvotes

r/databasedevelopment • • 1d ago

Relaxed reads in Raft: wait for the leader's no-op entry instead of the commit index

5 Upvotes

In standard Raft, a linearizable read waits until the state machine has applied the log up to the commit index. A relaxed read waits only for a lower bound: the no-op entry the leader wrote when it was elected. That entry is almost always applied already, so a relaxed read almost never waits, and it is still linearizable.

The standard ReadIndex makes a read see every log committed before the read arrives, including writes that have not replied to the client yet. Linearizability does not require the read to see such writes. A write that has replied is already readable from the leader's state machine, and every entry from earlier leaders comes before the no-op entry.

p50 read latency from the OpenRaft benchmark:

Apply delay per batchStandard readRelaxed read
none~1,100–1,300 µs~1,100–1,300 µs
500 µs (range over two runs)1,212–2,008 µs925–956 µs
2 ms (apply can't keep up)2.3 s1,213 µs

Limitation: it works only for reads on the leader. A read on a follower falls back to a standard read, and standard follower reads can get slower with relaxed reads enabled. Leader-only reads also need a term check on the last applied log; the design doc has a counterexample showing why.

The approach was proposed by neog24. It ships in OpenRaft 0.10.1 as an opt-in read mode.


r/databasedevelopment • • 2d ago

Is it worth learning more about database internals right now? (9 months in it)

16 Upvotes

For the past 9 months, I have been learning about databases purely about them. I started off focusing on database administration concept, then switched over to database internals. I began by understanding how hard disks and RAM actually work, and since then I tried to built my own page manager, buffer pool manager, and heap manager.

I want to keep learning, so I would love to get some answers on a few things from indrustry experts:

- Is it worth learning database internals right now, especially in today's AI era?

- Are there actual jobs for this kind of role?

- What is the typical salary range for remote jobs in this niche?

- What should I focus on learning next to get really good at this?

Thanks in advance for any advice!


r/databasedevelopment • • 6d ago

Next-Level Benchmarking Overfitting

Thumbnail databasearchitects.blogspot.com
10 Upvotes

r/databasedevelopment • • 12d ago

Building the Gravity Query Engine (pgrust)

Thumbnail
malisper.me
20 Upvotes

r/databasedevelopment • • 13d ago

MySQL large-transaction binary log optimization

Thumbnail
aws.amazon.com
16 Upvotes

r/databasedevelopment • • 15d ago

Monthly Release and Update Thread

10 Upvotes

This subreddit is primarily for discussing the implementation of databases, and not about sharing release announcements (either for the first time or your updates).

This thread is the exception!

Please tell us about the new database you (or your agent) built. Tell us about all the cool new features you added. Tell us about anything else you learned or worked on that you haven't gotten around to blogging about yet.


r/databasedevelopment • • 15d ago

Jepsen: ParticleDB 1.1.4

Thumbnail jepsen.io
32 Upvotes

r/databasedevelopment • • 16d ago

RamCourt Design Decisions

0 Upvotes

Hello everyone,

I'm the RamCourt author, and I decided to make a discussion about where RamCourt matches/differs from RAMCloud, the key-value storage solution it was inspired by. You can find that here:

https://github.com/OferMania/ramcourt-draft/discussions/13

Additional info: RamCourt grew out of my years working on RAMCloud at Stateless. RamCourt is an early-stage Rust key-value store: single node so far, with mandatory version-checks and expiry implemented, and replication and recovery not built yet.


r/databasedevelopment • • 29d ago

I Tried to Find How ClickHouse Implements String Hash Tables

17 Upvotes

Hi everyone,

I'm learning and exploring topics around database systems. This is a continuation from my last post here: https://www.reddit.com/r/Clickhouse/comments/1x0hu4s/i_tried_to_find_how_clickhouse_keeper_takes/

I think in one of the lectures in CMU's Database Systems course, Prof. Andy had mentioned that ClickHouse has different implementations of hash tables for different datatypes. I wanted to see what these are. So I tried to explore how String Hash Tables are implemented in ClickHouse.

I recorded a concept and code walkthrough here: https://youtu.be/wFCXI9P6Trg

A short summary of the points in the video:

  1. Why String keys can hamper performance of hash tables

  2. How ClickHouse uses String Adaptive Hash Tables to improve performance

  3. The code walk through the files: StringHashTable.h

Do give it a watch and consider supporting by liking the video or subscribing :)

This is partly for my own future reference and partly to share with others who might be interested in it. Would love feedback and corrections from people who know this stuff deeply.


r/databasedevelopment • • Oct 16 '26

Why I replaced RAMCloud-style version counters with random UUIDs (and what that doesn’t solve)

4 Upvotes

I am the RamCourt author. RamCourt grew out of my years working on RAMCloud at Stateless. RamCourt is an early-stage Rust key-value store: single node so far, with mandatory version-checks and expiry implemented, and replication and recovery not built yet. The part I most want feedback on is the last section: how to order a recreated key against its tombstone when the client-visible version is random, and whether the internal sequence should be per key or table-wide. Happy to be told where I’m wrong. Discussion thread here: https://github.com/OferMania/ramcourt-draft/discussions/7


r/databasedevelopment • • Oct 17 '26

Tuple key (primary key) really sould be mutable? I dont think so,

0 Upvotes

Hi, Im UkJin CHOI from South Korea, Im really passinate for building database engine. Claude code makes it possible. and I found this channel. I worked as DBA and Data engineer for years.

While building many services with PostgreSQL, SQLite, Azure SQL. I found that there are rarely cases in which primary key were update in buisiness logic.

Mutable keys expand undo logs, deepen WAL record and plus complecity.

So Im building a database with immutable tuple key specialized in OLTP database for 2month with Claude code.

Minimum feature were finished like SQLs, Catalog, Transaction (ReadRepeatable, Read Committed), MVCC, WAL, Indexes, Forein Key. Dropping them keeps the engine simpler and faster.

What do you think?

Am I on the proper way? do you think that it will works and company adopt it rather than PostgreSQL if it is mature?

I really appreciate any advice and opinion!

Thank you.


r/databasedevelopment • • Oct 15 '26

What changes when your search index lives on S3?

9 Upvotes

r/databasedevelopment • • Oct 14 '26

Building a Linearizable Log on READ COMMITTED Isolation

Thumbnail
materialize.com
9 Upvotes

r/databasedevelopment • • Oct 13 '26

Interview with Deepthi Sigireddi, Head of Databases at Supabase

Post image
10 Upvotes

Read the story of Deepthi Sigireddi and how she came into the world of databases and has spent more than 20 years scaling databases through Vitess at PlanetScale and now Multigres at Supabase. She's among the people who have worked a lot on scaling databases and are still working on some really cool database challenges.

Also, if you love learning about databases, The Consensus is really an amazing resource and has really great blogs about various topics of database systems.

https://theconsensus.dev/p/2026/10/08/deepthi-sigireddi-from-supabase.html?focus=1


r/databasedevelopment • • Oct 10 '26

Fencing an LSM on object storage: how SlateDB works and what breaks on "S3-compatible" stores

Thumbnail
blog.renatocron.com
13 Upvotes

I'm using SlateDB myself to save few bucks on RAM, here's some information about how it works internally. It's like a duckdb/leveldb, so it's embeded, but has persistent storage on S3, and I feel like we going into a "Storage Storage everything" era but not all S3 are the same, so there's some bugs I got as well when running under DigitalOcean Spaces


r/databasedevelopment • • Oct 09 '26

Keys and Values Don't Always Belong Together

Thumbnail
tidesdb.com
18 Upvotes

r/databasedevelopment • • Oct 08 '26

MariaDB Foundation Adds PostgreSQL to Its Engine‑Agnostic Testing Framework (TAF).

8 Upvotes

MariaDB Foundation continues expanding TAF with PostgreSQL support

More engines, more reproducibility, more value for contributors and the ecosystem.

https://mariadb.org/mariadb-foundation-adds-postgresql-to-its-engine-agnostic-testing-framework-taf/

Have fun benchmarking!


r/databasedevelopment • • Oct 08 '26

I Tried to Find How ClickHouse Keeper Takes Non-Blocking Snapshots

6 Upvotes

Hi everyone,

I'm learning and exploring topics around database systems. I wanted to see how ClickHouse Keeper takes a non-blocking snapshot of its storage.

I recorded a concept and code walkthrough here: https://www.youtube.com/watch?v=7J93WKEFinA&t

A short summary of the points in the video:

  1. Brute force approaches and why they don't work

  2. How ClickHouse Keeper uses HashMap + Doubly Linked List to solve this problem

  3. The code walk through the files: SnapshotableHashTable.h, KeeperStateMachine.cpp and KeeperSnapshotManager.cpp to see how it's actually implemented.

Do give it a watch and consider supporting by liking the video or subscribing :)

This is partly for my own future reference and partly to share with others who might be interested in it. Would love feedback and corrections from people who know this stuff deeply.

Thank you so much!


r/databasedevelopment • • Oct 07 '26

MECHLOVE Blueprint - 2/7. Storage (ZFS-style) and WAL (template-based logical with physical hints)

Thumbnail
6it.dev
6 Upvotes

2nd part of the 7-part MECHLOVE Blueprints. All the criticism is extremely welcome (hey, that's the point we're publishing it - to see if it stands).


r/databasedevelopment • • Oct 06 '26

Radical MVCC and Replay-Based Rebasing OCC (Re2OCC)

8 Upvotes

r/databasedevelopment • • Oct 05 '26

Thread Pool in Percona Server and MySQL (Part 1)

Thumbnail
percona.com
3 Upvotes