We’ve had a Python monolith (~200K LOC) for 8 years. Not perfect, but it handles 50K req/day fine. Rarely crashes. Easy to debug. Deploys take 8 min. New lead architect shows up, 3 months in, says it’s all gotta go. He wants 47 microservices in 6 months. The justification was basically that "monoliths don't scale," we need team autonomy, something about how a "service mesh and event bus" will make us future-proof, and that we're just digging debt deeper every day we wait.

The proposed setup is a full-blown microservices architecture with 47 services in separate repos, complete with sidecar proxies, a service mesh, and async everything running on an event bus. He's also mandating a separate database per service so goodbye atomic transactions all fronted by an API Gateway promising "eventual consistency." For our team of 25 engineers, that works out to less than half a person per service, which is crazy.

I'm already having nightmares about debugging, where a single production issue will mean tracing a request through seven different services and three message queues. On top of that, very few people on our team have any real experience building or maintaining distributed systems, and the six-month timeline is completely ridiculous, especially since we're also expected to deliver new features concurrently.

Every time I raise these points, he just shuts me down with the classic "this is how Google and Amazon do it," telling me I'm "thinking too small" and that this is all about long-term vision. and leadership is eating it up;

This feels like someone try to rebuild the entire house because the dishwasher is broken. I honestly can't tell if this is legit visionary stuff I'm just too cynical to see, or if this is the most blatant case of resume driven development ever.

I've recently bought this book, I have very less knowledge system design is it worth the read without any prerequisites.

Change my mind.

Not because microservices are bad.

But because most teams adopt them before:
- product market fit
- scale
- team scale
- operational maturity

For every team that genuinely needed microservices, i suspect there are many more that would ve been better off with a modular monolith.

Whats your experience?

At a CTO Craft Dinner in Toronto, I sat down with engineering leaders from more than a dozen tech companies and asked where AI has actually landed. The free-for-all is over and we need to be realistic.

One argument I keep hearing is that implementation is becoming commoditized while architecture becomes the real differentiator.

If that's true, should we be investing more in teaching engineers how to design systems instead of solving isolated algorithmic problems?

With AI writing more and more code, I'm wondering if grinding hundreds of LeetCode problems is still the best use of time.

For people who've recently landed jobs or switched companies:

How much did LeetCode actually matter?

Hi All,

I've been a software architect for last 12 years, 24 years yoe overall. I have worked on large enterprises as well as early stage startups.

Here are patterns I keep seeing repeatedly where projects are messed particularly in startups, which I wanted to share:

Premature microservices. Your team is 4 engineers and you have 8 services and thinking to build 4 more. You don’t have a scaling problem. You have a coordination problem. A well-structured monolith would let you move 3x faster right now. I would suggest go for modular monolith always.

No clear data ownership. Three services write to the same database table. Nobody knows which one is the source of truth. This becomes a nightmare at scale and during incidents. Again go for modular monolith, and if you want strictly then CQRS is way to go (but still overkill if you don't have that much scale)

Ignoring operational complexity. The architecture diagram looks awesome . But nobody thought about deployment, observability, or what happens at 3 AM when the message queue backs up.

Over-engineering for hypothetical scale. You have 5000 users, but only 500 MAUs. You don’t need Kubernetes, a service mesh, and event sourcing. Build for the next 10x, not the next 1000x.

Most of these are fixable without a rewrite. Usually it’s a few targeted changes that unlock the next stage of growth.

Happy to answer questions if anyone is dealing with similar challenges.

I'm practicing system design for FAANG interviews and holy shit, what is this depravity that i'm seeing in URL shorteners system design, why are they so much over-engineered? Is this really the bar that i need to complicate things into to pass an interview?

You really don't need 3 separate dbs, separate write/read services and 10 different layers for such a simple service.

My computer's old i7 can handle ~200k hashes per second. Any serious 16-32 core box can make multiple million hashes per second. I won't even get into GPU hashing (for key lookup).

1 million requests per second pretty much translates to 1-2 GB/s. Easily achievable by pretty much most network cards.
2-3 Billion unique urls are... 300-400 GB? mate you can even host everything on the memory if you wanted.

I mean such a service can be solo hosted on a shitbox in the middle of nowhere and handle so much traffic. The most you want is maybe a couple of redundancies. You can even just make a default hash map without any database solution.

Setting up ssl connection for high requests per second is more compute heavy than the entire service

It baffled me for years that software teams I worked with treated diagrams like an after thought. Maybe not quite the writing documentation level but still dreaded.

Loss of detail due to abstraction and maintainability are the main problems I noticed as a blocker to their wider-spread use.

Free-draw style apps and the output diagrams often end up as screenshots and lost somewhere, get outdated quickly and either become a noisy mess or too much abstraction.

I often gravitate towards dropping to a lower abstraction level to flesh things out.

I like the C4 model approach but I think it is too restrictive. The idea is impeccable, but in practice does not work, maybe its just me. Rather than thinking about the system I often ended up thinking how to best model the architecture to fit into a diagram. Which is unacceptable friction from a tool, in my opinion, tools should be meeting the user where they are.

I thought I could ease some of these with tooling so I created one for personal use but not-sure how far I got because I stared at it too long. Feature-set kind of exploded as I tried to integrate some AI into my old-ways to not fall behind, ended up spending more time to fix it then I like to admit but its at a "okay" polish state to share. If you want to give it a shot https://github.com/Mertcikla/tld

If you want to share your workflow, experience or share tips on how to maintain them. I really would love to discuss as I have been wrestling with this for a while now.

I’ve been working as a Software Architect primarily in the .NET ecosystem for a while, and I’ve noticed a fascinating trend: The architectural "culture war" seems to be cooling down. A few years ago, every conference was shouting "Microservices or death." Today, it feels like the industry leaders, top-tier courses, and senior architects have landed on the same "Golden Stack" of pragmatism. It feels like we've reached a state of Architectural Maturity.

The "Modern Standard" as I see it:

• Modular Monolith First (The Boundary Incubator): This is the default to start. It’s the best way to discover and stabilize your Bounded Contexts. Refactoring a boundary inside a monolith is an IDE shortcut; refactoring it between services is a cross-team nightmare. You don't split until you know your boundaries are stable.

• The Internal Structure: The "Hexagonal" (Ports & Adapters) approach has won. If the domain logic is complex, Clean Architecture and DDD (Domain-Driven Design) are the gold standards to keep the "Modulith" maintainable.

• Microservices as a Social Fix (Conway’s Law): We’ve finally admitted that Microservices are primarily an organizational tool. They solve the "too many cooks in the kitchen" problem, allowing teams to work independently. They are a solution to human scaling, not necessarily technical performance.

• The "Boring" Infrastructure:

  • DB: PostgreSQL for almost everything.

  • Caching: Redis is the de-facto standard.

  • Observability: OpenTelemetry (OTEL) is the baseline for logs, metrics, and traces.

• Scalability – The Two-Step Approach:

  • Horizontal Scaling: Before splitting anything, we scale the Monolith horizontally. Put it behind a load balancer, spin up multiple replicas, and let it rip. It’s easier, cheaper, and keeps data consistency simple.

  • Extraction as a Last Resort: Only carve out a module if it has unique resource demands (e.g., high CPU/GPU) or requires a different tech stack. But you pay the "Distribution Tax": The moment you extract, you must implement the Outbox Pattern to maintain consistency, alongside resiliency patterns (circuit breakers, retries) and strict idempotency across boundaries.

Is the debate over? It feels like we’ve finally settled on a pragmatic middle ground. But I wonder if this is just my .NET/C# bubble.

I’d love to hear from other ecosystems:

• Java/Spring Boot: Does the Spring world align with this "modern standard"?

• Node.js/TypeScript: With the rise of frameworks like NestJS, are you guys also moving toward strict Clean Architecture patterns, or is the "keep it lean and fast" vibe still dominant?

• Go/Rust: Are you seeing the same push toward Hexagonal patterns, or does the nature of these languages push you toward a more procedural, "flat" structure?

Is there a "Next Big Thing" on the horizon, or have we actually reached "Peak Backend Architecture" where the core principles won't change for the next decade?

Backend engineer here with mostly what I would consider mid to low-level scaling experience. I’ve never been a part of a team that has had to process billions of simultaneous data points. Millions of daily Nginx visitors is more my experience.

When I look at something like Apple’s FindMy app, I’m legitimately blown away. Within about 3-5 seconds of opening the app, each one of my family members locations gets updated. If I click on one of them, I’m tracking their location in near real time.

I have no experience with Kinesis or streams, though our team does. And my understanding of a more typical Postgres database would likely not be up to this challenge at that scale. I look at seemingly simple applications like this and sometimes wonder if I’m a total fraud because I would be clueless on where to even start architecting that.

FULL DISCLAIMER: This is an article I wrote that I wanted to share with others, it is not spam. It's not as detailed as the original article, but I wanted to keep it short. Around 5 mins. Would be great to get your thoughts.
---

Dropbox is a cloud-based storage service that is ridiculously easy to use.

Download the app and drag your files into the newly created folder. That's it; your files are in the cloud and can be accessed from anywhere.

It sounds like a simple idea, but back in 2007, when it was released, there wasn't anything like it.

Today, Dropbox has around 700 million users and stores over 550 billion files.

All these files need to be organized, backed up, and accessible from anywhere. Dropbox uses virtual servers for this. But they often got overloaded and sometimes crashed.

So, the team at Dropbox built a solution to manage server loads.

Here's how they did it.

Why Dropbox Servers Were Overloaded

Before Dropbox grew in scale, they used a traditional system to balance load.

This likely used a round-robin algorithm with fixed weights.

So, a user or client would upload a file. The load balancer would forward the upload request to a server. Then, that server would upload the file and store it correctly.

---

Sidenote: Weighted Round Robin

A round-robin is a simple load-balancing algorithm*. It works by cycling requests to different servers so they get an equal share of the load.*

If there are three servers, A, B, C, and three requests come in. A gets the first, B gets the second, and C gets the third.

Weighted round robin is a level up from round robin. Each server is given a weight based on its processing power and capacity*.*

Static weights are assigned manually by a network admin. Dynamic weights are adjusted in real time by a load balancer.

The higher the weight, the more load the server gets.

So if A has a weight of 3, B has 2, C has 1, and there were 12 requests. A would get 6, B would get 4, and C would get 2.

---

But there was an issue with their traditional load balancing approach.

Dropbox had many virtual servers with vastly different hardware. This made it difficult to distribute the load evenly between them with static weights.

This difference in hardware could have been caused by Dropbox using more powerful servers as it grew.

They may have started with an average server. As it grew, the team acquired more powerful servers. As it grew more, they acquired even more powerful ones.

At the time, there was no off-the-shelf load-balancing solution that could help. Especially one that used a dynamic weighted round-robin with gRPC support.

So, they built their own, which they called Robinhood.

---

Sidenote: gRPC

Google Remote Procedure Call (gRPC) is a way for different programs to talk to each other. It's based on RPC, which allows a client to run a function on the server simply by calling it*.*

This is different from REST*, which requires communication via a URL. REST also focuses on the resource being accessed instead of the action that needs to be taken.*

But gRPC has many more differences between REST and regular RPC.

The biggest one is the use of protobufs*. This file format* developed by Google is used to store and send data*.*

It works by encoding structured data into a binary format for fast transmission. The recipient then decodes it back to structured data. This format is also much smaller than something like JSON.

Protobufs are what make gRPC fast*, but also more difficult to set up since the client and server need to support it.*

gRPC isn't supported natively by browsers. So, it's commonly used for internal server communication*.*

---

The Custom Load Balancer

The main component of RobinHood is the load balancing service or LBS. This manages how requests are distributed to different servers.

It does this by continuously collecting data from all the servers. It uses this data to figure out the average optimal resource usage for all the servers.

Each server is given a PID controller, a piece of code to help with resource regulation. This has an upper and lower server resource limit close to the average.

Say the average CPU limit is 70%. The upper limit could be 75%, and the lower limit could be 65%. If a server hits 75%, it is given fewer requests to deal with, and if it goes below 65%, it is given more.

This is how the LBS gives weights to each server. Because the LBS uses dynamic weights, a server that previously weighted 5 could become 1 if its resources go above the average.

In addition to the LBS, Robinhood had two other components: the proxy and the routing database.

The proxy sends server load data to the LBS via gRPC.

Why doesn't the LBS collect this itself? Well, the LBS is already doing a lot.

Imagine there could be thousands of servers. It would need to scale up just to collect metrics from all of them.

So, the proxy has the sole responsibility of collecting server data to reduce the load on the LBS.

The routing database stores server information. Things like weights generated by the LBS, IP addresses, hostname, etc.

Although the LBS stores some data in memory for quick access, an LBS itself can come in and out of existence; sometimes, it crashes and needs to restart.

The routing database keeps data for a long time, so new or existing LBS instances can access it.

Routing databases can either be Zookeeper or etcd based. The decision to choose one or the other may be to support legacy systems.

---

Sidenote: Zookeeper vs etcd

Both Zookeeper and etcd are what's called a distributed coordination service*.*

They are designed to be the central place where config and state data is stored in a distributed system.

They also make sure that each node in the system has the most up-to-date version of this data*.*

These services contain multiple servers and elect a single server, called a leader, that takes all the writes.

This server copies the data to other servers, which then distribute the data to the relevant clients. In this case, a client could be an LBS instance.

So, if a new LBS instance joins the cluster, it knows the exact state of all the servers and the average that needs to be achieved.

There are a few differences between Zookeeper and etcd.

---

After Dropbox deployed RobinHood to all their data centers, here is the difference it made.

The X axis shows date in MM/DD and the Y axis shows the ratio of CPU usage compared to the average. So, a value of 1.5 means CPU usage was 1.5 times higher than the average.

You can see that at the start, 95% of CPUs were operating at around 1.17 above the average.

It takes a few days for RobinHood to regulate everything, but after 11/01, the usage is stabilized, and most CPUs are operating at the average.

This shows a massive reduction in CPU workload, which indicates a better-balanced load.

In fact, after using Robinhood in production for a few years, the team at Dropbox has been able to reduce their server size by 25%. This massively reduced their costs.

It isn't stated that Dropbox saved millions annually from this change. But, based on the cost and resource savings they mentioned from implementing Robinhood, as well as their size.

It can be inferred that they saved a lot of money, most likely millions from this change.

Wrapping Things Up

It's amazing everything that goes on behind the scenes when someone uploads a file to Dropbox. I will never look at the app in the same way again.

I hope you enjoyed reading this as much as I enjoyed writing it. If you want more details, you can check out the original article.

And as usual, be sure to subscribe to get the next article sent straight to your inbox.

I’ve been working on a web app where you can design distributed systems and actually simulate behavior, not just draw boxes.

What it does

• Drag/drop architecture components (API GW, LB, app, cache, DB, queues, etc.)

• Connect flows visually

• Run traffic simulation (inflow → processing → outflow)

• Inject chaos events and see impact

• Diagnose bottlenecks/failures and iterate

Why I built it

Most system design tools stop at diagrams. I wanted something that helps answer:

• “What breaks first?”

• “How does traffic behave under stress?”

• “What happens when chaos is injected?”

Tech highlights

• Flutter web app

• Canvas-based architecture editor

• Simulation engine with lifecycle modeling + diagnostics

• Chaos inference/synergy logic

• Real-time metrics feedback

Would love feedback from this community on:

1. What scenarios should I add next?

2. Which metrics are most useful in interviews vs real systems?

3. What would make this genuinely useful for practicing system design?

Site: https://paperdraw.dev/?utm_source=reddit.com

You know that moment when you hit “Send” on WhatsApp—and your message just zips across the world in milliseconds? No lag, no wait, just instant delivery.

I wanted to challenge myself: What if I had to build that exact experience from scratch?
No bloated microservices, no hand-wavy answers—just real engineering.

I started breaking it down.

First, I realized the message flow isn’t as simple as “Client → Server → Receiver.” WhatsApp keeps a persistent connection, typically over WebSocket, allowing bi-directional, real-time communication. That means as soon as you type and hit send, the message goes through a gateway, is queued, and forwarded—almost instantly—to the recipient.

But what happens when the receiver is offline?
That’s where the message queue comes into play. I imagined a Kafka-like broker holding the message, with delivery retries scheduled until the user comes back online. But now... what about read receipts? Or end-to-end encryption?

Every layer I peeled off revealed five more.

Then I hit the big one: encryption.
WhatsApp uses the Signal Protocol—essentially a double ratchet algorithm with asymmetric keys. The sender encrypts a message on their device using a shared session key, and the recipient decrypts it locally. Neither the WhatsApp server nor any man-in-the-middle can read it.

Building this alone gave me an insane confidence for just how layered this system is:
✔️ Real-time delivery
✔️ Network resilience
✔️ Encryption
✔️ Offline handling
✔️ Low power/bandwidth usage

Designing WhatsApp: A Story of Building a Real-Time Chat System from Scratch
WhatsApp at Scale: A Guide to Non-Functional Requirements

I ended up writing a full system design breakdown of how I would approach building this as an interview-level project. If you're curious, give it a shot and share your thoughts and if preparing for an interview its must to go through it

https://medium.com/@nishant.gargvii/faang-staff-level-google-calendar-system-design-0a6f884811c9

Hello everyone,

As a developer passionate about software architecture, I've noticed there's a real lack of dedicated tools for DDD workshops, especially in the context of remote work.

I decided to create a platform bringing together all the essential tools for Domain-Driven Design practice.

My project currently offers two main tools:

• Domain Storytelling: to visualize and communicate domain knowledge through collaborative stories

• Event Storming: to quickly discover business processes and identify bounded contexts

More tools will be added later to expand the toolbox.

It's free, open-source, and specifically designed for DDD practitioners.

GitHub project: https://github.com/poulainpi/ddd-toolbox

If you like the project, feel free to give it a ⭐ to support the development!

Mine was idempotency.

I used to think retries were enough.

Then I started working with:
- Payment webhooks
- Background workers
- Event-driven systems
- Push notifications

Eventually I realized retries are only safe if the operation itself can be repeated without changing the outcome.

That one concept changed how I think about APIs, message processing, and distributed systems.

What's the one backend concept that permanently changed how you build software?

We ran into something recently that made me rethink a system design decision while working on an event-driven architecture. We have multiple Kafka topics and worker services chained together, a kind of mini workflow.

The entry point is a legacy system. It reads data from an integration database, builds a JSON file, and publishes the entire file directly into the first Kafka topic.

The problem

One day, some of those JSON files started exceeding Kafka’s default message size limit. Our first reaction was to ask the DevOps team to increase the Kafka size limit. It worked, but it felt similar to increasing a database connection pool size.

Then one of the JSON files kept growing. At that point, the DevOps team pushed back on increasing the Kafka size limit any further, so the team decided to implement chunking logic inside the legacy system itself, splitting the file before sending it into Kafka.

That worked too, but now we had custom batching/chunking logic affecting the stability of an existing working system.

The solution

While looking into system design patterns, I came across the Claim-Check pattern.

Instead of batching inside the legacy system, the idea is to store the large payload in external storage, send only a small message with a reference, and let consumers fetch the payload only when they actually need it.

The realization

What surprised me was realizing that simply looking into existing system design patterns could have saved us a lot of time building all of this.

It’s a good reminder to pause and check those patterns when making system design decisions, instead of immediately implementing the first idea that comes to mind.

Hi
Excited to be able to announce that QO is almost ready to leave Early Access! This week I published a large patch that covers more than a year of work (lots of analytics, I've been tracking where ppl were getting stuck). Thank you a ton for your support, this game has seen a lot of love from this community. Game is almost done.

If you are interested in a highly intuitive visual method that faithfully describes all universal quantum computing and physics behind, this is for you. I am the Dev behind Quantum Odyssey (AMA! I love taking qs) - worked on it for about 10 years (3.5 in phd), the goal was to make a super immersive space for anyone to learn quantum computing through zachlike (open-ended) logic puzzles and compete on leaderboards and lots of community made content on finding the most optimal quantum algorithms. The game has a unique set of visuals (that was actually my PhD research) capable to represent any sort of quantum dynamics for any number of qubits and this is pretty much what makes it now possible for anybody 15yo+ to actually learn quantum logic without having to worry at all about the mathematics behind.

This is a game super different than what you'd normally expect in a programming/ logic puzzle game, so try it with an open mind.

Stuff covered

• Boolean Logic – bits, operators (NAND, OR, XOR, AND…), and classical arithmetic (adders). Learn how these can combine to build anything classical. You will learn to port these to a quantum computer.

• Quantum Logic – qubits, the math behind them (linear algebra, SU(2), complex numbers), all Turing-complete gates (beyond Clifford set), and make tensors to evolve systems. Freely combine or create your own gates to build anything you can imagine using polar or complex numbers.

• Quantum Phenomena – storing and retrieving information in the X, Y, Z bases; superposition (pure and mixed states), interference, entanglement, the no-cloning rule, reversibility, and how the measurement basis changes what you see.

• Core Quantum Tricks – phase kickback, amplitude amplification, storing information in phase and retrieving it through interference, build custom gates and tensors, and define any entanglement scenario. (Control logic is handled separately from other gates.)

• Famous Quantum Algorithms – explore Deutsch–Jozsa, Grover’s search, quantum Fourier transforms, Bernstein–Vazirani, and more.

• Build & See Quantum Algorithms in Action – instead of just writing/ reading equations, make & watch algorithms unfold step by step so they become clear, visual, and unforgettable. Quantum Odyssey is built to grow into a full universal quantum computing learning platform. If a universal quantum computer can do it, we aim to bring it into the game, so your quantum journey never ends.

Streams to watch:

khan academy style tutorials on qm/qc: https://www.youtube.com/@MackAttackx

Physics teacher wholesome stream with over 500hs in https://www.twitch.tv/beardhero

We made so many mistakes trying to mimic FAANG and adopt microservices back when the approach was new and cool. We ended up with an approach somewhere between microservices and monoliths for our v2, and learned to play to our strengths and deleted 2.3M lines of code along the way.