Written by Pratik Lotia

How we introduced automatically rotated workload credentials and migrated an existing service fleet to mutual TLS without a coordinated cutover.

One Service, Several Sets Of Credentials

Imagine building a new internal service. It calls another service, requires user context, reads from a database, requires access to a cloud resource and publishes events. Each dependency may have a different answer to the same question: how does the caller prove who it is and how do you manage access control?

With static credentials, the answer usually involves provisioning a secret, deciding where to store it, granting access to the consumers of the secret, setting up authorization rules and wiring it into the application. Repeat that for every dependency and for every Kubernetes cluster or cloud environment it deploys to. The service is eventually ready to run, but its credentials still need coordination across multiple teams making it a high friction and time consuming process. When a security control is cumbersome enough, your customers will often find a way to bypass it or adopt less practices which leads to other issues such leaking credentials in code, sharing tokens and lack of regular rotation.

We had accumulated versions of this problem across our infrastructure. Service authentication used static tokens. Database integrations had their own credential workflows. Services with an existing TLS requirement maintained separate certificate integrations. Depending on the system, credential onboarding could take weeks. There was also an operational problem where a service that received an unexpected burst of requests, couldn’t reliably identify the caller, let alone deny access to a caller.

The goal was a common identity system for workloads, with short-lived credentials issued and rotated automatically. We wanted the same foundation to work for service calls and infrastructure access, rather than asking each integration to introduce another secret.

Read on to find out how we succeeded in getting rid of static secrets for our modern database stack and getting 90% of our in-cluster Go grpc traffic to use authenticated calls in the form of mutual TLS.

Making The Work Worth Adopting

A project that changes how services authenticate needs support well beyond the security team. We had to make the case to leadership and partner teams that the investment would help them with product velocity. For product teams expanding an existing service, moving it to a new environment or launching a new one, use of a common identity system would significantly speed up product launches. Instead of designing another way to issue, distribute, and rotate client credentials, platform teams could build on the shared identity system and focus on deciding who should have access. This work stretched across multiple quarters and became one of the first real, sustained partnerships between Security, Platform and Product at Reddit. We chose mTLS to provide verified identities, confidentiality, and integrity through one shared integration, rather than asking teams to migrate again when they needed encrypted communication.

We started with teams that had needs like: a maintained TLS integration for sensitive services, fewer credential-provisioning steps, and a way to identify callers. Some early adopters already used TLS through a bespoke certificate path and wanted to move away from its scaling and maintenance constraints. For them, the value was replacing an existing operational dependency with something shared.

Performance was another reasonable concern. Modern TLS is fast but benchmarking your integrations for TLS connection establishment, certificate acquisition, latency and distinguishing those costs from steady-state request processing goes a long way in convincing product teams.

Adoption also needed an owner. Publishing a migration guide gave teams instructions, but it did not create room on their roadmaps. We eventually took on preparing migration changes ourselves, with service owners reviewing and validating the behavior they knew best. That decision substantially changed the amount of work we could complete.

Choosing SPIFFE And SPIRE

We had already invested in moving service communication to gRPC. Our identity model needed to fit into that foundation. We also wanted to avoid vendor lock-in. We chose SPIFFE as the identity standard and SPIRE as its implementation. SPIFFE defines how a workload identity is represented and how software obtains and verifies credentials for it. SPIRE performs the attestation and issuance work. The SPIFFE community’s book, Solving the Bottom Turtle, is a useful introduction to the problem and the design.

We considered introducing a service mesh that provides mTLS along with routing, retries, and other traffic controls. We already had investments in service discovery, transport, and application frameworks, and our immediate requirement was identity. Adopting SPIRE directly let us integrate with those systems without coupling the project to replacing the traffic platform or retrofitting a sidecar model across the fleet.

That choice left us responsible for the application integration. SPIRE would verify workloads and issue credentials while our libraries still had to retrieve them, handle rotation, configure TLS, and expose the peer identity to application code. Workloads receive a short-lived X.509 certificate carrying its SPIFFE ID which translates to the URI SAN, corresponding private-key material, and the trust bundle needed to verify peers. We use deployment and workload attributes to distinguish SPIFFE IDs at a pod level and policies can select the stable service attributes to define authorization.

Our deployment uses a nested hierarchy of issuers so workloads can share a trust foundation across k8s clusters while issuance is handled locally. We evaluated independent trust domains and federation as well. Federation provides useful isolation, but it also requires managing trust-bundle relationships as cluster environments change.

Cluster creation and node replacement are routine operations. We automated identity infrastructure setup as part of cluster provisioning, including the registration work needed for a new issuer to join the hierarchy. A security engineer should not have to manually arrange trust every time infrastructure creates a cluster.

Putting Identity Into The Service Framework

Most service owners should not need to know how to provision an intermediate CA or refresh an X.509 source. They need a supported way to start a server, connect to a dependency, and test the result.

We integrated identity into Baseplate, Reddit’s service framework, and our infrastructure-generation tooling. The infrastructure integration makes the SPIRE API available to the application. The library obtains credentials, configures the client or server, manages credential rotation without requiring a service restart and exposes authenticated peer information. Standard templates carry that setup into new services.

Lifecycle management matters just as much. During shutdown, a service may stop accepting new work while in-flight requests still make outbound calls. We encountered a case where the shared credential source could close before those calls finished. The fix was to keep it available through service draining and release it afterward. Authentication infrastructure had become part of the application lifecycle, so it needed the same care as a connection pool.

Developers also needed to exercise the integration in their normal development environment. We built on Snoodev, Reddit’s development environment, so changes could be tested with the same identity APIs used by the application. That gave migration reviews evidence beyond a successful CI compile.

The receiving service uses the verified identity for authorization. Authentication establishes who the peer is; policy determines whether that peer may connect or invoke a particular method. Keeping those responsibilities distinct helped us reason about both the API and the migration sequence.

Migrating Without A Coordinated Cutover

Our early adopters added this integration using a separate listener. A service could add a new TLS port while retaining its existing listener, then move clients over incrementally. That worked for the initial use cases. Expanding the pattern across the fleet exposed a mismatch with service discovery. Our discovery model did not represent separate plaintext and TLS destinations in the way a general dual-port rollout would require. We could change that model, or find a migration shape that kept existing addresses working. From experience, we knew that simplifying adoption will yield faster gains.

We chose a permissive listener for the general migration. It accepts a connection on the existing port, examines the initial bytes, and hands it to the appropriate TLS or plaintext path. Existing clients keep using the same address. TLS-capable clients can negotiate mTLS when the destination supports it. This let us deploy server and client support in the same migration change without requiring every upstream dependency to move first. It removed a large coordination dependency from the service graph. A migrated service could receive calls from older clients while beginning to use TLS with compatible dependencies. Our Kafka migration faced a similar constraint: change the infrastructure without requiring every client team to move together and we decided to continue a similar approach.

The compatibility path had a cost. A listener that supports TLS is not the same as one that requires it. Keeping the address and port unchanged also means port-level traffic counts cannot tell you whether a connection used TLS. We had to track the negotiated transport in application metrics and treat the eventual cutover as separate work.

Authorization needed a staged rollout too. Before this project, we did not have a complete, verified view of callers. Immediately rejecting every caller missing from a newly written policy would turn gaps in our dependency inventory into outages. Audit mode let owners compare proposed rules with observed calls before enforcement. Observing a caller is evidence that it exists, not approval that it should have access; owners still need to make that decision.

Metrics became part of the migration interface. A successful rollout meant more than a server starting: we checked credential acquisition, TLS setup, actual request traffic, and which peers were using the new path.

Large callers needed smaller batches. A service with many outbound dependencies could not be treated like a service with one client. Shared client libraries sometimes hid connection setup, and multiple logical clients could share an underlying connection. We separated library support from activation where needed, then enabled groups of dependencies and watched startup and request behavior.

Compatibility code also needed its own tests. Looking at the first bytes of a connection sounds trivial, but doing a blocking read in an accept loop can let one silent connection delay unrelated healthy clients. We fixed that behavior by separating acceptance from connection classification. The migration wrapper had to preserve the server’s availability properties as well as its protocol behavior.

Changing Who Did The Migration Work

The first approach was to find motivated adopters, help them migrate, and turn what we learned into documentation. It was useful for proving the integration, but it did not scale to every team independently finding time for another infrastructure project.

We changed the ownership model. The security team prepared the code and configuration changes, worked through validation, and brought service owners a pull request they could assess. That required understanding each service’s setup, not just changing a library version. AI-assisted coding helped with that repetitive discovery and editing. We encoded the migration process in a reusable set of instructions: inspect the service, identify clients and listeners, update infrastructure wiring, run tests, exercise the development environment, and prepare the pull request. The instructions also captured exceptions found during previous migrations.

We began with roughly five services per week. These were services we started working on, not necessarily changes merged and deployed that week. After building confidence in the process, we increased the pace to about twenty per week and expanded to higher-traffic services. AI-assisted coding, as some of you already have experienced, is not foolproof. It could update an obvious client and miss another one created through a shared library. Some libraries did not yet expose the TLS integration their consumers needed. Those cases required changes at the abstraction that owned the connection, followed by another pass over the consuming services.

We kept human review, testing, and deployment validation in the loop. Generating a plausible diff was useful, but it did not prove that the intended traffic was encrypted. The same telemetry used to track the rollout helped us find incomplete migrations and refine the instructions. Early in the rollout, TLS was often the first suspect when a migrated service had an incident. We rolled back changes where needed to restore service, then investigated; in most cases, the cause turned out to be elsewhere. As successful migrations accumulated, confidence grew and those suspicions became less frequent.

The result was a migration process that improved as it encountered more of the fleet. Automation reduced the cost of preparing changes; service knowledge and observed behavior still determined whether they were ready.

Beyond Service-To-Service Calls

A workload identity becomes more useful when other systems can consume it. A service that has already established its identity should not need to acquire another static credential merely to access a supported database or publish an event.

Different database-related backends required different approaches to support our identity model. Some systems understood URI SANs (our standard SPIFFE ID) directly. Others supported client certificates but expected a DNS subject alternative name or a subject common name (CN) when mapping a client to permissions. We added certificate composition support for those integrations, alongside extension support where additional verified attributes are needed.

Those fields were not interchangeable. A DNS SAN used by a particular database is a compatibility mechanism; it does not replace the SPIFFE identity or define a universal permission model. The issuer needs a trustworthy basis for any attribute it signs, and the receiving system still owns the authorization decision. We can share credential issuance without pretending every backend has the same access-control API.

Our TiDB integration is one production example. It supported clients to authenticate using workload credentials but we had to add upstream support to simplify mounting SPIFFE helpers and authorizing clients with regex support. It enabled us to remove the separate database credential provisioning step. Additionally, our integrations with event publishing and data infrastructure reuse the same identity foundation, while other database systems are at different stages of implementation.

Our internal LLM Platform is another early adopter. It verifies a backend workload’s identity and checks project permissions before allowing model access. A project identifier alone does not grant permission. Client services can use the platform without holding individual provider API keys, while the platform retains project-level access controls and usage attribution.

What Changed, And What Comes Next

Within four months of mass migrations, we migrated 95% of Go services by service count and 90% by in-cluster traffic coverage. For developers, the change is that supported integrations start with a workload identity and automatic credential rotation. Teams only need to decide what access a service should have, but they do not need to invent a credential lifecycle for every dependency. For operators, verified caller metrics provide a more useful starting point when investigating unexpected traffic.

We are continuing to reduce the attack surface by expanding language and backend support, extending protected communication across network boundaries, integrating SPIFFE identities with cloud-provider IAM roles and making authorization easier to apply consistently. Completing a migration also means verifying that the credential or transport it replaced is no longer needed, then retiring it. Adoption dashboards are useful, but they are not a substitute for that check.

That’s a wrap, if you enjoyed reading this article, I encourage you to check out some of my other articles:

• Zero Trust Odyssey

• Protecting Cat Memes from DDoS

• Tetragon Configuration Gotchas

• Identity Aware Proxies

• Day in the life of an Infrastructure Security engineer

Did you know we're actively hiring? If you made it this far then you're obviously interested in the types of problems we're solving. So why not solve them with us?

Written by Lukas Majercak, with thanks to Nick Garvey for his help with the investigation.
AI disclosure: all of the investigation was done before agents got good and AI was used to refine the wording.

Motivation

Over the past year, we moved Kafka from EC2 to Kubernetes using Strimzi, migrated its metadata layer from ZooKeeper to KRaft, and upgraded through several releases, from 3.6.X to 4.X.X. We tested these changes carefully, but some failure modes only emerged over time and under particular conditions.

Before Kubernetes, we rarely restarted brokers or ZooKeeper nodes outside planned maintenance. Now that we run Kafka on Kubernetes, our platform requires weekly node rotation, and pods can also be evicted during scale-down or under node resource pressure. Restarts are now therefore more frequent, with some occurring outside our maintenance schedule.

The first signs of trouble appeared in our support Slack channel. Users reported connectivity errors and growing consumer lag during cluster rolls. We initially attributed these symptoms to older clients or inadequate retry settings and our response often amounted to: “The cluster is restarting; it should finish in about an hour.” That explanation started to fall apart as reports accumulated.

We needed rolling restarts to be routine for node rotation, vertical scaling, configuration changes, and upgrades. Over time, we found that the interruptions were not confined to poorly configured clients, but some partitions experienced genuine brief server-side availability drops during rolls.

These interruptions had two costs:

1. User Impact: Write interruptions can increase latency, fill producer buffers, and cause delivery deadlines to expire. Applications that discard failed sends, or lose unacknowledged buffered records during a crash, can lose data. Read interruptions increase processing latency and consumer lag. If consumers crash, restart, or stop polling long enough to leave their groups, the resulting rebalances can extend the disruption.

2. Operational Overhead: Availability issues draw on-call engineers into investigations, sometimes escalating into incidents and follow-up reviews. These disruptions also erode our confidence in performing routine maintenance.

Background

Kafka divides topics into partitions, each of which can have multiple replicas. One replica acts as leader, and followers fetch records from it. Most of our topics use three replicas and min.insync.replicas=2, with producers configured for acks=all. Writes can therefore continue after one replica fails, provided the remaining two stay in the in-sync replica set (ISR) and the leader is available. A partition with only a single in-sync replica is considered “Under-Min-ISR”.

Two communication paths matter to this investigation:

1. Broker <-> Broker: Follower brokers send fetch requests to the leader. Leaders track follower replication progress. A follower that stops fetching, or fails to catch up within replica.lag.time.max.ms, can be removed from the ISR.

2. Broker <-> KRaft Controller: Brokers send heartbeats to the active KRaft controller. If the broker fails to renew its heartbeat lease within broker.session.timeout.ms, the controller can fence it and remove it from partition ISRs.

ZooKeeper vs KRaft

The migration to KRaft also changed how brokers receive leadership updates. In ZooKeeper mode, controllers communicated partition leadership changes to brokers through LeaderAndIsr RPCs. In KRaft, brokers learn these changes by consuming the metadata log. In KRaft, successful heartbeats to the controller do not prove that other brokers or clients can reach the broker’s advertised address.

Investigating ISR Drops

Misconfigured Topics

Some topics were configured with a replication factor of two, or used three replicas with min.insync.replicas=3. A single replica going offline therefore brings the partition below min.insync.replicas. We changed topics that were unintentionally configured this way and added validation to our topic-provisioning component.

Segment Recovery

This one is a doozy. It gave us our first concrete evidence of a server-side problem and helped explain some incidents, but it could not explain all the failures. When investigating Under-Min-ISR partitions, we found this broker log:

Producer state recovery took 1ms for snapshot load and 24789ms for segment recovery from offset ...

Almost 25 seconds? That can’t be good. Searching other broker logs turned up recoveries lasting as long as three minutes! Confusingly, not every long segment recovery resulted in Under-Min-ISR, but some Under-Min-ISR incidents correlated perfectly with long segment recoveries.

Producer-state recovery is a normal part of Kafka’s operation, and a long recovery does not necessarily affect availability. The problem arises when recovery overlaps with a partition state transition. We found two operations contending for the same partition lock:

1. Truncating a partition’s log: Partition.truncateTo -> … -> UnifiedLog.truncateTo -> UnifiedLog.rebuildProducerState (segment recovery here can take a long time)

2. Transitioning a partition replica to follower: ReplicaManager.applyDelta -> ReplicaManager.applyLocalFollowersDelta -> Partition.makeFollower

Partition.truncateTo and Partition.makeFollower both require the same lock.

Although the lock belongs to one partition, the delay can spread beyond it. Kafka processes a batch of follower transitions before starting the corresponding fetchers. If one transition waits behind a long-running truncation, fetching for other partitions in the batch can also be delayed. If those delays exceed the ISR lag threshold, otherwise healthy replicas can fall out of the ISR.

We found topics with large log segments (segment.bytes) and poorly batched records to be the biggest contributors to long producer-state recoveries. Today, long segment recoveries remain an issue in our infrastructure, but we’re managing the risk through monitoring and adjustments to topic configuration and producer batching.

Automatic Preferred Leadership Election

Reports of similar symptoms from other kafka operators were improved by disabling automatic preferred leader election. This feature periodically attempts to return leadership to preferred replicas when its balancing criteria are met. It seemed plausible that additional leadership changes during broker restarts might be contributing to the interruptions.

Unfortunately, disabling automatic preferred leader election did not eliminate the interruptions. We also observed failures during controller restarts, so broker leadership rebalancing could not fully explain the problem.

The Main Cause

Digging into Kafka-specific rabbit holes was fun, but we forgot the first rule of running a network application: it’s always DNS. After failing to find the root cause by examining Kafka’s replication and leadership behavior, we found new clues in the network logs, such as brokers attempting to fetch data from pod IPs that were no longer in use.

In our Strimzi-managed KRaft Kafka cluster, each broker publishes its listener hostnames and ports through advertised.listeners. These endpoints are distributed to other brokers through the KRaft metadata log. Our static controller quorum uses controller.quorum.voters to map controller IDs to hostnames and ports. Strimzi configures these DNS names through a headless Kubernetes Service: the names remain stable when pods are replaced, but the IP addresses they resolve to can change.

# Single broker addresses
advertised.listeners=REPLICATION-9091://test-broker-2.test-kafka-brokers.kafka.svc:9091,PLAIN-9092://test-broker-2.test-kafka-brokers.kafka.svc:9092

# Controller addresses
controller.quorum.voters=10@test-controller-test-10.test-kafka-brokers.kafka.svc.cluster.local:9090,11@test-controller-test-11.test-kafka-brokers.kafka.svc.cluster.local:9090,12@test-controller-test-12.test-kafka-brokers.kafka.svc.cluster.local:9090

This brings us to DNS caching. In our deployment, hostname lookups pass through two caches: the JVM, which caches successful lookups for 30 seconds, and Kubernetes NodeLocal DNSCache, which caps the normal cache TTL at 30 seconds.

Additionally, NodeLocal DNSCache also has serve_stale 3m enabled. This allows it to return expired DNS answers for up to three minutes after their TTL expires, refreshing them in the background. It does not mean every address remains stale for three minutes, but TTL expiry alone does not guarantee a fresh answer.

Your Kubernetes cluster configuration and specifically the Container Network Interface (CNI) you are using will determine your options for observing network traffic. We used Hubble, Cilium’s network observability tool, to identify DNS lookups and connection attempts to old pod IPs after replacement pods had received new addresses.

During controller pod replacement, stale DNS entries could prevent brokers from reaching the active controller. If successful heartbeats stop for longer than broker.session.timeout.ms, the controller will fence those brokers and remove them from partition ISRs. Broker replacement runs into a similar problem: after a returning broker becomes a partition leader, followers attempt to fetch from its old IP address. If they fail to reconnect and catch up in time, they are removed from the ISR.

We could have tweaked the timeout configurations that trigger heartbeat and ISR timeouts, but that merely moves the problem elsewhere.

We first disabled JVM DNS caching by setting sun.net.inetaddr.ttl=0. This removed one caching layer, but NodeLocal DNSCache and its stale-serving behavior remained. We also briefly attempted to bypass NodeLocal DNSCache (which earned us a stern talking-to from our Kubernetes team and a prompt reversal of that change). Ultimately, disabling JVM DNS caching became our first mitigation, and we set out to measure how much it improved rolling restarts and what remained to be fixed.

Results & Future Work

Turns out, observing and quantifying tail events such as brief Under-Min-ISR episodes is not simple. Metrics lack the granularity required, and logs are better, but are both hard to analyze and need to be complete. In our case for example, the trigger for the beginning of an Under-Min-ISR episode can both be a broker log shrinking the ISR, or a controller log fencing multiple brokers as their heartbeats time out. Similarly, we would need to identify an accompanying log for the end of each episode.

We worked around this by analyzing the KRaft metadata log, which includes information about in sync replica sets for topic partitions and their state changes over time. You can use the kafka-dump-log.sh tool to get a user-friendly representation of these events.

Test Setup and Methodology

• Kafka 4.0.0, Strimzi 0.49.1, 9 brokers and 3 controllers

• Kubernetes v1.32 with Cilium as the CNI

• 3 test topics with dummy traffic, replication factor of 3 and 16 partitions each

• Broker and controller pod rolls performed separately. Each roll triggered using the strimzi manual rolling-update annotation and repeated 20 times with brief breaks between each roll

• We measure availability as the percentage of total partition-seconds during the test that partitions spent at or above min ISR (2), parsed from the cluster metadata log

Disabling JVM caching eliminated Under-Min-ISR episodes during broker rolls and reduced their frequency and duration during controller rolls. Even more importantly though, our users have stopped complaining! This means they can focus on their applications, and we can perform routine maintenance with greater confidence.

Future Work - Kafka Leadership Eligibility and ClusterIPs

Disabling JVM caching has been good enough for us, but it sure feels like a band aid rather than a root fix. Ideally, external health checks would be tightly integrated with leadership eligibility in both the brokers and the controllers.

We have also experimented with using a per broker/controller ClusterIP Service instead of the current, shared headless Service. Strimzi doesn’t support this customization, so we prototyped on a forked version. For us, this removed DNS caching from the picture altogether, but your mileage may vary as the mechanism and the speed of propagating the new Pod IPs will depend on your CNI.

These tests were very promising - using the same methodology as earlier, we were unable to reproduce ANY Under-Min-ISR events! We have opened an enhancement proposal in Strimzi and hope to work with the community to enable using ClusterIPs as an option in the future.

Other Learnings

Cluster Metadata & Logs > Metrics

Short-lived errors can occur before the first Prometheus scrape or between subsequent scrapes. Logs and the cluster metadata revealed Under-Min-ISR events that our scraped metrics missed entirely.

PodMonitor > ServiceMonitor

We switched from ServiceMonitors to PodMonitors to keep scraping metrics during pod readiness transitions. This gives us better visibility into broker startup and recovery, when pods are running but not yet Ready.

PublishNotReadyAddresses

We also enabled publishNotReadyAddresses for our external listeners, a Kubernetes Service setting exposed by Strimzi. This allows broker and controller addresses to be published before their pods become Ready, so applications can discover them earlier during startup.

In Conclusion

Investigations like this are difficult to prioritize over work with predictable impact. We could have spent months tracing these failures only to discover that fixing them required impractical architectural changes or a patch we were unwilling to maintain. In this case, our investment paid off: we improved availability, gained a lot more understanding of our system, and made several improvements to our observability.

Written by u/LuckyStar1122

My day-to-day as a full-time engineer looks nothing like it did as an intern. A small part of that is because I’m on a new team working out of a new timezone, but it is primarily because the tools I use today are lightyears beyond what was available to me in the summer of 2024. 

Two years ago, I was on a Europe-based team, which meant I never got to snooze my alarm. Something about free lunch as an intern made working from home Monday-Thursday a non-starter, so I would wake up and hustle to make it into the office with plenty of time to caffeinate and prepare for my 8:45am daily standup. My whole team working out of Europe also meant back-to-back meetings in the morning followed by four hours of me adding questions to an ever-expanding doc titled “Questions” that I would launch rapid-fire at my extremely patient mentor the second standup ended.

I’m now on a west coast team, which means I can snooze my alarm and workout in the morning on days when I work from home. I work from home more often than not now, and it isn’t because of the weather, or the lunches, or the commute. I work from home because my primary workflow now involves talking out-loud to Codex. 

I used to snoop on incident channels as an intern and couldn’t wait to start up again as a FTE. What I saw when I came back felt a little dystopian. I had a great laugh over people talking to our incident management bot as if it were a human. I grew up watching Spy Kids and Smart House, and that’s exactly what the new incident channels reminded me of. I feel like a huge hypocrite now that I start nearly every morning saying “Hi, can you hear me?” to my computer. 

I thought talking to Codex would be better than typing because I tend to read too quickly and type too slowly, but I think the biggest help is from being able to start from a basically unformed thought. I’ve found that when typing to ChatGPT, I often start with the wrong question and spend way too much time polishing it before sending it off. As an intern, the way I’d seek answers was even worse. I’d transition from regexing Sourcegraph to inspecting git blame before clumsily drafting a message on Slack introducing myself and putting forth my best attempt at articulating what I believed to be the right question. Reflecting on the process now feels like looking at cave paintings.

AI is my teammate in ways I couldn't have even predicted two years ago. I have scheduled “standups” with Codex three times a day: one in the morning to plan my day, one in the afternoon to check my progress and reprioritize, and one in the evening to review what I've accomplished and where I'm still blocked. One of the hardest things for me to adjust to as a full-time engineer has been context switching. I took for granted being able to focus deeply on a single project throughout the entirety of my internship.

I’ve recently been able to offload a ton of that context switching to AI by keeping a running dialog throughout the day. It is so easy to stop mid-sentence and say “Oh, my Airflow PR got approved” or “my manager just responded and he said we should try ABC before resorting to XYZ” and then carry on with the rest of my sentence while my execution tracker and project logs get quietly updated in the background.

While having AI help me manage my workload is helpful, I try to put limits on what I let it do in terms of implementation. One of the best pieces of advice I received when I returned to Reddit as a full-time engineer focused on early-career AI use. It was suggested that, when starting a coding task I'd never done before, I implement the full feature on my own before setting AI loose on the same task on a fresh branch. I could then ask it to compare and contrast the two branches, explaining tradeoffs and design choices along the way.

This approach helped me learn faster than I thought possible over the course of my first six months at the company, and it kept me from falling into the trap of shipping AI slop I didn't understand. Comparing my own implementation against AI's forced me to think about why I'd made certain decisions, what alternatives I'd overlooked, and whether the code I was writing was actually good. 

I’ve found that using AI this way makes room for more interesting conversations with my teammates. As an intern, I’d wait desperately for one-on-ones to unleash all of my pent-up questions, and it sometimes felt more like taking than connecting. I think that’s expected for an intern, but as a FTE it is nice to have some of the technical burden lifted to make room for conversations ranging from professional growth to things that have nothing at all to do with engineering.

At the end of my day, after one last check-in with Codex to figure out where to pick things up tomorrow, I try to be done with work by 6pm so I have time to do something recharging. Now that it's getting colder, I've been reading or doing crafts after work. I just finished The Silent Patient, and now I'm reading Gone Girl, with The Martian next in the queue.

I really appreciate that Reddit supports a good work-life balance. I somehow love working here even more than I did when I first started, and I feel super grateful for the people and tools that have helped me grow. It has honestly been an insane time to enter the workforce. 

Two years ago, I was spending my afternoons GitHub stalking and stockpiling questions for my mentor. Now, I spend my mornings talking to my computer and my afternoons having actual conversations with my coworkers. I hope that two years from now I have a truly dystopian workflow where I don’t even have to speak out loud and can offload tasks to my cat, Domino (pictured below).

By Corey Roberts, Senior Software Engineer, Design System (iOS)

When Apple introduced Liquid Glass at WWDC, our reactions mirrored the rest of the iOS community’s: this looks amazing, and we have so many questions. Many of us have been iOS engineers long enough to remember the first major shift in Apple's design language: the move from texture-rich skeuomorphism of iOS 6 to the flat, motion-driven aesthetic of iOS 7. Assumptions about layout, controls, navigation, and typography went out the window. Layout code sprouted version checks, and making something feel cohesive across both paradigms was an awkward challenge with no grace period.

Liquid Glass reminded me of this experience. We knew this wasn’t going to be a simple “rebuild in Xcode and it works” trick. It changed parts of the app we’d spent years building around: the top and bottom navigation, the safe areas, and the window that houses every screen. Navigation became translucent instead of opaque, heights shifted (particularly on iPhone Air), and insets we’d tuned specifically for opaque bars no longer worked. These weren’t cosmetic issues: they were behavioral. A screen that behaved correctly on iOS 18 no longer looked the part under Liquid Glass.

We also couldn’t just move everyone to the new design paradigm. Reddit supports a wide range of iOS versions, and those users had to keep the exact same experience they knew. That’s why we leaned on Reddit’s design system, RPL, so the same components could render in both the pre-Liquid Glass and Liquid Glass paradigms.

The Initial Friction with Liquid Glass

The challenge with Liquid Glass was compatibility: we had to support two very different design paradigms at once. Simply updating to the new design and calling it a day would have broken the experience for everyone still using an older version of iOS. We needed to make a set of components look and behave correctly in both places while streamlining our APIs so we wouldn’t cause confusion, code duplication, or unintended behaviors. The problem that was most obvious, and one worth understanding before anything else: opaque content underneath the navigation bar defeats Liquid Glass entirely.

Prior to Liquid Glass, our navigation constructs were opaque, and we built our screens to be pinned to the safe area edges to avoid unnecessary rendering under the top and bottom navigation bars. Because our bars were opaque, it didn’t matter if nothing was rendered underneath: you couldn’t see it. Liquid Glass inverts that assumption. The top navigation had a diffusive blur that displayed incoming content contextually, and the bottom navigation became modular and translucent. With our current infrastructure, the effect was lost on us entirely. Instead of a translucent, diffusive bar, we got a flat, cut-off band: a jarring visual bug.

That one symptom is the whole problem in miniature: nothing was wrong with our old code. Rather, the rules underneath it changed, and the assumptions we made under the old rules now produced the wrong result. Multiply that across every screen, and you have this project!

Uncovering Bugs

We started our analysis in iOS 26 Beta 2 by building an alpha version of Reddit and looking through all of our screens for issues. We ended up creating categories for the kinds of visual defects we kept seeing:

1. Hardcoded navigation heights. Much of our layout code relied on static calculations based on OS and device in use. This was never a robust solution, and Liquid Glass (plus newer hardware, like the iPhone Air) made it obviously wrong. These screens were updated to use the top safe area insets.

2. Warped or misplaced icons. A handful of icons became stretched in Liquid Glass due to assumptions that were made with the size of a button. These were predominantly PNG images, many of which were not moved over to using our vector icon set. If an icon looked squished or oddly sized, it was almost always a bitmap that should have been a vector. In a similar vein, our bottom navigation tab for creating a post became misaligned due to using a custom tab item.

3. Static Colors in Glass. Liquid Glass views have a unique property where they adapt the colors of their content view based on the current trait environment. This conflicted with some of our navigation components, which use color tokens tied to the active Reddit theme, not the surrounding environment the color sits in.

4. Custom title views. Apple changed how content renders in the titleView of the navigation controller. In some instances, custom views could cause the diffusive blur that the navigation bar provides to extend across the entire screen! This would commonly happen when a title was being set using a leftBarButtonItem property. TopAppBar alleviated this by creating a title configuration and preset for these styled titles.

5. Not enough room. Related to title views, Apple prioritizes that view’s size over any leading or trailing buttons. Depending on the view's intrinsic size, this could push all of our trailing actions into an overflow menu. TopAppBar handled this by adjusting the constraints behind that behavior and by handling overflow gracefully. The glass platters that house the buttons also add substantial padding, so space is tight: we made sure any overflowed actions still surfaced in a reliable menu.

None of these issues are terribly exotic; they were mostly assumptions in our layout code, quietly invalidated by the new expectations set by iOS. A couple of them (notably, the stretched icons and static color tokens) were really pre-existing debt that Liquid Glass just happened to expose. The good news is that the foundation underneath all of it lived in one place we could fix and control: RPL.

RPL to Render Both Worlds

Despite all the bugs we found, we thankfully had a little runway. We leveraged UIDesignRequiresCompatibility, a property list flag that let iOS 26 keep the pre-Liquid Glass look, so we could build the foundation before exposing anything. Apple was clear in the docs that this was a temporary solution, so we set a timeline to launch publicly before Apple announced the next version of iOS, which typically happens the following June. That breathing room mattered, since it let us build the foundation properly instead of applying a fix onto every screen. And, because nearly all of that foundation lived in our design system, we had a universal place to get it right.

All of the issues mentioned above really allowed Reddit’s design system to shine. The core idea is simple: a component shouldn’t have to figure out whether it’s in a Liquid Glass environment. It should just know!

At a high level, a screen would need two layout paths, and we would check if that device was eligible to display Liquid Glass (i.e. iOS 26 with no compatibility flag set). However, given the breadth of screens within Reddit, it would be cumbersome to copy and paste the same condition and logic onto every screen. To combat this, we created a navigation layout strategy that encodes sensible defaults for both design paradigms: things like whether a screen can extend behind navigation bars, how safe areas are treated, and how content is inset. Screens then simply opt into a strategy instead of growing their own conditional branches that become harder to untangle and remove over time.

Using our previous example, the navigation bar was opaque on pre-Liquid Glass iOS, so laying out content just to the safe area edges was fine. Under Liquid Glass, this same layout looks like a visual bug since navigation is intended to have content diffused underneath. Encoding these constructs into a strategy means there’s a correct answer for each environment. It provides a nice layer of abstraction so teams don’t have to rediscover the problem. The best part is that they don’t even need to care about the environmental or implementation details! It just works, and that’s a win for reducing friction.

We took this a step further with our navigation foundation, named TopAppBar and BottomNav. Both of these rendered their content appropriately based on these layout strategies, and screens that used these components got this update for free (which is an awesome benefit for having a design system). Compatibility was always the goal, but Liquid Glass also gave us a reason to revisit a couple of interactions that never quite fit iOS's conventions. We found that iOS was more strict with custom interactions, and this interfered with the Create tab in BottomNav, which creates a new post. We decided to move this action to the TopAppBar for a more canonical iOS experience that's easier to maintain.

Similarly, we found that iOS’s native behavior of expanding a collapsed tab didn’t match the experience users were used to on Reddit. We wanted to build an experience that mirrored our pre-Liquid Glass animation of bringing the tab bar back after starting to scroll upwards, so we got creative: we set the minimization behavior of the tab bar based on a pan gesture event on the scroll view. If the velocity is high enough, it’ll set tabBarMinimizeBehavior to .never, then back to its original value. This trick allowed us to force the tab bar to expand, but still retain its minimization state if the user decided to scroll back down.

Testing and Evaluating Liquid Glass

We generally like to run experiments on changes that adjust the user experience, whether functional or visual. Liquid Glass wasn't one we could quietly A/B test. Whether the app renders in Liquid Glass or compatibility mode is decided at runtime, and it's a system-wide appearance change baked into the build, so we couldn't cleanly split people into “Liquid Glass” and “non-Liquid Glass” groups within a single release the way a traditional A/B test would. In practice, that meant one shot in production: when we flipped Liquid Glass on, everyone eligible would get it at once.

That left us two jobs: build confidence before launch, and get creative about how we measured the change.

For the first, we front-loaded several rounds of internal playtesting. We started by hardening the design system itself, fixing the issues that covered the largest surface area of visual regressions, then kept a timeline of when we'd progressively “unlock” Liquid Glass internally. We began small: announcing to engineering and product teams that they could install an alpha build, try the new experience, and drop feedback in our Slack channels. Constant communication with product, design, and engineering was a necessity, especially as we ramped up enablement across our different build pipelines.

To make that ramp possible, we worked extensively with our release engineering partners to build a Bazel configuration that could mutate the bundled property list to toggle UIDesignRequiresCompatibility on or off. This gave us a lot of control and flexibility over exactly which internal channels rendered Liquid Glass, without hand-maintaining separate builds.

For the second, we compared across time instead of across users. We froze all other experiment movement, shipped one build, and then the following week shipped an identical build whose only difference was Liquid Glass turned on. With everything else held steady, that week-over-week comparison let us isolate the impact of Liquid Glass itself: about as close to a controlled experiment as we could get for a change this global.

Alongside that comparison, we leaned on the community. We kept a close eye on r/bugs and our beta feedback for anything that looked off, triaging reports as they came in and fixing issues before widening the audience. For a change this visible, our users turned out to be some of the best regression testers we had!

Lessons Learned and the Future

Designing for Liquid Glass in the time we had wouldn't have been possible without a solid design system as our backbone. Consolidating these implementation details let feature teams keep shipping while we focused on the pre-Liquid Glass and Liquid Glass paths.

Liquid Glass turned out to be exactly the kind of change RPL was built for. All of the challenging parts, like layout, safe area insets, and navigation constructs, lived inside our design system. Because of this, we could take on an entirely new visual paradigm without asking every team to learn how to build for two different ones, and we could retain the same user experience for older versions of iOS while providing a new one for Liquid Glass. Components simply knew which world they were rendering in, and that’s the quiet promise of a design system paying off: a change this sweeping became something most of our engineers got for free!

And we’re not finished, either! iOS 27 optimizes Liquid Glass even further, and we’ll keep expanding where the material shows up in RPL. The foundation is in place though, and the next time Apple reshapes iOS, we can reach for the same playbook: let the design system speak both languages and keep shipping amazing designs for everyone.

We came out of that WWDC keynote with a lot of questions; we came out of this project with a design system refined enough that could answer most of them.

Written By Ivan Klimuk, Kim Holmgren, Jonathan Serrano, Jonard Doci, and Md Mansurul Bhuiyan

Background

In our previous post on the notifications recommender system, we described how Reddit finds and ranks posts to recommend through push notifications. Before that system selects a post, another component makes a decision about volume: how many recommendations should we send this user over the day?

We call this component the Trending Budgeting system. It assigns a daily budget for trending push notifications: the personalised post recommendations that help people discover conversations on Reddit. This budget applies to that recommendation channel, rather than to every kind of notification a user can receive.

A relevant recommendation can bring someone to a conversation they would otherwise have missed. But relevance alone does not tell us how often to send one. Someone might welcome one recommendation and find several more distracting. Another person may benefit from hearing about a wider selection of posts. Those responses can also change over time.

The useful signal is how a user’s expected response changes as we increase the budget. We model that response across several budgets, then apply a policy that weighs expected activity against the risk of a user turning off their notifications.

A budget determines how many trending push notifications we send a user each day. If the budgeting system selects three, we schedule three notifications, and the recommender selects a post for each one.

This gives us three related decisions:

• Budgeting: how many notifications to send.

• Pacing: when to send them throughout the day.

• Retrieval and ranking: which post to recommend in each one.

A scheduled opportunity gives the system a chance to send a notification. Pacing chooses which times to use the user’s budget, and retrieval and ranking select the content for those sends.

ML problem definition

We model two outcomes: how many days a user is active on Reddit, and whether they disable notifications. Activity captures return visits; disables provide a signal of a poor notification experience.

We want to understand how many notifications work well for each person. Sending more can help people discover interesting conversations, but too many can become annoying and lead them to turn notifications off.

Why use randomised data?

Historical notification logs contain many examples of users, assigned budgets and later outcomes. Training on those logs is tempting, but the way the budgets were assigned matters.

Consider a hypothetical system that gives new users one recommendation per day and established users five. A model trained to predict the assigned budget would implicitly learn those rules very accurately. But that would tell us little about whether either budget was a good choice.

Changing the prediction target to user activity helps, but does not remove the underlying problem. The two groups may differ in many ways beyond notification volume. A higher activity rate among established users would not establish that the larger budget caused it. The logs also contain no examples of new users receiving the higher budget under this simplified policy.

Our budgeting approach uses data from experiments that randomly assign users to fixed daily budgets. A user keeps their assigned budget during the collection period, while we observe subsequent activity and notification disables. Randomisation makes the groups comparable in expectation and gives us evidence about the effects of different budget assignments.

For modelling, each example contains:

• User features measured before the treatment.

• The assigned daily budget.

• Outcomes measured over a specified future window.

The treatment here is the assigned daily budget. We compare outcomes across the randomly assigned groups to estimate the effect of changing notification volume.

Each person still experiences only one assigned budget. The models learn across users to estimate responses to the alternatives, so randomised training data does not remove uncertainty from an individual prediction.

Predicting the response to each budget

The models take user features and a candidate budget as inputs. They predict outcomes such as the number of active days in a future window and the probability of disabling notifications. At inference time, we evaluate the same user's features at each possible budget. The result is a set of predictions describing how that user is expected to respond to different daily notification volumes.

This is an S-learner approach: the candidate budget is an input to an outcome model, and we compare its predictions across budget values. Randomised assignment gives those comparisons a causal basis; user features let us estimate how the response varies across users.

Balancing expected activity and disable probability

Outcome predictions leave a choice to make. A larger budget may increase expected activity and also increase the risk of a disable. Our chosen policy determines how to weigh those outcomes and how much additional value to require before increasing notification volume.

A simplified way to express this is:

Score = activity weight × E[active days | features, budget]

− disable weight × P(disable | features, budget)

E is the expected number of active days, and P is the chance of turning notifications off, given the user’s features and the chosen budget.

The weights express the relative importance of activity and disables while accounting for their different units. Both predictions refer to defined future windows, so those horizons matter when choosing the weights.

For each user, we calculate a score for every budget in the supported range, using the activity and disable predictions for that budget. We can then compare across budgets to understand the incremental gain from sending more notifications. Selection rules and thresholds determine which gains are sufficient, and caps limit the maximum number of notifications.

For a hypothetical user, moving from two to three notifications might produce a useful predicted activity gain with little change in disable risk. Moving from three to four might add very little activity while increasing that disable risk, so we’d send them three.

The policy weights are a practical way to express a trade-off, which still needs to be evaluated through online experiments.

Training and serving architecture

We compute outcome predictions in a daily batch and choose the budget when a notification opportunity reaches the ranking service. This separates the more static inference on a set user group’s likelihood of being responsive to notifications from the real-time policy decision of how to use that information.

Offline training and daily inference

We train the outcome models offline using the randomised budget assignments, user features, and labels on the outcomes described above.

Once trained, the models run in a daily batch inference job. For each user, we take their available features and evaluate the models at each supported budget. The output is a set of per-budget predictions for the outcomes used by the policy.

We store the predictions in a key-value (KV) store, keyed by user. Each entry contains predicted outcomes for several budgets. Keeping these alternatives available lets the online policy change the activity–disable trade-off without rerunning the budget models.

Applying the policy during ranking

When a scheduled opportunity reaches the ranking service, it fetches the stored predictions and applies the configured policy. The policy combines the expected activity and disable probability, evaluates the available budgets, and selects how many notifications to send. Budget enforcement and pacing then determine whether this slot is one of the sends for that user.

The policy runs at request time using batch-generated predictions. Changing the model requires a new inference run, while policy weights, thresholds and selection rules can change independently.

The upstream scheduling system generates a common set of opportunities throughout the day. Each user’s budget determines which proceed. Figure 2 shows a hypothetical example with six opportunities and a budget of two: pacing selects two slots for sending, and the other four exit before retrieval and ranking. Those early exits avoid the more expensive work of selecting a post.

Previously, budget selection and enforcement already ran before retrieval and ranking, but lived outside the ranking service. That separation made budgeting experiments harder to run. Moving the logic into the ranking service brought budgeting into the same service and experimentation framework as the recommendation pipeline, while retaining the early exit for opportunities that would not trigger a send.

The architecture gives each step a clear role: batch inference estimates the outcomes of different budgets, the online policy chooses the trade-off, and pacing and enforcement turn that choice into sends. Serving the alternatives keeps policy experiments independent of model training; enforcing the chosen budget before retrieval and ranking avoids selecting content for unused opportunities.

Offline evaluation challenges

Before running an online experiment, we want some idea of whether a new policy is an improvement. Traditional ML model metrics do not answer that, since the models can predict activity and disables well and still lead to budgets that perform poorly.

The direct approach would be to apply the candidate policy to past data and see how those users did. But those budgets were not assigned at random. The live policy chose them based on the same user traits that predict activity, so the users on each budget are not comparable to begin with.

Instead, off-policy evaluation (OPE) estimates what a policy would have produced had we deployed it, using the randomised budget experiments.

• The simplest version is direct matching: for each user, find the budget the candidate policy would pick, then keep the users whose assigned budget happened to be the same. Those users are a fair comparison, but there are not many of them, so the answer is right on average but noisy.

• A doubly robust estimate improves on that by using every user, not just the ones that matched. It predicts each user's outcome at the budget the policy picked, then uses the matched users to measure how far those predictions were off and corrects for it. It is right if the predictions are accurate, or if we know each user's chance of getting each budget. Randomisation gives us the second for free.

We built this into a tool that scores any proposed policy against the experiment data. To build confidence in the framework, we look back at policies that have already run online and compare what the offline estimate predicted against what the experiment measured.

Promising policies go through an online experiment, and we keep a long-term holdout to measure causal impact.

Future work

• Connecting offline and online metrics. A clearer link between offline evaluation and online outcomes would help us assess model changes with more confidence, opening up experiments with richer architectures such as neural networks and causal ML approaches such as T-learners.

• Coordinating notification types. We also want to extend budgeting beyond trending recommendations, managing volume across notification types to account for the user’s overall notification experience.

Conclusion

Randomised budget assignments help us estimate how behaviour changes when we send more or fewer recommendations. Modelling activity alongside disables brings both the benefit and the cost of additional sends into the decision.

Serving predictions for several budgets keeps that trade-off explicit and testable. We can change the policy without retraining the models, then use online experiments to test whether the new volume helps people find worthwhile conversations.

By Seth Epps

AI Disclosure: This is all artisanal, full of grammatical error, human written slop.

Fragmentation to Unification

gRPC Discovery and Communication Across Kubernetes Clusters at Reddit

In the before times, Reddit workloads were spread across a disparate set of Kubernetes clusters that varied in purpose based on any number of factors. They blurred the lines between team ownership, intent, and locality where workloads could be spread across multiple environments. Furthermore, while we gave owners a fairly simple API to expose their services, providing the target for clients was a constant pain-point. In order to target the service cross-cluster the client needed to know:

- what Kubernetes cluster the upstream they were targeting lived in;

- what is the hostname for that cluster ingress;

- what is the authority for the upstream exposed by that ingress

"How can I reach foo from cluster-bar when the service lives in cluster-baz”? or "I'm seeing Failed to dial target host for foo.ns.svc.cluster.local:80"? were frequently fielded questions by our team. These customer questions (and others like them) stem from the combination of an inconsistent/fragmented service exposure and the lack of full platform support. Every environment had a slightly different host naming convention and every service could define their own hostname/authority overrides.

Around early 2025 we started to see an uptick in adoption for our proxyless discovery platform Portal Transporter, but it had a fundamental limitation; it only worked for discovering services in cluster (not to mention the proliferation of standards [relevant xkcd]). Could we consolidate these two patterns, in cluster and out of cluster discovery, into the same platform and completely remove the need to define the "where" and let clients just declare "I want to reach foo"?. Furthermore, could we provide the control surface for infrastructure teams to spin up new compute clusters and let clients transparently discover them?

Envoy has [re]entered the chat

A core constraint our team was working with was that workloads were not assigned a unique local address that would be routable across our VPCs so across environment boundaries we would still need an ingress point to the cluster. Naturally, we reached for Envoy as it had served us well for many cross-cutting use-cases and was already the de-facto system to expose services within our private network. What differentiates this new Envoy deployment from its predecessors is that exposure of services would be completely handled behind the scenes. Service owners declare intent, however, they do not need (or even get) to specify any information about how it was exposed. This is a nuanced departure from how owners were previously exposing their services, but it shifts even more responsibility into the platform and infrastructure control plane. In particular, instead of one artifact being generated for discovery two artifacts are now produced; an in-cluster route and a cross-cluster route.

An absolute, non-negotiable, requirement to tie this new Envoy deployment into our discovery platform was that our developer APIs for declaring routing intent must maintain feature parity. That is to say, if someone institutes a routing policy for their service it must be supported regardless of caller locality, be it in or out of a cluster. This means that rules like load balancing policies, method matching, and cross-namespace policies all need full support and conformance across the platform.

Integrating Into Portal Transporter

Above, we've defined what it means to have a consistent routing behaviors between in cluster and out of cluster, but essentially nothing has changed from the outset. Routes are available, but we need some glue to bring them into the Portal xDS management server. Unfortunately, the cluster boundaries introduced another hurdle when trying to propagate the available routes to their intended consumers. We looked into many different ways to approach route propagation, such as a separate informer that could tap into the remote cluster API server or an orchestration control plane that would federate routes into target clusters, but they had very undesirable tradeoffs with respect to security and availability.

Since services, and to a larger extent routes, are fairly static / low-churn we ultimately landed on building up all the available routes in a background task and packaging them up into a Snapshot that is progressively rolled out to every cluster in our fleet. This was the sweet spot for managing risk and delivery speed of updated routes for all possible consumers.

A Snapshot contains three pieces of information:

- how to reach envoy;

- what is the relative weight assigned to routes served by this envoy;

- what routes are available at this envoy

Portal aggregates multiple snapshots into its route table, consolidating these three properties into weighted clusters that correspondingly point at logical clusters for the configured envoy.

Simplified Configuration and Traffic Shaping

One notable detail is that envoy routes are exposed by their virtualhost domain which, by construction, will always match the xDS listener address. By maintaining this contract, Portal constructs a response whether or not the target exists in a cluster such that services have a single unified target that clients will use without any knowledge or information about the locality. This also means that when a new environment spins up and a snapshot is generated, Portal provides the new endpoint assignment to the callers without a single configuration change. A beneficial byproduct for our customers is that they can cut down their deployment manifests dramatically as their configuration is completely portable; no more need for per environment overlays.

A common client configuration pattern we would see is

# cluster-bar
client:
  addr: dns:///foo.ns.svc.cluster.local:9090 

# cluster-baz
client:
  addr: dns:///path.to.envoy.internal:80
  authority: foo-custom-ingress.services.internal

Now, every deployment target shares the same configuration base, no matter the compute environment deployed to

# base
client:
  addr: xds:///foo.ns 

Since clients no longer need to provide details about the host when addressing services the platform empowers operators to shape traffic as needed. Operators can quickly onboard new workload clusters when scaling up capacity (for example to support growing machine learning demand) or divert traffic away from problematic clusters during an outage (for example when an availability zone is degraded), all without a single client configuration change.

Closing Remarks

Our journey into self discovery (see what I did there?) is far from over, but we've found ourselves experiencing a moment of bliss. Our clients are happy, our support channels are quiet, and best of all we've managed to make things easier; easier to deploy, easier to operate, and easier to adopt. At the end of the day, that's all anyone ever really wanted, right?

Jimmy Li

TLDR: Have you recently noticed that when you search on Reddit for fresh stories/events, we started to provide more timely and engaging post contents in the search results? And the subreddit recommendations, with their recent trend summaries under the Search Bar drop-down?

Behind this, is our COntextual Popularity TrEnding detection for Reddit content (a.k.a COPTER) system. This post described COPTER’s first production version. It focuses on the core design implemented under the contextual_popularity pipelines, together with its downstream Search integrations, and how it improves our search results on trending topics.

Context

Reddit Search could tell whether a result matched a query and whether content was generally successful, but it had no first-class signal for whether a post or topic is becoming popular right now. The gap was most visible for breaking news, live events, new releases, sports, elections, product launches, and fast-moving community threads, where semantically relevant but days or weeks old results outranked the posts carrying the live conversation. Relevance and trendingness are different questions: relevance measures query match, while trendingness measures whether attention to a post is unusual for its context and moment.

COPTER (COntextual Popularity TrEnding detection for Reddit) defines a post as trending when its engagement, projected to a fair comparison age, is exceptional relative to recent posts that share the same context (a community, i.e. subreddit_id, or a linked real-world topic/entity), post-age bucket, engagement metric, and residual-based growth-trajectory bucket. This separates the large from the truly exceptional: a modest post can be highly unusual for its community or linked topic at its age, while a large but old post is not trending if its trajectory is ordinary for comparable peers.

Two naive baselines fail in complementary ways:

1. Ranking by cumulative counts rewards age rather than momentum, is dominated by large communities and head topics, cannot reconcile the different scales of views, votes, and comments, and carries no trajectory information.

2. Ranking by a short recent-velocity window restores recency but reintroduces traffic-scale bias, is unstable at window boundaries, is noisy for very young posts, and misreads distribution pauses (e.g. local-time troughs, exposure lulls, notification gaps).

COPTER instead normalizes engagement for age and context and projects the full trajectory to a common comparison age, rather than trusting a single counter or the latest delta.

The COPTER V1 design prioritizes the following properties:

1. Fairness

2. Engagement understanding

3. Reliability and serving

Search is the first consumer, but the signal is intended as a reusable, platform-wide layer (supporting feed candidate generation and ranking, notifications, query suggestions, autocomplete, and typeahead, discovery and onboarding, community and topic recommendations, and event or story clustering).

Methodology

Overview

COPTER V1 is split into three main data cadences:

• an hourly observation pipeline that creates per-post engagement deltas and cumulative snapshots;

• daily community and topic modeling branches that fit growth curves, residual-conditioned projection parameters, and contextual baseline distributions; and

• hourly community and topic detection branches that project current engagement, assign trending levels, and publish downstream features.

The implementation is intentionally staged. Expensive model fitting happens daily, while the hourly detectors perform bounded lookups and projections for each eligible post and contextual group.

We will first explain these scoring stages in the following COPTER Scoring Workflow section below.

Then trending features supports the search pipeline in a few key places:

• Semantic search [Milvus] on trending posts

• Feature hydration (with relevance + safety + trending level gates)

• Value Model (with a trendingness dimension)

We will explain further in the Search Integration section.

COPTER Scoring Workflow

Formulation and notation. COPTER scores every eligible post p at snapshot time t under one or more contextual groupings g. A post has exactly one community grouping (g = subreddit_id) and zero or more topic groupings (g = wikidata_id, restricted to high-topicality, high-confidence Wikidata links); the two branches share identical machinery and differ only in g. For each metric m ∈ {views, upvotes, downvotes, comments} we observe a cumulative count x_m(p,t) and post age a(p,t) in hours. Ages are discretized into seven disjoint buckets b, each with a reference anchor age r_b at its upper boundary (Tab. 1), so all posts in a bucket are compared as if given equal time to grow. A single quality/scale floor defines the shared post universe, which bounds the candidate set without restricting either contextual view. Fig. 2 summarizes the scoring flow.

Tab.1 Age buckets Assignment and Reference Anchor Age

Stage 1: Hourly observation. At each hourly boundary (t), COPTER aggregates engagement events from the preceding window ([t-1, t]). For each post, it calculates the engagement accumulated during that hour (the deltas), such as views/upvotes/downvotes.

Each hourly partition therefore represents a deterministic set of per-post deltas. When a run is retried, COPTER replaces the same partition atomically, preventing duplicate counts.The downstream snapshot combines these hourly deltas to reconstruct the posts’ recent cumulative engagement trajectory.

Engagement deltas are collected independently of post metadata, so a fresh post's first-hour votes are not lost when its creation metadata lands late; eligibility and age filtering are applied only at snapshot construction. Validation gates enforce non-empty engagement, post uniqueness, age-bucket consistency, and delta-partition coverage before any snapshot is published.

Stage 2: Daily contextual modeling. For each pair of (g, m), where g is a subreddit or linked topic/entity and m is engagement metric (views/upvotes/comments/shares/.etc), a power-law growth curve is fit by ordinary least squares in log space,

log(x_m + 1) = α_g,m + β_g,m · log(a)

Here, x_m is the observed cumulative count for metric m; a is the post’s age in hours; α_g,m (alpha) is the fitted intercept, which sets the baseline log(count + 1) at one hour; and β_g,m (beta) is the fitted growth exponent, which describes how quickly engagement grows with age. The subscripts g, m indicate that each context–metric pair has its own fitted parameters.

The fitted engagement-plus-one value therefore scales as a^β once the transform is inverted (the +1 term admits zero counts; age is floored at a_min = 5 min). Each curve is fit locally per context and once per branch over pooled observations under a synthetic key __global__; local fits that are degenerate, under-supported, or that yield β ∉ [1e-5, 2.0] fall back to the branch-global curve. In the topic branch, entity-expanded rows are deduplicated to one observation per post before the global fit, so multiply-linked posts receive no extra weight.

A single average slope cannot describe posts already far above or below their curve, so COPTER conditions the projection on trajectory. For each observation it computes a base residual against the fitted curve,

ã = max(a, a_min); e = log(x_m + 1) − (α_base + β_base · log ã)

interprets e within (g, b, m), and partitions it by percentiles into residual trajectory buckets k; thresholds use only the latest observation per post so long-lived posts do not dominate the cut points. A separate projection slope β^proj_g,m,k is then fit per residual bucket (thresholds stay age-bucket-specific; the slope is shared across age buckets to keep the parameter table small).

At serving time each metric is projected forward to its bucket anchor using the trajectory-selected slope,

x̂_m = (x_m + 1) · (r_b / ã) ^ (β^proj_g,m,k) − 1

a deliberately local extrapolation (a 40-minute post is projected to 1 h, a 4-hour post to 6 h) that keeps the comparison horizon near the observed age. Projected observations are then pooled (the residual bucket is not an output key) into one contextual baseline per serving key (g, b, m) storing the percentile ladder {p95, p99, p99.5, p99.9, p99.99, p99.999}, the base curve, residual thresholds, the four trajectory slopes, and local/global provenance flags.

Reliability is enforced hierarchically rather than as a single hit-or-miss fallback. Sketch resolution adapts to support: an extreme-tail threshold is emitted only when the distinct-post count is sufficient, otherwise it is chained down to the nearest trusted rarity (an unsupported p99.9 collapses to p99.5, never fabricating tail precision). The pooled model is protected from low-signal groups at both ends, a minimum per-metric contribution to enter the global fallback, and a minimum unique-post support before a local extreme-tail bucket is trusted (e.g., a first-hour p99.99 bucket requires at least 200k unique posts, else it defers to p99.9).

Stage 3: Hourly detection. Two detectors (community and topic) consume the same validated snapshot and the newest contextual-baseline partition no older than two days, drop posts currently deleted, unlisted, neutered, or taken down, and for each (post, g, m): assign the residual bucket, project to the age anchor via x̂_m, and compare x̂_m against the contextual percentile ladder to yield a rarity tier ℓ_m ∈ {0,…,6} (0 below p95; 1–6 for p95…p99.999), subject to the per-tier support gate. Views, upvotes, and comments jointly trigger trendingness; downvotes are modeled for diagnostics but are never a positive trigger, and per-metric tiers are summarized into a max and median level. The community branch emits one row per post; the topic branch emits one row per (post, wikidata_id) and then collapses triggering topics into a single post row carrying max/median topic strength and a bounded list of entity-level explanations. Every row retains raw and projected metrics, selected trajectory buckets and slopes, support counts, and fallback flags, so each decision remains fully explainable.

Search Integration

Detection is only the first half of the system. Search turns COPTER's outputs into user-visible results through two complementary retrieval paths and a shared ranking path:

This separation matters. COPTER determines whether engagement is contextually exceptional; semantic retrieval determines whether a post matches the query; and the value model determines whether a bounded trending boost is appropriate after relevance and safety are considered.

Semantic Retrieval

The post-level flow combines approximate offline indexing with exact online feature validation:

1. The post_trending_level V1 GFS (Gazette Feature Storage, supporting Reddit’s internal feature batching/streaming) feature is indexed into the Milvus postsv5llm corpus as the trending_level field. Missing values become zero.

2. The query-to-post two-tower model (distilled from LLM) embeds the query and retrieves semantically related recent posts from Milvus.

3. Milvus can apply trending_level != 0 as a coarse index-time filter.

4. Search performs an exact Redis lookup for the current V1 trending level, applies the configured minimum level, and drops cache misses.

5. Surviving candidates are ordered by descending trending level, with vector similarity as the tie-breaker, before source mixing.

The split is important. Milvus provides efficient semantic retrieval over a large corpus, while Redis supplies a fresher exact value with a short TTL.

Entity Event Retrieval

The topic/entity retrieval channel provides a parallel route for events represented by multiple posts or communities. A downstream entity-to-post Redis cluster stores trending topic associations. At request time, Search matches query entities, retrieves their trending posts, and merges those candidates with other FPR sources while preserving entity-level trending features for scoring.

Together, the two paths capture two complementary signals: a post can be exceptional within its community, within a linked topic, or in both contexts simultaneously.

Feature Hydration and Ranking

The common hydration stage reads Redis for top reranker candidates and attaches the integer as the canonical candidate feature post_trending_level. The feature is not passed into the reranker inference model. It is consumed afterward by the trending-aware value model.

For a valid level from 1 through 6, the feature transformer:

1. maps the integer linearly into a configurable trending-score range;

2. checks that the candidate clears a query-relative relevance gate;

3. blocks the boost for unsafe NSFW, Drug/Violence/Porn-style, or related safety-tagged content;

4. scales the score using matched post-entity overlap when available; and

5. combines the ordinary value-model input and trending score using a harmonic mean.

For positive inputs a and b, the blend is:

harmonic_mean(a, b) = 2 * a * b / (a + b)

The harmonic mean rewards candidates that are strong on both ordinary Search value and trendingness. A very high trending level cannot fully compensate for poor relevance, and an unsafe or obviously off-topic post receives no trending boost.

Examples of Trending Posts Lifecycle

Fig.4 below shows how engagement and contextual trending levels evolve over seven days for six real posts from six different subreddits. The upper panels plot cumulative views, upvotes, and comments, while the lower panels show the highest trending tier across those metrics. Both axes use logarithmic scale to keep different magnitudes visible. From the example, we can see that trending levels can:

• Stays steady across the 7 days after being published;

• Fluctuations between neighboring levels due to organic traffic and time decay.

• Decline to lower levels (even 0) as engagement totals increase, as popularity is assessed relative to the post’s age and community baseline instead of monotonic raw counts.

Experiment Results

We ran live A/B test on real users in our search pipelines, and the topline metric are all up, including

• Days active (App), Days active, DAUq, Time on Site 📈

• Search Query Refinement Rate 📈

• Search fulfillment metrics 📈

• Search DAU 📈

• ROCS (rate of search conversations success) 📈

• SERP Dwell Rate (30s) 📈

• SERP Bounce Rate (3s/5s/10s) 📈

• Rolling DAU and Retention 📈

Some qualitative examples of how this changes the SERP quality on trending topics:

Before

After

Before

After

Future Works

COPTER V1 established the end-to-end system, but it leaves several areas for improvement:

• Hourly latency. Delta closure and detection happen once per hour, which can miss a substantial part of the earliest trend window.

• Power-law extrapolation. A power curve does not represent saturation; longer projections can overstate engagement for trajectories nearing a plateau.

• Daily model refresh. Community and topic behavior can change faster than the model parameters refresh.

• Internal-signal feedback. Exposure from ranking can increase engagement, making popularity partly endogenous to the system using it.

• Long-tail precision trade-off. Minimum sample and first-hour signal gates improve reliability but still reduce coverage for small or sparse contexts.

• No universal trend truth. Internal engagement alone cannot distinguish a platform-wide event from a community-specific burst or validate external importance.

• Entity dependence. Entity coverage and freshness depend on upstream linking quality.

• Presentation. Retrieval and ranking surface individual posts; richer story clustering and event presentation remain separate product work.

(spoiler alert: COPTER.v2 is coming soon!)

Acknowledgement

Huge shout out to the search relevance team for this effort, especially Cathleen Li, Andrey Kamenev, Xinyue Liu, Cliff Chen, Ray An, Shafi Bashar, Christophe Hivert, Brian Kimmig, Daniel Chung!!

By Jeff Mo and Christophe Hivert

At Reddit scale, candidate retrieval has to serve many product surfaces with different goals. Home Feed, Notifications, and ranking systems may all need personalized candidates, but each surface optimizes for a different mix of user actions, such as clicks, upvotes, comments, or long-dwell visits. SnooRec gives all surfaces a shared retrieval foundation, with task-specific user representations, a shared item index, and a serving-time control for mixing objectives.

At serving time, a product surface can mix representations of different kinds of intent, clicks, upvotes, or good visits, and make just one query against a shared approximate nearest neighbor index.

The same user and item representations are also reused as inputs in downstream ranking models.

SnooRec has served full production traffic since early 2025 and now supports multiple Reddit recommendation surfaces and ads ranking models.

Introduction

A straightforward approach to multi-task retrieval is to train separate retrieval models for each objective. In practice, this does not scale well as each objective can require its own item embeddings, ANN index, serving path, and retraining cycle. This increases infrastructure cost and slows experimentation.

SnooRec is Reddit’s multi-task retrieval architecture designed to solve this problem. The key idea is asymmetric specialization: task-specific capacity lives in the user tower, while the item tower remains shared. Each item has one embedding in a unified ANN index, and product surfaces can mix task-specific user embeddings at inference time to express different retrieval objectives.

SnooRec is used in two complementary ways in production. First, it powers candidate retrieval: a task-weighted user embedding queries a shared ANN index of item embeddings to retrieve personalized candidates. Second, SnooRec acts as a representation layer for similarity computation and ranking models: the same user and item embeddings, along with derived similarity features such as dot products, are used as alternate candidate sources, and as inputs into downstream rankers. This lets ranking systems benefit from the multi-task user representation learned by SnooRec even when SnooRec is not the final scoring model.

Asymmetric Design - One shared item space, many ways to query it

SnooRec is asymmetric on purpose. On the user side, we learn a separate expert for each engagement target. A click expert can represent what someone is likely to click, while an upvote or good-visit expert can represent a different kind of intent.

On the item side, we do the opposite: every candidate gets one shared embedding. Because all candidates live in the same embedding space, we only need one ANN index.

At request time, a product surface chooses a set of task weights, mixes the corresponding user embeddings, and sends the result to the shared index. The surface gets a retrieval policy tailored to its goals without retraining the item tower or rebuilding the index.

• In practical terms, SnooRec gives us:

• one item embedding per candidate;

• one shared ANN index;

• multiple task-aware user representations; and

• one ANN query per retrieval request.

The shared item embedding also acts as an anchor. Every task-specific user representation has to express its preferences in the same vector space, which makes it possible to linearly combine the user representations.

User Tower Architecture

The user tower processes categorical features, sequential interaction history, and real-time contextual signals into a shared base representation h̃ᵤ. This representation is then routed through a set of task-specific expert layers. Each expert Eₜ corresponds to an engagement target t ∈ 𝒯 and is implemented as a multilayer perceptron with layer normalization and ReLU activations.

The task-specific user embedding is defined as: uₜ = Norm(Eₜ(h̃ᵤ))

Candidate Tower Architecture

The candidate tower maps item-side features into a unified embedding space. Because the candidate tower is shared across tasks, the entire corpus of recommendable items can be indexed into a single approximate nearest neighbor (ANN) datastore.

The embeddings do more than retrieve candidates

SnooRec is used in two complementary ways.

First, it is a candidate generator. A task-weighted user embedding queries the shared ANN index and retrieves personalized candidates.

Second, it is a representation layer for downstream rankers. Ranking models can consume the user embedding, the item embedding, and interaction features derived from the pair, such as task-specific dot products or shorter Matryoshka prefixes.

This second use is important. SnooRec is not only the first stage of a recommendation funnel. It gives later stages a shared view of user intent and item similarity, so every ranker does not have to relearn those representations from scratch.

The same underlying representation can therefore support retrieval, early ranking, and as a feature in final ranking models.

Optimization Strategy

During training, only turn on the expert for the active task

Each training example comes with an observed action: a click, an upvote, a good visit, or another engagement target.

If the example is an upvote, we run and update the upvote expert. The other task-specific heads sit that example out. The shared user encoder and item tower still learn from the overall training stream, while the task-specific gradient stays focused on the action that produced the label.

Under the hood, we train with an InfoNCE contrastive objective and in-batch negatives. The model learns to place the observed user–item pair closer together than the negative candidates in the batch.

We use dot-product similarity during training because that is also what the ANN index uses at serving time. Put differently: we train the geometry that production will actually query.

Contrastive Loss Formulation

The parameters are optimized using an InfoNCE contrastive objective over in-batch negatives. For a query embedding u = uₜ⋆ and a positive candidate embedding v, the loss function scales with a learned temperature parameter τ:

Inference-Time Multi-Objective Mixing

At serving time, task weights are used to weigh different objectives. One surface might emphasize clicks. Another might put more weight on long-dwell visits or upvotes. SnooRec computes the task-specific user embeddings with nonzero weights, takes their weighted sum, normalizes the result, and sends one query to the shared ANN index.

Before normalization, a candidate’s score is the same weighted mixture of its task-specific similarity scores. Normalization scales every candidate score in that request by the same constant, so it does not change the resulting order.

This gives us a useful separation of concerns:

- the item index describes the shared content space;

- the user experts describe different kinds of intent; and

- the task weights describe what a product surface currently wants from that space.

Imagine a surface that wants to balance immediate clicks with longer, more meaningful visits. Rather than choosing one objective and building a new retrieval model around it, the surface can mix the click and good-visit user representations before querying the shared index.

Teams can change the retrieval objective at serving time without retraining SnooRec or rebuilding the candidate index.

Matryoshka Learning - One embedding, with several size budgets

SnooRec embeddings are used in multiple stages of the recommendation stack: full-size vectors for ANN retrieval, and dense embedding features for downstream ranking models. These consumers have different latency, memory, and scoring-cost constraints, so SnooRec uses a Matryoshka Representation Learning objective to make smaller vector prefixes useful on their own.

During training, we optimize the retrieval loss across multiple embedding dimensions:

The production ANN index uses the full embedding size (256) for candidate retrieval. Downstream rankers can consume the same user and item embeddings as dense features, either at full size or by using shorter prefixes (32 or 64).

This lets latency-sensitive rankers trade off feature cost and representation quality without requiring a separate embedding model. In practice, the same SnooRec representation can support retrieval, early ranking, and final ranking while giving each stage control over its embedding footprint.

Feature Dropout for Robust Embeddings

User interaction history can vary significantly in length, so does feature completeness across users. To make SnooRec more robust to missing or less dense sequence features, we randomly drop a small percentage of input features during training (10%). This encourages the shared user encoder to learn representations that remain useful when signals are sparse, delayed, or unavailable.

It has also made SnooRec easier to reuse across retrieval sources with different feature sets. For example, we were able to support a geo-specific retrieval source using regional features in the shared user encoder rather than training a separate retrieval model.

Controlled Exploration with Top-k Temperature Sampling

Recommendation pipelines often pass the highest-scoring retrieval candidates directly to downstream ranking stages. This is efficient, but it can also make retrieval overly deterministic. Over time, always selecting the top candidates can concentrate recommendations around a narrow set of familiar communities or content types, reducing opportunities for discovery.

SnooRec introduces a lightweight serving-time exploration mechanism after ANN retrieval. Instead of always taking the top-n candidates from the retrieved set, we sample from the top-k candidates using a temperature-scaled softmax over retrieval scores:

Lower temperatures keep the result close to deterministic top-ranked selection. Higher temperatures flatten the distribution, giving lower-ranked candidates within the retrieved set a greater chance of being selected. This gives product teams a single serving-time knob for adjusting the balance between immediate relevance and exploration.

In practice, the effect of temperature depends on the scale of the retrieval scores. The softmax distribution is controlled by score gaps relative to temperature: when score gaps are large compared to the temperature τ, sampling remains concentrated near the top and we only recommend high scoring candidates; when score gaps are small compared to the temperature τ, sampling becomes more exploratory.

To make this knob stable across serving traffic, we calibrate temperature using the historical standard deviation of retrieval scores: τ = α · σ̂ₛ. Here, σ̂ₛ is estimated from a sample of results, and α controls exploration strength. Values below 1 favor exploitation, while larger values increase exploration.

We also evaluated query-level calibration, where temperature is scaled by the score standard deviation for each individual retrieval request. In online experiments, this did not produce meaningful gains over the simpler global calibration. We therefore use the global calibration in production: it is simpler to serve, easier to tune, and provides a stable exploration control without retraining SnooRec or rebuilding the ANN index.

Evaluation

Public Benchmark: Taobao User Behavior Dataset

We verified the multi-task capacity of SnooRec using the public Taobao dataset, tracking a behavioral funnel consisting of Page View, Favorite, Cart, and Buy interactions.

The empirical results show strong cross-task generalization. Activating experts optimized for later funnel stages (Cart, Buy) significantly improves performance on early-stage behaviors (Page View), demonstrating that deep intent signals capture cleaner, more robust representation features.

Production Results at Reddit

An online A/B experiment deployed on Reddit's Home Feed evaluated SnooRec against our previous multi-task baseline (which utilized a shared encoder without explicit task differentiation). By isolating expert activations using one-hot weight vectors, we observed strict alignment between configured task weights and target online engagement metrics:

Note: Bold entries denote statistically significant lift over control (p < 0.05).

Tuning the system toward the good_visit expert yielded a +0.28% lift in topline user sessions alongside a +1.21% increase in long-dwell post consumption, validating the efficacy of runtime objective modification.

Exploration Trade-offs

We evaluate the impact of temperature sampling to observe the trade-off between community discovery and immediate user satisfaction.

Temperature sampling increased subreddit diversity, but it also increased explicit negative feedback. This was an important product lesson: exploration can improve discovery, but the temperature needs to be tuned per surface and monitored with guardrail metrics. In practice, SnooRec’s exploration controller gives teams a lightweight serving-time knob for balancing immediate relevance against longer-term discovery.

Cross-Surface Generalization

We evaluate SnooRec in two deployment modes: as a retrieval model that directly generates candidates, and as a representation layer whose user and item embeddings are consumed by downstream ranking models.

Conclusion

Recommendation systems have a natural tendency to accumulate: one model for clicks, another for upvotes, another for dwell, and another index behind each of them.

SnooRec gives us a different pattern.

Task-specific user experts capture different kinds of intent, while one shared item tower keeps Reddit content in a common embedding space. Product surfaces can mix objectives at serving time, query one ANN index, and reuse the same representations in downstream rankers.

Since early 2025, SnooRec has served as both a production retrieval system and a shared representation layer across multiple Reddit recommendation surfaces.

The result is a common foundation that lets teams share more and specialize where it matters – one model, one item index, and more ways to use it!

What’s Next

There is still plenty to explore along the themes 1) improving multi-embedding representations, and 2) providing more product levers within a single model.

On 1), we are exploring richer expert-sharing approaches, including mixture-of-experts architectures that allow learned routing to experts for related tasks to share useful capacity without collapsing into the same objective.

On 2), we’re looking into learning an inference-time popularity bias parameter that allows controlling how popular versus niche the retrieved candidates are. This helps us do both exploration as well as finding the most engage-able content, without needing to maintain two models

Want to help push this work further? Join Reddit to research and deploy our next model iterations (https://redditinc.com/careers)

Acknowledgements

We would like to thank everyone who has contributed to this work particularly Cathleen Li, Brian Kimmig, Javier Herrera, and Shafi Bashar.

Written by Andrew Gunsch and Randy Goodman.

Problems don’t have to be complex in order to have complex solutions. Few areas of Reddit’s infrastructure demonstrate that better than its Developer Platform.

On its surface, our system’s main function is simple: “run the JavaScript code that developers upload as Devvit apps”. But there are two wrinkles that make this problem much harder to solve than it is to understand.

Firstly, there are a lot of users out there uploading a lot of code. While we do provide the compute cycles for Devvit apps, we can’t afford to run all of the apps all the time. We need them scaled down when not in use. Second, a single VM doesn’t cut it either — the most popular of our apps will quickly run out of available CPU to serve all of their traffic if running on a single instance.

Essentially, we need to be able to scale up from zero instances of a user-created app, to an arbitrarily large number of instances of that app, all in real-time. For those of you thinking to yourselves “Finally, an enterprise use-case for Lambdas / Cloudroutines”, keep dreaming.

Our constraints:

• Scale-to-zero architecture, so we’re not burning compute cycles on apps users aren’t interacting with.

• Autoscaling up pods for high-traffic apps based on pod CPU and requests backing up.

• A target of 200,000+ deployments (Devvit app installations) per cluster.

• A cold-start time of <1 seconds, so that mod tools (infrequently-used installations) are reasonably responsive.

Living with Devvit’s choices from 2024

We have a service we call “compute-go” that has served us for the last two years, but is reaching its limits. We’ve got a few reasons we’re trying to rearchitect:

• We’ve had degenerate scaling behavior as the system has grown. Compute-go pods are “parents” of child Devvit-app pods, but incoming requests are evenly distributed across the fleet. We run ~20 instances of compute-go, which can mean an app getting 1 RPS gets spread over 20 pods!

• Compute-go has acquired a lot of complexity over time, with the team’s bus-number for it decreasing and its criticality remaining high. Its design came from a time when we imagined the network topology of Devvit apps looking really different, and carried complexity deep within its bones. After a few years of iterating on Devvit itself, we wanted to build something simpler modeled around its current assumptions and DX.

• We wanted to give ourselves more optionality on our compute infrastructure, so we can evaluate cost and performance tradeoffs of different runtimes.

• Finally, we wanted to reduce operational overhead, and are preparing to migrate Devvit apps to clusters better supported by Reddit’s internal infrastructure teams. A design that lets us have apps in multiple clusters and smoothly migrate them will help with that, and gives us future options for redundancy and scalability.

Rebuilding and hitting walls

There are out-of-the-box infra solutions to problems that are similarly shaped, but can’t handle the breadth of our scale. We dropped by the Kubernetes Slack to talk about the feasibility of scaling 150,000 different deployments using KEDA and were told “that would be the largest cluster they’d ever heard of”. And as much as we like using tools in interesting (and unintended) ways, we ran into a few practical limits that had us going back to the drawing board:

• The HPA Scaling Loop can only handle ~2,000 HPAs in a cluster before not being able to keep up with running every 15 seconds. Slowing the loop down to every 60 seconds could bring that up to ~10,000 HPAs, but at the cost of some responsiveness, and still significantly less than we need.

• Kubernetes Services are limited in managed clusters, with GKE and EKS both specifying hard limits of 10,000 per cluster.

• Managing too many Kubernetes Deployments can even start to become a bottleneck.

The recommended option we found here was to scale Kubernetes clusters horizontally — with limits this low, we would need a few dozen clusters, and tooling for managing them like cattle, not pets. We’ll add more clusters when we need, but with the vast majority of our apps asleep most of the time, we felt strongly we should get more out of our existing clusters before stamping out new instances.

So after an extensive period of design discussion, followed by answering management’s question of “Are you really really really REALLY sure?” the customary 14 times, we decided to roll a solution ourselves.

The re-design: 500,000 apps in one cluster

We landed on a design that separates components by config-time vs. run-time needs, and separates components by their scaling properties. Here’s what it looks like:

Config-time: the Orchestrator

With this new design, we separated out “getting the cluster into the right state” as a configuration-time concern — when moderators install a Devvit app into a subreddit, that creates the conceptual context for that app to run in. We create and configure a DevvitComputeContext resource (our own custom type) as a result for each — though to reduce pod count and pod churn in our cluster, we combine installations using the same app + version into the same compute context.

This service is designed like a Kubernetes controller — an informer and watch loop, reconciling the state of the cluster to the desired state. It watches our database of app configuration information, and continuously reconciles the DevvitComputeContext CR in the cluster to match. It also injects a long-lived JWT that is used to grant a specific Pod access to request that Devvit app’s code at startup.

Runtime: the app proxy

When a request comes into the cluster, the first component to touch it is devvit-app-proxy, which primarily just handles routing requests to app pods.

It keeps track of all of the cluster’s Pods using an informer, and routes requests to the right Pod based on its labels. If a Pod doesn’t exist for the given Devvit app version, it asks devvit-autoscaler to spin one up, waits until it becomes Ready, then delivers the request (usually around ~500ms!).

It also logs in-flight requests in Redis to help devvit-autoscaler make scaling decisions.

This lets us replace Kubernetes Services by using our own proxy as a routing layer, since we have consistent logic across the fleet for which pods match the label, eliminating one of our cluster scaling limitations.

This service is stateless and easily scales instances up and down based on its CPU. Its logic is simple, and most of the complexity comes in handling edge cases like quarantining and replacing pods when they start refusing connections.

Runtime: the autoscaler

The devvit-autoscaler service has most of the “brains” of this system, handling both 0 → 1 scaling for Devvit app versions (when devvit-app-proxy requests a first instance) and 1 → N (when requests back up or CPU gets high).

It maintains a pool of “standby pods”, all running the same NodeJS base image that we can load Devvit app code into. This lets us skip Pod creation and startup time, reducing median cold-start time from 4 seconds to 500ms. Autoscaler regularly runs a “replenishment” loop, ensuring the standby pool is sufficiently populated, and cleaning up any Pods that have gotten into a stuck/bad state. This optimization would be harder to keep if we were running a fleet of arbitrary Docker images, but we can easily support one base-image runtime and probably a small set of other runtimes with this speedup.

The autoscaler determines how many Pods/instances of each app version there “should be”, using two criteria: how many concurrent requests are being made to that app version, and how much CPU time that app version’s Pods are using.

This design lets us replace Kubernetes HPAs by taking advantage of using the same scaling criteria for all apps in the cluster — we can collect all of the pod CPU metrics with one query, and then all of the request metrics from Redis with one MGET, and make any scaling decisions in-memory. This is vastly more efficient than the normal HPA loop in two ways: we only have to consider currently-”awake” apps (usually <1% of all of them at any given time), and we don’t have to check every HPA definition and collect separate metrics for each since we’re using the same criteria across apps. This resolved our most critical cluster limitation!

The autoscaler also lets us replace Kubernetes Deployments — we were hoping to keep this one, but the practicalities of managing 200,000+ Deployments in GKE made this infeasible. So instead, autoscaler also takes on a primitive version of Deployment’s responsibilities, tracking desired replica counts for each DevvitComputeContext, and pulling Pods from the standby pool when scaling up is needed. This resolved the final cluster limitation we encountered.

In terms of scaling the autoscaler itself, we only need one instance of this running — though we keep a second alive for a quick fallback, using a k8s lease for leader-election and handoff. It can typically run its autoscaling loop in <100ms across the app fleet. And if it ever does start to bottleneck, we can always consider sharding the cluster’s Pods across a few instances of autoscaler.

Putting it to the test

A wise u/fuzzypercentage once said, “loadtesting is just a for-loop”, and I think that succinctly summarizes our strategy for stress-testing our hosting implementation. First, because ultimately we’re just using a big for-loop to send traffic to the system; second, because of the cyclic learn-and-fix nature of load testing.

We loaded one or two (or, in our case, 500k) dummy deployments, and then sent many times our production request traffic to those deployments to see what broke. When something inevitably broke, we’d ask “what?” and “how?”, and if we didn’t have enough information to answer those questions, we’d add instrumentation until we could. It was a very effective way to bolster the observability toolchain while getting the services production-ready.

Our load testing in this case centered around verifying that the proxy and autoscaler could keep up with high production traffic, rapid changes in cpu load, and that all of these services remained responsive when enormous numbers of deployments already existed within the cluster. At 500k+ deployments, we found that our observability tooling was failing before these services did.

Conclusion

We’re still ready to shard clusters horizontally when we really need to, but this design lets us push deep into a single Kubernetes cluster and scale for the next 10x of Devvit growth on the way to that point, while simultaneously making our architecture simpler to reason about.

It’s always a little daunting when deciding to implement something from the ground up. Our engineering intuition often has us asking ourselves the questions

• “Who else is working in this space?”

• “What solutions already exist?”

• “What design decisions did they make and why?”

• “Is our problem the same shape as theirs?”

• “Can we build our solution on top of theirs?”

When we find out we can’t re-use a lot of the existing work in this space, it then becomes tempting to try and change the problem so that it does fit the existing solutions, or even re-consider if we’re solving the right problem. It takes courage to make the investment into solving that problem anyway, especially if you have to break fresh ground doing so. It’s our hope that someone in the future will find themselves solving the same problems we are here, asking themselves the same questions, and ultimately may find some utility in our contributions to the developer ecosystem.

By Parineeta Sinha

Introduction

The Release Engineering team at Reddit is responsible for managing mobile releases, shipping beta and production versions, and coordinating testing efforts with Quality Engineering (QE) teams. 

Over the past decade, our process has evolved significantly. Initially we were releasing our iOS and Android applications once every two weeks. This process involved a two-week beta cycle designed to catch functional issues and crashes. Consequently, the mean time for code to move from the main branch to production was approximately one month. Our process included cutting a new branch every two weeks, baking it in beta for two weeks, and then starting the production ramp. The testing strategy relied on an offshore QA team that executed a suite of 3,000 manual tests, typically taking about a week to provide release sign-off.

In 2021, we transitioned to a weekly release cycle. This shift required us to cut a branch every week, but we still relied on the same manual QA pipeline.. We addressed potential testing gaps by allocating additional people for manual testing and successfully reducing the regression turnaround time from five days to three days. However, the increased release frequency necessitated removing open beta testing groups, limiting the beta environment to employees only.

In the subsequent years, we invested in our testing infrastructure. Moving away from manual testing to a pipeline of automated unit and integration tests enabled us to reduce our regression test cycle time from 3 days to a couple of hours. This helped us to bridge the gap between testing and development, and we shipped releases faster. We have been relying mostly on our suite of unit and integration tests to validate releases since. We also added open beta testing again in our release pipeline. This helped us test our changes with a select group of users and get early signals on crashes and functional issues through our beta feedback channels.

Operationalizing the Weekly Cadence

Since transitioning from a bi-weekly to a weekly release cycle in 2021, Every Wednesday, the Release Eng team cuts a new release branch and deploy a beta build following smoke test validation. This build undergoes a four-day stabilization period before moving to production on Monday.
On Monday, we kick off a "Go/No-Go" process for iOS and Android release candidates. Once we receive sign-off from all critical stakeholders, we proceed with production deployment:

• Day 1: The app is released to 1% of iOS users and 3% of Android users.

• Day 2: If no issues are encountered on Day 1, the rollout proceeds to 100% of users for iOS. For Android, we adopt a more gradual approach and ramp to 30% users at 500k adoption and 100% users at 1M adoption, if all critical app health metrics look good.

If the approval is delayed past 3:00 PM PST on Monday, the rollout is shifted to the following day. Hard freezes for code changes are enforced at 12:00 PM PST on Fridays to ensure weekend stability.

Cross-Functional Collaboration

When a version begins development, a dedicated channel is created to facilitate cross-functional coordination  in Slack. This channel is predictably named using our version number so it is easy to find. 

These channels serve as the centralized hub for all communications during the release process, covering development, beta, and production phases.

Key aspects include:

• Alerting and Triage: Automated alerts and issue reports from various detection channels like Crashlytics, manual and automated tests are surfaced here. Stakeholders are then pulled in or tagged to address these issues.

• Release Management: These channels are used for the entire release lifecycle, including late merges, beta release coordination, and hotfixes.

• Workflow Integration: The creation of these channels and the addition of necessary users is automated and is part of the standard release engineering workflow to ensure stakeholders are properly informed and aligned.

We use these channels for conversations about the upcoming release during development, in order to stabilize the beta and support its production rollout (to be covered in next weeks’ blog post)

Stabilizing the Beta

A late merge is an unusual occurrence that allows critical fixes into the current release candidate (Beta). Every request is reviewed by the Release Engineering team. The late merge process is designed to be technically rigorous, transparent, and decentralized, shifting the focus from management approvals to engineering peer reviews directly on the Pull Request (PR). The goal is to maintain release confidence while reducing administrative friction.

Criteria for Late Merge

Only specific types of changes are eligible for a late merge into the release branch:

• P0 Bugs: Critical issues that require immediate resolution.

• Publicly Announced Features: Fixes for gated features under feature flag already promised for the current release.

• Gating Fixes: Correcting ungated or incorrectly gated code.

Timing & Cutoffs

The method of merging depends on when the request is made:

• Until Friday Cutoff (12:00 PM PST): If the PR receives approvals from both groups and has no outstanding comments, the process uses automerge to merge the code and trigger a new beta build.

• After Friday Cutoff: The bar is higher. Only P0 break-fix changes are accepted. The Release team reviews these manually, and the team may manually cut a new beta build if necessary.

Quality Assurance and Triage Process

The QA process is tightly integrated with the release cycle. Once a release candidate is identified, the following testing protocols are initiated:

1. Automated Testing: The automated regression testing cycle begins by running a suite of ~1500 P0 and P1 test cases across all features on the release branch.

2. Manual Validation: In parallel, the QA team executes manual validations on user flows that cannot be automated eg: device specific behavior, visual quality, exploratory testing for edge cases etc.

Once regression reports are provided by both manual QA and automated QE, the release team reviews the findings and triages any identified regressions or P0/P1 issues on the release candidate.

Critical bugs are addressed using one of two methods:

• Feature Flag Toggle: The bug can be fixed by turning off the feature flag associated with the regression.

• Code Fix: If turning off the flag is not possible, a code fix requires to go through the late merge process. Or depending on where we are in the release it may require a Hotfix

Hotfix process

A hotfix is initiated for critical (P0/P1) bugs when toggling an associated feature flag is not a feasible resolution. If any fixes are made on the release branch, the candidate is sent to QA to run the manual smoke test suite, which includes P0 and P1 test cases across all features. The decision to do a hotfix release follows an exhaustive analytical approach to exhaust any other avenues that could help mitigate the issue in production.

Hotfix Post Mortems

Reddit utilizes a blameless post-mortem process, which is mandatory for all Hotfixes. The post mortem meetings are typically led by the team responsible for shipping the production fix. The primary objective is to reflect on the incident, specifically how it was identified and resolved, while surfacing process gaps, key learnings, and actionable improvements for the future. If you’d like to learn more about our incident reivew process, read https://www.reddit.com/r/RedditEng/comments/1se5gmu/incident_reviews_or_how_we_transform_outages_into/ 

Challenges of Ship Fast, Ship Confidently

Weekly releases require a delicate equilibrium between maintaining high development velocity and ensuring UI test alignment. When issues escape into production, resolution is costly. Feature flags or killswitches offer rapid mitigation, but traditional code fixes are expensive, necessitating resubmission and exposure to mandatory, time-consuming Apple and Play Store review cycles. Furthermore, while phased rollouts are critical for safeguarding stability, they inevitably introduce friction by slowing down feature adoption. Additionally, release rejections can derail the entire pipeline, significantly increasing on-call overhead and forcing teams into incident mode to resolve blockers as quickly as possible.

Test Strategy and Cross-Functional Collaboration

Feature teams typically ramp launches through controlled experiments rather than making a feature immediately available to everyone. They begin with a small, representative user cohort, monitor key product, performance, and quality metrics, and use the results to identify issues or unexpected behavior. If the experiment performs well, exposure is gradually increased across additional cohorts or markets, with safeguards and rollback plans in place at each stage. Once the feature meets success and stability criteria, it is fully enabled for all eligible users. Working with cross-functional feature and Quality Engineering (QE) teams to validate experiments before ramping up for iOS and Android users involves a structured, collaborative process focused on test strategy, validation activities, and continuous automation alignment.

The core strategy for validating experiments is driven by a partnership between Quality Engineering and feature teams. The overall goal is deployment safety and shipping confidently, especially since experiment ramp-ups are a major source of production bugs. Experiments are tested in both pre merge and post merge workflows by:

• Enabling experiments on demand while running UI Tests in Pull Requests

• Enabling experiments with > 45% exposure during nightly UI test runs

Participating in Playtests (Manual Validation)

While automated testing is the primary reliance for validating releases, manual efforts are still integrated, often serving as the "playtest" phase.

• Manual Validation Execution: The Quality Engineering team executes manual validations, which are critical for user flows that cannot be automated.

• Release Triage: Manual QA provides regression reports which are reviewed by the release team to triage regressions and P0/P1 issues on a release candidate.

• Developer Involvement: Feature developers actively collaborate on creating/reviewing new test cases in collaboration with the QE team which the QE team then prioritizes for automation to keep automated regression pipeline uptodate

Keeping Automation Aligned with New UI Changes

Maintaining test suite quality and ensuring UI test stability are recognized challenges of the weekly release cadence. Alignment efforts focus on empowering developers and improving framework stability.

• Simplifying Experiment Activation: To make it easier for developers to run experiments, the process for enabling experiments for automated tests running on CI has been simplified.

• Framework Stability: QE updates the automation framework to ensure greater stability, such as updating a consistent rule for activating experiments, which helps eliminate flaky tests in nightly regressions.

• Adapting to New Features/Experiments: Test plans and automation are continually updated to reflect new UI/experiment changes. 

Bridging the Gap: iOS vs Android Release Realities

The open beta cycle is a crucial step in Reddit's release strategy for iOS and Android, reflecting the distinct ecosystem constraints of each platform:

• iOS Open Beta: Testing is invite-only and constrained by a strict cap of 10,000 seats.

• Android Open Beta: Testing is significantly more scalable, with beta groups reaching sizes on the order of 200,000 seats.

Regardless of the platform, the primary goal of reintroducing open beta testing into the release pipeline in 2024 was to enhance pre-production stability and catch functional issues before the full rollout. Reddit identifies and prioritizes beta issues through a multi-layered approach that combines automated detection, manual validation, and centralized triage workflows.

Identification of Beta Issues
The identification process relies on following primary channels:

• Employee Dogfooding: Employee dogfooding at Reddit is central to the company’s product strategy, as it ensures teams align their work with the actual user experience. Employees are encouraged  and incentivized to log bugs for features that are ramped as experiments.

• Open Beta testing feedback: Open Beta testers report functional issues and provide feedback on features through our feedback subreddits. (Join Reddit’s Open Beta Tester program)

• Automated Regression: The pipeline runs a suite of approximately ~1500 P0 and P1 test cases on the release branch.

• Manual Validation & Playtests: Quality Engineering (QE) performs manual user-flow validation for scenarios that cannot be automated.

• Crash Monitoring: The team utilizes Crashlytics to monitor all crashes on released beta builds, which is integrated with CrashBot(internal tool for Crash alert and ticket filing) for automated detection and filing.

Conclusion

Successfully managing a weekly release cadence at scale is not just about the technical pipeline; it is an evolving practice of balancing speed with quality. By shifting from manual validation to robust automation, re-introducing open beta programs, and fostering a culture of blameless reflection and cross-functional collaboration, we have transformed our release process from a rigid cycle into a dynamic, reliable engine. While the challenges of shipping fast and confidently will always persist, our commitment to continuous process improvement, grounded in rigorous triage and data-driven decision-making ensures that every launch is an opportunity to learn, iterate, and ultimately deliver a better experience for our users.

By Michael Fullan

Solidifying the foundational infrastructure (part 1) and language picker screen (part 2) solved a lot of the localization problems we were dealing with in the Android app. But even with these pieces in place, each new language rollout came with its own set of challenges and learnings.

In this final part of the Android localization series, I’ll talk about some of those things we’ve learned along our journey to supporting 40 (!) languages!

RTL Support

To correctly support right-to-left (RTL) languages in a mobile app, the guiding principle is that the entire UI should essentially be a horizontal mirror image of its left to right counterpart.

Some aspects of this were immediately obvious, like ensuring that RTL language text flowed from the right to the left. But as we researched deeper we found that there’s a lot more to it:

• The cursor in text input fields should start on the right and move left as you type

• Arrows and other icons that indicate direction, reading/writing direction or motion should be flipped horizontally

• Toggles should be flipped horizontally. A toggle that you swipe right to enable in LTR should require a swipe to the left in RTL

• Swiping gestures should be a mirror of their LTR counterparts. If a left swipe moves you forward in an image gallery in LTR, a right swipe should do the same in RTL

• A11y - screen readers should respect RTL ordering

This felt like a lot when we were first starting out, so we did some baseline assessments on all of our supported platforms to see what we were up against. Web and iOS were both terribly broken when we turned on RTL layout direction, but Android was in surprisingly good shape before we did any fixes or changes!

Most of that is thanks to our extensive use of Jetpack Compose, which automatically handles most of the RTL mirroring for you. As we went through our app auditing each screen, the easiest solution when issues came up on legacy screens was often just to migrate the UI to Compose!

Audit Flow

Our basic process when testing our app for RTL-readiness was to take a screenshot of a screen in LTR mode, create a horizontal mirror image using a pure image transformation, and then compare that against the result when we turned on “Force RTL layout direction” in the OS Developer options menu. As an example, here are the results from our initial assessment of the language and translations settings screen:

When you compare the last two screenshots, it’s easy to see that the overall RTL layout looks correct. There were a few icons that needed to be flipped (the back arrow in the top app bar, the trailing caret in the list items), but this screen was largely good-to-go out of the box.

Other Strategies

Beyond this basic audit flow, there weren’t a lot of “silver bullet”-type strategies we found to prepare our app for RTL. A global find and replace for left/right -> start/end for paddings and margins helped, but most issues beyond that required some small local tweaks in the affected areas.

We did start to rely on RTL compose previews and screenshot tests to help prevent regressions, both of which are easy to set up.

@Preview(locale = "ar", name = "Arabic (RTL)")
val paparazzi = Paparazzi(deviceConfig = DeviceConfig(locale = "ar"))

Pseudolocales

Android supports two pseudolocales for localization testing that can be useful even before you have your app strings translated. They also make it easy to spot text that isn’t set up correctly for localization.

One thing to watch out for with pseudolocales is that the value of the “canary string” will also get pseudolocalized when it is resolved. Adding an additional check for this case will help prevent strange issues when using either pseudolocale for testing.

enum class Pseudolocale(val locale: Locale, val canaryString: String) {
 /**
  * Adds Latin accents to the base English UI text, expands the original text by adding non-accented text,
  * and brackets each message unit to expose potential issues from expanded text. Potential issues can be
  * layout breakage and badly formed message syntax, such as a sentence split into multiple parts
  * displaying as multiple bracketed messages.
  */
 AccentedEnglish(
   locale = Locale("en", "XA"),
   canaryString = "[éñ-ÛŠ one]",
 ),
 /**
  * Sets the text direction of the original left-to-right messages to the right-to-left direction,
  * which reverses the order of the characters in the original message.
  */
 RtlPseudo(
   locale = Locale("ar", "XB"),
   canaryString = "\u200F\u202Een-US\u202C\u200F",
 )
}

Other 1-off Language Quirks

One of the fun things about working in the localization space is getting to learn a little bit about different languages as we add them to the app. There have been a few interesting surprises that have come up along the way, so let me share a few tips that we’ve learned!

Bengali Numerals

The Bengali language has its own set of numerals, but with tech users highly habituated to English interfaces, it’s common to see Western digits (0-9) kept intact inside an otherwise Bengali UI.

It turns out that there’s an easy way to do this by specifying the Latin script numbering system as an extension to the language tag.

Locale.forLanguageTag("bn-IN-u-nu-latn")

Here bn is the Bengali language code, IN is the country code for India, u is the extension signifier, nu is the extension key (for numbering system), and latn is the extension value (for Latin script digits).

As long as you call AppCompatDelegate.setApplicationLocales with this full Locale object, everything works great! This is a really useful trick that we’ve since used for other languages as well.

Thai Line Breaks

The Thai language is written continuously with no spaces between words. Spaces are only used like punctuation, to mark the end of a sentence. 

When we did a first round of testing with Thai, I remember getting a bug report back like this that flagged this line overflow as being incorrect.

Now I’ve seen my fair share of localization overflow/truncation bugs, but initially I had no idea how to fix this. What were we supposed to do with continuous text this long? Where were we supposed to break the line?

The best solution for this type of issue with Thai is to use the zero-width space character (\u200B) within the Thai translations. It’s an invisible character that simply acts as a “soft break hint” to the rendering engine, indicating where it’s a grammatically safe place to wrap a line if needed. Working together with our translators, we were able to use this character where needed to address a variety of line-wrapping issues that came up like this.

Legacy Language Codes

Java is a great programming language. It’s over 30 years old at this point, and still strives for backward compatibility wherever possible. This is often a really great thing, but there were a few standard language codes that changed around the birth of Java.

As a result, Android inherits some strange backwards-compatibility behaviors when dealing with these specific language codes (check out the docs for more info).

val locale = Locale.forLanguageTag("id-ID")
locale.language // "in"
locale.toLanguageTag() // "id-ID"
locale.getLegacySafeLanguageCode() // "id"

private fun Locale.getLegacySafeLanguageCode() = toLanguageTag().substringBefore("-")

I was definitely “astonished” the first time I encountered this behavior, but as long as you know which APIs generate which results you can make things work correctly. After a few iterations of testing, the TL;DR here is:

• Use legacy codes for your resource directory names

• List modern codes in locales_config.xml

• SplitInstallManager.installedLanguages will report modern codes

• Use modern codes when requesting a language install

Conclusion

We’ve certainly come a long way on our localization journey, and we still have plenty of work ahead of us. It’s been an enjoyable learning process so far, especially getting to work with a top-notch team here at Reddit!

If you see any localization bugs on any of our platforms, please continue to post them to r/bugs. We’re always monitoring and will do our best to fix them!

Hey r/redditeng! I’m Lisa. You probably don't know me, but I've been here the whole time. 

For the last six years, I've been Chief of Staff to Chris Slowe, Reddit's founding engineer and CTO (now our first Technical Fellow). I'm also the person who founded r/redditeng over five years ago with Jerry, Punit, Braden, and a couple others who joined me in getting it off the ground. We wanted to give Reddit engineers a place to talk about what we build, how we build it, and what we learned along the way. We also wanted to show you what it’s like to work here. Oh, and we wanted to show you our cats, too!

My last day at Reddit is coming up, so before I go, I want to say thank you.

Thank you first to all of you for reading. It's still a little amazing to me that thousands of people chose to follow a subreddit about how we build Reddit. Some of you got excited to work here because of posts you saw! And some of you got hired. Thank you for the comments, the questions, the upvotes. It means a lot to our writers. 

Thank you to everyone who has written something for us over the years. You are amazing! We've published something every week since the sub was launched on 4/15/2021, and that only happened because people were willing to take time away from their actual jobs to explain their work, share what they'd learned, and put their ideas out into the world, which can feel a little scary, no matter how much you know. One of my favorite parts of doing this has been helping make that a little easier. You’re my people. You’ve always been. 

Thank you to our Legal and Comms teams, who have supported us when we've needed them and, just as importantly, trusted us to do this well. That trust gave our little volunteer team and our engineers room to talk honestly about our work while helping us navigate the moments when we needed a little extra guidance.

And thank you to Chris for everything and for trusting me to start this in the first place and supporting it for all these years. Like me, he believes our engineers should have a place to tell their own stories, in their own voices. He gave us the space to make that happen.

Most of all, thank you to the engineering branding team, past and present. What started as a handful of us trying something has become a real team of people who care about telling these stories and making this community good. One of the nicest things about leaving is knowing that r/redditeng (or Snoosweek) doesn't need me to keep going. But, damn it, I’m going to miss it all. 

I'm very proud of what we made here. I’m happy it’s going to carry on and I can be one of your readers. I may be finishing my time at Reddit but I haven’t finished loving the people, the platform, or this sub.  

Thanks for reading all these years. And please keep reading. There's a lot more good stuff to come.

P.S. Paying the cat tax with the cats who help(ed) me with my work over the years here

By Michael Fullan

In part 1 of this series, we explored some foundational pieces of the localization infrastructure for the Reddit Android app. Getting this foundation in place was great, but to address a lot of the bugs we were getting we had to completely rewrite our in-app language picker screen. What seems like a simple screen on the surface actually has quite a bit going on, so let’s dive in!

Display Names

We generally omit country codes from the display name of a language unless multiple variants of that language are supported. We’re also displaying the localized name of each language, which you can get programmatically using locale.getDisplayLanguage(locale).

Order

To organize the list, the primary approach involves placing all Latin-script languages at the top in alphabetical order, with all remaining languages positioned below. Any time I’m adding a new language, I’ll usually compare the order of our in-app picker against the order of languages that the system shows users in the per-app language selection screen (accessed via App info -> Language).

Selected Value

In order to determine “which language is currently selected”, the app language tag flow in the AppLanguageProvider component from part 1 works great! The only extra case to worry about is the “use device language” or “system default” option.

For that case, we can lean on AppCompatDelegate.getApplicationLocales(), knowing that an empty list signals that the app is set to this default setting.

class MyAppLanguageProvider ... {
 ...
 private val_isUseDeviceLanguageSettingEnabledFlow =
   MutableStateFlow(computeIsUseDeviceLanguageSettingEnabled())
 override val isUseDeviceLanguageSettingEnabledFlow: StateFlow<Boolean> =
   _isUseDeviceLanguageSettingEnabledFlow.asStateFlow()

 override fun isUseDeviceLanguageSettingEnabled(): Boolean = isUseDeviceLanguageSettingEnabledFlow.value

 override fun onConfigurationChanged() {
   ...
   _isUseDeviceLanguageSettingEnabledFlow.value = computeIsUseDeviceLanguageSettingEnabled()
 }

 private fun computeIsUseDeviceLanguageSettingEnabled(): Boolean =
  AppCompatDelegate.getApplicationLocales().isEmpty

}

Now with both flows in place, we can combine them by first checking the “is use device language setting enabled” value and falling back to the app language tag if needed.

The first time we tested this we did come across one edge case: switching from “use device language” to “English” on a device already in English. Since the app language doesn’t actually change in this scenario, there is no configuration change triggered that would normally invalidate our caches. This situation can happen from within your in-app language picker or outside of your app in the system per-app language selection settings.

Solving for this case on internal changes is pretty straightforward - just adding a cache-invalidating callback anytime a user selects a new language in the in-app picker. For external changes, it’s a bit harder as we have no signal that a change like this has happened. The best option we’ve come up with is to use a foreground listener with ProcessLifecycleOwner and invalidate our caches whenever the app comes back to the foreground.

Switching Languages

Before Android 13, the “standard” way of switching an app’s language was to use a technique known as “context wrapping”.

object LocaleHelper {
 fun wrap(context: Context, languageTag: String): Context {
   val locale = Locale.forLanguageTag(languageTag)
   val config = Configuration(context.resources.configuration)
   config.setLocale(locale)
   return context.createConfigurationContext(config)
 }
}

open class BaseActivity : AppCompatActivity() {
 override fun attachBaseContext(newBase: Context) {
   val sharedPrefs = newBase.getSharedPreferences("Settings", MODE_PRIVATE)
   val language = sharedPrefs.getString("selected_language", "en-US") ?: "en-US"

   val wrappedContext = LocaleHelper.wrap(newBase, language)

   super.attachBaseContext(wrappedContext)
 }
}

In all of your activities, you had to look up the user’s saved language value in shared preferences, wrap the Context object with that language, and pass it down the line. Then, whenever a user selected a new language, you would save that value to shared preferences and manually restart the Activity to get everything to re-run. This approach worked, but was a bit cumbersome.

Android 13 introduced a new API for switching the app language - AppCompatDelegate.setApplicationLocales.

fun onLanguageSelected(newLanguageTag: String) {
  AppCompatDelegate.setApplicationLocales(LocaleListCompat.forLanguageTags(newLanguageTag))
}

With this 1-liner, all you need to do is wrap the user’s new selection in a LocaleListCompat object and let the system handle the rest! It will handle storage for you (no more shared preferences), syncs with the system per-app language selection settings, is backwards compatible with pre-13 devices***, and manages any activity recreation for you.

The only complexity that comes into play now when switching languages is supporting Google Play language splits.

*** Ok, AppCompatDelegate is backwards-compatible as advertised to an extent, but we have seen issues where it doesn’t properly initialize or update the application context on pre-13 devices. If you have any strings that get resolved via the app context instead of activity contexts, you might need to stick with context wrapping 😞. This Droidcon talk from 2023 explores some of the differences in more detail. (We even tried wrapping just the application context within the Application class using AppCompat storage, but it didn’t load fast enough).

Language Splits

If you use App Bundles, Google Play will automatically break your app's resources into separate modules based on language. When a user downloads your app, Google Play delivers the base code only along with the specific language resources that match the user's current device settings.

When we did some analysis on first-install download savings, we saw a significant impact from enabling language splits.

Now this is the Reddit app - a very large app with 40+ languages - but 8 MB of savings is still considerable! It’s also really simple to opt-in to the language splits feature.

android {
   bundle {
       language {
           enableSplit = true
       }
   }
}

The only tradeoff comes with the complexity in managing on-demand downloads of new language resources when a user picks a new language in the in-app language picker. I’ve found the KTX extensions for the Google Play feature delivery library really useful in setting this up in an idiomatic Kotlin fashion.

SplitInstallManager Integration

fun updateAppLanguage(locale: Locale) = flow {
 val languageCode = locale.language

 if (isLanguageInstalled(languageCode)) {
   applyNewLanguageSetting(LocaleListCompat.create(locale))
   emit(LanguageInstallState.Installed)
 } else {
   ...
 }
}

Our new language change process begins by checking if the language is already present on the device. One tricky thing to note here is that the SplitInstallManager ignores country codes. That is, if your app includes resources for fr-FR and fr-CA, resources for both country codes are downloaded when requesting resources for fr.

private fun isLanguageInstalled(languageCode: String): Boolean {
 val installedLanguages = splitInstallManager.installedLanguages
 return languageCode in installedLanguages || languageCode == "en" ||
   (installedLanguages.isEmpty() && !isSplitBundle())
}

private fun isSplitBundle() = context.packageManager.getApplicationInfo(
 context.packageName,
 PackageManager.GET_META_DATA,
).metaData?.getBoolean("com.android.vending.splits.required") ?: false

To check if a language is already installed, we can look at splitInstallManager.installedLanguages directly. We also check if the language to switch to is English (en), since we know the base APK will contain all the default resources (and that’s where our English strings are).

If you install your app directly from Android Studio, it won’t be packaged as a split bundle in the same way a production release build would get downloaded from Google Play by a user. The last set of checks covers this scenario - a debug build where all languages are included. The SplitInstallManager returns an empty list in this case, which we can use as a signal that we don’t need to request a download when switching languages.

If we’ve determined that we do in fact need to download a new language, here’s what we need to do (filling in the else block from the first code sample):

try {
 val sessionId = splitInstallManager.requestInstall(languages = listOf(languageCode))

 val progressFlow = splitInstallManager.requestProgressFlow()
   .filter { it.sessionId == sessionId }
   .transformWhile { state ->
     emit(state)
     !state.hasTerminalStatus
   }
   .map { state ->
     when (state.status) {
       SplitInstallSessionStatus.DOWNLOADING -> LanguageInstallState.Downloading(
         bytesDownloaded = state.bytesDownloaded,
         totalBytes = state.totalBytesToDownload,
       )
       SplitInstallSessionStatus.INSTALLED -> {
         applyNewLanguageSetting(LocaleListCompat.create(locale))
         LanguageInstallState.Installed
       }
       ...
     }
   }

 emitAll(progressFlow)
} catch (e: SplitInstallException) {
 emit(LanguageInstallState.Failed(e.errorCode))
}

After requesting an install using the SplitInstallManager, we can then monitor progress via a convenient Flow. The rest of the code is simply mapping the internal states returned by the library into view states for the language picker screen with all the necessary metadata attached.

Testing

Testing this flow end-to-end is tricky. Like I mentioned before, doing a standard installation from Android studio won’t give you the setup you need to test out an app language download. To do so, you’ll need to use the ./gradlew install{app_variant}ApkSplitsForTesting command.

You can also simulate a network error to test that part of the flow.

val fakeSplitInstallManager = FakeSplitInstallManagerFactory.create(context)
fakeSplitInstallManager.setShouldNetworkError(true)

Do you need one?

Once we finished all this work to get our in-app language picker working correctly, I came across a case-study from LinkedIn on the Android Developers blog that made me wonder whether we needed this at all.

The approach they took for language changes was to link out to the system per-app language settings screen and essentially delegate all the work to the OS. Sounds kinda nice, right?

The whole use-case we’re trying to solve for here is users that want to set their apps to different languages than the system. The lack of historical OS-level support for this functionality is what gave rise to in-app language pickers (and context wrapping, etc.) in the first place. With the release of Android 13 (API level 33, Aug 15, 2022), this problem was solved, but only for users on newer versions of Android.

Given that, I think the best way to make this decision is to check your minSdk version. If it’s less than 33 like ours, then you want to keep your in-app language picker around for those users that don’t have access to the OS-level settings. If it’s 33 or higher, then I think it’s absolutely reasonable to remove your in-app picker and all its complexity. The only consideration might be that you prefer the UX of keeping users inside your app instead of forcing them to leave and come back, but that’s more of a subjective call.

What’s Next

In the final part of this series, we’ll explore our journey to supporting RTL languages as well as some interesting 1-offs we’ve seen as we’ve added more and more languages to the app!

By Michael Fullan

I recently had the opportunity to speak at DroidCon about our Android localization work at Reddit. Localization is a hard problem: when you take an app with a large number of screens and multiply that by the number of languages you support, the surface area for potential issues is massive. There’s some pretty good reference content in the official Android Developers documentation, but a lot of it is scattered around different pages and can make it difficult to synthesize into a correct localization setup for your app.

At Reddit, we’ve felt this pain. Even as recently as the beginning of this year, we were getting regular bug reports in r/bugs specifically about issues with our Android localization (example 1, 2, 3, 4, 5, 6, …). It was a mess. In this 3-part series, I’d like to share details about how we’ve dug our way out of the mess and things we’ve learned along the way. We’ve landed in a much better place today with a reworked localization infrastructure, RTL support in the app, and far fewer bug reports than we used to get. Let’s dive in!

Basic Localization Infrastructure

Most Android developers know this rule: the first step in localizing your app is to put all your user-facing strings into resource folders. This is what allows the OS to load the correct translations based on the user’s device settings. Any hard-coded strings will prevent that from working correctly. I’ve found that AI is really good at hunting down any hard-coded strings that may be leaking out in your production UI, which is especially useful in large projects like ours!

MyProject/
└── src/main/res/
    ├── values/            # Default (Fallback) Strings
    │   └── strings.xml
    ├── values-es/         # Spanish Translation Strings
    │   └── strings.xml
    └── values-fr-rCA/     # Canadian French Translation Strings
        └── strings.xml 

We have a lot of Android developers at Reddit, so we’ve tried to design a workflow that makes it as easy as possible. The guidance we give our developers is to only update the default English values/strings.xml file when editing strings, whether it’s an addition, removal, or modification. 

Within that default file, each string needs to be accompanied by a descriptive comment attribute that serves as additional context for a translator. This is really important for getting the most accurate localization possible, as strings may get translated differently in different languages depending on the use case (clickable button text vs. text label, etc.). We have a lint check to enforce this rule, but we’ve added an opt-out option where a string can be marked translatable="false" for delimiters, test tags, or other reasons. Introducing this rule led to thousands of baseline entries initially, but again AI proved to be really useful at inferring the context and usages for a string and coming up with a solid description (an otherwise very tedious and time-consuming manual process!).

Translation Pipeline

Once the source strings are in place, the rest of the pipeline is designed to be automated and completely hands-off for developers. We use a cloud-based Translation Management System (TMS) like a lot of other apps, but we’ve developed a Python CLI tool to use in CI that helps abstract the specific vendor we’re using. This tool gets reused for iOS, FE, and all our BE services, and prevents the need of having any third-party SDK directly inside our application.

A nightly job will go through and upload the latest version of our source strings to the TMS. Human translators can then come in and provide translations for each supported language given the source string and context. Lastly, the pipeline pulls down the latest translations from the TMS, inserts them into the correct resource directories, and creates a PR to merge in the updates.

Other Config Bits

With your translations in place, there are only two other configurations you need to specify to get your localized app up and running.

The first is the resourceConfigurations block in your build.gradle file. Here you’ll need to list out all your supported languages, and Gradle will ensure that any translations for unsupported languages (whether inside your project or in third-party libraries) get excluded from the build.

The other place you’ll need to list out your supported languages is in the locales_config.xml file. This file informs the system of the languages your app supports, which ties in to the Android 13+ per-app language setting feature. 

Newer versions of AGP can generate this automatically for you if you opt-in, but we’ve found that option to be a bit risky. The way it populates the autogenerated config is by scanning your project for the presence of any translation in a particular language. If you go the auto-route, you have to be careful to ensure that all locales in your app resources are ready to be published to users. We’ve chosen to stick with the manual setup for now, as it allows us to have translations for “in-progress” languages in the project and makes it easier to generate test builds before launching a new language.

BE Synchronization

The OS will do its job to show the correct translations for all of your app strings, so the next challenge becomes making sure that any strings you receive from your backend and show to users are correctly translated in the same language. If you mess this part up (like we have in the past), you’ll end up with inconsistent localization that is obvious to users and a very poor UX.

Thankfully, there’s a standard mechanism for a mobile client to request localized content from an API: the Accept-Language HTTP header. On Android, it’s easy to set up a basic interceptor to populate this header on each of your network calls.

class LanguageInterceptor : Interceptor {
 override fun intercept(chain: Interceptor.Chain): Response {
   val originalRequest = chain.request()
   
   val requestWithHeader = originalRequest.newBuilder()
     .header("Accept-Language", ???)
     .build()    

   return chain.proceed(requestWithHeader)
 }
} 

Before we get into the correct value to populate for the header, let’s review some basic localization vocabulary.

Quick Localization Vocabulary

• Language Code: represents a specific spoken or written language, completely independent of where it is being spoken

  • Typically two-letter lowercase codes, such as en, es, fr, or zh

• Region Code / Country Code: represents a specific physical country or geographical territory

  • Says nothing about the language; only identifies the place

  • Typically two-letter uppercase codes, such as US, MX, GB, or CN

• Language Tag: structured string that combines a language code and region code to pinpoint an exact linguistic variation

  • en-US represents the English language (en) as spoken in the United States (US)

  • en-GB represents the English language (en) as spoken in Great Britain (GB)

The distinction between language codes and language tags is important, as there are only ~200 standard language codes. Once you combine that with ~250 possible county codes, the number of potential language tags you’ll encounter from users’ devices becomes very large. I queried the number of unique language tags seen as the primary device language for Reddit Android users on a given day, and the result shocked me. We saw nearly 2500 unique language tags!

Obviously, you’re never going to be able to fully localize your app into 2500 different languages. Thus the way your app handles the case where a user’s language tag doesn’t exactly match one of your supported languages becomes critical.

Problem Scenario

Let’s imagine a user with a device set to es-US, or US Spanish. The Reddit app supports two different Spanish variants - es-ES and es-MX. When this happens, which resources will the OS resolve and show to the user? How does it make that decision? And most importantly for us (who need to inform the BE via Accept-Language), how can we know which one it decided to use?

From testing, I can tell you that there are a lot of APIs that don’t work the way we need in this situation.

• Locale.getDefault()

  • This one is ubiquitous and one I used to reach for all the time. But in this scenario, it will return es-US, the plain, unmapped language tag from the OS setting.

• AppCompatDelegate.getApplicationLocales()

  • One of the new Android 13 APIs, this will return an empty list to signify that the app is using the system default language (as opposed to a custom language set by the user specifically for your app).

• applicationContext.resources.configuration.locales, LocaleList.getDefault(), LocaleListCompat.getAdjustedDefault(), LocaleManagerCompat.getSystemLocales(applicationContext), ConfigurationCompat.getLocales(Resources.getSystem().configuration)

  • All do basically the same thing as Locale.getDefault(), returning es-US and not the actual language tag for the resources being shown to the user.

We could use one of these APIs and send es-US to the BE in the Accept-Language header, but that would force the BE to make its own decision between es-ES and es-MX that might not match the one the Android OS made. With no obvious system APIs giving us the answer we need, we had to dig deeper.

An Overview of CLDR

As we explored this problem, we learned that the OS is using something called the Common Locale Data Repository, or CLDR, when making its choice. CLDR is basically a giant database of lookup tables and rules for locale matching that’s maintained by the Unicode Consortium and used in almost all major operating systems and programming languages.

There’s a Java implementation in the ICU4J library, but when we tested adding it to our app we saw that it added 13 MB to the APK (even after the R8 code shrinker ran), making it a non-starter.

We then tried to develop our own simple internal algorithm to match certain country codes with es-ES and others with es-MX. This approach had a few issues:

1. Because the version of CLDR packaged with the OS varies across different Android releases, the platform's behavior changes depending on the scenario. This made it extremely difficult to try to write an algorithm that matched the system behavior in every case.

2. Maintaining this algorithm became challenging as we scaled to more and more supported languages in the app.

3. Scenarios where users have multiple device languages set were hard to reason about. Imagine a user with es-US (non-exact match) followed by en-US (exact match) on their device. How do we handle this case?

Ultimately, we realized we didn’t want to be in the business of trying to figure this out on our own. We’d love to just let the OS do its job and let that dictate the language tag we send to the BE to keep things in sync. After further research, we came across a community solution that’s not anywhere in the official Android Developers documentation.

Canary String

The non-obvious way to access the system-determined language tag is to use a technique called the “canary string”. This involves defining a “hidden” string resource for each of our supported languages that simply defines the language tag it belongs to.

private fun getResolvedLanguageTag(): String = context.getString(R.string.resolved_locale_canary)

• In res/values/strings.xml (Default/English), add: <string name="resolved_locale_canary">en-US</string>

• In res/values-es-rMX/strings.xml, add: <string name="resolved_locale_canary">es-MX</string>

• In res/values-es-rES/strings.xml, add: <string name="resolved_locale_canary">es-ES</string>

This felt pretty hacky, but it works like a charm! To avoid the string lookup hit on every network request, we set up a reusable component that caches the value in a StateFlow.

class MyAppLanguageProvider @Inject constructor(
 @ApplicationContext private val context: Context,
) : AppLanguageProvider {  
  
  private val _appLanguageTagFlow = MutableStateFlow(getResolvedLanguageTag())
  override val appLanguageTagFlow: StateFlow<String> = _appLanguageTagFlow.asStateFlow()

  override fun getAppLanguageTag(): String = appLanguageTagFlow.value
  
  override fun getAppLocale(): Locale = Locale.forLanguageTag(getAppLanguageTag())

  override fun onConfigurationChanged() {
   _appLanguageTagFlow.value = getResolvedLanguageTag()
 }

  private fun getResolvedLanguageTag(): String = context.getString(R.string.resolved_locale_canary)
} 

The cache is set up to be configuration-aware: as long as the Application class forwards any configuration changes it receives to onConfigurationChanged(), we can ensure that our value stays in sync with any system language changes.

This component is super useful! We know that the values returned are guaranteed to be one of our supported language tags and match the resources resolved and shown to users by the system. We can then use it to power the Accept-Language interceptor, any number or date formatters, telemetry, etc.

Final Notes

A couple of last things here: first, remember that Locale.getDefault() is risky! Consider adding a lint rule to warn developers and point them to this other component whenever they need a source of truth about the app language.

Also, consider setting up a unit test to make sure that your canary string setup never gets broken.

@RunWith(AndroidJUnit4::class)
class CanaryStringL10nTest {
 
 @Test
 fun verify_canaryString_matches_languageTag_forAllSupportedLocales() {
   // given
   val baseContext = InstrumentationRegistry.getInstrumentation().targetContext

   SupportedLocale.map { it.locale }.forEach { locale ->
     val config = baseContext.resources.configuration
     config.setLocale(locale)
     val localeSpecificContext = baseContext.createConfigurationContext(config)

     // when
     val canaryString = localeSpecificContext.getString(R.string.resolved_locale_canary)
     val expectedTag = locale.toLanguageTag()

     // then
     expectThat(expectedTag).isEqualTo(canaryString)
   }
 }
}

What’s Next?

Getting this foundation in place was great, but to address a lot of the bugs we were getting we had to completely rewrite our in-app language picker screen. Part 2 of this series will explore that rewrite and all the details that went into it!

By Liam Lu, Riccardo Ciovati, and Savannah Whelan

In Part 1 of this series, we shared our vision for a unified Cronet engine, our initial rollout for media traffic, and how we forked cronet-okhttp to fix request tag tracking and Flipper debugging.

With developer tooling functional, we turned our attention to evaluating performance in production — where we immediately ran into three more major challenges: fixing distorted telemetry, maintaining our library fork safely, and eliminating a +2.0% cold-start TTI regression.

Challenge 2: The Telemetry Trap

With request tags restored and Flipper working, we looked at our dashboards — and were shocked to see massive spikes in latency and response sizes.

The culprit was where we were measuring from. Moving execution to Cronet required moving measurement from OkHttp network interceptors up to application interceptors. But those layers see fundamentally different data:

• Network interceptors observe raw wire bytes before decompression, measuring true compressed size and actual network latency.

• Application interceptors observe decoded, uncompressed bodies after decompression. Measuring at this layer inflated reported response sizes (unzipped bytes) and added decompression/parsing time on device to reported latency.

Furthermore, OkHttp's EventListener does not fire under Cronet because Cronet owns the underlying connections.

To restore accurate metrics, we attached Cronet's RequestFinishedInfo.Listener directly to the engine. This listener provides raw transport metrics straight from Cronet: receivedByteCount for compressed wire size, alongside exact DNS, TCP/QUIC connect, TTFB, and transfer timings. Reusing our tag remapping from Challenge 1, we pass request annotations through to Cronet, allowing the listener to associate engine-level metrics back to individual GraphQL operations.

(Note: HttpEngine exposes limited timing metrics compared to Play Services or Embedded Cronet).

We also added startup tracing around Cronet initialization, placing start/end markers on the cold-start timeline alongside process start, DI initialization, and first frame rendering.

Challenge 3: The Reality of Maintaining a Fork

Forking cronet-okhttp solved our request tag issues, but maintaining a fork in production introduced important build and infrastructure tasks:

1. Classpath package collisions in A/B testing: To run clean A/B experiments comparing stock cronet-okhttp against our fork, both artifacts needed to exist in the app build simultaneously. However, both declared the package com.google.net.cronet.okhttptransport, causing duplicate class build errors. We built a Gradle plugin using ASM to relocate our fork's bytecode package to com.reddit.net.cronet.okhttptransport:

Kotlin

class PackageRemapper : Remapper() {
    private val OLD = "com/google/net/cronet/okhttptransport"
    private val NEW = "com/reddit/net/cronet/okhttptransport"
    override fun map(internalName: String): String =
        if (internalName.startsWith(OLD)) internalName.replaceFirst(OLD, NEW)
        else internalName
}

1. Upstream contract drift: When we updated OkHttp and Okio versions, the bridge crashed on empty responses because it assumed a response body could be null, whereas newer OkHttp versions require a non-null ResponseBody. We patched the bridge to return ResponseBody.EMPTY:

Java

ResponseBody responseBody;
if (bodySource != null) {
    responseBody = createResponseBody(request, status, contentType, contentLengthString, bodySource);
} else {
    responseBody = ResponseBody.EMPTY;
}

Challenge 4: Tackling the Shipping Blocker (+2.0% TTI)

With accurate telemetry in place and our fork running cleanly, the biggest performance hurdle stood out clearly: a +2.0% cold-start TTI (Time to Interactive) regression.

Cronet is a native C++ engine that requires loading libraries, starting listener threads, initializing disk caches, and loading QUIC configs. On cold start, the app fires startup-critical queries (auth, home feed) that blocked until the network client was ready. Because this is a fixed initialization cost, faster devices saw a higher percentage hit.

To win back startup performance, we systematically tested several levers:

• Eager background warmup: Instead of initializing Cronet lazily on the first request, we kick off engine initialization early during app launch on a background thread. This absorbs the initialization cost before or during request setup, recovering the full +2.0% TTI regression back to baseline.

• GraphQL preconnect: Our startup tracing revealed that even after the engine was warm, the subsequent GraphQL request was still paying for DNS resolution and TLS/QUIC handshakes. As soon as Cronet is ready, we proactively open a connection to our GraphQL endpoint before startup queries fire (TTI −0.47%).

• HttpEngine provider: On Android 14+ (API 34), obtaining the system-provided HttpEngine is faster than loading via Google Play Services (TTI −0.22%).

• Embedded Cronet: Bundling the native engine in the APK removes Play Services IPC lookups and dynamic module loading delays (TTI −0.48%).

• Stale DNS: Serving cached DNS answers immediately and refreshing asynchronously kept DNS off the hot path (TTI-neutral for startup, but maintained for media benefits).

Provider comparison summary

Act III: Production Rollout & Impact

With our optimizations and tooling folded in, we ran a 50/50 experiment across ~11.7 million devices per arm. TTI didn't just recover — it flipped to a win, alongside significant improvements in feed reliability and user engagement:

A 10.1% drop in feed failures was a major stability win, proving that transport-level reliability directly moves the needle on user retention and engagement.

Act IV: Where the Road Leads Next

Shipping HTTP/3 for GraphQL wasn't the finish line — it was the milestone that finally put all our major traffic (GraphQL, images, video) on a single, shared Cronet engine. That unified foundation makes our next set of network optimizations possible:

1. Cross-client request prioritization: Currently, traffic types handle priorities in isolation — GraphQL uses an app-level NetworkOrchestrator queue, Glide has its own image priority model, and video requests have no per-request priority. Consequently, background prefetching can compete directly with home feed queries. Sharing a single Cronet engine allows us to map app priorities to Cronet's five native priority tiers, scheduling parallel requests, socket assignments, and stream multiplexing accordingly.

2. 0-RTT connection resumption for read queries: QUIC 0-RTT allows returning clients to send data in the initial packet, removing a round trip from connection setup. Because 0-RTT payloads are vulnerable to replay attacks, CDNs restrict 0-RTT to idempotent HTTP GET requests. Since most of our GraphQL traffic currently uses POST, we are transitioning key read queries to GET before enabling 0-RTT.

Wrapping Up

Swapping the transport was the easy part — most of the work went into startup performance, developer tooling, and telemetry. Working through those requirements at scale directly improved app reliability for millions of Redditors, leaving us with a foundation we're excited to keep building on.

Special thanks to the folks from Android Platform, Data Science, Media Foundation, Feed Experience, Growth, GraphQL, and Security teams for reviewing PRs, specs, and readouts, chasing down startup regressions, and supporting us along the way — it truly took a village.

By Liam Lu, Riccardo Ciovati, and Savannah Whelan

A while back, we set out to move Reddit's GraphQL traffic on Android from our standard OkHttp/HTTP/2 setup over to HTTP/3. On paper, it looked like a clean win: swap the transport underneath, flip a feature flag, and watch latency drop.

It turned out to be much more complex than swapping a library.

In this two-part series, we share the story of our journey to HTTP/3 — why we wanted a unified network engine, how an initial victory with media gave us the confidence to tackle GraphQL, the engineering hurdles we ran into on our most critical code paths, and how we turned a cold-start regression into a production win across hundreds of millions of devices.

The Vision: A Single Engine for All Traffic

HTTP/3 replaces TCP with QUIC, a transport protocol built on top of UDP. For mobile apps, QUIC brings three headline advantages:

• Faster connection setup: QUIC folds transport and TLS handshakes together, saving round trips on flaky cellular links.

• No TCP head-of-line blocking: In HTTP/2 over TCP, a single lost packet stalls every multiplexed stream on that connection. QUIC streams are independent, so one dropped packet only stalls its own stream.

• Connection migration: QUIC connections use IDs rather than IP/port tuples, allowing active requests to survive Wi-Fi ↔ cellular network switches without dropping.

OkHttp doesn't speak HTTP/3 natively on Android, so adopting QUIC required a different stack. While individual libraries could be wired up with their own HTTP/3-capable transports (such as Apollo or Glide), configuring transports piecemeal would leave us with a patchwork of separate HTTP/3 integrations, duplicate connection pools, and fragmented metrics. We wanted the opposite: one shared engine underneath all of our traffic.

That narrowed our choices to two main options:

1. Cronet: Chromium's network stack, available via Google Play Services or bundled directly into the app.

2. HttpEngine: The system network client introduced in Android 14+ (API 34).

Beyond HTTP/3: What else Cronet unlocks

Getting to HTTP/3 was the main goal, but a big reason we chose Cronet was the broader feature set that comes with it. OkHttp is a great HTTP client, but Cronet ships with capabilities that are difficult or impossible to build on OkHttp alone:

• Stale DNS: On OkHttp, DNS resolution sits on the request's critical path. On mobile, slow resolvers cause tail latency and failures. Cronet's resolver can serve a cached (even expired) answer immediately and refresh in the background, taking DNS off the hot path — which matters when the first few requests decide how fast the feed appears.

• Request prioritization: The app has many requests in flight — feed, images, videos, analytics — competing for bandwidth. OkHttp has no shared notion of priority across callers, so prefetched images can crowd out the home feed query. Cronet supports native request prioritization, allowing us to tell the transport that the home feed query matters more than a pre-fetched avatar.

• 0-RTT connection resumption: QUIC's 0-RTT lets returning clients send data in the very first packet, removing a full round trip from connection setup on startup-critical queries.

• Connection migration: On TCP, a connection is pinned to an IP/port pair, so switching networks breaks it and forces in-flight requests to restart. QUIC uses connection IDs instead, allowing Cronet to carry live connections across network switches mid-scroll.

For a small app, these optimizations might be subtle. But at Reddit's scale — hundreds of millions of users on every device and network type, firing billions of requests a day — shaving milliseconds off the critical path and reducing failure rates compounds into a noticeable improvement in app responsiveness and reliability.

Act I: Testing the Waters with Media

We didn't jump straight to GraphQL. We first rolled out Cronet for media — images and video — in partnership with our Media Foundation team. Media was a much simpler starting point for two reasons:

• Simpler network path: Image and video requests go directly to the CDN through a minimal interceptor stack, unlike GraphQL's deep application and network interceptors.

• Direct last-mile benefits: Media is served straight from CDNs with minimal backend latency, so protocol wins (faster handshakes, no head-of-line blocking) show up clearly in metrics.

The media rollout was an immediate success. Image loading showed solid gains:

• p90 image load time: −3.57%

• Image success rate: +0.04% (meaningful at scale)

• Post views: +0.69%

Video playback saw similar improvements:

• Fast video starts (≤500ms): +1.37%

• Slow video starts (>1s): −14.53%

• Exits before playback: −14.99%

Media gave us confidence in HTTP/3, but moving GraphQL was always going to be a bigger undertaking. Because images and video load asynchronously after the app opens, Cronet's engine initialization overhead remained invisible — until we pointed Cronet at GraphQL, which sits squarely on the cold-start critical path.

Act II: Pointing Cronet at GraphQL

We knew moving GraphQL over to HTTP/3 would be a tougher challenge. It touches our most critical code paths at app launch, and we were already aware of wrapper limitations around request tracking. Even with those expectations, tackling the migration in practice brought up more complexity than anticipated.

These are the first major hurdles we had to clear before we could even evaluate performance:

Challenge 1: Going Blind (Request Tags & Flipper)

Before we could even evaluate GraphQL performance under Cronet, our developer tooling and telemetry broke.

First, request tags were dropped. We use the cronet-okhttp bridge library so the app can keep using OkHttp's Call/Request APIs. However, OkHttp and Cronet don't share an object model. When cronet-okhttp converted an OkHttp Request into a Cronet UrlRequest, it dropped OkHttp request tags (request.tag(...)). We rely on tags to attach metadata like GraphQL operation names and logging context. Without them, downstream telemetry lost context and started receiving nulls.

To fix this, we forked cronet-okhttp and added an extension hook: a RequestToUrlRequestMapper interface that runs right before the Cronet UrlRequest.Builder is finalized:

Java
/** Hook for customizing how an OkHttp Request is mapped to a Cronet UrlRequest. */
public interface RequestToUrlRequestMapper {
  void map(Request okHttpRequest, UrlRequest.Builder urlRequestBuilder);

  RequestToUrlRequestMapper NO_OP = (okHttpRequest, urlRequestBuilder) -> {};
}

Java
// Allow client code to customize the UrlRequest before building it.
requestMapper.map(okHttpRequest, builder);
return new CronetRequestAndOkHttpResponse(
    builder.build(), createResponseSupplier(okHttpRequest, callback));

This gave us a clean seam to read tags off the OkHttp Request and attach them as Cronet request annotations.

Second, Flipper network debugging broke. Our team relies on Flipper to inspect network traffic locally, which attaches as an OkHttp network interceptor. Under cronet-okhttp, requests stopped appearing in Flipper because cronet-okhttp installs its bridge as the last application interceptor. That bridge handles execution via Cronet and returns the response directly, terminating the chain before reaching network interceptors:

None
OkHttp call
  │
  ▼
Application interceptors
  ├─ Auth / Headers / Tracing
  ├─ FlipperInterceptor (Custom) ──► Mirrors request + response to Flipper
  └─ Cronet bridge ──► Cronet ──► Response   (terminal: chain stops here)
─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ✗ Unreachable
Network interceptors  ← Flipper was originally registered here

We resolved this by writing a custom FlipperInterceptor registered as an application interceptor directly before the Cronet bridge, manually re-adding pre-bridge headers (Host, Content-Type, Content-Length) and safely peeking response bodies.

What's Next?

Restoring request tags and fixing Flipper got our local developer tooling back on track, but the GraphQL rollout was just getting started. Once we looked at production telemetry, we hit our next wave of challenges: metric numbers were wildly changed, maintaining our fork required custom bytecode tooling, and Cronet's native startup created a +2.0% cold-start TTI regression.

In Reddit's Journey to HTTP/3 on Android (Part 2), we dive into how we fixed our telemetry, eliminated the startup penalty, the results we saw in production, and where we're taking our network stack in the future.

Written by Dominic Valenciana (u/kiricon)

For a while, everyone had been asking: What can a coding agent do?

Then agents became capable enough that a more practical question started to matter: What am I supposed to do while I wait for one to finish?

The answer seemed obvious: start another agent chat.

Parallel development has long been normal across teams, but parallel development for individuals was still far from the default. Most of the tools available at Reddit assumed one engineer, one repository, and one agent session at a time.

Getting antsy, I frankensteined together OpenCode, Cursor CLI agents, Git worktrees, Markdown instructions, and Bash scripts. The result was a basic terminal interface where I could run as many agents as I could keep track of, each operating with a high degree of autonomy.

Parallel agentic development at scale felt incredible. It also made every problem with agentic development at Reddit scale with it. Mo agents, mo problems.

Workspace collisions, slow CI feedback, scattered context between sources, interrupted sessions, inconsistent behavior during complex operations, and mismatched tool configuration all multiplied with the number of active agents. Four agents could produce four streams of work, but they could also produce four streams of friction.

My first attempt at solving this was to write more instructions: `AGENTS.md` files, skills, and increasingly detailed checklists. Those tools remain valuable for ambiguous work, but common deterministic operations should not require an agent to relearn them on every run. Loading the model with more instructions cost money, slowed it down, and still produced nondeterministic results.

It was time to go a layer deeper.

Prompts can suggest a process. A system can enforce one.

Isolation, continuity, validation, and safety needed to become guarantees rather than reminders. Common context and system access needed to be part of the environment, not another skill to configure or another enormous tool surface for the model to navigate.

Snoocode is Reddit’s solution to those problems. It is a batteries-included desktop application for working with multiple coding agents. It is not another model, another harness, or a replacement for the agent tools Reddit engineers already use. It is a unifying layer that runs those tools inside one Reddit-aware development application: multiple models, multiple harnesses, and our internal integrations under one roof that we control.

Here is the simplest way I have found to explain the layers. The model is the pilot, and the harness gives it a body. Together they form a coding agent. Snoocode is the six-story mech that the agent climbs into, equipped with the Reddit-specific tools, controls, and the environment it needs to work.

Snoocode Lets Us Own Our Destiny

As I built Snoocode, proprietary and open-source tools began shipping many of the same primitives we had discovered we needed. That convergence validated the problems while showing how quickly generic agent tooling could become a commodity.

With voluntary internal contributions, Snoocode repeatedly reached feature parity with much larger proprietary and open-source projects. That did not prove that we had built the one perfect agent product. It proved that the generic layer was buildable, the good ideas would converge, and the moat around agent-development tooling didn’t exist.

At Reddit’s scale, the traditional “buy versus build” calculation looks different. We could continue buying the models and harnesses that the wider industry builds well while cheaply owning the layer that determines how they operate inside Reddit.

This theory was tested when much of Reddit engineering had settled into a single agentic coding product. When the commercial relationship changed, access changed with it, and engineers had to reconstruct their workflows elsewhere on short notice.

Other capable agents existed. The disruption came from how much setup and habit had become coupled to one vendor. For Snoocode users, the underlying agent changed; the surrounding workflow mostly did not.

Vendor agnosticism  for coding agents is a core philosophy for this project. A software company should be able to replace a model or harness without replacing its development workflow. If we own the machine, changing the pilot should not require rebuilding the cockpit overnight.

For Reddit, there is another reason to own that layer: no third party can build it for us.

Our design, project management, development, testing, and delivery workflows include internal systems with no public equivalent. Even where we use common external products, only we can encode how they connect to our internal tools and what their signals mean here. A vendor cannot design tools or plugins for systems it has never seen.

Development at Reddit is its own terrain. No matter how capable the pilot or its body becomes, it still needs a machine designed for the environment in which it will work.

The principle is simple: buy the commodity, build the seam, and keep the freedom to change what sits on either side.

A Side Project Became Shared Infrastructure

When I shared the prototype, it went “viral” internally, at least by internal developer-tool standards. People downloaded it, modified it, and began using it for their own work. A personal setup became a shared experiment developed in public inside Reddit.

Within the first few months, it had been used across 94 repositories, and 26 engineers had contributed to Snoocode itself. Engineers used it against real Reddit workflows, found weak points, and helped improve the shared platform.

The community supplied the pressure that turned a productive hack into a product. It also gave us a development loop no external vendor can reproduce. The engineers using Snoocode can add the feature they need, fix the bug blocking them, and adapt the product as the industry changes. Maintainers provide coherence and remain responsible for what merges.

That direct loop lets a narrowly focused internal product move quickly toward Reddit’s needs. We do not have to persuade a vendor that our problem belongs on its roadmap or wait for a generalized solution designed for thousands of other customers.

Snoocode is now officially staffed, and we are expanding the project to increase developer velocity across Reddit. The goal is not to force engineers away from useful agent tools; those tools are components of Snoocode. The goal is to make the Reddit-native experience so comprehensive and fluid that managing a fragmented toolchain feels unnecessary by comparison.

How Snoocode Works

Snoocode acts as a control plane around any number of supported harnesses—in effect, a “harness for harnesses.”. It bootstraps and configures them, normalizes the concepts that benefit from being shared, and preserves their native capabilities where abstraction would get in the way.

Open-source projects such as [T3 Code] (https://github.com/pingdotgg/t3code) which was built in parallel with Snoocode has been a major inspiration and validation that our ideas are on the right track. They demonstrate how much value a focused interface can add around several coding agents. Snoocode takes that pattern further for Reddit by owning the automatic bootstrapping, internal integrations, and workflow on both sides of the harness.

From Setup to Shipping

Without Snoocode, starting a task can mean creating a worktree, signing into a harness, choosing a model, configuring internal development tools, and loading repository-specific skills. Each step adds context and another failure point, while the resulting session remains tied to one harness.

Snoocode turns that assembly into shared infrastructure. Engineers choose a task, repository, harness, and model; Snoocode supplies the isolated workspace, internal tools, and repository knowledge through one consistent experience.

Because we own the application—not just the agent’s prompt—we can also build experiences third-party harnesses cannot: mentioning coworkers from the company directory, visually attaching application elements, or surfacing local development resources directly beside the chat.

The result is more than a better-equipped agent. It is a cohesive development environment that gives every harness the same Reddit-specific capabilities while adding features no harness could provide on its own.

The common layer is where much of the leverage comes from. Internal integrations, identity, permissions, and deterministic workflows can be built once and reused across every supported backend. Improvements benefit every harness at once instead of being rebuilt in personal scripts or repeatedly interpreted from prompts. Several durable sessions can progress without sharing a checkout or losing their history, while the engineer moves between them and steps in when judgment is required.

This lowers the barrier to useful agentic development while reducing duplicated setup, model context, cost, and variance. It also lets Reddit adopt better models and harnesses without reconstructing the development environment around each one.

Because the contract belongs to us, the same core can eventually power whichever surface fits the work: IDE chats, terminal interfaces, desktop, web, mobile, cloud agents, or local containerized sandboxes. We do not have to wait for every vendor to understand our systems before we can bring its agent into them.

In the mech analogy, this is the payoff: Reddit owns the cockpit, instruments, attachments, and hangar. We can improve the machine for everyone at once, deploy it wherever engineers need it, and swap the pilot when a better one arrives.

Snoocode’s job is not to pick the winning agent. It is to ensure that whichever agent we use arrives ready to work at Reddit.

Key Takeaways

• Models are largely stateless and increasingly interchangeable. The durable value lives in the workflow, context, and environment around them.

• Generic agent tooling is converging quickly. Its moat is limited, and agents themselves make bespoke tooling cheaper to build.

• Software companies should not tie one of their most important development workflows to a single vendor.

• No third party understands a company’s systems and engineering practices well enough to build its ideal development environment for it.

• Context hydration, prompts, and skills remain useful, but they can only take an agent so far. Deterministic workflows belong in the system.

• As models improve, the limiting factor will increasingly be the environment in which they work.

How we catch distributed system failures before users do

Written by Hiram Silvey u/space4268 and Shashank Veerapaneni u/colorsoda

Reddit on Fire: How This All Came to Be

Let me paint you a picture that many engineers are familiar with: You’re sitting at home sipping your coffee when suddenly the echoing thock of thousands of fingers hitting CTRL+R simultaneously pierce your eardrums in the form of the dreaded emergency page. You run over to your laptop, declare an incident, open dashboards, and start investigating. You find the worst has come to pass: total failure of the system. That’s right: reddit.com is unavailable. You pull in more engineers as necessary and all work together to restore service as quickly as you possibly can. As this story repeats over time, one crucial detail becomes quickly apparent: Reddit is a complex, fast-moving, non-linear distributed system, and nobody can fit the entire consistently-up-to-date detailed view of it within their brain. When one component breaks, its knock-on effects cannot consistently and reliably be predicted.

So… what do?

A natural conclusion to this is that we need better isolation between systems, and we need better testing to verify that. It doesn't matter how fireproofed my little room is if there's an open window or door for the fire to escape through, right? So, we add tests. We add as many tests as we can. We brainstorm all the ways something could fail and we add tests for them. This works well, and we do find and fix many vulnerabilities ahead of time. Unfortunately, after everything we do, that emergency page still arrives. Brand new novel issues continue to arise in unexpected ways, ad nauseam. We need something more.

Let's take a step back and approach this from another angle. The fundamental issue with standard testing is that it requires knowledge of the vulnerabilities beforehand. How do we flip this on its head and write tests to discover the vulnerabilities we don't yet know about? Enter: chaos testing. At a high level, chaos testing is the practice of intentionally introducing controlled failures into a system to verify that the system behaves as expected. A failed chaos test is new information: a dependency we thought was optional is actually critical, a fallback path does not work, or an error that should have been contained is escaping into the user experience. Once we find that pattern, we can fix it and turn it into a regression test. Because Reddit is a complex, fast-moving, non-linear distributed system, though, we can’t test each individual component we know about and reasonably expect to catch everything. Instead, we test the system itself and see how it truly behaves.

Introducing: FIB! Fault Injection in Baseplate

Reddit regularly holds company-wide hack-a-thon events called “Snoosweeks.” These events allow engineers to work on projects they believe are important that may not otherwise have explicit long-term plans and funding. During one of these weeks, I proposed a solution to the problem highlighted above: automated chaos testing. The proposal was simple: build request-level fault injection to allow us to systematically verify the behavior of our production systems continuously and proactively.

FIB, or Fault Injection in Baseplate, is our first step towards accomplishing our goal of developing a robust chaos testing system. Fault injection is a specific facet of chaos testing, providing the capability to inject predefined faults into, in our case, service requests. Baseplate is Reddit’s common service framework, and it comprises many internal libraries that services throughout our stack utilize.

FIB comprises 2 components:

1. Baseplate library middleware responsible for injecting faults.

2. A fault configuration request header, propagated throughout our entire backend infrastructure via Baseplate.

As requests are made between backend services, the Baseplate library middleware is responsible for:

1. Reading the fault configuration request header (if present).

2. Comparing the outgoing request to the specified configuration.

3. If matching, injecting a fault at the transport protocol layer (i.e. gRPC client interceptor).

If you are familiar with chaos engineering, you already know there are many different ways faults can be injected into complex systems. The well-known chaos mesh tool, for example, offers the ability to simulate Kubernetes faults at various levels, such as pod faults, network faults, file I/O faults, etc. We chose to inject faults at the request level because the vast majority of failures surface themselves to client services in the form of request failures. If a disk fails and the pod running the upstream service is unable to complete the request because it cannot read some data, that ultimately will make it back to the client in the form of a request failure. If the network connection between two services is degraded, that, too, will surface as a request failure to the client service. If the upstream service process is killed due to insufficient machine resources, that will also surface as a request failure to the client service. You get the picture.

In order to simulate the full range of possible symptoms we need to be prepared for (see examples in the next section), FIB supports matching on address, host, and/or method (including negative matching), as well as percentage-based injection. This means we can add a particular fault header onto many requests but only actually inject faults on some percentage of total matching requests, in order to more accurately simulate degradation of pieces of a distributed system rather than a full outage.

See the open-source implementation here.

Improving Reddit’s Resilience

We started with important user-facing surfaces, including Feeds and post detail pages, because reliability problems there are immediately visible. The user expectation is simple: open Reddit, see content, and keep browsing even when some backend dependency is unhealthy. Behind that experience, though, a single request can touch several backend systems, and we wanted to find out what our users experience when these systems are misbehaving.

Finding Degradation Bugs

The first pass was practical: engineers injected scoped failures while walking through important surfaces and noted where the experience degraded badly or unexpectedly. Rough edges were smoothed and the user experience improved.

The most valuable findings did not mitigate total outages. They were often quieter experience bugs where Reddit could still show something useful if the failure was handled as degradable.

Examples of fixes and fallback behavior we added include:

• Media related failures now degrade the media parts of the experience instead of preventing core surfaces from loading.

• Failures in non-critical systems related to styling, flairs, stats etc. fall back gracefully instead of breaking a feed or throwing error toasts.

• Post detail views preserve already loaded comments instead of replacing them with an empty or error state when additional requests fail.

• Viewer specific failures now have a smaller blast radius: e.g. if a dependency only powers moderator or author-only affordances, its failure should affect those affordances.

Those findings turned into fixes, better fallback behaviors, regression coverages and improved user experience.

Making It Repeatable

We made the workflow easier for other developers by adding developer-only controls in internal development builds, so teams could reproduce degraded dependency scenarios easily. We also updated our mobile and web UI automation frameworks to support chaos testing directly.

Those automated tests stress the app or site with a scoped fault configuration applied only to the request paths under test. Then they do what a user would do: load core surfaces, browse content, move from a listing into a detail view, and verify that those surfaces still load under controlled backend failures. Today those chaos tests run daily across mobile and web flows, exercising a growing set of request-level failure scenarios.

A Reliability Signal

We saw useful directional signals after the resiliency work matured. For the Feeds service, high-urgency pages dropped by roughly half when comparing January-June 2026 with January-June 2025. We saw a similar shift in incident participation: Feeds team was less often pulled into upstream incidents as an affected downstream customer, because covered paths were more likely to keep serving a degraded but usable experience. Chaos testing was one of the major contributors to that improvement: it helped us reproduce failure modes earlier, prioritize graceful-degradation fixes, and reduce incident impact and blast radius when dependencies misbehaved.

A True Platform: Moving Beyond Manual Fault Injection

While this early user-facing reliability work proved that request-level fault injection is useful as a sharp tool, the long-term vision has always been a full platform. Manually choosing dependencies and maintaining per-surface fault lists works for an initial reliability push, but it cannot scale to every team and request path at Reddit. We need an autonomous system.

The next step is to make the system more data-driven. Instead of asking each team to spend effort to run their own fault injection tests, we want to derive candidate experiments from service dependency graphs, production traffic shape, and criticality metadata. Not all services at Reddit are created equal. At Reddit, we classify services into tiers from 0-3, with 0 being the most critical and 3 being the least critical. When a tier 0 service fails, we expect the entire platform to become unusable. For example, if our public ingress is down, we expect traffic to be unable to reach any internal service backends. When a tier 1 service fails, we expect a core experience of the platform to become unavailable, though some other functionality still may be usable. For example, if the comments service is unavailable, we expect users to be unable to read/write comments, though we still expect users to be able to load their homefeed. Failures of this caliber manifest as high severity incidents with expectations to resolve even outside of business hours. So, we already know what the impact will be if a tier 0 service fails; there is little to discover there. The interesting tests are the ones that expose a mismatch between dependency criticality tier and user impact. A tier 2 or 3 service failing should never make the site unusable for users. If a tier 0 or 1 service has an unexpected hard dependency on a tier 2 or 3 service, though, that’s exactly what can happen.

We also want to reduce adoption path friction. Convincing teams across Reddit to invest in setting up, running, and maintaining a new set of tests requires real effort for both the maintainers of the chaos system as well as the teams using it. Reddit already has a mature experimentation system, though, and integrating with that directly gives us the opportunity to fit these tests into a familiar format. It also gives access to key metrics to better understand the impacts of degraded experiences that we may not discover otherwise. While teams can leverage this to more easily create their own chaos tests, the key innovation is that these experiments will also be created dynamically based on the data highlighted above. Affected teams will be notified according to results, but otherwise their input on crafting the tests shouldn’t actually be needed.

Request-level fault injection via FIB gives us the primitives. The platform work is about dynamically choosing what to inject, where to inject it, how often to run it, and how to route the resulting findings to the teams best positioned to fix them.

Conclusion

Distributed systems will always surprise us. The goal is to make those surprises smaller, earlier, and less user-impacting. The framework lets us break carefully chosen request paths on purpose, observe how the rest of Reddit behaves, and turn the results into concrete reliability work.

For us, the biggest shift was cultural as much as technical. Instead of treating graceful degradation as something we hope exists, we can test it continuously. Instead of learning about dependency criticality from incidents alone, we can discover mismatches in controlled test runs. And instead of letting those lessons live only in postmortems, we can encode them into daily checks that keep Reddit usable when parts of the system inevitably have a bad day.

Written by Aleksandr Krivoshchekov and Walther Lee.

Prometheus has a simple idea at its core: metric_name + labels = time_series. Any two samples with the same metric name and labels map to the same series, but adding even a single extra label initiates a completely new series.

In Kubernetes, that extra label is usually pod. It’s noisy and always changing. It spams our system with more and more series, slowing down queries and draining memory. It’s mostly pointless for our users. But it’s also crucial for Prometheus to evaluate queries correctly.

All these problems with pod are partially solved in Prometheus, but they are still hard to avoid entirely. We could add periodic preaggregations known as recording rules to improve latency. We could sample and keep only a fraction of our metrics to get back some memory. We could even start all over again and try a different database. Each of these options would help for a while, but each of them also create new bottlenecks. What we wanted was an option that kept our existing model with minimal impact for our users.

This post is about that option, which dropped our total number of series used in production queries by 88%, reduced latency by 80%, and achieved 99.85% fidelity over 1M metrics per second in a single instance.

The problem

As the observability team at Reddit, we ran into the challenge of controlling our series count a while ago. If we instrument an application with counter series and send it to Kubernetes, those counters will show up in a pod. And since deployments can have multiple replicas, these counters have to be stored separately to be able to identify them by their own labels. Otherwise, a pod that measures 5 counts in a metric will conflict with another one that counts only 3.

# The label "pod" is necessary to keep track of counts from different targets
http_client_requests_total{pod="pod-a"} 5
http_client_requests_total{pod="pod-b"} 3

In its simplest form, this way of storing counters should scale with our deployments. If we have three replicas, Prometheus should store three counters that we can then add up at query time to get the total number. The reality, however, is not that simple.

Pods have very active lifecycles; they can be restarted after a rollout, shuffled by a node rotation, or killed by Kubernetes because they failed a check. A broad range of infrastructure events can make our pods come and go, and since each will start exposing its own metrics, Prometheus has to keep up with the growth at least until old entries are cleared out. Suddenly, a rollout can double our number of counters just by deploying a whole fleet of new pods and then double them again if we rollback.

Services at Reddit expose billions of metrics at any given time, so these infra events can cause a big impact. A large deployment with thousands of pods often needs millions of counters to monitor values like the total number of errors seen by users. Letting that payload double with every rollout compromises our observability when teams need it more.

This scaling pattern with metrics in Prometheus is unfortunate because applications in Kubernetes are actually designed to be unchanged by these events. A well designed deployment should be able to scale up and down, and to handle unhealthy pods seamlessly, without any impact in user experience. It comes as a surprise to product teams that events that don’t affect their applications do affect their ability to monitor them. 

On top of that, product teams don’t even care for these separation of metrics. They want to see the overall state of their deployments: the total number of requests and errors, the total latency. Single pods are treated as ephemeral and fungible, so the fact that they can influence their observability is an unexpected drawback.

Separation of metrics is an implementation detail that teams don’t care about, so we wanted to find a new solution that could match that expectation. We decided to start with a simpler goal in mind: 

• Find an efficient way of aggregating all counters in real-time to drop only the labels added by our infrastructure layer, like pod.

Aggregate at query time

The trivial solution would be just letting it grow and aggregate at query time. For example, with the expression:

• sum without (pod)(rate(http_client_requests_total[1m]). 

This is basically what we had before doing anything, and the main problem is still the unexpected changes in the number of counters that the query will load. While unexpected, this growth wouldn’t be so bad if it was intuitive. We think it’s reasonable to accept that the number of metrics will go up if we scale up a deployment. But teams don’t expect their query latency to go up just because they rolled out a change.

In practice, aggregations at query time are not consistent in latency or throughput, and the standard way to work around that in Prometheus is using recording rules.

Just drop the labels in recording rules

The single most common request we get in observability from other teams is if we can just drop the pod label. Unfortunately, that takes us back to the original problem: if two pods record different counts for the same metric, there’s no way to know which one we should keep. Probably both.

What teams really mean is to add up the counts without this label that split them in the first place. For that purpose, Prometheus uses recording rules to aggregate common queries periodically in the background. However, this has multiple downsides:

• At a big scale, evaluating a recording rule takes almost as much resources as evaluating the query, so using recording rules helps with latency, but it shifts the heavy payload from query time into a continuous load in the background.

• It generates toil, because teams have to define their rules in advance, and it’s hard to predict what information they will need, especially for incidents.

• And finally, they can still timeout and fail if the number of series spikes after multiple infra events.

In our experience, recording rules were a good option for shifting latency, but they were not a solution for our high volume of aggregations.

Aggregating at scrape time

The main bottleneck that nudged us to move away from recording rules was redundancy. If we use rules to drop labels, those evaluations will run periodically and load data that was scraped just a few seconds ago. We figured that we could do better than that by just aggregating the samples right when we receive them at scrape time.

Another good reason was the fact that scrapes are performed by Prometheus concurrently, as opposed to rule evaluations that compute operations in sequence across the different counters. Concurrent aggregations meant a more efficient use of our resources, so we decided to start by giving that a try.

However, the first problem we found is that counters in Prometheus are cumulative; their values can only go up or reset to 0, so when a query sees a sequence like 10, 14, 5, 8, it assumes that the pod restarted after 14 and then counted up to 5: 

http_client_requests_total    10 14 (reset)  5  8 

# Prometheus assumes the actual deltas were
http_client_requests_total   +10 +4 (reset) +5 +3 = 22 

If we just add up cumulative counters, a single restart will show up as a huge increment right after the reset:

http_client_requests_total{pod="pod-a"}    5 10 15 20    #  +5  +5  +5  +5 = 20
http_client_requests_total{pod="pod-b"}    7 12 17 22    #  +7  +5  +5  +5 = 22
http_client_requests_total{pod="pod-c"}    9 14  2  7    #  +9  +5  +2  +5 = 21

# in a query with increase(...) we would expect the deltas to be
increase(http_client_requests_total[1m])                   +21 +15 +12 +15 = 63

# however if we simply add them up, the series turns into
http_client_requests_total{}              21 36 34 49    
increase(http_client_requests_total[1m])                   +21 +15 +34 +15 = 85

A change in pod-c that was only 2 counts after the restart becomes 34 after the sum. The change becomes even more catastrophic when adding up counters during a rollout, as shown in the panels below:

In the end, we realized that adding up cumulative counters is an inherent but well-known limitation in Prometheus. But it’s fixable. By turning cumulative values into a sequence of increments or deltas we can solve the problem upfront, exactly how recording rules handle it.

Instead of adding up the raw values, recording rules use functions like rate and increase to account for resets and return only the delta over a step interval, and as long as all deltas are calculated over the same window, they can be safely added up.

For example, in the panels below, a pod restart shows up as a gap and a reset in the cumulative counter on the left side, but on the right side, the deltas just keep the gap. Working with deltas, aggregations don’t need special checks to add them up, even between restarts.

Switching to deltas completely solved the problem of aggregating counters at scrape time. We just had to keep track of resets and the last value seen for each counter to compute the deltas after every scrape!

However, this approach introduces a major catch: converting to deltas requires a fixed time interval to calculate them, but our teams query vastly different time windows throughout the day. While a 1-minute delta works perfectly for short few-hours-long graphs, it breaks down on a longer view. Ultimately, we would either have to lock everyone into a single interval or find a way to convert these deltas back into cumulative counters.

This delta-to-cumulative challenge isn’t new. Open-source projects like OpenTelemetry and VictoriaMetrics already offer processors to convert metrics between these formats. However, cumulative counters are stateful. This means that all aggregations must happen inside a single collector. If you split the load across two replicas that started at different times, their cumulative totals won't match, making it impossible for one to seamlessly fill in gaps if the other fails.

So our goal was no longer how we can aggregate our counters at scrape time. The new goal was: 

• How can we turn deltas back into counters in a way so that multiple replicas can agree on the value without requiring consensus? And how can we use the counters in one to deduplicate and fill gaps in another?

The ZigZag

We asked other engineers and read some posts and white papers that gave us an anchor of the error margin that other approaches were achieving. Some were using streaming in external components, probability algorithms, proxies, remote-write and processors. Overall, most of them expected an error margin around 9%.

But before trying any of that, which would require more time and changes and deployments, we decided to try something simpler. From studying how other projects convert cumulative counters to deltas it’s clear that the goal is to shorten the span of time a counter keeps a total count, so what’s the shortest span we can have while still being considered cumulative? The scrape interval! 

If we use the scrape interval as the span window for our deltas, and then we force the counter to reset immediately after, then the counter kind of resembles a delta. It’s still cumulative, but now it looks like a zigzag.

Keeping in mind that we were just trying to get close to the 9% error margin, and since that sounds pretty small, we decided to give it a try. We implemented the aggregation step at scrape time using deltas in Prometheus, and then added them up into a zigzag counter and ran some queries.

last = 0
for every 15s:
delta = delta_since_last_flush()
if delta < last:
last = delta
else:
last = delta + last
flush(last)

And it worked! The panels below show the delta below and its generated zigzag cumulative counter above.

Deduplication for zigzag counters

Zigzag counters let us store the aggregated data back in Prometheus again, but this isn’t too different from what we mentioned before with OpenTelemetry processors. The main difference is that now our data looks kind of the same between replicas.

The zigzags on each replica oscillate at different frequencies and are definitely not identical, but now the scale and values are somehow comparable. We can easily use one to fix gaps in the other, and the rates they return are very accurate! They have an average error margin of 0.15% with just 5 samples, exceeding our expectations by far.

We let this run for a few months in a test instance and it worked great. Aggregating a little over 4M series, driving query load and throughput down to just 3% of our old value for the metrics names with more series, and it didn’t even seem to use more resources. The operations are really light: add up or set a new number. And the new allocations are also minimal. It’s equivalent to adding just one more pod value to the label.

Zigzagging worked well for requests querying recent data, mostly because that data usually comes from a single replica in our system, but after a few months, we noticed an issue with queries spanning multiple hours.

Since Prometheus stores data in 2-hour blocks, long queries connecting multiple blocks had issues jumping between zigzags from different replicas. And we had the same problem deduplicating samples. For example:

25.31 @ 1775649538
50.48 @ 1775649568
25.49 @ 1775649598
25.42 @ 1775649607 // Note how the value goes down again
52.39 @ 1775649628
23.89 @ 1775649658 

The spikes happen because, even though both samples [email protected] and 25.42 @...607 recorded almost the same value at the same time, the second one is slightly lower, triggering the queriers to treat it as a restart. When a rate or increase is calculated, it reads as the following deltas:

50.48 @ 1775649568 
25.49 @ 1775649598 // a change of ~25 in 30s
25.42 @ 1775649607 // a change of ~25 in 9s!
52.39 @ 1775649628 // again, ~25 in only 21s

The extra sample ends up doubling the rate of 25 every 30s, and the inconsistency will last until the rate interval passes that point.

We fixed that by adding an extra step in our function to build zigzagging counters. Technically, counters following the algorithm above don’t have to be zig-zags. A counter whose rate is slowing down would record lower and lower deltas, resulting in a counter that goes continuously down, signaling a reset in every single scrape.

We decided to block that option and force our counters to go down only once in a row, essentially forcing a real zigzag pattern.

With that constraint, now we can walk through our deduplicated samples and explicitly exclude the ones that break the pattern. These values couldn’t have come from the same replica, so we know that a switch was made. The switch loses some accuracy, but this isn’t different from a normal deduplication algorithm that breaks samples’ resolution.

Let's say a change in replicas while deduplicating causes a second drop, resulting in the following two streams:

Replica 1: 22 50 21 
Replica 2:          20 48 19 52 20

Replica 1: 22 50 21 50 
Replica 2:             48 19 52 20

First, we tried discarding the last drop: 20 in the first stream and 19 in the second. This was a small change because samples are streamed for deduplication, so we just had to watch for two consecutive drops and remove the second. However, that doesn’t work for the second stream. After the drop we have:

Samples: 22 50 21 48 19 52 20
Delta:      28 21 27 19 34 20

Samples: 22 50 21 50 48 52 20
Delta:      28 21 29 48  4 20  # 48 and 4 break the steady rate!

Instead, we can fix both inconsistencies by keeping a look forward and discarding the first drop instead:

Samples: 22 50 20 48 19 52 20
Delta:      28 20 28 19 34 20
Samples: 22 50 21 50 19 52 20
Delta:      28 21 29 19 34 20 # Seamless transition

Production results

This algorithm was implemented and tested in a Prometheus fork and deployed to some of our instances in production.

For our largest metric, it shrank the number of series to only 2% its previous value, and the average throughput and latency across all queries went down to 12% and 20%, respectively.

In terms of resources, we didn’t find any significant memory overhead other than the one added by the new zigzag series ingested. CPU usage increased by 6%, but can be easily offset by replacing previous recording rules to aggregate the same data.

Mean error margin between raw and aggregated data was 0.15% with a standard deviation of 0.13%.

This was measured in an instance scraping 1M+ samples per second, using 70Gi and 16 cores before the implementation. It had 17M series before aggregation, and resulted in 2M new zigzag series.

Limits and Tradeoffs

This algorithm doesn’t solve every metric problem. It is limited to counters and counter-like types like Prometheus classic histograms, but gauges, summaries, and native histograms need different aggregations.

It also doesn’t change the existing Prometheus expectation around counter resets. If a pod restarts between scrapes and the new process reaches a value higher than the previous sample, Prometheus cannot know that a reset happened:

# Scrape #1
http_client_requests_total{pod="pod-a"} 90 

# "pod-a" restarts
http_client_requests_total{pod="pod-a"} 0 

# Scrape #2
http_client_requests_total{pod="pod-a"} 140 

# After the second scrape, Prometheus will assume the delta is +50, not +140 

That’s the same behaviour we see today with regular counters.

There is also a replica-local visibility limitation. Our aggregations run independently within each Prometheus replica using only the samples that the replica scraped. If only one of two replicas loses connection to a target, it will miss samples that were successfully aggregated in the other. Resulting in different values that we can’t compare or use to fill gaps between them.

This wasn’t a big tradeoff for us because it is exactly how recording rules work already. So we can think of it as a lightweight implicit recording rule:

# record: cluster:http_client_requests_total:sum
# expr: sum without (pod, instance) (http_client_requests_total)
cluster:http_client_requests_total:sum
==
# An internal series produced by our algorithm.
http_client_requests_total:materialized 

Finally, in our implementation we decided to keep the raw metrics. The algorithm only adds the new series as an optimization path for common aggregation queries, while raw ones remain available for debugging or per-pod queries. In the future, we could drop them after aggregation to reduce memory, which is an option we never had with rules.

What’s next?

Aggregating metrics during scraping is an idea that we had been chasing for a long time. What surprised us most was that we didn't need a massive rewrite or a fancy new storage format to achieve that. We already had all the necessary pieces of the puzzle, we just lacked a small but clever way to put them together.

The production results exceed what we thought was possible without switching databases, or at least without adding proxy-layers. Our algorithm cut query latency by 80% and dropped the total number of series loaded by 88%. And the best part is that it requires no changes for our users: all the same metrics, the same PromQL expression, and the same mental model.

That makes the next step pretty clear. We want to take this feature out of our fork and bring it to open source Prometheus, where our design can be discussed, tested, and improved by the community. By doing so, we hope that these changes can help others and spare them the scaling troubles we faced.

Written by Sarah Thomas

Hi, r/RedditEng!  I’m one of the leads of Reddit’s Women Engineers Employee Resource Group (ERG), WomEng. I’m here to tell you about how Reddit celebrated International Women in Engineering Day this year, and the idea that shaped the programming.

Like many ERGs, WomEng exists to build community among employees with shared identity, and provide a support network for its members. For ERG leaders, the challenge is translating these abstracts into real value and resources for members. 

For a group of women engineers, this can manifest in different ways. Our members share lived experiences, but also technical backgrounds and career interests. So while value can come from bringing members together into physical spaces, it can also be career-oriented and come from technical skill-building and visibility.

This perspective was reinforced in a conversation between WomEng leadership and Sabrina Farmer (former CTO of GitLab and former VP at Google) at Grace Hopper Celebration 2024. One idea from that conversation kept coming up in our planning and discussions:

Shared experiences are important, but having a community where people learn from one another and celebrate each other's accomplishments can be an even more tangible way to build a sense of belonging.

International Women in Engineering Day (June 23rd) provided an opportunity to put this idea into practice. Throughout the day, we hosted a mix of technical and lighthearted programming. Executives participated in segments like “Guess the PR Comment: AI or Human?” and “Can you match the vibe?” where they were challenged to vibe code alternative themes and features for the Reddit feed. We also organized Speed "Dating" networking events in our offices and in-person meetups across multiple cities to help members make new connections.

The centerpiece of the day was a series of mini technical talks delivered by fellow Women Engineer Snoos. Rather than bringing in an external keynote speaker, we turned the spotlight inward.

Members presented on projects and topics they were excited to teach others about. The talks ranged from technical deep dives like “Improving Image Performance on Android” (u/jessekrema) to career advice like “How Giving Back Can Propel Your Career” (u/tbonejones_). Every presentation reflected something the speaker had built or become an expert in.

Choosing our members as the speakers gave attendees the chance to learn something directly applicable to their day-to-day work while also celebrating the breadth of knowledge and experience that already exists within WomEng. 

International Women in Engineering Day gave us a chance to celebrate our WomEng Snoos, but it also reinforced something we hope continues to define WomEng @ Reddit: a community where members connect not only through shared experiences, but also by sharing our expertise, celebrating one another's work, and learning from each other.

~~~~~~~~~~~~~~~~~~~~~~~~

Thanks for reading! Below are bonus clips from the “Guess the PR Comment: AI or Human?” and the “Can You Match the Vibe” segments featuring our CTO, Amit and Senior Director of Engineering, Growth, Shweta!

Written by Alexey Bykov, Staff Software Engineer at Reddit & Google Developer Expert for Android technology

Reddit serves approximately hundred of millions playbacks a day on Android.
In our last two posts: Improving video playback with ExoPlayer and Taking ExoPlayer Further: Reddit's performance techniques we covered ExoPlayer performance and what you can do to improve startup latency, rebuffering, video quality & stability. (Check them out if you haven't yet, they also show how your production metrics may improve after every optimisation)

But bundling those performance practices into a reusable component with a safe and clear API turned out to be a different challenge. Over time, every new video integration became harder, and we kept finding edge cases that were difficult to support without breaking existing behaviour.

In this article, we'll share how we rewrote our abstraction on top of ExoPlayer from scratch: a better API for teams building video features, and even better performance and stability.

Goal: One obvious way to play video

Our main requirement for the API was simple: engineers integrating video shouldn't need to be video or performance experts. They shouldn't have to think about prefetching, player creation, prewarming, or lifecycle. Video should be fast by default and take a few lines of declarative code to integrate.

ExoPlayer is considered one of the best open-source video players among all platforms. It gives you a dozen valid ways to get to the same playable video, and at our scale, that freedom stops being a feature and starts being a challenge.

Tim Peters put it best in the Zen of Python: "There should be one, and preferably only one, obvious way to do it."

Architecture

Playback engine: ExoKit

To bundle all of the optimisations together, we built an agnostic playback engine called ExoKit.

It abstracts away ExoPlayer, makes the API stricter, owns all player communication, and keeps every playback in a central place with a single app-wide video state and effects handling.

It supports all of the performance features we covered earlier, such as:

• Player pooling and prewarming the pool on the app start, since creating a player can still take up to ~200ms according to our production traces

• Decoder reuse for identical videos, which saves up to ~80ms more

• Warming up videos during first composition (not to be confused with prefetching), so the first keyframes are decoded and rendered before the user scrolls to the video.

You can read more about these optimisations and their impact on our metrics here.

On top of that, it offers two APIs: a declarative one for rendering and an imperative one for managing playback state.

@Composable
fun PostVideo(
    mediaId: String,
    url: String,
    modifier: Modifier = Modifier,
) {
    val key = remember { PlaybackKey(mediaId, "feed") }

    // Imperative API: express playback intent.
    val actions = rememberPlaybackActions()
    PlayButton {
        actions.action(key, PlaybackAction.Play)
    }

    // Declarative API: render from playback truth.
    val state = rememberVideoPlaybackState(mediaId)
    if (state.isBuffering()) {
        LoadingIndicator()
    }

    // Declarative API: describe the video surface.
    Video(
        modifier = modifier,
        props = VideoProps(
            url = url,
            playbackKey = key,
        ),
        surfaceLifecycle = rememberLifecycle(),
    )
}

But even though it looks minimalistic enough, it still leaves a lot of responsibility on engineers. They have to decide where to play the video (which gets tricky with autoplay on), manage the player lifecycle, fire telemetry and coordinate between the declarative and imperative APIs.

To make it easier, we landed on a much more opinionated API. 

Self-hosted UI Components

Our first step was to abstract the imperative API from the feature layer: one less decision means one less way to get it wrong. But that logic had to live somewhere.

Rather than building reusable abstractions and inject them into every screen's ViewModel, we went the opposite way: every UI component became fully independent and self-hosted.

To ensure every composable can access product-related context, such as analytics, every block implements the following contract:

interface Component<Props : Any> {
    u/Composable
    fun Content(props: Props, modifier: Modifier)

    fun key(props: Props): Any = props
}

Implementations live in an impl module and can also access product-related context.
Here is a simplified example of what a lower-level composable might look like inside a component implementation:

u/Composable
// Very simplified version of the code for mute button
fun MuteButton(
    playbackKey: PlaybackKey,
    modifier: Modifier = Modifier,
) {
    // Imperative API
    val playbackActions = rememberPlaybackActions() // manage playback
    val globalActions = rememberGlobalActions() // global state, e.g. settings

    // Declarative API
    val audioSettings by rememberAudioSettings()
    val playbackState by rememberPlaybackState(playbackKey)

    // no sound -> hide, loading -> settings fallback, has sound -> real state
    val muted = when (playbackState.audio) {
        AudioTrackState.HAS_NO_SOUND -> return
        AudioTrackState.UNKNOWN -> !audioSettings.isEnabled(playbackKey.surfaceId)
        AudioTrackState.HAS_SOUND -> playbackState.isMuted
    }

    IconButton(
        modifier = modifier,
        onClick = {
            globalActions.action(
                GlobalAction.SetSurfaceAudioSetting(
                    playbackKey.surfaceId,
                    audioEnabled = muted,
                ),
            )
            playbackActions.action(
                playbackKey,
                PlaybackAction.Mute(!muted),
            )
            // More things, like handling product-related telemetry
        },
    ) {
        Icon(if (muted) Icons.VolumeOff else Icons.VolumeUp)
    }
}

The same pattern applies to every clickable or interactive media component. Every UI component also usually has its own ViewModel.

At the screen level, media becomes a set of declarative components that can be arranged like any other UI:

u/Composable
fun SimpleVideoScreen(
    screenState: SimpleVideoScreenState,
    videoComponent: Component<VideoProps>,
    playComponent: Component<PlayProps>,
    muteComponent: Component<MuteProps>,
    seekbarComponent: Component<SeekbarProps>,
    modifier: Modifier = Modifier,
) {
    val key = screenState.playbackKey

    Box(modifier.fillMaxSize()) {
        // 1. Video
        videoComponent.Content(props = screenState.toMediaProps())

        // 2. Play
        playComponent.Content(props = key.toPlayProps())

        // 3. Mute
        muteComponent.Content(props = key.toMuteProps())

        // 4. Seekbar
        seekbarComponent.Content(props = key.toSeekbarProps())
    }
}

Usually, the Media Foundation team, which focuses full time on media performance and developer experience, implements the components whose internals use both declarative and imperative APIs, whereas feature teams interact with the player through the declarative API.

This new setup gave us three big benefits:

• Product screens stay simple. They only need to arrange the media components and pass in a few props. Nothing more.

• Playback works the same way across the whole app. Every component shares the same playback state wired by playback key, actions, settings and telemetry pipeline under the hood, so users get one consistent experience no matter where they are.

• Performance. Screen-level state does not need to update for every media-related event. Only the affected media component recomposes.

Since introducing a component-based model in our app, AndroidX has made a new addition to their Media3 UI. This new model might serve as a good starting point if you’re thinking of solving similar problems today.  One key difference remains: our media components are tied to global playback state, while Media3 UI’s are tied to one Player. So the challenges still remain the same: player creation and the rest of the lifecycle ownership.

Experimentation Setbacks

Choreographer-based Seekbar

I personally found the seekbar a challenging component to implement smoothly, and there are a few interesting decisions we made.

So what does “smooth” mean in practice?
Basically, don't do more work than the screen can render. A 60Hz display takes roughly 16.7ms per frame; at 120Hz (which is not a rare thing anymore), that reduces to  ~8.3ms to render a frame. And if it goes beyond this limit, the user sees a jank.

This is where Choreographer can help. It runs right before Android draws the next frame, using the display’s real refresh rate. If the previous frame takes too long to draw, it waits for the next frame instead of firing again. That means we do not stack seekbar updates on top of an already-janky UI.

Estimated position or player position?
In our custom seekbar we predict the position by extrapolating from the last update, and only update it if the video is playing.
If your application's use cases are limited to video, ExoPlayer.getCurrentPosition() is a cheap operation, and it already uses an estimated position under the hood.

long elapsedTimeMs = SystemClock.elapsedRealtime() - positionUpdateTimeMs;
long estimatedPositionMs =
    Util.usToMs(positionUs) + (long) (elapsedTimeMs * playbackParameters.speed);

Polling this function will give you a smooth, speed-aware position for free. My recommendation is to still drive the updates using Choreographer and not use an arbitrary delay within a Coroutine or an Handler. 

Reddit's production experience
Moving seekbar updates out of screen-level state and driving them with Choreographer made the full-screen video experience measurably smoother.

On our full video screen, the slow-frame rate dropped by 7.3%.

Playback error 1004 & MediaSource reuse

While experimenting with our newly written ExoKit in production, there was a noticeable jump in playback error 1004 with "Unknown error" message. It affected a small share of devices and was challenging to reproduce locally. Our initial hypothesis was that it was a device-specific playback issue.

The root cause was our MediaSource cache. In some cases, the same cached MediaSource was reused across different player instances. This is something that I wouldn't recommend doing, as every media source is bound to the playback handler of the player it was originally attached to. Otherwise, you risk hitting an error which is thrown here.

Reddit's production experience
We fixed this by keying cached media sources by player id, so a MediaSource can only be reused by the same player instance.

Another possible fix was to move all playbacks onto a single playback thread. That also avoids the handler mismatch, but our production data showed that it made overall startup performance worse:

• % Video started in less than 250 ms: −0.199%

• % Video started in less than 500 ms: −0.132%

• % Video started in more than 1 sec: +1.165%

• % Video started in more than 2 sec: +0.634%

Rebuffering problems

If your app saves and restores playback position, be careful. In our old player we found that at least 36% of plays had a very short stall, under one second.

The main reason came from old choices made years ago. The old player restored the saved position as a check on every play and every move between screens. But the saved position did not match the position the player held at run time. Here is why: if you call player.pause() and then save player.position, and later call player.seek(savedPosition), the two numbers may differ by a few milliseconds. player.pause() does not finish right away. If you want the true position, wait for a playback state change first.

Reddit's production experience
We stopped saving positions this way. Instead we lean on the run time cache kept in the media source, and we reuse the player for videos the user watched before. This removed all of these tiny stalls.

Saving position on your own is still useful when the system kills the process of your app and you need to restore it for a long video.
If you cover this case, watch the order of your calls. Always call seek() first and prepare() after. If you do it the other way, the player loads key frames and chunks you don't plan to play.

Trade-off: Only one playable video at a time

Simultaneous playback is possible on Android, but hardware decoders are finite and shared across the whole device. A high-end phone might decode two or three videos at once, a low-end one just a single video, and an app sitting in Picture-in-Picture can quietly hold a decoder you were counting on. When you run out, you either fall back to software decoding (higher CPU usage, dropped frames) or fail playback with errors like 4001 or 4003.

Reddit's production experience
We could have partly mitigated this by checking the device performance class (a challenge in itself, since not all devices support it), but the development and testing cost wasn't worth the benefit. Instead, ExoKit runs a state machine that picks one active playback by priority (at Reddit it's based on how much of the video unit is visible, but it could also be based on how much of the screen's playable zone it fills).

We expected every surface to start faster after the rewrite. But surfaces that went from playing several videos to playing one improved their startup latency even more than the ones that already played a single video in the control group.

Conclusion

Besides an improved developer experience, the rewrite reduced perceived start latency by 65% at P50 and by 20% at P90. (An additional factor which helped here was the removal of a lot of unnecessary IO work from the startup path. In production traces, a single Main → IO dispatch before playback could add up to 11ms at the P99.)

Kudos to Merve Karaman, Ahmed Nawara, Irene Yeh and Vikram Aravamudhan for making this rewrite possible. We used to talk about this as a dream 2-3 years ago, and now it's our new reality.

Thanks to the following folks for helping me review this article: Iaroslav Khramov, Nicholas Ngorok

Authors:  Sreeprasad Govindankutty, Alex Okolish, Jerry Chu

This is the third post in our P0 media safety series. Previously, we covered Reddit's P0 Media Safety Detection and the Evolution of Reddit's In-House P0 Media Detection.

Introduction: The Blind Spot

Protecting our communities and users from high-risk, P0 content is a foundational priority for Reddit’s Safety team. Reddit's P0 media detection stack has historically relied on hash-matching: perceptual hashing algorithms like PhotoDNA, PDQ and hashsets from NCMEC, StopNCII, and Take It Down that compare uploaded media against databases of known violating content. This approach is precise: if an image has been reported and hashed before, we catch it. 

Our previous blog post identified a key focus area for future work: "Incorporate AI and ML to detect previously unseen media." This post describes how we did that by integrating Google's Content Safety API into our Content Classification Service (CCS), first for images and then for video. Our work has meaningfully improved our ability to identify and action harmful content of CSAM (Child Sexual Abuse Material).

Why Google's Content Safety API

Google's Content Safety API uses an ML model trained to classify raw media and return a priority score. Crucially, it operates on the media itself, independent of whether the content has been reported before. This makes it a natural complement to our existing hash-matching infrastructure. Hash-matching brings precision on known content; ML classification improves recall on unseen content. Together, they cover more ground than either approach alone.

How We Fit a New Signal Into Our Existing Pipeline

Instead of building a new service, we integrated the API directly into our existing media moderation flow. Media upload events are routed into image and video processing paths, where CCS prepares the media for classification. Then it sends the prepared media to Google's Content Safety API. The returned priority score from Google is then emitted into our eventing and analytics systems and used to create review tickets for Trust & Safety operations. The diagram below is a simplified view of the workflow.

After classification, the pipeline emits analytics events to our data warehouse, records telemetry, and creates review tickets based on the returned priority bucket for human review.

Key Technical Decisions

Image preprocessing. We resize images balancing Google's recommendations against preserving enough signal quality for accurate classification.

Rollout gating. A feature-flag-based fractional rollout controls blast radius. Only a configurable fraction of uploads call Google’s API, allowing us to ramp gradually and roll back instantly if needed.

The Image Integration: Design and Results

We ran a time-boxed evaluation over two weeks in Q4 2025. Reddit production image traffic was sent through the API, with priority scores logged to our data warehouse. Sampled review tickets were created across all priority score tiers, then reviewed and labeled by our Trust & Safety operations team using standard enforcement guidelines.

Because our sampling rates were deliberately biased toward higher-priority scores, we applied Inverse Probability Weighting (Horvitz–Thompson estimator) to correct for the bias and project true violation volumes in each bucket.

Image results

Meaningful CSAM recall lift. The API prioritized confirmed CSAM images that were not flagged by hash-matching. These were net-new detections: content that would have remained on the platform without this signal. The recall improvement over our hash-matching baseline was significant.

Priority scores sort risk effectively. The highest priority scores concentrated on CSAM with high precision. And the lower scores carried near-zero signals. This clean separation made it straightforward to route tickets to different Ops queues with appropriate review priorities.  

Higher actionability than hash-matching. When we compared the overall actionable rate of Google Content Safety tickets against PhotoDNA tickets (aggregating all violation types), the ML-based signal was substantially more actionable. After full rollout, the CSAM review queue maintained strong actionability.

Extending to Video

With the image results validated, video was the next frontier. Tens of thousands of video messages per day were flowing through CCS with no ML scoring. We built an event-driven callback for video, mirroring the image path with the addition of ffmpeg re-encoding and Google's video-specific constraints.

We rolled out at 100% for all priority levels over a multi-week evaluation window, creating review tickets for every video except the ones with the lowest scores. This contributed to the net-new CSAM coverage on video. The API surfaced confirmed CSAM videos that our existing video hash-match systems would have missed. The recall lift was smaller than the image evaluation (video upload volume is much lower), but directionally the same: the API catches what hashing cannot.

Impact and Takeaways

Hashing finds what is known; ML classification finds what is new. The combination is more effective than either approach alone. On images, the API delivered a meaningful CSAM recall lift. On video, it also added net-new CSAM coverage.

Stratified pilot design pays off. Deliberately varying sampling rates across priority scores let us measure both precision and projected volume before committing to full rollout, while keeping operational costs manageable. 

Future Work

1. Exploring the API's embeddings endpoint for additional signal types beyond the current priority classification.

2. Continuous monitoring of model drift via lower-score sentinel sampling to catch degradation early.

3. Evaluating Google's AI image detection API for additional classification capabilities beyond the Content Safety API.

If you're interested in working on problems like these, check out our careers page. Reddit Safety org is always looking for engineers who care about making the internet safer.

Hi friends - The job market is rough, and a lot of good people are looking for work. Good news is, Reddit is hiring across a wide range of functions, not just Engineering.

If you're looking, or know someone who is, take a look: redditinc.com/careers

Feel free to share with your network, group chats, Discord servers, fantasy leagues, family text threads, and other highly sophisticated professional recruiting channels. And not just with your engineering friends. Let's look out for one another!

Good luck out there. We're rooting for you.

Written by Shadi Altarsha, Transport team

When I started my career in infrastructure engineering, I wasn't sure how my work connected to the people actually using the product. I imagined myself deep in low-level systems that nobody knew existed, the kind of work that only surfaces when something breaks at 3 AM. 

Four years later, I've come to understand what I believe is one of the most important responsibilities of an infrastructure engineer: Keeping infrastructure concerns off application and platform engineers’ plates, so they can focus on their domain instead of paying the tax of context-switching into ours.

The way you deliver that experience is by doing the hard work of managing complexity, and building simple interfaces and abstractions so other teams don't have to think about it.

This post is about one such effort: how we removed an entire Envoy proxy from the serving path of Reddit's Home feed, Search rankings, and Notifications, and in doing so, improved availability, cut hundreds of CPU cores, and made onboarding new services trivial. The technical story involves gRPC service mesh, cross-namespace routing, CPU-aware load balancing, and a migration pattern for safely moving thousands of requests. But the underlying theme is simpler: sometimes the best thing infrastructure can do is disappear.

The Problem

When you open Reddit, your personalized experiences are powered by Ranking Platform, which is a set of gRPC services handling tens of thousands of requests per second across multiple Kubernetes namespaces and clusters. Traffic comes from our primary API gateway (GraphQL), Notification Platform, Answers, and other clients.

All of this traffic used to flow through an Envoy Gateway, a reverse proxy deployed and maintained by the Ranking Platform team, routing requests to the right service based on gRPC method and custom headers.

Here is the thing with great tools like Envoy: adopting one without deep experience is like giving a team a Formula 1 car when they have only ever driven regular sedans. A sedan is forgiving. You can change your own oil, and when something breaks it stalls on the side of the road. An F1 car has hidden sharp edges. It takes a pit crew to keep it on the track, and when it does fail, the failure is loud.
At first it feels like a huge upgrade. Then the complexity sneaks in. Configuration explodes, operational overhead grows, debugging becomes non-trivial, and before you know it, the team is not just the driver anymore. They need to be the mechanic and the pit crew too, on-call for a big dependency they did not originally sign up to own.

Why doesn't the Transport Infrastructure team just own these gateways? 
For one, we are a small Transport team supporting a large fleet of services across the company. More critically, the value of this kind of infrastructure comes from consistency across the fleet, which is hard to achieve when each team manages it independently.
We love Envoy and our Ingress deployment serves us well, but for gRPC service-to-service communication, we chose simplicity. So about a year ago, we adopted a proxyless service mesh instead.

Portal Transporter

Portal Transporter is Reddit's xDS-based control plane. It is a Kubernetes controller that watches GRPCRoute resources, EndpointSlices, and services, and then builds an xDS snapshot that it advertises directly to gRPC clients. The client dials an address like xds:///ranking.ranking-platform and gets back everything it needs, including routing rules, endpoint lists, load balancing configuration,  without any proxy sitting in the middle.

Instead of Client -> Proxy -> Backend, the architecture becomes Client -> Backend. The gRPC client itself handles routing and load balancing, programmed remotely by the control plane. This means you don’t need a sidecar, or a gateway.

If you want to understand the system in depth, my colleague Sotiris Nanopoulos gave a talk at gRPConf last year about it: Building a gRPC Proxyless World: How Reddit Scaled Resilience with xDS. It covers the control plane architecture and how we handle client-side observability as well. I would really recommend watching it if this topic interests you.

For this post though, all you need to know is this: Portal Transporter lets a service owner define their routing table once, and every client in the fleet gets it automatically. No client code changes needed. That is the foundation everything else in this post builds on.

The Goal

Here is the thing, to migrate one of the most important customers in the company to our infrastructure golden path, you cannot just show up at their door and say "hey folks, please use our toys." 

I approached this migration by asking myself a series of questions:

• In an ideal world where engineering cost is irrelevant, what is my perfect end goal?

• How much effort do my team and the team I am migrating have to pay to reach that end goal?

• Okay, that is very expensive. How can I find a common ground that achieves 90% of the dream goal without burning everyone out?

• And most importantly: how can I make the migration safe for both my team and my customer, and build trust in the process?

After working through these questions, here is what I landed on:

• Migrate the Ranking Platform to Portal Transporter without changing any of their existing architecture. I did not want to walk in and tell the Ranking team to restructure their services. The migration should be invisible to them as much as possible and the clients should only dial one address similar to how they used to do with Envoy.

• Cross-namespace routing was our biggest blocker. When I sat down and studied the Envoy Gateway configuration in detail, I realized we could not do this yet. The Ranking Platform team runs a cross-namespace setup, with services spread across multiple Kubernetes namespaces but unified behind a single Envoy routing table. Portal Transporter did not support that at the time.

• There were other challenges too. Envoy was a major source of SLO metrics and alerts for the Ranking team. They had even written a custom WASM filter to add tailored metrics to Envoy, which made replacing the observability layer harder than just swapping the data path. We needed to bring equivalent or better observability to the table before we could ask anyone to move.

• And above all, I had to do this safely. This is not some low-traffic internal tool, it serves traffic that powers some of Reddit's most important user experiences. A bad migration here would be very visible.

On the positive side, I was not coming empty-handed. I personally believe standardization is valuable, even if not everyone shares that belief as strongly, so I knew I needed something more concrete to bring to the table: CPU-aware load balancing (ORCA), which lets gRPC clients route requests based on how busy each server actually is. In practice that means better pod balance under load and real CPU savings, which are the things teams actually optimize for.

In the upcoming sections, I will walk through how we tackled each of these blockers and how we arrived at the architecture below.

The missing piece: cross-namespace routing

Before we could migrate anything, we had to solve a fundamental gap in Portal Transporter. The Ranking Platform's services all live in separate Kubernetes namespaces but are served through a single unified routing table. Clients call one address and the routing layer figures out which namespace to send the request to based on the gRPC method and headers. Portal Transporter had no concept of this. It assumed every GRPCRoute lived in the same namespace as the service it was routed to.

To make things harder, we already offer a public API to service owners across Reddit that supports exactly this kind of parent-child relationship across namespaces for the Ingress layer. Our ingress infrastructure relies on it, and our cross-cluster routing is also using it. Whatever we built for Portal had to fit into that same API. I would not introduce a new, ambiguous API or force changes on teams already using it. 
One API surface, two backends, that was the constraint.

This took a lot of iteration to get right. My first instinct was to look at what the Gateway API spec already offered. The Gateway resource has a model for cross-namespace routing, and it seemed like a natural fit. But we realized it was built on different assumptions than our infrastructure. So we stepped back and looked for something simpler.

What we ended up with is actually quite clever, and I say that knowing it took me three attempts and a lot of help from my colleagues to get there. We decided to use the GRPCRoute's backendRef field to reference other GRPCRoutes from different namespaces. Here is a simplified example:

The root GRPCRoute, let's say in the reddit-service-streaming-pipedream namespace, defines the main address that clients dial. But instead of routing directly to a service, some of its rules point to GRPCRoutes in other namespaces as backend references:

apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: ranking-platform
  namespace: reddit-service-streaming-pipedream
spec:
  rules:
- matches:
- method:
service: reddit.rankingplatform.ranking.v1.Ranking
backendRefs:
- kind: GRPCRoute
name: ranking-comments
namespace: reddit-service-ranking-comments
- kind: GRPCRoute
name: ranking-channels
namespace: reddit-service-ranking-channels
- kind: GRPCRoute
name: ranking-onefeed
namespace: reddit-service-ranking-homefeed

Each GRPCRoute in its own namespace then defines its own specific routing rules and points to its actual service. When Portal Transporter's control plane encounters these backend references, it expands them by pulling in the child routes and building a combined routing table that it serves to clients as a single xDS snapshot.

This design maps cleanly to the parent-child model our public API already supports for ingress. Service owners define their setup once, and it generates the right resources for both our ingress layer and Portal Transporter. No breaking changes or any special cases.

I want to be honest about the process though. This was not a clean straight path. The first design went through code review and was ultimately closed. The second attempt at combining GRPCRoutes had the right idea but the wrong execution. It was only on the third iteration, with significant input from my teammates, that we landed on the backendRef approach. It took several months from the initial proposal to the merged PR. 
I think there is something worth saying about that: good infrastructure design is not about getting it right the first time. It is about iterating and maybe discovering the real requirements as part of the process and it is about having colleagues who will tell you when an approach is not working, and being willing to throw away code that you spent weeks on (it is not easy when your “ego” is involved, by the way) and I am grateful my team gave me that.

Metrics and Alerts

Envoy was a major source of SLO metrics and alerts for the Ranking team. This was not a small detail but a real blocker.

At Reddit, we offer a standardized set of gRPC metrics and recording rules that teams can use to build dashboards and alerts. The Ranking Platform has a specific challenge: they use the same gRPC service and method across all their components, and they rely on custom headers like x-route to direct traffic to separate deployments. To get the breakdowns they actually needed, we had to add a custom layer on top of the standard metrics.

There was also the client-side SLO question. Client-based SLOs are an area we are still investing in, so there was no off-the-shelf path I could lean on for this migration. Aggregating metrics across multiple client sources is also not trivial, which meant I had to be pragmatic about scope.

This is where the "ideal world vs reality vs common ground" framework came back. A perfect, like-for-like replication of the existing observability layer would have taken months. But I could deliver a solution that gave the Ranking team most of what they actually needed:

• We built custom server and client metrics using grpc-go's StatsHandler, with the required headers as labels.

• We injected these metrics into the Ranking Platform components on the server side and into the clients.

• We used the server-side metrics to build alerts for each ranking tenant.

• We use the largest client as a stand-in for client-side SLO measurement.

Is it a one-to-one replacement for Envoy's observability? No. Does it produce a reliable outcome that the Ranking team can actually use? Yes. And honestly, I think that is the right tradeoff because perfect parity would have taken months of additional work and delayed the entire migration. 

Good enough, delivered now, was the better call.

De-risking the rollout

Here is a secret on how you make your managers and users happy at the same time: reliability. If there is one north star we operate under at Reddit Infrastructure, it is reliability. Everything else flows from that.

How do you deliver reliability when you are shuffling many RPCs?

Before migrating any traffic, I wanted to assure correctness to the best of my knowledge. And I want to be honest here, I really do not understand everything going on with the Ranking Platform and its clients. I do not know exactly what Notification Platform is calling or how Answers gives you all the information you need about the best daily shoe recommendations (spoiler: it is the Adidas Evo SL, thank me later). So what could I do to build confidence without pretending I understood every edge case?

I broke it down into three requirements:

1. Whatever development environment the engineers who work on Ranking components use must support Portal Transporter, not Envoy.

2. The Ranking Platform CI needs to run tests against my new setup and Envoy at the same time, and I should expect 1:1 correctness if my routing table is right.

3. My migration knobs should be able to turn traffic up and down between Envoy and the proxyless mesh in under two minutes.

For number one, we designed Portal to be development-friendly from day one, so there was no fundamental issue here. I just had to do some tweaks to make cross-namespace routing work in our development environment, but nothing major.

For number two, thanks to the Ranking Platform team, they already had very well thought-out smoke tests in their CI running against Envoy. All I needed to do was hook up the same calls via Portal alongside the existing tests and hope for the best. To my relief, it worked, the routing table was correct and the smoke tests passed on both paths.

Number three is where it gets interesting. We built a gRPC sampling client that has been part of Portal Transporter's migration toolkit since the early days. The idea is simple: we spin up two gRPC clients at the same time, one for the existing path (Envoy) and one for the new path (Portal Transporter). We use our dynamic configuration framework to provide a deterministic sampling knob that allows us to shift traffic between the two on the fly, straight from a UI. If something looks wrong at 10%, flip it back to 0% in seconds.
Under the hood, the implementation is actually quite neat. One of our engineers had the idea to override the ClientConnInterface interface in grpc-go, so that when Invoke is called, it checks against our rate sampler to decide which client handles the RPC. This means the sampling is completely transparent to the application code and the service owner does not need to change anything.

Teaching Servers to Tell Clients How Busy They Are

Remember when I said standardization itself is valuable, but I'm bringing something extra to the table? This is it.

With Envoy out of the picture, your gRPC clients now connect directly to server pods. The default load balancing strategy is round robin where every pod gets roughly the same number of requests. Sounds fair, right? The problem is that "fair" and "efficient" aren't the same thing as not all pods are equal. Some run on busier nodes, some share CPU with noisy neighbors, some just got unlucky with garbage collection timing. Round robin doesn't care, it sends traffic to an overloaded pod just as happily as it sends traffic to an idle one.
We implemented ORCA (Open Request Cost Aggregation) which is a standard that lets servers whisper back to clients: "hey, here's how busy I actually am." Each server pod reports its CPU utilization in gRPC response trailers, and the client uses these signals to compute weights. 
More loaded? Fewer requests. Less loaded? More requests. 
This is Client-Side Weighted Round Robin, and it's beautiful in its simplicity.

For Ranking Platform, the impact was significant: tighter CPU distribution across pods, improved availability, and a meaningful reduction in the number of CPU cores needed (we're talking about hundreds of cores saved).

My colleague wrote an excellent deep dive on ORCA at Reddit: Cheaper, Safer Scaling of CPU-Bound Workloads. If you're curious about the weight formula, the observability setup, and the operational lessons, go read that. It's worth your time.

After the migration

• We secured another nine on Ranking Onefeed.

• Hundreds of cores reclaimed across Ranking workloads, thanks to ORCA.

• Latency stayed flat

• Envoy Gateway fully decommissioned

• Simplified architecture

• Standardization: Ranking now uses the same infra golden path as the rest of Reddit

Conclusion

I started this post by saying that sometimes the best thing infrastructure can do is disappear. I think we did that here.

The Ranking team didn't have to change their architecture. The Transport team moved one of the most important deployments in the company to the golden path. The migration was seamless and honestly, the best compliment we got was that nobody realized it happened. For an infrastructure team, that silence is very important because it means trust, and in infrastructure, trust is your currency.

We improved availability for our users, saved a lot of CPU cores, and deleted a system that no team should have had to own in the first place. 

This work was also recognized internally, and that recognition mattered to me and the team. Migrations of this scale are never the work of one person. They happen because engineers across Transport, Ranking Platform, and the surrounding teams chose to invest in each other, and I am grateful to all of them.

A very happy ending 😄

Written by Will Johnson

Hello, my name is Will Johnson, and I’m a web engineer on Reddit’s Design System team. My team is responsible for Reddit's Design System, RPL, its corresponding component libraries, and helping other teams develop front-end experiences that adhere to our design system principles on all of Reddit's platforms.

Recently, I took on a project to resolve a persistent platform issue regarding web component dependencies that frustrated both Redditors and our engineering team. After responding to several internal bug reports, my team and I decided it was time to take a deeper look into the issue.  In this post, I share how I tackled that problem and what I learned along the way.

Missing Components

Reddit.com runs on a web component-based framework called LitElement, so we have quite a few template functions that render markup that contains a mix of native and custom elements. Since web components require JavaScript to render, developers were responsible for ensuring each page included the necessary component imports. Below is an example of what the markup and component imports would look like.

Example of server HTML template

<header>
 <reddit-header-small is-logged-in hide-on-scroll></reddit-header-small>
</header>
<main>
 <article>
   <h1 class="sr-only">Today's top posts</h1>
   <shreddit-post permalink="/r/cats/comments/abc123/fluffy/">
     <div slot="title">My cat learned to open doors</div>
   </shreddit-post>
   <shreddit-post permalink="/r/programming/comments/xyz789/rust/">
     <div slot="title">Why I rewrote everything in Rust</div>
   </shreddit-post>
 </article>
 <aside>
   <comment-forest-empty-state post-id="abc123">
     <p>No comments yet. Be the first!</p>
   </comment-forest-empty-state>
 </aside>
</main> 

// Register custom elements by importing their modules.
import 'reddit-header-small';
import 'shreddit-post'; 
import 'comment-forest-empty-state';

Historically, an engineer would write the page's markup, review the result in their development environment, and add any necessary dependencies to the import list if the page didn't display correctly. However, whenever an engineer removed a dependency or decided to load it asynchronously, the application could break. While "feature" may seem like a generic term, it has a specific meaning in our web codebase. Our web application is mostly server-rendered, where each page has a root feature and several ancestor feature nodes. Each feature has HTML markup and includes the dependencies necessary for that page to render. To visualize what happens when a dependency is missing, let's look at the example below. In that example, removing the modal’s import results in a darkened screen appearing without any additional context.

Having to keep track of your dependencies by hand created two problems: the first was a series of bugs in which various components appeared not to render, and the second was the need for increased internal support to debug these issues. To solve this problem, our team had the following idea: what if all client imports were automatically managed, so developers wouldn’t have to track them manually? By using import attributes, we could create a system that treats them as signals for client dependencies.

The big refactor: Adding Support For Import Attributes

To get things started, I needed to add the missing client imports to every template file that existed in our repository. I decided to enable our no-missing-imports rule from Lit Analyzer and create a script to parse the warnings and convert them into actual import statements. These imports would use a type attribute to indicate that they were client files, where the import would look something like this:  `import u/reddit with {type: ‘client’}`.After adding these imports across the codebase, I launched the development server to evaluate the system's response.

Unfortunately, the new client imports initially broke development and rollup builds. Node only supported type: json, and the rollup server builds picked up client imports, causing various issues. I chose to use plugins to strip these imports rather than more complex alternatives like dynamic imports or converting them to another format.

While the esbuild plugin for Node was simple, adding support for Rollup was a bit tricky. The TypeScript plugin performs type checking during the buildStart phase, which is too early for standard transformation hooks. I bypassed this by using type stubs in the monolith build to maintain type safety across specific builds while removing incompatible typings. With the server stable, I moved on to the parser.

Constructing the Shreddit AST Parser

Before coding, I aligned the team around a design document.

The node plugin approach was relatively straightforward: I used esbuild to omit client import attributes, but the rollup parser introduced an interesting problem. Whether I wrote my own transformer or used an off-the-shelf plugin, the imports remained unchanged. Although we have a rollup plugin to strip client imports, the TypeScript plugin sets up a file watcher and performs type checking during the buildStart phase of the plugin lifecycle. Using `buildStart`meant that block of code would always run before any plugin that uses a transform hook to load. To get around this limitation, I ended up using type stubs to remove incompatible typings in our monolith rollup build, which meant it did not affect the type safety of our client- and server-specific builds. Once the development server was operating correctly, I shifted my focus to building the parser.

Constructing the Shreddit AST Parser

I initially used the Recast AST parser to visit file dependency trees recursively, but later refactored to oxc-parser to cut runtime from 40 to 10 seconds. Before that, I relied on a few heuristics to reduce processing time, such as excluding child features, since Shreddit’s Orchestrator automatically handles their assets, and excluding components from a common client file that’s loaded on every page.

As I added more functionality to the parser, I discovered that each file contained metadata that could be used to parse the tree more efficiently. I decided to turn that into a class that would act as a cache, preserving visited files and telling me everything I needed to know about a file without revisiting it.

Metadata captured for visited files

export type Filemetadata = {
 features: Set<string>;
 dependencies: Set<string>;
 clientImports: Set<string>;
 // eslint-disable-next-line u/typescript-eslint/no-explicit-any
 exports?: any[];
 imports: Map<string, Set<string>>;
 // eslint-disable-next-line u/typescript-eslint/no-explicit-any
 ast?: any; // Store parsed AST for later access
};

type Entrypointmetadata = {
 skippedFiles: Set<string>;
 eligbleExports?: Set<string>;
 assets: Set<string>;
 clientImports: Set<string>;
}; 

With the basics in place, I decided to run the script and determine how the system was working so far. The image below captures the initial run and some of the work that the parser was able to do. As I looked more closely, I realized that some unexpected dependencies were sneaking into various features. These new dependencies caused the bundle size to increase dramatically. Were the exclusion heuristics working as expected? Was the parser incorrectly parsing the dependency tree? Or maybe these dependencies were correct after all. Either way, we needed a way to reduce import bloat.

Tree-Shaking Barrel Files

After completing my investigation, I realized that we had multiple barrel files in a few of our shared libraries that were not getting parsed accurately. There had been some discussion about this being a pattern we’d like to move away from, so my initial plan was to conduct a large refactor across the codebase. I decided to leverage an LLM to perform the refactor, so the work was done pretty quickly. But I soon realized that this change would require many reviews to get merged, a lint rule to curb its usage, and would ultimately still lack proper support in the parser.

To combat this problem, I decided to implement tree shaking to determine whether a module was actually requested before capturing its dependencies. The design of the algorithm involved peeking at the barrel file’s imports and temporarily holding them while I determined whether any modules had been imported by upstream files. While this incurs the overhead of parsing the barrel file rather than a direct import, it provides a sufficient mechanism to reduce the number of imports the parser detects. You can check the diagram below for a more detailed breakdown of how this workflow behaves.

Experimentation & Mitigations

With the final known roadblock resolved, I set out to begin experimentation by exposing a small subset of users to our newly generated imports. Unfortunately, the initial report showed several performance regressions. We were shipping more JavaScript, effectively slowing the rendering of various pages.

To unblock the project, I investigated which new dependencies were absent from production, using a staging utility to identify unused components. While I was considering a platform-level asset-exclusion API, another engineer used my work to address performance bloat by converting shared templates into features, powered by our feature-inclusion API, to exclude unnecessary assets from specific pages.

After finishing the migration, I created a document outlining the performance benchmarks we needed to meet. The goal was to prove that our changes were effective, keep partner teams informed about systemic impacts, and reach agreement on acceptable performance levels. The improvements were substantial, and we were finally in a position to roll out the feature to all users.

Conclusion

Overall, this project establishes a highly accurate dependency tree across our web codebase. It included improved code hygiene by systematically adding 4,000 previously missing client imports and removing 600 unused ones. Not only did this project eliminate a class of bugs in which UI components failed to appear due to incomplete dependency listings, but it also provided a foundational framework for future performance improvements.

Having an accurate dependency tree immediately enabled critical performance optimizations, including unused asset pruning, an optimized CSS delivery project leveraging the newly created AST parser, and a significant component-reduction initiative on a core surface. Adoption was made easy for engineers by including automated developer safety checks during commit and Continuous Integration (CI).

Additionally, we improved performance metrics across multiple areas, including Time to First Byte, First Contentful Paint, Total Page Load, Largest Contentful Paint, and Interaction to Next Paint, with strong statistical significance.

We’re hopeful that this platform will allow us to make additional improvements around serving more user-specific asset bundles in the future.