r/sre • • Mar 10 '23

BLOG A ‘unofficial’ investigation into Datadog’s latest outage. And a lesson on multi-cloud reliability

Thumbnail
overmind.tech
0 Upvotes

r/sre • • Apr 15 '22

[Visual summary] How Uber's infra handled growth from 2014-2016

Post image
5 Upvotes

r/sre • • Jul 21 '25

HUMOR Some Lame SRE jokes :)

11 Upvotes

1

Why did the on-call engineer miss his own birthday party?

Because the real surprise was a memory leak.

——————

2

A Knock knock joke (I like them a lot)

Knock knock

Who’s there?

Spike.

Spike who?

Latency Spike… or as we coffee-lovers call it:

Latte-ency Spike ☕

———

3

Why do on call engineers didn’t get promoted ?

Bcoz they missed their 1:1 with managers due to another call

————

4

What is the email subject for leave of on call engineers ?

OOI (out of incidents)

————

5

What’s the favourite bed time story for an on call engg ?

‘Once upon a time …. the system was up.’

————

6

Both SRE and his wife have load balancer in their browser history

  • he searched it in AWS console

  • his wife searched it on Amazon.com


r/sre • • Aug 23 '22

Hello everyone,

0 Upvotes

Where can I find a video tutorial for Structural Reliability Analysis of a Rectangular Plate?

This is for my thesis. I find a lot of papers regarding this where I can definitely understand the results. But I don't find anywhere it describes the step by step procedure software wise. Can anyone help me regarding this?

Thanks a lot.


r/sre • • Aug 25 '26

DISCUSSION Change fail rate isn't rising because of AI. It's rising because your batch sizes doubled!

Thumbnail
leaddev.com
0 Upvotes

A lot of engineering leaders I talk to are having some version of the same experience right now. The AI tooling is working, and it’s exciting. Teams are shipping things in days that used to take weeks. At the same time, everything feels like it’s falling apart – more bugs than they’ve ever had, more incidents, and engineers who are busier and more stressed.

If that’s you, it might honestly be more of an arithmetic problem than a quality one.


r/sre • • Dec 27 '25

Is the current hype around AI for SRE more about future potential than present usefulness?

0 Upvotes

Genuine question: Recently, I read the news that resolve.ai valuation is now a billion dollar after their recent funding. There has been too much hype about AI SRE since sometime but this was unbelievable. A lot of SREs and other people in this field that i have spoken to  seem skeptical of these tools ,about AI handling real-world incidents . Even i believe that the product is at a very nascent stage and will take atleast 1 year to be actually useful.  I don’t know what investors are actually betting on here. Am i missing something?


r/sre • • Apr 11 '21

How to create something like https://www.katacoda.com/

0 Upvotes

How to create some thing like https://www.katacoda.com/

What would it require at infrastructure level and application/front end level.

Any suggestions?


r/sre • • May 11 '25

I’m done applying. I’ll fix your cloud/SRE problem in 48 hours for free.

0 Upvotes

I’m a Site Reliability Engineer with 3 years of experience stabilizing cloud chaos , scaling infrastructure, optimizing observability, and putting out production fires nobody else could trace.

But after months of getting ghosted by hiring pipelines, I’m flipping the script.

Here’s the deal:
Give me one real, gnarly infra or SRE issue I’ll solve it in 48 hours. Free. No strings.

Dealing with stuff like:

  • ML workloads starving your GPU nodes and breaking autoscaling?

  • CI runners hogging ephemeral disks and silently failing deploys?

  • OpenTelemetry or Datadog showing 0% CPU... right before your pod dies?

  • Terraform state files locking up during high-frequency changes?

  • Real-time APIs randomly timing out under load but only during inference spikes?

  • S3 buckets quietly serving stale model files after a blue/green deployment?

  • IAM policies growing into unmanageable beasts breaking least privilege by accident?

  • Docker build cache exploding and pushing deploy times past 15 minutes?

  • EKS upgrades failing because of legacy node taints?

  • GitHub Actions burning free minutes due to missing cache keys?

  • Broken rollback logic that works in staging but fails in production?

  • Load balancers routing traffic unevenly across AZs during scale events?

  • Secrets leaking from ENV vars in ephemeral test environments?

  • Lambda cold starts doubling after a version bump and nobody knows why?

These are the problems I love solving and the kind of fires I’ve put out before.

Reply here or DM me your toughest infra/SRE pain. I’ll pick a few, solve them fast, and share anonymized fixes publicly.

You get a real solution. I get to prove what I can do no fluff, just execution.

Let’s build.


r/sre • • May 22 '22

K8s pods without K8s API

0 Upvotes

r/sre • • 20d ago

HIRING [Hiring] Senior Site Reliability Engineer – Backend Platform | Japan, Tokyo

0 Upvotes

Our client is a leading global technology company that develops large-scale digital platforms and innovative marketing solutions for customers worldwide.

We are looking for an experienced Senior Site Reliability Engineer (SRE) to drive system reliability, operational excellence, and automation across critical cloud-based platforms.

Key Responsibilities

  1. SRE Strategy and Reliability Engineering

Define and implement SRE strategies, including Service Level Objectives (SLOs), Service Level Indicators (SLIs), error budgets, and reliability targets.

Align reliability initiatives with business objectives and customer expectations.

2. Incident Management and Response

Establish and continuously improve incident management processes, including on-call rotations and escalation procedures.

Lead incident response activities and facilitate blameless postmortems.

Reduce mean time to recovery (MTTR) and implement preventive measures to avoid recurring incidents.

3. Reliability in the Software Development Lifecycle

Collaborate with development teams to integrate reliability practices throughout the software development lifecycle.

Promote design reviews, chaos engineering, and production readiness assessments.

4. Observability and Monitoring

Design and implement comprehensive observability solutions covering monitoring, logging, distributed tracing, and alerting.

Establish actionable insights into system health, availability, and performance.

5. Automation and Operational Efficiency

Drive automation initiatives to reduce repetitive operational tasks and improve deployment reliability.

Develop self-service capabilities that enable engineering teams to operate more efficiently.

6. Infrastructure Architecture and Optimization

Partner with architecture and platform teams to design scalable, fault-tolerant infrastructure.

Contribute to cloud infrastructure optimization and cost-efficiency initiatives.

7. Capacity Planning and Performance Optimization

Lead capacity planning and performance optimization for high-volume marketing platforms.

Ensure systems can reliably support large-scale campaign execution and real-time personalization workloads.

8. Reliability Reporting and Stakeholder Collaboration

Track and report reliability metrics, incident trends, and operational health.

Communicate technical findings and their business impact to engineering leaders and other stakeholders.

Mandatory Qualifications

・ 8+ years of experience in software engineering, DevOps, or site reliability engineering

・ Proven track record of building and leading high-performing SRE or platform engineering teams in a distributed, multi-timezone environment

・ Deep expertise in cloud platforms (GCP preferred, AWS/Azure acceptable) including compute, networking, storage, and managed services

・ Strong knowledge of containerization and orchestration technologies (Kubernetes, Docker) and Infrastructure as Code (Terraform, Ansible)

・ Hands-on experience with observability tools and practices (Prometheus, Grafana, Datadog, ELK Stack, or similar) and defining meaningful SLOs/SLIs

・ Experience with CI/CD pipelines, deployment strategies (blue-green, canary), and release engineering best practices

・ Strong programming/scripting skills in languages such as Python, Go, or Java for automation and tooling development

・ Excellent communication skills with the ability to collaborate effectively across engineering, product, and business stakeholders

・ Strong incident management experience with demonstrated ability to lead high-pressure situations calmly and effectively

Languages
English: Fluent
Japanese: Optional / a plus

Work Environment
Fast-paced, dynamic global environment with collaborative teams across multiple locations

Salary: ¥8M – ¥11M JPY per year (Middle, Senior level)
Location: Hybrid (4 days in the office, 1 day remote)
Office Location: Tokyo, Japan
Working Hours: Flexible schedule with core hours from 11:00 AM to 3:00 PM
Visa Sponsorship: Available
Language Requirement: English only

Apply now or contact us for further information:
[email protected]


r/sre • • May 12 '26

DISCUSSION Found out our webhook system was silently losing successful payments during deploys

0 Upvotes

We found a pretty nasty issue in our webhook pipeline a few months ago.

A merchant reached out saying a customer had successfully paid for a yearly subscription but never got access to the product. No email, no provisioning, nothing.

At first glance everything looked normal. The payment existed in the database, status was marked successful, charge ID was there, fully committed.

But the webhook event tied to it just, didn’t exist lol. Not failed. Not queued. Not retrying. Just gone.

Took a while to figure out what was happening, but eventually we traced it back to deploys. A pod would receive the request, start processing it, then get terminated mid-flight during rollout. Grace period expired, pod died, request vanished without ever making it into the retry system.

At our scale that tiny edge case was happening more than we were comfortable with.

The big thing we changed was moving event creation into the database transaction itself instead of relying on application code.

Now whenever a payment gets committed, a Postgres trigger writes the corresponding event into a domain_events table inside the same transaction. So either both happen, or neither happens. No weird “payment succeeded but event disappeared” state anymore.

From there we stream changes directly from Postgres using CDC instead of polling. Events go through Kafka, which basically acts as a durable buffer if downstream systems are unhealthy.

One thing we learned the hard way is that queues aren’t really the source of truth here, the database is. Kafka is just holding events in transit.

We also changed how retries work. Earlier, retries mostly lived in process memory, which sounds fine until the process disappears. Now execution state gets journaled so retries survive crashes, deploys, pod deaths, all of it.

For actual delivery we eventually stopped trying to do everything ourselves and moved the last-mile webhook delivery to Svix. Signing, retries, endpoint state, all of that is harder than it looks when you’re running payments infrastructure.

We still messed up v1 though. Originally the event payload stored payment state directly when the event was created. That worked until refunds happened between event creation and delivery, which meant merchants occasionally got stale webhook data. We ended up switching to fetching fresh state right before delivery instead.

The one thing we still monitor obsessively is replication lag on the CDC side. If consumers fall behind badly enough, WAL growth becomes its own production problem pretty quickly.

Curious how others are handling that part, especially around replication lag and replay systems.

you can read the full blog at https://dodopayments.com/engineering/building-webhooks-never-fail


r/sre • • Dec 28 '25

CAREER HELP! - DevOps / SRE Role Resume

Post image
1 Upvotes

Hello everyone,

Any suggestions for this resume?

P.S: I’m based in Chennai, India. I’d also appreciate guidance on the salary range I can realistically expect with my current experience.


r/sre • • Mar 13 '24

[HIRING] [FTE] Sr.SRE at Service based company. Salary 160k-180k with a bonus of 20-30k a year.

2 Upvotes

Send me resume if interested.


r/sre • • Sep 16 '25

HIRING We're hiring Forward Deployed Engineers at SigNoz

0 Upvotes

Apply here: https://jobs.ashbyhq.com/SigNoz/4b8cd389-88c0-4301-b770-5bc7332f773c

🚀 23k+ ⭐ on GitHub, 6k+ members in Slack — want to help supercharge it?

We’re an open-source, OpenTelemetry-native observability platform (traces + metrics + logs). YC-backed. Fully remote—no offices.

What you’ll do

🔧 Design & implement observability in customers infra: OTel instrumentation, tailored dashboards, real-world optimization
📝 Write crisp integration guides, troubleshooting docs & best practices engineers actually follow
💻 Help instrument customer codebases (Go/Python/Node/Java), setup Otel agents, ensure successful rollouts
🧩 Spot patterns across deployments and feed them into product defaults, templates & tooling

You’ll thrive if you

🛠️ Have 2–6 yrs in DevOps/SRE/Platform/Solutions Eng
🐳 Know containers, Kubernetes, IaC, and at least one cloud (AWS/GCP/Azure)
💻 Enjoy hands-on coding across stacks
✍️ Care about clear, actionable technical writing

Not a fit if you

🙈 Prefer working in isolation vs partnering with engineers
📝 Avoid documentation
🚫 Shy away from hands-on implementation

Why SigNoz

🌍 Build a global dev-infra product with a 200+ contributor OSS community
⚡ High ownership, talk to users daily
🌱 Backed by YC & top Bay Area VCs, remote-first

Location: Remote - India

Compensation range: ₹30L - ₹40L INR


r/sre • • Apr 07 '24

HELP Is SRE that bad ?

0 Upvotes

I like Cloud and am working in it, but recently, I saw an overflooded amount of posts talking about how SRE is bad and stressful. They have to be available 24 x 7 and have to work anytime a Cloud infrastructure goes down.

Is that so ?

Is SRE really that bad ? Or is it exaggerated ? How do I find companies which have bad SRE jobs, like from their JD ?


r/sre • • Aug 20 '24

Is it possible to transition from sre to swe?

1 Upvotes

I want to know if I’m reducing my options in a career if I go into sre as a new grad


r/sre • • Jul 25 '25

Pre-mortem

0 Upvotes

I just invented a new word: pre-mortem.

It's like post-mortem, but before it hit the production. Someone notice root cause by chance, before it happened and avoided post-mortem all together.

Like "or, won't it be a problem if those to things start to override each other?", and everyone else like 'oh, that big..." and it didn't happened, and was just a small boring change. Instead of a bloody report, postmortem, public apology and commit description like 'fixing the problem which cost company 3 hours global outage and a week of confusion'.

It's pre-mortem, and they are way cooler than post-mortems.


r/sre • • Dec 03 '24

CPU throttling in Kubernetes

0 Upvotes

Here's a blog on how to catch and manage CPU throttling when it's happening in K8s to improve application performance.


r/sre • • Dec 09 '23

DISCUSSION DevOps vs SRE vs Platform Engineering

Thumbnail
youtube.com
0 Upvotes

r/sre • • May 16 '26

DISCUSSION LLMs solve about 1 in 3 real root-cause cases on a realistic benchmark. Mostly wrong on the hard ones.

0 Upvotes

Hi team: Sharing something I came across --

Here is what the 2025-26 research actually says about llms doing root cause analysis. Because the demos and the on-call reality are far apart and imo this is the right room to be honest about it.

On OpenRCA, an MSFT and Tsinghua benchmark built to look like real production, llm agents went from solving roughly 1 in 10 real failure cases in early 2025 to roughly 1 in 3 by early 2026 (that is a real jump).

It is also still mostly wrong on the very hard, multi-part failures. Both halves are true tbh and the second half is more top of mind when you / I / SREs are the one paged.

One detail that should make the industry skeptical is that when the system saw a cleaner, reduced slice of the signals, accuracy went up. On a realistic messy slice it dropped. Goes without saying, our production telemetry is the messy slice and everyone's is.

The useful finding is that the lever is not model size, it is structure. A 2026 study ran the full benchmark across several models and the two most common failure modes, hallucinated readings of the data and stopping the search too early, showed up across every model regardless of how capable it was.

Raw model on raw telemetry is near useless. Model plus retrieval plus an SOP that bounds where it can go is genuinely useful as a first responder, tho not as the final word.

So, here is my honest read. Use agentic SRE to compress a mountain of telemetry into a ranked set of suspects in minutes, then a human makes the call - that's the reality of today. It does not replace the engineer and the research does not claim it does.

I've been frequenting this sub off late and as the field evolves, I am curious what would actually make you trust one of these agents on your stack, the headline accuracy number, or the structure around the model, or anything else?


r/sre • • Jan 07 '26

DISCUSSION Claude Code Cope

13 Upvotes

Okay. I feel like certain roles within the software development life cycle are Coping pretty hard with how advanced AI has gotten. For context I’m a 24yr old QA engineer at a f500, specifically I do performance testing and work a lot with SRE/infra teams. As someone who actually keeps up with ai unlike my colleagues I’ve come to the realisation my role is pretty much automated using Claude code. The new browser plugin can manually go through apps and has complete access to network traffic allowing it to generate non trivial performance test scripts in any language.

I pointed this out on the QA subreddit and got pretty negative reaction. Personally my job is only safe for few years due to archaic practises and adoption lag at my bloated f500 company.

What would you do in my situation? I’m attempting to move into the SRE team now. Should I mention to my manager that my job is automated and explain my worries? Would you even bother upskilling to become an SRE in this day and age?


r/sre • • Jan 11 '22

SRE vs DevOps: Infographics

Post image
0 Upvotes

r/sre • • Nov 08 '24

Python for SRE

0 Upvotes

What level of python skill needed for a SRE role?


r/sre • • Feb 06 '25

PROMOTIONAL SigNoz vs. New Relic. Is It Really That Much Better? What's the Catch?

Thumbnail
signoz.io
0 Upvotes