r/sre • u/jameslaney • Mar 10 '23
BLOG A ‘unofficial’ investigation into Datadog’s latest outage. And a lesson on multi-cloud reliability
r/sre • u/jameslaney • Mar 10 '23
r/sre • u/hatchikyu • Apr 15 '22
r/sre • u/incidentjustice • Jul 21 '25
Why did the on-call engineer miss his own birthday party?
Because the real surprise was a memory leak.
——————
A Knock knock joke (I like them a lot)
Knock knock
Who’s there?
Spike.
Spike who?
Latency Spike… or as we coffee-lovers call it:
Latte-ency Spike ☕
———
Why do on call engineers didn’t get promoted ?
Bcoz they missed their 1:1 with managers due to another call
————
What is the email subject for leave of on call engineers ?
OOI (out of incidents)
————
What’s the favourite bed time story for an on call engg ?
‘Once upon a time …. the system was up.’
————
Both SRE and his wife have load balancer in their browser history
he searched it in AWS console
his wife searched it on Amazon.com
r/sre • u/Dependent_Trouble_99 • Aug 23 '22
Where can I find a video tutorial for Structural Reliability Analysis of a Rectangular Plate?
This is for my thesis. I find a lot of papers regarding this where I can definitely understand the results. But I don't find anywhere it describes the step by step procedure software wise. Can anyone help me regarding this?
Thanks a lot.
r/sre • u/Suspicious_Orchid770 • Aug 25 '26
A lot of engineering leaders I talk to are having some version of the same experience right now. The AI tooling is working, and it’s exciting. Teams are shipping things in days that used to take weeks. At the same time, everything feels like it’s falling apart – more bugs than they’ve ever had, more incidents, and engineers who are busier and more stressed.
If that’s you, it might honestly be more of an arithmetic problem than a quality one.
r/sre • u/Actual_Storage_3698 • Dec 27 '25
Genuine question: Recently, I read the news that resolve.ai valuation is now a billion dollar after their recent funding. There has been too much hype about AI SRE since sometime but this was unbelievable. A lot of SREs and other people in this field that i have spoken to seem skeptical of these tools ,about AI handling real-world incidents . Even i believe that the product is at a very nascent stage and will take atleast 1 year to be actually useful. I don’t know what investors are actually betting on here. Am i missing something?
r/sre • u/myoutlet101 • Apr 11 '21
How to create some thing like https://www.katacoda.com/
What would it require at infrastructure level and application/front end level.
Any suggestions?
r/sre • u/LongjumpingRole7831 • May 11 '25
I’m a Site Reliability Engineer with 3 years of experience stabilizing cloud chaos , scaling infrastructure, optimizing observability, and putting out production fires nobody else could trace.
But after months of getting ghosted by hiring pipelines, I’m flipping the script.
Here’s the deal:
Give me one real, gnarly infra or SRE issue I’ll solve it in 48 hours. Free. No strings.
Dealing with stuff like:
ML workloads starving your GPU nodes and breaking autoscaling?
CI runners hogging ephemeral disks and silently failing deploys?
OpenTelemetry or Datadog showing 0% CPU... right before your pod dies?
Terraform state files locking up during high-frequency changes?
Real-time APIs randomly timing out under load but only during inference spikes?
S3 buckets quietly serving stale model files after a blue/green deployment?
IAM policies growing into unmanageable beasts breaking least privilege by accident?
Docker build cache exploding and pushing deploy times past 15 minutes?
EKS upgrades failing because of legacy node taints?
GitHub Actions burning free minutes due to missing cache keys?
Broken rollback logic that works in staging but fails in production?
Load balancers routing traffic unevenly across AZs during scale events?
Secrets leaking from ENV vars in ephemeral test environments?
Lambda cold starts doubling after a version bump and nobody knows why?
These are the problems I love solving and the kind of fires I’ve put out before.
Reply here or DM me your toughest infra/SRE pain. I’ll pick a few, solve them fast, and share anonymized fixes publicly.
You get a real solution. I get to prove what I can do no fluff, just execution.
Let’s build.
r/sre • u/pithagobr • May 22 '22
Hello everyone, Please read my new article https://sreblog.substack.com/p/kubernetes-pods-without-the-kubernetes?sd=fs
r/sre • u/FunMuted6440 • 20d ago
Our client is a leading global technology company that develops large-scale digital platforms and innovative marketing solutions for customers worldwide.
We are looking for an experienced Senior Site Reliability Engineer (SRE) to drive system reliability, operational excellence, and automation across critical cloud-based platforms.
Key Responsibilities
SRE Strategy and Reliability Engineering
Define and implement SRE strategies, including Service Level Objectives (SLOs), Service Level Indicators (SLIs), error budgets, and reliability targets.
Align reliability initiatives with business objectives and customer expectations.
2. Incident Management and Response
Establish and continuously improve incident management processes, including on-call rotations and escalation procedures.
Lead incident response activities and facilitate blameless postmortems.
Reduce mean time to recovery (MTTR) and implement preventive measures to avoid recurring incidents.
3. Reliability in the Software Development Lifecycle
Collaborate with development teams to integrate reliability practices throughout the software development lifecycle.
Promote design reviews, chaos engineering, and production readiness assessments.
4. Observability and Monitoring
Design and implement comprehensive observability solutions covering monitoring, logging, distributed tracing, and alerting.
Establish actionable insights into system health, availability, and performance.
5. Automation and Operational Efficiency
Drive automation initiatives to reduce repetitive operational tasks and improve deployment reliability.
Develop self-service capabilities that enable engineering teams to operate more efficiently.
6. Infrastructure Architecture and Optimization
Partner with architecture and platform teams to design scalable, fault-tolerant infrastructure.
Contribute to cloud infrastructure optimization and cost-efficiency initiatives.
7. Capacity Planning and Performance Optimization
Lead capacity planning and performance optimization for high-volume marketing platforms.
Ensure systems can reliably support large-scale campaign execution and real-time personalization workloads.
8. Reliability Reporting and Stakeholder Collaboration
Track and report reliability metrics, incident trends, and operational health.
Communicate technical findings and their business impact to engineering leaders and other stakeholders.
Mandatory Qualifications
・ 8+ years of experience in software engineering, DevOps, or site reliability engineering
・ Proven track record of building and leading high-performing SRE or platform engineering teams in a distributed, multi-timezone environment
・ Deep expertise in cloud platforms (GCP preferred, AWS/Azure acceptable) including compute, networking, storage, and managed services
・ Strong knowledge of containerization and orchestration technologies (Kubernetes, Docker) and Infrastructure as Code (Terraform, Ansible)
・ Hands-on experience with observability tools and practices (Prometheus, Grafana, Datadog, ELK Stack, or similar) and defining meaningful SLOs/SLIs
・ Experience with CI/CD pipelines, deployment strategies (blue-green, canary), and release engineering best practices
・ Strong programming/scripting skills in languages such as Python, Go, or Java for automation and tooling development
・ Excellent communication skills with the ability to collaborate effectively across engineering, product, and business stakeholders
・ Strong incident management experience with demonstrated ability to lead high-pressure situations calmly and effectively
Languages
English: Fluent
Japanese: Optional / a plus
Work Environment
Fast-paced, dynamic global environment with collaborative teams across multiple locations
Salary: ¥8M – ¥11M JPY per year (Middle, Senior level)
Location: Hybrid (4 days in the office, 1 day remote)
Office Location: Tokyo, Japan
Working Hours: Flexible schedule with core hours from 11:00 AM to 3:00 PM
Visa Sponsorship: Available
Language Requirement: English only
Apply now or contact us for further information:
[email protected]
r/sre • u/aagarwal1012 • May 12 '26
We found a pretty nasty issue in our webhook pipeline a few months ago.
A merchant reached out saying a customer had successfully paid for a yearly subscription but never got access to the product. No email, no provisioning, nothing.
At first glance everything looked normal. The payment existed in the database, status was marked successful, charge ID was there, fully committed.
But the webhook event tied to it just, didn’t exist lol. Not failed. Not queued. Not retrying. Just gone.
Took a while to figure out what was happening, but eventually we traced it back to deploys. A pod would receive the request, start processing it, then get terminated mid-flight during rollout. Grace period expired, pod died, request vanished without ever making it into the retry system.
At our scale that tiny edge case was happening more than we were comfortable with.
The big thing we changed was moving event creation into the database transaction itself instead of relying on application code.
Now whenever a payment gets committed, a Postgres trigger writes the corresponding event into a domain_events table inside the same transaction. So either both happen, or neither happens. No weird “payment succeeded but event disappeared” state anymore.
From there we stream changes directly from Postgres using CDC instead of polling. Events go through Kafka, which basically acts as a durable buffer if downstream systems are unhealthy.
One thing we learned the hard way is that queues aren’t really the source of truth here, the database is. Kafka is just holding events in transit.
We also changed how retries work. Earlier, retries mostly lived in process memory, which sounds fine until the process disappears. Now execution state gets journaled so retries survive crashes, deploys, pod deaths, all of it.
For actual delivery we eventually stopped trying to do everything ourselves and moved the last-mile webhook delivery to Svix. Signing, retries, endpoint state, all of that is harder than it looks when you’re running payments infrastructure.
We still messed up v1 though. Originally the event payload stored payment state directly when the event was created. That worked until refunds happened between event creation and delivery, which meant merchants occasionally got stale webhook data. We ended up switching to fetching fresh state right before delivery instead.
The one thing we still monitor obsessively is replication lag on the CDC side. If consumers fall behind badly enough, WAL growth becomes its own production problem pretty quickly.
Curious how others are handling that part, especially around replication lag and replay systems.
you can read the full blog at https://dodopayments.com/engineering/building-webhooks-never-fail
r/sre • u/Thyprasat28 • Dec 28 '25
Hello everyone,
Any suggestions for this resume?
P.S: I’m based in Chennai, India. I’d also appreciate guidance on the salary range I can realistically expect with my current experience.
r/sre • u/masked_rumblings • Mar 13 '24
Send me resume if interested.
r/sre • u/MithunArunan • Sep 16 '25
Apply here: https://jobs.ashbyhq.com/SigNoz/4b8cd389-88c0-4301-b770-5bc7332f773c
🚀 23k+ ⭐ on GitHub, 6k+ members in Slack — want to help supercharge it?
We’re an open-source, OpenTelemetry-native observability platform (traces + metrics + logs). YC-backed. Fully remote—no offices.
What you’ll do
🔧 Design & implement observability in customers infra: OTel instrumentation, tailored dashboards, real-world optimization
📝 Write crisp integration guides, troubleshooting docs & best practices engineers actually follow
💻 Help instrument customer codebases (Go/Python/Node/Java), setup Otel agents, ensure successful rollouts
🧩 Spot patterns across deployments and feed them into product defaults, templates & tooling
You’ll thrive if you
🛠️ Have 2–6 yrs in DevOps/SRE/Platform/Solutions Eng
🐳 Know containers, Kubernetes, IaC, and at least one cloud (AWS/GCP/Azure)
💻 Enjoy hands-on coding across stacks
✍️ Care about clear, actionable technical writing
Not a fit if you
🙈 Prefer working in isolation vs partnering with engineers
📝 Avoid documentation
🚫 Shy away from hands-on implementation
Why SigNoz
🌍 Build a global dev-infra product with a 200+ contributor OSS community
⚡ High ownership, talk to users daily
🌱 Backed by YC & top Bay Area VCs, remote-first
Location: Remote - India
Compensation range: ₹30L - ₹40L INR
r/sre • u/AsishPC • Apr 07 '24
I like Cloud and am working in it, but recently, I saw an overflooded amount of posts talking about how SRE is bad and stressful. They have to be available 24 x 7 and have to work anytime a Cloud infrastructure goes down.
Is that so ?
Is SRE really that bad ? Or is it exaggerated ? How do I find companies which have bad SRE jobs, like from their JD ?
r/sre • u/Big_Mountain9707 • Aug 20 '24
I want to know if I’m reducing my options in a career if I go into sre as a new grad
r/sre • u/amarao_san • Jul 25 '25
I just invented a new word: pre-mortem.
It's like post-mortem, but before it hit the production. Someone notice root cause by chance, before it happened and avoided post-mortem all together.
Like "or, won't it be a problem if those to things start to override each other?", and everyone else like 'oh, that big..." and it didn't happened, and was just a small boring change. Instead of a bloody report, postmortem, public apology and commit description like 'fixing the problem which cost company 3 hours global outage and a week of confusion'.
It's pre-mortem, and they are way cooler than post-mortems.
r/sre • u/Background-Fig9828 • Dec 03 '24
Here's a blog on how to catch and manage CPU throttling when it's happening in K8s to improve application performance.
r/sre • u/Kickass_Wizard • Dec 09 '23
r/sre • u/gaurav_sherlocks_ai • May 16 '26
Hi team: Sharing something I came across --
Here is what the 2025-26 research actually says about llms doing root cause analysis. Because the demos and the on-call reality are far apart and imo this is the right room to be honest about it.
On OpenRCA, an MSFT and Tsinghua benchmark built to look like real production, llm agents went from solving roughly 1 in 10 real failure cases in early 2025 to roughly 1 in 3 by early 2026 (that is a real jump).
It is also still mostly wrong on the very hard, multi-part failures. Both halves are true tbh and the second half is more top of mind when you / I / SREs are the one paged.
One detail that should make the industry skeptical is that when the system saw a cleaner, reduced slice of the signals, accuracy went up. On a realistic messy slice it dropped. Goes without saying, our production telemetry is the messy slice and everyone's is.
The useful finding is that the lever is not model size, it is structure. A 2026 study ran the full benchmark across several models and the two most common failure modes, hallucinated readings of the data and stopping the search too early, showed up across every model regardless of how capable it was.
Raw model on raw telemetry is near useless. Model plus retrieval plus an SOP that bounds where it can go is genuinely useful as a first responder, tho not as the final word.
So, here is my honest read. Use agentic SRE to compress a mountain of telemetry into a ranked set of suspects in minutes, then a human makes the call - that's the reality of today. It does not replace the engineer and the research does not claim it does.
I've been frequenting this sub off late and as the field evolves, I am curious what would actually make you trust one of these agents on your stack, the headline accuracy number, or the structure around the model, or anything else?
r/sre • u/acewithacase • Jan 07 '26
Okay. I feel like certain roles within the software development life cycle are Coping pretty hard with how advanced AI has gotten. For context I’m a 24yr old QA engineer at a f500, specifically I do performance testing and work a lot with SRE/infra teams. As someone who actually keeps up with ai unlike my colleagues I’ve come to the realisation my role is pretty much automated using Claude code. The new browser plugin can manually go through apps and has complete access to network traffic allowing it to generate non trivial performance test scripts in any language.
I pointed this out on the QA subreddit and got pretty negative reaction. Personally my job is only safe for few years due to archaic practises and adoption lag at my bloated f500 company.
What would you do in my situation? I’m attempting to move into the SRE team now. Should I mention to my manager that my job is automated and explain my worries? Would you even bother upskilling to become an SRE in this day and age?
r/sre • u/Sea_Asparagus5286 • Nov 08 '24
What level of python skill needed for a SRE role?
r/sre • u/RandomThoughtsAt3AM • Feb 06 '25