I wonder what took them so long. PagerDuty seems to have become one of those heavyweight products that are so content in their illusion of market dominance that they have stopped innovating. But until the enterprise CFOs wake up and ask why is this costing us 5k per month, they are going to stay in their bubble.

I last used PD 3 years ago, and the UI had not changed in years, looked like something out of a 90s app. Pricing was our way or the highway.

No wonder people are leaving it for other solutions.

I can't believe how fast things are moving. Seeing Zuck saying his AI is replacing mid level engineers, the non stop offshore hiring, the fact my team is 50% is in Latin America now it's all so scary man, all the h1b visa stuff and the nonstop AI scares. I read a post that a few people are considering jumping ship to the medical field.

Im genuinely terrified of the future now. I wanted to change jobs, but i'd rather just be comfortable with this one till they lay me off with severance even though it's not ideal.

i hate this.

Saw this post published yesterday about Netflix's logging arch https://clickhouse.com/blog/netflix-petabyte-scale-logging

Pretty cool, does anyone know if netflix did a blog or talk that goes deeper?

It says they have 40k microservices?!?! Can't even really imagine dealing with that

we have three observability tools. datadog for metrics and apm. splunk for logs. sentry for errors.

looked at the bill last month. $47k for datadog. $38k for splunk. $12k for sentry. our actual aws infrastructure costs $52k.

we're spending more money watching our systems than running them. that's insane.

tried to optimize. reduced log retention. sampled more aggressively. dropped some custom metrics. saved maybe $8k total but still paying almost $90k a month to know when things break.

leadership asked why observability costs so much. told them "because datadog charges per host and we autoscale" and they looked at me like i was speaking another language.

the worst part is we still can't find stuff half the time. three different tools means three different query languages and nobody remembers which logs are in splunk vs cloudwatch.

pretty sure we're doing this wrong but not sure what the alternative is. everyone says observability is critical but nobody warns you it costs more than your actual infrastructure.

anyone else dealing with this or did we just architect ourselves into an expensive corner.

I run SRE/Ops at a small tech company and we had a doozy today.

A "Claude Hole" is when engineer is troubleshooting or developing code with Claud/llm that they don't understand and end up in a different zip code from the actual solution.

Example: We had an engineer today run into a bug with CNPG template, due to a really simple value miss they didn't set the AWS account number correctly in the service account annotation. Fairly easy to spot due to the cluster throwing IAM errors.

They somehow ended up submitting a PR changing the OICD for EVERY SERVICE ACCOUNT in there org. SRE blocked the PR and spent the next hour trying to figure out what the hell this engineer was actually trying to do.

On of the SRE's described it as goaltending which I thought was apt.

Stay safe our there buddies , shits getting weird.

Side note, mods we need a horror story flare .

Just read through Netflix's writeup about moving from centralized SRE owned incident response to empowering all engineers to declare and manage incidents: https://netflixtechblog.com/empowering-netflix-engineers-with-incident-management-ebb967871de4

This really resonates with challenges we've been facing during peak shopping seasons. We had a similar problem where only our SRE team would declare incidents, which meant a lot of issues that should have been escalated weren't, especially when the business side engineers hit problems during Black Friday or holiday rushes. The whole "engineers don't want to deal with incident paperwork" thing is so real.

What I found interesting was their focus on making the process intuitive rather than just adding more tooling. We've been working on something similar, trying to reduce the friction between "something's wrong" and "incident declared." The part about moving from an underutilized incident template to actual ownership across teams really hits home. Anyone else dealing with this kind of cultural shift around incident ownership? Curious how other commerce folks have handled the seasonal traffic aspect of this.

Anthropic's SRE team gave a talk at QCon last week worth reading if you're thinking about AI for incident response.

Alex Palcuie has been using Claude as his first tool in incident response since January. The New Year's Eve example is good: HTTP 500s on Claude Opus 4.5, looked like a bug, turned out to be 4,000 accounts created simultaneously all hammering the API at once. Claude found the fraud pattern in seconds. Palcuie says he would have filed it as a bug and never paged account abuse.

The failure mode is just as specific. Every time their KV cache broke and caused a request spike, Claude called it a capacity problem. Add more servers. Every single time. It has no idea the KV cache has broken this exact way before.

His framing is AI at the observation layer is genuinely superhuman, which I agree with. AI at the orient-and-decide loop mistakes correlation for causation reliably enough that you can't trust it there yet, again I agree.

The scar tissue point is the one I keep coming back to. The model doesn't know your system's history. That context lives in people. If AI handles more incidents, the next generation of engineers never builds it and nobody's figured out how to encode ten years of "we've seen this before" into a model that's never been paged at 3am.

https://www.theregister.com/2026/03/19/anthropic_claude_sre/

My boss announced he’s leaving a couple of weeks ago and just found out I’ll be the one to replace him.

Big company with a stream of incidents and tickets that don’t stop. Burnout almost derailed the whole team a couple of time in 2023 and I don’t want it to happen under me.

I’ve dealt with burn out before and want to be the type of boss who cares about the well-being of my team. I know how to manage burnout personally (meditation, healthy habits), but looking for tips on how to fight it in an org.

Saw an email with a word doc full of "critical misalignments" and "savings opportunities" generated by the COO's wife and sent to me and the Sr. devs. Read through it and it suggested setting our already-fragile CPU/Ram based ECS scaling policies from 25% utilization -> 50% for big savings!! I wrongly assumed that he would be smart enough to know that suggestion was crap as we have seen it cause issues even at 40%. He proceeded with it anyways and without telling anyone. Busy Friday rolls around and low and behold, shit is down and people are calling us.

I set it back to what it was and tell him we really need to move to latency based scaling but get waved off.

His response on how to communicate the cause? Unexpected increase in customer load and we have "permanently adjusted the new baseline in response!"

Fml

We had an internal meeting last week about promising a 99.9% uptime SLA to a new enterprise customer. Everyone was nodding like "yep, that's reasonable." Then I did the math on what 99.9% actually means: ~43 minutes of downtime per month.

The funny part is we’d already blown through that on Saturday during a P1. I had to be the one to break the news in the meeting. The room got real quiet.

There was even a short debate about pushing for another nine (99.99%). I honestly had to stop myself from laughing out loud. If we can’t keep three nines, how on earth are we going to do four?

In the end we decided not to make the guarantee and just sell without it. Curious if anyone else here has had to be the bad guy in an SLA conversation?

Google released two early-release chapters from the SRE Book 2nd Edition this week.

One is the new "AI for SRE" chapter. It's on O'Reilly publication behind a paywall, but a free trial works. Read it last night, sharing the takeaways for anyone who doesn't to read the full thing.

The condensed version:

1. AI is not a human replacement. The book is firm on this. We still need humans for the high-stakes calls and to maintain the AI itself.

2. Don't give AI full access on day one. Build trust the way you would with a junior engineer. Let it suggest fixes first, fix small issues next, only then expand its scope.

3. If the agent can take an action, it must have a rollback. If there is no undo path, the access should not be granted. This is the line I think most teams shipping agents are skipping right now.

4. When the agent fails or gives a bad suggestion, flag it. The chapter leans on the same principle as good postmortem culture, more feedback and more context means better future execution.

5. During incidents, the time-saver is not the fix, it is the searching. The chapter frames the agent as the thing that finds the right answer fast across tabs, runbooks, and prior incidents, instead of the thing that pushes the fix.

6. Dashboards tell you something is broken. AI is positioned as the layer that tells you why, by reading the tickets and the user feedback that the dashboards do not capture.

7. The framing that stuck with me most: AI does not reduce SRE workload, it raises the reliability ceiling. Cheaper reliability does not mean less work, it means higher reliability demanded across more services. Jevon's paradox applied to ops.

What I would add as a practitioner: the 5-level maturity model they propose is useful, but the gating criteria between levels is where the real engineering lives. "Agent suggested 50 fixes, 47 were good" sounds great until you ask which 3 were wrong and what they would have broken. Most teams I see skipping straight to autonomous remediation are not doing that work.

Worth a read if you are scoping AI in operations in the next year.

(Disclosure: I run Sherlocks, which builds in this space. This is not a pitch for it.)

FT reported this week that Amazon had a 13-hour AWS outage after an AI coding tool decided, autonomously, to delete and recreate an infrastructure environment. No human caught it in time.

Their SVP sent an all-hands. Senior sign-off now required on AI-assisted changes.

Where do you actually draw the approval gate? We landed on requiring human sign-off before the AI executes anything with real blast radius, not because it's the safe/boring answer, but because we kept asking "what's the failure mode if this is wrong?" and the answers got uncomfortable fast. That feels right.

What I don't have a clean answer to yet: how do you make that gate fast enough to not become the new? If the human-in-the-loop step just becomes another queue, you've traded one problem for another.

Who's you letting AI agents execute infra changes autonomously, or is everything still human-approved? Where would or are you drawing the line?

Article: https://www.ft.com/content/7cab4ec7-4712-4137-b602-119a44f771de
Interesting post on X: https://x.com/AnishA_Moonka/status/2031434445102989379

https://github.com/nklmilojevic/sofka/

A reimagining of k9s with one generic object pipeline instead of a renderer per resource kind - so every CRD works on day one, Flux CD is built in, and the async UI never blocks on the cluster.

License: MIT/Apache 2

Opus 5.5 has been giving me some excellent suggestions and it's really helping me link together theory and my observability stack as I build it at work. I just asked it to score what we have based on each chapter of the new 2nd edition SRE book and then provide suggestions to move that score up and the advice is very solid.

SRE interviews are getting on my nerves now.Somehow I am supposed to learn AWS and GCP and Terraform and CI/CD and k8s and leetcode in python or golang and architecture and observability and gitops and mlops and keda and kustomize and Thanos and cryptography and processes setups and then focus on culture and stakeholder management.

All while I am told no to lookup syntax and then being told that Change Management is a business lingo phrase and you are a 2nd tier engineer and hence you cannot push the teams to make changes for supporting reliability.

Is this even worth it anymore? I am interviewing actively and being told how “culture doesnt matter” and how the sre team should take over the operational charge of the systems, accountability without authority.

Are sre here really keeping all this information on their finger tips or do you understand the concepts well but lean on googling stuff when required?

I am seriously considering getting out of the ecosystem entirely. I cant tell if I am an idiot or the industry is that problematic.

Edit:

I have 9 yoe primarily in SRE.

Here are some of the experiences I have had:

First: I am discussing how I setup preview environments and how they could lower issues in production but at a cost of infra and such, I gave the design around the pipeline, the gitops setup and the environment promotion setups. Only to be rejected because I couldn’t mention the exact syntax for doing it in github actions.

Second:- Talked about how setting up observability is one the first tasks I pick when setting up a SRE function. It’s mostly non intrusive, and gets quick results and the executive buy in for more projects like infra automation. Laid down the setups for the infra monitoring,Thanos,LGTM setup, golden paths and alerts and escalation matrix. Only to be told that the SRE function should begin by writing instrumentation libs for 200+ devs as a single SRE.

Third:- Coding: tell me n letter palindromic substring from a given string. This one i did feel bad about , but honestly I still don’t understand how that going to help me setting up a release process.

Fourth: Change Management ,what?. Turns out its a business lingo for a team which spends everyday yelling at each other asking what changed yesterday.

Fifth: Dont care about your influence in the engineering culture as a Staff SRE. Why are you not leading a team? . Doesn’t matter how RACI solved friction between the pillars and broke down silos stopping growth.

and many more I can count.

I can design systems and processes but getting rejected just because you can’t tell whats the best AWS service to achieve something or you haven’t lead a k8s upgrade just sounds weird.

Howdy, r/sre!

Recently I announced my new blog series on "52 Weeks of SRE", where each week I'll go in-depth on a different SRE concept. The reception was amazing here, and I was excited to work no this next topic, one which I work with daily: Monitoring.

Check out the post on Monitoring Fundamentals here: https://jpereira.me/week-2-monitoring-fundamentals/

There is also a companion blog post where I go in-depth on deploying a monitoring stack with docker, and apply the best-practices taught in Monitoring Fundamentals to instrument a microservice and create dashboards and alerts in Grafana. Check it out here: https://jpereira.me/building-and-deploying-a-robust-monitoring-solution-for-your-applications/

Stay tuned for next week where I'll be talking about Service Level Objectives!

Thank you for the amazing reception on this series so far, and as always any feedback is much appreciated :)