r/Observability • • 3h ago

You may not need to retain everything just in case you need it. Keep a lightweight historical record of your Linux infrastructure and investigate what actually happened when things go wrong.

Thumbnail
github.com
2 Upvotes

Low-overhead, retrospective Linux observability

clifr (command line flight recorder) and has two parts:

  • clifrd runs as a service and records processes, process starts and exits (through eBPF, so a process that lives 5 ms still shows up), TCP connections with the process that owns them, files held open, mounts, kernel messages and the usual system counters. It keeps 72 hours in /var/lib/clifr, 2 GB max.

  • clifr rebuilds the output of the normal commands for a past moment or a time range. It doesn't run the command, it rebuilds the output from the recording.clifr --at "02:45" "ps aux" clifr --since "02:00" --until "03:00" "ss -tanp dst 10.0.4.21 and dport = :5432" clifr --since "1 hour ago" events # starts, exits, OOM kills, mounts clifr diff --at "14:00" --at "14:20" "ss -ltnp"

There are about 40 commands so far: ps, top, pidstat, lsof, ss, netstat, free, vmstat, iostat, mpstat, df, ip, dmesg, systemctl list-units, lsblk, and more and all are 100% accurate.

The first thing people asked last time was the cost, so here are the numbers:

Quiet serverMixed loadBusy server
CPU0.35% of one core0.5-0.6% of one core1.9% of one core
RAM~90 MiB~80-120 MiB~290 MiB (335 max)
Disk~`20 MiB/day~100 MiB/day~4 GiB/day
Networknone, no ports opennonenone

Disk use depends heavily on how busy the server is.

clifr is not a monitoring tool and has no alerts. It's not a security audit tool either: it doesn't record file reads or writes, and root can stop it. It's for the "what was going on at 02:45" question, after the fact.

Status: it's a prototype, version 0.1.0. It hasn't run on a real production server yet (a friend starts one this week), and all the load has been synthetic. AGPL-3.0, no telemetry, nothing leaves the box.

So if you have a test or dev box (RHEL 8/9, Rocky, Alma), I'd really appreciate it if you tried it and told me what breaks or what looks wrong. The most useful thing for me is comparing what clifr shows with what you know was really happening.

I appreciate all your comments even if they are harsh


r/Observability • • 1d ago

How to calculate the true cost of enterprise AI agents: why cost per completed task beats cost per token, and what an AI observability platform has to capture

1 Upvotes

Token price on a model's rate card is a poor guide to what enterprise AI agents really cost in production. An agent plans, calls tools, re-reads a growing context and sometimes retries, so spend grows faster than the number of steps, and tasks that fail still cost money. Cost per token and cost per request both count calls rather than results, so neither can tell a cheap success from an expensive failure.

For AI agent cost, the unit that has worked better for me is cost per completed task:

```

(model tokens + retries and failed attempts + tool and API calls

+ infrastructure + evaluation and telemetry overhead + human review)

÷ tasks that actually completed

```

Failed and abandoned attempts stay in the numerator, and only completed tasks count in the denominator. That asymmetry is deliberate: it makes an unreliable agent look as expensive as it really is. The catch is that you have to define what "completed" means for each task type (a resolved support case, a finished analysis) before calculating anything, otherwise every later number is arguable.

The components sit in different systems (model provider invoice, tool and API invoices, infrastructure, evaluation jobs, reviewer time), so the calculation only holds together if your AI observability data gives you one common record. At minimum, that means:

- step-level token usage linked by trace to the task that triggered it, so retries and loops show up as part of one task

- an outcome label for every task: completed, failed, or escalated to a person

- tool and API usage with a price attached

- attribution by task type, agent, model and team

- the overhead of observability and evaluation itself

Two more things worth watching. First, look at the median together with the high percentiles, split by agent and task type, because a few runaway tasks (loops, very long contexts) can dominate spend while the average looks fine. Second, read cost next to completion rate and output quality, since a cheaper model that completes fewer tasks can actually raise the cost per completed task.

For those running AI agents in production: how do you define the unit of work? Do you attribute cost per task, per agent or per team? And do you include evaluation, telemetry and human review in the number, or only model and tool spend?

Disclosure: I work at Bonree, an observability vendor. This post names no products.


r/Observability • • 1d ago

How to calculate the true cost of enterprise AI agents: why cost per completed task beats cost per token, and what an AI observability platform has to capture

Post image
1 Upvotes

Token price on a model's rate card is a poor guide to what enterprise AI agents really cost in production. An agent plans, calls tools, re-reads a growing context and sometimes retries, so spend grows faster than the number of steps, and tasks that fail still cost money. Cost per token and cost per request both count calls rather than results, so neither can tell a cheap success from an expensive failure.

For AI agent cost, the unit that has worked better for me is cost per completed task:

```

(model tokens + retries and failed attempts + tool and API calls

+ infrastructure + evaluation and telemetry overhead + human review)

÷ tasks that actually completed

```

Failed and abandoned attempts stay in the numerator, and only completed tasks count in the denominator. That asymmetry is deliberate: it makes an unreliable agent look as expensive as it really is. The catch is that you have to define what "completed" means for each task type (a resolved support case, a finished analysis) before calculating anything, otherwise every later number is arguable.

The components sit in different systems (model provider invoice, tool and API invoices, infrastructure, evaluation jobs, reviewer time), so the calculation only holds together if your AI observability data gives you one common record. At minimum, that means:

- step-level token usage linked by trace to the task that triggered it, so retries and loops show up as part of one task

- an outcome label for every task: completed, failed, or escalated to a person

- tool and API usage with a price attached

- attribution by task type, agent, model and team

- the overhead of observability and evaluation itself

Two more things worth watching. First, look at the median together with the high percentiles, split by agent and task type, because a few runaway tasks (loops, very long contexts) can dominate spend while the average looks fine. Second, read cost next to completion rate and output quality, since a cheaper model that completes fewer tasks can actually raise the cost per completed task.

For those running AI agents in production: how do you define the unit of work? Do you attribute cost per task, per agent or per team? And do you include evaluation, telemetry and human review in the number, or only model and tool spend?

Disclosure: I work at Bonree, an observability vendor. This post names no products.


r/Observability • • 1d ago

Knight Capital lost ~$440M in 45 minutes with the alerts firing. I think the failure was decision rights, not telemetry. Numbers and caveats from a chapter I just finished.

Post image
0 Upvotes

r/Observability • • 1d ago

OSS Backstage Temporal Plugin

Thumbnail gallery
1 Upvotes

We’ve expanded our AI governance suite with a new plugin that allows teams to monitor Temporal Technologies workflows right within Spotify Backstage.
Combine it with backstage-plugin-litellm-govai for a complete governance and execution view:
https://github.com/acarmisc/backstage-plugin-temporal

https://github.com/acarmisc/backstage-plugin-litellm-govai

Contributions are welcome, both issues or PRs ;)


r/Observability • • 1d ago

Cloudflare's analytics dashboard wasn't keeping up with my site metrics. So I built an atlas for my 30 day site traffic.

Thumbnail
reddit.com
1 Upvotes

r/Observability • • 2d ago

rrweb, the open-source session replay library, just passed half a billion npm downloads. Thank you!

6 Upvotes

Hi from the rrweb team 👋

We just passed half a billion npm downloads, and we want to thank everyone here! A big thank you to everyone who has used rrweb in their products and built on top of it.

A few more stats from rrweb recently:

  • 📥 542.4M npm downloads in the last 12 months

  • 🌍 BuiltWith detected rrweb on 790K live websites, including 11% of the world's top 1,000 (it only sees scripts a page loads directly, so the real number is higher)

  • ⭐ 20K+ GitHub stars

  • 🚀 2.0 launched in June after nearly four years of work

  • 📦 2× as many releases this year as in 2024 and 2025 combined

For those of you who don't know rrweb: the name stands for "record and replay the web". It's an open-source library that records the browser state and user interactions on a web page as structured events, then replays them so you can see exactly what happened.

If you're interested, here's the blog post: https://rrweb.com/blog/half-a-billion-downloads

One personal ask: we're doing our best to make working on rrweb full time sustainable. What are you really missing in the library?


r/Observability • • 2d ago

Five security releases shipped last week across

Post image
2 Upvotes

r/Observability • • 3d ago

Meet Scout — HawkEye's New Mascot! 🦅

Post image
0 Upvotes

Hey everyone!

As we continue building HawkEye, we wanted to give the platform a little personality.

Meet Scout, our new mascot!

We wanted something that represented what HawkEye is all about: keeping a watchful eye on your websites and APIs, catching problems early, and making monitoring less of a headache.

Scout seemed like the perfect fit.

We'll be incorporating him into our website, dashboards, and notifications over time.

What do you think? Would you like to see Scout make appearances throughout the app?


r/Observability • • 3d ago

Metrics, logs and alerts for my 3-node Talos homelab

4 Upvotes

My 3-node Talos homelab had no monitoring, so I only found out something was wrong when an app stopped working.

So I added VictoriaMetrics for metrics and VictoriaLogs for logs from every node and pod. Both show up in Grafana, and alerts go to my phone on Telegram.

Healthchecks[.]io also watches the alerting itself, so I hear about it if the alerts stop. The whole stack uses about 1.3 GiB of RAM across the three nodes.

Repo: https://github.com/prateekjaindev/homelab

Write-up: https://blog.prateekjain.dev/metrics-logs-and-alerts-for-a-3-node-talos-homelab-with-victoriametrics-0f4a91b9d44e?sk=751eedebb5d54cc15561d59abb7e52d8


r/Observability • • 3d ago

Reworking span names for an on-call walkthrough

17 Upvotes

I'm putting together an onboarding walkthrough for our document-processing service and picked one request to read without opening the code.

In Braintrust, the trace shows document lookup, model extraction, and validation as separate spans. The sequence is there, but a few names still come straight from helper functions. `process_result` makes sense if you know the implementation. It doesn't tell the next on-call engineer that we're validating fields returned by the model.

Considering operation names like `lookup_document`, `extract_fields`, and `validate_extracted_fields`, with function names kept as attributes. Looking at the trace this way also made me question whether some of our smaller helper spans are making the sequence harder to scan.


r/Observability • • 4d ago

What finally moved your longest incident call forward?

Thumbnail reddit.com
1 Upvotes

r/Observability • • 4d ago

Got laid off recently, looking for referalls/ jobs

0 Upvotes

Hi Guys,

I have about 2.5 years of experience with Dynatrace and automations using python. I am looking for SRE/ Observability jobs in India. Could you please provide any relevant referrals, leads, or potential openings?

Apart from dynatrace, I also have some experience with AlOps, Change Analytics and Callout (Major incident) tools.


r/Observability • • 4d ago

High Cardinality: I acked a page at 3am and woke up in an observability stack where the three pillars want to date me

Thumbnail
highcard.love
6 Upvotes

A little isekai dating sim where you get paged into another world and have to date Logs, Metrics and Traces.

Every choice you make gets logged as telemetry into chDB WASM running in your browser tab. Each love interest only sees their own projection of your data, which is why none of them really get you. You can unlock a SQL console and query your own love life to find secret routes.

There may or may not be a polycule ending.


r/Observability • • 4d ago

Observability is not plug and play

4 Upvotes

"What is the best observability system?"

This is a frequently asked question.

The obvious answer is "it depends". There is also the more fundamental point - that observability is not plug and play. No matter what system you procure there is a lot of upfront planning and ongoing input required to make it work.

https://observability-360.com/article/ViewArticle?id=observability-is-not-plug-and-play


r/Observability • • 4d ago

Exporter of software vulnerabilities

Thumbnail
reddit.com
1 Upvotes

r/Observability • • 4d ago

If Datadog Is So Expensive, Why Is It Barely Profitable?

Thumbnail
telemetrymachine.com
44 Upvotes

r/Observability • • 5d ago

What evidence should an operations agent produce before you act on its root cause analysis?

2 Upvotes

AI agents are increasingly used for incident investigation. In most demonstrations the output is a single conclusion, such as “the root cause is X”. A conclusion without its basis is of limited use during an incident: the on-call engineer must repeat the investigation to decide whether to trust it, and the time saved is lost.

My working position is that an operations agent should be required to return an evidence chain together with each conclusion, with four parts:

1.  Topology. The dependency path the agent considered and any recent changes on it.

2.  Metrics. Which signals moved, and when relative to the first symptom.

3.  Errors and logs. Which failures appeared first, and on which path.

4.  A hypothesis referencing all three, so that each link can be verified individually.

(The image shows the structure. The example in it is fictional.)

From this position I draw three practical conclusions, which I would like to test against your experience:

● Evidence is a precondition for action, not an optional extra. A conclusion that cannot be traced to data should be treated as a hypothesis, regardless of how confident it sounds.

● Autonomy should be tiered by reversibility and blast radius, not by model confidence. Reading data and attaching diagnostics can be autonomous. Changing configuration or restarting shared services should require approval, with the evidence presented in the approval request.

● Agents should be allowed to report uncertainty. “Two candidate causes, evidence for each” is a more useful output than a single confident answer.

Known weaknesses of this approach: it depends on topology and telemetry being complete, so gaps in either skew the chain without being visible; and an evidence chain can be assembled to support a wrong hypothesis as easily as a right one, so it makes errors checkable rather than impossible.

Questions for people who carry a pager:

● Which part of an evidence chain do you check first, and which part do you not trust?

● Would you permit an agent to execute a remediation, and under which approval conditions?

● Where have AI-assisted RCA tools failed you in practice: false confidence, hidden assumptions, or missing data?

Disclosure: I work at Bonree, an observability vendor. This post is about the design question, not a product.


r/Observability • • 5d ago

What evidence should an operations agent produce before you act on its root cause analysis?

Post image
1 Upvotes

AI agents are increasingly used for incident investigation. In most demonstrations the output is a single conclusion, such as “the root cause is X”. A conclusion without its basis is of limited use during an incident: the on-call engineer must repeat the investigation to decide whether to trust it, and the time saved is lost.

My working position is that an operations agent should be required to return an evidence chain together with each conclusion, with four parts:

1.  Topology. The dependency path the agent considered and any recent changes on it.

2.  Metrics. Which signals moved, and when relative to the first symptom.

3.  Errors and logs. Which failures appeared first, and on which path.

4.  A hypothesis referencing all three, so that each link can be verified individually.

(The image shows the structure. The example in it is fictional.)

From this position I draw three practical conclusions, which I would like to test against your experience:

● Evidence is a precondition for action, not an optional extra. A conclusion that cannot be traced to data should be treated as a hypothesis, regardless of how confident it sounds.

● Autonomy should be tiered by reversibility and blast radius, not by model confidence. Reading data and attaching diagnostics can be autonomous. Changing configuration or restarting shared services should require approval, with the evidence presented in the approval request.

● Agents should be allowed to report uncertainty. “Two candidate causes, evidence for each” is a more useful output than a single confident answer.

Known weaknesses of this approach: it depends on topology and telemetry being complete, so gaps in either skew the chain without being visible; and an evidence chain can be assembled to support a wrong hypothesis as easily as a right one, so it makes errors checkable rather than impossible.

Questions for people who carry a pager:

● Which part of an evidence chain do you check first, and which part do you not trust?

● Would you permit an agent to execute a remediation, and under which approval conditions?

● Where have AI-assisted RCA tools failed you in practice: false confidence, hidden assumptions, or missing data?

Disclosure: I work at Bonree, an observability vendor. This post is about the design question, not a product.


r/Observability • • 5d ago

An open-source wiki where runbooks show live Prometheus metrics and Loki logs

Thumbnail
gallery
13 Upvotes

Hi r/Observability, I'm David, one of the two developers of Oxynote. My friend Simon and I also maintain ttlcache, a Go caching library with about 1,300 GitHub stars.

I've worked as a software engineer for 8 years, including 2 as an SRE and 3 as a team lead. On every team, the knowledge you needed during an incident was in someone's head or buried in a Slack thread. The runbooks we did have were stale, and the dashboard they linked to had often been renamed or deleted.

So we built Oxynote, a self-hosted wiki for engineering teams. A page can hold a live chart that runs its own PromQL or SQL query each time someone opens it. Since v0.10 this week, it can also show live Loki logs, filtered with LogQL. You write what the numbers mean right next to them, so the runbook for "checkout latency is high" has the latency chart, the error logs, and the steps on one page. Changes to important pages go through a draft that a teammate reviews, with a diff of both the text and the queries.

The name comes from oxygen + note: docs that breathe. Live data gives people a reason to open the wiki every day, and docs people open every day stay up to date.

It's Apache-2.0 and runs as a single Docker image on your own server, so your docs and data never leave it. GitHub: https://github.com/oxynote/oxynote

We started it about a year ago and made it public about 3 months ago. I'm happy to answer questions, and I welcome any feedback.


r/Observability • • 5d ago

How do you monitor small Spring Boot apps without running a whole monitoring stack?

Thumbnail
reddit.com
0 Upvotes

I keep running into the same thing with small Spring Boot projects. The app is tiny, but proper monitoring means a collector, a store, dashboards and alert rules, and that setup ends up bigger than the app. So most of my small projects just ran with no monitoring at all, and I only found out something was wrong when someone told me.

How do you handle this for small services? Do you skip it, use Actuator and look at the numbers by hand, or set up the full stack anyway?

I got curious enough that I tried a different approach: a Spring Boot starter that watches the app from inside. One dependency, no agent, no database, and you get a live dashboard with per-route rate, error rate and p50/p95/p99 latency, plus spike and brute-force detection. It only sees its own JVM and has no OpenTelemetry or Prometheus export yet.

Is in-process detection like this a sane default for small services, or does it always end up wanting to live in the backend where you can change rules without a redeploy?

Code: https://github.com/eswar-7116/streamguard


r/Observability • • 5d ago

Observability Has Spent 20 Years Finding Problems. It's Time to Start Fixing Them.

Thumbnail
groundcover.com
0 Upvotes

r/Observability • • 6d ago

Hand instrumenting every path stopped scaling once agents started writing most of our code

0 Upvotes

We did the cost exercise everyone’s doing. Trimmed custom instrumentation, moved a pile of hosts to infra only, saved real money. Held up fine for a while. Then the service we’d thinned out broke and we were reconstructing the whole thing from logs, which is where we were years ago.

The part that bugs me is that the trim wasn’t really the problem. The instrumentation model is. Somebody has to decide ahead of time which path matters, write the span, keep it current. That was already annoying. Now most of our backend diffs come out of a coding agent, and nobody is going back to add spans to code they didn’t write.

So I’m looking at things that don’t ask you to pick in advance. Honeycomb’s wide events approach, and Hud, which runs in process at the function level and which the agent can query over MCP when something regresses. Both are another decision about what leaves prod.

For anyone who’s been through a proper trim: what did you refuse to give up, and did the instrumentation model change, or did you just buy back what you cut?


r/Observability • • 6d ago

Temporary runtime instrumentation for production debugging — useful or risky?

Thumbnail reddit.com
2 Upvotes