When the Agent Should Wake You Up
When the Agent Should Wake You Up
An agent that tells you everything is a log file with a notification sound. An agent that tells you nothing is a liability. The interesting work is in between, and almost nobody designs it on purpose. It accumulates: someone gets surprised by a failure, adds an alert, and the alert stays forever because removing one feels like inviting the surprise back.
The result is a channel nobody reads. Not because the people are careless, but because they learned, correctly, that the channel is mostly noise. Once that lesson is learned it applies to the one message that mattered too.
The asymmetry nobody prices
The usual argument for alerting on everything is that a missed incident costs more than a false alarm. That is true for any single comparison and wrong in aggregate.
A false alarm does not cost one interruption. It costs a small amount of trust in every future alert from the same source. Spend that enough times and you have built a system that pages reliably into a void. The missed incident you were trying to prevent now arrives through a channel that has been trained into irrelevance, and it looks exactly like the eleven things last week that did not matter.
So the real comparison is not "one false alarm versus one missed incident." It is "this alert, plus its effect on the credibility of every alert after it." An alert that fires often and resolves itself is not cautious. It is actively eroding the thing you will need later.
Three questions before anything pages a human
A condition earns an interruption when all three of these hold. Most conditions fail at least one.
Is it real? Confirmed by an independent check, not just claimed by whatever raised it. An error in a log is a claim. A failing health endpoint checked directly is evidence. Our own rule is that nothing gets escalated until it is verified with a second look, because the walk-back costs more credibility than the delay ever saves.
Is it actionable right now? If the recipient cannot do anything until Monday, it is not a page, it is an item for a list. A disk at 80 percent with two weeks of headroom is a ticket. The same disk with four hours of headroom is a page. Same metric, different question answered.
Will it not fix itself? A single transient error that the retry logic is designed to absorb is the system working. Alerting on it means alerting on normal operation. Wait for the retries to be exhausted, then speak.
What actually deserves the ping
The list is shorter than people expect, and it is mostly about the system's ability to keep working rather than about any individual error.
Something is stuck and will stay stuck: retries exhausted, a queue that stopped draining, a job that has not completed in several expected cycles. Something crossed a threshold that is irreversible or expensive: money moving, data being deleted, a quota about to be hit. Something needs a decision only a human is allowed to make, which for an agent is the entire permission boundary. And the one everyone forgets: something that should have happened did not.
That last one is the highest-value alert most teams do not have. Monitoring failures means a process that never started reports perfect health. A run that does not happen cannot throw. If a job is supposed to produce an artifact daily, the check that matters is not "did it error" but "is today's artifact here." Absence is a symptom, and it needs a detector of its own because nothing else will generate one.
Severity is a routing decision
Once the list is short, the remaining question is how loud each item should be, and the honest answer is that most of them are not loud at all.
Wake someone up for things that are actively getting worse while nobody is looking. Send an in-hours notification for things that are stable but need a decision today. Put everything else in a digest that gets read when convenient. The digest is not a graveyard, it is the correct destination for the large category of things worth knowing and not worth interrupting for.
The discipline is that the top tier stays small enough to stay meaningful. If everything is urgent, the tiers are decorative and you are back where you started.
Give the human a decision, not a fragment
An alert that says a job failed makes the recipient do the entire investigation before they can act. At 3am, that is most of the response time, spent reconstructing context the system already had.
What was expected, what happened instead, what has already been tried automatically, and what the likely next action is. Four lines. The point is that someone reading it half awake can decide whether this needs them now or can wait, without opening anything. An agent is unusually well positioned to do this, because it has the surrounding state at the moment it decides to escalate. Throwing that away and sending a bare error is wasting the one advantage it has.
Tuning is the job, not the setup
Escalation rules are never right on the first pass, and the failure mode is that nobody revisits them. Two habits keep it honest.
Every alert that fires and turns out not to need a human is a bug in the alert. Not an annoyance to absorb, a defect to fix. Either tighten the condition, add the verification step it was missing, or demote it.
Every incident that surprised you is a missing detector. Ask what observable would have caught it earlier and whether anything is watching that observable now. Usually the answer is that the signal existed and nothing was looking at it.
Both habits push in the same direction: fewer alerts, each one meaning more. The goal is not an empty channel. It is a channel where every message is worth reading, so that the one that matters at 3am is read like it matters.
Escalation ladders, severity routing, and the verification step that keeps an agent from crying wolf are covered in One Agent, One Company ($9.97), including the false alarm that taught us to confirm before we page.