The Threshold That Lived in Two Places
Config drift is boring — right up until your agent and your infrastructure disagree about what "too high" means.
Here's the pattern. You give an agent an operational rule: "alert when queue depth passes 10,000." The agent stores that threshold in its own config, because that's where its instructions live. Meanwhile, the actual monitoring system — the one wired to the pager — has its own copy of the same number. Two sources of truth, born identical, guaranteed to diverge.
At a company we'll call a warehouse-robotics outfit, that divergence took about six weeks. Someone tuned the monitoring threshold up to 15,000 during a seasonal spike and never tuned it back. The agent, still carrying 10,000 in its head, kept quietly evaluating the old rule. Result: the agent flagged "incidents" nobody else could see, and — worse — when the humans finally silenced it as noisy, a real backlog sailed past 15,000 with nobody's copy of the rule doing its job.
The failure isn't the drift itself. Drift is physics; every duplicated value drifts eventually. The failure is designing an agent that holds operational values instead of reading them.
The fix: edit-vs-deploy separation
The rule we landed on: an agent may know where a threshold lives, but it may not own a copy of it. Its config holds a pointer — "the queue-depth alert threshold is defined in the monitoring system, rule X" — not the number itself. When the agent evaluates anything, it reads the live value first.
That sounds like a small implementation detail. It's actually an autonomy-design decision, and it splits into two distinct permissions:
- Read the value — always allowed. The agent should pull live config on every evaluation, never from memory, never from its own notes.
- Change the value — a gated deploy action. If the agent believes the threshold is wrong, it doesn't edit anything. It opens a proposal: current value, suggested value, evidence, blast radius. A human approves; the change lands in the one canonical place.
Edit and deploy are different verbs. Most agent failures in this family come from blurring them — an agent that "remembers" a setting has silently become a second deploy target, one with no changelog, no review, and no rollback.
Why agents make this worse than cron jobs did
An old shell script with a hardcoded threshold at least looked like a hazard. Agents are more dangerous here precisely because they're articulate: an agent will confidently explain its reasoning using the stale number, and the explanation reads as authoritative. The polish hides the drift. You can't spot the wrong value by tone.
So the verification step has to be structural, not vibes-based: any operational value an agent cites in an alert or report should carry its provenance — where it was read from and when. If the agent can't answer "where did this number come from, live, just now," the number doesn't get used.
The takeaway
Audit your agents for owned copies of operational values: thresholds, rate limits, on-call rotations, escalation timeouts. Every one you find is a second deploy target waiting to drift. Replace the copy with a pointer, gate changes behind a human-approved proposal, and make provenance mandatory in anything the agent reports.
One source of truth. The agent reads it; it never quietly becomes it.