Chapter 1 · agent-neuron · Lab

Agreement

Forty invented alerts and three cheap detectors. One expensive call, and it waits until two of them agree.

The storm

Forty alerts arrive for five invented services, all in under six seconds. That’s 40 alerts. The last lands at 5555 ms.

Three cheap detectors look at every one. Two are shell scripts: one scans log lines, one checks a list of recent deploys. The third, a few lines of JavaScript, checks error rates, and only for critical alerts. Flag the alert or stay quiet? Each one decides for itself.

The expensive part sits behind them, and it only runs once two different detectors have flagged the same alert. In the recording it’s a stand-in, a fixed 500 ms wait where a model call would go. Its result goes to a diagnosis neuron: a whole small network sealed inside one neuron. It answers with one verdict.

And one more rule. A newer alert for the same service makes the older work for it stale. The flag still waiting gets replaced. The call already running gets cut.

Clock5.58 s of 5.58 s

At 5.58 s: 40 alerts, 52 flags, 17 calls started, 7 finished, 10 cut, 0 running, 0 flags waiting, 11 replaced, 7 expired, 7 diagnoses.

Status board

One row per service: its alerts so far, the latest alert, the flags waiting at hypothesis, any call running for it and its diagnoses.
Service	Alerts	Latest alert	Flags waiting	Call	Diagnoses
gateway	8	#37 info: request queue depth growing	none	none	2
search	9	#39 info: query latency above target	none	none	0
checkout	9	#40 info: cart submit failing for some users	none	none	3
payments	7	#38 info: payment retries piling up	none	none	2
inventory	7	#36 info: warehouse feed delayed	none	none	0

Network

Neuron: hypothesis

Kind: model stand-in
Description: a fixed 500 ms wait standing in for a model; acts once enough detectors agree
Options: threshold 2, within 500
Calls started: 17
Finished: 7
Cut short while running: 10
Waiting flags replaced: 11
Waiting flags expired: 7

Timeline

alerts

logs

deploys

metrics

hypothesis

diagnosis

signal
answer
quiet
drop

5.58 s deploys says nothing about alert 40
5.58 s logs says nothing about alert 40
5.58 s metrics says nothing about alert 40
5.56 s alerts sends alert 40 (checkout)
5.14 s deploys says nothing about alert 39
5.14 s logs says nothing about alert 39

Recording, invented dataFigure 1. One run of the invented alert storm, recorded and played back. Pick a recording, then play, pause or drag the clock. Nothing runs here: every view is worked out from the recorded events.

Agreement is the rule

The rule itself is small. One detector’s flag waits, and if a second detector flags the same alert within 500 ms, the call starts. Otherwise the flag gives up, unless a newer alert for the same service replaces it first.

Across forty alerts, here’s what it did.

Clock5.58 s of 5.58 s

At 5.58 s: 40 alerts, 52 flags, 17 calls started, 7 finished, 10 cut, 0 running, 0 flags waiting, 11 replaced, 7 expired, 7 diagnoses.

not arrived yet
no detector flagged it
a flag is waiting
a call is running
its call finished
its call was cut short
its flag was replaced by a newer alert
its flag expired

No detector flagged: 5 alerts

One detector flagged: 18 alerts

Two detectors flagged: 17 alerts

Three detectors flagged: 0 alerts

None.

Pick an alert to see what happened to it.

Recording, invented dataFigure 2. Each invented alert, by how many detectors flagged it; the recording you pick shows what happened to it.

The honest economics

The demo’s own summary counts it as calls started against alerts. At threshold 2 that’s 17 calls for 40 alerts. So 23 calls never started.

That isn’t 17 useful calls, though. 10 of them were cut short: a newer alert for the same service arrived while they ran. Only 7 finished. Each became a diagnosis.

In this recording a call cut short still counts as spent. The work started, and nobody used the answer: that’s the demo’s accounting rule. Time is the other way to count it. The cut calls ran 1521 ms of the stand-in’s 5028 ms.

Clock5.58 s of 5.58 s

At 5.58 s: 40 alerts, 52 flags, 17 calls started, 7 finished, 10 cut, 0 running, 0 flags waiting, 11 replaced, 7 expired, 7 diagnoses.

gateway · 5 calls

search · 0 calls

checkout · 8 calls

payments · 4 calls

inventory · 0 calls

finished cut running▏ an alert for the service

The calls so far
Call	Service	Alert	Agreed	Started	Outcome	Ended
1	gateway	1	deploys and metrics	0.03 s	cut	0.18 s
2	gateway	2	deploys and metrics	0.18 s	cut	0.33 s
3	gateway	3	deploys and metrics	0.33 s	finished	0.83 s
4	checkout	5	logs and deploys	0.55 s	cut	0.70 s
5	checkout	7	logs and deploys	0.70 s	finished	1.20 s
6	payments	9	metrics and logs	0.93 s	cut	1.09 s
7	payments	10	metrics and logs	1.09 s	finished	1.59 s
8	checkout	16	logs and deploys	1.99 s	cut	2.14 s
9	checkout	17	deploys and logs	2.14 s	cut	2.29 s
10	checkout	19	logs and deploys	2.29 s	finished	2.79 s
11	gateway	24	metrics and deploys	3.04 s	cut	3.19 s
12	gateway	26	metrics and deploys	3.19 s	finished	3.69 s
13	payments	29	logs and metrics	3.78 s	cut	3.93 s
14	payments	30	metrics and logs	3.94 s	finished	4.44 s
15	checkout	33	deploys and logs	4.23 s	cut	4.39 s
16	checkout	34	logs and deploys	4.39 s	cut	4.54 s
17	checkout	35	deploys and logs	4.54 s	finished	5.04 s

Recording, invented dataFigure 3. Every call the expensive neuron started, beside the invented alerts for its service, drawn from start to finish or cut.

Three recordings, side by side

Recording

Threshold 1Threshold 2Threshold 3

Each recording is a separate run at its own threshold.

Calls, split by outcome, against the alerts	✓ finished ✕ cut, grey: alerts	✓ finished ✕ cut, grey: alerts	✓ finished ✕ cut, grey: alerts
Alerts	40	40	40
Flags	52	52	52
Calls started	52	17	0
Finished	21	7	0
Cut short as stale	31	10	0
Waiting flags replaced	0	11	31
Waiting flags expired	0	7	21
Diagnoses	16	7	0
Inputs dropped at the sealed neuron	5	0	0
Calls minus alerts	+12	−23	−40
Final verdict	root gateway, deploy dep-8b21, 14 min (call 52, alert 35, deploys)	root gateway, deploy dep-8b21, 14 min (call 17, alert 35, deploys + logs)	none
Events, last at (ms)	306, 5583	261, 5585	264, 5592

Recording, invented dataFigure 4. The same forty invented alerts, recorded three times, once at each threshold. Separate runs, so the times differ by a few milliseconds.

At threshold 1 every flag starts a call. So an alert two detectors agree on gets two calls. That’s 52 calls for 40 alerts. More calls than alerts, and 31 of them cut.

At threshold 3 nothing starts at all. Not because three is strict in general: in this data, no alert was ever flagged by all three detectors.

Threshold 2 gets threshold 1’s verdict with 17 calls, not 52. In this recording, anyway.

Limits and counter-evidence

It’s a recording. Every figure plays back one run on one machine. The demo is timing-bound: alerts for one service arrive either close together or far apart, and a detector script has to answer quickly. Run it again and the times move by a few milliseconds. Take t = 2000 ms. The threshold 2 recording has counted 24 flags there, the threshold 3 recording 22. The only reason: two flags landed at 1993 ms in one and 2002 ms in the other.
The waste is built in: 10 of 17 calls were cut, and every one was an agreeing alert followed inside its burst by a newer alert for the same service, while the stand-in takes 500 ms. Change the burst spacing or the call length and that share changes. The figure shows a mechanism, not a rate.
The recording counts a cut call as spent. That’s an accounting choice, and the project lists it as an open question. What a real model provider charges for a cancelled call, this page doesn’t know.
Agreement went deaf to two services. Search is flagged only by deploys and inventory is flagged only by logs, so at threshold 2 neither ever reached the expensive call: 14 lone flags, no calls. The invented fault sits where two detectors can see it, so here it didn’t matter. In a new kind of failure it could be exactly the one that matters.
The verdict agrees by construction. In the invented dependency map every service depends on gateway, so every diagnosis says gateway, deploy dep-8b21, 14 minutes. The diagnosis neuron just looks the answer up. That shows sealing and one verdict out, not a network finding a cause.
No alert had three flags, and metrics only checks critical alerts. So threshold 3’s zero is the data talking. A different detector set would give a different answer.
The pattern isn’t new. Independent specialists that each see every input and decide for themselves when to act are close to a blackboard system [Hearsay-II], and to content-based publish/subscribe [The Many Faces of Publish/Subscribe]. Alert managers already group alert storms [Prometheus Alertmanager]. What I’m testing is narrower: whether the rule (how many must agree, how long to wait, what makes work stale) belongs on each neuron’s wrapper, not in a central controller.
The concept has connections that strengthen and weaken, but none of that is built, and no recording sets a strength. Every detector counts the same. Nothing learns yet.

Sources

Hearsay-II The Hearsay-II Speech-Understanding System: Integrating Knowledge to Resolve Uncertainty. Computing Surveys 12(2), June 1980, pp. 213 to 253, published version; read 2026-11-04.
Independent knowledge sources read and write a shared blackboard; a monitor notices changes to it and schedules the knowledge sources interested in them; a scheduler runs the waiting action with the highest priority.
- The monitor and the scheduler (p. 221 and p. 222, Figure 4 and its text)
The Many Faces of Publish/Subscribe The Many Faces of Publish/Subscribe. ACM Computing Surveys 35(2), June 2003, pp. 114 to 131, published version; read 2026-11-04.
Publish/subscribe decouples the communicating parties in time, space and synchronization; the topic-based and content-based schemes; content-based subscription works on the content of the events themselves.
- Abstract, p. 114; Table 1 and §4.2, content-based, p. 121
Prometheus Alertmanager Alertmanager. The project’s own documentation; read 2026-11-04.
Grouping puts alerts of a similar nature into one notification, and inhibition suppresses some alerts while certain others are firing. Used only to say that alert storms already have mainstream tools.
- Grouping and Inhibition

Each claim is paraphrased from the page named, read on the date shown. Everything else on these pages is my own concept and my own recordings of invented data, which are not sources. Where a sentence is my own reading, it is marked “My view”.