The 3 AM Problem

This is Part 3 of a 5-part series on Cisco EEM. Part 1 covered fundamentals; Part 2 built self-healing applets. A recurring lesson there was that the best automated response is often "capture everything, then escalate" rather than swing a hammer. This part is entirely about that capture — because the failures that never get fixed are the ones nobody can catch in the act.

Here’s the failure mode every network engineer knows and hates. A monitoring alert fires at 03:12. By the time it escalates through the paging system, wakes you, and you’ve fought through the VPN and SSH’d into the box, it’s 03:24. The CPU is back to 4%. The interface counters look clean. The BGP session is Established. Everything is fine.

And it’ll happen again in three nights, and you still won’t have anything to look at.

This is the 3 AM problem: the most damaging failures are often transient, and transient failures destroy their own evidence. Your monitoring told you that something happened. It can’t tell you why, because "why" lived in the process table, the drop counters, the interface queue, and the log ring — all of which have moved on by the time a human arrives. Polling every five minutes, even every one minute, samples the aftermath, not the event.

EEM solves this precisely because it’s on the box, watching in real time, with no human and no network path in the loop. It reacts in the same second the condition appears, and it can freeze exactly the state you’d have killed yourself to capture live. This article is a toolkit for doing that well — the triggers, the capture techniques, the offload, and the sharp edges that will bite you if you’re careless with flash and file handles.

The core pattern: trigger → freeze → timestamp → offload

Every diagnostic-capture applet is a variation on four beats:

1. Trigger the instant the condition appears (SNMP threshold, syslog pattern, counter, track).
2. Freeze a snapshot of the relevant state — the show commands whose output you’d want if you’d been standing there.
3. Timestamp it so multiple events don’t overwrite each other and you can correlate to your other systems.
4. Offload it off the box (or at least to persistent storage) so a reload doesn’t take your evidence with it.

Most people get beats 1 and 2 and stop. Beats 3 and 4 are what separate a capture that helps from a capture you find corrupted, overwritten, or gone after the very reload you were trying to diagnose. We’ll build all four.

Building block: the timestamped filename

Before any trigger, solve the filename problem, because you’ll reuse it everywhere. If every capture writes to flash:capture.txt, the second event overwrites the first, and during a flapping problem you lose everything except the last occurrence — often the least interesting one.

EEM exposes built-in variables you can compose into a unique filename. The event-publish time and the hostname are the two you want:

event manager applet CAPTURE-TEMPLATE
 event none
 action 1.0 cli command "enable"
 action 2.0 set fname "flash:$_event_pub_sec-$_event_pub_msec.txt"
 action 3.0 cli command "show clock | append $fname"
 action 4.0 syslog msg "EEM: capture written to $fname"

$_event_pub_sec and $_event_pub_msec give you the event’s publish time in seconds/milliseconds since epoch — monotonic, unique per event, and directly correlatable to timestamps in your syslog server and SIEM. Compose them into the filename and every capture lands in its own file. (event none here just means "manually triggered"; it’s how you test an applet with event manager run CAPTURE-TEMPLATE before wiring it to a real event.)

On IOS-XE you can also shell out to a more human-readable timestamp, but epoch-based names sort correctly and never collide, which matters more than readability when you’re grepping a flash full of captures at 9 AM.

Applet 1: The CPU spike autopsy

CPU spikes are the archetypal 3 AM transient. The trigger is an SNMP threshold; the value is capturing the process-level detail that tells you which process, driven by what, ate the CPU — detail that’s gone seconds later.

event manager applet CPU-AUTOPSY
 event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.6.1 get-type exact entry-op ge entry-val 85 poll-interval 5 ratelimit 60
 action 1.0 set fname "flash:cpu-$_event_pub_sec.txt"
 action 1.1 syslog priority warnings msg "EEM 3AM: CPU >=85% - autopsy to $fname"
 action 2.0 cli command "enable"
 action 3.0 cli command "term length 0"
 action 4.0 cli command "show clock | append $fname"
 action 5.0 cli command "show processes cpu sorted 5sec | append $fname"
 action 6.0 cli command "show processes cpu history | append $fname"
 action 7.0 cli command "show interfaces summary | append $fname"
 action 8.0 cli command "show ip traffic | append $fname"
 action 9.0 cli command "show logging | append $fname"
 action 10.0 cli command "term length 24"
 action 11.0 syslog priority critical msg "EEM 3AM: CPU autopsy complete - $fname ready for review"

The details that make this actually work:

• show processes cpu sorted 5sec — the 5-second window is deliberate. A process spiking right now shows at the top of the 5-second sort; the 1-minute average would dilute a sharp transient into invisibility. During a spike, this one line is usually the whole answer.
• show processes cpu history — the ASCII history graph shows you the shape of the spike (a spike vs. a step vs. a climb), which tells you a lot about cause.
• term length 0 before the captures and term length 24 after — without this, show commands with long output can hang on a --More-- prompt inside the applet, and your capture stalls. This is one of the most common reasons a capture applet "just doesn’t work." Set the length to 0 for the duration, restore it after.
• show ip traffic and show interfaces summary — because a CPU spike driven by a traffic event (a punt storm, an ACL log flood, an ARP storm) shows its fingerprints here, not in the process list alone.

When you get paged now, the answer is often already sitting in flash: before you’ve finished typing your password.

Applet 2: The interface-drop / error-burst capture

Counters are the cruelest transient. Input drops, CRC errors, output-queue drops — they tick up during a microburst or a marginal link, then sit at a static number that tells you nothing about when or how fast. EEM can trigger on the counter itself and snapshot the rate in the moment.

The cleanest trigger for this is event interface, which watches a named counter and fires on a threshold or rate:

event manager applet DROP-BURST-CAPTURE
 event interface name GigabitEthernet0/1 parameter input_drops entry-op ge entry-val 100 entry-type increment poll-interval 10 ratelimit 30
 action 1.0 set fname "flash:drops-$_event_pub_sec.txt"
 action 1.1 syslog priority warnings msg "EEM 3AM: input_drops burst on Gi0/1 - capturing to $fname"
 action 2.0 cli command "enable"
 action 3.0 cli command "term length 0"
 action 4.0 cli command "show clock | append $fname"
 action 5.0 cli command "show interfaces GigabitEthernet0/1 | append $fname"
 action 6.0 cli command "show interfaces GigabitEthernet0/1 controller | append $fname"
 action 7.0 cli command "show platform hardware ... buffers | append $fname"
 action 8.0 cli command "show policy-map interface GigabitEthernet0/1 | append $fname"
 action 9.0 cli command "term length 24"
 action 10.0 syslog priority notifications msg "EEM 3AM: drop capture complete - $fname"

What’s important here:

• entry-type increment — the applet fires when the counter increases by 100 within the poll interval, not when it reaches 100 in absolute terms. That’s the difference between catching a burst and firing once forever after the counter first crosses a static line. For transient bursts, increment is almost always what you want.
• show policy-map interface — if the drops are QoS output-queue drops, the policy-map counters are where the story is. Capturing them in the moment shows which class overflowed.
• The controller/buffer commands (platform-dependent — adjust for your hardware) catch the physical-layer and buffer-exhaustion cases that a plain show interface summarizes away.

This applet turns "the counter says 4,000 drops and I have no idea when" into "here are five timestamped snapshots showing 100-drop bursts every night at 03:10, correlated with a QoS class that’s undersized."

Applet 3: The pre-reload black box

When a device reloads unexpectedly — crash, watchdog, power event — the single most valuable thing is state from just before it went down. IOS crashinfo helps, but you can supplement it: an applet that captures the crashinfo context on the way back up, while it’s freshest.

Trigger on the reload/restart syslog to capture context immediately at boot:

event manager applet POST-RELOAD-CONTEXT
 event syslog pattern "%SYS-5-RESTART" maxrun 120
 action 1.0 set fname "flash:reload-$_event_pub_sec.txt"
 action 1.1 syslog priority critical msg "EEM 3AM: device restarted - capturing post-reload context to $fname"
 action 2.0 cli command "enable"
 action 3.0 cli command "term length 0"
 action 4.0 cli command "show version | append $fname"
 action 5.0 cli command "show logging onboard | append $fname"
 action 6.0 cli command "dir crashinfo: | append $fname"
 action 7.0 cli command "show processes memory sorted | append $fname"
 action 8.0 cli command "show environment all | append $fname"
 action 9.0 cli command "term length 24"
 action 10.0 syslog priority critical msg "EEM 3AM: post-reload context saved - correlate with crashinfo"

Points that matter:

• maxrun 120 — the maximum runtime for the applet. Post-reload the box may be busy, and some of these commands (especially onboard logging / OBFL) can be slow; give the applet room to finish rather than being killed at the default timeout.
• show logging onboard (OBFL) — onboard failure logging survives reloads and often contains the environmental/hardware breadcrumb (a voltage or temperature excursion) that a software crashinfo won’t show. This is frequently the difference between "unexplained reboot" and "PSU was browning out."
• show environment all captured at boot catches a thermal or power condition that may still be present and about to cause the next reload.

Pair this with your existing crashinfo collection and you’ve got both the software crash context and the environmental context, timestamped and in one place.

Applet 4: Multi-trigger correlation capture

The most sophisticated 3 AM captures fire only when several things are true at once — which slashes false positives and captures genuinely correlated state. EEM supports multiple events in one applet with boolean correlation via event tag and the trigger statement.

Consider: you only care about high CPU if it coincides with a spike in punted traffic (the classic "something is punting to CPU" scenario). Two tagged events, correlated with AND:

event manager applet PUNT-STORM-CAPTURE
 event tag cpu snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.6.1 get-type exact entry-op ge entry-val 80 poll-interval 5
 event tag drops interface name GigabitEthernet0/1 parameter input_drops entry-op ge entry-val 50 entry-type increment poll-interval 5
 trigger
  correlate event cpu and event drops
 action 1.0 set fname "flash:punt-$_event_pub_sec.txt"
 action 1.1 syslog priority critical msg "EEM 3AM: CPU+drops correlated - likely punt storm, capturing $fname"
 action 2.0 cli command "enable"
 action 3.0 cli command "term length 0"
 action 4.0 cli command "show clock | append $fname"
 action 5.0 cli command "show processes cpu sorted 5sec | append $fname"
 action 6.0 cli command "show platform punt-statistics | append $fname"
 action 7.0 cli command "show ip traffic | append $fname"
 action 8.0 cli command "show interfaces GigabitEthernet0/1 | append $fname"
 action 9.0 cli command "term length 24"
 action 10.0 syslog priority critical msg "EEM 3AM: punt storm capture complete - $fname"

The value of correlation: high CPU alone fires on plenty of benign things (a big show tech, an SNMP walk). High CPU and a drop burst together, within the same window, is a much stronger signal of an actual problem — and the capture reflects exactly that coincident state. The trigger/correlate syntax supports and, or, and occurrence counts, so you can express fairly rich conditions without dropping to Tcl. (We’ll push correlation further, and into Tcl, in Part 5.)

Offload: getting evidence off the box before it’s lost

A capture on flash: is vulnerable — a reload can eat it, and flash fills up. For anything you care about, push it to a remote server as part of the applet. Add an offload step after the capture completes:

 action 20.0 cli command "copy flash:$fname scp://netops@192.168.37.50/captures/ " pattern "assword"
 action 21.0 cli command "<scp-password-or-use-key-auth>"
 action 22.0 syslog priority notifications msg "EEM 3AM: capture offloaded to netops server"

Realities to design around:

• Prefer SCP/SFTP with key auth over embedding a password in the applet. On platforms that support it, key-based auth keeps a credential out of running-config (where it’d be readable). Embedding credentials in an applet is a compliance problem in any audited environment.
• The copy command is interactive — the pattern/response pairs handle its prompts. This is fiddly and platform-dependent; test it thoroughly, because a copy that hangs on an unexpected prompt will stall the applet until maxrun.
• If the WAN might be down (the very condition you’re capturing), the offload will fail — which is fine, as long as the local copy persists and you retrieve it later. Don’t let a failed offload prevent the local capture; order the actions so local write always happens first.

Flash hygiene: the applet that captures the captures

Diagnostic applets have a failure mode of their own: they fill flash, and a full flash causes its own outage. Self-defense is a scheduled cleanup applet that prunes old captures so your diagnostics never become the incident:

event manager applet CAPTURE-CLEANUP
 event timer cron cron-entry "30 4 * * *"
 action 1.0 cli command "enable"
 action 2.0 cli command "delete /force /recursive flash:cpu-*.txt"
 action 3.0 cli command "delete /force /recursive flash:drops-*.txt"
 action 4.0 syslog priority notifications msg "EEM: old capture files pruned from flash"

Tune what it deletes and when — and ideally only prune files you’ve confirmed are offloaded. But have a cleanup story from day one. More than one engineer has diagnosed a mysterious device failure only to find the root cause was their own capture applet filling the filesystem.

The 3 AM capture checklist

• Timestamp every filename with $_event_pub_sec so events never overwrite each other.
• term length 0 before captures, restore after — or long show output hangs the applet on --More--.
• Capture before you remediate — if this applet is paired with a self-healing action, freeze state first (the Part 2 lesson).
• Use increment triggers for bursty counters, absolute thresholds for sustained conditions. They catch different things.
• Set maxrun generously on applets that run many show commands or run post-reload, so they aren’t killed mid-capture.
• Offload off-box, local-write-first, key auth not embedded passwords.
• Prune flash on a schedule so your diagnostics don’t become the outage.
• Correlate multiple events when a single trigger is too noisy — coincident conditions are stronger signals and cleaner captures.
• Test with event manager run (via an event none variant) so you validate the capture logic before trusting it to fire at 3 AM.

The payoff

The 3 AM problem is really an evidence problem: the failure announces itself and then destroys the proof. EEM closes that gap because it’s already there, on the box, in the moment — no human, no network path, no delay. Build the trigger tightly, freeze the right show output, timestamp it, get it off the box, and keep flash clean, and you convert "it happened again and I still don’t know why" into a directory of timestamped autopsies that make root cause obvious.

The best part: you diagnose the problem while you sleep. The applet did the hard part — being present at exactly the wrong moment — so you don’t have to be.

---

What’s the transient that haunted you longest before you finally caught it — and what show command turned out to hold the answer? The "I wish I’d been capturing X all along" moments are the ones worth sharing.