Heisenbugs: The Case of the Visibly Invisible Rogue Virtual Machine
A Heisenbug is defined by frustrated programmers trying to mitigate a defect as:
A bug that disappears or alters its
behavior when one
attempts to probe or isolate it.
In the case of a hardened rogue virtual machine (VM) sitting somewhere on your network, trying to probe or isolate it yields a similar frustration index for the IDS analyst as compared to that of the pissed off code jockey unable to locate a bug in a trace.
In many cases, simply nuking it off the network is not good enough. You want to know where it is (logically and physically,) how it got there, and whose it is.
Here’s the scenario I was reminded of last week when discussing a nifty experience I had in this regard. It’s dumbed down and wouldn’t pass a
Here’s what transpired for about an hour or so one Monday morning:
1) IDP console starts barfing about an unallocated RFC address space emergence being used by a host on an internal network segment. Traffic not malicious, but it seems to be talking to the Internet, some DNS on (actual name) "attacker.com" domain…
2) We see the same address popping up on the external firewall rulesets in the drop rule.
3) We start to work backwards from the sensor on the beaconing segment as well as the perimeter firewall.
4) Ping it. No response.
5) Traceroute it. Stops at the local default with nowhere to go since the address space is not routed.
6) Look in CAM tables for interfaces usage in the switch(es). Coming through the trunk uplink ports.
7) Traced it to a switch. Isolate the MAC address and isolate based on something unique? Ummm…
8) On a segment with a collection of 75+ hosts with workgroup hubs…can’t find the damned thing. IDP console still barfing.
9) One of the security team comes back from lunch. Sits down near us and logs in. Reboots a PC.
10) IDP alerts go dead. All eyes on Cubicle #3.
…turns out that he was working @ home the previous night on his laptop upon which he uses (on his home LAN) VMWare for security research to test for how our production systems will react under attack. He was using the NAT function and was spoofing the MAC as part of one of his tests. The machine was talking to Windowsupdate and his local DNS hosts on the (now) imaginary network.
He bought lunch for the rest of us that day and was reminded that while he was authorized to use such tools based upon policy and job function, he shouldn’t use them on machines plugged into the internal network…or at least turn VMWare off ;(
/Hoff
Recent Comments