Rational Survivability

Update 030712: I’m going to follow this post up with yet another post mortem that includes lessons learned and more details as I can supply them.  I will point out two things:

1. It’s pretty clear that the secondary/tertiary stages of this infestation which led to multiple alerts from my readers is related to the massive WordPress attack you can read about here.  It’s important to note, however, that the first incident (which was chalked up impoperly to a false positive) and a second started with similar symptoms back in late July.  I simply didn’t have the data to correlate.  They were different variants.
2. The support from vendors and the security community has been outstanding.  People with no vested interest in the health of my personal blog have gone out of their way to help, even my hosting provider, Dreamhost (although we got off on a rather rocky footing 😉

I do owe both FireEye (who spotted the original attack) and Dreamhost additional data which I will attempt to retrieve.  I also owe Rich Mogull an apology/explanation regarding why I didn’t immediately take the blog down, risking further infection — I legitimately thought we’d fixed it, but because of the stealth of the malware, I was wrong.  Once I realized I couldn’t contain/isolate it, I did take it down…and then wiped the entire blog/database.

At any rate, thanks for bearing with me though this.  It’s been invaluable to me and I hope you found some value in all of this.

It certainly was interesting and gave me some unique insight into the psychology, behavior, biases and opinions of the community/industry that I didn’t fully appreciate prior.

—

This is an update that I originally included with the post describing the malicious infestation of malware on my WordPress site here.  I’ve split it out for clarity.

The last 12 hours or so have been fun. I’ve had many other folks join in and try to help isolate and eradicate the malware that plagued my WordPress install (read the original post below.)

I was able to determine that the Dreamhost password compromise in January (correlated against logs) was responsible for the (likely) automated injection of malicious PHP code into a plug-in directory that had poor permissions.  This code was BASE64 encoded. It was hard to find.

Further, as was alluded to in my earlier version of this post, the malware itself was adaptive and would only try (based on UA and originating IP) to drop it’s Windows-based trojan executable ONCE by way of a hidden iFrame. Hit it again and you’d never see it.

It was a variant of the Blackhole Exploit kit.

If you ran any up-to-date AV solution (as evidenced by the 6 different brands that people reported,) visiting my site immediately tripped an alert.  I run a Mac and up until today didn’t have such a tool installed. I clearly do now as a detective capability.  This was a silly thing NOT to do as it costs basically nothing to do so these days.

When I made a backup of the entire directory, my VPS hosting provider THEN decided to run a security scan on the directory (serendipity) and notified me via email that it found the malware in the directory 🙁 Thanks.  Great timing.  The funny thing was that all the activity last night and uploaded telemetry must have set something off in Google because only late last night — 30+ days later — did Google flag the site as potentially compromised.  Sigh.

At any rate, I ended up nuking my entire WordPress and mySQL installations and doing a fresh install. I’ve rid myself of almost every plug-in and gone back to a basic theme.  I’ve installed a couple of other detective and preventative tools on the site and will likely end up finally putting the site behind CloudFlare for an additional layer of protection.

Really, I should have done this stuff LONG ago…this was my personal failure.  I owe it to the kindness and attentiveness of those who alerted me to the fact that their AV sensors tripped.

The interesting note is that most of the security pros I know who run Macs and have visited my site in the last 30 days never knew I was infected.  If this were a Mac-targeted malware, perhaps they may have been infected.  The point is that while I’m glad it didn’t/couldn’t infect Mac users, I do care that I could have harmed users with other operating systems.

Further, the “ignorance is bliss” approach is personally alarming to me; without a tool which many security pros sleight as “useless,” I would never have know I was infected.

If anything, it should make you think…