Rational Survivability

t’s Freaky Friday.  I’m not inspired to crank out a poem.  Instead, courtesy of BoingBoing, I found this fascinating example of applied cryptography (my apologies, Bruce)…an Internet-enabled Chastity Belt featuring encryption and distributed key management:

From the Italian site promoting this abberation of science:

"The site timelock.rules.it (NoScript didn’t like this site — use at your own risk)
has a program [Timelock, $20] that allows someone to use encryption to
lock themselves up for a set or random amount of time, or even to send
the key to their chastity belt over the internet to a trusted keyholder."

The keyholder has set the Hide Timer option so you have no idea how
much time has been set. You feel the fear and the anxiety, but, with
trembling fingers, you close the lock. Your fate is now entirely in the
hands of your keyholder. Only they know how much time has been set.
Only they know the lockword, which may grant you early release. The
need to touch yourself is already overwhelming but there is nothing you
can do about it. All is as it should be.

Oh my. 

I believe Amrit Williams
beta-tested this and reverse engineered the firmware via JTAG,
connecting it to the ‘Net using SCADA along with visualization and
"input" interfaces thanks to a set of VR goggles, a nintendo power
glove and a Novation AppleCat 300 baud modem that auto-dials "Uncle
Percy’s House of Pain and Panna Cotta"  sending DTMF tones that spell
"STICKY" in morse code.

Maynor notified me that he’d also verified a wireless vulnerability
exists in the software, despite the fact that it has no wireless
interface.  He ordered one, anyway.

I guess I was wrong about how Information Security is dead.  I should have said it’s just become a perverted (yet cryptographically secure) version of itself.

Ten bucks says they use the "Blowfish" algorithm (sorry again, Bruce.) 😉

/Hoff

This isn’t going to be a fancy post with pictures.   It’s not going to be long.  It’s not particularly well thought out, but I need to get it out of my head and written down as tomorrow I plan on beginning a new career. 

I am retiring from the Information Security rat race and moving on to something fulfilling, achievable, impacting and that will make a difference.

Why?

Mogull just posted Information Security’s official eulogy titled "An Optimistically Fatalistic View of The Futility of Security."

He doesn’t know just how right he is.

Sad, though strangely inspiring, it represents the highpoint of a lovely internment ceremony replete with stories of yore, reflections on past digressions, oddly paradoxical and quixotic paramedic analogies, the wafting fragility of the human spirit and our unstoppable yearning to all make a difference.  It made me all weepy inside.   You’ll laugh, you’ll cry.  Before I continue, a public service announcement:

I’ve been instructed to ask that you please send donations in lieu of flowers to Mike Rothman so he can hire someone other than his four year old to produce caricatures of "Security Mike."  Thank you.

However amusing parts of it may have been, Rich has managed to catalyze the single most important thought I’ve had in a long time regarding this topic and I thank him dearly for it.

Along the lines of how Spaf suggested we are solving the wrong problems comes my epiphany that this is to be firmly levied on the wide shoulders of the ill-termed industrial complex and practices we have defined to describe the terminus of some sort of unachievable end-state goal.  Information Security represents  a battle we will never win.

Everyone’s admitted to that, yet we’re to just carry on "doing the best we can" as we "make a difference" and hope for the best?  What a load of pessimistic, nihilist, excuse-making donkey crap.  Again, we know that what we’re doing isn’t solving the problem, but rather than admitting the problems we’re solving aren’t the right ones, we’ll just keep on keeping on?

Describing our efforts, mission, mantra and end-state as "Information Security" or more specifically "Security" has bred this unfaithful housepet we now call an industry that we’re unable to potty train.  It’s going to continue to shit on the carpet no matter how many times we rub it’s nose in it.

This is why I am now boycotting the term "Information Security" or for that matter "Security" period.  I am going to find a way to change the title of my blog and my title at work.

Years ago I dredged up some research that came out of DARPA that focused on Information Assurance and Information Survivability.  It was fantastic stuff and profoundly affected what and how I added value to the organizations I belonged to.  It’s not a particularly new, but it represents a new
way of thinking even though it’s based on theory and practice from many
years ago.

I’ve been preaching about the function without the form.  Thanks to Rich for reminding me of that.

I will henceforth only refer to what I do — and my achievable end-state — using the term Information Survivability.

Information Survivability is defined  as “the capability of a system to fulfill its mission, in a timely manner, in the presence of attacks, failures, or accidents to ensure
that the right people get the right information at the right time.

A survivability approach combines risk management and contingency planning with computer security to protect highly distributed information services and assets in order to sustain mission-critical functions. Survivability expands the view of security from a narrow, technical specialty understood only by security experts to a risk management perspective with participation by the entire organization and stakeholders."

This is what I am referring to.  This is what Spaf is referring to.  This is what the Jericho Forum is referring to.

This is my new mantra. 

Information Security is dead.  Long live Information Survivability.  I’ll be posting all my I.S. references in the next coming days.

Rich, those paramedic skills are going to come in handy.

/Hoff

Despite the consistent heel nipping assertions that all I want to do is have people throw away their firewalls (I don’t,) I think Shrdlu nailed it with a comment posted on Lindstrom’s blog.  I’ll get to that in a second.  Here’s the setup.

Specifically, Pete maintains that Spaf’s comments (see here) are an indicator that security isn’t failing, rather we are — and by design.  We’re simply choosing not to fix the things we ought to fix:

This is a simple one, from Dr. Eugene Spafford’s blog:

We know how to prevent many of our security problems — least privilege,
separation of privilege, minimization, type-safe languages, and the
like. We have over 40 years of experience and research about good
practice in building trustworthy software, but we aren’t using much of
it.

So,
we have resources that are unallocated – we have time, money, and
bodies we could throw at the security problem. We have the know-how and
the tools to reduce the risk. And yet, we aren’t doing it.

If security were "failing" there would be evidence of people either
giving up entirely and reducing their IT investments and resources, or
spending more money on success.

An interesting perspective and one I’m bound to agree with.

Here’s Shrdlu’s comment which I think really nails the reason I am going to continue to press the issue regardless; I think the general apathetic state of the security industry (as Pete suggests also) is the first obstacle to overcome:

Cherchez l’argent, mes amis. Mix in Spaf’s argument with Pete’s and
add Marcus and Bruce, and you’ve got the answer: people don’t think
security is failing enough to spend money doing something about it. The
externalities aren’t intolerable. The public isn’t up in arms; if
anything, security breaches have reached the same level of public
semi-awareness as bombing in Iraq — it happens every day, everyone
agrees how awful it is, and then they go back to their lattes.

We’re not going to fire or retrain a generation of cheap programming
labor to Do the Right Thing and redesign systems. Not until it hurts
enough, and let’s face it, it doesn’t. All the FUD and hand-wringing is
within the security industry. We’re doing our jobs just well enough to
keep things from melting down, so why should anyone pay more attention
and money to something that’s mediocre but not a disaster?

There’s not a whole lot more that needs to be said to embellish or underscore that argument.

I’ll be over here waiting for the next "big thing" to hit and instead of fixing it, we’ll see SoX part Deux.

See, Shrdlu’s not the only one who can toss in a little French to sound sophisticated 😉

/Hoff

Just as I finished up a couple of posts decrying the investments being made in lumping device after device on DMZ boundaries for the sake of telling party guests that one subscribes to the security equivalent of the "Jam of the Month Club," (AKA Defense-In-Depth) I found a fantastic post on the CERIAS blog where Prof. Eugene Spafford wrote a fantastic piece titled "Solving Some of the Wrong Problems."

In the last two posts (here and here,) I used the example of the typical DMZ and it’s deployment as a giant network colander which, despite costing hundreds of thousands of dollars, doesn’t generally deliver us from the attacks it’s supposedly designed to defend against — or at least those that really matter.

This is mostly because these "solutions" treat the symptoms and not the problem but we cling to the technology artifacts because it’s the easier road to hoe.

I’ve spent a lot of time over the last few months suggesting that people ought to think differently about who, what, why and how they are focusing their efforts.  This has come about due to some enlightenment I received as part of exercising my noodle using my blog.  I’m hooked and convinced it’s time to make a difference, not a buck.

My rants on the topic (such as those regarding the Jericho Forum) have induced the curious wrath of technology apologists who have no answers beyond those found in a box off the shelf.

I found such resonance in Spaf’s piece that I must share it with you. 

Yes, you.  You who have chided me privately and publicly for my recent proselytizing that our efforts are focused on solving the wrong sets of problems.   The same you who continues to claw disparately at your sacred firewalls whilst we have many of the tools to solve a majority of the problems we face, and choose to do otherwise.  This isn’t an "I told you so."  It’s a "You should pay attention to someone who is wiser than you and I."

Feel free to tell me I’m full of crap (and dismiss my ramblings as just that,) but I don’t think that many can claim to have earned the right to suggest that Spaf has it wrong dismiss Spaf’s thoughts offhandedly given his time served and expertise in matters of information assurance, survivability and security:

As I write this, I’m sitting in a review of some university research
in cybersecurity. I’m hearing about some wonderful work (and no, I’m
not going to identify it further). I also recently received a
solicitation for an upcoming workshop to develop “game changing” cyber
security research ideas. What strikes me about these efforts —
representative of efforts by hundreds of people over decades, and the
expenditure of perhaps hundreds of millions of dollars — is that the
vast majority of these efforts have been applied to problems we already
know how to solve.

We know how to prevent many of our security problems — least
privilege, separation of privilege, minimization, type-safe languages,
and the like. We have over 40 years of experience and research about
good practice in building trustworthy software, but we aren’t using
much of it.

Instead of building trustworthy systems (note — I’m not referring to
making existing systems trustworthy, which I don’t think can succeed)
we are spending our effort on intrusion detection to discover when our
systems have been compromised.

We spend huge amounts on detecting botnets and worms, and deploying
firewalls to stop them, rather than constructing network-based systems
with architectures that don’t support such malware.

Instead of switching to languages with intrinsic features that
promote safe programming and execution, we spend our efforts on tools
to look for buffer overflows and type mismatches in existing code, and
merrily continue to produce more questionable quality software.

And we develop almost mindless loyalty to artifacts (operating
systems, browsers, languages, tools) without really understanding where
they are best used — and not used. Then we pound on our selections as
the “one, true solution” and justify them based on cost or training or
“open vs. closed” arguments that really don’t speak to fitness for
purpose. As a result, we develop fragile monocultures that have a
particular set of vulnerabilities, and then we need to spend a huge
amount to protect them. If you are thinking about how to secure Linux
or Windows or Apache or C++ (et al), then you aren’t thinking in terms
of fundamental solutions.

Please read his entire post.  It’s wonderful. Dr. Spafford, I apologize for re-posting so much of what you wrote, but it’s so fantastically spot-on that I couldn’t help myself.

Timing is everything.

/Hoff

{Ed: I changed the sentence regarding Spaf above after considering Wismer’s comments below.  I didn’t mean to insinuate that one should preclude challenging Spaf’s assertions, but rather that given his experience, one might choose to listen to him over me any day — and I’d agree!  Also, I will get out my Annie Oakley decoder ring and address that Cohen challenge he brought up after at least 2-3 hours of sleep… 😉 }

I’m going to play devil’s advocate again as I ponder a point.  Roll with me here.  I’m slightly conflicted.

Jeff Hayes blogged about an interesting encounter in a sports bar he had with the head of physical security for an international accounting firm.  It turns out that as part of a casusal conversation, this person disclosed some very interesting facts about his company’s security:

It turns out this guy handles physical security for a major
international accounting firm. He travels around North America doing
premises and access control assessments and deployments. He described
to me, without me asking specific questions, the technology they use,
the problems they deal with including the push-back they get from each
office complaining about burdensome security, their budgets, his
working environment, how he moved up the company ladder and his
qualifications or lack thereof, and a number of other tidbits that
would prove valuable to anyone doing surveillance.

It would appear that this guy had one too many and the apparent level of detail disclosed seems excessive.  Jeff’s point about confidence and accelerated reconnaissance for targeted profiling seem to be quite relevant in this scenario.  This person was being reckless and was potentially endangering his company.

However, let’s look at this a little differently to illustrate a counterpoint.

This encounter sounds like what many of us read and talk about under the guise of non-attribution at many of the security forums and "professional" security gatherings we attend and participate in with our "peers."  You know the ones where we all sit around, hoping that the badges actually represent the fact that the organizers have appropriately vetted and authenticated that the person wearing it is who they say they are…

Moreover, it sounds a lot like the conversations at the bar after said forum roundtables.  We share our collective experiences in order to gain insight and intelligence so we can improve our security posture, accelerate our intelligence on short-listing vendors and not make mistakes by learning from others.

How about those Visio diagrams you show on the whiteboard to VARs when they send their SE’s in for work and pitches?

It gets even more interesting when you have CISO’s/CSO’s (like I do) talk to the press and do case studies describing technologies and solutions deployed.  Some CISO’s don’t mind doing so after making a tactical risk-based decision that what they reveal does not expose the company adversely.  Others simply don’t talk at all about what they do.

I understand there exists the potential that by disclosing that you use
vendor ABC or technology XYZ that someone could exploit that knowledge
for malignant purposes.  I suppose this is where the fuzzy area (I’m sorry Mr. Hutton!) of
thin-slicing and quickly assessing risk comes into play.   What is the likelihood that this
information when combined with a vulnerability (in policy, architecture, deployment) in the presence of a
threat might become a risk to my company?

I use Check Point NGX R65.  I run it on a Crossbeam X-Series.  It filters a bunch of packets.  I use Cisco routers.  Is that information you couldn’t have found out with a network scan, fingerprinting and enumeration?  Have I made your job of attacking me orders of magnitude easier?

Ah, the slippery slope is claiming me as a victim…

Have you seen the Military Channel?  I watched several fantastic Navy/Marine-sponsored documentaries on Carriers, NextGen APC’s, new weapons systems…all of which are deployed.  Is Al Qaeda now in a more advantageous position because they know how the de-desalinization plant on a fast frigate functions?

Everyone in a company is both a sales and marketing rep as well as a
potential security breach waiting to happen. Most businesses like
people to present their company in a good light. We want people to know
that we work for a good employer. What we don’t want people to do is to
tell others how crappy our employer is. Likewise, we probably don’t
want our security personnel describing the details of our security
systems, policies and procedures.

So Jeff’s right, but I guess that depends upon the level of "details" he’s referring to?  Is Jeff’s point still valid when we’re talking about a breakfast conversation at an Infragard meeting?  How about the forums over at SecurityCatalyst.com?  There’s that level of trust and judgment factor again.  How about an ISAC gathering?  Aren’t we all supposed to share knowledge so we can help one another? 

Where do we draw the line as to who gets to say what and to whom?  Those policies either have to get really fuzzy or very, very black and white…which goes to Jeff’s point:

Loose lips have been known to sink ships; they can also hurt organizations.

Yes they have.  They’ve also been known, when appropriately pliable with a modicum of restraint, to float the boat of someone whose time, energy and budget you’ve been able to save by sharing relevant experience.  Let’s be careful not to throw the baby out with the bilge water.

So, how do you establish "trust" and assess risk before you talk about your experience with technology you’ve deployed or are thinking about deploying?  What about policies and procedures?  How about lessons learned?

Obviously anybody who answers is not a true "security guy" 😉

/Hoff

On September 24-25th, InfoWorld will host their Virtualization Executive Forum in NYC which promises "…two days of
technical breakout sessions, case studies and industry expertise on
server, desktop, application, storage and file virtualization
technologies."

Here’s the overview:

Designed for those
who are evaluating where to begin and for those already implementing
virtualization technologies, InfoWorld’s Virtualization Executive Forum
features:

• Analyst perspectives on innovative uses of virtualization adoption
  rates and trends, and policy-based datacenter automation
  
• In-depth
  sessions examining Virtual Machines and Security, Open Source
  Virtualization, Business Continuity/Disaster Recovery, and more.
• Industry
  Keynotes from IT end users addressing the challenges, pitfalls,
  results, and benefits of their implementations
• A
  spotlight on Green IT practices and its potential for cost savings and
  reducing power and cooling needs in large datacenters.
  

In
addition to the in-depth case studies and industry panels you have come
to expect from InfoWorld’s Executive Forums, this fourth edition has
added another key ingredient to the mix: more opportunities for you and your peers to  collaborate and share experiences.

For an "executive forum" they have an interesting split-track breakout agenda; one track features case studies and the other focuses on technical presentations and panels.

Here’s the rub, did you notice that the word "security" appears only twice in the entire agenda, once in the keynote address and once more in a case-study breakout session on day two regarding applications of virtualization.  While I recognize that this is supposedly targeted at "executives," let’s take a look at the technical track breakout topics:

• 
  Vendor Crossfire: x86 Server Virtualization
• Getting Started with Server Virtualization
• Technical Track: Physical to Virtual Migration
• Leveraging Virtualization for Information Availability and Business Continuity
• Lessons from Big Iron: The Power of RISC UNIX Virtualization
• Open Source Hypervisor: Zeroing in on Xen
• VM Management and Monitoring
• Scaling Virtual Infrastructure

Not a mention of security in the bunch.  This is asinine. If you’re at all curious as to why security is an after-thought in emerging markets, look no further than this sort of behavior. 

…and don’t just tell me that security is "assumed."

If the executives who attend this two day forum walk away with a head full of fun new ideas and cautionary tales regarding virtualization and the closest thing to security they got was the valet guarding the doughnuts during the break, don’t anybody get surprised in 18 months when the house of cards come tumbling down.

InfoWorld, what the hell!?  How about ONE session — even a panel — titled something as simple as "Virtualization and Security – A Discussion You Need to Have."

In fact, you’re welcome to at least just print out my presentation from a couple of days ago and give it to your attendees.  At least they’ll walk away with something relating to security and virtualization.  850+ people from my blog already have more information on security and virtualization *for free* than is being presented at the forum.

Listen, I feel so strongly about this that I’ll speak for free on the topic — I’ll pay my own hotel, airfare, etc…and you can keep the doughnuts during the break.

By the way, I find it deliciously ironic that when I clicked on the "Visit Virtualization Portal" link in the above graphic, I was greeted by this little gem:

I’m sure this is probably running on a "real" server.  A virtualized instance would never have this sort of problem, right? 😉

/Hoff

Nope.  Haven’t changed my mind.  Sorry.  Harrington stirred it up and Chuvakin reminded me of it.

OK, so way back in April, on the cusp of one of my normal rages against the (security) machine, I blogged how Data Leakage Protection (DLP) is doomed to be a feature and not a market. 

I said the same thing about NAC, too.  Makin’ friends and influencin’ people.  That’s me!

Oh my how the emails flew from the VP’s of Marketing & Sales from the various "Flying V’s" (see below)  Good times, good times.

Here’s snippets of what I said:

Besides having the single largest collection of vendors that begin with
the letter ‘V" in one segment of the security space (Vontu, Vericept,
Verdasys, Vormetric…what the hell!?) it’s interesting to see how
quickly content monitoring and protection functionality is approaching
the inflection point of market versus feature definition.

The "evolution" of the security market marches on.

Known by many names, what I describe as content monitoring and
protection (CMP) is also known as extrusion prevention, data leakage or
intellectual property management toolsets.  I think for most, the
anchor concept of digital rights management (DRM) within the Enterprise
becomes glue that makes CMP attractive and compelling; knowing what and
where your data is and how its distribution needs to be controlled is
critical.

The difficulty with this technology is the just like any other
feature, it needs a delivery mechanism.  Usually this means yet another
appliance; one that’s positioned either as close to the data as
possible or right back at the perimeter in order to profile and control
data based upon policy before it leaves the "inside" and goes "outside."

I made the point previously that I see this capability becoming a
feature in a greater amalgam of functionality;  I see it becoming table
stakes included in application delivery controllers, FW/IDP systems and
the inevitable smoosh of WAF/XML/Database security gateways (which I
think will also further combine with ADC’s.)

I see CMP becoming part of UTM suites.  Soon.

That being said, the deeper we go to inspect content in order to
make decisions in context, the more demanding the requirements for the
applications and "appliances" that perform this functionality become.
Making line speed decisions on content, in context, is going to be
difficult to solve. 

CMP vendors are making a push seeing this writing on the wall, but
it’s sort of like IPS or FW or URL Filtering…it’s going to smoosh.

Websense acquired PortAuthority.  McAfee acquired Onigma.  Cisco will buy…

I Never Metadata I Didn’t Like…

I didn’t even bother to go into the difficulty and differences in classifying, administering, controlling and auditing structured versus unstructured data, nor did I highlight the differences between those solutions on the market who seek to protect and manage information from leaking "out" (the classic perimeter model) versus management of all content ubiquitously regardless of source or destination.  Oh, then there’s the whole encryption in motion, flight and rest thing…and metadata, can’t forget that…

Yet I digress…let’s get back to industry dynamics.  It seems that Uncle Art is bound and determined to make good on his statement that in three years there will be no stand-alone security companies left.  At this rate, he’s going to buy them all himself!

As we no doubt already know, EMC acquired Tablus. Forrester seems to think this is the beginning of the end of DLP as we know it.  I’m not sure I’d attach *that* much gloom and doom to this specific singular transaction, but it certainly makes my point:

  August 20, 2007

EMC/RSA Drafts Tablus For Deeper Data-Centric Security
The Beginning Of The End Of The Standalone ILP Market
by
Thomas Raschke

with
Jonathan Penn, Bill Nagel, Caroline Hoekendijk

EXECUTIVE SUMMARY

EMC expects Tablus to play a key role in its information-centric security and storage lineup. Tablus’ balanced information leak prevention (ILP) offering will benefit both sides of the EMC/RSA house, boosting the latter’s run at the title of information and risk market leader. Tablus’ data classification capabilities will broaden EMC’s Infoscape beyond understanding unstructured data at rest; its structured approach to data detection and protection will provide a data-centric framework that will benefit RSA’s security offerings like encryption and key management. While holding a lot of potential, this latest acquisition by one of the industry’s heavyweights will require comprehensive integration efforts at both the technology and strategic level. It will also increase the pressure on other large security and systems management vendors to address their organization’s information risk management pain points. More importantly, it will be remembered as the turning point that led to the demise of the standalone ILP market as we know it today.

So Mogull will probably (still) disagree, as will the VP’s of Marketing/Sales working for the Flying-V’s who will no doubt barrage me with email again, but it’s inevitable.  Besides, when an analyst firm agrees with you, you can’t be wrong, right Rich!?

/Hoff

Serendipity is a wonderful thing.  I was in my local MA bank branch on Monday arranging for a wire transfer from my local account to a Wells Fargo account I maintain in CA.  I realized that I didn’t have the special ABA Routing Code that WF uses for wire transfers so I hopped on the phone to call customer service to get it.  We don’t use this account much at all but wanted to put some money in it to keep up the balance which negates the service fee.

The wait time for customer service was higher than normal and I sat for about 20 minutes until I was connected to a live operator.  I told him what I wanted and he was able to give me the routing code but I also needed the physical address of the branch that my account calls home.  He informed me that he couldn’t give me that information.

The reason he couldn’t give me that information was that the WF "…computer systems have been down for the last 18 hours."  He also told me that "…we lost a server somewhere; people couldn’t even use their ATM cards yesterday."

This story was covered here on Computerworld and was followed up with another article which described how Phishers and the criminal element were spooling up their attacks to take advantage of this issue:

August 21, 2007   (IDG News Service)  — Wells Fargo & Co.
customers may have a hard time getting an up-to-date balance statement
today, as the nation’s fifth-largest bank continues to iron out service
problems related to a Sunday computer failure.

The outage knocked the company’s Internet, telephone and ATM banking
services offline for several hours, and Wells Fargo customers continued
to experience problems today.

Wells Fargo didn’t offer many details about the system failure, but
it was serious enough that the company had to restore from backup.

"Using our backup facilities, we restored Internet banking service in about one hour and 40 minutes," the company said in a statement today. "We thank the hundreds of team members in our technology group for working so hard to resolve this problem."

Other banking services such as point-of-sale transactions, loan
processing and wire transfers were also affected by the outage, and
while all systems are now fully operational, some customers may
continue to see their Friday bank balances until the end of the day,
Wells Fargo said.

I chuckled uneasily because I continue to be directly impacted by critical computer systems failures such as two airline failures (the United Airlines and the TSA/ICE failure at LAX,) the Skype outage, and now this one.  I didn’t get a chance to blog about it other than a comment on another blog, but if I were you, I’d not stand next to me in a lightning storm anytime soon!  I guess this is what happens when you’re a convenient subscriber to World 2.0?

I’m sure WF will suggest this is because of Microsoft and Patch Tuesday, too… 😉

So I thought this would be the end of this little story (until the next time.)  However, the very next day, my wife came to me alarmed because she found a $375 charge on the same account as she was validating that the wire went through.

She asked me if I made a purchase on the WF account recently and I had not as we don’t use this account much.  Then I asked her who the vendor was.  The charge was from Google.com.  Google.com?

Huh?  I asked her to show me the statement; there was no reference transaction number, no phone number and the purchase description was "general merchandise."

My wife immediately called WF anti-fraud and filed a fraudulent activity report.  The anti-fraud representative described the transaction as "odd" because there was no contact information available for the vendor.

She mentioned that she was able to see that the vendor executed both an auth. (testing to see that funds were available) followed then a capture (actually charging) but told us that unfortunately she couldn’t get any more details because the computer systems were experiencing issues due to the recent outage!

This is highly suspicious to me.

Whilst the charge has been backed out, I am concerned that this is a little more than serendipity and coincidence. 

Were the WF anti-fraud and charge validation processes compromised during this "crash" and/or did their failure allow for fraudulent activity to occur?

Check your credit/debit card bills if you are a Wells Fargo customer!

/Hoff

I was just leaving the office for a client dinner last night when I noticed I
couldn’t get to my TypePad blog, but I chalked it up to a
"normal" Internet experience.   

When I fired up Firefox this morning (too much wine last night to care) I was surprised to say the least.

I am just awestruck by the fact that yesterday’s PG&E  power outage in San Francisco took down some of the most popular social networking and blogging sites on the planet.  Typepad (and associated services,) Craigslist, Technorati, NetFlix etc…all DOWN. (see bottom of post for a most interesting potential cause.)

I’m sure there were some very puzzled, distraught and disconnected people yesterday.  No blogging, no secondlife, no on-line video rentals.  Oh, the humanity!

I am, however, very happy for all of the people who were able to commiserate with one another as they apparently share the same gene that renders them ill-prepared for what is one of the most common outage causalities on the planet: power outages.

Here’s what the TypePad status update said this morning:

Update: commenting is again available on TypePad blogs; thank you for your patience.  We are continuing to monitor the service closely.

TypePad blogs experienced some downtime this afternoon due to a
power outage in San Francisco, and we wanted to provide you with the
basic information we have so far:

• The outage began around 1:50 pm Pacific Daylight Time
• TypePad blogs and the TypePad application were affected, as well as LiveJournal, Vox and other Six Apart-hosted services
• No data has been lost from blogs.  We have restored access to blogs as well as access to the TypePad application. There
  may be some remaining issues for readers leaving comments on blogs; we
  are aware of this and are working as quickly as possible to resolve the
  issue. (See update above.)
• TypePad members with appropriate opt-in settings should have
  received an email from us this afternoon about the outage.  We will
  send another email to members when the service has been fully restored.
• We will also be posting more details about today’s outage to Everything TypePad.

We are truly sorry for the frustration and inconvenience that
you’ve experienced, and will provide as much additional information as
possible as soon as we have it. We also appreciate the commiseration
from the teams at many of the other sites that were affected, such as
Craigslist, Technorati, Yelp, hi5 and several others.

I don’t understand how the folks responsible for service delivery of these sites, given the availability and affordability of technology and hosting capability on-demand, don’t have BCP/DR sites or load-balanced distributed data centers to absorb a hit like this.   The management team of Sixapart has experience in companies that understand that the network and connectivity represent the lifeblood of their existence; what the hell happened here in that there’s no contingency for power outages?

Surely I’m missing something here.

Craigslist and Technorati are services I don’t pay for, so one might suggest taking the service disruption with a grain of SLA salt (or not, because it still doesn’t excuse not preparing for issues like this with contingencies)  but TypePad is something I *pay* for.  Even my little hosting company that houses my personal email and website has a clue.  I’m glad I’m not a Netflix customer, either.  At least I can walk down to Blockbuster…

Yes, I’m being harsh, but I there’s no excuse for this sort of thing in today’s  Internet-based economy.  It affects too many people and services but really does show the absolute fragility of our Internet-tethered society.

Common sense obviously didn’t make the feature list on the latest production roll.  Somebody other than me ought to be pissed off about this.  Maybe when Data Center 3.0 is ready to roll, we won’t have to worry about this any longer 😉

/Hoff

Interestingly, one of the other stories of affected sites relayed the woes of 365 Main, a colocation company, whose generators failed to start when the outage occurred.  I met the the CEO of 365 Main when he presented at the InterOp data center summit on the topic of flywheel UPS systems which are designed to absorb the gap between failure detection and GenStart.  This didn’t seem to work as planned, either. 

You can read all about this interesting story here.  This was problematic because the company had just issued a press release about a customer’s 2-year uninterrupted service the same day 😉

Valleywag reported that the cause of the failure @ 365 Main was due to a drunk employee who went berserk! This seemed a little odd when I read it, but check out how the reporter from Valleywag is now eating some very nasty Crow … his source was completely bogus!

The only thing worse than when people find out you’re in the "computer industry" and ask you to diagnose why their USB-powered combo blender/Easy-bake oven keeps giving them the BSOD is when they find out you’re in the "computer security" field and ask you to diagnose why their Symantec (nee Norton) Uber Blocking Pop-Up Personal Firewall prevents them from connecting to AOL.

Sometimes, however, I feel compelled to volunteer myself when I know I can quickly help so I can feel good about "giving back" and make the world a more secure place.

Today was such a day.

I took the kids to our local candlestick bowling joint en route to a matinee screening of "Hairspray" the movie (very good, by the way.)  As the kids were knocking down frames thanks to the bumpers in the gutters, I went to the ATM for monetary reinforcement in order to buy the requisite pop and pizza.

As I approached the machine, the floor manager — noticing that I was going to use the ATM — scurried to plug the machine in so I could use it.  Noticing that it was a Tranax unit since this particular marque has been in the news lately due to security concerns, I happily queried the manager as to whether or not they had changed the default password on the machine.

I don’t really know why I did this.  Perhaps because I wanted to settle a bet with myself or just to show off my mad security current event skillz.  Honestly, I think I just wanted to see what would happen under controlled circumstances.  Nevertheless, I asked and waited patiently for a response as the machine whirred and clicked.

She looked at me puzzled and asked what I meant and why.  At which point I was going to be content in alerting her to the potential that someone could easily use the Internet to gain 10 seconds of courage and rip them off by re-programming the ATM to think it was giving out $5 bills instead of $20 bills by gaining access to the admin. interface via the default password.

At the exact moment I said this, the machine finished booting as she walked away shrugging her shoulders wondering no doubt why this tattooed idiot in bowling shoes was trying to "help."  As she did this, the screen started blinking alerting me that the cash magazine was empty and if would I like to enter the Administrator mode.

I called her back over to the ATM and said "watch" at which point I was queried for the administrative password which I dutifully keyed in as "######" (not shown so I don’t enable those idiots who can’t manage to find the real number via Google.)  The myriad of administrative options was splayed out before me and we walked through the various scenarios that might appear should we execute.

Das machine was owned and now she understood.

We agreed that this was a bad thing and that she should unplug the machine until the owner who serviced the unit could be contacted.  I suggested that she find a way to make sure that nobody could plug it back in easily and I walked her through changing the password.

I figured I’d done a good deed and proceeded go out into the parking lot and scour my car for loose change so I could at least buy the kids a soda since I could no longer get cash and I didn’t exactly trust their security to use my credit card at this point.

I returned to find the manager giving me back the $23 I paid for bowling in return for the security lesson.

I thanked her for the trade and got the hell out of there before she asked me how to update the anti-virus signatures on the point of sale terminal that took credit card payments…

The moral of the story?  Don’t be afraid to offer a little security help every once in a while.  You never know, it might earn you $23 and some free bowling.  Karma.  Nice.

Now I’m going to visit the Mobil station down by the highway…they have the same machines.  I could always use some free gas 😉

As Cutaway would say…"Go forth and do good things."

/Hoff