Double D'Oh!

We all make mistakes—some are just more obvious than others. But it’s a graceful recovery that separates the gurus from the geeks.

• By Em C. Pea
• April 01, 2001

What do you do when Something Stupid happens? I’m not talking about accounts payable sending the payment for the servers to the guy who keeps the soda machine stocked, or when Auntie thought it fabu to wear faux fur to the Oscar after-party. No, I’m talking about Something Stupid.

For instance, let’s take my example of freedom to innovate in the technology biz. During two days in January, not one but two Somethings Stupid happened to our friend in Redmond. It got hacked and was a victim of its own sloppy architecture.

First, Microsoft’s Web sites were flooded with denial-of-service attacks, those bogus requests that suck up bandwidth until the servers are crippled. Microsoft brought in outside help to plot strategies against future DOS attacks. There are only a limited number of countermeasures against this weapon, but give Microsoft credit for how it handled this Something Stupid.

It’s the other Something Stupid that had Auntie snickering. The day before the hacks, Microsoft’s Web sites were unreachable for hours. Seems it had four DNS servers on one subnet and some unfortunate tech screwed up the configuration on their gateway. Result? “You can’t get there from here.”

Keep in mind that Microsoft prides itself on “eating its own dog food,” implementing its own products and designs on its own network. The network architects must have sent out for turkey sandwiches the day they came up with this design. Just to twist the knife a little, Auntie will now quote twice from Microsoft’s Windows NT Server 4.0 Networking Guide, Chapter 9, “Managing Microsoft DNS Servers”:

Generally, plan to install the primary and secondary servers on different subnets to provide continual support for DNS name queries if one subnet should go down.
The minimum number of DNS servers needed to serve each zone is two — a primary and a secondary — to provide database redundancy. As with any fault tolerant system, the computers should be as independent as possible, for example, by placing the primary and secondary servers on different subnets.

This Something Stupid has two components:

1. Human error. I admit a morbid curiosity as to whether the tech remains in the employ of Microsoft, but let’s be realistic: people make mistakes, and let he or she cast the first stone who has never forgotten to pull the floppy before rebooting, failed to save the i386 directory to the hard drive, or not set all objects equal to Nothing.
2. Less-than-brilliant DNS physical implementation. In the course of human events and enterprise architectures, we all do the occasional boneheaded Something Stupid. It’s how we recover that distinguishes run-of-the-mill IT geeks from the gurus who get renewed at a higher rate.

Being a well-known iconoclast, Auntie proposes a modest yet revolutionary solution. Raise your hand high and say, “It was me! I screwed up! My fault! Right over here!” Doing this will stun the powers-that-be long enough for you to add, more quietly, “And here’s how we can show that we’ve fixed it and put into place a mechanism to prevent this Something Stupid from happening again.”