In-Depth

Fear and Loathing in Las Vegas

In a city of high stakes none were higher than during this tale of database corruption. But with tenacity and resourcefulness (and a little luck), this MCSE came out a big winner.

• By Rodney Landrum
• April 01, 2001

As I've spoken to other engineers and DBAs, I'm sure I've said many a time that the keys to longevity in the turbulent landscape that is Information Technology are tenacity and resourcefulness. Of course, as I found out, the realities of IT can put those virtues to the test.

Trudging to an airport restroom to change out of gaping, split-seamed khakis that had been ripped while climbing out of a luggage-laden airport taxi, I’m sure I didn’t appear to be someone on his way to Las Vegas to compete for a prestigious Industry Solution Award from Microsoft. I’m also sure that those in the know would have had to stifle a snicker to find out I had missed the flight by 20 minutes. I most likely appeared to have about as much resourcefulness and tenacity as a Styrofoam cooler. But after the sluggish start to this fateful journey, the silicon gods must have been smiling on me, as I did manage to get myself and my traveling companion to Las Vegas, albeit after 12 cumulative hours on stand-by as penance for missing the flight. The companion, by the way, was the president of the company.

Special Delivery
The server I had lovingly built up during the past several weeks was safely nestled like a football under my left arm for most of the trip. Its immediate purpose, before it would be converted to a VPN or fax server, was to show off the company’s SQL Server-based application to a panel of judges and several hundred Microsoft Healthcare Users Group (MSHUG) conference attendees. My immediate purpose was to make sure it made the trip unscathed because when it was finished “wow”-ing them in Vegas, it was to fly on to Texas, where it would be used to demonstrate the same application to a large group of executives in a large company who were interested in making a large software purchase. Other than server bodyguard, my job was a simple one: Point on the right icon at the right time and press the page-down button when queued by “El Presidente.”

We arrived at our booth early. Thankfully, all of the peripheral equipment we’d rented was waiting for us. I immediately set up and confirmed that the server had sustained the jostling and jouncing of being bumped across America. All that remained was to fight back the nervous energy and jitters that had been steadily growing as I saw just how enormous this show had turned out to be.

Appropriately, Las Vegas has every known cure for jitters. The next two days, as we waited for our turn to present to the judges, were a blur of taxi rides, neon lights and complimentary beverages. I also had the task of finding a special slot machine to play my wife’s $20, which she had given me along with a hopeful, “Good luck!”

Without a Hitch
Finally, our time came, and I dutifully displayed my rehearsed point-and-click savvy to the eager, note-jotting judges. After the 15-minute presentation, I was, simultaneously, overcome by relief and dry mouth. After a congratulatory pat on the back from my boss, I headed for the nearest soda machine.

My mind was reeling from the questions we’d been hit with by the judges. One question, in particular, was, “Can your application be configured to launch other applications?” Well, we had designed our menu structure to be able to embed other applications, like Microsoft Excel, as links directly into our database. But we’d only tried it a few times and had refrained from doing a live demo of this ability. The judges seemed content with that.

OK, What Did You Do?
When I returned from my break, I saw my boss feverishly clicking on myriad SQL errors popping up on the screen. He didn’t offer an explanation or even acknowledge my presence. He just kept trying in vain to get back into the program. I knew not to ask. Eventually he relinquished the mouse with a statement just above the audible range:

“Can you fix it?” That’s when instinct took over like an ER doctor pulling a trauma patient back from the edge.

“OK. What did you do?” I asked as I moved quickly into position.

“I tried to embed a link to Excel in the database,” was his only reply.

I starting digging and within three minutes had the ugly truth: The database was corrupt. The 200MB database was corrupt. The 200MB database—with no local disk dump and no connection to the office to get the backup—was corrupt. I had to inform the next of kin, and he didn’t like it one bit. The large demo in Texas was scheduled for the next afternoon. As we were consoling one another, the announcement came over the loud speakers, “The show floor will be closing in 15 minutes. Winners for the Industry Solution Awards will be announced tomorrow morning. Thank you, everyone, for attending.”

Translation: “Scram! And make sure you leave our equipment behind!”

“At least we made it through the presentation,” I chided. The man who had signed my paycheck for the past two years said, with a look, “Get to work. You have 15 minutes.”

Is Anybody Home?
At this point, resourcefulness kicked in. Tenacity would show up much later in the evening. I tried everything I could think of. My tier-1 tech support background instructed me to bounce the server — no good. Database Consistency Checks (DBCC) revealed errors of the irreparable variety. I saw the floor workers stripping keyboards and monitors away from other booths. Our time was up. I turned to my boss and said in a contrived, yet somehow believable, tone, “Don’t worry, I’ll take care of it.”

It was about 5 p.m. in Las Vegas; therefore, it was 8 p.m. back in our Pensacola, Florida office. Surely someone would still be there at 8 p.m. — that was the fortunate side of working at a company of workaholics. Sure enough, a programmer was still there. I hurriedly explained my dilemma. All I needed was for her to take the backup copy of our database, compress it to an optimistically small file and put it in an accessible location. She said she’d have it posted in half an hour.

Intergalactic Inspiration
I had a relatively powerful machine at my disposal — but, without a keyboard and mouse at the very least, it was useless; a modem would be required as well. Armed with the company credit card and carte blanche to get the job done, I set out to piece together the missing components. On the way through the hotel, I ran into a few meandering Klingons. You see, the Hilton, where we were staying, was also the home of “Star Trek: The Experience.” I couldn’t avoid stopping in for a short visit at Quark’s Bar — you know, just to say I had been there. From my vantage point in the bar, I spotted an even more interesting site: a 24/7 copy shop right across the way. I was instantly relieved to think I might not have to carry a computer and, now, a brand-new monitor back home with me after all. I raised a blue-frothy, cosmic toast to my fellow patrons and teleported over to the copy shop.

The first thing I noticed was the external 28.8Kbps modem attached to one of the rental workstations. After much pleading, which soon to turned to groveling, I convinced the patron using the system to hurry along. I had also, using similar tactics, convinced the copy shop clerk to let me disassemble her workstation and use the parts for the next three hours. (The gold corporate credit card may have helped.)

I brought my machine down and packed into a very small alcove where I had access to everything I needed, including a phone line. I dialed into our RAS server at work and easily found the file. It had compressed down to 15MB. I started the download from the command prompt where I could use “xcopy” with the /Z option. If I were to get cut off, which I assumed I would at least once, I could re-establish the connection, start “xcopy” again and pick up where it had been cut off. I asked the clerk to watch the machine and I went off to play my wife’s $20.

Hitting it Big
I knew I’d be there for a while, so I took my time, trying to find the perfect machine upon which to play the $20. Quarter by quarter, I had played down to only three, so I figured I’d move to the dollar machines. If I was going to lose, I might as well lose a little faster. I dropped all $3 in, pulled the handle and prepared to walk away. To my utter astonishment, I hit triple sevens. Twenty minutes and $275 later I was back at my computer with only 300KB remaining on the download. When it was finished, I uncompressed the file, restored the database, and was able to get back into our program.

Life was good.

I didn’t tell my boss until the next morning that I had everything back together. We met at the awards ceremony, and I’m sure I looked quite haggard. When it came time to present the award for our category, I felt the butterflies returning. I knew that I wouldn’t be the one to physically accept the award if we won, but I was nervous just the same. We did win, for the first time after several years of coming very close, and the trip to Texas went off without a hitch. But the best part of the adventure wasn’t winning the award, or getting a large client, or even scoring big at the slots. It was leaving Vegas with the realization that — in the face of seemingly overwhelming adversity — resourcefulness and tenacity (and, perhaps a little intergalactic help) can bring you through.