Top Guns Under Fire
They were a smooth-running, well-oiled IT machine—then a sticky, application-specific problem tested their mettle.
- By Bob Wheeler, Jr.
- March 01, 2003
YOU KNOW HOW you crave that rare and unusual feeling where everything
seems to be going OK? Like when your office network was purring and your
confidence level in your remote networks was in the 99 percent range?
I was recently blessed with that rare feeling here in our home office.
I savored it and inhaled deeply, sucking in all of the conceit I could
muster. The recent migration in our engineering firm from NetWare to Windows
2000 was now complete, stable and deemed successful by the senior staff.
My new firewalls and routers were working great in all offices, and I
had a competent IT staff. There was no doubt—we, the IT team, were Top
An Easy Fix
The feeling lasted until a network engineer knocked on my door
and informed me that one of our remote offices was having serious problems
accessing time sheets on our Web server in the home office. The problem
seemed to be excessive disconnects and time-outs. There’s no WAN or VPN
connection for the remote office running in a separate domain but simply
a 768kbps DSL Internet connection. The Win2K clients in the remote office
were attempting to access a Windows NT IIS Web application via Internet
Explorer. A quick check of the Web server indicated no problems with other
remote offices. Sounded like an easy fix for the Top Guns.
The LAN operator in the remote office—normally a full-time engineer or
designer who volunteers to monitor the local network—reported that the
problem was steadily getting worse and that there had been no recent configuration
or hardware changes. Outlook was starting to lock up the client computers
and Web pages were taking forever to open. The HR director informed me
that it was a payroll day and they needed timesheets from the remote office
I stalled and pressured my Top Guns for a quick solution. The remote
office was four hours away, so there was no chance of dispatching my network
engineer for a quick fix. We immediately narrowed the problem to a faulty
Internet connection, router or hardware firewall. More time passed before
the division director for the remote office called; he was alarmed that
he couldn’t use FTP to upload computer-aided drafting drawings for a client.
And meanwhile, the HR director was still waiting on an answer. The LAN
operator reminded me that she didn’t have a lot of answers for users who
had a lot of questions. Naturally, this caused our Top Gun status slip;
we were now Lower Guns.
From Top Guns to Cap Pistols
With no WAN or VPN connection, I had no way to access the remote
office except via the Internet, which was broken on the remote side. My
network engineer was able to squeeze out enough bandwidth to access the
remote firewall and routers. It was painfully slow but still better than
our last option—a 56kbps dial-up. Strangely, the router and firewall looked
good, although pinging got slow responses.
More time passed. The Lower Guns reviewed the situation. The phone was
ringing off the hook. We eventually isolated the problem to the Internet
connection itself. I recommended to HR and remote division directors that
the users fax their time sheets to the home office. That wasn’t pretty.
I retreated to the safety of my office. And we were demoted again—now
we were simply Guns.
This was a time we could really have used the packet analyzer we discussed
buying. Instead, we opted for more Microsoft licenses. My network engineer
contacted the DSL Internet provider for the remote office. The provider
claimed everything was peachy, thank you very much. My network engineer
debated with him. The tech support request was escalated several levels
until we were connected with an engineer who’d actually heard of “packets.”
We pleaded with the engineer to analyze the Internet traffic for the remote
office. “No problem,” he said. “It’ll take a few days.” Ah, the joy of
DSL connectivity. It’s cheap—and so was the support that went with it.
The Big Cheese Arrives
Next, the CEO showed up at my door, wanting to know what was going
on. I smiled and reassured him that everything would be fine. He winked
at me, knowing I needed his support. I was grateful for his confidence.
The growling in my stomach reminded me that I missed lunch—again. It was
time to put this to rest. No way were the Guns going down shooting blanks.
I reminded my staff of the infamous “failure is not an option” remark
from the movie Apollo 13. They suggested a solution involving duct tape,
garbage bags and a modem. Not funny. My network engineer revisited his
dialogue with the DSL provider and negotiated a quick-and-dirty traffic
analysis. While waiting forever for them to call back, the phone continued
ringing off the hook from employees. The waiting list for my network engineer
was growing, and my LAN and CAD administrators were trying to fill in
The Big Break
Finally, the provider called and informed us that there was an
immense amount of broadcast traffic flowing through the DSL connection.
My network engineer and I exchanged deer-in-the-headlights looks. The
remote office hosted no Web, FTP or teleconferencing services, and the
firewall reported no hacking of any kind. After receiving the information,
we thanked the provider and scratched our heads. We also remembered that
the LAN operator stated that there were no hardware or configuration changes.
The remote division director was increasingly anxious to know when the
Internet connection would be functional. Again, I stalled. The network
engineer and I started sifting through the firewall log reports obtained
from the remote office. We were looking for broadcast traffic and noticed
that some unconventional ports were open and active. We traced the port
usage back to a user.
We then developed an unorthodox plan. We accessed a specific workstation
in the remote office via Symantec pcAnywhere. From that workstation, we
accessed the suspected user’s local hard drive across the local network
to see what the user had loaded. Bingo! After clandestinely stomping around
on the user’s hard drive, we discovered a Webcast application.
Next, we accessed another user’s drive. The Webcast software was there
as well. Could it be? We contacted the LAN operator, who claimed no knowledge
of the Webcast application. We instructed the LAN operator to immediately
uninstall the application from all client computers. Meanwhile, in the
home office, we monitored Internet bandwidth for the remote office. Victory!
The bandwidth slowly started increasing. After about two hours, the bandwidth
was back at 100 percent.
Oh, What a Tangled Webcast…
The Webcast application had essentially choked off all the available
I queried the remote users regarding the Webcast application. They said
the application provided them with real-time news, sports and stock statistics.
They loved it and were disappointed to see it go, although it had become
painfully slow. One of the users had obtained the application and passed
it around discreetly, and none of the remote users realized the application
could cause a problem. The LAN operator was embarrassed but not at fault.
We reconfigured our firewalls to block this traffic.
Lesson learned: Know what applications your users are running,
and don’t allow unauthorized programs on their machines.
This story had a happy ending, but reminded me that “monitoring” is a
relative term. Supporting remote offices can be both a logistical and
technical challenge, even with the best of teams. It took several days
to finally correct all aspects of the problem. In a perfect world I’d
have IT technicians, administrators and engineers at all my sites, along
with the fastest and most secure connectivity. Then I wake up.
Small but Powerful
For now I’ll settle for my IT staff and LAN operators, who truly are Top
Guns. They may be a small group, but they can hold their own and are humbled
by the confidence and support of our employees. You don’t have to work
for a huge company to be viewed as a professional or to find technology
challenges. It’s not the amount of money that determines success, but
the quality of people working on a problem.