In-Depth

Thin Clients, Fat Heads

This business owner’s thin-client Windows network was impregnable. Or so he thought, until he met Bob…

• By Kevin Kohut
• May 01, 2004

“We’re really crawling here. Is there something wrong?” I tell her that we’re already checking it out, and we should have things back to normal soon.

Another client, then another, calls, each with the same complaint. One more comes in. This time my customer offers an explanation: “I think there’s a virus on the network!”

Virus? No Way!
A virus? There’s no way we could have a virus. Yes, this was all happening in the midst of a serious virus outbreak, and yes, many large corporations had already been infected, but not our systems. Let me explain.

My company, EasyOffice Network (EON), provides clients with comprehensive, thin-client–based, fully-managed IT solutions. We supply, build, configure, manage and retain ownership of all computer systems used by our clients. I’ve built most of these IT infrastructures from the ground up and have employed best practices in how we manage everything from desktop machines to file severs to routers to firewalls. I’ve been doing this for a long time and am proud of a long track record of keeping my clients’ systems in good working order.

We’re able to do this thanks to several key elements of our service model:

The capabilities of Terminal Services in a Windows 2000/XP environment.

The licensing agreement we have with Microsoft that allows us to rent the use of Microsoft software to our clients.

The ability to remotely administer the computers and network systems in place at our client sites.

The fact that our high-end data center systems are used to support several clients.

The proprietary expertise we incorporate into our system configurations.

Because we use a server-based computing model with thin-client workstations, the technical requirements for one of our IT solutions are quite different than those for a traditional office network environment. To illustrate, let’s look at a typical EON solution.

Our customer, Little Guy Industries (LGI), has two locations: their main office in Los Angeles and a smaller branch office in San Jose. We put a file server, an application server, a SQL server and a domain controller in the Los Angeles office. In San Jose, we placed a domain controller (which also serves as a file server) and an application server. The application servers run Terminal Services. All servers are configured with real-time virus-checking software.

Thin Is In
For the clients, we use LGI’s existing Dell computers—a mixture of older Celeron- and Pentium III-based machines. We rebuild all the Dells with a standard Windows XP Professional configuration. These machines are used to do one thing, and one thing only: run the Remote Desktop Client to connect to an application server. They won’t be used for Internet browsing; they can’t even get out to the Internet! As such, we don’t install any kind of virus scanning software on clients—remember, all program activity takes place on the application server through a terminal services session; nothing actually takes place on the client machine.

The offices are each connected to the Internet via DSL, protected by industry-standard firewall appliances. Interoffice network communication is handled by a hardware-based VPN. Only the servers have access to the Internet, and then only through a firewall.

Now that you have an idea about the typical EON environment, let’s see what happens when trouble’s afoot. Suppose it’s a Tuesday morning, and word of a new virus threat makes the rounds. One of our standard procedures is to check the firewall traffic logs to see what kind of packets are hitting our networks from the outside, and sure enough, we find evidence of attempted attacks. We contact our upstream Internet provider (if they haven’t already contacted us first) and ask them to keep tabs on the traffic going through their networks.

We also check our server logs for evidence of virus activity. They show lots of attempts but no infections. Thanks to the faithful real-time virus scanning software installed on each server, no one suffers any ill effects from this attack.

We’ve survived one of the more destructive and widespread virus attacks that have come down the pike in a long time. Feeling ever so smug in our prowess over infectious bytes, we send an e-mail to all our clients detailing how severe this virus attack was to many Fortune 500 corporations—and emphasizing how not one of our clients suffered even as much as a sniffle.

Virus Outbreak No. 2
Several weeks later, another virus outbreak. We dutifully carry out our standard procedures, and again our servers are undaunted. We’re thinking about sending out another boastful e-mail—but wait! Our clients start reporting problems—slow performance, unable to connect, dropped terminal server sessions.

OK, so we hold off on sending the “We beat the virus” e-mail to our customers and start troubleshooting. Could these problems be related to the virus outbreak? No, we think, just an untimely coincidence. There’s got to be something we can find to explain these problems.

After observing heavier than normal network traffic, we find that there is, indeed, evidence of virus activity on our internal networks. So I ask my techs to pore through the firewall logs again. But this time, I tell them to look at activity from inside our NAT-compliant, firewall-protected, private networks. “But the only way into our private networks is through public Internet gateways, and we’ve already proven that nothing came through any of our firewalls,” my senior tech points out.

I repeat my instructions. He be-grudgingly complies with my request. Later that day I get an e-mail from my tech. He tells me that he was able to narrow down where all this virus mayhem began—at one of our client sites (let’s refer to this client as “Bob”), from inside the private network. As I’m contemplating how this could have happened—how a virus could just appear in a private network without any trace of it coming through the firewall—I get another e-mail. This one is from Bob.

He tells me that he thinks his laptop is infected with a virus and wants to know what to do about it. “What laptop?” I think to myself. We never sold or discussed a laptop computer with this client. I e-mail him back. I tell him how to discover the MAC address of his NIC and give this bit of information.

I pass this info on to my tech, and he confirms that Bob’s laptop is the source of all this extra network traffic. Further analysis reveals that none of our other internal subnets started having problems until after Bob’s laptop started doing its thing.

Virus? Way!
Sure enough, all our virus issues started with Bob’s laptop. He brought it into the office, plugged it into the network, and that was it. Our servers were undaunted. They were used to being buffeted with attacks. But our thin-client workstations? They were never designed to be in a hostile environment. Their only purpose in life is to connect with a Remote Desktop Connection to a Terminal Server, all inside a private network.

OK, so we needed to get these thin-client machines working properly again. And we needed to do this in a way that minimized downtime. Thankfully, this was pretty easy. Remember, all the applications used by our customers are run from terminal servers, and all their data files are stored on servers, as well. Because the thin clients didn’t have anything installed on them other than the operating system and remote desktop clients, all we had to do was re-image them, a 10-minute process using our disk imaging software.

But, even at only 10 minutes a pop, with dozens of machines to re-image in disparate locations, it still took the better part of a day to complete the task. No data was lost, and our clients suffered little actual downtime.

Lessons Learned
But Bob’s little laptop adventure did make us think about revamping our security procedures—and our contracts. From now on, we’ll be stricter in enforcing our policies and making sure our clients are held accountable for following them.

For starters, we reworked our standard contract to include specific language about our customers’ responsibilities regarding their own equipment. But anyone who’s in this line of work knows that a contract is only good after the fact. Because our goal is to have satisfied clients, we also revamped our thin-client image to include anti-virus software. Yes, it costs us more; but, hey, our customers are worth it.

We’re also examining more sophisticated technology solutions to address this area, specifically in the areas of DHCP and network switching. If we can prevent a non-EON client device from getting a valid IP address in the first place, we’ve pretty much nipped the problem in the bud. Again, this increases costs, and at some point we’ll have to pass them on to the client.

A typical EasyOffice Network thin-client arrangement. (Click image to view larger version.)

Are YOU Ready for Bob?
“OK, Kevin,” you may ask, “nice story and all, but what does this have to do with me?” Well, if you’re an IT pro responsible for maintaining computer systems (or even if you’re not), realize that you can’t just rely on technology to keep things running smoothly. This latest wave of viruses may not have caused you or your users any trouble, but eventually you’re going to have deal with a Bob of your own. And all the anti-virus software in the world won’t protect you. Are you ready for Bob?