Implementing this emerging data communication security standard can tighten up your network. Here’s how its default policies work.

IPSec, Your Private Communications Security Cop

Implementing this emerging data communication security standard can tighten up your network. Here’s how its default policies work.

• By Roberta Bragg
• August 01, 2000

My teenage son Fred (all names have been changed to protect the innocent) recently got a job as a security guard. It’s not enough that I have to worry about his flaming out in a spectacular automobile wreck or dying of alcohol poisoning, now I have to worry about his being gunned down while he’s protecting the secret warehouse of a national retail chain. (The location of the warehouse has been secret since a whole truck of Beanie babies or Furbies or something got stolen a couple of Christmases ago.)

Never one to just sit back and imagine the horrors he might encounter, I’ve been learning what he and his buddies (they call themselves the “private security enforcement cops”) do. Turns out, their job is much like an IP Security (IPSec) agent that enforces the IPSec policies you can implement in Windows 2000. Let me explain.

IPSec is an emerging data communication security standard. You may have used it yourself to implement a Virtual Private Network (VPN) between two routers. Win2K can use IPSec to secure communications between computers in your network and beyond. It can also be used to encrypt the data in an L2TP tunnel in a Win2K VPN.

Understanding what IPSec is, however, and how to implement it isn’t something we can cover in 3,000 words or less. There are four natural ways to divide and conquer the subject:

• An intimate discussion of the protocol itself.
• IPSec combined with L2TP for VPN.
• IPSec default policies.
• IPSec custom policies.

Since this is my first column for you on IPSec, I’ll focus on a discussion of the default policies. I’ll use these to help you understand the possibilities that IPSec presents, and show you how to incorporate IPSec policies for greater network security—without becoming an IPSec guru. I’ll leave the other topics for future columns.

To use IPSec, you don’t have to understand a great deal about the protocol, but you do have to know the default IPSec policies that are available, what they do, and how to use them. In order to fully use the power of IPSec and troubleshoot it when things go wrong, you need to invest some time in understanding this complex protocol and how it’s implemented in Win2K. To maximize your effort and begin securing network communications faster, study the default polices first. An understanding here will allow you to lock down a server and require encrypted communications, allow encrypted communications with a client, or allow the server to act as a client and negotiate encrypted communications with another server.

Please note that I didn’t say to blindly implement these policies on your network. You need to test them in a controlled environment to make sure you understand them. Also, when exploring, you should always exit each policy page by using the Cancel button. Never, ever “Assign” a policy until you understand what it will do. It’s possible to unwittingly create your own “Denial of Service” attack using default policies. On the other hand, by modifying these policies you can intricately tune your policies at the protocol level so that nothing travels on the wire to or from your systems that you don’t control. (Of course, if you’re not careful, the control can turn absolute; then nothing will travel on the wire.)

What can IPSec do for your network communications? IPSec can be configured to manage authentication, authorization, and encryption. It also has a tunnel mode. When we talk about secured communications, we’re talking about implementing some or all of these features.

No Data Transmission Security

When you’re driving, anyone can look in and see who’s in your car. We don’t get to travel incognito in Wonder Woman’s invisible plane. It’s like this on a Win2K network. By default, there’s no security established for data communications. Oh, users have to be authenticated before they can use the systems, and their access is authorized or not, but once the connection is made and authorized, most data travels from computer to computer without the benefits of encryption. Moreover, unless some kind of port filtering or a firewall has been configured to protect it, any available type of data can be transmitted.

If you want to change that picture, follow Fred and me as we discover how to establish default policies.

Fred’s First Day: Default Policies Apply

When Fred started his new job, he put on his uniform (Oh, my! He looks just like a real cop from a distance), drove south, and reported for work. Turns out he got guard duty in a shack outside the truck entrance to the warehouse. It took the boss all of five minutes to give him the low-down:

• Rule 1: If the truck’s paperwork is in order and matches your list here, send ’em on in.
• Rule 2: If the truck’s paperwork isn’t in order, make them park and wait. Pass the information on. Something has to be cleared up.
• Rule 3: If they’re not on your list, send ’em packing.

Fred was given the established policy, he had his orders, and that’s all there was to it. The policy was created at headquarters and pushed out across the company’s many locations. Just like Win2K Group Policy: Once set, everyone follows the lead.

When you start working with IPSec, you’re almost as lucky. IPSec policy isn’t turned on by default, but default policies exist. Turn them on (or “assign them,” in IPSec policy lingo) and they’ll work.

But hold on—don’t turn on these policies until you understand how they work; you don’t want your own domain-wide Denial of Service attack. IPSec policies are that powerful. They’re designed to control communications between computers. Set the wrong policy at one computer and not at another and it’s possible to shut down communications. Let the brave and the stupid beware: Use Group Policy to establish your control on multiple computers and you might be looking for another job—say, one that’s a tad less technical.

So how do you properly set default IPSec policies? They’re not at some secret location—in fact, you’ve probably passed them a zillion times on your way to some other policy implementation. But halt! Remember to play with IPSec policies on a test computer. For our purposes, select the Local Security Policy from the Administrative Tools Menu. Navigate in the Tree pane to and select “IP Security Policies on Local Machine.” In the Scope pane you’ll see three default policies (see Figure 1).

• Client (Respond Only)
• Secure Server (Require Security)
• Server (Request Security)

Figure 1. In your tour of the default policy, you’ll find three rules displayed, which allow you to define what type of traffic is allowed.

You should also see a description and a column designated “Policy Assigned.” (Yours should all say, “No,” unless they’ve been implemented previously.)

Before you investigate the property pages of these policies, right-click on one and note the “Assign” selection. Default IPSec policies are present, as you can see, but they’re not in effect until they’re assigned. Remember this. When you create your own IPSec Policy, it won’t be used until it’s assigned. This is different from the normal group policy application. If I give you a new right on this system or change the password policy at a domain controller, the policy will be implemented at the next policy refresh. An IPSec policy won’t be, unless it’s been assigned.

To implement any of the default policies on a computer, select “Assign” from this drop-down list and wait for or request a policy refresh. You don’t need to make any additional settings. The following discussion is simply for the purposes of allowing you to learn about IPSec and when to implement default policies, not as a tutorial for how to configure them. To remove any of the default policy implementations, select “unassigned” from this drop-down list and wait for or request a policy refresh. If you’ve implemented IPSec Group Policy at a higher level, remember: the usual Group Policy inheritance rules apply.

Fred’s default policy was simple to understand, and these are too. Double-click on the policy and click the General tab to get a helpful description.

The Client (Respond Only) policy allows the systems that it’s set on to operate in the normal manner for all communication, unless a server requests secured communications. Then the client can participate in negotiating communications. Notice the word “negotiate.” Just because the client system is set to respond to a server’s request for secure communications doesn’t mean it will be able to communicate. Client and server must be able to agree on the security that’s enabled, and what types of communication are allowed. Further, only the type of traffic the server wants to be secured will be. Other types of communication between client and server might take place with no security enabled or might not take place at all. Remember, policy is set on two machines—server and client—and they must be able to agree on the particulars in order for communication to occur. (Imagine a truck that doesn’t have its paperwork in order. For the truck whose papers were correct, entry was swift. For the one that wasn’t, negotiation could take awhile. It might even result in no entry at all.)

The Secure Server (Require Security) policy requires the server to insist on a negotiated conversation. Moreover, every communication is based on first establishing that the client is trusted. That is, a Kerberos trust is required. No unsecured communication at all is allowed with untrusted clients. So what happens if your client has no IPSec policy set and your server does? The server will listen to your request, but respond only by requesting negotiated communications using IPSec. Even if the client computer is joined in the domain (trusted), since no policy has been set on the client, the client can’t respond. In the case of failed negotiation, further communication isn’t allowed. If we think of Fred’s guardpost as the server and a truck full of merchandise as the client, it’s as if Fred asked the truck driver at the gate for his paperwork, and the truck driver had none. It really doesn’t matter to Fred whether or not the truck is on his list. No paperwork, no entry—that’s the rule.

By now you can probably guess what the server (Request Security) policy states: The server will always request security, but if the client doesn’t respond in kind, he still gets to drive his truck through the gate. (Fred would get fired for that one.) This policy is the hardest to understand. I’ll explain more on that shortly.

How Policies Work Together

Remember that policies require negotiation. It takes two computers to play. Let’s see how these policies can work together on our network.

Scenario One: Secure Server/Accountant to Accounting Database
Our first scenario is easy to understand. If I set Secure Server (Require Security) for my accounting database server, Client (Respond Only) for all accountants’ Win2K Professional systems, and no policy for everyone else, what do I have? That’s right. No communications will take place unless you’re an accountant using his or her policy-assigned client computer (or have physical access to an accountant’s computer). When an accountant uses his or her computer to access the accounting database (presuming that person has authentication and access permissions; remember, we’re not talking about replacing OS- or file-level protections, just securing communication on the wire), the conversation is secured.

To determine what we mean by “secured,” we’re going to have to dig beneath the initial pages of the policy. IPSec offers a variety of options. For a list see the sidebar, “How IPSec Protects Data Communications.” To see what default policies are set, let’s take a short tour.

First, open the Secure Server property pages by double clicking on this default policy. (See Figure 1.) The first thing to notice is the Rules tab. There are three rules displayed, and all three rules are checked. (An unchecked box would indicate a rule not used by the policy.) In an IPSec policy you can design rules for managing communications depending on the protocol used. Remember this well: Only the communications that have a policy set for them will be managed. The converse is also true: If you don’t have a policy for a communication protocol you’re using, yet you have a policy required, no communication can take place using that protocol.

Now, look at the rules set: one for “All IP Traffic,” another for “All ICMP Traffic,” and a third for . All IP traffic and all Internet Control Message Protocol (ICMP) traffic has specific rules defined, and one is a default rule to cover everything else. Incidentally, you can’t affect the application of rules by changing the order of the list. IPSec rules are all loaded at the same time and applied according to their settings. Placing the default rule first in the list and allowing it to pass traffic other than IP or ICMP wouldn’t allow unsecured IP traffic.

Open a rule by double-clicking. Rules contain a list of filers and filter actions. If a match with the filter is found, the action will occur.

The All IP Traffic rule list (see Figure 2) includes two filters. One, however—for All ICMP traffic—is unchecked, so it doesn’t apply. The other is set to match all IP packets. Open it and you’ll find that broadcast, multicast, Kerberos, RSVP and ISAKMP or IKE traffic are exceptions. I think you can probably agree that broadcast and multicast packets shouldn’t require negotiation. Kerberos is necessary if the server is going to authenticate the client. What about RSVP and ISAKMP? RSVP is the resource reservation protocol, used if you’ve established requirements for reserving bandwidth on your network. You’ll probably want that to apply so the default policy will allow it. ISAKMP or IKE (Internet Key Exchange) is used for key management. To encrypt communications between two computers, a shared key must be available to both client and server. IKE is used in the key management process. If this IP traffic were blocked, keys couldn’t be shared, and if encryption were required, no communication could take place.

Figure 2. The IP Filter List that allows you to designate what network traffic will be secured with a given rule.

Remember: This is a default policy; you can change it, but to do so requires understanding all of the protocols used on your systems, how IPSec works, and what the required policies are for communication with this computer on your network.

Double-click to open the filter within the filter list (there’s only one). Here’s where specific IP addresses for source and destination, protocols, and descriptions can be defined. Note that while the source address by default is set here for “My IP address” (the server’s address) and destination is set for “Any IP address” (the client), you can make choices that affect only a specific IP address or a specific IP subnet. A Protocol tab allows you to select protocol as well as the definition of from and to ports. To filter by protocol, you must know the required TCP and/or UDP ports for each protocol you wish to manage.

Other tabs on the rules property pages allow you to examine Filter Action, Authentication Methods, Tunnel Setting, and Connection Type.

Filter Actions tell you what happens if the rule applies (the packet meets the rule filter). For our rule only one is checked: “Require Security.” You can double-click this for further information. Security Methods in Figure 3 tells you that security is negotiated, 3DES (an encryption algorithm) is preferred, and that unsecured communications are accepted, but only responded to with IPSec.

Figure 3. The Security Methods dialog lets you know what happens if the rule filter applies.

So all communications between our accountant’s computer and the accounting database server will be encrypted, but the strength of the encryption will be negotiated. Just for fun, double-click on the 3DES security method (see Figure 4), then click the “custom” button.

Figure 4. The “custom” button lets you specify how to implement session keys.

Note that a new key is generated every 100,000 kilobytes and every 900 seconds. Geez… even if the attacker cracks the key, he’s got to do a lot of key cracking unless it’s an awfully short message. Are you smellin’ what I’m cookin’?

You’re right. IPSec has incredible possibilities for securing network communications, and I’m glad there are default policies to use, because it’s incredibly complex.

If you’re designing your own policy, you should note that three filter actions are possible: Permit (unsecured packets can pass), Request security, and Require security. Remember also to look underneath the filter action (double-click) and note the Security Method (Permit, Block, or negotiate security).

Authentication Methods allows the setting of an authentication method. Our default policy is set to use the standard Win2K authentication method, Kerberos, but you can select certificates or a preshared key. Of these, the preshared key choice is going to be the least secure. After all, selecting a preshared key exposes the key in the interface. That’s right. You enter it in clear text, and it shows up in the IPSec interface. It’s not exposed during network communications, but anyone who can bring up the IPSec policy can see it.

Tunnel setting is used to specify settings if an IPSec tunnel is required—not the case for our default policy.

Finally, Connection Type allows us to specify which connections are affected. The choices are all connections—only LAN connections or only remote access connections. You can choose only one of these for each rule, but you can create multiple rules. Our default policy, as you would expect, affects all network connections.

When you’re done checking out the default settings of the IP rule, make sure to cancel out of the property pages.

I’ll leave the exploration of the ICMP rule to you.

To implement this policy, assign the Secure Server policy on the accounting database server and assign the Client policy on the accountant’s computers. Do nothing on all other computers.

Scenario 2: Request Security Server/Respond Client to the Employee Database
The Server (Request Security) policy is the hardest for most people to understand. Why would you set this policy? Wouldn’t your security requirements be, either you need it or you don’t? Why set the server to request security and then say, oh, well, you can play anyway? Remember our goal here. We want to be able to secure communications, not just set a barrier at the front gate of the warehouse. It’s possible that you’ll want to pick and choose which conversation should be secured.

The entry of trucks into Fred’s warehouse must always be the result of negotiation (paperwork in order), but what about drivers entering to use the vending machines? It turns out Fred has different rules for his different “clients.” A driver (different from a truck client)—even a driver of a truck that’s not on the list—can enter the first section of the warehouse to use the phones, get a drink, or the like.

In the Server (Request Security) policy, it’s actually the client that’s requesting the security. The server is really responding to the client’s request for a secured conversation, not the other way around. When would this be useful? Suppose you have an application server that manages the employee database. As you can imagine, a large number of requests for information (a large number of changes to the database), takes place every day. If you required secured communication for all of this, you’d be adding a lot of overhead as well as more administration on every client computer. Do all communications need to be secured? If, under close inspection, you decide that only changes to employee benefits should be secured, this policy is for you. Requests for an employee phone number or supervisor name, for example, can be adequately controlled through database permission settings, and it’s unlikely you’d need to encrypt that data as it travels over the network.

To implement your policy, assign IPSec policy, “Server (Request Security)” on the employee database server and assign the “Client (Respond Only)” policy to the client systems used to make the employee benefit changes. For all other clients do nothing. The assigned client computers will communicate normally with all other servers, but when the employee database server requests it, the client will respond in kind. All communications between these client computers and the employee database are secured. None of the communication between other client computers and the database is.

A Last Bit of Advice

Rather than visit each approved client computer to assign the policy, I’d use Group Policy and establish a policy for all of the approved client computers. That way I can assign the policy once and avoid multiple trips to client computers to set up, troubleshoot, and maintain the policy. What’s that? You don’t know how to use Group Policy? That, I’m afraid, is a story for another day. You see, I’ve pestered Fred’s security company so much that they now want to see about offering computer security as a service to customers.