Implementing this emerging data communication security standard can tighten up your network. Here’s how its default policies work.
IPSec, Your Private Communications Security Cop
Implementing this emerging data communication security standard can tighten up your network. Here’s how its default policies work.
- By Roberta Bragg
- August 01, 2000
My teenage son Fred (all names have been changed to protect
the innocent) recently got a job as a security guard.
It’s not enough that I have to worry about his flaming
out in a spectacular automobile wreck or dying of alcohol
poisoning, now I have to worry about his being gunned
down while he’s protecting the secret warehouse of a national
retail chain. (The location of the warehouse has been
secret since a whole truck of Beanie babies or Furbies
or something got stolen a couple of Christmases ago.)
Never one to just sit back and imagine the horrors he
might encounter, I’ve been learning what he and his buddies
(they call themselves the “private security enforcement
cops”) do. Turns out, their job is much like an IP Security
(IPSec) agent that enforces the IPSec policies you can
implement in Windows 2000. Let me explain.
IPSec is an emerging data communication security standard.
You may have used it yourself to implement a Virtual Private
Network (VPN) between two routers. Win2K can use IPSec
to secure communications between computers in your network
and beyond. It can also be used to encrypt the data in
an L2TP tunnel in a Win2K VPN.
Understanding what IPSec is, however, and how to implement
it isn’t something we can cover in 3,000 words or less.
There are four natural ways to divide and conquer the
- An intimate discussion of the protocol itself.
- IPSec combined with L2TP for VPN.
- IPSec default policies.
- IPSec custom policies.
Since this is my first column for you on IPSec, I’ll
focus on a discussion of the default policies. I’ll use
these to help you understand the possibilities that IPSec
presents, and show you how to incorporate IPSec policies
for greater network security—without becoming an IPSec
guru. I’ll leave the other topics for future columns.
To use IPSec, you don’t have to understand a great deal
about the protocol, but you do have to know the default
IPSec policies that are available, what they do, and how
to use them. In order to fully use the power of IPSec
and troubleshoot it when things go wrong, you need to
invest some time in understanding this complex protocol
and how it’s implemented in Win2K. To maximize your effort
and begin securing network communications faster, study
the default polices first. An understanding here will
allow you to lock down a server and require encrypted
communications, allow encrypted communications with a
client, or allow the server to act as a client and negotiate
encrypted communications with another server.
Please note that I didn’t say to blindly implement these
policies on your network. You need to test them in a controlled
environment to make sure you understand them. Also, when
exploring, you should always exit each policy page by
using the Cancel button. Never, ever “Assign” a policy
until you understand what it will do. It’s possible to
unwittingly create your own “Denial of Service” attack
using default policies. On the other hand, by modifying
these policies you can intricately tune your policies
at the protocol level so that nothing travels on the wire
to or from your systems that you don’t control. (Of course,
if you’re not careful, the control can turn absolute;
then nothing will travel on the wire.)
What can IPSec do for your network communications? IPSec
can be configured to manage authentication, authorization,
and encryption. It also has a tunnel mode. When we talk
about secured communications, we’re talking about implementing
some or all of these features.
No Data Transmission Security
When you’re driving, anyone can look in and see who’s
in your car. We don’t get to travel incognito in Wonder
Woman’s invisible plane. It’s like this on a Win2K network.
By default, there’s no security established for data communications.
Oh, users have to be authenticated before they can use
the systems, and their access is authorized or not, but
once the connection is made and authorized, most data
travels from computer to computer without the benefits
of encryption. Moreover, unless some kind of port filtering
or a firewall has been configured to protect it, any available
type of data can be transmitted.
If you want to change that picture, follow Fred and me
as we discover how to establish default policies.
Fred’s First Day: Default Policies Apply
When Fred started his new job, he put on his uniform
(Oh, my! He looks just like a real cop from a distance),
drove south, and reported for work. Turns out he got guard
duty in a shack outside the truck entrance to the warehouse.
It took the boss all of five minutes to give him the low-down:
- Rule 1: If the truck’s
paperwork is in order and matches your list here, send
’em on in.
- Rule 2: If the truck’s
paperwork isn’t in order, make them park and wait. Pass
the information on. Something has to be cleared up.
- Rule 3: If they’re not
on your list, send ’em packing.
Fred was given the established policy, he had his orders,
and that’s all there was to it. The policy was created
at headquarters and pushed out across the company’s many
locations. Just like Win2K Group Policy: Once set, everyone
follows the lead.
When you start working with IPSec, you’re almost as lucky.
IPSec policy isn’t turned on by default, but default policies
exist. Turn them on (or “assign them,” in IPSec policy
lingo) and they’ll work.
But hold on—don’t turn on these policies until you understand
how they work; you don’t want your own domain-wide Denial
of Service attack. IPSec policies are that powerful. They’re
designed to control communications between computers.
Set the wrong policy at one computer and not at another
and it’s possible to shut down communications. Let the
brave and the stupid beware: Use Group Policy to establish
your control on multiple computers and you might be looking
for another job—say, one that’s a tad less technical.
So how do you properly set default IPSec policies? They’re
not at some secret location—in fact, you’ve probably passed
them a zillion times on your way to some other policy
implementation. But halt! Remember to play with IPSec
policies on a test computer. For our purposes, select
the Local Security Policy from the Administrative Tools
Menu. Navigate in the Tree pane to and select “IP Security
Policies on Local Machine.” In the Scope pane you’ll see
three default policies (see Figure 1).
- Client (Respond Only)
- Secure Server (Require Security)
- Server (Request Security)
|Figure 1. In your tour of the
default policy, you’ll find three rules displayed,
which allow you to define what type of traffic is
You should also see a description and a column designated
“Policy Assigned.” (Yours should all say, “No,” unless
they’ve been implemented previously.)
Before you investigate the property pages of these policies,
right-click on one and note the “Assign” selection. Default
IPSec policies are present, as you can see, but they’re
not in effect until they’re assigned. Remember this. When
you create your own IPSec Policy, it won’t be used until
it’s assigned. This is different from the normal group
policy application. If I give you a new right on this
system or change the password policy at a domain controller,
the policy will be implemented at the next policy refresh.
An IPSec policy won’t be, unless it’s been assigned.
To implement any of the default policies on a computer,
select “Assign” from this drop-down list and wait for
or request a policy refresh. You don’t need to make any
additional settings. The following discussion is simply
for the purposes of allowing you to learn about IPSec
and when to implement default policies, not as a tutorial
for how to configure them. To remove any of the default
policy implementations, select “unassigned” from this
drop-down list and wait for or request a policy refresh.
If you’ve implemented IPSec Group Policy at a higher level,
remember: the usual Group Policy inheritance rules apply.
Fred’s default policy was simple to understand, and these
are too. Double-click on the policy and click the General
tab to get a helpful description.
The Client (Respond Only) policy allows the systems that
it’s set on to operate in the normal manner for all communication,
unless a server requests secured communications. Then
the client can participate in negotiating communications.
Notice the word “negotiate.” Just because the client system
is set to respond to a server’s request for secure communications
doesn’t mean it will be able to communicate. Client and
server must be able to agree on the security that’s enabled,
and what types of communication are allowed. Further,
only the type of traffic the server wants to be secured
will be. Other types of communication between client and
server might take place with no security enabled or might
not take place at all. Remember, policy is set on two
machines—server and client—and they must be able to agree
on the particulars in order for communication to occur.
(Imagine a truck that doesn’t have its paperwork in order.
For the truck whose papers were correct, entry was swift.
For the one that wasn’t, negotiation could take awhile.
It might even result in no entry at all.)
The Secure Server (Require Security) policy requires
the server to insist on a negotiated conversation. Moreover,
every communication is based on first establishing that
the client is trusted. That is, a Kerberos trust is required.
No unsecured communication at all is allowed with untrusted
clients. So what happens if your client has no IPSec policy
set and your server does? The server will listen to your
request, but respond only by requesting negotiated communications
using IPSec. Even if the client computer is joined in
the domain (trusted), since no policy has been set on
the client, the client can’t respond. In the case of failed
negotiation, further communication isn’t allowed. If we
think of Fred’s guardpost as the server and a truck full
of merchandise as the client, it’s as if Fred asked the
truck driver at the gate for his paperwork, and the truck
driver had none. It really doesn’t matter to Fred whether
or not the truck is on his list. No paperwork, no entry—that’s
By now you can probably guess what the server (Request
Security) policy states: The server will always request
security, but if the client doesn’t respond in kind, he
still gets to drive his truck through the gate. (Fred
would get fired for that one.) This policy is the hardest
to understand. I’ll explain more on that shortly.
How Policies Work Together
Remember that policies require negotiation. It takes
two computers to play. Let’s see how these policies can
work together on our network.
Scenario One: Secure Server/Accountant
to Accounting Database
Our first scenario is easy to understand. If I set Secure
Server (Require Security) for my accounting database server,
Client (Respond Only) for all accountants’ Win2K Professional
systems, and no policy for everyone else, what do I have?
That’s right. No communications will take place unless
you’re an accountant using his or her policy-assigned
client computer (or have physical access to an accountant’s
computer). When an accountant uses his or her computer
to access the accounting database (presuming that person
has authentication and access permissions; remember, we’re
not talking about replacing OS- or file-level protections,
just securing communication on the wire), the conversation
To determine what we mean by “secured,” we’re going to
have to dig beneath the initial pages of the policy. IPSec
offers a variety of options. For a list see the sidebar,
“How IPSec Protects Data Communications.” To see what
default policies are set, let’s take a short tour.
First, open the Secure Server property pages by double
clicking on this default policy. (See Figure 1.) The first
thing to notice is the Rules tab. There are three rules
displayed, and all three rules are checked. (An unchecked
box would indicate a rule not used by the policy.) In
an IPSec policy you can design rules for managing communications
depending on the protocol used. Remember this well: Only
the communications that have a policy set for them will
be managed. The converse is also true: If you don’t have
a policy for a communication protocol you’re using, yet
you have a policy required, no communication can take
place using that protocol.
Now, look at the rules set: one for “All IP Traffic,”
another for “All ICMP Traffic,” and a third for .
All IP traffic and all Internet Control Message Protocol
(ICMP) traffic has specific rules defined, and one is
a default rule to cover everything else. Incidentally,
you can’t affect the application of rules by changing
the order of the list. IPSec rules are all loaded at the
same time and applied according to their settings. Placing
the default rule first in the list and allowing it to
pass traffic other than IP or ICMP wouldn’t allow unsecured
Open a rule by double-clicking. Rules contain a list
of filers and filter actions. If a match with the filter
is found, the action will occur.
The All IP Traffic rule list (see Figure 2) includes
two filters. One, however—for All ICMP traffic—is unchecked,
so it doesn’t apply. The other is set to match all IP
packets. Open it and you’ll find that broadcast, multicast,
Kerberos, RSVP and ISAKMP or IKE traffic are exceptions.
I think you can probably agree that broadcast and multicast
packets shouldn’t require negotiation. Kerberos is necessary
if the server is going to authenticate the client. What
about RSVP and ISAKMP? RSVP is the resource reservation
protocol, used if you’ve established requirements for
reserving bandwidth on your network. You’ll probably want
that to apply so the default policy will allow it. ISAKMP
or IKE (Internet Key Exchange) is used for key management.
To encrypt communications between two computers, a shared
key must be available to both client and server. IKE is
used in the key management process. If this IP traffic
were blocked, keys couldn’t be shared, and if encryption
were required, no communication could take place.
|Figure 2. The IP Filter List
that allows you to designate what network traffic
will be secured with a given rule.
Remember: This is a default policy; you can change it,
but to do so requires understanding all of the protocols
used on your systems, how IPSec works, and what the required
policies are for communication with this computer on your
Double-click to open the filter within the filter list
(there’s only one). Here’s where specific IP addresses
for source and destination, protocols, and descriptions
can be defined. Note that while the source address by
default is set here for “My IP address” (the server’s
address) and destination is set for “Any IP address” (the
client), you can make choices that affect only a specific
IP address or a specific IP subnet. A Protocol tab allows
you to select protocol as well as the definition of from
and to ports. To filter by protocol, you must know the
required TCP and/or UDP ports for each protocol you wish
Other tabs on the rules property pages allow you to examine
Filter Action, Authentication Methods, Tunnel Setting,
and Connection Type.
Filter Actions tell you what happens if the rule applies
(the packet meets the rule filter). For our rule only
one is checked: “Require Security.” You can double-click
this for further information. Security Methods in Figure
3 tells you that security is negotiated, 3DES (an encryption
algorithm) is preferred, and that unsecured communications
are accepted, but only responded to with IPSec.
|Figure 3. The Security Methods
dialog lets you know what happens if the rule filter
So all communications between our accountant’s computer
and the accounting database server will be encrypted,
but the strength of the encryption will be negotiated.
Just for fun, double-click on the 3DES security method
(see Figure 4), then click the “custom” button.
|Figure 4. The “custom” button
lets you specify how to implement session keys.
Note that a new key is generated every 100,000 kilobytes
and every 900 seconds. Geez… even if the attacker cracks
the key, he’s got to do a lot of key cracking unless it’s
an awfully short message. Are you smellin’ what I’m cookin’?
You’re right. IPSec has incredible possibilities for
securing network communications, and I’m glad there are
default policies to use, because it’s incredibly complex.
If you’re designing your own policy, you should note
that three filter actions are possible: Permit (unsecured
packets can pass), Request security, and Require security.
Remember also to look underneath the filter action (double-click)
and note the Security Method (Permit, Block, or negotiate
IPSec Protects Data Communications
IPSec provides cryptography-based protection.
A selection of algorithms can be used.
Keys are generated at the time of use
and can be changed with alarming frequency.
Key generation and distribution is automatic.
Keys can be dynamically rekeyed during
a conversation. This means an attacker
who obtains a compromised key doesn’t
acquire one that unlocks the entire
Multiple security benefits are provided:
- Integrity. Data is protected from
modification in transit. Individual
packets are signed and checked before
being used by the receiving computer.
A changed packet therefore can be
discarded. What you send is what you
- Authentication. This isn’t a substitute
for authentication of the user to
the computer but rather authenticates
from computer to computer. It proves
the origin of the data. It assures
the identity of the computer that
is sending the packets. By varying
the authentication methods allowed,
computers that understand IPSec but
that aren’t Win2K systems can be allowed
to converse with Win2K systems. Each
computer knows to whom it’s talking.
- Confidentiality is provided via
encryption. Only the desired system
gets to read the message. Even if
the packets are captured, the data
they contain is unreadable.
- Non-repudiation ensures that the
only possible sender of the message
is the one identified. There’s no
way the sender can deny that he sent
- Anti-replay or replay prevention
guarantees that a captured message
can’t be used later by an attacker
to establish a session.
Authentication Methods allows the setting of an authentication
method. Our default policy is set to use the standard
Win2K authentication method, Kerberos, but you can select
certificates or a preshared key. Of these, the preshared
key choice is going to be the least secure. After all,
selecting a preshared key exposes the key in the interface.
That’s right. You enter it in clear text, and it shows
up in the IPSec interface. It’s not exposed during network
communications, but anyone who can bring up the IPSec
policy can see it.
Tunnel setting is used to specify settings if an IPSec
tunnel is required—not the case for our default policy.
Finally, Connection Type allows us to specify which connections
are affected. The choices are all connections—only LAN
connections or only remote access connections. You can
choose only one of these for each rule, but you can create
multiple rules. Our default policy, as you would expect,
affects all network connections.
When you’re done checking out the default settings of
the IP rule, make sure to cancel out of the property pages.
I’ll leave the exploration of the ICMP rule to you.
To implement this policy, assign the Secure Server policy
on the accounting database server and assign the Client
policy on the accountant’s computers. Do nothing on all
Scenario 2: Request Security Server/Respond
Client to the Employee Database
The Server (Request Security) policy is the hardest for
most people to understand. Why would you set this policy?
Wouldn’t your security requirements be, either you need
it or you don’t? Why set the server to request security
and then say, oh, well, you can play anyway? Remember
our goal here. We want to be able to secure communications,
not just set a barrier at the front gate of the warehouse.
It’s possible that you’ll want to pick and choose which
conversation should be secured.
The entry of trucks into Fred’s warehouse must always
be the result of negotiation (paperwork in order), but
what about drivers entering to use the vending machines?
It turns out Fred has different rules for his different
“clients.” A driver (different from a truck client)—even
a driver of a truck that’s not on the list—can enter the
first section of the warehouse to use the phones, get
a drink, or the like.
In the Server (Request Security) policy, it’s actually
the client that’s requesting the security. The server
is really responding to the client’s request for a secured
conversation, not the other way around. When would this
be useful? Suppose you have an application server that
manages the employee database. As you can imagine, a large
number of requests for information (a large number of
changes to the database), takes place every day. If you
required secured communication for all of this, you’d
be adding a lot of overhead as well as more administration
on every client computer. Do all communications need to
be secured? If, under close inspection, you decide that
only changes to employee benefits should be secured, this
policy is for you. Requests for an employee phone number
or supervisor name, for example, can be adequately controlled
through database permission settings, and it’s unlikely
you’d need to encrypt that data as it travels over the
To implement your policy, assign IPSec policy, “Server
(Request Security)” on the employee database server and
assign the “Client (Respond Only)” policy to the client
systems used to make the employee benefit changes. For
all other clients do nothing. The assigned client computers
will communicate normally with all other servers, but
when the employee database server requests it, the client
will respond in kind. All communications between these
client computers and the employee database are secured.
None of the communication between other client computers
and the database is.
To read the current IPSec RFC specifications,
rfcsearch.html. Be sure to read
them all. The current specifications
- 2085, HMAC-MD5 IP Authentication
with Replay Prevention
- 2104, HMAC: Keyed Hashing for Message
- 2403, The use of HMAC-MD5-96 within
ESP and AH
- 2404, The Use of HMAC-SHA-1-96 within
ESP and AH
- 2401, Security Architecture for
the Internet Protocol
- 2402, IP Authentication Header (AH)
- 2405, The ESP DES-CBC Cipher Algorithm
with Explicit IV
- 2406, IP Encapsulating Security
- 2407, The Internet IP Security
Domain of Interpretation for ISAKMP
- 2410, The NULL Encryption Algorithm
and Its Use with IPSec
- 2411, The IPSec Document Roadmap
- 2451, The ESP CBC-Mode Cipher Algorithms
The sheer number of RFCs should clue
you into something; this isn’t a subject
that you’re going to comprehend all
of the ramifications of in one quick
Excellent white papers exist on Microsoft’s
You’ll find several sections of the
Windows 2000 Server Resource Kit of
- In the TCP/IP Core Networking Guide,
Chapter 8, “Internet Protocol Security.”
- In the Distributed Systems Guide,
Chapter 14, “Cryptography for Network
and Information Security.”
- In the Deployment Guide, Chapter
11, “Planning Distributed Security.”
Last, I cover the topic in Chapter
15 of my new book, MCSE Windows 2000
Network Security Design (New Riders,
ISBN 0-73570-984-X, $49.99).
A Last Bit of Advice
Rather than visit each approved client computer to assign
the policy, I’d use Group Policy and establish a policy
for all of the approved client computers. That way I can
assign the policy once and avoid multiple trips to client
computers to set up, troubleshoot, and maintain the policy.
What’s that? You don’t know how to use Group Policy? That,
I’m afraid, is a story for another day. You see, I’ve
pestered Fred’s security company so much that they now
want to see about offering computer security as a service