Thump. Thump. Is This Thing On?

Analyzing network traces may not the most exciting task, but it can be the fastest way to get to the root of a problem.

An Ethernet network is really nothing more than a big antenna connecting lots of little adapter transmitters, which—like thousands of tiny Howard Sterns all trying to talk at the same time—form the basis of Carrier Sense-Multiple Access/Collision Detect. For all its simplicity, a network is a nasty thing to troubleshoot. When a server problem arises that looks like it might be network-related, it’s not often that you hear someone say, “Here’s a tricky little problem. Let’s check a packet sniffer to see what’s happening.” More often you’ll hear, “Whoa, this looks really nasty. I sure hope we don’t have to break out the darned packet sniffer.”

Analyzing a network trace may seem like a tedious way to resolve a problem—something akin to going through a Warner Bros. cartoon one frame at a time trying to figure out how they get Bugs Bunny to sing opera—but it’s often the fastest way to find a solution. The key to making a sniffer useful is becoming familiar with standard transactions so you can recognize problems as they occur. In this column, I’ll trace a typical Windows network connection to provide an idea of what it should look like and why.

To do the traces in this column, I used an open-source product called Ethereal, available at http://www.ethereal.com. I prefer Ethereal to the Windows 2000 Network Monitor utility because of its ability to display all traffic seen by the adapter (“promiscuous mode” sniffing), as well as the sophistication of its parsers for analyzing traffic contents.

Header Information
A single transmission unit is called a frame. The first 14 bytes in the frame comprise the Ethernet header. The header has two 6-byte MAC addresses, one for the destination adapter and one for the source adapter, along with a 2-byte code indicating the payload type. 0x0800 designates an IP payload.

The next 20 bytes or so contain the Internet Protocol (IP) header. The first byte in the header is normally 0x45, indicating the IP version number (4) and the number of 4-byte sections in the header (5*4=20, the standard IP header length.) The data following the IP header is called a packet.

In addition to the source and destination host addresses, the IP header contains a Time to Live (TTL) value limiting the router hops a packet can accommodate (128 maximum), a Differentiated Services Field that defines acceptable handling characteristics by routers (described in RFC 2474), a checksum that verifies that the payload hasn’t been corrupted and a flag indicating if the payload has been fragmented to fit in the Ethernet frame. Some notorious perimeter intrusion exploits take advantage of IP packet fragmentation, so many firewalls reject IP packets with the fragmentation flag set.

Application Tracing
All we’ve seen so far is packaging, the electronic equivalent of airbills and foam containers. Now we’ve arrived at the true payload, and it’s time to start analyzing a transaction stream.

Figure 1 shows an Ethereal trace of a transaction involving a user who maps a drive to a server via the NET USE command. The 42 frames in this transaction zip by in just more than four-tenths of a second. (I’ve disabled NetBIOS over TCP/IP in this example to avoid extraneous broadcasts and name-resolution requests.) The user does a DIR of the mapped drive and then issues the NET USE * /d /y command to tear down the connection.

Here’s the cast of characters: W2KS-VM1 is a Win2K domain controller and DNS server with the address 192.168.0.100. W2KS-VM2 is a Win2K member server with the address 192.168.0.101. VM-PRO1 is a Win2K Professional workstation with the address 192.168.0.115. (The VM in each name stands for Virtual Machine to indicate that they’re actually VMWare sessions. If you haven’t worked with VMWare, run—don’t walk—to www.vmware.com to download a 30-day trial.)

Ethereal Transaction
Figure 1. The 42 frames of this top-level Ethereal transaction trace display in less than half a second. (Click image to view larger version.)

When the user issues the command “net use * \\w2ks-vm2\data,” here’s what happens:

Client: DNS Query—Give me an “A” record for W2ks-vm2.company. com.

The client converts the flat name W2KS-VM2 into a Fully Qualified DNS Name (FQDN) by appending the DNS suffix assigned to the computer when it joined the domain—in this case, company.com. The FQDN is included in an “A” record request sent to a DNS server.

A flag in the DNS query designates this as a recursive query, which tells the DNS server to go hunt down the resource record and deliver it back to the resolver. This is the electronic equivalent of Tony Soprano telling a lieutenant, “Take care of this little problem for me.”

DNS Server: DNS Reply—Here’s the resource record you requested.

The “A” record returned by the DNS server includes a TTL value that tells the resolver how long to cache the record. You can view the contents of the resolver cache using ipconfig /displaydns. The DNS server also sets a flag in the response telling the resolver that the server is authoritative for the designated zone, indicating that the resource record came from a physical zone file—not from the server’s cache.

Client: ARP Request—Give me the MAC address for 192.168.0.101.

The client needs the MAC address of the target server so it can buid an Ethernet frame. It obtains this address using Address Resolution Protocol (ARP). ARP consists of broadcasting the IP address, then waiting for the host to respond. If a router sits between the client and the host, the router responds to the ARP with its own MAC address. The MAC address is six octets: The first three are assigned by IEEE to the adapter vendor, and the second three are assigned sequentially and uniquely by the vendor. No two adapters should ever have the same MAC address; but some manufacturers get a little sloppy in their accounting practices, and accidents happen.

Server: ARP Reply—I’m 192.168. 0.101, and my MAC address is 00505640007a.

The target Ethernet adapter replies to the ARP request. The client caches the reply. If the address is not used again within two minutes, it’s removed from cache. If it’s reused, it remains in cache an additional 10 minutes. You can view the cache using arp -c.

Client: PING Request—Can you hear me, 192.168.0.101?

The Tcpip driver tests the result of the ARP by sending out an ICMP Echo Request, or ping. The Echo Request payload consists of letters of the alphabet. You can adjust the payload size by using ping –l . Older Windows stacks were susceptible to crashes if you overstuffed the receiving buffer. New stacks don’t have this bug, but Windows limits the maximum payload to 65,500 bytes, just in case.

Server: PING Reply—I can hear you.

Unless ICMP has been blocked by a firewall or client policy, the target adapter should respond with a copy of the payload in the Echo Request.

Client: TCP SYN—I’d like to connect to port 445.

When humans meet, they make eye contact, shake hands, exchange names, then alternately speak and listen—controlling the conversational flow with nuances of vocal inflection. TCP-based applications control connections using flags in the TCP header.

A client initiates a connection by sending a SYN flag. The server responds with SYN and ACK flags, and the client responds with a single ACK flag. During this transaction, the two entities exchange random 32-bit numbers that act as session identifiers. An entity acknowledging a transmission includes the last number assigned by its partner, incrementing the number by 1. This tells the receiver when a datagram arrives out of sequence or doesn’t arrive. This is how TCP guarantees in-sequence delivery of messages larger than a single datagram.

This particular connection request is directed at TCP port 445, the port used by Win2K for Server Message Block (SMB) transactions. This so-called “direct hosting” SMB port replaces the legacy port 139 Session Service used by earlier versions of Windows.

Server: TCP SYN-ACK—I’m waiting for your connection to port 445.

A SYN-ACK indicates that the server received the connection request and has allocated buffer space with the size indicated in the Window Size field. These buffer sizes are flexible, allowing the two machines to adjust to changes in network latency and packet loss. If the client doesn’t respond immediately, the server continues to send SYN-ACK replies up to the limit set in the Registry.

Client: TCP ACK—I’ve completed the connection to port 445.

Classic teardrop attacks and other Denial-of-Service exploits take advantage of this portion of the TCP handshake by sending bazillions of SYN requests to a port without responding to the resulting SYN-ACK or by directing the SYN-ACK to some innocent third party. This eventually causes the target to run out of memory and cease functioning. Win2K has special Registry entries that help protect against this type of attack. See Knowledge Base article 324270, “How to harden the TCP/IP Stack Against Denial of Service Attacks.”

Client: SMB Negotiate Protocol Request—What dialect of SMB should we use?

The SMB protocol has a history that goes back to the mid-1980s, with the older dialects considered as medieval as Chaucerian English. The client suggests a list of dialects (from oldest to newest) it supports.

SMB Negotiate Protocol Response—Let’s use version 0.12.

All modern Windows servers and clients use SMB version 0.12. This dialect also forms the basis of the Common Internet File System (CIFS) protocol that Microsoft recently published. Obtain a copy at http://download.microsoft.
com/download/win2000srv/protocol/0.6a/NT5XP/EN-US/cifs.exe
.

Client to Domain Controller: Kerberos Session Ticket Request—I need a session ticket for W2KS-VM2.

Kerberos clients are required to submit session tickets when connecting to Kerberos validating servers. To obtain this session ticket, the client validates its own identity using a preauthenticator, a timestamp encrypted with a session key extracted from the Ticket Granting Ticket (TGT) issued by the DC when the user first logged onto the domain.

Domain Controller: Kerberos Session Ticket Reply—Here’s your session ticket.

The Kerberos session ticket returned by the DC has two cipher portions: one encrypted with the user’s password hash, and one encrypted with the target server’s password hash. Both portions contain a copy of a session key that uses encrypt authenticators used in the next two packets.

Client to Server: Session Setup andX Request—I’d like to set up an SMB session.

Two Security Support Providers for Authentication
Figure 2. In the Session Setup SMB, the client offers two Security Support Providers (SSPs)—Kerberos and NTLM—for authentication. (Click image to view larger version.)

The “Function andX” structure of SMB requests hearkens back to the original days of NetBIOS, when network commands were stored in data structures called Network Control Blocks then plowed into the network controller via a hardware interrupt. The “andX” construct permits two NCBs to be transmitted in the same SMB.

In the Session Setup SMB, the client offers two Security Support Providers (SSPs) for authentication: Kerberos and NTLM. In case the target server prefers Kerberos, the client includes a copy of the session ticket and an authenticator.

Server: Session Setup andX Response—I’ve established an SMB session for you.

The reply contains a Kerberos authenticator for the client to validate the target server’s identity. This prevents man-in-the-middle attacks. The reply also contains a User ID that acts as a thread for future SMB communications over this connection. The SMB header contains a Multiplex ID that pairs the response to the original request, which permits SMB partners to exchange requests and replies out of sequence in the same session.

Client: Tree Connect andX Request—I’d like to connect to the IPC$ share.

The client makes an initial SMB connection to the Inter-Process Communication service stub, which takes the form of a hidden share called IPC$. For an in-depth look at this and similar transactions, get the Macmillan Technical Publishing book DCE/RPC over SMB, by Luke Kenneth Casson Leighton.

Server: Tree Connect andX Response—I’ve set up a Tree connection.

The server assigns a Tree ID that the client uses in subsequent transactions to the IPC interface.

Client: Transaction 2 Request—Give me a referral for the \Data Dfs link point.

Win2K and Windows XP clients always start by assuming that a share is a Dfs link point. Older clients assume that a share is a share and only send a Dfs referral request if the initial share connection fails.

Server: Transaction 2 Response—Nope, \Data is not a Dfs link point.

Unabashed, the client obtains a fresh Kerberos session ticket and initiates another SMB session to connect to the Data share. With that done, the sequence continues.

Client: Tree Connect andX Request—I’d like to connect to the \Data share.

If you look at the TCP header at this point, you’ll see that the client continues to use the original TCP connection (as shown by the session and acknowledgement numbers) with the PSH and ACK flags set, indicating that the client has received the last datagram and is sending a new one. The ability to piggyback acknowledgements with later transmissions reduces network traffic.

Server: Tree Connect andX Response—I’ve established a tree connect for you.

The server includes a Tree ID for the client to use when communicating with this share in the future. This acts as a “handle” to the share point.

Client: TCP ACK—Thanks.

Client: Create andX Request—I’d like to open the Desktop.ini file inside Data.

A Create andX SMB tells a server to open a designated file and return a file handle. The request includes flags to indicate the type of access required by the client. In this case, the Read flags are set. Another set of flags is used to request an opportunistic lock (oplock). “Opportunistic” means the server is free to hold off on the request until it can lock the file. If the client doesn’t free the oplock, the only way to break it is to tear down the entire session.

Server: Create andX Response—Sorry. Desktop.ini doesn’t exist.

Client: Create andX Request— Are you sure? Try again.

Server: Create andX Response—Yup. I’m sure. It’s not there.

Client: Transaction2 Request—Well, maybe \Data really is a Dfs link. Please check again.

Server: Transaction2 Reply—Nope, \Data really isn’t a Dfs link.

The network client now tries the whole Tree Connect sequence over again just to make sure there hasn’t been a mistake. When it’s satisfied that \Data really is a share, the user sees a success message for the drive mapping. In Win2K, the next available drive letter is assigned to the connection. In Windows .NET and XP, the last available drive letter is assigned.

Using a Mapped Drive
Let’s say that the user now changes to the mapped drive and issues a DIR command. Here’s what happens.

Client: Transaction2 Request—Give me the property information on the share.

The Transaction2 SMB has a variety of options for checking the status of a network file system. The Subcommand field contains a code that defines the option. In this case, option 0x003 designates QUERY_FS_VOLUME_INFO. The CIFS documentation from Microsoft lists the available options.

Each Transaction2 SMB option has a set of suboptions that Microsoft calls InformationLevel and Ethereal displays as “Level of Interest.” The text entry in Ethereal corresponds to the option number. In this case, option 0x102 corresponds to SMB_QUERY_FS_VOLUME_ INFO, which Ethereal displays as Query FS Volume Info.

Server: Transaction2 Response—Here’s your volume info.

The response includes the volume name, the volume’s eight-digit serial number and the volume’s creation date.

Client: Transaction2 Request—What are the volume attributes?

This is QUERY_FS_VOLUME_INFO option 0x105.

Server: Transaction2 Response—The volume uses FAT.

The response also includes the maximum file name length, which is 255 characters when connecting to NT or Win2K servers. This limit also applies to path lengths, a fact that’s often used to hide files by burying them deep inside a set of long folder names.

The server returns attributes for the share. In this case, the attributes indicate that case-in-file names are preserved, along with persistent changes to ACLs. This appears to be a bug, as FAT doesn’t contain ACLs.

Client: Transaction2 Request—What are the basic attributes for this file object?

Server: Transaction2 Reply—It’s a folder.

The response includes the creation date, last access date, last write date and last modification date. This information is displayed to the user in the console.

Client: Transaction2 Request—Double check the volume attribute.

The client repeats the request for QUERY_FS_VOLUME_INFO, option 0x102.

Server: Transaction2 Reply—Repeats response.

The server sends another response with the volume name, the eight-digit volume serial number and the volume creation date.

Client: Transaction2 Request—Enumerate the folder contents.

Classic DOS/Windows enumerates a directory tree with the FindFirstFile and FindNextFile function calls. You’ll sometimes hear this called FindFirst/FindNext. The associated SMB commands are FIND_FIRST2 and FIND_NEXT2.

The FIND_FIRST2 command searches a directory to find files and folders that match a specified pattern, such as *.txt. Unlike the FindFirstFile function call, which only returns simple file names, the FIND_FIRST2 SMB returns a list of file names and their attributes. A flag in the request determines how many attributes to include in the list.

Server: Transaction2 Reply—Here’s a list of the items in the folder.

Even if a folder is empty, it will have the dot (self) and dot-dot (parent) folders.

The amount of information that can be returned by the server in response to FIND_FIRST2 is limited to the maximum TCP datagram size, which is 16K or 16,384 bytes in Win2K. If there’s more to send after filling up the FIND_FIRST2 reply, the client sends a FIND_NEXT2 SMB to obtain another 16K of information.

Client: Transaction2 Request—How much space is left in the volume?

This SMB was sent because the DIR command needs to display space used by the listed files and total free space.

Server: Transaction2 Reply—Here’s the information you requested.

The response includes the total clusters (allocation units) in the volume behind the share and the number of free clusters. It also lists the number of sectors per cluster and the number of bytes per sector. The application does the math and displays the result.

TCP ACK—Thanks.

The user now issues a NET USE * /d /y command to break the connection.

Client: Tree Disconnect Request—I’m done with Tree ID 4102.

Server: Tree Disconnect Reply—I’ve torn down Tree ID 4102.

Client: Logoff andX Request— I’m done with Session ID 6146.

Server: Logoff andX Reply— I’ve torn down Session ID 6146.

Client: TCP FIN—I’m done with this TCP session.

Server: TCP FIN-ACK—I’ve torn down the TCP session.

Client: TCP ACK (from client)—So long.

If the user simply turns off the computer without gracefully exiting the connection, it eventually times out and the server tears down the session and sends a TCP Reset (RST) to the client.

Wrapping It Up
This analysis of a standard SMB connection only scratches the surface of the information that can be obtained using a packet sniffer. In production, you’ll probably need to reconfigure switches to permit promiscuous sniffing and to configure filters to help isolate transactions from the background streams. Once you get this configuration in place, you have what amounts to a fluoroscope of your system. If you find time to go through a few traces, you can get a feel for what standard and non-standard transactions look like. It might not be exciting work, but it beats spending the afternoon reading your company’s cafeteria plan documentation.

Featured