Disk drives are the slowest components of your system. Using a disk defrag tool can help them deliver optimum performance.

Defragmenting Devices

Disk drives are the slowest components of your system. Using a disk defrag tool can help them deliver optimum performance.

• By Michael Chacon
• August 01, 1999

Last month I presented an overview of how NTFS manages files in partitions. Because of the cluster-based allocation process, adding and deleting files to and from an NTFS partition causes them to become fragmented over time. Now that you better understand how the file system functions, I’ll introduce utilities that work with NTFS to bring these fragments together into, ideally, contiguous streams.

Tools for the Toolbox

Several utility programs provide disk defragmentation services, most notably Norton Speed Disk (part of Symantec’s Norton Utilities for NT) and Executive Software’s Diskeeper. I know people who use these and other products, and most are happy with what they have—so you have a good selection to choose from. Personally, I use Diskeeper, so I’ll use it here as an example of how the defragmentation process works.

One reason I like Diskeeper is because it was developed for the VAX VMS operating system more than a decade ago. Windows NT has a close historical and architectural relationship with VMS (David Cutler was deeply involved in the development of both OSes). Contributing to this historical relationship, Executive Software helped develop some of the components of the NTFS implementation, and Diskeeper was built upon these components. Of course, these components are also available to other disk management utility companies that use them to provide defragmentation services. You may also be aware that Windows 2000 will contain a manual version of Diskeeper. The full Diskeeper add-on will continue to be available to run automated defragmentations. At any rate, let’s take a deeper look.

As you may recall from last month, fragmentation is a natural process with both the FAT and NTFS file systems. Figure 1 shows an image of a partition as seen through Diskeeper. Other disk defragmentation utilities display similar information.

Figure 1. Diskeeper's main interface displays a color-coded image of a partition.

Based on the color-coded legend and the color display, you can immediately see fragmented files and randomly distributed directories on this partition. Although this provides a general overview of the partition condition, you’ll need to use the text mode window to view specific information (see Figure 2).

Figure 2. To view specific information about your partition, you'll need to use Diskeeper's text mode.

This textual information gives us a better idea of the condition of this drive. We can see that it’s mildly fragmented from a file perspective. Regardless of how mild or severe the file fragmentation, the process of removing that fragmentation is similar. Diskeeper works on each file individually. Because it doesn’t keep a scratch file for the process, Diskeeper needs only enough contiguous disk space for any two given fragments. The process is always concerned with creating one fragment out of the two. It will then take this new fragment and write it to another contiguous space on the partition. If there are multiple fragments for one file, the process will continue until the file is completely contiguous or there isn’t enough contiguous space left on the partition to hold the complete file. The benefit of this linear fragment-by-fragment process is that even if a file can’t be made completely contiguous, Diskeeper can still reduce the fragmentation. For example, a file in 12 fragments might only be reduced to four. Diskeeper accepts this best effort and then moves on to the next file. Although this isn’t the optimum state, it’s still an improvement over the original condition of the file.

The Trouble with Directories

One factor that commonly prevents the complete defragmentation of files is the presence of directories. As you can see from Figure 1 and verify from the text window in Figure 2, the directories on this drive are scattered across the partition. This raises an important aspect of NTFS partitions: Directories are files with special attributes that include all the file names. If those file names consume more than the 1K record size in the Master File Table (MFT), one or more runs will be created to contain the rest of the names. These runs will be written to random locations in the partition, as are all files. However, directories can’t be defragmented or moved while the system is accessing the partition. So the random location of static directories means that there will be fewer available areas of free space on the partition. As new files and runs are created, it’s more likely that they’ll also become fragmented immediately as they bump into the directories. Figure 3 shows the partition after the normal defragmentation process.

Figure 3. The normal defragmentation process eliminates the fragmentation, but the directories remain.

As you can see, the file fragmentation is gone, but the directories represented by the light blue lines remain. This means that as new files are created or existing ones grow, they’ll become fragmented almost immediately. The solution is to move the directories to one location on the disk periodically when the system isn’t accessing the partition, such as during the boot process. With Diskeeper, this is accomplished by running a boot-time defragmentation process from the Advanced Tools menu option. This configures the Registry setting to run the directory defragmentation during the boot process. One other function available when running in this mode is to defrag the paging file, which again is something you can’t do when NT is actually running.

During the boot process, the MFT length and offset is determined and then the Logical Cluster Numbers (LCNs) are remapped to the directories as they’re moved to a contiguous area of the partition.

You may notice that, although the program reports zero percent fragmentation, all the files and directories aren’t completely consolidated. File consolidation doesn’t necessarily improve performance (see Figure 4).

Figure 4. After running the boot-time process, the program reports zero percent fragmentation, but all files and directories still aren't completely consolidated.

There’s a distinction between consolidated files and individually contiguous files. It’s the contiguous characteristic of each individual file that’s important, not how they might all be neatly organized at the beginning of the partition. Files on a disk usually aren’t read from the disk in order, particularly on a server. Records in a database are frequently read in order, but they’re usually contained within a database that the file system considers an individual file, such as Microsoft SQL Server or Exchange. Diskeeper writes bit for bit to the new location and doesn’t alter the contents of the file in any manner other than the LCN attribute mappings. Therefore, Diskeeper doesn’t address internal record fragmentation in databases used by applications such as SQL Server or Exchange. Tools that do address internal fragmentation, such as EBDUTIL for Exchange, copy the entire database file to a new location. If there isn’t enough free partition space to contain the copy of the database file, you’ll get cluster fragmentation.

Defragmenting offline is a good idea before you run these database tools. Remember, directories aren’t consolidated to make them read faster. They’re consolidated because they can’t be defragmented when the system is up, and they can block the contiguous runs of files as they are created or grow during use.

Keep Your Drive Slim and Trim

Although a defrag program is essential for network administrators, one of the best things you can do for a hard drive is simply not fill it up completely with data. Microsoft recommends that you always keep 20 percent of the disk free so the file system can breathe. If you surpass this, NTFS will spend a significant amount of access time looking for free space and spreading new or growing files across the partition, diminishing performance. When a new NTFS partition is created, 12.5 percent of the partition is “reserved” for the MFT. It’s important to understand that this space isn’t preallocated, and it can be used by NTFS for attribute runs instead of MFT records if there’s no other space on the partition to hold them. This means that when the MFT needs to add a new record to store attributes for a new file, the records will be stored in a different area in the partition and the MFT will become fragmented (see Figure 5). 

Figure 5. When the MFT needs to add a new record to store attributes for a new file, the records will be stored in a different area in the partition, causing the MFT to become fragmented.

Once an MFT becomes as badly fragmented as shown in Figure 5, the ballgame is over. There’s no way to defragment the MFT other than reformatting the partition and restoring the files from a backup device. In the worst case, if the MFT on the system partition is highly fragmented, NT may not even be able to boot at all because the bootstrap code can’t locate the NTLDR file.

Discipline is Key

The best method of dealing with defragmentation is to run the process on a regular basis. In fact, Diskeeper recommends running the defragmentation process continuously. Diskeeper is designed as an ongoing process rather than as a tool to be used only when fixing a problem. You want to prevent the problem in the first place. The process is scheduled with a low priority so that it will acquiesce to other processes and run only while the system is idle. This keeps the partitions neatly trimmed and hedged. Users can access most of the files concurrently with Diskeeper. Each file is locked only for few milliseconds at the end of processing. If the user’s application tries to write to a file that Diskeeper is writing, the request is suspended until the file is released, and then the request is serviced. If a file has been exclusively locked by the application, Diskeeper will skip that file and return to deal with it on the next pass.

Remember, as directories are created they create walls that are barriers to contiguous space. Check your partitions for directory fragmentation on a regular basis and, when you have an opportunity, reboot your system and run Diskeeper to consolidate your directories. Your disk drives are the slowest component of your system. Keeping them in a contiguous state will help them deliver optimum performance.