In-Depth

Exchange Storage Rules: 15 Ways to Simplify, Solidify and Save

Exchange Server 2003 offers advances in scalability, allowing for dramatic server and storage consolidation efforts—if it's properly configured. These best practices will send you in the right direction.

• By Doug Barney
• April 01, 2005

Exchange Server 2003 offers IT professionals major advantages. Not only is it far more secure than its predecessors, thanks to Microsoft's Trustworthy Computing initiative, but it also offers advances in scalability, allowing for dramatic server and storage consolidation efforts.

IT departments should plan carefully for Exchange 2003, and create a forward-thinking plan that includes fewer (but higher-powered) servers, a simpler network infrastructure, and a more rational, clear and ultimately more cost- effective approach to storage. Following are 15 configuration tips culled from nearly two dozen white papers, reams of case studies, several books on storage and Exchange, Microsoft's own internal best practices, as well as the author's experience as a former storage analyst for Nucleus Research.

Getting Started
1. Let's Get Hierarchical
A major upgrade like Exchange 2003 allows IT to take a fresh look at other network elements, including taking a novel approach to storage. Information Life Cycle Management (ILM) concepts can shape your Exchange storage infrastructure in a positive way. Similar to hierarchical or tiered storage, ILM is concerned with the entire life cycle of data, from date of creation to its ultimate disposal, and how to design the storage systems accordingly.

ILM sets policies for clear levels of storage. Data needed day to day, such as fresh e-mail, can be held by highly available Fibre Channel storage arrays. Older or less important information can be moved to less expensive array solutions. Toward the end of its lifecycle information can be moved to an affordable online ATA solution or to tape. Afterwards the data can be moved to remote facilities for true disaster recovery. Define as many layers of storage as your business requires.

2. Exchange 2003 Storage Design Configurations—Evaluate Your Environment
Building the right server, network and storage architecture is the key to getting the most out of Exchange 2003.

While great savings are possible, the maximum savings are based on a close evaluation of the existing environments and user profiles.

The key is to calculate I/Os per second (IOPS). Every incoming or outgoing e-mail creates server I/Os. Larger messages, such as those with big attachments, naturally create more I/Os. To properly configure your server and storage requirements, you need to esimate how many IOPS your e-mail usage generates.

Are most users accessing mail over the Internet using POP3-compatible clients? With POP3, most e-mail processing occurs on the client, so there's little strain on the server. In this case, calculate about 0.1 IOPS per user.

Moving up the scale, perhaps yours is a server-based setup, but your company is a manufacturing firm staffed largely with factory floor workers who rarely use e-mail. Figure 0.25 IOPS per user for this scenario.

If your company is dominated by office workers, with a moderate use of e-mail, figure approximately 0.5 IOPS per user, or double the previous example.

The most e-mail-intensive users are in high-tech organizations for whom messaging is mission-critical. Get ready for 1 IOPS per user in these environments.

Once you've calculated your approximate IOPS load, you need to ensure your database disk subsystem will be able to handle the anticipated I/O, with room to spare. Testing I/O is the key to deciding how many mailboxes a single server can handle. If peak utilization doesn't task the server, and average utilization is a fraction of capacity, you can feel confident in perhaps doubling the mailboxes on that server.

And if you have a centralized storage solution, you can expand mailboxes without worrying about the limitations of direct-attached storage.

An Exchange Toolbox

Microsoft has some valuable tools for Exchange and Windows when you're re-evaluating your Exchange storage. A few to get you started: Jetstress: You can test the capacity of your existing or proposed storage with Jetstress, a utility written by Microsoft that can run against an entire disk subsystem, simulates Exchange disk I/O load, and can help you decide if the proposed storage system is reliable and fast enough. Diskpar: A command line utility that allows you to align your disks properly to optimize performance and take advantage of the power of a SAN. Available in the Windows 2000 Server Resource Kit, Diskpar sets the starting offset in the master boot record. DiskPart: The Windows Server 2003 version of Diskpar has a slightly different name, DiskPart, but a similar function. You can enable storage configuration from a script, remote session or other command prompt. DiskPart augments the Disk Administrator graphical user interface. DiskPart differs from many command-line utilities in that it does not operate in a single line mode. Instead, the utility is first invoked and then commands are read from standard I/O. Commands can be directed to any disk, partition, or volume. More Exchange-specific tools, including Jetstress, are available here: www.microsoft.com/exchange/tools/2003.asp. — D.B.

3. How Much Storage Do You Need?
Calculating Exchange 2003 storage needs isn't rocket science. In fact, at the most basic level, you can use a simple formula: total number of users * average mailbox size + size of public folders = disk space required.

But not every user is the same, and the typical user profile is a factor in sizing storage for Exchange. Some shops use e-mail infrequently.

And even those that send and receive a lot of messages may not use a lot of attachments and added Exchange services.

As a guideline, figure an average user in the previous scenario requires 0.5 IOPS. Power users not only send more mail, but also collaborate by sharing large files, use calendaring/scheduling, and perhaps even video conferencing.

Once you've defined user profiles, calculate the total IOPS required by multiplying each user by his predicted use of IOPS.

Base your calculations on peak IOPS. Think of your company on Monday morning: employees boot their PCs, process all the spam the filters didn't catch, respond to legitimate e-mail and start working on projects. This is the IOPS load your storage infrastructure needs to accommodate.

4. Understand Exchange I/O
Exchange 2003 is characterized by frequent, small I/O transactions, as one might expect with e-mail, where most messages are relatively small. Exchange is based on 4KB random reads and writes to the database, and sequential writes to the logs.

Since Exchange transfers data in these small blocks, native SCSI can actually be a faster way to connect servers to storage than Fibre Channel. However, Fibre Channel offers more flexibility as IT moves to centralized storage.

5. Go Granular
Exchange 2003 rids itself of the old monolithic database structure of Exchange 5.5. The newer software can segment message databases in a number of ways, supporting a more flexible storage infrastructure.

Exchange message storage consists of databases and Exchange storage groups (ESGs). Each ESG should be filled before the next one is populated with databases, thus efficiently filling all ESGs and making the most of concurrent operations.

Multiple ESGs also allow IT to work on a particular ESG while the rest remain active, increasing uptime. In fact, IT can recover a particular ESG without taking down the system.

Smaller, more granular databases are also faster to back up and restore.

6. Tackle Transaction Logs
Increasing server capacity also increases the number of transaction logs per server. Transaction logs are created per storage group, not per database.

The time it takes to replay transaction logs significantly impacts the time it takes to restore a server. Calculate the time to replay the logs, monitor the average number of logs per day, and adjust recovery plans accordingly.

Logs and Databases: While not an absolute requirement, it is strongly recommended that you hold Exchange databases and accompanying logs on different disks. This not only protects your critical data in the event of a hard drive failure, but improves I/O speed at the same time.

By separating logs and databases, IT can restore the database from a backup, and then replay all the logs to insure the data is fresh and up to date, and all transactions are accounted for.

If logs and databases are on the same disk, and that disk crashes, you must restore from the latest backup, which is by definition not up to date.

Circular Logs: With circular logs, older log files are continually deleted. It's important to make sure that circular logging is disabled if you plan to implement advanced backup and restore techniques, because you'll need a more complete log history to do a proper restore. While fuller logs take more space, it's worthwhile; otherwise your only backup is the latest full backup, and later transactions won't be accounted for.

7. Fix the File System
Best practices call for using NTFS, rather than FAT, on Exchange 2003 servers. Not only does NTFS support Volume Shadow Copy Services (VSS), which eases IT and end-user message restoration, it's faster (supporting disk compression), more scalable (supporting files and disks up to 16TB), and more secure (supporting encryption) than FAT.

Consolidate
8. Use Fewer Servers
Exchange 2003, when coupled with advanced storage technology, can break the cycle of expanding server capacity by building out or scaling out—in other words, simply adding new servers when they run out of gas, and then balancing the processing load across multiple servers. Users typically bought a new server for Exchange when the old one grew to support 500 users.

Much of this was due to storage and backup limitations. In simple direct-attached storage environments, Exchange databases were usually kept to 40GB to limit the backup window. However, more efficient storage and backup technologies can dramatically increase database sizes.

Exchange is not stateless, and thus needs constant connection between clients and servers to support inboxes, calendars and Active Directory, and to sort mail. For stateless applications, which don't need persistent client and server connections, scaling out can be an effective strategy—if you're willing to support more and more servers and more and more licenses. But scaling out isn't an effective solution for stateful applications such as Exchange 2003.

Centralized storage makes it possible to scale up instead. Here, you swap out multiple older servers for a smaller number of modern high-end servers. The key is to deploy servers with room to grow, whether through adding more users, services or applications.

The average Exchange 2003 server supports some 500 users, but a decent modern server can handle more than seven times that many. In fact, a large, well-equipped, top-of-the-line server can handle more than 5,000 Exchange 2003 users.

9. Realize the Beauty of SAN
A zero-latency, block-level storage area network is the best thing you can do for your Exchange installation—if you have the dough.

A SAN supports storage consolidation, offering a quick return on investment—if built properly. Unlike direct-attached storage, where each server has its own disk and you upgrade servers or disks as capacity runs out, a SAN removes the storage from discrete servers and centralizes it as part of a separate network. With a SAN you don't need to add new servers just because you reach disk capacity.

Rules of Thumb for I/O Sizing The key to right-sizing your Exchange architecture is correctly estimating I/Os per second (IOPS).

The SAN can also become the backup device, and a disk-based SAN can restore data far more quickly than tape, which can then become a deeper level of archive. The SAN, in fact, can write data directly to the tape subsystems and eliminates the need for a separate backup server. This is the brand of magic called serverless backup.

10. Consolidate Virtually
A great way to consolidate servers is to create virtual servers with a tool such as VMware or Microsoft's Virtual Server. With either, you can house a number of virtual servers and robust applications on a single high-end server, which acts as if it were multiple servers. This greatly eases maintenance, management and administration because there's less hardware, fewer network adapters, storage connections, power supplies and so on.

Backup and Restore
11. Get in Front of Backup
A system can't be considered trustworthy if it can't be restored quickly after an unplanned failure. That's where a backup plan comes in. No matter how much fault tolerance and storage redundancy you add, there's always the possibility that a failure will overcome your best defenses. Hardware and storage redundancy is not, in and of itself, protection against a corrupted Exchange database. When a catastrophic or dramatic failure hits, the only option is to restore the servers, then restore the data from backup.

Traditional wisdom dictated that you back up logs and all related databases together, since all databases in an ESG share a common log. Consequently, you had to put up with slow backups and degraded performance of Exchange servers. The granularity of Exchange 2003 allows you to break this mold, increasing flexibility, and allowing for quicker small backup (and restore) jobs.

Your SLAs determine your backup strategy. Decide how quickly you need to restore data by putting a value on the data and on the workers accessing it.

How much would the company lose per hour, minute or even per second in the case of a disaster? How long can your organization do without its data and working applications?

Your SLA for Exchange should specify the maximum time to restore a database. This can limit the number of users per server.

Before designing your Exchange backup architecture, document all your systems, including disk configuration, all hardware and software details, and server names. Put in place rigorous backup procedures, including:

• Backing up everything related to Exchange 2003.
• Splitting your final backup, whether tape, ATA drives, or something else, into two or three copies, stored in separate locations.
• Making your primary backup to disk. Secondary backup can be to tape via direct SCSI-attached tape drives to each server.

12. What to Back Up?
Everything critical to Exchange 2003 operation should be safely backed up. Make a list of everything the Exchange admin needs.

Besides the obvious (databases and logs), make sure to have fresh replicas of:

• Relevant Active Server pages
• Scripts
• Configuration data
• Management software and agents
• Server and storage inventory list

Truly static data, such as OS and application files, and certain scripts, need not be regularly backed up, as with dynamic data such as Exchange databases and logs, System State information and the Active Directory NTDSUTIL file.

13. Use Volume Shadow Copy Services
Exchange 2003 leverages many choice Windows Server 2003 features, including eight-node clustering and VSS.

With VSS, shadow copy takes a snapshot of the disk or storage group capturing a single point in time—while the system remains online. As a result, VSS can get back data quickly without disrupting the overall system.

VSS is ideal for end users, who can use it to recover data themselves. You should point the end-user data to the network, and activate VSS. Once in place, users can invoke VSS to recover a lost message or file, even if the file was open when it was lost.

VSS only works with NTFS, not FAT, and can make snapshots of entire NTFS volumes.

Availability
14. Cluster Up for Safety
Just as with server consolidation, plan the groundwork for high availability before implementing the storage component of an Exchange 2003 architecture. Many of these high availability underpinnings come from Windows Server 2003, including Microsoft Cluster Services (MSCS), which supports up to eight active nodes in Windows Server 2003, and Network Load Balancing (NLB), which can optimize overall performance as it protects uptime. These features help you survive memory and CPU problems, as well as application failures, with limited or no downtime.

The Standard version of Windows Server 2003 doesn't support clustering, and shouldn't be considered by IT organizations with critical needs or availability (although Standard does support Network Load Balancing). Those companies should be running Exchange 2003 on either Windows 2003 Enterprise Edition or Datacenter Edition, which support up to eight-node failover clusters.

MSCS provides high-availability for supported applications such as Exchange 2003 through failover. In other words, when there's a failure on one server, it tries to repair itself and restart. If unsuccessful, it quickly fails over to another, with minimal end-user disruption.

Clustering greatly eases server maintenance and configuration. You can take an active server offline, switch its tasks to another server, repair or reconfigure the original server and return it to active duty without disrupting messaging operations.

15. Plan for Disaster Recovery
How much downtime can you accept? What is the cost of that downtime? Is your Exchange installation interacting with customers via e-commerce? Does it support top executives whose time and data is especially valuable? How dependent is your operation on messaging?

You can calculate these costs by looking at the pure hourly rates of those affected, or better yet, the amount of revenue each employee produces. Consider opportunity costs as well: What's the long term impact of lost business and connections?

This analysis is the basis of the SLA that dictates how fail-proof your system needs to be and how quickly it needs to be brought back online.

All Exchange 2003 storage planning stems from this analysis and this SLA.

True disaster recovery requires a current data set be held in a remote location, so it's not destroyed in a natural or manmade disaster. Exchange databases and logs can be backed up on remote arrays, which would dramatically speed recovery compared to slower tape.

Exchange 2003 can increase productivity and deliver impressive ROI, but only if properly configured and supported by an intelligent, efficient storage infrastructure. Following these rules will help you get there.