In-Depth

Building the Perfect SAN

As prices continue to drop, storage area networks are becoming increasingly common. Here’s what you need to know to design and build your enterprise’s SAN. We’ve even done the testing for you.

IT administrators, you try your best to stay current with the technologies that get you through the day. Learning the newest technologies, especially those of the expensive variety, are normally put on the back burner. Let’s face it: Unless you manage one of the “Networks of the Rich and Famous,” you probably don’t have the luxury of testing expensive storage solutions when first available. Instead, you wait a few years for the dust to settle and the technology to be more readily available to the masses.

While many of us techno-geeks love to play with the latest offerings, being patient does have its advantages. Remember the early days of SCSI? During the turbulent era of non-standardized SCSI, mixing and matching SCSI components from different hardware vendors was like mixing oil and water. The safest bet was to marry your organization to a particular vendor and incorporate its integrated solution suite on your network. When storage area networking emerged as the hot storage technology a couple of years ago, it was clear that history was set to repeat itself.

Initially, standards were slow to develop, and mixing SAN hardware was often not recommended or supported. To implement a SAN that worked, many organizations chose to work with a single vendor or consulting firm on implementation.

Today, the hardware needed to implement a SAN has finally fallen in line with many industry standards, several of which were brought about by the Storage Networking Industry Association (SNIA) and the Fibre Channel Industry Association (FCIA). You can learn more about the standardization process, as well as storage certification, by visiting SNIA’s Web site at www.snia.org or FCIA’s Web site at www.fibrechannel.org.

Reviewed in this article:

Brocade SilkWorm 2400
$5,000 to $10,000, depending on options
Brocade Communications Systems
408-487-8000
http://www.brocade.com

Gadzoox Capellix 2000
Less than $400 per port
Gadzoox Networks Inc.
408-486-5600
http://www.gadzoox.com

Nishan IPS 3000
$30,000
Nishan Systems
408-519-3700
http://www.nishansystems.com

Vixel 7100
$6,995
Vixel Corp.
425-806-5509
http://www.vixel.com

ADIC Pathlight 5000
$12,465
Advanced Digital Information Corp.
425-881-8004
http://www.adic.com

Crossroads 10000
$18,995 to $19,595
Crossroads Systems Inc.
512-349-0300
http://www.crossroads.com

(Also see "What About NAS?" in this issue.

The beauty of standardization is that it’s led to independence, which means you can research the SAN that’s right for your organization, get the required hardware and put it all together. Of course, if it were really that easy, we IT folks wouldn’t be making the big bucks; but with an understanding of SAN essentials, storage network design and implementation is a project any experienced network administrator can undertake successfully.

Who Needs a Storage Network?
For many organizations, data defines worth. If access to company data is down, the organization may lose anywhere from thousands to millions of dollars per hour. Even for smaller companies, data loss can be devastating. I recently spoke with representatives of a company that had a single database failure that went undetected over a weekend. The estimated revenue loss was $200,000. While your organization may be willing to gamble on data loss, odds are that your insurance company isn’t: Many insurance companies are starting to base premiums on data recovery standards. For example, if you can fully recover from a disaster within six hours, you can acquire a specific rate; whereas if the recovery would take two to three days, the premium is significantly higher.

These problems and the need for quick recovery are prime examples of why storage networks are so important. The primary purpose of a storage network is to isolate your storage infrastructure from the public network. This provides the following advantages:

  • Data storage and backup is consolidated on its own network.
  • SANs can offer up to 2Gbps bandwidth, allowing for speedy backup and recovery.
  • Storage networks can have built-in redundancy (storage and access path), allowing for automated fault-tolerant data access.

With all this talk about storage networks, let’s see what they’re all about.

SAN Topologies
Like LANs, SANs are networks that can be configured in specific topologies. There are three basic SAN topologies:

  • Point-to-point
  • Arbitrated loop
  • Switched fabric

Point-to-Point Topology
Point-to-point is the most basic of all SAN topologies and requires little hardware to implement. As shown in Figure 1, a point-to-point SAN simply incorporates two devices connected by a single fibre channel cable. This is the equivalent of using a crossover cable to connect two computers. As with a crossover cable, the two devices have the entire bandwidth of the SAN for communications. To set up a point-to-point SAN, you connect the first component’s receive connector to the second component’s transmit connector, and vice versa. Due to its simplicity, this topology is rarely employed unless you only have the budget for a new library and some cabling and plan to purchase a switch the following year.

Point-to-point SAN
Figure 1. Point-to-point is the most basic SAN topology.

Arbitrated Loop Topology
Arbitrated loop topology, also referred to as fibre channel-arbitrated loop (FC-AL), is the SAN equivalent of a token ring network topology. In this configuration, all devices connected to the loop share the bandwidth of the single-loop connection. In this topology, all SAN components are arranged in a loop to which up to 127 devices can be connected (Figure 2). To accomplish this, the transmit fiber of the first component is placed into the receive port of the second component. This repeats around the loop until all ports are connected. Another way to physically configure this topology is to use a hub that supports FC-AL.

Arbitrated loop topology
Figure 2. In arbitrated loop topology, all devices in the loop share bandwidth.

Switched Fabric Topology
Because only one device can transmit over the loop at a time, the FC-AL topology is limited in performance and scalability. The most common topology that offers optimal network performance and scalability is the switched fabric topology. Switched fabric is just like switched Ethernet and offers switched point-to-point connections between SAN network components. This topology allows each component to utilize the entire network bandwidth. Also, with switched fabric, devices can be easily added and removed from the SAN without interruption. With an FC-AL SAN, each time a device is added or removed, the entire network needs to reinitialize. A switched fabric SAN is shown in Figure 3.

Switched fabric topology
Figure 3. Switched fabric topology allows each component to use the entire network’s bandwidth.

That’s the logical side of SANs; now let’s look at the physical aspects.

Key SAN Ingredients
SAN components are connected with fibre channel cable, which can be purchased in two forms: copper and fiber-optic. (Fibre channel uses the spelling “fibre” so as not to be fully associated with fiber optics.) Your choice of cable should primarily be dictated by the needs of your SAN. Fiber-optic cable costs much more than copper and is more fragile, but has its advantages.

If you need to maintain high bandwidth over a great distance or if electromagnetic interference (EMI) is a concern, your likely choice is fiber optic. If you’re looking for durable cable to be used for connecting local storage devices, copper may be your best bet.

Each server connecting to the SAN will need a fibre channel host bus adapter (HBA). The HBA requirement is pretty logical. If you want to connect a server to an external SCSI device, you need a SCSI adapter. The same can be said for needing an HBA to connect to a SAN. There is, however, one major difference between external SCSI storage and fibre channel SANs: With external SCSI, you have to select the appropriate SCSI card for the external SCSI bus and devices and then connect everything together with SCSI cables and proper termination. With fibre channel, HBAs are more generic. Unlike SCSI, your HBA choice shouldn’t cause any compatibility concerns, regardless of whether you’re using copper or fiber-optic cabling. This is because of the need for one additional device—the Gigabit Interface Converter (GBIC).

Gigabit Interface Converter
GBICs are modular transceivers that adapt the signal sent to or from the SAN hardware to the SAN transmission medium (fiber or copper). For any given fibre channel cable, you may see a GBIC on each end. One GBIC connects the HBA to the cable; the other GBIC connects the cable to a SAN switch, router or storage device. Some switches, however, don’t support GBICs; instead, they have an integrated connector that only supports one cable type. If you’re looking for versatility, make sure your switch has ports for GBICs.

Because GBICs adapt different HBAs to different fibre channel mediums, there are a few unique types of GBICs. So unfortunately, you can’t mosey on down to your local computer shop and shout, “I’ll take three GBICs—to go!” GBICs are identified primarily by their connector types (think back to narrow vs. wide SCSI). To interface with copper mediums, there are GBICs with DB-9 connectors, which are old and rarely used anymore. There are also GBICs with High Speed Serial Data Connectors (HSSDC)s. HSSDCs look similar to USBs and are the most common GBIC for copper mediums. For fiber-optic connections, there are multimode and singlemode GBICs, with the choice of GBIC based on the type of fiber employed in the network. A multimode GBIC is shown in Figure 4.

Multimode GBICs
Figure 4. Multimode GBICs are one choice for fiber-optic connections.

Now let’s look at some of the devices that may exist on a SAN.

Which Switch?
SAN switches and hubs work the same as their Ethernet counterparts, al-lowing you to network devices on the SAN, such as storage arrays, libraries and other switches and servers. With a hub, all devices share network bandwidth, while a switch provides dedicated point-to-point connections, just like Ethernet. However, your choice of switch will be dictated by the SAN topology employed. For example, some switches support arbitrated loop networks (FC-AL), while others support switched fabric or either topology. The type of topology a switch supports is determined by the type of ports it contains. F_Ports are used in switched fabric topologies and FL_Ports with arbitrated loop. Some switches have ports that can act as either F or FL ports. These ports are known as universal, or U_Ports. To support cascading of switches and the growth of your SAN, many switches also contain E_Ports, which are used to interconnect switches.

There’s one other major SAN component of which to be aware—the router or bridge. Some vendors call this piece a router, while others call it a bridge; but no one wants to go out on a limb and call it a “brouter.” The majority of vendors refer to this piece as a router, so that’s what I’ll call it.

With fibre channel, the job of a router is much different than with IP networks. Fibre channel routers translate fibre channel frames to frames of another transport, such as SCSI. Aside from a switch, the router is perhaps the most important piece of your SAN infrastructure. Because this device provides fibre-channel-to-SCSI translation, you can connect legacy SCSI devices, such as storage arrays or libraries, to your fibre channel SAN. Many newer libraries include built-in fibre channel HBAs, so a router isn’t necessary; but for moving your existing libraries to a SAN, purchasing a router is money well spent. If you’re starting to feel overwhelmed by the myriad switch and router possibilities, don’t worry. The most popular fibre channel switches and routers are covered shortly.

Zoning Laws
Another hot topic accompanying the rising popularity of SANs is the use of zoning. It’s easiest to think of zoning as the SAN equivalent to virtual LANs (vLANs). With LANs, you can set up vLANs on a switch to segment the single physical switch into multiple logical switches. This makes some switch port connections unable to see connections to other switch ports. With zoning, you can apply the same concept to SAN switches.

With networked storage, security may be a primary concern. For example, you may not want servers in the Development Organizational Unit (OU) to be able to access storage in the Finance OU. By setting up zoning on your SAN switches, your storage infrastructure can be configured so that Finance servers connected to the SAN can only see disk arrays allocated to Finance.

There are two primary ways to configure zoning on SAN switches. One is by port. For example, you can allow devices on Switch A Port 5 to communicate with devices connected to Switch A Port 9. The other is by World Wide Name (WWN). WWNs are unique, 64-bit identifiers for devices or ports. Some devices with multiple ports have a WWN for each port, allowing more granular management. Because of their length, WWNs are expressed in hexadecimal, so a typical WWN would look like this:

3A:08:7C:98:56:D9:02:44

With WWN, zoning is configured on a device or device-port basis, allowing you to move the device on the SAN and change its associated switch port without affecting zoning configuration. However, if the device fails and has to be replaced, you’ll have to reconfigure the zoning so that the WWN of the replacement device is associated with the correct zone.

Remember: Without zoning, all servers connected to the SAN have the potential to see all storage devices on the SAN. Configuring zoning allows you to limit what storage devices particular servers can see. If you have plans to expand your SAN to be shared by multiple departments, consider zoning a necessity.

Bridging the Gaps
For many organizations that require high data availability, disbursing storage across two or more remote sites offers the security of being able to maintain data availability even after a disaster. Availability can be achieved through technologies such as clustering, replication, snapshots and traditional backup and recovery. To configure a storage infrastructure that traverses geographical boundaries, organizations need a protocol that uses WAN’s economic advantages.

The cheapest transmission medium is the Internet, which requires IP. Wouldn’t it be cool to be able to bridge SANs in two sites through the Internet? That’s what Fibre Channel over IP (FCIP) is all about. In order for this to happen, a device able to do the fibre-channel-to-FCIP translation is needed. Some fibre channel switches have integrated FCIP ports allowing this. Remember, however, that FCIP doesn’t provide any means to directly interface with a fibre channel device; it’s a method of bridging two fibre-channel SANs over an IP network.

Internet SCSI (iSCSI), on the other hand, has a slightly different purpose. With iSCSI, hosts connected to the IP network can directly access iSCSI storage. Proponents of iSCSI believe that, with the rising popularity of gigabit Ethernet, SANs may someday be devoid of fibre channel altogether and will, instead, run on a gigabit IP backbone.

As with SAN and NAS (see "What About NAS?"), think of FCIP and iSCSI as complementary, rather than competing, solutions. They can reside together on the same network, and many storage vendors are working to build solutions to bridge these two technologies.

With their high degree of flexibility and scalability, SANs are here to stay. But don’t run off and grab the first SAN switch you see. Next is an evaluation of switches and routers that have been thoroughly tested.

Switch Shootout
Before buying, make sure that your potential hardware not only meets your needs, but doesn’t back you into a corner. The right switch for you should be one that’s easy to program and manage, works with all your devices, and has the right technology incorporated to grow with your SAN and emerging SAN technologies. With these criteria in mind, I put the following switches to the test:

  • Brocade SilkWorm 2400
  • Gadzoox Capellix 2000
  • Nishan IPS 3000
  • Vixel 7100

Each switch operates at 1Gbps and is considered to be in the entry-level class. For a higher price, most switch vendors today offer switches with 16 or more ports that run at 2Gbps.

Switch Speed Ports Management
Brocade SilkWorm 2400 1Gbps 8: F, FL, E • Web GUI
• Telnet
Gadzoox Capellix 2000 1Gbps 8: FL, E (expandable to 11 ports) • Web GUI
• Telnet
• Serial port
Nishan IPS 3000 1Gbps 8: F, FL, E, iSCSI, iFCP • Web GUI
• Telnet
Vixel 7100 1Gbps 8: F, FL, E • Telnet
• Serial port
Port key: F used in switched fabric topology; FL used in arbitrated loop topology; E used to connect or cascade switches.

Brocade SilkWorm 2400
Brocade is the SAN industry giant. If you have a SAN in place, odds are you have some product that was OEM’d by Brocade. Brocade has OEM agreements with many large vendors, including Dell, EMC, HP, IBM and StorageTek. According to IDC, Brocade has 90 percent market share of all SAN fibre channel switches and more than 60 percent of the total SAN infrastructure market. Based on this track record alone, going with Brocade products is an easy and safe choice—and its SilkWorm 2400 switch is no exception. In what is still a volatile tech market, many companies want to commit to a SAN vendor they know will be around for the next five to 10 years. With Brocade’s established market presence, you can count on it to be around well into the next decade.

The SilkWorm was a snap to set up, and I was able to configure zoning for my SAN within minutes with Brocade’s easy-to-use Web GUI (see Figure 5). However, configuring zoning through a telnet session through the command line wasn’t as straightforward. This switch offers Universal Ports (U_Ports), which allow ports to autodetect their connection state, permitting the switch to be used easily in both FC-AL and switched fabric implementations. In terms of setup and performance, I found the features and ease of use of this switch impressive.

Brocade Silkworm 2400
Figure 5. Configuring zoning with Brocade’s Web GUI is simple.

Note: Brocade has replaced the SilkWorm 2400 with the SilkWorm 3200, an 8-port entry fabric switch.

Of course, Brocade’s success and track record will mean more damage to your wallet, so if you’re looking to build a SAN on a budget, you may want to go with another switch, as this top-of-the-line product comes with a top-of-the-line price.

Gadzoox Capellix 2000
I found the Gadzoox Capellix 2000 to be as enjoyable as the SilkWorm 2400 for setup and initial deployment. However, it only supports FC-AL. So if the intent of your SAN is to run multiple backups to different devices on the SAN, you may want to consider another switch. If most of the storage devices on the SAN connect to only a couple of servers, it may be acceptable.

Gadzoox Capellix 2000
Figure 6. Gadzoox’s Capellix 2000 can be expanded to an 11-port switch.

The switch came with Gadzoox’s Ventana SANtools embedded in its firmware, which I found to be intuitive and easy to use. As with the Brocade switch, configuring zoning was a piece of cake. One nice feature is that a 3-port expansion plug can be had for $1,295, allowing you to turn your 8-port switch into an 11-port switch fairly affordably.

Nishan IPS 3000
I must admit that when I first started this test, I believed that all competitors would bow to mighty Brocade; but I was more than pleasantly surprised with Nishan Systems’ IPS 3000 storage switch.

I found this switch to be the jack of all trades—and master of them all, as well. It isn’t marketed as a fabric switch, but rather as a storage switch. The reason is that you can plug just about anything into it (with the right GBIC, of course). So if you want to run your storage network over Gigabit Ethernet, no problem! Prefer fibre channel? No problem! This switch also supports Internet Fibre Channel Protocol (iFCP), which allows you to bridge two or more remote SANs over an IP network using two or more IPS 3000 switches.

Unlike FCIP, which can also bridge fibre channel switches over an IP network, iFCP provides better fault isolation and scalability due to its “SAN routing” capabilities and the ability to create autonomous domains. Also, iFCP gives you the ability to network native IP storage devices and fibre channel devices on the same IP-based SAN. If iSCSI is in your future, it’s also supported by the IPS 3000. The bottom line is that if you’re looking for options, this switch is definitely a great choice.

Of course, paper capabilities vs. real-world performance aren’t always the same. But I was surprised at how easy it was to set up this switch in my SAN, as well as configure zoning with the IPS 3000’s drag-and-drop-style GUI interface. While I tested a single switch, Nishan Systems’ SANvergence Manager Java-based software gives you the ability to manage and monitor all of your Nishan switches from a single console. This was especially useful.

Nishan IPS 3000
Figure 7. Nishan’s IPS 3000 has a simple, easy-to-use GUI for management tasks.

The switch supports both switched fabric and arbitrated-loop fibre channel networks, in addition to supporting iFCP, iSCSI and Gigabit Ethernet. Its Web-based Java GUI was easy to navigate, and I found its telnet-based management simple as well. From a support perspective, Nishan bent over backward to address my questions.

Vixel 7100
The Vixel 7100 was the last switch tested, and a disappointment. While this switch offered the same features as the SilkWorm 2400 and Capellix 2000, its ease of use wasn’t easy at all. I found zone configuration, for example, to be extremely cumbersome. Furthermore, Web documentation was practically non-existent, adding to configuration woes.

Vixel 7100
Figure 8. Once implemented, Vixel’s 7100 performs well.

After the initial setup, the switch performed well. If your SAN is implemented by consultants familiar with this switch, you shouldn’t have problems. However, if you need to rebuild your SAN following a disaster, you may want to consider implementing a more user-friendly switch.

Adding SCSI to the SAN
A major requirement for most SAN implementations is support for SCSI devices on the fibre channel network. My SAN lab literally has dozens of SCSI disk arrays and libraries available, so I was able to put the routers of the top two vendors, ADIC and Crossroads, through thorough testing. The routers I tested were:

  • ADIC Pathlight 5000
  • Crossroads 10000

ADIC Pathlight 5000
The ADIC Pathlight 5000 was easy to set up, making it simple to bridge my SCSI devices to the SAN. ADIC’s Web site offers loads of online documentation, including its 316-page installation and administration guide. The guide stepped me through the setup process and then moved into documenting the Pathlight’s many management tools. For a do-it-yourselfer like myself, the guide was an absolute dream.

ADIC Pathlight 5000
Figure 9. ADIC’s Pathlight 5000 provides an easy setup, as well as thorough documentation.

The Pathlight 5000 offers 2Gbps link speed, allowing it to work with today’s top-of-the-line fibre channel switches, and has a simple GUI-based management interface. The interface offers a tree view of each of its four SCSI channels, permitting you quickly to observe the SCSI devices attached to each channel. It also has its own type of zoning for SCSI devices, called Virtual Private SANs (VPS). With VPS, you can configure the router to allow only specific HBAs attached to the SAN access to particular SCSI devices connected to the Pathlight 5000 router.

Crossroads 10000
The Crossroads 10000 router was easier to get going out of the box than the Pathlight 5000. Its embedded Web GUI, known as Crossroads Visual Manager (CVM), was simple to connect to and configure. In addition to its ease of use, this router’s loaded with features, permitting it to grow with your business needs. It’s expandable to support up to 12 SCSI channels (both SE/LVD and HVD SCSI interfaces) and up to 2Gbps fibre channel links, making it a truly versatile router. It can also act as a data mover, which allows it to interface with backup programs in support of both LAN-free and server-free backups. With server-free backups, an application server’s data stored on the SAN can be backed up without placing any load on the server itself.

Crossroads 10000
Figure 10. The Crossroads 10000's embedded Web GUI provides simple connection and configuration.

Final Report
As you see, there’s quite a bit to think about when considering reworking your storage infrastructure. As storage and data recovery requirements for organizations continue to increase, so will the need for SANs. When planning to implement a SAN in your organization, you should ensure that the SAN components are compatible with your organization’s current and future needs and are also compatible with your planned operating systems and applications. Windows Server 2003, for example, has very limited support for FC-AL SANs, and Microsoft strongly recommends against them. To ensure that your planned SAN is right for you, consider asking these questions of prospective SAN hardware and software vendors:

  • What’s important now? SAN equipment is expensive, so prioritize the SAN build-out with the needs of your organization.
  • Does the proposed SAN meet my performance requirements? If the primary goal of your SAN is to move data quickly, make sure that it lives up to its promise. Performance requirements will likely dictate your choices of SAN hardware, such as the choice to go with copper or fiber.
  • What SAN topologies are supported by the switch or hub? Make sure that the SAN topology employed can scale with your organization, servers and applications.
  • Is the proposed SAN compatible with my existing backup and storage management products? A SAN can easily have a ripple affect with other storage management applications. If you plan to keep your existing applications, make sure that they support your planned storage configuration. Otherwise, the cost of replacing your current storage management applications may greatly exaggerate your storage budget.

Businesses today live and die by their data. Over the next several years, your organization’s success will likely have some direct relation to the availability of its data. Guaranteeing data availability is an expensive but often necessary task. A properly designed SAN could be considered life insurance for not only your organization, but possibly your job as well.

Featured