In-Depth
Building the Perfect SAN
As prices continue to drop, storage area networks are becoming increasingly common. Here’s what you need to know to design and build your enterprise’s SAN. We’ve even done the testing for you.
- By Chris Wolf
- March 01, 2003
IT administrators, you try your best to stay current with the technologies
that get you through the day. Learning the newest technologies, especially
those of the expensive variety, are normally put on the back burner. Let’s
face it: Unless you manage one of the “Networks of the Rich and Famous,”
you probably don’t have the luxury of testing expensive storage solutions
when first available. Instead, you wait a few years for the dust to settle
and the technology to be more readily available to the masses.
While many of us techno-geeks love to play with the latest offerings,
being patient does have its advantages. Remember the early days of SCSI?
During the turbulent era of non-standardized SCSI, mixing and matching
SCSI components from different hardware vendors was like mixing oil and
water. The safest bet was to marry your organization to a particular vendor
and incorporate its integrated solution suite on your network. When storage
area networking emerged as the hot storage technology a couple of years
ago, it was clear that history was set to repeat itself.
Initially, standards were slow to develop, and mixing SAN hardware was
often not recommended or supported. To implement a SAN that worked, many
organizations chose to work with a single vendor or consulting firm on
implementation.
Today, the hardware needed to implement a SAN has finally fallen in line
with many industry standards, several of which were brought about by the
Storage Networking Industry Association (SNIA) and the Fibre Channel Industry
Association (FCIA). You can learn more about the standardization process,
as well as storage certification, by visiting SNIA’s Web site at www.snia.org
or FCIA’s Web site at www.fibrechannel.org.
The beauty of standardization is that it’s led to independence, which
means you can research the SAN that’s right for your organization, get
the required hardware and put it all together. Of course, if it were really
that easy, we IT folks wouldn’t be making the big bucks; but with an understanding
of SAN essentials, storage network design and implementation is a project
any experienced network administrator can undertake successfully.
Who Needs a Storage Network?
For many organizations, data defines worth. If access to company
data is down, the organization may lose anywhere from thousands to millions
of dollars per hour. Even for smaller companies, data loss can be devastating.
I recently spoke with representatives of a company that had a single database
failure that went undetected over a weekend. The estimated revenue loss
was $200,000. While your organization may be willing to gamble on data
loss, odds are that your insurance company isn’t: Many insurance companies
are starting to base premiums on data recovery standards. For example,
if you can fully recover from a disaster within six hours, you can acquire
a specific rate; whereas if the recovery would take two to three days,
the premium is significantly higher.
These problems and the need for quick recovery are prime examples of
why storage networks are so important. The primary purpose of a storage
network is to isolate your storage infrastructure from the public network.
This provides the following advantages:
- Data storage and backup is consolidated on its own network.
- SANs can offer up to 2Gbps bandwidth, allowing for speedy backup
and recovery.
- Storage networks can have built-in redundancy (storage and access
path), allowing for automated fault-tolerant data access.
With all this talk about storage networks, let’s see what they’re all
about.
SAN Topologies
Like LANs, SANs are networks that can be configured in specific
topologies. There are three basic SAN topologies:
- Point-to-point
- Arbitrated loop
- Switched fabric
Point-to-Point Topology
Point-to-point is the most basic of all SAN topologies and requires
little hardware to implement. As shown in Figure 1, a point-to-point SAN
simply incorporates two devices connected by a single fibre channel cable.
This is the equivalent of using a crossover cable to connect two computers.
As with a crossover cable, the two devices have the entire bandwidth of
the SAN for communications. To set up a point-to-point SAN, you connect
the first component’s receive connector to the second component’s transmit
connector, and vice versa. Due to its simplicity, this topology is rarely
employed unless you only have the budget for a new library and some cabling
and plan to purchase a switch the following year.
 |
| Figure 1. Point-to-point is the most basic SAN
topology. |
Arbitrated Loop Topology
Arbitrated loop topology, also referred to as fibre channel-arbitrated
loop (FC-AL), is the SAN equivalent of a token ring network topology.
In this configuration, all devices connected to the loop share the bandwidth
of the single-loop connection. In this topology, all SAN components are
arranged in a loop to which up to 127 devices can be connected (Figure
2). To accomplish this, the transmit fiber of the first component is placed
into the receive port of the second component. This repeats around the
loop until all ports are connected. Another way to physically configure
this topology is to use a hub that supports FC-AL.
 |
| Figure 2. In arbitrated loop topology, all devices
in the loop share bandwidth. |
Switched Fabric Topology
Because only one device can transmit over the loop at a time, the
FC-AL topology is limited in performance and scalability. The most common
topology that offers optimal network performance and scalability is the
switched fabric topology. Switched fabric is just like switched Ethernet
and offers switched point-to-point connections between SAN network components.
This topology allows each component to utilize the entire network bandwidth.
Also, with switched fabric, devices can be easily added and removed from
the SAN without interruption. With an FC-AL SAN, each time a device is
added or removed, the entire network needs to reinitialize. A switched
fabric SAN is shown in Figure 3.
 |
| Figure 3. Switched fabric topology allows each
component to use the entire network’s bandwidth. |
That’s the logical side of SANs; now let’s look at the physical aspects.
Key SAN Ingredients
SAN components are connected with fibre channel cable, which can
be purchased in two forms: copper and fiber-optic. (Fibre channel uses
the spelling “fibre” so as not to be fully associated with fiber optics.)
Your choice of cable should primarily be dictated by the needs of your
SAN. Fiber-optic cable costs much more than copper and is more fragile,
but has its advantages.
If you need to maintain high bandwidth over a great distance or if electromagnetic
interference (EMI) is a concern, your likely choice is fiber optic. If
you’re looking for durable cable to be used for connecting local storage
devices, copper may be your best bet.
Each server connecting to the SAN will need a fibre channel host bus
adapter (HBA). The HBA requirement is pretty logical. If you want to connect
a server to an external SCSI device, you need a SCSI adapter. The same
can be said for needing an HBA to connect to a SAN. There is, however,
one major difference between external SCSI storage and fibre channel SANs:
With external SCSI, you have to select the appropriate SCSI card for the
external SCSI bus and devices and then connect everything together with
SCSI cables and proper termination. With fibre channel, HBAs are more
generic. Unlike SCSI, your HBA choice shouldn’t cause any compatibility
concerns, regardless of whether you’re using copper or fiber-optic cabling.
This is because of the need for one additional device—the Gigabit Interface
Converter (GBIC).
Gigabit Interface Converter
GBICs are modular transceivers that adapt the signal sent to or
from the SAN hardware to the SAN transmission medium (fiber or copper).
For any given fibre channel cable, you may see a GBIC on each end. One
GBIC connects the HBA to the cable; the other GBIC connects the cable
to a SAN switch, router or storage device. Some switches, however, don’t
support GBICs; instead, they have an integrated connector that only supports
one cable type. If you’re looking for versatility, make sure your switch
has ports for GBICs.
Because GBICs adapt different HBAs to different fibre channel mediums,
there are a few unique types of GBICs. So unfortunately, you can’t mosey
on down to your local computer shop and shout, “I’ll take three GBICs—to
go!” GBICs are identified primarily by their connector types (think back
to narrow vs. wide SCSI). To interface with copper mediums, there are
GBICs with DB-9 connectors, which are old and rarely used anymore. There
are also GBICs with High Speed Serial Data Connectors (HSSDC)s. HSSDCs
look similar to USBs and are the most common GBIC for copper mediums.
For fiber-optic connections, there are multimode and singlemode GBICs,
with the choice of GBIC based on the type of fiber employed in the network.
A multimode GBIC is shown in Figure 4.
 |
| Figure 4. Multimode GBICs are one choice for
fiber-optic connections. |
Now let’s look at some of the devices that may exist on a SAN.
Which Switch?
SAN switches and hubs work the same as their Ethernet counterparts,
al-lowing you to network devices on the SAN, such as storage arrays, libraries
and other switches and servers. With a hub, all devices share network
bandwidth, while a switch provides dedicated point-to-point connections,
just like Ethernet. However, your choice of switch will be dictated by
the SAN topology employed. For example, some switches support arbitrated
loop networks (FC-AL), while others support switched fabric or either
topology. The type of topology a switch supports is determined by the
type of ports it contains. F_Ports are used in switched fabric topologies
and FL_Ports with arbitrated loop. Some switches have ports that can act
as either F or FL ports. These ports are known as universal, or U_Ports.
To support cascading of switches and the growth of your SAN, many switches
also contain E_Ports, which are used to interconnect switches.
There’s one other major SAN component of which to be aware—the router
or bridge. Some vendors call this piece a router, while others call it
a bridge; but no one wants to go out on a limb and call it a “brouter.”
The majority of vendors refer to this piece as a router, so that’s what
I’ll call it.
With fibre channel, the job of a router is much different than with IP
networks. Fibre channel routers translate fibre channel frames to frames
of another transport, such as SCSI. Aside from a switch, the router is
perhaps the most important piece of your SAN infrastructure. Because this
device provides fibre-channel-to-SCSI translation, you can connect legacy
SCSI devices, such as storage arrays or libraries, to your fibre channel
SAN. Many newer libraries include built-in fibre channel HBAs, so a router
isn’t necessary; but for moving your existing libraries to a SAN, purchasing
a router is money well spent. If you’re starting to feel overwhelmed by
the myriad switch and router possibilities, don’t worry. The most popular
fibre channel switches and routers are covered shortly.
Zoning Laws
Another hot topic accompanying the rising popularity of SANs is
the use of zoning. It’s easiest to think of zoning as the SAN equivalent
to virtual LANs (vLANs). With LANs, you can set up vLANs on a switch to
segment the single physical switch into multiple logical switches. This
makes some switch port connections unable to see connections to other
switch ports. With zoning, you can apply the same concept to SAN switches.
With networked storage, security may be a primary concern. For example,
you may not want servers in the Development Organizational Unit (OU) to
be able to access storage in the Finance OU. By setting up zoning on your
SAN switches, your storage infrastructure can be configured so that Finance
servers connected to the SAN can only see disk arrays allocated to Finance.
There are two primary ways to configure zoning on SAN switches. One is
by port. For example, you can allow devices on Switch A Port 5 to communicate
with devices connected to Switch A Port 9. The other is by World Wide
Name (WWN). WWNs are unique, 64-bit identifiers for devices or ports.
Some devices with multiple ports have a WWN for each port, allowing more
granular management. Because of their length, WWNs are expressed in hexadecimal,
so a typical WWN would look like this:
3A:08:7C:98:56:D9:02:44
With WWN, zoning is configured on a device or device-port basis, allowing
you to move the device on the SAN and change its associated switch port
without affecting zoning configuration. However, if the device fails and
has to be replaced, you’ll have to reconfigure the zoning so that the
WWN of the replacement device is associated with the correct zone.
Remember: Without zoning, all servers connected to the SAN have the potential
to see all storage devices on the SAN. Configuring zoning allows you to
limit what storage devices particular servers can see. If you have plans
to expand your SAN to be shared by multiple departments, consider zoning
a necessity.
Bridging the Gaps
For many organizations that require high data availability, disbursing
storage across two or more remote sites offers the security of being able
to maintain data availability even after a disaster. Availability can
be achieved through technologies such as clustering, replication, snapshots
and traditional backup and recovery. To configure a storage infrastructure
that traverses geographical boundaries, organizations need a protocol
that uses WAN’s economic advantages.
The cheapest transmission medium is the Internet, which requires IP.
Wouldn’t it be cool to be able to bridge SANs in two sites through the
Internet? That’s what Fibre Channel over IP (FCIP) is all about. In order
for this to happen, a device able to do the fibre-channel-to-FCIP translation
is needed. Some fibre channel switches have integrated FCIP ports allowing
this. Remember, however, that FCIP doesn’t provide any means to directly
interface with a fibre channel device; it’s a method of bridging two fibre-channel
SANs over an IP network.
Internet SCSI (iSCSI), on the other hand, has a slightly different purpose.
With iSCSI, hosts connected to the IP network can directly access iSCSI
storage. Proponents of iSCSI believe that, with the rising popularity
of gigabit Ethernet, SANs may someday be devoid of fibre channel altogether
and will, instead, run on a gigabit IP backbone.
As with SAN and NAS (see "What
About NAS?"), think of FCIP and iSCSI as complementary, rather
than competing, solutions. They can reside together on the same network,
and many storage vendors are working to build solutions to bridge these
two technologies.
With their high degree of flexibility and scalability, SANs are here
to stay. But don’t run off and grab the first SAN switch you see. Next
is an evaluation of switches and routers that have been thoroughly tested.
Switch Shootout
Before buying, make sure that your potential hardware not only
meets your needs, but doesn’t back you into a corner. The right switch
for you should be one that’s easy to program and manage, works with all
your devices, and has the right technology incorporated to grow with your
SAN and emerging SAN technologies. With these criteria in mind, I put
the following switches to the test:
- Brocade SilkWorm 2400
- Gadzoox Capellix 2000
- Nishan IPS 3000
- Vixel 7100
Each switch operates at 1Gbps and is considered to be in the entry-level
class. For a higher price, most switch vendors today offer switches with
16 or more ports that run at 2Gbps.
| Switch |
Speed |
Ports |
Management |
| Brocade SilkWorm 2400 |
1Gbps |
8: F, FL, E |
• Web GUI
• Telnet |
| Gadzoox Capellix 2000 |
1Gbps |
8: FL, E (expandable to 11 ports)
|
• Web GUI
• Telnet
• Serial port |
| Nishan IPS 3000 |
1Gbps |
8: F, FL, E, iSCSI, iFCP |
• Web GUI
• Telnet |
| Vixel 7100 |
1Gbps |
8: F, FL, E |
• Telnet
• Serial port |
|
|
| Port key: F used in switched fabric topology; FL used
in arbitrated loop topology; E used to connect or cascade switches.
|
Brocade SilkWorm 2400
Brocade is the SAN industry giant. If you have a SAN in place,
odds are you have some product that was OEM’d by Brocade. Brocade has
OEM agreements with many large vendors, including Dell, EMC, HP, IBM and
StorageTek. According to IDC, Brocade has 90 percent market share of all
SAN fibre channel switches and more than 60 percent of the total SAN infrastructure
market. Based on this track record alone, going with Brocade products
is an easy and safe choice—and its SilkWorm 2400 switch is no exception.
In what is still a volatile tech market, many companies want to commit
to a SAN vendor they know will be around for the next five to 10 years.
With Brocade’s established market presence, you can count on it to be
around well into the next decade.
The SilkWorm was a snap to set up, and I was able to configure zoning
for my SAN within minutes with Brocade’s easy-to-use Web GUI (see Figure
5). However, configuring zoning through a telnet session through the command
line wasn’t as straightforward. This switch offers Universal Ports (U_Ports),
which allow ports to autodetect their connection state, permitting the
switch to be used easily in both FC-AL and switched fabric implementations.
In terms of setup and performance, I found the features and ease of use
of this switch impressive.
 |
| Figure 5. Configuring zoning with Brocade’s Web
GUI is simple. |
Note: Brocade has replaced the SilkWorm 2400 with the SilkWorm 3200,
an 8-port entry fabric switch.
Of course, Brocade’s success and track record will mean more damage to
your wallet, so if you’re looking to build a SAN on a budget, you may
want to go with another switch, as this top-of-the-line product comes
with a top-of-the-line price.
Gadzoox Capellix 2000
I found the Gadzoox Capellix 2000 to be as enjoyable as the SilkWorm
2400 for setup and initial deployment. However, it only supports FC-AL.
So if the intent of your SAN is to run multiple backups to different devices
on the SAN, you may want to consider another switch. If most of the storage
devices on the SAN connect to only a couple of servers, it may be acceptable.
 |
| Figure 6. Gadzoox’s Capellix 2000 can be expanded
to an 11-port switch. |
The switch came with Gadzoox’s Ventana SANtools embedded in its firmware,
which I found to be intuitive and easy to use. As with the Brocade switch,
configuring zoning was a piece of cake. One nice feature is that a 3-port
expansion plug can be had for $1,295, allowing you to turn your 8-port
switch into an 11-port switch fairly affordably.
Nishan IPS 3000
I must admit that when I first started this test, I believed that
all competitors would bow to mighty Brocade; but I was more than pleasantly
surprised with Nishan Systems’ IPS 3000 storage switch.
I found this switch to be the jack of all trades—and master of them all,
as well. It isn’t marketed as a fabric switch, but rather as a storage
switch. The reason is that you can plug just about anything into it (with
the right GBIC, of course). So if you want to run your storage network
over Gigabit Ethernet, no problem! Prefer fibre channel? No problem! This
switch also supports Internet Fibre Channel Protocol (iFCP), which allows
you to bridge two or more remote SANs over an IP network using two or
more IPS 3000 switches.
Unlike FCIP, which can also bridge fibre channel switches over an IP
network, iFCP provides better fault isolation and scalability due to its
“SAN routing” capabilities and the ability to create autonomous domains.
Also, iFCP gives you the ability to network native IP storage devices
and fibre channel devices on the same IP-based SAN. If iSCSI is in your
future, it’s also supported by the IPS 3000. The bottom line is that if
you’re looking for options, this switch is definitely a great choice.
Of course, paper capabilities vs. real-world performance aren’t always
the same. But I was surprised at how easy it was to set up this switch
in my SAN, as well as configure zoning with the IPS 3000’s drag-and-drop-style
GUI interface. While I tested a single switch, Nishan Systems’ SANvergence
Manager Java-based software gives you the ability to manage and monitor
all of your Nishan switches from a single console. This was especially
useful.
 |
| Figure 7. Nishan’s IPS 3000 has a simple, easy-to-use
GUI for management tasks. |
The switch supports both switched fabric and arbitrated-loop fibre channel
networks, in addition to supporting iFCP, iSCSI and Gigabit Ethernet.
Its Web-based Java GUI was easy to navigate, and I found its telnet-based
management simple as well. From a support perspective, Nishan bent over
backward to address my questions.
Vixel 7100
The Vixel 7100 was the last switch tested, and a disappointment.
While this switch offered the same features as the SilkWorm 2400 and Capellix
2000, its ease of use wasn’t easy at all. I found zone configuration,
for example, to be extremely cumbersome. Furthermore, Web documentation
was practically non-existent, adding to configuration woes.
 |
| Figure 8. Once implemented, Vixel’s 7100 performs
well. |
After the initial setup, the switch performed well. If your SAN is implemented
by consultants familiar with this switch, you shouldn’t have problems.
However, if you need to rebuild your SAN following a disaster, you may
want to consider implementing a more user-friendly switch.
Adding SCSI to the SAN
A major requirement for most SAN implementations is support for
SCSI devices on the fibre channel network. My SAN lab literally has dozens
of SCSI disk arrays and libraries available, so I was able to put the
routers of the top two vendors, ADIC and Crossroads, through thorough
testing. The routers I tested were:
- ADIC Pathlight 5000
- Crossroads 10000
ADIC Pathlight 5000
The ADIC Pathlight 5000 was easy to set up, making it simple to
bridge my SCSI devices to the SAN. ADIC’s Web site offers loads of online
documentation, including its 316-page installation and administration
guide. The guide stepped me through the setup process and then moved into
documenting the Pathlight’s many management tools. For a do-it-yourselfer
like myself, the guide was an absolute dream.
 |
| Figure 9. ADIC’s Pathlight 5000 provides an easy
setup, as well as thorough documentation. |
The Pathlight 5000 offers 2Gbps link speed, allowing it to work with
today’s top-of-the-line fibre channel switches, and has a simple GUI-based
management interface. The interface offers a tree view of each of its
four SCSI channels, permitting you quickly to observe the SCSI devices
attached to each channel. It also has its own type of zoning for SCSI
devices, called Virtual Private SANs (VPS). With VPS, you can configure
the router to allow only specific HBAs attached to the SAN access to particular
SCSI devices connected to the Pathlight 5000 router.
Crossroads 10000
The Crossroads 10000 router was easier to get going out of the
box than the Pathlight 5000. Its embedded Web GUI, known as Crossroads
Visual Manager (CVM), was simple to connect to and configure. In addition
to its ease of use, this router’s loaded with features, permitting it
to grow with your business needs. It’s expandable to support up to 12
SCSI channels (both SE/LVD and HVD SCSI interfaces) and up to 2Gbps fibre
channel links, making it a truly versatile router. It can also act as
a data mover, which allows it to interface with backup programs in support
of both LAN-free and server-free backups. With server-free backups, an
application server’s data stored on the SAN can be backed up without placing
any load on the server itself.
 |
| Figure 10. The Crossroads 10000's embedded Web
GUI provides simple connection and configuration. |
Final Report
As you see, there’s quite a bit to think about when considering
reworking your storage infrastructure. As storage and data recovery requirements
for organizations continue to increase, so will the need for SANs. When
planning to implement a SAN in your organization, you should ensure that
the SAN components are compatible with your organization’s current and
future needs and are also compatible with your planned operating systems
and applications. Windows Server 2003, for example, has very limited support
for FC-AL SANs, and Microsoft strongly recommends against them. To ensure
that your planned SAN is right for you, consider asking these questions
of prospective SAN hardware and software vendors:
- What’s important now? SAN equipment is expensive, so prioritize
the SAN build-out with the needs of your organization.
- Does the proposed SAN meet my performance requirements? If
the primary goal of your SAN is to move data quickly, make sure that
it lives up to its promise. Performance requirements will likely dictate
your choices of SAN hardware, such as the choice to go with copper or
fiber.
- What SAN topologies are supported by the switch or hub? Make
sure that the SAN topology employed can scale with your organization,
servers and applications.
- Is the proposed SAN compatible with my existing backup and storage
management products? A SAN can easily have a ripple affect with
other storage management applications. If you plan to keep your existing
applications, make sure that they support your planned storage configuration.
Otherwise, the cost of replacing your current storage management applications
may greatly exaggerate your storage budget.
Businesses today live and die by their data. Over the next several years,
your organization’s success will likely have some direct relation to the
availability of its data. Guaranteeing data availability is an expensive
but often necessary task. A properly designed SAN could be considered
life insurance for not only your organization, but possibly your job as
well.