Minding the Store
As SAN and NAS storage drops in price, more companies are implementing them. What's your role in managing these devices?
- By Bill Heldman
- October 01, 2002
Users are a disk-hungry bunch, aren't they? Consider a few scenarios:
- Users need a place to back up their critical data—preferably
on some server disk that you currently back up with your formal server
- The e-mail system is so heavily utilized that your e-mail database
is growing by leaps and bounds. You just can't get your boss (and others)
to give up their multi-megabyte inboxes!
- The Home and Shared directories on your file servers are gargantuan.
- You've introduced a geographic information system and the spatial
images take up massive quantities of disk space.
- Your Oracle and SQL Server databases are colossal.
Where are you going to put all this data? You could keep buying servers
with a few hard drives in them, but at some point your servers begins
to procreate like rabbits and your farm overrunneth. The biggest question
you need to ask yourself in this situation: "Am I maximizing the usage
of all this disk space?" Some studies that show that only about 30 to
50 percent of disk space is utilized on all these servers. And that unused
disk is expensive!
Also, even though RAID array controller cards have brought a tremendous
measure of fault tolerance to servers, there are two problems associated
with RAID systems:
- If more than one drive in a RAID 5 array fails, you're toast—you've
got to rebuild the array and restore from tape.
- Admins often don't get it when it comes to RAID. They don't have hot
or cold spares available and at the ready or they incorrectly rebuild
the mirror or stripe set with parity.
- Additionally, even though most Admins routinely back up their systems,
they don't often validate the backups by performing routine restoration
tests to make sure the data on the tape can actually be restored. (I'm
raising my hand as I write this paragraph—been there, done that,
bought the soundtrack.)
On top of all that, a $1,700 hardware RAID array controller card, while
better than nothing at all, is still a single point of failure (SPOF).
While your users' data is certainly safer when on a drive hooked to a
RAID card, it still isn't completely safe.
So, is there a solution that can help offset some of these issues? Is
there a way that we can provide ample, inexpensive disk space for our
users, a place where all of our clients—Linux, Unix, Windows, even
mainframe—can go to get their data?
Storage Area Network (SAN)
A SAN is actually a collection of equipment that makes possible
the consolidation of disk storage. Following are some of the critical
elements in a SAN:
- Storage array—This is a device that has a lot of disks
in it. Typically these devices are rated in the number of supported
Terabytes (TB). Most high quality (expensive) storage arrays provide
some sort of fault-tolerance software in them that continuously monitors
the disks, proactively moves data off of bad sectors and even "phones
home" if a disk is going to go bad so that the vendor's service department
can bring a new one out.
- Network Attached Storage (NAS)—A NAS device (discussed
below) is essentially a server with a lot of disks in it using a highly
scaled down OS.
- Fiber-channel—The fibre-channel standard is used to connect
devices to the storage array. Typically servers that run applications
will connect to the storage array to access their data. Most connections
are done through some form fibre-channel.
- Host Bus Adapters (HBAs)—Think of these as fibre-channel
network cards, HBAs go into your servers so that you can connect them
to the storage array.
- Fibre-channel switches—There are two categories of fibre-channel
(FC) switch: normal and "director-class". The director-class switch,
invented and patented by McData (www.mcdata.com)
allows for more intelligent monitoring and traffic management of the
FC switch fabric.
- Software—The value-add and the most important feature
of a high-quality storage array installation is the software that's
used to run, manage and report on the array. Big companies like EMC
really shine in this area, but their software is pricey. Other big players
such as Legato and Veritas have gotten into the storage array software
game. There are a variety of needs: reporting, representing NetBIOS-type
share information to Windows clients, monitoring disk health, managing
disk space, real-time copying to a remote device backup software, etc.
The combination of all of the above constitutes what most people call
a SAN. The idea is that you have users who connect to the SAN via software
running on the storage device that allows them to do so, even though they're
not fibre-channel-attached and then to access data that's managed by servers
that are fibre-channel-attached. This is a SAN in its idealized form—there
are other variations such as servers that connect to a FC switch, which,
in turn, talks to the storage array; NAS devices that connect to the storage
array and so forth. The reason we call a SAN what we do is that with a
little bit of architectural expertise you can set up what are called "islands
of storage" that provide highly available and fault-tolerant data space
and that can act as backups to one another.
Network Attached Storage (NAS)
There are simple solutions out there that allow you to snap on
a piece of disk space for your users without having to get into the whole
SAN-box (ha-ha). The solution is called Network Attached Storage (NAS),
and a wide variety of vendors today offer NAS boxes. NAS solutions typically
utilize a mini-OS with enough intelligence to get the data to users (it
ain't Windows folks) although big players such as EMC provide very robust
NAS solutions. Admins use a set of tools (which vary from vendor to vendor)
to manage the NAS.
NAS devices support a wide variety of file systems and generally support
all "open systems" clients. When we say "open systems" we mean Linux,
all flavors of Unix, Windows, Netware and Macintosh along with the various
file systems that these OSes require. We do not mean mainframe clients.
If you want a storage device that can host both mainframe and open systems
clients you need a larger, much more expensive class of device such as
one that IBM or EMC could provide.
Some NAS devices use the Network Data Management Protocol (NDMP) to allow
administrators to back up the device. NAS devices do not support niceties
such as e-mail, file management or user authentication services.
We're talking basic, inexpensive, reliable easy-to-manage storage. Basic
NAS boxes are extremely plug-and-play. (See www.networkappliance.com
as an example of a company heavily involved in this type of NAS device).
Admins interested in getting started with SAN/NAS technology and who
have minor data consolidation issues to worry about should investigate
A larger storage array differentiates from its NAS cousin in the
software that it runs. If you purchase an EMC Symmetrix or Clariion or
a Hitachi Data Systems (HDS) 9900, for example, you are paying largely
in part for the software that makes that bad boy so highly fault-tolerant.
There are expensive up-front software costs, as well as equally expensive
ongoing software maintenance fees. However, if any component of the array
goes south, it is the vendor that makes sure that you continue operation.
EMC and IBM guarantee five-nines uptime (99.999%) while HDS (and its certified
reseller—Sun Microsystems) guarantee 100% uptime. How do they do
it? Redundancy, redundancy, redundancy. Plus good software. If there appears
to be a problem with a section of disk, the operating system will proactively
move the data to a good sector and mark the bad place as bad. It will
continuously monitor the array. If the bad sector gets worse—that
is, it spreads—the OS will actually phone home to tell the vendor
that it has a problem and to please bring out a new disk.
One night, I was a server administrator on call in a shop that used EMC
Symmetrix. The night operator called me at about 2:00 a.m. to tell me
that there was an EMC technician at the door—our Symm had a bad disk
and the guy wanted to replace it. The operator asked me what I thought
she should do. "Let the guy in!" I said. We had no clue the disk was failing—all
of the data was accessible and just fine.
A storage array device typically comes with a certain amount of
fibre-channel ports (fibre-channel in the mainframe world is provided
through the FICON standard). If the number of hosts you plan to connect
to the array exceeds the amount of ports in your array, then you've got
to augment with a FC switch.
A fibre-channel switch is used to connect a lot of devices to the SAN,
or to connect devices a long way away from the SAN to the SAN, or both.
FC switches aren't any different than IDF/MDF switches in the sense that
you buy them by the number and by the kind of ports they have in them.
There is one noticeable difference however: For what you pay for one FC
switch you could probably equip an entire IDF or MDF. FC switches are
priced in the tens of thousands of dollars.
Host Bus Adapters
HBAs are also quite expensive. Plan on spending in excess of $1,000
for a typical HBA. Note that you won't need HBAs for a little NAS device
that you purchase and hang on the network. You get into HBAs when you
have a sophisticated storage array that comes with some FC ports, but
not enough for you to hang all your hosts onto the SAN.
Generally your mission-critical data will live on the storage array and
your servers that talk to the data will need to be attached to the SAN
via FC. We call such servers "channel-attached." As a rule of thumb, you
should plan on putting two HBAs in any such mission-critical server.
Fiber-optic cable is generally utilized as the connection between the
SAN and the HBA, though copper is also supported in fibre-channel technology.
Note that tape back-up devices such as tape libraries are completely
capable of connecting to a SAN via HBAs and are thus equipped to provide
tape backup of the data on the SAN.
Backin Up My Words
SAN technology is complicated, high-tech and extremely expensive.
But it's also the best way to go if you're considering protecting your
user's mission critical data. Another reason to consider SAN/NAS solutions
revolves around the idea behind data consolidation. You want to get all
that file, home-directory and shared-directory off of your servers and
put it onto a central location. These are the main reasons you'd consider
a SAN/NAS solution.
There are major players that support both mainframe and open systems
on the same storage array. If you don't have a requirement for mainframe
storage, you may be able to save significant dollars with a downscaled
storage array device (such as the EMC Clariion or offerings from Compaq,
HP and others).
When you build a SAN, you're building a little storage network that is
comprised of several different hardware components that are managed with
software specially designed for the SAN.
If you're considering designing a more complex SAN/NAS solution that
a simple 1U NAS box, you'd be wise to see experienced counsel before proceeding.