The 64-Bit Question

Itanium servers, which utilize 64-bit architecture, are ready for the datacenter. This guide can help you decide whether your company should consider adding these high-muscle machines.

• By Bill Boswell
• July 01, 2002

Here’s a riddle: What’s a little bigger than a Wheat Thin, has more transistors than the adult population of the United States, draws as much power as a small vacuum cleaner, and costs as much as a top-quality HDTV? Give up? It’s Intel’s 64-bit McKinley CPU, soon to be released as Itanium 2. McKinley is the newest member of the IA64 (Intel Architecture 64-bit) processor family.

After years of bluster and prototype product releases, 64-bit Intel-based servers and workstations are finally ready for real-world production applications. The choice of an operating system for these is relatively limited. Five Linux distributions (http://www.linuxia64.org) have IA64 versions, and there’s an IA64 version of HP/UX, which makes sense because Hewlett-Packard was the co-developer of the IA64 architecture. As for Windows, there’s an IA64 version of Windows XP and two members of the upcoming Windows .NET Server family, Enterprise Server and Datacenter Server, will have IA64 versions.

Let’s see take a look at the reasons why vendors of databases and other data-intensive applications are starting to migrate their apps to native IA64 versions and why we need to be ready for the new technology when it arrives in our server rooms.

Supercharged Memory Addressing
An IA64 machine’s biggest advantage comes from its ability to directly address boatloads of memory. IA32 machines can address only 232 bytes of memory (about 4GB). Windows 2000 Advanced and Datacenter servers support the Physical Address Extension (PAE) in the Pentium processor, which opens four more memory lines for a total of 236 bytes, or 64GB; but PAE memory handling is relatively clumsy and requires applications to use function calls in the Address Windowing Extensions (AWE).

IA64 machines, on the other hand, have a much larger address space and don’t require special gymnastics to use it. The current crop of IA64 processors has a 44-bit address bus that supports 16TB (terabytes) of virtual memory. IA64 Windows divides this memory space in half, giving 8TB to user applications and 8TB to the operating system.

Physical memory is limited due to constraints imposed by the chipset and motherboard design. The 460GX chipset that accompanies the original Itanium limits memory to 16GB per processor. McKinley uses the new i870 chipset, which supports 64GB of RAM per processor and also supports a total of 512 processors. (Non-OEM versions of IA64 Windows .NET Datacenter Server support 64 processors. IA64 Enterprise Server supports eight.)

The first McKinley processor has a 1GHz clock speed, which may seem slow compared to the 2.4GHz Xeon, but McKinley has a 128-bit system bus running at 400MHz with on-board L1 and L2 and up to 3MB of L3 cache. The L2 and L3 cache have a 5-tick latency, nearly instantaneous when compared to the time needed to access the external cache on a Pentium. Also, 64-bit applications compiled with suitable optimizers will run faster than their 32-bit counterparts, even though the IA64 processor is slower, thanks to the superior pipelining and larger number of registers in McKinley.

The i870 chipset also supports Infiniband, the successor to PCI for I/O architecture. Infiniband provides a true channelized switching fabric rather than the shared memory bus used by PCI. Each Infiniband device gets its own clear communications path to the CPU and main memory with stratospheric speeds up to 6GBps. Visit www.inifiniband.org for details.

Better Booting
There’s more to the IA64 story than simply a more capable processor. IA64 machines have a variety of new features, including an improvement to the boot process.

When you start an IA32 machine, it goes through a complicated ballet trying to locate a device with bootstrap code (floppy, CD-ROM, fixed disks, or PXE.) Changing the search order requires modifying CMOS settings or, if you have a newer BIOS, selecting from a special startup menu. Once the system locates a bootable device, it makes a series of INT13 disk calls to find an active partition with a boot sector. Executable code in the boot sector locates a secondary bootstrap loader that actually loads the operating system. For DOS and Win9x, the secondary bootstrap loader is Io.sys. For the NT family of products—including Windows 2000, XP and .NET—the secondary bootstrap loader is Ntldr.

To do its job, Ntldr relies on hardware information gleaned by Ntdetect. com and a path to the operating system kernel files stored in Boot.ini. Ntldr also requires an SCSI driver if the SCSI interface doesn’t have a BIOS. The SCSI driver is stored in Ntbootdd.sys. If any of these files aren’t present or they get corrupted or overwritten, the operating system won’t boot. Anyone who’s spent hours of their youth trying to configure a multiboot Windows machine will attest to the clumsiness of this IA32 boot scheme.

IA64 machines avoid all that ugliness by storing boot information in non-volatile RAM (NVRAM), also called firmware. These firmware settings are controlled by the Extensible Firmware Interface, or EFI. For the full EFI specification, see developer.intel.com/technology/efi/EFISpec_v102.htm.

Because the initial bootstrap code is stored in firmware, IA64 Windows doesn’t need Ntdetect.com or Boot.ini. It still requires a secondary bootstrap loader, though, which is called Ia64ldr.efi. The .efi extension indicates a file executable in the EFI operating environment.

The EFI contains a small command processor, or shell, accessed via a boot menu. Here’s an example menu:

EFI Boot Manager ver 1.02 [12.36A]

Please select a boot option:
Microsoft Windows .NET Standard Server
Acpi(PNP0A03,0)/Pci(5|0)/Mac(0003478991556)
EFI Shell [Built-in]
Boot option maintenance menu

The EFI shell recognizes a fairly comprehensive list of commands. You can get a partial list by typing “?” at the shell prompt. For a full list, visit cedar.intel.com.

When the EFI Shell first loads, it lists the available partitions on the drives. You can get the same list using the MAP command. Here’s an example MAP listing for a machine with several partitions:

Device mapping table

fs0 : VenHw(Unknown Device:80)/HD(Part1,SigBD1C1A20-685C-01C1-507B 9E5F8078F531)
fs1 : VenHw(Unknown Device:80)/HD(Part5,Sig1FD4472C-D516-11D5-8473 806D6172696F)
fs2 : VenHw(Unknown Device:FF)/CDROM(Entry1)
blk0 : VenHw(Unknown Device:00)
blk1 : VenHw(Unknown Device:80)
blk2 : VenHw(Unknown Device:80)/HD(Part1,SigBD1C1A20-685C-01C1-507B 9E5F8078F531)
blk3 : VenHw(Unknown Device:80)/HD(Part2,Sig1FD44720-D516-11D5-8473 806D6172696F)
blk4 : VenHw(Unknown Device:FF)
blk5 : VenHw(Unknown Device:FF)/CDROM(Entry1)

The long number in a few of these entries is the partition’s Globally Unique Identifier (GUID). Each partition has its own GUID. The blk# number identifies the sequence of the partition on the disk. If a partition has a file system that EFI can read (FAT, FAT32, and Joliet CD-ROM), it’s assigned an fs# alias.

Inside the EFI shell, you can access partitions in the same way you access logical drives in DOS. Enter the block name followed by a colon. For example, enter blk3: to go to the third partition. If the partition has a file system readable by EFI, you can list the files and folders using dir or ls.

If you have a new machine with no pre-installed operating system or you want to install Windows as a second operating system, you must use the EFI shell to launch Setup. IA64 systems don’t recognize El Torito bootable CDs. To initiate Setup, select the fs# alias representing the CD-ROM, then launch the Setup boot loader, Setupldr.efi.

More Powerful Partitioning
Another major feature of IA64 machines involves a wholesale change to the way disks are partitioned. A standard IA32 computer uses the venerable Master Boot Record (MBR) to identify partitions. A partition table in the MBR lists the Cylinder/Head/Sector (CHS) location for each partition along with partition-type information (primary, extended, non-DOS and so on) and whether the partition’s active (bootable).

An MBR-based partition table has a maximum of four entries, forcing vendors to tuck information in crannies between the MBR and first partition, install hidden partitions or to load proprietary partition managers. This chaos goes away on IA64 machines, thanks to a new partition scheme called the GUID Partition Table (GPT).

On a GPT-based disk, each partition is assigned two GUIDs. One of them uniquely identifies the particular partition. The other represents a partition type. There are only a handful of recognized partition types. I’ll talk about them in just a moment.

A GPT can hold up to 128 partitions. A mirrored copy of the GPT is stored at the end of the drive, which avoids another problem with MBR disks, where a single failed sector can render the drive unusable. There are no hidden partitions, no special disk structures, no arcane rules for creating logical drives, no hidden OEM partitions, and no undocumented machinations for booting multiple operating systems.

GPT disks also support very large partitions thanks to a 64-bit Logical Block Address scheme. A logical block corresponds to one sector, or 512 bytes, yielding a maximum theoretical capacity of eight zettabytes, enough storage to assign a name to every star in the universe. (See physics.nist.gov/cuu/Units/prefixes.html for a list of scientific notation prefixes.) IA64 Windows XP and .NET support volume sizes up to 16 exabytes based on NTFS limitations.

Standard GPT Partitions
Here’s a brief list of the standard partition types supported by GPT disks and what they’re used for.

EFI System Partition (ESP) Each operating system on an IA64 machine requires a secondary bootstrap loader. The path to this loader file is stored in firmware and appears on the EFI Boot Menu. As I mentioned previously, the secondary bootstrap loader for IA64 Windows is Ia64ldr.efi.

Secondary bootstrap loaders can’t be stored on the same partition as the operating system. EFI only has a FAT/FAT32 driver and the operating system partition could be formatted with NTFS for Windows or EXT2/EXT3 for Linux. Instead, secondary bootstrap loaders are stored in a special partition called the EFI System Partition (ESP). The ESP is formatted as FAT.

IA64 Windows stores Ia64ldr.efi in a folder called \EFI\Microsoft\WINNT50. This folder also contains a file called Fpswa.efi. This is the Floating Point Software Assistance handler, which contains floating-point exceptions required by the operating system. (See developer.intel.com/design/itanium/ downloads/245415.htm for more information.)

If you run multiple copies of Windows .NET or XP on an IA64 machine, each copy of Ia64ldr.efi gets its own unique folder in the ESP. For example, the second copy would be stored in \EFI\Microsoft\WINNT50.0, the third in \EFI\Microsoft\WINNT50.1 and so on.

The ESP uses 1 percent of the disk space with a minimum size of 100MB and a maximum size of 1GB. A GPT disk can contain only one ESP. The ESP doesn’t appear in the Disk Management console and not as a drive in Explorer.

Microsoft Reserved Partition (MSR) In IA32 Windows, an MBR disk is called a basic disk. When you convert a basic disk to a dynamic disk, the partition information is transferred to a Logical Disk Manager (LDM) database located in the last cylinder of the disk or the last 1 MB, whichever is smaller. All dynamic disks on a machine have an identical copy of the LDM database. This permits them to participate in shared volume configurations such as RAID 1 (mirroring), RAID 0 (striping), RAID 5 (striping with parity), and volume spanning.

IA64 versions of XP and .NET treat a GPT disk as a basic disk. If you convert a GPT basic disk to a dynamic disk, the GPT information is transferred to the LDM database. The GPT specification doesn’t permit the kind of ad hoc partitioning used to hide the database on IA32 machines, so IA64 Windows automatically creates a Microsoft Reserved Partition (MSR) to store the LDM database. The MSR partition is created even if the GPT disk is left as a basic disk.

The MSR partition size depends on physical disk capacity. On disks up to 16GB, the MSR is 32MB. On disks of more than 16GB, the MSR is 128MB. The MSR is formatted as FAT16. It isn’t exposed by Explorer, but can be seen in the Disk Management console.

Microsoft Data Partition This partition type is assigned by IA64 Windows .NET or XP when creating new partitions using Explorer. Other operating systems can see the contents of Microsoft Data Partitions unless you use NTFS permissions to lock them down.

OEM Partitions Some vendors love to ship proprietary BIOS setup utilities and diagnostic tools in hidden partitions. For them, the GPT specification includes a special OEM partition type. Ordinarily, an OEM partition doesn’t appear as a drive in Explorer, but can be seen in the Disk Management console. Vendor utilities find the partition using its GUID.

Working and Playing with Others
IA64 versions of Windows must boot from a GPT disk. However, they permit using MBR to partition additional disks, although this feature is primarily for backward compatibility so that you can take an MBR disk from an IA32 machine and read it in an IA64 machine. IA32 versions of Windows .NET and XP can’t read a GPT disk, so you can’t do the reverse and remove a GPT disk from an IA64 machine and put it in an IA32 machine.

IA64 Windows can’t partition removable media disks such as Jaz, Orb or Zip using GPT. The GPT specification describes these as superfloppies and only permits MBR partitioning. Detachable drives such as Universal Serial Bus (USB) and IEEE 1394 drives, or shared SCSI/Fibre Channel drives in a cluster, must also be partitioned as MBR rather than GPT.

A GPT disk could potentially be damaged by utilities that expect to see and write to an MBR. The GPT specification anticipates this problem and requires including a protective MBR on the first sector of a GPT disk. This MBR performs no real function other than to act as a sacrificial goat for disk utilities.

They Ain’t Cheap
The first McKinley servers will be very expensive. The CPUs themselves cost upward of $4,000 for models with a 3GB L3 cache; the advanced chipsets, high-speed memory, 64-bit PCI adapters and Infiniband peripherals (when they come to market) also cost quite a bit more than their 32-bit counterparts. Still, lots of top-flight technology is waiting in the wings to take advantage of 64-bit processing, and prices will decline as volumes improve. Even so, widespread IA64 adoption may take a few years. Look for the first production 64-bit applications (mostly large database applications) to appear near the end of this year. By the end of 2003, mainstream applications will have been ported to native IA64 code. By 2006, it’s possible that you could be retiring the last of your feeble quad-Xeon servers.