In-Depth

Smoke, but No Fire for TCO

Total Cost of Ownership (TCO) is a concept touted by Microsoft and various Linux vendors as proof that their products are cheapest to run. But TCO claims aren't what they're cracked up to be, and most IT shops never use TCO, or just plain do it wrong.

• By Stuart J. Johnston
• October 01, 2005

The debate about which operating system delivers the best total cost of ownership is over—over-rated, that is.

TCO has been a major battlefront in the Windows vs. Linux marketing wars for years. But it's a complex task to actually determine what costs comprise TCO, and it's even trickier to figure out how to compare one system with another.

In fact, many companies that claim to have achieved a lower TCO with one system over the other cannot or will not disclose how they actually did the measurement. And to protect proprietary advantages, many on both sides of the debate are reticent to open the TCO kimono.

Also, many shops that claim to have done thorough TCO comparisons may not have been as thorough or complete as they think—or as their vendors would like you to believe.

Indeed, many IT organizations only get part way down the path to a rational, detailed comparison before picking a platform. And—no surprise—the winner is often the one IT staffers are most familiar with.

At the end of the day, incumbency and familiarity count for a lot. That means Windows servers will continue to dominate for now, although lines between the two systems' capabilities have begun to blur as Linux has matured. And that just makes it harder to come up with any real numbers to compare them.

The Problems with Surveys
Most, if not all, TCO studies have to be taken with a grain of salt. Maybe even a block of it, many analysts say.

First, there's the issue of who commissioned the study.

"I have trouble finding a study without it quickly being disqualified by who funded it," says Rob Enderle, principal analyst at consultancy Enderle Group.

However, the problems with any TCO survey or study go much deeper than whether an interested party sponsored it.

Analyst Rob Enderle, of Enderle Group, says many, if not most, TCO studies are tainted by their source of funding, which is often a company with something to gain.

"[TCO studies] are like a house of cards … where's the proof?" asks Michael Cherry, lead analyst for operating systems at consultancy Directions on Microsoft, pointing out that even subtle psychological biases can spin a study's results.

In addition, accusations often are made that one side cheated and stacked the deck in favor of their system, particularly in non-production simulations.

Apart from questions of bias lurks an even bigger analysis problem: How do you model any specific-enough scenario to provide meaningful data without being so specific as to make one company's comparison to another company's scenario unrealistic?

When evaluating a TCO survey or case study, there are a number of questions you should ask in order to get a proper comparison between the case study and your particular situation. Here's a starter list:

• What do you measure?
• For how long a period?
• How many parameters should you examine?
• How do you quantify and allocate costs?
• What is a legitimate expense to include and what isn't?
• How do you model a real-life situation?
• Where do you get your data?
• What's a reasonable lifecycle for a given system or application?

Some TCO "studies" are so simplistic as to provide a spreadsheet with blanks where users fill in costs for mostly big-ticket items such as software or staffing. While these may be useful in a very broad-brush sort of way, many analysts criticize them for being too general and therefore not a good basis for decision-making.

"Nobody's infrastructure is the same, so it's unrealistic that you could do a vanilla survey that fits everybody," says JupiterResearch senior analyst Joe Wilcox.

Wilcox and other analysts point to comparisons that use "list" prices for Microsoft software. In reality, many enterprises participate in Enterprise Agreements and other volume deals, which can dramatically lower costs based on volume and commitment.

There are others issues as well. Are survey respondents self-selecting, as can often happen in a Web-based or call-in survey? That can prejudice survey results because only interested parties respond. Phone-call-out or in-person randomized surveys are generally more accurate but more time consuming and, thus, more expensive to conduct.

That brings up another important question: What's the sample size? The smaller the sample, the less accurate the results.

Results can also be off because of the number and depth of questions or number of variables in the model. The possibilities for variables and options are dizzying.

"You end up with a number that's as accurate as an EPA mileage rating … 'Your mileage may vary,'" says Directions on Microsoft's Cherry.

Ultimately, analyst Wilcox questions how much weight even a thorough TCO analysis should get in the decision-making process. "I think cost containment is the wrong priority for making technology decisions," Wilcox says. "You should be thinking about the business."

That same sentiment is echoed by Robert Rosen, president of SHARE, the giant IBM user group. "People aren't looking at IT as a cost center any more [which means] TCO isn't the driving issue … it's the total economic picture," Rosen adds.

The Devil Is in the Details
Most of the TCO studies publicly available were either funded by vendors or are at least a year out of date, or both.

One that's not is the Yankee Group's "2005 North American Linux and Windows TCO Comparison Survey."

Released last spring, it follows Yankee Group's 2004 survey, which was roundly condemned by the Linux community because a portion of it was hosted on a Web site owned by a Microsoft Certified Gold partner. While Yankee Group defended its methodology, the 2005 survey was done with no involvement by any outside parties.

It also has twice as much detail as the 2004 survey. The firm's March 2005 survey reached 550 IT decision makers and asked respondents 50 questions, double the number in the 2004 survey. Despite that, the report's conclusions were similar in many respects to 2004. Neither report actually spells out any bottom line numbers comparing TCO for Linux vs. Windows, instead identifying trends.

In 2005, for instance, a study by Yankee Group analyst Laura DiDio found that "there is no universal clear-cut TCO basis to compel the corporate masses to do a wholesale switch from Windows to Linux as there is for a migration from Unix to Linux, and there is no indication that users are replacing Windows with Linux."

While that's good news for Windows, Linux isn't going away. The same report found that "from 2004 to 2005, Linux maintained—but did not expand—its healthy 15 percent market share—compared to 73 percent market share for various versions of Windows servers."

DiDio also found that while more than half of all respondents said they'd done a thorough TCO comparison in advance, when asked, "on average 75 percent could not answer explicit questions."

What's missing, DiDio says, is information and metrics that you would expect most IT shops to normally gather and evaluate, such as how long on average it takes to recover from a system crash, how many calls are made to the help desk in a typical day, the average hourly cost for downtime for Windows and Linux, and what kinds of workloads a company puts on Linux vs. Windows servers.

"If you don't have those estimates, how do you do a TCO? That's scary," DiDio adds. Without that information, she wonders how those IT shops run their businesses, much less calculate TCO.

Michael Cherry, of Directions on Microsoft, says TCO comparisons are problematic due to the huge number of variables involved.

Familiarity Breeds Respect
The answer is that many companies do not really conduct TCO comparisons, even when they say they do. Witness the percentage of respondents in the Yankee Group's 2005 survey who said they had performed TCO analyses but couldn't cite detailed costs needed for the calculation. Despite the TCO lip service, other criteria tend to win out in choosing a product, according to both analysts and IT managers.

Applications are the biggest criteria for platforms, and it's often the case that a key application is available on one platform and not the other. "The TCO is roughly the same [for Windows and Linux so] the deciding factor is 'Are the applications users need available [on the platform]?'" says SHARE's Rosen.

But perhaps most important is the current staff's familiarity with the system. It's a touchy subject.

"One thing we find in our research is so much of what gets adopted has to do with [IT users'] familiarity and their skill sets [and not TCO concerns]," quietly admits a vice president at one of the largest analysis firms who asked not to be identified.

And while measurement before the fact is suspect, TCO measurement after the fact is almost non-existent. "All of the TCO [analysis] that I've ever seen was forward-facing projections … I never saw an audit of TCO [after the fact]," says a highly experienced manager of technology and operations for one of the largest county governments on the West Coast. And when he went looking to see what other IT organizations had done, it was "like nailing jelly to a tree."

"It was impossible to find anyone that actually did TCO [calculations] … so I came to the conclusion that TCO is a marketing tool," the manager adds.

That TCO takes a back seat to more important issues is true for both platforms, it turns out, one vendor reports. "We've never had anybody ever complete [a TCO study]," says Don Keeler, CTO of Lumen Software, a Linux portal technology vendor in Kansas City, Mo. "We thought originally there would be a need to do TCO evaluations, but users really just wanted to get their problems solved," Keeler continues, adding, "It's the application [that sells Linux] because we don't lead with Linux at all."

Windows incumbency can also seal deals. "Everybody we had knew Windows," says Nick Brookins, former IT director for Computer Builders Warehouse, a custom assembler of PCs and servers headquartered in Warren, Mich., of that company's 2003 decision to consolidate both its internal Linux and Windows-based servers onto a single platform. While he reckons the company saved somewhere around $600,000, Brookins says that's an estimate based primarily on decreased staffing levels. "We had two separate staffs [prior to moving everything onto Windows], and when we staffed up, it was a lot easier to find Windows IT talent."

All of which underlines the unsettling conclusion that TCO, while often discussed, is rarely calculated.

"Most organizations don't do TCO [and instead] they make decisions on other bases," agrees John Rymer, vice president in Forrester Research's application development and integration group.

That, he says, is short-sighted. "TCO is a long-term cost analysis. If you can quantify those longer-term costs and benefits, you can use that information in a lot of places," Rymer adds.

Home-Court Advantage
For the time being, the older and more entrenched Windows has what Wilcox refers to as "home-court advantage." His point, and one with which most analysts agree, is that incumbency most often beats new challengers—no matter who the incumbent is.

"When I look at Linux, the playing field isn't level because Microsoft is the incumbent," says Wilcox.

"Forty-eight percent [of companies] have Windows end-to-end. That's a big number [and] if you're running Windows already, you're probably going to get some cost savings from standardization," Wilcox says. "You can't ignore switching costs [such as] moving data, training, additional support, management and on and on … it all adds up and it makes TCO analysis that much more difficult."

Certainly, Yankee Group and other firms feel TCO studies are still a valuable tool when constructed using proper survey techniques that gather enough detailed information.

What even the Yankee Group studies show, however, is that Linux is making inroads into the data center. Eleven percent of the companies surveyed in DiDio's 2004 report planned to migrate entirely from Windows to Linux in the data center. An additional 4 percent planned to migrate all their Unix systems to Linux.

"The metamorphosis of Linux from a free, hobbyist software environment to a major revenue-producing operating system is occurring with the same surety and swiftness of a neighborhood undergoing gentrification," DiDio says.

To date, however, many, if not most, of the Linux deployments in enterprise environments still come at the expense of more expensive Unix boxes, and not Windows Server. "[Linux] is brought in by corporations' Unix groups, so the first thing that's displaced is Unix," says Wilcox.

Hearing Footsteps
Longer term, however, perhaps in as few as two years, Microsoft may be struggling to maintain its dominance inside the walls of data centers that it only recently breached itself. Over time, TCO studies or not, it's obvious that in the future Windows will be sharing more of the data center's rack-mount space with Linux.

In the long run, all of the angst and number twiddling may not matter a lot. Given the dearth of accurate information on TCO, many IT customers have come to the conclusion that Linux will soon match Windows in terms of capabilities, if it hasn't already.

For many IT decision-makers, that's why TCO is pretty far down the list of considerations when it comes to upgrade or replace systems, or build new ones.

Several analysts say rightly so.

"Linux vs. Windows TCO studies are great business for analysts and wicked sales propaganda for companies, but I don't think high-tech vendors or their customers should take much stock in them," JupiterResearch's Wilcox said on his Weblog after being interviewed for this article.

Many industry observers say you should use TCO models and surveys as a tool, the same way you would ROI analyses, tête-à-tête product reviews/shoot outs, business needs analyses, and similar-business comparisons. In other words, be skeptical, but don't ignore them completely, because they can provide value.

"TCO is just a data point," says John Hogan, vice president of strategic marketing for Novell. "What is perhaps more effective is when you can steer a customer to someone who has a similar situation."

"You can't look at costs in a vacuum, so they're [IT managers] not looking at TCO but instead are looking for the net gain," says Rosen.

Amanda Morgan agrees with both Hogan and Rosen—but only to a point. After all, she's global campaign manager for Microsoft's "Get the Facts" program, which has spearheaded Microsoft's TCO studies strategy. TCO is not the only criteria for evaluating systems investments, she concedes, but it is an important one that customers ask about frequently.

"CIOs are really in a tough space, especially in non-technology companies," Morgan says. "They see large amounts of money [being spent on IT] and really want to know about their ROI over time." But, she admits, "It's not the sole element of our conversations with customers."

Note: Several large and midsize companies whose Windows TCO success case studies are posted on Microsoft's site declined to be interviewed for this article.