The Director of Microsoft’s Certification and Skills Assessment Program educates us on benefits, new exams, and the quality question.

Donna Senko Talks Back

The Director of Microsoft’s Certification and Skills Assessment Program educates us on benefits, new exams, and the quality question.

Education lies at the heart of Microsoft’s certification efforts. After all, without skilled people trained to implement, administer, and manage Windows NT, most of us might still be Novell customers. That’s why it was appropriate that the Redmond company look to somebody with an academic background when the time came to hire an executive to direct the certification program.

Donna Senko holds a doctorate in German language and linguistics. She has taught at the college and university level in the U.S. and Germany. But she also has a business background based in training, most recently in a lengthy stint at Santa Cruz Operations. As group manager of training, curriculum development, certification, and technical publications, Senko restructured SCO’s “ACE” certification program and re-designed its training and curriculum businesses. Prior to that she held management positions at Hewlett-Packard.

Since January 1997, she has directed Microsoft’s certification and skills assessment programs worldwide—an effort that currently involves 360,000 participants in 100 countries.

That job hasn’t necessarily been easy. While the certification program has experienced outrageous growth in the past two years (from 100,000 in February 1997 to 360,000 as of October 1998), some MCPs have accused Microsoft of pushing quantity over quality, cutting back on benefits, and allowing unskilled people to obtain titles too easily, thereby diluting the certification’s value.

Editors Linda Briggs and Dian Schaffhauser caught up with Senko during MCP Magazine’s TechMentor conference to discuss all of these topics.

MCP Magazine: In a discussion in the MCP Magazine online forums with [Microsoft’s Marketing Manager Eckhart Böhme], one of the topics that kept coming up was MCP benefits. The participants seemed a little disgruntled about the level of benefits right now; I’m sure you’ve heard that feedback yourself. One of the things Ecki mentioned is that a survey is underway now exploring benefits. Can you expand a little bit on that?

Senko: Actually, we did the survey last year, when we were looking at the whole program and re-evaluating the program and its benefits. We looked at two factors. One was the usage of the benefits and how many [Microsoft Certified Professionals] use the particular benefits. And the other was about what benefits were the most important to MCPs. Based on those two factors, we made the changes to the program.

So, for example, in the case of the [technical] support benefit, we found the usage rate incredibly low. There were a few people who were very interested in that benefit, but something like one percent of the eligible people actually took advantage of that benefit. The average usage rate of those people was between one and two support calls that they used out of the 10-pack they were given—really low.

Also, we did a survey of over 2,000 MCPs worldwide—2,000 people who responded worldwide. And we had them actually rate the different benefits that they receive and put them in a ranking order of which benefits they consider the most important. Based on that, we determined the new benefits package. One thing that came out high on the list was technical information and access [to it] through Web sites—through the Internet. So that is the direction we decided to start moving in. We’ve only just started.

Can you share with us at all what kinds of things you plan to add to that MCP site?

This past year our goal was to set up the infrastructure and start with some of the basics, so that people could download their logos and use the transcript tool. That was our starting point—with some technical information [also].

The goal really is to use that as our pipeline of technical information to the MCPs, because that’s the thing that they’re really hungry for, and that’s the best way of us providing it to them. So that’s the direction.

It’s interesting that you found usage of the technical support calls so low, yet now MCPs are demanding that you give that benefit back.

Well, I have mixed feelings about that. I think with individuals who were using them, they probably were one of the very important benefits. But I do think that a few people who are dissatisfied can sound like a number of people. And so we really tried to take a broad look at how we’re going to satisfy the greatest number of people—what makes the most sense for the program and for individuals worldwide.

MCPs who are part of the Certified Solution Provider channel can get support that way.

Developing the MCSD Program

The new Microsoft Certified Solution Developer track seems to be attracting a lot of attention. Are you on schedule for releasing the new exams?

The major exam, the Solution Architecture exam [70-100], should be in beta form in November, which has actually been our target date. We have been saying Fall and it’s [scheduled for] November right now. We’re on track with that; the other exams will [be released] roughly one exam into beta per month until they’re rolled out. We feel strongly that we have a solid program in place now, and we’re going to meet those commitments.

Can you talk about marketing efforts for the new MCSD? There seems to be a general feeling that with more marketing, the title would take off.

You’re going to hear a lot about the MCSD this [fiscal] year. Now that we have the new program in place and [in the process of] being rolled out, we have a number of initiatives planned. One is Visual Studio Developer Training Month [which took place in October, 1998.—Ed.]. We’ll be offering an exam voucher for any core exam for the MCSD to any participant in that training wherever the Visual Studio training initiative is launched.

That was initially planned in North America. But quite a number of subsidiaries [outside the U.S.] have come to us and expressed interest there too; so we’ll honor the same commitment to the subsidiaries.

We’re going to ‘incent’ current MCSDs to migrate to the new program. These are people who are already in our program; we’re dedicated to helping them move to the current technology. We’ll be having promotions for them, probably beginning in the winter timeframe. It will most likely be in the format of a two-for-one exam promotion.

The bottom line is that it will be easier for individuals who are currently MCSDs to migrate to the new program than to update in the existing program. For example, rather than going from the WOSSA exams to the [Windows Architecture exams], we extended the life of the WOSSA exams, so that people can skip right into the new program. We really want to encourage everyone to move to the new program. However, people will have through July 1, 2000 to move. So there’s a lot of [leeway].

We also have an awareness campaign planned around it, and a couple of other ideas, too, that by the end of the year we’ll probably be rolling out. So this is really the year that you’ll be hearing about the Solution Developer.

Will we see something in the Visual Studio product box about education and certification?

I’m not certain at this point.

What about working with large corporations to encourage them to bring MCSDs up through the ranks?

One of the things that we’re doing at the same time is expanding the definition of what an MCP is. [As of October 1, 1998, people who pass a single Microsoft exam, with the exception of Networking Essentials, earn the MCP title.—Ed.] Because the new structure of the MCSD program is really quite different than the old structure, we want individuals to have a stepping-stone from being an MCP to becoming an MCSD. So we have broadened the definition of an MCP to accomplish that purpose. There are other reasons too, but this really [encourages] people to move on.

On Retirements and New Exams

Can you give readers an explanation of how Microsoft decides when to retire an exam? For example, the SQL Server 4.2 exams are still live, as well as 6.0 and 6.5. [The SQL 4.2 exams are now in the process of being retired.—Ed.] We assume that’s because the product is still well-represented in the marketplace.

There are a number of factors that we look at. One is exactly what you just stated—how well the product is still represented in the marketplace. Another thing that could affect our decision is how close together releases are and how radically different the releases might be. We also look at the base of people who have certain exams or what the ‘run rate’ is on a certain exam. If a lot of people are still taking a particular exam, then we feel there’s a need to maintain that exam for a certain period of time.

Typically, for a major product, we would have two versions of an exam available at the same time. For example, we have Windows NT 3.51 and 4.0 exams right now. When we release NT 5.0 exams, then we’ll probably retire the 3.51 exams.

So we really look at a wide range of criteria: the product, the status of the product, status of the certification track, if it affects the certification, and the run rate on the particular exam.

But at the same time, [we don’t want to allow] people to stagnate. We try to find the right balance: Constantly trying to move people forward into new technology, while being realistic, too, as to which version of the technology they might be using.

Are you starting to see people dropping out of the program as they don’t re-certify? It’s really only in the last year or so that you’ve been retiring exams.

We found last year that we had a very low number of people who didn’t upgrade. If you looked at the whole base of MCPs, roughly one percent of the population was de-certified last year.

The majority of people are staying in the program, which is really exciting. We went back and analyzed that at the end of last year, and we were very pleased with what we saw.

Our current numbers reflect people who are current in the program. We really want to make sure that people who are not active in the profession don’t maintain a certification for the wrong reasons. For the most part we hope that people are going to move ahead.

There’s been some confusion about whether readers should follow the Windows NT 4.0 certification track or wait for the NT 5.0 track. Can you provide any insights on the pros and cons of both of those approaches?

I would really recommend that individuals go ahead with the Windows NT 4.0 track. This is our current technology, and it will be around for quite a while. It’s a very sound, stable technology. It doesn’t make sense for an individual to wait until the next release, which is several months out at this point. [Also], if someone is current on NT 4.0, we always have incentive to migrate them to the next version, which would be NT 5.0. If they’re in the pipeline it’s easier to move up. There really isn’t a good reason not to move forward with 4.0.

Can you expand on areas where we should expect to see retirements over the next six to twelve months in terms of exams?

No.

(She laughs.)

We have already stated that when NT 5.0 exams become available, the 3.5 exams will retire. Other than that I couldn’t give you the details on the plan right now.

Someone asked during the [TechMentor Windows NT 5.0] keynote whether there was going to be an NT 5.0 Server and NT 5.0 in the Enterprise exam.

We’re [going to be] doing a complete job task analysis—JTA—around NT 5.0 and the job functions of the MCSE. So we don’t know the answer to that yet. But we’re looking at all of NT 5.0 very seriously, and that might mean some revisions to the track in general—things might look a little bit different than they do in the 4.0, for example. We’re now doing that research; we’re several months away from having the results.

Just to clarify for readers, a job task analysis is what you do at the very beginning of a track.

Right. That’s what we did when we began looking at [revamping] the Solution Developer track, and it took us in a totally new direction in that particular case. Because jobs evolve, the technologies change, and what the individuals do is always changing. That’s how we keep our tracks current. We’re not just replacing exams all the time; we’re looking at the whole picture. What are individuals doing in those jobs?

Do you have any plans to provide a certification with an emphasis on database management work?

There is a data warehousing exam planned, but that’s not to say that there will be a whole track planned around that. That exam will be one of the electives for the MCSD. We’ve looked at the whole database area, along with a number of other areas. You have to realize that every year, [Microsoft] product teams come to us and say, “You know, it makes sense to do [this or that] certification.” And we have to prioritize and say, “What is the most meaningful for the marketplace today?”

So we’re very cautious about introducing new certifications. We’ve introduced the new MCSD and the MCP+Site Building [certifications] this year. We’ve looked at a number of others. One [exam idea] that has come up several times is in the area of databases—database administration, most likely. But we haven’t made any definite plans to move in that direction at this point. We want to keep the message clear and make sure we’re getting the top priorities first.

So product groups come to the certification group and say, “Hey, do an exam on my product”?

Yes. It’s really refreshing compared to years ago [when the certification group had to educate product groups on the value of certification]. I think, in general within Microsoft, people really understand the meaning of certification and the value that it offers for supporting Microsoft technologies.

Exam Piracy and Keeping Quality

We left this topic for last because we know it’s near and dear to your heart—exam piracy. First, now that you have the new exam NDA [non-disclosure agreement] in place, have you had occasion to use it against a company or individual?

We haven’t yet used it, although our intention is to use it when it’s called for.

If I could just back up a second, the NDA is part of a greater effort. We do consider exam piracy to be an issue, and we’re confronting it aggressively on two fronts. One front [involves] making some process changes, such as adding the NDA. This will mean that people are making a breach of contract if they disclose exam items.

Another process change is our retake policy. Now, when individuals take the exam the first time, they can take it a second time within one week; after that they have to wait two weeks every time they want to retake the exam.

And there are some other process changes that we’re looking into, as well as working with both of our testing vendors [Sylvan Prometric and VUE] on increasing security at the test sites.

On the other side, we’re using progressive techniques in testing, which makes it a lot more difficult to steal exam items. [With] simulations, adaptive tests, or any type of testing in which individuals actually have to do a task, there’s no way to steal that [exam question]. The only thing you can really do is walk away and say, “You need to be able to do such and such a task.” And you know what? Those are exam objectives, which we already publish. So those aren’t secrets.

We want people to know: This is what you need to be able to do to pass certain exams. [The exam process] is geared towards people being able to do job functions. By using some of those techniques, we believe we’ll mitigate the problem somewhat.

Also, we’re now creating item banks for our exams with many more items, so that the chance of [one individual] seeing all the items is much lower. We’re [creating] multiple forms of the beta exams, so that a person who takes the beta exam doesn’t see all of the items. We’re also seeding items, which means that the exams are moving targets when we’re pulling items out and putting items in.

We’re trying to hit this on a number of fronts, because ultimately the value of the credential is the most important thing. We want to support anybody—any MCP—who has attained a certification by helping to maintain the value and the integrity of the program and of the certification that someone worked so hard to gain. It’s very important to do whatever we can to [accomplish] that.

Vendors are now selling exam simulation products in which they’ve introduced product simulation engines. So you’re saying, regarding the product simulations, that if someone studies what these products do and knows how to answer that question—“This is how you add a user to the network”—that’s the point.

That’s the point. Ultimately, we want people to be able to do the task. And if they can do the task—in this case, add the user—they’ll pass that particular item on the exam.

If you learn to do the task, that’s really the key. Obviously, the broader the focus, the better. Sitting there with the real product is always the way to go, because we can phrase a question or make derivations of a question in a couple different directions. And so if you understand the context and the broader scope, you’re more likely to be able to pass the exam, as opposed to having learned one particular way of doing something. Hands-on use of a product—that’s the preferred way [to learn].

Are you concerned that some vendors are undermining the value of certification?

I’m concerned that there is that perception in the industry. I’m doing whatever possible to change that—where there’s a problem, to try to get the problem fixed; where’s there’s misinformation out there, to try to get it corrected.

So is the problem that vendors really are undermining certification, or is that the perception in the MCP community?

I think there’s a mix. But I also know that you can hear about something from one or two people, and it looks like a big problem; but it’s not always that wide scale. I’m trying to be objective about looking at where we have problem areas and how we go about fixing them. And we’re looking at things in a positive light: OK, how do we move the program forward? How do we maintain the quality? What’s the best way of getting that?

What should an MCP do who comes upon a “brain dump site?”

I recommend that they e-mail [the URL] to mcp@msprograms.com. We look at those. We’re aware of the major ones at this point.

And you act upon that information?

We do. And as we move forward, we will be acting more. We’ll be acting where we can and where appropriate.

Is there anything else you want to explain to readers?

The message I would really like to get out there is that the value of the certification and the integrity of the program are really key to us. Those are things that are very, very important to me, because I know how hard people have worked to gain that certification. It’s because of the value of it that the MCP program has such a high regard in the industry. And we’re going to do everything we can to maintain that high level. That’s why we have a number of the initiatives that I’ve talked about [both on] the process side and the exam development side—along with some of the legal issues, such as the non-disclosure.

Do you feel pressure from above at Microsoft to bring up the numbers of certifications—to make it less tough to become an MCSE, for example?

I haven’t felt that pressure at all. Really, I think the numbers have grown themselves. I think the reason they’re growing is because we have that high bar of quality. And I believe that the executive team [at Microsoft] recognizes that. I personally have never once had any executive come to me and say, “How are you going to increase your numbers of MCSEs?” That is really not the directive I’ve been given.

And when we introduce new methods of [testing] to maintain the quality and innovative exam types, these are well received internally. I think everybody understands the value issues, the value proposition.

So what is the directive you get from executives at Microsoft?

We’re here in support of product sales. So we need to be creating the right strategies and the right certification tracks to do that, to support our products and to support our technologies. That’s really the direction they want me to follow. Nobody comes to me and tells me I need to produce so many MCPs. I set the quotas. I do it by looking at our numbers and the new tracks we’re introducing. But nobody comes to me and gives me some magic number.