In-Depth

Gaining Control Through Enterprise Process

How new policies and procedures can help you gain control over processes, ranging from patch management to topology changes.

“Can you get another multi-site Exchange organization built … by tomorrow …?”

“Rebooting the file server will fix that problem. I know there are users on it, but no one will know I did it.”

“I know no one else is using Visual Basic .NET, but I really want the new features. Can’t you install it for me?”

Life as an IT person is filled with these little temptations from users, managers and ourselves. They come with the best of intentions, and sometimes we make inappropriate changes just to get people off our backs. It’s not until months down the road that we find the accumulation of all these undocumented changes creates a major headache when something breaks.

When Microsoft Certified Professional Magazine named me their Editor for a Day, they asked, “Bring us the issues from down in the trenches. Give us a topic our users need to hear about.” So, rather than focus on the next best technology from Microsoft or next new server platform, I thought it best to step back and take a sweeping view of the process and procedures that govern how we do our daily jobs.

Working in the IT department for Raytheon Company in Aurora, Colorado for nearly five years, I’ve experienced some great successes. I have also been a part of some spectacular failures in technology deployment driven from inappropriate schedules, lack of management buy-in, or poorly written documentation.

Throughout these five years I’ve played a minor part in redesigning our core IT processes, taking us from an environment of constant firefighting and uncoordinated changes to one where systems are maintained to a known state, and where nothing changes without group agreement.

Why Enterprise Process?
As a general concept, “enterprise process” is difficult to envision; but the goals it aims to achieve are easy to understand when related to the tasks that make up your work day. Think for a minute about what you do:

  • When you bring a new system online, are the steps to get it up repeatable?
  • Is the knowledge required to fix today’s problem easily transferable to new teammates?
  • Are you comfortable that today’s changes to your environment won’t affect tomorrow’s uptime?

From a management perspective, the establishment of enterprise processes ensures that expensive technology lessons are learned only once and that each employee’s daily tasks are completed in a consistent, repeatable fashion from which others can learn.

More Info
For more information on Microsoft’s view of Enterprise Process and Change
Management, take a look at http://www.microsoft.com/technet/
itsolutions/techguide/msm/
smf/smfchgmg.mspx
.

From the perspective of the IT trenches, establishment of common processes agreed upon by both management and IT ensures that our environment, servers, workstations and applications won’t change from a baselined state without our consent. It also protects us from an often incessant stream of inappropriate requests brought about by over-zealous management or unreasonable time schedules.

Sounds like a great idea, huh? How do you start? The first step is to baseline your environment.

Bring It to Baseline
Not long ago Microsoft released Internet Explorer patch MS03-048 with four different versions. This patch used a different installation file depending on whether the system was running IE Version 5, Version 5.5, Version 6 Service Pack 1, or Version 6 on Windows Server 2003.

Keeping Track of “Builds”

In my program area at Raytheon, we started with the concepts of server and workstation baselining firmly ingrained into our user’s minds. Office users, administrative assistants and management types are given our “standard build.” This build includes Microsoft Office, Adobe Acrobat, Lotus Notes and a few other miscellaneous tools needed by everyone in the program.

As we’re primarily a software development shop, our developers have a different build. Our “Dev Build” is a superset of the “Standard Build” with additional tools for software developers.

All software that makes up our servers—our standard build, dev build and additional software installations—is stored in our software tracking database in Microsoft Access. We use this database to report software licensing metrics to management and locate machines where patches or software updates might have conflicts prior to deployment.

At that time, the Raytheon network in Aurora encompassed more than 1,200 desktops and dozens of servers and laptops, yet large sections of the network hadn’t agreed to undergo a baseline control process, so the installation of this critical patch took more than a month to complete, as the IT department scrambled to identify which machines needed which patch. Many of the workstations were patched manually by techs in the field rather than through an automated process.

Raytheon Aurora is made up of a number of separate software development programs. Fortunately, my program agreed early on to complete a computer baselining activity where nearly 400 workstation and server configurations were captured into a central database. All workstations are imaged with standard software builds, and any individual changes to that build are captured to the database.

Because of our baseline, it took just 48 hours to patch more than 95 percent of our systems. Only one of the four patch variants needed to be deployed to our computers, and zero software conflicts were experienced.

Enterprise Process Decision-Making Flowchart
Figure 1. Enterprise Process Decision-Making Flowchart.

Start with the Servers
Bringing your computers to baseline sounds like an ominous task. But the process of baselining your environment needn’t occur overnight. Since there are fewer of them, start with your servers. Documenting every detail of their hardware configuration and installed software can involve little more than Microsoft Word and a screen capture utility.

If you’re running Windows 2000 Server or Windows Server 2003 and your company security policy allows, use Terminal Services to connect to your servers and take screen shots of every system configuration screen. By pasting these screen shots into a Word document, even the most complicated configurations need take a only a few hours to complete.

Compile your screen shots along with other pertinent information like hardware composition, installed software and patches, and software installation location into a central file location on your network—this is your “baseline.”

For larger environments, tools such as Systems Management Server automate the process of ascertaining the software and hardware spec for each piece of server equipment in your environment.

Next, the Workstations
At Raytheon Aurora, we’ve created our own “build tank.” To this location, new workstations arriving from HP as well as previously used ones needing re-imaging are delivered. Once there, each computer’s software configuration is wiped clean and one of two “standard images” are squirted onto the system using Symantec Ghost.

Each and every computer leaves our build tank with a known configuration, including known and tested software and drivers. Because of this baselining of software and drivers on each system, our number of help-desk tickets has been significantly reduced.

Once servers are completed, the process of baselining workstations can begin. Standardize on a workstation image and use Ghost or HP Altiris to replicate this image to your workstations in a consistent and repeatable fashion.

If you’ve just started this process, keep “swap” machines available to immediately deliver a fresh machine to each user as you pull their old machine for re-imaging. Once the machine’s re-image is complete, it becomes the swap machine for the next user. For machines with special software or patch requirements, start a database that details how each machine’s software build deviates from your baseline, and information about all non-standard software products.

If your environment contains major groupings of applications, consolidate on as few images as possible. Maintenance of as few workstation images as possible means less work to update your “golden images” as you get new patches or software updates.

Your build tank need not be complicated or expensive. At Raytheon Aurora, our build tank is nothing more than a three-shelf metal rack rescued from the recycle bin. Zip-tied to this rack are 14 individual cable drops that connect back to a central KVM and network switch. The entire rack sits on its own subnet with a file server that hosts our Ghost server and the image files. The entire assembly was built for less than $3,000.

Admin Rights: Use ‘Em or Lose ‘Em
Once you’ve obtained a workstation baseline, it’s reasonable to consider granting admin rights to your users as the end of your baseline. Notwithstanding who sits in front of the computer all day, the ultimate owner of that workstation is the company, and the ultimate reason for that workstation’s existence is for the benefit of the company.

Our users used to view admin rights as a privilege granted to them along with their medical and dental benefits. I can’t tell you how many times I’d go to a user’s desk to find the reason for their system crash was a recently installed MP3 player.

It’s worth saying again: The widespread use of admin rights is the death knell of workstation stability.

To maintain stability, you’re going to need management buy-in. Involve management in your baselining plan. Discuss in terms of dollars and cents the value of lost company productivity and increased workstation downtime attributed to workstation mismanagement. Management should understand the additional cost involved when users retain admin rights they likely don’t even need.

In our case, we obtained upper management buy-in on our process early on. The process is as follows:

  • Initially, no one gets admin rights.
  • To obtain admin rights, an employee is required to review a short training presentation with their manager that details the responsibilities required of a person with admin rights.
  • The individual and their manager then send an e-mail to a special mailbox agreeing to the policy and certifying that they won’t make unauthorized changes to their workstation.
  • Once we receive and file the e-mail, we grant them admin rights.

We explain in the training presentation that admin rights can and will be revoked by IT should evidence be found of rule-breaking. Removal of admin rights means that the user can’t perform their job function; the employee’s made aware of the results of not being able to complete their job.

In the three years this policy’s been in effect, we’ve had virtually no issues requiring us to revoke a person’s admin rights.

Decision-Making by Committee: The Change Control Board
After agreeing on a firm baseline, you next need a process to modify that baseline. Enter the Change Control Board (CCB). In parallel with the baselining of your environment, you should develop a plan for establishment of a regular CCB meeting where changes to baseline configurations are discussed and approved by a committee of peers.

At Raytheon Aurora, many Windows system administrators (SAs)—including myself for a time—initially saw the establishment of the CCB as a slap in the face to our own capabilities as talented SAs. However, as I saw—and you’ll see—there are real benefits in having a “support group” that shares the responsibility for each change to your environment.

Because of the size and complexity of the IT department at my company, the process of establishing the CCB, defining the rules, determining what should and shouldn’t be brought to the CCB, and who should be in attendance was a multi-year process. It involved a lot of arguing and nearly resulted in the loss of some very talented SAs.

Most of the time involved with setting up our CCB was spent dealing with political issues and calming the nerves of SAs who were used to making independent changes to the environment, often without anyone else’s knowledge.

Once our group realized how useful the CCB could be in ensuring that every change was agreed upon through group consensus, and how our jobs were somewhat protected as part of that group, we grew to appreciate its usefulness.

Your CCB can meet as often as necessary, but the meeting times should be regularly scheduled and rarely, if ever, modified. The establishment of set times for your CCB meeting ensures that individuals with political agendas can’t modify the meeting time in order to approve a change they haven’t clearly thought through in the first place.

Check out “CCB Rules of Engagement” for some guidelines you can use to setup the rules for your own CCB.

12 Change Control Board Rules of Engagement

1. The CCB should be composed of regular representatives from your major IT groupings: technical support, systems administrators, networks, network security, and IT management. The CCB should also be open to other interested parties, including those not from IT.

2. Changes should be submitted in writing, using a standard Change Request Form sufficiently prior to the CCB date and time that your peers can review the change and provide comments.

3. Completed Change Request Forms should be e-mailed to CCB participants when completed. Any interested party within or outside the IT department should have the ability to receive these Change Request Form notifications.

4. Issues with any particular change shouldn’t be brought up during the CCB, but instead worked out prior to the meeting. Minor edits or clarifications, though, can be brought up in the meeting.

5. For changes where issues can’t be resolved through peer communication, the issue should be elevated to the Technical Interchange Board (discussed later).

6. The CCB isn’t intended as an approval body for major project plans. Large projects and their accompanying test plans, implementation plans, schedules, and user training guides should first be approved by the Engineering Review Board (discussed later).

7. Once the ERB has approved a major project plan, the individual changes to your environment and their timeframe for implementation should then be approved by the CCB.

8. Consider inviting one non-technical member of management to the CCB to represent their concerns such as customer impact and risk mitigation. The presence of this management liaison ensures that CCB decisions are carried to your company management team through the most direct route.

9. A list of minor changes not requiring CCB approval should be agreed upon and documented prior to the first CCB meeting.

10. Change Request Forms should be stored in electronic form. The ongoing record of change requests will serve as the beginning of a searchable repository of IT knowledge. If your company uses a work order tracking system to coordinate help desk tickets, use this system so all your IT knowledge is consolidated in a single searchable database.

11. Some provision for emergency changes should be made, because even the best-laid plans go to waste in an emergency. Your process for Change Request approval should include some method to short-circuit the process during crises.

12. When a crisis does occur, have a standard Incident Report Form similar to your Change Request Form. This form is brought to the CCB for review at the conclusion of the crisis. The sum total of your Incident Report documents will populate your database with a history of lessons learned.

 

Setting up a CCB
The CCB is the committee equivalent of “the buck stops here.” No change occurs in your computing environment without approval of your CCB.

Any change to your enterprise relevant enough to be approved by the CCB should be documented into a standardized Change Request Form. This Change Request Form is submitted to the CCB members through some medium (typically e-mail) sufficiently prior to the CCB meeting time that your fellow SAs have time to review the change and reply with comments.

In order for a Change Request to be approved by the CCB, all comments or issues with the change must be adjudicated prior to the CCB meeting. At the CCB, each individual Change Request Form is reviewed by the members in attendance and approved or “rubber-stamped” by a process of acclimation.

At Raytheon Aurora, we found it easiest to automatically approve all changes unless one of the following three conditions aren’t met:

  • The CCB Chair determines that not enough pre-approval due diligence has been completed on the part of the submitter. This could be missing documentation, lack of communication with fellow SAs, the network team, company management, or missing user documentation.
  • A member of the CCB determines that the time for the change is inappropriate. The individual wishing the change in implementation date should to provide an appropriate alternate date and time so no Change Requests are left permanently in limbo.
  • A member of the CCB flags the Change Request as having a remaining issue not adjudicated by the Change owner.

As any member of the CCB can flag a Change Request and delay the implementation of the change, the onus of responsibility lies on the Change owner to ensure that:

  • The change is truly necessary, and therefore good for the business.
  • They’ve researched and documented what they believe is the best possible solution to the problem or issue at hand.
  • They’ve coordinated with their peers to resolve any possible conflicts that could come out of the change.

These three points are the key to why your organization needs a CCB. Lacking a CCB, a company is missing a system of checks and balances on administrators. Since your SAs typically have advanced rights and privileges in your Windows domain, and since SAs could make changes without fully thinking them through, the “peer pressure” of an authoritative body of your peers ensures that changes are researched, documented, and implemented carefully and thoroughly.

When we first setup the CCB at Raytheon Aurora, we set up some skeleton rules but made the decision to work on the “details” later as we learned what was appropriate and what wasn’t for Change Control.

This process of organizationally learning what was and was not pertinent was a challenging one. For months, the individuals involved with our CCB struggled with issues such as:

  • What changes need to be brought before the board?
  • In an emergency, what should I do first? Should I first work on the documentation, get approval, or simply begin working on the emergency?
  • If I can’t get consensus on a change I want to implement, where do I turn?

The answers to these questions will be different depending on your company’s size, tolerance of downtime, and your user culture. You should, however, have some understanding of the impact of these issues as you begin writing the document that defines your CCB.

Find Help and Avoid Group-Think: The Technical Exchange Board
Setting up an Active Directory at Raytheon Aurora was—and is—a very hot topic. With our AD implementation touching the daily workflow of nearly everyone in our IT department, every person in that department seemed to have an opinion on how it should be implemented.

The actual task of implementing AD was handed to an engineer in our Operations department. The engineer had opinions on how the AD should be implemented, as did many others. The wide range of opinions virtually ensured that no Change Request for its implementation would ever be approved by the CCB.

For situations like this, we have an established process where conflicting opinions can be resolved in a timely manner using a formal structure. That process takes place in the form of our Technical Exchange Board.

Unlike the Change Control Board, where people come together every week to rubber-stamp agreed-upon changes to the infrastructure, the Technical Exchange Board uses a less formal approach to resolving sticking points among the greater IT group.

Your Technical Exchange Board should come together on an as-needed basis and remain solely a gathering of peers for the purposes of mediation. Its purpose is to provide a structure for open and candid conversation between technically-minded peers, with the intention of coming to a group consensus on a most-appropriate solution for the technical problem at hand. The chair of the board should be responsible solely for maintaining productivity within the group and guiding the conversation to the most appropriate conclusion.

The Technical Interchange Board should be a tool for mediation and cross-IT education; not an approval body. This ensures that decisions made there are done by common group agreement in preparation for CCB or Engineering Review Board approval.

At Raytheon Aurora, we’ve convened the Technical Exchange Board for topics ranging from implementation of Microsoft Software Update Services using SMS, to deployment of a new DNS structure for the company, to the aforementioned AD implementation. In each case, everyone in the room eventually came to an agreement on a best-possible solution—even when that solution required repeated meetings.

Reaching Process Consensus: The Engineering Review Board
The final component of a successful enterprise process deployment is the establishment of an Engineering Review Board (ERB). This board, comprised of a small group of high-level systems engineers and members of management, is responsible for the approval of new processes and the review and subsequent approval of large projects.

Within your process documents is where the various review boards discussed earlier are created, where they’re given their authority, and where the procedures that guide your workflow are defined. The ERB is responsible for the review and approval of these process documents.

Typical membership on an ERB includes individuals with understanding of the “big picture.” They’re the individuals who understand how the approval of process documents will affect the performance of you, your computer resources, and ultimately the business as a whole.

In addition to creating and approving process, these individuals are also typically the body of authority for large projects and their associated project schedules and documentation. These long-term projects drastically change your computer environment and users’ experience. It’s here that large project schedules, impacts, risks, costs, and benefits are evaluated and approved.

At Raytheon Aurora, we’ve brought dozens of documents before our ERB for approval. We’ve sent documents detailing the process of requesting and obtaining domain admin rights and SMS rights, for establishing the formal process for obtaining call metrics for our help desk, and for detailing our formal Service Level Agreement between IT and the programs in our company.

Each of these process documents was reviewed by the members of the ERB, analyzed to determine its relevance to our core business, and approved for use. The ERB and its authority in approving each of our processes brings legitimacy to our organization.

Worth the Time and Effort
It took us a few years to get all this process structure up and functioning the way it is today. It took just as long to get buy-in from the IT people who found the process a hindrance to “getting work done”. However, I can honestly say our company has benefited from the experience.

Even though our approval process has a tendency to slow down change implementation, our records have proven that more inappropriate or inherently risky changes have been prevented than time-critical ones have been slowed down. In our old working environment, our management would expect us to jump to deploy new technology that may or may not jibe with everything else on the network. In our new working environment, all requests for new network services require pre-implementation due diligence and an plan of action before they’re green-lighted.

Enterprise process may be a check on your virtually unlimited power as a domain admin, but it will save you when you need time and the peer support to ensure your job is done correctly.

Featured