4 steps to a successful Technology Evaluation

When you get an Enterprise Architecture established, it's critical to institute a Technology Evaluation process, to methodically analyze and introduce new technologies as needed. As an organization, you'll want to ensure new software purchases are adequately governed, to continue striving towards the EA nirvana of reduced IT and business process complexity.

As I see it, there are four key steps to running a successful technology evaluation:

1.) Gather Requirements
This is the most critical step of the evaluation. The requirements must be detailed enough to quantitatively score the vendors' abilities to meet them, and be from each stakeholder group in the business and IT. This must happen first, so that you get a good idea of everyone's requirements before vendor demos happen. If not, everyone's requirements later will look oddly enough like the shiniest vendor's brochure.

Additionally, don't forget to weight your requirements. Some requirements are simply more important than others, so you'll want to adequately weight them with percentages to reflect these priorities when the vendor ratings are applied later.

2.) High Level Evaluation
Next is to put together the long list of possible software vendors that could meet your requirements. You may already know of some from your past experiences, but other good sources of vendor information are Forrester, Gartner, and any procurement partners your company may work with.

Working with this long list, perform a high level software vendor analysis. If you don't have much hands-on experience with the software options, schedule some vendor demos, or send each vendor a Request for Information (RFI). By the end of this phase, trim down your vendor list to 4 - 5 vendors so you don't get stuck in analysis paralysis in step 3.

3.) Detailed Evaluation
This is where most of your time will be spent during a technology evaluation. The two most important components of the detailed evaluation are:

• Sending a Request for Proposal (RFP) to each vendor that contains all of your detailed requirements, with a quantitative list of possible responses for as many requirements as possible, so you keep your detailed reading of free-form text boxes to a minimum. An example of a quantitative response list would be: 5 (Meets requirement out of the box), 4 (Meets requirement with minimal configuration), 3 (Meets requirement with significant configuration), 2 (Meets requirement with customization), 1 (Meets requirement with third-party tool integration), 0 (Does not meet requirement).
• Conducting detailed vendor demos. Be sure to give the vendors plenty of guidance in advance on specifically what you want to see during the demo, and capture your team's feedback on the vendor immediately after each demo to make sure nothing is forgotten.

Sometimes you will want to speak with customer references as well, especially if you want to validate some implementation questions (such as how well the software really does integrate with another tool).

4.) Vendor Selection and Communication
You've done it! All the information you need to make a rational, informed technology selection has been gathered, and in a quantitative fashion. Now combine the detailed RFP responses with the qualitative vendor demo feedback, and lock your team in a room for an hour or two (or three) and come out with a decision.

Don't forget to communicate the technology decision to the rest of the organization. Since the software was chosen in partnership with Enterprise Architecture, it should be reusable by other groups in the company with similar requirements, so make sure they know about it.