What are the biggest risks in a software evaluation?

Many good points in your blog but it doesn't start at the beginning. A comprehensive needs analysis - that fleshes out needs vs. wants and functionality vs. bells-and-whistles - is crucial before selecting the software. Without it, scope creep is an ever-present danger that results in delays and cost overruns.

This is a great place to ask this type of question as there are many people here that are very knowledgeable on the topic.

There are so many different ways to screw this up, but I think the biggest risk is lack of well thought out requirements. Companies often assemble a requirements document based on a list of features from various product websites or they sit through demonstrations and come up with ideas. This will create a certain shortsightedness that will lead to a poorly developed requirements document. If you don't get this part right in the beginning you can end up with an implementation that is not aligned with the company objectives - even if the best vendor is selected it will end in frustration, change orders, finger pointing and heads on the cutting block.

Another extremely huge risk, that is basically based on the requirements, is allowing a department to take control of the project when they are not the benefactor of the project. For example, I have seen countless examples of serial failed implementations due to IT's over-involvement or senior management's over-involvement in the requirements. If it is is CRM software, for example, and your objective is to make the sales people more efficient then you need to respect the needs of the sales people who will be using it and reflect this in the requirements. If it has great reporting and integrates nicely to the existing database then this is good, but it also has to help the sales person enter and access information quickly - if the sales person has to enter data into nine different screens then the database integration and the pretty report for management are absolutely useless.

In short, spend the extra time to get the requirements correct and don't let certain stakeholders take control and select something that benefits them despite the rest of the company. IT should never be responsible for the requirements document unless they are the benefactor of the software.

I agree with Ken. Without a clear picture of the need you expect this software to fill, there is a much higher likelihood of implementing a solution that will ultimately fail.

When developing a needs analysis, I would suggest that you come at it from a process perspective. In order to do so, you will need to engage the appropriate business subject matter experts (SME) in your company.

I also agree with Chris that, while I.T. has a role in the software selection, they should not take the lead in the process. Even with the best of intentions, they generally do not have the depth of business knowledge to ensure that you BUSINESS NEEDS are being addressed. As the saying goes "the devil is in the details" and a cross-functional team of SMEs can help flesh out those details.

It’s very gratifying to see a change from so much of the usual mistaken conventional wisdom regarding software acquisition/evaluation. Both the original blog and the comments instead refreshingly demonstrate considerable on-target understanding. As someone who has performed more than 20 such evaluations and also has had the advantage of analyzing the process from both software and legal perspectives to train hundreds in several seminars on appropriate Beyond the Textbook™ acquisition/evaluation methodology, let me add a couple of cautionary points.

It’s good that everyone is emphasizing the importance of first identifying business needs and getting the requirements right. While absolutely essential, such admonitions can be illusory. What too many people (though possibly none of those contributing here) mean by requirements actually is high-level design—the requirements of the product, system, or software they frequently mistakenly think is needed to meet their (often inadequately defined) business needs. One should be looking to the vendor for product design, which should be their expertise, rather than locking them into the buyer’s presumption of the design, which probably is not the buyer’s expertise.

The second caution refers to a far more insidious and seldom-recognized pitfall that undercuts practically all conventional software acquisitions/evaluations, even when REAL business requirements have been identified adequately. With the best of intentions, buyers invariably, inadvertently, and unknowingly remove most of their legal advantages by shifting the legal responsibility burden from the vendor back unto themselves.

Consider the typical acquisition/evaluation process:

1. Here’s what I need (hopefully REAL business requirements).
2. Tell me what you’ll sell me.
3. I evaluate your proposed solution to determine if it is adequate and superior to other proposals for meeting my needs.

Can you see why this puts the buyer on the legal hook instead of the vendor?

The biggest risk with corporate developed software is the usability of the interface. Most users today are exposed to software, and more users interfaces, in their role as a consumer than in their corporate life. Corporate products are usually generations behind in usability.

In the day of the iPad, users do not know how to articulate what they want in an interface, but they assume that someone on the technical team will deliver a high-quality interface. Unfortunately, most programming teams don't have a lot of formal experience in usability design and testing. The corporate team can often deliver products that are robust and can perform at high speed, but even the few development teams that can produce a world-class user interface don't know how to develop a compelling business case for the cost of better "look and feel" elements.

Call it the Discovery Process or the Needs Evaluation, the danger is that you have spent too much time focusing on the software and not enough on the real needs. Vendors, Customers, and Employees are all impacted with a software implementation. You have to know the questions to ask and the way to ask them considering humans have various needs, wants, and knowledge.

The risk that you are not willing to accept that you only know what you know. I know a fair amount about cars and in my younger years worked on them all the time. You know what? When I need a brake job, I take the car to a mechanic. He's the expert. Takes him a fraction of the time it would be, and costs me a fraction of what it would cost if I did it. And he's not going to screw it up. I probably will. Heck, If I tried it myself, I'd probably end up taking it to him to fix my mess.

Oh, and I can focus on what I really want to do and what I do best. I'm not putting my safety and the safety of those around me in the hands of a amateur - me.

So with an investment such as software, you're dealing with the future of your business! Get the wrong match to your need and you've got a mess. Get the right match and you'll be making lots of deposits in the bank account.

Bingo! Everyone’s happy and my banker and sharehoulders like what I'm doing.

This is a great question. Biggest risk in software evaluation is lack of clear requirements and hence mapping of these requirements to software which are being evaluated.

Failure to recognize usability problems is a common problem with software evaluations. There is a saying "You can't see the picture when you're inside the frame." What seems to be easy to use software for the designers may confuse and mystify actual users. They don't know that you have to do "A" first then "B". They see a button for B and click on it and it doesn't work.

A lot of software contains idiotic, unhelpful, jargon-riddled text as error messages, which is often overlooked in a software evaluation.