Where are there reports that validate the frequent assertion that 65%+ of major IT projects fail?

Having survived many a 'red project', I offer that we should take these reports on project success/failures with a healthy dose of skepticism.

While the reports mention a % of projects that fail for reasons noted above, the fact is that the very context of projects change the very next moment after a project is initiated. Business needs change, competitive landscapes change, target markets change, project teams change, sponsors change.......

Given the above challenges, adopting an agile approach, keeping the project scope realistic and achievable and keeping project teams small can help.

Large projects, in themselves are a recipe for disaster. Their sheer scale increases the exposure to change from all directions as noted above.

Michael is right when he says that the definition of failure is not always clear. I would go even further and say that it is almost impossible for major IT projects to determine either they are successful or not.

So if you see stats showing that 70% of CRM implementations fail and you're looking for a new CRM system, that doesn't mean that you have 70% chances to fail as well. Actually, it could be 100% for you, or 20% (it's highly unlikely to be 0%).

One thing to keep firmly in mind, as you look for failure reports, is the fact that people who are responsible for the failures don't often do a "mea culpa" about them. Indeed, in many cases, the failure is kept under very tight wraps until the responsible party has moved on. When the failure is made public, the causes are often laid to things that protect the real culprits.

Here's an extreme but relevant example; as a contractor quite a few years back, I managed the computer center and staff for the Executive Office of the President: in the back of the computer room was a huge communications switch that came with two full-time manufcturer's technicians (on my budget.) Because none of my staff or their government counterparts ever seemed to do anything with this device, we wondered what it was for. One day, we looked behind it (as part of our task of upgrading the computer facilities) and found to our amazement, that it was connected to power (so the lights stayed on) but to nothing else.

When I asked about it, my govt. mentor told me, on the QT (meaning if you talked about it, you were toast) that it had been the project of a former govt. manager who had been kept on to manage another part of the EOP organization. I kept my mouth shut until the person was cashiered a few months later by the incoming administration. The next week, the device was disconnected and moved out, being no longer a potential embarrassment to its mentor. By my reckoning, it had cost the government approximately $500K (in 1980s dollars) just to keep it in place.

While this is admittedly extreme, examples of the same kind of thing are all over both public and private sectors, so finding and fully understanding them won't be easy.

I find that most studies of IT projects, because of the reasons cited above and many others, do not answer the question effectively. Moreover, many fail to note the sea-change in IT software development projects as a result of the large adoption in the last few years of agile software development practices.

The best source for reasonable studies, imho, is The Standish Group, which has been accepting reports from major IT shops over the last few decades. Their definition of failure is approximately "not on time, over budget, or never completed", which admittedly does not take into account the frequently unrealistic deadlines characteristic of many IT projects. However, they have certified that when agile development processes are adopted, failures by their definition drop to almost zero, despite the fact that the agile development process itself offers very few ways to measure progress while the process is going on, the goal of the project is very likely to change during the process, and the particulars of the process seem to indicate a fair amount of inefficiency and risk.

The Standish Group also seems to indicate that incidences of "not on time" and of "over budget" in the non-agile case are highly correlated, and instances of "never finish" are relatively rare.

As most of the contributors to the answer of this question are saying to improve your chances of success, you must up front, before the start of any project, clearly define the definitions of success and failure. I can not give a true figure, but from my experience up to 70% of initial implementations do not meet the "success" criteria or expectations.
In most IT projects I have been involved in, many of them rescues after the fact, the issues for most failures go back to the original scope, definitions of the project and the methods and practices implemented to track and manage those projects.
Yes projects fail, but in most cases it is due to inadequate planning, inadequate definitions of scope and clarification of deliverables which in turn induces scope creep and dilutes the value of any planning. If any major changes are instigated for a project, my suggestion has always been to replay and redefine all the deliverables. In many cases also the unrealistic timeframes imposed by management and hierarchy due to reporting, financial or other unrelated deadlines.
Yes many fail, but success is achievable if you empower the right team to focus on the key components of the project and deliver as planned. That is the key.
I always remember the saying "you don't plan to fail, you failed to plan".

Of the track there, I should answer your question.
I don't believe you will find many real and complete failure reports, because in most situations I have observed either, the blame is distributed, or deliverables diluted and project de-scoped after the event. Often financial penalties are imposed and put down to savings or reduced scope of requirements and unfortunately in many cases its to protect the image of the organisations involved and/or the executives in charge.

Its disappointing, but many of the projects out there are being managed without much implementation planning and by people who may be great techs but probably do not have the required skills to manage a large complex project.

@Don

The Standish Group publishes the CHAOS Report that talks about this specific topic. This is the link to their latest report which is set to be updated shortly. They say 68% of projects don't succeed and don't fulfill business needs. They state that this rate has not changed for the past several years. Only this upcoming report shows improvement. But as a lot of other have stated, project failures can mean a lot of different things.

www.standishgroup.com
http://www1.standishgroup.com/newsroom/chaos_2009.php

@Krishnan
You have hit the nail on the head. My experiences are very similar. You cannot measure success on a moving target with changing scale and priorities.

@the Agile fan-boys
My experience is that no methodology is a panacea. I saw one Agile project burn over $4M and not deliver what the initial premise for the investment was.

My diagnosis? The Agile team drifted off scope because the user reps on the team lost sight of the original target and the governance processes were not in place to identify this. So they delivered something that worked and looked pretty, but it did not do the job for which it was intended.

What is failure? When is failure measured? If you get a client's approval for a job, in which the proposal asked for partial pre-payment, is the Job a win on approval, on receipt of partial pre-payment, full payment or when all the services are done?

The mythical IT project. Did it have mission creep (or as in construction; change orders). Were you fulfilling those original goals but got side-tracked by the creep?

Finally, who is claiming the win or the failure? Perspective and perception are crucial elements in any project. Just look at politics - depending on your side, it is either a win or loose, even if it was a winner or loser..