Share what you know with millions of people

Try to implement a lights out data center policy. Statistically 40% of your unplanned downtime will be operator error. So, keep inexperienced or hurried hands out of the racks with "remote console access". In lieu of a crash cart use the following (Avocent, Raritan, et.al.):
* KVM over IP (for Windows and other GUI-based interfaces including BIOS)
* Serial over IP (for networking gear or Unix, anything with command line)
* Managed power strips (remotely turn outlets off/on)
* Service Processors (example):
* iLO (for HP)
* ALOM (for Sun)
* DRAC (for Dell)
* RSA (for IBM)
* SMASH & IPMI (generic protocols)

Virtualize critical applications and even storage (VMware or Citrix). If done properly, your users won't even notice when a piece of server hardware goes offline.

Use offsite backups. Restoring from tape backups is slow and less reliable. (DoubleTake, Yotta280)

Server and desktop lifecycle management (LANDesk, Altiris) This includes automation and simplification of:
* Server/Desktop provisioning
* Patch management
* Software license management (license recovery/reallocation is the ROI here)
* Server/desktop re-purposing and de-commissioning

Also, I'm a really big fan of F5's traffic management solutions, PresiNET monitoring, PaloAlto Networks firewall and Ubiquity Networks wireless.

Hope that is helpful!

Tony Elam

IT/IS Manager, Patterson Companies

Posted on Feb. 16, 2010

Adam,

There are many items that we need to implement from a process and procedures standpoint that will help your datacenter run smoothly. Some of the things we do internally to help eliminate downtime:

- Document and adhered to a policy for change management. Any production software or hardware is not changed without a request with business and IT/IS management approval. This will eliminate the cases where an employee with request or make a change thinking that it will not impact anything. Also, if you are a public company you have to adhere to this for SOX and other compliance standards

- Maintain the controlled environment as much as you can. Therefore do not hang out in your server room or use it for a storage location. You would also need to have all equipment that is frequently touched such as backup devices and media, desktop switches and such are all located outside of the server room. The less you alter your controlled environment the less your systems that control those settings have to change cycles, etc.

- Understand the business needs and/or corporate goals to ensure that you place the proper solution the first time. This will take working with business leaders, project manager, members of management, etc. The real value that IT professionals bring to the table is to understand those requirements and translate them into a rock solid solution that meets the requirements you have for your IT strategy.

- The more due diligence you perform on the front end of a project traditionally leads to a smooth implementation and smooth maintenance after you complete the project and the software/device goes into production. So make sure to properly document your requirements, ask for customer references, check out sites like 'Focus' (what do I get for the free plug?), understand the technology as much as possible prior to any agreement, ask for proof-of-concepts. I have had a couple of projects that after getting delayed multiple times we know more about the solution and second guess if it is the proper solution, or reconfigure the solution when we should be implementing or supporting the solution.

- I do not like the saying that if it is not broke do you fix it. It may not be broke, but that does not mean that you do not need to review your vendor partners, management systems, infrastructure providers, etc. You do not always need to change things, but it is a good idea to keep an idea out for new processes or solutions that can provide gains over currently implemented solutions.

Thanks and please feel free to share any findings that you have come across.