What are the most critical aspects when chosing a website solution?

Great question. The priority of the most critical aspects can vary greatly between companies depending on the objectives for the website (lead generation, sales support, client support/retention, etc.). Regardless, here are few critical aspects for any type of website:
1. Functionality and features - diverse and appropriate website features to meet company needs and objectives.
2. Extensibility and customization - ability to customize and supplement the core website system.
3. Compatibility and integration - flexibility to integrate with various other complimentary systems (CRM, ESP, web analytics, etc.).
Typically, your CMS is separate from your CRM and ESP (e-mail service provider) and it is best to keep those functions on separate systems specific for each function. But it is important that all three systems can talk to each other. Depending on the desired level of marketing and sales automation, your ESP will drive leads to various landing pages on your CMS which will communicate back with your ESP and CRM for progressive nurturing and sales/marketing hand-off. We have found luck using Joomla CMS with Mail Chimp ESP and ACT! for Sage CRM, although we may be changing CRM to achieve more integration with our CMS sites.

So I think you have to ask yourself what type of business owner you are talking about. When I hear "business owner" I think of SMBs - companies that are 2-50 employees. For smaller businesses that may not have an internal IT department or have access limited technical resources, there is even a more basic question of what resource can I even turn to design, build and host my website - even if the primary purpose of the website is simply brochure ware. From this initial question flow a lot of basic functional requirements, including things ranging from ability to edit the site (CMS) to providing e-Commerce integration. I believe one class of features is probably universal to all businesses (e.g. CMS) and class functionality may be industry specific (e.g. e-Commerce).

I have not used or researched Business Catalyst yet so I do not have an opinion, good or bad. The idea in general of having an all-in-one solution sounds great but I would be interested in seeing the strength of each individual module. I have often found in the past that all-in-one solutions may not achieve the expected quality in each function compared to other systems which focus on one core function. Although, Adobe would definitely be a brand that could prove that experience wrong.

Thanks for great responses. I understand the question is quite broad. OK, how's about if an "all in one solution" like Business Catalyst: CMS, Email Marketing, eCommerce, Report and Analytic and CRM which could fit your own online needs with affordable hosting fee. Do you consider trying?