Share what you know with millions of people
Focus is the best place to turn what you know into remarkable content
1
How trustworthy are storage benchmarking stats?
Groups like the SPC tote themselves as being vendor-neutral, but each vendor will obviously tweak their hardware prior to testing, right? Does the SPC have a way to regulate this, and do you think these results are reliable?
Events
- Dos and Don'ts of Small Business Marketing May 29 @ 11 am PT
- Lead Nurturing 202: The Next Generation May 31 @ 11 am PT
- The Tricks to Paid Media June 6 @ 11 am PT
- Display Advertising for Brand Awareness June 20 @ 11 am PT
Related Activity
3 Answers
From a systems architecture perspective, storage benchmarks provide a good deal of value. Without getting too technical, vendor tweaking should not skew test results to the point that they’re unreliable. That said, Like an x-ray to a doctor, the results of these tests have to be interpreted by someone of sufficient technical understanding to determine what they mean to each application of storage technology.
They're trustworthy to the extent that you understand what they represent. Juraj makes some key points about the applicability of the configurations to real world scenarios.
From my perspective, SPC and other high-end benchmarks show what is possible with a particular hardware and software combination. If a vendor has a number of top 10 entries in the categories that I care about, then it is likely that their product will function well enough for me in that category relative to other entrants or solutions. While there is no guarantee that the product which can be made to perform the highest can *more easily* made to perform that way relative to a product that didn't win, but it is more likely.
These numbers serve to provide relative value, and as such, they are not totally useless. (Unfortunately, almost all the stats we tout are like that: Pay attention to the RPMs associated with the horsepower of the car you drive, and see if that represents a real world situation for you)
-ASB: http://XeeMe.com/AndrewBaker
A couple points:
Most storage benchmarks don't reflect real world workloads, so their value is limited to the couple of performance markers they measure.
Most benchmarks are run on hardware that is not representative of real world configurations, with hardware configurations that are only available in ideal scenarios.
Finally, performance is largely based on the tuning and configuration that your team will do onsite and at levels usually higher than the hardware level.
I found the only way to make an fully informed decision is to run benchmarks based on the workloads in your environment. Most vendors will provide loaner hardware to test on.
Answer This Question