(c)iStock.com/erhui1979

Cloud Spectator, an analyst firm, has put together the second of its infrastructure as a service (IaaS) pricing reports, and found Amazon Web Services’ (AWS) crown slipping.

The latest study, which collected over one million data points over a 24 hour period on processor and memory, examined performance alongside other patterns, such as instability and unpredictability. Performance tests were conducted on five different virtual machine sizes across 15 IaaS providers.

The results were described by the analysts as “eye-opening.” AWS’ T2 micro, with burst performance features, revealed a controlled period of high performance alongside a controlled period of lower performance, putting it right at the bottom of the table. The high degree of fluctuation was caused by the burst performance, which occurred for three quarters of an hour during the 24 hour testing.

In contrast, CenturyLink, SoftLayer and Verizon showed steady performance over the 24 hour testing period. SoftLayer recorded the lowest variability, while DigitalOcean and Rackspace recorded the highest performance figures.

Picture credit: Cloud Spectator

Anne Liu, an analyst at Cloud Spectator, argues both performance values and performance variation should be taken into consideration when selecting a VM. She wrote: “Comparing VMs using pricing, performance and price-performance can lead to significantly different results, and therefore it is critical for customers to select the most fitting comparison criteria.

“Knowing what the most important criteria [are] and using those criteria to compare and select VMs are essential,” she added.

The 24 hour continuous testing, she noted, was to test a provider’s requirements, and due to the large performance fluctuations, the performance level may drop below a customer’s minimum performance requirement.

The first part of the report, released in January, found Amazon EC2 offers “significant” cost advantages over a long term investment.