Picture credit: “Mongo DB at OSCON”, by “Garrett Heath”, used under CC BY / Modified from original

Updated May 5: A report from United Software Associates (USAIN) has found MongoDB to be top of the pile of NoSQL database providers in benchmark testing.

The research tested three leading products – Cassandra, CouchBase and MongoDB – through Yahoo!’s cloud standard benchmark, YCSB. USAIN wanted to assess the durability of each, going on the theory that most applications should prioritise durability over performance, not accepting data loss. The databases were put through the ringer on three types of performance metric; throughput optimised, durability optimised, and balanced.

In workload A of 50% read and 50% update with throughput optimised, under the YCSB benchmark MongoDB hit 160,719 operations per second, ahead of Cassandra (134,839) and Couchbase (106,638). With workload B’s 95% read and 5% update, MongoDB again came out on top with 196,498, ahead of Couchbase (187,798) and Cassandra (144,455).

However with durability optimised, MongoDB soared ahead on workload A, with 31,864 ops a second compared with Cassandra (6,289) and Couchbase (1,236). It was a similar story on workload B, with MongoDB (114,455) ahead of Cassandra (54,864) and Couchbase (18,201).

For the balanced tests, there was no equivalent configuration for Couchbase so it had to sit the tests out. Again MongoDB performed more strongly than Cassandra on workload A (114,245 against 77,676) and workload B (183,152 against 71,643).

The overall conclusion from USAIN was that, not surprisingly, MongoDB provided greater performance in every test, in some instances by as much as 25 times. In Couchbase’s default setting of optimised for throughput, MongoDB again outperformed it. The reason USAIN gave for this disparity was the method the two databases employed: MongoDB handles write conflicts in the database, while Couchbase instructs app developers to detect and handle conflicts in their code, meaning additional trips to retry updates.

Kelly Stirman, MongoDB director of products, noted the importance of the data durability test. “For most organisations, and for most applications, data loss is a cardinal sin,” he told CloudTech. “When real-world factors are considered, the gap between MongoDB and the NoSQL providers is obvious.”

Stirman added: “The report validated what we already know and have seen in hundreds and thousands of applications. Our advice to potential customers is always the same; follow best practices, do your research and talk directly to the people who make the software you’re working with.”

Of course, it’s horses for courses. Back in June 2014 Couchbase released a benchmark testing report, this time from Thumbtack Technology, which put its database at the top of the pile ahead of MongoDB and DataStax, arbiters of Cassandra. Earlier this year the company released another report comparing its Server 3.0.2 with MongoDB 3.0. It’s worth noting as well that USAIN has its place on a MongoDB partner page here.

Couchbase CEO Bob Wiederhold noted this when he hit back against the report in a blog posting. “Benchmarks need to make every effort to achieve an apples-to-apples comparison,” he wrote. “Apples-to-apples comparisons can be difficult because the architectures and operational setups of each product are so different, but significant effort should be made to achieve this.”

He added: “Vendors will continue to sponsor and publish benchmarks, and they’ll continue to gear them toward the use cases the vendor supports best. All of this is just another indicator of the rising importance of NoSQL, which is growing fast.”

You can take a look at the full report here (email required).