A common question we’re asked here at ObjectRocket is: “How do you compare to X?”. This is a very good question, as you should always look for the best performing option for your business! We’re always looking for ways to make our platform better and show what we can do.
Why is this important?
When we deploy changes in our environment we needed a way to make sure that there were clear benefits to doing so, and along the way we found Rally. It started out with simple tests against the same Elasticsearch version against hardware changes, then moved on to comparing minor versions against each other, and even farther comparing major versions and so on. Needless to say, it’s a huge help in measuring how we’re pushing performance little by little as we make changes behind the scenes.
As time went on, we decided to use the same tools to compare ourselves from the outside. We started by creating an EC2 instance (m4.xlarge 16GB RAM, 4vCPUs) in US-East, using the default external interface, running CentOS 7 and installed Rally v0.4.3.
Once setup, we started creating off the shelf instances on our platform, along with matching instances on Elastic Cloud and Compose.io. Here’s what we went with:
|Provider||ES Version||Data Nodes||Region||Memory||Disk||Price|
|Compose.io||2.4.2||3||US-East-1 (AWS)||8GB||75GB Amazon GP2 SSDs||Monthly rate: $1,359|
|ObjectRocket||2.4.4||3||US-East-IAD1||8GB||64GB SSDs||Monthly rate: $720|
|Elastic Cloud||2.4.3||3||US-East||8GB||192GB SSD||Monthly rate: $587|
The versions mismatch slightly, as we just chose the highest default available. We contacted support at Compose.io to give us 8GB of RAM, as the closest we could get by default was 7.47GB. We’d also hoped to test using ES 5, but Compose let us know their launch of ES 5 would be within the next quarter, so we went with ES 2.4.
As there are quite a few options within Rally, we went with the following:
--track pmc --pipeline=benchmark-only --target-hosts=<hosts> --pipeline=from-distribution --distribution-version=<version> --client-options="basic_auth_user:'<user>',basic_auth_password:'<password>'"
The notable portions are specifying the version we’re testing against, setting benchmark-only so Rally doesn’t try to download and stand up a local Elasticsearch install, and adding HTTP authentication.
We ran 3 tests randomly during different times of the day. One between 10am and 12pm CT, another between 2pm and 4pm CT, and the last between 8pm and 10pm CT. We did this to make sure there’d be no performance drops or spikes in case any deployment was on shared hardware. We then averaged the numbers together and compared them.
This is by no means an end all, be all comparison, but rather what we imagine a potential customer might do by testing their application against different providers. We plan on automating Rally to run tests like these using it’s tournament configuration to easily compare going forward.
With that said, here’s what we found! The chart below highlights the total time it took to run the tests. Lower numbers here are better.
The following charts cover the 90th percentile latency and service times for each test, with the exception of
index-append due to their extremely high execution times. The results for those will be in the raw output linked below, but they’re in line with the rest of the results shown.
With the performance results laid out side by side, its clear to see that we’re all fairly close to each other! However we do vary in one other very important way, and that’s in price.
You can take the monthly price from above for each deployment and do some quick math to figure out which is better for your business, but we’ve got another chart below that we think makes the differences a bit clearer.
How we came to these values is fairly simple. We first inverted the test times, then divided that number by the monthly cost. Then we divided the result by the Elastic Cloud result to normalize the rest of the values.
((1/test time)/cost)/ESC Result.
So, now that we know where we stand, we can begin to work towards being the absolute best option for hosting Elasticsearch in both performance and price. We’ve already started some exciting things internally, and we hope to show some of them soon!
If you’d like to see the raw CSV output, we’ve uploaded them here.