Clustered Elasticsearch Indexing, Shard, and Replica Best Practices

Some of the most common sources of support tickets we see on the ObjectRocket for Elasticsearch platform are related to indexing, shard count, and replication decisions. Elasticsearch is awesome at spreading data across your cluster with the default settings, but once your cluster begins to grow, the defaults can get you in trouble. Let’s go over some of the basics of sharding and provide some best practices for indexing and shard count.

An Intro to Elasticsearch Sharding

There are tons of docs out there about how shards in Elasticsearch work, but the basic concept of sharding is splitting up the your data into a number of chunks so that searches can operate on multiple parts in parallel. In order to facilitate clustering and parallelization of index functions, each index in your Elasticsearch instance is sliced up into some number slices; These slices are called shards. Let’s look at some of the key behaviors for shards:

Now that we’ve set some ground rules, let’s look at a small example. I’ve got an index created with a shard count of three and a replica setting of one. As you can see in the diagram above, Elasticsearch will create 6 shards for you: Three primary shards (Ap,Bp, and Cp above), and three replica shards (Ar, Br, and Cr).

Elasticsearch will ensure that the replicas and primaries will be placed on physically different hosts, but multiple primary shards can and will be allocated to the same host. Now that we’re talking hosts, let’s dive into how shards are allocated to your hosts.

Shard Allocation and Clustered Elasticsearch

As mentioned above, by default, Elasticsearch will attempt to allocate shards across all available hosts. At ObjectRocket, each cluster is made up of master nodes, client nodes, and data nodes. It’s the data nodes in our architecture that form the “buckets” that the shards can be assigned to.

Using our example above, let’s take those six shards and assign them to an ObjectRocket for Elasticsearch cluster with 2 data nodes (the minimum). In the diagram below, you can see that for each shard, the primary will land on one data node, while the replica is guaranteed to be on the other node. Keep in mind that the examples here show just one possible allocation, the only thing that’s definite is that a replica will always be placed on a different data node than its primary.

Now, let’s extend this example and add a third data node. What you see is that two shards will be moved to the new data node. You’re now left with 2 shards on each node.

Finally, let’s add a new index to this cluster with a a shard count of two and the number of replicas set to two. What you’re left with is two new primaries (Xp and Yp) and four replicas (Xr0, Xr1, Yr0, Yr1), that could be spread across the cluster as seen in the picture.

That’s it. Elasticsearch does all of the hard work for you, but there are some pitfalls to look out for.

Pitfall #1 – Massive Indexes and Massive Shards

The most common and easiest to mitigate issue in Elasticsearch is a massive index with massive shards. We see it very often that a user starts out with a very manageable single index and as their application grows, so does their index. This then leads to huge shards, because shard size is directly related to the amount of data in the cluster

The first issue this causes is poor efficiency in cluster utilization. As the shards get larger and larger, they get harder to place on a data node, since you’ll need a large block of free space on a data node to place a shard there. This leads to nodes with a lot of unused space. For example, if I have 8GB data nodes, but each shard is 6GB, I’ll be stranding 2GB on each of my data nodes.

The second issue is “hot spotting”. If your data is consolidated into few shards than complex queries will not have the opportunity of being split across a larger number of nodes and executing in parallel.

Don’t be stingy with indexes

The first and easiest solution is to use multiple indexes. Spreading your data across multiple indexes will increase the number of shards in the cluster and help spread the data a little more evenly. In addition to just an easier game of “Tetris” when Elasticsearch places shards, multiple indexes are easier to curate. Also the alias capabilities in Elasticsearch can still make multiple instances look like a single index to your app.

Most of the Elastic Stack will create daily indexes by default, which is a good practice. You can then use aliases to limit the scope of searches to specific date ranges, curator to remove old indexes as they age, and modify index settings as your data grows without having to reindex the old data.

Increase shard count as your index size increases

In addition to adding indexes more frequently, you can also increase the shard count as your index sizes start to increase. Once you see shard sizes starting to get a little too large, you can update your index template (or whatever you use to create new indexes) to use more shards for each index. However, this only helps if you’re regularly creating new indexes, which is why this recommendation is listed second. Otherwise, you’ll have to reindex to modify shard count, which is not impossible, but a little more work than managing multiple indexes.

Our rule of thumb here is if a shard is larger than 40% of the size of a data node, that shard is probably too big. In this case, we recommend reindexing to an index with more shards, or moving up to a larger plan size (more capacity per data node).

Pitfall #2 – Too many indexes/shards

The inverse is far too many indexes or shards. After reading the previous section, you may just say “Fine, I’ll just put every doc in its own index and create a million shards”. The problem there is that indexes and shards have overhead. That overhead manifests itself in storage/memory resources as well as in processing performance.

Since the cluster must maintain the state of all shards and where they’re located, a massive number of shards becomes a larger bookkeeping operation which will have an impact on memory usage. Also, since queries will need to be split more ways, there will be a lot more time spent in scatter/gather for queries.

This one is a little harder to give exact guidance on, since it is highly dependent on the size of the cluster, use case, and a few other factors, but in general we can mitigate this with a few recommendations.

Shards should be no larger than 50GB

In general, 25GB is what we target large shards, but 50GB is where we have the conversation with our customers about reindexing. This has as much to do with the performance of the shard itself as it does with the process of moving that shard when you need to. Whenever rebalancing, shards may need to be moved to a different node in the cluster. Moving 50GB of data can take a significant amount of time and then you’ve got that capacity tied up on two nodes during that entire process.

Keep shard size less than 40% of data node size

As mentioned above, the second metric we look at for shard size is what percentage of the data node capacity a shard takes up. On the ObjectRocket service, we offer different plan sizes that are related to the amount of storage on the data nodes. We try to size the cluster and the shards to ensure that each of the largest shards don’t take up more than 40% of a data node’s capacity. In a cluster with a number of indexes at a mix of sizes, this is fairly effective, but in a cluster with a single or very few indexes that are very large, we are even more aggressive and try to keep this below 30%.

The idea here is to make sure that you’re not stranding capacity on a data node. If you’re shards are 45% the size of the data node, for example, you’ll need a data node at roughly half utilization to be able to place that shard. That’s a lot of spare capacity to leave lying around!


Selecting the right shard and indexing settings can be a moving target, but by planning ahead, making some good decisions up front and tuning as you go, you can keep your cluster healthy and running optimally. We help businesses refine their Elasticsearch instances all the time. Contact us for an Elasticsearch with Kibana consultation with our Elasticsearch DBAs.

Exit mobile version