I’ve Got Nodes in Different Area Codes

By February 25, 2020 No Comments

One of CockroachDB’s most amazing capabilities is the ability to replicate across regions and selectively determine which data resides in each region. Today, we’ve automated that capability on the ObjectRocket platform! Taking advantage of CockroachDB’s multi-region capabilities is now just as easy as adding an instance. In other words… it’s pretty easy.

Nodes in US-east, US-west, EU-west…

To create a multi-region cluster on the ObjectRocket platform:

  1. Create an initial instance in one of your target regions
  2. Add a region via the instance list or the instance details page
  3. Add more regions as needed

Image from Gyazo

That’s really it. We’ll handle connectivity behind the scenes and make sure your clusters are replicating, but all you need to know is:

  • Clusters in different regions can be different sizes, plans, or flavors: Customize each region based on your needs.
  • Only one instance per Provider and Region combination: If you need more capacity in an existing region, you can simply add nodes/scale the instance in that region.
  • Whitelist IPs must be set per region/instance: We would expect that you’ll be connecting to the local instances from other local resources, so we let you set IPs per region. If you’re having trouble connecting, make sure you have your IP whitelisted in the region you’re connecting to.
  • Queries to any region can access data in any other region, but there will be a performance penalty for data that has to be accessed cross-region. We talk a lot below about strategically placing replicas, but that doesn’t change the fact that the data is always available everywhere.

Control How and Where your Data Replicates

Now that you have multiple regions configured we can dig into the various ways that Cockroach Labs allows you to control where your data is replicated. Cockroach Labs has a wealth of amazing documentation on this topic, so you can definitely check them out for the details, but here we’ll provide a quick overview of the capabilities that multi-region CockroachDB brings to the table.

First a little terminology:

  • Ranges: CockroachDB automatically splits up your data so it can be distributed across the cluster. This is similar to a shard in other datastores.
  • Replicas: Each range is replicated a programmable number of times (default 3, except for system tables, which are 5). Each replica is stored on a different node.
  • Locality: In CockroachDB, a locality describes the location of a node. You can set locality based on continent, region, datacenter. Localities can also be multi-tiered. On the ObjectRocket platform, we take care of this and set the locality on all of the nodes for you.
  • Leaseholder: For each range, one of the replicas holds the “range lease”. This replica, referred to as the “leaseholder”, is the one that receives and coordinates all read and write requests for the range.

Default option: Do nothing and get good results

By default, CockroachDB will automatically try to balance all of your replicas across localities. This will ensure that your data is evenly distributed across all regions for resiliency and good average latency. 

One additional thing CockroachDB does in the default case is activate a feature called “Follow-the-workload”, which will attempt to locate the leaseholder for a replica in the region seeing the highest workload. The goal is to optimize latency in the region with the greatest load.

Customizing with Replication Zones

However, if you have specific latency or data sovereignty needs, you can customize the placement of data with a feature called replication zones in CockroachDB. CockroachDB’s Replication Zones allow you to be prescriptive about replication at the various levels:

  • Database level
  • Table level
  • Index level (for secondary indexes)
  • Row level (using partitions)

More granular replication zones take priority over less (a row replication zone will override database level replication zone).

In each zone you can specify:

  • Range control – control how many ranges to maintain and when to split/combine them into smaller/larger chunks 
  • Number of replicas to maintain
  • Constraints on required and prohibited localities/attributes for replicas in the zone
  • Lease preferences to provide constraints specifically for leaseholders

Try it out for yourself

We could keep diving down that rabbit hole, but we’ll save that for another blog. For now, go check out our CockroachDB offering. We continue to be excited about what CockroachDB brings to the table and with the ability to geographically distribute your data, we think you will be too!

Steve Croce

Steve Croce

Steve Croce is currently a Senior Product Manager and Head of User Experience at ObjectRocket. Today, Steve leads the UX/UI team through rebuilding out the platform’s user interface, scopes the company’s product and feature roadmap, and oversees the day to day development for ObjectRocket's Elasticsearch and PostgreSQL offerings. A product manager by day, he still likes to embrace his engineer roots by night and develop with Elasticsearch, SQL, Kubernetes, and web application stacks. He's spoken at KubeCon + CloudNativeCon, OpenStack summit, Percona Live, and various ObjectRocket events.