Company

Database Scaling Techniques – What it Takes to Keep Up with Traffic

By November 20, 2017 August 18th, 2022 No Comments

Databases are essential to modern business. Every day, you’re generating data that needs organizing, indexing, storing, and eventually, retrieving. Your constant stream of customers and users feeds this data stream, growing your database, and increasing the number of requests on the system. Sometimes, though, traffic is higher than we expect, and those who aren’t prepared for a spike in server requests will ultimately suffer the consequences.

 

Nothing’s worse than seeing your traffic increase, only to have your servers go down because they can’t handle the load. To avoid that mishap, you first need to understand how database scalability works, and how to leverage it to your advantage.

Vertical vs. Horizontal Scaling

Let’s start with a crash course in “scaling.” What we’re talking about is enabling your system to handle more server requests and to answer them faster. This can be done in one of two ways: 1. Upgrade the server (vertical scaling) 2. Add more servers (horizontal scaling).

What is Vertical Scaling?

Upgrading an individual server (installing a more powerful processor, adding ram, plugging in more hard drives, etc.) is called “vertical scaling,” and it gets the biggest bang for your buck out of the server. But no matter what you equip a server with, it has limits. Similar to a stack overflow, an overload of server requests will shut down a server (f.y.i., this is the principle upon which a DDOS attack is based).

What is Horizontal Scaling?

Adding additional servers will help ease the load on individual servers and help answer the server requests. This is called “horizontal scaling,” and it’s the basis for cloud-based computing. Multiple servers are linked together, communicating with each other to make sure each request is answered and everyone has access to the service. The hardware of any given server unit doesn’t have to be overly robust, since it doesn’t carry the weight of the traffic by itself.

Database Scalability Concerns

Obviously, your ability to scale your database (whether vertically, horizontally, or both) is limited by your available capital. A day-one startup isn’t going to have the server capacity of Google or Amazon. That said, no matter the size of your business, you need to consider how you will be able to handle anticipated traffic, spikes in traffic, and how to maximize uptime. So here are the basic techniques to making that happen: – Determine your average traffic, and use that as your baseline – Scale vertically to maximize the effectiveness of an individual server – Scale horizontally to handle traffic spikes and improve uptime – Ensure uptime by setting up redundancies – Consider outsourcing your database needs

How to Handle Anticipated Traffic

To handle anticipated traffic, use your average traffic levels as a baseline to decide how many resources you need to add to your database to maintain functionality. If you know you have 50,000 visitors a day to your website or your app, you need to have the equipment to accommodate that, bare minimum.

Keeping it at the minimum won’t do you any favors, though.

How to Handle Spikes in Traffic

If you plan on your product/app/service doing well at all, traffic is going to increase, and there are going to be spikes, or unanticipated traffic. To be prepared for these spikes in traffic, you will need to add more resources than you’re using on a consistent basis. In other words, you’re going to want overflow parking to handle your metaphorical peak summer hours. The most reliable way to do that is to scale horizontally. Failing to do so means you’re going to be dealing with some server downtime.

How to Guarantee Uptime

The best way to guarantee maximum uptime is through redundancy of equipment (in other words, horizontal scaling). Obviously, downtime is something you want to avoid as much as possible, but that’s easier said than done. Tech giants like Google guarantee a 99.9% uptime for their services—they have to, because there would be riots in the streets if Google ever went down, so they take steps to prevent that. They can guarantee this because of redundancies—they have more than one set of servers running at the same time. So even if one set of servers goes down (too many requests, fire in the warehouse, power outage, etc.), there are others both on and off site to compensate and keep things running. Be aware that, as you grow, guaranteeing uptime requires redundancy of equipment, data, and services.

What it Takes to Run it In-House

If you’re considering running your database in-house, you need to be aware of what that entails. It entails equipment: servers (lots of them), and all the hardware, software, and physical space to accommodate them. You’ll need a way to cool them, since they generate heat, and that heat builds up. You’ll need to power them. You’ll need to maintain them, and fix or replace pieces that break. Perhaps most importantly, you’ll need staff (the IT crew) to keep it running smoothly.

You can obviously see how this can all add up. That’s not to say running your own servers isn’t a viable option; it gives you greater control over the database and how it’s maintained. It just means more startup costs, which is why companies frequently switch from outsourcing to running their own database later down the line. Read more about the total cost of ownership.

Outsourcing the Database

For those that would rather not deal with the difficulty and expense of maintaining their own systems, or who don’t feel like they have the expertise to do it themselves, there are options. Database as a service (DBaaS) is a thing, and there are experts that can handle these things for you. That way, you can focus on what you do best, and leave the operations side of things to people who specialize in it.

Scaling with ObjectRocket

Here are some links for more information about scaling at ObjectRocket: