The following blog is based on the podcast interview of Jeff Garbus, hosted by HC Biz during the Health IT Expo, which took place in New Orleans from May 30th, 2018 through June 1, 2018. Listen to the full interview here.
In this day and age, data is everything. Trade secrets, legal pleadings, tax returns, credit card information, payroll, website content, and so on and so forth. Take a moment to truly digest this information and answer this question honestly: How much money would your business lose every second your database is down?
The purpose of this blog is to provide you with a brief overview of how what you don’t know about your data can significantly hurt your business, and in a worst-case scenario, it can affect your personal life too.
The importance of tuning scalability.
One of the biggest concerns we get as a remote database administrator company is how to ensure efficient continuity as a business grows. For example, let’s say that whenever you entered a database query, you used to have a 10 second response time, but now that there’s been an increase in volume, there’s a 20 second response time. Why are things running so slowly?
CPUs may be spiking and your business may not realize it. They could be running at 98% capacity, not knowing that at 99%, it will sort of stop. Your business could be walking along a cliff. Everything may look good, but it’s all about to fall off.
This is why health checks are so important. Done well and done regularly, you’ll be able to gauge how your business is doing and foresee issues before they happen, such as scalability issues, database corruption, or a failure to implement an efficient disaster recovery plan.
How to prevent database corruption.
One of the most common mistakes a CIO can engage in is not knowing if the data stored in the company’s database is actually accessible, i.e. not corrupt. If you were to ask for all of the data in your database, are you 100% sure you’d be able to get all of it?
After 30 years in business, we can tell you this much: We could walk into an organization with about 100 SQL servers, and it’s usual for one or two to have a corrupt database.
Sometimes, it’s just an index that’s not working properly anymore, but other times, the problem is data that you’re no longer going to be able to access. When this happens, it needs to be cleaned up sooner rather than later, because bad data tends to snowball.
How can a CIO find out if there’s database corruption within their system? Depending on the technology - SQL, Oracle, Sybase- all have built-in tools that allow you to identify if there’s any issues.
This is something that needs to be run before you age out your backups. If you don’t do this beforehand, you run the risk of not being able to restore from a backup in case of a disaster.
One of the first things we do at Soaring Eagle as part of a health check is to take a look at the preventive maintenance regimen that the existing DBA has in place. Sometimes, all they’re doing are backups. Other times they are doing index rebuilds for all databases. And sometimes, they’re not even backup up the system databases because they don’t realize it’s an important thing to do.
An efficient DBA should thoroughly understand that the database consistency needs to be validated prior to aging out a back up. If they don’t, that’s a huge red flag.
Prioritizing backups.
It’s often the case that people will think their data is backed up, only to find out once they attempt to recover it, that it has been failing for months and nobody noticed.
It’s crucial for any business to make sure that backups are taken periodically as a business decision, not a technical one. If this is not currently a priority, how much data would you be able to afford to lose if everything melted right now?
Make sure backups go off-site and make sure you can retrieve them and load into a machine at your company.
How often a business should back up information varies from industry to industry. At Soaring Eagle, we do it quarterly. Depending on the nature of your industry, you may have to do it more often.
In addition to backing up your data, test your process regularly: Ask your IT people: “Hey, can you please restore last Thursday’s backup to my dev server? I want it to come to 3:00PM in 8 seconds.”
If they don’t know how to do it, that’s a problem. If they can’t get it done, that’s a also problem. Randomly asking your DBAs for backup information when you don’t actually need it is the best way to ensure that everything’s ready to be recovered when you do.
If your industry’s regulations require that you back up data on a much more regular basis, you can build a process that will automate it. The cost may be high, but not doing so and having a problem is a lot more costly.
Case in point: We once had a client who spent half a million dollars on a backup system that failed. It wasn’t until November that they realized their system had stopped backing up data since March. They ended up having to furlough 1,200 of their 1,400 employees. Are you willing to take that risk?
Implementing a Disaster Recovery (DR) plan.
All kinds of things can cause a data center to go down: natural disasters, single points of failure, getting hit by lightning.
When this happens, not only do you have to take into account the lost information, but also the time invested in trying to recover it. How long can you afford to be down?
To maximize recovery efforts, make sure your application servers have failovers too, ‘cause if the site goes down, you want to know that all the components can be brought up at the remote site.
The failover should not require 10 people to get involved. Make these decisions early, get the process put in place, get the application failover in place. It will save you a lot of aggravation.
You’d be surprised to see the amount of companies that have to close their doors after catastrophic data failures. The number is close to 65%, and that includes very large corporations.
What if I just store everything in a cloud-based system?
A lot of companies are moving to the cloud or considering doing so, based on the myth that once you move everything to a cloud-based provider, everything is taken care of. This myth is very far from the truth.
What cloud providers usually offer is simply an infrastructure. If you pay extra, you may get backups; but they’re not providing data management, performance, index rebuild, or database consistency validation.
Just moving everything to the cloud won’t solve database administration problems. For this, you need 24/7 operational database support.
What if I have an in-house IT staff?
Even if you have in-house personnel who can manage your database, they won’t be able to do so on a continuous basis. Expecting a handful of DBAs to do that for you will likely end up in them getting overwhelmed. Having to be on-call around the clock has a big impact on their lives.
Having a remote 24/7 DBA support significantly cuts your turnover rate. Quality of life at your business goes up, because companies like Soaring Eagle keep people from getting burned by doing the grunt work.
We’ve been doing what we do for over three decades because we’re tech geeks who know what we’re doing. We have the longevity and a long list of books under our names to prove it.
To perform a health check of your database performance, you can reach me at jeff@soaringeagle.guru or contact us here. Time is money, and we’re here to keep your business running full throttle.
