An ounce of prevention is work a pound of cure

 

The most common number of dedicated Database Administrator's we see in a team is two – just enough administrators to cover each other on vacations. The second most common is one. The third most common is zero.

 

In small organization, finding the space in the budget for necessary redundancy can be difficult, thus we have a tale of two organizations.

 

One of our small healthcare data clients, located outside of Seattle, WA, couldn’t find the budget for a second Database Administrator, so instead they contracted our services. For two years, we were their backup DBA. We woke up when their servers in the night, so their in-house DBA was always rested in the morning. We performed their off hours patching and weekend maintenance so their DBA could recharge. When their in-house DBA went on vacation, we filled in, deploying new queries, packages, and reports to their production systems, and handling requests from the business. We were also available for questions when the only DBA on their team hadn’t encountered a specific issue before.

 

Another organization, who came to us later, specialized in manufacturing. Located in Philadelphia, PA, They had a single DBA, and no backup. Their Database Administrator handled the alerts on off hours and patching, often letting their servers fall behind due to lack of resources. When they were sick or on a rare vacation, deployments were put on hold. If an unfamiliar issue arose, they would work with what they had.

 

Within six months of each other, both teams lost their only DBA. The former to health issues, and the latter to another business.

 

The manufacturing business was in trouble. Jobs, reports, and whole databases were being run under the old DBAs account. When it was disabled through active directory by their networking team, they suddenly become unavailable. The DBAs account was also the only sysadmin account on the Microsoft SQL Server. No one in the company new how to fix it.

 

Moreover, the old DBA had kept their company processes in his own head. With no coworker to speak of, he saw no use in documentation.

 

Not only did they not know how to replace their DBA, but they also did not know how to regain access to their systems, and they were not going to be able to recover the lost business knowledge.

 

Unfortunately, this was a costly process. Regardless of the consulting fees, the data their plant relied on was inaccessible for three days before they finally decided to call us, costing them time, money, and stress. They also lost irreplaceable knowledge of their internal system. The worst part? It was preventable.

 

The healthcare data business, on the other hand, was prepared – or really, we were prepared to step up. We took on the same role that we had stepped into time and time again as their DBA had been out of office. We already had all the business knowledge for the position documented for our own team.

 

They were able to take their time and hire the correct fit, and when that happened our team was able to bridge the gap, training their new DBA in the internal processes.

 

It is far more costly to fix an issue than it is to prevent it.