What are the 5 things that a CIO should ensure that their team is doing especially if they do not have a DBA on staff?
In my experience the worst thing that many CIOS do is: they assume.
They assume that their DevOps team knows and or has the time to do the regular proper maintenance of their environment. In our world assuming is a word that should not be uttered. Make certain your team can report to you at any moment whether maintenance jobs are run, reports are running smoothly, and business process don’t fail without notification. Examples of maintenance jobs are of course all backups, full, log and differentials. Failover detection, index jobs missed and unused, database consistency checker, CPU alerts, and unexpected and predictive space alerts should all be in your auditing and maintenance jobs repertoire. Please don’t forget networking and security alerts.
CIOS often believe that DevOps know how to write queries that perform well today and in two years from now when the data, the databases and the environments grow. Some coders can code in every language and code well. Some assume that the rules apply to all of the languages when it comes to code. But if you code several old-fashioned loops, and stack stored procedures and transactions into nasty queries, using too many tables, improperly using views or not indexing for performance then the query that runs in 30 seconds to a minute today might take several minutes or hours to run when your database grows to 10s of millions of rows. The larger and more complex the processes need to be the more work it is to find and clean up the queries later. When your customers and users start complaining about performance that constrains their daily tasks and interferes with the business-necessary processing goals you will need to recode. That can be costly. With tools Like Solarwinds DPA you can evaluate long-running queries and with a more senior team at the ready you can have the code reviewed.
Another area where CIOS overlook is a process to review the documentation for the entire environment. They don’t notice that the documentation doesn’t exist until either an employee leaves or when no one remembers how, when and where reports, or processes should run. Sometimes no one knows why they run this can turn into a literal management nightmare.
Management needs to create and audit document processes, ensure upgrades are labeled assigned to an owner, are readable and easy to find. Not doing this can be very costly a few years down the road.
Include documentation time for creation and validation for all of our projects. No matter if you are creating a simple report or a large-scale application. Include the information, why it runs, who has the skill to run it, and how to fix it when needed. Throwing code up into a version control system does not cover all of these steps. Make sure there are headers at the minimum that answer the questions above. It is better to create documents that are assigned for upgrades, testing, and who has access. These should be living documents; don’t file them away and forget about them. This documentation is of course needed for disaster recovery certainly. As you build the system, application out then any missing details can cause bad data, missing data, overworked databases and servers. Plus cost you timely review, audits and damages based on upgrades to code that destroys what was built before. Inside our software Soaring Eagle FLIGHT center we have a secondary application with its own MFA connection requirements where you can store the runbook and other documents along with the contact information of all of the pertinent personnel.
Allowing DevOps teams to make changes directly in the database, fast tracking quality assurance, ignore proper testing protocols. An excellent example of this is this year’s patches sent up by Microsoft and CrowdStrike. Both of these are famous for causing major disruptions for their customer’s businesses. Don’t skip the testing. I have seen CIOs direct a person that is stressed, tired and threatened to write and change major code initiates directly into the Production Database. When things go wrong they blame the coder. This is simply business malpractice. If you are a business leader and you request changes in production OWN IT. You are the responsible party and the person working for you is trying to follow your orders and please you. With responsibility, great respect and ownership should go hand in hand. Having secondary support team to help with the grunt work and a software that proves that the work is completed every day can free up your team to allow them to properly review and test code.
In summary you know the best IT business practices, even though budgets are tight, people are overworked, try your best to follow the Professional Standards of your seat in the business. There is help out there. Many of the grunt jobs can be handled by third party MSPs. A solid MSP has gone through these issues many times and can assist you, working inside your budget and within your priorities and goals.
