For this month’s T-SQL Tuesday, Pat Wright invites us to write about weighing risk vs. reward when it comes to database design decisions. While Pat’s prompt mentions design, he also references risky decisions during production incidents. I’ve been involved with more of these reactive decisions than proactive ones and have learned that it’s better to slow down to avoid making problems worse.
Check out the full invite by clicking the T-SQL Tuesday logo.
Don’t be a Cowboy
Pat mentions needing to decide on how to handle production incidents within seconds. As urgent as issues may seem in the moment, they’ll often feel more urgent than they actually are. Remember, when everything is high priority, nothing is high priority. Rushing into split-second decisions can lead to hours of cleanup if that decision is wrong.
Let’s say a client is down and calls into support to report the problem. Support might identify the problem as a known issue that has a documented script template to use as a workaround. If they take that script template, populate it with values specific to the client, and send it to you to run, should you blindly run it?
No!
You should always know what’s in the script you’re running and what it’s going to do. This is where the experience of being in a DBA role and knowing the context of your environment is more valuable than what other associates (or what AI) can tell you. Maybe this script doesn’t actually apply to the issue. Maybe the support associate fat-fingered a value or sent over the wrong script. It is your responsibility to catch it.
If I’m the one receiving a script that needs to be executed, I want to know what the issue is, where the script fix template came from, and solid evidence to back up why the changes are being requested. Even then, I’ll still double-check all that myself (trust, but verify). After all, I’m the one pressing the button to make the change, so the fingers are going to be pointed at me first if things go wrong.
Not Just the Calls Inside the House
A similar situation may be when dealing with third-party vendors and code they want released. If you’re being given an update to implement, check it for yourself before deploying. I’ve run into scenarios where there were typos, wrong tables being updated, and more. Once again, trust but verify.
C-Y-A
We all have situations where we’re overwhelmed, and the easy way out is to just take whatever comes through your email and Slack and say “Looks good to me.” The lesson I learned early on was that dealing with production issues is one area where you can’t afford to be lazy. Dot your I’s and cross your T’s because it’s not worth the risk not to.
Thanks for reading!
