A friend recently told me of a frustrating weekend database issue that caused some full disk drive alerts. I thought it would be worth sharing, as there are some important technical and personal lessons that can be learned.
If you’ve been around databases long enough, you’ve been in a situation where disk space was running low for log files. The cause and fix aren’t always the same, but this story shows some examples of what not to do.
In my friend’s case, log files kept growing out of control over the weekend. Out of all the ways to approach the issue, my friends coworker managing the databases at the time decided to sign on, try shrinking logs, and sign back off. It may be no surprise that a short time later, alerts were being sent that logs were still filling up. At this point, it’s time to investigate what’s causing unusual log growth, right? Wrong.
Since shrinking logs didn’t solve the problem, the next decided step towards a fix was to put in a request to increase the drive space so that there would be enough space to stop the alerts. It didn’t take long for logs to fill up again and cause more alerts due to full disks.
Eventually, my friend was brought in to do more thorough investigation. Once the queries running against the database were investigated, an issue from a recent release was found, a quick change was made, and things were back under control.
Shrinking files and expanding drives were both band-aid fixes that arguably did more harm than good. The query running wild was going to fill up logs regardless. I’m not saying there’s never a time to add more disk space, add more memory, etc. but don’t ever blindly use it as a crutch while crossing your fingers, expecting it to solve a problem.
Now, for the main point of this post. There was concern about what the coworker was doing with the steps taken to address the issue. Were they being lazy and not wanting to investigate the cause of the issue? Perhaps, perhaps not. Either way, treat it as a learning opportunity and don’t assume someone is purposely kicking the can down the road. Maybe someone found themselves in a position they weren’t ready for. Maybe someone is just trying to do their best to help out.
Far too often we assume the worst in people. Especially around the holidays, give the benefit of the doubt and assume people mean well. You might learn that, more often than not, they do.
Thanks for reading!
Chad Callihan 