For T-SQL Tuesday this month, Marlon Ribunal invites us to write about an unforgettable SQL Server outage. Click the T-SQL Tuesday logo to the right to read over the invite post.
I’ve written about a few outages and issues in the past and wanted to write about a different one for this post. One that came to mind was from awhile ago when a code deployment appeared to be successful on the surface, but in reality turned into a time consuming mess.
What time is it?
We had code deployments fairly often, and 99% of the time they worked well. They were well tested, and my biggest concern was making sure my alarm woke me up so I could sign on to press the big red button to deploy.
For the deployment in question, it all went to plan. I woke up, prepped the deployment, had time to get coffee, and came back to deploy the code successfully during the window. All was well. Or so I thought.
Later on that morning, reports started coming in from users about times not being accurate for some pieces of data. After taking a look at the tables where complaints were being made, it was clear some records meant for local time (but not all) were getting saved in UTC time.
The source of the problem wasn’t SQL Server itself, but another part of the release. Once identified, it wasn’t too hard to stop the bleeding. But the incorrect data still needed to be cleaned up, and trying to pick out the records that had the wrong time without incorrectly modifying the records that already had the “right” time was…complicated.
It’s Five O’Clock Somewhere
If you’ve never had to deal with a system setup with multiple time zones, consider yourself lucky. It’s not fun when you have an environment where a database has some tables using UTC time and other tables using local time. Then another database on the server has error logs that might be dated local time but only local to where that server is located. Meanwhile, the error messages themselves reference a different time altogether.
Of course, we got the data cleaned up eventually, and a lesson was learned for additional testing to catch that type of problem in the future. While the outage itself wasn’t too long, the cleanup process was one of the worst I’ve had to be a part of. Hopefully, I never run into anything like that again.
Thanks for reading!
Chad Callihan 