I still don’t know why I gave into my manager wanting me to postpone my scheduled vacation by a week, so he has some billable hours to report in December.
So here I was, stumbling into the office after a terrible drive through heavy traffic and a 20km detour because of a roadblock caused by a serious accident. As expected most of my customers administrators where either still not in office or in a meeting. I start the computer and fire up Outlook and the Operations Manager Console, and… Uh-Oh, this can’t be good.
I was looking at about 1000 critical alerts, all less than 36 hours old and of the same type.
It turned out that a new Windows worm has appeared over the weekend, infecting computers without a previously released patch and/or old versions of Antivirus Software.
To say I was surprised of this sort of fallout would be … a huge lie, since I kept telling every responsible stakeholder for months this could be happening anytime at the way they are handling update and patch distribution.
And it continued to play out just as I had expected it would. Someone found the single patch that would prevent an infection, and everyone with an Administrator account jumped at any server they could get a hold on to install exactly this single patch manually, instead of installing the other 28 – 79 missing patches along the way.
It also reminded some of them that I had published a process on how to identify and classify servers for automatic update distribution. After all this document was available to everyone for the last few months and presented to management for consideration.
Two days later most of the monitored servers are responding again, which doesn’t imply they are no longer infected, just patched. But as far as I can tell we are only monitoring about 60% of all servers after all.
Since there is no active WSUS on the network, all patches had to be downloaded from Microsoft to the servers which took hours. The worm was putting so much additional traffic on the network that the proxy servers collapsed several times. User where constantly complaining about service and accessibility.
The last company I witnessed this kind of chaos after a virus out brake was during the Blaster era (2001). It took them less than a month after this incident to implement strict policies and processes to prevent this from happening again. And as far as I know it never has since.
Knowing there are still companies like my current customer out there, that don’t have a clue about processes, security and management only fill me with the assurance that there is still a great potential for future business, if only they would listen to their consultants.
That’s why the pay us, right?