About McAfee’s new business strategy

This post is going to be a rant because I really need to vent. If you are expecting a factual recount of the events that came to pass or have no feeling for the linguistic phenomenons known as irony and sarcasm this post is not for you so please leave this blog through the following link.

If not, please read on.

In a move that took many by surprise late yesterday afternoon in CET McAfee astonished the world as it pushed a virus definition update for their Enterprise product line that bricked Windows 2000 and Windows XP machines.

It seemed that McAfee had decided it was time for businesses to upgrade their outdated machines to newer versions of the Microsoft Operating System and just tried to nudge everyone gently in that direction with catastrophic consequences.

What McAfee did not anticipate is that their well intended update marked svchost.exe, a critical process in Windows since it’s birth, as a virus and quarantined or even deleted it. The result of that action was that people experienced Blasterworm-like symptoms, PC rebooted, stuck in reboot loops or not booting at all anymore.

The result of this was that big enterprises suddenly had over thousand machines that were failing, all because the virus scanner turned against them (Skynet anyone?).

So you start to wonder, how does this blatant bug in an update get past something we call testing. Simple, it doesn’t. By simply installing the venerable update the updated machine explodes in front of your eyes, everything goes haywire. Had this update been tested on Windows 2000 and Windows XP such a grave mistake would never have gotten through testing so the only conclusion I can make is that they just did not test it. We’re talking about one of the biggest security firms here, with thousand and thousand of companies buying their products such as virus scanners to protect them, not nuke them.

Poor us over at Inter-Actief experienced the same problem too, At 17h00 CET our machines pulled the defective update from McAfee resulting in five bricked machines. This could have been way worse because since the symptoms resembled so much the Blasterworm symptoms we initially assumed it had to be some variant to W32.Blaster we got infected with. Only an hour later did we discover, by chance, that the problems started exactly at the time McAfee was set to update. A quick search for #mcafee on Twitter revealed the ugly truth.

By some miracle the other machines in our network had somehow failed to pull the update at 17h00 CET and were still functional. We immediately logged on to our firewall and routers, pulled up the configuration screen and told it not to route our Windows XP machines’ traffic from and to the internet anymore in order to prevent any from the still functioning machines to be able to reach the McAfee update servers and fetch the defective update.

After that we wrote a Group Policy which we pulled into the Active Directory that told all our clients to disable the McAfee Update Scheduler so that when we turned our internet connection back on they wouldn’t update and break.

Now we still had 5 broken machines we needed to fix. At that time, around 20h30-ish CET there was still no word from McAfee or a fix, people were frantically disabling the McAfee Update Scheduler, deleting the evil DAT-file from ePO and anything else they could think of.

We needed to figure out a way to get rid of McAfee and it’s broken virus definitions and then restore svchost.exe. How do you get rid of an anti-virus scanner? You uninstall it. We booted in Safe Mode (try hitting F8 during boot for fun) and when you boot Safe Mode you anti-virus scanner is not started since Windows only starts essential processes (though it is slightly ironic that Windows considers an anti-virus scanner not to be an essential process). Once booted we browsed to the program-files directory, %PROGRAMFILES% and renamed the McAfee folder to McAfee_whores. After that we rebooted normally. Why did we do that? Once that folder is renamed McAfee can’t find itself during normal boot because it expects to find it’s binaries in a folder named McAfee, not McAfee_whores. We now logged in as a local Administrator, renamed the folder back to McAfee, ran the uninstaller and after that restored svchost.exe from a working machine back into the %WINDIR%\system32 folder. One more reboot and we were done, the machine was working again.

Thanks to the way we have stuff installed in our network McAfee was immediately automatically reinstalled after that last reboot through our network but because of the earlier Group Policy the Update Scheduler was turned of. So now we had functional machines, that weren’t updating anymore. Some with fairly new virus definitions, the restored systems with quite a bit older virus definitions but at least they all had a working anti-virus scanner again which let Windows be instead of killing it, which is quite a step forward.

By the time we had finished this dance with our machines it was now 22h00 CET and McAfee had released a temporary fix, a DAT-file which one could install on working machines to prevent it from eating up svchost.exe and that update was being pushed to all their update-servers too so with a few hours everyone should be able to get the new update.

They also released a note saying that ‘An earlier update this evening could cause minor to significant performance issues on affected machines’ and that they apologised for that but that unfortunately, the affected machines would have to be manually fixed by a system administrator. I don’t know if you noticed the ‘minor to significant performance issues’ but spontaneous reboots and bricked systems… seriously, understatement of the year.

After all the hell was over and helping a few other people over the phone with the same problem I have come up with the following new publicity lines for a new McAfee spot:

5, the number of hours it took us to fix our broken update. 5, the number of your PC’s we bricked. 5, the number of hours it took you to fix our mistake. 5, the number of hours of sleep you got thanks to us. 5… the new fragrance, by McAfee.

I also figured they would need some new company slogans so there goes:

McAfee, empowering you to fix our mistakes.

McAfee, disconnecting people.