Wednesday, December 12, 2007

CCP Tries To Explain Themselves: Fails

Dr. Erlendur S. Thorsteinsson, EVE Online Software Group's director, has a lengthy post trying to explain the Best Bug Ever.
Shortly after releasing EVE Online: Trinity at 22:04 GMT on Wednesday, 5 December, we started receiving reports that the Classic to Premium graphics content upgrade was causing problems to players by deleting the file C:\boot.ini, which is a Windows system startup file. In some cases the computer was not able to recover on the next startup and would not start until the file had been fixed. In this dev blog I want to tell you how this happened.
He goes on to answer a few questions.
Why doesn't Windows protect its system startup files?
That's a good question, one that I have asked myself in these last few days and wish I knew the answer. But of course I'm not going to blame Microsoft for our mistake. Windows doesn't protect those files and therefore software developers must take care not to touch them. We should have been more careful.
I have to take offense to this answer. The question that needed to be asked: why was a file named the same as a critical Windows system file knowing full-well that EVE Online (like most games) will be played and PATCHED on an account with administrative privileges?

This could of been Linux and an fstab file with the same outcome; a PC that doesn't boot correctly. It baffles me that someone this high up in the company would even attempt to answer this question and state "I'm not going to blame Microsoft". I'm sorry, Dr. Erlendur S. Thorsteinsson, but it sure sounds like you are saying part of the blame goes to Microsoft.

Of course the answer to why the file was named boot.ini:
The answer is really "legacy"; it has been like that since 2001 when the file was introduced on the server and later migrated over to the client in 2002, so this file has been with us for over 6 years. We are reviewing all filenames and changing the name of any file that conflicts with Windows.
Or as I like to call it: lazy-assedness.

Reading through the comments, many EVE Online players are giving CCP props for full disclosure. Unfortunately, CCP really doesn't have a choice at this point in EVE Online's life with all the other drama that has swirled around the game.

The entire post details a breakdown of the most basic principals that guide any software project, from a Hello World! to Google. And I can't believe they DON'T have a single machine setup in their testing environment that mirrors what someone would be using at home (Windows XP installed on a single drive, game being played and patched on an administrator account).

All told, in the end, 215 users seem to have been affected. That is 215 too many.


  1. Anonymous6:45 PM

    "This could of been Linux and an fstab"
    You would be a fool to be using the root user as your normal day to day user.

  2. i was pretty happy with the answers. although i wasn't one of those 215 that probably had major issues. so it is easy for me to say, "ok, i forgive you ccp".

    i did like to hear that ccp was working with geek squad to service people that ran into boot issues. geek squad isn't the best, but it was a good gesture to provide that.

    all in all they handled it the best they could. yes, it shouldn't even have happened but what else are they supposed to do now at this point?

  3. Anon, I would like to see you patch the game without being logged in as root.

  4. Anonymous9:05 AM

    @Heartless it seems XP on a primary partition will recover, so the failure mode is only observed on XP/Win2K on a less common configuration than that straw man.

    It did mention this in the devblog.

  5. Hmmm I must of misread it then. I will have to take another look later. Didn't know a deleted boot.ini would self-recover.

  6. Looking back over the post, I don't know how they can say that Windows XP, primary partition or not, recovers by itself from a deleted boot.ini. And the post says specifically that Windows XP recovers if it is installed on the first partition of the drive. Any machine that isn't built by hand, most likely does not have Windows on the first partition. For example: Dell computers have a Dell partition as the very first partition on the drive.

    I will test this here shortly with Windows on the first partition of a drive.

  7. Well interesting. Windows does recover from a deleted boot.ini if the Windows install is on the first partition of the drive.

    However, it does not restore the file which could be troublesome at some point.

    I am guessing the bootloader is simply able to go to the first partition if no boot.ini is found.

    Still does not get CCP off the hook in my book. Dell, HP, etc. all have small first partitions that forces Windows onto the second partition and they are all huge companies that sell a ton of PCs. Not having a test PC that mirrors this sort of setup is inexcusable.

  8. Anonymous5:34 PM

    Sure, the "bug" was due to bad testing and lazy handling of legacy files. Probably bad coding standards as well.

    But no matter how you try to cover it still happens from time to time. It shouldn't but it does. Everyone who works in IT can probably tell a story or two that bears similarity to what the CCP patch did.

    Sure that doesn't fully get them off the hook, however by giving a full closure they got some of their credibility back (at least for me). I wish more companies would take up the habit of doing that.

    So the credit where the credit is due. Shit happened and CCP should get kudos for not trying to sweep it under the rug

  9. Meh, CCP has a long ways to go before I consider this "full disclosure". Added onto all of the other issues swirling around CCP, this is just another sign that they do not have control over their company, their processes, or EVE Online.

  10. Anonymous7:33 PM

    Hi Heartless,

    I don't understand why you say you take offence to that response from CCP. Here's the way I see it:

    Thorsteinsson gets asked the question "Why doesn't Windows protect its system startup files?" (implying Microsoft may be partly to blame)

    He answers "I don't know. I don't blame Microsoft for our mistakes. We should have been more careful."

    And you say: "I'm sorry, Dr. Erlendur S. Thorsteinsson, but it sure sounds like you are saying part of the blame goes to Microsoft."

    It really really doesn't seem that way around to me. This is a total "Sorry! We messed up." response. He's not blaming Microsoft in any way.

    That's my 2 cents anyway.


    -- Dave

  11. He didn't have to answer the question and secondly he answered the question incorrectly. Microsoft does protect it's files. But like any other operating system out there, to install, patch, or un-install most programs requires administrative (or root) access. Therefore, my comment about an fstab file in Linux stands.

    To state that again. He could of completely ignored that question, but he chose not to, he answered it poorly, and ended up making it seem as though "hey they aren't completely to blame, but they don't protect their files so it is part of their fault".


Join the conversation; leave a comment!