The sorry saga

After a few years of excellent service from NACAA’s web hosting providers (Flexi e-Solutions Pty Ltd, ACN 119 070 147), we were struck with three web site “crashes” in two weeks (2014/2/15 to 2014/2/28). The causes of the crashes are unclear: “hardware errors” is what we are being told. Each time, Flexi migrated NACAA’s web site to a new server, without notifying us that they were doing so. Each time, they used out-of-date database backups for the migration. The effect was to wind the clock back several days, erasing content, updates, registrations and new membership details. Why this was done is also unclear: the “hardware errors” did not destroy our up-to-date data. Each time the data were still sitting on the previous server. I was able to use the previous data to manually restore the lost registrations and content, and (most of the?) new memberships. Twice. Each time took me many hours. After the second “crash”, I also put in place regular database backup procedures, since it was clear that Flexi’s backups were not reliable enough to allow rapid disaster recovery. I also started setting up an off-site backup solution. In the meantime, Flexihostings advised me that they would migrate the site back to our original server. I explained to them what steps they should take to ensure that this migration would have minimal impact on the operation of NACAA’s website. I received no responses to my requests. The third “crash” happened on the evening of Friday 2014/2/28, and wound back all my hard work, with the clock going back to 2014/2/17 (shortly after the first crash). We lost two weeks of changes, including my on-site backups. Flexi took more than 24 hours before they finally responded to my requests for access to the backups and up-to-date code base on the third server. They may have taken longer if not for an email to their CEO “Albert”. In the meantime, I had no choice but to take the site off line as soon as it was accessible on the Saturday afternoon. [1] On Sunday morning I was able to download the site content from just prior to the third “crash”. I used this to test the quality of the backup on my development system, then deployed it to the live site. I then manually restored the missing content and functionality, for the third time. The site seems to be working fine now, though there are some issues with the database server that need to be resolved. Please contact me if you spot any problems. Of course, the timing couldn’t have been worse. Registrations for NACAA XXVI were in full swing, with lots of new members signing up. I extend my apologies to everyone who has been affected by this mess. I have no idea whether this is the last “emergency” migration we will suffer. I hope to talk to someone in authority at Flexi soon and find out what their plans are to avoid any repeats of this adventure. In the meantime, I am investigating alternative hosting options, though moving to a new, untested provider this close to the event may be jumping from the frying-pan into the fire. Update 2014-03-16: Apparently the hardware error was caused by “disk sector errors”; ie, a hard disk crash. It’s not clear whether Flexi have any form of disk redundancy (RAID) for their servers. It would appear not, even though it has been industry best practice for decades [2]. It is also not clear whether Flexi have any pro-active health monitoring system in place; hard errors are almost always preceded by weeks or months of soft errors, and so are easily predicted. It is highly unlikely that “disk errors” occurred on three different servers in a two week period, after years of stable operation. The third “crash” migrated the NACAA site back to the original server. My guess is that this was the migration that we were advised would happen, gone badly wrong, with no notice, and with no attempt to follow the guidelines I had provided. “Disk errors” also doesn’t explain why the backups used for each recovery were several days old [3]. No attempt was made to communicate with the affected customers before or after the event, unless they raised a support ticket. Contrast this with the service provided when I was migrated to a new server in 2013: lots of emails letting me know the state of the migration as it happened. It’s clear that customer support at Flexi has gone backwards in the last few months, in spite of the assurances of its CEO “Albert”. The good news is that, finally, their technical support person “Cliff” has followed the instructions I provided many days ago, and fixed the final problem with our database server. I hope to restore full site functionality over the next few days. Flexi continues to refuse our requests for meaningful compensation for the impact on NACAA, and the many tens of hours of my time wasted. They have offered to allow us to have free hosting, but only if we endorse the company as a “sponsor”. Considering their behaviour over the last few months, I don’t think that NACAA can endorse the services and support provided by Flexihostings, particularly in comparison to our real sponsors. Update 2014-03-18: I spoke too soon. The database problem was only partly solved. The fix to the latest problem is obvious (it’s a simple permissions issue with MySQL and CPanel). I’m now being asked to write an essay, full of screen dumps and lots of details, for something that should take five minutes to fix. Update 2014-03-26: Flexi are still not being very helpful. They’ve increased their compensation offer to nine months (~$60). They still haven’t fixed the database permissions problem, even though I’ve provided them with all the information they need to do so. Looks like I’m going to have to waste another few hours of my life explaining to them how to do their job, again. There is still no one in authority at Flexi willing to talk to me. Not even “Albert”. Update 2014-03-31: Flexi has finally figured out that there is a “privileges” issue with the repaired database and the NACAA CPanel account. There is still no one in authority at Flexi willing to talk to me, in spite of more requests. Bottom line: if you are thinking of using Flexihostings to host your web site, don’t. Update 2014-04-06: Our site went down again, for nearly 24 hours. Over the last few days, Flexi’s “senior admins” have been trying to fix the “privileges” issue with the NACAA database and our CPanel account. Their first attempt was laughable; they tried to grant permissions to the wrong account. Last night they tried a different approach, and locked out the account that the web site needs to access the database. I didn’t spot the problem till late Sunday morning. I contacted their “help” desk, then waited a few hours for them to fix it. By late evening, I went in to see what they had done. I couldn’t fix their mistake myself, but was able to work around it in a few minutes [4]. As I’ve said earlier, if you are thinking of using Flexihostings to host your web site, you should keep in mind that their senior technical staff are, based on the evidence I have seen today, completely incompetent. Caveat emptor. I’ve written to “Albert” again. No response yet. Steve Russell, NACAA General Secretary, Webmaster [1] I was using the web site that Friday evening. I still recall the “not again!” feeling as I watched the site slowly crash around me. Saturday morning was even worse. I watched as the Flexi “support” team tried various backups. Each time, the NACAA site would come up, then shut down again, as they tried to find a backup that “worked”. Each time I watched more and more of our content disappear as they wound the clock back. The worst part was watching the backups disappear. [2] I have three RAID disk systems in my home network. It’s not rocket science. Storage technology has come a long way since the 1980s. [3] Backup practices have evolved since the 1980s too. A simple dictum is that things that change frequently should be backed up frequently. Our database was undergoing dozens of changes per day at the time of the “crashes”. It appears that Flexi’s backups were not keeping pace with a load as enormous as ours. Heaven help anyone who needs to cope with a hundred or more updates per day! [/sarcasm] 🙂 [4] After taking weeks to convince them that they have a problem, Flexi has been trying to fix the database permissions problem by adjusting the database privileges for the MySQL account that is used by Drupal to access our content database. This is not the problem: it’s a CPanel issue. Their first “solution” was to grant all privileges to the runtime account. This didn’t break anything, but it’s lousy security. The privileges I had set up for the account were all that were needed. Granting more permissions breaks the “minimum privileges” principle. It also didn’t fix the problem, since it’s not a runtime (Drupal) issue. I honestly don’t know what they did in attempting their second solution. All I know is that they managed to lock out our runtime account from accessing our database. The workaround was to create a new account (which was necessary anyway, as our existing account name was exposed to the public for 24 hours, which is a significant security breach.) I then gave the new account the necessary access privileges, and updated Drupal to use the new account. Total time: less than 5 minutes. I doubt that Flexi are aware of what the problem that they created is, let alone have any idea how to fix it. My guess is that they weren’t even aware of the latest problem until I notified them. Alas, I have little or no hope that they will be able to fix the original permissions problem.