If you tried to visit our site at any time since Tuesday morning, you were likely met with an error message of some sort, or a temporary message about our site being down. Temporary turned into five days as our hosting company, Dreamhost failed us spectacularly this week.
If you follow us on Facebook or Twitter, you got the play by play of our annoyance and desperation through the week as we submitted support ticket after support ticket with no response other than cryptic “we’re working on it” messages from reply robots.
The short story is, the server our site is hosted on had a hardware failure on Tuesday morning. Dreamhost attempted to move everyone on that server to new hardware, but that process failed, forcing them to fix up the old server, restore from backup, then move to a new server. While our site was half moved from one server to another, then back, then restored from backup (kind of), various things within the WordPress install reported back failure errors to anyone trying to visit the site.
Over the days we received various automated replies about things supposedly being back “tomorrow” and some people having service, but slow. With our random WordPress error messages and our inability to get a handle on where exactly Dreamhost was in their process (since we never got a human response to any of our requests for information), we were left to twiddle our thumbs and fume.
Misery loves company and we weren’t alone, here’s some of the chatter from Twitter:
Dear @dreamhost, return our website to its former glory, s’il vous plait. We have many students who need us! Merde! Quel disastre!
— Int’l Mime Academy (@IntlMimeAcademy) June 9, 2012
Our corporate website is still having technical problems. We’re reexamining our relationship with @dreamhost. Please stand by…
— Metacortechs (@MetacortechsInc) June 9, 2012
Our website is still down! Dear @dreamhost, take off your awesome webishades for a second and fix us!!
— Webishades! (@webishades) June 9, 2012
Our site is still down. It’s Day #6. We blame the mimes. And @DreamHost
— No Mimes Media (@NoMimesMedia) June 9, 2012
— Steve Peters (@vpisteve) June 9, 2012
Late Saturday (remember, we went down on Tuesday), we received a message that indicated that the various server moves might be done. We were having a consistent WordPress error message regarding one plugin. We visited some of our new Twitter friends sites and saw they were coming up. So we went into FTP and removed the plugin we were getting the error message on and hey presto, we were back online.
At this point, we’re still not positive where we stand regarding the server. Have we been moved to a new functioning server? Is the movement of sites done? Our last robot reply from Dreamhost indicates that we should be on new hardware at this point.
New hardware be damned though, we’re once and for all done with Dreamhost at this point. One of our readers who does this web stuff for his real life job has kindly volunteered to move us to a new hosting company. That move will probably take place Monday and itself will likely result in some downtime. However, we have experience with Bluehost (the service we’re moving to) and have had no problems and good service from them, so we hope to have things sorted next week.
After that, we’re planning to work on more robust backups and some scheme to mirror the site elsewhere so we won’t be left dead in the water like this again.
So we’re nearing the end of our long nightmare. Thanks to everyone for hanging in there, your understanding, and everyone’s offers of help.