Photoblog Status Updates

June 01, 2008 - 9:10pm PST

Dear Photoblog.com members,

Yesterday evening at 4:55pm, the datacenter that houses Photoblog's servers experienced a significant disaster, but luckily none of the servers in the datacenter were damaged and no data was lost. Here is the email we received from them:
This evening at 4:55 in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room Thankfully, no one was injured. In addition, no customer servers were damaged or lost. We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department. This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock. We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well. There is no impact in any of our other five data centers. I am sorry that this accident has occurred and apologize for the impact.

Sincerely,

Douglas J. Erwin
Chairman & Chief Executive Officer
Here are a couple news articles detailing what happened:
Explosion At ThePlanet Datacenter Drops 9000 Servers
Houston hosting provider The Planet chronicles weekend power outage and recovery process
Fire at The Planet takes down thousands of websites

Communication with the server company has been terrible so far, and they haven't given us much info as to when they'll have power on and the servers back online. We understand how frustrated you all must be, and how many emails we've probably missed with the servers down. However, please understand that we're working as hard as we can with the limited info we've been given to bring Photoblog back online for all of you.

The company has been very unclear as to how progress is going, so there isn't much I can report right now. You can read their announcements in this forum thread. Since our servers are offline, we cannot contact any of our members directly, or post announcements on our website. Instead, we're forwarding the Photoblog.com domain to this website to make announcements until our server is back online.

We cannot access Photoblog.com email accounts right now, but if you would like to contact me directly for any reason, please send emails to this temporary email account: michaelzhang@temporaryforwarding.com

I'm truly sorry for this incident, and apologize for not forwarding the domain to notify all of you of this situation earlier, since we thought power would be brought back sooner. I hope to bring all of you good news soon!

Sincerely,

Michael Zhang
Photoblog.com



June 01, 2008 - 11:30pm PST

For the latest news from the datacenter, please visit this page they've set up: Datacenter updates. Sadly, regarding the most recent announcement (June 1 – 11:00pm), our servers are Phase 1, so their ETA for power being restored to our servers, with temporary generators, is sometime tomorrow evening (June 2nd).

Once again, we're terribly sorry for all the downtime that has occurred in the past day.



June 02, 2008 - 11:30am PST

We discovered this morning that only one of our servers (the main one) was in the affected datacenter. However, it was in Phase 1, the floor that was more affected by the outage. The company is estimating that our server will be back online sometime this evening.