Site Downtime on 24 February

青山素子 · February 24, 2009, 10:55:09 PM

Today our site experienced an outage that lasted several hours. Once our staff became aware of the outage, our emergency backup site was activated for community support.

First, I'd like to let you all know that the outage wasn't the result of a hacking attempt or anything so glamorous. Rather, it was the very boring condition of failed hardware.

Specifically, the main power supply in our master database server decided to die. While some attempts were made to resurrect the machine, the power supply refused to perform its primary duty of supplying power. As a result, our database server was deprived of electrons and would not boot.

We have moved our database drives to our replica database server and got it booted back up. All the databases crucial for running our services have checked out okay. If you notice any problems that didn't exist previously, especially database errors, let us know.

lovearat · February 24, 2009, 10:59:41 PM

I know ya'll often go unappreciated for all the great work and time put into running this site. So I want to give ya'll my heartfelt Thank you for all that ya'll do. And for working so hard to get the site back online.

Apllicmz · February 24, 2009, 11:52:34 PM

thank you
i try to seach she not work
can check please

Nice work

fords8 · February 25, 2009, 12:41:16 AM

Glad everything is back up and running!

_Anthony_ · February 25, 2009, 01:06:30 AM

Glad to have it back though

青山素子 · February 25, 2009, 01:27:53 AM

I believe our search server was running on the slave SQL server, which is now disabled. I'm not sure if we'll be able to get it to return until we get the new power supply and boot it back up.

fords8 · February 25, 2009, 01:35:04 AM

No worries Motoko-chan! Do what ya got to do to get back to 100% . Thanks for the updates also!

EDIT: I hope this didn't hurt CodeFest at all too. They didn't have to take time away from that to deal with this?

青山素子 · February 25, 2009, 01:55:19 AM

No, those that could attend (I was unable to attend due to work) were still able to meet. I did keep them informed of developments by phone.

fords8 · February 25, 2009, 02:05:30 AM

Quote from: Motoko-chan on February 25, 2009, 01:55:19 AM
No, those that could attend (I was unable to attend due to work) were still able to meet. I did keep them informed of developments by phone.

Great! Looking forward to hearing what they get done.

metallica48423 · February 25, 2009, 03:34:32 AM

We all got here tonight -- the last will be here in the early afternoon tomorrow

Thanks for asking.

We'll try to get the search issue resolved as fast as we can

Aleksi "Lex" Kilpinen · February 25, 2009, 03:39:54 AM

Considering the fact that you actually lost hardware, you guys sure got back on line fast I think...

metallica48423 · February 25, 2009, 04:03:47 AM

Well, we have two database servers - a master and slave used for replication. We pretty much just swapped the hard drives from one to the other. Problem is, the sphinx search index was on the second server, which is now the one without a power supply

Amacythe · February 25, 2009, 04:16:26 AM

Quote from: fords8 on February 25, 2009, 02:05:30 AM
Quote from: Motoko-chan on February 25, 2009, 01:55:19 AM
No, those that could attend (I was unable to attend due to work) were still able to meet. I did keep them informed of developments by phone.

Great! Looking forward to hearing what they get done.

Well, thus far we managed to get everyone here who was on an airline flight, and those who got here early have managed not to kill each other. We've had a few discussions about how much we love this project, and how we all want to double our pay (It's a joke since we are all unpaid volunteers!) but alas, we haven't gotten drunk, nor have we had any of the famous orgies that most 'business' meetings tend to manage.

Ok, seriously... metallica will be posting some of the details of our conferences to the Blog as time permits.

PacificWx · February 25, 2009, 04:20:34 AM

Great work getting the site back - as had been mentioned in this thread, you guys should be congratulated about how quick you got the site back up considering the power supply failure.

Great work!

Dzonny · February 25, 2009, 06:02:49 AM

I'm glad smf is online again...

TW1ST3D · February 25, 2009, 07:30:36 AM

Wow !!! I was actually having symptoms of SMF Withdrawl Disorder...............

SleePy · February 25, 2009, 09:46:14 AM

Quote from: TW1ST3D on February 25, 2009, 07:30:36 AM
Wow !!! I was actually having symptoms of SMF Withdrawl Disorder...............

Your not the one who spent over 12 hours in travel time to get somewhere

* SleePy injects SMF into himself.

Brettflan · February 25, 2009, 09:52:30 AM

Glad to see things back up and running.

fords8 · February 25, 2009, 09:56:05 AM

Traveling does stink. But at least you are with people that like the samething you do. Now that is some coding power in one room! I wish I was there just to learn somethings!

LiroyvH · February 25, 2009, 11:54:46 PM

Quote
As a result, our database server was deprived of electrons and would not boot

Lmfao, that's the most nice phrased explanation i've ever seen for a failing power supply

Good job getting it back up

News:

Site Downtime on 24 February

PacificWx