Simple Machines Community Forum

Simple Machines => News and Updates => Topic started by: Compuart on July 02, 2009, 02:05:49 PM

Title: Server downtime July 2nd
Post by: Compuart on July 02, 2009, 02:05:49 PM
July 2nd, between 11 am and 2 pm EST we've experienced server problems causing simplemachines.org to be unreachable. We are still looking into what happened, but suspect a power failure might have caused this issue.

The site has been checked for errors and should be working properly. If you find something out of the ordinary, please let us know.

We apologize for the inconvenience the outage might have caused.
Title: Re: Server downtime July 2nd
Post by: N3RVE on July 02, 2009, 02:06:57 PM
Thanks :)

-[n3rve]
Title: Re: Server downtime July 2nd
Post by: Drumstick on July 02, 2009, 02:19:39 PM
Ok, thanks for the info ;)!
Title: Re: Server downtime July 2nd
Post by: kat on July 02, 2009, 02:26:07 PM
I got a 404, a few minutes back, but all's cool, now.
Title: Re: Server downtime July 2nd
Post by: skyhintack on July 02, 2009, 02:27:01 PM
I'm able to connect to the site every few refreshes or so, but the server keeps resetting my connection. I'm not sure where that issue lies, but I'll type in simplemachines.org and it will almost load the site, but then I get the error.

Thanks  ;)
Title: Re: Server downtime July 2nd
Post by: GravuTrad on July 02, 2009, 02:58:44 PM
derek will surely explain us the problem soon. ;)
Title: Re: Server downtime July 2nd
Post by: mirahalo on July 02, 2009, 03:02:11 PM
everything seems fine for me except for customize page: http://custom.simplemachines.org/

it gives me a 404 error

but besides that everything is normal :)
Title: Re: Server downtime July 2nd
Post by: ReVoKe on July 02, 2009, 03:05:23 PM
everything works fine but the search :(

i get this when trying to search on a topic:

An Error Has Occurred!
Unable to access the search daemon
Title: Re: Server downtime July 2nd
Post by: N3RVE on July 02, 2009, 03:07:49 PM
Quote from: 130860 on July 02, 2009, 03:02:11 PM
everything seems fine for me except for customize page: http://custom.simplemachines.org/

it gives me a 404 error

but besides that everything is normal :)

I don't get that, it loaded normally.

-[n3rve]
Title: Re: Server downtime July 2nd
Post by: Chas Large on July 02, 2009, 03:09:48 PM
I'm seeing the Search Error too. 8pm UK Time, if that helps.
Title: Re: Server downtime July 2nd
Post by: metallica48423 on July 02, 2009, 03:24:37 PM
I can now verify that the cause of this was a power failure.  The datacenter was undergoing a routine UPS inspection when one of the UPS systems dropped load, causing power outages within the datacenter.

Here's the current status as far as I can tell:

- Email is down (the devel server is not fully up yet)
- Search is down (db02 is not fully up yet)

So notifications and search, as of this time, will not work.  We're still working on this, however, getting the site accessible was the number one priority, first and foremost :)

Thanks :D
Title: Re: Server downtime July 2nd
Post by: sAce on July 02, 2009, 03:25:40 PM
all k, except mods area
Title: Re: Server downtime July 2nd
Post by: metallica48423 on July 02, 2009, 03:28:39 PM
The customize site should be working properly.  I have had no problems with it and have downloaded some mods to test :)
Title: Re: Server downtime July 2nd
Post by: Kill Em All on July 02, 2009, 03:39:41 PM
I get the search error to. Maybe a possible power surge screwed some things up?
Title: Re: Server downtime July 2nd
Post by: mirahalo on July 02, 2009, 03:45:06 PM
Quote from: metallica48423 on July 02, 2009, 03:28:39 PM
The customize site should be working properly.  I have had no problems with it and have downloaded some mods to test :)

OK, first, I tried with opera and it show me a 404, then I tried with IE8 and worked fine!   

this is weird.... it supposedly be the other way around...  :P
Title: Re: Server downtime July 2nd
Post by: metallica48423 on July 02, 2009, 03:50:06 PM
Quote from: Kill Em All on July 02, 2009, 03:39:41 PM
I get the search error to. Maybe a possible power surge screwed some things up?

See my post above:

- Email is down (the devel server is not fully up yet)
- Search is down (db02 is not fully up yet)
Title: Re: Server downtime July 2nd
Post by: Cassiel on July 02, 2009, 04:05:29 PM
Wow, that was back up quick. Awesome job. :D
Title: Re: Server downtime July 2nd
Post by: SteveFJ on July 02, 2009, 04:05:46 PM
Oh I know what caused this problem.  :P

I'm with o2, and they've been down nearly all day. Leakage from them - that's what it is. :o

Thanks for being on the case, and for returning to normal.  :)
Title: Re: Server downtime July 2nd
Post by: metallica48423 on July 02, 2009, 04:10:35 PM
Search and email should both be functional now
Title: Re: Server downtime July 2nd
Post by: program5 on July 02, 2009, 04:59:40 PM
The only thing I saw was a blank page with nothin' in it.
Since I am a paranoid person, I immediately started making scenarios :D

Good to see You're back.
Title: Re: Server downtime July 2nd
Post by: Yigal on July 02, 2009, 05:04:20 PM
Thanks for the heads up :P
Title: Re: Server downtime July 2nd
Post by: rapvideosguy on July 02, 2009, 07:07:22 PM
thanks man for the heads up
Title: Re: Server downtime July 2nd
Post by: Arantor on July 02, 2009, 07:11:58 PM
I seem to have missed the worst of it, as it only kicked in just as I was going out for the day...
Title: Re: Server downtime July 2nd
Post by: [Core] on July 02, 2009, 09:22:26 PM
Works fine for me
Title: Re: Server downtime July 2nd
Post by: JBlaze on July 02, 2009, 09:31:08 PM
Very nice job all. I fell asleep shortly after it went down, so I got to see metallica48423 in action so to speak lol.

Title: Re: Server downtime July 2nd
Post by: zaclondon on July 03, 2009, 06:03:12 AM
I couldn't access the server for a good few hours yesterday afternoon UK time. I tried to find an alternative download source, including finding your mirrors in a Google cached page of your download page, but since they are under simplemachines.org domain they didn't work either!

Might it be worth having truly alternative download locations so this doesn't happen again?
Title: Re: Server downtime July 2nd
Post by: SteveFJ on July 03, 2009, 06:46:24 AM
Quote from: zaclondon on July 03, 2009, 06:03:12 AM
Might it be worth having truly alternative download locations so this doesn't happen again?

I think what needs to be worked out there is frequency of occurrence vs the cost.

The downtime was a very short period really, the developers were on it pretty sharpish. That's their spare time for which they don't get paid as it's all free. For that they are applauded. Running back up sites elsewhere would mean more spare time out the window. For the amount of times this sort of thing happens, it might not be worth it. The SMF support is superior to many a paid system.
Title: Re: Server downtime July 2nd
Post by: Stallyon on July 03, 2009, 06:59:12 AM
I keep getting the error "Get off that damn computer and help me!" oh ... wait ... that's my wife!
Title: Re: Server downtime July 2nd
Post by: Norv on July 03, 2009, 05:52:25 PM
Ah! Nice job overall!

Funnily enough my ISP had a problem around the same time (leaving me with no internet connection). Shortly after the problem seemed solved, simplemachines.org was not loading... So while thinking to a few nice words for the same ISP, I turned to play some single-player RPG instead.
Title: Re: Server downtime July 2nd
Post by: SleePy on July 03, 2009, 07:53:15 PM
Quote from: zaclondon on July 03, 2009, 06:03:12 AM
I couldn't access the server for a good few hours yesterday afternoon UK time. I tried to find an alternative download source, including finding your mirrors in a Google cached page of your download page, but since they are under simplemachines.org domain they didn't work either!

Might it be worth having truly alternative download locations so this doesn't happen again?

I do have a copy of our publically usable mirrors file setup on my site. It links to our download mirrors.
http://sleepycode.com/index.php/action,getsmf.html
Title: July 5 2009 Re: Server downtime July 2nd
Post by: KirkhamsEbooks on July 05, 2009, 02:43:56 PM
July 5 2009

I'm getting this right now is there something else going on?

Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.
Title: Re: Server downtime July 2nd
Post by: JBlaze on July 05, 2009, 03:00:38 PM
Quote from: KirkhamsEbooks on July 05, 2009, 02:43:56 PM
July 5 2009

I'm getting this right now is there something else going on?

Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

If you are getting this here on simplemachines.org, then please send an email to [email protected]

If this is happening on your forum, then please open a support topic.
Title: Re: Server downtime July 2nd
Post by: Joe N on July 06, 2009, 09:39:17 AM
Works Fine

Shouldn't the backup site be closed now?
Title: Re: Server downtime July 2nd
Post by: Peludo_08 on July 07, 2009, 12:08:36 AM
Quote from: Drumstick on July 02, 2009, 02:19:39 PM
Ok, thanks for the info ;)!
Title: Re: Server downtime July 2nd
Post by: Yigal on July 08, 2009, 03:48:38 PM
Quote from: Joe N on July 06, 2009, 09:39:17 AM
Works Fine

Shouldn't the backup site be closed now?

There's a back up site? o.o
Title: Re: Server downtime July 2nd
Post by: ethankcvds on July 08, 2009, 03:55:05 PM
Quote from: Yigal on July 08, 2009, 03:48:38 PM
Quote from: Joe N on July 06, 2009, 09:39:17 AM
Works Fine

Shouldn't the backup site be closed now?

There's a back up site? o.o

Yes there is but it's only used when the main site has problems like the SQL database server going down.
Title: Re: Server downtime July 2nd
Post by: Yigal on July 08, 2009, 04:04:55 PM
Oh, where can I find this link?
Title: Re: Server downtime July 2nd
Post by: JBlaze on July 08, 2009, 04:06:57 PM
Quote from: Yigal on July 08, 2009, 04:04:55 PM
Oh, where can I find this link?

IIRC, http://www.simplemachinesforum.com or something like that.

Title: Re: Server downtime July 2nd
Post by: metallica48423 on July 08, 2009, 04:53:52 PM
http://www.simplemachinesforum.org
Title: Re: Server downtime July 2nd
Post by: Yigal on July 08, 2009, 04:58:27 PM
Quote from: metallica48423 on July 08, 2009, 04:53:52 PM
http://www.simplemachinesforum.org

Thank you.  Just a question: Will we have to sign up again once it is re-opened?
Title: Re: Server downtime July 2nd
Post by: SleePy on July 08, 2009, 05:06:25 PM
Yes, that is a off site location totally independent of our server setup.
Title: Re: Server downtime July 2nd
Post by: Yigal on July 09, 2009, 10:59:18 PM
In that site, there is support available for users, correct?
Title: Re: Server downtime July 2nd
Post by: IchBin™ on July 09, 2009, 11:02:47 PM
Only when this site is not available.
Title: Re: Server downtime July 2nd
Post by: riker on July 17, 2009, 07:40:14 AM
Was the backup site not used then ?
Title: Re: Server downtime July 2nd
Post by: Arantor on July 17, 2009, 07:59:14 AM
Yes, the backup site was in use while the main site was down two weeks ago.
Title: Re: Server downtime July 2nd
Post by: rune-coders on July 17, 2009, 05:45:06 PM
Sorry to hear this :(
Title: Re: Server downtime July 2nd
Post by: riker on July 18, 2009, 05:22:25 AM
Quote from: Arantor on July 17, 2009, 07:59:14 AM
Yes, the backup site was in use while the main site was down two weeks ago.

Thank you for highlighting it was two weeks ago but my question was still valid :)
Title: Re: Server downtime July 2nd
Post by: Arantor on July 18, 2009, 06:06:34 AM
I know - and I answered your question.
Title: Re: Server downtime July 2nd
Post by: riker on July 19, 2009, 03:52:13 AM
Quote from: Arantor on July 18, 2009, 06:06:34 AM
I know - and I answered your question.
Yes making sure you added the two weeks, suttle ;)