Simple Machines Community Forum

Simple Machines => News and Updates => Topic started by: dschwab9 on January 19, 2008, 02:02:07 AM

Title: Recent Downtime
Post by: dschwab9 on January 19, 2008, 02:02:07 AM
As you all know, we just had quite a long downtime.  Just wanted to post an update to let you guys know what happened and assure you that we weren't hacked or anything like that.

Things started out as a higher than normal CPU load and slightly sluggish server and pretty quickly went south.  I arrived at the datacenter expecting a runaway process or something was making the machine unreachable.  Instead, what I found was a server that was hot enough to fry eggs on and a seriously corrupted file system.  Below is, best I can tell, the chain of events that occured:

1.  The thermal management system on the motherboard wasn't properly throttling fan speed, possibly for quit a while.  This caused the CPU temperature to hoover near the danger zone off approximately 65 degrees C.
2.  CPU load rose higher than normal, possibly caused be heavy traffic and/or a bug on the latest site upgrade.
3.  The increase in CPU load caused a proportional increase in heat being produced.
4.  The CPU temperature rose to over 75 degrees C, well into the critical range.
5.  The CPU became flakey and ultimately caused the entire box to come down, corrupting nearly every file that was open at the time.

Then, the attempted repair:
1.  Ran fsck on the disks, repaired numerous errors, and began to restore corrupted files.
2.  Site appears to be relatively stable in maintenance mode.   In preporation to come out of maintenance mode and call things "good", we rebooted the box to see if any more errors pop up.  A few more file system errors were found and repaired.
3.  Another reboot was performed to see if the box would come up cleanly.  This time, we are greated with thousand upon thousands of file system problems, many of which were unrecoverable.
4.  At this point, I made the decision to leave the datacenter, go home, and bring in another server to replace the entire machine.
5.  New machine was installed and all apps reconfigured.
6.  A couple of critical directories were not properly backed up, so the old server was disassembled, the drives mounted in another machine, and a reconstruction done on that data.

So, it has been a looonnggggg night.  In order to prevent future problems like this, the fan speed control has been disabled on all of the servers.
Title: Re: Recent Downtime
Post by: shadow82x on January 19, 2008, 02:04:30 AM
Great to see SMF is finally up again. Some hard work there eh Derek?

Also I suppose avatars were cleared?
Title: Re: Recent Downtime
Post by: Aaron on January 19, 2008, 02:08:18 AM
Wow Derek, you've certainly been busy. Great job on bringing us back this fast, considering circumstances. Thanks for all the effort! :)
Title: Re: Recent Downtime
Post by: JayBachatero on January 19, 2008, 02:09:16 AM
Just as a side note.  Due to the file corruption some attachments were lost.  Including mods and themes.  Mod authors please look over your mods and check to see if any files are missing.

If you experience any errors or anything just let us know and we will look into it.  We are still working on a few things like downloads.  The page should be back up later on tonight. :)

BTW THANKS DEREK.  YOU THE MAN :)
Title: Re: Recent Downtime
Post by: nick09 on January 19, 2008, 02:10:40 AM
well I'm happy to see the site is back.
Title: Re: Recent Downtime
Post by: EdwinK on January 19, 2008, 02:21:38 AM
Glad to see the site up again ;)
Title: Re: Recent Downtime
Post by: LightningMk6 on January 19, 2008, 02:29:20 AM
Well done on getting the site up and running again.
Title: Re: Recent Downtime
Post by: I B D on January 19, 2008, 02:29:34 AM
As it was mentioned the Theme Site may be missing files from your themes, we have just checked themes and all that is missing is thumbnail images, if you have a theme on the Theme Site can we please ask that you update your theme with a new image where possible.  All archive files are still there and are downloading fine.
Title: Re: Recent Downtime
Post by: The Tornado on January 19, 2008, 02:31:37 AM
well
i was trying to open the site all the day
thanks GOD it's here again ....

but i saw that u v updated the forums to smf 2.0 Beta 2 !!
when can it be available for members to update their forums to this version ?
Title: Re: Recent Downtime
Post by: TheWrath! on January 19, 2008, 02:32:21 AM
hoo-ah for smf. good jobs fixing the problem guys!
Title: Re: Recent Downtime
Post by: Zetan on January 19, 2008, 02:32:43 AM
Good job on a hard mission.. just a heads up,
The Search seems to be broken.
Title: Re: Recent Downtime
Post by: shadow82x on January 19, 2008, 02:34:56 AM
Quote from: The Tornado on January 19, 2008, 02:31:37 AM

but i saw that u v updated the forums to smf 2.0 Beta 2 !!
when can it be available for members to update their forums to this version ?

There is no set time. Its ready when its ready. Currently a testing beta is aviable to charter members. (http://www.simplemachines.org/charter/)
Title: Re: Recent Downtime
Post by: I B D on January 19, 2008, 02:35:00 AM
Quote from: The Tornado on January 19, 2008, 02:31:37 AM
when can it be available for members to update their forums to this version ?

im afraid you will have to wait for a newer version of smf2 before its being released in public, also until its released as the stable it should not be used on a live environment
Title: Re: Recent Downtime
Post by: gffb on January 19, 2008, 02:35:30 AM
Nice to see you back bt when searching I get the following message.

The search API could not be found! Please contact the admin to check they have uploaded the correct files.
Title: Re: Recent Downtime
Post by: Gary on January 19, 2008, 02:36:05 AM
There will be at least one more charter beta before a public release.
Title: Re: Recent Downtime
Post by: nick09 on January 19, 2008, 02:36:47 AM
Quotewhen can it be available for members to update their forums to this version ?

months,years possible.

but the current 2.0 is not recommended to try on major boards only test ones.

besides you have to be a charter members to even try the beta.
Title: Re: Recent Downtime
Post by: CiOooo on January 19, 2008, 02:38:53 AM
how a loooong night for u and a long time for us ;)

nice to see you back ;)
Title: Re: Recent Downtime
Post by: Cheryl619 on January 18, 2008, 06:36:16 PM
We Missed You Guys :)
Title: Re: Recent Downtime
Post by: Eugeniu on January 18, 2008, 06:46:12 PM
Ah, that sounds terrible :(. Not to mention buying a new server out of nowhere probably wasn't fun either...

Glad to see that SimpleMachines is back online :).
Title: Re: Recent Downtime
Post by: Dude111 on January 18, 2008, 06:47:13 PM
Glad the site is back :)

SMF is awesome!!
Title: Re: Recent Downtime
Post by: Gary on January 18, 2008, 06:49:01 PM
Quote from: nick09 on January 19, 2008, 02:36:47 AM
months,years possible.

I wouldnt go as far as years.
Title: Re: Recent Downtime
Post by: G6™ on January 18, 2008, 06:52:14 PM
Incredible people, your asking for releases when staff and tech admins been struggling hard to get this site back up for us.
It's really a shame and dissrespectfull to even start to ask in this topic where they try and give us info about what happend.

Thank you all people involved in the hard work done, I hope you didnt suffer to much so you can have some relaxed time ahead.
Thank you SMF team, you really deserv a big basket of cheese and wine  :)
Title: Re: Recent Downtime
Post by: RoarinRow on January 18, 2008, 06:53:12 PM
Glad you'all are back online!   8)
Title: Re: Recent Downtime
Post by: nick09 on January 18, 2008, 06:55:24 PM
Quote from: Gazmanafc on January 18, 2008, 06:49:01 PM
Quote from: nick09 on January 19, 2008, 02:36:47 AM
months,years possible.

I wouldnt go as far as years.

* enables quick reply

well its possible. :P
Title: Re: Recent Downtime
Post by: CountryLady on January 18, 2008, 06:56:40 PM
Thanks for giving us a quick explanation and for all the work you do.
At times like this, the value of a TEAM is obvious. Derek, Thanks for your exceptional dedication to SMF and we users.

I just LOVE SMF~!

Title: Re: Recent Downtime
Post by: pagnor on January 18, 2008, 07:03:27 PM
Nice to see you back up guys, wouldnt say i was skiving but now your up i dont have an excuse not to carry on lmao
Title: Re: Recent Downtime
Post by: dschwab9 on January 18, 2008, 07:05:32 PM
Quote from: Eugeniu on January 18, 2008, 06:46:12 PM
Ah, that sounds terrible :(. Not to mention buying a new server out of nowhere probably wasn't fun either...

Glad to see that SimpleMachines is back online :).

I actually already had another server that was intended to be an additional cluster node.
Title: Re: Recent Downtime
Post by: Ben_S on January 18, 2008, 07:06:30 PM
Sounds like a nightmare day, good job on getting it back up.
Title: Re: Recent Downtime
Post by: Smoky "Rider" Blue on January 18, 2008, 07:24:37 PM
Glad to see everything going again.. I just checked my Kane Krush theme, and it has lost it's files and thumbnail.. will try to reupload them. I think I saved them to disk somewhere.. cross your fingers  ;)
Title: Re: Recent Downtime
Post by: ph4ever on January 18, 2008, 07:31:11 PM
Thanks everyone for all your hard work!!
Title: Re: Recent Downtime
Post by: MiniK on January 18, 2008, 07:41:24 PM
I watched the downtime ever since it was the "Yay it works!" Apache message. I feel so emotional. *sniffles*
Title: Re: Recent Downtime
Post by: dschwab9 on January 18, 2008, 07:43:58 PM
The "Yay it works" message was when I finally got up off the hard floor in the datacenter and went home to sit on the couch :D
Title: Re: Recent Downtime
Post by: metallica48423 on January 18, 2008, 07:48:37 PM
lol and i bet you wished you drank and/or had a few beers at that point :P

nah, j/k
Title: Re: Recent Downtime
Post by: Smith6612 on January 18, 2008, 07:57:05 PM
I figured something was up as I had never seen the page's generation time on this site go above 3 seconds at the max. Yesterday, they were around sometimes a minute just to generate, well, there goes one fried server. Great to see SMF back.

Also, the only reason I'd ever really leave any forum I go to is because we canceled our DSL connection here, but that won't be anywhere near soon as Cable in my area is always down and laggy now. I guess paying for the 7Meg/768k DSL for $40 paid off :P
Title: Re: Recent Downtime
Post by: I B D on January 18, 2008, 08:02:55 PM
see I was enjoying not having to worry about what the next theme related question was gonna be ;)

only jesting i was pestering derek over a messenger :P

oddly ims till waiting fro replys
Title: Re: Recent Downtime
Post by: YCW_jkt on January 18, 2008, 08:09:25 PM
Thanks for the update... and all the hard work...

I'm assuming that my inability to download mods is related to this downtime...?

Thanks to all!
Title: Re: Recent Downtime
Post by: SleePy on January 18, 2008, 08:21:21 PM
Quote from: YCW_jkt on January 18, 2008, 08:09:25 PM
Thanks for the update... and all the hard work...

I'm assuming that my inability to download mods is related to this downtime...?

Thanks to all!

That is most likely due to the files being missing, contact the mod author and let them know to reupload their mods.

Edit, actually a better idea may be to wait a couple of days. Give the author time to reupload their mods.
Title: Re: Recent Downtime
Post by: Elysia on January 18, 2008, 08:23:23 PM
Good to see you back up and running again, and thanks for being so fast. :) 

BTW the board search here isn't working for me either just in case someone needs to know.
Title: Re: Recent Downtime
Post by: winrules on January 18, 2008, 08:39:11 PM
We are aware there is a problem with downloading many mods and themes. It is something we are working on. Please do not PM mod authors yet.
Title: Re: Recent Downtime
Post by: melbinnc on January 18, 2008, 09:13:18 PM
thanks for all the hard work!
Title: Re: Recent Downtime
Post by: SaltedWeb on January 18, 2008, 09:56:41 PM
Great Job........ we are very lucky.
Title: Re: Recent Downtime
Post by: ApplianceJunk on January 18, 2008, 10:00:35 PM
Good job!!

Thank you,
Title: Re: Recent Downtime
Post by: Fiery on January 18, 2008, 10:18:15 PM
Great job getting it up and running again!

Title: Re: Recent Downtime
Post by: lovearat on January 18, 2008, 11:23:55 PM
You did an awesome job getting the forum back online!  I know was an awful lot of work. And I want you to know how much it is appreciated.  :)
Title: Re: Recent Downtime
Post by: christicehurst on January 19, 2008, 02:24:59 AM
Yay, the forum is back!
Title: Re: Recent Downtime
Post by: karlbenson on January 19, 2008, 02:35:34 AM
FYI to all those who have contacted me.
I have reuploaded all my mods/screenshots.
http://custom.simplemachines.org/mods/index.php?action=profile;u=63186
Title: Re: Recent Downtime
Post by: Sarge on January 19, 2008, 03:02:49 AM
Amazing job, Derek! And done within 24 hours...! It tells us something about both your technical abilities and dedication to SMF. In other words: simply great! ;)

Title: Re: Recent Downtime
Post by: uberjon on January 19, 2008, 03:28:48 AM
Quote from: dschwab9 on January 18, 2008, 07:05:32 PM
Quote from: Eugeniu on January 18, 2008, 06:46:12 PM
Ah, that sounds terrible :(. Not to mention buying a new server out of nowhere probably wasn't fun either...

Glad to see that SimpleMachines is back online :).

I actually already had another server that was intended to be an additional cluster node.

now the task to test the server that failed to figure out if anything is damaged or just a full crash + corruption ;)


but i must admit. i literally "freaked" when that messed up code stuff came up on the index...
Title: Re: Recent Downtime
Post by: BPLive on January 19, 2008, 03:36:10 AM
so is this a bug for 2.0 beta  >:(

ok trying to  make myself laugh...  had a bad day.

great job getting back.  I was told my server shares a place in one of our racks in Atlanta, hope my machine didn't get wet.  ;)
Title: Re: Recent Downtime
Post by: Sadil on January 19, 2008, 03:47:07 AM
Thank you for your hard work on getting the server back up. :) I think I nearly crapped my pants when I saw the "Yay it works!" message - thought for sure things had been hacked and ran like an idiot to my forum to make backups, haha. :P (Never know!)

In any case, I hope things will get back to normal soon. Thanks!
Title: Re: Recent Downtime
Post by: Bad-Dj on January 19, 2008, 04:06:29 AM
Good job getting it back you deserve a drink ;)
Title: Re: Recent Downtime
Post by: calamine on January 19, 2008, 05:40:11 AM
good to see smf back on track :)

but many of images for preview of the themes r missing :(
Title: Re: Recent Downtime
Post by: JOHNFS on January 19, 2008, 06:34:45 AM
Hi,
I can't seem to be able to download language packs for 1.1.4. the download page for them issues warnings that might indicate the files don't exist.

Could someone maybe confirm this?

Thanks!
Title: Re: Recent Downtime
Post by: Eddy Matthews on January 19, 2008, 06:51:19 AM
Doing a search results in:

An error has occured:

Unable to access the search daemon


Regards
Eddy
Title: Re: Recent Downtime
Post by: padexx on January 19, 2008, 06:58:23 AM
Quote from: winrules on January 18, 2008, 08:39:11 PM
We are aware there is a problem with downloading many mods and themes. It is something we are working on. Please do not PM mod authors yet.

Some All of my themes are not downloadable. You said you are working on this. Do I have to reupload all the files by myself?
Title: Re: Recent Downtime
Post by: H on January 19, 2008, 07:23:54 AM
Thanks to all involved in bringing us back online :)
Title: Re: Recent Downtime
Post by: i2Paq on January 19, 2008, 08:49:50 AM
Great job guys!

Just one thing: going into Theme stats gives:
QuoteWarning: require_once(/home/simple/public_html/custom/themes/Sources/Shared/Stats.php) [function.require-once]: failed to open stream: No such file or directory in /home/simple/public_html/custom/themes/index.php on line 164

Fatal error: require_once() [function.require]: Failed opening required '/home/simple/public_html/custom/themes/Sources/Shared/Stats.php' (include_path='.:/usr/local/lib/php') in /home/simple/public_html/custom/themes/index.php on line 164
Title: Re: Recent Downtime
Post by: Looking on January 19, 2008, 09:04:58 AM
Cannot access any attachments or mods - I get either 404 or blank page
Title: Re: Recent Downtime
Post by: Grudge on January 19, 2008, 09:21:20 AM
We've lost a lot of attachments and we're not sure whether we're able to restore them at the moment, we're looking into it.
Title: Re: Recent Downtime
Post by: Looking on January 19, 2008, 09:24:34 AM
Wow, sad.
Title: Re: Recent Downtime
Post by: Harro on January 19, 2008, 09:27:45 AM
I had no idea what was going on, but glad to see everything is working now!
Thanks a lot for the updates and all the hard work that was done to get everything back up and running again!!
Title: Re: Recent Downtime
Post by: nick09 on January 19, 2008, 09:39:06 AM
Quote from: karlbenson on January 19, 2008, 02:35:34 AM
FYI to all those who have contacted me.
I have reuploaded all my mods/screenshots.
http://custom.simplemachines.org/mods/index.php?action=profile;u=63186

sweet now to get that lightbox mod.
Title: Re: Recent Downtime
Post by: Gary on January 19, 2008, 09:58:02 AM
Please note, we are going through figuring out which customisations have missing files and authors will be contacted to get them back up at some point. ;)
Title: Re: Recent Downtime
Post by: nick09 on January 19, 2008, 09:59:14 AM
Quote from: Gazmanafc on January 19, 2008, 09:58:02 AM
Please note, we are going through figuring out which customisations have missing files and authors will eventually be contacted to get them back up. ;)

i know mine.

i just need to reupload.
Title: Re: Recent Downtime
Post by: Gary on January 19, 2008, 10:18:42 AM
If you wanna go through your own customisations and reupload the files, that'll save us alot of hassle. :P
Title: Re: Recent Downtime
Post by: LiroyvH on January 19, 2008, 10:20:43 AM
Glad to see it's back :)

Good work Derek!
Title: Re: Recent Downtime
Post by: nick09 on January 19, 2008, 10:29:56 AM
Quote from: Gazmanafc on January 19, 2008, 10:18:42 AM
If you wanna go through your own customisations and reupload the files, that'll save us alot of hassle. :P

yeah i only needed to fix a mod's download.

so you can ignore my mods.
Title: Re: Recent Downtime
Post by: HecKel on January 19, 2008, 10:57:12 AM
I have only one mod submitted :P

I have to upgrade that mod after the exams, this can be an excuse to do it faster :P I hope you did lost him :P

Great work with the community :) You installed Beta 2 now or it was before?
Title: Re: Recent Downtime
Post by: Gary on January 19, 2008, 11:02:06 AM
It was Beta 2 just before the site went down.
Title: Re: Recent Downtime
Post by: winrules on January 19, 2008, 11:02:17 AM
Quote from: panic on January 19, 2008, 06:58:23 AM
Quote from: winrules on January 18, 2008, 08:39:11 PM
We are aware there is a problem with downloading many mods and themes. It is something we are working on. Please do not PM mod authors yet.

Some All of my themes are not downloadable. You said you are working on this. Do I have to reupload all the files by myself?
Unless we are able to restore the attachments (which is highly unlikely) you will have to reupload your mod/theme files. At this point we are working on figuring out what attachments are missing so we can send authors messages.

However if you are unable to download a mod please do not PM or email the mod author. Doing this will just get authors bogged down with PMs.
Title: Re: Recent Downtime
Post by: TheWrath! on January 19, 2008, 11:05:32 AM
Quote from: winrules on January 19, 2008, 11:02:17 AM
Quote from: panic on January 19, 2008, 06:58:23 AM
Quote from: winrules on January 18, 2008, 08:39:11 PM
We are aware there is a problem with downloading many mods and themes. It is something we are working on. Please do not PM mod authors yet.


Some All of my themes are not downloadable. You said you are working on this. Do I have to reupload all the files by myself?
Unless we are able to restore the attachments (which is highly unlikely) you will have to reupload your mod/theme files. At this point we are working on figuring out what attachments are missing so we can send authors messages.

However if you are unable to download a mod please do not PM or email the mod author. Doing this will just get authors bogged down with PMs.


should we post in that particular mods thread to let them know of that problem? so we can help you if you have not gotten to that mod that there is a problem?
Title: Re: Recent Downtime
Post by: winrules on January 19, 2008, 11:17:30 AM
Quote from: TheWrath! on January 19, 2008, 11:05:32 AM
should we post in that particular mods thread to let them know of that problem? so we can help you if you have not gotten to that mod that there is a problem?
No. We'll take care of notifying mod authors.

For an update on the theme site please see here: http://www.simplemachines.org/community/index.php?topic=217551.0
Title: Re: Recent Downtime
Post by: Sun11 on January 19, 2008, 11:58:17 AM
ok, thnx, i will see when its back working
Title: Re: Recent Downtime
Post by: Mysteerie on January 19, 2008, 03:32:14 PM
This might have been mentioned already (I'm not going to read every post), but the search function doesn't work anymore. I get the following error:

An Error Has Occurred!
Unable to access the search daemon
Title: Re: Recent Downtime
Post by: TheWrath! on January 19, 2008, 03:36:04 PM
(http://img245.imageshack.us/img245/9857/smfsearchsl2.th.jpg) (http://img245.imageshack.us/my.php?image=smfsearchsl2.jpg)


that is what i get to. i see that, Mysteerie. I wanted to see that for myself and here is evidence of that problem.

thank you to the the "SMF Teamies" for working on fixing this problem.
Title: Re: Recent Downtime
Post by: I B D on January 19, 2008, 03:49:03 PM
yes it is known and the site team are working on it
Title: Re: Recent Downtime
Post by: TheWrath! on January 19, 2008, 04:39:25 PM
all right.

thank you runicwarrior
Title: Re: Recent Downtime
Post by: Smith6612 on January 19, 2008, 04:51:03 PM
You know what I find a little amusing/weird? This place fried their old server 2 days ago, their server is back up, and now, it looks like x10 Hosting fried their database hard drive on their Absolut server. Wierd...
Title: Re: Recent Downtime
Post by: dschwab9 on January 19, 2008, 06:01:28 PM
I'm still working on the attachments.   Things aren't looking good at the moment though.
Title: Re: Recent Downtime
Post by: rsw686 on January 19, 2008, 06:26:29 PM
dschwab9 I hope you don't mind if I ask a couple of questions about the failure.

What was the motherboard brand / model that had the thermal management failure? If this was an Intel chip I thought they had built into the core throttle control to prevent overheating.

Was there a reason to use the fan speed thermal management from the beginning? If you are in a datacenter noise shouldn't be much of an issue. I personally would trade cooler operating temps and longer component life for some noise.
Title: Re: Recent Downtime
Post by: Smith6612 on January 19, 2008, 06:35:06 PM
Quote from: rsw686 on January 19, 2008, 06:26:29 PM
dschwab9 I hope you don't mind if I ask a couple of questions about the failure.

What was the motherboard brand / model that had the thermal management failure? If this was an Intel chip I thought they had built into the core throttle control to prevent overheating.

Was there a reason to use the fan speed thermal management from the beginning? If you are in a datacenter noise shouldn't be much of an issue. I personally would trade cooler operating temps and longer component life for some noise.

As the same with my home computer (my gaming one). I'd rather have my computer get noisy when I fire up games rather than have the fans sit at low speed and make my Graphics card and CPU overheat.
Title: Re: Recent Downtime
Post by: Chriss Cohn on January 19, 2008, 09:14:36 PM
Hi, good its up agian.... but following error:
When i will try to use the search then, "            Unable to access the search daemon" appears...

Regards, Christian
Title: Re: Recent Downtime
Post by: Gary on January 19, 2008, 09:21:45 PM
We know about that.
Title: Re: Recent Downtime
Post by: dschwab9 on January 20, 2008, 01:34:07 AM
Quote from: rsw686 on January 19, 2008, 06:26:29 PM
dschwab9 I hope you don't mind if I ask a couple of questions about the failure.

What was the motherboard brand / model that had the thermal management failure? If this was an Intel chip I thought they had built into the core throttle control to prevent overheating.

It's an Asus board with an Athlon 64 X2 processor

Quote from: rsw686 on January 19, 2008, 06:26:29 PMWas there a reason to use the fan speed thermal management from the beginning? If you are in a datacenter noise shouldn't be much of an issue. I personally would trade cooler operating temps and longer component life for some noise.

Apparently it was on by default.   it's now off on all the servers
Title: Re: Recent Downtime
Post by: dschwab9 on January 20, 2008, 01:39:31 AM
Also, fyi, I'm making some progress on recovering recent attachments from the corrup hard drive
Title: Re: Recent Downtime
Post by: Gary on January 20, 2008, 02:58:05 AM
Awesome! :D
Title: Re: Recent Downtime
Post by: dschwab9 on January 20, 2008, 03:29:04 AM
all attachments (or at least 99% of them) have been recovered and are being uploaded now.   Should all be there in an hour or two.
Title: Re: Recent Downtime
Post by: dschwab9 on January 20, 2008, 04:41:09 AM
attachments and avatars have been restored
Title: Re: Recent Downtime
Post by: 4LP3RUZ1 on January 20, 2008, 05:15:27 AM
you rock!

asus mobo and x2 cpu... thats the same as my comp! :)

Q-fan was the problem?
Title: Re: Recent Downtime
Post by: dschwab9 on January 20, 2008, 05:23:17 AM
Quote from: alperuzi on January 20, 2008, 05:15:27 AM
you rock!

asus mobo and x2 cpu... thats the same as my comp! :)

Q-fan was the problem?

Whatever they call it.   That thing that adjusts the speed of the fan.   I suspect the temperature sensor is in such a place where the reading isn't accurate.   There was another box that was running really hot with low fan speed also.
Title: Re: Recent Downtime
Post by: Aaron on January 20, 2008, 06:10:03 AM
Quote from: dschwab9 on January 20, 2008, 03:29:04 AM
all attachments (or at least 99% of them) have been recovered and are being uploaded now.   Should all be there in an hour or two.

You're the man, Derek! Great work! :D
Title: Re: Recent Downtime
Post by: Alan S on January 20, 2008, 08:10:45 AM
I am not sure if this is related to the downtime but i have found a number of corrupt mod packages on the mod site , Message Preview On Hover and Sarcasmics Smiley sets to name a few , I tried to opern them in WinRar and i got a unexpected end of archive error.
Title: Re: Recent Downtime
Post by: Peter De Decker on January 20, 2008, 08:16:42 AM
glad to see the site back up!
Title: Re: Recent Downtime
Post by: TheWrath! on January 20, 2008, 09:10:51 AM
Quote from: dschwab9 on January 20, 2008, 04:41:09 AM
attachments and avatars have been restored

good job.
Title: Re: Recent Downtime
Post by: rsw686 on January 20, 2008, 09:36:18 AM
Quote from: dschwab9 on January 20, 2008, 01:34:07 AMIt's an Asus board with an Athlon 64 X2 processor

Apparently it was on by default.   it's now off on all the servers

Quote from: dschwab9 on January 20, 2008, 05:23:17 AM
Whatever they call it.   That thing that adjusts the speed of the fan.   I suspect the temperature sensor is in such a place where the reading isn't accurate.   There was another box that was running really hot with low fan speed also.

First off don't take this the wrong way. I just want to point a few things out so this doesn't happen to others.

What a bad combination of a lot of things. Asus boards are horribly unreliable. I have 3 for 3 where something has failed. An AMD chip with no built in thermal protection. And user error of not checking all the bios settings and stress testing with CPU Burn In or the like.

Well on the bright side least you were able to recover most of the data. This is one of those experiences that you never want to have to do again.

Also if your not using ECC memory it would be wise to do so. Memory prices are soo cheap that you can pickup Kingston 2x2GB of DDR2 ECC memory for $110, same as the non EEC memory.
Title: Re: Recent Downtime
Post by: Dannii on January 20, 2008, 09:52:02 AM
Thanks for your dedication Derek. :)

Btw, now is the time to finally implement that AJAX CPU temperature feature into the admin CP yes?
Title: Re: Recent Downtime
Post by: i2Paq on January 20, 2008, 01:35:35 PM
Quote from: dschwab9 on January 20, 2008, 03:29:04 AM
all attachments (or at least 99% of them) have been recovered and are being uploaded now.   Should all be there in an hour or two.

I've downloaded a lot of Mods for the 1.1.4 smf release the last couple of day's, so if any of them is missing do let me know and I will put them up for dowload/upload.
I've also downloaded a few themes for 1.1.4, same case, let me know.

Great job anyway!

Regards, Norman.
Title: Re: Recent Downtime
Post by: 4LP3RUZ1 on January 20, 2008, 02:34:13 PM
@rsw686

Asus boards are good quality, I've had 2 and never had a problem with them. I use the Q-fan feature and find it to be a great addition to my quiet PC. I have no problems with thermal monitoring and find the Athlon X2 to be a cool and reliable CPU. The thermal diode is actually inside the CPU, so the CPU does its own monitoring, but again, its all setup from the BIOS.
Title: Re: Recent Downtime
Post by: dschwab9 on January 20, 2008, 02:51:17 PM
Quote from: rsw686 on January 20, 2008, 09:36:18 AM
What a bad combination of a lot of things. Asus boards are horribly unreliable. I have 3 for 3 where something has failed. An AMD chip with no built in thermal protection. And user error of not checking all the bios settings and stress testing with CPU Burn In or the like.

I could post a list of 20 motherboards here and I wouldn't get a response from someone about each and every one of them being horrible.   FYI:  not sure if it was mentioned in my original post, but the power supply fan had also failed, which has nothing to do with ASUS or the quality of their boards.   I will say I have been using their stuff for years and haven't never been a problem.

Also, about using AMD being a bad idea - I can't go for that.   AMD has some of the top performing CPU's out there and is probably in use in 50% of the web servers out there.

Quote from: rsw686 on January 20, 2008, 09:36:18 AM
Also if your not using ECC memory it would be wise to do so. Memory prices are soo cheap that you can pickup Kingston 2x2GB of DDR2 ECC memory for $110, same as the non EEC memory.
You obviously don't know a whole lot about servers.   You can't just buy any stick of RAM and stick in any motherboard.  ECC RAM won't work in a board that doesn't support it.   Also, ECC RAM or not is totally unrelated to CPU temperature.
Title: Re: Recent Downtime
Post by: dschwab9 on January 20, 2008, 02:52:42 PM
Quote from: alperuzi on January 20, 2008, 02:34:13 PM
@rsw686

Asus boards are good quality, I've had 2 and never had a problem with them. I use the Q-fan feature and find it to be a great addition to my quiet PC. I have no problems with thermal monitoring and find the Athlon X2 to be a cool and reliable CPU. The thermal diode is actually inside the CPU, so the CPU does its own monitoring, but again, its all setup from the BIOS.

Isn't there another diode other then the CPU though?   The BIOS reports a CPU temp and a "System temp".   On this machine, the system temp was relatively low, so I suspect that diode was near the front of the case where the air intake it.
Title: Re: Recent Downtime
Post by: rsw686 on January 20, 2008, 02:56:27 PM
Quote from: alperuzi on January 20, 2008, 02:34:13 PM
@rsw686

Asus boards are good quality, I've had 2 and never had a problem with them. I use the Q-fan feature and find it to be a great addition to my quiet PC. I have no problems with thermal monitoring and find the Athlon X2 to be a cool and reliable CPU. The thermal diode is actually inside the CPU, so the CPU does its own monitoring, but again, its all setup from the BIOS.

That's fine. I'm just saying that out of the 4 Abit, 1 Intel, 1 Gigabyte, and 3 Asus boards I've bought the only ones that have failed are all of the Asus boards.

I know the diode is in the CPU. I was just stating that Intel chips starting with the P4 don't rely on the motherboard the CPU will shut down itself if it exceeds the thermal limit. AMD only relays this to the motherboard and lets it decide.

Quote from: dschwab9 on January 20, 2008, 02:51:17 PMYou obviously don't know a whole lot about servers.   You can't just buy any stick of RAM and stick in any motherboard.  ECC RAM won't work in a board that doesn't support it.   Also, ECC RAM or not is totally unrelated to CPU temperature.<br />

Actually I do. I build my own servers, have a rack in the basement, and host my own sites. The ASUS board for the AMD X2 chip takes DDR2 ECC or non ECC memory. 95% of the boards produced now support ECC memory. There is no reason not to use it if your data is critical. Look at the servers Dell builds, they all come with ECC memory.

ECC is different than buffered memory. Most don't support buffered memory, but they do support ECC memory.
Title: Re: Recent Downtime
Post by: 青山 素子 on January 20, 2008, 03:01:27 PM
Quote from: rsw686 on January 20, 2008, 09:36:18 AM
Asus boards are horribly unreliable. I have 3 for 3 where something has failed.

It depends on the model. ASUS is sourced by a lot of OEMs (as is MSI) and they are fairly decent. (Their lower-cost Asrock aren't as much.) It simply sounds like you've had a bad batch.


Quote from: rsw686 on January 20, 2008, 09:36:18 AM
An AMD chip with no built in thermal protection.

True of the older Athlon and Athlon XP models, but the 64 series does have a built-in thermal sensor and protection. This is probably why the chips weren't hot charcoal. (Max die temp for that series is 90c I believe.)


Quote from: rsw686 on January 20, 2008, 09:36:18 AM
And user error of not checking all the bios settings and stress testing with CPU Burn In or the like.

Usually, you should expect that the BIOS defaults are safe settings. If you have to check each and every one you are either way overboard on control or you need to change the board maker you are using.


Quote from: dschwab9 on January 20, 2008, 02:52:42 PMIsn't there another diode other then the CPU though?   The BIOS reports a CPU temp and a "System temp".   On this machine, the system temp was relatively low, so I suspect that diode was near the front of the case where the air intake it.

It depends on the board layout. There usually is a different sensor so you can get the general ambient inside the case. Usually, that sensor is by or inside the southbridge. Depending on the case design , that could be near the intake.



Quote from: rsw686 on January 20, 2008, 02:56:27 PM
Actually I do. I build my own servers, have a rack in the basement, and host my own sites. The ASUS board for the AMD X2 chip takes DDR2 ECC or non ECC memory. 95% of the boards produced now support ECC memory. There is no reason not to use it if your data is critical. Look at the servers Dell builds, they all come with ECC memory.

Which ASUS board and which chipset? You do realize there are multiple chipsets available for that processor, right?


BTW, I wouldn't take on Derek, he's a professional admin over a huge setup. He knows his stuff.
Title: Re: Recent Downtime
Post by: dschwab9 on January 20, 2008, 03:01:50 PM
Don't have the manual in front of me, so don't know if it supports it or not, but that's irrelevant to the problem we had.
Title: Re: Recent Downtime
Post by: Douglas on January 20, 2008, 03:02:58 PM
Quote from: dschwab9I could post a list of 20 motherboards here and I wouldn't get a response from someone about each and every one of them being horrible.

Also, about using AMD being a bad idea - I can't go for that.&nbsp;  AMD has some of the top performing CPU's out there and is probably in use in 50% of the web servers out there.
I have to chime in with what Derek posted and am in complete agreeance with both of the above quoted statements.

HW failures occur for a multitude of reasons.  Might want to ease off a bit unless you were there and know the whole story (even I don't know the whole story, nor do I want to).

Also, AMD is what powered the sites in my sig below.  These sites, even on a clustered environment choked to death even more so on Intel based machines.
Title: Re: Recent Downtime
Post by: dschwab9 on January 20, 2008, 03:06:30 PM
For anyone who wants to analyze the location of the temp sensors, here's a pic of the failed machine
Title: Re: Recent Downtime
Post by: CVAN on January 20, 2008, 03:27:08 PM
I'm still receiving an error message:

Connection Problems

Sorry, SMF was unable to connect to the database. This may be caused by the server being to busy. Please try again later.

Is anyone else still unable to access their forums?
Title: Re: Recent Downtime
Post by: Harro on January 20, 2008, 03:29:32 PM
That has nothing to do with the downtime this site had :)
You can always ask for support about that in the support boards.
Title: Re: Recent Downtime
Post by: IchBin™ on January 20, 2008, 03:33:42 PM
Haven't had a single problem here. Thanks for the hard work Derek.
Title: Re: Recent Downtime
Post by: Fiery on January 20, 2008, 07:58:15 PM
Quote from: dschwab9 on January 20, 2008, 03:06:30 PM
For anyone who wants to analyze the location of the temp sensors, here's a pic of the failed machine

Just take it all out ;)
Title: Re: Recent Downtime
Post by: GravuTrad on January 20, 2008, 09:07:12 PM
hardware downtime is always a hard thing to recover. Great job derek and thanks for all. ;)
Title: Re: Recent Downtime
Post by: 4LP3RUZ1 on January 21, 2008, 04:19:19 AM
Quote from: dschwab9 on January 20, 2008, 02:52:42 PM
Quote from: alperuzi on January 20, 2008, 02:34:13 PM
@rsw686

Asus boards are good quality, I've had 2 and never had a problem with them. I use the Q-fan feature and find it to be a great addition to my quiet PC. I have no problems with thermal monitoring and find the Athlon X2 to be a cool and reliable CPU. The thermal diode is actually inside the CPU, so the CPU does its own monitoring, but again, its all setup from the BIOS.

Isn't there another diode other then the CPU though?   The BIOS reports a CPU temp and a "System temp".   On this machine, the system temp was relatively low, so I suspect that diode was near the front of the case where the air intake it.
Yes, theres two on the motherboard and one inside the CPU I think. I'm not sure which diode the BIOS reads as CPU though, its own or the one in the cpu.
Title: Re: Recent Downtime
Post by: Trekkie101 on January 21, 2008, 07:35:31 PM
Regular backups :P
Title: Re: Recent Downtime
Post by: Dannymh on January 22, 2008, 01:41:58 AM
Not sure if anyone has noticed but it seems that the themes files are corrupted, I haven had any succesfful downloads or the ones I have , have had corrupted zip files
Title: Re: Recent Downtime
Post by: Smith6612 on January 22, 2008, 01:46:02 AM
That's the result of the site going down. The theme authors I'm sure will re-upload them shortly.
Title: Re: Recent Downtime
Post by: Dannymh on January 22, 2008, 02:06:45 AM
Awesome cheers mate
Title: Re: Recent Downtime
Post by: Gary on January 22, 2008, 07:27:49 AM
If you can be specific on which themes are actually corrupted, we can take care of it and contact the authors to re-upload the packages.

Though, many of them, if not all, have been restored now.
Title: Re: Recent Downtime
Post by: GravuTrad on January 22, 2008, 03:16:42 PM
Quote from: Dannymh on January 22, 2008, 01:41:58 AM
Not sure if anyone has noticed but it seems that the themes files are corrupted, I haven had any succesfful downloads or the ones I have , have had corrupted zip files

It' said in the few first posts of this subject.....

Other question for me:

Which is exactly the system use for smf site servers hosting?

And more the backups...
Title: Re: Recent Downtime
Post by: Dannymh on January 22, 2008, 05:50:12 PM
Quote from: Gazmanafc on January 22, 2008, 07:27:49 AM
If you can be specific on which themes are actually corrupted, we can take care of it and contact the authors to re-upload the packages.

Though, many of them, if not all, have been restored now.

The list is pretty long man. I have not had 1 successful theme download (I checked with other downloads from this site that all worked.

I also just retried one of them and didn't want to go back through them all. here is the one i re-tried
http://custom.simplemachines.org/themes/index.php?lemma=335

Looks like it may be best to just ask everyone to re-uplaod them

Dan
Title: Re: Recent Downtime
Post by: Gary on January 22, 2008, 06:13:46 PM
Alright thanks for the report, but do try to find as many as possible.
Title: Re: Recent Downtime
Post by: dschwab9 on January 22, 2008, 06:32:18 PM
Are they downloading an appearing as corrupted files or not downloading at all?
Title: Re: Recent Downtime
Post by: Gary on January 22, 2008, 06:35:57 PM
They download as corrupt files.
Title: Re: Recent Downtime
Post by: Dannymh on January 22, 2008, 07:42:45 PM
Quote from: dschwab9 on January 22, 2008, 06:32:18 PM
Are they downloading an appearing as corrupted files or not downloading at all?

They download but the archives are corrupted.

When you try to extract them the archives have bytes missing. Looks like the failure might have corrupted a whole batch of them.
I will list as many as I can find.

So they all download but are missing data.

So far have found 4

Title: Re: Recent Downtime
Post by: Gary on January 22, 2008, 08:31:11 PM
Yes, please do report any you find, to either myself or RunicWarrior. We will then try to get the theme packages back online ASAP.
Title: Re: Recent Downtime
Post by: Dannymh on January 23, 2008, 05:47:31 PM
I will download some more and test them when I get a chance, unfortunately with my recent site upgrade I have let my normal work falter a bit so need to catch up, once I am back on top of things I will test as many as humanly possible
Title: Re: Recent Downtime
Post by: GravuTrad on January 23, 2008, 06:53:20 PM
Quote from: GravuTrad on January 22, 2008, 03:16:42 PM
Quote from: Dannymh on January 22, 2008, 01:41:58 AM
Not sure if anyone has noticed but it seems that the themes files are corrupted, I haven had any succesfful downloads or the ones I have , have had corrupted zip files

It' said in the few first posts of this subject.....

Other question for me:

Which is exactly the system use for smf site servers hosting?

And more the backups...

No more precision about my question?
Title: Re: Recent Downtime
Post by: dschwab9 on January 23, 2008, 07:07:11 PM
We currently have a cluster of 3 machines and will be adding 3 more in the coming weeks.   Database and file servers are Intel Core 2 Quad and the web servers are AMD Athlon 64 X2's.
Title: Re: Recent Downtime
Post by: Chriss Cohn on January 26, 2008, 05:56:24 PM
Quote from: dschwab9 on January 23, 2008, 07:07:11 PM
We currently have a cluster of 3 machines and will be adding 3 more in the coming weeks.   Database and file servers are Intel Core 2 Quad and the web servers are AMD Athlon 64 X2's.
Well please don't get me wrong, but does that mean if i have a Forum with "just" 125.000+ Members, i really need 6?! Servers to power it?
That would be 1000$ and more per month for "just" 1/10 million users..... I always thought that SMF is consuming very less ressources......

Regards, Christian
Title: Re: Recent Downtime
Post by: Harro on January 26, 2008, 06:07:33 PM
Lots depends on the traffic you have.
Also have to keep in mind of all the extra stuff we have here. Including all the download, mods, themes.
So perhaps it's just 125.000 members, but I'm sure we have a LOT more visitors...
Title: Re: Recent Downtime
Post by: Compuart on January 26, 2008, 06:15:11 PM
Also there are thousands of admins around world opening their SMF admin panel and thereby requesting a check directed to simplemachines.org to see if there are any updates available.
Title: Re: Recent Downtime
Post by: Ol' Wombat on January 26, 2008, 06:23:31 PM
Quote from: dschwab9 on January 19, 2008, 02:02:07 AM
...
1.  The thermal management system on the motherboard wasn't properly throttling fan speed, possibly for quit a while.  ..

Fairly known issue with a number of motherboards - disable it and all will be well  :-*
Title: Re: Recent Downtime
Post by: dschwab9 on January 26, 2008, 06:24:25 PM
How many posts and members are irrelevant.  You could have 100,000 members and 10 million posts and operate on one server if those members aren't doing anything.   We're currently serving approximately 5 million requests to about 30,000 unique visitors per day.
Title: Re: Recent Downtime
Post by: Smith6612 on January 26, 2008, 06:34:06 PM
Hey, at least be happy that SMF is buying more servers and not overloading servers :)
Title: Re: Recent Downtime
Post by: GravuTrad on January 26, 2008, 07:00:16 PM
Quote from: dschwab9 on January 23, 2008, 07:07:11 PM
We currently have a cluster of 3 machines and will be adding 3 more in the coming weeks.   Database and file servers are Intel Core 2 Quad and the web servers are AMD Athlon 64 X2's.

Mounted how?
Title: Re: Recent Downtime
Post by: dschwab9 on January 26, 2008, 08:33:30 PM
If you mean physically mounted:  In a rack

If you're talking about file systems:  NFS
Title: Re: Recent Downtime
Post by: GravuTrad on January 26, 2008, 11:19:20 PM
thanks. backup servers too?
Title: Re: Recent Downtime
Post by: dschwab9 on January 27, 2008, 12:35:31 AM
Backups are currently rsync'ed offsite.   I'm working on implementing a dedicated fileserver that will have two RAID arrays, so there will be an onsite nightly backup and then a weekly offsite snapshot.   The current backup system is a little less than reliable and very short on disk space which is why we had a number of corrupt and missing files after the hardware problems.
Title: Re: Recent Downtime
Post by: VistaX on January 27, 2008, 04:52:22 AM
Nice job on getting it back online!
Title: Re: Recent Downtime
Post by: GravuTrad on January 27, 2008, 09:25:01 AM
Quote from: dschwab9 on January 27, 2008, 12:35:31 AM
Backups are currently rsync'ed offsite.   I'm working on implementing a dedicated fileserver that will have two RAID arrays, so there will be an onsite nightly backup and then a weekly offsite snapshot.   The current backup system is a little less than reliable and very short on disk space which is why we had a number of corrupt and missing files after the hardware problems.

Thanks for this precision (that's exactly the sense and the why of my question), and thanks for this new backup system which is for me really necessary.
Title: Re: Recent Downtime
Post by: dschwab9 on January 27, 2008, 02:03:00 PM
The biggest problem with backups is you usually don't realize they have problems until you need them ::)
Title: Re: Recent Downtime
Post by: GravuTrad on January 27, 2008, 03:43:58 PM
 :D ;) right. ;)
Title: Re: Recent Downtime
Post by: Ben_S on January 27, 2008, 04:00:22 PM
Quote from: Chriss Cohn link=topic=217388.msg1399382#msg1399382
I always thought that SMF is consuming very less ressources......

It is, my forum runs happily on the hardware in my sig does about 380,000 SMF powered page views / day with a smile on it's face.
Title: Re: Recent Downtime
Post by: dschwab9 on January 27, 2008, 04:45:05 PM
Ben, not sure how you do it, but currently we have one quad core database server with 4 gigs of RAM and one dual core web server with 2 gigs, and the database server sits at 40-40% utilization and the web sever is maxed out a lot of the time, and that's with caching at level 3.   We did just get things running with memcached recently, which I'm real happy about.
Title: Re: Recent Downtime
Post by: Ben_S on January 28, 2008, 05:59:59 AM
I admit it's starting to get a little bit hairy during and after a game, I'll probably look into using something like Mathopd, lighty, tux etc for static content, apache serving 250+ requests causes a few issues. Tux is looking like the best option, although the possibility of security issues with it running in kernel space is a bit scary but hey, if everything is kept upto date there shouldn't be any issues but I think I'll give it a miss.

Memcached looks interesting, although from what I can tell would only be of real benefit if it's being used in a multi server setup so wouldn't really be much use for me?
Title: Re: Recent Downtime
Post by: 青山 素子 on January 28, 2008, 02:16:23 PM
Memcached will benefit single server configurations as well. It's a cache engine, but can run on distributed setups.
Title: Re: Recent Downtime
Post by: Ben_S on January 28, 2008, 02:19:55 PM
Whats the benefit over say, APC?

Just switched for lighty for most of my static content (still need to go through the main site files to change that) and load has about halfed, which is nice.
Title: Re: Recent Downtime
Post by: 青山 素子 on January 28, 2008, 02:22:01 PM
Memcached seems more stable on 64-bit systems.

We're actually using it at work for a huge template-driven e-commerce solution we built, and it works wonderfully well.
Title: Re: Recent Downtime
Post by: Tony Reid on January 28, 2008, 03:00:41 PM
Quote from: Ben_S on January 28, 2008, 02:19:55 PM
Just switched for lighty for most of my static content (still need to go through the main site files to change that) and load has about halfed, which is nice.

Any thing special to watch out for/be aware of when setting this up Ben?
Title: Re: Recent Downtime
Post by: Ben_S on January 28, 2008, 06:55:45 PM
Thus far everything seems to be working fine as static.redand.... No doubt as soon as I go away for a period of time, lighty will start segfaulting and my images will disappear.
Title: Re: Recent Downtime
Post by: kasxh on January 30, 2008, 05:21:33 AM
Someone has the theme dark dirge? Download this damaged

thanks
Title: Re: Recent Downtime
Post by: Gary on January 30, 2008, 07:30:29 AM
A couple of files seemed to have errors in them. Contact the author.
Title: Re: Recent Downtime
Post by: pothead on February 08, 2008, 01:14:00 PM
recently....
i have noticed many of my memebers having a delay to get into my forum...
even i as admin have to wait to get in and this is increasing...

i didnt not read everything her ein details..but i jus wanted to know if anyone else is experiencing this..
at the moment i can not get in..cleaning all applications...rebooting...cache clearing nothing works...

any kind of infor would be appreciated....


www.potheadpoet.com
Title: Re: Recent Downtime
Post by: Gary on February 08, 2008, 01:15:07 PM
Post a support topic in the support boards as that is relating to your site. This topic is an announcement about the server crash we had here on this site.
Title: Re: Recent Downtime
Post by: sledmods on February 09, 2008, 10:21:48 AM
Any reason why my site isn't back up www.sledmods.com

seems my index file won't load
Title: Re: Recent Downtime
Post by: rsw686 on February 09, 2008, 10:29:48 AM
Quote from: sledmods on February 09, 2008, 10:21:48 AM
Any reason why my site isn't back up www.sledmods.com

seems my index file won't load

I still don't get how this gets asked so many times. Simple Machines does not host any of your sites. They just provide the software to do so. You need to contact your host or post in the support board.
Title: Re: Recent Downtime
Post by: sledmods on February 09, 2008, 10:33:33 AM
Thanks for the help.

I was on the phone with my host.  Can't load the index file but can load others.

Since I am not that technical i thought the forum software that i use could give me a hand.

Sorry if i offended you.
Title: Re: Recent Downtime
Post by: Sarge on February 09, 2008, 10:35:26 AM
Quote from: sledmods on February 09, 2008, 10:21:48 AM
Any reason why my site isn't back up www.sledmods.com

seems my index file won't load

This is NOT a support board!

Quote from: Gazmanafc on February 08, 2008, 01:15:07 PM
Post a support topic in the support boards as that is relating to your site. This topic is an announcement about the server crash we had here on this site.
Title: Re: Recent Downtime
Post by: sledmods on February 09, 2008, 10:41:20 AM
Got it.

Thought my issue was related to this topic

will post it over there