Simple Machines Community Forum

Simple Machines => News and Updates => Topic started by: LiroyvH on October 24, 2011, 04:38:09 PM

Title: Server maintenance
Post by: LiroyvH on October 24, 2011, 04:38:09 PM
Dear users,


We have planned required maintenance to our server infrastructure.
This maintenance causes us to have downtime several times, both short and a bit longer in time.
(Exact timeframe cannot be predicted)
We will of course do our utmost best to keep the interruptions as short as possible.

The maintenance is scheduled to start at:
Sunday, October 30 2011.
The first maintenance session will most likely be the one causing the longest interruption.
Other dates will be released when they are planned.


For more information, please see all details about this maintenance in the Team Blog article:
http://www.simplemachines.org/community/index.php?topic=457092


To be able to receive quick updates regarding the progress and status, please:
Follow us on Twitter (http://www.twitter.com/simplemachines)
Add us on Facebook (http://www.facebook.com/smforum?ref=ts)

If you require urgent support when our community is down, please visit our backup forum. Make sure to bookmark it!!
http://www.simplemachinesforum.org/
Please do not open support topics there unless it is really urgent and cant wait a few hours.


Thank you :)


Regards,
- Servers Team
Simple Machines
Title: Re: Server maintenance
Post by: Looking on October 24, 2011, 04:47:17 PM
Out of curiosity what kind of hosting setup runs this community?
Title: Re: Server maintenance
Post by: LiroyvH on October 24, 2011, 04:49:23 PM
I will not go in to full details, but we have several servers that do different jobs such as virtual machine hosts, webservers, SQL servers, file servers, mail servers, etc.
All of them combined is what powers Simple Machines.

We own all hardware.
Title: Re: Server maintenance
Post by: Looking on October 24, 2011, 05:23:32 PM
Thanks for the reply, that must be quite a setup to maintain with updates and monitoring user activity here.

It also explains the below page stats staying on par:
QuotePage created in 0.077 seconds with 25 queries.
Page served by: 10.0.100.135 (10.0.100.135)
Title: Re: Server maintenance
Post by: SleePy on October 24, 2011, 05:34:39 PM
It isn't always easy, but both of our primary server admins are very experience professionals in this :)
Title: Re: Server maintenance
Post by: Robert. on October 25, 2011, 10:47:09 AM
Thanks for the news, glad to hear simplemachinesforum.org will still be available :) Good luck folks ;D
Title: Re: Server maintenance
Post by: LiroyvH on October 31, 2011, 02:18:46 PM
Quick note:
Welcome back!

The upgrades didnt went as smoothly as they should have gone, as you may have noticed ;)
We will post more details later.

As some of you may have noticed: the MOD site currently replies with "access denied".
We apologise for the inconvenience, but we have to make some changes before opening that up again. This should not take too long.

Hang in there! :)
Thank you for your patience!
Title: Re: Server maintenance
Post by: Antes on October 31, 2011, 02:21:48 PM
Thanks for info.

I can access the mod site now.
Title: Re: Server maintenance
Post by: LiroyvH on October 31, 2011, 02:22:45 PM
You mean customize?
You shouldnt be able to access the mods section :P
Title: Re: Server maintenance
Post by: Illori on October 31, 2011, 02:23:38 PM
Quote from: Antes on October 31, 2011, 02:21:48 PM
I can access the mod site now.

http://custom.simplemachines.org/mods/

no you cant
QuoteOops, what happened?
Sorry, but you're not allowed to access the file you requested.
Title: Re: Server maintenance
Post by: Robert. on October 31, 2011, 02:24:16 PM
Good news and thanks all for your hard work :)
Title: Re: Server maintenance
Post by: Antes on October 31, 2011, 02:38:43 PM
edit : now mine gone too thank you Illori :P
Title: Re: Server maintenance
Post by: SleePy on October 31, 2011, 02:45:55 PM
It should be fixed now.  But it may go off-line intermediately.
Title: Re: Server maintenance
Post by: Herman's Mixen on October 31, 2011, 02:57:43 PM
/me gives CoreISP and Derek a nice break after doing this maint everything is smooth again ;)
good work and keep it up :P
Title: Re: Server maintenance
Post by: gisfreak on October 31, 2011, 10:17:57 PM
good work, and congrats
Title: Re: Server maintenance
Post by: LiroyvH on October 31, 2011, 10:39:21 PM
Thank you :)

So this has been quite a nightmare for us system administrators.
We will post a more detailed report later, but first of all I would like to thank:

- dschwab9 (Derek) for spending a significant ammount of time both in and out of the datacenter, patching configurations, comming up with workarounds, upgrading hardware, etc. :)
- SleePy for spending a lot of time looking through configs and helping where he could :)
- WireSix Technicians, for bearing with us on letting them run to our file server multiple times :)
  They have been a great help to allow us to continue doing remote jobs
- I guess myself (sounds so wrong :P) for spending tons of time on transferring files, udating configs, figure out temporary workarounds for our hardware issues, making patches and modifications, installing software and mitigating DDoS attacks, all to get us in a running state again and working with Derek with the datacenter jobs that had to be done.
- Anyone else that may have indirectly helped us

And thank you all for your patience in this matter!! :)

Happy forum-ing!!


- Liroy
Title: Re: Server maintenance
Post by: mashby on October 31, 2011, 10:44:05 PM
Thanks to all who got everything back up! Running nicely at the moment. :)
Title: Re: Server maintenance
Post by: SleePy on October 31, 2011, 11:00:36 PM
Well maybe I am under estimating myself.  But both Derek and Liroy deserve a good thanks for the lack of sleep from this all :D.  I just annoyed them and kept them thinking ;)
Title: Re: Server maintenance
Post by: dschwab9 on November 01, 2011, 02:48:29 AM
Just a heads up - the mail server and backend utility server that processes the mail queue are still down, so no emails are going out from the site.  We're currently working on recovering those.

Due to a hardware malfunction with the RAID controller in our NFS server, it's taking forever to migrate everything.
Title: Re: Server maintenance
Post by: Oldiesmann on November 01, 2011, 02:58:17 AM
Thanks for the update Derek. Hopefully you'll have things back up and running like normal again soon.
Title: Re: Server maintenance
Post by: Antes on November 01, 2011, 04:45:52 AM
That's why I'm staying in SMF... You guys doing something because you want to.

I salute you with all my respect and thank you for your all hard work :)
Title: Re: Server maintenance
Post by: Dzonny on November 01, 2011, 06:12:13 AM
Thans Derek, Liroy and SleePy for the great job. :)
Title: Re: Server maintenance
Post by: Angelina Belle on November 01, 2011, 09:59:41 AM
Thank you liroy, derek, and SleePy.  We look forward to hearing when the RAID controller is replaced, and everything can get back to normal.
Title: Re: Server maintenance
Post by: Apllicmz on November 02, 2011, 05:42:53 PM
Thanks all
good work

Title: Re: Server maintenance
Post by: Deaks on November 02, 2011, 07:15:42 PM
wait mail servers are down?!?
Title: Re: Server maintenance
Post by: SleePy on November 02, 2011, 07:38:00 PM
Mail was down at the time of Dereks posting.  They are operational at this point.
Title: Re: Server maintenance
Post by: Deaks on November 02, 2011, 07:38:39 PM
ah ok :)
Title: Re: Server maintenance
Post by: Herman's Mixen on November 03, 2011, 01:02:02 AM
You know.. they work also behind the scene ;)
Title: Re: Server maintenance
Post by: dschwab9 on November 03, 2011, 02:21:38 AM
Everything is fully functional at this point.  We will be repairing and rebuilding the file server with the hardware issue and ultimately moving the data back to that box, which has more disks and IP capacity than the file server we are currently on.
Title: Re: Server maintenance
Post by: moguns on November 30, 2011, 10:00:21 PM
This is crazy.. you guys need to get the server stuff fixed. Making godaddy look good here
Title: Re: Server maintenance
Post by: live627 on December 09, 2011, 07:32:21 PM
Quote from: moguns on November 30, 2011, 10:00:21 PM
This is crazy.. you guys need to get the server stuff fixed. Making godaddy look good here
Yes, manually dealing with dedicated servers is crazy.
Title: Re: Server maintenance
Post by: LiroyvH on December 11, 2011, 05:21:49 PM
Lol, someone has to do it :P


Anyway, we are going offline again tomorrow to proceed with the upgrades!
We do not have a estimate time as it depends on various factors. We can also not exactly predict how long it will be offline for, but please expect it to take a couple of hours.

In the meanwhile, you may always visit our backup board for urgent issues :)

Thank you


- Servers Team
Title: Re: Server maintenance
Post by: LiroyvH on December 12, 2011, 09:30:51 PM
Due to planning issues we unfortunately have to postpone the maintenance.

We will hopefully proceed tomorrow instead of today.
Sorry for the inconvenience!

We will keep you posted :)
Title: Re: Server maintenance
Post by: 青山 素子 on December 13, 2011, 03:28:17 PM
Quote from: moguns on November 30, 2011, 10:00:21 PM
This is crazy.. you guys need to get the server stuff fixed. Making godaddy look good here

When you can get GoDaddy to deal with your 4+ dedicated systems (with multiple virtual instances on nearly all boxes) with special internal networking configuration and over 4mbit of constant traffic, then you can say that. As it is, their hosting solutions can barely keep a small shared website operating.

simplemachines.org isn't some small website, you know. It's well into where you can throw around the word "enterprise" and not look like a total moron.
Title: Re: Server maintenance
Post by: sshahnawaz on December 24, 2011, 08:01:08 AM
GoDaddy is a web host, right?
Title: Re: Server maintenance
Post by: sawz on December 24, 2011, 08:16:25 AM
Quote from: Mysterious_Stuff on December 24, 2011, 08:01:08 AM
GoDaddy is a web host, right?

they like to think so but i wouldn't consider them a serious host.
so don't use them to host your site, you'll regret it eventually.
Title: Re: Server maintenance
Post by: LiroyvH on December 27, 2011, 10:19:57 PM
Please be advised we are planning on moving forward this Thurdsay.
Please keep in mind this is not definite, though it is approximately 90% sure we will be moving forward.

Simple Machines will be completely unreachable during the upgrades.
Updates will be posted to twitter and to some extent to Facebook.

Thank you for your ongoing patience in the matter :)
We want this fixed as much as all of you do ;)
Title: Re: Server maintenance
Post by: LiroyvH on December 29, 2011, 10:11:27 PM
After some trouble at the beginning, we are back online now :)
Powered by more servers and a new configuration.

Please be advised we are !not! 100% finished yet.
There's one big task left, though you should only notice a few minutes downtime from that, most likely tomorrow.

Thank you for your patience! :)

Thank you to:
- dscwhab9
- SleePy
- myself I guess :p
- Some help from WireSix with a hardware issue that popped up
Title: Re: Server maintenance
Post by: busterone on December 29, 2011, 10:15:55 PM
Thanks for the hard work.   :)
Title: Re: Server maintenance
Post by: ApplianceJunk on December 29, 2011, 10:16:25 PM
Thanks dscwhab9, SleePy and CoreISP.
Title: Re: Server maintenance
Post by: Deaks on December 29, 2011, 10:36:29 PM
hey where my thanks for pestering you on MSN keeping you sane :P

Seriously though cheers guys we all love ya :)
Title: Re: Server maintenance
Post by: Robert. on December 30, 2011, 03:40:32 AM
Thanks for the update :)
Title: Re: Server maintenance
Post by: karlbenson on December 30, 2011, 03:31:51 PM
Indeed, good work people!
Title: Re: Server maintenance
Post by: Herman's Mixen on December 31, 2011, 09:25:35 AM
Thanks guys, you all did a great job keep on going :P
Title: Re: Server maintenance
Post by: dschwab9 on January 05, 2012, 05:24:00 AM
Hey guys, just a quick note that we will be performing some final configuration changes to complete this project today, 1/5/2012. We don't expect a lengthy outage, but might have a few minutes here and there of the site being unreachable or sluggish.
Title: Re: Server maintenance
Post by: Joker™ on January 05, 2012, 06:18:10 PM
Working awesome on my side. Great job team.
Title: Re: Server maintenance
Post by: Robert. on January 05, 2012, 06:24:31 PM
I agree, working great here too. :) Great work and thanks to all involved! :D
Title: Re: Server maintenance
Post by: Angelina Belle on January 06, 2012, 11:48:34 AM
Things are so much better now than they were in December. I'm hoping that's due to your work protecting us from the incessant  DDOS attacks.
Title: Re: Server maintenance
Post by: Joshua Dickerson on January 06, 2012, 01:01:02 PM
It was dschwab9 and CoreISP's new year's resolution to make the servers stable :D Unlike everyone else, you are actually fulfilling that resolution!
Title: Re: Server maintenance
Post by: karlbenson on January 10, 2012, 02:15:20 PM
Don't jinx it!
Title: Re: Server maintenance
Post by: admirable on February 12, 2012, 06:36:41 PM
Nice dear all the team of SMF.... Rocks... :laugh: