Regarding our downtime

[metallica48423]

As many of you may have noticed, Simple Machines experienced downtime for approximately 8 hours today.

This stemmed from some important files being accidentally overwritten. Because of this, it caused the server to become unresponsive.

There is no evidence that any data was compromised.

We have restored the files from backups and are back up and running. If you notice any issues that didn't exist previously, please let us know in this topic.

We apologize for any inconveniences this has caused.

metallica48423
Project Manager

[Akyhne]

Yeah, I didn't have anything to do for 8 hours

[metallica48423]

Just as a note -- Fixed the missing smileys

[weightman]

How did the files get overwritten? Why did it take 8 hours to recover? No backup?

Just curious.

[shadow82x]

Yeah, I didn't have anything to do for 8 hours

Tell me about it. LOL

[GravuTrad]

very strange overwriting...

how could it be done?

[Bigguy]

Nice to see we're back up and running now though.

[青山 素子]

How did the files get overwritten?

We are investigating this. Also, there is an ongoing plan to streamline the way files are updated to prevent this kind of thing from happening easily in the future, and to keep site upgrades from being as rough as they have been in the past.

Why did it take 8 hours to recover? No backup?

Our main server admin was out of contact range, and I (as backup) was dealing with my own issues at the office. I was unable to get to things until I did, and it took some time to determine the cause and the extent of the problem before recovering. The last thing I wanted to do was cause further damage and lengthen the outage.

[weightman]

We are investigating this. Also, there is an ongoing plan to streamline the way files are updated to prevent this kind of thing from happening easily in the future, and to keep site upgrades from being as rough as they have been in the past.

Sounds good. I have always wondered how many people SMF allows access to the server running this site. I sort of envisioned a problem in that regard. I also noticed upgrades seemed a little slow and unorganized. Glad to see you are on top of it! It was a long day without SMF! Also, FYI, once I was able to access the SMF web install page. I don't know if that is a security risk or anything but it was definitely a surprise!

Our main server admin was out of contact range, and I (as backup) was dealing with my own issues at the office. I was unable to get to things until I did, and it took some time to determine the cause and the extent of the problem before recovering. The last thing I wanted to do was cause further damage and lengthen the outage.

Totally understandable.

[lax.slash]

I get this error randomly:

Fatal error: Call to undefined function smf_seed_generator() in /home/simple/public_html/community/index.php on line 87

Glad to see the site is back up, either way.

[Fustrate]

In reference to that error, did Subs.php get restored to a Beta 4 file instead of Beta 4 Public? IIRC, that's when smf_seed_generator was added.

[GravuTrad]

I get this error randomly:

Fatal error: Call to undefined function smf_seed_generator() in /home/simple/public_html/community/index.php on line 87

Glad to see the site is back up, either way.

i confirm that bug. i had it too randomly...

[metallica48423]

It is occuring randomly. 

The code *is* in Subs.php.

I believe it may be an SMF bug but we need to track down why it only occurs randomly

[Fustrate]

Because it only gets re-seeded randomly. I'm 95% certain that's the problem.

Code Select Expand

// Seed the random generator.
if (empty($modSettings['rand_seed']) || mt_rand(1, 250) == 69)
smf_seed_generator();

[Oldiesmann]

Yes, but that function is defined, and Subs.php is included with every page view in SMF, so there's no reason for that error to occur.

[Fustrate]

ooo... debug mode! Yahtzee! Winner gets to take the unneeded blame!

[Running With Scissors]

I was wondering what was up. I was hoping you were upgrading to smf 2.0 RC1 or something

[PeeaichpeeBB]

How did the files get overwritten?

We are investigating this. Also, there is an ongoing plan to streamline the way files are updated to prevent this kind of thing from happening easily in the future, and to keep site upgrades from being as rough as they have been in the past.

Why did it take 8 hours to recover? No backup?

Our main server admin was out of contact range, and I (as backup) was dealing with my own issues at the office. I was unable to get to things until I did, and it took some time to determine the cause and the extent of the problem before recovering. The last thing I wanted to do was cause further damage and lengthen the outage.

out of contact range???

you need a longer string methinks!

[yollyp]

Just trust the experts.  They knew what they are doing and there's no reason for them to just let this downtime happen without any significant corrections and fixing. Let the experts do their job.

Season's greetings to all.

[metallica48423]

I *may* have fixed the smf_seed_generator issue. 

Please let me/us know if this error happens to anyone again.