Topic: SMF admin panel backup is crap! Can we have a patch release to remove it please?

[Angelina Belle]

As it is, it is general purpose.  If you want to back up only one prefix, you specify the prefix.
You have to type in all the "settings.php stuff" yourself.

It would be great to have someone look through the code critically.  It is not complicated.

[emanuele]

• The backup can end in the middle due to timeouts, and the poor user is none the wiser.

What I have right now is a code that is as good as phpMyAdmin is (as far as I can test on my own computer).
It still rely on apache_reset_timeout or set_time_limit, I'll check later other ways to do the backup.

Just added something like this to the backup page (wording needs a lot of love... ):
http://img337.imageshack.us/img337/8521/screen61r.png

• SMF backs up in chunks of 250 records per INSERT. If there are 250*N records, you end up with a superfluous INSERT with no data.

That shouldn't happen with the code I wrote.

• Field names should be within ` because sometimes there are reserved keywords used as field names.

That is already like that in 2.0 as far as I can see:

Code: [Select]

$data = 'INSERT INTO `' . $tableName . '`' . $crlf . "\t" . '(`' . implode('`, `', $fields) . '`)' . $crlf . 'VALUES ';

Code: [Select]

$schema_create .= ' `' . $row['Field'] . '` ' . $row['Type'] . ($row['Null'] != 'YES' ? ' NOT NULL' : '');

[IchBin™]

I think putting a message about our tool in the backup page not working or not being reliable looks pretty amateur'ish. I think I'd prefer we point to the scripts that Unknown wrote. And just update those tools if they need it.

[Angelina Belle]

Knowing about a bug and not admitting it looks ?.
I think it would make sense to own up to "a bug" in the backup page.

Disable it in 2.1, if it can't be replaced "in time".

[Antechinus]

[unknown]'s backup and restore scripts are reliable for me, and they are written in php.  Are they reliable for larger forums?

Corn job running straight from mysqldump. Sorted. Easy. Works. No extra shiz required.

[Antechinus]

If it's been repaired (if needed) to avoid the problems I listed, it might just fit the ticket. Should it be included as a standalone utility in SMF (not directly called by SMF), or just offered for download? That is, do we want to encourage users to back up/restore using phpMyAdmin wherever possible, and only use this utility as a last resort? Should such a utility be optimized in any way for SMF (what ways)? Should it only back up/restore {smf_prefix} tables, in case the user has only one shared database, or can it be a general purpose utility?

Why phpMyAdmin? Honestly, it's not the best way of tackling backups and restoring.

[IchBin™]

Knowing about a bug and not admitting it looks ?.
I think it would make sense to own up to "a bug" in the backup page.

Disable it in 2.1, if it can't be replaced "in time".

I'm not talking about that. I'm saying let's not show that we have half assed code in our product. Either completely remove the option and perhaps point to the right tool, or fix the one so we don't have to put such a message in our product.

[emanuele]

Disable it in 2.1, if it can't be replaced "in time".

It can be replaced by something that works better, it cannot be replaced by something that works *in every possible condition*, but not because "we" are not good at writing such tool, but simply because it's impossible.

SMF is written in php and run in a very wide range of server configurations (from the cheap free overseller host to dedicated servers full of ram, space and almost without limits.
The point is: even on such super-duper server a backup made with php is *not* always reliable. It cannot be. But the same is true for *any* php script attempting to do a backup on large databases (phpMyAdmin included...and their authors should know something about MySQL).

I already wrote many of the reasons why such backups are not reliable, now it's up to you: do you want something or should we drop it completely?
Keep in mind that the *only* 100% reliable alternative is the command line, of course many people using SMF don't have access to such tool...

Knowing about a bug and not admitting it looks ?.
I think it would make sense to own up to "a bug" in the backup page.

Disable it in 2.1, if it can't be replaced "in time".

I'm not talking about that. I'm saying let's not show that we have half assed code in our product. Either completely remove the option and perhaps point to the right tool, or fix the one so we don't have to put such a message in our product.

On the code I'm using it's not a problem of fix anything.
As I said before I did a backup of a 2.8 GB database with that code (on my localhost), the point is that in a real-life scenario this backup is unreliable because it took me 20 minutes to do it. And the same would be true for phpMyAdmin.

[青山 素子]

I'm saying let's not show that we have half assed code in our product.

Looking at the page in the new doc wiki, and ugh... It's totally non-professional in tone and way too conversational. Can I just rewrite it in whole or do I need to run a total rewrite through somewhere first?

[Angelina Belle]

Regarding the reliability of MySql backup written in php:
Thanks for pointing that out, emanuele.  I was not aware.  I obviously have not tested out [unknown]'s script on a larg forum.
I've only seen that it works better than the inbuilt utility on my little forum running at 1&1, and I don't know why.

Would it be possible to include, with whatever backup thingy we eventually go with, some kind of guideline that the admin can read if they care to do so?
This backup is likely to work in the following cases, and unlikely to work in the following cases and here's how you can check if it worked properly.
Is there any way that the script itself can figure out if the backup failed?

Regarding the wiki:
It's a wiki.  Generally, on a wiki, we write first and ask questions later. Be bold.  But respectufl.
If two people care deeply about the wording of one document, they should discuss it in the documentation helpers board.
If there is a big problem, I will lock the document until "wiki war" is over.

We have two sets of standards for documents.  Those documenting the actual "pages" that admins see while using the software should be dry and non-personal.

How-tos, tutorials,  and FAQs can be written in a more "friendly" style.

[MrPhil]

Why phpMyAdmin? Honestly, it's not the best way of tackling backups and restoring.

If there is a better tool that is available on nearly every system, let's recommend it. My first priority would be to suggest tools that are likely to be on every system. If we package a better backup and restore utility with SMF, fine, let's tell people to use it (but it has to be bug free!). If it's a tool that many users are going to have to download and install (separately from SMF), I'm not so sure that's the right road to take.

[MrPhil]

It can be replaced by something that works better, it cannot be replaced by something that works *in every possible condition*, but not because "we" are not good at writing such tool, but simply because it's impossible.

SMF is written in php and run in a very wide range of server configurations (from the cheap free overseller host to dedicated servers full of ram, space and almost without limits.
The point is: even on such super-duper server a backup made with php is *not* always reliable. It cannot be. But the same is true for *any* php script attempting to do a backup on large databases (phpMyAdmin included...and their authors should know something about MySQL).

Granted, it is impossible to prove that any given non-trivial program is 100% correct. However, the issue at hand is that SMF's current backup is known to be riddled with serious bugs that would have been revealed with more than cursory testing.

As I said before I did a backup of a 2.8 GB database with that code (on my localhost), the point is that in a real-life scenario this backup is unreliable because it took me 20 minutes to do it. And the same would be true for phpMyAdmin.

The only problem I know of with phpMyAdmin (and maybe straight MySQL command line itself) is that it's subject to process timeouts. That can be cured by breaking up the job into smaller pieces in some manner. Otherwise, it seems to be quite reliable.

A further note (in favor of an SMF-specific backup): any backup that takes more than a second or two to back up the entire database should automatically put the forum into maintenance mode so that the database is not changed during the backup. This prevents the backup from being inconsistent.

[MrPhil]

Knowing about a bug and not admitting it looks ?.
I think it would make sense to own up to "a bug" in the backup page.

Disable it in 2.1, if it can't be replaced "in time".

Knowing about a serious bug and not saying anything about it (hiding it) verges on criminal behavior. Simple Machines could actually be found liable for damages if someone were to sue for loss to their business. Listing a relatively minor bug and its probable impact, with the promise that we're working on it and will have a fix soon, is permissible. Putting out software with known serious bugs, whether documented or not, is unacceptable. The software or function should be disabled, or release held up until it's fixed. This applies to all levels of SMF, not just 2.1. We've got to get out of the mindset that serious bugs in downlevel branches are OK, because we have a new leading edge version out. Unless we want to go out and cyber-attack SMF 1 installations to force them to upgrade to SMF 2, we have to accept that people will be running 1.1 and even 1.0 for a long time to come. I wouldn't be surprised if there were still YaBB (SE) installations running!

[Kindred]

Actually, I disagree with the last contention, MrPhil.

Bugs in previous versions which are known and acknowledged (or more importantly, fixed) in new releases can require people to upgrade. Sorry... but if someone refuses to upgrade when there is a fixed version out there, then what happens is on their own head. If someone doesn't keep track of the things fixed in upgrades, then THEY are acting stupidly.

So, fixing it (my preference) to some reasonable standard or removing it (I hope not) in 2.1 is acceptable.
1.0 is at end of life... ignore it.
1.1 has been stated, numerous times, to be in security release mode only...  If people really want to make a stink about it, then kill the function in 1.1 with the next security release, but I see no point in wasting time on a version which is outdated and replaced.
(honestly, if it were up to me, I would encourage the immediate sunset of 1.0 and the planned sunset of 1.1 by next year (instead of hanging on to outdated versions like we seem to do) -- but that's neither here not there, for this issue.)

and as for your statement "would have been revealed with more than cursory testing", I disagree....  I don't think I have actually had it fail on me, across 8 sites on 5 different hosts, machines, etc. I'll admit, it does seem to be an issue with some folks. Not sure what the common factor is which causes it to fail for them, but I'd hardly say that it was so obviously broken that it should have been immediately obvious (considering the length of our beta process, really?)

[MrPhil]

Testing was clearly inadequate. For example, the code outputs INSERTs with 250 records at a time. Testing should have been done with 499, 500, and 501 records for some table, just to make sure the 250 splitting works OK. This would have revealed a problem at 500 (250*N) records with an INSERT with no data.

I'll agree that it's hard to design a test to pick up everything. Who would think to test a hex field with value 123e4567, which MySQL will barf on if it's not in quotes? (tries to handle it as a floating point number) Older versions did not put ` around field names, which might not be a problem with vanilla SMF, but does show a problem with mods which add fields with reserved keywords as names. Good coding practice would try to account for that, but even thorough testing on vanilla SMF might not have displayed the problem.

As for your contention that older versions need not be fixed, I must respectfully disagree. At the very least, the next/last 1.0 update should disable backup (if it's not to be fixed) and the next 1.1 security update should do the same. If people want to stay at older 1.x levels, that's their lookout, but I feel that SM has an obligation towards its community to make available patches to fix bugs which seriously compromise the system's functioning, even if, strictly speaking, they're not security related. Needless to say, 2.0 should get the full treatment, and not wait for some future 2.1.

[Kindred]

well, obviously, I disagree...   but I'll leave it to the active team folks to decide how they specifically want to handle it.

[Sir Osis of Liver]

If a feature has limited functionality, but that functionality is considered to be of sufficient value to put up with the cases where it doesn't work, I can understand keeping it, but only if the feature displays appropriate error messages when it fails.  The very bad thing about the admin backup is that it always appears successful, making users believe they have a good backup when they actually do not. 

[Norv]

I feel that SM has an obligation towards its community to make available patches to fix bugs which seriously compromise the system's functioning, even if, strictly speaking, they're not security related. Needless to say, 2.0 should get the full treatment, and not wait for some future 2.1.

This is how it is. For as long as they're in active development, older versions may receive backported fixes for major/critical issues, that could affect critical functionality or compromise the system's functioning. We have done that in the past, for SMF 1.0.x even, and of course SMF 1.1.x, as well as 2.0.x. It does apply even to 1.0.x, on some issues. And of course, during ALL the time until 1.0.x reaches EOL. Whether an issue is critically needed or not, is the question, not that we do that. We might also not do it the same way, too.

It has been under question whether this is indeed the case of the backup functionality. It doesn't compromise one's system, though. There are cases of hitting resource limits and such, and there is no warning, error, message, something to tell the user that it may go wrong too. That's wrong, in the sense that when it happens, it gives a false sense of security.


Please do share your feedback on Ema's tool presented here:

• The backup can end in the middle due to timeouts, and the poor user is none the wiser.

What I have right now is a code that is as good as phpMyAdmin is (as far as I can test on my own computer).
It still rely on apache_reset_timeout or set_time_limit, I'll check later other ways to do the backup.

Just added something like this to the backup page (wording needs a lot of love... ):
http://img337.imageshack.us/img337/8521/screen61r.png

• SMF backs up in chunks of 250 records per INSERT. If there are 250*N records, you end up with a superfluous INSERT with no data.

That shouldn't happen with the code I wrote.

• Field names should be within ` because sometimes there are reserved keywords used as field names.

That is already like that in 2.0 as far as I can see:

Code: [Select]

$data = 'INSERT INTO `' . $tableName . '`' . $crlf . "\t" . '(`' . implode('`, `', $fields) . '`)' . $crlf . 'VALUES ';

Code: [Select]

$schema_create .= ' `' . $row['Field'] . '` ' . $row['Type'] . ($row['Null'] != 'YES' ? ' NOT NULL' : '');

Thank you very much for looking into it, Ema. Perhaps having it available for testing would be even better, when possible, for people to take it for a spin.
As for wording, can you propose an alternative, better wording?

[Norv]

Ok, I'm interested. Why was the backup unusable? Was it truncated, or was there some other problem?

Here is the original thread -

http://www.simplemachines.org/community/index.php?topic=474878.msg3320443#top

Let me get this straight.
1/  the user asks how to restore their backup.
2/  we are, instead, telling them the backup is "corrupted". (?)
3/  this discussion is opened, to report that another backup was corrupted. (?)

I must be missing something here?

[Sir Osis of Liver]

I think the point was to caution the OP that the backup may not be useable.  Not knowing the experience level of the user, importing a bad backup could further damage the forum db.  These support topics are problematic because it's often difficult to determine exactly what the user is dealing with, and the unreliability of admin backup must be considered.