Settings.php, database update, caching, hostname lookup, editing controls

[MrPhil]

Since the developers refuse to address these issues as bug fixes, perhaps asking for them as features will work.

1. Fix the "Settings.php is empty" problem once and for all. It's still turning up several times a week. The original code is incredibly stupid. Pardon my French, but why the ****** are we rewriting the entire file just to change one value? Why are we emptying out the file first, to satisfy an ancient version of Unix, and risking a race condition? I told you guys years ago how to fix it by using a separate include file to hold just the "last database problem timestamp", and that's just one way of doing it, I'm sure.

2. Fix the "database may require an update" message. This is confusing to users and scares them. If you're going to try to keep file and database version numbers in synch, each file version should include the oldest acceptable database version it can use. That way, you can suppress unnecessary warnings about updates being needed. See my fixes for an acceptable method.

3. Fix the caching, or remove it. There are far too many problems reported with corrupted cache files.

4. Make hostname lookup optional (ship with it disabled). There are far too many problems reported due to excessive CPU load from this.

5. This one might be better as a mod, so feel free to detach and move it: I wish SMF would include edit/removal controls that would prevent modifying or erasing posts after a certain amount of time or after there are any replies. Admins/mods could of course edit or remove content if deemed necessary. Even better would be that existing material could be struck out and new material appended by posters, but the original can't be removed.

[Shambles]

I go along with points 1, 2 and 5, plus the remaining pair inbetween.

[Kindred]

emanuele,  I forget, did you implement Mr Phil's fix in 2.1?

[MrPhil]

A couple more related ones:

6. Lots of people, for some strange reason, like to type up their posts in Word, and cut and paste it over to their forum. MS Smart Quotes cause all sorts of problems, because they occupy code points reserved for specific control codes in all the standard code pages (everything except Windows-12xx). This forum is jammed with pleas for help because "my posts are being cut off".

If SMF developers will look around at the Real World, they will see on other systems that there is often an editor button to "paste from Word". SMF needs to implement this ASAP. I don't know if it's possible to query the code page for something that was just pasted into the browser, if the poster doesn't explicitly use a "paste from Word" button. If it is, do a proper translation of all codes to the forum's code page. If you don't know what the source code page is (and there is no "paste from Word" or it wasn't used), if your forum is not Windows-12xx itself, and the data includes x80-x9F bytes, assume it's Windows Smart Quotes. Translate them to the closest thing available on your forum code page -- ASCII equivalents if Latin-x, or Unicode characters if UTF-8. Note that this range of bytes is not quite consistent, even in Windows-12xx pages, so you'll have to come up with a set of characters that will please the most users (perhaps not a fixed set, but dependent on what the forum's code page is, e.g., if a Cyrillic encoding, assume it's coming from Windows-1251 Cyrillic).

7. While you're at it, if your forum is UTF-8, scan all input for invalid characters in the xA0-FF range and fix them in a similar manner to the Smart Quotes. Since those single byte characters can vary wildly by code page, perhaps it would be best to ask the poster if they know what code page the source was. If they have no idea, ask the language being used, and offer the page translated (code points) in all the reasonable encodings for that language (e.g., Russian language should be in Cyrillic, so try all the single byte Cyrillic translation tables until the poster says, "da, that's the one!"). The range of choices might be narrowed down if there are valid UTF-8 characters belonging to a certain alphabet in the rest of the post... at least, SMF could say, "I see Cyrillic characters in your post, but there are others that are not valid UTF-8. Did you paste in text from a Cyrillic source?"

This one could be a major hit on SMF code size, so it might be better as a mod, but at the least, standard SMF should recognize invalid UTF-8 input (due to pasting) and give the user a chance to do something about it. It's not reasonable to silently accept invalid characters and then puke all over itself later when it tries to display the text. Some browsers apparently do this already, by themselves (at least for Smart Quotes), but you can't guarantee that all your forum's users will be using a smart enough browser.

[Kindred]

Mr Phil,

You do need to stop demanding that SMF do things "ASAP"

If you are so gung ho to have it done, then - by all means, go to github and contribute your fixes.

In case you have not looked at 2.1, we have totally replaced the wysiwyg editor...
and I believe that 2.1 is also making utf-8 the default.

[MrPhil]

If something causes problems for users, and an inability to use the software "as promised", then it's a bug and should be fixed ASAP. I'm posting here in Features Requests only because SMF seems to be disinclined to fix bugs unless it's a red-lights-flashing, sirens-blaring, our-lawyer-wants-to-talk-to-you emergency. So, are fixes for these things I've listed going to appear in 1.1.18 and 2.0.4?

If I sign up for Github access and contribute fixes and features, is there any guarantee that fixes (once validated) will go out in the next release? I won't bother with this if a fix sits in limbo for years.

[MrPhil]

8. A not uncommon request is to change a forum name hard coded into many posts over to a new forum name (due to a move). Currently, administrators have to run direct SQL updates to the database. It would be nice to add an optional field in repair_settings.php to do a find and replace of the forum name while you're using it to update after a move. Even better would be a general find and replace facility (in messages and posts) in Admin > DB Maintenance. Just yesterday there was a request for how to clean up formatting junk left over from a migration from phpBB -- this facility could make it easier.

9. Avoid hard coding the forum name in the first place by having a standard BBCode [forum] that would be replaced by $boardurl (default) or something to the choice of the administrator.

[Kindred]

there is never any guarantee about anything....

but it's more likely to go out if you contribute it than if you sit back and demand that things be fixed by someone else.

[Arantor]

1. Fix the "Settings.php is empty" problem

No argument. I can understand the logic of why it was done - primarily to avoid loading another file on a fatal error. But yes, it should have been fixed.

2. Fix the "database may require an update" message.

No argument. There is no practical reason to actually store the version, or if there is, it should be updated - except no manual update ever does that anyway, even if the package came with instructions on doing it.

3. Fix the caching, or remove it. There are far too many problems reported with corrupted cache files.

It's not that hard to fix. It just requires dropping PHP < 5 support and using the atomic operation that is file_put_contents to atomicise it, followed by not having a fatal error on inclusion.

Though there is something freaky about a file which passes file_exists() but fails on the *next line* being an include/require statement.

4. Make hostname lookup optional (ship with it disabled). There are far too many problems reported due to excessive CPU load from this.

I'd possibly even consider removing it entirely if you're really going for it. Thing is... it's not the option itself that's broken; it's actually the host configuration and poor rDNS at the *host* level, not the software level that's wrong.

5. This one might be better as a mod, so feel free to detach and move it: I wish SMF would include edit/removal controls that would prevent modifying or erasing posts after a certain amount of time or after there are any replies.

There are already controls about editing posts after a certain amount of time. I forget if moderators explicitly can override that though.

The others I can see an argument in favour of though.

6. Lots of people, for some strange reason, like to type up their posts in Word, and cut and paste it over to their forum.

Correct.

If SMF developers will look around at the Real World, they will see on other systems that there is often an editor button to "paste from Word"

If people making bold proclamations will look around at the real world, they would understand that this is NOT the reason for Paste From Word.

The prime reason that the other editors (note, they're all WYSIWYG editors) need this is not because of encoding, but because Word is going to give them the formatting etc. in a strange and mashed up format that needs some parsing to do something with it.

As I have pointed out to you on multiple occasions - this very forum has demonstrated repeatedly that it works JUST FINE to convert to UTF-8. The browser is required to do the conversion to UTF-8 if the form properly indicates that it is UTF-8 and only supposed to send UTF-8.

The issues posted here, repeatedly, are because 1) forums are not set up as UTF-8 by default and 2) the converter has issues. It also doesn't help that the mb-string compatibility stuff can be damaged as can the code in Subs.php that handles some of this stuff.

If a forum is UTF-8 from the start, I've NEVER seen an issue with it. This forum proves it over and over again.

7. While you're at it, if your forum is UTF-8, scan all input for invalid characters in the xA0-FF range and fix them in a similar manner to the Smart Quotes.

The bulk of this issue is related to the UTF-8 converter, and/or converter from other software, not doing its job properly, not the core posting code.

You do need to stop demanding that SMF do things "ASAP"

Agreed. As I found, it just eats away at your credibility.

If something causes problems for users, and an inability to use the software "as promised", then it's a bug and should be fixed ASAP. I'm posting here in Features Requests only because SMF seems to be disinclined to fix bugs unless it's a red-lights-flashing, sirens-blaring, our-lawyer-wants-to-talk-to-you emergency. So, are fixes for these things I've listed going to appear in 1.1.18 and 2.0.4?

Hahahaha don't be silly. We all know that this stuff isn't going to be fixed in core SMF, mostly because most of it isn't really broken.

Even better would be a general find and replace facility (in messages and posts) in Admin > DB Maintenance

Yes, it probably would.

Just yesterday there was a request for how to clean up formatting junk left over from a migration from phpBB -- this facility could make it easier.

Actually, that should largely be down to the importer because you'd have to do a lot more than simple find/replace. Regex find replace is only part of the story there, too. Far better to clean it up where possible on import rather than try and find WHERE body RLIKE anything.

but it's more likely to go out if you contribute it than if you sit back and demand that things be fixed by someone else.

You know as well as I do that it makes precisely zero difference if commits won't be accepted. It's also why so many forks have sprung up.

[Antechinus]

4. Make hostname lookup optional (ship with it disabled). There are far too many problems reported due to excessive CPU load from this.

Not arguing with the rest of it, but hostname lookup is already optional, and has been optional since at least 1.1.x (not sure about 1.0.x). But yes, shipping with it disabled would be a good idea.

[Kindred]

AFAIK, contributions and commits are being accepted.

That was one of the main goals of dropping svn and going to github, IIRC

[Arantor]

Yes... but it still requires commits to be accepted. And given the crap storm over someone just changing the copyright line, it sort of defeats the point..

[Kindred]

not quite...   that update was made and confirmed by the same person instead of being reviewed and accepted (or not) and was made without consideration for the legal matter of changing a copyright.

Other commits, of actual code, are being accepted, (I believe)

[Arantor]

At a rate of a few per month, and the commits that are being accepted are all from people who are either developers in the project or ex-team members. I don't see a single third party patch being accepted.

In fact, even the non-developer commits were done by people who previously had SVN access at one point or another anyway...

[vbgamer45]

I agree with these changes should be done. Would save a lot of support time and have done for my own projects.
I think #1 is the most critical to fix it is unacceptable for file to have a chance to be rendered empty.

[Arantor]

If you've already done them, why not share them with the community?

[vbgamer45]

Mainly did the first four since they caused me issues on my sites may not be the best fix but works for me.
The issue that we have to have a list of fixes on third party sites means something is wrong.
http://www.catskilltech.com/freeSW/SMF/fixes1.1/

[Arantor]

So why aren't these people who fix these things either 1) posting them here for people to find and/or 2) actually putting their money where their mouth is and contributing to SMF core?

I've done some fixes to the above for my own stuff, but my own stuff is sufficiently different that it would not be useful to provide a diff anyway.

[Kindred]

At a rate of a few per month, and the commits that are being accepted are all from people who are either developers in the project or ex-team members. I don't see a single third party patch being accepted.

In fact, even the non-developer commits were done by people who previously had SVN access at one point or another anyway...

Well, I don't have full access to the commit submissions, but, to the best of my knowledge, no commutes by anyone other than those folks have ever been submitted......

[Arantor]

And therein lies the problem. No-one *wants* to commit anything because there's always been this mentality, so no-one *is* submitting anything.