Hello all
First let me say thanks for helping in advance
PROBLEM:
Just installed, made a few posts, all has went well until tonight.
Been playing with members making posts. When a member makes a post not all of the post will show up. Example I made a post that had 10 sentences and only 2 show. I did toggle a few settings today but for the life of me don't remember putting a limit of characters in a post.
Latest version being used, just installed yesterday 2.02
edited
Was that a one off or does it happen all the time ?
Thanks for the reply.
Did a lot of testing last with newly created members rotating posts through various boards and it seems to be one particular post. All members in all boards can post what they want and as much as they want. But, no mater who posts including admin this one post will only allow 2 sentences to be posted. When I write the post, I can view and all is there and looks good - but, when I hit the post button only 2 sentences show. Strange because there is no foul language . Anyway here is the post:
It's those quote marks, they're invalid code under some circumstances.
Take a backup, then convert the forum to UTF-8 from the maintenance area.
It's hard to see what exactly the original quotation marks are/were, but it's likely that the quotation marks are Microsoft "Smart Quotes" and are invalid in many browsers. Depending on the browser, a Smart Quote might be interpreted as a control code, which could terminate the post at that point. It won't do any good to convert to UTF-8 (both database and page display and any non-English language files), but it won't hurt.
Since this is such a common problem (cutting and pasting text from MS Word), I've asked the SMF developers to put in code to scan for Smart Quotes characters and convert them to HTML entities. For the time being, all you can do is edit the posts to replace the Smart Quotes with standard ASCII characters or HTML entities. See my sig > projects > Change "Smart Quotes" for a complete list of Smart Quotes to look for.
Quote from: MrPhil on May 12, 2012, 11:00:08 AM
It's hard to see what exactly the original quotation marks are/were, but it's likely that the quotation marks are Microsoft "Smart Quotes" and are invalid in many browsers. Depending on the browser, a Smart Quote might be interpreted as a control code, which could terminate the post at that point. It won't do any good to convert to UTF-8 (both database and page display and any non-English language files), but it won't hurt.
Since this is such a common problem (cutting and pasting text from MS Word), I've asked the SMF developers to put in code to scan for Smart Quotes characters and convert them to HTML entities. For the time being, all you can do is edit the posts to replace the Smart Quotes with standard ASCII characters or HTML entities. See my sig > projects > Change "Smart Quotes" for a complete list of Smart Quotes to look for.
Thanks for the reply but that was copied to notepad first to remove all that then pasted to thread, not sure but that doesn't seem to be the problem. Not an expert but I was taught notepad does remove word coding. Am I wrong on that?
Quote from: Arantor on May 12, 2012, 10:34:33 AM
It's those quote marks, they're invalid code under some circumstances.
Take a backup, then convert the forum to UTF-8 from the maintenance area.
I went to the "Convert the database and data to UTF-8" Section, there is a drop down menu to select options. Which option do I select?
Thank you
Quote from: spinfactor on May 12, 2012, 02:17:45 PM
I was taught notepad does remove word coding. Am I wrong on that?
Partially wrong. While cutting and pasting to Notepad (or, usually, anything done via Window's clipboard buffer) will remove the codes for
italic,
bold, etc., they will
not translate "Smart Quotes" characters into other characters (e.g., standard ASCII). What happens is that you end up with single byte codes in your text that aren't valid in the encoding that SMF uses (typically either Latin-1 or UTF-8). Some browsers may fudge the results and display these particular characters as Smart Quotes characters under some encodings, but you can't count on it.
Quote from: spinfactor on May 12, 2012, 02:17:45 PM
I went to the "Convert the database and data to UTF-8" Section, there is a drop down menu to select options. Which option do I select?
Before you go and convert your SMF installation to UTF-8, you should be aware that
if the root cause of your problem is that Smart Quote characters were cut and pasted in, it is highly unlikely that a conversion to UTF-8 will work. You
might be able to successfully convert existing database entries (posts) to UTF-8 if you tell phpMyAdmin that the existing database is CP-1252 rather than Latin-1, but it will do nothing for you in the future the next time someone cuts and pastes in a Word document containing Smart Quotes.
Quote from: MrPhil on May 12, 2012, 03:09:28 PM
Quote from: spinfactor on May 12, 2012, 02:17:45 PM
I was taught notepad does remove word coding. Am I wrong on that?
Partially wrong. While cutting and pasting to Notepad (or, usually, anything done via Window's clipboard buffer) will remove the codes for italic, bold, etc., they will not translate "Smart Quotes" characters into other characters (e.g., standard ASCII). What happens is that you end up with single byte codes in your text that aren't valid in the encoding that SMF uses (typically either Latin-1 or UTF-8). Some browsers may fudge the results and display these particular characters as Smart Quotes characters under some encodings, but you can't count on it.
Thank you, I learned something here
Quote from: MrPhil on May 12, 2012, 03:14:03 PM
Quote from: spinfactor on May 12, 2012, 02:17:45 PM
I went to the "Convert the database and data to UTF-8" Section, there is a drop down menu to select options. Which option do I select?
Before you go and convert your SMF installation to UTF-8, you should be aware that if the root cause of your problem is that Smart Quote characters were cut and pasted in, it is highly unlikely that a conversion to UTF-8 will work. You might be able to successfully convert existing database entries (posts) to UTF-8 if you tell phpMyAdmin that the existing database is CP-1252 rather than Latin-1, but it will do nothing for you in the future the next time someone cuts and pastes in a Word document containing Smart Quotes.
Gotcha, well this sounds far more difficult than my abilities. Besides, my past experience shows when you tinker with something that effects are none familiar to you the chances of a major screw-up are plentiful. Thanks, guess I'll live with this and hope members don't copy and paste. Shame
Maybe someone should come up with a mod to sanitize cut and pasted Smart Quotes into HTML entities? This may involve creating a new BBCode tag to handle entities, e.g., [ent=laquo]. We can then nudge the developers to build this into the base product (hopefully not waiting until 2.1). There may be work going on for this already, and if not, I'll try to take a look at it this weekend.
Sweet, I like that. There's always room for improvement my father always said, fingers crossed.
I did some experimenting, and at least some browsers on some PCs will translate cut and pasted Smart Quotes into UTF-8 equivalents for forums set up as UTF-8. I don't know what will happen if the database is converted to UTF-8, but I suspect that any Smart Quotes already in the database won't be translated to UTF-8. Since this translation of Smart Quotes on the way in (or out) is probably done on the whim of the browser author, and I haven't heard of any official standard to handle it, I hate to recommend doing something major like converting your database and forum to UTF-8. There's no telling when that will work and when it won't.
Try this. It translates Smart Quotes "on the way out" to HTML entities. It should work for any single byte encoding (as well as UTF-8), but has only been tested on a UTF-8 SMF 2.0.2 forum, so use at your own risk and be ready to back it out. On UTF-8, it tries to determine if a byte sequence is already a legitimate UTF-8 character, and if so, doesn't touch it. Otherwise (and on single byte encodings such as Latin-1), it changes anything in the range x80 through x9F to HTML entities. UTF-8 is the only multibyte encoding I've tried -- it will probably fail on UTF-16 or other multibyte encodings, but very few people use those (and they should not install this patch).
It only is applied to text that is potentially subject to BBCode expansion (it is called from parse_bbc). Text that does not undergo BBCode processing (e.g., topic subjects) will not be fixed up. Perhaps someone with the time and inclination can fix that (it doesn't depend on BBCode; that was just a convenient place to call it from). This patch is just a Quick'n'Dirty; others are welcome to fix it up or redo it in more elegant code.
All changes are in Sources/Subs.php. Applying this to SMF 1.x will probably work, but you may have to work to find where in parse_bbc() to apply the changes.
Find
// Never show smileys for wireless clients. More bytes, can't see it anyway :P.
and change to
// Just in case it wasn't determined yet whether UTF-8 is enabled.
if (!isset($context['utf8']))
$context['utf8'] = (empty($modSettings['global_character_set']) ? $txt['lang_character_set'] : $modSettings['global_character_set']) === 'UTF-8';
// always fix up "Smart Quotes" going out to the browser, no matter what the encoding
$message = fix_SmartQuotes($message);
// Never show smileys for wireless clients. More bytes, can't see it anyway :P.
Find
// Just in case it wasn't determined yet whether UTF-8 is enabled.
if (!isset($context['utf8']))
$context['utf8'] = (empty($modSettings['global_character_set']) ? $txt['lang_character_set'] : $modSettings['global_character_set']) === 'UTF-8';
and change to (comment out):
// // Just in case it wasn't determined yet whether UTF-8 is enabled.
// if (!isset($context['utf8']))
// $context['utf8'] = (empty($modSettings['global_character_set']) ? $txt['lang_character_set'] : $modSettings['global_character_set']) === 'UTF-8';
At the very end of the file, find
?>
and insert before it:
// Translate MS Smart Quotes into HTML entities, so browsers don't choke
// on them as control codes. Some browsers treat Latin-1 as CP-1252, and
// some pages may already be in CP-1252; in either case this should be safe.
// This is done upon text output, and only for text which can have BBCodes.
// Database functions which might be searching or sorting text on UTF-8, and
// are run against the text before this routine is called, might still fail.
// author: SMF's MrPhil (see catskilltech.com)
function fix_SmartQuotes($message) {
global $context;
// sometimes a numeric entity is used, because the named entity
// may not yet be widely supported
$SQ_list = array(
// UTF SQ usage, UTF usage
// codepoint
0x80 => '€' , // 20AC Euro, reserved control
0x81 => '?', // unused, reserved control
0x82 => '‚' , // 201A Low-"9" opening quotation mark, Break Permitted Here
0x83 => '& #402;' , // 0192 ƒ Florin/script f/folder, No Break Here
0x84 => '„' , // 201E Low-"99" opening quotation mark, Index
0x85 => '…', // 2026 Ellipsis, Next Line
0x86 => '†', // 2020 Single dagger, Start of Selected Area
0x87 => '‡', // 2021 Double dagger, End of Selected Area
0x88 => 'ˆ' , // 02C6 Circumflex ^ accent (combining?), Character Tabulation Set
0x89 => '‰', // 2030 o/oo per mille, Character Tabulation with Justification
0x8A => '& #352;' , // 0160 &Scaron S + caron accent, Line Tabulation Set
0x8B => '‹', // 2039 Single left angle quote < (guillemet) Partial Line Down
0x8C => 'Œ' , // 0152 OE ligature, Partial Line Up
0x8D => '?', // unused, Reverse Line Feed
0x8E => '& #381;' , // 017D &Zcaron Z + caron accent, Single Shift Two
0x8F => '?', // unused, Single Shift Three
0x90 => '?', // unused, Device Control String
0x91 => '‘' , // 2018 "6" opening quotation mark, Private Use One
0x92 => '’' , // 2019 "9" closing quotation mark/apostrophe, Private Use Two
0x93 => '“' , // 201C "66" opening quotation mark, Set Transmit State
0x94 => '”' , // 201D "99" closing quotation mark, Cancel Character
0x95 => '•' , // 2022 Solid bullet, Message Waiting
0x96 => '–' , // 2013 En-dash, Start of Guarded Area
0x97 => '—' , // 2014 Em-dash, End of Guarded Area
0x98 => '˜' , // 02DC Tilde ~ accent (combining?), Start of String
0x99 => '™' , // 2122 Trademark TM, reserved control
0x9A => '& #353;' , // 0161 &scaron s + caron accent, Single Character Introducer
0x9B => '›', // 203A Single right angle quote > (guillemet), Control Sequence Introducer
0x9C => 'œ' , // 0153 oe ligature, String Terminator
0x9D => '?', // unused, Operating System Command
0x9E => '& #382;' , // 017E &zcaron z + caron accent, Privacy Message
0x9F => 'Ÿ' , // 0178 Y + diaeresis/umlaute accent, Application Program Command
);
$new_message = '';
if ($context['utf8']) {
// we are in multibyte UTF-8 mode, so need to skip legitimate UTF-8
// sequences that may contain x80-9F bytes inside them
// note that strlen($message) can vary as entities replace char bytes
for ($i = 0; $i < strlen($message); $i++) {
$c = ord($message[$i]);
// lead byte 110x xxxx, followed by one 10xx xxxx, or
// 1110 xxxx two
// 1111 0xxx three ?
// if so, is legitimate UTF-8 sequence, don't modify
$utf8_seq = 0; // not UTF-8 (zero 10xx xxxx bytes to follow)
$cm = $c & 0xE0;
if ($cm == 0xC0) {
$utf8_seq = 1;
} else {
$cm = $c & 0xF0;
if ($cm == 0xE0) {
$utf8_seq = 2;
} else {
$cm = $c & 0xF8;
if ($cm == 0xF0) {
$utf8_seq = 3;
}
}
}
for ($j = 0; $j < $utf8_seq; $j++) {
// j+1st following byte should be 10xx xxxx
// but first, are we running off the end of $message?
// shouldn't happen with well-formed UTF-8 characters...
if ($i+$j+1 >= strlen($message)) {
$utf8_seq = 0;
break;
}
$cm = ord($message[$i+$j+1]) & 0xC0;
if ($cm != 0x80) {
$utf8_seq = 0;
break;
}
}
// skip over next $utf8_seq bytes as a legitimate UTF-8 sequence
// or process single byte as possible Smart Quote
if ($utf8_seq == 0) {
if ($c >= 0x80 && $c <= 0x9F) {
$new_message .= $SQ_list[$c]; // replace by HTML entity
} else {
$new_message .= chr($c); // use original character
// note that originally malformed UTF-8 won't be fixed
}
} else {
$new_message .= substr($message, $i, $utf8_seq+1); // use original bytes
$i += $utf8_seq; // end of loop adds another 1
}
} // end of for loop through $message
} else {
// we are in a single byte mode, so go ahead and fix any
// x80-9F bytes
for ($i = 0; $i < strlen($message); $i++) {
$c = ord($message[$i]);
if ($c >= 0x80 && $c <= 0x9F) {
$new_message .= $SQ_list[$c];
} else {
$new_message .= chr($c);
}
}
}
return $new_message;
}
There are five (5) entries in the $SQ_list table where you need to close up & and # (remove the space) to form proper HTML entities. This is because numeric entities get processed in the forum display into the actual characters.
Add: That last chunk of text is available as an attachment SQ_fix.txt. Just cut and paste it into place (just before the closing ?> ). There is no fixup needed as described above.
Have you had a chance to try this code? Please let me know if it works for you, or if you experience problems. If it works well, it might get packaged up into a mod.
Please, has this been tested by anyone using SMF 2.0.2? does it work on 2.0.2?
It was developed and tested (briefly) on SMF 2.0.2. Just be sure to back up the file before editing it, in case something goes wrong.
Ok. I will do that ASAP. though I am not good in php coding. :-*
No PHP coding is involved. All you have to do is edit one file, looking for a match in a group of lines, and making some changes listed.
Thanks mrphil. The problem is solved now with your patch. Hope it remains a permanent solution. Many thanks.
Ahh, I thought this thread was closed. Could have sworn it was closed, that's why I started a different thread yesterday. Anyway, thanks for looking into this further. I'll give it a shot
BTW, where so I make these changes? How do I find this find to make the changes? Sorry, this is all new to me
QuoteBTW, where so I make these changes?
In his posting above MrPhil advised that these changes were in
Subs.php which is located in the
Sources folder
;)
Yes I got the name of the file, that was clearly established, thank you.
Where does one locate the sources folder please?
Better yet, is there a tutorial on this that I possibly over looked when browsing for help?
Find your SMF installation on your site. It's where Settings.php lives. The Sources folder is under that folder.
shhh, I knew it was simple. I must be blind. Thank you sir
Mr. Phil......
I'm confused with this part:
There are five (5) entries in the $SQ_list table where you need to close up & and # (remove the space) to form proper HTML entities. This is because numeric entities get processed in the forum display into the actual characters.
Maybe if you have time sir please explain a little more, sorry?
Not sure what the $SQ_list table is or where to find.
Not sure exactly what this means, sorry again, this is all new for me:
close up & and # (remove the space)
Five of the entries contain '& #1234;' type entries. I had to add the space so that SMF would not immediately interpret the HTML entity. You have to remove the space to get '&#nnnn;' type entries. See, I can't even do it here.
Add: The $SQ_list table is in the new code to add to the Subs.php file.
Now I'm really confused. Sorry and thank you for your time. You know, I just realized one of the post I was trying to add showed up here all 100 percent but not on mine. Strange
What is the $SQ_list table and where do I find it please?
Can anybody else tell me what is the $SQ_list table and where do I find it please?
I think Mr. Phil wants me to remove some spaces in five locations other than make changes to that file, correct, no?
Thanks in advance
QuoteCan anybody else tell me what is the $SQ_list table and where do I find it please?
I think Mr. Phil wants me to remove some spaces in five locations other than make changes to that file, correct, no?
I must admit I found this a little confusing as well as it was a little ambiguous.
My understanding is that he was only referring to the code that he advised adding to Subs.php in this post:
http://www.simplemachines.org/community/index.php?topic=476538.msg3333723#msg3333723
No where else.
I would add the code in that post and try it. If thats what you've done then I think thats it.
If I'm wrong then maybe someone else (or MrPhil) can advise. ;)
You take the code posted in that post and add it to your file. Part of that code is the aforementioned $SQ_list, and you need to remove some spaces in the code where it says, because it can't be posted properly in the forum otherwise.
Sigh. I have added that last new function as an attachment to the earlier post. You can download the file and cut and paste it into Subs.php without having to do any of the manual fixing (closing up of some spaces). Just be careful to compare the finished result with what was listed in the post, to make sure your editor didn't do anything funny with entities. And of course, the 5 &#nnnn; entries will already have their spaces closed up.
Why not just make a mod package out of it?
Some day. I wanted to let a few people play with it first to see if it consistently worked for them. I especially want to make sure it works for all combinations of encodings and database histories. So far I've only gotten one report back (successful). If a few more report success, I'll submit it as a mod. I need to read up first on how to package a mod, as this will probably need to cover all 1.x and 2.x systems.
Thank you MrPhil.
Do I copy your code in the downloaded file and paste that code at the end of the Subs.php file just before this ?>?
Yes. Just paste it in before the ?>. Do an eyeball check on it to see if it looks like the listing in the post. Note that there are a couple of other changes needed that you still have to edit manually.
ok, this is getting more and more confusing. I thought the down load was the fix so I'm going back one more time to read all over again.
It covers that last big chunk of code that you were having such a horrible time with. It does not cover the other two relatively small pieces of code change.
OK Sir, I think I got this. Soooooo (fingers crossed) do the change code thing in this quote below, and then add the downloaded code to the end of file?
Quote from: MrPhil on May 13, 2012, 11:46:36 AM
I did some experimenting, and at least some browsers on some PCs will translate cut and pasted Smart Quotes into UTF-8 equivalents for forums set up as UTF-8. I don't know what will happen if the database is converted to UTF-8, but I suspect that any Smart Quotes already in the database won't be translated to UTF-8. Since this translation of Smart Quotes on the way in (or out) is probably done on the whim of the browser author, and I haven't heard of any official standard to handle it, I hate to recommend doing something major like converting your database and forum to UTF-8. There's no telling when that will work and when it won't.
Try this. It translates Smart Quotes "on the way out" to HTML entities. It should work for any single byte encoding (as well as UTF-8), but has only been tested on a UTF-8 SMF 2.0.2 forum, so use at your own risk and be ready to back it out. On UTF-8, it tries to determine if a byte sequence is already a legitimate UTF-8 character, and if so, doesn't touch it. Otherwise (and on single byte encodings such as Latin-1), it changes anything in the range x80 through x9F to HTML entities. UTF-8 is the only multibyte encoding I've tried -- it will probably fail on UTF-16 or other multibyte encodings, but very few people use those (and they should not install this patch).
It only is applied to text that is potentially subject to BBCode expansion (it is called from parse_bbc). Text that does not undergo BBCode processing (e.g., topic subjects) will not be fixed up. Perhaps someone with the time and inclination can fix that (it doesn't depend on BBCode; that was just a convenient place to call it from). This patch is just a Quick'n'Dirty; others are welcome to fix it up or redo it in more elegant code.
All changes are in Sources/Subs.php. Applying this to SMF 1.x will probably work, but you may have to work to find where in parse_bbc() to apply the changes.
Find
// Never show smileys for wireless clients. More bytes, can't see it anyway :P.
and change to
// Just in case it wasn't determined yet whether UTF-8 is enabled.
if (!isset($context['utf8']))
$context['utf8'] = (empty($modSettings['global_character_set']) ? $txt['lang_character_set'] : $modSettings['global_character_set']) === 'UTF-8';
// always fix up "Smart Quotes" going out to the browser, no matter what the encoding
$message = fix_SmartQuotes($message);
// Never show smileys for wireless clients. More bytes, can't see it anyway :P.
Find
// Just in case it wasn't determined yet whether UTF-8 is enabled.
if (!isset($context['utf8']))
$context['utf8'] = (empty($modSettings['global_character_set']) ? $txt['lang_character_set'] : $modSettings['global_character_set']) === 'UTF-8';
and change to (comment out):
// // Just in case it wasn't determined yet whether UTF-8 is enabled.
// if (!isset($context['utf8']))
// $context['utf8'] = (empty($modSettings['global_character_set']) ? $txt['lang_character_set'] : $modSettings['global_character_set']) === 'UTF-8';
At the very end of the file, find
?>
and insert before it:
// Translate MS Smart Quotes into HTML entities, so browsers don't choke
// on them as control codes. Some browsers treat Latin-1 as CP-1252, and
// some pages may already be in CP-1252; in either case this should be safe.
// This is done upon text output, and only for text which can have BBCodes.
// Database functions which might be searching or sorting text on UTF-8, and
// are run against the text before this routine is called, might still fail.
// author: SMF's MrPhil (see catskilltech.com)
function fix_SmartQuotes($message) {
global $context;
// sometimes a numeric entity is used, because the named entity
// may not yet be widely supported
$SQ_list = array(
// UTF SQ usage, UTF usage
// codepoint
0x80 => '€' , // 20AC Euro, reserved control
0x81 => '?', // unused, reserved control
0x82 => '‚' , // 201A Low-"9" opening quotation mark, Break Permitted Here
0x83 => '& #402;' , // 0192 ƒ Florin/script f/folder, No Break Here
0x84 => '„' , // 201E Low-"99" opening quotation mark, Index
0x85 => '…', // 2026 Ellipsis, Next Line
0x86 => '†', // 2020 Single dagger, Start of Selected Area
0x87 => '‡', // 2021 Double dagger, End of Selected Area
0x88 => 'ˆ' , // 02C6 Circumflex ^ accent (combining?), Character Tabulation Set
0x89 => '‰', // 2030 o/oo per mille, Character Tabulation with Justification
0x8A => '& #352;' , // 0160 &Scaron S + caron accent, Line Tabulation Set
0x8B => '‹', // 2039 Single left angle quote < (guillemet) Partial Line Down
0x8C => 'Œ' , // 0152 OE ligature, Partial Line Up
0x8D => '?', // unused, Reverse Line Feed
0x8E => '& #381;' , // 017D &Zcaron Z + caron accent, Single Shift Two
0x8F => '?', // unused, Single Shift Three
0x90 => '?', // unused, Device Control String
0x91 => '‘' , // 2018 "6" opening quotation mark, Private Use One
0x92 => '’' , // 2019 "9" closing quotation mark/apostrophe, Private Use Two
0x93 => '“' , // 201C "66" opening quotation mark, Set Transmit State
0x94 => '”' , // 201D "99" closing quotation mark, Cancel Character
0x95 => '•' , // 2022 Solid bullet, Message Waiting
0x96 => '–' , // 2013 En-dash, Start of Guarded Area
0x97 => '—' , // 2014 Em-dash, End of Guarded Area
0x98 => '˜' , // 02DC Tilde ~ accent (combining?), Start of String
0x99 => '™' , // 2122 Trademark TM, reserved control
0x9A => '& #353;' , // 0161 &scaron s + caron accent, Single Character Introducer
0x9B => '›', // 203A Single right angle quote > (guillemet), Control Sequence Introducer
0x9C => 'œ' , // 0153 oe ligature, String Terminator
0x9D => '?', // unused, Operating System Command
0x9E => '& #382;' , // 017E &zcaron z + caron accent, Privacy Message
0x9F => 'Ÿ' , // 0178 Y + diaeresis/umlaute accent, Application Program Command
);
$new_message = '';
if ($context['utf8']) {
// we are in multibyte UTF-8 mode, so need to skip legitimate UTF-8
// sequences that may contain x80-9F bytes inside them
// note that strlen($message) can vary as entities replace char bytes
for ($i = 0; $i < strlen($message); $i++) {
$c = ord($message[$i]);
// lead byte 110x xxxx, followed by one 10xx xxxx, or
// 1110 xxxx two
// 1111 0xxx three ?
// if so, is legitimate UTF-8 sequence, don't modify
$utf8_seq = 0; // not UTF-8 (zero 10xx xxxx bytes to follow)
$cm = $c & 0xE0;
if ($cm == 0xC0) {
$utf8_seq = 1;
} else {
$cm = $c & 0xF0;
if ($cm == 0xE0) {
$utf8_seq = 2;
} else {
$cm = $c & 0xF8;
if ($cm == 0xF0) {
$utf8_seq = 3;
}
}
}
for ($j = 0; $j < $utf8_seq; $j++) {
// j+1st following byte should be 10xx xxxx
// but first, are we running off the end of $message?
// shouldn't happen with well-formed UTF-8 characters...
if ($i+$j+1 >= strlen($message)) {
$utf8_seq = 0;
break;
}
$cm = ord($message[$i+$j+1]) & 0xC0;
if ($cm != 0x80) {
$utf8_seq = 0;
break;
}
}
// skip over next $utf8_seq bytes as a legitimate UTF-8 sequence
// or process single byte as possible Smart Quote
if ($utf8_seq == 0) {
if ($c >= 0x80 && $c <= 0x9F) {
$new_message .= $SQ_list[$c]; // replace by HTML entity
} else {
$new_message .= chr($c); // use original character
// note that originally malformed UTF-8 won't be fixed
}
} else {
$new_message .= substr($message, $i, $utf8_seq+1); // use original bytes
$i += $utf8_seq; // end of loop adds another 1
}
} // end of for loop through $message
} else {
// we are in a single byte mode, so go ahead and fix any
// x80-9F bytes
for ($i = 0; $i < strlen($message); $i++) {
$c = ord($message[$i]);
if ($c >= 0x80 && $c <= 0x9F) {
$new_message .= $SQ_list[$c];
} else {
$new_message .= chr($c);
}
}
}
return $new_message;
}
There are five (5) entries in the $SQ_list table where you need to close up & and # (remove the space) to form proper HTML entities. This is because numeric entities get processed in the forum display into the actual characters.
Add: That last chunk of text is available as an attachment SQ_fix.txt. Just cut and paste it into place (just before the closing ?> ). There is no fixup needed as described above.
Everything except
QuoteAt the very end of the file, find
?>
and insert before it:
(code removed)
There are five (5) entries in the $SQ_list table where you need to close up & and # (remove the space) to form proper HTML entities. This is because numeric entities get processed in the forum display into the actual characters.
Instead,
At the very end of the file, find
?>
and paste
before it the contents of the file SQ_fix.txt.
OK, got it and sorry for the hassle and thank you very very much for your patience. I'm tired now so should be tomorrow I'll attempt this. First I need to read up on what to back up for before this change. ;)
MrPhil, it didn't work. Still cuts off everything whenever their are " ( ' characters involved....thanks for your efforts
I uploaded the changed Subs.php file if you wish to review because I know you put work into this, if not, I don't blame you. Thanks again
The Subs.php you sent me had been modified for other mods, but you had not made any of the changes for Smart Quotes. Attached is a revised copy. Keep a copy of the existing one just in case...
P.S. I had an editor warning that it might have changed the "end of line" convention. Check that the file looks OK when you upload it. If not, you might have to upload in binary instead of ASCII or vice-versa.
OH lord, not sure what happened. I'm somewhat new at this, maybe I got confused and uploaded wrong file. Please forgive me. However, I did download your file and uploaded it to the sources folder. NO Work. I tried copy and past an article, all words will display until it gets to those "/')(& characters which is where the article gets cut off and nothing beyond the character will display.
P.S. I did check it and all looked normal
Groan. Attach Subs.php (the right one!) to your next post and I'll look at it.
OK, thank you sir
Well, I'm not seeing any important difference between yours and mine, certainly nothing that should cause it to fail. You're absolutely sure you have this code in .../Sources/Subs.php? You have only one copy of your forum (no chance of looking in the wrong forum)? Your forum is in UTF-8 or Latin-1 encoding (which)? Your "bad" text causing the failure was cut and pasted in from Word, and has "Smart Quotes" characters like the nice quotation marks and such? Look at the page source (View > Page Source)... have any of the bad characters been changed to HTML entities such as «? If so, my code is working. Is there any chance that there are other invalid characters in the cut and pasted text? Not just Smart Quotes, but other valid Latin-1 but invalid UTF-8 text (if your forum is UTF-8).
1. Yes sir, I use filezilla and I'm sure the Subs.php file is being dropped in the sources folder.
2. I only have this one forum, no others
3. After viewing page source I identified two places where this appears in red:
4. Is there any chance that there are other invalid characters in the cut and pasted text?
just the standard " or ' or ( or ) or/ seems to be the ones that cause this.
5. Not just Smart Quotes, but other valid Latin-1 but invalid UTF-8 text (if your forum is UTF-8). All I noticed thus far were the characters above
So characters like " and ' (Smart Quotes versions, not the standard ASCII) are not being turned into HTML entities like »? Once again, is your forum UTF-8 or Latin-1? Did you clear your SMF cache (erase all .php files from cache directory) and clear your browser cache after uploading the Subs.php code? Did you make sure the capitalization wasn't messed up -- you didn't add it back to your site as subs.php, did you?
1. I don't know if my SMF forum is UTF-8 or Latin-1. Been searching to determine but nothing yet. I just did the install and it is what it is, sorry. Still looking through Goggle and SMF forums.
2. I did clear browser cache twice and nothing.
3. I also cleared my SMF cache
4. So characters like " and ' (Smart Quotes versions, not the standard ASCII) are not being turned into HTML entities like »?
All I know is whenever you paste an article with " or ' or - characters in the body the character used and everything beyond that point will not display. The full article will preview nicely but will not post properly.
5. Did you make sure the capitalization wasn't messed up -- you didn't add it back to your site as subs.php, did you?
What do you mean capitalization wasn't messed up? Sorry but I don't understand
1. In your browser, View > Character Encoding (or similar wording, possibly Character Set or Charset). It will tell you what your forum is requesting. For example, this forum (simplemachines.org) shows up as "UTF-8".
4. This is text from Microsoft Word? The quotes are the curved kind, looking like 6's and 9's, not the straight ' and "? Those are "Smart Quotes", which will wreck your forum display, especially if your forum is UTF-8.
5. The file is Subs.php with a capital S. I was asking you to check whether you made a mistake and put it in the Sources/ directory as subs.php (small s). If you did, it's still using the old unmodified Subs.php and your updated subs.php is being ignored.
1. Western (150-8859-1)
4. They appear to be the straight kind, eyes are going bad but they sure look straight to me.
5. Yes I checked that in the beginning, all is good
OK, so you're still at Latin-1. The only text which is giving problems is cut and pasted from a Word document? If so, could you paste a small sample into your next post here so we can see if it's using Smart Quotes? Sometimes the curves are very subtle. I think this forum will be able to display them, but check the results anyway.
What browser are you using (make and version)? Many browsers these days actually use CP-1252 when Latin-1 is requested -- maybe yours isn't, as is choking on what it thinks are control codes.
Edit post
Browser is Firefox 12.0 the latest ver
OK, yes those are curved quotes and apostrophes, so they're Smart Quotes. Did you cut and paste directly from Word into simplemachines.org, without going through any intermediate steps? If the last Subs.php I sent you is in the right place under the right name, and the database and pages are displayed in Latin-1/ISO-8859-1 (and that's what you requested), then I don't know what's wrong with your system. I take it you did not end up trying to convert to UTF-8? You were asking about how to do it.
I did not convert or attempt to do so.
Did you cut and paste directly from Word into simplemachines.org, without going through any intermediate steps?
I cut and paste from an old non smf forum we had to our new smf forum directly. Even if I paste to notepad before the smf forum the results will be the same.
yes the Sub.php file is in the right place for sure
If the text went through another forum or web application before arriving at SMF, who knows what happened to the original Smart Quotes! My changes to Subs.php are assuming actual Smart Quotes byte codes, and I suspect yours may have already been changed to something else. If they were changed to UTF-8, you should be seeing multiple (usually 3) funny accented characters for each Smart Quote. There's a chance that they are UTF-8, but one of the bytes is still a control code that is killing the display. If the control code is the second or third byte, I would still expect to see a funny accented character before the text is cut off. Was there, and you edited it out?
Have you tried cutting and pasting text directly from Word (with Smart Quotes) into SMF? Does it work then? If it does, is the source of your problem text Word or another forum? If it's not Word, my fix may not work.
No funny code at all appeared, just cut off.
I typed this in word: This is just a test just a "test"
copied and paste it in the forum and got this:
This is just a test just a
OK, please upload the attached copy of Subs.php. It contains a few lines to dump out the bytes in your input (raw) message so I can see what's going on. Please repeat your "just a test" test with it. You shouldn't see any change to the display, but if you look at the page source (HTML) you should see a comment <!-- fix_SmartQuotes: ... followed by whether or not it's UTF-8 and all the byte values. Please cut and paste that line into your next post. After that, you can restore your system by editing out all the lines between (and including) // =================, or just restoring an earlier copy of the file.
OK sir, this is what I saw in two locations, the first was located at the top of the page source and very first line and the second was about half way down
<!-- fix_SmartQuotes: is NOT UTF-8 encoding, bytes: 4D 45 42 41 20 4D 65 6D 62 65 72 73 20 6D 75 73 74 20 73 69 67 6E 20 69 6E 20 74 6F 20 76 69 65 77 20 61 6C 6C 20 62 6F 61 72 64 73 -->
<!-- fix_SmartQuotes: is NOT UTF-8 encoding, bytes: 54 68 69 73 20 69 73 20 6A 75 73 74 20 61 20 74 65 73 74 20 -->
Is the second one where it fails and gets cut off? The first line says "MEBA Members must sign in to view all boards" and the second line says "This is just a test ". Was this second one cut off at a Smart Quote? What was its history -- was it cut and pasted directly from Word? It looks like it got cut off either while going into the database or being retrieved from it. It didn't survive to get to the point where my routine could work on it!
Well, at this point, you certainly could try converting your forum to UTF-8, and hope for the best. I know that some browsers (no guarantee that all) will convert Smart Quotes on the fly to UTF-8 characters. Maybe that will work. I would leave in the fix_SmartQuotes patch I gave you just in case some Smart Quotes in existing posts survive the conversion process and end up being sent to the browser. You can either revert to your previous copy (without the hex dump) or edit the file to remove // ===== and everything in between those two lines.
Note that writing a post in Word, only for the purpose of cutting and pasting it to SMF, is kind of pointless. All the formatting is destroyed anyway, and needs to be put back in. The only reason would be that you're going to keep a Word document with that content, anyway. If you just want to paste the same text in multiple forums, I would write it in Notepad so that it's plain text (no formatting, no Smart Quotes).
Is the second one where it fails and gets cut off? I've lost you on this one, no idea what second one you are talking about, sorry.
At this point I have no idea where the original article was created or created on. All I know is when a member copies an article from my old forum to this one where ever there is a ' or " character it will not display anything from there forward.
As for: This is a test "just a test"
That was done in word as you asked me too, copied and pasted in the forum resulting in the exact same behavior.
OK, this isn't good, not much of a forum if members can't copy and paste an article without most of the article missing, beginning to wish I had invested the money into vBulletin. Thank you for your help, you really reached out far beyond what most folks would have. Very kind of you sir. If I can ever repay the favor please send a PM and I'll do my best to assist you. Off to research on how to convert this thing over as that seems to be the last stop before I ditch this thing for vBulletin. Thank you again and God Bless
This is just a test -- was there more to this line? Was this line cut and pasted directly from Word? Was it cut off at a quotation mark? Something very strange is happening on your system compared to others. You're only the second one to try my fix for Smart Quotes and report back (the other one was successful). Now, once you paste Word text with Smart Quotes into another forum or something, who knows what happens to the Smart Quotes. They may or may not be converted to something else at that point. The only other suggestion would be to go ahead and try converting to UTF-8, and maybe you'll be lucky.
It would be unfortunate to lose you, but if you're confident that another forum can properly handle your requirements (especially cut and pasted Smart Quotes), maybe that would be the necessary impetus to get the SMF developers off their duffs and fixing this thing right. Of course, for any forum software, a lot depends on what your particular browser is doing with these pasted Smart Quotes characters on input and in what form they survive their round trip to the database. For example, Firefox 12 on a UTF-8 page will apparently convert Smart Quotes to the equivalent UTF-8 characters, but will IE7 (say) do the same? I'm not even going to ask you to look inside your database and try to find the records for your test posts and see if they are already cut off in the database.
Good luck and best wishes with your forum work, and I hope you find something that works for you.
Yes, IE and others will do exactly the same in converting to UTF-8 and has done for years (even IE6 gets this right)
It is no accident that I ripped out all the ISO handling code in its entirety over a year ago in my SMF fork, because I've long, long since known that if the forum is in UTF-8, the browsers will do their job in getting it right when served with a UTF-8 page and a form that specifically indicates it should return UTF-8...
(This is why I've just been telling everyone to convert because it works, consistently, on newly posted items. No matter the browser.)
Quote from: MrPhil on May 27, 2012, 08:04:38 PM
This is just a test -- was there more to this line? Was this line cut and pasted directly from Word? Was it cut off at a quotation mark? Something very strange is happening on your system compared to others. You're only the second one to try my fix for Smart Quotes and report back (the other one was successful). Now, once you paste Word text with Smart Quotes into another forum or something, who knows what happens to the Smart Quotes. They may or may not be converted to something else at that point. The only other suggestion would be to go ahead and try converting to UTF-8, and maybe you'll be lucky.
It would be unfortunate to lose you, but if you're confident that another forum can properly handle your requirements (especially cut and pasted Smart Quotes), maybe that would be the necessary impetus to get the SMF developers off their duffs and fixing this thing right. Of course, for any forum software, a lot depends on what your particular browser is doing with these pasted Smart Quotes characters on input and in what form they survive their round trip to the database. For example, Firefox 12 on a UTF-8 page will apparently convert Smart Quotes to the equivalent UTF-8 characters, but will IE7 (say) do the same? I'm not even going to ask you to look inside your database and try to find the records for your test posts and see if they are already cut off in the database.
Good luck and best wishes with your forum work, and I hope you find something that works for you.
This is a test "just a test" was typed in word, copied and pasted into the forum and the result was:
This is just a test
the remaining text starting with the quote disappeared. Remember this was only a test at your request. This was first noted after a member copied an article from the original forum and pasted it in the new forum. But I tested this by coping articles from various sites and every time it's cut off at the " or '......... of course I delete the article as I'll have no part of of plagiarism. It was just a test and I'm deeply disappointed.
So be it then, I'll research how to convert after I return from church tonight. If that doesn't work then off to VB it is. Thanks agin
OK I'm going to convert in the morning,
It says: After converting your data and database to UTF-8, you will need UTF-8 compatible language files.
Where do I get UTF-8 compatible language files please?
http://download.simplemachines.org/?smflanguages;lang=english
Assuming you want english, other languages available. ;)
Thank you
After download the language package I noticed there is no read me file. I assume being the contents of the language pack is titled Theme I replace the whole them pack on the server, correct?
BTW, thanks for pointing out the language download, not used to a forum offering download so I didn't look thoroughly. The two mods I'm using came from the admin panel. Thank you for being patient with me.
You drop the files into the Themes area, so that all the language files end up in Themes/default/languages/
OK then, I open the downloaded theme folder and upload the contents inside that folder to the theme folder on the server, right?
Yup. That way all the files should go into the Themes/default/languages/ folder - you don't need to install them into every theme, just the default one.
Converted to UTF-8, uploaded English language pack, problem solved! Whoooweee! Thank you everyone. Looks like SMF and me will be friends for a while. Again thanks to all of you for helping me, especially MrPhil who spent many hours assisting. God Bless!
/me would note that that was his basic advice two weeks ago in this very thread.
Quote from: Arantor on May 28, 2012, 04:51:27 PM
/me would note that that was his basic advice two weeks ago in this very thread.
Thank you Arantor, you were correct, sorry I didn't listen. Was overwhelmed with new data that I've never seen before