Post not complete after hitting POST Button

[MrPhil]

The Subs.php you sent me had been modified for other mods, but you had not made any of the changes for Smart Quotes. Attached is a revised copy. Keep a copy of the existing one just in case...

P.S. I had an editor warning that it might have changed the "end of line" convention. Check that the file looks OK when you upload it. If not, you might have to upload in binary instead of ASCII or vice-versa.

[Johnny B]

OH lord, not sure what happened. I'm somewhat new at this, maybe I got confused and uploaded wrong file. Please forgive me. However, I did download your file and uploaded it to the sources folder. NO Work. I tried copy and past an article, all words will display until it gets to those "/')(& characters which is where the article gets cut off and nothing beyond the character will display.

P.S. I did check it and all looked normal

[MrPhil]

Groan. Attach Subs.php (the right one!) to your next post and I'll look at it.

[Johnny B]

OK, thank you sir

[MrPhil]

Well, I'm not seeing any important difference between yours and mine, certainly nothing that should cause it to fail. You're absolutely sure you have this code in .../Sources/Subs.php? You have only one copy of your forum (no chance of looking in the wrong forum)? Your forum is in UTF-8 or Latin-1 encoding (which)? Your "bad" text causing the failure was cut and pasted in from Word, and has "Smart Quotes" characters like the nice quotation marks and such? Look at the page source (View > Page Source)... have any of the bad characters been changed to HTML entities such as «? If so, my code is working. Is there any chance that there are other invalid characters in the cut and pasted text? Not just Smart Quotes, but other valid Latin-1 but invalid UTF-8 text (if your forum is UTF-8).

[Johnny B]

1. Yes sir, I use filezilla and I'm sure the Subs.php file is being dropped in the sources folder.

2. I only have this one forum, no others

3. After viewing page source I identified two places where this appears in red:  

4. Is there any chance that there are other invalid characters in the cut and pasted text?
     just the standard " or ' or ( or ) or/  seems to be the ones that cause this.

5. Not just Smart Quotes, but other valid Latin-1 but invalid UTF-8 text (if your forum is UTF-8).  All I noticed thus far were the characters   above

[MrPhil]

So characters like " and ' (Smart Quotes versions, not the standard ASCII) are not being turned into HTML entities like »? Once again, is your forum UTF-8 or Latin-1? Did you clear your SMF cache (erase all .php files from cache directory) and clear your browser cache after uploading the Subs.php code? Did you make sure the capitalization wasn't messed up -- you didn't add it back to your site as subs.php, did you?

[Johnny B]

1. I don't know if my SMF forum is UTF-8 or Latin-1. Been searching to determine but nothing yet. I just did the install and it is what it is, sorry. Still looking through Goggle and SMF forums.

2. I did clear browser cache twice and nothing.

3. I also cleared my SMF cache

4. So characters like " and ' (Smart Quotes versions, not the standard ASCII) are not being turned into HTML entities like »?
All I know is whenever you paste an article with " or ' or - characters in the body the character used and everything beyond that point will not display. The full article will preview nicely but will not post properly.

5. Did you make sure the capitalization wasn't messed up -- you didn't add it back to your site as subs.php, did you?
What do you mean capitalization wasn't messed up? Sorry but I don't understand

[MrPhil]

1. In your browser, View > Character Encoding (or similar wording, possibly Character Set or Charset). It will tell you what your forum is requesting. For example, this forum (simplemachines.org) shows up as "UTF-8".

4. This is text from Microsoft Word? The quotes are the curved kind, looking like 6's and 9's, not the straight ' and "? Those are "Smart Quotes", which will wreck your forum display, especially if your forum is UTF-8.

5. The file is Subs.php with a capital S. I was asking you to check whether you made a mistake and put it in the Sources/ directory as subs.php (small s). If you did, it's still using the old unmodified Subs.php and your updated subs.php is being ignored.

[Johnny B]

1. Western (150-8859-1)

4. They appear to be the straight kind, eyes are going bad but they sure look straight to me.

5. Yes I checked that in the beginning, all is good

[MrPhil]

OK, so you're still at Latin-1. The only text which is giving problems is cut and pasted from a Word document? If so, could you paste a small sample into your next post here so we can see if it's using Smart Quotes? Sometimes the curves are very subtle. I think this forum will be able to display them, but check the results anyway.

What browser are you using (make and version)? Many browsers these days actually use CP-1252 when Latin-1 is requested -- maybe yours isn't, as is choking on what it thinks are control codes.

[Johnny B]

Edit post

Browser is Firefox 12.0 the latest ver

[MrPhil]

OK, yes those are curved quotes and apostrophes, so they're Smart Quotes. Did you cut and paste directly from Word into simplemachines.org, without going through any intermediate steps? If the last Subs.php I sent you is in the right place under the right name, and the database and pages are displayed in Latin-1/ISO-8859-1 (and that's what you requested), then I don't know what's wrong with your system. I take it you did not end up trying to convert to UTF-8? You were asking about how to do it.

[Johnny B]

I did not convert or attempt to do so.

Did you cut and paste directly from Word into simplemachines.org, without going through any intermediate steps?
I cut and paste from an old non smf forum we had to our new smf forum directly. Even if I paste to notepad before the smf forum the results will be the same.

yes the Sub.php file is in the right place for sure

[MrPhil]

If the text went through another forum or web application before arriving at SMF, who knows what happened to the original Smart Quotes! My changes to Subs.php are assuming actual Smart Quotes byte codes, and I suspect yours may have already been changed to something else. If they were changed to UTF-8, you should be seeing multiple (usually 3) funny accented characters for each Smart Quote. There's a chance that they are UTF-8, but one of the bytes is still a control code that is killing the display. If the control code is the second or third byte, I would still expect to see a funny accented character before the text is cut off. Was there, and you edited it out?

Have you tried cutting and pasting text directly from Word (with Smart Quotes) into SMF? Does it work then? If it does, is the source of your problem text Word or another forum? If it's not Word, my fix may not work.

[Johnny B]

No funny code at all appeared, just cut off.

I typed this in word: This is just a test just a "test"

copied and paste it in the forum and got this:

This is just a test just a

[MrPhil]

OK, please upload the attached copy of Subs.php. It contains a few lines to dump out the bytes in your input (raw) message so I can see what's going on. Please repeat your "just a test" test with it. You shouldn't see any change to the display, but if you look at the page source (HTML) you should see a comment <!-- fix_SmartQuotes: ... followed by whether or not it's UTF-8 and all the byte values. Please cut and paste that line into your next post. After that, you can restore your system by editing out all the lines between (and including) // =================, or just restoring an earlier copy of the file.

[Johnny B]

OK sir, this is what I saw in two locations, the first was located at the top of the page source and very first line and the second was about half way down

<!-- fix_SmartQuotes: is NOT UTF-8 encoding, bytes: 4D 45 42 41 20 4D 65 6D 62 65 72 73 20 6D 75 73 74 20 73 69 67 6E 20 69 6E 20 74 6F 20 76 69 65 77 20 61 6C 6C 20 62 6F 61 72 64 73 -->

<!-- fix_SmartQuotes: is NOT UTF-8 encoding, bytes: 54 68 69 73 20 69 73 20 6A 75 73 74 20 61 20 74 65 73 74 20 -->

[MrPhil]

Is the second one where it fails and gets cut off? The first line says "MEBA Members must sign in to view all boards" and the second line says "This is just a test ". Was this second one cut off at a Smart Quote? What was its history -- was it cut and pasted directly from Word? It looks like it got cut off either while going into the database or being retrieved from it. It didn't survive to get to the point where my routine could work on it!

Well, at this point, you certainly could try converting your forum to UTF-8, and hope for the best. I know that some browsers (no guarantee that all) will convert Smart Quotes on the fly to UTF-8 characters. Maybe that will work. I would leave in the fix_SmartQuotes patch I gave you just in case some Smart Quotes in existing posts survive the conversion process and end up being sent to the browser. You can either revert to your previous copy (without the hex dump) or edit the file to remove // ===== and everything in between those two lines.

Note that writing a post in Word, only for the purpose of cutting and pasting it to SMF, is kind of pointless. All the formatting is destroyed anyway, and needs to be put back in. The only reason would be that you're going to keep a Word document with that content, anyway. If you just want to paste the same text in multiple forums, I would write it in Notepad so that it's plain text (no formatting, no Smart Quotes).

[Johnny B]

Is the second one where it fails and gets cut off?  I've lost you on this one, no idea what second one you are talking about, sorry.

At this point I have no idea where the original article was created or created on. All I know is when a member copies an article from my old forum to this one where ever there is a ' or " character it will not display anything from there forward.

As for: This is a test "just a test"

That was done in word as you asked me too, copied and pasted in the forum resulting in the exact same behavior.

OK, this isn't good, not much of a forum if members can't copy and paste an article without most of the article missing, beginning to wish I had invested the money into vBulletin.  Thank you for your help, you really reached out far beyond what most folks would have. Very kind of you sir. If I can ever repay the favor please send a PM and I'll do my best to assist you.  Off to research on how to convert this thing over as that seems to be the last stop before I ditch this thing for vBulletin. Thank you again and God Bless