News:

SMF 2.1.4 has been released! Take it for a spin! Read more.

Main Menu

Posts truncated after a "bad" character

Started by Dave En Velo, January 16, 2017, 10:49:53 AM

Previous topic - Next topic

Dave En Velo

I am a new user of SMF with a tiny base of users.

We don't understand why there is not some kind of workaround for the problem of posts being truncated when the text is pasted in from Word or other applications under Windows.  We have good looking text in Word and when we cut and paste into the Post box it all looks good, even on preview.  However, if a quote or double quote is in the text then, when we come to view the Post the quote and everything after it has not been Posted.

I have looked through this forum and the general advice seems to be to remove the quote and double quote yourself.  This seems a rather error prone activity.  Are there software architectural reasons why any dodgy characters cannot be filtered out?  We would be happy with simple replacement of bad characters with ?, anything is better than Posts occasionally getting truncated and you only knowing when a reader Posts that he doesn't understand your topic.

I had hoped there would be some kind of enhancement to remedy this issue but if there is I cannot find it.

Thanks for you help.

PS I am using Windows 7 and the problem seems to be the same under Chrome, IE and Firefox.   I am using a Bitnami stack of SMF 2.0.12 running in Google Cloud.  This works great.


Dave En Velo

OK this explains why the problem occurs and I had largely understood that already.  What I do not understand is why there cannot be a software solution within SMF.  My users think it is pretty naff.  To them SMF seems quite able to throw the bad characters back in Preview but is unable to Post them.  They tell me it is a bug.

Illori

if you convert to UTF-8 as suggested in that page, it will resolve the issue.

Arantor

Quote from: Dave En Velo on January 16, 2017, 01:00:36 PM
OK this explains why the problem occurs and I had largely understood that already.  What I do not understand is why there cannot be a software solution within SMF.  My users think it is pretty naff.  To them SMF seems quite able to throw the bad characters back in Preview but is unable to Post them.  They tell me it is a bug.

We did fix this permanently in the next version. (2.1.0)

For the interim the issue is actually your database chopping off the data because it thinks it's invalid because Word quotes are actually invalid ISO 8859-1 characters. This is actually a Microsoft problem that everyone else just has to deal with. This is why it works in the preview (because the browser silently fixes it, in violation of HTTP spec, and in violation of what SMF instructs it to do) but not when it gets saved.

Convert to UTF-8 and it will go away. Ideally it should have been installed with UTF-8 in the first place but I can see that didn't happen.

-- former member of development team

Kindred

in other words... it's not actually an SMF problem in the first place...   but there is already a software solution (convert to UTF-8). :D
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

Dave En Velo

OK thanks for the rapid support, I think I understand the issues.  My forums are relatively short lived (a few months) and I start new ones as required so I can easily move to a new release or upgrade to UTF-8 support. 

I was going to ask Bitnami to use UTF-8 in their stack but I see, in their support forum, that they have already addressed this issue (perhaps I should have looked there first). They document the process of Forum Maintenance and converting the database to UTF-8.  It all seems pretty straightforward.  Indeed I have just spun up a test version in Google Cloud, converted the database without any downtime and the problem seems resolved.

My users felt it was a bug since they could do the same posting in phpBB and it worked fine.  I suppose the real message here is that UTF-8 support is pretty essential if your users are likely to be using Word and cutting and pasteing into SMF Posts.

Thanks for all the help.

Arantor

We know this, which is why 2.1 forces UTF-8 as opposed to 2.0 giving users the choice ;)

Kindred

Using Word is pretty much always a mistake anyway....

the junk code generated by pasting form word into a webpage with wysiwyg is stupendous...
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

Advertisement: