Hi
I am having issues with iPhone smileys on my forum, whenever a user inserts a iPhone smiley the post cuts off... So the first iPhone smiley basically decides the length of the post. Users report having this issue both when using Tapatalk and when posting directly on the website.
I have googled, and searched this forum without much luck. Can anyone point me in the right direction? Debugging is made more difficult by the fact that I do not have a iPhone... But any pointers on what might cause posts to cut is greatly appreciated.
I am using SMF 2.0.8, with Norwegian UTF-8 language.
Thank you.
Link to an example where this is happening?
Quote from: ‽ on July 30, 2014, 05:43:27 AM
Link to an example where this is happening?
Thank you your quick response. Here are some examples:
<links removed>
There are no strange symbols in the posts, they just cut. I've checked the database, and it doesn't contain the rest of the text either.
What's the collation of the body column in smf_messages?
utf8_general_ci
Interesting. I was going to suspect a mismatch between database encoding and everything else but you're UTF-8 through and through.
This may well prove to be a bug in SMF itself (since the posting routine predates the iPhone!)
That was my initial suspicion as well, but I didn't find any. If it is indeed a bug in the SMF system I find it strange that I can't seem to find anyone else with the same problem. I'm guessing a lot of people use their iPhone when posting.
I do know that whenever someone sends me iPhone smileys my BlackBerry doesn't understand any of them and just prints squares.
Any hints to what source file I can start digging in? Or where I go from here?
BTW: The forum is normally closed for guests, i opened it temporary. Can I close it again?
Sure, you can definitely close it again.
The thing about such emoji is that they are special characters and something doesn't like them. I've not seen them used here, though.
What mods do you have installed?
My mods:
Tapatalk SMF 2.0 Plugin 4.0.1
SMF Gallery Lite 3.7
SimplePortal 2.3.5
Theme:
Blue Star Theme
In addition to that I have written a custom scheduled task and action feature.
are you posting form your iphone to your site through tapatalk or directly?
(and, either way, the reason it might not have been noticed is that this is the first time I have heard of someone using the iphone emoji rather than the standard smilies)
Quote from: Kindred on July 30, 2014, 07:16:11 AM
are you posting form your iphone to your site through tapatalk or directly?
(and, either way, the reason it might not have been noticed is that this is the first time I have heard of someone using the iphone emoji rather than the standard smilies)
Hi, I have not posted myself. But I've had a couple of members that have experienced the error, and they have tested and confirmed that it indeed was the iPhone smileys that caused the post to cut. One member tried posting both through Tapatalk and directly, with the same result. The post cut off at the first iPhone smiley.
It should be very easy to replicate, but I don't have, or have access to an iPhone... I had one of the members send the iPhone emotions to me on e-mail, and they show up as black squares in my webmail, like this:
Hm, that was interesting! When I tried to copy and paste the black squares from my webmail I got the follow error message when previewing this message:
The following error or errors occurred while posting this message: The message body was left empty. And all my text duplicated and screwed up the entire entry.
I am going to try and get the HTML code for the symbol and add it to the censored words. Let you know how that goes.
well, that is the point... I don't think that iPhone emojis have an html code...
wow... actually, I see LOTS of articles on going the other way - converting html to emoji, but - with albeit a quick search, nothing about converting iphone emoji back to html.
edit - found one..
http://www.iamcal.com/emoji-in-web-apps/
and, from this article it looks like it is a definite bother and would require new code to handle...
blech.
I hate apple.
You are right, gawd damn Apple...
Thank you for the article, I think I'm going to try and test with implementing "Emoji for PHP" solution in my posting source. But it seems like more of a work around than a solution. Maybe a more permanent solution to this is something to consider for future SMF versions? Seems like the use of emojis isn't going to decline any time soon.
On second thought, I think I'm going to try and filter them out and remove them instead. Supporting them is not really important, but I can't have them cutting my members posts. That gets a bit frustrating.
I'm thinking a regex replace on the submitted post title and body. Any ideas and pointers would be greatly appreciated, and thank you for pointing me in the right direction :-)
Okey, it seems that I can use http://htmlpurifier.org/ to clear out the unwanted symbols. But I am having some problems locating where the text is actually saved, I tried applying it to the beginning of function Post2 in Sources/Post.php, but this only seems to apply to the preview...Any ideas?
touchpal keyboard on android has emoji's
i can enter them just fine, post the message and the message seems to be 100% empty as there was no other text in it.
phpmyadmin has nothing as the message body at all. [tested with current 2.1 which is utf-8 by default]
I'm posting from iPhone and this is an emoji
Well, it seems real :o
I posted a bigger message with more text and emojis...
Quote from: margarett on July 30, 2014, 01:50:11 PM
Well, it seems real :o
I posted a bigger message with more text and emojis...
yep tested again with android/touchpal and can duplicate the same with 2.1. i just think that utf-8 cant handle the characters so it removes them?
So I have added this to the beginning of the Post2 function in Source/Post.php:
require_once '/path/to/library/HTMLPurifier.auto.php';
$config = HTMLPurifier_Config::createDefault();
$purifier = new HTMLPurifier($config);
$_REQUEST['message'] = $purifier->purify($_REQUEST['message']);
Right after:
// No need!
$context['robot_no_index'] = true;
And I no longer get an error when previewing a post with emojis, they are just removed and all text is kept. But no change when I post, then all text after the first emojis is lost. How can I also apply this filter when posting? I've read trough the function but am no closer to understanding why...
The truncation happens all the same on my setup and posting from an iPad (so it's the iOS rather than the iPhone alone). The lost text seems to include the space preceding the first emoticon encountered.
OK, so I'll do some analysis on this later tonight when I'm back at home, and I'll pass the details to the team to incorporate into the next patch since this strikes me as a fairly important detail.
I would not blindly throw things like HTML Purifier at it since it neuters other things that you might legitimately want to add as an admin.
There's a nice conversion table located here (http://apps.timwhitlock.info/emoji/tables/unicode) that could help.
I don't get why those are futzing around with it though, they're not high enough to be pushing UTF-8mb4 as far as I can see.
Need to sit and watch this one with a packet sniffer to get actual traffic and a bytesafe catcher during post processing on the server to see what is going on - but I'm not at home and don't have any iDevices with me.
I'm not sure why SMF is truncating the emoji characters and anything after it, but I do know that iOS uses a specific font that is pre-installed on the device to display and utilize the emojis.
Once the issue with the message being truncated is fixed, in order to display them properly, a third-party font must be used. There are a few that can be found on GitHub.
It's truncating it because it doesn't like it. It smells like an encoding issue since SMF is jittery around invalid codes for crap like that, which is why I want to apply a packet sniffer and watch what physical bytes are sent and exactly why SMF is choking on it.
As for display, that's a separate ball game since for iOS users it'll already be legitimate. Right now I'm simply interested in why it's choking.
It would be nice to display them correctly, but the most pressing issue is not cutting of the posts. I've had a few users who have had their whole introduction post lost because they started it with "Hi (emoji)". Not a good first impression to give to new users.
As for this bug/issue should I file it in the bug system?
Don't worry, I got this one. I'll report it on the official tracker once I have more information.
Quote from: ‽ on July 30, 2014, 05:57:30 PM
Don't worry, I got this one. I'll report it on the official tracker once I have more information.
Thank you all for your attention on this :)
Any news on this?
Sorry, I forgot to mention about this.
I worked out what it was. Fixing it is, however, a much larger problem.
Technical explanation: most normal UTF-8 stuff we deal with is 3-byte, the iPhone emoji are 4 byte. Standard UTF-8 doesn't cover 4 byte encoding, we have to specifically use a thing called utf8mb4.
Now, if all the database were converted to utf8mb4 everywhere, and the set-names query updated to set names to utf8mb4, there's a reasonable chance it would work as expected.
Part of me is wondering if converting it to an entity form before saving it to the database (to avoid all of the above) but that fall foul of other checks and balances inside SMF.
Thank you the explanation :) I did have some luck using an HTML-purifier, however I did not manage to implement it correctly. You mentioned that it might also clean out some stuff that the admin wanted to insert, which is a very good point. What are your thoughts about writing a "SMF-complaint" "cleaner" to at least stop the posts from getting cut?
I've tried sending these symbols to my outlook.com mailbox, it does not display the symbol correctly however it does not cut the post. It instead shows a unfilled black square. Same symbol I saw on my old Blackberry whenever someone would send me some of these emotions.
There's already a cleaner involved in the bowels of SMF, you do not need to implement your own cleaner to validate HTML or anything. In any case if some kind of sanitiser is fixing this, I'd be tempted to call it a bug because it shouldn't be pruning the characters.
MySQL only does because 4-byte UTF-8 characters are not supported in the standard UTF-8 character set encoding or collation.
An unfilled black square indicates a valid character with no glyph available to render the character, but that it has correctly acknowledged the character as it should.
I read a couple of posts on MySQL and utf8mb4, I think I'm going to do some experimentation on my virtual test setup. I found an article explaining the conversion from utf8 to utf8mb4, and it seems like quite a tedious task. http://mathiasbynens.be/notes/mysql-utf8mb4
I'll setup a test bed with SMF, MySQL (or MariaDB (https://mariadb.com/kb/en/mariadb/mariadb-vs-mysql-compatibility/) as I am currently using) with utf8mb4 and do some testing :)
You will need to modify Load.php as a minimum to update the SET NAMES query. Everything else... off hand no idea.
Quote from: ‽ on August 05, 2014, 05:54:33 AM
You will need to modify Load.php as a minimum to update the SET NAMES query. Everything else... off hand no idea.
Thanks for the hint, I rather enjoy digging in PHP code :) So it will be a fun thing to play with after the wife and kids have gone to bed and I've grabbed myself a beer. Let you know how it goes when I have found the time for it :)
Have fun :) The big problem is that there's all kinds of edge cases with the UTF-8 handling that people may not remember or be aware. Let me know how you get on :)
If it helps I've experienced the exact same issue, but only on my SMF 2.0.8 test installation. My (albeit heavily modified) SMF 2.0 RC3 installation works absolutely fine with iPhone emoji :-\
Interesting, since that code has not changed since then... I do know a future-proof patch has been discussed by developers however.
Meanwhile there are *dozens* of security holes that have been fixed since RC3. You really should investigate upgrading.
Quote from: Arantor on September 03, 2014, 06:16:04 PM
Meanwhile there are *dozens* of security holes that have been fixed since RC3. You really should investigate upgrading.
Thanks, yes I've been pulling in a number of the changes, but the code in my site is so heavily modified that it would take me a few days to sort it all out. It's non-public facing though, so less of a concern! :)
I think that changing the type of the body column in the smf_messages table to a BLOB would allow the database to store the iPhone emoticons correctly (depending on your database setup). This may not play nicely with the search functionality and I don't recommend messing around unless you really know what you're doing. Best to wait for an official patch for this one I reckon!
It would but it would also have other side effects too.
It would break search for two of the kinds of search available. Presumably of course one would be in UTF-8 mode already but if not, it would break the convert-to-UTF-8 system too.
Converting internally to UTF8mb4 (and adjusting certain other calls) would fix the worst of this but that requires MySQL 5.5.3 which is quite a steep requirement in practice.
As I said, I know there's been some discussion in the team about it and I believe one of the developers has a patch that needs more testing but I gather it does solve the problem.
Quote from: Arantor on September 04, 2014, 06:48:19 AM
It would but it would also have other side effects too.
It would break search for two of the kinds of search available. Presumably of course one would be in UTF-8 mode already but if not, it would break the convert-to-UTF-8 system too.
Converting internally to UTF8mb4 (and adjusting certain other calls) would fix the worst of this but that requires MySQL 5.5.3 which is quite a steep requirement in practice.
As I said, I know there's been some discussion in the team about it and I believe one of the developers has a patch that needs more testing but I gather it does solve the problem.
That sounds perfect to me. Thanks for the info Arantor!
Sorry to resurrect an old post, but I've been unable to find this mentioned in any newer posts.. I've just recently installed 2.0.17, one of the admins uses iPhone almost exclusively... and this still seems to be an issue....
I hate to migrate to a release candidate, but perhaps this has been fixed there?
After reading this thread I've seen no solid fix, but a mention that some other dev might have a patch. If there's a 'good' way to fix this in 2.0.17 I'll take a shot at creating a Mod for it...
UPDATE: I thought I had set everything to UTF-8 however the 'convert to UTF-8' option was still available. Appears my data was not UTF-8. Hopefully that will resolve the issue...
Converting to utf8 solves the problem.
Have you tried that?
Thanks... I noticed that I wasn't fully UTF-8 (updated my other post with that)... I am now, waiting for confirmation of the fix from the other admin as I don't have an iOS device.
I'll post results, at least so the next person that comes across this knows the fix...
Just posting a follow up. Fully migrating to UTF8 fixed the main issue (truncating). Emoji's from iPhone keyboard don't translate great, but that's a whole nother issue.
Quote from: braindead0 on January 13, 2021, 08:26:08 AM
Just posting a follow up. Fully migrating to UTF8 fixed the main issue (truncating). Emoji's from iPhone keyboard don't translate great, but that's a whole nother issue.
there is a post that helps with that, though it needs updated but it is a start in the right direction
https://www.simplemachines.org/community/?topic=544488.0
it halfway works to translate emojis
eventually it'll annoy me enough and I'll see how to fix it :laugh:
Although dull, the truth is they are just characters. Most browsers support all of them, but some might be uglier than others....
With the right virtual keyboard/character selector, you can properly use & display all of them already.
A sampler:
😁😂😍😴🙅🚀🚂🚕🚽🛀🛌🛵
To the browser, they are characters, no different from any of these:
ᚠᛇᚻ᛫ᛒᛦᚦ᛫ᚠᚱᚩᚠᚢᚱ᛫ᚠᛁᚱᚪ᛫ᚷᛖᚻᚹᛦᛚᚳᚢᛗ
ಬಾ ಇಲ್ಲಿ ಸಂಭವಿಸು ಇಂದೆನ್ನ ಹೃದಯದಲಿ
Hello there!
⠊⠀⠉⠁⠝⠀⠑⠁⠞⠀⠛⠇⠁⠎⠎⠀⠁⠝⠙⠀⠊⠞⠀⠙⠕⠑⠎⠝⠞⠀⠓⠥⠗⠞⠀⠍⠑
আমি কাঁচ খেতে পারি, তাতে আমার কোনো ক্ষতি হয় না।
ကျွန်တော် ကျွန်မ မှန်စားနိုင်တယ်။ ၎င်းကြောင့် ထိခိုက်မှုမရှိပါ။
Կրնամ ապակի ուտել և ինծի անհանգիստ չըներ
𐌼𐌰𐌲 𐌲𐌻𐌴𐍃 𐌹̈𐍄𐌰𐌽, 𐌽𐌹 𐌼𐌹𐍃 𐍅𐌿 𐌽𐌳𐌰𐌽 𐌱𐍂𐌹𐌲𐌲𐌹𐌸
ᛖᚴ ᚷᛖᛏ ᛖᛏᛁ ᚧ ᚷᛚᛖᚱ ᛘᚾ ᚦᛖᛋᛋ ᚨᚧ ᚡᛖ ᚱᚧᚨ ᛋᚨᚱ
Few browsers display the black mini-versions of emojis anymore. The only one that does this still, I think, is IE.
hm....so essentially you could add them like smileys with the character codes ???
Yep.
On Windows 10, press & hold the Win key plus a period... Win-.
holy crap I didn't know about that
stop doing magic :laugh:
(Please keep it a secret, or GigaWatt will be using them everywhere... ::))
Okay, even I didn't know this one 🤷♂️ And I thought I had Win 10 pretty well figured out.
Though, I've never been one to use emojis much.
Quote from: shawnb61 on January 13, 2021, 02:37:23 PM
(Please keep it a secret, or GigaWatt will be using them everywhere... ::))
;D ;D ;D ;D ;D ;D ;D ;D ;D ;D ;D ;D
(https://i.imgur.com/B9WHVla.gif)
Not to bust your bubble, but I don't post emojis anywhere since they're not actually a part of the reduced UTF-8 character set in MySQL, so, from what I know, they're encoded as HTML in the DB. I don't even allow them in my forum, LOL :D. Not planning on breaking something. Will wait for 2.1 and full emoji support ;).
And for the record Shawn, it doesn't work on Server 2019 / LTSC 2019 ;D ;D ;D.
Quote from: GigaWatt on January 13, 2021, 04:38:45 PM
Not to bust your bubble, but I don't post emojis anywhere since they're not actually a part of the reduced UTF-8 character set in MySQL, so, from what I know, they're encoded as HTML in the DB. I don't even allow them in my forum, LOL :D. Not planning on breaking something. Will wait for 2.1 and full emoji support ;).
You are correct, all 4-byte characters, such as emojis & some CJK & some ancient languages & symbols, are encoded as html in mysql when using a utf8 charaset. Not sure that's a reason not to use them, though. They still work & are displayed fine - as in this thread 🧵. The only exception is they're not really searchable in 2.0. (Yes, they are searchable in 2.1.)
2.0 & 2.1 both default to using the same utf8 set. A big part of the problem is that real utf8 support came much later, in mysql 5.5, dubbed "utf8mb4". SMF still claims support of mysql 4.1 & greater.
https://mysqlserverteam.com/mysql-8-0-when-to-use-utf8mb3-over-utf8mb4/
https://wiki.simplemachines.org/smf/SMF2.0:Requirements_and_recommendations
Quote from: GigaWatt on January 13, 2021, 04:38:45 PM
And for the record Shawn, it doesn't work on Server 2019 / LTSC 2019 ;D ;D ;D.
Interesting. I knew Server was a subset of Win10, intended for rack-mount usage, not desktop usage, so yeah, I guess it makes sense some of the fun user things don't work there...
Quote from: shawnb61 on January 13, 2021, 04:59:29 PM
They still work & are displayed fine - as in this thread 🧵.
Was thinking more as in a "better safe than sorry" kind of a way... in case the 2.1 upgrader just won't budge unless I clear the DB of all of the emojis... not a likely scenario, but still, I'd like to be on the safe side. I already went through hell when upgrading from 1.1.x to 2.0 because all of the custom stuff the previous admin did... I was stuck with trial and error for months :(. The fact that I've never done anything like that before didn't help, and now, I'd probably do it in like a day or two, but still, I get nervous when it comes to upgrades, LOL :D.
I'm still on utf8_general_ci and I really haven't looked into cPanel to see what version of MySQL is running and if it supports utf8mb4... I'll eventually get to it, when 2.1 is final and I can finally prepare myself mentally for the upgrade :D.
Quote from: shawnb61 on January 13, 2021, 04:59:29 PM
Interesting. I knew Server was a subset of Win10, intended for rack-mount usage, not desktop usage, so yeah, I guess it makes sense some of the fun user things don't work there...
I use it as a desktop OS :P :D. The kernel is more stable than the regular releases and it doesn't have any apps from the AppStore, like CandyCrush ::). Disable some services intended primarily for server use and you've got a decent "almost bare bones" OS. I use either that or LTSC 2019, either one is fine, both have no apps out of the box (the server versions don't even support apps, but you can install the AppStore on LTSC 2019 if you like), which is actually exactly what I'm looking for ;).
Oh, and the best part, you can disable updates <3... i.e. you can set the OS to check for updates manually, both on LTSC 2019 and Server 2019 ;).
(https://i.imgur.com/ApntMb2.png)
Not sure you actually can, I'm under the understanding that some versions do allow you to set a long delay for updates, but unless you are running enterprise licensing you can't really stop them. ( Except, if you completely disable the update service... )
Quote from: Aleksi "Lex" Kilpinen on January 14, 2021, 06:15:44 AM
Not sure you actually can, I'm under the understanding that some versions do allow you to set a long delay for updates, but unless you are running enterprise licensing you can't really stop them.
Well... both Server 2019 and LTSC 2019 are Enterprise products :D. The full name of LTSC 2019 is Windows 10 Enterprise LTSC 2019. So yeah, you can disable them ;).
Quote from: Aleksi "Lex" Kilpinen on January 14, 2021, 06:15:44 AM
( Except, if you completely disable the update service... )
Doesn't work. There's a service that checks if the Windows Update service is running and enables it if it isn't ::)... and there's a bunch of other services that are no primarily meant to check if this "check if Windows Update is running" service is running, as well as the Windows Update service itself, but they do check if everything is OK in "Windows Updates land" ::), so... no, on anything other than LTSC 2019 or Server 2019, automatic updates are pretty hard to disable.
I've been using this tool for the past year or so to disable automatic updates on anything other than LTSC 2019 or Server 2019. So far, so good, does the job :).
https://www.sordum.org/9470/windows-update-blocker-v1-6/
Emojis do not & will not cause problems with the 2.1 upgrader. They're in html form: ascii text.
Quote from: shawnb61 on January 14, 2021, 11:46:53 AM
Emojis do not & will not cause problems with the 2.1 upgrader. They're in html form: ascii text.
Yeah, I've seen how they're stored in the DB.
Hm... you're right... I might think about letting users use emojis on the forum ;).