Topic: Emojis are causing PMs to not send full content

[mark7144]

I've noticed that when you insert emojis (via phone keyboard) into a PM, everything after that point fails to reach the recipient. However, the full contents actually shows in the email notification they get.

Does anyone know how I can fix this?

[GigaWatt]

Probably a UTF-8 database problem. See if the content column in the private messages table is set as UTF-8.

I would advise you not to insert emojis in posts and PMs. I had a similar problem on my forum and I've asked the members (in a separate thread) not to insert emojis in posts, but to use the standard smileys available in the post editor.

Basically, emojis are some sort of unicode characters and I have no idea if they are UTF-8 standardized. They work on mobile devices, but they are not like normal smileys, like the ones on forums created by 2 or more ANSI characters. They are separate symbols.

[mark7144]

Thanks for replying.

So currently under smf_personal_messages, the 'body' collation is set as 'utf8_general_ci'. So to fix this do I simply change that to 'utf8mb4_unicode_ci'? Is that relatively safe to do?

[Illori]

are you using SMF 1.1.*? if so i would suggest you upgrade to SMF 2.0. i believe SMF 2.0 has support for emojis at least to the point of not breaking the post like you are seeing.

[mark7144]

I do want to upgrade but I'm holding back for now as it's such a daunting task. My website is heavily integrated using SSI code and my theme is basically custom - I made it mobile friendly myself which took a long time. Also, this particular forum of mine is past its peak now so I don't mind just trying to keep it chugging along.

[Illori]

do keep in mind that SMF 1.1.* is not getting security patches any longer.

[mark7144]

Yeah, I admit that's a little unsettling.

[GigaWatt]

So currently under smf_personal_messages, the 'body' collation is set as 'utf8_general_ci'. So to fix this do I simply change that to 'utf8mb4_unicode_ci'? Is that relatively safe to do?

I would advise you not to do that, you may break forum PMs (make characters question marks or unreadable characters). "utf8_general_ci" is the way to go. If you decide to do this, do a database backup first.

And I just noticed that this post is in the 1.1.x. As far as I know, SMF 1.1.x doesn't officially support UTF-8 out of the box (someone correct me if I'm wrong). There are "hacks" that make 1.1.x UTF-8 compatible, but I wouldn't go further than "utf-8_general_ci".

Please post results if you happen to try it and it works .

[mark7144]

It was already set as utf8_general_ci originally. Did you mean to say that?

Anyway, I actually tried changing it to 'utf8mb4_unicode_ci' before I read your post. And while emojis only show as question marks, it did resolve the major flaw of missing private message content. So far not noticing any other issues thankfully.

Please let me know your thoughts?

[GigaWatt]

It was already set as utf8_general_ci originally. Did you mean to say that?

No, I meant if you set the column from "utf-8_general_ci" to "utf8mb4_unicode_ci", post results .

As for the other part of your post... good to know . I wouldn't set this collation as this might cause problems in future versions of SMF and database conversions from one version to another, but it's your forum, your choice .

If there was a way to block emojis from mobile devices, I would implement it on my forum, but as far as I know, there isn't one, so I just hope the members obey the "no emoji request" i posted in a thread in their PMs. As for the public part of the forum, I usually do manual edits, although I haven't done one in a while, not since I posted the request.

Just to be clear, there are no problems with emojis on SMF 2.0.x. On some browsers they load, on others, they don't (I think PPAPI browsers loaded them correctly, NPAPI didn't... well, not all of them, I usually use Pale Moon, a Firefox fork and it loaded the emojis correctly), but the topics and posts are accessible and readable. Though, I have to admit, I haven't looked in the error log :S.

So, the bottom line is, I asked the members not to use emojis as a precaution. I don't know what decisions will be made in future versions of server OSes, database managers, SMF, PHP, etc. Most likely, things will go in an upward direction (will support these symbols out of the box in future versions), but better safe than sorry . Besides, it's not such a big deal, they could always use the ANSI combinations to generate smileys. Sure, it's a bit more complicated, but hey... that's life .

[mark7144]

Ok great. Thanks for your help here

[GigaWatt]

No problem .

And don't forget to mark the topic as solved .