Advertisement:

Author Topic: Can't post unicode emoji  (Read 7208 times)

Offline Kolya

  • Full Member
  • ***
  • Posts: 409
  • systemshock.org
Can't post unicode emoji
« on: April 28, 2017, 06:53:45 PM »
When trying to post unicode emoji (as can be copied from here for example: http://getemoji.com/) the error is thrown:
Quote
The following error or errors occurred while posting this message:
The message body was left empty.

1. That error is incorrect
2. SMF supports unicode or does it?

It breaks in Post.php on validating the input:
Code: [Select]
if (htmltrim__recursive(htmlspecialchars__recursive($_REQUEST['message'])) == '')
$context['post_error']['no_message'] = true;

When this check is disabled, posting unicode emoji becomes possible.
However the post cannot be previewed which is a separate bug.

Offline Arantor

  • Resident Overthinker
  • SMF Friend
  • SMF Legend
  • *
  • Posts: 68,041
    • Arantor on GitHub
Re: Can't post unicode emoji
« Reply #1 on: April 28, 2017, 07:00:27 PM »
Version of SMF? Using UTF-8?

Emoji is tricky, but it should be supported even when your database probably doesn't (because MySQL's idea of UTF-8 usually doesn't cope with emoji, which is what the utf8mb4 mode is for, though SMF can support it without that)
To assume is to hope that those who came before had the presence of mind and capacity to implement the dreams of those who would come after.

You either die a hero or live long enough to see yourself become the villain. It seems you have chosen which, and now I must do the same.

Offline Kolya

  • Full Member
  • ***
  • Posts: 409
  • systemshock.org
Re: Can't post unicode emoji
« Reply #2 on: April 28, 2017, 07:08:05 PM »
Version: SMF 2.0.13
I've run the function to "Convert HTML-entities to UTF-8 characters" in the past, if that is what you mean.

//Reading further I found that I have this language installed: English   ISO-8859-1   385   en_US
//Installing "SMF 2.0.13 english british-utf8" now

Offline Gluz

  • Native Language Support Specialist
  • Sr. Member
  • *
  • Posts: 958
  • Gender: Male
Re: Can't post unicode emoji
« Reply #3 on: April 28, 2017, 08:58:52 PM »
That was patched in the 2.0.10 update, as can see in the Changelog:
Code: [Select]
SMF 2.0.10   April 22 2015
===============================================================================

March 2015
-------------------------------------------------------------------------------
 ! Forum Maintenance - Topics fails if header is collapsed
 ! Fix for unsupported UTF8mb4 characters

And as you can see in this post, just the preview is broken:
https://www.simplemachines.org/community/index.php?topic=537329.msg3818330#msg3818330

I remember another thread about something similar, and the team says that they fix the posting part, the unicode characters save correctly in the database, but somehow they forgot to look at the previews and that is still broken, not a major issue, but sometimes is kind of annoying that if you want to preview the post, you need to take out the unicode characters.


And now that I see the thead again, it seems that Windows 8.1 and 10 added support to emojis and it shows up as image instead as a simple vector like in Windows 7.


For your problem, maybe the collation of the database is not correctly set to utf8_general_ci or utf8mb4_general_ci as pointed out by Margarett.
Need help with PHP, HTML, CSS, Themes or MODs?

Quote
Think in spanish, habla en inglés.

Offline Arantor

  • Resident Overthinker
  • SMF Friend
  • SMF Legend
  • *
  • Posts: 68,041
    • Arantor on GitHub
Re: Can't post unicode emoji
« Reply #4 on: April 29, 2017, 02:30:17 AM »
The code explicitly requires the database not be mb4 by way of converting it to an entity before processing. I know, I'm the one who submitted that to the team. This is what the fix mb4 stuff does in Load.php where $smcFunc is defined.

Does it work as you'd expect here?
To assume is to hope that those who came before had the presence of mind and capacity to implement the dreams of those who would come after.

You either die a hero or live long enough to see yourself become the villain. It seems you have chosen which, and now I must do the same.

Offline Kolya

  • Full Member
  • ***
  • Posts: 409
  • systemshock.org
Re: Can't post unicode emoji
« Reply #5 on: April 29, 2017, 05:27:33 AM »
😛 Yeah, it works here as expected, including the preview.
What can I do now to get the same behavior in my forum? Should I set the database collation to utf8mb4_general_ci? Or install an UTF-8 language pack? Or both?

BTW, I'm writing a unicode emoji picker in Javascript. So I will have something to give back to the community.
Thanks for your help.

Offline Arantor

  • Resident Overthinker
  • SMF Friend
  • SMF Legend
  • *
  • Posts: 68,041
    • Arantor on GitHub
Re: Can't post unicode emoji
« Reply #6 on: April 29, 2017, 08:13:20 AM »
You do not need to change the collation at all.

If you're using the forum with a UTF-8 database but a non-UTF-8 language pack it could easily not work as expected.
To assume is to hope that those who came before had the presence of mind and capacity to implement the dreams of those who would come after.

You either die a hero or live long enough to see yourself become the villain. It seems you have chosen which, and now I must do the same.

Offline Kolya

  • Full Member
  • ***
  • Posts: 409
  • systemshock.org
Re: Can't post unicode emoji
« Reply #7 on: April 29, 2017, 11:45:13 AM »
I changed the language to "SMF 2.0.13 english british-utf8".
Then I looked at a database backup and it uses UTF8 as well.

Code: [Select]
DROP TABLE IF EXISTS `smf_messages`;
/*!40101 SET @saved_cs_client     = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `smf_messages` (
  `id_msg` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `id_topic` mediumint(8) unsigned NOT NULL DEFAULT '0',
  `id_board` smallint(5) unsigned NOT NULL DEFAULT '0',
  `poster_time` int(10) unsigned NOT NULL DEFAULT '0',
  `id_member` mediumint(8) unsigned NOT NULL DEFAULT '0',
  `id_msg_modified` int(10) unsigned NOT NULL DEFAULT '0',
  `subject` varchar(255) NOT NULL DEFAULT '',
  `poster_name` varchar(255) NOT NULL DEFAULT '',
  `poster_email` varchar(255) NOT NULL DEFAULT '',
  `poster_ip` varchar(255) NOT NULL DEFAULT '',
  `smileys_enabled` tinyint(4) NOT NULL DEFAULT '1',
  `modified_time` int(10) unsigned NOT NULL DEFAULT '0',
  `modified_name` varchar(255) NOT NULL DEFAULT '',
  `body` mediumtext NOT NULL,
  `icon` varchar(16) NOT NULL DEFAULT 'xx',
  `approved` tinyint(3) NOT NULL DEFAULT '1',
  `edit_reason` tinytext NOT NULL,
  `show_thumbnail` int(1) NOT NULL,
  PRIMARY KEY (`id_msg`),
  UNIQUE KEY `ID_MEMBER` (`id_member`,`id_msg`),
  UNIQUE KEY `topic` (`id_topic`,`id_msg`),
  UNIQUE KEY `ID_BOARD` (`id_board`,`id_msg`),
  KEY `ID_TOPIC` (`id_topic`),
  KEY `participation` (`id_member`,`id_topic`),
  KEY `showPosts` (`id_member`,`id_board`),
  KEY `ipIndex` (`poster_ip`(15),`id_topic`),
  KEY `approved` (`approved`),
  KEY `id_member_msg` (`id_member`,`approved`,`id_msg`),
  KEY `current_topic` (`id_topic`,`id_msg`,`id_member`,`approved`),
  KEY `related_ip` (`id_member`,`poster_ip`,`id_msg`)
) ENGINE=MyISAM AUTO_INCREMENT=111424 DEFAULT CHARSET=utf8;
/*!40101 SET character_set_client = @saved_cs_client */;

Still no luck. The same emoji that work here produce the empty message error on my own board.
I'm running PHP Version 5.5.12 if that is relevant.

Offline Arantor

  • Resident Overthinker
  • SMF Friend
  • SMF Legend
  • *
  • Posts: 68,041
    • Arantor on GitHub
Re: Can't post unicode emoji
« Reply #8 on: April 29, 2017, 11:56:57 AM »
So you have UTF-8 language files and a UTF-8 database. What does SMF think it's using?
To assume is to hope that those who came before had the presence of mind and capacity to implement the dreams of those who would come after.

You either die a hero or live long enough to see yourself become the villain. It seems you have chosen which, and now I must do the same.

Offline Kolya

  • Full Member
  • ***
  • Posts: 409
  • systemshock.org
Re: Can't post unicode emoji
« Reply #9 on: April 29, 2017, 01:54:07 PM »
Is that  a rhetorical question? Where can I look that up?

The forum language is set to "English British (UTF-8)". The pages are delivered with UTF-8 encoding as well. And the detailed version check doesn't show any outdated forum files.

Offline Kolya

  • Full Member
  • ***
  • Posts: 409
  • systemshock.org
Re: Can't post unicode emoji
« Reply #10 on: April 29, 2017, 02:05:41 PM »
I just noticed that it works fine when entering emoji on the quick reply box. But not via the regular editor page. That throws the empty post error.
And you know what? It's the same problem here.

Try pasting the following string to the quick reply box and hit "Preview". Then hit "Preview" again on the editor page.

📗test

Offline Kolya

  • Full Member
  • ***
  • Posts: 409
  • systemshock.org
Re: Can't post unicode emoji
« Reply #11 on: May 06, 2017, 03:32:45 AM »
I don't want to be that triple posting guy, but it would be nice to get some confirmation that you guys acknowledged this as a bug (maybe created a ticket on whatever bugtracker you're using?).

To summarise: On  SMF 2.0.13 posting unicode emojis works from the quick reply (including preview) but not from the regular editor page, which throws an error that "The message body was left empty.".

Offline live627

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 5,265
  • Gender: Male
  • Cat: Destroy!
    • live627 on Facebook
    • live627 on GitHub
    • live627 on LinkedIn
    • @live627 on Twitter
    • livemods
Re: Can't post unicode emoji
« Reply #12 on: June 07, 2017, 09:09:56 PM »
🎂
Try not to become a man of success, but rather try to become a man of value.
- Albert Einstein

Offline live627

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 5,265
  • Gender: Male
  • Cat: Destroy!
    • live627 on Facebook
    • live627 on GitHub
    • live627 on LinkedIn
    • @live627 on Twitter
    • livemods
Re: Can't post unicode emoji
« Reply #13 on: June 07, 2017, 09:14:03 PM »
That w worked on the full editor. I'll need to look into this more later.
Try not to become a man of success, but rather try to become a man of value.
- Albert Einstein

Offline lurkalot

  • Sr. Member
  • ****
  • Posts: 978
  • Gender: Male
  • Tinyportal Support
    • guitaristguild on Facebook
    • Tinyportal on GitHub
    • @GuitaristGuild on Twitter
    • Guitarist Guild
Re: Can't post unicode emoji
« Reply #14 on: June 08, 2017, 03:26:06 AM »
This one doesn't 😜 It gives this error on preview here,

The following error(s) occurred while posting this message:

The message body has been left empty.

Doesn't seem to do it in SMF 2.0 beta 3 though.  The smiley is from the Emoji keyboard of an Apple iPad  Copy and paste it into a post here and try and preview it.  Or modify and preview it.

Offline Arantor

  • Resident Overthinker
  • SMF Friend
  • SMF Legend
  • *
  • Posts: 68,041
    • Arantor on GitHub
Re: Can't post unicode emoji
« Reply #15 on: June 08, 2017, 03:33:04 AM »
This should have been fixed in the week or so before 2.1 beta 3 landed.
To assume is to hope that those who came before had the presence of mind and capacity to implement the dreams of those who would come after.

You either die a hero or live long enough to see yourself become the villain. It seems you have chosen which, and now I must do the same.

Offline lurkalot

  • Sr. Member
  • ****
  • Posts: 978
  • Gender: Male
  • Tinyportal Support
    • guitaristguild on Facebook
    • Tinyportal on GitHub
    • @GuitaristGuild on Twitter
    • Guitarist Guild
Re: Can't post unicode emoji
« Reply #16 on: June 08, 2017, 03:45:00 AM »
This should have been fixed in the week or so before 2.1 beta 3 landed.

I was testing this in the test board here a couple of days ago, after someone posted one on one of my forums, and I immediately thought my forum had gone T/U after moving it to new hosting.  Then found it was causing problems here as well.   :)

I then tried it on 2.0 beta 3 and all seems fine on there, previewed ok.  It's just 2.0.xx

Offline live627

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 5,265
  • Gender: Male
  • Cat: Destroy!
    • live627 on Facebook
    • live627 on GitHub
    • live627 on LinkedIn
    • @live627 on Twitter
    • livemods
Re: Can't post unicode emoji
« Reply #17 on: June 08, 2017, 03:18:05 PM »
It works using Windows 10... I guess then different systems do it differently. I won't be able to fix it.
Try not to become a man of success, but rather try to become a man of value.
- Albert Einstein

Offline Kolya

  • Full Member
  • ***
  • Posts: 409
  • systemshock.org
Re: Can't post unicode emoji
« Reply #18 on: June 18, 2017, 02:23:51 PM »
I am on Windows 10 as well. Try copying this string to the quick reply box and hit Preview, then hit Preview again on the full editor: 🙂test
The preview from the quick reply box works, the full editor gives the error.

Offline Gwenwyfar

  • Customizer
  • Full Member
  • *
  • Posts: 687
  • Gender: Female
    • Gwenwyfar on GitHub
Re: Can't post unicode emoji
« Reply #19 on: June 18, 2017, 05:42:50 PM »
I just noticed that it works fine when entering emoji on the quick reply box. But not via the regular editor page. That throws the empty post error.
And you know what? It's the same problem here.

Try pasting the following string to the quick reply box and hit "Preview". Then hit "Preview" again on the editor page.

📗test
Same behavior for me, both here and on my own forum, on linux mint/FF. I tested with 😜

Quick reply preview ok, full editor preview gives an error.

Edit: Also quick edit gives the same error, full edit is ok.

Offline Gluz

  • Native Language Support Specialist
  • Sr. Member
  • *
  • Posts: 958
  • Gender: Male
Re: Can't post unicode emoji
« Reply #20 on: June 18, 2017, 06:34:47 PM »
The fix for 2.0.10+ pretty much is the same as for 2.1 beta 3, but it would work only if the forum is in UTF8. I see that in the github and tested in my test forum, I already passed that into my live forum as well.

https://github.com/SimpleMachines/SMF2.1/commit/5b73e8dcc330dfb4d87fec54b7a5518a46272fbf

Basically in /Themes/all_themes_that_have_it/Post.template.php search for all instances of:
Code: [Select]
.php_to8bit().php_urlencode()
And delete that part, with it, the previews should work.
Need help with PHP, HTML, CSS, Themes or MODs?

Quote
Think in spanish, habla en inglés.

Offline feline

  • SMF Hero
  • ******
  • Posts: 1,618
  • Gender: Female
    • Product developer site
Re: Can't post unicode emoji
« Reply #21 on: June 19, 2017, 02:24:41 AM »
Better choice ..

replace:
Code: [Select]
.php_to8bit().php_urlencode()
with:
Code: [Select]
.html_entity_decode.php_urlencode()
That works perfect ...
PortaMx has released the official release of PortaMx-Forum 1.2 with a built in SEF engine, a integrated Portal and full support for the EU Cookie Law.
Follow us on our Community


Offline Arantor

  • Resident Overthinker
  • SMF Friend
  • SMF Legend
  • *
  • Posts: 68,041
    • Arantor on GitHub
Re: Can't post unicode emoji
« Reply #22 on: June 19, 2017, 11:46:58 AM »
Don't you want brackets on the decode call?
To assume is to hope that those who came before had the presence of mind and capacity to implement the dreams of those who would come after.

You either die a hero or live long enough to see yourself become the villain. It seems you have chosen which, and now I must do the same.

Offline feline

  • SMF Hero
  • ******
  • Posts: 1,618
  • Gender: Female
    • Product developer site
Re: Can't post unicode emoji
« Reply #23 on: June 19, 2017, 04:00:43 PM »
No .. this I have tested with a Android handy/tablet and a MS Lumia ..
Works perfect ...  ;)
PortaMx has released the official release of PortaMx-Forum 1.2 with a built in SEF engine, a integrated Portal and full support for the EU Cookie Law.
Follow us on our Community


Offline albertlast

  • Development Contributor
  • Jr. Member
  • *
  • Posts: 171
Re: Can't post unicode emoji
« Reply #24 on: June 19, 2017, 04:34:22 PM »
what is the motivation for doing this decode/encode stuff?
Because this stuff cost cpu and traffic.

Offline Arantor

  • Resident Overthinker
  • SMF Friend
  • SMF Legend
  • *
  • Posts: 68,041
    • Arantor on GitHub
Re: Can't post unicode emoji
« Reply #25 on: June 19, 2017, 06:13:00 PM »
I'm not sure but I genuinely don't see how that snippet works correctly. Maybe it ends up doing the same as the 2.1 equivalent change ;D
To assume is to hope that those who came before had the presence of mind and capacity to implement the dreams of those who would come after.

You either die a hero or live long enough to see yourself become the villain. It seems you have chosen which, and now I must do the same.

Offline Gluz

  • Native Language Support Specialist
  • Sr. Member
  • *
  • Posts: 958
  • Gender: Male
Re: Can't post unicode emoji
« Reply #26 on: June 20, 2017, 01:33:13 AM »
It doesn't, it throws JavaScript error about html_entity_decode not being defined if used without brackets, and that is not a function with brackets.

That part it's failing to do anything.

If you just use the fix for 2.1, it works the same but without errors.
Need help with PHP, HTML, CSS, Themes or MODs?

Quote
Think in spanish, habla en inglés.

Offline feline

  • SMF Hero
  • ******
  • Posts: 1,618
  • Gender: Female
    • Product developer site
Re: Can't post unicode emoji
« Reply #27 on: June 20, 2017, 02:01:47 AM »
Prewiev Screenshot without html_entity_decode (not_correct)

Preview Screenshot width html_entity_decode (correct)

More questions?
PortaMx has released the official release of PortaMx-Forum 1.2 with a built in SEF engine, a integrated Portal and full support for the EU Cookie Law.
Follow us on our Community


Offline Arantor

  • Resident Overthinker
  • SMF Friend
  • SMF Legend
  • *
  • Posts: 68,041
    • Arantor on GitHub
Re: Can't post unicode emoji
« Reply #28 on: June 20, 2017, 03:49:48 AM »
I'm just struggling to understand how a JavaScript property can be called as a function without using the operator to tell it to use it as a function.

In other words, as gluz said.

Now, if you had the brackets in there as I suggested, I could see it maybe working.
To assume is to hope that those who came before had the presence of mind and capacity to implement the dreams of those who would come after.

You either die a hero or live long enough to see yourself become the villain. It seems you have chosen which, and now I must do the same.

Offline Gluz

  • Native Language Support Specialist
  • Sr. Member
  • *
  • Posts: 958
  • Gender: Male
Re: Can't post unicode emoji
« Reply #29 on: June 20, 2017, 04:08:43 AM »
That preview without html_entity_decode have php_urlencode or not?

Where it's defined html_entity_decode in SMF? because the JavaScript console says it's not defined.

With the github fix it displays the Emojis fine in a brand new SMF install, and with your code it displays the Emojis but also generate that error in the JavaScript console.
Need help with PHP, HTML, CSS, Themes or MODs?

Quote
Think in spanish, habla en inglés.