Do emojis render correctly on SMF 2.1?

[Biology Forums]

Do emojis render correctly on SMF 2.1?

[Aleksi "Lex" Kilpinen]

I do believe they should, but don't take this as a promise.

[Bigguy]

All seems fine on both my sites.

[Biology Forums]

What makes 2.1 capable of rendering them that 2.0 and 1.x don't necessarily have?

[Gwenwyfar]

I believe they should work anywhere that utf-8 is accepted. The rest depends on the browser. Ex: 🌸

[Aleksi "Lex" Kilpinen]

What makes 2.1 capable of rendering them that 2.0 and 1.x don't necessarily have?

If I understood right, 1.1 and 2.0 may break them in how they are saved to the DB.

[Arantor]

1.1 will break them, 2.0 shouldn't, 2.1 really really shouldn't break them.

[Biology Forums]

I just checked on SMF 2.0.15, emojis do render correctly.

What should I look at to possibly make them work on 1.x? Which function, for example, should I start my focus?

[Arantor]

You can look at what $smcFunc['htmlspecialchars'] does, but I don't know if SMF 1.1 uses its equivalent everywhere correctly,

[Biology Forums]

Thank you everyone. I'll start by analyzing $smcFunc['htmlspecialchars'] then look for the smf 1.x equivalent then piece together what I need.

[Biology Forums]

So here's the fix.

Find in Load.php (1.x)

Code Select Expand

'htmlspecialchars' => create_function('$string, $quote_style = ENT_COMPAT, $charset = \'ISO-8859-1\'', '
global $func;
return ' . strtr($ent_check[0], array('&' => '&'))  . 'htmlspecialchars($string, $quote_style, ' . ($utf8 ? '\'UTF-8\'' : '$charset') . ')' . $ent_check[1] . ';'),

Replace with:

Code Select Expand

'htmlspecialchars' => create_function('$string, $quote_style = ENT_COMPAT, $charset = \'ISO-8859-1\'', '
global $func;
return ' . ($utf8 ? '$func[\'fix_utf8mb4\'](' : '') . strtr($ent_check[0], array('&' => '&')) . 'htmlspecialchars($string, $quote_style, ' . ($utf8 ? '\'UTF-8\'' : '$charset') . ')' . $ent_check[1] . ($utf8 ? ')' : '') . ';'),


'fix_utf8mb4' => create_function('$string', '
$i = 0;
$len = strlen($string);
$new_string = \'\';
while ($i < $len)
{
$ord = ord($string[$i]);
if ($ord < 128)
{
$new_string .= $string[$i];
$i++;
}
elseif ($ord < 224)
{
$new_string .= $string[$i] . $string[$i+1];
$i += 2;
}
elseif ($ord < 240)
{
$new_string .= $string[$i] . $string[$i+1] . $string[$i+2];
$i += 3;
}
elseif ($ord < 248)
{
// Magic happens.
$val = (ord($string[$i]) & 0x07) << 18;
$val += (ord($string[$i+1]) & 0x3F) << 12;
$val += (ord($string[$i+2]) & 0x3F) << 6;
$val += (ord($string[$i+3]) & 0x3F);
$new_string .= \'&#\' . $val . \';\';
$i += 4;
}
}
return $new_string;'),

Thank you, Arantor

[Kindred]

you know what the BETTER fix is?  upgrade your SMF to a modern version that does not have known, unpatched security issues....

[Biology Forums]

you know what the BETTER fix is?

found it! https://www.simplemachines.org/community/index.php?topic=566029.msg4011519#msg4011519

[albertlast]

This "fix" is more a hack than something else,
modern setup with postgres oder mysql with mb4 didn't use this any more.

[Arantor]

The best solution is to native to something that actually support ps it properly rather than using a workaround.

@albertlast, when I wrote that "hack" mb4 support was not especially widespread in hosting communities that SMF users actually use, but it's good to know that you think so highly of my work. I guess we should stop supporting everyone except those who can run bleeding edge versions of things.

[albertlast]

To be ownest i was thinking on study and not at you and
i only want to describe how 2.1 works internaly since we are here in 2.1 part.

[Biology Forums]

This "fix" is more a hack than something else,
modern setup with postgres oder mysql with mb4 didn't use this any more.

So 2.0.x used a hack to make emojis supportable? That's interesting.

I'm in the process of learning the 2.x system, I'm currently educating myself about 2.x. Hopefully by the time I'm comfortable, 2.1 will be officially released and I can make the switcharoo

[Arantor]

Yes, it used a hack I came up with (which you've found) because the issue is that emojis require 4-byte UTF-8 storage and this is what utf8mb4 is, while SMF 2.0 doesn't technically support it at all, and when I put the hack together, MySQL on a lot of shared hosts didn't support it either... so I came up with a way to safely encode emoji into entities so they could at least be stored and displayed.

[Biology Forums]

@Arantor -- Curious though, why you didn't tell me right from the get-go? In other words, why did you make me look for it? Is it because you weren't confident with your own solution?

[Arantor]

Because being around here isn't my day job and I was typing that on an iPad in between doing my actual job.