Chinese and other special characters.

[nolefan]

Hey folks...
I have a fresh install of SMF running with the lightning-dk theme by padexx ( http://raoulschinasaloon.com ).
Technically, SMF supports UTF-8 so if I type Chinese characters, they should show up... unfortunately, it's not the case. Which doesn't make sense

Do I have to install the Chinese language pack before it can display Chinese characters?
I've done a search on the topic but none of the answers seemed to address my particular problem...

[SleePy]

Did you convert your forum to UTF-8?
Did you download the utf-8 language packs?

[nolefan]

Did you convert your forum to UTF-8?
Did you download the utf-8 language packs?

I didn't read anything about converting the forum to UTF-8... I figured that was the default option. ... me confused...

[SleePy]

SMF supports UTF-8. But by default it is disabled.

This document should help explain more.
http://docs.simplemachines.org/index.php?topic=865.0

[nolefan]

Hey there,

thanks for all the help and I apologize profusely ahead of time for being such a nag but I just can't seem to get this taken care of.

The forum is in EnglishUTF-8, the DB is in English UTF8 and so is the theme. If I type something in French, chinese, japanese and preview it, there is no problem... then, if I press "post" it shows up as a question mark...
I've looked everywhere I could for an explanation, a fix and nothing... please please please...

[nolefan]

anybody ?

[Sarge]

Read the "How to convert to UTF-8?" section in the document that SleePy gave you a link to.

Have you converted your forum with Admin > Forum Maintenance > Convert the database and data to UTF-8?

[nolefan]

Read the "How to convert to UTF-8?" section in the document that SleePy gave you a link to.

Have you converted your forum with Admin > Forum Maintenance > Convert the database and data to UTF-8?

Hi Sarge,

I have read everything in the link SleePy gave me and spent loads of time in the admin section of my forum. I don't have anything in the forums maintenance that says " Convert Database and data to UTF-8".

the options I currently have are:
Optimize all tables to improve performance.
Check all files against current versions.
Find and repair any errors.
Recount all forum totals and statistics.
Empty out unimportant logs.
Convert HTML-entities to UTF-8 characters

[SleePy]

Did you try the default theme?
Not all themes support different character sets.

[nolefan]

Did you try the default theme?
Not all themes support different character sets.

I did. The problem as I finally found out was that the DB itself was in latin_swedish_ci for whatever reason ( freaking Fantastico install!!!! ) and I'm working on changing it to utf8_general_ci.
SMF itself is installed in 8tf-8 so the whole thing is a bit messy...

[Sarge]

Have you converted your forum with Admin > Forum Maintenance > "Convert the database and data to UTF-8"?

I don't have anything in the forums maintenance that says " Convert Database and data to UTF-8".

Have you tried deleting the entry global_character_set = UTF-8 in smf_settings and going to Admin > Forum Maintenance again? If you do that, the link I mentioned should appear.

Open phpMyAdmin (if you can access it for your hosted account), select your forum database and click on the SQL icon. It looks like this:



Enter the query (copy/paste from below) into the box and click the Go button.

Code Select Expand

DELETE FROM `smf_settings` WHERE `variable` = 'global_character_set' LIMIT 1;

The query above works if your forum tables prefix is smf_ (which is the default). You can find the exact prefix for your forum in Admin > Server Settings > Database Tables Prefix -- if it is different from smf_ then you have to modify the query before running it in phpMyAdmin.


[edit]
Also check the variables mentioned in this post:
http://www.simplemachines.org/community/index.php?topic=165442.msg1056581#msg1056581

[nolefan]

thank you , thank you , thank you, thank you!!!!!

it worked!

[Sarge]

Great! I'm so happy for you!

If you need assistance in the near future with the same issue, feel free to post in this topic again.

Enjoy.

[wtt]

I also encountered the same problem. However, some sentences here are  too small. It is hard to read that. Just my 2 cents.

[Sarge]

However, some sentences here are  too small. It is hard to read that.

You mean the font size?

[wtt]

Hello Sarge, Yes, I meant the "Font Size"
"DELETE FROM `smf_settings` WHERE `variable` = 'global_character_set' LIMIT 1;" Is this statement right?

BTW, Are there step by step solution for this problem? I encountered the same problem here...

However, some sentences here are  too small. It is hard to read that.

You mean the font size?

[wtt]

"Have you tried deleting the entry global_character_set = UTF-8 in smf_settings "
How can I do that?

[wtt]

"The problem as I finally found out was that the DB itself was in latin_swedish_ci for whatever reason ( freaking Fantastico install!!!! )" The same situation in my side. This seems to be a bug of Fantastico installation. I guess the guy who developed the software came from Latin countries...

[wtt]

How did you do that? I encountered the same problem, but I couldn't find the way to do that right now...

"I did. The problem as I finally found out was that the DB itself was in latin_swedish_ci for whatever reason ( freaking Fantastico install!!!! ) and I'm working on changing it to utf8_general_ci.
SMF itself is installed in 8tf-8 so the whole thing is a bit messy..."

[Sarge]

You don't have to post multiple times. If there's no reply after your last post, simply edit it.

Yes, it seems to be a Fantastico bug. It sets global_character_set as UTF-8 even if the tables are in latin1.

First, make a backup of your forum database. Then delete global_character_set from smf_settings (read reply #10 in this topic about how to do that). Then go to SMF Admin > Forum Maintenance and run "Convert the database and data to UTF-8?" -- SMF will convert the forum tables and set global_character_set to UTF-8.

[wtt]

Hello Sarge, Thanks so much for your guide and I am sorry for my previous posts as I was a little bit freaking out. Anyway, any other folks would not encounter the same situation as me with your post here!

You don't have to post multiple times. If there's no reply after your last post, simply edit it.

Yes, it seems to be a Fantastico bug. It sets global_character_set as UTF-8 even if the tables are in latin1.

First, make a backup of your forum database. Then delete global_character_set from smf_settings (read reply #10 in this topic about how to do that). Then go to SMF Admin > Forum Maintenance and run "Convert the database and data to UTF-8?" -- SMF will convert the forum tables and set global_character_set to UTF-8.

Hello Sarge, I have done this to my database.
"DELETE FROM `smf_settings` WHERE `variable` = 'global_character_set' LIMIT 1;"
However, I found that it is not the solution for this problem.

Have you converted your forum with Admin > Forum Maintenance > "Convert the database and data to UTF-8"?

I don't have anything in the forums maintenance that says " Convert Database and data to UTF-8".

Have you tried deleting the entry global_character_set = UTF-8 in smf_settings and going to Admin > Forum Maintenance again? If you do that, the link I mentioned should appear.

Open phpMyAdmin (if you can access it for your hosted account), select your forum database and click on the SQL icon. It looks like this:



Enter the query (copy/paste from below) into the box and click the Go button.

Code Select Expand

DELETE FROM `smf_settings` WHERE `variable` = 'global_character_set' LIMIT 1;

The query above works if your forum tables prefix is smf_ (which is the default). You can find the exact prefix for your forum in Admin > Server Settings > Database Tables Prefix -- if it is different from smf_ then you have to modify the query before running it in phpMyAdmin.


[edit]
Also check the variables mentioned in this post:
http://www.simplemachines.org/community/index.php?topic=165442.msg1056581#msg1056581

You don't have to post multiple times. If there's no reply after your last post, simply edit it.

Yes, it seems to be a Fantastico bug. It sets global_character_set as UTF-8 even if the tables are in latin1.

First, make a backup of your forum database. Then delete global_character_set from smf_settings (read reply #10 in this topic about how to do that). Then go to SMF Admin > Forum Maintenance and run "Convert the database and data to UTF-8?" -- SMF will convert the forum tables and set global_character_set to UTF-8.

[Sarge]

Then go to SMF Admin > Forum Maintenance and run "Convert the database and data to UTF-8?" -- SMF will convert the forum tables and set global_character_set to UTF-8.

[wtt]

I knew that and I have done that successfully with your previous post. Cheers!

[Sarge]

So, is it working OK now?

[wtt]

Yes. I got that work before your last two posts.
Thanks a lot!

So, is it working OK now?

[polytalks]

Have you converted your forum with Admin > Forum Maintenance > "Convert the database and data to UTF-8"?

I don't have anything in the forums maintenance that says " Convert Database and data to UTF-8".

Have you tried deleting the entry global_character_set = UTF-8 in smf_settings and going to Admin > Forum Maintenance again? If you do that, the link I mentioned should appear.

Open phpMyAdmin (if you can access it for your hosted account), select your forum database and click on the SQL icon. It looks like this:



Enter the query (copy/paste from below) into the box and click the Go button.

Code Select Expand

DELETE FROM `smf_settings` WHERE `variable` = 'global_character_set' LIMIT 1;

The query above works if your forum tables prefix is smf_ (which is the default). You can find the exact prefix for your forum in Admin > Server Settings > Database Tables Prefix -- if it is different from smf_ then you have to modify the query before running it in phpMyAdmin.


[edit]
Also check the variables mentioned in this post:
http://www.simplemachines.org/community/index.php?topic=165442.msg1056581#msg1056581

where can I find entry global_character_set = UTF-8 in smf_settings?? and which smf setting that you are regarding to?

thanks

[Relyana]

Brilliant ! This should be in "tips and tricks". You have no idea how long I searched for a fix to my problems with the Romanian diacritics (ok in preview, not ok in submitted posts). 

Thank you !!! 

[Sarge]

... it seems to be a Fantastico bug. It sets global_character_set as UTF-8 even if the tables are in latin1.

Therefore, it's not really suitable for Tips and Tricks, because the issue does not happen when you upload SMF files by yourself and install SMF using install.php (not using an auto-installer like Fantastico).

We have contacted Netenberg, the developer of Fantastico. At the time of writing, we are still waiting for a response from them.

[Relyana]

You're right, forgive my enthusiasm. My forum is no longer "connected" to Fantastico. 
The weird thing is that soon after I fixed the utf-8 character issue some of the posts disappeared .... I mean they were there in their threads but with no words or smileys or anything ... just blank posts. It didn't bother me though, they're gone for good now and everything's fine.   

[Sarge]

You're right, forgive my enthusiasm. My forum is no longer "connected" to Fantastico. 
The weird thing is that soon after I fixed the utf-8 character issue some of the posts disappeared .... I mean they were there in their threads but with no words or smileys or anything ... just blank posts. It didn't bother me though, they're gone for good now and everything's fine.   

Test with accented and other special characters (for example ä, 中文, " etc). If you still experience any issues, start a new topic (and post the topic link here).

[Relyana]

Test with accented and other special characters (for example ä, 中文, " etc). If you still experience any issues, start a new topic (and post the topic link here).

Thanks Sarge but everything's perfect (or maybe it's just my proverbial luck.  )

[smurfyforum]

I am having this same problem - but I do not want to convert to UTF8, how do I solve this without converting?

[Sarge]

I am having this same problem - but I do not want to convert to UTF8, how do I solve this without converting?

What special characters are you having issues with?

[smurfyforum]

I have a member form finland and when he posts the apostrophies and stuff come out different characters, also like my user name here if I try to add the ™ to my name on my profile I get that error -

Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation 'like'
File: /home/mysite/public_html/mysite/forum/Sources/Subs-Members.php
Line: 904

[smurfyforum]

oh line 904 in my subs-member says

      LIMIT 1", __FILE__, __LINE__);

[Sarge]

oh line 904 in my subs-member says

      LIMIT 1", __FILE__, __LINE__);

That's the last line of that query. Copy the lines before that (starting with the line that says db_query) and paste them here.

[smurfyforum]

ok here is the whole section

Code Select Expand



// Make sure they don't want someone else's name.
$request = db_query("
SELECT ID_MEMBER
FROM {$db_prefix}members
WHERE " . (empty($current_ID_MEMBER) ? '' : "ID_MEMBER != $current_ID_MEMBER
AND ") . "(realName LIKE '$checkName' OR memberName LIKE '$checkName')
LIMIT 1", __FILE__, __LINE__);
if (mysql_num_rows($request) > 0)
{
mysql_free_result($request);
return true;
}

// Does name case insensitive match a member group name?
$request = db_query("
SELECT ID_GROUP
FROM {$db_prefix}membergroups
WHERE groupName LIKE '$checkName'
LIMIT 1", __FILE__, __LINE__);
if (mysql_num_rows($request) > 0)
{
mysql_free_result($request);
return true;
}

// Okay, they passed.
return false;
}




[Sarge]

Is line 904 before or after this:

Code Select Expand


// Does name case insensitive match a member group name?


[smurfyforum]

Is line 904 before or after this:

Code Select Expand


// Does name case insensitive match a member group name?


904 is before - that is line 911

[smurfyforum]

I think this might be the problem??

[Sarge]

Let's see...

When using UTF-8, SMF issues "SET NAMES utf8" before dealing with database data; that SQL command overrides latin1 or any other charset in the MySQL configuration and database setup.

When SMF is not using UTF-8, it uses the MySQL default charsets. In your case, it seems that MySQL is using UTF-8, but the database data are in latin1.

You can force SMF to use latin1 explicitly. One way to do this is to add $db_character_set in Settings.php (in your main forum directory). Make a backup copy of Settings.php before modifying it!

1. Download Settings.php from your forum directory to your computer.
2. Open it in a text editor.
3. Search for $db_character_set in the file.
4.
    a) if there's no line that starts with $db_character_set, find this (at the end of the file):

Code Select Expand


?>

    and before that line, add this:

Code Select Expand


$db_character_set = 'latin1';

    b) if $db_character_set is already set (to 'utf8', maybe?), set it to 'latin1' as above.
5. Save the file and upload it to your forum directory. Make sure that it replaces the original Settings.php.

Again: make a copy of Settings.php before modifying it!

[smurfyforum]

THAT WORKED!!! THANK YOU SOOOOOOOOOOOOOOOOOOOOOOO MUCH!! This has been making me nuts for a month now!! LOL!!!

[Sarge]

Glad it helped!

A word of advice: Next time, please start new topics for your own issues. Topics marked as solved are not checked by the Support Team. (I noticed your reply only because I had set notifications for this topic.) Also, asking for support about your own issues in other people's topics is generally not recommended.

Thank you!

[albanian]

I wont to convert Albanian Character  how do i ?  If AN Albanian Can help me in albanian is more simple for me,,,, My email [email protected]

[Sarge]

I wont to convert Albanian Character  how do i ?  If AN Albanian Can help me in albanian is more simple for me,,,, My email [email protected]

Hello Erion, we have an Albanian language board here.

[albanian]

wow   thank's  very much

[cold77]

Thank you Sarge is worked for me!

Have you converted your forum with Admin > Forum Maintenance > "Convert the database and data to UTF-8"?

I don't have anything in the forums maintenance that says " Convert Database and data to UTF-8".

Have you tried deleting the entry global_character_set = UTF-8 in smf_settings and going to Admin > Forum Maintenance again? If you do that, the link I mentioned should appear.

Open phpMyAdmin (if you can access it for your hosted account), select your forum database and click on the SQL icon. It looks like this:



Enter the query (copy/paste from below) into the box and click the Go button.

Code Select Expand

DELETE FROM `smf_settings` WHERE `variable` = 'global_character_set' LIMIT 1;

The query above works if your forum tables prefix is smf_ (which is the default). You can find the exact prefix for your forum in Admin > Server Settings > Database Tables Prefix -- if it is different from smf_ then you have to modify the query before running it in phpMyAdmin.


[edit]
Also check the variables mentioned in this post:
http://www.simplemachines.org/community/index.php?topic=165442.msg1056581#msg1056581

[Smoking Mountains]

I'm having the same problem of not able to display chinese characters. I used the "Fantastico De Luxe" to install the SMF as well. I've tried Sarge's suggestion by running this code below

DELETE FROM `smf_settings` WHERE `variable` = 'global_character_set' LIMIT 1;

and then "Convert HTML-entities to UTF-8 characters" in "Forum Maintenance".

But i'm not as lucky. 

Attached is the server variables and settings. Any suggestion? Thank you!

[Smoking Mountains]

i guess this shows the problem? should i change all the "latin1_swedish_ci" to "utf8_general_ci"? if so, how do i do that?

thanks.

[Norv]

Please post a link to your forum.
Also, if you have in Admin > maintenance the option to "Convert the database and forum to utf8", perhaps you can try that, but only after making a complete backup of the database. (preferably using phpMyAdmin's "Export" feature for backup).

[Sarge]

I've tried Sarge's suggestion by running this code below

DELETE FROM `smf_settings` WHERE `variable` = 'global_character_set' LIMIT 1;

and then "Convert HTML-entities to UTF-8 characters" in "Forum Maintenance".

1) Are you sure it wasn't "Convert the database and data to UTF-8"? The other option ("Convert HTML-entities to UTF-8 characters") should appear only after the conversion, not before.

2) Also, if you have a copy of Settings.php from before the conversion, see if there is a line for $db_character_set (see reply #40 in this topic for more details.)

But i'm not as lucky. 

3) What encoding does your browser use when visiting your forum? In Firefox it can be seen at View menu > Character Encoding, and in Internet Explorer at View > Encoding.

i guess this shows the problem? should i change all the "latin1_swedish_ci" to "utf8_general_ci"? if so, how do i do that?

Yes, that probably means that the conversion didn't happen. Please see my three questions above (don't modify anything for the moment.)

Oh, and absolutely get a full backup of the database before making any more changes, as Norv said.

[Smoking Mountains]

Thank you Norv. My site was just created, so it is pretty much empty. But here is a link to the post that is supposed to be in Chinese. hxxp:forums.lanbud.com/index.php?topic=5.msg6#new [nonactive]

[Smoking Mountains]

1) Are you sure it wasn't "Convert the database and data to UTF-8"? The other option ("Convert HTML-entities to UTF-8 characters") should appear only after the conversion, not before.

Oh, and absolutely get a full backup of the database before making any more changes, as Norv said.

Thanks Sarge. Please see attached image. I'm backing up the forum now.

[Sarge]

I see that your forum is brand new, so you don't really have anything to backup. At worst, you could simply delete and/or reinstall it

Please see my second question:

2) Also, if you have a copy of Settings.php from before the conversion, see if there is a line for $db_character_set (see reply #40 in this topic for more details.)

[Smoking Mountains]

2) Also, if you have a copy of Settings.php from before the conversion, see if there is a line for $db_character_set (see reply #40 in this topic for more details.)

i just checked this. that line was already
$db_character_set = 'utf8';

and i changed it to
$db_character_set = 'latin1';

but it didn't make a difference.

[Smoking Mountains]

aha, it is working now!!!!

after i changed the "utf8" to "latin1" per your suggestion, and went to "Forum Maintenance", there was the option to convert the database, after i did that, i did "Convert HTML-entities to UTF-8 characters" as well.

wow. problem fixed!

thank you so much!!!

i was chatting with the web host tech support guy and he told me "we can't help you."

[Sarge]

after i changed the "utf8" to "latin1" per your suggestion, and went to "Forum Maintenance", there was the option to convert the database, after i did that, i did "Convert HTML-entities to UTF-8 characters" as well.

Be sure to change that line back to "utf8", if this wasn't done automatically by the conversion. Anyway, I'm glad to know it's fixed now.

Also, due to the way support is done here, please be sure to start your own topics for your own support questions. Thank you!

[allanmiller99]

Thanks! Its working for me also.

Great help!

retireincebu.com