Special Characters (Alt Codes) Become Question Marks

[WantSome]

Hello

I have searched around the forum and haven't found the solution to this anywhere else.  Please forgive me if this is a stupid question.

I recently had a few users comment that 'special characters' (sometimes they call them 'alt codes') are not working.  They give me the example of hearts and spades.  Now, to be honest I have no idea what they're talking about because this is not something I personally ever use.  One person helpfully commented that the  •  seems to work (and it appears to work here too - I copy/pasted it).

The hearts and spades turn into a bunch of question marks (? ? ? ?). 

After searching around here I can confirm that I am already using utf-8 and that mySQL is also set to utf-8.  So I don't think those are my issue.  (Also, I'm running 2.0.9, I do have a few mods and will post here if someone says they are relevant).

Could it be that the language files are out of date?  Some of them appear to be out of date but I don't use those languages (English appears up to date and that is the only language on my forum).

Any advice appreciated.

Thanks!
WS

[Mortissimov]

Check your language files. If you use UTF-8 encoding, your files should be saved in UTF-8 without BOM. If not, you should change it.

[Kindred]

Url?

[WantSome]

Hi, thanks for the replies.

I don't know what BOM is - how do I check this?  *is dumb*

The url is  http://wantsome.co.uk/forum/index.php?action=forum  guests should be able to see the thread discussing the special characters at http://wantsome.co.uk/forum/index.php?topic=8743.0

[Mortissimov]

I don't know what BOM is - how do I check this?

Just open the PHP files with a simple text editor, for example Notepad++. Than you can see Encoding menu item. Click on it, and now you can see the encoding of the file, and you can change it here as well.

[Steve]

WantSome, guests can't see your second link.

[WantSome]

I don't know what BOM is - how do I check this?

Just open the PHP files with a simple text editor, for example Notepad++. Than you can see Encoding menu item. Click on it, and now you can see the encoding of the file, and you can change it here as well.

Hey navMartin,  it looks like some of the language files are coded ANSI.  Is this causing the issue?  Don't suppose there is a shortcut to fixing all language files quickly without downloading, converting and uploading each one singly?

I should mention also that I see the correct characters appear when writing / editing the post, but that they turn into question marks upon posting. 

EDIT:  Steve, I've made that board visible to guests for now.  I'd appreciate any help!

[margarett]

First things first, check you tables collation in phpmyadmin. If they are "utf8_general_ci" (or similar) you need to make sure that your language files are also utf8. You need to pick the correct language pack from our downloads section, not edit all files by hand

[WantSome]

Hi margarett, I checked that yesterday and yes it uses utf_8_general_ci

And then just to check - If I get the english utf-8 files from the downloads area, and upload them (where? to theme/default/languages) that's all I need to do?

Seems too easy

Edit:  knew it would be - I downloaded the zip file for the latest english utf-8 files from here but when I open the files in notepad++ it tells me they are encoded in ANSI still.  What did I do wrong?

EDIT2:  On the off chance notepad++ was wrong I unpacked the language files onto the server anyway (couldn't do any harm, right?)  Nothing appears to have changed and the special characters still don't work... *sadface*

[margarett]

Strange. I just picked http://download.simplemachines.org/index.php?thanks;filename=smf_2-0-9_english-utf8.zip
And notepad++ tells me UTF-8 W/o BOM

[Illori]

i thought english was a special case and the utf-8 language files were not really required to be used?

[WantSome]

Although notepad++ tells me ANSI I loaded the language pack on the forum and it still doesn't appear to work. 

I thought it might be a theme thing but it doesn't work in any theme. 

Any other ideas?

Thanks!

[margarett]

Are you trying to write new special characters or are you viewing the already existing ones? Because I think that the old ones will stay there...
Also, you need to choose the english-UTF8 language from your admin panel so that the browser knows the collation in use.

[WantSome]

Hi guys

This is still not working so I'm at a loss.  I have chosen english-utf8 from the admin panel and uploaded the latest language pack to the server.

When I compose the code, it appears the browser knows the special characters.  When I preview the post, the special characters appear correct.  But when I post it, and view what I've posted, the characters have turned into a ? instead of a symbol.

Any other ideas would be greatly appreciated.

[margarett]

That would indicate that the collation on the database is different then...

When you preview, the text only travels on PHP. When you submit, it is sent to the database. When you view what you've posted, you are viewing content retrieved from the database.

2 things:
* can you please check the post directly on the database, see how the symbols show up in phpmyadmin? It's the column "body" of the table smf_messages. You need to find the ID of a message that has this problem
* can you check if your Settings.php has this line in it?

Code Select Expand

$db_character_set = 'utf8';
If not, can you add it in a line before ?> and try again to create a post with symbols?

[WantSome]

Hi margarett

The settings.php has that line in it near the bottom.  It also says default language is english_british_utf8.

I checked phpmyadmin and the text that should look like characters looks like this instead:

Code Select Expand


<br /><br />?<br />?<br />?<br />?

So... still no luck

[margarett]

So it's already messed up when submitting to the database...

I'm sorry, I'm out of ideas. I really hate these collations issues

[Illori]

i wonder if the column in the db table is the right collation?

[margarett]

Nice idea

Speaking of which, maybe you can backup your database and try to run the Maintenance task to convert to UTF8? Can't hurt. I guess...

[WantSome]

i wonder if the column in the db table is the right collation?

You genius.  The table itself was somehow set to latin1_swedish_ci.  How did that happen? I don't even know, but switching it back to utf8_general_ci seems to have cracked it!  Thanks Ilori.

Thanks for all your help, margarett!