Missing posts after changing Language file?

[SergeantAsh]

I have just changed the language file in use on my forum from English to 'English (British)' and randomly the body of many of the forum posts has just been blanked, the post is still there as is it's subject but the actual post text is empty.

Anyone any ideas if this data has now been lost? and why it happened in the first place?

[CapadY]

Be sure all language files are there.
You can copy them again manualy via FTP.

After that, clear your forumcache:
Admin - Maintenance - Forum Maintenance - Routine, run the last option

[SergeantAsh]

Just ran that task and it completed successfully, however the posts with missing content are still blank - it is only certain members that have blank posts, and only in some threads, not all?

I used the automatically Language Search and Download tools within the Admin panel to get the new language pack, it was so as to get rid of the odd symbols that appeared when members tried to post Pound (£) signs.

Should I try and revert back to the other language file I have installed?

Thanks for your help.

[SergeantAsh]

Oddly, I've just reverted back to the 'English' ISO language set and the pound signs do now appear, and the missing posts have reappeared!!

Bit strange because the Language pack I have switched back to wouldn't show Pound symbols correctly? Now it does?

Any ideas?

[CapadY]

No idea about showing a pound sign Not realy known with the english languages.

I had misread your question, I thought textstrings weren't shown.
Post's should always be shown, even when you select the Japanese language so I think it is very weird at all.

[SergeantAsh]

I've got this issue once again - my English Pound currency symbols are showing as � for some reason! Any ideas? Is it worth converting my forum to UTF8? Are there any risks associated with this?

[SergeantAsh]

Interestingly, I have a feed of topics shown on the frontpage of my portal (SimplePortal) and in the frontpage posts the pound currency symbol appears fine!?

[Norv]

Your forum is not UTF8, if I understand correctly?

What version exactly is your forum?

What charset and collation do your database tables have (actually your fields in the tables)?

[SergeantAsh]

Thanks for your reply Norv.

No my forum isn't currently UTF8 but I've noticed the option within the Admin panel to convert it. Here's the stats from my Database:

Character Set: utf8 -- UTF-8 Unicode
Collation: utf8_general_ci
Table type: MyISAM

My forum is currently configured to use a language file called:

English   ISO-8859-1     en_US

Would the correct step forward be to use the SMF UTF8 conversion within the Admin panel? If so (and having backed up everything) what are the risks?

[Norv]

How about the columns of a table like smf_messages? If you select it in phpmyadmin, and look at the structure, what collation do the columns have? (those who have one)
How about a table with "sp_" in the name? (those who are created by simple portal), what collation have its fields?

[SergeantAsh]

Interestingly my tables are a mix of the following...

latin1_swedish_ci
utf8_general_ci

My 'Messages' table is 'latin1_swedish_ci'. I haven't ever set these, they were setup initially by SMF 1.1 RC2 from which I went to SMF 2.0 RC3. The SP_ tables are UTF8.

Should I go with the forum conversion option to UTF8 via the Admin Panel option? Will that also amend my DB tables for me to UTF8 or should I do that manually?

[Norv]

I can't answer yet.
So, please don't convert to UTF8.

Even if the table smf_messages is latin1_swedish_ci, how are the columns in it? If you select the table, and see its structure, what collation do the columns have?
How about smf_admin_files?

[SergeantAsh]

I can't answer yet.
So, please don't convert to UTF8.

Even if the table smf_messages is latin1_swedish_ci, how are the columns in it? If you select the table, and see its structure, what collation do the columns have?
How about smf_admin_files?

The varchar and text columns within the SMF_MESSAGES table are all 'latin1_swedish_ci'.

[Norv]

In your Settings.php file, is there a line saying something like:

$db_character_set = 'utf8';

or similar, some other character set?

[SergeantAsh]

In your Settings.php file, is there a line saying something like:

$db_character_set = 'utf8';

or similar, some other character set?

Nope, no character set related bits within the settings.php file

[Norv]

If you look in the messages table, browse to a message which contains the euro sign, one which you cannot see on your board, can you see it properly in phpmyadmin?

[SergeantAsh]

Viewing a specific topic with £ (pound) symbols within PHPMyAdmin shows that they appear correctly as the symbol (£), interestingly hough single quotes appear as...

Code Select Expand

'

[Norv]

Okay, thank you.

Can you please check your PM? I would appreciate to see an example post with the problem... I'm afraid I didn't notice any garbked characters in the former one.

[SergeantAsh]

Okay, thank you.

Can you please check your PM? I would appreciate to see an example post with the problem... I'm afraid I didn't notice any garbked characters in the former one.

Sorry for the late reply Norv, I'm sending you a link to the page with the pound symbol issues...

[Aleksi "Lex" Kilpinen]

Any updates on this? Have you been able to sort things out?

[SergeantAsh]

Any updates on this? Have you been able to sort things out?

No Lex unfortunately not - everything appears to look fine to Norv yet it doesn't to me and some of my members. Would it be worth me converting to UTF8, is that an advantage anyway?

[Aleksi "Lex" Kilpinen]

It's not really an advantage - more of a disadvantage on some perspectives, unless you really need support for utf-8 character sets.
You could ofcourse, make a duplicate install of your current forum - make sure to copy over the current database as well, and then play around with the copy - that you could safely convert as well if needed.

I would wager that this could possibly be because your posts and DB partly use UTF-8 already, but your forum tells the browsers it is in ISO and so the browsers fail to show the posts correctly - or something, I'm pretty much guessing here honestly.

[Aleksi "Lex" Kilpinen]

Actually - I just had a thought! How was your forum installed in the first place?

[SergeantAsh]

Actually - I just had a thought! How was your forum installed in the first place?

I manually installed it but I started with 1.1 RC2 and upgraded to 2.0 RC3 - is it worth me altering the tables perhaps?

[Aleksi "Lex" Kilpinen]

Okay, if it was manually installed then my thought won't probably lead anywhere. I just thought, you could try (for laughs) to put your forum in maintanance mode for a while (just so that no one will cause further problems by mistake while testing), add the $db_character_set = 'utf8'; line to your settings.php and see if the nonfunctional posts now appear correctly or not.

I don't really know if this could do any good, but just a thought to further diagnose things.

[SergeantAsh]

Okay, if it was manually installed then my thought won't probably lead anywhere. I just thought, you could try (for laughs) to put your forum in maintanance mode for a while (just so that no one will cause further problems by mistake while testing), add the $db_character_set = 'utf8'; line to your settings.php and see if the nonfunctional posts now appear correctly or not.

I don't really know if this could do any good, but just a thought to further diagnose things.

Very sorry for the late reply - I only just returned from holiday.

I have just tried this and miraculously EVERYTHING IS NOW WORKING as it should and symbols are appearing correctly. Should I leave the line in there or remove and await your next suggestion to permanently correct this?

Many thanks so far!

[Aleksi "Lex" Kilpinen]

Just leave the line there, but remember that you should now on use UTF8 language packs if not already

[SergeantAsh]

Just leave the line there, but remember that you should now on use UTF8 language packs if not already

Thanks Lex, I'd prefer a more permanent fix though - would it be worth proceeding with the DB conversion to UTF8 via the admin panel?

[Aleksi "Lex" Kilpinen]

Just leave the line there, but remember that you should now on use UTF8 language packs if not already

Thanks Lex, I'd prefer a more permanent fix though - would it be worth proceeding with the DB conversion to UTF8 via the admin panel?

That is a permanent fix, and if you check, you will probably notice that with the line in there you don't have the option to do the DB conversion anymore, as if everything is working correctly with that line in settings.php - you don't need (really shouldn't actually) do the conversion anymore.
If you would convert a virgin install to UTF8, it would convert the DB tables, their contents, and add that line to settings.php

[SergeantAsh]

Ok fair enough, thanks very much for your help! I hope that future versions look out perhaps for this oddity?

[SergeantAsh]

Bit of an update, having inserted this line into my settings.php, my members are complaining that they're once again no longer able to see the pound (GBP £) symbol, I've got a link to their thread that I'll PM to the SMF experts that responded above - please ask if you'd also like a copy of the thread.

[Aleksi "Lex" Kilpinen]

You are probably still using an ISO language pack, since your source indicates var smf_charset = "ISO-8859-1";
Perhaps, if you uploaded an UTF-8 language pack, and switched to use that, it would work out OK - I'm not really an expert on different charsets and their differences though, so I'm not completely sure about this. You shouldn't lose anything if you try though

[Illori]

if you can find the post in your database you can check to make sure it is showing up correctly there and not corrupted in some way. I am not an expert on this step though.

[SergeantAsh]

Thanks guys, I have changed my language pack to a UTF8 version and it now appears to be working fine - could you take a look at the page PM'd to you to see if it looks right for you too?

[Aleksi "Lex" Kilpinen]

msg523052 Seems fine So I guess we solved it

[SergeantAsh]

All looks good to me, thanks guys, good work!!

[SergeantAsh]

Guys...more problems - since making these changes things have gone from bad to worse - modifications aren't working because they don't support UTF8, login cookie problems have occurred, and on top of that I'm getting tens of thousands of errors being logged in my logs...

Mainly this one...

--------------------

Database Error: Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '='

SELECT passwd, id_member, id_group, lngfile, is_activated, email_address, additional_groups, member_name, password_salt,
openid_uri, passwd_flood
FROM cc_smf_members
WHERE member_name = 'Bad Kidâ?¢'
LIMIT 1

--------------------

How can I properly migrate to UTF8 or is there a better alternative? Does RC4 resolve any of this?

[Arantor]

Perhaps, if you uploaded an UTF-8 language pack, and switched to use that, it would work out OK - I'm not really an expert on different charsets and their differences though

No it wouldn't.

The proper way to migrate to UTF-8 is to use Admin > Maintenance > Database > Convert to UTF-8

[Aleksi "Lex" Kilpinen]

I know that, but was thinkinng that these problems came to be by an upgrade losing the already existing UTF8 setup, since that happened to many others, and then doing the conversion again would not have probably been a good idea.

[Arantor]

From what I've seen the majority of issues seem to be because people upload UTF-8 language packs and don't actually convert the forum at the same time.

[SergeantAsh]

From what I've seen the majority of issues seem to be because people upload UTF-8 language packs and don't actually convert the forum at the same time.

Ok, I'm confused as to how we are where we are.

My forum started its life as a fresh 1.1 RC2 installation, the DB was created by the installer and so any preference of language pack/DB table collation was out of my hands.

I upgraded the forum to 2.0 RC3 which seems to have mangled the table collations and forum so as to knock the English '£' pound sign out of joint, it wasn't working and so we appended the UTF8 charset meta tag to the pages of my forum - I then changed the language pack that the forum was using to UTF8 also. Now I'm getting numerous DB errors that relate to a mix of table collation such as...

Database Error: Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '='

Whats the best way to proceed here? Should I remove the modifications that I've made and upgrade the forum properly to UTF8 or is there a simpler way to get my pound symbol back and get everything working once again?

I appreciate any help!

[Arantor]

Backup then convert the forum to UTF-8 properly because the database isn't UTF-8 right now.

[SergeantAsh]

Am I best off doing that than trying to salvage the database format that I had prior to upgrading? Are there any disadvantages to upgrading to UTF8? I see some mods aren't compatible?

And of course, are there are advantages?

[Arantor]

UTF-8 makes things a touch slower, but handles multiple character sets and special characters much better.

Only badly written mods aren't compatible with UTF-8, it's basically laziness on the part of mod authors *most* of the time (I won't say *all* because there are times it's not their fault, but speaking as a mod author, UTF-8 support is not generally hard to do)

Yes, you might well be worth trying to salvage it as it was.

[SergeantAsh]

UTF-8 makes things a touch slower, but handles multiple character sets and special characters much better.

Only badly written mods aren't compatible with UTF-8, it's basically laziness on the part of mod authors *most* of the time (I won't say *all* because there are times it's not their fault, but speaking as a mod author, UTF-8 support is not generally hard to do)

Yes, you might well be worth trying to salvage it as it was.

Ok, so for a UK based forum that has international visitors but generally those from countries where their charset doesn't contain a vast number of 'special' characters, what table collation should my tables be, and the forum itself? Am I safe in simply changing the table collations within the DB etc?

[Arantor]

It should have been latin1_swedish_ci as it's indicating, that covers the UK just fine. You only really need it for heavily accented stuff or special characters like CJK languages use.

[SergeantAsh]

Ok thanks Arantor - I'll have a browse through the tables and set them all to this collation - I'll then change the language pack back as it should be, remove the charset meta tag that we added and see what happens?

At the moment, my tables are a random looking combination of...

- latin1_swedish_ci
- utf8_general_ci

Should I just change them all to 'latin1_swedish_ci'?

I installed your login validator too, hopefully that'll stop the login attack! Good work!