Upgrade from SMF 1.1.16 to SMF 2.0.2 - language/character problem.

[wildd]

Hello, our forum is located in Latvia and therefore people are writing all posts in Latvian language.

We had problems with lost buttons in forum and some other things was broken so we finally decided to move to SMF 2.0.2

We duplicated DB, upgrade worked well, all looks good, also all lost buttons like "Save change profile", etc are back, but we have strange thing with characters. There seems to be some problem with encoding.

There are screen shot from one board OLD vs NEW.
hxxp:img832.imageshack.us/img832/6315/oldforum.png [nonactive]
hxxp:img192.imageshack.us/img192/6534/newforumq.png [nonactive]

DB VARIABLES OLD

character set client    utf8
character set connection    utf8
character set database    utf8
character set filesystem    binary
character set results    utf8
character set server    utf8
character set system    utf8
character sets dir    /usr/share/mysql/charsets/
collation connection    utf8_general_ci
collation database    utf8_general_ci
collation server    utf8_general_ci
completion type    

DB VARIABLES NEW

character set client    utf8
character set connection    utf8
character set database    utf8
character set filesystem    binary
character set results    utf8
character set server    utf8
character set system    utf8
character sets dir    /usr/share/mysql/charsets/
collation connection    utf8_general_ci
collation database    utf8_general_ci
collation server    utf8_general_ci
completion type    

As you see all settings in DB are the same.

In both page source I see

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
var smf_charset = "UTF-8";

The only different thing is that in old forum under Languages we have set "Latvian" (we did translation by our self), but in new forum there is "English", but as I understand this is related only for the forum things like buttons, labels, etc. So this has nothing to do with topic titles, posts.

Only weird thing that I see, is that in new forum under Languages there is English language with "Character Set ISO-8859-1", not UTF-8 what is like WTF for me, but okay.

Also tried "Convert HTML-entities to UTF-8 characters" but that didn't do anything.

So any ideas?

Thanks!

[Oldiesmann]

You need to download and install the English-UTF8 language pack.

Go to Admin -> Configuration -> Languages -> Add Language
Type "English" in the box and click Search
Click "install" next to the one with a description of "SMF 2.0.2 English-utf8"
Make sure it says "Yes" in the "Writable" column for each file and that the box in the "copy" column is checked for each file.
Click "Install" at the bottom of the page
Now go back to Admin -> Configuration -> Languages and select the radio button next to the newly-installed language pack to make it the default.

[wildd]

Hi, thanks for reply, tried what you said, but nothing got fixed.

Searched for english-utf8, installed it (checked to copy all files including few that wasn't automatically checked - all files were showing green as writable). Then selected english-utf8 as default language pack for all users, and it shows that all users are using it, but forum is still messed up.

Any other solutions?

P.S. Just now tried new topic with text and there all characters are showing right ;/
Maybe there is some encoding convert that I can try to old DB?

[wildd]

Although I thought that the language what is set in forum doesn't mean nothing, maybe it means?

I checked both settings.php

Old

$language = 'latvian';      # The default language file set for the forum.
$db_character_set = 'utf-8';

New

$language = 'english-utf8';      # The default language file set for the forum.
$db_character_set = 'utf8';

Maybe before upgrade I need to change back to english or something in old forum and then proceed with upgrade?

[wildd]

Seriously forum with "279,460 Members" and no one reads this forum/board and hasn't any ideas/thoughts?

Today tried this
A) In 1.1.16 changed forum language back from "Latvian" to "English", so settings.php

$language = 'english';      # The default language file set for the forum.
$db_character_set = 'utf-8';

All looks good, charachers are right.
B) In 1.1.16 used maintenance "'Convert the database and data to UTF-8", then "Convert HTML-entities to UTF-8 characters". After this step forum characters were wrong, but continued to next steps.
C) Upgrade from 1.1.16 to 2.0.2. The characters were wrong.
D) In 2.0.2 added and changed language to English-UTF8, all looks still wrong.

What else I can try. Maybe someone knows why forum characters are wrong after step b)?

[Illori]

we are all volunteers so maybe those that know the answer were busy elsewhere and could not get to reply to your thread.

also you really should not bump within 24 hours.

[wildd]

Ok, thanks for help. I figured out that there had been some bug with SMF 1.1.16 and although my DB was set as UTF-8, the data was inserted (SMF?) in latin1 or latin2.

So the problem is that I have UTF-8 tables and columns and data what is encoded in UTF-8, but before insertion it was encoded in some other encoding.

I find solution for this and it almost worked and I was so happy, but steel some things like topic titles remained messed with "?"

UPDATE smf_messages SET subject=CONVERT(CONVERT(CONVERT(subject USING 'latin1') USING BINARY) USING 'utf8')

Executing this on topic posts columns, private message columns, board name and description columns worked fine, but as I said, topic titles are still with "?", which makes me think that maybe they were encoded not in latin1 but maybe latin2 or something else.

In which SMF 1.1.16 PHP files I need to look to find out in what topic titles were encoded before inserting in DB?

[JimM]

Not sure if it is of any help but my collation in a non-utf8 install is latin1_swedish_ci. 

[Crazy F]

I'm not able to update from 1.1.16 to 2.0.2. The upgrade always stop when it goes to update the DB. Can it be because of this settings???

$language = 'english';      # The default language file set for the forum.
$db_character_set = 'utf8';

What should I do to fix this?

Thanks in advance for any help!