Character problem after upgrade from 1.x.x to 2.0.x

[croboy]

I read topics about upgrade and problems with character set.
I have 1.1.16 and UTF-8. All tables have collation "utf8_general_ci"
Default language is Croatian.

These are topics I found
http://www.simplemachines.org/community/index.php?topic=472141.0
http://www.simplemachines.org/community/index.php?topic=467055.0
http://www.simplemachines.org/community/index.php?topic=351080.0

Last topic  http://www.simplemachines.org/community/index.php?topic=351080.0
is suggesting to install english UTF-8 language. I tried that but it does not help.
I installed "english UTF-8 language". When I change language from Croatian to English UTF-8 characters on forum are not good.

I tried to upgrade from 1.1.16 with default Croatian language but when upgrade is finished all Croatian characters are messed up. After upgrade in language option I see all languages are ISO-8859-2. I tried forum option to convert to UTF-8 but it does not work.

I would appreciate help and suggestion.

I think this is very big problem for upgrading from 1.X.X.to 2.X since obviously a lot of people is having a problem with special/local characters.

[croboy]

I just tried this
SELECT value FROM smf_settings WHERE variable LIKE 'global_character_set';
but get nothing.

In settings.php I have

$language = 'croatian';   
$db_character_set = 'utf-8';

I learned that before upgrade I have to change
$db_character_set = 'utf-8';   to:   $db_character_set = 'utf8';

[croboy]

Now I tried this
INSERT INTO smf_settings (variable, value) VALUES ('global_character_set', 'UTF-8');
(it is suggested in http://www.simplemachines.org/community/index.php?topic=351080.msg2382692#msg2382692)
but all I got wass messed up charatcers. I tried conversion to UTF-8 but it didn't help.

[kat]

I tried forum option to convert to UTF-8 but it does not work.

I tried conversion to UTF-8 but it didn't help.

Does that mean that you managed to get it to convert?

Or, are you still having hassles, with that?

Is there anything relevant in your error-logs?

[croboy]

I haven't solved it. I would happily post the solution here.

I found out that encoding in DB is iso-8859

I tried to replace characters in database dump using Notepad++ but after I restored DB characters were not OK.

I tried to convert dump encoding using Dreamweaver but it is not working too.

I tried to upgrade using Croatian non UTF language pack. I didn't work OK.

I have no idea what to do.

[kat]

This is all new stuff, to me. I went to my site's phpmyadmin and checked the encoding options. Odd thing... UTF-8 Croatian isn't there! Even the Croatian language is missing. Loads of other languages, there. But, for some reason, not Croatian.

Is it there, on yours? I'd imagine it would be. But, it's worth checking.

[croboy]

This is all new stuff, to me. I went to my site's phpmyadmin and checked the encoding options. Odd thing... UTF-8 Croatian isn't there! Even the Croatian language is missing. Loads of other languages, there. But, for some reason, not Croatian.

Is it there, on yours? I'd imagine it would be. But, it's worth checking.

No, official Croatian UTF-8 for phpmyadmin does not exists.
Collation in my tables is utf8_general_ci

I did try with installing English UTF-8 language pack on 1.1.16 it messed up everything.

I spoke to some web hosting admins but they don't have idea how to solve this.
I know there is php encoding, database encoding and DB link encoding. All this is really messed up.

[kat]

It seems, to me, that you have to get them to install the Croatian language on the server, then.

I'm surprised they couldn't figure that out, really...

[croboy]

It seems, to me, that you have to get them to install the Croatian language on the server, then.

I'm surprised they couldn't figure that out, really...

I didn't explain well. I spoke to some admins I know. Not the one that maintance shared hosting I use.

A friend that is programmer told me that Croatian UFT-8 does not exists at all.

[kat]

Ah. That might be a problem, then.

Thinking about it, though, It's curious that everything was OK, until you did the upgrade, isn't it?

Sure is weird.

Let me do a bit of research...

[croboy]

What I see as a problem is not just my case. I wrote those 3 threads in forst post.

I also found other threads with same topic
http://www.simplemachines.org/community/index.php?topic=498578.0
http://www.simplemachines.org/community/index.php?topic=357401.0

It seems a lot of people is having problems with encoding when upgrades from 1.x.x to 2.0.x

Does anyone know all places where encoding is defined?
I know about settings.php -> $db_character_set = 'UTF8';

Is there some other place in mySQL DB or in forum?

[kat]

Yeah. If you look in phpmyadmin, on your site's Cpanel, you can see the "Server connection collation", there.

[croboy]

I can't find it in Cpanel but i Found in forum database on tab Operations. DB is set to "latin1_swedesh_ci"
I made test DB and tried different export from original DB  and imports do test DB togehter with change of collation of test DB.
Even when I define empty test DB as ISO-8859-2 (Latin1_croatian_ci) tables are imported as UTF8.
Probabably becuase that is defined in dump.

BTW, what is correct name for ISO-8859 in SMF settings and mysql dump?

In dump I have this
/*!40101 SET NAMES utf8 */;

Maybe I could change it to ISO-8859? I just don't know what is correct name.

CREATE STATEMENT says:
ENGINE=MyISAM DEFAULT CHARSET=utf8;

[croboy]

I found these links. MIght help someone

http://ronaldbradford.com/blog/migrating-mysql-latin1-to-utf8-the-process-2010-03-06/
http://ronaldbradford.com/blog/migrating-mysql-latin1-to-utf8-preparation-2-2010-02-22/
http://ronaldbradford.com/blog/migrating-mysql-latin1-to-utf8-preparation-2010-02-11/

Now I see how complex is all this.

[kat]

I'm going to ask the localisation and languages team, if they can take a look at this. Their experiences might shed some light, on this.

At this precise moment, I'm totally baffled.

[croboy]

I'm going to ask the localisation and languages team, if they can take a look at this. Their experiences might shed some light, on this.

At this precise moment, I'm totally baffled.

OK. Tnx a lot for taking look at this. I believe solution would be helpful to other SMF users.
This guy Ronald Bradford wrote really great article about migration from Latin to UTF (links from my previous post).
Unfortunatelly it's quite complex for avarage SMF user and it requires shell access which most users on shared hostings don't have.

[kat]

I bet nobody on shared hosting does.

[emanuele]

I found out that encoding in DB is iso-8859

Did you try removing $db_character_set from Settings.php?
If the db is ISO, then you shouldn't have that value.

[croboy]

Just to say that I solved it. Just in short for now.

One of the problems was that 1.1.16 had in settings UTF-8.
When I changed it to UTF8 characters were OK.
Of course before that I made dump, some change and then upload.

One more thing. When you check things be sure to check it as gueast. In my account I had Croatian as language even though default was Croatian UTF-8.

More detail tommorrow.


One more tip not related to this:
How to add Google Analytics
http://it.i88.ca/2012/09/add-google-analytics-asynchronous.html

[Dzonny]

These character problems can be quite weird. Yes, utf8 is the right string to be used in Settings.php file, rather than utf-8 which is normally used anywhere. Have no idea why's that so. Just for the record, there is a Serbian/Croatian/Bosnian Local Support Board, so you may ask this kind of questions there as it's more probably that you'll get relevant answer there faster than here, as some membesr from that board may had the same problems.

Anyway, I'm glad to see that you solved your problem