Scandinavian letters æ,ø,å

[nicowilh]

Hi,

I have converted my forum to SMF 2.0 and also upgraded my language to norwegian UTF-8.

After I "Converted HTML-entities to UTF-8 characters" text that included the scandinavian letters "æ", "ø" and "å" dissapeared from the forum.

You find examples of how:

http://psykologidanmark.net/Forum/index.php?board=19.0

[Aleksi "Lex" Kilpinen]

I can see your news

Kommende studenter: Samle informasjon om tidligere opptakelser for studenter med utenlandsk bachelor. Ønsker du å bidra, kontakt RPiD. Informasjon vil bli samlet på: http://www.psykologidanmark.net/Studie/Opptakkandidat/Tidligereopptak.htm

Your menu

    Hovedside
    Hjelp
    Søk
    Kalender
    Medlemmer
    Logg inn
    Registrer

Your board names

Sosialt og faglig - Køben

All perfectly fine...

Make sure your forum is set to to use an UTF-8 language by default, and double check that your own profile has an UTF-8 language selected

[nicowilh]

Hey,

Thanks for your reply.

You found two examples where I have written the text again after converting to UTF-8. But i am not able to do this for all the missing text, since it is a lot!

I have made sure that my forum uses UTF-8 by defoult, and double checked that my own profile uses UTF-8.

If you look at the link i sent you you will find a lots of examples where text is missing: http://psykologidanmark.net/Forum/index.php?board=19.0

Nicolai

[Aleksi "Lex" Kilpinen]

OK, did you check if the text was ok before you did the HTML entity converting? It might not have been needed at all.
Do you have a backup of the time before you did it?

[nicowilh]

Thanks for your reply,

Before i did the HTML converting everything was fine.

I have a back up of the board (SMF 1.1.14) before i upgraded to 2.0.

Is it easy to convert everything back to 1.1.14 using a backup? Is it necessary?

Nicolai

[Aleksi "Lex" Kilpinen]

If everything was fine before you did it, then you probably should not have needed to do it.

If you have no backup from after the upgrade, then reverting to a backup would probably be safest to do by
first importing the 1.1.14 backup to a new empty database, then connecting your existing 2.0 installation to that new database (so, edit settings.php or run repair_settings.php to change the DB info ) and then run the 2.0 upgrade script on this new database with the 1.1 backup imported in it. (So you only need the upgrade.php and the *.sql files from the large upgrade package to do this).

[nicowilh]

Thanks for your reply Aleksi,

By uploading 1.1.14 to a new empty database, what do you mean exactly? Is it ok for me to upload the backup to the same locatian as the other 2.0 forum, but with a different name, eg. forum instead of Forum?

Nicolai

[Aleksi "Lex" Kilpinen]

You wouldn't need to touch the existing files at all,
just use an old database backup - and I hope you have one...

The files on your server provide the forums functionality and look, and also hold uploaded attachments and acataras - The topics, posts, PMs, members etc. data is held in a database that is not among the files, and can not be backed up through FTP.

How do I backup my database?
How do I restore a database backup?

[nicowilh]

Hi Aleksi,

"When you have successfully entered phpMyAdmin select the correct database from the left": Can an I use the same database as for the smf 2.0 files, or do i need to create a new one, not to mix the old database backup of SMF with the existing database for 2.0 SMF?

What character set of the file should i use: ASCII or Utf8 ("Once you have selected the correct database select the import tab and locate your file. Ensure you have the right character fileset (usually utf8 or ascii), and then select go")

Nicolai

[Aleksi "Lex" Kilpinen]

I suggest you create a new database, just in case (you can remove the old one once you have all this figured out succesfully, but keep it there for now if you have the space)

The character set would depend on the original character set of the backup - but it is most probably one of the two mentioned.

[nicowilh]

Thanks Aleksi,

I tried to create a new database in phpMyAdmin by using "Copy database to", and marking "Create database before copying". But even though the operation was successful, i could not find a new database. And in stead of uploading backup to a new database in phpMyAdmin I uploaded it to the same database as SMF 2.0. Everything seems to have worked fine with the upload, but i am a bit afraid i did something wrong in the first place, trying to "copy database"?? What do you think?

I followed your suggestion and now everything is back in order with the SMF, that is, no missing text. But the scandinavian letters are now replaced by "?" in stead. What do you suggest?

[Aleksi "Lex" Kilpinen]

Hmm.. Now - let me think - did you upload a 1.1.14 database backup directly in to a 2.0 database?

I don't think that would work out OK... The reason I suggested you to use an empty db, is just because the 1.1 and 2.0 db's will be using the same prefixes and all, but their content's will not match at all - and if you are going back to a 1.1 backup, you will need to upgrade it before you would be able to use it on a 2.0 installation...

[nicowilh]

Thanks for your reply,

I uploaded the database backup directly to the same database in phpMyAdmin as the 2.0 database were located. Than i run the upgrade.php to upgrade from 1.1.14 to 2.0. This time, i did not "convert HTML-entities to UTF-8 characters", so text did not dissapear. In stead æ, ø, and å has been replaced by "?".

http://www.psykologidanmark.net/Forum/index.php

Nicolai

[Norv]

It seems I can see the characters just fine now,

på mai 25, 2011

startlønn for psykologer

And the charset on the page is UTF8. Are there still "missing" characters?

[Norv]

Correction, some are not displaying fine at all.
http://www.psykologidanmark.net/Forum/index.php?board=19

If I understand correctly:
- when you restored the backup and upgraded, you did so with the SAME forum files
- and didn't run converting to UTF8 anymore.

Can you please try to: make a copy of your Settings.php file from forum directory (for safekeeping in case something goes wrong), then open the Settings.php file and remove a line you should have at the end, a line saying:

Code Select Expand


$db_character_set = 'utf8';

Save and reupload this Settings.php file without this line.

[nicowilh]

Hi Norv,

I can not find the line you are talking about: $db_character_set = 'utf8';

I only found the following line: # The default language file set for the forum. $language = 'norwegian-utf8';   

I think you understood me right, that i uploaded the backup of the smf database to phpMyAdmin, and than followed the instructions for upgrading to smf 2.0 as i did the first time, now without "converting HTML-entities to UTF-8 characters".

But æ, ø, and å turn out like "?"

Nicolai

[Aleksi "Lex" Kilpinen]

If it is not there, try adding it in there just for kicks, and tell us how the scandics look after that...

[nicowilh]

Hi Aleksi,

I tried it, and "?" turned into "Ã".

Do you think it is easier to find a solution to this?

Nicolai

[Aleksi "Lex" Kilpinen]

I'm confused to be honest... I think I'll be happy to let Norv continue with this if he has any better ideas...

[Norv]

Can you please: log in phpmyadmin, select the database, and export its structure to a file? Please note: the structure only, NOT the data (make sure 'data' is unchecked when exporting). And attach that file here.

Also: remove that line you added last time from Settings.php (thank you for trying!), and make sure to reset the language on the forum as well, to norvegian (non-utf8). When you do, please let us know how it behaves.

Do you still have the initial backup of the database, made before upgrading?

[nicowilh]

Attached is the file you requested.

I have removed the line I added to settings.php and reset the language to norwegian (non-utft8). Now "æ, ø, å" lookes like "Ã...".

I still have the initial backup of the database made before ugrading.

In phpMyAdmin i find that MySQL charset is set to: UTF-8 Unicode (utf8). Is this necessary to change?

I also find that some files have Collatian: utf8 bin, while others have Collatian: latin1 swedish.

[Norv]

It seems you have converted from phpbb at some point, or is that a different installation? May I ask, when exactly was that - I mean, from which version of phpbb and to which version of SMF have you converted? (if that was the case )

All data fields are utf8, with collation set to either utf8_unicode_ci either database default; even though sometimes tables default is latin1; and in phpbb they were utf8_bin. Now the latter shouldn't matter at all. However a problem is: if you had the option to convert to utf8 in SMF, before upgrading, as you mention in the OP, it means that SMF didn't have utf8 set... despite the data being clearly stored in utf8 fields... That's why it would be very good to go a little back in time, and please tell, when did you convert from phpbb (*IF* that was indeed the case), to what version of SMF, and if you remember... did you choose 'utf8' in the converter when doing so?
Also... when you converted from phpbb, was SMF installed as utf8 in the first place?

Also, please tell: before the actions in the original post here, that is, before first upgrading to 2.0 and converting SMF to utf8, were the characters being displayed correctly? All of them?

Sorry for the bunch of questions. Any information you could give can be useful, though.

[nicowilh]

Hey Norv,

Thanks for your reply!

I have never converted from phpbb. So this is a different installation.

The letters were displaying fine with 1.1.14.

Nicolai

[Norv]

Okay.

Can you please log in phpmyadmin, SMF 2.0 database, and run the following SQL query on it:

Code Select Expand


select value from smf_settings where variable = 'global_character_set';


[nicowilh]

Hi,

I runed the following SQL query:

"select value from smf_settings where variable = 'global_character_set'"

SELECT value
FROM smf_settings
WHERE variable = 'global_character_set'
LIMIT 0 , 30

MySQL returned an empty result set (i.e. zero rows). (Query took 0.0004 sec)

Nicolai

[Norv]

Do you have an utf8 language set as default for the forum at this moment?

[nicowilh]

Not any longer. I switched it right now.

[Norv]

Please:
1) log in phpmyadmin, and run from the SQL tab:

Code Select Expand


SHOW VARIABLES;

and if it works, paste here or save all results starting with 'character_set_' and respectively 'collation_'. This doesn't change anything, just shows the MySQL server variables.

2) run in phpmyadmin the following query:

Code Select Expand


SELECT `subject`, `body` FROM `smf_messages` WHERE id_msg = 606;

It shouldn't change anything, just show you the subject and text of a message (which currently is displayed 'broken'). Does it display correctly in phpmyadmin?

3) in the file Settings.php, make an addition:

Code Select Expand


$db_character_set = 'latin1';


Can you please tell the results?

[nicowilh]

character_set_client    utf8
character_set_connection    utf8
character_set_database    utf8
character_set_filesystem    binary
character_set_results    utf8
character_set_server    latin1
character_set_system    utf8
character_sets_dir    /usr/share/mysql/charsets/
collation_connection    utf8_unicode_ci
collation_database    utf8_unicode_ci
collation_server    latin1_swedish_ci

I run the following query in phpMyAmin: SELECT `subject`, `body` FROM `smf_messages` WHERE id_msg = 606;, but the text did not display correctly in phpMyAdmin: "Danske retningslinjer for 2 ?r praksis!"

I made the addition "$db_character_set = 'latin1';" in Setting.php, but it did not make any difference, so i took it away again.

I found somebody with a similar problem in the forum where they found a solution: http://www.simplemachines.org/community/index.php?topic=166743.msg1064592#msg1064592

[Norv]

It all depends what character set your data really is. This seems to be the relatively opposite of the situation there...

If you run in phpmyadmin:

Code Select Expand


SET NAMES 'utf8';
SELECT `subject`, `body` FROM `smf_messages` WHERE id_msg = 606;

(both in the same time, same run)
Can you please tell what does it display?

[nicowilh]

Hi,

The sentence still appear with a "?": "Danske retningslinjer for 2 ?r praksis!"

Nicolai

[Norv]

Okay, one more try please (though, well)

Code Select Expand


SET NAMES 'latin1';
SELECT `subject`, `body` FROM `smf_messages` WHERE id_msg = 606;


[nicowilh]

No change

[Norv]

Okay, thank you for the patience. We might get nowhere by guessing, and it depends on other settings too, sorry about that.

We can continue, or, at your choice, I can take a look at the data myself. I would need phpmyadmin access or a database backup (the initial 1.1.14 backup would be best). Please don't worry if you're not comfortable with doing that, it's absolutely fine if you don't agree, of course. If you agree please let me know.

[nicowilh]

Hey,

No problem. 

I have sent you accessinformation for phpMyAdmin as a personal message.

How can i send you a copy of the 1.1.14 database backup?

Thank you very much for your help.

[Aleksi "Lex" Kilpinen]

Hey, to protect you and your users I removed the SQL from your post.

Please, send Norv a PM to discuss this further ( Login details etc. )
http://www.simplemachines.org/community/index.php?action=pm;sa=send;u=211029