Advertisement:

Author Topic: Convert to UTF-8?  (Read 2901 times)

Offline Petter B

  • Semi-Newbie
  • *
  • Posts: 85
Convert to UTF-8?
« on: March 07, 2017, 03:14:34 AM »
I run a swedish forum and have never had problems with our letters åäö.
Only in some parts of the Tapatalk-app they look weird. Now I installed SMFpacks Social Login and when someone register a new account via FB, åäö doesn't work.
I get Petter Bergström instead of Petter Bergström

Is there an easy way to fix this or do I need to convert the forum to UTF-8?
I guess it´s not now because the language file says: Swedish ISO-8859-1

There is a setting in the forum admin: Convert database och data till UTF-8.
Should I press the convert-button? (feels scary...)
and do I in that case need to do all this first:
Quote
How to convert to UTF-8[edit]
This procedure will work well if you have used the recommended character sets in your language files in the past.
1. Start with a backup of your database. Character set conversions nearly always work correctly, but it is best to be prepared for the unexpected.
2. Check your default language file (Administration Center » Languages » Edit Languages ) and make a note of the character set used.
3. Go to Forum Maintenance > Convert the database and data to UTF-8 (this option will only be available if SMF detects a database version which supports UTF-8).
4. Select the character sets for your data (member posts) and database. By default, SMF will choose the character set of your default language file.
5. Press proceed, and your database will be converted. Depending on the size of your database, the conversion process might stop temporarily from time-to-time to avoid overloading the server. If this was successful, your forum should be converted to UTF-8.
6. For each of language pack currently in use, replace it with the UTF-8 version of that language pack.
7. Once all the UTF-8 language packs have been installed, convert the language settings of each user by running this query:
UPDATE smf_members
SET lngfile = CONCAT(lngfile, '-utf8')
WHERE lngfile != ''
8. In your Administration Center, change the default language, ensuring that you choose the UTF-8 version.
9. Check to see if all your data was properly converted.
10. If any of your posts contain HTML entities, you will want to convert these to UTF-8 as well by running "Convert HTML-entities to UTF-8 characters"

Offline Illori

  • Project Manager
  • SMF Master
  • *
  • Posts: 47,163
Re: Convert to UTF-8?
« Reply #1 on: March 07, 2017, 05:05:34 AM »
yes you need to convert to UTF-8.

Offline Petter B

  • Semi-Newbie
  • *
  • Posts: 85
Re: Convert to UTF-8?
« Reply #2 on: March 07, 2017, 05:17:27 AM »
OK thanks.
So I just do the list above?
I have a very big forum, are there chances this will break it?

Offline Illori

  • Project Manager
  • SMF Master
  • *
  • Posts: 47,163
Re: Convert to UTF-8?
« Reply #3 on: March 07, 2017, 05:27:30 AM »
then do a backup of your database in case of any issue you can always revert back.

Offline Petter B

  • Semi-Newbie
  • *
  • Posts: 85
Re: Convert to UTF-8?
« Reply #4 on: March 07, 2017, 08:17:42 AM »
Instead of downloading the BIG database, is it OK to just make copy of it in the phpMyAdmin?
(don´t remember the command for this...)

Offline Illori

  • Project Manager
  • SMF Master
  • *
  • Posts: 47,163

Offline shawnb61

  • Charter Member
  • Jr. Member
  • *
  • Posts: 271
    • sbulen on GitHub
Re: Convert to UTF-8?
« Reply #6 on: March 07, 2017, 11:39:29 PM »
That looks like an excellent set of instructions... 

Something to look out for:
I recall some mods needed to be reinstalled, specifically the ones that made entries to the language files. 
Address the process rather than the outcome.  Then, the outcome becomes more likely.   - Fripp

Offline shawnb61

  • Charter Member
  • Jr. Member
  • *
  • Posts: 271
    • sbulen on GitHub
Re: Convert to UTF-8?
« Reply #7 on: March 07, 2017, 11:46:27 PM »
Instead of downloading the BIG database, is it OK to just make copy of it in the phpMyAdmin?
(don´t remember the command for this...)

That - very helpful - feature is under 'Operations' in phpMyAdmin. 

I do that all the time on my test environment under localhost.   

I have never tried it on my ISP, the main reason is that in the past, any DBs not created under their database cPanel applet were not properly accessible.  (E.g., if created using a CREATE DATABASE in pmpMyAdmin it ended up in limbo...  They really wanted DBs created via cPanel...)   

So...   I would experiment with the phpMyAdmin COPY function to confirm it works as expected & the results are usable. 
Address the process rather than the outcome.  Then, the outcome becomes more likely.   - Fripp

Offline Petter B

  • Semi-Newbie
  • *
  • Posts: 85
Re: Convert to UTF-8?
« Reply #8 on: March 09, 2017, 04:32:46 AM »
Tried this in my dev-server now but got:

Quote
Duplicate entry 'vår-4328' for key 'PRIMARY'
Fil: /home/d/dev/www/Sources/ManageMaintenance.php
Rad: 705

NOTE: It appears that your database may need to be upgraded. Your forum files belonging to the SMF version 2.0.13, while your database belong to version 2.0.2. It is very possible that this error can be resolved if you're running the latest version of Upgrade.php.

What/where is Upgrade.php and should I run it?

Offline Illori

  • Project Manager
  • SMF Master
  • *
  • Posts: 47,163
Re: Convert to UTF-8?
« Reply #9 on: March 09, 2017, 05:09:13 AM »
you dont need to run upgrade.php there are no database changes to apply. you do need to fix the duplicate entry though.

admin -> security and moderation enable include database query in the error log, run the convert again and check your error log for what query is failing then go from there to fix the issue.

Offline Petter B

  • Semi-Newbie
  • *
  • Posts: 85
Re: Convert to UTF-8?
« Reply #10 on: March 10, 2017, 02:46:58 AM »
Got this...  :o

Offline Petter B

  • Semi-Newbie
  • *
  • Posts: 85
Re: Convert to UTF-8?
« Reply #11 on: March 10, 2017, 03:40:19 AM »
After a few tries I put forum in maintainance mode and got to run the conversion for a couple of minutes but then I got 500 (Internal Server Error)...  >:(

And now I can not log in as admin since my password contains the letter ä.
Glad it´s only my test forumet that I screwed up..  :P

I´m gonna forget this whole UTF8-thing, forum works fine without it. Except for social login via FB. Is there a workaround for that?
« Last Edit: March 10, 2017, 04:20:33 AM by Petter B »

Offline Kindred

  • The Mean One
  • Support Specialist
  • SMF Legend
  • *
  • Posts: 54,679
  • Gender: Male
    • Kindred-999 on GitHub
Re: Convert to UTF-8?
« Reply #12 on: March 10, 2017, 08:35:14 AM »
well 2.1 is ONLY UTF-8, so you'll have to do it sometime...

as for facebook, you would have to ask the mod author
Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

Offline Petter B

  • Semi-Newbie
  • *
  • Posts: 85
Re: Convert to UTF-8?
« Reply #13 on: March 13, 2017, 04:08:21 AM »
Thanks, giving the test forum another try.
About this:
Quote
UPDATE smf_members
SET lngfile = CONCAT(lngfile, '-utf8')
WHERE lngfile != ''

-should I run it exactly like this or put the name of the language file in it somewhere?

Offline Arantor

  • Resident Overthinker
  • SMF Friend
  • SMF Legend
  • *
  • Posts: 67,343
    • Arantor on GitHub
Re: Convert to UTF-8?
« Reply #14 on: March 13, 2017, 04:12:54 AM »
Exactly like that. It works for all languages.
To assume is to hope that those who came before had the presence of mind and capacity to implement the dreams of those who would come after.

You either die a hero or live long enough to see yourself become the villain. It seems you have chosen which, and now I must do the same.

Offline Petter B

  • Semi-Newbie
  • *
  • Posts: 85
Re: Convert to UTF-8?
« Reply #15 on: March 13, 2017, 06:10:35 AM »
Hmmm.
Seems like only 14 rows got affected, and no difference in forum, still strange characters...

"Affected rows: 15 (Query took 0.3936 sec)"
« Last Edit: March 13, 2017, 06:21:22 AM by Petter B »

Offline @rjen

  • Jr. Member
  • **
  • Posts: 145
  • Gender: Male
    • FJR-club Nederland
Re: Convert to UTF-8?
« Reply #16 on: March 13, 2017, 08:06:00 AM »
Did you check this post:

http://www.simplemachines.org/community/index.php?topic=357401.msg2435528#msg2435528

Quote
Changing the users' language settings is not done in any file. You go into phpMyAdmin and in the SQL tab run the given query to change everyone's <language> to <language>-utf8. Of course, first you will check to see if any (or all) are already -utf8 (which means the help page needs updating!). After running the SQL to update the entries, list them all again to make sure all where changed and that you don't have any unexpected language files that you need to add (e.g., you had "albanian", but you forgot to load "albanian-utf8". Make sure that your default language is likewise updated. Make sure that Settings.php has the entry for "UTF8" now added to it, so that the pages will be displayed with UTF8 encoding rather than Latin-1.

Running SMF 2.0 with Tinyportal 1.3R at www.fjr-club.nl
Testing SMF 2.1 beta 3 with Tinyportal 2.0 at http://test2.fjr-club.nl/

Offline Petter B

  • Semi-Newbie
  • *
  • Posts: 85
Re: Convert to UTF-8?
« Reply #17 on: March 13, 2017, 08:22:29 AM »
Thanks! It seems like the field is empty for all except 15 members.
How can I change the command:
UPDATE smf_members
SET lngfile = CONCAT(lngfile, '-utf8')
WHERE lngfile != ''

So that everyone becomes Swedish UTF-8 ( sv_SE.utf8  ) ?

Offline Arantor

  • Resident Overthinker
  • SMF Friend
  • SMF Legend
  • *
  • Posts: 67,343
    • Arantor on GitHub
Re: Convert to UTF-8?
« Reply #18 on: March 13, 2017, 08:39:49 AM »
You don't change every record, blank means "use the forum default". So if the forum default is that, they'll use that.
To assume is to hope that those who came before had the presence of mind and capacity to implement the dreams of those who would come after.

You either die a hero or live long enough to see yourself become the villain. It seems you have chosen which, and now I must do the same.

Offline Petter B

  • Semi-Newbie
  • *
  • Posts: 85
Re: Convert to UTF-8?
« Reply #19 on: March 13, 2017, 01:29:20 PM »
Yes, I have set swedish UTF-8 as standard.
OK, what might be wrong then?
Still strange letters or words cut of after åäö...