News:

SMF 2.0.19 has been released! Please update. Read more.

Main Menu

[3.0] Full UTF8 support

Started by Suki, September 19, 2011, 11:18:43 AM

Previous topic - Next topic

Fustrate

As long as PHP is compiled with multibyte support (which it is by default, if my memory serves correctly) then you just use the multibyte functions instead, such as mb_strlen().

http://www.php.net/manual/en/ref.mbstring.php
Steven Hoffman
Former Team Member, 2009-2012

spiros

Quote from: AngelinaBelle on November 08, 2011, 08:04:56 PM
I understand that converting code to work with UTF8 will take some work -- strlen() won't be dependable (depends on the language), etc. How difficult will it be for mod writers to convert their code?

Sometimes it is very easy, i.e.
http://www.simplemachines.org/community/index.php?topic=49410.msg2412108;topicseen#msg2412108

Oldiesmann

Someone posted a simple solution to the strlen issue on the PHP manual - strlen(utf8_decode($string)). We could probably use that as a last resort if mb_strlen and iconv_strlen aren't available.

I don't know anything about iconv so I can't say whether it would be a performance hit or not.

As far as I know, converting the tables to UTF-8 doesn't have any effect on performance.
Michael Eshom
Christian Metal Fans

spiros

By the way, joomla has gone UTF-8 only since version 1.5.

agridoc

Going UTF-8 as default: YES
Abandoning ISO/ANSI: Not so sure.

It needs thought and much work. Fantastico caused quite some problems with pseudo-UTF8 installatin in the past. This is a sticky topic in Greek support board (Google translation, not perfect but the problem can be understood)

As spiros noticed many mods don't work well with UTF-8, many mod writers have no experience with UTF-8 and table collations.

No problem for a new installation. A difficult task would be database conversion in upgrades, it's a procedure that must be done with care, prone to problems and possible data loss. Unless convinced otherwise in the future,  I believe that it should be a separate process.

Voices such as Daniel Hofverberg 's should not be neglected, although ISO-8859-1 SMF installations are the easiest case, almost no difference with UTF-8.

Heroic declarations as "throw these mods out" is something that many admin's wouldn't like to hear.
  For Greek aeromodellers and our friends around the world  - Greek Button sets for SMF - Greeklish to Greek mod
Δeν αφιερώνω χρόνο για μηνύματα σε greeklish.

spiros

Let me put it this way:

ISO-8859-1 is not compatible with everything and is bound to cause internationalization problems.
UTF-8 is compatible with everything (including the scripts one traditionally uses ISO-8859-1 for).

Joomla is a paradigm of a major open source project gone full and only UTF-8. It is perhaps the most popular open source project to date.

My question is not whether one should use UTF-8 or not; rather, why it has been so long and it has not been implemented yet, and, when does one plan to do so.

Let's focus on the future, because the future is already here.

Angelina Belle

Everything I have heard tells me SMF 3.0 will have full UTF-8 support.

So part of the future includes how to properly convert existing forums to UTF-8, with as few errors as possible.
Never attribute to malice that which is adequately explained by stupidity. -- Hanlon's Razor

spiros

I converted mine 4-5 years ago with minor issues (without the SMF converter). I think the current built-in converter is quite good anyway.

Angelina Belle

That is excellent to hear.

What tips should we offer to mod writers who will have to rewrite to handle UTF8 properly?
Never attribute to malice that which is adequately explained by stupidity. -- Hanlon's Razor

spiros

Future-proofness (as close as they can get to immortality) instead of oblivion. One mod I had to "rewrite" myself: I just had to add a single line in the SQL code.

Angelina Belle

Never attribute to malice that which is adequately explained by stupidity. -- Hanlon's Razor

Dzonny

Mods can easely adjust to utf-8 forums with changing collation of tables and some minimal changes in files if it's needed. So i guess that mods will not be our main problem.

Angelina Belle

Do you think it would be helpful to provide a "guide to making your mod UTF-8 ready", or do you think that would just be silly?
Never attribute to malice that which is adequately explained by stupidity. -- Hanlon's Razor

Dzonny

That may be usefull, of course. Many mod authors don't use utf-8, so they don't know how to make a mod utf-8 compatible. However if full utf-8 support will be provided from 3.0 version i think that there must be some notice in Customization approval guidelines about this (required utf-8 compatibility) when 3.0 goes public.

Angelina Belle

Yep.  As a doc writer, I am thinking a little bit ahead. If I can persuade savvy mod-writers to share some secrets NOW, then I won't be scrambling later.
Never attribute to malice that which is adequately explained by stupidity. -- Hanlon's Razor

agridoc

In Customization approval guidelines it should be added that the mod should work properly at least with UTF-8, better with both UTF-8/ANSI, as we are still in 2.0x and there is quite some time and there will be quite some changes in the roadmap. for 3.0 final.

There are many excellent otherwise coders that have little knowledge of database collations and proper UTF-8 implementation. So we see that new tables, or new fields in existing tables are created with latin1_swedish_ci collation regardless of SMF installation being UTF-8 or not.

SMF matured for proper UTF-8 with version 1.1 RC3, Compuart did an excellent job then.
Foreign languages will benefit from UTF-8. Although SMF has some multilingual possibilities with ISO/ANSI and a little cheating, UTF-8 has clear advantages in proper sorting and better search. Also, as many other software have matured for UTF-8 or selected UTF-8 only bridging will usually be without problems.

One question
As far as I know, SMF from version 1.1 RC3 up is fully compatible with UTF-8. So the main question is leaving or not ISO/ANSI. Do I miss anything?
  For Greek aeromodellers and our friends around the world  - Greek Button sets for SMF - Greeklish to Greek mod
Δeν αφιερώνω χρόνο για μηνύματα σε greeklish.

Fustrate

If people don't understand how to make their modifications UTF-8 compatible, we can certainly help them learn.

In order to reduce development and compatibility headaches, though, it's best to use UTF-8 only, instead of allowing a mix of encodings and hoping that most users choose UTF-8.
Steven Hoffman
Former Team Member, 2009-2012

Angelina Belle

OK. 
I'm just an ignorant doc writer.  So if you write down the most important points, I can try to turn your gibberish into something the newer coders might understand.
Never attribute to malice that which is adequately explained by stupidity. -- Hanlon's Razor

agridoc

Quote from: Fustrate on March 30, 2012, 01:20:53 PMIn order to reduce development and compatibility headaches, though, it's best to use UTF-8 only, instead of allowing a mix of encodings and hoping that most users choose UTF-8.

Most users will use the default in installation. So with . With the addition of a warning for ISO/ANSI no problem with new installations.


@ AngelinaBelle: Some integrated (not Wiki) guidelines for proper language set selection depending on SMF installation would be useful for screens in
Administration Center » Languages
I find it also related to proper Localization.


Is it necessary to wait for 3.0 to make UTF-8 the default preselection?  Removing ISO/ANSI is something different. Adding UTF-8 as default in one next 2.0 subversion, after some additions, will reduce the following cases until SMF 3.0 will be ready.

What needs much thought and care are existing forum installations.

If I started my SMF today I would choose UTF-8. However, my SMF started 7 years ago, when SMF was not UTF-8 ready, as well as hosts and other software. Although i have helped others to convert, I don't find the time to do mine, it's not only SMF involved. The converter works well for all databases with SMF_prefix but needs careful planning and time, sometimes not so easy to find. I have seen in the past other software that had gone UTF-8 only without proper development and proper converter.

These cases are different, also as I have read, many admins would not like to be forced to UTF-8, although most understand that sooner or later will have to do the big jump.
  For Greek aeromodellers and our friends around the world  - Greek Button sets for SMF - Greeklish to Greek mod
Δeν αφιερώνω χρόνο για μηνύματα σε greeklish.

spiros

Quote from: agridoc on March 30, 2012, 02:34:40 PM
These cases are different, also as I have read, many admins would not like to be forced to UTF-8, although most understand that sooner or later will have to do the big jump.

Well, this is the case exactly with Joomla. People had to go along with UTF-8 default (and only option) when upgrading to 1.5 whether they liked it or not. I did not see anybody getting any harm out of it.

Advertisement: