News:

SMF 2.1.2 has been released! Take it for a spin! Read more.

Main Menu

[3.0] Full UTF8 support

Started by Suki, September 19, 2011, 11:18:43 AM

Previous topic - Next topic

Suki

Hi all, the Devs are considering going with full UT8 support instead of the current ANSI/UTF8.


We will like to hear all opinions about this, please try to consider all angles that can possibly influence this such as server requirements,  database sizes, hosting restrictions, etc.


Please share your thoughts on this :)
Disclaimer: unless otherwise stated, all my posts are personal and does not represent any views or opinions held by Simple Machines.

Making tough decisions, so you don't have to.

Robert.

I always use the latin set, but going full utf8 would be a good idea though

live627

It's absolutely a no brainer. Chaarset support  problems would mostly vanish, large mods don't have to ****** around with including ANSI/UTF languages, translators need only to translate into utf-8 encoding.

青山 素子

I have to agree with this. If it is technically possible, moving to full Unicode support would be beneficial. Most hosting providers have upgraded SMF's dependencies to versions that support this.
Motoko-chan
Director, Simple Machines

Note: Unless otherwise stated, my posts are not representative of any official position or opinion of Simple Machines.


Xarcell

+1

As it was said, a no brainer.

青山 素子

Quote from: Xarcell on October 05, 2011, 01:09:04 PM
+1

As it was said, a no brainer.

It certainly wasn't when SMF 2.0 was being designed. Back then, PHP4 was still widely used (still is in some areas...) and proper Unicode support was difficult to come by without a ton of effort. With the huge move to PHP5 and cleaned up support, it would be silly to not fully support Unicode in a widely-used software.
Motoko-chan
Director, Simple Machines

Note: Unless otherwise stated, my posts are not representative of any official position or opinion of Simple Machines.


Dzonny

+1
It would be easier if new members don't have to deal with charset problems, full utf8 support would fix many possible issues.

Daniel Hofverberg

Of course it's preferable with full UTF-8 support for those that do want it. However, I do not want to be forced to use UTF-8, as I prefer good old ISO-8859-1.

Nightwish

Quote from: Daniel Hofverberg on October 11, 2011, 03:54:45 AM
Of course it's preferable with full UTF-8 support for those that do want it. However, I do not want to be forced to use UTF-8, as I prefer good old ISO-8859-1.
Bad idea.

Seriously, it's 2011. The web should be UTF-8 only. Period. 7/8bit character sets are a relict of ancient days and should die. They are among the most annoying things a developer has to deal with - Unicode makes things so much easier. Time to move on and forget old habits.
Every program has at least one bug and can be shortened by at least one instruction -- from which, by induction, one can deduce that every program can be reduced to a single instruction that doesn't work.
EoS - SMF-based forum under development.

Fustrate

Quote from: Daniel Hofverberg on October 11, 2011, 03:54:45 AM
Of course it's preferable with full UTF-8 support for those that do want it. However, I do not want to be forced to use UTF-8, as I prefer good old ISO-8859-1.

Do you have a reason for preferring it, or do you just not want to change? Honest question.
Steven Hoffman
Former Team Member, 2009-2012

Daniel Hofverberg

As the rest of my web site is using ISO-8859-1, using UTF-8 for just the forum would cause problems with the integration. That would make SSI.php and other aspects a pain to deal with, unless changing the character set on the entire site. I also don't see any specific need for UTF-8 on my site, as all characters I need to use for the Swedish language is present in Latin1.

As my site consist of closer to 1000 pages, moving the entire site over to UTF-8 with no real benefit doesn't really sound too pleasing...

Fustrate

I believe there are PHP functions for converting between character sets, though I don't use them often enough to remember their names.

Found it, though: http://www.php.net/manual/en/function.iconv.php

You'd just use that on anything from the forum, to convert from UTF8 to ISO-8859-1
Steven Hoffman
Former Team Member, 2009-2012

青山 素子

Quote from: Daniel Hofverberg on October 11, 2011, 03:54:45 AM
Of course it's preferable with full UTF-8 support for those that do want it. However, I do not want to be forced to use UTF-8, as I prefer good old ISO-8859-1.

Could be worse, it could be windows-1252 (ugh).


Quote from: Daniel Hofverberg on October 11, 2011, 05:10:21 AM
I also don't see any specific need for UTF-8 on my site, as all characters I need to use for the Swedish language is present in Latin1.

Then you shouldn't see a difference, actually. The only possible issues would be with characters outside Latin-1 like the "smart quote" and such. Heck, you could probably send your Latin1 pages as UTF8 without any changes as HTML entities would still work the same way for characters.
Motoko-chan
Director, Simple Machines

Note: Unless otherwise stated, my posts are not representative of any official position or opinion of Simple Machines.


spiros

I could not agree more. There are so many mods which do not support UTF-8 and one has to hack them in order to work in a UTF-8 forum. This should NOT be happening.

Kindred

well, that would be a MOD problem, not an SMF issue....
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

spiros

Indeed, but if SMF does not enforce strict guidelines (i.e. mod compatibility with UTF-8), many mod developers are inclined to ignore it.

Nightwish

Quote from: spiros on November 05, 2011, 08:37:13 AM
I could not agree more. There are so many mods which do not support UTF-8 and one has to hack them in order to work in a UTF-8 forum. This should NOT be happening.
Then trash these mods, period. No mod can be important enough to stand in the way of a modern design and getting rid of ancient character set support *is* an important part of a modern design.

Seriously, it's 2011, people who still insist on ancient 8bit character set support must have been sleeping under a rock for the past 10 years. Most modern web applications are UTF-8 only, because it is, by far, the easiest way to support multiple languages.
Every program has at least one bug and can be shortened by at least one instruction -- from which, by induction, one can deduce that every program can be reduced to a single instruction that doesn't work.
EoS - SMF-based forum under development.

spiros

I thought that php 6 (what happened to it by the way?) was meant to be the version to fully support UTF-8 in its core. If that had happened, there should not have been many excuses left.

Fustrate

iirc, they turned PHP 6 into PHP 5.4
Steven Hoffman
Former Team Member, 2009-2012

Angelina Belle

I understand that converting code to work with UTF8 will take some work -- strlen() won't be dependable (depends on the language), etc. How difficult will it be for mod writers to convert their code?

If someone is trying to integrate an ISO-8859-1 website with a UTF8 SMF forum, and was using the iconv functions to convert EVERYTHING, what kind of performance hit would that take?

What are the implications of switching to using all UTF8 in MYSQL tables? Any affect on performance?

I have never made the switch to UTF-8 myself, since I don't feel I really understand the implications.
Never attribute to malice that which is adequately explained by stupidity. -- Hanlon's Razor

Advertisement: