The translator should add the Turkish language translation as follows

[Arantor]

You already have a solution for you, add the .utf8! It's just the solution for everyone else that's trickier.

[Seyit replikacep.com]

You already have a solution for you, add the .utf8! It's just the solution for everyone else that's trickier.

I want permanent solution I know how to add manually

[Steve]

Would uninstalling Pretty Urls solve the problem?

[Seyit replikacep.com]

Would uninstalling Pretty Urls solve the problem?

actually there is no problem, there is no problem in English, it causes problems in Turkish language

permanent solution add .utf8 to language file

[Arantor]

You don't understand, this might be a solution today but it might break other users (Turkish) setup and might even break on you if your host upgrades their server in a certain way.

It's really more complicated than you're suggesting.

[Seyit replikacep.com]

Then do a test to see if there is a possibility of error in the installation or elsewhere, I think this is appropriate.

[Arantor]

Do you have any idea how many variations this is?

This is every version of Linux for the last 5 years.

[m4z]

Hard disagree, actually.

The mod asks the operating system "how do I translate these letters". Said system libraries are ending up misconfigured because of core SMF code. The mod isn't calling setlocale or deciding what locale should be used, SMF is.

Ok, that wasn't obvious to me from your previous post. Your posts are often hard to understand for someone who know less about, well, all of SMF, because you explain very little.

is it that hard? you just add one .uft8

Is it that hard to understand that the problem doesn't exist for most other people, and that in only occurs with some mods?

First, you demanded to change something that works fine for basically everybody else, without giving any explanation why it should be changed (re-read your first post).
Second, your posts are very short and riddled with spelling and grammatical errors. It took you 6 posts before people were able to begin to understand your problem.
Third, you seem to ignore half of the questions you're asked (or you answer them in a way that it's not obvious what you're answering or referring to).

We're all volunteers, and you're not only not helping us to help you, you're making us (me, at least) not want to help you.

If I understand Arantor correctly this time, whether the default (without the ".utf8" suffix) causes problems depends on the server locale settings. Different Linux distributions and even distribution versions might have different defaults. Then there's the server hosters, who might make their own adjustments, and then there's the individual server admin who might change things. All that will have an effect on the outcome.

Whether or not the problem is in core SMF, it does not seem to cause issues there (or you didn't take your time to explain them). So I assume it's unlikely things will be changed.

[Arantor]

because you explain very little.

I don't remember how much of everything everyone knows, and frankly I have neither the time nor the patience to explain every thing every single time to everyone. But I'll come back to that later.

Is it that hard to understand that the problem doesn't exist for most other people, and that in only occurs with some mods?

Given that the OP is almost certainly reading this through Google Translate which will make something of a mess of the back and forth, it genuinely might be that hard to understand. cf. understanding what the OP's original problem even was.

It never helps to be condescending and dismissive towards the users - a mistake I only make after I already gave them an opportunity and by which time I'm frustrated that they're not able to get help. They're doing the best they can with what they have. Not everyone is smart like you.

works fine for basically everybody else

Debatable. The actual reality is that SMF doesn't do much of the locale-specific stuff properly and has a 'good enough' aspect to it mostly because when some of the original locale-specific stuff was written, the locale libraries were just crap and so SMF did it itself.

The most obvious manifestation in the OP's case is links straight up breaking because SMF botches translation between upper and lower case letters in a target language, or transliterating between accented and non-accented letters. While this is most manifest inside a language like Turkish if you're using a pretty URLs mod, literally any instance of switching upper to lower case or vice versa, or munging accented letters will be hit and miss in any language not English.

German is one of the better handled ones because over time there were enough very-technical German people to iron out the issues. Turkish has not had that benefit. Other languages' mileage varies.

Second, your posts are very short and riddled with spelling and grammatical errors. It took you 6 posts before people were able to begin to understand your problem.

As far as I know, the OP does not speak English at all. What you're getting is the best result of Google Translate, or some other service. Note that while languages like German are reasonably well handled in Google Translate, Turkish has historically not been one of those as well supported.

Different Linux distributions and even distribution versions might have different defaults. Then there's the server hosters, who might make their own adjustments, and then there's the individual server admin who might change things. All that will have an effect on the outcome.

Sort of. Since we're apparently at the point where it needs an explanation, I might as well actually do so.

So, operating systems amongst other things are usually responsible for providing information and supporting infrastructure for 'given that I am set up for the user, I know what language they are using, I presumably know how to format numbers for them and outline the correct time format' and things like that.

One of the things that happens as a result of all this is the locale database. The locale database covers a whole bunch of things - it covers as mentioned things like time formatting, date formatting, number formatting. When SMF was first written, none of these things were in any way consistent, so SMF literally rolled its own. You still see the remnants of this today.

But one of the other things the locale database does is provide transliteration services. What the rules are for a given language for ordering letters (e.g. the correct handling for ø, where in Norwegian (Bokmal at least if not Nynorsk, I don't remember) and Danish, this is a separate letter ordered after z, but in other languages it is more often approximated to the oe digraph), what the rules are for case shifting, transliteration, currency symbol handling and so on.

If that weren't bad enough you have to remember that there's also character encoding. The world has not universally adopted UTF-8 because there are times this is wholly not going to fly. (Most of Windows runs internally on a UTF-16 variant for example.) So here's your first drama: some vendors ship their copy of the locale database in local encoding, so tr_TR without a suffix might ship in ISO-8859-9 or it might be UTF-8... whether the .utf8 suffix is in use is complicated, since they might ship tr_TR in UTF-8 without the UTF-8 suffix. Or they might ship both versions. And they might not use .utf8 as a suffix, they might use .utf-8.

The whole thing is a mess.

Let me give you a practical example of what I mean. I have a Docker container for this stuff, it runs based on Ubuntu but the only specific change I made for this was to install the en language pack.

Code Select Expand

root@smf20:/# locale -a
C
C.UTF-8
POSIX
en_AG
en_AG.utf8
en_AU.utf8
en_BW.utf8
en_CA.utf8
en_DK.utf8
en_GB.utf8
en_HK.utf8
en_IE.utf8
en_IL
en_IL.utf8
en_IN
en_IN.utf8
en_NG
en_NG.utf8
en_NZ.utf8
en_PH.utf8
en_SG.utf8
en_US.utf8
en_ZA.utf8
en_ZM
en_ZM.utf8
en_ZW.utf8
The first three you can reasonably assume will be there. Perfectly fine for English, naturally.

But as you can see, the language-pack-en Debian package installs a bunch of languages, some of which get both a UTF-8 and non-UTF-8 edition (e.g. en_AG, en_IN) added, the rest only get a .utf8 version.

These will behave differently for things like this transliteration. Any instance of $smcFunc['strtoupper'] and $smcFunc['strtolower'] is *potentially* affected as well as any use of iconv, and some other ancillary functions I don't remember off hand.

Out of interest I installed the Ubuntu/Debian package language-pack-tr which as suspected *only* installs tr_TR.utf8.

So for one of the more popular distributions, what SMF has in its configuration is demonstrably wrong. But I wouldn't be surprised if Red Hat/CentOS did kind of the opposite, I haven't test-driven in a while but I have a memory of this being the case. It's been about a decade since I did any real investigation into this.

But one is not beholden to yum for their packages and plenty of hosts do do their own thing, which translates to that the .utf8 packages are more likely to be common and installed. Which leads back to my original comment that it probably should be done but that there's a non-trivial chance for some users that this will break in the opposite way and you have no way to know which is which.

And you can see from my example that .utf8 as a suffix is not 100% consistent. The setlocale page even provides some more outlandish examples that were historically valid in some contexts, such as German being made up of whichever was first out of 'de_DE@euro', 'de_DE', 'de', 'ge', 'deu_deu' (looking across the examples)

What might be worth doing is allowing $txt['lang_locale'] to be an array and include multiple possible fallbacks, in this case tr_TR.utf8 then tr_TR.utf-8 then tr_TR and try loading them in that order (since setlocale allows for doing this). I wouldn't necessarily offer all possible permutations up, however.

(or you didn't take your time to explain them)

This. You see, there are two choices. Either a) I can explain there's a problem, provide some hints in between all the other things in my life (considering that, for example, my Thursday 10th March when this thread started was a full work day ending at 5.30pm, followed by packing for a trip, arriving at the hotel at 11pm then carrying on working until 2am, sleeping until 6am, then a full work day until arriving back at the hotel at 7pm for dinner, and then travelling back today), or I can b) answer a fraction of the topics and lose what little patience I do have by doing everyone's homework for them.

You see, this is not even remotely the first time I've done even this specific problem; the last time I actually looked was about a decade ago when I solved this problem in Wedge, where we set the locale loader to load .utf-8 then .utf8 which was pretty consistent for the hosting environments we took the time to examine.

Ultimately, I lack the patience in general to do the legwork. The reason I lack the patience is because of the sniffy dismissive attitude the team has in general about accepting its responsibilities. Too many times I've done the legwork to bring about the change I wanted to see and the net result from the team was to stiff me for the bill at the end.

End of the day I don't really care if SMF makes this change or not. It literally does not affect me. But even as much as I have been known to find the OP annoying with the repetitive statements about what he thinks SMF should do, the reality is that he does have a legitimate problem and that the problem is kinda SMF's fault.

I say kinda because it's a complete disaster in general as we've seen but someone has to go to bat for the users who are just doing the best they can with what they have. Honestly, most people would have just reported there's a bug, would have no way of knowing where to start - and likely neither would the mod author - which would have just lead to this not ever getting solved, and then the user going off to another platform that did work.

Becuase, hilariously, that's one of the biggest causes of defection: people just not getting answers to their problems, because too few of the questions ever really get under the noses of the people who know enough to answer them properly. And after a while of assuming it'll never get solved, they either make do or go elsewhere entirely, often the latter if the forum is otherwise successful.

So I assume it's unlikely things will be changed.

And you seem surprised that I don't feel the need to explain anything every time, when I can assume it's unlikely to be changed because it doesn't affect you. And you seem to be under the illusion that people who don't speak English have to meet your personal standards of minimum communication fluency in order to get help.

If you bury things like this in the language boards, they are guaranteed to never get in front of the people who need to see them in order to get solved.

[Seyit replikacep.com]

is it that hard? you just add one .uft8

Is it that hard to understand that the problem doesn't exist for most other people, and that in only occurs with some mods?

First, you demanded to change something that works fine for basically everybody else, without giving any explanation why it should be changed (re-read your first post).
Second, your posts are very short and riddled with spelling and grammatical errors. It took you 6 posts before people were able to begin to understand your problem.
Third, you seem to ignore half of the questions you're asked (or you answer them in a way that it's not obvious what you're answering or referring to).

We're all volunteers, and you're not only not helping us to help you, you're making us (me, at least) not want to help you.

If I understand Arantor correctly this time, whether the default (without the ".utf8" suffix) causes problems depends on the server locale settings. Different Linux distributions and even distribution versions might have different defaults. Then there's the server hosters, who might make their own adjustments, and then there's the individual server admin who might change things. All that will have an effect on the outcome.

Whether or not the problem is in core SMF, it does not seem to cause issues there (or you didn't take your time to explain them). So I assume it's unlikely things will be changed.

I regret opening this topic.
i don't speak english i'm sorry

I use google translate, some of my writings are not fully translated

[Steve]

I regret opening this topic.

Do not feel that way. You have a problem that you would like looked at and the only way is to open a topic.

[m4z]

because you explain very little.

I don't remember how much of everything everyone knows, and frankly I have neither the time nor the patience to explain every thing every single time to everyone. But I'll come back to that later.

I get that. I wasn't criticizing, only pointing out.

Is it that hard to understand that the problem doesn't exist for most other people, and that in only occurs with some mods?

Given that the OP is almost certainly reading this through Google Translate which will make something of a mess of the back and forth, it genuinely might be that hard to understand. cf. understanding what the OP's original problem even was.

It never helps to be condescending and dismissive towards the users - a mistake I only make after I already gave them an opportunity and by which time I'm frustrated that they're not able to get help. They're doing the best they can with what they have. Not everyone is smart like you.

I don't understand what is condescending or dismissive about trying to point out our (or rather, my) problems with this thread and (given my understanding at the time) trying to explain why a fix 1.) isn't easy, 2.) is unlikely (as far as I can tell), and how the OP is making it hard for us (me) to help them or even want to help them.

(My phrasing was mirroring (even quoting) the OP.)

works fine for basically everybody else

Debatable. The actual reality is that SMF doesn't do much of the locale-specific stuff properly and has a 'good enough' aspect to it mostly because when some of the original locale-specific stuff was written, the locale libraries were just crap and so SMF did it itself.

The most obvious manifestation in the OP's case is links straight up breaking because SMF botches translation between upper and lower case letters in a target language, or transliterating between accented and non-accented letters. [...] literally any instance of switching upper to lower case or vice versa, or munging accented letters will be hit and miss in any language not English.

My point was based on the fact that we haven't had similar reports, with one of our Turkish translators saying he hasn't seen the problem. (I don't know enough about SMF internals to know how/where the problem would manifest, but at least it doesn't seem to be blatantly obvious.)

Second, your posts are very short and riddled with spelling and grammatical errors. It took you 6 posts before people were able to begin to understand your problem.

As far as I know, the OP does not speak English at all. What you're getting is the best result of Google Translate, or some other service. Note that [...] Turkish has historically not been one of those as well supported.

Thanks, I didn't know (the extent of) that.

Different Linux distributions and even distribution versions might have different defaults. Then there's the server hosters, who might make their own adjustments, and then there's the individual server admin who might change things. All that will have an effect on the outcome.

Sort of. Since we're apparently at the point where it needs an explanation, I might as well actually do so. [...]
The world has not universally adopted UTF-8 because there are times this is wholly not going to fly. (Most of Windows runs internally on a UTF-16 variant for example.) So here's your first drama: some vendors ship their copy of the locale database in local encoding, so tr_TR without a suffix might ship in ISO-8859-9 or it might be UTF-8... whether the .utf8 suffix is in use is complicated, since they might ship tr_TR in UTF-8 without the UTF-8 suffix. Or they might ship both versions. And they might not use .utf8 as a suffix, they might use .utf-8.
[...] some of which get both a UTF-8 and non-UTF-8 edition (e.g. en_AG, en_IN) added, the rest only get a .utf8 version.

These will behave differently for things like this transliteration. Any instance of $smcFunc['strtoupper'] and $smcFunc['strtolower'] is *potentially* affected as well as any use of iconv, and some other ancillary functions I don't remember off hand. [...]

Thanks, that was (in parts) what I assumed from your earlier post.

What might be worth doing is allowing $txt['lang_locale'] to be an array and include multiple possible fallbacks, in this case tr_TR.utf8 then tr_TR.utf-8 then tr_TR and try loading them in that order (since setlocale allows for doing this). I wouldn't necessarily offer all possible permutations up, however.

Sounds reasonable. Sesq knows infinitely more about Unicode than I do, so he might come up with something.

(or you didn't take your time to explain them)

This. You see, there are two choices. Either a) I can explain there's a problem, provide some hints in between all the other things in my life [...], or I can b) answer a fraction of the topics [...]

1. This (everything after the quote) was directed at the OP. Sorry if that wasn't clear.
2. I know the feeling, I too do SMF in my free time.
3. I realize you might also be saying that the OP might be in a similar situation. I also know that; I too report bugs to SMF and other software.

I was trying to say (to the OP): Report the problem as precisely as possible, give all (possibly) relevant info and details about the environment etc., then it is easier to pinpoint the problem and find a solution. Don't "demand" a fix, don't "shout" in all caps, we're all doing this in our free time. Don't be impatient when we ask more questions, we're just trying to find out what exactly the problem is.
We're not saying "We don't believe you" or "We don't want to help you", we're saying "Some pieces of the puzzle are still missing, and we need those".

[...] The reason I lack the patience is because of the sniffy dismissive attitude the team has in general about accepting its responsibilities. [...]

I swear I only am like this when I feel somebody doesn't appreciate the time I put into trying to solve their problems.

But even as much as I have been known to find the OP annoying with the repetitive statements about what he thinks SMF should do, the reality is that he does have a legitimate problem and that the problem is kinda SMF's fault.

I don't disagree, (up until my rant) I was still trying to figure out the details of whether the problem is in SMF or a mod. I personally can't fix the problem, I can only try to get more info from the person with the problem, and try to communicate what I think will happen next.

people just not getting answers to their problems, because too few of the questions ever really get under the noses of the people who know enough to answer them properly. And after a while of assuming it'll never get solved, they either make do or go elsewhere entirely, [...].

I have been on that side too, and I don't like it either.

But on the other side, the developers (or myself in my day job as a sysadmin) have to try to cherry-pick the most important (or most visible) problems from the unsurmountable torrent of reports, because working on all the small things will not bring the project forward (ot at least not in the desired direction).

So I assume it's unlikely things will be changed.

And you seem surprised that I don't feel the need to explain anything every time, when I can assume it's unlikely to be changed because it doesn't affect you.

I wrote that under the assumption the problem was not in core SMF.

And you seem to be under the illusion that people who don't speak English have to meet your personal standards of minimum communication fluency in order to get help.

My core problem was with the perceived attitude of the OP. Everything else I mentioned because I was trying to explain to the OP that it is very hard to understand them, and that makes it even harder to find and possibly fix the problem they are trying to report and get fixed.

If you bury things like this in the language boards, they are guaranteed to never get in front of the people who need to see them in order to get solved.

I know, I'm wearing the "proxy" hat in a few SMF roles (German language board,
translation/English source string issues, ...).

I regret opening this topic.
i don't speak english i'm sorry

I use google translate, some of my writings are not fully translated

Ok, thanks for confirming.

As Steve said: Please keep reporting problems. But please also realize that (with only a handful of developers) most bugs take a long time to fix. And if we can't even identify all the details of the problem in the first place, how can we fix it at all?

(I think we have these details for this problem now, thanks to Arantor.)

[Arantor]

I get the frustration with people who need help and who, for whatever reason, are making it hard to help. I really do. It's one of the banes of my life. But... getting cranky doesn't help. I should know! I'm trying to be less cranky with the people who are doing their best and trying to meet me in the middle, but it does leak out from time to time.

And the 'is it core' vs 'is it a mod' is such a huge question mark. Is it a core bug if the only *major* time you see it is in a mod? Well incidentally the bug Sesq fixed with the Q&A would run into issues with the locale if the locale were set wrong, so... it kinda does affect core too.

Just fixing it is hard

[Max22]

Tracked as #7388

[Seyit replikacep.com]

Tracked as #7388

Do you think they found a solution?

[Sesquipedalian]

Do you think they found a solution?

#7392

[Arantor]

I'm not *entirely* clear that adding .UTF8 will help (when the common locale packages are .utf8 as demonstrated) but in general this looks like an improvement!

[Sesquipedalian]

We try .UTF-8, .UTF8, .utf-8, and .utf8. The system will chose whichever one matches its particular variant.

Beyond that, our $smcFunc['strtolower'](), $smcFunc['strtoupper'](), $smcFunc['ucwords'](), and $smcFunc['ucfirst']() functions are now all locale aware and handle special casing situations correctly, so any mods that call those functions should produce the expected results once that PR has been merged. However, if a mod uses mb_strtolower(), etc., then that mod will still produce incorrect results because the mb_* functions are not aware of locales.

[Arantor]

I wasn't sure from the patch, but yeah, cycling through all of them is good.

Mods should generally not have ever been doing bare mb_* calls for the simple reason that mb_* wasn't ever expected to be available.

[Max22]

Do you think they found a solution?

Yes, I do.
It should be ready for the version 2.1.3.