[4822] I18N issue with $txt['members']

[ke]

From Themes/default/BoardIndex.template.php (version 2.0):

Code Select Expand

', $context['common_stats']['total_posts'], ' ', $txt['posts_made'], ' ', $txt['in'], ' ', $context['common_stats']['total_topics'], ' ', $txt['topics'], ' ', $txt['by'], ' ', $context['common_stats']['total_members'], ' ', $txt['members'], '. ', !empty($settings['show_latest_member']) ? $txt['latest_member'] . ': <strong> ' . $context['common_stats']['latest_member']['link'] . '</strong>' : '', '<br />

In German, $txt['members'] should read Mitgliedern here because the preposition von ($txt['by']) requires dative case. However, with many other occurrences of $txt['members'], the nominative form Mitglieder is required. A correct translation is not possible this way. Other languages probably have similar issues. To fix this, I propose that the above message get its own $txt entry, rather than composing it from many small strings.

[Angelina Belle]

The SMF developers and localization team will discuss this. Thanks for bringing it up.

[Angelina Belle]

In the example you give, would it also be proper to use dative form of the noun "topics" following the preposition "in"?   (English: "in 23 topics").  If not in German, then how about in other languages, like Slavic and Balkan languages, Hungarian, Armenian, and Greek? (as well as Latin and Sanskrit).

I have reported this as a feature for 2.1 -- to distinguish between nominative and dative case of nouns. It could be tough for English-speakers to remember to get this one right.

[poolhall]

I have reported this as a feature for 2.1 -- to distinguish between nominative and dative case of nouns. It could be tough for English-speakers to remember to get this one right.

there are 6 cases in Russian, even more in some other languages (14 in Estonian, I believe). There must be a more complex solution.

[emanuele]

Could you list where these cases are so we can have a more comprehensive view of the problem?

[Dzonny]

Serbian language has 7 cases. As i can see Finnish has 15 (info), and Russian 6 of them (info).
From what i think, the best would be to change those strings to merged ones, but i'm not sure about that completly because some languages which don't need to change nouns to different cases will have to translate strings on the same way, so we'll have the few exactly the same translated strings.
I'm about to test this and see what's the best to do. Feel free to post all your thoughts and suggestions about this.

[poolhall]

Could you list where these cases are so we can have a more comprehensive view of the problem?

okay, a few examples:

Case	Example word:      topic	Example:      January	Example:      author	Example:  permissions
Именительный (Nominative)	тема	январь	автор	права доступа
Родительный (Genitive)	темы	января	автора	прав доступа
Дательный (Dative)	теме	январю	автору	правам доступа
Винительный (Accusative)	тему	январь	автора	права доступа
Творительный (Instrumental)	темой	январём	автором	правами доступа
Предложный (Prepositional)	теме	январе	авторе	правах доступа

[Dzonny]

Well, we don't need all of those cases. We would need just nominative and dative for "members" if i'm right?

[poolhall]

Well, we don't need all of those cases. We would need just nominative and dative for "members" if i'm right?

In theory, it would be every possible case (not just nominative and dative) with every possible noun used in language strings. In reality it would be less, for example, in English it is just "1 reply" or "2 or more replies", in Russian it would be 21 ответ (nominative), 22 ответа (genitive singular), 25 ответов (genitive plural), so with numerals there would be 2 possible cases (but 3 noun variations).

[Aleksi "Lex" Kilpinen]

Quoting myself from the team boards

In Finnish:

by 27 members =  27 jäseneltä ( possibly jäsentä )

=> Currently used translation for "19 Posts in 3 Topics by 2 Members."   
=> "19 viestiä 3 aihetta kirjoittanut 2 jäsentä." 
=> Which would be "19 messages 3 topics written by 2 members" in english.

from X-number of members = Jäseneltä / Jäseniltä depending on context.
to members = Jäsenille
to 1 member = Jäsenelle

Literally "Many members" = Monia jäseniä / Monta jäsentä
One member = Yksi Jäsen
Two members = Kaksi Jäsentä

So in short - No, making extra strings for these does not sound like a viable option IMO. What would sound more doable, would be to allow some flexibility for translators to rewrite the whole sentence. ( For example, containing the whole thing in one language string with expressions to show the right data )

So - I agree with the OP. It should be made one big language string instead of a puzzle of many incompatible parts.

[emanuele]

Could you list where these cases are so we can have a more comprehensive view of the problem?

okay, a few examples:
[SNIP]

My question was more: can you point us to the places (the strings or the positions in the templates) in SMF where you know a string is misused because should be translated in two different ways for your language? Sorry for the misunderstanding.

[Aleksi "Lex" Kilpinen]

As we have an open bug report on this, and as we are discussing the fix for this particular case already - please report any similar cases you might find elsewhere in SMF, so they can be worked out at the same time Thank you.

The bug report for those interested: http://dev.simplemachines.org/mantis/view.php?id=4822

[poolhall]

Could you list where these cases are so we can have a more comprehensive view of the problem?

okay, a few examples:
[SNIP]

My question was more: can you point us to the places (the strings or the positions in the templates) in SMF where you know a string is misused because should be translated in two different ways for your language? Sorry for the misunderstanding.

In Russian there are no such places, I believe. The translation of a noun every time would be the same, but cases would be different. Example, in boardview, in Replies / Views column, the words views / replies have a wrong case in Russian ~50% of the time.

[poolhall]

From practical point of view, I think it would be enough in Russian language strings to take care of 3 cases for some nouns (such as member, topic, replies, views, names of the months and days) used with numerals.

[Angelina Belle]

So this would be adding 2 more cases for about 30 nouns, and asking developers to use them correctly in templates?
And, something different in other languages, which have different rules?

[emanuele]

From practical point of view, I think it would be enough in Russian language strings to take care of 3 cases for some nouns (such as member, topic, replies, views, names of the months and days) used with numerals.

Fine, but please give practical references to code in SMF or at least to a page and a position where the current translation for a string (e.g. $txt['members']) doesn't fit with the Russian grammar rules like ke did in the first post.

Is the string in the first post of this topic problematic in Russian too?
There are other similar situations in SMF? If so where?

Of course I'm not asking you to review the entire SMF, just provide the examples you have at your hands, in future if you discover others you will be able to post here again.

[poolhall]

emanuele, the same example from the ke's post:

the code in BoardIndex.template.php:

Code Select Expand

', $context['common_stats']['total_posts'], ' ', $txt['posts_made'], ' ', $txt['in'], ' ', $context['common_stats']['total_topics'], ' ', $txt['topics'], ' ', $txt['by'], ' ', $context['common_stats']['total_members'], ' ', $txt['members'], '. ', !empty($settings['show_latest_member']) ? $txt['latest_member'] . ': <strong> ' . $context['common_stats']['latest_member']['link'] . '</strong>' : '', '<br />

results in the following string in my forum: "2,337,893 Сообщений в 13,418 Тем от 10,107 Пользователей.", which means "2,337,893 messages in 13,418 topics by 10,107 members."

This is how these three are translated in index.russian-utf-8.php:

Code Select Expand

$txt['posts_made'] = 'Сообщений';
$txt['topics'] = 'Тем';
$txt['members'] = 'Пользователей';

Because the case in Russian changes depending on a qualifying numeral, in this particular example the two of the three nouns are incorrect (it must be: "2,337,893 Сообщения в 13,418 Темах от 10,107 Пользователей."). If I refresh the page, all 3 words may become correct or incorrect because there are 3 possible variations for the first word and two variations - for the two other words.

There are other similar situations in SMF? If so where?

Everywhere where single word nouns, such as members, messages, topics, replies, views, etc. are used with their qualifying numerals.

[Angelina Belle]

This is adding 60 words, at least? The English-speaking developers will simply not use them "properly", even if they exist in the russian translations. They will never understand how to choose between all the cases of the words.

How will we get it right in some future version of SMF?

[Dzonny]

So - I agree with the OP. It should be made one big language string instead of a puzzle of many incompatible parts.

As Lex said, we will make new string with whole phrase, so translators can deal with that string we're talking about. However, we would like to hear if there is some other specific phrase which should be changed in order to get better translation?

@Angel, you're right, we can't just add 60 strings and ask translators to translate them. It would be better to solve this with merging few smaller strings in one bigger IMO, like we did in this case. But i don't think it's necessery if translators can deal with these phrases on their way. In serbian we did manage to do so, even if we have 7 cases.

In order to get best feedback, we should maybe post in translation board and ask translators to say their opinion about this problem?

[Illori]

I think a thread was posted there already asking for feedback in this thread.

[opinions needed] issue with $txt['members']

[Dzonny]

I think a thread was posted there already asking for feedback in this thread.

[opinions needed] issue with $txt['members']

Sorry, i've missed that, as we discuss about this problem in several boards.

[poolhall]

This is adding 60 words, at least? The English-speaking developers will simply not use them "properly", even if they exist in the russian translations. They will never understand how to choose between all the cases of the words.

How will we get it right in some future version of SMF?

I think so, about 60. How it should be done, I don't know. Maybe it could be a function that would look up for a correct case depending on a qualifying numeral.

Merging shorter strings into longer ones will not work in Russian for the reasons stated in my previous post.

[emanuele]

emanuele, the same example from the ke's post:

Thanks and sorry for the confusion, I'm not so good with foreign languages.

It's what is explained here, correct?
Now I see the complexity...

[Aleksi "Lex" Kilpinen]

The merging to one big string would allow,completely rewording and rearranging the sentence. It could even be made a simple list like Discussions: (number of topics), messages: (number of posts), Users: ( number of members)
This would not work in russian then?

[Aleksi "Lex" Kilpinen]

To add: I don't really think it's viable to build real custom code for all languages out there to suit their specific exceptions - so a compromise, a middle ground, will probably be where we should aim with this.

[poolhall]

It's what is explained here, correct?
Now I see the complexity...

yes

The merging to one big string would allow,completely rewording and rearranging the sentence. It could even be made a simple list like Discussions: (number of topics), messages: (number of posts), Users: ( number of members)
This would not work in russian then?

this construction does not require a coordination, so yes, it would. But I would not call it a sentence those are not grammatical constructions.

[Aleksi "Lex" Kilpinen]

That's my point exactly It's originally a "sentence" that can be translated to anything else that works - does not need to remain a sentence

[poolhall]

That's my point exactly It's originally a "sentence" that can be translated to anything else that works - does not need to remain a sentence

would you then change the English sentence "2,337,893 messages in 13,418 topics by 10,107 members" to "Messages: 2,337,893, topics: 13,418, members: 10,107"?

[poolhall]

and would you then be willing to date your posts like this: "Month: September, Day: 6, Year: 2011"?

LOL

[Aleksi "Lex" Kilpinen]

No, but a translator not able to translate it correctly, could reword the "translation".

It would work like "word word VALUE1 word VALUE2" and so on, and the translator only needs to preserve the VALUEs, and can show them in any working context.

[Aleksi "Lex" Kilpinen]

Oh, and by the way - I fully understand the problem.
I'm a finnish translator myself

[poolhall]

Sure, a translator could only do that. But we're talking about a possible new feature that could be created by developers, not translators.

I just changed my Facebook language to Russian and saw Facebook does that:

21 пользователю это нравится (21 people like this)
20 пользователям это нравится (20 people like this)

[Angelina Belle]

Of course, such things can be done, if you have enough developers, enough translators, and enough money to pay them.

[emanuele]

"Unfortunately" we don't pay anyone......AFAIK, otherwise I want my money!!!

It's also a performance problem: I did few tests this morning with a script based on another one found in the developer's blog. Concatenation (what we are doing now) is obviously the fastest way to deal with strings, sprintf is more or less two times slower, a dedicated function dealing with different rules according to the language will be for sure slower than sprintf, how much slower it would depend on the language.

I have no idea how FB deal with this kind of situations...
Just curious: how did you "teach" FB to deal with all these variations?

[poolhall]

Don't get me wrong, I am not sure myself what is the best way to handle this problem, and I am not asking or proposing that such function should be developed.

I have no idea how FB deal with this kind of situations...
Just curious: how did you "teach" FB to deal with all these variations?

As I said I use Facebook in English. I change it to Russian to see how it is done there and saw that FB uses the proper case of the word "пользователи" (people) with various cardinal numerals. Though this word, its plural form, has only two variations with numerals, and I didn't notice Facebook does this for other nouns.

[Aleksi "Lex" Kilpinen]

Like I said earlier, I do think that building new features only to make sure all translations for all languages can be made grammatically correct is probably not a real option. First of all, the devs would have to know those languages in order to do that, and secondly it would mean a lot of custom code added to the core...

[Dzonny]

I agree with you all, but the most important for now is to know where exactly issues are. When we know about it we will discuss about how those problem should be resolved. I dont think that all 60 words is used in different cases though. I will test this during the day.

[emanuele]

Don't get me wrong, I am not sure myself what is the best way to handle this problem, and I am not asking or proposing that such function should be developed.

Don't worry, I'm just thinking aloud.

As I said I use Facebook in English. I change it to Russian to see how it is done there

Sorry, I thought you contributed to translate FB in Russian, never mind.

First of all, the devs would have to know those languages in order to do that

Yep, that is one of my concern too. I was going to write we would need translator-developers, but then I started writing about other things.

I agree with you all, but the most important for now is to know where exactly issues are.

Right!

[poolhall]

here is info on how it's done in Drupal 6 for Russian: http://api.drupal.ru/api/function/format_plural/6

[emanuele]

The page (the server) doesn't load here...I'll check again tomorrow.

[ke]

Thanks for taking care of this, guys.

I agree with you all, but the most important for now is to know where exactly issues are. When we know about it we will discuss about how those problem should be resolved. I dont think that all 60 words is used in different cases though. I will test this during the day.

Basically, all $txt elements that consist of a single word (or other fragments) are suspicious. The only exception I can think of is labels for buttons and other navigational elements whose whole label indeed consists of just one word. But whenever even the tiniest message is concatenated from more than one $txt entry, as in the example in my original post, you can be sure that it will create problems for translation into one language or another. Translators may be able to work around these issues in many cases. For example, to answer Angelina's question:

In the example you give, would it also be proper to use dative form of the noun "topics" following the preposition "in"?

Yes, in, like von, requires dative case (in this context!). However, unlike Mitglieder/n, the nominative and dative form of Themen happen to be the identical.

But in general, word-by-word translation is not possible.

Therefore, in my opinion, developers should aim for all $txt elements to be complete message templates. Nowhere in the codebase should they enter into concatenation with each other.

[Joshua Dickerson]

ke, I am not sure if your last statement would hold true for everything, but you're absolutely right about looking in to more cases.

[emanuele]

Yesterday evening I started a new branch that I would like to integrate into 2.1, and changed some strings to sprintf:
https://github.com/emanuele45/playpen/commits/l10n/
here you can see the list of changes so far:
https://github.com/emanuele45/playpen/compare/master...l10n

I hope to be able to spot more places where something should be changed, and I hope to receive suggestions from you too since I'm sure I'll not be able to find all of the strings that need to be replaced with something else.

Thinking more about the future, keeping in mind SMF translation structure, looking at how drupal does it (just because poolhall posted a link to drupal's api) and looking at how I named the $txts in my changes one possible approach could be:

Code Select Expand

function define_plural ($string, $count, $find, $replace)
{
    global $txt;

    if (empty($count))
        $return = $txt[$string . '_zero'];
    elseif ($count == 1)
        $return = $txt[$string . '_one'];
    else
        $return = sprintf($string . '_many', $count);

    if (!empty($find) && count($find) == count($replace))
        foreach ($find as $key => $val)
            $return = str_replace($val, $replace[$key], $return;

    return $return;
}

Unfortunately that function should be placed in a language file and adapted by each localizer/translator to match each language plural rule.

At the moment I don't have any other idea, but that's for the future, for now I'd really like to find all the current places where sprintf could give a better result than what we have now.

[Angelina Belle]

In SSI.php,

Code Select Expand

sprintf($txt['welcome_newest_member'], ' ' . $context['common_stats']['latest_member']['link'])

Code (replace) Select Expand

sprintf($txt['welcome_newest_member'],  $context['common_stats']['latest_member']['link'])

Because

Code Select Expand

txt['welcome_newest_member'] = 'Please welcome %1$s, our newest member.';

Your work toward better translation of plural is a good start, emanuel, but I think this will need more study before it is added to SMF.  The world plural situation is more complex than that.  Consider polish, which has much more complicated rules than other languages.
http://doc.qt.nokia.com/qq/qq19-plurals.html
Nokia has attempted to provide an easy plural tool for its QT framework. I don't know what the implementation looks like, of course.

[emanuele]

That's why I said his function should stay in the language file, so that any translation could have it's own way to handle plural, for example for Polish could be something like:

Code Select Expand

function define_plural ($string, $count, $find, $replace)
{
    global $txt;

    if ($count == 1)
        $return = $txt[$string . '_one'];
    else
    {
        $last_digit = substr($count, -1);
        if (in_array($last_digit, array(2, 3, 4))
            $return = sprintf($string . '_many_1', $count);
        else
            $return = sprintf($string . '_many_2', $count);
    }

    if (!empty($find) && count($find) == count($replace))
        foreach ($find as $key => $val)
            $return = str_replace($val, $replace[$key], $return;

    return $return;
}

That's not something that I would really make it generic and valid for any language. I think the best option is to allow each language to handle their own set of rules.

Of course it's just an idea, nothing really solid.

[Angelina Belle]

I guess I follow.  Polish is still more complicated than that, of course. The "many_2" form comes back to haunt us every decade, I think.
The nokia idea is to use "%n noun" in strings, and the replacer is smart enough to replace %n with the number, and replace "noun" with the appropriate plural, based on the value of the number.  This requires an entire dictionary, however, which SMF does not have.

[emanuele]

The code above will result in exactly the table presented in the page you posted, so:
11 domów
12 domy
13 domy
14 domy
15 domów
16 domów
17 domów
18 domów
19 domów
21 domów
22 domy
23 domy
24 domy
25 domów
26 domów
27 domów
...
101 domów
102 domy
103 domy
104 domy
105 domów
etc.

ETA: and of course, since this would be custom code for each language, each language could code their own grammar rules.
If a language has 3 way to define the "many" then they could introduce such a system without affecting any other language.
I remember Dr. Deejay once reported that Chinese doesn't have Arabic numbers, so for the Chinese the function could be (taking *as example* the easiest version):

Code Select Expand


function define_plural ($string, $count, $find, $replace)
{
    global $txt;

    $count_number = array(
        0 => 'zero', //the equivalent Chinese version of course
        1 => 'one', //the equivalent Chinese version of course
        2 => 'two', //the equivalent Chinese version of course
        3 => 'three', //the equivalent Chinese version of course
        4 => 'four', //the equivalent Chinese version of course
        5 => 'five', //the equivalent Chinese version of course
        6 => 'six', //the equivalent Chinese version of course
        7 => 'seven', //the equivalent Chinese version of course
        8 => 'eight', //the equivalent Chinese version of course
        9 => 'nine', //the equivalent Chinese version of course
    );

    if (empty($count))
        $return = $txt[$string . '_zero'];
    elseif ($count == 1)
        $return = $txt[$string . '_one'];
    else
        $return = sprintf($string . '_many', $count_number[$count]);

    if (!empty($find) && count($find) == count($replace))
        foreach ($find as $key => $val)
            $return = str_replace($val, $replace[$key], $return;

    return $return;
}

[Angelina Belle]

OK.  I guess I just didn't read it through hard enough.
and it is easy enough to deal with all the different pluralizations of all languages, like Russian and Inuit (or whatever).

So that seems like an excellent proposal. 

[emanuele]

In SSI.php,

Code Select Expand

sprintf($txt['welcome_newest_member'], ' ' . $context['common_stats']['latest_member']['link'])

Code (replace) Select Expand

sprintf($txt['welcome_newest_member'],  $context['common_stats']['latest_member']['link'])

Because

Code Select Expand

txt['welcome_newest_member'] = 'Please welcome %1$s, our newest member.';

Fixed that one and added another series of things more or less related.

[Bugo]

I'm using simple function for this.

Example 1 with BoardIndex.template.php, find

Code Select Expand

$board['is_redirect'] ? $txt['redirects'] : $txt['posts']
replace with:

Code Select Expand

rspack_declension($board['posts'], $board['is_redirect'] ? $txt['rspack_redirect'] : $txt['rspack_post'])

Example 2, find

Code Select Expand

$context['common_stats']['total_posts'], ' ', $txt['posts_made'], ' ', $txt['in'], ' ', $context['common_stats']['total_topics'], ' ', $txt['topics'], ' ', $txt['by'], ' ', $context['common_stats']['total_members'], ' ', $txt['members']
replace with

Code Select Expand

$context['common_stats']['total_posts'], ' ', rspack_declension($context['common_stats']['total_posts'], $txt['rspack_post']), ' ', $txt['in'], ' ', $context['common_stats']['total_topics'], ' ', rspack_declension($context['common_stats']['total_topics'], $txt['rspack_topic']), ' ', $txt['by'], ' ', $context['common_stats']['total_members'], ' ', rspack_declension($context['common_stats']['total_members'], isset($txt['rspack_user1']) ? $txt['rspack_user1'] : $txt['rspack_user'])

English language file:

Code Select Expand

$txt['rspack_post'] = 'posts, post';
$txt['rspack_user'] = 'members, member';
$txt['rspack_redirect'] = 'redirects, redirect';

Russian language file:

Code Select Expand

$txt['rspack_post'] = 'сообщений, сообщение, сообщения';
$txt['rspack_user'] = 'пользователей, пользователь, пользователя';
$txt['rspack_user1'] = 'пользователей, пользователя';
$txt['rspack_redirect'] = 'переходов, переход, перехода';

Ukrainian:

Code Select Expand

$txt['rspack_post'] = 'повідомлень, повідомлення, повідомлення';
$txt['rspack_user'] = 'користувачів, користувач, користувача';
$txt['rspack_user1'] = 'користувачів, користувача';
$txt['rspack_redirect'] = 'переходів, перехід, переходи';

Subs.php:

Code Select Expand


function rspack_declension($num, $str)
{
$exp = explode(',', $str);

$num = (($num < 0) ? $num-$num*2 : $num)%100;
$dig = ($num > 20) ? $num%10 : $num;

if (sizeof($exp) == 2) {
        $exp[2] = $exp[0];
    }

return trim((($dig == 1) ? $exp[1] : (($dig > 4 || $dig < 1) ? $exp[0] : $exp[2])));
}

[emanuele]

I closed the issue.
But since the discussion here is much more complex I'll not move the topic in order to facilitate further discussions.