[2659][Bug] Problem with Users Logging in

[poolhall]

Hello everyone,

After moving to a new hosting and converting the database from cp1251 to utf8 I'm having a problem with logging in users with Cyrillic (Russian) symbols in their nicknames.  They all get a message that their passwords are incorrect. If I manually change passwords for such users, problem disappears. The problem is aggravated in that those users can't reset password as most of them state they never received an email, even though email works perfectly on the board.
I compared 'passwd' fields of the members tables in the databases on the new and old hosting and these fields are identical.

MySQL settings:
version 5.0.67
collation connection utf8_general_ci
collation database utf8_general_ci
collation server utf8_general_ci

What can I do about that? Any suggestion would be highly appreciated.

Another problem that I've been experiencing since the move is that if I search for a member with Russian (Cyrillic) symbols in its name from the admin panel, the search returns nada while the same search being run from the forum index finds a user.


Update: I forgot to mention, it's SMF 2.0 Beta 4

[Mr.Claw]

Ok, finally we've found a part of code, which is a 100% bug.

at the source file Sources/LogInOut.php we have:
line ~ 287:

/* some code before */
   else
   {
      $sha_passwd = sha1(strtolower($user_settings['member_name']) . un_htmlspecialchars($_POST['passwrd']));
   }

   // Bad password!  Thought you could fool the database?!
   if ($user_settings['passwd'] != $sha_passwd)
   {
/* some code after */

I think the guy who decided to store at the DB not the hash of passwd, but the hash of login+passwd was a real (but not appreciated yet) genius.

I will remind you, that we had made upgrade our SMF database from cp1251_general_ci to utf8_general_ci.

What we have now? Now we have 'member_name' in utf8, and sha1 hash of cp1251 member_name+passwd. It could be ok, if our smf forum was english, or polish, or any other latin alphabet language, but unfortunately it is.... Russian.

Dear SMF developers, please note, that hash("some_cp1251_string") != hash ("the_same_utf8_string"). It is true only in case of first 256 symbols which are the same as in Latin-1.

I don't even say that it's abnormal to glue login and password before hashing passwd. (do you know any benifits of it? i don't)

The cure is:

        // cp1251 => utf8 migration bugfix by Mr.Claw (tntclaus at gmail dot com)
        $other_passwords[] = sha1(strtolower(utf8_to_cp1251($user_settings['member_name'])) . un_htmlspecialchars($_POST['passwrd']));
        $other_passwords[] = sha1(strtolower($user_settings['member_name']) . un_htmlspecialchars(utf8_to_cp1251($_POST['passwrd'])));
        $other_passwords[] = sha1(strtolower(utf8_to_cp1251($user_settings['member_name'])) . un_htmlspecialchars(utf8_to_cp1251($_POST['passwrd'])));

You should add above code right before

// Whichever encryption it was using, let's make it use SMF's now .
        if (in_array($user_settings['passwd'], $other_passwords))

which is at line ~ 342 of Sources/LogInOut.php

and finally we should add function utf8_to_cp1251 to get old encoding of member_names. you can put it anywhere in LogInOut.php or somewhere else you'd like.

function utf8_to_cp1251($str, $type = 'w' )
{ // $type: 'w' - encodes from UTF to win 'u' - encodes from win to UTF
    static $conv='';
    if (!is_array ( $conv )){
        $conv=array ();
        for ( $x=128; $x <=143; $x++ ){
        $conv['utf'][]=chr(209).chr($x);
        $conv['win'][]=chr($x+112);
    }
    for ( $x=144; $x <=191; $x++ ){
        $conv['utf'][]=chr(208).chr($x);
        $conv['win'][]=chr($x+48);
    }
        $conv['utf'][]=chr(208).chr(129);
        $conv['win'][]=chr(168);
        $conv['utf'][]=chr(209).chr(145);
        $conv['win'][]=chr(184);
    }
    if ( $type=='w' ) return str_replace ( $conv['utf'], $conv['win'], $str );
    elseif ( $type=='u' ) return str_replace ( $conv['win'], $conv['utf'], $str );
    else return $str;
}

Now we can say that LogInOut.php had been patched to work with MySQL upgraded to utf8 from cp1251.


Another problem with unsuccessful search of Russian-named users can be solved with setting correct locale (php manual can be found at http://ru.php.net/manual/ru/function.setlocale.php [nofollow]). Locale settings also affect results of php sha1() and md5() functions.

[poolhall]

As we believe this is a bug in SMF, we request this topic be moved to the bug tracker.

[†MavN†]

Many people have this bug in smf 2.0 and smf 1.1.* too. Attention is required!!!

[poolhall]

I dare to bump this thread as the bug described here is very serious affecting the entire Russian SMF community.

On the Russian support website it's been two years to a thread filled with users complaining about the issue and still, no universal solution has been offered.

I call on the mods to act on this, please.

[poolhall]

Conversion of database from CP1251 to UTF-8 causes users with Cyrillic usernames be unable to login using their correct passwords.

The problem is caused by SMF-style authorization where SMF sticks username and password together and uses its hash to record it into a database. Since hash of a cp1251 string is defferent from hash of the same string in UTF-8, the resultant hashes of username+password don't match existing database records after data has been converted to UTF-8.

Please see this thread for more details.

Thanks!

[poolhall]

Was it logged, declined or just being ignored?

[metallica48423]

not being ignored.  it takes someone with knowledge in utf8 and character encoding to properly test this.

patience is a virtue.

Thanks for your report.

[poolhall]

Thank you.

[SleePy]

Do you got some example hash values?

I can't wrap my mind around why the hash values would be different here.

[Mr.Claw]

Do you got some example hash values?

http://www.allboxing.ru/forum/test.php [nofollow]

I can't wrap my mind around why the hash values would be different here.

hash functions work not with symbol, but with its binary implementation, which depends on type of encoding.

e.g. "A" can be implemented in encoding "blah-1" as 0x05 and in encoding "blah-2" - as 0x15.

setlocale("blah-1");
$A_blah_1 = some_hash("A");  // equal to some_hash(0x05);

setlocale("blah-2");
$A_blah_2 = some_hash("A"); // equal to some_hash(0x15);

$A_blah_1 != $A_blah_2; // cuz some_hash(0x05) != some_hash(0x15)

now, look at Cyrillic implementation in utf-8
http://www.unicode.org/charts/PDF/U0400.pdf [nofollow]

and compare it to cp1251 yourself

[SleePy]

Eeh, thats good enough for me
Bug #2659: UTF-8 conversion can break logins

[poolhall]

Thanks SleePy! 

[†MavN†]

SleePy
This bug in smf 1.1.* too. Please correct description in bug tracker

[Mr.Claw]

Eeh, thats good enough for me
Bug #2659: UTF-8 conversion can break logins

Note I attached an image, I don't have the language set installed so it shows up as ? for me. But the hash part is the important thing.

no, utf-8 chars look pretty good

[poolhall]

SleePy,

This is about your and ThorstenE's comments on the bug tracker page pertaining to this bug.

Please note that the picture you attached is misleading. The utf-8 encode string and the cp1251 encode string in the test.php example look and are EXACTLY the same. You don't need any additional language set installed to see that - just change character encoding in your browser from UTF-8 to CP1251 and then go back to UTF-8 to compare the strings.

@ThorstenE's comment:
There are/were NO question marks in database. In both cases there are normal Cyrillic symbols. But resultant hashes are different because of the reasons given above.

We're gonna keep the test.php on our server to let the devs look at the problem closely.

Thank you.

[Mr.Claw]

plain php script of http://allboxing.ru/forum/test.php [nofollow]
is here: http://allboxing.ru/forum/test.plain [nofollow]

you are free to analyze it

[SleePy]

Ahh, I thought I was missing a language pack on my computer for it to interpret the language correctly.

[Oldiesmann]

pool, did you install and switch to the Russian-UTF8 language pack after switching to UTF8?

[poolhall]

pool, did you install and switch to the Russian-UTF8 language pack after switching to UTF8?

I believe we converted first, then added the UTF-8 language pack, but I will give you a more definite answer tomorrow.