Compilation failed: nvalid UTF-8 string at offset 1238

[ddabcd277]

Hello,

I have this error:

2: preg_replace() [<a href='function.preg-replace'>function.preg-replace</a>]: Compilation failed: invalid UTF-8 string at offset 1238
File: /home/www/myforum.com/www/Sources/Subs.php
Line: 2513

What does it mean? How to fix it?

I am running SMF 2.0.2 and only one mod installed "Quick Spoiler".

Thanks,

[MrPhil]

UTF-8 non-ASCII characters have to follow a strict definition for their bit patterns, so that it is possible for the database, PHP, and browser to find multibyte characters that have been cut in half or otherwise corrupted. Single byte characters from the "upper half" (x80 - xFF) of Latin-1 or other single-byte encodings are invalid in UTF-8. Your error says that it encountered a character that did not follow the rules. Most likely there is a Latin-x accented or special (upper half) character lurking somewhere in your database or hard coded into a template or something (very bad practice). Perhaps the mod was never tested with UTF-8? If you are using a non-default theme, does this error happen on the default theme?

You might start trying to track down this bogus character by carefully examining all the code touched or added by the Quick Spoiler mod. It's most likely in there. Has your database always been UTF-8, or did you convert recently? There's also a chance that the mod code used PHP single-byte string and character functions instead of the mb_ (multibyte) forms, leading to data corruption.

[ddabcd277]

Hello MrPhil,

Thank you for your reply.

In fact my forum has 14 pages only with this error from the last time I cleared all the errors log. (sometimes at the end of the last year.) And I see they are various. For example:

- http://www.myforum.bg/index.php?action=register
- http://www.myforum.bg/index.php?
- http://www.myforum.bg/index.php?topic=1597.0;prev_next=next
- http://www.myforum.bg/index.php?action=forum
- http://www.myforum.bg/index.php?topic=912.0
- http://www.myforum.bg/index.php?topic=1916.msg6401
etc..

Don't know is this information important.

If you are using a non-default theme, does this error happen on the default theme?

I am using the Default SMF curve theme.

You might start trying to track down this bogus character by carefully examining all the code touched or added by the Quick Spoiler mod. It's most likely in there.

I doubt I can manage with that.  Can you give me an example what should I look for? What should I do step by step. I don't understand very much from php and I find it quite difficult.

It's most likely in there. Has your database always been UTF-8, or did you convert recently?

Not sure about that but I think the answer is "Yes". At first the forum was version 1.1.1x but last year I have converted it to 2.0. 

[MrPhil]

I looked through the Quick Spoiler code, and it looks pretty clean. They are using hooks instead of code edits. As this mod says that it is only for SMF 2.x, I presume it was installed after you upgraded to SMF 2? If not (it was from an earlier version for SMF 1.x?), it's possible there is something bad in there.

Assuming your Sources/Subs.php is unmodified, line 2513 is doing a string replacement in the smiley parsing routine. There's one thing that concerns me -- it defines $non_breaking_space as either xC2A0 (UTF-8) or xA0 (non-UTF-8). The test is rather complex, and I'm wondering if due to your upgrade from 1.1 to 2.0 something is not being set correctly, thus giving xA0 for the value when you're in UTF-8 mode, which could cause errors. Since the code is a mess and poorly commented, I can't tell if it's looking for a Latin-1 (single byte) non-breaking space or looking for a UTF-8 non-breaking space in the search pattern.

Code Select Expand

$non_breaking_space = $context['utf8'] ? ($context['server']['complex_preg_chars'] ? '\x{A0}' : "\xC2\xA0") : '\xA0';
If $context['utf8'] is false, or $context['server']['complex_preg_chars'] is true, you will get the wrong (Latin-1 instead of UTF-8) non-breaking space. What PHP version are you running on? Sources/Load.php should set complex_preg_chars to true if you're at PHP 4.3.3 or higher. I'm wondering if the test is backwards (should be

Code Select Expand

$non_breaking_space = $context['utf8'] ? ($context['server']['complex_preg_chars'] ? "\xC2\xA0" : '\x{A0}') : '\xA0';
instead)? That test is in many places in SMF, so it's hard to believe it's broken! Context utf8 is set two different ways in 3 routines -- apparently it's supposed to be true if you're using UTF-8.

Anyway, you might add some temporary code to print out $context['utf8'] and $context['server']['complex_preg_chars'], to see what $non_breaking_space is being set to. That's where I'd start. One thing you might do before anything else is to check that your message text includes the proper non-breaking space code, which for UTF-8 is a two-byte character. Don't cut and paste in a Word document, as it will contain Latin-1 non-breaking spaces, which are invalid in UTF-8.

[ddabcd277]

I looked through the Quick Spoiler code, and it looks pretty clean. They are using hooks instead of code edits. As this mod says that it is only for SMF 2.x, I presume it was installed after you upgraded to SMF 2? If not (it was from an earlier version for SMF 1.x?), it's possible there is something bad in there.

Yes, the mod was installed after I upgrade to version 2.x. I don´t know is there something bad. 

Assuming your Sources/Subs.php is unmodified, line 2513 is doing a string replacement in the smiley parsing routine. There's one thing that concerns me -- it defines $non_breaking_space as either xC2A0 (UTF-8) or xA0 (non-UTF-8). The test is rather complex, and I'm wondering if due to your upgrade from 1.1 to 2.0 something is not being set correctly, thus giving xA0 for the value when you're in UTF-8 mode, which could cause errors. Since the code is a mess and poorly commented, I can't tell if it's looking for a Latin-1 (single byte) non-breaking space or looking for a UTF-8 non-breaking space in the search pattern.
Code: [Select]
$non_breaking_space = $context['utf8'] ? ($context['server']['complex_preg_chars'] ? '\x{A0}' : "\xC2\xA0") : '\xA0';
If $context['utf8'] is false, or $context['server']['complex_preg_chars'] is true, you will get the wrong (Latin-1 instead of UTF-8) non-breaking space. What PHP version are you running on? Sources/Load.php should set complex_preg_chars to true if you're at PHP 4.3.3 or higher. I'm wondering if the test is backwards (should be
Code: [Select]
$non_breaking_space = $context['utf8'] ? ($context['server']['complex_preg_chars'] ? "\xC2\xA0" : '\x{A0}') : '\xA0';
instead)? That test is in many places in SMF, so it's hard to believe it's broken! Context utf8 is set two different ways in 3 routines -- apparently it's supposed to be true if you're using UTF-8.

Anyway, you might add some temporary code to print out $context['utf8'] and $context['server']['complex_preg_chars'], to see what $non_breaking_space is being set to. That's where I'd start. One thing you might do before anything else is to check that your message text includes the proper non-breaking space code, which for UTF-8 is a two-byte character. Don't cut and paste in a Word document, as it will contain Latin-1 non-breaking spaces, which are invalid in UTF-8.

Can you look at the Subs.php file and see if there is something wrong in it and add those things you write about? Will be very gratefull if you do that! 

[MrPhil]

Your Subs.php appears to be standard, except that the end-of-line is different (CRLF instead of newline) for each line (no problem).

Given that the offending character is at offset 1238, I think it must be in the $message (post body) rather than in the regular expression template. Let's start with the $message text, as that will likely be more productive. Can you tell where the offending post came from? Was it cut and pasted from a PC? If so, it could easily contain an invalid character (not valid for UTF-8). Anything non-ASCII (character codes x80 through xFF) will be invalid. Common offenders would be "Smart Quotes" as used in MS Word, and nonbreaking spaces. If the original text in the post contains characters that are invalid for UTF-8, all bets are off (although it's curious that people have been cutting and pasting Word docs into UTF-8 SMF for a long time, with sometimes bad results, but I've never seen it blow up PHP code like this).

Let me poke around a bit to see if there is some test I can do to discover if there are bad UTF-8 characters in the $message text. In the mean time, can you see if you can isolate it to a specific post? If you disable smileys, this section of code won't be run, and maybe you will see a ?-in-diamond invalid character mark or two.

[MrPhil]

You can give this a try. It will output a comment <!-- HERE string is NOT valid UTF-8: text --> or <!-- HERE string IS valid UTF-8: text --> which you can look for in your page source (browser View > Page Source). At least we can narrow it down to the offending post, assuming that it can output something before the PHP error makes it go belly up. I'll check both the regex pattern and the message, though I suspect it's in the message. Once we've narrowed down the post and shown this can do the job, we'll worry about finding the bad character(s).

Find // Replace away! and after that line insert:

Code Select Expand


echo "<!-- HERE regex pattern ";
echo (is_utf8($smileyPregSearch)) ? "IS" : "is NOT";
echo " valid UTF-8 $smileyPregSearch -->";
echo "<!-- HERE message ";
echo (is_utf8($message)) ? "IS" : "is NOT";
echo " valid UTF-8 $message -->";


Two lines after the original // Replace away! line is the } in column 1 marking the end of the function. Insert after that line:

Code Select Expand


// Returns true if $string is valid UTF-8 and false otherwise.
function is_utf8($string) {
   
    // From http://w3.org/International/questions/qa-forms-utf-8.html
    return preg_match('%^(?:
          [\x09\x0A\x0D\x20-\x7E]            # ASCII
        | [\xC2-\xDF][\x80-\xBF]             # non-overlong 2-byte
        |  \xE0[\xA0-\xBF][\x80-\xBF]        # excluding overlongs
        | [\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}  # straight 3-byte
        |  \xED[\x80-\x9F][\x80-\xBF]        # excluding surrogates
        |  \xF0[\x90-\xBF][\x80-\xBF]{2}     # planes 1-3
        | [\xF1-\xF3][\x80-\xBF]{3}          # planes 4-15
        |  \xF4[\x80-\x8F][\x80-\xBF]{2}     # plane 16
    )*$%xs', $string);
   
} // function is_utf8


I have not actually tried this, so no promises.

[ddabcd277]

Many Thanks for your help.  I have just added the string you wrote about in Subs.php and will observations write in the next comment

[ddabcd277]

Sorry, for the late reply.

Unfortunately not much of a progress. No of the output massages appear (<!-- HERE string is NOT valid UTF-8: text --> or <!-- HERE string IS valid UTF-8: text -->) in the page source. Here three example of topics where the error appears. Check it yourself if you wish:

http://www.garminclub.bg/index.php?topic=722.0
2: preg_replace() [<a href='function.preg-replace'>function.preg-replace</a>]: Compilation failed: invalid UTF-8 string at offset 1238
File: /home/www/garminclub-bg.com/www/Sources/Subs.php
Line: 2519

http://www.garminclub.bg/index.php?topic=2134.0
2: preg_replace() [<a href='function.preg-replace'>function.preg-replace</a>]: Compilation failed: invalid UTF-8 string at offset 1238
File: /home/www/garminclub-bg.com/www/Sources/Subs.php
Line: 2519

http://www.garminclub.bg/index.php?action=register
2: preg_replace() [<a href='function.preg-replace'>function.preg-replace</a>]: Compilation failed: invalid UTF-8 string at offset 1238
File: /home/www/garminclub-bg.com/www/Sources/Subs.php
Line: 2519

Maybe this will also be interesting to you. I have never seen such a big error before:

Code Select Expand

http://www.garminclub.bg/index.php?action=post;topic=1821.0+%5BPLM=0%5D%5BR%5D+GET+http://crackgarmin.com/index.php?PHPSESSID=de20bc829a27e77e04d2802b4de57ff3&action=register+%5B0,8338,9404%5D+-%3E+%5BR%5D+POST+http://crackgarmin.com/index.php?action=register+%5B0,12684,11688%5D+-%3E+%5BR%5D+POST+http://crackgarmin.com/index.php?action=register2+%5BR=302%5D%5B0,0,615%5D+-%3E+%5BR%5D+GET+http://crackgarmin.com/index.php?action=login2;sa=check;member=7200+%5BR=302%5D%5B0,0,323%5D+-%3E+%5BR%5D+GET+http://crackgarmin.com/index.php?action=profile;u=3267+%5B0,0,8286%5D+-%3E+%5BN%5D+GET+http://crackgarmin.com/index.php?action=profile;u=3267+%5B0,0,8286%5D+-%3E+%5BN%5D+GET+http://crackgarmin.com/index.php?action=forum+%5B0,0,21091%5D+-%3E+%5BN%5D+GET+http://crackgarmin.com/index.php?board=6.0+%5BR=302%5D%5B0,0,293%5D+-%3E+%5BN%5D+GET+http://www.garminclub.bg/+%5B0,0,41285%5D+-%3E+%5BN%5D+GET+http://www.garminclub.bg/index.php?action=post;topic=1821.0+%5B0,0,13580%5D+-%3E+%5BN%5D+GET+http://www.garminclub.bg/index.php?topic=1820.0+%5B0,0,24004%5D+-%3E+%5BN%5D+GET+http://www.garminclub.bg/index.php?topic=1820.0;prev_next=prev+%5B0,0,26949%5D+-%3E+%5BN%5D+GET+http://www.garminclub.bg/index.php?topic=1820.0+%5B0,0,24004%5D+-%3E+%5BN%5D+GET+http://www.garminclub.bg/index.php?board=18.0+%5B0,6361,51881%5D+-%3E+%5BL%5D+POST+http://www.garminclub.bg/index.php?action=login2+%5B0,0,12828%5D+-%3E+%5BN%5D+GET+http://www.garminclub.bg/index.php?topic=1821.0;prev_next=prev+%5B0,0,23696%5D+-%3E+%5BN%5D+GET+http://www.garminclub.bg/index.php?topic=437.0+%5B0,0,23638%5D+-%3E+%5BN%5D+GET+http://www.garminclub.bg/index.php?topic=1821.0;prev_next=next+%5B0,0,24062%5D+-%3E+%5BN%5D+GET+http://www.garminclub.bg/index.php?topic=1820.0+%5B0,0,24004%5D+-%3E+%5BN%5D+GET+http://www.garminclub.bg/index.php?topic=437.0;prev_next=next+%5B9980,0,26949%5D
2: preg_replace() [<a href='function.preg-replace'>function.preg-replace</a>]: Compilation failed: invalid UTF-8 string at offset 1238
File: /home/www/garminclub-bg.com/www/Sources/Subs.php
Line: 2519

Big mess isn't it? From the 25 of April till now I have 40 error in the error logs. Hope this info leads you to some clues. If you need more please write.

[MrPhil]

Hmm. If it's always offset 1238 on different operations, perhaps it's nothing to do with the post's message body. Is your original line 2512 in Subs.php // Replace away!? This is in the processing of "smileys" -- did you add additional smileys at any point? You can remove the code you added for me -- it seems to be blowing up before it can run that code.

The "big error" looks like your server may be having a bad time. GET and POST are low level HTTP commands that I don't think you usually see in these error messages.

So what was the history of this forum, especially with regards to changing character encodings, and when in this history did you start getting these errors? Have you done any manual editing of files? Are you ever in UTF-8 mode on your PC when you edit? I wonder if you did some editing while encoding was Latin-1 (ISO-8859-1) and you inserted a hard coded Latin-1 character (byte value x80 through xFF). Such a character is invalid UTF-8, and is probably best replaced by an HTML entity (e.g., &raquo; instead of >>).

As a temporary desperation move, you could disable the whole function:

Code Select Expand


// Parse smileys in the passed message.
function parsesmileys(&$message)
{
/*  <====== add this
global $modSettings, $txt, $user_info, $context, $smcFunc;
....
// Replace away!
$message = preg_replace($smileyPregSearch, 'isset($smileyPregReplacements[\'$1\']) ? $smileyPregReplacements[\'$1\'] : \'\'', $message);
*/  <====== add this
}

If the errors go away (and smileys don't get expanded into graphics), we'll at least have confirmed that the whole problem is in this function. If the errors still happen, the problem is elsewhere (please attach your entire Subs.php file). Why would expansion of smileys happen during registration?

[ddabcd277]

If it's always offset 1238 on different operations, perhaps it's nothing to do with the post's message body.

Yes, always at - offset 1238 on line 2019.

Is your original line 2512 in Subs.php // Replace away!

Yes.

This is in the processing of "smileys" -- did you add additional smileys at any point?

Yes, I did. Should I remove them?

You can remove the code you added for me -- it seems to be blowing up before it can run that code.

Ok, I did that.

The "big error" looks like your server may be having a bad time. GET and POST are low level HTTP commands that I don't think you usually see in these error messages.

Phew...

So what was the history of this forum, especially with regards to changing character encodings, and when in this history did you start getting these errors? Have you done any manual editing of files? Are you ever in UTF-8 mode on your PC when you edit? I wonder if you did some editing while encoding was Latin-1 (ISO-8859-1) and you inserted a hard coded Latin-1 character (byte value x80 through xFF). Such a character is invalid UTF-8, and is probably best replaced by an HTML entity (e.g., &raquo; instead of >>).

Well, the history of this forum is long and full of inexperience.  Briefly the version on which the forum started was 1.1.1 then it was upgraded to versions 1.1.5 till last year a decide it to upgrade it to 2.0. Unfortunately, during the conversion to 2.0 something got screwed up. Something in the database. And most of the attached files, images etc got corrupted. Don't know is this information useful to you.

I have also once re installed the forum because I got many emails from people can't registering in the forum and it was suggested to me to re install it. (I have made some small cosmetic editing before that.) I have re installed it and the only mod I have installed till now is the Bugo spoiler cause I like it. Nothing more is touched to the forum cause I am afraid to touch anything to it because there is always some kind of problems with it...

Unfortunately, I still got the emails from people who can't register in the forum so I decided I will try to fix all the errors appearing in the error log one by one hoping this would be fixed. 

Do you think if I change the language settings to Latin not UTF-8 this error going to disappear?

As a temporary desperation move, you could disable the whole function:
....
If the errors go away (and smileys don't get expanded into graphics), we'll at least have confirmed that the whole problem is in this function. If the errors still happen, the problem is elsewhere (please attach your entire Subs.php file). Why would expansion of smileys happen during registration?

The file Subs.php is a couple of posts above.

Ok, I will disable the function and inform you about the result in the next post.

[MrPhil]

Hmm. The parsesmiley() code doesn't look any different, i.e., yours is unmodified. It still would be interesting to disable this function section by section until we find what it's unhappy about. BTW, put the /* */ inside the enclosing { } ... we still need the function defined, just doing nothing. Then we can /* */ out smaller sections to narrow it down. It may well be something outside of this code that's causing the problem.

You say you added new smileys? Just for grins and giggles, try removing them and seeing if it works now. If it does, do any of the new code contain Latin-1 accented or Word (non-ASCII) characters? That will cause UTF-8 to barf (although why in this function I don't know yet). Beware of cutting and pasting any text or PHP code from Word, as its "Smart Quotes" can really mess you up (e.g., changing -- to an em-dash). I'm curious as to why there is no new code in parsesmileys(), which goes through a list of all the smiley codes to replace them with images.

[ddabcd277]

Ok, think we (you) localize where the error is. Good! From yesterday till now after disabling the smileys function (from the beginning till the end) no error have appeared in the error log. So now what?

You say you added new smileys? Just for grins and giggles, try removing them and seeing if it works now. If it does, do any of the new code contain Latin-1 accented or Word (non-ASCII) characters? That will cause UTF-8 to barf (although why in this function I don't know yet). Beware of cutting and pasting any text or PHP code from Word, as its "Smart Quotes" can really mess you up (e.g., changing -- to an em-dash). I'm curious as to why there is no new code in parsesmileys(), which goes through a list of all the smiley codes to replace them with images.

Sorry, I couldn't understand exactly what you mean. Can you give a simple example what should I do.

For example I have now placed again the original Subs.php (with no smileys function disabled) and changed the smileys from "default" to "classic" in the admin panel and I can see that the error is not appearing. But this do not solve the issue because on the place of the old smileys, which I have added additionally, I have now only text in brackets in the posts.   

[MrPhil]

OK, so it's definitely in parsesmiley(). That's a start. You can re-enable it whenever you want (remove /* and */).

Since it failed to output (<!-- HERE comments), I think we'll have to do some digging deeper, sectioning parsesmiley() to find exactly what's failing. But first, what smiley mod(s) did you add? I'd like to take a look at them, to see if there's something incompatible with UTF-8. Did you have to do any editing by hand to get the mod(s) to install? Were there any errors reported by the SMF mod installer? Did any of this involve cut and paste? My suspicion is that somewhere in the new smiley code are one or more characters that are invalid UTF-8. Did any of this smiley code pass through Word or another word processor? Never, ever use Word (or anything else with "Smart Quotes") to edit computer code! I'm worried that at some point you pasted some code over to Word as a temporary holder, and pasted it back into SMF. In the process, Word changed some of the characters into invalid UTF-8 characters. Did that happen?

What is this "text in brackets" you are talking about? Is it something like :) or is it [smile] or is it something else? I would expect :) as long as parsesmiley() is disabled, but [smile] looks like "alt text", which means smiley tags are being processed somewhere, but the graphics (.gif) files are missing or corrupted.

[ddabcd277]

Since it failed to output (<!-- HERE comments), I think we'll have to do some digging deeper, sectioning parsesmiley() to find exactly what's failing. But first, what smiley mod(s) did you add? I'd like to take a look at them, to see if there's something incompatible with UTF-8. Did you have to do any editing by hand to get the mod(s) to install? Were there any errors reported by the SMF mod installer? Did any of this involve cut and paste? My suspicion is that somewhere in the new smiley code are one or more characters that are invalid UTF-8. Did any of this smiley code pass through Word or another word processor? Never, ever use Word (or anything else with "Smart Quotes") to edit computer code! I'm worried that at some point you pasted some code over to Word as a temporary holder, and pasted it back into SMF. In the process, Word changed some of the characters into invalid UTF-8 characters. Did that happen?

Thank you for the advice! Haven't though about such thing.

Here the smileys folder. Have a look at it if you want:
http://www.2shared.com/file/PyrtYIyS/Smileys.html

In general I have added the smileys from the "Add Smiley" option one by one after enabling "Enable customized smileys" option in the "Default". Then in the "Tooltip or description" I have wrote for each smiley a description in Bulgarian. Maybe this one causes the error? The description? Haven't though about the encoding when writing the description...

BTW I have forgot to write I have now also disabled the "Enabled customized smileys" option before changing to classic at the moment...

What is this "text in brackets" you are talking about? Is it something like or is it [smile] or is it something else? I would expect as long as parsesmiley() is disabled, but [smile] looks like "alt text", which means smiley tags are being processed somewhere, but the graphics (.gif) files are missing or corrupted.

The smiley code. For example (ok) is a hand with thumb up.

As I wrote I have disabled "Enabled customized smileys" option and changed to Classic now. I haven't added yan smileys to the Classic smileys. So the codes stays in the posts instead of smileys because no smiley is associated to it in the classics smileys codes...

[MrPhil]

Something is not letting me download your file. If this is an SMF mod, just give me the mod number of it. If it is available for free, please give me the link. If it is a paid mod, can you put it on your site somewhere and PM a link to me.

You're not sure if you used Word (or another Smart Quotes word processor) while installing this mod? If you did, it might have corrupted certain characters. Perhaps you should just uninstall the mod and install a fresh copy, being careful to look for error messages, and NOT to use Word at any point in any manual editing needed. Note that if your original installation was corrupted (by using Word), SMF may not be able to uninstall the mod and you'll have to "parse" the instructions to back it out manually.

[ddabcd277]

Ok, here it is...

No, I haven't installed any paid mods. As I wrote now I have only installed Bugos' quick spoiler. No other mods. Before reinstalling the forum I have installed many mods but can't remember which ones and were there any issues with them.

What about the description? Could that cause the error?

[MrPhil]

Both those archives have only .gif files in them. I need the PHP code that's added to various places in SMF, especially in parsesmiley() function. The code you showed (Sources/Subs.php) had no changes for the new smileys.

Can you uninstall the smileys mod and remove it? Just to see if the error goes away. If it does go away, carefully re-install the mod, being careful not to run any of the code through any word processor (especially MS Word). If the error comes back, there is something wrong with that mod, in that it includes text that is not UTF-8 compatible.

Originally, you said you only had Quick Spoiler and you were getting the error. Did you (or did you not) add another mod at some point, for additional smileys? How was that added -- by the SMF mod installer (package manager) or by hand? If you did add smileys, it looks like something went wrong there. There must have been instructions on how to change your SMF code to handle these new smileys -- you can't just drop in the image files and have them work. Just to confirm, your database and forum are in UTF-8 encoding? I don't think you said that anywhere, but that you're getting UTF-8-related errors sounds like you are.

[ddabcd277]

...you can't just drop in the image files and have them work

I am doing it from the "Add Smiley" option. No code changing.





Ok, I think I got what your point is - there is some problem in the encoding that is causing the error. I will experiment in the next few days and reinstalled all the smileys in the forum. Hopefully this error won't appear again.

Appreciate all your help. Thank you. I will mark the topic as solved not to bother this one any more.