Unexpected "Fetch Simple Machines Files" Scheduled Task Error

[sah62]

I'm trying to debug some unexpected events/errors that are showing up in my forum's error log whenever the "Fetch Simple Machines Files" scheduled task runs at its scheduled time. Forum version is 2.0.19, no mods installed.

First, I see an entry like this in my web server log:

Code Select Expand

203.130.163.121 - - [05/May/2022:06:02:04 -0400] "GET /smf/index.php?topic=15804.msg102160 HTTP/1.1" 200 6376 "https://www.mysite.com/smf/index.php?action=search2" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15"
203.130.163.121 - - [05/May/2022:06:02:05 -0400] "GET /smf/avatars/allery/BlazeBoard/blank.gif HTTP/1.1" 404 117 "https://www.mysite.com/smf/index.php?topic=15804.msg102160" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15"
203.130.163.121 - - [05/May/2022:06:02:06 -0400] "GET /smf/index.php?scheduled=task;ts=1651744800 HTTP/1.1" 200 85 "https://www.mysite.com/smf/index.php?topic=15804.msg102160" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.4 Safari/605.1.15"
This particular IP address is associated with an active forum user. Why would the scheduled task show up like this?

Second, when I try to run the scheduled task manually I get this error:

Code Select Expand

Database Error
Incorrect string value: '\xDA\xA9\xD8\xAFFile: /var/www/mysite/smf/Sources/ScheduledTasks.php
Line: 1338
The code involved:

Code Select Expand

1321                 // Get the file
1322                 $file_data = fetch_web_data($url);
1323
1324                 // If we got an error - give up - the site might be down.
1325                 if ($file_data === false)
1326                 {
1327                         log_error(sprintf($txt['st_cannot_retrieve_file'], $url));
1328                         return false;
1329                 }
1330
1331                 // Save the file to the database.
1332                 $smcFunc['db_query']('substring', '
1333                         UPDATE {db_prefix}admin_info_files
1334                         SET data = SUBSTRING({string:file_data}, 1, 65534)
1335                         WHERE id_file = {int:id_file}',
1336                         array(
1337                                 'id_file' => $ID_FILE,
1338                                 'file_data' => $file_data,
1339                         )
1340                 );

Is this describing an error with the data that's been retrieved? If so, how might it get corrupted like this?

[Doug Heffernan]

Database Error
Incorrect string value: '\xDA\xA9\xD8\xAFFile: /var/www/mysite/smf/Sources/ScheduledTasks.php
Line: 1338

What is the collation character set to for your database?

[Aleksi "Lex" Kilpinen]

First, I believe the 200 for the GET in your log is fine.
200 means OK, and the reason it shows up with a user IP is because the user triggered the task.
Second, the incorrect string value makes me think the DB is saying "no, don't know what to do with this" because of a character set mismatch. Is your forum running in UTF-8, and if so is the database in UTF-8 as well?

[Arantor]

So the GET is fine... unless you have 1) set up a real cron to call it and 2) deactivated the normal calling, this is how scheduled tasks always get called - by a user calling them in the background. AFAIK it has never been documented how to perform scheduled tasks via real cron.

Probably a better question is what entries there are actually in the admin_info_files table. At least one mod historically added entries there and the domain subsequently lapsed causing the files to error out.

[sah62]

Database Error
Incorrect string value: '\xDA\xA9\xD8\xAFFile: /var/www/mysite/smf/Sources/ScheduledTasks.php
Line: 1338

What is the collation character set to for your database?

I recently inherited admin responsibility for this forum, so I won't be surprised to find some configuration oddities. It's set to latin1_swedish_ci for some strange reason. I assume something like utf8mb3_general_ci would be more appropriate?

[Arantor]

Is the individual column in the table set to latin1_general_ci? The bytes it's complaining about are valid in that encoding. (You can't just rely on what the database server default, the database default, or the table default is, you need to check the actual column itself.)

[sah62]

Is the individual column in the table set to latin1_general_ci? The bytes it's complaining about are valid in that encoding. (You can't just rely on what the database server default, the database default, or the table default is, you need to check the actual column itself.)

No, the data column in the smf_admin_info_files table was also set to latin1_swedish_ci.

[Arantor]

Does your SMF think it should be in UTF-8 mode?

[sah62]

OK, changing the collation fixed the database error - thanks for the clue.

When the scheduled task runs automatically, I see a large number (three pages worth) of "undefined index" errors in my error log. Here's the first of them:

Code Select Expand

https://www.mysite.com/smf/index.php?scheduled=task;ts=1651744800
8: Undefined index: is_guest
File: /var/www/mysite/smf/Sources/Load.php
Line: 1561
The last few are in Subs.php. This is the very last one:

Code Select Expand

https://www.boss302.com/smf/index.php?scheduled=task;ts=1651744800
2: Use of undefined constant WIRELESS - assumed 'WIRELESS' (this will throw an Error in a future version of PHP)
File: /var/www/boss302/smf/Sources/Subs.php
Line: 901
What might cause these?

[Arantor]

OK, so now you may have damaged your database, because I didn't ask you to change anything, I asked you to confirm what state your system *thinks* it should be.

And since the previous question I asked wasn't answered, I think trying to actually spend any further effort debugging is a waste of everyone's time.

[sah62]

Does your SMF think it should be in UTF-8 mode?

Hmm, no, the only installed language is English with the ISO-8859-1 character set.

[Arantor]

Again, that wasn't the question I asked. I don't care what languages you have installed.

I am more interested in whether your settings file has an entry mentioning character set, as that is what determines primarily what database connection should be made.

And I think there is a mismatch between what your settings say and what your database has. It is perfectly possible to have a 2.0 site in UTF-8 mode without the UTF-8 language pack (it's just not a good idea)

[Doug Heffernan]

OK, so now you may have damaged your database, because I didn't ask you to change anything,

No, the database is not damaged from being converted to utf-8.

And since the previous question I asked wasn't answered, I think trying to actually spend any further effort debugging is a waste of everyone's time.

No, is not a waste of time at all. If you do not want to help anymore fine, but please stop being so arrogant and pompus.

I recently inherited admin responsibility for this forum, so I won't be surprised to find some configuration oddities. It's set to latin1_swedish_ci for some strange reason. I assume something like utf8mb3_general_ci would be more appropriate?

@sah62, having latin1 as default collation is not very useful as it does not support inserting unicode and non-English words. Please have a look at this link to make sure that you have converted to utf-8 properly.

https://wiki.simplemachines.org/smf/UTF-8_Readme#:~:text=Go%20to%20Forum%20Maintenance%20%3E%20Convert,of%20your%20default%20language%20file.

Can you please post here the list of mods that you have installed?

[Arantor]

OK, so now you may have damaged your database, because I didn't ask you to change anything,

No, the database is not damaged from being converted to utf-8.

Read again the state of play: the database is itself not currently in UTF-8 but SMF appears to be using a UTF-8 connection (since collation is a factor of connection, even a UTF-8 database can reject UTF-8 content if it doesn't know to connect as UTF-8). So the state of whether SMF is in need of a conversion, or confused about a prior conversion is something that needs to be established before advising anything.

If you have a mismatch of collation types (which is now possible by converting a single table, or worse, a single column) you can make it so much worse by running the converter!

I have been fixing broken SMFs for over a decade, but you must know more about this than me I guess.

No, is not a waste of time at all. If you do not want to help anymore fine, but please stop being so arrogant and pompus.

Ignoring my questions completely as well as not answering the questions I do ask, with things that are at best misleading... is a waste of everyone's time. I'm also not in the habit of giving out advice when I don't know enough about the situation to suggest whether the advice will make it worse.

[Doug Heffernan]

I have been fixing broken SMFs for over a decade, but you must know more about this than me I guess.

Ok, I will bite for this time only. I never said that. And you can be fixing broken Smf forums for 10 decades for all I care. Maybe you should start fixing your ego and arrogance as well.

Ignoring my questions completely as well as not answering the questions I do ask, with things that are at best misleading... is a waste of everyone's time.

So not answering your questions is a waste of everyone 's time? You represent everyone now? Wow, you are really full of yourself, completely self-absorbed.

@sah62, what is your php version?

[sah62]

I'm sorry if I failed to answer a specific question. Please understand that it was done out of ignorance and our posts crossing in time.

I don't see any specific settings for the character set in the Settings.php file, and I don't see any other character set settings in the Admin interface. If I'm missing something obvious, please, point me to it.

PHP version is 7.4.3. There are no mods installed.

[Doug Heffernan]

Can you have a look at this link to make sure that you did a thorough and proper utf-8 conversion?

https://wiki.simplemachines.org/smf/UTF-8_Readme#:~:text=Go%20to%20Forum%20Maintenance%20%3E%20Convert,of%20your%20default%20language%20file

[sah62]

Can you have a look at this link to make sure that you did a thorough and proper utf-8 conversion?

https://wiki.simplemachines.org/smf/UTF-8_Readme#:~:text=Go%20to%20Forum%20Maintenance%20%3E%20Convert,of%20your%20default%20language%20file

Working on it. I'm getting a "The Simple Machines site is not responding. Please try again later" error when trying to install the UTF-8 version of the "SMF 2.0.19 english" language file, though.

[Steve]

No more personal attacks please.

[Doug Heffernan]

Working on it. I'm getting a "The Simple Machines site is not responding. Please try again later" error when trying to install the UTF-8 version of the "SMF 2.0.19 english" language file, though.

How are you trying to install the language pack?

[sah62]

Working on it. I'm getting a "The Simple Machines site is not responding. Please try again later" error when trying to install the UTF-8 version of the "SMF 2.0.19 english" language file, though.

How are you trying to install the language pack?

Through the admin interface. Languages->Add Language as shown in the attached image.

[Aleksi "Lex" Kilpinen]

Before experimenting with further changes to the system, two things I would recommend:
- Make a backup
- Document what ever changes you have made recently, and what changes you are making while trying to fix things

This will make it easier to recover, in case something goes terribly wrong.

[Doug Heffernan]

Working on it. I'm getting a "The Simple Machines site is not responding. Please try again later" error when trying to install the UTF-8 version of the "SMF 2.0.19 english" language file, though.

How are you trying to install the language pack?

Through the admin interface. Languages->Add Language as shown in the attached image.

Can you try to add it manually via a ftp program as mentioned in this guide?

https://wiki.simplemachines.org/smf/Languages_-_How_do_I_add_a_new_language#:~:text=Administrators%20using%20SMF%202.0.,Configuration%20%3E%20Languages%20%3E%20Add%20Languages.

[sah62]

Thanks, that worked. The UTF-8 conversion is complete.

Any thoughts on the "undefined index" errors?

[Doug Heffernan]

Thanks, that worked. The UTF-8 conversion is complete.

Glad to hear that it worked.

Any thoughts on the "undefined index" errors?

Can you please attach here your Load.php file?

What is your php version?

[sah62]

I'm using PHP 7.4.3. File attached.

[Doug Heffernan]

I'm using PHP 7.4.3. File attached.

That file looks fine to me. You said that you have not any mods installed, what about any manual edits to the files? If you have, make a copy of them and then overwrite your forum files with a fresh set of the 2.0.19 package, minus the upgrade files. Clear the error log and see if the errors will come back.

If they do, I can take a look at this for you if you want.

[sah62]

I installed fresh copies of Load.php and Subs.php yesterday and there were no errors when the task ran automatically this morning. I don't see any differences when comparing what I had to the new files using diff, but something worked. Thanks for the help!

[Steve]

Marking solved then. If the problem reoccurs, or you have further questions about this specific topic, by all means mark it unsolved and ask away.

[Doug Heffernan]

I installed fresh copies of Load.php and Subs.php yesterday and there were no errors when the task ran automatically this morning. I don't see any differences when comparing what I had to the new files using diff, but something worked. Thanks for the help!

Glad to hear that everything is fine now. Enjoy your Smf forum