Simple Machines Community Forum

SMF Development => Bug Reports => Fixed or Bogus Bugs => Topic started by: Orbfighter on June 30, 2015, 01:31:58 PM

Title: scheduled_fetchSMfiles() truncates files, breaking Latest Packages interface
Post by: Orbfighter on June 30, 2015, 01:31:58 PM
Versions of SMF: 2.0.8 - 2.0.10 (as best as I can tell)

Error Caused From: Sources/ScheduledTasks.php, In function scheduled_fetchSMfiles(), line 1292

        $smcFunc['db_query']('substring', '
            UPDATE {db_prefix}admin_info_files
            SET data = SUBSTRING({string:file_data}, 1, 65534)
            WHERE id_file = {int:id_file}',
            array(
                'id_file' => $ID_FILE,
                'file_data' => $file_data,
            )
        );

For reasons I don't really understand, the SQL query apparently truncates files it downloads from the simplemachines.org server as it uploads them to the database, to 65,534 characters. This is problematic because the download http://simplemachines.org/smf/latest-packages.js at the time of this writing, is 65,706 bytes, cutting off the javascript function function smf_packagesBack() and causing a general JS syntax error when trying to load the latest packages info in the Packages Manager.

To fix this issue on my end, I simply changed the SUBSTRING function in the query to allow for a larger amount of characters, and ran the Fetch Simple Machines Files scheduled task again. After doing this the Latest Packages interface displayed as normal and the full file was loaded in the database.
Title: Re: scheduled_fetchSMfiles() truncates files, breaking Latest Packages interface
Post by: Illori on June 30, 2015, 03:08:01 PM
we had seen this around the release of 2.0.10 or so, since it resolved itself it was not looked into much further that i can see. maybe the file should be fixed on the site unless they wish to fix it in SMF itself.

for team members reference topic id 535850 has some further info on this.
Title: Re: scheduled_fetchSMfiles() truncates files, breaking Latest Packages interface
Post by: dcmouser on July 17, 2015, 03:16:58 PM
I got a database error today due to the sql update line above (smf 2.0.10):
"Database Error: Data too long for column 'data' at row 1"
Not sure how to fix or the ramifications..
Title: Re: scheduled_fetchSMfiles() truncates files, breaking Latest Packages interface
Post by: Kindred on July 17, 2015, 03:23:45 PM
 just don't use the inline mod downloader...   come to the smf customization site, download the mod to your computer and the upload it directly to the package manager.

unfortunately, the issue is related to the content that mod authors put on our server here... so there is not much else you can do.
Title: Re: scheduled_fetchSMfiles() truncates files, breaking Latest Packages interface
Post by: Orbfighter on July 17, 2015, 04:43:07 PM
That's well and good but this issue is more serious than regarding the installation of customization/mods. I noticed and reported this bug because the forum was failing to notify me that there was a new patch of 2.0 to install (as it previously had.) I was still running 2.0.8 when I realized there was a new version, and yet the system had no idea, and I tracked it down to this line-of-code. If there are security patches such as 2.0.9 that are not getting applied because of this issue, that goes beyond a minor inconvenience. It should at the very least not break the update notification based on the content that mod authors put on your server.
Title: Re: scheduled_fetchSMfiles() truncates files, breaking Latest Packages interface
Post by: Joshua Dickerson on August 05, 2015, 01:54:15 PM
Quote from: Kindred on July 17, 2015, 03:23:45 PM
just don't use the inline mod downloader...   come to the smf customization site, download the mod to your computer and the upload it directly to the package manager.

unfortunately, the issue is related to the content that mod authors put on our server here... so there is not much else you can do.
Truncate it here and check the file size before you make the file.
Title: Re: scheduled_fetchSMfiles() truncates files, breaking Latest Packages interface
Post by: Kindred on August 05, 2015, 02:56:16 PM
yeah... are you volunteering to do the work on the customization site?
Title: Re: scheduled_fetchSMfiles() truncates files, breaking Latest Packages interface
Post by: Illori on August 05, 2015, 02:59:10 PM
since the file contains html, you cant really truncate it easily. if you do then you will break the html. the size of the description box on the mod site needs to be modified to be smaller then this would not happen.
Title: Re: scheduled_fetchSMfiles() truncates files, breaking Latest Packages interface
Post by: Joshua Dickerson on August 06, 2015, 03:35:06 PM
Quote from: Kindred on August 05, 2015, 02:56:16 PM
yeah... are you volunteering to do the work on the customization site?
No.

Quote from: Illori on August 05, 2015, 02:59:10 PM
since the file contains html, you cant really truncate it easily. if you do then you will break the html. the size of the description box on the mod site needs to be modified to be smaller then this would not happen.
True, but not impossible.

Or have a smaller, shorter description for those files.
Title: Re: scheduled_fetchSMfiles() truncates files, breaking Latest Packages interface
Post by: live627 on August 06, 2015, 08:11:42 PM
There must be a better way to handle this. Why are we saving it to the database? It is a FILE. Why not download it to the file cache?
Title: Re: scheduled_fetchSMfiles() truncates files, breaking Latest Packages interface
Post by: Illori on August 06, 2015, 08:36:30 PM
and what would happen if the cache is cleared and it included this file? it is only populated when the fetch files scheduled task is run. if it could be stored as a file and not cause issues if the file is deleted that is fine.

i am not even sure this file is really called in 2.1... i know it does not show the latest packages etc in 2.1 like it does in 2.0. so maybe we dont need to fix it in 2.1 because of that.