2.1 Upgrader, Attachments & Avatars - Helpful Utilities

[shawnb61]

Occasionally after running the 2.1 upgrader, attachments & avatars are missing.  Clicking on attachment links shows a 404, & users may end up with the blank, default, 2.1 avatar.

Since we still see support issues with this, I figured it might be helpful to share related info in one place, instead of scattered around in various topics.  Also to share a couple utilities that I hope help.

Background
Prior to 2.1, there were a number of issues & inconsistencies with attachments & avatars...
 - The lack of a file extension on attachment records often led to problems with various utilities making assumptions about the contents of the file.  Things like file downloads/uploads might render attachments unusable and even unfixable.
 - The attachment files in the file system were named differently in different SMF versions over time...  E.g., the early attachment files had the original file name embedded, e.g., 888_VG-99_v104_features_pdfdf4cc5a73f5b2ec07bf8237ab450da9e
 - As a result, several variants of these filenames were honored; SMF would have to try each possible spelling of the filename in the file system to open an attachment.
 - Custom avatars were stored and retrieved as attachments.  There were pretty big performance issues with this, so an option was added to SMF 2.0 allow support for a custom avatar directory.
 - The custom avatar directory was named by the user, so the name could vary.  (Many folks followed the instructions shared in early 2.0 days exactly & named their custom avatar folder "/avs".)
 - Any avatars uploaded prior to configuration of the custom avatar directory would remain attachments until a special hidden utility was run.
 - So... SMF2.0 custom avatars may or may not be stored & retrieved as attachments, and may or may not be stored in the /custom_avatar folder.

For background on the custom avatar folder, read item #1 in this list:
https://www.simplemachines.org/community/index.php?topic=293441.0

2.1 Changes
The 2.1 upgrader goes through the smf_attachment table & addresses the issues above:
 - A .dat extension is added to all attachments found in the smf_attachments table.  I.e., the .dat is not added to stray files in the file system.
 - All attachments, including thumbnails, are renamed to a consistent "#_hash.dat" format.  2.1 only honors attachments and thumbnails with that naming.
 - If a custom avatar folder had not been configured, one would be created, named '/custom_avatar'.  If one had been previously configured with a different name, that would be kept & used.  I.e., some forums may have different custom avatar folder names, even in SMF2.1.
 - All custom avatars that are still attachments are renamed and moved to the custom avatar folder.

You can read the attachment processing in GitHub to see the logic used, starting here:
https://github.com/SimpleMachines/SMF/blob/374c3b0f164c8dca3a7138c531a00abe5811cf29/other/upgrade_2-1_mysql.sql#L375

If the 2.1 upgrader is run with incorrect attachment or custom avatar folder settings or permissions, the upgrader cannot find the attachment & avatar files to apply the fixes above.  Although everything else will look OK, you will be missing attachments &/or avatars in 2.1.  Possibly all of them...

Fixing via rerunning the upgrader
THE BEST SOLUTION FOR POST-UPGRADE ATTACHMENT & AVATAR ISSUES IS TO PROPERLY CONFIGURE THE FOLDERS AND THEN RERUN THE UPGRADER.

The upgrader has an option to "Rerun attachment conversion" - it's very important to check that.  The attachment process can be very time and resource consuming, and is only rerun when this box is checked.
You cannot view this attachment.

.dat utilities
There are a number of ".dat" utilities shared that add .dat to the filename in the file system.  These are 90% solutions, and do not address all of the issues listed above.  E.g., old filenames will not be fixed.  Also, any custom avatars stored as attachments will not be fixed. 

After running one of these .dat utilities, you can no longer fix these issues via re-running the upgrader, because the filenames will no longer match what the upgrader is looking for...  I.e., these utilities may hurt more than help.

They do have their place...  These utilities are perfect when:
 - You are 100% certain there are no old school filenames used.
 - You are 100% certain there are no custom avatars stored as attachments.  I.e., a custom avatar folder was configured, and further, the special hidden utility to move files from the attachments folder to the custom avatar folder has been run.

Attachment Integrity Check
SMF's Attachment Integrity Check in the admin control panel attempts to sync up your smf_attachments table with the files in the file system.  Extraneous attachment table rows, with no associated files in the file system, will be deleted.  Extraneous files in the file system, with no corresponding row in the attachment table, will be deleted.

So here's the danger...  The attachment integrity checker, like the upgrader, bases its actions on the contents of the smf_attachments table & the database's attachment folder settings...

Running the attachment integrity check when your forum's attachment & avatar folders are incorrect can actually delete... everything...

The attachment integrity check is a very helpful utility that should be run occasionally to delete the funky gunk that accumulates in the system from invalid uploads or abandoned posts over the years.

On the other hand, you should only run it when you know your configuration is solid...

IT WILL NOT BE HELPFUL FIXING PROBLEMS POST 2.1 UPGRADE...  BAD THINGS HAPPEN...

Two new utilities

I have written two new utilities, one to help diagnose what happened & how to fix, and one that emulates the 2.1 upgrader attachment & avatar logic as a standalone utility. 

smf_attachment_dirs.php - Found here: https://raw.githubusercontent.com/sbulen/sjrbTools/refs/heads/master/smf_attachment_dirs.php
 - This is a diagnostic utility only; it just displays information.  It only works in SMF2.1 & 3.0.
 - Its purpose is to help you figure out if your folders & settings are configured correctly.
 - It displays all critical attachment settings.
 - It displays file system information for any folder that starts with /att*.
 - It validates the contents of attachmentUploadDir.
 - It checks for errors that cause discrepancies between the file system & attachments, e.g., attachment records missing files, files missing attachment records, invalid filenames (doesn't match "#_hash.dat"), and file in the wrong folder.
 - The checks are not as thorough as SMF's Attachment Integrity Check; its focus is letting you know where the file system is out of sync.
 - It only works in SMF2.1 & 3.0.

smf_attachment_fix.php - Found here: https://raw.githubusercontent.com/sbulen/sjrbTools/refs/heads/master/smf_attachment_fix.php
 - This utility emulates the 2.1 attachment & avatar upgrade logic as closely as possible.
 - It should ONLY be used if it is impossible to rerun the upgrader for some reason.
 - It won't work if you've already run a .dat utility, as the filenames it's looking for won't be there.
 - It only works in SMF2.1 & 3.0.
 - BACKUP DB & FILE SYSTEM BEFORE USING THIS UTILITY!  If something goes sideways & you cannot recover, that's on you.

[Illori]

if you are running upgrade.php or another script that edits attachments should the forum be put in maintenance mode to make sure nothing happens while the script is running?

[shawnb61]

if you are running upgrade.php or another script that edits attachments should the forum be put in maintenance mode to make sure nothing happens while the script is running?

Yep.  Especially if it's large or highly active. 

[Illori]

which can be enabled here in the admin panel https://wiki.simplemachines.org/smf/SMF2.1:Server_settings#General