Problem with attachments upgrade

[Sir Osis of Liver]

Large forum was migrated to new server and upgraded from 2.0.19 to 2.1.4, upgrade completed normally.  Seeing this in /attachments -

You cannot view this attachment.

.dat extension has been added to pdf files, but not png or jpg files.  The forum is quite old (2003), contains over 78,000 attachments in 3 directories, most are pdfs.  FTP server is only displaying 9998 files in each directory, so I'm currently unable to view most of the attachments, and hesitate to run script to add .dat extension.  Any ideas?

[shawnb61]

First suspicion is that none of the files w/o .dat extensions have matching attachment records, e.g., does an attachment record exist with an id_attach of 1000?

Do the attachment directories look ok in the acp?  Counts look right?

[Illori]

.dat extension has been added to pdf files, but not png or jpg files. 

i bet your computer is set to open a .dat file with adobe reader, that is why it shows the pdf icon, these i bet are not even pdf files.

[Sir Osis of Liver]

No, filetype association for .dat is set for Notepad.exe in FileZilla.  The files are not pdfs, they're image files, but if I d/l them and try to open one with .dat extension it opens Adobe Acrobat which tells me it's an invalid or damaged file.  Renaming the file to test.jpg it opens normally in image editor.  There are three installs of this forum, 2.0.19 production is on old server, attachment filenames are normal, we migrated it to new server, 2.0.19 install displays filenames normally.  Upgraded to 2.1.4, that's where I'm seeing the filenames as pdfs. 

Attachment directories in ACP show file counts that are close to what I see in FZ, which includes temp and thumb files.  The problem seems to be the files that had .dat extension added are somehow screwed up.  That's a larger concern because most of the 78,000 attachments I can see have the extension and are shown as pdfs.

I've attached two of the files, they're both good jpgs, open and display normally here.

[Kindred]

No. If you download and try to open, it uses acrobat...  because you have acrobat defined as the program to use for .dat files on your conputer.
Filezilla is irrelevant.

[shawnb61]

e.g., does an attachment record exist with an id_attach of 1000?

Please answer the question above!

More questions:
 - When you browse the attachments in the ACP attachment browser, can you open them properly?
 - Do they look OK in the forum?

The .dat extensions are a good thing.  As Kindred says, the fact that your computer locally tries to open them as a .pdf is irrelevant.  (The same is true of ANY attachment file on ANY forum you've ever worked on...  If you download the .dat, your local computer will try to open it as a .pdf if it has that association...  And you cannot open images until you rename them...)

Further, the .dats suggest the upgrader completed successfully.

Everything might be OK...  I haven't seen something that indicates a problem yet...

[Sir Osis of Liver]

No. If you download and try to open, it uses acrobat...  because you have acrobat defined as the program to use for .dat files on your computer.
Filezilla is irrelevant.

That must be a default setting in windows, nothing I would have done.  Checked a few other 2.1.4 forums, they also display the pdf icon in FileZilla, just never noticed it before (it's rather tiny).

e.g., does an attachment record exist with an id_attach of 1000?

Please answer the question above!

No, there's no id_attach 1000 in attachments table.

More questions:
 - When you browse the attachments in the ACP attachment browser, can you open them properly?

They're displayed in attachments browser.

- Do they look OK in the forum?

That will take some searching, all of the images I've looked at are posted with img tags.

Further, the .dats suggest the upgrader completed successfully.

Well, that was the original thing I was looking at.  The first 6 and last 17 of the 9998 files I can see in /attachments don't have the .dat extension.  There are 26k files in that directory.  /attachments2 contains 52k files, looks like all the 9998 I can see have the extension.  /attachments3 has 151 files, all have extension.  I'm trying to get SFTP access so I can view all the files, but that's been a slog so far.

[shawnb61]

I'm pretty sure the upgrader only renames valid attachments, i.e., files with entries in smf_attachments.  That's why I asked whether 1000 had a record.  If not, I suspect that somehow the file is an orphan (e.g., message deleted), and thus it remained unnamed.

There are a # of reasons why the filenames might look incomplete, usually due to abandoned posts, or abandoned attachments, etc.  They can accumulate on forums that have been around a while.  Some of these are due to bugs that were fixed way back in 1.x or 2.0.x.  Note that you have more issues on the lowest numbered attachments.  Those are likely quite old.

If all the directories are properly configured, the 2.1 attachment folder clean utility normally cleans that all up. 

As near as I can tell, that forum looks fine. 

[Sir Osis of Liver]

Don't think it would do any harm to run the script to add .dat extension to files that don't have it.  There are some file names that end with _thumb, they would be changed to _thumb.dat unless I modify the script, which I'd rather not do.  Would be more comfortable with this if I could see all the files, but not having any luck with sftp access.

[shawnb61]

It looks like the upgrader successfully ran to completion.

I wouldn't run that utility.  I suspect it would do more harm than good. 

If anything, I'd run the integrity check.  It will likely delete those funky files.

[Sir Osis of Liver]

Bad things have happened when running the integrity check. 

Does the upgrader add the .dat extension to files that already have it?

[shawnb61]

The integrity check will delete attachment records if there is no file in the file system.

If the folders are incorrect, e.g., after a move, it can't find any files - so it deletes them all.

I.e., it's safe - and helpful - to run the attachment integrity check when attachment folders are configured properly.

It cleans up the gunk left behind by unusual circumstances.

And no, the upgrader doesn't re-add .dats.

It is always safe to rerun the upgrader.

In fact, the 2.1 upgrader has a checkbox for reprocessing attachments.  This allows you to correct your attachment directories & rerun.  There's no need for a separate script.

[Sir Osis of Liver]

The integrity check will delete attachment records if there is no file in the file system.

Well, that's what I'm afraid of.  If files without .dat extensions are not found by integrity check the records will be deleted.  If upgrader did not add extension to all files (it's happened more than once) we could lose a lot of content.  Until I can get sftp access (no luck so far) I can only see 20k of 78k files.

[shawnb61]

The integrity check will delete attachment records if there is no file in the file system.

Well, that's what I'm afraid of.  If files without .dat extensions are not found by integrity check the records will be deleted.  If upgrader did not add extension to all files (it's happened more than once) we could lose a lot of content.  Until I can get sftp access (no luck so far) I can only see 20k of 78k files.

While that is true, it does prompt you to confirm any updates made first.  You have to tell it to do so... 

I hope this might help, it's been something I've been thinking of doing for a while.  Yes, another utility:
https://raw.githubusercontent.com/sbulen/sjrbTools/refs/heads/master/smf_attachment_dirs.php

Right click & download, place in your folder root & execute.  Delete it when done.

This is an inquiry-only utility, no updates.  It's safe.  It will:
 - Display attachment settings & directories
 - Display & validate the crucial attachmentUploadDir setting
 - Compare your file system to your attachment records, and highlight files missing attachment records & attachment records missing files
 - Identify which records are missing .dats
 - Highlight any files that are in the wrong folder

Give it several seconds to execute if there are lots of attachments.  It only displays errors found.

Hope this helps,

[Sir Osis of Liver]

Will give it a go.  Host support went ahead and added .dat extension to files with root commands before I could get to it.  I've modified my .dat script to display a count of total files and fixed files when it completes, will try running it when we do final migration, which should be next day or two.

[Sir Osis of Liver]

What is the effect of adding .dat extension to _thumb file?

120524_b1e117b890bd4ae014a23bbaa2f8408a9686b2c2_thumb.dat

[shawnb61]

I think the _thumbs can be safely deleted.  This is true for *_thumb, *.dat_thumb & *_thumb.dat.  I suspect they're temp files from mid thumbnail process that get left around when issues are encountered.  I just delete them, and there are no issues viewing the associated attachments or thumbs.  Plus, they're all 0 bytes...

SMF 2.1+ thumbnails have their own .dat file in the file system and their own attachment record.  Those are the real thumbnails...

All SMF 2.1+ attachments, including thumbnails, are in the #_hash.dat format, e.g., 999_abcdef0123abcdef0123abcdef0123abcdef0123.dat

Which brings me to a related point...  I don't think simply adding a .dat to a file makes it usable under all circumstances to SMF.  It works most of the time, but not all of the time.  Some of the old 1.x & 2.0 SMF attachments have part of the original filename within them, e.g., "999_OwnersManualcdef0123abcdef0123abcdef0123"...

During the 2.1 upgrade, that gets renamed to the #_hash.dat.  Those are found properly.

So I don't believe that just renaming "999_OwnersManualcdef0123abcdef0123abcdef0123" to "999_OwnersManualcdef0123abcdef0123abcdef0123.dat" will work.  SMF still won't find it.

This won't impact newer forums, but is definitely an issue on older forums. 

You *MUST* rerun the upgrader to guarantee a proper #_hash.dat rename.  This is why I don't recommend using that utility, unless you know for certain there are no "old format" SMF attachment names around. 

If the utility reproduced the upgrader logic (https://github.com/SimpleMachines/SMF/blob/374c3b0f164c8dca3a7138c531a00abe5811cf29/other/upgrade_2-1_mysql.sql#L490) then you're probably OK.  If not, I'd avoid it.

[shawnb61]

FYI, new version of the utility uploaded; improvements to error checks.  Grab the new one if you're going to use it.

[Sir Osis of Liver]

Will try it soon as I get a chance.  FTP reports 78455 attachment files in 3 directories on production forum. 

Admin > Attachments and Avatars > File Maintenance reports following -

Production forum tradgang.com/tgsmf -

File Attachment Statistics

Total Attachments: 41759
Total Avatars: 522
Total Size of Current Attachment Directory: 471345.09 kB
Total Space Available in Current Attachment Directory:  No maximum directory size is currently set

new.tradgang.com/tgsmf21 -

File attachment statistics

Total attachments: 41,527
Total avatars: 521
Total size of all attachment directories: 31,511,518.35 KB
Current attachment directory:
    /home/tradgang/public_html/tgsmf21/attachments3
Total size of current attachment directory: 316,574.40 KB
Total space available: No maximum directory size is currently set
Total files in current attachment directory: 263
Total files remaining: No directory file limit is currently set


Host support is seeing 78509 files in three directories on prod forum, 64549 on new server where they were copied and upgraded.  The only numbers that are in agreement are the file attachment statistics, which are a bit more than half of the actual files.

[Sir Osis of Liver]

You can view the utility (snip)  It's running on the 2.1.4 install.