"Attachment Directory can not be located" when trying to add an attachment

[casey_mrc]

Thank you to the experienced gurus on here who are continuing to give advice. I'm somewhat concerned that this 21 year old Forum (2003-2014, VBulletin, 2014-2024 SMF) might be in serious trouble, hopefully not!

If you have access to the old server, do a complete backup of all files and databases to your local computer.  If not, ask your host for a complete account backup from old server.  Do you have a backup of the production database?

I have FTP access to the old server and have a taken a local backup of the Attachments Directory (10,000 files, 3Gb). I don't have Control Panel access any longer and I can't see the SQL Database file via FTP to back it up. I'm not sure where this is located, as I normally backup via the CP or SMF Admin. I have an out-of-date SQL backup (2023) in worst case.

There is a "snapshot" backup directory with of all files on the old server, taken around the time of the migration. I'm downloading that to my PC as I type this, which might take some time! There ought to be snapshots of the database too, but the relevant 6 snapshot folders (rotating each day) are EMPTY.

The host has now sorted out my FTP access to the new server - the new server IP I was given doesn't work for some reason, but they have given me a path which works.

The good news is, the Attachment directory is there on the new server, but is much larger (38823 files, nearly 10Gb) than the size of the Attachments directory on the "pre-migration" instance. I have looked back at a 2021 backup of this directory (which is easily at hand) and it is 29,816 files and 4.2Gb, suggesting the  new server instance is probably about right.

So, it seems the "connection" of these attachments to the relevant forum posts has been broken during the transfer and/or during the related UK2 fix of my reported "unable to make new attachments" ticket.

I have raised a ticket with UK2, but as all the attachments appear to be present, is there a way of "re-connecting" them to the correct forum posts or? I'm asking here, just in case UK2 don't come up with a fix.

I have also requested a complete backup of all files and databases immediately prior to the migration. They must have done that surely? It is standard practice before any major work......

[casey_mrc]

I can't help but think the clue is here:

You cannot view this attachment.

It is a fact that /public_html/forum/attachments exists, I can see it in FTP view.

It is a fact that /home/sites/31a/e/ed77b10c29/public_html/forum/attachments is now being populated with NEW attachments, since the migration, along with the 10Gb of files that were migrated over. BUT, the forum only shows the new attachments, the "size" and "files" has incremented since the screenshot above.

[Sir Osis of Liver]

Your host appears to have trashed your forum.  The more they do, the worse it gets.  You must obtain a complete dump of your current production database, preferably pre-migration.  If you can get that, and you have a complete backup of attachments, then most of your content is secure.  Next thing to do is run (not walk) to a different host.  The forum can be rebuilt from scratch as long as you have the database and attachments.

FYI, the database is not a file and not accessible via ftp.  You need control panel access to mysql databases or phpmyadmin.

[Sesquipedalian]

Indeed. This topic reads like something out of a web hosting horror story.

[casey_mrc]

That's not good to hear 

You must obtain a complete dump of your current production database, preferably pre-migration. 

I've requested it, awaiting a response.

Lets say I have an up to date copy of the Attachments directory and I'm only able to get a copy of the database that is, say, 6 months older than the Attachments file contents, what impact would that have on starting again from scratch? Hopefully not the scenario I'll face, but just in case, would be good to know consequences.

I've not done a fresh install of SMF since 2014, when migrating from vBulletin. I know there are some good SMF installation guides around though. Assume basic steps would be;

install SMF (use latest v2.1.4? Compatible with the v2.1.2 I was running wrt attachment formats, etc?)
populate attachments file with backed-up files
config SMF with database details
Pray....

I'm not a programmer (apart from some machine code stuff with 8080 microprocessors, back in the 1980's, some recent coding for Arduino microcontrollers and some ancient BASIC), but I'm pretty good with logic.

In simple terms, do I assume the database contains the "instructions" to link a post with relevant attachments (a table entry?), enabling SMF to display everything correctly in a browser environment? Where is the text of posts actually stored? I'd like to understand more of the structure of SMF and how it works, is there a (simple-ish) guide somewhere. Bear in mind the only knowledge I have of databases is of simple ones, like those provided with MS Works many moons ago  .

[Sir Osis of Liver]

Post text is in messages table, attachments are linked to posts in attachments table.  If you rebuild the forum with an older database, newer attachments will be ignored because the posts don't exist.  You can rebuild the forum on new host by doing a fresh 2.0.19 install with new database, drop all database tables, import the production database, upload attachments, and run repair_settings.php.  Once it's up and running correctly, you can upgrade to 2.1.4.  You can go directly to 2.1.4, but anything you get from your current host is questionable, so I'd rather see it running in current version before attempting the upgrade.  I do see that https://mazdarotaryclub.com/forum/ is running 2.1.2, but that would seem to be the mess your host created.

[casey_mrc]

That's very helpful.

In trying to understand what has happened. The forum boards, messages, membership, etc are all OK, it's just the attachments that are missing.

If there is an attachment table in the database, then is is a "simple" matter that the table can't find the attachments? The attachments are there, so is it "just" a "path" in the table that needs to fixed?

[Sir Osis of Liver]

The attachments table contains id_attach and id_msg to link it to the post, and file_hash, which is the hashed filename you see in /attachments directory.  The settings table contains attachmentUploadDir, which is the server path to /attachments directory.  You can set the correct path in Admin > Attachments and Avatars > Attachment Settings > Manage attachment directories and make it the current directory.
 

[casey_mrc]

Thanks. That's helpful. I will look into this whilst host tries to sort things out.

No time today, off to help friend at a day at Brands Hatch race track, setting up his triple rotor  700bhp+ Mazda RX-8 for the race season ahead 

[casey_mrc]

This has been dragging on for too long. The host has tried (unsuccessfully) to fix the missing attachments issue.

They have now agreed to grant me cPanel access to the pre-migration instance of the forum so I can backup the home directory files and the database.

I will also proceed to backup (copy) the contents of any posts/topics made since the flawed migration. Once that is done, the host has agreed to run the migration again, to see if the attachments issue recurs. If it does, I will have to use the backups to move to a new host.

Any recommendations on a good host and service to use (preferably UK based)? Currently on a stackCP shared hosting service with UK2. I have had issues with TSO recently, so not going to use TSO again.

MySQL database is ~1Gb
Attachments folder is ~10Gb

Forum is over 20 years old, previously vBulletin, migrated (by me) in 2014 to SMF.

Thanks again for your help and advice at this somewhat stressful time.

[Steve]

Any recommendations on a good host and service to use (preferably UK based)?

We have a dedicated board for hosts: https://www.simplemachines.org/community/index.php?board=4.0

[Sir Osis of Liver]

Crocweb is very good, inexpensive, and has excellent support.  I've moved approx. two dozen forums there from other hosts.

[casey_mrc]

Well, it looks like we have a happy ending. I asked a friend (Stew) with SQL and coding skills to take a look at the problem and he has fixed the problem in a few hours this morning (where our hosts had failed for the past 4 weeks  )

In case this helps anyone else, and to close out this sage, here is Stew's description of his fix;

Whilst the problem was in the DB, it was not as a result of our first guess and probably had got into its current state as a result of both the original website migration and the attempts to fix it subsequently.

There is a table called `smf_attachments`. This table records each individual attachment, the ID of the message (forum post) that it is attached to, the ID of the attachment directory it is stored in and a bunch of other meta data. I got hold of a back-up of both the old and new DB and was able to compare. The table in the old DB had ~34K of records, where as in the new DB there were only 342. I suspect that one of those repair operations had 'helpfully' deleted records for all the 'broken' attachments.

Going back a step to the original issue caused by the move - unable to make any new attachments. As mentioned, the smf_attachments table contains an ID for the 'folder' (attachment directory) that holds the given attachment. SMF allows you have to multiple, with a specific ('current') one used for all new uploads. Prior to the move you had a single upload directory with an ID of 1. After the move, you still had the old folder configured (but now inaccessible), but then added configuration for several more (all duplicates) based on the details that UK2 had supplied of the new attachment directory path with one (having an ID of 2) set as the active one. I guess this is when new attachments started working again correctly, which resulted in new records being added to the smf_attachments directory with a folder ID of 2.

My fix = I have edited all the records in my copy of the old smf_attachments table to have a folder ID of 2, removed ~340 records that are currently present in the new database (for reasons I am not entirely sure), then dumped those records into a file and uploaded into the live forum DB smf_attachments table. There were ~360 pre-existing records in the live DB that had a folder ID of 1 so I have updated them also to have the correct ID of 2. That is where we are at now.


The result? All attachments are correctly attached to the relevant posts and the admin attachment directory screen now looks like this;

You cannot view this attachment.

All ~38,400 old pre-migration files, plus ~340 post-migration attachments are now showing in the directory.

Phew!

I'll leave things for a week or two, for users to confirm no other problems, then get onto updating to v2.1.4.

[Sir Osis of Liver]

Next thing to do is run (not walk) to a different host.