News:

SMF 2.1.4 has been released! Take it for a spin! Read more.

Main Menu

Bad links for SOME attachments

Started by SethStudent, November 03, 2019, 06:26:17 PM

Previous topic - Next topic

SethStudent

The only thing I found in the cPanel labeled CPU was this, hopefully that's what you're talking about. I just want to make sure you know that the attachment problem is not new, it's just that I only recently found out about it from a new member. Attachment links prior to 07/16/18 are broken and while I haven't looked at every attachment since that date, the newer ones seem okay from what I've seen so far. If there was a CPU problem on 07/16/18 I doubt there'd be a record of it any more.

I'll also take a look at the links in the Attachment browser this afternoon, randomly checking attachments dated after 07/16/18 to see if they're all functional.

Illori

did anything that you can recall happen in July of 2018? server move/server crash etc?

SethStudent

Quote from: Illori on November 25, 2019, 01:49:11 PM
did anything that you can recall happen in July of 2018? server move/server crash etc?

No, not that I can think of. I mentioned in my post back here I even thought maybe an update or something:

"I looked at dates in the Package directory thinking maybe I did a patch or something around 07/06/18 that would affect the db, but no. 2.0.12 and .13 were installed Feb 2017. 2.0.14 and .15 were installed 08/20/18."

I don't know of any server crashes, no moves. My forum has been on the same hosting/server since Feb. 2017. I have a few Wordpress sites hosted in that same public_html directory and they are all fine, were never offline as far as I know.

Since the database controls everything, you'd think there would be visible evidence of some sort of differences in the attachment table between working and non-working links.

Illori

the database does not control everything. for examples attachments are in a folder on your server, they are partly controlled by the database but if not found on the server you will run into the issue you are facing now.

SethStudent

Gotcha, thanks. Yes they are in the folder. When this first began I downloaded the Attachments folder, was able to identify the first two "missing" attachment files, add the appropriate extension (jpg) so I could open them in Photoshop, save as a jpg and reattach since a member wanted to see them.

When I'm going through all the other stuff this afternoon—shawnb's audit steps and randomly checking the newer attached after July 201—I'll also compare the files in the attachment dir with the Attachments browser list and database. I won't be able to check every one of them, but enough to get an idea if the "missing" ones are actually in the dir.

SethStudent

Quote from: shawnb61 on November 25, 2019, 01:26:37 PM
Let's see if the audit turns up anything...

So far I've only checked one attachment thoroughly, it took quite a bit of time and I'll check a couple more tomorrow.

Example from post 12857, dated 061919, attachment #2146 HillHouse.jpg [404 error]

In current Attachment Dir (total 1,627 files as of today):   
2146_436898194c0c8184112ce8407ebe11623ae74bb9.dat 06/29/19

In upgraded site (1,615 files, older tally from 11/14):
2146_436898194c0c8184112ce8407ebe11623ae74bb9.dat 11/14/19

In current Attachment Browser, on pg. 8 of 20 pages:
HillHouse.jpg 720x540  246.57kB Deb   June 19, 2019, 08:32:12 AM
in The Hill House 5 views

In upgraded Attachment Browser. pg. 25 of 66 pages:
HillHouse.jpg 720x540 246.57KB   Deb June 19, 2019, 03:32:12 pm
in The Hill House 2 views (upgrade may have set back to zero)

Checked the "Configure multiple attachment directories" and there was only one folder, the correct one /speakingofseth/attachments. It did say 1636 files and then Files Missing (repair) which I was afraid to click. Since I'm doing this on a test site, I may just click it tomorrow. I can always restore the site.

Went through the database thru the control panel, every attachment has id_folder as 1, also all approved 1.  id_member are mostly 0, just the avatars have a real member number. The number of attachments matched the number of attachments in the directory.

I'm done for the day, I'm out of ideas and need to be somewhere else anyway.

shawnb61

Look into some old ones, before July 2018.
Address the process rather than the outcome.  Then, the outcome becomes more likely.   - Fripp

Sir Osis of Liver

Just got on, will have a look.  Thinking about this while eating my supper (and having a few beers) I don't think there's anything wrong with your database or attachment files.  Have you tried connecting your production db to a clean 2.0.15 install?
Ashes and diamonds, foe and friend,
 we were all equal in the end.

                                     - R. Waters

Sir Osis of Liver

Were the images here - https://speakingofseth.com/index.php?topic=530.msg8610#msg8610 - all attached at the same time?  None display thumbs, but the thumb icons and links work on some, not others.
Ashes and diamonds, foe and friend,
 we were all equal in the end.

                                     - R. Waters

aegersz

#29
Quote from: Sir Osis of Liver on November 24, 2019, 04:31:53 PM
The different attachment counts is something I've seen before, and the primary reason I don't trust the maintenance task.  In your db screenshot some of the mime types are incorrect.  Any correlation with missing attachments?

You don't trust it ? I found it very reliable and have used it (in Dev) to remove attachments when i deleted members via the database (i know, i was experimenting).

I think you can safely run it without fear BUT having said all that, it's a big deal so I feel your fear.

EDIT: I just ran it on my live system and it fixed 7 reported file size errors and it gave me the option to do so prior than to just automatically fixing things so there's your safety net.
The configuration of my Linux VPS (SMF 2.0 with 160+ mods & some assorted manual tweaks) can be found here and notes on my mods can be found here (warning: those links will take you to a drug related forum). My (House) music DJ dedication page is here

shawnb61

The attachment integrity check deletes attachment records it finds confusing, e.g., if an attachment folder got changed somehow.  When dealing with some issues, it can make a bad situation much worse.

It can be helpful routine maint when things are setup properly.  But not when diagnosing issues.
Address the process rather than the outcome.  Then, the outcome becomes more likely.   - Fripp

SethStudent

Quote from: Sir Osis of Liver on November 25, 2019, 08:48:12 PM
Have you tried connecting your production db to a clean 2.0.15 install?

No, that's a great idea though. I'll do that today on another test site.

Quote from: Sir Osis of Liver on November 25, 2019, 10:41:46 PM
Were the images here - https://speakingofseth.com/index.php?topic=530.msg8610#msg8610 - all attached at the same time?  None display thumbs, but the thumb icons and links work on some, not others.

I had to switch over to Safari to see the thumb missing image icons. Yes, Attachment browser says all were attached at the same time--10:30:02 on Feb 14, 2017. The first three are bad, the next four are ok (except for the missing image icons), the last two are bad.

Quote from: shawnb61 on November 25, 2019, 08:41:57 PM
Look into some old ones, before July 2018.
Quote from: Sir Osis of Liver on November 25, 2019, 08:48:12 PM
I don't think there's anything wrong with your database or attachment files.

I'll go ahead and do the audit on some of these particular images anyway since they're before July 2018. I originally thought that was the day separating which attachments functioned and which ones didn't, but came across that later one which makes me think there are more bad links that I first realized. Especially when I saw that the Attachment browser on my production site shows 20 pages of attachments, and the upgrade 2.1 test AB shows 66 pages. So more than 2/3 of my attachment links aren't working.

I'll update after I've done the 2.0.15 clean install and the audit of some of the attached files (screen capture).

Quote from: shawnb61 on November 26, 2019, 10:48:18 AM
The attachment integrity check deletes attachment records it finds confusing

Yep, that's what it did to mine.

SethStudent

UPDATE: I think I know what's going on. In viewing the Attachment folder from my live site Speaking of Seth on my computer, in Mac Finder, under file "kind", some files were labeled as documents and some were labeled as TextEdit.app Document, which stood out. The ones that were labeled as TextEdit did not have the .dat extension on the file name. See 1st attachment. These are the ones that "worked" in the associated messages—the ones missing the extension. The ones with .dat extensions were the ones that don't work, at least in the 15 or so attachments I've been investigating.

I looked at the Attachments folder for the test site I'd upgraded to 2.1, and all the files have the .dat extension. But the links also work (as far as I can tell, I have't looked at every single one). The upgrade somehow fixed them. See 2nd attachment. All functional attachments on my Seth forum, from July 14, 2019 to today, do not have the .dat ext. I have no memory of anything happening on July 14.

Right now I have some other work I need to get done, but this afternoon I will do the clean install of .15 and see if that fixes the links. If not, since the files with .dat are not recognized, I'll experiment by manually removing the extensions from a few bad link attachment files to see if that fixes anything.

shawnb61

The .dat extension is added as part of the 2.1 upgrade. 

In 2.0, the .dat MUST NOT be present; in 2.1, the .dat MUST be present.  That's why the 2.1 upgrade works better - some of your files already have that extension. 

The very first thing I'd try is to simply remove the .dat in 2.0 on a trial attachment or two & see if that works.
Address the process rather than the outcome.  Then, the outcome becomes more likely.   - Fripp

SethStudent

Quote from: shawnb61 on November 26, 2019, 02:04:19 PM

In 2.0, the .dat MUST NOT be present...

The very first thing I'd try is to simply remove the .dat in 2.0 on a trial attachment or two & see if that works.

Thank you Shawn!

Wow then I wonder how they got in there to begin with? I'll definitely try the extension removal test. If I have to go that route, it will take some time since there are so many attachments, but from the Attachments browser tallys it seems that about 1/3 of my attachments may have the extension. That's a lot better than 2/3.  ???

I'll update later today.

shawnb61

Don't do that by hand.  Once you confirm it works, do the rename via a wildcard.

Of course, back everything up again first.  Again.  Twice.
Address the process rather than the outcome.  Then, the outcome becomes more likely.   - Fripp

aegersz

Quote from: shawnb61 on November 26, 2019, 10:48:18 AM
The attachment integrity check deletes attachment records it finds confusing, e.g., if an attachment folder got changed somehow.  When dealing with some issues, it can make a bad situation much worse.

It can be helpful routine maint when things are setup properly.  But not when diagnosing issues.

I see. Good point. maybe have the delete function as an added option one has to check box it, one day ?
The configuration of my Linux VPS (SMF 2.0 with 160+ mods & some assorted manual tweaks) can be found here and notes on my mods can be found here (warning: those links will take you to a drug related forum). My (House) music DJ dedication page is here

SethStudent

Quote from: shawnb61 on November 26, 2019, 02:45:59 PM
Don't do that by hand.  Once you confirm it works, do the rename via a wildcard.

Of course, back everything up again first.  Again.  Twice.

I've become a Master Backer Upper, lol. Not being very familiar with the technical aspects, I guess I don't understand—the file names only need to be changed in the Attachments folder, correct? I don't see a way to search and replace in the File Manager of the cPanel, although I could ask the host for advice.

Does anything need to be changed in the database? Skimming visually through the Attachments table, the images just have the names without the .dat extension. Even a search turned up no .dat extensions.

shawnb61

The files need to be renamed in the file system.  The host should be able to help. 

If not, check back.
Address the process rather than the outcome.  Then, the outcome becomes more likely.   - Fripp

SethStudent

Quote from: shawnb61 on November 26, 2019, 04:35:29 PM
The files need to be renamed in the file system.  The host should be able to help. 

If not, check back.

OK, this is very exciting for me. I figured out a way to fix it in less than 10 minutes. In case anyone else ever needs to do this (god forbid), and with no waiting for the host company to answer a support ticket: After I put the forum into Maintenance Mode,

1. I downloaded the Attachments folder (once again).
2. Opened it on my Mac.
3. In Finder/Preferences, I checked Show all filename extensions; uncheck Show warning before changing an extension.
4. In the Attachments folder, I selected all, then right clicked: I chose Rename x items; Find: .dat  Replace with: (blank); clicked Rename.
5. I renamed the original Attachments folder in the site directory and uploaded the newly edited one.

All of the attachments I've checked so far, various dates, are alive and well. Life is good.

Thank you so much for your help, I never could have gotten this figured out if it wasn't for all the replies from you and others at SM. I know you all have personal lives and jobs, and your devotion to Simple Machines is very obvious. Honestly, last night I was starting to feel like this was a problem that would never be solved. After 3+ weeks of stressing over this, I can finally get on with my life.  ;D

Case closed.


Advertisement: