News:

Want to get involved in developing SMF, then why not lend a hand on our github!

Main Menu

Image attachments aren't preview/loading correctly in some posts after RC3

Started by kerbob, March 11, 2010, 09:30:34 AM

Previous topic - Next topic

kerbob

I verfied FileZilla on Windows and Unix both treat files without extensions as ASCII.

Road Rash Jr.

I use FileZilla too and although I've used it to back up my forum I have not had occasion to upload the backup files. Now that I know better, I'll not go there LOL but it makes backing up a useless effort.
I just tried to download a pic in binary mode, then uploaded that same pic, now it is not recognised by SMF.  ::)
Never argue with an Idiot like myself, they just drag you down to their level then beat you with experience.

kerbob

Binary more should work for the JPG, GIF, and PNG files.  So, you're saying that, even with the checkbox circle below cleared, that downloading and then re-uploading an image screws it up?


Kays

With that check box cleared, it should transfer binary files as binary.

If at first you don't succeed, use a bigger hammer. If that fails, read the manual.
My Mods

AncientDragonfly

Quote from: Kays on March 12, 2010, 12:11:19 AM
Sorta, I am aware of how it works. I was just curious if it will still transfer ASCII files as ASCII and not transfer them as binary.

The automatic mode depends on a list of which files are to be considered ASCII or binary.  FileZilla has a default list which contains common extensions for ASCII (am, asp, bat, c, cfm, cgi, conf, etc.), with a provision for adding new ones of your own.  For instance, if I wanted all .log files transferred as ASCII, I would add it to the list.  Anything that is on the list will be transferred as ASCII, and anything not on the list, as binary.  CuteFTP probably works the same way.

The problem is that the files don't have an extension, and most of the time, a file without an extension will be plain text, since most programs create files with their own specific extension. 

Quote from: kerbob on March 12, 2010, 12:16:48 AM
I'm screwed and feel like I let the people of my forum down.  I've called 1and1 and they're going to see if they have a backup that is over 48hours old.  :-\  My forum is now in maintenance mode.

You shouldn't kick yourself over it; it was an innocent mistake.  I hope you can get the backup restored.  It sounds like your forum depends heavily on attachments.   :(

QuoteI cannot believe this has happened to no one else!!  Check your FTP clients folks!!
If anyone knows a way to convert BIN files corrupted via ASCII transfer from Unix to Wndows, please let me know.  :-[

It did; it happened to me, and that's how I came upon this thread.  I was just fortunate enough to be working on a test forum, with my live forum still intact.  I was testing the RC3 update with a duplicate of the database, and noticed that none of the user-uploaded avatars were working.  In the admin attachments section, when I clicked on the individual avatar files, a window popped up and told me they couldn't be displayed because they were corrupted.  Came here looking for answers, and yours matched.  That's why I appreciate you posting about it.

Unfortunately, I don't think there's any way to convert back, because the 8th bit is stripped on ASCII transfers.  ASCII xferred as binary, yes; bin to ASCII, no.   :(

Quote from: Kays on March 12, 2010, 07:33:42 AM
With that check box cleared, it should transfer binary no-extension files as binary.

FTFY  ;)  Mine did, I tested it. 

I remember the security issue that prompted this (I think, last summer?).  A future solution for this with SMF might be to have SMF assign an extension for attachments (and uploaded avatars), but it would need to be carefully considered to avoid colliding with common ASCII extensions.   

Aha, this has been discussed before, over at the FileZilla project: http://forum.filezilla-project.org/viewtopic.php?f=2&t=12225&p=46454&hilit=ascii+default#p46454


Kays

Thanks for that. :)

That's not much of a discussion. More like a comment. All that I was curious about was if there was no extension , if it would treat ASCII files as ASCII and binary files as binary. But that link does sorta answer that question.

I have posted this on the team boards and am still waiting for a reply from the devs. But I can see it being a real pain in the butt to re-add file extensions.

If at first you don't succeed, use a bigger hammer. If that fails, read the manual.
My Mods

kerbob

+1 1AND1!!!!1!  FTW!!

1and1 was able to restore my attachments directory to the state prior to my overwriting the files.  8)

MAN O MAN... this should be in BIG BOLD letters in the Ops guide for anyone maintaining a SMF site.  Because of the encrypted file names, the extension is removed relegating your FTP client to resort to its default config for extensionless files.

Yeah, we've got several posters that spend a lot of time contstructing their posts and attachments, so the attachments are a big deal to me.  As an example, check out this thread: http://www.westknoxrc.com/index.php?topic=625.0

Back in the saddle again.  ;D

kerbob

I was digging through the SMF directories and there seems to be a catch 22..

There are SMF files that are extensionless (at least in some of the mods for sure) that NEED to be transferred as ASCII as well as ones that NEED to be transferred as BINARY (such as the attachments directory).

Now.. I'm confused again. 
What is the best way to backup my site correctly?!?

Kays

Glad that got sorted out. :)

That transfer question is what I was trying to get an answer to. It looks like it might transfer as binary, but it is recoverable. Do a test of a few files and try to open them.

If at first you don't succeed, use a bigger hammer. If that fails, read the manual.
My Mods

AncientDragonfly

Quote from: Kays on March 12, 2010, 12:29:56 PM
Thanks for that. :)

That's not much of a discussion. More like a comment. All that I was curious about was if there was no extension , if it would treat ASCII files as ASCII and binary files as binary. But that link does sorta answer that question.

Yeah, basically, it doesn't know how to treat them, as ASCII or binary, unless it's told.

Quote
I have posted this on the team boards and am still waiting for a reply from the devs. But I can see it being a real pain in the butt to re-add file extensions.

Thanks for that.  Maybe something to think about for the next major update.  It's not a bug with SMF, just a bad set of circumstances.

Quote from: kerbob on March 12, 2010, 12:37:56 PM
+1 1AND1!!!!1!  FTW!!

1and1 was able to restore my attachments directory to the state prior to my overwriting the files.  8)

Yay!

Quote
MAN O MAN... this should be in BIG BOLD letters in the Ops guide for anyone maintaining a SMF site.  Because of the encrypted file names, the extension is removed relegating your FTP client to resort to its default config for extensionless files.

Absolutely!  That could save a lot of headaches, and potentially forum disasters, for those diligent enough to read the manual and backup guide.

Quote
Yeah, we've got several posters that spend a lot of time contstructing their posts and attachments, so the attachments are a big deal to me.  As an example, check out this thread: http://www.westknoxrc.com/index.php?topic=625.0

Back in the saddle again.  ;D

I see.  Nice forum.   :)  Glad you could get it all back.  Nice of you to post a message letting your users know what was up, too. 

Road Rash Jr.

Quote from: kerbob on March 12, 2010, 12:37:56 PM
1 1AND1!!!!1!  FTW!!

1and1 was able to restore my attachments directory to the state prior to my overwriting the files.  8)

Back in the saddle again.  ;D

So what exactly was done to fix this? Inquiring minds want to know ;D
Never argue with an Idiot like myself, they just drag you down to their level then beat you with experience.

AncientDragonfly

Quote from: Bionic Rooster on March 12, 2010, 01:08:37 PM
Quote from: kerbob on March 12, 2010, 12:37:56 PM
1 1AND1!!!!1!  FTW!!

1and1 was able to restore my attachments directory to the state prior to my overwriting the files.  8)

Back in the saddle again.  ;D

So what exactly was done to fix this? Inquiring minds want to know ;D

The host, 1and1, restored a backup from a couple of days ago for him.


Road Rash Jr.

Quote from: Kays on March 12, 2010, 12:52:06 PM
Glad that got sorted out. :)

That transfer question is what I was trying to get an answer to. It looks like it might transfer as binary, but it is recoverable. Do a test of a few files and try to open them.

I ftp'd a file in binary and ascii and tried to open with various graphic programs. NADA. If I add the extension .jpg Windows photo gallery will open it but all other report it as not supported file format.
Even saving the file with WPG as a .jpg, as is or renaming it produced a "not supported file format" error.

I have no idea what the security issues where to do it SMF way, but it's a real pain and perhaps SMF should return to saving attachments in their proper format.
Never argue with an Idiot like myself, they just drag you down to their level then beat you with experience.

Road Rash Jr.

Quote from: AncientDragonfly on March 12, 2010, 01:12:20 PM

The host, 1and1, restored a backup from a couple of days ago for him.

Cool, so as long as there is a server backup we're fine but if we back it up via ftp we're screwed :o
Never argue with an Idiot like myself, they just drag you down to their level then beat you with experience.

AncientDragonfly

Quote from: kerbob on March 12, 2010, 12:43:57 PM
I was digging through the SMF directories and there seems to be a catch 22..

There are SMF files that are extensionless (at least in some of the mods for sure) that NEED to be transferred as ASCII as well as ones that NEED to be transferred as BINARY (such as the attachments directory).

Now.. I'm confused again. 
What is the best way to backup my site correctly?!?

Maybe SMF needs to put together a quick standalone script that will zip all the forum files, and let the user download and upload the zip, with an option for the upload to be unzipped on the server.  Any changes that the user needs to make can be put into the zip file before it's uploaded, effectively replacing the files on the server.  Of course, this would only be necessary for major file replacements.  A standalone script would not require another update, and could just go on the "Tools" list.

Leave your checkbox unchecked for extensionless files.  ASCII files uploaded as binary don't get corrupted; the only problem with them is end of line characters (CR, CR/Left), and there are lots of converters for that (Unix to DOS, DOS to Unix); some editors have them built in.


kerbob

Quote from: Bionic Rooster on March 12, 2010, 01:21:34 PM
Cool, so as long as there is a server backup we're fine but if we back it up via ftp we're screwed :o
Correct, if you're getting a server backup done you're ok.


1AND1 restored a full copy of my sites file structure into a FTP directory as it was the night before I hosed things.  All I did was download the Attachments directory as binary files and re-uploaded them at the site.  That solved the problem.. I think and I removed the site from 'maintenance mode'.

I still need to manually drill into each folder to check to see.. are there ANY other files that should be transferred as Binary that do not have an extension?  I have already found very many without extensions that need to be transferred as ASCII.  Unless they re-add the extensions to the attachments folder, I would recommend everyone just avoid using FTP as a backup method.

kerbob

Note to all FTP users: If you are backing up your site via FTP and you're not changing the mode of transfer for files without extensions.. your doing it wrong and risk having bad backups.  The process below is specific to FileZilla but applies to any FTP client that auto determines the transfer file type (ASCII or BIN) for certains files.

SMF BACKUP PROCEDURE USING FILEZILLA FTP
1.) Make certain that FTP client is configured to use transfer type AUTO for all files and that "Treat files without extension as ASCII file" is checked and "Treat dotfiles as ASCII files" is checked.
2.) Re-check Step 1.  Highlight all remote folders on your ftp site EXCEPT the Attachments folder and download them. Why?  These folders contain ASCII files without extensions and must be transferred as ASCII.
3.) Change the FileZilla configuration so that "Treat files without extension as ASCII file" is unchecked.
4.) Re-check Step 3.  Highlight the remote Attachments folder and download it.  Why? This folder must be downloaded as Binary because it contains Binary files (jpg, gif, png, etc).

Does anyone know if there is any other folder within SMF that may have BINary files that have no extensions?

kerbob

All, I'm marking this topic as solved for me.  Kays has posted this up on the Team Boards and is following up with it there.

The root issue is.. if you use FTP (regardless of which client you use) to backup your site, you have to switch modes for files without extensions:
- Attachments directory files without extensions should be transferred as BINARY
- All other directories that have files without extensions should be transferred as ASCII

Thanks Kay for following up on this.  I'm hoping someone can write a nifty FTP backup script for us so that this is easier to do on a weekly basis.

Thanks,
Kerbob

AncientDragonfly

Quote from: Kays on March 12, 2010, 12:29:56 PM
Thanks for that. :)

That's not much of a discussion. More like a comment. All that I was curious about was if there was no extension , if it would treat ASCII files as ASCII and binary files as binary. But that link does sorta answer that question.

I have posted this on the team boards and am still waiting for a reply from the devs. But I can see it being a real pain in the butt to re-add file extensions.

Kays, there has been at least one more victim of Filezilla, and possibly a second:
http://www.simplemachines.org/community/index.php?topic=372577.0
http://www.simplemachines.org/community/index.php?topic=372619.0

Do you think you could get a sticky notice put up in the "Install and Upgrade Help" board with something about this problem?  Maybe something added to the "Before You Upgrade" section of the manual?

The devs can discuss it till the cows come home, but that doesn't help the people that are losing pieces of their forums now, while a fix is being discussed.

Got any strings you could pull?   :) 

AncientDragonfly

I put together a post about this issue and how to avoid it in the Install and Upgrade Help section, with some additional stuff about making a mirrored test forum: http://www.simplemachines.org/community/index.php?topic=374178.0


Advertisement: