Filezilla has changed one of their default settings which will destroy your forum's avatars and attachments if you depend on it to transfer your files.
There is a setting under the Edit menu --> Settings --> Transfers --> File Types: "Treat files without extension as ASCII file."
(https://www.simplemachines.org/community/index.php?action=dlattach;topic=370199.0;attach=137471;image)
This should be unchecked when transferring directories that contain avatars or attachments because these are binary files.
kerbob has written instructions for backing up your site using Filezilla.
Quote from: kerbob on March 12, 2010, 03:21:21 PM
Note to all FTP users: If you are backing up your site via FTP and you're not changing the mode of transfer for files without extensions.. your doing it wrong and risk having bad backups. The process below is specific to FileZilla but applies to any FTP client that auto determines the transfer file type (ASCII or BIN) for certains files.
SMF BACKUP PROCEDURE USING FILEZILLA FTP
1.) Make certain that FTP client is configured to use transfer type AUTO for all files and that "Treat files without extension as ASCII file" is checked and "Treat dotfiles as ASCII files" is checked.
2.) Re-check Step 1. Highlight all remote folders on your ftp site EXCEPT the Attachments folder and download them. Why? These folders contain ASCII files without extensions and must be transferred as ASCII.
3.) Change the FileZilla configuration so that "Treat files without extension as ASCII file" is unchecked.
4.) Re-check Step 3. Highlight the remote Attachments folder and download it. Why? This folder must be downloaded as Binary because it contains Binary files (jpg, gif, png, etc).
Does anyone know if there is any other folder within SMF that may have BINary files that have no extensions?
It is also a good idea to make changes (updates, modifications, etc.) on a test forum before changing your live forum, because you have an opportunity to check that everything is working correctly before you make irreversible changes to your live forum. Using a test forum prevented me from losing my avatars and attachments because I noticed the problem on my test forum first, and left my live forum alone until I found the problem.
To make a test forum which is a mirror of your existing forum:1. Backup your current forum's files and database. If you are using Filezilla, be sure to follow kerbob's instructions above. I like to use phpMyAdmin to back up my database, but the SMF Admin section also provides a method to do this.
2. Create a new space within your website for the test forum, and upload the files you backed up. Again, if you are using Filezilla, be sure to follow kerbob's instructions above.
3. Create a new database through your web host's control panel with phpMyAdmin. Make note of the database name, username, and password which you created when you made the new database.
4. In the new database, use phpMyAdmin to restore your backed up database from the live forum.
- Select the database
- View the structure
- Check the box to select all the tables and then choose the action "drop" under the dropdown. That will leave the database itself but drop all data.
- Under the mysql tab, restore your backup by importing the database from the live forum. That will put everything from your live forum into your test forum's database.
Tutorial videos are here:
How to drop tables: http://www.charlottezweb.com/tutorials/phpma-deltable.html
How to import: http://www.charlottezweb.com/tutorials/phpma-import.html
5. Upload the file "repair_settings.php (http://download.simplemachines.org/index.php?thanks;filename=repair_settings.php)" to the root directory for your test forum. (For example, if you put your test forum in the directory, www.example.com/testforum/, put repair_settings.php in that directory. I would recommend using a unique name for your test forum directory, as "testforum" is too easily guessed. If you're of the tinfoil hat variety, I would also recommend password protecting the directory with .htaccess.)
6. Run repair_settings.php by going to http://www.example.com/testforum/repair_settings.php and change all the entries to match your test forum. What is repair_settings.php (http://wiki.simplemachines.org/smf/What_is_repair_settings.php)?
If you do not do this, any changes to the database, theme, posts, etc. that are done through the forum software will be made to your live forum instead and defeat the purpose of having a test forum.
7. Important: DELETE repair_settings.php from your site; it can be used to get your database password, and to change crucial settings that make the forum work.
8. Test your test forum.
Thanks to kerbob for the Backup instructions and for bringing this issue to light. There is more discussion in the thread: http://www.simplemachines.org/community/index.php?topic=370199.0
Thanks to Jason of CharlottezWeb.com (http://www.charlottezweb.com/) for the instructions and video tutorials (I hope he doesn't mind me linking them) and for being a super great host for SMF forums.
There is more information about restoring a MySQL database here: Restoring a MySQL Database (http://wiki.simplemachines.org/smf/Restoring_a_MySQL_Database)
And yet more about backing up and restoring here: [Tutorial] ''How to backup & restore your forum in several different ways'' (http://www.simplemachines.org/community/index.php?topic=184356.0)
I had this issue previously. Since then I had been really careful about it. Thanks for the heads up once again. I am sure this will help alot of users around. :)
Thanks for the tip - this has been an issue with FileZilla since SMF 1.1.9 / 2.0 RC1-1.
Quote from: (F.L.A.M.E.R) on April 02, 2010, 11:57:15 AM
I had this issue previously. Since then I had been really careful about it. Thanks for the heads up once again. I am sure this will help alot of users around. :)
It will help, if it can stay up where people who take the trouble to look can see it. I see a new thread from yesterday about more lost avatars and attachments.
What is the proper way to ask for the thread to be stickied?
You should ask one of the support team members to do it for you. They are the one who have the permissions to do it.
You may also hit the "Report to moderator" and write there what you want done. Maybe that will send a message to everyone.
Thanks, (F.L.A.M.E.R). I'll use "Report to moderator" since so many staff members have "do not PM me" in their sigs. Just didn't want to report my own post and maybe get myself banned. :)
The reason for 'no PMs' in our sigs (both when I was staff and not) is simple: so many people send support PMs expecting an answer when they should use the forum.
Quote from: Arantor on April 11, 2010, 10:09:34 AM
The reason for 'no PMs' in our sigs (both when I was staff and not) is simple: so many people send support PMs expecting an answer when they should use the forum.
I know. :) Just makes me hesitant to PM anyone, whether it's for support or not.
I can't speak for anyone else, but I know that if you PM me and it isn't support, good chance of getting a reply. But I have to just say 'no PMs' otherwise most people don't listen.
Topic Stickied as requested. Good tip. ;D
Thanks for letting me know, Arantor. I sympathize. There may be a PM from me sometime in your future.
Quote from: Chas Large on April 11, 2010, 10:17:29 AM
Topic Stickied as requested. Good tip. ;D
Thanks, Chas; that was fast service. Let's hope it helps some people.
I hope so too. You beat me to recommending this topic to another member who may well have suffered the same problem. If only my memory wasn't limited to 64K, I might remember a few more things :D
Quote from: Chas Large on April 11, 2010, 02:31:33 PM
I hope so too. You beat me to recommending this topic to another member who may well have suffered the same problem. If only my memory wasn't limited to 64K, I might remember a few more things :D
Are you a Commodore 64? ;D
Could be a ZX Spectrum; or indeed any machine based on a 16-bit address bus (i.e. M6502, M6800, Z80, 8080 type processor amongst many others)
Now you're all taking the proverbial P*** :D
I'll have you know I started out with a ZX80. 4K memory (2KROM + 2KRAM). I wrote a word processing program for it. It displayed Word Proce.... and ran out of memory (LOL)
Ah, the good old ZX80, with integer handling, a Z80 processor clocked at 3.57MHz IIRC, much improved when ZX81 came out :)
My first was a VIC-20. Display was on a TV, and my black and white just wasn't cutting it, so my first color TV was bought just for the VIC. I wrote some simple graphics programs in BASIC - simple (by today's standards) was all the machine was capable of - and typed in a lot of programs from magazines. http://en.wikipedia.org/wiki/Type-in_program
Ok, I haven't backup in a week so I am gonna backup today but I want to make sure the image shown:
http://www.simplemachines.org/community/index.php?topic=377117.msg2594267#msg2594267
is correct? SO that it doesn't not mess up the avatars and attachments.
Of course by far the fastest way to back up folders, if you have cPanel access, is to use the cPanel file manager. This is a lot faster than FTP and has no problems with any folders or their content.
Ok, the setting should be uncheck ONLY on attachments and avatar folder, is there other folders I should be worried about that need to be download as Binary?
It doesn't actually hurt to transfer everything as binary, really.
The whole concept of text transfer as text is an outdated concept that's really not an issue with modern editors; it's only for the olden days when editors couldn't handle Linux and Windows file endings (like Windows Notepad still can't, heh), if you're using a modern editor like Notepad++, it's a non issue.
Quote from: Arantor on May 03, 2010, 06:18:13 AM
It doesn't actually hurt to transfer everything as binary, really.
The whole concept of text transfer as text is an outdated concept that's really not an issue with modern editors; it's only for the olden days when editors couldn't handle Linux and Windows file endings (like Windows Notepad still can't, heh), if you're using a modern editor like Notepad++, it's a non issue.
I will try everything as binary the next time I back up. It's better than sorry not to try both methods. For now I am leaving it AUTO and check "Treat files without extension as ASCII file" and uncheck it for attachment and avatars only.
When you say binary...you mean not auto and and what about "Treat files without extension as ASCII file"?
Text mode == ASCII mode, they're one and the same.
When I'm using either FileZilla or WinSCP, I have it set to treat *everything* as binary.
Quote from: Dismal Shadow on May 02, 2010, 06:26:11 PM
Ok, I haven't backup in a week so I am gonna backup today but I want to make sure the image shown:
http://www.simplemachines.org/community/index.php?topic=377117.msg2594267#msg2594267
is correct? SO that it doesn't not mess up the avatars and attachments.
No, that image in your other post is
incorrect.
This image from the first post in this thread is correct:
(https://www.simplemachines.org/community/index.php?action=dlattach;topic=370199.0;attach=137471;image)
As Arantor says, however, you should be ok to download ASCII (plain text) files using binary mode. Auto mode is just a list of extensions that specifies which files should be treated as ASCII but since the files we are talking about don't have extensions, it can't be used in this situation.
Basically, if you open a file in a text editor and it looks like non-alphabetic, non-numeric, non-standard-punctuation characters, it is a binary file (see attachment).
ASCII can be transferred as binary without corruption.
Binary files will be corrupted if they are not transferred as binary.
QuoteASCII can be transferred as binary without corruption.
Not entirely true, in fact, it depends what you define as 'corruption'.
If you mean 'without corruption' as the actual, byte for byte file it will be corrupted.
The file will be silently converted for line endings. For example, go grab the SMF install package and open any SMF file in Windows Notepad. Not Notepad++, not anything fancy, just normal Notepad. You get a mess where everything is on one line.
Now download the same file from FTP in ASCII mode and open it. Now it's not a mess. Why? Because FileZilla has silently converted the line endings on your behalf.
If you have ANY doubt, use binary mode. I expect the files I send and receive to be the actual files, not a modified-in-ANY-way version.
Ok uncheck the one from the image and set it to binary mode instead of auto. I open with textwrangler, seem fine. No lines messed up.
That could be because TextWrangler can handle Windows, Linux and Mac line endings, which would hide if the file format has been changed on you without you being aware of it.
Quote from: Arantor on May 03, 2010, 06:27:38 PM
QuoteASCII can be transferred as binary without corruption.
Not entirely true, in fact, it depends what you define as 'corruption'.
If you mean 'without corruption' as the actual, byte for byte file it will be corrupted.
The file will be silently converted for line endings. For example, go grab the SMF install package and open any SMF file in Windows Notepad. Not Notepad++, not anything fancy, just normal Notepad. You get a mess where everything is on one line.
Well, yes, that is true in the sense it wouldn't pass a CRC (or some other kind of integrity) check, but it will be recoverable with one of the many line-ending converter programs, unlike a binary file like an image, which, once it's lost every 8th bit, it's gone for good.
Quote
Now download the same file from FTP in ASCII mode and open it. Now it's not a mess. Why? Because FileZilla has silently converted the line endings on your behalf.
If you have ANY doubt, use binary mode. I expect the files I send and receive to be the actual files, not a modified-in-ANY-way version.
But you said:
Quote from: Arantor on May 03, 2010, 06:30:01 AM
Text mode == ASCII mode, they're one and the same.
When I'm using either FileZilla or WinSCP, I have it set to treat *everything* as binary.
:P (I'm on FreeBSD - binary doesn't mess up my line endings.)
Quote from: Dismal Shadow on May 03, 2010, 06:39:07 PM
Ok uncheck the one from the image and set it to binary mode instead of auto. I open with textwrangler, seem fine. No lines messed up.
Good. Glad to hear it.
/me thinks people don't understand what text mode is.
It DOESN'T eat every 8th bit.
What it does is convert line endings depending on what the server and client are using.
Windows uses \r\n (ASCII codes 13, 10, aka carriage return and newline) to mean end of line
Linux, *nix generally, including niche OSes derived from that heritage (including AmigaOS, randomly) uses \n
Mac historically used \r, don't know if it still does
Some images are fine, in fact, because they contain neither \r or \n, but it's incredibly rare for that to be the case.
Thus, text == ASCII mode, because it's not stripping bit 7 unless it's REALLY stupid, because it doesn't foul up extended bit characters, or UTF-8 characters for that matter (except line endings which are the same across UTF-8 due to UTF-8 reusing the same 0-127 sequence)
Binary wouldn't mess up your line endings. It would preserve it, that's kind of the point!! Text mode WILL screw your line endings unless you happen to be on the same OS as the server in which case I'd hope your client was smart enough to realise.
/me doesn't want to argue with Arantor, so will just include a few links with accompanying text.
http://www.faqs.org/rfcs/rfc959.html
Quote
RFC 959
File Transfer Protocol
[snip]
3.4.3. COMPRESSED MODE
There are three kinds of information to be sent: regular data,
sent in a byte string; compressed data, consisting of
replications or filler; and control information, sent in a
two-byte escape sequence. If n>0 bytes (up to 127) of regular
data are sent, these n bytes are preceded by a byte with the
left-most bit set to 0 and the right-most 7 bits containing the
number n.
http://www.faqs.org/rfcs/rfc1635.html
QuoteRFC1635 - How to Use Anonymous FTP
- You may set BINARY mode to transfer executable programs or files of data. Type "binary" to do so. Usually FTP programs assume files use only 7 bits per byte, the norm for standard ASCII-encoded files. The BINARY command allows you to transfer files that use the full 8 bits per byte without error, but this may have implications on how the file is transferred to your local system.
http://courses.wccnet.edu/computer/mod/na36c.htm
QuoteFTP was developed at a time when typical modem speeds were 110 to 300 bits per second (as compared with 28,000 to 56,000 today). Since ASCII only used 7 bits, long files could be transmitted more quickly by not sending all the unused bits.
The big drawback is that if a file that uses all 8 bits in each byte is accidentally sent using ASCII transfer, it will lose 1/8 of its information content. In most files, even the loss of one bit is enough to make it invalid, and losing 1/8 makes them totally unreadable. So ASCII transfer can be fatal to a file's health!
With today's higher speeds, the time lost by sending all 8 bits of an ASCII file is practically unnoticeable. But FPT has incorporated features into ASCII transfer that make it useful for other reasons, so the two modes remain.
http://mc-computing.com/winexplorer/ftp.html
QuoteBinary vs ASCII
The internet was developed to transfer information in 7-bit packets.
Computers store data in 8-bit bytes.
Normal ASCII text data is stored using only 7 bits per byte (the 8th bit is always zero). (This kind of document is created using programs like notepad.exe) As a result, it is fairly easy to transfer this type of data over the internet. This is one reason the html pages are always written in ASCII.
Other types of data (images, programs, MS Word documents, and the like) use all 8 bits to store data. As a result, special algorithms are required to transfer 8-bit data over a 7-bit interface.
When you are using FTP, you need to be aware of the type of data being transferred - ASCII or binary (7-bit or 8-bit).
If you try to transfer 8-bit data in ASCII mode, data WILL be lost (the 8th bit will be set to zero).
http://www.ietf.org/mail-archive/web/ietf/current/msg40628.html
Quote
To: Sandeep Srivastava , ietf at ietf . org
Subject: Re: [RFC 959] FTP in ASCII mode
From: John C Klensin (http://en.wikipedia.org/wiki/John_Klensin)
Date: Tue, 21 Feb 2006 03:34:27 -0500
[snip]
> I knew that a FTP transfer in ASCII mode does EOL and EOF
> conversions based on the OS of the receiving system.
No, it doesn't. That was part of the point. It does no EOF
conversions at all. The command and data channels were
separated for several reasons, but the desire to stay out of the
EOF business was an important one. And the server is required
to convert whatever line-end convention it uses to CRLF, and any
characters it uses to ASCII, and transmit that over the wire.
If the client then converts from CRLF and ASCII to some local
convention, that is its business, not that of the protocol. In
other words, there are, at most, conversions to and from CRLF
and ASCII. There are no FTP-specified conversions based on the
properties of the receiving system.
/me thinks this esoteric discussion may confuse people who just want to make sure they don't lose their files.
Bottom line, or tl;dr: Transfer your avatars and attachments directory as binary files.
Originally, yes that was the case. But 959 has been superceded multiple times since 1985, meaning that in the current specifications, it's not a strict 7-bit safe only environment.
If it were, every instance of SMF would be broken by FTP clients that treat PHP files as 'text' because there is a function in Subs.php that has 8-bit characters in it. As would any language file that isn't English.
Quote from: Arantor on May 05, 2010, 06:07:58 PM
Originally, yes that was the case. But 959 has been superceded multiple times since 1985, meaning that in the current specifications, it's not a strict 7-bit safe only environment.
Do you mean to say that the only difference between an ASCII transfer and a binary transfer is what happens with line endings? John Klensin says that any translation besides that to CR/LF for transfer (specified by network ASCII) is performed by the client for the local system, not by the server (if the server follows protocol). So wouldn't that imply that between two identical systems, there would be no difference in line endings once the transfer is complete? So if I transfer a binary file between 2 FreeBSD systems in ASCII, the LFs will be converted to CR/LFs for transfer, then the client will convert them back to LFs, so the file would essentially be the same, wouldn't it? So a binary file (say, like an avatar) transferred as ASCII between identical systems will not be corrupted? (No, I haven't tested this yet, I'll have to do that later sometime.)
Arantor, I am the kind of person who likes to understand things, not just take someone else's word for them (not at all implying, though, that your word isn't good), and there is no question that you are far smarter than I am or ever will be, so how about sharing the source of these new (to me) revelations?
I have found RFC 2640 (http://tools.ietf.org/html/rfc2640), which is titled "Internationalization of the File Transfer Protocol," and says this:
QuoteAbstract
The File Transfer Protocol, as defined in RFC 959 [RFC959] and RFC
1123 Section 4 [RFC1123], is one of the oldest and widely used
protocols on the Internet. The protocol's primary character set, 7
bit ASCII, has served the protocol well through the early growth
years of the Internet. However, as the Internet becomes more global,
there is a need to support character sets beyond 7 bit ASCII.
This document addresses the internationalization (I18n) of FTP, which
includes supporting the multiple character sets and languages found
throughout the Internet community. This is achieved by extending the
FTP specification and giving recommendations for proper
internationalization support.
but it also says this:
Quote1 Introduction
As the Internet grows throughout the world the requirement to support
character sets outside of the ASCII [ASCII] / Latin-1 [ISO-8859]
character set becomes ever more urgent. For FTP, because of the
large installed base, it is paramount that this is done without
breaking existing clients and servers. This document addresses this
need. In doing so it defines a solution which will still allow the
installed base to interoperate with new clients and servers.
This document enhances the capabilities of the File Transfer Protocol
by removing the 7-bit restrictions on pathnames used in client
commands and server responses, RECOMMENDs the use of a Universal
Character Set (UCS) ISO/IEC 10646 [ISO-10646], RECOMMENDs a UCS
transformation format (UTF) UTF-8 [UTF-8], and defines a new command
for language negotiation.
The recommendations made in this document are consistent with the
recommendations expressed by the IETF policy related to character
sets and languages as defined in RFC 2277 [RFC2277].
and this:
Quote2 Internationalization
The File Transfer Protocol was developed when the predominate
character sets were 7 bit ASCII and 8 bit EBCDIC. Today these
character sets cannot support the wide range of characters needed by
multinational systems. Given that there are a number of character
sets in current use that provide more characters than 7-bit ASCII, it
makes sense to decide on a convenient way to represent the union of
those possibilities. To work globally either requires support of a
number of character sets and to be able to convert between them, or
the use of a single preferred character set. To assure global
interoperability this document RECOMMENDS the latter approach and
defines a single character set, in addition to NVT ASCII and EBCDIC,
which is understandable by all systems. For FTP this character set
SHALL be ISO/IEC 10646:1993. For support of global compatibility it
is STRONGLY RECOMMENDED that clients and servers use UTF-8 encoding
when exchanging pathnames. Clients and servers are, however, under
no obligation to perform any conversion on the contents of a file for
operations such as STOR or RETR.
The character set used to store files SHALL remain a local decision
and MAY depend on the capability of local operating systems. Prior to
the exchange of pathnames they SHOULD be converted into a ISO/IEC
10646 format and UTF-8 encoded. This approach, while allowing
international exchange of pathnames, will still allow backward
compatibility with older systems because the code set positions for
ASCII characters are identical to the one byte sequence in UTF-8.
This seems to mean it is only talking about pathnames changing, not the ASCII spec. But are the local systems changing the way things are done depending on their localization? They know that some characters in that localization require all 8 bits? Or the local system determines that a file has non-ASCII characters, and adapts accordingly?
I have found RFC 3659 (http://tools.ietf.org/html/rfc3659), which updates (but does not obsolete) 959, and it says this:
QuoteThis document also uses notation defined in STD 9, RFC 959 [3]. In
particular, the terms "reply", "user", "NVFS" (Network Virtual File
System), "file", "pathname", "FTP commands", "DTP" (data transfer
process), "user-FTP process", "user-PI" (user protocol interpreter),
"user-DTP", "server-FTP process", "server-PI", "server-DTP", "mode",
"type", "NVT" (Network Virtual Terminal), "control connection", "data
connection", and "ASCII", are all used here as defined there.
which is defined there as:
QuoteASCII
The ASCII character set is as defined in the ARPA-Internet
Protocol Handbook. In FTP, ASCII characters are defined to be
the lower half of an eight-bit code set (i.e., the most
significant bit is zero).
This is knowledge-seeking on my part, not a "my * is bigger than your *" challenge to you.
Quote from: Arantor on May 05, 2010, 06:07:58 PM
If it were, every instance of SMF would be broken by FTP clients that treat PHP files as 'text' because there is a function in Subs.php that has 8-bit characters in it. As would any language file that isn't English.
Then if the 8th bit in ASCII transfers can now be 0 or 1, how come the images in the Avatars and Attachments directory get broken?
OK, hereafter I'm going to throw out the rulebook and the specs and go purely on what I have seen and observed because when it comes down to it, quoting spec after spec doesn't solve the issue - because it's how things implement the spec that matters.
(Heck, if everyone observed the spec right, how come IE6 is such a pile of ****?)
Anyway.
There are two VERY DIFFERENT issues here.
One is 8-bit sanctity, one is line ending sanctity.
The former is essentially a non issue for modern FTP servers and clients because funnily enough the spec is ignored, and they send 8 bit files as is. In today's UTF-8 world, it's pretty much a necessity to send 8-bit files, because UTF-8 is a full 8-bit text format.
The latter is still very much an issue because line ending conversions get done. THIS is what breaks attachments, not loss of the 8th bit.
How do I know this? I've seen attachments broken by this, specifically I've observed the differences in the files, and viewed them in more tolerant viewers. The files get broken part way through, not totally. If 8th bit loss were the case, on average 50% of the file would be damaged since the odds of any one bit being set are non exclusive 50%. But that's not the case.
From experience, FTP between two Linux servers hasn't damaged such files. That isn't to say it wouldn't but in my experience that was the case.
Quote from: Arantor on May 05, 2010, 08:37:13 PM
OK, hereafter I'm going to throw out the rulebook and the specs and go purely on what I have seen and observed because when it comes down to it, quoting spec after spec doesn't solve the issue - because it's how things implement the spec that matters.
Ok, fair enough.
Quote
(Heck, if everyone observed the spec right, how come IE6 is such a pile of ****?)
Microsoft wanted to write their own spec for the internet - they thought they were an exception to the existing specs, that they were powerful enough to write their own, and that everyone else would adjust to it because they were Microsoft.
This was obvious in their early implementations of FrontPage - I was working tech support at a web hosting company at the time, and we had many, many customers call to find out why their pages written with FrontPage which displayed perfectly in IE (3? 4?) would not display in Netscape. Try telling a bunch of customers who thought Microsoft invented the internet that FrontPage and IE didn't follow the already established standards! ::)
Microsoft has since realized that they need to work with the internet technical community, instead of trying to take it over. :)
You might find this thread on the ASCII issue interesting (that is, unless you're bored with the whole thing by now): http://www.securityfocus.com/archive/1/437948/100/0/threaded
Quote
Anyway.
There are two VERY DIFFERENT issues here.
One is 8-bit sanctity, one is line ending sanctity.
The former is essentially a non issue for modern FTP servers and clients because funnily enough the spec is ignored, and they send 8 bit files as is. In today's UTF-8 world, it's pretty much a necessity to send 8-bit files, because UTF-8 is a full 8-bit text format.
The latter is still very much an issue because line ending conversions get done. THIS is what breaks attachments, not loss of the 8th bit.
How do I know this? I've seen attachments broken by this, specifically I've observed the differences in the files, and viewed them in more tolerant viewers. The files get broken part way through, not totally. If 8th bit loss were the case, on average 50% of the file would be damaged since the odds of any one bit being set are non exclusive 50%. But that's not the case.
From experience, FTP between two Linux servers hasn't damaged such files. That isn't to say it wouldn't but in my experience that was the case.
Ok, fwiw, I tested 2 files on my BSD machines. I did this with both the command line FTP client and with Filezilla, with the same result. One was a plain text file, one was a png file. I downloaded both using both ASCII and binary. The ASCII downloaded png file wouldn't open in GIMP; GIMP suggested that the file might be corrupt. I then opened both the ASCII-downloaded and the binary-downloaded png files with Diffuse, a file comparison editor (similar to Diff, but graphical), and the only differences in the two files that I could see were some blank lines in one but not the other. There were about 8-10 instances of this in the whole file. With the plain text file, there were no differences apparent in Diffuse.
This suggests you're right about the line ending translation being what messes up the binary files instead of the 8th bit, since every other character was the same, and none of the 8-bit characters had been converted to standard ASCII characters. If the 8th bit had been changed to 0 on all the bytes, I would have expected to see only alpha-numeric and punctuation characters in the ASCII-downloaded binary file.
Anyway, thanks for the stimulating discussion, and for bringing my FTP/ASCII knowledge up to date. I never mind having what
I think I know challenged if it might be wrong.
*nods* Experience definitely wins - and it's great to get more experience on the matter, and good to have references to things to read up on.
That's the thing - you appreciate the journey as well as the destination itself, and it's been a pleasure discussing this :)
LOL so on or off, auto or binary that's all we needed to know ! :P
If in doubt, binary.
Here's a question: in the original post, it says the option should be UNCHECKED in the image. Then directly after, there are some instructions that state the option should be CHECKED. So which is it, checked or unchecked??
EDIT: Doh... I'm an idiot. Guess I couldn't make my brain go past what I thought was a confusing error. ::)
nice glitch, really!
that check box could be completely removed as it should transfer everything in BINARY until known extensions transfer in ASCII.
Consequentially this is a complete mess and got it all wrong :o
thanks for sharing
I use Filezilla
Thanks for this interesting information. Really helpful
Quote from: Forum Guy on May 31, 2010, 03:52:35 AM
nice glitch, really!
that check box could be completely removed as it should transfer everything in BINARY until known extensions transfer in ASCII.
Consequentially this is a complete mess and got it all wrong :o
So, it is enough to transfer all files in Binary and it will be alright?
This information should be on the forum backup page in the forum admin. This problem could be completely resolved if the forum backup function really did backup the forum, including attachments and avatars. Give it all to me, in one nice tar.gz. Simple, no caveats. The tricks only come when you have to extract the file and decide where to put the contents. That's easier to do than discovering you didn't actually have a correct and complete backup in the first place!
Quote from: qwasty on November 09, 2010, 01:09:08 AM
This information should be on the forum backup page in the forum admin.
Well, perhaps - but then again, this post is to support users of one specific FTP client, and it really isn't SMFs place to support 3rd party software IMO. Something general to remind about FTP backups, and how to transfer files could be done of course.
Right, I've complained about this before when I was surprised to find out the backup function didn't really do a backup. It just gave me bits and pieces, and I had to go hunt down the other bits and pieces separately before I had a full backup. That would be easy to fix, and would probably save a lot of forums.
Well, the point of the admin panel backup is, that it is a Database backup. And all you need to do to save an installation, with all of it's actual contents is the database backup. Agreed, that won't help you save attachments, avatars, themes etc, but for real emergency situations the DB is the biggest worry for most admins.
I made this mistake a year ago and am just finding this thread. Does anyone know if i re-upload all the attachment files under the new filezilla settings. replacing existing files if this will correct the images?
I have shifted my forum from one host to another. [www.merapahadforum.com]
While I got all the post back but lost all my attacments. Yes I did use Filezilla.
After reading this thread I have unchecked the Filezilla setting as mentioned and downloaded all the attachements in my windows machine. I have uploaded all the attachments on my new host (deleting the old one) but I could not get them back.
Then I went to old host and create the tar backup using
tar -cvf att.tar /home/username/public_html/attachments
downloaded it using sftp and uploaded to new host. The result remained same. I have also done chmod 777 for attachment directory.
Am i missing something, somewhere.
I am using RC4
I could sort it out. It was a file path related problem
Is filezilla truly a good ftp program to use, as I read somewhere that it can sometimes leave out some type of file, like pictures or something. I cannot remember the actual topic completely but it was something about images?
Quote from: crotalusco on November 23, 2010, 01:36:30 AM
I made this mistake a year ago and am just finding this thread. Does anyone know if i re-upload all the attachment files under the new filezilla settings. replacing existing files if this will correct the images?
If the files have been corrupted by an ASCII download, that is, if data has been stripped from them, it appears that there is no getting them back. Uploading them with the correct settings will not restore the missing data.
If there is a piece of software which can restore the files that someone knows of, it would probably do a lot of people a big favor to post the name of it.
Quote from: Alison Black on December 22, 2010, 06:18:01 PM
Is filezilla truly a good ftp program to use, as I read somewhere that it can sometimes leave out some type of file, like pictures or something. I cannot remember the actual topic completely but it was something about images?
Alison, go back and read this thread for the details.
I like Filezilla quite a bit as an ftp program, but it has to be configured correctly. But then,
all software must be configured correctly to work as one intends it to, unless it's a "one trick pony" kind of software.
QuoteIf there is a piece of software which can restore the files that someone knows of, it would probably do a lot of people a big favor to post the name of it.
There isn't, because it has no way of knowing the content of the original file to be restored. If there were some signature of the original file (md5 of the file contents), there's a chance but if you realise that for every instance of \r or \n in the file - and that's likely to be a lot in images, actually, there are 2 combinations, and those 2 combinations are exponential - so if there was a file with 10 original entries of either \r or \n, each of those is now an entry of \r\n, and you need to go through every combination, which is 2^10 = just 10 original values is 1024 combinations. 1024 combinations to try, calculate new signature, it's computationally expensive.
And there are usually a LOT more than 10 instances in a large file... often thousands. 2^1000 is a very large number.
Theoretically, it could be done, like the NSA might do with Osama Bin Laden's encrypted files if they were to find them, but practically speaking, for the little guys (us), it's foolish to think about. (Just an example, not meaning to start a political discussion.) Your web host's backup, if there is one, is the best possibility.
/me laughs at the whimisical idea of someone starting a distributed computing project, like Folding or SETI, to get their forum attachments and avatars back.
I still like the idea of SMF just giving people a real backup when it says it's giving people a backup in the administration panel. That ought to be plan A, if plan B is using the web host's backup, and plan C is becoming a terrorist leader so the NSA will repair your files for you.
Which would lead to even more support issues as enough people have trouble with backups timing out as it is, and most server configurations would not support sending every attachment through the admin panel without hitting either the PHP or Apache timeouts. (cPanel runs its own web server and so has no such direct limitation)
Which is what volume splitting is for, so you have smaller files that are more manageable to download without timeouts.
Sure, but either you prepare the volumes up front (which means doubling the data stored on disk while you download it), or you prepare it on the fly, which can lead to memory timeouts as people already experience when gzipping their content.
Ideally, SMF needs to have the entire backup code ripped out and rewritten from scratch to support this kind of thing though that's clearly not going to happen prior to 2.1 at the very earliest.
I think it would be really cool if SMF could figure out a way to "chunk" it's database, so older bits of it get stored separately from newer bits. That way, you wouldn't have to backup the entire thing every time, just the parts that have changed. Of course, that leads into the idea of diffs and deltas, which may not be used directly, but could provide some models for simplified techniques to achieve something similar.
On low to moderate traffic forums, it becomes feasible to email a backup every day automatically. Wouldn't that be cool?
I'd hate to get that by email every day when there are much more efficient ways of doing it, like automating a scheduled task on my PC... like I already do now, but that's just me.
The problem with deltas is that to restore the database you have to rebuild it incrementally rather than a single restore motion, and I'd personally hesitate to build such an option for users that will inevitably get it wrong, which is why I abandoned development of so many mods over the time I was modding, knowing the support issues, so ultimately I'd suggest SMF not take that course as a standard feature personally.
It's not impossible to chunk, of course, just the restoration is sufficiently complicated it would confuse most users here, I think, knowing the number of times I got fed up with people who didn't even read what was put in front of them multiple times in large red letters. That said, it gives me an idea for the other thing I'm working on...
i using filezilla,,,its work.
Good tip about Filezilla, the settings were indeed wrong. Could that be why some of my members can't access the forum anymore? I just upgraded to RC4 and some members say they get these messages
- unable to verify url, please go back and try again
- Maintanance Mode and we're attempting to restore an older back up.
The problem doesn't appear on a Mac. One member has a PC and can't access with that, but the Mac doesn't give problems..
Any tips?
Quote from: Omebolle on January 13, 2011, 07:08:12 AMCould that be why some of my members can't access the forum anymore? I just upgraded to RC4 and some members say they get these messages
- unable to verify url, please go back and try again
- Maintanance Mode and we're attempting to restore an older back up.
The problem doesn't appear on a Mac. One member has a PC and can't access with that, but the Mac doesn't give problems..
Any tips?
I don't think so (I could be wrong), but you might want to search this forum for the error messages, and if you don't find anything that fixes it, start a new topic. ;)
thanks for info 8)
Just a thought... Wouldn't it be far simpler to give these files an extension that can then be removed by the forum code upon detection?
Quote from: headstone on March 12, 2011, 09:43:31 AM
Just a thought... Wouldn't it be far simpler to give these files an extension that can then be removed by the forum code upon detection?
Funny you should say that. The reason the files are stored without extension is to prevent the files being abused; there was a vulnerability a bit back that allowed files with extensions to be triggered by the server itself, which lead to a lot of forums being hacked.
Honestly, though, it's a bit of a poor showing that FileZilla assumes anything without an extension is 'likely a text file' by default... which is where the problem comes in.
Yes, in theory, it can be modified to have a generic extension that won't normally be executable - but it's a change that you won't see in SMF, certainly not before 2.1 at the earliest (if at all)
Quote from: Arantor on March 12, 2011, 12:09:54 PM
Honestly, though, it's a bit of a poor showing that FileZilla assumes anything without an extension is 'likely a text file' by default... which is where the problem comes in.
Yes, I agree. And if I remember correctly, this was an unannounced change in an upgrade from its previous default behavior where it assumed it was not a text file, which further compounded the problem for people who had the previous version installed. Upgrades shouldn't change user settings without some sort of notice to the user. (imo)
Quote from: Arantor on May 03, 2010, 06:30:01 AM
Text mode == ASCII mode, they're one and the same.
When I'm using either FileZilla or WinSCP, I have it set to treat *everything* as binary.
including .htaccess?
Can I uncheck both "Treat files without extension as ASCII file" and "Treat dotfiles as ASCII file"
and then upload everything as binary?
thanks
OMFG ...
This should be it the admin panel.
I have moved my site 3 times and wondered why 50% of the attachments never made it.
now my question that I didn't see a answer on is, why don't all said images have extensions? this would make life so much easier :P
but now I know what I been doing wrong, ThankYou for the heads up ;D
The attachments don't have file extensions to make it harder to upload an executable file like a shell script and use it to take over the server ;)
Quote from: ryry46d9 on May 09, 2011, 04:02:20 AM
OMFG ...
This should be it the admin panel.
I have moved my site 3 times and wondered why 50% of the attachments never made it.
Yes.
Thanks.My brother is searching for this board.
Quote from: Ljungberg on March 17, 2011, 12:58:10 AM
Quote from: Arantor on May 03, 2010, 06:30:01 AM
Text mode == ASCII mode, they're one and the same.
When I'm using either FileZilla or WinSCP, I have it set to treat *everything* as binary.
including .htaccess?
Can I uncheck both "Treat files without extension as ASCII file" and "Treat dotfiles as ASCII file"
and then upload everything as binary?
thanks
Yes.
Both for upload and download...everything binary.
Most of the time binary is quite safe to use on everything, although I do not know if htaccess and other text only files like robots.txt would ever have any real benefit from being treated as binary... The most important thing is to make sure that Filezilla uses binary as default for files without extension, and for files with extensions it doesn't recognise, as this would make sure your image attachments are treated as binary like they should be.
This is Stuart from Minneapolis (America's best city). I came across this thread and found it interesting. Hopefully, we can get new ideas from each other as we all are here for sharing our knowledge and experience. :)
I just transferred my forum and used FileZilla....
Couldn't see attachments and avatars....
Thanks to everyone as yes the ascii transfer in auto mode was the culprit. Re-transferred everything without the treat files w/o extensions as ascii...and set it to transfer everything as binary and it worked.
This should be a required part of any move SMF to new host documentation. Again, thanks...
I'm glad we made a big enough deal about this that you were able to find it and make use of the information here.
Why would they do that? I mean do you know any extensionless files that AREN'T binary files? Other way around yeah maybe, .htaccess no name, just file with extension its ASCII... but FILE. ect.... chances are its binary, default to that unless told otherwise....geez... you would think OS guys would know better.... guess not, they must work for Apple lol :(
I had this issue previously. Since then I had been really careful about it. Thanks for the heads up once again. I am sure this will help alot of users around.
EDIT by (F.L.A.M.E.R):
Links/Images for advertisement removed.
Quote from: outlawsys on October 15, 2011, 01:40:41 AM
Why would they do that? I mean do you know any extensionless files that AREN'T binary files? Other way around yeah maybe, .htaccess no name, just file with extension its ASCII... but FILE. ect.... chances are its binary, default to that unless told otherwise....geez... you would think OS guys would know better.... guess not, they must work for Apple lol :(
Complain to FileZilla, not Apple. Who- who am I kidding? That was funny!
you all are bringing back memories. I still have a working Commodore 64 :)
Thanks for posting this.. It is helpful for those of us who would otherwise not know such things and be banging head on table.
I'm confused! I get the bit about checking the ASCII box to transfer all files and folders except Attachments and Avatars, and then unchecking the box to transfer those folders separately.... but then you all start going on about binary! I've left the Auto option checked throughout the whole backup as per the picture in the first post of the thread - should I have changed this to Binary as well?
Does anyone know if this is also an issue when backing up Wordpress files?
Thank you for your post!This can help a lot of people who doesn't know what to do on such situation.
Quote from: Sabrinova on April 22, 2012, 09:53:31 AM
I'm confused! I get the bit about checking the ASCII box to transfer all files and folders except Attachments and Avatars, and then unchecking the box to transfer those folders separately.... but then you all start going on about binary! I've left the Auto option checked throughout the whole backup as per the picture in the first post of the thread - should I have changed this to Binary as well?
Does anyone know if this is also an issue when backing up Wordpress files?
set everything to binary and you'll be fine. ASCII changes the line endings, and binary does not.
Ya i use filezilla.For transferring a site in Live i think it no 1
Wooooow thanks for share this info. Now i know why happens this :/
I am very grateful this post was stickied! I would have had no clue about any of this! Thanks many times over!! :)
My question is same as above. Do I also uncheck it for avatars folder? You mention to uncheck it for attachments directory but didn't mention avatars folder. I'm just assuming to also uncheck it for avatars folder since avatars are the topic subject.
You can just transfer everything as binary and it will be ok, as Arantor says here:
http://www.simplemachines.org/community/index.php?topic=374178.msg2610981#msg2610981
Or use your host's cpanel filemanager to make a zip file of everything, then download the zip. If you do this, make sure to read the popup window that shows what has been zipped because any error messages will show up in here. (For instance, if you don't have enough space for the zip file in your web space, it will fail.)
Thank you all for this thread as I had the very same problem. Godaddy was able to restore my forum directory from a backup and I have now moved it to my new domain name correctly and all now works okay. I thought, for a moment, I had lost all of my attachments and avatars but this thread alerted me to the problem and I was able to recover. That was a close call!
Glad to hear all is working now, SatTrkr. Let us know if you have any other issues.
Thanks for the fix. I was worried that I'd lost all my photos and avatars after I moved hosts.
I just reuploaded my 'attachments' folder contents in 'binary' rather that 'ACSII'
FYI - my filezilla options was under 'Transfer > Transfer Type > select Binary'.
I am using FileZilla 3.10.2 version the option 'Treat Files Without Extension' is checked by default. Do i need to uncheck it ?
I don't thinks so because i have been using same settings since long without any problem in my SMF forum like Avatar, Images etc you mentioned.
Correct me if i am wrong.
Hoping for a reply. Thanks :)
you are wrong.
If understood correctly, I need to use these settings for the whole forum-folders except for attachments?
(https://i.postimg.cc/SKfsjvbk/Screenshot-346.jpg)
For attachments folder these are the settings?
(https://i.postimg.cc/nhsSHy79/Screenshot-347.jpg)
Please correct me if I'm wrong... thanks! :)
In most cases, you should be fine using binary or auto for everything really. Just as long as the attachment files (or other non-text files) don't get sent as ASCII.
In some previous backups I have lost avatars and attachments and the settings were to default (auto/checked ASCII file).
Now if I set to binary and leave checked ASCII file would be fine?
Or I must also unchecked ASCII file? :-\
You should be able to use the settings in your second picture for everything.
Thanks @Lex, you are very kind. :)
You are welcome :)