Advertisement:

Author Topic: CR + LF file corruption after FTP transfer  (Read 1774 times)

Offline Aleksi "Lex" Kilpinen

  • Support Specialist
  • SMF Super Hero
  • *
  • Posts: 16,542
  • Gender: Male
  • Don't worry, I'm n00b friendly
    • Aleksi.Kilpinen on Facebook
    • aleksi-kilpinen on LinkedIn
Re: CR + LF file corruption after FTP transfer
« Reply #20 on: March 17, 2018, 02:31:59 AM »
For example, upload a config file with wrong line endings, and you could potentially render a server inoperable.

As far as I know, both Linux and BSD support CR + LF (Windows) line endings. Everything should work without a problem if the server is set up correctly... even if the line endings don't match the native Linux/BSD ones.
But do it the other way around, and Windows boxes will crash.
It's also said to be faster than binary.

This doesn't make any sense. In binary mode, the FTP client just transfers the files without analyzing them first to see if they are a text file or not... ASCII mode should be slower.
I'm not sure of this either way, but that's what I have heard. The whole protocol was originally designed for transferring text files.

Anyway, there is some progress on the topic at hand ;).

I did some thinking about the corrupt attachment issue... I may have a solution ;). I just need some information on how the hash of the attachments is calculated (i.e. their file name in the attachments folder). I noticed that even if I upload the same file twice in a certain topic, the hash values for both of the files differ in the database. This was kind of disappointing, since I thought that the hash values are only check sum based, but I guess I was wrong.

My idea was to let the tool (I started coding something) do every possible reconstruction scenario of a file, calculate the check sum and each time a reconstruction scenario is complete, compare the check sum with the generated check sum in SMF. Since there is an attachment integrity tool, I assume there must be some sort of a check sum calculated by the scrip or at least taken into account when calculating the hash value of the attachment. I just need the algorithm that calculates the hash value, reverse it, discard anything that is not the check sum of the file and compare this value to the check sum of the reconstructed file, which is of course calculated in the same way that the script calculates the check sums ;). I know there could be hundreds of possible reconstruction scenarios for each file, especially if the file contained lots of 0D 0A binary strings, but with modern CPUs, I don't think it'll be such a big problem ;).
This sounds like a cool experiment really, might even work.
A Finnish Support Specialist
 Happily running multiple SMF 2.0 installations.

"Before you allow people access to your forum, especially in an administrative position, you must be aware that that person can seriously damage your forum.
 Therefore, you should only allow people that you trust, implicitly, to have such access." -Douglas

Offline GigaWatt

  • Jr. Member
  • **
  • Posts: 201
  • Gender: Male
    • Macedonian electronics forum
Re: CR + LF file corruption after FTP transfer
« Reply #21 on: March 17, 2018, 03:23:24 PM »
But do it the other way around, and Windows boxes will crash.

Yeah, true ;).

But, on the other hand, I don't think there are many Windows servers out there. Personally, I wouldn't run my forum on a Windows box :D.

In any case, you might be right... it might be safer to work in ASCII mode, but do it only if you're certain that the files you're transfering are text.

I'm not sure of this either way, but that's what I have heard. The whole protocol was originally designed for transferring text files.

Had no idea this is the case. File Transfer Protocol (at least in my head) associates with... well... just that, a protocol for transfering files, not a protocol specifically designed for transfering text files.... although the Wiki page mentions both ASCII and binary (image) mode so, I guess historically, both modes might have been a part of the original standard.

This sounds like a cool experiment really, might even work.

Yeah, I'm hoping it will ;).

I'll start digging in the script, see how the hash is calculated and will write back as soon as I've made some progress ;).
"This is really a generic concept about human thinking - when faced with large tasks we're naturally inclined to try to break them down into a bunch of smaller tasks that together make up the whole."

Online Kindred

  • The Mean One
  • Support Specialist
  • SMF Legend
  • *
  • Posts: 55,891
  • Gender: Male
    • Kindred-999 on GitHub
Re: CR + LF file corruption after FTP transfer
« Reply #22 on: March 17, 2018, 04:19:31 PM »
You'd be wrong.
  Most corporate businesses run on Windows and IIS.
Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

Offline GigaWatt

  • Jr. Member
  • **
  • Posts: 201
  • Gender: Male
    • Macedonian electronics forum
Re: CR + LF file corruption after FTP transfer
« Reply #23 on: March 17, 2018, 04:53:51 PM »
Had no idea. Good info though ;).
"This is really a generic concept about human thinking - when faced with large tasks we're naturally inclined to try to break them down into a bunch of smaller tasks that together make up the whole."

Offline Aleksi "Lex" Kilpinen

  • Support Specialist
  • SMF Super Hero
  • *
  • Posts: 16,542
  • Gender: Male
  • Don't worry, I'm n00b friendly
    • Aleksi.Kilpinen on Facebook
    • aleksi-kilpinen on LinkedIn
Re: CR + LF file corruption after FTP transfer
« Reply #24 on: April 06, 2018, 05:13:32 AM »
Have you had any progress with this? :)
A Finnish Support Specialist
 Happily running multiple SMF 2.0 installations.

"Before you allow people access to your forum, especially in an administrative position, you must be aware that that person can seriously damage your forum.
 Therefore, you should only allow people that you trust, implicitly, to have such access." -Douglas

Offline GigaWatt

  • Jr. Member
  • **
  • Posts: 201
  • Gender: Male
    • Macedonian electronics forum
Re: CR + LF file corruption after FTP transfer
« Reply #25 on: April 06, 2018, 10:17:55 AM »
Some. I'm working with a member on my forum on the issue. Apparently, SMF uses SHA1 and MD5 for it's hash values, but it also uses a time stamp and a random value to generate the hash. I (we) think this is the part that actually generates the hash (located in Subs.php).

Code: [Select]
// Just make up a nice hash...
if ($new)
    return sha1(md5($filename . time()) . mt_rand());

Sure, I could get the time stamp value (it's stored in the database), but what about the random value :S. I can't recreate it, thus, I can't get the MD5 and SHA1 signature, which of course means that the one the program will calculate is practically useless :S.

Is the mt_rand value stored somewhere in the database? From what we've analyzed, it's not, but just in case, I thought I'd ask.
"This is really a generic concept about human thinking - when faced with large tasks we're naturally inclined to try to break them down into a bunch of smaller tasks that together make up the whole."