Simple Machines Community Forum

SMF Support => SMF 2.1.x Support => Topic started by: chadon on December 18, 2020, 09:56:54 AM

Title: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 18, 2020, 09:56:54 AM
I tried to upgrade the Mustang forum in my signature for the last 36h with 3 attempts but never managed to get it done till the end.

It mostly stops every 30 seconds with a message saying the server is not responding and asking me to try again and it always starts again until it doesn't.

The best I did was to reach about 60 %, till the begining of step 5. That was yesterday and I waited about 12h for it to start again but it never did.

Last night, I started to upgrade again from the begining and right now I am at 55% general progress,  76% step progress with Exécuter : "Cleaning up after old UTF-8 languages" (57 de 69)  in step 4 with the server not responding error and keep clicking on try again but nothing happens since the last 30 mn.

It's a large forum with over a million posts, I already posted here asking help after I had big problems when I upgraded from SMF 2.0 to 2.1 RC1. I was afraid to upgrade again because of that until yesterday because I had enough of all the bugs and the nightmare starts again.

All files from the RC3 version have been uploaded in the server including the French language ones.

What should I do? Should I wait hours or days till the upgrade process starts again by itself and reaches 100%, modify something in SMF's files or could there be something wrong with my VPS server's configuration?
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 18, 2020, 12:17:52 PM
OK here are the news.

After 896 minutes, I reached the same point as yesterday. My forum is in French so I'm not sure if I will use the correct English terms.

General progess: 64%
Progress of this step: 7%

JSON conversion:

Done 1 out of 13 tables
Current table: "log actions"

It was stuck for about 10 hours yesterday on that table before I started all over again.

The log_actions table has a size of 35.6 MiB.

I don't have any error message or the link to try again so what can I do to get it done? Just wait?

Thanks.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: Sir Osis of Liver on December 18, 2020, 12:26:11 PM
You should have a choice of resuming or restarting upgrade.  2.0 upgrader can resume an incomplete upgrade and finish it successfully, I would think 2.1 can do it as well. 
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 18, 2020, 12:48:41 PM
The 2.1 & 2.0 upgrades are almost exactly the same.  2.1 has a utf8 conversion & a json conversion (additional steps) - but the rest is basically the same.

Those "30 seconds" error messages can mean one of two things:
- It's still executing, give it more time
- It died and is never coming back

Unfortunately, it is impossible to know which.  I generally wait at least 10 minutes.  Beyond 10 minutes, I'm fairly sure it died.  (If you know how to look at the processes on your computer, you can tell for certain if it is still working or dead...)

The last thing I'd try is to run the upgrader via the CLI.  The command line is much better at giving you helpful error info when there is a failure.

To do so, go to a command line prompt, navigate to the folder where your settings.php & upgrade.php files are, and issue the following command:
Code:
php upgrade.php

There are some options, which you can query using --help.  You'll see something like this:
Code:
$ php upgrade.php --help
SMF Command-line Upgrader
Usage: /path/to/php -f upgrade.php -- [OPTION]...

        --language=LANG         Reset the forum's language to LANG.
        --no-maintenance        Don't put the forum into maintenance mode.
        --debug                 Output debugging information.
        --backup                Create backups of tables with "backup_" prefix.


Note sometimes your host offers multiple versions of php, so you need to specify the right version when running the command.  You might need to ask your host what the proper path to use is.  If this is the case, your command line would end up looking something like this:
Code:
/opt/php56/bin/php upgrade.php

CLI should give you a much better error to work from.

At first guess, you may have some corrupted JSON strings in log_actions that is causing the upgrade to fail.  But we need to know for sure.  With a proper error message, we will know how to fix the problem.

Hope this helps,
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 18, 2020, 01:19:34 PM
Waitaminute...

If you were on RC1, why is it even attempting the JSON conversion again? 

Check out your forum...  You may be done...
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 18, 2020, 01:52:21 PM
Oh you are right. I disabled the maintenance mode with ftp and the forum is showing RC3 on the footer. The home page, section and topic pages all seem to work.

The only issue is that I get a database error message when I try to login as the administrator.

QuoteDatabase Error
Please try again. If you come back to this error screen, report the error to an administrator.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 18, 2020, 02:05:19 PM
The error log file is full of this error message:

[18-Dec-2020 20:01:29 Europe/Paris] PHP Warning:  md5_file(/335650_c81155677793117f7543bda495afb40733dae9c9): failed to open stream: No such file or directory in /home/mustvco/public_html/fr/Sources/ShowAttachments.php on line 156
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 18, 2020, 02:11:40 PM
Here is the result when we click on an attachment link:
https://www.mustangv8.com/fr/index.php?action=dlattach;topic=58146.0;attach=519789

Quote<br />
<b>Notice</b>:  Undefined offset: 27 in <b>/home/mustvco/public_html/fr/Sources/Subs.php</b> on line <b>4451</b><br />
<br />
<b>Warning</b>:  md5_file(/519789_6c35a3a0a0e4d47dc66f24fbe936e7b6692a02f5): failed to open stream: No such file or directory in <b>/home/mustvco/public_html/fr/Sources/ShowAttachments.php</b> on line <b>156</b><br />
<br />
<b>Notice</b>:  Undefined offset: 27 in <b>/home/mustvco/public_html/fr/Sources/Subs.php</b> on line <b>4451</b><br />
<br />
<b>Warning</b>:  md5_file(/519790_2dcb3c9da67ce19d5bfbd728d91d72d68001ef6e.dat): failed to open stream: No such file or directory in <b>/home/mustvco/public_html/fr/Sources/ShowAttachments.php</b> on line <b>182</b><br />
File not found.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 18, 2020, 08:16:31 PM
Two issues here...

First issue, attachments.  Are you using the RC3 version downloaded from this site?  Or...  Did you download files fresh from GitHub?

Second issue, SMF doesn't realize your JSON conversions had already taken place.  This appears to have been a bug in RC1.  To fix, using phpmyadmin or similar, add the following to your settings table:
   variable: json_done
   value: 1

Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 18, 2020, 08:36:58 PM
Regarding the attachment error, I think that somehow your attachment directory is incorrect.  Those are the errors you get if you are pointing to the wrong attachment directory.  Not sure how that would have happened... 

Two ways to fix...  Either take a peek at the directory under Admin | Forum | Attachments and Avatars | Attachment Settings and correct. 

OR... 

Download the latest repair_settings.php and use that to double-check all your directories.  Download from here:
     https://download.simplemachines.org/index.php?tools
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 18, 2020, 08:54:20 PM
"First issue"
I downloaded it from this site just before I did the upgrade.

"Second issue"
Thanks, I just did that and I will try to upgrade again.

About the attachment error , I can't login or access the admin area at the moment but I will see what happens if the upgrade goes all the way threw.

Thank you for your help Shawn.

Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 03:37:35 AM
I don't think you need to run the upgrader again.

Just fix the attachment directories.  If you cannot use the admin panel, then use repair_settings.php.

Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 19, 2020, 10:34:07 AM
I did run the upgrader again last night and it went all the way without a problem.

I also ran repair_settings.php, there was no value in the attachment directory field so I added it but it didn't make any difference.

I'm also still unable to login without getting a database error. I tried to repair the database but with no success.

If you have no suggestions, I think I will be good for a backup restoration and another upgrade.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 11:33:46 AM
If you do restore back to 2.1 RC1, make sure you set the value of json_done in the settings table before running the upgrader.  That is definitely part of the problem.

The attachment issue is caused by your attachment directory entry being wrong.  The current version of repair_settings.php (from the download link above) should help you fix it.  I would try that again.  (Don't forget to hit Save!)  Or you could edit it directly using a tool like phpmyadmin in the settings table.  The table is JSON encoded, but it's pretty easy to do if you have a single attachment folder.
  variable: attachmentUploadDir
  value: {"1":"foldername"}

We don't know which error is preventing you from logging on.  I would look in the error log for a database-related error. 
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 19, 2020, 12:34:24 PM
I used the repair_settings.php file from the download link and entered the right path of the attachment directory but it didn't fix the issue.

I checked the settings table and here is the content

{"1":"\/home\/mustvco\/public_html\/fr\/attachments","3":"\/home\/mustvco\/public_html\/fr\/attachments2"}

I'm not sure if this is right but I replaced the default attachments directory by an attachments2 one from the admin pannel of the forum early this year.

QuoteWe don't know which error is preventing you from logging on.  I would look in the error log for a database-related error.
If you are talking about the error log file located in the SMF directory, nothing is generated when I try to log in.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 02:26:10 PM
Quote from: ForumMustang.com on December 19, 2020, 12:34:24 PM
{"1":"\/home\/mustvco\/public_html\/fr\/attachments","3":"\/home\/mustvco\/public_html\/fr\/attachments2"}
Are you using one or two directories?  Which one is correct?

If you execute the following in the SQL tab of phpmyadmin, it will tell you which folder is being actually used:
select distinct id_folder FROM smf_attachments
The id_folder will correspond to the # you see in the attachmentUploadDir you see above.

This also helps you tie everything together...  The current folder in use is ID'd here:
SELECT *  FROM smf_settings WHERE variable LIKE '%attachmentupload%'

Quote from: ForumMustang.com on December 19, 2020, 12:34:24 PM
If you are talking about the error log file located in the SMF directory, nothing is generated when I try to log in.
No, I mean in the SMF log_errors table.  Since you cannot access the ACP, you may need to look at the entries via phpmyadmin.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 19, 2020, 04:24:10 PM
QuoteAre you using one or two directories?  Which one is correct?
The "attachments" directory is the one I used for years. It is still used by the forum to access the old attachments but new uploads since a year are sent to the "attachments2" directory.

QuoteIf you execute the following in the SQL tab of phpmyadmin, it will tell you which folder is being actually used:
It shows me a list of 26 rows.

Yes, I should have said that the attachments2 directory was set to have a new subdirectory each month. I just checked with ftp and this was set up like that since february 2019 so I must in fact be using 26 directories. I did that because the forum has XXX thousands of attachments and it was unmanageable to have them in one directory.
That might explain why repair_settings didn't fix the issue but if so, what should I enter?

QuoteThis also helps you tie everything together...  The current folder in use is ID'd here:
This one:
{"1":"\/home\/mustvco\/public_html\/fr\/attachments2"}

That's what I entered with the repair_setting file but the files are in subdirectories like for example:
attachments2/2019/01/file.jpg

QuoteNo, I mean in the SMF log_errors table.  Since you cannot access the ACP, you may need to look at the entries via phpmyadmin.
The table is empty:
QuoteMySQL returned an empty result set (i.e. zero rows). (Query took 0.0003 seconds.)

Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 05:31:46 PM
Do you have a way to get the contents of that field from BEFORE the upgrade?

Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 19, 2020, 05:37:19 PM
I have a backup on my computer but I don't know how to read it from there.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 05:42:58 PM
If the backup ends in .sql, you can just open it in Notepad++, and search for "attachments2".  I want to determine if it just has the 2 entries like we see above, or, if it has 26 entries.  I suspect the latter.  We may be able to just copy & paste the text from the old backup. 


Regarding the logon issue...  One thing to check...  If you look at the structure of the smf_members table, what is the length of the password_salt field?   You can use the Structure tab in phpmyadmin.

(It should look like the attached...)
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 19, 2020, 06:01:24 PM
It ends in sql.gz but it has a size of 355MB as it is so I doubt my computer will have enough memory to open it once unziped.

For the password_salt field I have varchar(5)
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 06:29:34 PM
Aha!  OK, so for the logon issue, you need to change the length of that field to 255.  You will then be able to logon.  (Note: A recent fix has gone in for this...)

Next to fix the attachment dirs...

Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 06:47:15 PM
To fix the attachment folders, I think the first thing to try is to actually re-enter them by hand, under Admin | Forum | Attachments & Avatars | Attachment Directories.

The order you add the attachment folders MUST BE the same order in which they were created/used.  So get a list of all 26 folders, newest to oldest, and enter them. 
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 19, 2020, 07:12:22 PM
Quote from: shawnb61 on December 19, 2020, 06:29:34 PM
Aha!  OK, so for the logon issue, you need to change the length of that field to 255.  You will then be able to logon.  (Note: A recent fix has gone in for this...)

Next to fix the attachment dirs...
Wow! You're the man Shawn, I managed to open the gate. Thank you very much.

Quote from: shawnb61 on December 19, 2020, 06:47:15 PM
To fix the attachment folders, I think the first thing to try is to actually re-enter them by hand, under Admin | Forum | Attachments & Avatars | Attachment Directories.

The order you add the attachment folders MUST BE the same order in which they were created/used.  So get a list of all 26 folders, newest to oldest, and enter them. 

What you sent me in PM didn't work but I'm starting to see the light.

I will see what I can do in the admin panel.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 07:19:32 PM
Sometimes the admin panel does funky things with the numbering...

Those numbers in the attachmentUploadDir entry must align with the numbers from the distinct query above...
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 19, 2020, 07:29:51 PM
In the admin section it says that some folders don't exist but it also shows they contain thousands of files.

On the 2 last folders, the november and december ones it says Files Missing (Repair)

I clicked on repair and here are the results:

61916 fichiers/avatars ont une entrée mais n'existent plus sur le disque --- they have an entry but don't exist on the disk
95 fichiers n'ont plus de message auquel ils sont associés --  don't have associated messages
7559 fichiers joints sont dans le mauvais dossier --- are in the wrong directory
Les fichiers 13 n'ont pas d'entrée correspondante dans la base de données. (Ceux-ci seront supprimés) -- don't have a matching entry in the database, they will be removed

Should I process? I m worried about the  61916 files
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 19, 2020, 07:31:30 PM
Quote from: shawnb61 on December 19, 2020, 07:19:32 PM
Sometimes the admin panel does funky things with the numbering...

Those numbers in the attachmentUploadDir entry must align with the numbers from the distinct query above...
I checked the directories with ftp and they are matching what you guessed so I don't understand.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 07:36:21 PM
Do NOT run the attachment repair tool until folders are properly entered.  It DELETES all the entries it doesn't think make sense.....   Making it IMPOSSIBLE to fix...
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 19, 2020, 07:41:09 PM
OK, I won't but I have found the problem. The months' directories are with one number they should all be with 2. It should be 2020/01 instead of 2020/1.

I'm changing them all and will let you know.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 07:42:02 PM
Fingers crossed...
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 19, 2020, 07:47:40 PM
I can't do it from the admin panel, I guess I have to do it from MySQL?

Oops, there seems to be a problem.
Unable to move or rename. Please verify that the path is correct or that this directory does not contain any sub-directories.
Unable to move. A directory already exists at that path.
....
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 07:49:35 PM
Just fix the string I sent.  Much easier...
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 19, 2020, 07:58:39 PM
That's what I did and it's fixed. Now I have 5 directories out of the 8 last months with files missing error messages. I started the repair tool and here are the results:

309 attachments/avatars have an entry but no longer exist on disk
95 attachments no longer have a message associated with them
37573 attachments are in the wrong directory
22 files do not have a corresponding entry in the database. (These will be deleted)
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 08:15:59 PM
What does it look like in the ACP when you browse Attachment Directories?  Do all 26 folders make sense?
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 08:17:35 PM
The 35k "wrong directory" ones likely still need to be fixed....
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 19, 2020, 08:47:33 PM
Quote from: shawnb61 on December 19, 2020, 08:15:59 PM
What does it look like in the ACP when you browse Attachment Directories?  Do all 26 folders make sense?
They all do except one base directory path showing:
/home/mustvco/public_html/fr/attachments2/2019/1

It should be showing:
/home/mustvco/public_html/fr/attachments2

But when I click on save, the first one keeps coming bath.

I tried to fix it in MySQL but I can't find anything about it.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 08:56:14 PM
The entries in the settings table are
  'attachment_basedirectories' - list of all basedirectories (I think you only have one)
  'basedirectory_for_attachments' - current basedirectory

Note the following query may help you tie things together, which attachments & folders belong to which entry:SELECT id_folder, min(id_attach) as minatt, max(id_attach) as maxatt FROM smf_attachments GROUP BY id_folder ORDER BY id_folder
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 08:59:19 PM
Yes, maintaining base directories in 2.1 is quirky (same with 2.0...).  You cannot change anything.  You can delete (by blanking it out & saving) and you can add.  You cannot modify. 

Written up here, as an FYI:
https://github.com/SimpleMachines/SMF2.1/issues/5977
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 19, 2020, 09:29:18 PM
Quote from: shawnb61 on December 19, 2020, 08:56:14 PM
The entries in the settings table are
  'attachment_basedirectories' - list of all basedirectories (I think you only have one)
  'basedirectory_for_attachments' - current basedirectory

Note the following query may help you tie things together, which attachments & folders belong to which entry:SELECT id_folder, min(id_attach) as minatt, max(id_attach) as maxatt FROM smf_attachments GROUP BY id_folder ORDER BY id_folder
Thanks, that worked but the attachment numers increased since:

Here are the results:

309 attachments/avatars have an entry but no longer exist on disk
95 attachments no longer have a message associated with them
37118 attachments are in the wrong directory
22 files do not have a corresponding entry in the database. (These will be deleted)
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 09:58:33 PM
Does the acp give you any hint which files are "in the wrong folders"?
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 19, 2020, 10:09:18 PM
No but there is another big issue. When I browse files and try to sort them by date our click on other pages, I get a white page with this content:

QuoteForbidden
You don't have permission to access this resource.

Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.

When I read my private messages, I see the last page but I get the same Forbiden error when I changes pages or when I want to send a private message.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 10:38:18 PM
Are there other changes underway?  Did your host make changes?  What happened in addition to an RC3 upgrade?

If you do a quick search, that is most often associated with a host turning on an overaggressive mod_security rule, e.g.:
https://www.simplemachines.org/community/index.php?topic=572656.0

To fix, you normally have to ask your host to disable that...


Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 19, 2020, 10:41:15 PM
Back to the attachments issue, I wrote up a little doc to help you audit your attachment folders. 

This should help you understand which id_folder should be associated with which directory on disk, and how that relates to the list of folders you see in the ACP. 

This should help you ID the 36K attachments in the "wrong folder". 

I would look at each physical folder on disk, find an attachment ID for a doc in that folder, and use that to tell me which id_folder that should be associated with in the ACP.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 19, 2020, 11:10:28 PM
Quote from: shawnb61 on December 19, 2020, 10:38:18 PM
Are there other changes underway?  Did your host make changes?  What happened in addition to an RC3 upgrade?

If you do a quick search, that is most often associated with a host turning on an overaggressive mod_security rule, e.g.:
https://www.simplemachines.org/community/index.php?topic=572656.0

To fix, you normally have to ask your host to disable that...
lol Do you have a solution for everything? I disabled mod_security and the problem is solved. But is it safe to run without it?

Quote from: shawnb61 on December 19, 2020, 10:41:15 PM
Back to the attachments issue, I wrote up a little doc to help you audit your attachment folders. 

This should help you understand which id_folder should be associated with which directory on disk, and how that relates to the list of folders you see in the ACP. 

This should help you ID the 36K attachments in the "wrong folder". 

I would look at each physical folder on disk, find an attachment ID for a doc in that folder, and use that to tell me which id_folder that should be associated with in the ACP.
That looks more complicated, I'm going to read that. Thank you.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 20, 2020, 12:28:57 AM
OK if I understood right, here are my discoveries.

I have the folder "attachments2/2020/05" with files missing in the ACP.
It's the 19th folder in the ACP.
In the smf_attachments table, the folder ID 19 has minatt: 491405 and maxatt: 496843
In the physical folder "attachments2/2020/05" via ftp, the file names start with: 491405_.... but ends with the file name: 500981_...

If I understand right, there shouldn't be any file name with numbers higher than 496843. How is that possible? It's not like files were moved or renamed during the upgrade?
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 20, 2020, 01:35:43 AM
I have no idea how or when that would have happened...

What I would do is see if moving one or two of those to the right folder fixes it.  Find what message 500981 is used in.  View the message.  Move that one file to the proper folder # (you can confirm the folder # on the attachment record).  View the message again...  If the attachment then appears properly, you're good.

If it works, move more, spot checking & confirming along the way.

I don't know of any process that moves the files or changes the folder ids.  Pretty sure the upgrader would not do that.   (Though I'm going to spend some time trying to duplicate this....)

You can navigate directly to a message easily by adding:
/index.php?msg=56765

...once you know the message #.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 20, 2020, 07:25:37 AM
QuoteI don't know of any process that moves the files or changes the folder ids.  Pretty sure the upgrader would not do that.
Wouldn't the upgrader mess up the smf_attachments table and change the value of maxatt?

If I replace the value of maxatt 496843 with 500981 in the table, wouldn't it work? It would at least be easier done than moving files and checking where they have to go.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 20, 2020, 07:45:49 AM
I have these reports that don't look good in the ACP Error log:

QuoteType d'erreur: Base de données (database)
Database Error: Unknown column 'filename' in 'field list'


SELECT code, filename, description
FROM smf_smileys https://www.mustangv8.com/fr/index.php?/trackback/
/home/mustvco/public_html/forum/Sources/Subs.php (Ligne 2489)

And

Quote8192: Function create_function() is deprecated https://www.mustangv8.com/fr/index.php?/trackback/
/home/mustvco/public_html/forum/Sources/Subs.php (Ligne 1305)   
Invité
Aujourd'hui à 11:58:00
209.90.225.218
5e71eecdd0efb53c28f8f482178c5dc7
Informations de traçage
Type d'erreur: Générale
8192: Function create_function() is deprecated https://www.mustangv8.com/fr/index.php?/trackback/
/home/mustvco/public_html/forum/Sources/Subs.php (Ligne 1225)   
Invité
Aujourd'hui à 11:58:00
209.90.225.218
5e71eecdd0efb53c28f8f482178c5dc7
Informations de traçage
Type d'erreur: Générale
8192: Function create_function() is deprecated https://www.mustangv8.com/fr/index.php?/trackback/
/home/mustvco/public_html/forum/Sources/Subs.php (Ligne 1214)
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 20, 2020, 08:26:44 AM
Here is the structure of the smf_attachments table in the attached image.

The id folders with missing images in the ACP panel are:

2
19
20
21
25
26

The id folder 1 is the base directory but what is the id folder 27? I only have 26 in the ACP.

The id folder 2 and 3 have 0 files when I view them with ftp.

It seems to me that the database should be cleaned instead of moving files, doesn't it?
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 20, 2020, 11:11:23 AM
Maxatt doesnt exist in the db, its calculated based on values found.

I do have a theory why/how the upgrader might do this.  Hint: it uses json_done to determine whether attachments have been renamed yet.  This might be a byproduct of that missing flag.

I see two broad paths for what to do next....    Restore & rerun the upgrader (fixing known issues first, eg, setting json_done & increasing the length of password_salt).  We would have fewer issues to clean up.

Or, try to fix these errors.  Regarding attachments, to align id_folder with the folder on disk, we either need to move the file or change id_folder.  Either would need to be done by hand.  If you have a gui like File Manager, moving the files would be trivial.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 20, 2020, 11:37:29 AM
If I restore and upgrade without the json step, are you sure it won't mess up the attachments again?

I don't know what is a gui or how to use one so I would have to do it by hand. I think I'd rather restore and upgrade but not if there is still a chance to mess up the attachments.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: Illori on December 20, 2020, 11:39:59 AM
GUI is a graphical user interface, rather then using a command line to do the task.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 20, 2020, 11:55:47 AM
I have a co-administrator who did an upgrade with the forum's database in local and he also had issues with MySQL but it went smoothly with MariaDB.
I have the option to do a MySQL/MariaDB Upgrade on my server, should I try that before to do a new upgrade? I never used MariaDB and don't know anything about it.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 20, 2020, 12:01:43 PM
Quote from: ForumMustang.com on December 20, 2020, 11:37:29 AM
If I restore and upgrade without the json step, are you sure it won't mess up the attachments again?

No.  There is a chance we still have the attachments issue.  We don't know root cause yet.  That will take time.

I suspect the create_function warnings are due to a mod, pretty sure there are none in 2.1.

If you wanted to try to update id_folder in the attachments table, these queries might help:

To update one:
update smf_attachments set id_folder = 11 where id_attach = 12

To update a range:
update smf_attachments set id_folder = 8 where id_attach >= 10 and id_attach <= 12

And you can confirm you did it correctly by rerunning:
SELECT id_folder, min(id_attach) as minatt, max(id_attach) as maxatt FROM smf_attachments GROUP BY id_folder ORDER BY id_folder


BACKUPS... BACKUPS... BACKUPS...  It's easy to type a typo...  Not all mistakes will be reversible.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 20, 2020, 12:03:08 PM
Quote from: ForumMustang.com on December 20, 2020, 11:55:47 AM
I have a co-administrator who did an upgrade with the forum's database in local and he also had issues with MySQL but it went smoothly with MariaDB.
I have the option to do a MySQL/MariaDB Upgrade on my server, should I try that before to do a new upgrade? I never used MariaDB and don't know anything about it.

No.  There are some known issues with MariaDB in SMF that do not exist with MySQL.

These are not MySQL issues.  We shouldn't add another layer of changes & potential problems. 
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 20, 2020, 12:04:54 PM
I would attempt to fix the id_folders.  Give that a few hours of effort.  If it fails, fall back to the restore/upgrade. 
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 20, 2020, 12:23:17 PM
QuoteI suspect the create_function warnings are due to a mod, pretty sure there are none in 2.1.
I had mods before the upgrade but none have been installed since.

Quote from: shawnb61 on December 20, 2020, 12:04:54 PM
I would attempt to fix the id_folders.  Give that a few hours of effort.  If it fails, fall back to the restore/upgrade. 
OK, I will try that, thanks.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 20, 2020, 01:56:32 PM
I just discovered that one of my attempts to upgrade created 8 backup tables in the database and one is the attachments one and the structure is how it should be. How should I do to restore it?
I tried te rename the    smf_attachments by smf_attachments2 and rename backup_smf_attachments by    smf_attachments but there is no change in the ACP.
I checked the content of smf_attachments and it's how it's supposed to be, I cleared the cache from the ACP
What am I missing?
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: albertlast on December 21, 2020, 02:50:37 AM
Quote from: ForumMustang.com on December 20, 2020, 07:45:49 AM
Quote8192: Function create_function() is deprecated https://www.mustangv8.com/fr/index.php?/trackback/
/home/mustvco/public_html/forum/Sources/Subs.php (Ligne 1305)   
Invité
Aujourd'hui à 11:58:00
209.90.225.218
5e71eecdd0efb53c28f8f482178c5dc7
Informations de traçage
Type d'erreur: Générale
8192: Function create_function() is deprecated https://www.mustangv8.com/fr/index.php?/trackback/
/home/mustvco/public_html/forum/Sources/Subs.php (Ligne 1225)   
Invité
Aujourd'hui à 11:58:00
209.90.225.218
5e71eecdd0efb53c28f8f482178c5dc7
Informations de traçage
Type d'erreur: Générale
8192: Function create_function() is deprecated https://www.mustangv8.com/fr/index.php?/trackback/
/home/mustvco/public_html/forum/Sources/Subs.php (Ligne 1214)

could you upload your Subs.php?
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 21, 2020, 10:21:53 AM
Oh! The problem is coming from the path but how is it possible?

The path in the error is showing:
/home/mustvco/public_html/forum/Sources/Subs.php
It should be :
/home/mustvco/public_html/fr/Sources/Subs.php

In the "forum" directory it's an old test forum running under SMF 2.0.15

I just tried to run repair_settings.php but everything is set with the "fr" directory".
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: albertlast on December 22, 2020, 03:49:37 AM
so everything is fixed?
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 22, 2020, 07:35:47 AM
No, I was hoping to get answers to my questions but I don't need them anymore. I discovered the damages were worse than what I thought so I'll restore the database and restart the upgrade all over today.
I was browsing the forum and noticed that non of the attachments posted since January 2020 could accessed from the topics. It affected all subdirectories in the 2020 directory, not only the ones with files missing.
I will let you know if I succeed... and if I fail.  :D
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 22, 2020, 12:12:49 PM
Dont forget to:
- add the json_done entry
- change the length of password_salt

BEFORE starting the upgrader.

If any issues whatsoever, consider using the CLI for better error reporting.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 22, 2020, 12:51:25 PM
Yes it's done.

The upgrader has been going for 4h but 3h where the wait to see what would happen if I didn't click on the link "try again". Nothing.  ;D

What is CLI? I'm no good with abbreviations.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: Illori on December 22, 2020, 01:02:47 PM
command line interface
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: albertlast on December 22, 2020, 04:42:10 PM
normaly some of the upgrade steps should be internaly skipped when you already on rc release,
so would be nice to know which step take so long in your setup.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 22, 2020, 05:33:56 PM
You can find some info on running via the CLI on the wiki, here:
https://wiki.simplemachines.org/smf/Upgrading

It's much better at reporting errors.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 22, 2020, 05:44:15 PM
From memory, the longest ones were the database steps, the cleaning up of the indexes and utf8.

Anyways, it went all the way after many hours and the forum is now open since about an hour and working very good. I keep my fingers crossed.
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: shawnb61 on December 22, 2020, 05:52:08 PM
Attachments look good?
Title: Re: Upgrade fail on large forum RC1 to RC3
Post by: chadon on December 22, 2020, 07:48:19 PM
Yes, it looks all good in the ACP and in all the topics I have seen.

Thank you very much Shawn for your help and your patience.  :)