News:

Bored?  Looking to kill some time?  Want to chat with other SMF users?  Join us in IRC chat or Discord

Main Menu

Hard times updating 2.0.19->2.1.2

Started by sinnerman, May 23, 2022, 09:33:13 AM

Previous topic - Next topic

sinnerman

Hi everyone,

I'm trying to upgrade a "relatively large" (~2GB db) forum to 2.1:
  • It's been on 2.0.15 for quite some time and I got it up to 2.0.19 just for the sake of following the versions.
  • It's sitting on a dedicated server running CentOS 7.9, with Webmin/Virtualmin for a control panel. I have 3 php versions 5.4.16, 7.2.24, 7.4.28 installed. Up until now it was served using 5.4 as much work was needed to be able to switch to php7. Now I've updated or uninstalled all the plugins that couldn't run on php7 (like tinyportal 1.1) and I was able to have it in working condition on 2.0.19 with php 7.4
  • also Apache 2.4, mariadb 10  and using Cloudflare

OK so got it working in 2.0.19 with php7 and that is where I have a snapshot to be able to restart my test environment. Got the large update for 2.1. Started the update and then I was getting error messages every 30 seconds.
The error log was being populated with this error "AH01797: client denied by server configuration: " on every page call but it was working. just mentioning..
Also found this thread and many, many others but didn't help. At that point I'm about 10 hours in so I deployed a windows mouse clicker for the next 8 hours..talk about manual labor.

Anyways the update never completed. The "internal"/server clock stopped at 6 hours something.
Restarted the whole proccess removed some more plugins but it never completed again.
The error and access logs are no help. On the error log there is constantly the AH01797 message which also appears when it's working fine with 2.0.19. The access log is filled with messages like this one:
ip - - [21/May/2022:04:58:53 +0300] "GET /upgrade.php?step=3&substep=0&data=eyJjdXJzdGVwIjozLCJsYW5nIjoiZW5nzaCIsInJpZCI6MjE2NCwicGFzcyI6MzE5MDUsImRlYnVnIjp0cnVlLCJqcyI6MX0=&xml&filecount=2&substep=101&&_=16530757163 HTTP/1.1" 200 51 "http://url/upgrade.php?step=1&substep=0&data=eyJjdXJzdGVwIjoxLCJsYW5nIjoiZW5nbGlzaCIsInJpZCI6MjE2NCwicGFzcyI6MzE5MDUsImRlYnVnIjowLCJqcyI6MX0=" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.41 Safari/537.36"
1

Right now the main page shows maintanance. Thought I'd try through CLI this time without resetting the environment and had the following:

[root@s public_html]# /usr/bin/php74 upgrade.php
 * Updating Settings.php... Successful.
 *  +++  Updating old values done.
 +++  Changing default values done.
 Successful.
 *  +++  Removing all karma data, if selected done.
 Successful.
 *  +++  Emptying error log, if selected done.
 Successful.
 *  +++  Adding login history... done.
 +++  Copying the current package backup setting... done.
 +++  Copying the current "allow users to disable word censor" setting... done.
 +++  Converting collapsed categories... done.
 +++  Parsing board descriptions and names done.
 +++  Dropping "collapsed_categories" done.
 +++  Adding new "topic_move_any" setting done.
 +++  Adding new "enable_ajax_alerts" setting done.
 +++  Adding new "alerts_auto_purge" setting done.
 +++  Adding new "minimize_files" setting done.
 +++  Collapse object done.
 +++  Adding new "DEFAULTMaxListItems" setting done.
 +++  Adding new "loginHistoryDays" setting done.
 +++  Enable some settings we ripped from Theme settings done.
 +++  Adding new "httponlyCookies" setting done.
 +++  Adding new "samesiteCookies" setting done.
 +++  Calculate appropriate hash cost +++  Disable Moderation Center Security if                                                                                                                                                              it doesn't exist done.
 +++  Adding new profile data export settings done.
 +++  Adding settings for marking boards as read done.
 Successful.
 *  +++  Adding more space to the mime_type column. done.
 +++  Converting legacy attachments. done.
 +++  Note attachment conversion complete done.
 +++  Fixing invalid sizes on attachments done.
 +++  Fixing attachment directory setting... done.
 Successful.
 *  +++  Adding new columns to log_group_requests done.
 +++  Adjusting the indexes for log_group_requests done.
 Successful.
 *  +++  Adding support for <credits> tag in package manager done.
 +++  Adding support for package hashes done.
 +++  Adding support for validation servers done.
 +++  Add Package Validation to Downloads Site done.
 +++  Ensure The Simple Machines Customize Site is https done.
 +++  Add validation to Simple Machines Customize Site done.
 Successful.
 *  +++  Altering the session_id columns... done.
 Successful.
 *  +++  Adding new columns to topics .. done.
 Successful.
 *  +++  Adding a new column "callable" to scheduled_tasks table done.
 +++  Adding new scheduled tasks done.
 +++  Adding a new task-related setting... done.
 +++  Remove old tasks added by modifications... done.
 +++  Adding the new table done.
 Successful.
 *  +++  Adding new columns to boards... done.
 Successful.
 *  +++  Adding the boardindex_max_depth setting. done.
 Successful.
 *  +++  Removing manage_boards permission done.
 Successful.
 *  +++  Adding new columns to categories... done.
 Successful.
 *  +++  Adding the count to the members table... done.
 +++  Adding the new table for alerts. done.
 +++  Adding alert preferences. done.
 +++  Upgrading post notification settingsPHP Fatal error:  Uncaught Error: Call to undefined function allowedTo() in /pathto/public_html/Sources/Subs-Db-mysql.php:708
Stack trace:
#0 /pathto/public_html/Sources/Subs-Db-mysql.php(494): smf_db_error()
#1 /pathto/public_html/Sources/Subs-Db-mysql.php(802): smf_db_query()
#2 /pathto/public_html/upgrade.php(2184) : eval()'d code(43): smf_db_insert()
#3 /pathto/public_html/upgrade.php(2184): eval()
#4 /pathto/public_html/upgrade.php(1674): parse_sql()
#5 /pathto/public_html/upgrade.php(357): DatabaseChanges()
#6 {main}
  thrown in /pathto/public_html/Sources/Subs-Db-mysql.php on line 708

open to your input and/or guidance.

Thank you in advance!

edit1: I think I should restart from 2.0.19 and redo the upgrade.php with debug on straight from CLI. I'll do that and post an update until I think of something else. gonna check the mariadb logs too.

Doug Heffernan

Quote from: sinnerman on May 23, 2022, 09:33:13 AMThe error log was being populated with this error "AH01797: client denied by server configuration: " on every page call but it was working. just mentioning..

I have come across this before, on another script. It was caused by the .htaccess file.  The client had converted from Apache 2.2 to 2.4, and it allowed access to the script only from certain ip 's. The old 2.2 version syntax for protecting the page was causing the error.

Most likely you must update the contents of your .htaccess file to reflect the 2.4. version, to solve the aforementioned issue.

Quote from: sinnerman on May 23, 2022, 09:33:13 AMAnyways the update never completed. The "internal"/server clock stopped at 6 hours something.

I want to make sure that the edits mentioned in this post did not work as per your saying. Is that correct?

https://www.simplemachines.org/community/index.php?topic=582045.msg4122461#msg4122461

Regarding your database size, you can try to trim it by running the the Empty out unimportant logs option from the Maintenance before the upgrade.

sinnerman

Quote from: Doug Heffernan on May 23, 2022, 11:14:23 AMI have come across this before, on another script. It was caused by the .htaccess file.  The client had converted from Apache 2.2 to 2.4, and it allowed access to the script only from certain ip 's. The old 2.2 version syntax for protecting the page was causing the error.

Most likely you must update the contents of your .htaccess file to reflect the 2.4. version, to solve the aforementioned issue.

Thank you, I've found that too already, not worried about it right now.

As for you other question yes I did change the value to 1000 and made no difference.

I have tried quite a few more times, here is a quick log:
reset
update to 2.0.17
update to 2.0.18
uninstall bad behaviour
uninstall SMF4Mobile Mod
uninstall simple audio video embedder
uninstall smfpacks wysiwyg editor

--took stable backup at 2.0.18 running php 5.4

uninstall JDALIP - Join Date And Location In Posbit
uninstall Remove images from quotes
uninstall tiny portal 1.2
switch to php 7.4
install tiny portal 2.2.2
install SMF 2.0.19 Update

--upgrade to 2.1.2 -> failed again
RESET

uninstall JDALIP - Join Date And Location In Posbit
removed Forum Firewall manually - link

--took stable backup at 2.0.18 running php 5.4

uninstall tiny portal 1.2
uninstall tapatalk
switch to php 7.2
install tiny portal 2.2.2
install SMF 2.0.19 Update
uninstall @mention

--took stable backup at 2.0.19 running php 7.4
update to 2.1.2
same results

when I say same result I mean that I run
php74 upgrade.php --debug --language=english
the log goes as previous times and ends with the same error:

+++  Dropping old notification fields from the members table done.
 +++  Creating alert prefs for watched topicsPHP Fatal error:  Uncaught Error: Call to undefined function allowedTo() in /path/to/public_html/Sources/Subs-Db-mysql.php:708
Stack trace:
#0 /path/to/public_html/Sources/Subs-Db-mysql.php(494): smf_db_error()
#1 /path/to/public_html/Sources/Subs-Db-mysql.php(802): smf_db_query()
#2 /path/to/public_html/upgrade.php(2184) : eval()'d code(33): smf_db_insert()
#3 /path/to/public_html/upgrade.php(2184): eval()
#4 /path/to/public_html/upgrade.php(1674): parse_sql()
#5 /path/to/public_html/upgrade.php(357): DatabaseChanges()
#6 {main}
  thrown in /path/to/public_html/Sources/Subs-Db-mysql.php on line 708

Some guidance would be welcome as I'm running out of moves.

sinnerman

This allowedto() function is defined in security.php which is not required in subs-db-mysql file.
found this: https://www.google.com/search?q=allowedto()+smf+site:www.simplemachines.org&sxsrf=ALiCzsbC439gMPHNB7nxmvSgS5ExjC9Xrg:1653495681926&sa=X&ved=2ahUKEwiG14i2h_v3AhVYi_0HHUAYBs4QrQIoBHoECAkQBQ&biw=1536&bih=702&dpr=1.25
added a require once just to see what happens and I get a fatal error this time.

the definition of the function is indeed on sources/security.php

sinnerman

OK we got through that barrier.
for anyone interested the problem was with MariaDB config and more specifically with max_allowed_packet.
I had it at 1M and changed it to 1024M after seeing these in the error logs (that I had to enable too and resuffer everything):
2022-05-25 19:41:17 378797 [Warning] Aborted connection 378797 to db: 'ngstage' user: '' host: 'localhost' (Got a packet bigger than 'max_allowed_packet' bytes)

Doug Heffernan

Quote from: sinnerman on May 25, 2022, 12:48:39 PMOK we got through that barrier.
for anyone interested the problem was with MariaDB config and more specifically with max_allowed_packet.
I had it at 1M and changed it to 1024M after seeing these in the error logs (that I had to enable too and resuffer everything):
2022-05-25 19:41:17 378797 [Warning] Aborted connection 378797 to db: 'ngstage' user: '' host: 'localhost' (Got a packet bigger than 'max_allowed_packet' bytes)

I take it that the issue has been solved now?

sinnerman

that specific issue is indeed solved but I'm keeping my fingers crossed as to what comes next..I'll update accordinly.  :)

sinnerman

Update completed. What a mess it is now but it's working.

Doug Heffernan

Quote from: sinnerman on May 25, 2022, 02:47:37 PMUpdate completed.

Glad to hear that the upgrade is working.

Quote from: sinnerman on May 25, 2022, 02:47:37 PMWhat a mess it is now but it's working.

What is a mess now if I may ask?

sinnerman

Quote from: Doug Heffernan on May 25, 2022, 03:09:43 PMWhat is a mess now if I may ask?

Actually it didn't work fine and up to today I haven't had much time getting into it.
Now I can't post Greek nor on 2.0.19 nor on 2.1.2..greek characters showing up like this: �α
I have come to understand it's about character sets. The forum has English installed and Greek but only GReek in UTF-8.
Also I do declare the encoding in .htaccess ( AddDefaultCharset utf-8)
All Greek post are displaying fine and also the quick edit works. When I use the full editor there is a problem.
Tried installed a thrird party editor, the exact same problem.
I believe it's a PHP 7.4 conversion issue just haven't figured it out yet. Any ideas?
Should I erase the UTF-8 in smf_settings table from the db? link

Kindred

The database collation for tables AND columns has to ALSO be utf8

Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

sinnerman

Every table is already utf-8_general_ci and the database collation is utf8mb4_general_ci..
I also tried from admin maintanance to convert to utf-8..
also installed english-utf8 too..

Doug Heffernan

Quote from: sinnerman on May 27, 2022, 02:17:00 PMEvery table is already utf-8_general_ci and the database collation is utf8mb4_general_ci..

The columns should be utf-8 as well. Have a look at this link and double check that the utf-8 conversion has been done properly/thoroughly.

https://wiki.simplemachines.org/smf/UTF-8_Readme

The issue that you are having is utf-8 related imo.

sinnerman

#13
followed everything from that wiki and nothing helped.
I did delete all non utf8 english files, had the language section give me 500 error, cleaned cache and it worked. I mean it goes into the language menu again..now I see Greek and English both in UTF-8. Remember Greek was always in UTF-8 while English wasn't.
I tried the Convert the database and data to UTF-8 once more.
Checked the html output and also declares UTF-8 twice once in javascript as an smf var and once in html.
Checked all the tables and columns, everything is utf8_general_ci.
Every old post is fine and displays fine.

The problem is still that when using a full editor (not the quick reply) greek characters as transferred and stored as garbage character in the db. they also show up as garbage though phpmyadmin. After that they are "properly" displayed exactly the same way in the forum.
Quick edit works, old posts work, only new submits though the editor are somehow scrambled on the way to db. I smell a php-apache-mariadb issue..

also deleting global_character_set from smf_settings table didn't help either.

Doug Heffernan

Quote from: sinnerman on May 28, 2022, 11:33:11 AMfollowed everything from that wiki and nothing helped.
I did delete all non utf8 english files, had the language section give me 500 error, cleaned cache and it worked. I mean it goes into the language menu again..now I see Greek and English both in UTF-8. Remember Greek was always in UTF-8 while English wasn't.
I tried the Convert the database and data to UTF-8 once more.
Checked the html output and also declares UTF-8 twice once in javascript as an smf var and once in html.
Checked all the tables and columns, everything is utf8_general_ci.
Every old post is fine and displays fine.

The problem is still that when using a full editor (not the quick reply) greek characters as transferred and stored as garbage character in the db. they also show up as garbage though phpmyadmin. After that they are "properly" displayed exactly the same way in the forum.
Quick edit works, old posts work, only new submits though the editor are somehow scrambled on the way to db. I smell a php-apache-mariadb issue..

also deleting global_character_set from smf_settings table didn't help either.

Are you using any other mods that add to/modify the editor? That is the only thing that I can think of that can cause the issue that you are describing. That is, if the utf-8 conversion has been done properly, which it has as per your post.


Kindred

Is utf8 declared in your settings.php file?
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

sinnerman

Quote from: Kindred on May 28, 2022, 12:19:08 PMIs utf8 declared in your settings.php file?

yes it is
$db_character_set = 'utf8';

sinnerman

I'll throw a large 2.0.19 update at it and see what happens.

if nothing then i'll start again from 2.0.14 with php5.4 and see at which point the issue is created.

sinnerman

Exactly as it smelled, a PHP issue.
Solution found.  default php74 settings (at least on webmin/virtualmin):

[mbstring]
; language for internal character representation.
; This affects mb_send_mail() and mbstring.detect_order.
; http://php.net/mbstring.language
;mbstring.language = Japanese
mbstring.language = all

; Use of this INI entry is deprecated, use global internal_encoding instead.
; internal/script encoding.
; Some encoding cannot work as internal encoding. (e.g. SJIS, BIG5, ISO-2022-*)
; If empty, default_charset or internal_encoding or iconv.internal_encoding is used.
; The precedence is: default_charset < internal_encoding < iconv.internal_encoding
mbstring.internal_encoding = UTF-8
mbstring.http_input = auto
mbstring.http_output = UTF-8
mbstring.encoding_translation = On
mbstring.detect_order = UTF-8
mbstring.substitute_character = none;
mbstring.func_overload = 0
mbstring.strict_encoding = Off

change to:
[mbstring]
; language for internal character representation.
; This affects mb_send_mail() and mbstring.detect_order.
; http://php.net/mbstring.language
;mbstring.language = Japanese
mbstring.language = all

; Use of this INI entry is deprecated, use global internal_encoding instead.
; internal/script encoding.
; Some encoding cannot work as internal encoding. (e.g. SJIS, BIG5, ISO-2022-*)
; If empty, default_charset or internal_encoding or iconv.internal_encoding is used.
; The precedence is: default_charset < internal_encoding < iconv.internal_encoding
mbstring.internal_encoding = UTF-8
mbstring.http_input = auto
mbstring.http_output = auto
mbstring.encoding_translation = Off
mbstring.detect_order = UTF-8
mbstring.substitute_character = none;
mbstring.func_overload = 0
mbstring.strict_encoding = Off

Doug Heffernan

Quote from: sinnerman on May 28, 2022, 01:14:49 PMExactly as it smelled, a PHP issue.
Solution found.  default php74 settings (at least on webmin/virtualmin):

Glad to see that you got it solved and thank you for posting the solution. It will help other users too who might run into the same issue. Marking this as solved.

Advertisement: