Customizing SMF > SMF Coding Discussion

mergeSMF.php - Script to merge two SMF forums, v2.0.17

(1/2) > >>

shawnb61:
Here is a link to the latest version of the mergeSMF.php script: mergeSMF.php

History:
Original version for SMF 1.0.x by Oldiesmann at SimpleMachines.org
Modified version for SMF 1.1RC3 by Resourcez at resourcez.biz (way back in 2006)
Modified version for SMF 2.0.6 by bfeist (fall, 2013). Let's start this new version at 2.0.
Modified version for SMF 2.0.17 by shawnb61 (Sept, 2020).  Tested with SMF 2.0.17 & php 7.3.

Description:
 - This script will merge two SMF forums. It will take the boards, topics, members, messages, etc., from a SECONDARY forum and merge them into a PRIMARY forum.
 - You need to be comfortable with PHP, and wrapping your head around databases, etc., to use this script. It works, but it will take some trial and error to get everything right.
 - Given SMF's flexibility, there will be wrinkles... Anticipate issues. Backups, backups, backups...
 - We are all indebted to Oldiesmann for the deep institutional SMF knowledge baked into the script & approach.
 - This utility is not officially supported by the SMF team. I was asked to do a test run for my forum, so I brought this script current to do so, and am now sharing it here in case others find it useful.

Prerequisites:
1. BACKUP everything, file systems & databases, twice.  In addition, ask your host to do so.  If things go sideways, you must be able to restore everything.
2. Practice all the steps below in a test forum.  If you cannot do so, do not attempt this.
3. Primary & secondary forums must be in perfect working order.  E.g., all routine maintenance should be run, integrity checks run. A broken forum before merging = a horrifically mangled forum after merging...
4. Collations & charsets must be aligned across the forums & tables.
5. Forums must be offline and frozen.  A single update in the middle of execution can break everything...

Instructions:
1. Restore your two forums and get them working side-by-side on your site, both installations within the same database (just use a different database prefix like smf2_ for the SECONDARY one prior to the DB restore).
2. Edit mergeSMF.php in a text editor and change the prefix to match the database prefix of the PRIMARY smf board (define('PRIMARY_DB_PREFIX', 'smf_') - change smf_ to whatever you used...).
3. Also set $secondary_suffix to be the value appended to the end of member names & group names & other records in order to make them unique, in case of conflicts with the primary forum.
4. Save this modified file, put it in the directory for the SECONDARY smf installation and run it (just like you would run install.php or the converter).
5. Follow the instructions, clicking the "Continue" link as needed.
6. Pay careful attention to any errors reported.
7. When completed, copy all files in your SECONDARY attachments/to_move_to_primary directories to the primary attachments directories.
8. Copy all files in your SECONDARY custom_avatar/to_move_to_primary directory to the primary custom_avatar directory.  Be careful to confirm the name of the directory, it is different for different forums.
9. Copy all files in your SECONDARY avatars directory to the primary avatars directory.
10. Once you're done, if everything worked, login to your PRIMARY SMF board, go to Admin -> Forum Maintenance and click on "Recount all totals and statistics" - this will update everything for you.
11. Check for any members in the new (merged) PRIMARY SMF installation that end with your $secondary_suffix. If desired, these members can be merged with the members of the same name using a tool like SMF Admin Toolbox (https://github.com/Spuds/SMF-Admin-Toolbox)
12. Carefully audit all board permissions, membergroups, and subscriptions.
13. Clear your cache.

Approach:
 - This script basically makes the contents of the secondary forum unique, and then copies the contents over to the primary forum.  Default & duplicate content are removed from the secondary forum.  Remaining content is made unique by incrementing the keys safely beyond values used in the primary forum (e.g., board IDs), or by appending the $secondary_suffix to the value (e.g., member names).
 - Records merged: categories, boards, topics, members, messages, PMs, attachments, polls, poll_choices, log_actions, membergroups, permission_profiles, moderators, notifications, subscriptions, log_subscribed, log_reported, log_reported_comments, ban_groups and ban_items.
 - Records not merged: Anything not named above... Packages, themes, logs, calendars, search tables, custom_fields.  Any tables that are new in 2.1, e.g., support for likes.
 - Member names, real names & email addresses will be made unique by adding the $secondary_suffix.  Duplicate emails will have the suffix, thus will not work.
 - When this script is completed, the secondary forum is unusable.

Enhancements included with this version:
 - Many new record types added: Permissions, subscriptions, moderation activity, moderators, bans, & associated logs.  The idea was to bring member history along with each member.
 - Category & board hierarchy & order are now preserved. The secondary forum appears to be 'appended' intact to the end of the primary forum in the board index.
 - Multiple attachment directories supported.
 - Avatars & custom avatars are brought over.
 - Mismatches of columns between the primary & secondary forums are handled; no records dropped.
 - There is no notion of 'bigger' vs 'smaller' forums. I have merged a massive forum into a smaller one & vice-versa.
 - Now uses mysqli.
 - Improved error reporting; improved reporting of actions taken.
 - 2.1 and 2.0 support.  2.1 tested with 2.1 RC3.

Notes on Permissions:
The basic idea is to only bring over what is unique to the secondary forum:
 - Profiles: Only unique profiles will be brought over.  Default & standard profiles should be defined in the primary forum.
 - Membergroups: Only non-default, non-guest, membergroups will be brought over.  Default & guest behavior should be defined in the primary forum.
 - Post-count-based membergroups will not be brought over - what is in the primary forum should apply.
 - Assignments to post-count-based groups are not brought over.

Other Considerations:
 - If you had a custom search index, it will need to be rebuilt.
 - Links to other posts, including quote links, will be incorrect unless you use separate utilities to fix them.

Other Utilities that may be of help:
 - SMF_URLs_Paths.php - I used this to update all the links in the old secondary posts to properly point to the new forum.
 - smf_fix_log_actions.php - I used this to clean up log_actions before running the merge. Checks & fixes all string lengths. UTF8 conversions can break string lengths in serialized strings.
 - smf_mark_all_read_merge.php - I used this to mark all secondary boards read by active primary board users. In my instance, the secondary forum was a predecessor to the current forum, and the content was dated.
 - SMF_Topic_Link_Diag.php - Inspects quote links. Read only.
 - SMF_Topic_Link_Fixer_Merge.php - Post-merge quote link fixer. Fixes quote links for a specific range of posts, incrementing the message #s by a value you provide. Intended to fix the old secondary forum posts AFTER the merge.
 - SMF_Topic_Link_Fixer.php - Fixes quote links. Note if it can't find the message, it strips the link. Good to run BEFORE doing the merge, so links are solid before you begin. Do not run this immediately after a merge or it will break your links irreparably, as the message numbers would be incorrect. Run the merge version to increment the message numbers properly after the merge.

If you don't understand what these are doing, you probably shouldn't run them...

Revision history:
2.0 Nov 2/2013 - Initial updated script for use with 2.0.6
2.0.1 Nov 2/2013 - Added an initial step to confirm initiation of the script.
2.0.2 June 17/2014 - Fixed an error that skipped step 8.
2.0.17 October 2020 - New version, more record types.  2.1 support.

Edited 10/21/20: 2.1 support & multiple attachment folder support.
Edited 11/24/20: Added attachment with before & after screenshots.

Dzonny:
Great work man, well done!

Diego Andrés:
Really nice, tried one of the old ones a few years ago and was quite an interesting experiment.

Bigguy:
Very awesome. Thanks for all your work.

Doug Heffernan:
Nice work shawn.

Navigation

[0] Message Index

[#] Next page

Go to full version