mergeSMF.php - Script to merge two SMF forums

Started by bfeist, November 02, 2013, 03:39:27 PM

Previous topic - Next topic

bfeist

Hi all,
I know i'm a newbie and everything, but with Oldiesmann's permission, and with great pleasure, I present an updated version of a script that will merge two SMF forum databases into one.

Original version for SMF 1.0.x by Oldiesmann at SimpleMachines.org
Many thanks for throwing it together and giving us something to work with.
Modified version for SMF 1.1RC3 by Resourcez at resourcez.biz (way back in 2006)
Modified version for SMF 2.0.6 by bfeist (fall, 2013). Let's start this new version at 2.0.

Description:
This script will merge two SMF forums. It will takes the boards, topics, members, messages, etc from a SECONDARY forum and merge it into a PRIMARY forum. The PRIMARY forum has to be larger than the secondary (because topics will be numbered based on the max topic number in the PRIMARY forum).
You need to be comfortable with PHP, and wrapping your head around databases etc to use this script. It works, but it will take some trial and error to get everything right.

Known Problems:
This script merges topics by adding SECONDARY topics after the largest PRIMARY topicID. This results in all secondary messages appearing to be newer than any in the PRIMARY forum (in spite of the topic date). This is due to the way SMF sorts by default (to avoid DB performance hits) based on topicID rather than date. I believe this can be remedied by using a mod that allows a different sort method, but you take the performance hit.
It would be great if someone felt like writing a cool script that literally spliced the topicIDs of both PRIMARY and SECONDARY forums together based on topic date. This would result in a true merge of forums.

Instructions:
1. Download SMF and install it in two different places on your site, both installations within the same database (just use a different database prefix like smf2_ for the SECONDARY, smaller one). It's important that the SECONDARY board is smaller than the primary board.
2. Edit this file in a text editor and change line 25 below to match the database prefix of the PRIMARY smf board, your main, largest board (define('PRIMARY_DB_PREFIX', 'smf_') - change smf_ to whatever you used...).
4. Save this modified file, put it in the directory for the SECONDARY, smaller smf installation and run it (just like you would run install.php or the converter).
5. Follow the instructions, clicking the "Continue" link as needed (there are 10 steps to avoid it timing out).
6. Copy all files in your SECONDARY attachments/to_move_to_primary directory to the primary attachments directory.
7. Once you're done, if everything worked, login to your PRIMARY SMF board, go to Admin -> Forum Maintenance and click on "Recount all totals and statistics" - this will update everything for you.
8. Check for any members in the new (merged) PRIMARY SMF installation that start with "todelete_". These members need to be merged with the members of the same name using a tool like SMF Admin Toolbox (https://github.com/Spuds/SMF-Admin-Toolbox)

Revision history:
2.0 Nov 2/2013
Initial updated script for use with 2.0.6

2.0.1 Nov 2/2013
Added an initial step to confirm initiation of the script.

2.0.2 June 17/2014
Fixed an error that skipped step 8.

margarett

Se forem conduzir, não bebam. Se forem beber... CHAMEM-ME!!!! :D

QuoteOver 90% of all computer problems can be traced back to the interface between the keyboard and the chair

shadow82x

Colin B
Former Spammer, Customize, & Support Team Member

dannbass

How would you tell the databases that you're importing to with prefix go to? that's the part I can't wrap around my head. Let's say you created MERGE as the database and smf1_ and smf2_ as your prefixes... when you are importing the old databases, how would you tell each one to go where?

dannbass

I figured it out and it worked like a charm! Thank you so much!! the only problem is that the mod to merge users was removed :(

EDIT: This mod does the trick http://custom.simplemachines.org/mods/index.php?mod=2626

Herman's Mixen

Is there any permissions todo this from Oldiesmann and or ReSources please let them post here if its oke Team...
according to this topic... the file must be removed !!

http://www.simplemachines.org/community/index.php?topic=332894.msg3408891#msg3408891
Met vriendelijke groet, The Burglar!

 House Mixes | Mixcloud | Any Intelligent fool can make things bigger, more complex, and more violent.
It takes a touch of genius - and a lot of courage - to move in the opposite direction. - Albert Einstein

Former Godfather of our dutch community ;)

Oldiesmann

Yes, bfeist has my permission to post this, as per the first post in this topic. He contacted me to ask for my permission before starting this topic.
Michael Eshom
Christian Metal Fans

Herman's Mixen

Than that's cleared out ... also point the other topic to this one i guess ... goodluck all ;)
Met vriendelijke groet, The Burglar!

 House Mixes | Mixcloud | Any Intelligent fool can make things bigger, more complex, and more violent.
It takes a touch of genius - and a lot of courage - to move in the opposite direction. - Albert Einstein

Former Godfather of our dutch community ;)

bfeist

danbass, I heard back about what happened to the Admin Tool Box mod that lets you merge users. The author keeps it on Github now instead of the mod site. I'm not sure why. Here's the link:
https://github.com/Spuds/SMF-Admin-Toolbox

bfeist

Quote from: dannbass on November 03, 2013, 01:13:25 AM
How would you tell the databases that you're importing to with prefix go to? that's the part I can't wrap around my head. Let's say you created MERGE as the database and smf1_ and smf2_ as your prefixes... when you are importing the old databases, how would you tell each one to go where?

I know that this can be confusing. Can you think of a better way to describe the process?
Let me try:
You run this script from the smaller secondary SMF site. It modifies the secondary database so that it can be merged with the first, then it performs the merge by copying all modified secondary records to the primary site database. The resulting secondary database can be discarded.

dannbass

Quote from: bfeist on November 05, 2013, 09:53:32 PM
danbass, I heard back about what happened to the Admin Tool Box mod that lets you merge users. The author keeps it on Github now instead of the mod site. I'm not sure why. Here's the link:
https://github.com/Spuds/SMF-Admin-Toolbox

Good to know that! too bad I didn't know that before, I ended up just going into each message and changing the poster with the mod I found.

Quote from: bfeist on November 05, 2013, 09:56:22 PM
I know that this can be confusing. Can you think of a better way to describe the process?
Let me try:
You run this script from the smaller secondary SMF site. It modifies the secondary database so that it can be merged with the first, then it performs the merge by copying all modified secondary records to the primary site database. The resulting secondary database can be discarded.

Thanks, I didn't know that! I thought that you import the secondary database into the primary database, with a different prefix of course, and then run the script from the smaller SMF site. That is with both databases in the same database. I didn't know that you could let the script do that for you.

What I did was this, I changed the prefix of the secondary database on phpMyAdmin and then exported it. Then I imported that database into the primary one and then run the script from the secondary folder. That's what I ended up doing.

bfeist

Sorry, I've caused even more confusion. By database, I mean SMF table set. The script requires that the tables for both forums are within the same MySQL database with different prefixes (example: smf_ vs smfsecondary_).

Diego Andrés


SMF Tricks - Free & Premium Responsive Themes for SMF.

JTVaughn

Bit of a bump. But I couldn't see any newer topics about this script.

I'm intrigued by merging, but don't want to mess up my 2 forums. Would this work just as well if I downloaded both databases, uploaded to different hosting, then merged to test things out? I know I'd have to mess with the database paths, but is that the best way to test it? Or is there a local way of doing it?

bfeist

I recommend you do a dry run on separate hosting (or local hosting) anyway just to make sure you have the steps straight.

Then, take your live forums and place them both down for maintenance. Do all the steps again locally with the latest live data, then upload the resulting database to your live site.

If you're giving it a shot today. Let me know if you get stuck, I'm happy to help.

JTVaughn

Thanks. No, it wont be today, just something I'm investigating as a possibility for some time in the future. Don't know when I'll test, but I'm sure I'll post again if I need help.

JTVaughn

Actually, one possible hitch... it used to be 1 forum but I split it into 2, if I were to merge back into 1 would I have to manually  merge all the users back together 1 by 1? Or can you group merge them using that tool posted above? Would topics merge back together? (i.e. the longer threads would add the extra posts back onto the shorter threads)?

Some more detailed info about what your tool does/doesn't do would be really useful :)

bfeist

Quote from: JTVaughn on March 22, 2014, 12:12:59 PM
Actually, one possible hitch... it used to be 1 forum but I split it into 2, if I were to merge back into 1 would I have to manually  merge all the users back together 1 by 1? Or can you group merge them using that tool posted above? Would topics merge back together? (i.e. the longer threads would add the extra posts back onto the shorter threads)?

Some more detailed info about what your tool does/doesn't do would be really useful :)

If a user in the SECONDARY forum already exists in the PRIMARY forum,  the member is renamed todelete_[membername]. After the merge is complete and the forum is up and running, you use SMF Admin Toolbox (https://github.com/Spuds/SMF-Admin-Toolbox) to merge the "todelete" users together with their counterparts from the first forum.

Topics will not merge back together. Reread the "known issues" in the first post in this thread.


JTVaughn

Thanks, starting to get my head around it all now :)

JTVaughn

Sorry, I have another stupid question.

So, say you've already got 2 large forums installed with tonnes of posts/members. Forum1 is far larger than Forum2. Using your merge script, it takes the data in Forum2 and merges it with Forum1, so that when you visit Forum1 it's got all of the data from Forum2 also in it. Right?

What happens to Forum2? Does it stay in place same as before you ran the script? And Forum1 would be changed so much that it wouldn't be an easy process to reverse (without deleting the database and loading a backup from pre-merge), right?

If this is the case, is there any way to merge them into one forum but in a totally different location/database, so that the original Forum1 and Forum2 stay as they are? At least temporarily.

Advertisement: