mergeSMF.php - Script to merge two SMF forums

[margarett]

What happens to Forum2? Does it stay in place same as before you ran the script?

I assume yes. Usually the source has data only read from.

If this is the case, is there any way to merge them into one forum but in a totally different location/database, so that the original Forum1 and Forum2 stay as they are? At least temporarily.

Create a copy of forum 1 (target forum), duplicate database, duplicate files and run repair_settings in it. So you should have now 3 forums. The third forum will be your temporary merged forum and both other forums are untouched.

[JTVaughn]

That's a great idea, thanks. So I could experiment on merging Forum2 with Forum3(copy of 1) without affecting the live version of Forum1. And if it works as planned, I can then delete Forum1 and move Forum3 over (or run the script again using Forum1).

[margarett]

Yup.

You should be careful, though, this is usually a very server-intensive operation. If you have a lot of users online, it is possible that some inconsistencies occur (eg: a user creates a post after the copy operation for members but before copy of posts). It is always advisable to choose a "dead period" of your forum, take it out for some minutes while the tool runs (maintenance mode = 2 in Settings.php) and take it out of maintenance mode after.

[bfeist]

I highly recommend that you do this in a sandbox environment, perhaps even locally in a development environment.

Also, the way the script is written, both forums data has to be house in the same database with a different table prefix. So if your larger Forum1 starts with smf_, export it from your live database and import it to your sandbox environment. You then need to export Forum2's datadatabase to SQL, modify the table names to start with smf2_ for example, then import those tables into the same sandbox database. This will result in a database that contains two sets of SMF forum tables, one starting with smf_, the other with smf2_

Hope this helps.

Ben

[JTVaughn]

When you say you have to merge the smaller one with the bigger one, is that by amount of posts or amount of topics? My "smaller" one has more total posts, but they're in less topics.

[bfeist]

Hmmm, I'll have to take a close look at the code to answer you. I'll have a look in the morning.

[JTVaughn]

Hmmm, I'll have to take a close look at the code to answer you. I'll have a look in the morning.

Hope you can find the answer

I just tried merging the smaller forum (more posts, less topics) with the larger one (less posts, more topics). The process seemed to work. I checked the tables for the newly merged larger board and I can see threads which only used to exist on the smaller board. But when I go to the larger board (after recounting the totals) none of the topics or boards are there. The categories moved over, but no boards or topics. I wonder if this means more posts = larger forum, so I actually did them the wrong way around? Could that be why the boards/topics are in the database but not visible?

I'll try and go back to the start and merge them in the opposite direction, see what happens

[JTVaughn]

haha, so I tried the opposite way around ("larger" forum with more posts less topics, "smaller" forum with less posts more topics). All of the posts did indeed move from the smaller one to the larger one... problem is none of the boards did, and all of the posts have attached themselves to random topics all over the place. Also, all of the posts are attributed to the right username but none of those usernames are attached to accounts, so they all look like guests posts.

It's a great idea for a script, but I'm not sure it works in a live forum environment

[bfeist]

The problem is that the original script does things in kind of a "lazy way" in that all it's really doing is looking for the maxvalue of the posts table on the larger forum, then adding that amount to the postIDs of the smaller forum and merging them in with all of the supporting stuff (like boards, users etc). This results in things like sorting being out of order without further mods.

The right way to do a merge script would be to do an actual line-by-line merge of every table, threading the values together by date. There would be no concept of "smaller" or "larger" forum, instead an actual splicing of all tables in each forum would occur, new postIDs, topicIDs, userIDs, etc would be generated for both forums. Coding-wise this is a complete rewrite.

Since I wrote this script for myself, I haven't seen many other people using it. Someone above mentioned that merging forums is a common request. If there's enough demand for it maybe I'll take a stab at the rewrite. But so far in the 8 months since I've written this version, only one or two people have given it a shot. Thoughts?

[JTVaughn]

I don't have an answer for that. I'd guess that maybe there aren't many people that run two similar enough forums who'd want to merge them. Maybe there ARE a lot of people who'd like the ability to merge but are worried about all their data being corrupted. Who knows.

I'd happily donate a little money if it helped you do a complete re-write and make this awesome, but I dunno if I could give enough to make it worth your while.

[trot12345]

My partner and i discovered the idea out and it worked well like a dream! Thanks a lot a great deal!! the one dilemma can be that the mod to help mix users was taken out

[bfeist]

Great to hear you had success. Here's the mod to merge users:
https://github.com/Spuds/SMF-Admin-Toolbox

[Dzonny]

Just mentioning, you can always make a backup of forum files and database before merging and then restore it wherever you want, so you have working copy of forum before merging.

[JTVaughn]

I did all my testing on duplicate copies of the forum directories/databases so I didn't harm the originals. Then I could just drop all the tables and re-import the databases each time and try again.

[JTVaughn]

By the way, do you HAVE to use SMF 2.0.6+ for this to work? I just realised we're still on 2.0.4. Though I don't see why that would cause problems for something like this. Am I wrong?

(Yes, I know I should update soon)

[margarett]

You should be on 2.0.6 because of security fixes introduced meanwhile.

Other than that, it has to work fine, there were no database changes since 2.0 RC-something

[JTVaughn]

Is it possible to only merge the boards/topics/posts and nothing else? i.e. no users/attachments etc? 99% of my users are the same on both

And a question about the script: how does it work with assigning posts to users? Does it just say "post was made by member ID #3 on one board, so merge it over and assign to member ID #3 on the other board"? Is that why the posts were all funky and attributed to the wrong people when I tried it?

[bfeist]

Is it possible to only merge the boards/topics/posts and nothing else? i.e. no users/attachments etc? 99% of my users are the same on both

And a question about the script: how does it work with assigning posts to users? Does it just say "post was made by member ID #3 on one board, so merge it over and assign to member ID #3 on the other board"? Is that why the posts were all funky and attributed to the wrong people when I tried it?

You can comment out different sections of the script if you only want to merge certain tables.

There's a good chance you have different memberIDs on your different boards. It's easiest to do it the way the script already does it:
- All duplicate users from secondary board are renamed as <username>_todelete during the merge.
- You then use a separate tool to merge users, thus removing the _todelete users: https://github.com/Spuds/SMF-Admin-Toolbox

[ExSpirit]

There's a problem because of ID_TOPIC - those numbers are not changed, so messages are messed up - assigned to wrong topic at the end...

[bfeist]

There's a problem because of ID_TOPIC - those numbers are not changed, so messages are messed up - assigned to wrong topic at the end...

This happens in step 8. The approach is to find the highest topic ID of the larger board, and add that number to all topics in the smaller board before merge. It's possible you have your boards mixed up and are trying to merge the larger into the smaller by accident.