Hi all,
I know i'm a newbie and everything, but with Oldiesmann's permission, and with great pleasure, I present an updated version of a script that will merge two SMF forum databases into one.
Original version for SMF 1.0.x by Oldiesmann at SimpleMachines.org
Many thanks for throwing it together and giving us something to work with.
Modified version for SMF 1.1RC3 by Resourcez at resourcez.biz (way back in 2006)
Modified version for SMF 2.0.6 by bfeist (fall, 2013). Let's start this new version at 2.0.
Description:
This script will merge two SMF forums. It will takes the boards, topics, members, messages, etc from a SECONDARY forum and merge it into a PRIMARY forum. The PRIMARY forum has to be larger than the secondary (because topics will be numbered based on the max topic number in the PRIMARY forum).
You need to be comfortable with PHP, and wrapping your head around databases etc to use this script. It works, but it will take some trial and error to get everything right.
Known Problems:
This script merges topics by adding SECONDARY topics after the largest PRIMARY topicID. This results in all secondary messages appearing to be newer than any in the PRIMARY forum (in spite of the topic date). This is due to the way SMF sorts by default (to avoid DB performance hits) based on topicID rather than date. I believe this can be remedied by using a mod that allows a different sort method, but you take the performance hit.
It would be great if someone felt like writing a cool script that literally spliced the topicIDs of both PRIMARY and SECONDARY forums together based on topic date. This would result in a true merge of forums.
Instructions:
1. Download SMF and install it in two different places on your site, both installations within the same database (just use a different database prefix like smf2_ for the SECONDARY, smaller one). It's important that the SECONDARY board is smaller than the primary board.
2. Edit this file in a text editor and change line 25 below to match the database prefix of the PRIMARY smf board, your main, largest board (define('PRIMARY_DB_PREFIX', 'smf_') - change smf_ to whatever you used...).
4. Save this modified file, put it in the directory for the SECONDARY, smaller smf installation and run it (just like you would run install.php or the converter).
5. Follow the instructions, clicking the "Continue" link as needed (there are 10 steps to avoid it timing out).
6. Copy all files in your SECONDARY attachments/to_move_to_primary directory to the primary attachments directory.
7. Once you're done, if everything worked, login to your PRIMARY SMF board, go to Admin -> Forum Maintenance and click on "Recount all totals and statistics" - this will update everything for you.
8. Check for any members in the new (merged) PRIMARY SMF installation that start with "todelete_". These members need to be merged with the members of the same name using a tool like SMF Admin Toolbox (https://github.com/Spuds/SMF-Admin-Toolbox)
Revision history:
2.0 Nov 2/2013
Initial updated script for use with 2.0.6
2.0.1 Nov 2/2013
Added an initial step to confirm initiation of the script.
2.0.2 June 17/2014
Fixed an error that skipped step 8.
Hi.
Nicely done. Kudos for the effort ;)
Agreed, great work. :) This is a common request.
How would you tell the databases that you're importing to with prefix go to? that's the part I can't wrap around my head. Let's say you created MERGE as the database and smf1_ and smf2_ as your prefixes... when you are importing the old databases, how would you tell each one to go where?
I figured it out and it worked like a charm! Thank you so much!! the only problem is that the mod to merge users was removed :(
EDIT: This mod does the trick http://custom.simplemachines.org/mods/index.php?mod=2626
Is there any permissions todo this from Oldiesmann and or ReSources please let them post here if its oke Team...
according to this topic... the file must be removed !!
http://www.simplemachines.org/community/index.php?topic=332894.msg3408891#msg3408891
Yes, bfeist has my permission to post this, as per the first post in this topic. He contacted me to ask for my permission before starting this topic.
Than that's cleared out ... also point the other topic to this one i guess ... goodluck all ;)
danbass, I heard back about what happened to the Admin Tool Box mod that lets you merge users. The author keeps it on Github now instead of the mod site. I'm not sure why. Here's the link:
https://github.com/Spuds/SMF-Admin-Toolbox
Quote from: dannbass on November 03, 2013, 01:13:25 AM
How would you tell the databases that you're importing to with prefix go to? that's the part I can't wrap around my head. Let's say you created MERGE as the database and smf1_ and smf2_ as your prefixes... when you are importing the old databases, how would you tell each one to go where?
I know that this can be confusing. Can you think of a better way to describe the process?
Let me try:
You run this script from the smaller secondary SMF site. It modifies the secondary database so that it can be merged with the first, then it performs the merge by copying all modified secondary records to the primary site database. The resulting secondary database can be discarded.
Quote from: bfeist on November 05, 2013, 09:53:32 PM
danbass, I heard back about what happened to the Admin Tool Box mod that lets you merge users. The author keeps it on Github now instead of the mod site. I'm not sure why. Here's the link:
https://github.com/Spuds/SMF-Admin-Toolbox
Good to know that! too bad I didn't know that before, I ended up just going into each message and changing the poster with the mod I found.
Quote from: bfeist on November 05, 2013, 09:56:22 PM
I know that this can be confusing. Can you think of a better way to describe the process?
Let me try:
You run this script from the smaller secondary SMF site. It modifies the secondary database so that it can be merged with the first, then it performs the merge by copying all modified secondary records to the primary site database. The resulting secondary database can be discarded.
Thanks, I didn't know that! I thought that you import the secondary database into the primary database, with a different prefix of course, and then run the script from the smaller SMF site. That is with both databases in the same database. I didn't know that you could let the script do that for you.
What I did was this, I changed the prefix of the secondary database on phpMyAdmin and then exported it. Then I imported that database into the primary one and then run the script from the secondary folder. That's what I ended up doing.
Sorry, I've caused even more confusion. By database, I mean SMF table set. The script requires that the tables for both forums are within the same MySQL database with different prefixes (example: smf_ vs smfsecondary_).
What about suscriptions?
Bit of a bump. But I couldn't see any newer topics about this script.
I'm intrigued by merging, but don't want to mess up my 2 forums. Would this work just as well if I downloaded both databases, uploaded to different hosting, then merged to test things out? I know I'd have to mess with the database paths, but is that the best way to test it? Or is there a local way of doing it?
I recommend you do a dry run on separate hosting (or local hosting) anyway just to make sure you have the steps straight.
Then, take your live forums and place them both down for maintenance. Do all the steps again locally with the latest live data, then upload the resulting database to your live site.
If you're giving it a shot today. Let me know if you get stuck, I'm happy to help.
Thanks. No, it wont be today, just something I'm investigating as a possibility for some time in the future. Don't know when I'll test, but I'm sure I'll post again if I need help.
Actually, one possible hitch... it used to be 1 forum but I split it into 2, if I were to merge back into 1 would I have to manually merge all the users back together 1 by 1? Or can you group merge them using that tool posted above? Would topics merge back together? (i.e. the longer threads would add the extra posts back onto the shorter threads)?
Some more detailed info about what your tool does/doesn't do would be really useful :)
Quote from: JTVaughn on March 22, 2014, 12:12:59 PM
Actually, one possible hitch... it used to be 1 forum but I split it into 2, if I were to merge back into 1 would I have to manually merge all the users back together 1 by 1? Or can you group merge them using that tool posted above? Would topics merge back together? (i.e. the longer threads would add the extra posts back onto the shorter threads)?
Some more detailed info about what your tool does/doesn't do would be really useful :)
If a user in the SECONDARY forum already exists in the PRIMARY forum, the member is renamed todelete_[membername]. After the merge is complete and the forum is up and running, you use SMF Admin Toolbox (https://github.com/Spuds/SMF-Admin-Toolbox) to merge the "todelete" users together with their counterparts from the first forum.
Topics will not merge back together. Reread the "known issues" in the first post in this thread.
Thanks, starting to get my head around it all now :)
Sorry, I have another stupid question.
So, say you've already got 2 large forums installed with tonnes of posts/members. Forum1 is far larger than Forum2. Using your merge script, it takes the data in Forum2 and merges it with Forum1, so that when you visit Forum1 it's got all of the data from Forum2 also in it. Right?
What happens to Forum2? Does it stay in place same as before you ran the script? And Forum1 would be changed so much that it wouldn't be an easy process to reverse (without deleting the database and loading a backup from pre-merge), right?
If this is the case, is there any way to merge them into one forum but in a totally different location/database, so that the original Forum1 and Forum2 stay as they are? At least temporarily.
QuoteWhat happens to Forum2? Does it stay in place same as before you ran the script?
I assume yes. Usually the source has data only read from.
QuoteIf this is the case, is there any way to merge them into one forum but in a totally different location/database, so that the original Forum1 and Forum2 stay as they are? At least temporarily.
Create a copy of forum 1 (target forum), duplicate database, duplicate files and run repair_settings in it. So you should have now 3 forums. The third forum will be your temporary merged forum and both other forums are untouched.
That's a great idea, thanks. So I could experiment on merging Forum2 with Forum3(copy of 1) without affecting the live version of Forum1. And if it works as planned, I can then delete Forum1 and move Forum3 over (or run the script again using Forum1).
Yup.
You should be careful, though, this is usually a very server-intensive operation. If you have a lot of users online, it is possible that some inconsistencies occur (eg: a user creates a post after the copy operation for members but before copy of posts). It is always advisable to choose a "dead period" of your forum, take it out for some minutes while the tool runs (maintenance mode = 2 in Settings.php) and take it out of maintenance mode after.
I highly recommend that you do this in a sandbox environment, perhaps even locally in a development environment.
Also, the way the script is written, both forums data has to be house in the same database with a different table prefix. So if your larger Forum1 starts with smf_, export it from your live database and import it to your sandbox environment. You then need to export Forum2's datadatabase to SQL, modify the table names to start with smf2_ for example, then import those tables into the same sandbox database. This will result in a database that contains two sets of SMF forum tables, one starting with smf_, the other with smf2_
Hope this helps.
Ben
When you say you have to merge the smaller one with the bigger one, is that by amount of posts or amount of topics? My "smaller" one has more total posts, but they're in less topics.
Hmmm, I'll have to take a close look at the code to answer you. I'll have a look in the morning.
Quote from: bfeist on April 06, 2014, 12:29:07 AM
Hmmm, I'll have to take a close look at the code to answer you. I'll have a look in the morning.
Hope you can find the answer :)
I just tried merging the smaller forum (more posts, less topics) with the larger one (less posts, more topics). The process seemed to work. I checked the tables for the newly merged larger board and I can see threads which only used to exist on the smaller board. But when I go to the larger board (after recounting the totals) none of the topics or boards are there. The categories moved over, but no boards or topics. I wonder if this means more posts = larger forum, so I actually did them the wrong way around? Could that be why the boards/topics are in the database but not visible?
I'll try and go back to the start and merge them in the opposite direction, see what happens
haha, so I tried the opposite way around ("larger" forum with more posts less topics, "smaller" forum with less posts more topics). All of the posts did indeed move from the smaller one to the larger one... problem is none of the boards did, and all of the posts have attached themselves to random topics all over the place. Also, all of the posts are attributed to the right username but none of those usernames are attached to accounts, so they all look like guests posts.
It's a great idea for a script, but I'm not sure it works in a live forum environment :)
The problem is that the original script does things in kind of a "lazy way" in that all it's really doing is looking for the maxvalue of the posts table on the larger forum, then adding that amount to the postIDs of the smaller forum and merging them in with all of the supporting stuff (like boards, users etc). This results in things like sorting being out of order without further mods.
The right way to do a merge script would be to do an actual line-by-line merge of every table, threading the values together by date. There would be no concept of "smaller" or "larger" forum, instead an actual splicing of all tables in each forum would occur, new postIDs, topicIDs, userIDs, etc would be generated for both forums. Coding-wise this is a complete rewrite.
Since I wrote this script for myself, I haven't seen many other people using it. Someone above mentioned that merging forums is a common request. If there's enough demand for it maybe I'll take a stab at the rewrite. But so far in the 8 months since I've written this version, only one or two people have given it a shot. Thoughts?
I don't have an answer for that. I'd guess that maybe there aren't many people that run two similar enough forums who'd want to merge them. Maybe there ARE a lot of people who'd like the ability to merge but are worried about all their data being corrupted. Who knows.
I'd happily donate a little money if it helped you do a complete re-write and make this awesome, but I dunno if I could give enough to make it worth your while.
My partner and i discovered the idea out and it worked well like a dream! Thanks a lot a great deal!! the one dilemma can be that the mod to help mix users was taken out
Great to hear you had success. Here's the mod to merge users:
https://github.com/Spuds/SMF-Admin-Toolbox
Just mentioning, you can always make a backup of forum files and database before merging and then restore it wherever you want, so you have working copy of forum before merging.
I did all my testing on duplicate copies of the forum directories/databases so I didn't harm the originals. Then I could just drop all the tables and re-import the databases each time and try again.
By the way, do you HAVE to use SMF 2.0.6+ for this to work? I just realised we're still on 2.0.4. Though I don't see why that would cause problems for something like this. Am I wrong?
(Yes, I know I should update soon)
You should be on 2.0.6 because of security fixes introduced meanwhile.
Other than that, it has to work fine, there were no database changes since 2.0 RC-something ;)
Is it possible to only merge the boards/topics/posts and nothing else? i.e. no users/attachments etc? 99% of my users are the same on both
And a question about the script: how does it work with assigning posts to users? Does it just say "post was made by member ID #3 on one board, so merge it over and assign to member ID #3 on the other board"? Is that why the posts were all funky and attributed to the wrong people when I tried it?
Quote from: JTVaughn on April 18, 2014, 01:30:52 PM
Is it possible to only merge the boards/topics/posts and nothing else? i.e. no users/attachments etc? 99% of my users are the same on both
And a question about the script: how does it work with assigning posts to users? Does it just say "post was made by member ID #3 on one board, so merge it over and assign to member ID #3 on the other board"? Is that why the posts were all funky and attributed to the wrong people when I tried it?
You can comment out different sections of the script if you only want to merge certain tables.
There's a good chance you have different memberIDs on your different boards. It's easiest to do it the way the script already does it:
- All duplicate users from secondary board are renamed as <username>_todelete during the merge.
- You then use a separate tool to merge users, thus removing the _todelete users: https://github.com/Spuds/SMF-Admin-Toolbox
There's a problem because of ID_TOPIC - those numbers are not changed, so messages are messed up - assigned to wrong topic at the end...
Quote from: ExSpirit on June 17, 2014, 06:52:06 AM
There's a problem because of ID_TOPIC - those numbers are not changed, so messages are messed up - assigned to wrong topic at the end...
This happens in step 8. The approach is to find the highest topic ID of the larger board, and add that number to all topics in the smaller board before merge. It's possible you have your boards mixed up and are trying to merge the larger into the smaller by accident.
I found the problem - step 7 contained wrong link (?step=9 instead of ?step=8). Fixed and it worked great now. :)
Quote from: ExSpirit on June 17, 2014, 04:51:43 PM
I found the problem - step 7 contained wrong link (?step=9 instead of ?step=8). Fixed and it worked great now. :)
Oops. Thanks for the catch. Just updated the original post to 2.0.2 to address that issue.
I used this script to merge two SMF 2.09 forums this past weekend, and wanted to say it worked beautifully, with just a few small hiccups that I thought I'd share in case someone else wants to use it.
Well, to be specific, I converted 14 phpBB boards to SMF 2.09 then merged all 14 of those forums into the one forum. The entire project took only six hours, thanks to this script.
Before you run this script:
1. You MUST backup or make a copy of your smaller database/forum (the one you are merging FROM). This script will rename all your duplicate member names in that smaller database BEFORE it imports them to the larger database. If for some reason you need to continue to use the smaller database/forum or revert to it, there's no real way to revert it without a gazillion manual database edits changing all those member's names from todelete_name back to name....
2. You MUST make a backup copy of your smaller forum's settings.php file. The script overwrites this file with the larger database's connection information and some other stuff, it's basically useless to continue running the smaller forum if you wanted to. Make a backup copy of this file before you use this script!
3. Check the number of columns on the smf_members table on both forums before you run the merge script. If the number of fields in the member's table (and field names) don't match, it won't actually import the members into the larger forum from the smaller forum, even though it looks like it did successfully import them all from the script's messages and HAS successfully imported all the messages and posts. For example, if your larger forum has some mods that have added additional custom fields that your smaller forum does not have in its members table, you will need to manually add those fields to the relevant table so that both member's tables have the same number of fields (and name them the same). Note the script does not give you any kind of warning that the fields don't exist and the import failed on the members table.
4. If the member groups on the smaller forum do not already exist on the larger forum, it's a very, very good idea to create those same membergroups on the larger forum before you import from the smaller one. If you don't, the imported members likely will be assigned an undesirable membergroup... :o
I also used the Admin Toolbox script linked elsewhere in this thread to merge my duplicate members. A few quick notes about it:
1. When using the Admin Toolbox to manually merge duplicate members, be aware that sometimes the avatar for that particular member account just disappears into the ether somewhere... about half of the todelete_name duplicate forum members lost their avatar and had to reset it.
2. If you get an error message saying that something doesn't exist or can't be found or something like that (sorry can't remember the exact error message) when merging two members, go to your admin =>maintenance => forum maintenance menu => routine and run the "clear file cache" option, then reattempt to merge . For some reason the Admin Toolbox mod doesn't seem to clear its own buffer/cache/whatever nicely all the time, and will occasionally act like some field or another that does exist, doesn't.
Sorry to necro this thread, but hopefully the above will help someone else with using this script. It's a wonderful script that saved me countless hours and greatly simplified this messy project, and for that you have my eternal thanks!
I really appreciate all the work put into this script! Thank you again!
GG
Thanks for your kind words and the great details you've provided into the issues you encountered.
Thanks for a good script.
p.s. Not work with child board
Mmm... I tried a 11 times, but don´t work... in localhost or server online... maybe the autor or any member developer can Updated??....... ;)
Please??... this is a realy dream for my... Please, Update the script... ;)
Thanks... and Regards
Waiting to the last version...(https://www.simplemachines.org/community/proxy.php?request=http%3A%2F%2Fmedia.simplemachinesweb.com%2Fsmf%2Fdefault%2Fimages%2Fpost%2Fthumbup.gif&hash=cdb957ce8273308383cd65570638a246a18adaaa)
Have a Great Time 8).-
You need to be more specific about the issues you encountered. Several other persons used this before you, there are no database changes in 2.0 since RC versions so it is supposed to still work...
Any time I search merging I wind up here, but if I'm understanding it correctly, I don't think it will work.
I took over the active userbase of a dying, abandoned PHPbb forum. It's been going about 4 years and is pretty large. I tried everything I could to get the data but crapped out and started from scratch. The old one was kind of a hot mess of categories and topics anyway about half of which didn't make any sense.
So now we have shiny, new "Community 2.0" with a completely different structure and all kinds of features and everyone loves it. After different theories on how to archive the PHPBB forum, I found I've actually had some success with the crawler converter. So now what I'd like to do is convert the new phpbb one to SMF and merge them, but due to the horrible structure that doesn't have anything to do with the new one, I just want to stick the whole thing under an "archived" category of the new board, and even start moving board contents en masse to their respective "new" locations.
BUT - the "crappy" forum is the larger one. Has the need to merge into the larger one been overcome?
Can I "pad" the smaller one with thousands of posts perhaps in a hidden area?
Any other thoughts on getting the big one to be just a stepchild of the small one?
Thanks for looking.
Hi all,
First thanks for all the hard work on this topic -- bfeist ! ;D
And second, where does this stand today?
I am a newb when it comes to scripting, but I am looking to merge two forums and well, here I am.
I followed this post/link by bfeist from a similar topic:
"Hi all,
I've posted a tool that will merge two SMF forums here. Have a look.
http://www.simplemachines.org/community/index.php?topic=513684.msg3630822"
And ended up here, but for the life of me I cannot find the link that takes me to the aforementioned tool.
I get that I am not that smart and I apologize for that if I am asking a dumb question. Can anyone provide an update?
Thankyou again in advance.
R
it is an attachment to the first post in this topic.
GAAAGAGAEREGEARGEADRGADFgadfaarrrrgghghhghgh!
I told you I was stupid.
I'm looking all over for a link. Never thought of looking for an "attachment." :-[
Thanks
r
Hello, all.
I came across this topic while looking for some guidance about merging two separate SMF boards with an overlapping membership. Sadly, I can't get the script attached to the OP to run.
Is there an updated version available somewhere? Both of my boards are using 2.0.15, with the latest patch installed.
Thank-you, in advance. :)
what errors are you getting?
^ and which version of PHP are you using?
I'm not getting any errors; it's just remaining a blank page.
PHP version is 7.0.30
Quote from: DragonLady on July 29, 2018, 03:04:12 PM
I'm not getting any errors;
Not even in your PHP error log file? I find that disturbing...
If your server is setup correctly, a blank white page will generate errors in your server side errorlog.
I suspect that the merge script does not support php7
As I need to merge these two live boards, I've turned off most of the logging and caching to make fewer tables to merge....
But there's nothing on the forum error log, and I don't have access to the server logs.
It would not generate errors in the forum Errorlog. Why do you not have access to the server? How do you load files, then? Look for a file named error.log or errorlog. Or soething similar.
If you don't have a server side error log, then you need to reconfigure your server to make one. But as I said, this script probably does not support php7.... and probably doesn't support mysqli either
QuoteIf you don't have a server side error log, then you need to reconfigure your server to make one
I'm on shared hosting, and my hosts have started charging extra for a lot of the stuff that's traditionally been included. As I intend to change hosting when this account runs out, I really don't want to have to pay for more features if I can help it.
When I try to run this script, all I get is a blank page.
I just figured there might be an updated version somewhere.
this is not an officially supported script... so, unless the original script was updated by the author, it is not likely that it was ever updted for the php7 changes.
As has been said before, a blank white page indicates a server side failure of php. It's a server side error, which means that it never makes it to the script and can not generate errors in the forum error log (actually, I doubt that this script would ever generate into the forum error log anyway)
The ONLY way you are going to actually find out what is happening is to look at the server side error log.
If your host has removed access to the error log, then you should flee that host ASAP since it means that they are complete and utter idiots.
if you have the two databases, then move to a new host and do the import/merge there
I have another hosting account elsewhere. I'll try installing a sandbox version of SMF, and see if this script works there. If so, I can just export the merged database & use it on the other account. But if not, I can get the errors from my cpanel there.
QuoteIf your host has removed access to the error log, then you should flee that host ASAP since it means that they are complete and utter idiots.
Yeah. They're the reason I have all these issues in the first place. :( :(
Okay; it's running in the sandbox on the other server just fine.
It will take a few days to move my databases over there and get 'em installed, but when I do I'll let you all know how this turns out.
I'm really, really hoping it works as well as the folks upstream have reported. :)
I take it this will not work for 2.1 at all running php 7.1 ???
Hmmm, maybe it will work.
An updated version of this script may be found here:
https://www.simplemachines.org/community/index.php?topic=575102.0