Advertisement:

Author Topic: Converting big forum from Yabb to SMF  (Read 53418 times)

Offline SleePy

  • Site Team Lead
  • SMF Master
  • *
  • Posts: 30,341
  • Gender: Male
  • Thats his happy face.
    • jdarwood007 on GitHub
    • @jdarwood on Twitter
    • SleePy Code - My personal site
Re: Converting big forum from Yabb to SMF
« Reply #20 on: July 31, 2008, 09:38:46 PM »
Give this one a try.

I believe I fixed some issues in it. The good and bad karma should of be handled it looks like.
Jeremy D — Site Team / SMF Developer
Support the SMF Support team!
Profiles:
GitHub

Offline evgenydeep

  • Semi-Newbie
  • *
  • Posts: 99
Re: Converting big forum from Yabb to SMF
« Reply #21 on: August 03, 2008, 02:58:05 PM »
SleePy, thanks, I saw you added block_size. Something other?
I have tried your version, but it strangely not ALTER tables sometimes, which produce errors on next steps. I'm not trying to investigate why and where... I simple return to old one version, which works almost well. Almost becasue i still not converted my forum :) But now I'm serious, and i want to fight and finish it finaly, becasue if it will be Yabb forum, my server will not survive on autmn, when many users will come to my forum...


I have other issue.  I looked on the convertion process and realize that on step 8(function convertStep8), it become unexpected slowly on the query like
UPDATE `forum_smf`.smf_log_topics\n                    SET ID_TOPIC = 38\n                    WHERE tempID = 1111015939
I have almost 4mln rows in smf_log_topics and around 30 000 topics in smf_topics.
So, to really speed up the process, is only needed to add index to the tempID column, and then drop it.
Otherwise this 8 step can lasts months :)

How do you think?

Offline evgenydeep

  • Semi-Newbie
  • *
  • Posts: 99
Re: Converting big forum from Yabb to SMF
« Reply #22 on: August 03, 2008, 05:12:44 PM »
Other one issue/bug:
On STEP 4 find next code:

      if ($_GET['substep'] >= -4)
      {
         $knownColumns = array(
            'temp_toName' => 'DROP temp_toName',
         );
         alterTable('personal_messages', '', $knownColumns, false, true);

         pastTime(-5);
      }

I suppose it should be 'DROP COLUMN temp_toName' instead 'DROP temp_toName'
?

Offline evgenydeep

  • Semi-Newbie
  • *
  • Posts: 99
Re: Converting big forum from Yabb to SMF
« Reply #23 on: August 03, 2008, 05:32:16 PM »
3) On step 8(function convertStep8), during query
         $knownKeys = array(
            'PRIMARY' => 'ADD PRIMARY KEY (ID_TOPIC, ID_MEMBER)',
         );
         $knownColumns = array(
            'tempID' => 'DROP COLUMN tempID',
         );
         alterTable('log_topics', $knownKeys, $knownColumns, '', true, true);

I got following error:


<br />
<b>Notice</b>:  Undefined variable: endl in <b>/home/forumgr/public_html/smf102/yabb21_to_smf.php</b> on line <b>800</b><br />
Duplicate entry '20706-8050' for key 1
                ALTER TABLE `forumgr_smf102`.smf_log_topics
                ADD PRIMARY KEY (ID_TOPIC, ID_MEMBER),
                DROP COLUMN tempIDhosting#


Why to not check by aditional one sql-query - is there duplicates or not?
Can You tell me correct SQL query(-ies), 1) to check for duplicates and then 2) delete duplicates, but let one recored will remain only one from duplicates.
?
Thanks.

Offline SleePy

  • Site Team Lead
  • SMF Master
  • *
  • Posts: 30,341
  • Gender: Male
  • Thats his happy face.
    • jdarwood007 on GitHub
    • @jdarwood on Twitter
    • SleePy Code - My personal site
Re: Converting big forum from Yabb to SMF
« Reply #24 on: August 03, 2008, 11:05:52 PM »
Well there should hopefully not be duplicate topics :|
It does happen though, the flat file converters dont make it very fun because they remove key during conversion that is supposed to help against this.
This may work to show all duplicate topics in the log_topics table. Replace smf_ with your prefix

Code: [Select]
SELECT ID_TOPIC
FROM smf_log_topics
WHERE ID_TOPIC != 0
GROUP BY ID_TOPIC, ID_MEMBER
HAVING ID_TOPIC > 1

But it should show the id of a topic that has more than 1 ID_TOPIC and ID_MEBER (both are the same).
The next part would be to find and delete those.
Jeremy D — Site Team / SMF Developer
Support the SMF Support team!
Profiles:
GitHub

Offline evgenydeep

  • Semi-Newbie
  • *
  • Posts: 99
Re: Converting big forum from Yabb to SMF
« Reply #25 on: August 04, 2008, 02:40:17 AM »
Moreover, I have the same bug with poll_choices:
<b>Notice</b>:  Undefined variable: command_line in <b>/home/forumgr/public_html/smf102/convert.php</b> on line <b>1148</b><br />
Duplicate entry '275-0' for key 1
                ALTER TABLE `forumgr_smf102`.smf_poll_choices
                ADD PRIMARY KEY (ID_POLL, ID_CHOICE)hosting#

And moreover, I saw such bugs have not me only. Try to search string 'Duplicate entry' on SMF community forum and you will notice that many users have the same bug. So, it would be good before adding any PRIMARY KEY(UNIQUE KEY) check - is there duplicates or not, and if there - then delete them.

BTW:  your sql is not working, I used

SELECT ID_TOPIC, ID_MEMBER, COUNT(*)
FROM smf_log_topics
GROUP BY ID_TOPIC, ID_MEMBER
HAVING COUNT(*) > 1

but on 2mln rows table without index it working VEEEEEEEERY :-) slow

and for polls:
SELECT ID_POLL, ID_CHOICE, COUNT(*)
FROM forumgr_smf102.smf_poll_choices
GROUP BY ID_POLL, ID_CHOICE
HAVING COUNT(*) > 1

Offline evgenydeep

  • Semi-Newbie
  • *
  • Posts: 99
Re: Converting big forum from Yabb to SMF
« Reply #26 on: August 04, 2008, 03:53:21 AM »
18 step

      if ($_GET['substep'] >= -2)
      {
         $knownKeys = array(
            'poll' => 'ADD UNIQUE INDEX poll (ID_POLL, ID_TOPIC)',
         );
         alterTable('messages', $knownKeys, '', true);

         pastTime(-3);
      }

Is it error? Is it should be messages table? But in messages table there is no field ID_POLL

Offline evgenydeep

  • Semi-Newbie
  • *
  • Posts: 99
Re: Converting big forum from Yabb to SMF
« Reply #27 on: August 04, 2008, 03:59:16 AM »
it looks that on 18 step should be not messages, should be topics, becasue only in topics exist ID_POLL, ID_TOPIC
Yeah... and on clean install poll index exist in topics only.

Guys... :-) How many users already converted Yabb to SMF? :)

Offline evgenydeep

  • Semi-Newbie
  • *
  • Posts: 99
Re: Converting big forum from Yabb to SMF
« Reply #28 on: August 04, 2008, 04:10:29 AM »
Another thing: during Cleaning up steps(14 and 15), no indexes creating. May be problems with the substeps again?

Offline evgenydeep

  • Semi-Newbie
  • *
  • Posts: 99
Re: Converting big forum from Yabb to SMF
« Reply #29 on: August 04, 2008, 04:19:10 AM »
I decided to remove duplicates by following SQL

CREATE TABLE forumgr_smf102.smf_poll_choices_temp AS SELECT * FROM forumgr_smf102.smf_poll_choices WHERE 1 GROUP BY ID_POLL, ID_CHOICE;
DROP TABLE forumgr_smf102.smf_poll_choices;
RENAME TABLE forumgr_smf102.smf_poll_choices_temp TO forumgr_smf102.smf_poll_choices;

and Indexes should be added manually also. It will be good to add this into converting script. At fiest - check, is there duplicates or not, and then if yes - remove them.


Offline evgenydeep

  • Semi-Newbie
  • *
  • Posts: 99
Re: Converting big forum from Yabb to SMF
« Reply #30 on: August 04, 2008, 04:27:49 AM »
On step 8

         $knownKeys = array(
            'PRIMARY' => 'ADD PRIMARY KEY (ID_TOPIC, ID_MEMBER)',
         );
         $knownColumns = array(
            'tempID' => 'DROP COLUMN tempID',
         );
         alterTable('log_topics', $knownKeys, $knownColumns, '', true, true);


Here ADD PRIMARY KEY (ID_TOPIC, ID_MEMBER) , but I look, on clean installed SMF
PRIMARY KEY (ID_MEMBER, ID_TOPIC)

Is it important?

Offline SleePy

  • Site Team Lead
  • SMF Master
  • *
  • Posts: 30,341
  • Gender: Male
  • Thats his happy face.
    • jdarwood007 on GitHub
    • @jdarwood on Twitter
    • SleePy Code - My personal site
Re: Converting big forum from Yabb to SMF
« Reply #31 on: August 04, 2008, 12:57:18 PM »
evgenydeep,

Many users have converted to SMF from Yabb :P
Just not all of them have been big boards where these minor issues come poping out.

Thanks for the info on the query for finding duplicates. Did that table have a index for                  ID_TOPIC itself? If this is missing it would explain why it took a while. Its not a unique index or anything so adding it shouldn't cause issues but would speed up that query. As well a limiter on it would help.

As for step 18, are you sure you where using the .php file I gave you?
Lots of those steps got changed because of the inequality worked better the opposite way preventing the columns from being dropped to early.

Edit,
Attaching the working copy I have from SVN.
If you find bugs let me know, The hardest part is getting these flat file converters to work right especially from big boards where we don't go through 10 steps in a second :P
« Last Edit: August 04, 2008, 12:59:01 PM by SleePy »
Jeremy D — Site Team / SMF Developer
Support the SMF Support team!
Profiles:
GitHub

Offline evgenydeep

  • Semi-Newbie
  • *
  • Posts: 99
Re: Converting big forum from Yabb to SMF
« Reply #32 on: August 04, 2008, 02:47:41 PM »
Thanks for the info on the query for finding duplicates. Did that table have a index for                  ID_TOPIC itself? If this is missing it would explain why it took a while. Its not a unique index or anything so adding it shouldn't cause issues but would speed up that query. As well a limiter on it would help.
About what table you discuss? smf_log_topics? Sorry, not understand.

Quote
As for step 18, are you sure you where using the .php file I gave you?
Lots of those steps got changed because of the inequality worked better the opposite way preventing the columns from being dropped to early.
Yes, i look into your file also, here is the same bug :-)
On 18 step should be not messages, should be topics, becasue only in topics exist ID_POLL, ID_TOPIC

Offline SleePy

  • Site Team Lead
  • SMF Master
  • *
  • Posts: 30,341
  • Gender: Male
  • Thats his happy face.
    • jdarwood007 on GitHub
    • @jdarwood on Twitter
    • SleePy Code - My personal site
Re: Converting big forum from Yabb to SMF
« Reply #33 on: August 04, 2008, 05:58:18 PM »
yes sorry, I was asking about the smf_log_topics. It should have a primary index and another index of id_topic.

What line you seeing that on?
In fact in that file I gave you, its slightly different.
This is what I am seeing:
Code: [Select]
        if ($_GET['substep'] <= -3)
        {
            $knownKeys = array(
                'poll' => 'ADD UNIQUE INDEX poll (ID_POLL, ID_TOPIC)',
            );
            alterTable('messages', $knownKeys, '', true);

            pastTime(-4);
        }

If you look it is now substep -3 in step 18. The inequality is going the other way as well (This seemed to fix issues with changes to the table being done before the conversion was complete).
Jeremy D — Site Team / SMF Developer
Support the SMF Support team!
Profiles:
GitHub

Offline evgenydeep

  • Semi-Newbie
  • *
  • Posts: 99
Re: Converting big forum from Yabb to SMF
« Reply #34 on: August 05, 2008, 03:38:21 AM »
Should be as quoted.  Sholuld be not messages table, must be topics table. Becasue only in topics table exist fields ID_POLL, ID_TOPIC.
Code: [Select]

$knownKeys = array(
'poll' => 'ADD UNIQUE INDEX poll (ID_POLL, ID_TOPIC)',
);
alterTable('topics', $knownKeys, '', true);


Quote
If you look it is now substep -3 in step 18. The inequality is going the other way as well (This seemed to fix issues with changes to the table being done before the conversion was complete).
Aha, thanks, I see. I'll look on it.

Offline evgenydeep

  • Semi-Newbie
  • *
  • Posts: 99
Re: Converting big forum from Yabb to SMF
« Reply #35 on: August 05, 2008, 03:44:04 AM »
One another bug. Serious bug.
In "Converting membergroups" step function convertStep1().
I notice that NoPost Groups has not been converted.

How it now:
Code: [Select]
elseif (preg_match('~\$NoPost\[(\d+)\] = [\'|"]([^|]*)\|(\d*)\|([^|]*)\|([^|]*)~', $group, $match) != 0)
How it should be:
Code: [Select]
elseif (preg_match('~\$NoPost\[(\d+)\} = [\'|"]([^|]*)\|(\d*)\|([^|]*)\|([^|]*)~', $group, $match) != 0)

Offline SleePy

  • Site Team Lead
  • SMF Master
  • *
  • Posts: 30,341
  • Gender: Male
  • Thats his happy face.
    • jdarwood007 on GitHub
    • @jdarwood on Twitter
    • SleePy Code - My personal site
Re: Converting big forum from Yabb to SMF
« Reply #36 on: August 05, 2008, 02:00:13 PM »
Ahh, I thought I had fixed the id_poll index issue. I fixed it

Thanks for the suggestion in the regex. I see from the other regex in membergroup conversion it looks right :)
Jeremy D — Site Team / SMF Developer
Support the SMF Support team!
Profiles:
GitHub

Offline Gilrod

  • Semi-Newbie
  • *
  • Posts: 81
Re: Converting big forum from Yabb to SMF
« Reply #37 on: August 05, 2008, 02:40:27 PM »
Didn't you see my post :)

Offline SleePy

  • Site Team Lead
  • SMF Master
  • *
  • Posts: 30,341
  • Gender: Male
  • Thats his happy face.
    • jdarwood007 on GitHub
    • @jdarwood on Twitter
    • SleePy Code - My personal site
Re: Converting big forum from Yabb to SMF
« Reply #38 on: August 05, 2008, 02:57:15 PM »
Which one? :P
Jeremy D — Site Team / SMF Developer
Support the SMF Support team!
Profiles:
GitHub

Offline Gilrod

  • Semi-Newbie
  • *
  • Posts: 81
Re: Converting big forum from Yabb to SMF
« Reply #39 on: August 05, 2008, 03:30:40 PM »
The one with the error in the 2.2 thread  :P

this one
« Last Edit: August 05, 2008, 03:32:21 PM by Gilrod »