PHPbb to SMF -- Not converting all topics

[jaijaymama]

I did get it worked out.  I switched to a new host and they ended up dumping my database (full backup, by the way, and it was 313MB!). I did get convert - and now everyone's happy! 

[ediww]

today was the D-day. no turning back.

500k+ postings, 5k+ users. phpmyadmin says for both bgmamma and smf (they are still in one database)
64 table(s)       Sum       11,142,906       --         824.7 MB         5.3 MB

debian sarge (current testing), apache 1.3.33, local mysql server (4.0.24), php4.3.10, not counting any uploads et cetera
p4 3.06,2gb of ram, pata disks with softwre raid1, with 27MB/s (hdparms says).

done in around 2,5 hours (05:30 am to 08:00 am), "pure" converter time, tweaked the script a little to avoid the timed wait
setTimeout("doAutoSubmit();", 0);
on line 2928
since was the only user.
(it is not a good idea to do so, except if you have many posts and you are the only user, and you are in a hurry).


SO:
I.for sure there are missing topics. strange: i've got the same number of messages, but i really can not find the messages from a particular topic. i've still got both my boards running, if anyone of the devs is interested, can show where and what.

II.second problem, if in phpbb2 is a "topic" which is only a link to a moved topic (numviews=-1), it gets converted but points to nowhere (generates error).

III. the announcement topics. better, leave them just as sticky.

IV.there is a problem with the polls:
2: max(): Array must contain atleast one element in Poll.php line 362. going to check it right now.
i guess it is because nobody voted or something, but it is not for that topic.

that's all. going right now to see that poll s**t. so far so good.

[ediww]

V. duplicate username resulted from "illegal" char.

in phpbb i've had two different users with usernames "eli" and 'eli===". the second got converted as "Eli" also, resulting with two users with same username, and different ID_MEMBER

[Grudge]

ediww,

Sorry that you've had some problems, I'll try to address some of them:

Firstly, could you go to the SMF admin centre and run "Find and Fix any Errors". This should sort out the moved topics, hopefully also the polls and maybe the other bits. The poll error is because their don't seem to be any options for that poll - so maybe the conversion hasn't fixed that?

As for usernames with duplicate names - we really should of protected against that! I'm afraid you will have to manually edit that using phpMyAdmin - although we could give you another converter to fix these things if you don't mind running it again?

I'm concerned about the "missing" messages - could you run the repair error thing and let us know how that got on.

Cheers,

Grudge

[ediww]

grudge,

so far, and it becomes fun to dig. i've got a problem with the polls, and i've manually fixed it - there was only one really important poll. there was entries for who voted, the poll existed, but the options (w/o &'" or any special chars) was missing.

i've put by hand them in the db and everything is ok (it was a poll about choosing a moderator so it was important to do quickly). so, they are suspicious to me, but i will check them better later. for now, maybe the poll import have problems, but in that moment i do not have a problem with poll import

but with the topics...

i've got two, not, 3 types of problems.

1 is solved already- this was the "moved" pointers. just checked for topics with -1 replies and removed them.

2
next, i've pulled from the database the topic id's from the phpbb and smf (all who have moved field = 0). removed all the new topics (tbe board is live) and compared the id's.

then, pulled the message (post) id's from the database, and checked the numbers, topic per topic.

nd i've got about 300 topics, which have the messages imported but the record in the topics database is missing. checked with simple shell scripts, no error with the numbers. did not check the content, but several random picked are equal.

3 th i've got 2 topics which have messages in phpbb but the id's are not the same in the smf and phpbb2.

i will check with the moderators, it is possible that these topics was one, but split to two.

wwell edi
PS i will leave the repair options for now.. is there a doc explaining what is done with them in detail somewhere - or i must check the sources?

[Grudge]

ediww,

Glad to see you are working through it

The most important problem seems to be these messages without topic entries. I assume you are saying that the messages table has entries with an ID_TOPIC value, but no entry in the topics table exists with this value - very odd.

I do suggest you look at the repair options - the source code is in RepairBoards.php. I would first say backing up my be advised before trying to do it. In the case of messages without topic entries, repair will actually CREATE those entries for you, and put them in a "salvage board". You can then check they are correct and move them into the relevant actual boards. Of course you could edit RepairBoards.php to make sure they just get put in the correct board, as ID_BOARD is now stored in the messages table.

On a side note, to help us fix these problems in the future, could you please post an example line from phpBB which *didn't* get converted over (Assuming your phpBB install is still in place). For example, if a message with a topic_id of 364 didn't have the topics entry created, do:

Code Select Expand


SELECT * FROM phpBB_topics WHERE topic_id = 364


And post the result here, so I can see if there was an obvious reason for the error.

For the other problems, personally I would have removed the "moved topics" pointers by deleting entries with a moved_topic_id value which wasn't equal to 0, as there is a *chance* that phpBB could put the wrong numReplies value in for a topic which wasn't moved.

The third problem does sound like someones split the topic - especialy if one of the numbers is very high compared to the other. The converter desn't do any clever stuff creating additional entries at conversion.

With the polls are you saying that any options with special characters in them were missing? You have any examples of what characters they were?

Cheers,

Matt

[ediww]

grudge,

i've fixed the missing topics "by hand" - using phpmyadmin, and some mysqldump|grep, i've managed to make the missing entries quite easily, fixing the posts/forums and polls attached. i really don't wanted to sort out 315 threads by hand - the effort would be at least the same. everything is just fine after that sewing finished;) i will double-check with the repair script.

If you need something, please, pm me and tell me exactly what (and where to send)- i do still have both smf and phpbb databases.

actually, i will perform an test import on other machine tomorrow if i do have some time, from the same phpbb2 dump. it will be interesting to see if the missing things will be the same.

but, let me know if this will be of any help (i.e. someone will fix something in the import scripts).

wwell edi
PS just to make it clear: the import _IS_ good, and in the matter of fact, it is much cleaner than the result from other boards import scripts.

[wobo]

After having moved my phpBB to SMF - although not quite as large a beast as edi's! - I had similar results:

1. Missing messages
Strolling through the new smf I realized there were topics with a reply number of '-1'. After checking both phpBB and smf database I realized that all those topics were topics whose messages were moved to another board. In phpBB you could leave a "shadow" topic which - on klick - redirects the user to the new location of the thread. These 'shadow' topics are the ones which turned up with "NoOfReplies=-1" in SMF.

2. Polls
All converted polls which were closed befor convertion contained no data, neither the questions nor the results. The only data displayed are the total of given votes. I have not found any cause or solution to this, only the common fact that it effects only closed polls. A bit annoying but not really a killer.

That was all there was. Everything else was converted. There were no problems with duplicate names because of special chars.

wobo

[[Unknown]]

The moved topics (replies = -1) - how should they be converted?  Should a message be made for them, or should they be ignored?

-[Unknown]

[Grudge]

In the converter for 1.1, I made it so moved topic notices would just be ignored. The alternative would be to make a topic out of them with a "moved" notice, but then you'd have to hardcode the text for said topic, making it annoying for non-english forums.

[ediww]

in phpbb you can have some special chars in the usernames - like & ' et cetera. they do convert as html'zed entities. if it can not be allowed, let's at least have some info about these users, some list or, maybe, email message to the users affected? i've checked the login username before and after conversion (using the phpbb's and the new, smf tables) and did get the differences... but after user compliant.

"-1" topics are pointers. if you move, in phpbb, a topic, you have a choice to leave a "pointer" - which is in fact a link to the moved topic. not a bad feature, imho. it will not be bad to have similar thing in smf.

wwell edi

[Saleh]

ok I did a creepy script that will replace phpBB shadows (pointers) with a move topic:
[convertShadows.php]

Code Select Expand


<?php
include_once 'Settings.php';

mysql_connect($db_server, $db_user, $db_passwd);
mysql_select_db($db_name);

$fetch = mysql_query("
    SELECT ID_TOPIC, ID_BOARD, ID_FIRST_MSG
      FROM {$db_prefix}topics
    WHERE numReplies = -1 AND ID_FIRST_MSG = ID_LAST_MSG
    ")or die(mysql_error());

$i = 0;
$move = 'moved'; //Change it to your own language

while ($row = mysql_fetch_assoc($fetch)) {
   $fetch_2 = mysql_query("
         SELECT * FROM {$db_prefix}messages
           WHERE ID_MSG=$row[ID_FIRST_MSG] LIMIT 1
   ")or die(mysql_error());
   $msg = mysql_fetch_assoc($fetch_2);
   
   $topicID_fetch = mysql_query("
   SELECT ID_TOPIC FROM {$db_prefix}topics
      WHERE ID_LAST_MSG='$msg[ID_MSG]'
        AND numReplies != -1
     LIMIT 1
    ")or die(mysql_error());

   $topic_id = mysql_fetch_row($topicID_fetch);
   $topic_id = $topic_id[0];

   mysql_query("
       INSERT INTO {$db_prefix}messages
           VALUES(NULL, $row[ID_TOPIC], $row[ID_BOARD], $msg[posterTime], $msg[ID_MEMBER],
           '$move: $msg[subject]', '$msg[posterName]', '$msg[posterEmail]', '$msg[posterIP]',
           $msg[smileysEnabled], $msg[modifiedTime], '$msg[modifiedName]',
           '[iurl]$boardurl/index.php?topic=$topic_id.0[/iurl]', 'moved')
   ")or die(mysql_error());
   
   $id_fetch = mysql_query("
      SELECT ID_MSG FROM {$db_prefix}messages
         WHERE posterTime = $msg[posterTime] AND
          body = '[iurl]$boardurl/index.php?topic=$topic_id.0[/iurl]'
          AND icon = 'moved'
      LIMIT 1
    ")or die(mysql_error());
   
   $last_id = mysql_fetch_row($id_fetch);

   mysql_query("
       UPDATE {$db_prefix}topics
           SET
               ID_FIRST_MSG=$last_id[0],
               ID_LAST_MSG=$last_id[0],
               numReplies=0
       WHERE ID_TOPIC=$row[ID_TOPIC]
    ")or die(mysql_error());
   $i++;
}
echo "<b>$i</b> Shadow topics where converted succesfully!";
?>
you have to put this script in the root dir of SMF and run it ONCE!

just change the value of $move to your language..

I know it's very bad and I don't like it but it works
suggestions to improve it?
of course this is tested on 1.0! I don't know if it's compatible with 1.1 or not.

[dzver]

After edi's recommendation I've moved my forum site to smf. Then I've tried to post new topic. As a result i got topic with one old (not related message) and my new posting inside. I've deleted not related one and I've got strange result - topic with -1 replies and no messages. Then i manually deleted all topics with -1 replies (26 of 2750) and then all messages with no topic (54 of ~130000).

mysql> delete from smf_topics where numReplies=-1;
Query OK, 23 rows affected (0.00 sec)

mysql> delete from smf_messages where id_topic not in (select id_topic from smf_topics);
Query OK, 54 rows affected (0.90 sec)

I had no time problems - on overloaded hosting, athlon 1800+ it took about 15-20 minutes to convert the database (130487 msg).

My current problem is that all users are able to see all hidden forums. I am not exactly sure how permissions system in smf works, but i will try to understand and fix this too.