Big phpBB Forum Conversion

[Deriel]

Greetings,

I thinking about converting a big phpBB forum (1.1+ M posts and 15+k users) to SMF. I'm worried abaoiu performance in general and search performance in specific. Somebody could, please, explain to me exactly how the search works in SMF ?

[A.M.A]

Do not worry too much about search .. you can limit it by using one of these:
wait time between searches
Require minimum number of characters in search
Fuse Box

[Deriel]

Hummm.... ok. But the search feature is implemented how? FULLTEXT? wordlist table? LIKE in SQL queries? By my searchs, this would be the biggest SMF forum and I'm a bit worried (no previous experience with SMF) with overall perfomance.

(sorry for the bad english, I'm from Brazil  )

[[Unknown]]

Hummm.... ok. But the search feature is implemented how? FULLTEXT? wordlist table? LIKE in SQL queries? By my searchs, this would be the biggest SMF forum and I'm a bit worried (no previous experience with SMF) with overall perfomance.

Well, first of all I should note that this is not the largest SMF forum.  As well, I want to make sure you know that the converter has been designed to be able to convert large numbers of posts, etc.

A lot of people don't like word lists, and they only help so much.  Additionally, they limit what you can search for significantly (four letters at least, no wildcards, etc.)  The same problem is present with FULLTEXT match searches.  For this reason, SMF does not use FULLTEXT or a word list.

It is, however, very possible to make it use one of these (FULLTEXT I would suggest), especially with newer versions of MySQL.  Significantly, I would note than MySQL 4.1.x would be best.

So, the search currently uses a combination of temporary tables and LIKE clauses.  It may also use RLIKE depending on your options and selections.

Look here:
http://www.redandwhitekop.com/forum/index.php

Additionally, the search functionality can be disabled for guests, and other groups - it is also possible to use things like the "Fuse Box" to disable search under high load conditions (high load averages, that is.)

--> Converters and Importers.

-[Unknown]

[Oldiesmann]

The largest SMF board that I know of is the Coppermine Support Board with over 10,000 members...

[Deriel]

Well... I think that Valinor (after my conversion from phpBB to SMF) will be the largest: 1,099,281 messages and 16,027 users. But I have a small problem with the convertion script. It convert too few posts each time and isn't nice click a thousand times in the Continue button

Can somebody change it to me (or help me to change it)? No need to worry with server load, it's a dedicated server just to the fórum

[[Unknown]]

Well... I think that Valinor (after my conversion from phpBB to SMF) will be the largest: 1,099,281 messages and 16,027 users. But I have a small problem with the convertion script. It convert too few posts each time and isn't nice click a thousand times in the Continue button

Well, the continue button should be "clicked" automatically with JavaScript, but it still is somewhat long a process for a million posts.  If you want to increase the limits, find:

Code Select Expand

$result = mysql_query("
SELECT $select
FROM {$from_prefix}posts AS p
LEFT JOIN {$from_prefix}posts_text AS pt ON (pt.post_id = p.post_id)
LEFT JOIN {$from_prefix}topics AS t ON (t.topic_id = p.topic_id)
LEFT JOIN {$from_prefix}users AS u ON (u.user_id = p.poster_id)
LIMIT $_REQUEST[start], 500");

Replace the last line with:

Code Select Expand

LIMIT $_REQUEST[start], 2500");

Just below that, you should see:

Code Select Expand

mysql_query("
INSERT INTO {$to_prefix}messages
(" . implode(', ', array_keys($messages_table)) . ")
VALUES (" . implode('),
(', $rows) . ")");

$_REQUEST['start'] += 500;
if (mysql_num_rows($result) < 500)
break;

Change both 500's to the same number (2500 in this case).  I suggest not making it *too* high, because it may time out - which will cause unpredictable behavior.

Also remember that it's best to recount all forum statistics from the administration panel after conversions (even though the converters are meant to, in most cases, calculate them properly.)

-[Unknown]

[Jerry]

Once a long time ago when I converted from phpBB to invision, thier converter script automatically loaded a certain number of posts every few seconds like
Post 1 through 500 .. . .
Post 501 through 1000. . .
posts 1001 through 2000
Posts Conversion Complete

Would something like this be possible with this script? That way no clicking maybe that would be useful here. even setting the script at 2500, isnt that still alot of clicking for that amount of posts?

[Metho]

Would something like this be possible with this script?

Well, the continue button should be "clicked" automatically with JavaScript

And it does.

- Methonis

[Deriel]

Automatic Continue? Hummm... now I see a Javascript error:

Code Select Expand

Line: 104
Char: 7
Error: 'document.autoSubmit.b' is null  or not an object

Line 104 is the *** one:

Code Select Expand

<script language="JavaScript" type="text/javascript"><!--
window.onload = doAutoSubmit;
var countdown = 3;

function doAutoSubmit()
{
if (countdown == 0)
document.autoSubmit.submit();
else if (countdown == -1)
return;

*** document.autoSubmit.b.value = "Continue (" + countdown + ")";*** countdown--;

setTimeout("doAutoSubmit();", 1000);
}
// --></script>


I'm using Windows XP SP2 with Internet Explorer 6 (but I tested it with Firefox 1.0 too).

[[Unknown]]

It seems the form's name got missed somehow.  Thanks for finding this - I hadn't tried the phpBB converter recently (sorry.)

Look for:

Code Select Expand

<form action="', $_SERVER['PHP_SELF'], '?step=', $current_step, '&amp;start=', $position, '" method="post">

Replace:

Code Select Expand

<form action="', $_SERVER['PHP_SELF'], '?step=', $current_step, '&amp;start=', $position, '" method="post" name="autoSubmit">

-[Unknown]

[Deriel]

 

Done! Everything working. No more manual clicks

[notyourself]

It seems the form's name got missed somehow.  Thanks for finding this - I hadn't tried the phpBB converter recently (sorry.)

This problem is also present in the IPB 2 converter.  I was trying to do it today with a fairly large forum [350,000 posts & 5,000 users] and it was taking forever because I had to keep clicking.  I changed the code to what you posted above and now it works great.  Maybe the conversion code was copied from the phpBB mixup and it's present in most of the current ones?

Thanks for helping me out

[[Unknown]]

This problem is also present in the IPB 2 converter.  I was trying to do it today with a fairly large forum [350,000 posts & 5,000 users] and it was taking forever because I had to keep clicking.  I changed the code to what you posted above and now it works great.  Maybe the conversion code was copied from the phpBB mixup and it's present in most of the current ones?

Indeed it was - several of the converters, I'm afraid, were affected by this (some not yet public, even) and they've all been fixed.

-[Unknown]