Report: Successful YaBB 1 Gold 1.1 Conversion

[BigMoneyJim]

I recently converted a YaBB 1 Gold SP1.1 (.pl version in /cgi-bin/, although I don't think that matters in converting) BBS to SMF 1.0.3 .

The site has about 1300 users and 42,000 messages in 2600 topics over 14 boards. The board had been up for about 3 years and is getting around 15,000 hits/day. The whole fileset--software and data--was about 83 Mb. The site mostly worked fine but had various glitches with "new" messages not showing properly on the index page and some users' profiles stopped tracking their last posts. The site seemed a tad slow, and the page loaded slowly so you had to wait a second or so for all the posts to draw in. (After converting the comparison is that the old site was very slow, but not as slow as other YaBB sites I've seen.)

I don't run the board, but I took it upon myself to check out replacement software. First in my mind was phpBB because it was GPL, free and in my perception the most popular choice among board operators. I wanted to see what the alternatives were...I forget how I wound up at YaBB SE's site, but that led me here, and I saw an active development and active support community working on (from a nontechnical forum member's point of view) a slicker faster version that looks and works almost exactly like the old board. I like that because the posters were familiar with the old software and there were no feature complaints that SMF didn't already address. (Well, except "ignore user", but phpBB didn't have that, either.) I installed test versions of YaBB1G1.1, phpBB and SMF to compare them and test converting from the old to the new. SMF took the conversion much better and more completely, and I detest how phpBB handles "new" posts as I don't always read all then new posts each session. And the SMF "classic" theme looked like it had a low learning curve for nontech posters used to the old forum since it's almost exactly like YaBB1G.

My research generated interest. The board owner agreed SMF looked like the best choice and set up a new server account, gave me access to set it up there and uploaded a snapshot of the YaBB1G files. I was a bit worried that a glitchy 3-year old board with 42k posts might have some corruption issues, but the conversion script handled it without a problem...it just took a little longer is all. We twiddled with the settings, figured out what did and didn't transfer and then decided to do it for real. We put both boards in maintenance mode, he uploaded the new snapshot and changed the domain name over. I converted again, clearing the new board as I went.

The users seem universally happy with the upgrade, and we keep discovering cool new features. Oh, and the new board is lighting fast compared to the old YaBB1G. Some people blame this on Perl & CGI, but those hits are more of a configuration and purpose issue. The real speed demon here is going from a text-file-based site to a database-backed site.

Other notes: Afterward the database size was 40MB and the software directory was about 7MB. The old site was a Windows server, and the new site was Linux. It didn't affect the conversion, but at one point I tried to get the old YaBB1G working on the new board and had tons of trouble. I think there were multiple issues that I won't delve into here.

ISSUES:

(Non-SMF) The biggie was the DNS cutover. Some people got the new address in an hour while a few stragglers were still being sent to the old site 48 hours later. We put up messages at the old site saying we moved and that we had no control over when the name would switch over. If we had put those up when we started the move there would have been less confusion; as it was users going to the old site were getting a small cryptic 404 error that didn't indicate it was a 404 error. I spent a couple of hours trying to figure that out thinking the new board was coughing up a funny error for a couple of users.

(SMF) An annoying thing I found about SMF is that there are too many places to change the site URL: server settings, smilies, themes and avatars...and as far as I can tell you have to edit the database or use [Unkown]'s repair_settings.php to fix the avatar URL. I ran into this because I installed SMF with [host-ip-address]/~user/forums/index.php then changed to [temporarydomain.tld]/forums/index.php and finally [realdomain.tld]/forums/index.php . But after a few iterations I found "grep -r"-ing the forum files and searching the database helped me track them all down. repair_settings.php looks like it will handle the job, too. I used it once to look at my settings but didn't change them because at that point the site was live and busy and I just had one value to change.

(Conversion process) During the conversion process the users who had YaBB1G builtin avatars seemed to lose their avatars during the conversion. This is not technically correct, but in effect this is what happens. Look to the advanced section for more detail and a resolution. A possible resolution I didn't try is to copy the old avatars into the avatars/ directory...that might work but isn't quite "proper" with respect to how the software is designed.

(Conversion process) During the conversion process the IM "Outbox" was cleared. No biggie for us, and all the Inbox IM's were kept.

(Conversion process) Links to the original message at the top of quote boxes were stripped in the conversion process. No biggie. This makes sense given the technical issues, but I thought I'd mention it. New quotes still get the link back to the original message, of course.

ADVANCED:

(Avatars) It turns out the YaBB1G "builtin" avatar settings were transferred but meaningless as-is to SMF. Users that had picked builtin avatars in YaBB1G had the filename of the avatar in their avatar column in the _members db table. I made an avatars/Original/ directory and copied the old avatars into it. Now I had to fix the avatar settings to point to the correct place; for each filename I ran a database command to update, for example, "bart.gif" to "Original/bart.gif". Don't do this for "blank.gif", although it probably doesn't hurt if you do it accidentally. I used SQL commands to update the database, and here is an example:

Code Select Expand

UPDATE smf_members SET avatar='Original/bart.gif' WHERE avatar='bart.gif';

(old URLs, intro) YaBB1G was at /cgi-bin/yabb/YaBB.pl . The new site couldn't go there without major contortions, and the old link arguments wouldn't have worked on the new board, anyway. But there are a fair number of internal and external URLs pointing to posts on this board, and I hate to simply redirect them to the index of the new board with a "sorry, we moved, update your bookmarks and links" message if I can avoid it. By looking at the old and new URLs and how things are stored I learned that the old software referenced all messages by topic number and start message, and SMF can work the same way. (old URL: /cgi-bin/yabb/YaBB.pl?board=[boardname];num=[topic_ID];start=[start_message] ; new URL: /smf/index.php?topic=[topic_ID].[start_message]) What's really great about this is that the topic ID doesn't change when you move it to a different board, so the link can follow the message through reorganizations. However, the topic numbers change during the conversion process, and I had 2600+ topics to cross reference. (There are potential issues if threads are later split, and if the new board lists a different number of messages per page than the old board then the starting message may put the reader on a different page than originally intended. I didn't attempt to fix these; I doubt we'll be splitting old threads, and if the messages per page changes at least we got the reader to the right topic.)

(old URLs, part 1) First part of this is to somehow get a list of what each old topic ID goes with which new topic ID. After failed attempts at figuring out a conversion algorithm I realized that the conversion script must at some point know which goes with which, and I was right. It creates a tempID column in the _boards table, and the tempID is the old topic number! However it drops this information during the conversion process. I commented out the code that drops the tempID column so I could get to the cross-reference information after the conversion. The code that drops this info in the SMF 1.2 version of yabb_to_smf.php starts at line 1339 and looks like this:

Code Select Expand

        if ($_REQUEST['start'] == 0)
        {
                mysql_query("
                        ALTER TABLE {$to_prefix}boards
                        DROP tempID");
        }

I commented it out as so:

Code Select Expand

/*
        if ($_REQUEST['start'] == 0)
        {
                mysql_query("
                        ALTER TABLE {$to_prefix}boards
                        DROP tempID");
        }
*/

Note that tempID is used and dropped several times in several tables. But in the version of the script I had there was only one tempID dropped from the _boards table. If you don't understand what this is doing, you probably shouldn't try it. Anyhoo, so after making the alteration and running the conversion script, the _boards table still has column tempID in it. I made a query to list the tempID and ID_TOPIC from _boards. Now I have my topic ID cross reference!

Code Select Expand

tempID  ID_TOPIC
1024863332      1
1024907739      2
1024908326      3
1024909372      4
1024909774      5
1024974175      6
1024978255      7
[and so on for all topics]

Now that I have my cross-reference table I drop the tempID column from the boards table. I don't know if would hurt anything to leave it there, but I don't need it and the board doesn't need it so I got rid of it. Quick manual conversion exercise: given the above list, an old url of YaBB.pl [nofollow]?board=news;num=1024909774;start=15 would translate to a new url of index.php?topic=5.15 .

(old URLs, part 2)Okay, I have the list and can manually translate old URLs to new URLs that will take you to the correct topic and likely the correct page in the topic. Let's hack a Perl script to replace YaBB.pl [nofollow] that will translate and redirect old URLs to the new ones. Warning: This is an ugly, uncommented, hacked-together script. If I get around to making a nicer, easier-to-edit one I'll come back and post it here.

Code Select Expand

#!/usr/bin/perl

use CGI qw (standard);

$query = new CGI;
my $oldtopic = $query->param('num');
my $start = $query->param('start') || "0";

%topics = qw(
1024863332      1
1024907739      2
1024908326      3
1024909372      4
1024909774      5
1024974175      6
1024978255      7
);

my $newparam = $topics{$oldtopic} || 0;
if ($newparam) { $newparam = "?topic=" . $newparam . "." . $start; }

my $site = "http://mydomain.tld";

my $url = "/forums/index.php";
if ($newparam) { $url = $url . $newparam; }

print "Content-type: text/html\n\n";
print '
<html>
<head>
  <title>Forum MOVED</title>
<META http-equiv="refresh" content="7;URL=';

print $url;
print '">
</head>
<body>
<b>The Forums have moved to a different location on this server
.
<p>
You will be redirected automatically in a few moments.
<p>
Please update your bookmarks. We apologize for the inconvenience.
<p>
<a href="';

print $site . $url;

print '">';

print $site . $url;

print '</a>
</b>
</body>
</html>
';


Notice I was able to copy the list as posted into the perl code with qw(); . I only included 7 topic cross-references, but with 2600+ topics listed the file gets pretty long and about 42kb in size.

I want to redo this script. I plan to put the cross-ref list at the end as a subroutine (for readability and editability) and return a HTTP 301 permanent redirect code instead of using META refresh. But it's working for me so I may not get around to redoing it.

Another option would be to query the database for the topic cross-reference data. This would be advisable for larger sites. I would probably make a new table for the cross-reference data, although I suppose you might get away with leaving tempID in the boards table and cross-referencing from there.

Another approach might be to redirect the parameters to a php script and do the magic there. Perhaps even an SMF mod. But I'm pretty sure I'll never get around to it.

[Oldiesmann]

The easiest way to do this would be to use MySQL to pull the appropriate topic from the database for that tempID. Assuming your host has the MySQL Perl API installed, there's a nice tutorial at http://www.ucl.ac.uk/is/mysql/perl/ that explains how to get information from a MySQL database using Perl.