Converting from YaBB 2.2.3

[trepanne]

OK, following sleepy's suggestion in this post, I first upgraded my YaBB2 RC forum to the current version (YaBB 2.2.3).  The upgrade seems to have gone fairly successfully.

Then I tried to convert the YaBB 2.2.3 forum to SMF.  This is not working.  It's failing with this error message:

Code Select Expand

Successful.
Converting membergroups... Unsuccessful!
This query:
DELETE FROM `forum`.permissions
WHERE id_group > 4;
Caused the error:

Table 'forum.permissions' doesn't exist

I'm using the convert.php file from this sticky post, and i'm using the yab22_to_smf.php file from this sticky post.

Thanks for helping!

[AzJose]

I used this one tonight on yabb 2.2.2 and worked barring a couple of minor issues.

This Message

Hopes this helps and hope the SMF guys don't mind me trying to help another user 

[trepanne]

Well thanks a lot AzJose; I certainly appreciate all the help I can get.

As it turns out, though, my problem was careless user error... I had blown away the DB & recreated it empty, so of course the code wasn't able to find the table it was looking for.

But I did use the file you pointed me to, and it successfully converted my forum.  Hooray!

However, the conversion process did not succeed in maintaining the attachments from the old site.  Anybody got a suggestion for how to fix that?

TIA

[trepanne]

It looks like the conversion process didn't copy over the attachment files, so I did that manually.  Still not working though.

As well, for posts that had been edited... both the original text and the edited text are deleted in the converted version.

SMF looks pretty jazzy; I'd like to get it up & running.  However, preserving fidelity of the post text and the attachments is about the highest priority I've got.

Any help would be much appreciated.

[SleePy]

Was there data in the attachments table?

[trepanne]

Was there data in the attachments table?

nope, a complete blank:

Code Select Expand


Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 7306909
Server version: 5.0.33-log FreeBSD port: mysql-server-5.0.33

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> use forum
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

mysql> select * from attachments;
Empty set (0.00 sec)

[SleePy]

Looks like that is our issue. It didn't even convert any attachments.

In your yabb files. In either Paths.pl or Variables/Paths.pl

Is there a line in there that has a "uploaddir" variable?

[trepanne]

In your yabb files. In either Paths.pl or Variables/Paths.pl
Is there a line in there that has a "uploaddir" variable?

Yes, it's set to ~/web/public/forum/Attachments (but actually an absolute path that is correct, not a relative path).

[SleePy]

I am guessing that SMF is using the same path for attachments?

That would be the only explanation here.

[trepanne]

I am guessing that SMF is using the same path for attachments?

How would I verify?

Pretend that you're talking to a very slow child, and you won't be too far wrong...

[SleePy]

In the SMF database go to the smf_settings table.
Search for the variable "attachmentUploadDir".

Is that the same path as your yabb one?

[trepanne]

Code Select Expand

In the SMF database go to the smf_settings table.
Search for the variable "attachmentUploadDir".

Is that the same path as your yabb one?
No, it isn't. The path is set to ~/web/public/smf/attachments (i.e. the correct path).

[SleePy]

I am at a lost here then. There should of been data converted then.

If you go to your SMF database.
There is a smf_convert table.

Search by type and see if there is any one that exists with "msg_attach".
I wonder if they didn't even get transferred over from the original conversion.

Edit, I think I possibly see the bug now. Give me a second to make the changes.

Edit2, Try the attached one.

[trepanne]

OK, I think we're getting somewhere.

Now the conversion process copies over a bunch of files into smf/attachments/ as you'd expect.  However, each file copied over is only 7.0K in size, and the original filename has been rewritten as something like
"1_d41d8cd98f00b204e9800998ecf8427e" (i.e. an incrementing index, then an underscore, then a hex hash, with no file extension).

I tried using diff on some of these files, and the ones I checked all were identical.  Examining the contents with cat, I see the desired filenames (along with some other metadata from PDF attachments) listed as data inside the files.

Does that help?

Thanks for the support, Sleepy.  Am I the only one using this converter?

[SleePy]

Well that is the encoded file name. SMF encodes the file name for protection and security. It shouldn't be putting any data into the files. It keeps that stored in the database.

I don't think you are the only one using this converter. Although the 2.2 converter for Yabb is fairly new so keep that in mind. I rewrote it totally and I did make a few mistakes it looks like with attachments. I didn't have any attachments or didn't see any attachments in the test board I had

Try this file if you don't mind. I hopefully found that bug that made them all the same file size. I wasn't getting the correct file name copied over so it was bugging out.

[trepanne]

All right sleepy, this is working, thanks.

Now I've got the actual files stored inside smf/attachments/, and browsing the forum, they are properly linked to their posts.  They can be downloaded & viewed correctly.  So far as I can tell, this functionality is now working.  Thanks!

There's another bug I can see, though.  In posts which have been modified after initial posting, the modifications aren't getting converted over to SMF; instead the posts are truncated where they were modified.  As well, it looks like posts which contain certain kinds of whitespace characters are getting truncated there too.

I've got posts where text has been cut&pasted from HTML documents; looks like SMF truncates right at the newline.

Any idea what might be causing this?  Thanks again.  I really do appreciate the help; what I've seen of SMF really makes me want to convert.

[SleePy]

Well that is odd.

In that sample text you sent me. One of those posts was edited right? I believe it was the 2nd or 3rd reply that was edited?

That seems odd though :|

[trepanne]

In that sample text you sent me. One of those posts was edited right? I believe it was the 2nd or 3rd reply that was edited?

I'm afraid you must have me confused with somebody else, because I haven't sent you any sample text.  Would it be helpful if I did so?

In addition to this "edited post" bug, I was also trying to figure out why the conversion process is truncating posts that contain newline characters cut&pasted from rendered web pages.

I tried cutting & pasting the offending text into a new post on SMF, and that was no problem.  It was just the conversion process that choked on this.

[SleePy]

Oh sorry that was. To many people

It would help to see a regular post then an edited post. I don't get what could of changed that caused this, but we could try and fix it.

As well the sample text for the site thing. The converter isn't touching any line endings, so I wonder if its just an issue how SMF is able to handle it since it didn't go through its cleanup functions that posts normally go through.

[trepanne]

Sleepy,

I sent you a PM with some sample text that's getting mangled in conversion.

Thanks for the help!

[trepanne]

Okay, following up on this:

We're discussing two apparent bugs:
1) The YaBB converter seems to truncate edited posts, and
2) The YaBB converter seems to truncate posts with certain kinds of whitespace.

I am unable to replicate problem (1).  I suspect that it's actually not true; it's just that the post edits contained bad whitespace, so got truncated.

The actual bug is (2).  I opened up a Python interpreter, then cut&pasted a post from the old YaBB forum that was getting truncated as a Python string so that I could analyze each character of the whitespace.

It looks like the conversion process is choking on '\xc2\xa0' in UTF-8, which is what an HTML ' ' translates to.

So instead of setting up my new SMF forum with UTF-8 character set, as I'd been doing, I instead tried setting it up as ISO-8859-1.  This seems to fix all the problems (although I'll be looking around for any more bugs).

I think I can get by with ISO-8859-1, because all the forum content is in English... but just a heads-up, it seems that the YaBB converter isn't compatible with UTF-8, if that's an issue for anybody else.

Thanks again for your support, it's much appreciated.

[SleePy]

Hmmm..

Doesn't make much sense why the converter isn't understanding the UTF-8 characters.
I guess we could try to replace those characters with a preg_replace if you think that is going to work for you.

If you are using non latin characters, then you are bound to have issues if you are not using UTF-8 for SMF.

[trepanne]

Sleepy, my problem is basically solved.  I really don't need nonroman characters, so latin charset is good enough for me.  I'm just providing some feedback on what I found to help you or anybody else who encounters a similar problem in the future.

Thanks very much, you really helped me get it working.  We're subjecting SMF to serious testing in the process of switching over.

[schlafe]

I am hesitant to start a new thread as my issue relates directly to the aforementioned problems.  I have almost 100% successfully converted a YaBB 2.5 AE forum over to SMF via IamTheBoy's instructions located here:http://www.simplemachines.org/community/index.php?topic=457735

I utilized the yabb24_to_smf.php and converter.php located in mrtoner's reply #11 here:
http://www.simplemachines.org/community/index.php?topic=344465.msg2778613#msg2778613

Direct link to yabb24_to_smf.php and converter.php zip file: http://www.simplemachines.org/community/index.php?action=dlattach;topic=344465.0;attach=156540

I can confirm the issue of posts getting truncated was resolved by using the  ISO-8859-1 standard rather than UTF-8.

Attachments have transferred and are properly associated with every post that has not been edited.

The only remaining issue is that attachments associated with YaBB posts that have been edited hare no longer properly associated with the posts and do not appear in the thread.

Any advice with regard to isolating the cause and potentially resolving the issue would be greatly appreciated.

Thank you,

josh