[SMF Converter] XMB - 1.9.8

Started by JayBachatero, January 08, 2007, 09:21:48 AM

Previous topic - Next topic

rcpp

Quote from: amesfan on January 26, 2010, 12:04:32 PM
I checked and I have a 64M memory limit, so it looks that size is well below it.  Is there anything else that would cause this?

Hmmm... This message means your query tried to allocate 900k of memory, but more or less 1MB is already allocated. This is usually a server side problem:

http://dev.mysql.com/doc/refman/5.0/en/out-of-memory.html [nofollow]


You could have a look at your mysql's logs.

amesfan

Thank you for the help, I actually figured out a way around it.  I found I have over 2gb of attachments on my forum, and it turns out that a group of about 500 are all bmp's at least 500kb in size, so it was going way over the 64m limit even though the error was different.  I alleviated this by changing the limit in the php file from 500 to 50 so it would process much less files at once.  It took much longer to convert the attachments, but ultimatey completed!

Looks like the conversion was successful all the way around, I had to rebuilt the thumbnail cache, but even U2U's migrated over!

Thanks again!

rcpp

Quote from: amesfan on January 26, 2010, 03:35:40 PM
Thank you for the help, I actually figured out a way around it.  I found I have over 2gb of attachments on my forum, and it turns out that a group of about 500 are all bmp's at least 500kb in size, so it was going way over the 64m limit even though the error was different.  I alleviated this by changing the limit in the php file from 500 to 50 so it would process much less files at once.  It took much longer to convert the attachments, but ultimatey completed!

Looks like the conversion was successful all the way around, I had to rebuilt the thumbnail cache, but even U2U's migrated over!

Thanks again!

I'm glad I could help! And I will ask you 2 things:

1. Which version was your XMB forum?
2. Does it have the [file]##[/file] tags? If yes, have you been able to fully convert its content to a thumbnail?

Cheers!

amesfan

My forum was XMB 1.9.10 Karl.  The actual reason I converted to SMF was I was unable to upgrade to 1.9.11, and unable to find a way around it without losing data.  Plus I was looking to integrate my Gallery2 and chat into a message board, of which I could not do with XMB.

My posts dig not have file tags, would I look anywhere else for them?

All the images converted over and displayed with paperclip icons.  I used the method elsewhere on this board to assign dimensions and generate thumbnails.  The only issue I had with conversion is many of the titles on my topics have been changed.

rcpp

Quote from: amesfan on January 30, 2010, 12:54:37 PM
My posts dig not have file tags, would I look anywhere else for them?

No where, I guess. What happened in my case was that a few posts had phots attached to them, which had a thumbnail automatically posted inside the [file] tag inside the post body.

Quote
All the images converted over and displayed with paperclip icons.  I used the method elsewhere on this board to assign dimensions and generate thumbnails.  The only issue I had with conversion is many of the titles on my topics have been changed.

I see! I've seen some plugins to generate thumbails, but Im not so sure if they come by default with SMF. So dar, Im thinking about using regex to change from [file] to something SMF understand, though I could change them by hand (only a few posts, 20 something, have images attached).

I will work some more over it and see how far I can get with my small SQL and PHP skills.

Cheers!

rcpp

I tried this to create the proper links for the posts inside the quote blocks, but did not work:


        SUBSTRING(REPLACE(p.message, 'rquote=([0-9]*&)(tid=[0-9]*&)(author=[\w|\W]*)',
                                     'quote (author=[\w|\W]*) (link=topic=([0-9]*)(\.msg[0-9]*\#msg[0-9]*)) (date=[1-9]*)'),
                                      1,65534) AS body,


this I put inside "xmb_to_smf.sql"

any tip? how can I test this in MySQL? I know that the regular exp. works because I tested in python, so I think it might be something sql~mysql related.


Thanks!

rcpp

Hi all! I manage to make a simple python script to convert the XMB 1.9.11 link format to SMF 1.1.11.

The convertion works, however, the script has some problems:

Quote
No control over execution time  (mysqld  job)
High CPU usage - between 78 and 96% for mysqld while the script is running
No user interaction - you put your database config. inside the script.

TODO:
Quote
- Configuration by file
- Timer
- Conection / executing time control
- Interface
- Some error messages like "Missing MySQLdb module"
"Config error" and connection/query errors.

To execute simply type "python xmb-to-smf-quote-fix.py [nofollow]" (without the quotes, of course). Use at your own risk.




Norv

Oh, thank you very much for taking the time to share your solution, rcpp!
To-do lists are for deferral. The more things you write down the later they're done... until you have 100s of lists of things you don't do.

File a security report | Developers' Blog | Bug Tracker


Also known as Norv on D* | Norv N. on G+ | Norv on Github

rcpp

#188
You are welcome! Sorry for the late³ reply.

We are about to move from xmb to smf and Im on the process of testing everythig And my script works on part.... it does change the quote form, but it does not build de link. If I look in the source, there is no <a http://localhost/index.php?topic=1111.msg1111#msg1111 [nofollow] > Quote from ... </a> .

Only "Quote from:username link=topic=111.msg=1111#msg11111 date=1111111"


Of course, 1 is just for ilustrating.  ;D

Nevertheless, how does SMF creates the links? How it brings the username, topic and msg ids and paste them into the href?

Cheers!




EDIT:

I looked into SMF code and I've found where it builds the ?topic=...  part. However, I still can't get how it parses the code. Because, as far as I understand, it should have guessed the ?topic=... part (from the [quote ... ] ), but it didn't.

rcpp

I gave up of using an external tool for that, and decided to use the converter code. However, Im kind of lost on where to call a function there that would perform the search/replace task.


$body=array(0=>"[quote=61238&amp;tid=806&amp;author=Princess Moon]",
            1=>"[quote=61&amp;tid=2346&amp;author=ABCDFGH]");
$matriz['body']=$body;

$msgid_re='~([0-9]*)&amp;~';
$topicid_re='~tid=([0-9]*)&amp;~';
$author_re='~author=([\W|\w]+)\]~';

    for ($i = 0, $size=sizeof($matriz); $i <= $size; ++$i){
        echo $i ."\n";

        $msgid_xmb=preg_match($msgid_re, $matriz['body'][$i], $msgid);
        $topicid_xmb=preg_match($topicid_re, $matriz['body'][$i], $topicid);
        $author_xmb=preg_match($author_re, $matriz['body'][$i], $author);

        $smf='[quote author='.$author[1].' link=topic='.$topicid[1].'.msg='.$msgid[1].'#'.$msgid[1].']';

        $matriz['body'][$i]=preg_replace('~((\[quote=[0-9]*&amp;)(tid=[0-9]*&amp;)(author=[\w|\W]+)\])~',$smf, $body[$i]);
        print_r( $author);
        print_r( $topicid);
        print_r( $msgid);
        print_r( $matriz['body']);
        echo "\n";
        }
?>


This is the code I made, just for testing the regex. I dont know if there is a place where I'd put that that is specific for converting the messages from the board.

Nonetheless, Im about to install Eclipse and play with its debug tool and see if I can manage to find myself when it starts to convert the messages, since the regex above would be applied on the 'body' column.


Cheers!

Norv

Sorry for the delay. I'm not entirely certain what is the actual result you are trying to change...

Nonetheless: eclipse and the debugger might not help as much as you may wish... because the code of the actual conversion is in the sql file which ships with the convert.php script, and convert.php is parsing it and eval()-ing it.
Perhaps what you want to see is: xmb_to_smf.sql lines 129 - 147
specially this bit: (which is a PHP chunk)

---{
$ignore = true;
$row['subject'] = stripslashes($row['subject']);
$row['body'] = preg_replace('~\[align=(center|right|left)\](.+?)\[/align\]~i', '[$1]$2[/$1]', stripslashes($row['body']));
---}

(for the converter format, ---{ and ---} are marking a PHP block of code, like you can see ---* for SQL, and others for comments - they're just a few)
When evaluated, $row will be here the result of the request which follows immediately: the chunk of SQL (a SELECT from posts table which follows).
To-do lists are for deferral. The more things you write down the later they're done... until you have 100s of lists of things you don't do.

File a security report | Developers' Blog | Bug Tracker


Also known as Norv on D* | Norv N. on G+ | Norv on Github

rcpp

Quote from: Norv on September 21, 2010, 02:54:00 PM
Sorry for the delay. I'm not entirely certain what is the actual result you are trying to change...

Nonetheless: eclipse and the debugger might not help as much as you may wish... because the code of the actual conversion is in the sql file which ships with the convert.php script, and convert.php is parsing it and eval()-ing it.
Perhaps what you want to see is: xmb_to_smf.sql lines 129 - 147
specially this bit: (which is a PHP chunk)

---{
$ignore = true;
$row['subject'] = stripslashes($row['subject']);
$row['body'] = preg_replace('~\[align=(center|right|left)\](.+?)\[/align\]~i', '[$1]$2[/$1]', stripslashes($row['body']));
---}

(for the converter format, ---{ and ---} are marking a PHP block of code, like you can see ---* for SQL, and others for comments - they're just a few)
When evaluated, $row will be here the result of the request which follows immediately: the chunk of SQL (a SELECT from posts table which follows).


okay. that means I dont need a "for" condition to apply the regex into $row=['body']? and I could use it only once? i.e:



---{
$ignore = true;
$row['subject'] = stripslashes($row['subject']);
$row['body'] = preg_replace('~\[align=(center|right|left)\](.+?)\[/align\]~i', '[$1]$2[/$1]', stripslashes($row['body']));

$row['body']=array(0=>"[quote=61238&amp;tid=806&amp;author=Princess Moon]",
            1=>"[quote=61&amp;tid=2346&amp;author=ABCDFGH]");

$msgid_re='~([0-9]*)&amp;~';
$topicid_re='~tid=([0-9]*)&amp;~';
$author_re='~author=([\W|\w]+)\]~';


$msgid_xmb=preg_match($msgid_re, $matriz['body'][$i], $msgid);
$topicid_xmb=preg_match($topicid_re, $matriz['body'][$i], $topicid);
$author_xmb=preg_match($author_re, $matriz['body'][$i], $author);

$smf='[quote author='.$author[1].' link=topic='.$topicid[1].'.msg='.$msgid[1].'#'.$msgid[1].']';

$row['body'][$i]=preg_replace('~((\[quote=[0-9]*&amp;)(tid=[0-9]*&amp;)(author=[\w|\W]+)\])~',$smf, $body[$i]);


---}



On the version we have installed (after 19.8) xmb quote is defined as written below:


[rquote=112233&amp; tid=1234&amp;author=John Doe] [/rquote]


so, there is no linking inside the quotes. The Python  scipt I made can change is, but there is no linking neither.


changing rquote to quote is way less harder. I think I even posted my modification here.


cheers and thanks for the reply!



rcpp

#192
Updating!


With the Pyhton script I have made previously I was able to create the quote links. It was just that SMF needs the timestamp for the posts. And I missed it.  ;D

Now that I've added the date to the string used on the update clause, it corrects the links and everything works fine.  8)

Sorry for doing that on Python and not PHP, but since I believe the script will reach 97% of processig in PHP as well, I don't think it would be wise to run it on your ordinary production server - specially if you use shared hosting (my case).

The only 2 problems that remain are:

1. XMB posts attached files to the body of the post with a file BBcode - smf dont know how to deal with it and prints [\file]  id of the attachment [\/file]

2. XMB when mergin topics somehow change the topic ID to the most recent one (at least in our case), making Topic A be called Topic B after the merge. That causes 2 problems:

The topic link title and all its inner link titles are called Topic B.
If you rename Topic B to A, as it should be called, it will change only the link within the topic ist, not  inside Topic A.

example:

http://dl.dropbox.com/u/4399839/smf-exemplos/topicName.png [nofollow]

http://dl.dropbox.com/u/4399839/smf-exemplos/topicTitleInside.png [nofollow]



I think this is "doable". I just have to think a little more.

cheers!


EDIT: forgot the second problem:

the 2nd problem is that all the links on the post title, inside the thread, will have a wrong id.



PS.: if you want to run it on a small database, rename database.conf.orig to database.conf and configure it.


rcpp

Bump!

As for 12/10 until today I managed to migrate from XMB 1.9.11(musebrasil.com/velho) to SMF 1.1.11 (musebrasil.com/forum ). The only remain problems are with merged topics, where the merged topic has now the ID from the other one (i.e. B merged with A and now A has B id) ,  making a little confusion with titles and subjects (inside the threads) - hence why I still have the old board on, so users can search their old topic and tell us which topic they are seeking. Nothing that a simple SQL update can't fix. Another problem is that pools has no items or votes, though I already know  how to add them through queries.

I can't remember now if there is another problem, but from 0 to 10, I'd say this migration was 9.0.  8)


- No message loss.
- No user account problems.
- No topic loss.
- No PM loss.

And my Python quick fix works!  8) (now I run it on the XMB before converting, not that make much different though).


There is another small query I run before converting that changes rquote to quote, but it is a simple string replace MySQL command, nothing that you can't do using the official MySQL documentarion.


Cheers!

flewfruit

Hi, I want convert my XMB 1.9.11 to SMF. But how to do this? I don't understand this tut, can someone explain it please better for me :(

I have to install SMF first, right?

rcpp

Quote from: flewfruit on August 25, 2011, 10:12:40 AM
Hi, I want convert my XMB 1.9.11 to SMF. But how to do this? I don't understand this tut, can someone explain it please better for me :(

I have to install SMF first, right?

Sorry for the late reply. I dont check this board that much. The way I did is kind of complicated if you cant deal with command line and some SQL code. I recommend trying to run XMB to phpBB converter and then phpBB to SMF.

Cheers!


Advertisement: