Quote tags not recognized

[Burpee]

I'm not sure if this should be in converters and importers or in here.

I recently changed from phpBB to SMF.

My problem is that in the majority of the posts on my board the [ quote ] tags are not recognized properly.
When I do however modify the post, it shows the proper [ /quote ] to close it off.
When I save the post then, it suddenly shows the quote properly.

I'm wondering how I could fix this without having to resave 1200 posts. Already looked through the forum maintenance and there is nothing to "repair" this in repair_settings.php either. I have already looked at the MySQL information which is like this:

Code Select Expand

[quote="username"]content[quote][/quote]
Seems okay to me, but I could be mistaken.

[Peter Duggan]

I have already looked at the MySQL information which is like this:

Code Select Expand

[quote="username"]content[quote][/quote]
Seems okay to me, but I could be mistaken.

No, that's wrong... it should be:

(can't get this to show right!)

But you should be able to run a SQL query to change that throughout your forum if they're all the same.

[Peter Duggan]

So are you actually getting the extra quote tag before the closing quote tag or is that just how your post here (like mine) came out despite the code tags? IOW, is it effectively 'open quote, content, open quote, close quote' or 'open quote, content, close quote'?

[Burpee]

The resulting effect in the browser is:
[ quote="username" ] content

So the open-quote is not recognized or something, but the close-quote is?

[[Unknown]]

I'll see if I can find what's wrong.

Edit: Fixed for RC1.

-[Unknown]

[Burpee]

In the meantime, any advice on what I could do to fix this?

[Dietrich]

Yeah, I would like to know this too. The problem is just that most phpBB quotes begin with

Code Select Expand

[quote="nickname"][quote][/quote]Now it would be handy if I can run a query that replaces those tags with a simple

Code Select Expand

[quote][/quote] tag.

[Vitat]

SMF not understand phpBB tags.
Needed some script to implement to phpBB2SMF converter, which change

Code Select Expand

[quote="nickname"]
to

Code Select Expand

[quote author=nickname][/quote]

[[Unknown]]

I'll see if I can find what's wrong.

Edit: Fixed for RC1.

-[Unknown]

As I said, yes it should support quote=.  There was something wrong, but it will in RC1.

-[Unknown]

[packman]

As I said, yes it should support quote=.  There was something wrong, but it will in RC1.

-[Unknown]

Is it the phpbb2smf converter or the main SMF codet that's finxed in RC1, i.e. if I import my phpBB2 database using the current converter, the quotes will all display correctly once I apply the RC1 code, or I would need to import my phpBB2 database using the RC1 converter for it to work properly?

Chris

[Oldiesmann]

The next version of the convertor will be released with RC1, so that's what he means...

[[Unknown]]

Umm, no, it was a bug in the bbc parsing engine of SMF.  Your existing conversion will work properly under RC1.

-[Unknown]

[packman]

Edit: Fixed for RC1.

Quote tags still don't seem to be recognised correctly in RC1. I'm still seeing [ quote="aardvark"] tags with the quoted text not shown as quoted.

The additional [ quote][ /quote] at the end of the quoted text seems to be OK now though.

(I've added an extra space after the tag open bracket to ensure that the text doesn't mess up the display!)

[[Unknown]]

Can you give an example quote that has problems?  See if it parses correctly here?

-[Unknown]

[Burpee]

[ quote="username" ]testing text[ /quote ]

testing text

The problem is that when you create a new post with the same thing, it does work
Old posts however are not "converted" to an actual quote block

My solution was to be found in:
http://www.simplemachines.org/community/index.php?topic=14930.msg125925#msg125925
Using phpMyAdmin

[packman]

When you enter a quote manually in phpBB you use:

[ quote]This is some quoted text[ /quote]

If you select text from a previous post for quoting you get:

[ quote="windy"]Some quoted text from the user called windy[ /quote]

Looking at the post_text field in the phpbb_posts_text table shows that quoted text from a user is stored as:

[ quote:49c994073e="windy"]uploading a new zip[ /quote:49c994073e]

The 49c99.... bit appears to be the same as the 'bbcode_uid' field in the phpbb_posts_text table, but I don't understand the significance of this field/value.

After running the convertor I end up with

[ quote="windy"]This is some quoted text from the user called windy

but there is no trailing [ /quote]

[Oldiesmann]

I've seen that stupid phpBB bug before although it mainly seems to happen on phpNuke sites (annoying, isn't it?). Not sure why it's leaving off the ending quote tag.

[Unknown] - Sorry about that. I didn't pay attention to the date of the post, so I thought you meant it had been fixed for the next version.

[[Unknown]]

SMF doesn't even mess with the

.

-[Unknown]

[packman]

SMF doesn't even mess with the

.

-[Unknown]

Here's an original phpBB2 discussion thread. If you look at the fourth message down (from administrator), you'll see what the quotes look like in phpBB2.

http://www.weather-watch.com/phpBB2/viewtopic.php?t=4880&start=120

Here's the converted thread on my test system...

http://www.marton-blackpool.info/smf/index.php?topic=4880.120

There are many other examples...in fact I don't remember seeing a correctly converted quote. I didn't see any errors during the conversion either.

So something is messing with the quote 

Strangely the messages in SMF have kept their open quote tags but lost the close quote tags.

Chris

[[Unknown]]

Here's an original phpBB2 discussion thread. If you look at the fourth message down (from administrator), you'll see what the quotes look like in phpBB2.

http://www.weather-watch.com/phpBB2/viewtopic.php?t=4880&start=120

Here's the converted thread on my test system...

http://www.marton-blackpool.info/smf/index.php?topic=4880.120

There are many other examples...in fact I don't remember seeing a correctly converted quote. I didn't see any errors during the conversion either.

So something is messing with the quote 

Strangely the messages in SMF have kept their open quote tags but lost the close quote tags.

Chris

Can you try this query in phpMyAdmin?

SELECT post_text, REPLACE(post_text, CONCAT(':', bbcode_uid), '')
FROM phpbb_posts_text
WHERE post_id = 32540
LIMIT 1;

-[Unknown]

[packman]

Can you try this query in phpMyAdmin?

SELECT post_text, REPLACE(post_text, CONCAT(':', bbcode_uid), '')
FROM phpbb_posts_text
WHERE post_id = 32540
LIMIT 1;

-[Unknown]

Before:

Code Select Expand

[quote:6932865ed6="Vergil83"][quote:6932865ed6="windy"]
but chris is away again for a week (he is a very busy guy!)[/quote:6932865ed6]
i found chris  :lol:  http://news.bbc.co.uk/1/hi/england/3545218.stm[/quote:6932865ed6]
I told the BBC that I didn't want any publicity so they put someone else's picture on the web site. This is me....

[img:6932865ed6]http://www.weather-watch.com/chris.jpg[/img:6932865ed6]

After:

Code Select Expand

[quote="Vergil83"][quote="windy"]
but chris is away again for a week (he is a very busy guy!)[/quote]
i found chris  :lol:  http://news.bbc.co.uk/1/hi/england/3545218.stm[/quote]
I told the BBC that I didn't want any publicity so they put someone else's picture on the web site. This is me....

[img]http://www.weather-watch.com/chris.jpg[/img]

[packman]

I've just taken a look at some records migrated to SMF that have quotes in them. If I look at a migrated record in SMF that has a quote in it, I get the [ quote="metoo"] at the start of the quoted string but not the [ /quote] at the end of the string.

However, if I modify a message that contains a quote, the editable message text shows the [ /quote] in the correct place. When I save the unedited message back, the quoting is miraculously fixed. Looking at examples of the SMF message record before and after modifying/saving shows...

Before

Code Select Expand

[quote="windy"]i am willing to change the way these numbers are calculated, instead of being :
the coldest or warmest temperature[/quote]

After

Code Select Expand

[quote=&quot;windy;&quot;]i am willing to change the way these numbers are calculated, instead of being :<br />the coldest or warmest temperature[/quote]

So there seem to be two things going on...

1) The double quotes are being translated into &quot's. I've manually modified a record to include &quot and it then seems to display correctly in SMF.

2) End of lines are being translated into <br /> tags.

The first of these appears to fix the quoting problem, but I'm wondering if the second translation will have some currently unidentified impact?

[[Unknown]]

Wait, the converter doesn't change \n to <br />??  It should always do that.

It should also have the &quot;'s in the message - these are expected, and should be parsed fine:

Something...

(in the database, that will show as [ quote=&quot;me&quot; ]Something...[/ quote ])

It looks like it should be converted properly, I'm not sure why it would ever remove the [ /quote ].  The "after" is what *should* be used in the "body" column on the messages table... it should not at all be stripping anything...

As for the fixing, SMF automatically does that when you post/edit...

-[Unknown]

[packman]

The converter code doesn't appear to do very much to the body text of each message. It seems to simply transfer the content of the phpBB message body straight into the SMF body text field after removing the bbcode_uid values. So no &quot or <br /> conversions are carried out.

[[Unknown]]

The converter code doesn't appear to do very much to the body text of each message. It seems to simply transfer the content of the phpBB message body straight into the SMF body text field after removing the bbcode_uid values. So no &quot or <br /> conversions are carried out.

Right, and I just fixed that... however, if it's not doing any of that, it's surely not removing the

's!

-[Unknown]

[packman]

however, if it's not doing any of that, it's surely not removing the

's!

The converted data in the smf_messages file contains both the incorrectly formatted start tag and the correctly formatted end of quote tag.

It's when the messages are displayed by SMF that the code formatting the message for display isn't recognising the incorrectly formatted start of quote tag and therefore displays it as if it were normal message text. The message formatting code does recognise the end of quote tag and presumably doesn't display it. So you see the start of quote tag, but not the end of quote tag.

[[Unknown]]

Ah, I undertstand.  So the [ /quote ] was there, it's just that it's not parsing it.  Okay, that makes sense....

In the converter, find:

Code Select Expand

'body' => "REPLACE(pt.post_text, CONCAT(':', pt.bbcode_uid), '')",

Which should be:

Code Select Expand

'body' => "REPLACE(REPLACE(REPLACE(pt.post_text, CONCAT(':', pt.bbcode_uid), ''), '\\n', '<br />'), '\"', '&quot;')",

Not to mention:

Code Select Expand

'signature' => "REPLACE(user_sig, CONCAT(':', user_sig_bbcode_uid), '')",

Which might also better be:

Code Select Expand

'signature' => "REPLACE(REPLACE(REPLACE(user_sig, CONCAT(':', user_sig_bbcode_uid), ''), '\\n', '<br />'), '\"', '&quot;')",

Does that resolve it?  By the by, how fluent are you in PHP?

-[Unknown]

[packman]

Does that resolve it?  By the by, how fluent are you in PHP?

You're a star...it does solve it 

I've not coded much PHP, but I know the language well enough to act as a debugger for other people's code. I can often identify which bit of code is causing a problem and sometimes suggest a fix. I usually come unstuck with writing fixes because I don't know how the whole system works, e.g. exactly what database fields are used for or exactly how system/API calls function.

I'm from the old school of programming...I've used dozens of languages since I first entered my first bits of binary machine code onto a 256 byte Intel 4004 system way back in 1976. Once you've seen a few languages you can usually make sense of new languages fairly quickly!

[[Unknown]]

I've never done machine code, but I did do a fair amount of assembly... wrote a PCX loader completely in it which I was quite proud of .

-[Unknown]

[packman]

Seriously off topic...

I wrote a RTTY (Radio Telex using Baudot code) terminal program in assembler way back in the early 80's. The machine I was using didn't have timers available so I had to count machine cycles in a timing loop for code to detect and create the two frequencies that were used. I had no way to test it, so once it was written I put it out on air. I was stunned that I could decode other signals and they could decode mine. The only mistake I made was to transpose a couple of letters in my code lookup table.

Real time programming without interrupts was tricky, but fun. For example, I had to use the machine cycles avilable whilst the stop bits were being sent to do all the display/keyboard handling! Luckily RTTY only ran at 45.45baud, so I had around 30mS to do all the user interaction

PHP is easy by comparison 

[[Unknown]]

How would you like to take a whack at some of the other problems in the phpBB converter if I give you where and a basic gist of how to fix them?

It would save me a lot of time, make them get done faster, and if you have the time to do it you seem to be more than knowledgable enough (about phpBB and about PHP) to do so.

-[Unknown]

[packman]

I've started to mod the converter to try to add in the bits that I requested. I've got the code in to migrate the phpBB ranks and just need to add a check to make sure there are some ranks in phpBB before I wipe out the default SMF post based groups.

I'll try adding the other stuff I requested and if I can do that I'll hopefully learn enough about the converter to have a go at tackling any other problems that are awaiting fixes.

I'll get back in touch when I've made some more progress.

Just one question...which I should have asked before I started modding...has any code in the converter been changed since RC1 was released, i.e. am I working from a valid version? I've got the quote tags fix as described in this thread in my version.

[[Unknown]]

You're working on an older version, but I can use the repository to merge in the changes made since...

-[Unknown]

[packman]

At what point is it worth trying the merge?

So far I've added extra steps to migrate phpBB post based ranks and groups (but not added users to groups or set up board permissions using the groups yet). Are the mods in the repository applicable to RC2 (or whatever comes next), or would the converter still work with RC1? I'm just conscious that the work I've done so far is essentially adding new code, but I think I will have to start modding more existing code to add users to groups. Is it worth me working on the latest code from the repository or carrying on with what I'm doing?

[[Unknown]]

All of the changes made since RC1 should be fine to convert even to RC1; so yes, I can merge changes in and it won't conflict with RC2/1.1.

Some of the major changes were described here.  If you want, I can give you a diff file. (or the latest code, although it'd be easier for me to merge your changes in with cvs than otherwise, I'd guess.)

-[Unknown]

[packman]

I made some more mods last night that aren't fully finished/tested yet. When I get to my next stable version perhaps you could do the merge and then let me have a copy of the latest code to continue the mods?

I'm part way through doing the groups/permissions migration. After that I'm going to look at migrating all of the installed phpBB smileys. Hopefully, I'll have enough experience of the code by then to tackle any outstanding bugs.

[packman]

It's probably time for a merge with the repository...and some pointers at known problems to take a look at. So far I've added the following new features...

- Migration of phpBB2 Ranks as SMF Post Based MemberGroups. These are migrated before members are imported so that people end up in the correct groups. The first 5 new groups get 1 to 5 stars and then every other group just gets 5 stars (otherwise things get silly!) Post Based Groups are not given any board access/permissions when the boards are migrated.

- Migrate phpBB2 Groups as non-Post Based SMF MemberGroups. To avoid name clashes the AdditionalGroups are prefixed with 'phpbb2_'. The prefix can be removed manually (if required) after migration.

- Migrate SMF Members into the correct phpBB2 Groups. Note that you'll still need to set up the permissions for the new groups. I've had a brief look at mapping permissions from phpBB2 to SMF and it's quite tricky!

- Migrate phpBB2 avatars as SMF avatars. The SMF Member record is given the correct avatar and all the avatar files are copied from the phpBB2 directories to an appropriate SMF directory. phpBB2 gallery avatars are placed in a directory of the same name under '...smf/avatars'. phpBB2 uploaded avatars are placed in a directory called '.../smf/avatars/phpBB2/'

- Migrated the phpBB user setting that hides the email address from public view

- Migrated the phpBB user setting that allows the user to hide their online status

- Migrated the phpBB user timezone setting

- Inactive phpBB users are now marked as inactive SMF users. The registration processes are different between the two systems, so inactive users will have to be manually activated in SMF if required.

[[Unknown]]

- Migrate phpBB2 Groups as non-Post Based SMF MemberGroups. To avoid name clashes the AdditionalGroups are prefixed with 'phpbb2_'. The prefix can be removed manually (if required) after migration.

I'm not sure I understand; is this only to prevent converting an "Administrator" group?  Unless I'm mistaken, it's perfectly valid to have two groups with the same name.

- Migrate SMF Members into the correct phpBB2 Groups. Note that you'll still need to set up the permissions for the new groups. I've had a brief look at mapping permissions from phpBB2 to SMF and it's quite tricky!

Certainly so... but maybe we/you/us can come up with something .

- Migrated the phpBB user timezone setting

How correctly is this migrated ^_^?  SMF stores it as a time offset, instead of a GMT offset....

Sounds good, though.... you can send it to me at [email protected] (I'd prefer attachments by email) and I'll look at it..

-[Unknown]

[packman]

Whilst phpBB can have a Rank and a Group with the same name, SMF doesn't appear to allow this, i.e. I can't persuade Admin Centre to create a Regular Group and a Post Based Group with the same name. I guess I could probably create two SMF groups with the same name by direct database inserts in the convertor. However, phpBB ranks are purely descriptive, i.e. they can't grant permissions like in SMF, and I've assumed that the additionalGroups will be used for setting permissions, so I thought it safest to avoid any confusion between groups of the same name.

Being a GMT based being, I tend to forget that people do different things with timezones and time offsets  I've removed that for now.

Permissions...I gave up on this until I got a better feel for how SMF permissions work. SMF permissions are a lot more flexible than phpBB permissions, but also spread across a number of tables. I tried working out what went into which table, but got more than a little confused

An email will be on it's way after I post this.

[Burpee]

I'm guessing the following is related to this topic:

http://forum.vuurwerklook.com/index.php/topic,344.0.html

[size=14]Tijdszones[/size]

I'm thinking it doesn't recognize the tags because it doesn't specify px or pt.
So there's two things:
1. What do I do to fix this for my forum?
2. Does the phpBB2 to SMF convertor change this correctly? The size in phpBB is in px by default (I checked)

[[Unknown]]

Oh interesting.  The way to fix that will be to replace:

[size=##]

With:

[size=##px]

But, I don't know a simple way to do that with MySQL (all versions, at least..) off hand.

-[Unknown]

[Burpee]

I'm even more clueless... 
Been trying to do some searches but I have absolutely no clue what to search for

[packman]

Like [Unknown] said, I don't think there is a way to change this directly just using SQL.

I think you'll need to write some code (PHP or Perl) to search the smf_messages table for records that contain the 'size' tag and then do a preg_replace on the body field. I think you'd need...

$body = preg_replace("/\[size=([0-9]?[0-9])\]/i", "\[size=\\1px]", $body);

...but given that this is regexp I'd only trust it after extensive testing!!

[[Unknown]]

Like [Unknown] said, I don't think there is a way to change this directly just using SQL.

I think you'll need to write some code (PHP or Perl) to search the smf_messages table for records that contain the 'size' tag and then do a preg_replace on the body field. I think you'd need...

$body = preg_replace("/\[size=([0-9]?[0-9])\]/i", "\[size=\\1px]", $body);

...but given that this is regexp I'd only trust it after extensive testing!!

* [Unknown] loves regular expressions .

$body = preg_replace('~\[size=([789]|[012][1-9])\]~is', '[size=$1px]', $body);

The reason for this special matching is that we don't want to match, neccessarily, [ size=2 ], which might really be to have that "html font size".  But, most people wouldn't use font sizes lower than 7 anyway... anything higher (not too high!) should be px .

-[Unknown]

[packman]

You beat me to it by 10 minutes. I had to tweak mine a couple of times to make it work. I wasn't a million miles out...I just ended up with a couple of extra slashes that I didn't need! I told you I didn't like regexp

I'll add it into the converter.

[packman]

I've fixed the timeOffset now. It's timeOffset = phpBB_user_timezone - phpBB_board_timezone.

Whilst it's worth getting right, a lot of users seem to be confused about time zones. I've got users from New Zealand who say they're GMT-12 when they should be GMT+12. I've got a user in Belgium who is convinced he's in GMT+13. Interestingly phpBB doesn't seem to support users in the Kiribati Line Islands who apparently live in GMT+14.

[[Unknown]]

Whilst it's worth getting right, a lot of users seem to be confused about time zones. I've got users from New Zealand who say they're GMT-12 when they should be GMT+12. I've got a user in Belgium who is convinced he's in GMT+13. Interestingly phpBB doesn't seem to support users in the Kiribati Line Islands who apparently live in GMT+14.

That's specically why, at least as far as I am concerned, SMF uses the method it does - it's less confusing.  Most everyone can add... but even I sometimes forget which time zone I am in.

-[Unknown]

[Burpee]

* [Unknown] loves regular expressions .

$body = preg_replace('~\[size=([789]|[012][1-9])\]~is', '[size=$1px]', $body);

The reason for this special matching is that we don't want to match, neccessarily, [ size=2 ], which might really be to have that "html font size".  But, most people wouldn't use font sizes lower than 7 anyway... anything higher (not too high!) should be px .

-[Unknown]

Another question
How do I perform this action on my own database?
My guess is to create a .php file, create a mysql query and then... insert this somewhere?

* Burpee feels noob 

[[Unknown]]

Code Select Expand

<?php

include_once('/path/to/SSI.php');

$result = mysql_query("
   SELECT body, ID_MSG
   FROM {$db_prefix}messages}");
while ($row = mysql_fetch_assoc($result))
{
   $body = preg_replace('~\[size=([789]|[012][1-9])\]~is', '[size=$1px]', $row['body']);

   mysql_query("
      UPDATE {$db_prefix}messages
      SET body = '$body'
      WHERE ID_MSG = $row[ID_MSG]
      LIMIT 1");
}
mysql_free_result($result);

?>

You're lucky you only have 5000 posts.... this isn't the most efficient way, mind you.

-[Unknown]

[Burpee]

Warning: mysql_fetch_assoc(): supplied argument is not a valid MySQL result resource in /home/vuurwerk/public_html/forum/sizefix.php on line 8

Warning: mysql_free_result(): supplied argument is not a valid MySQL result resource in /home/vuurwerk/public_html/forum/sizefix.php on line 18

Maybe there's something wrong with the query?

[[Unknown]]

Oops, take out that extra } - it's a typo.

Bad:
   FROM {$db_prefix}messages}");

Good:
   FROM {$db_prefix}messages");

-[Unknown]

[Burpee]

Hmm.... DB is now completely messed up... don't worry tho made a backup
I think that somehow the post ID's are also modified by the query...

[[Unknown]]

I think that somehow the post ID's are also modified by the query...

Strange, they shouldn't be.  Then again, you shouldn't be "kissing" me for an invite.

-[Unknown]

[Burpee]

Strange, they shouldn't be.  Then again, you shouldn't be "kissing" me for an invite.

But... but...