YABb 2.1 conversion errors

[vfrjon]

OK, without the -f and the script filename only has worked!

I added the debug code as advised, but did not see any output from that code.
I got a bunch of errors, all identical which I assume are related to the member records with no birthdays...........

PHP Notice:  Undefined index:  bday in D:\server\www\myserver.dev\public_html\ht
docs\newforum\yabb21_to_smf.php on line 333

Then it displays.....

Successful.
Successful.
Successful.
Successful.
. Successful.
... Successful.
...................................... Successful.
..... Successful.
..................... Successful.
........................ Successful.
..........


as its processing. Thats as far as it had got by the time I had to leave for work, so I will report back on the final results.

One thing I have noticed between doing the conversion on the webserver and doing it locally is the results in the database. As an example, running the conversion on the webserver, the dateRegistered column of the member table is filled out, yet on my local machine it doesnt, its filled with zeros 

[vfrjon]

Well, its been running now for about 8 hours (via Command-line) and no sign of finishing.

CMD window shows pretty much the same as above,

The PHP warnings re undefined index, then

.Successful.
Successful.
Successful.
Successful.
. Successful.
............. Successful.
............................................................................ Successful.
................. Successful.
......................................................... Successful.
.............................................................................................. Successful.
............................................................................... {and continuing on next line}
.............................................................

The last line is increasing by one ' . ' about every 5 minutes.

Seems like this is taking an extraordinary long time to convert seeing it has 387 members, 13,189 posts, 1902 topics.
Should I let it keep running, or time to abort and add some debug code?

[vfrjon]

This is crazy!
Each time I run the convert process, I get a different result.

I cleared the forum tables, reinstalled and re-ran converter via command line. This time it made it all the way through, with a few errors.............

PHP Notice:  Undefined index:  bday in D:\server\www\myserver.dev\public_html\ht
docs\newforum\yabb21_to_smf.php on line 333
PHP Notice:  Undefined index:  bday in D:\server\www\myserver.dev\public_html\ht
docs\newforum\yabb21_to_smf.php on line 333
.
.
LOTS OF THESE
.
.
.
PHP Notice:  Undefined index:  bday in D:\server\www\myserver.dev\public_html\ht
docs\newforum\yabb21_to_smf.php on line 333
PHP Notice:  Undefined index:  bday in D:\server\www\myserver.dev\public_html\ht
docs\newforum\yabb21_to_smf.php on line 333
PHP Notice:  Undefined index:  bday in D:\server\www\myserver.dev\public_html\ht
docs\newforum\yabb21_to_smf.php on line 333
PHP Notice:  Undefined index:  bday in D:\server\www\myserver.dev\public_html\ht
docs\newforum\yabb21_to_smf.php on line 333
. Successful.
Successful.
Successful.
Successful.
Successful.
... Successful.
..... Successful.
... Successful.
PHP Warning:  file(D:\\server\\www\\myserver.dev\\public_html\\htdocs\\cgi-bin\\
yabb/Messages/0.txt): failed to open stream: No such file or directory in D:\ser
ver\www\myserver.dev\public_html\htdocs\newforum\yabb21_to_smf.php on line 1193
........ Successful.
.. Successful.
.................................................... Successful.
Successful.
Successful.
. Successful.
Successful.
Successful.
Successful.
* Recalculating forum statistics... ..Unsuccessful!  MySQL error message:
Duplicate entry '0' for key 1
Converting membergroups...Converting members...Converting settings...Converting
personal messages...Converting boards and categories...Converting mark read data
...Converting topics (part 1)...Converting topics (part 2)...Converting notifica
tions...Converting posts (part 1 - this may take some time)...Converting posts (
part 2)...Converting posts (part 3)...Converting attachments (if the mod is inst
alled)...Cleaning up (part 1)...Cleaning up (part 2)...Converting polls and poll
choices...Converting polls and poll choices (part 2)...Converting poll votes...

D:\server\php>

And now when I try and run new forum/index.php , I get a blank page. No source code, nothing.
The data base still contains similar data to what has been described earlier, with tempID still in the Topics table.

Attached are the convert and Yabbtosmf php files I used

and the command I made on the CLI is
d:\server\php\php.exe D:\server\www\myserver.dev\public_html\htdocs\newforum\convert.php --path_to=D:\server\www\myserver.dev\public_html\htdocs\newforum --path_from=D:\server\www\myserver.dev\public_html\htdocs\cgi-bin\yabb --convert_script=yabb21_to_smf.php --db_pass=root --debug

[SleePy]

Give this yabb file a try.

[vfrjon]

D:\server\php>php.exe D:\server\www\myserver.dev\public_html\htdocs\newforum\con
vert.php --path_to=D:\server\www\myserver.dev\public_html\htdocs\newforum --path
_from=D:\server\www\myserver.dev\public_html\htdocs\cgi-bin\yabb --convert_scrip
t=yabb21_to_smf.php --db_pass=root --debug
PHP Notice:  ob_end_clean(): failed to delete buffer. No buffer to delete. in D:
\server\www\myserver.dev\public_html\htdocs\newforum\convert.php on line 2519
Array
(
   

• => D:\server\www\myserver.dev\public_html\htdocs\newforum\convert.php
      [1] => --path_to=D:\server\www\myserver.dev\public_html\htdocs\newforum
      [2] => --path_from=D:\server\www\myserver.dev\public_html\htdocs\cgi-bin\yab
  b
      [3] => --convert_script=yabb21_to_smf.php
      [4] => --db_pass=root
      [5] => --debug
  )
  
  
  
  Converting membergroups.... Successful.
  Converting members...PHP Notice:  Undefined index:  bday in D:\server\www\myserv
  er.dev\public_html\htdocs\newforum\yabb21_to_smf.php on line 334
  PHP Notice:  Undefined index:  bday in D:\server\www\myserver.dev\public_html\ht
  docs\newforum\yabb21_to_smf.php on line 334
  PHP Notice:  Undefined index:  bday in D:\server\www\myserver.dev\public_html\ht
  docs\newforum\yabb21_to_smf.php on line 334
  >>
  >>
  lots of these............
  >>
  >>
  >>
  PHP Notice:  Undefined index:  bday in D:\server\www\myserver.dev\public_html\ht
  docs\newforum\yabb21_to_smf.php on line 334
  PHP Notice:  Undefined index:  bday in D:\server\www\myserver.dev\public_html\ht
  docs\newforum\yabb21_to_smf.php on line 334
  PHP Notice:  Undefined index:  bday in D:\server\www\myserver.dev\public_html\ht
  docs\newforum\yabb21_to_smf.php on line 334
  PHP Notice:  Undefined index:  bday in D:\server\www\myserver.dev\public_html\ht
  docs\newforum\yabb21_to_smf.php on line 334
  PHP Notice:  Undefined index:  bday in D:\server\www\myserver.dev\public_html\ht
  docs\newforum\yabb21_to_smf.php on line 334
  PHP Notice:  Undefined index:  bday in D:\server\www\myserver.dev\public_html\ht
  docs\newforum\yabb21_to_smf.php on line 334
  . Successful.
  Converting settings... Successful.
  Converting personal messages.... Successful.
  Converting boards and categories... Successful.
  Converting mark read data.... Successful.
  Converting topics (part 1).......... Successful.
  Converting topics (part 2)........ Successful.
  Converting notifications....... Successful.
  Converting posts (part 1 - this may take some time)...PHP Warning:  file(D:\\ser
  ver\\www\\myserver.dev\\public_html\\htdocs\\cgi-bin\\yabb/Messages/0.txt): fail
  ed to open stream: No such file or directory in D:\server\www\myserver.dev\publi
  c_html\htdocs\newforum\yabb21_to_smf.php on line 1190
  ......... Successful.
  Converting posts (part 2)..... Successful.
  Converting posts (part 3).......................................................
  .................................. Successful.
  Converting attachments (if the mod is installed)... Successful.
  Cleaning up (part 1)... Successful.
  Cleaning up (part 2).... Successful.
  Converting polls and poll choices... Successful.
  Converting polls and poll choices (part 2)... Successful.
  Converting poll votes... Successful.
  * Recalculating forum statistics... ..Unsuccessful!  MySQL error message:
  Duplicate entry '0' for key 1
  
  D:\server\php>
  

[vfrjon]

Sleepy, above are the results of the last convert with the file you supplied.

[SleePy]

Well I can fix the bdary error easily

Does the forum run or work?
Does things work if you go to the Admin panel (will need to login) and then Forum Maintenance. Running the "Find and Repair Errors" and "Recount Totals and Stats" tools?

[vfrjon]

No, I cant get to the forum.

When I go to www.myserver.dev/public_html/htdocs/newforum/index.php I get a blank page. There is a pause while loading and looking at the 'page source' it displays no code. If I clear the tables, install the board again, it works with a new forum instal.

[vfrjon]

I imported the local database to the web based forum to see if I could access the forum. It worked.

When I log in, I get............

The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET SQL_MAX_JOIN_SIZE=# if the SELECT is okay
File: mywebsite/newforum/Sources/BoardIndex.php
Line: 85

Running report errors, I get

The following errors are fouling up your forum:
Found topics and/or messages with topic or message IDs of 0.
Message #1 is in non-existent topic #1.
Message #1 is in non-existent topic #1.
Cached word 'smf' is linked to a non-existent topic.
Cached word 'to' is linked to a non-existent topic.
Cached word 'welcome' is linked to a non-existent topic.

Repair errors "YES"........

The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET SQL_MAX_JOIN_SIZE=# if the SELECT is okay
File: mywebsite/htdocs/newforum/Sources/RepairBoards.php
Line: 173

smf_topics still contains data filled TempID
smf_members MemberID's are all still '0'
___________dateRegistered are all '0'
___________no PM infos

                       

[vfrjon]

Would it be worth upgrading to  Yabb2.2 and then converting that?

[SleePy]

I am guessing that the topic ids and message ids didn't get linked up right or fixed correctly.

I need to think of an idea to fix this, I don't understand why it has worked so well in the past and then recently we have had all these issues just appear. I know the SMF database for 1.1 hasn't changed since it was released as final for 1.1.

[vfrjon]

Sleepy appreciate all your help,

Ive created a new Yabb2.2 board, and updated all the data from the old board to it, so I still have a working copy of yabb2.1 if required.

Ive tried converting yabb2.2 to SMF with the following error............

Converting membergroups... Successful.
Converting members...
Notice: Undefined index: bday in mywebsite/htdocs/newforum/yabb22_to_smf.php on line 333
Notice: Undefined index: bday in mywebsite/htdocs/newforum/yabb22_to_smf.php on line 333
Notice: Undefined index: bday in mywebsite/htdocs/newforum/yabb22_to_smf.php on line 333
Notice: Undefined index: bday in mywebsite/htdocs/newforum/yabb22_to_smf.php on line 333
(lots of these)
Notice: Undefined index: bday in mywebsite/htdocs/newforum/yabb22_to_smf.php on line 333
Successful.
Converting settings... Successful.
Converting personal messages...
Notice: Undefined offset: 7 in mywebsite/htdocs/newforum/yabb22_to_smf.php on line 485
Notice: Undefined offset: 7 in mywebsite/htdocs/newforum/yabb22_to_smf.php on line 492
Notice: Undefined offset: 7 in mywebsite/htdocs/newforum/yabb22_to_smf.php on line 485
Notice: Undefined offset: 6 in mywebsite/htdocs/newforum/yabb22_to_smf.php on line 491
(Lots of these)
Notice: Undefined offset: 7 in mywebsite/htdocs/newforum/yabb22_to_smf.php on line 492
Notice: Undefined offset: 7 in mywebsite/htdocs/newforum/yabb22_to_smf.php on line 485
Notice: Undefined offset: 6 in mywebsite/htdocs/newforum/yabb22_to_smf.php on line 491
Successful.
Converting boards and categories... Successful.
Converting mark read data... Incomplete.
Converting boards and categories... Successful.
Converting mark read data... Incomplete.
Converting topics (part 1)... Successful.
Converting topics (part 2)... Unsuccessful!
This query:

    DELETE `vfr_smf_forum`.vfrforum_log_topics
    WHERE ID_TOPIC = 2;

Caused the error:

    You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'WHERE ID_TOPIC = 2' at line 2

Looking at the tables on the db, it seems to have filled out a lot more info. For example, Members all have their own unique ID, Registration dates are filled in

Should we continue trying to convert 2.2 or go back to 2.1

I really need to get this board running.


yabb2.2tosmf.php and converter were downloaded from http://www.simplemachines.org/community/index.php?topic=249777.0

[vfrjon]

OK, that was an easy fix, should have been DELETE FROM, not DELETE

Converter now finishes only displaying the errors as shown in above post about birthdays and undefined offset.
I can log in to the forum, topics and post are there but are showing as having no replies, altho they do display.

When I try to run fix and repair in maintanance, It reports errors about

Topic (numbers 1561 to 1900) contains no (actual) messages.
Topic #352 has the last message ID 12656, which is incorrect.
Topic #504 has the last message ID 11655, which is incorrect.
Topic #969 has the last message ID 11498, which is incorrect.
The subject of topic number 2 to 1460 is currently not stored in the subject cache.
Poll #1 has been given a vote from member #0 , who is now missing.

Trying to fix these, causes a long pause, then a blank screen, with

http://www.myserver.dev/htdocs/newforum/index.php?action=repairboards;fixErrors;sesc=357ca78d2515cf88190ea0e5b1e49409

in the address abr of the browser.

[vfrjon]

Ran the converter again, and this time with another result............SUCCESS!!

Sort of........

It made it all the way through and when I went to Admin panel and ran "Find and fix errors", I got the expected BIG_SELECTS error, so I edited the sources/RepairBoards.php and with a bit of experimenting found where to place the " db_query("SET @@SQL_BIG_SELECTS = 1", __FILE__, __LINE__);   "
Found and fixed all the errors and now the board appears to be running OK with all users and posts visable.

Only problem I can see so far is the PM's
The table on the data base (XXXX_personal_messages) is all screwed up
This is a sample of what it shows........

ID_PM       ID_MEMBER_FROM       deletedBySender       fromName       msgtime       subject  body       temp_toName
   
1            0                                1                            Hey Hey        0                   60.231.40.44         FlyGA
   
2            0                                1                            Hey Hey        0                   210.10.229.143         thetonester

So obviously when you display the PM's the data doesnt relate to the corresponding column
Is there anyway to repair this, perhaps run only a section of the convert again to extract the correct data which can then be imprted to the database as I really dont want to have to run the complete convert again in case I dont get a happy result again.

[SleePy]

Well, We can cut up the converter here most likely and try to have it just do pms since we have had luck so far.

It looks like the pm conversion is a little out of sync though for converting the data correctly.
Give this file a try, it will only convert pms. This is from the yabb22_to_smf.php file. Hopefully it is converting the pms correctly. If it isn't, then I might need a another sample pm file to make sure I am reading the text file right.

[vfrjon]

Thanks!

Running this script, I get this.......

Notice: Undefined offset: 7 in mywebsite/forum/yabb22-pms_to_smf.php on line 162
Notice: Undefined offset: 7 in  mywebsite/forum/yabb22-pms_to_smf.php on line 169
Notice: Undefined offset: 7 in  mywebsite/forum/yabb22-pms_to_smf.php on line 162
Notice: Undefined offset: 7 in  mywebsite/forum/yabb22-pms_to_smf.php on line 169
Notice: Undefined offset: 7 in  mywebsite/forum/yabb22-pms_to_smf.php on line 162
Notice: Undefined offset: 7 in  mywebsite/forum/yabb22-pms_to_smf.php on line 169
Successful.
Recalculating forum statistics... Unsuccessful!
This query:

    SELECT t.ID_TOPIC, COUNT(m.ID_MSG) AS numMsg
    FROM `vfr_smf_forum`.vfrforum_topics AS t
    LEFT JOIN `vfr_smf_forum`.vfrforum_messages AS m ON (m.ID_TOPIC = t.ID_TOPIC)
    GROUP BY t.ID_TOPIC
    HAVING numMsg = 0
    LIMIT 0, 200;

Caused the error:

    The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET SQL_MAX_JOIN_SIZE=# if the SELECT is okay

Big_Selects is not a major issue, I just need to find where to shove the " db_query("SET @@SQL_BIG_SELECTS = 1", __FILE__, __LINE__);   " like I did in the RepairBoards.php (Or perhaps you could tell me  ) Either that or I run repair / recount etc from Admin.


Its partially worked............

I have PMs in my inbox, same number as in YABB.

Looking at my .msg file in YABB,

1198758920|rpleum|jon|||Circuit|1198758919|Hi,<br /><br />Could you send the electronic circuit with all the IC's to [email protected]?<br />thanks a lot! |1198758920|0|77.249.185.230|s|r||

Nothing in my Outbox

Looking at my .outbox file in YABB,

1198808115|jon|rpleum|||Re: Circuit|1198808114|Sure, I think I have one I drew that starts at the USB input, and goes thru to the 40 outputs to the matrix. Its on a hard drive at home, so I will dig it up and post it on the forum. <br /><br />Ive decided to create a new forum for Hardware (at present its hidden whilst in developement) but should be up soon.<br /><br />Jon|1198808115|0|124.186.41.133|s|||

Looking at another members .msg file in YABB,

jon|Re: Westsim|1179141939|Thanks Josh|1179141941|211.26.76.166

Looking at same members .outbox file in YABB,

joshstone|hi|1198413076|hi|1198413077|121.221.193.9|1

So it seems there are 2 file formats in the folder.

Bear in mind that I have just converted from YABB2.1 to YABB2.2 but just copied the Members and Messages folders to the new install.  Im going crazy trying to work out what happens here, which is why I have edited this post so many times! Perhaps the 2 versions of YABB use different formats, and it converts them the first time you access them in YABB2.2.............



temp_toName still contains data in the XXX_personal_Messages table
and the rest of the data is all out of order
The First 430 or so are like this

ID_PM   ID_MEMBER_FROM  deletedBySender  fromName              msgtime  subject                 body        temp_toName
93          0                         1                      Re: Westsim            0            211.26.76.166                     joshstone

The rest (about a 60) or so are like this. They are all mine, which leads me believe that they ARE being changed by YABB as these files were placed into the directory from a backup tonight.

ID_PM   ID_MEMBER_FROM  deletedBySender  fromName        msgtime         subject     body                                                                       temp_toName
496       313                        1                     rpleum             1198758919    Circuit      Hi,<br /><br />Could you send the electronic circu...     jon

The forum is locked, so I am the only one who has been there, and I have checked my PM's in YABB. I will go and compare the files from the backup.................

My backup from the YABB2.1 board has this format
Member|subject|timestamp|body|another timestamp|IP Address
And the same file from YABB2.2 has this format
Timestamp|Member(Author)|Member(recipient)|another timestamp|body|another timestamp|IP Address||(s)|(r)|(null)|(null)]


They may not be timestamps, they may be post references. Not sure what  they are, but I guess you will   

I had a look at the outbox folders and they too change with the YABB update in the same way.

Perhaps the converter needs to look at the data, decide what format it is in, then process it accordingly.

More than happy to zip up and email some of the complete files (.outbox .msg .ims etc) of a couple of members for you to look at, just let me know which ones you want. Or, I could give you ftp access to the YABB folder on the server.



Jon

[SleePy]

Ok, those to formats wont make the flat file conversion very easy to do
That will be a pain to even work with really.

Some of those messages don't even have the no smiley information (the new one may have it in |s|).
The only way It may work is if we can detect the pm formats.

Are the PMs this important? It sounds like you have everything else. Is it going to be worth it to try and dig this around with these two formats to get the pms converted?

[vfrjon]

I was wondering if it would be easier for me to restore the data files from the earlier version (The one that starts with the member name and try and convert those as they 'should' all be the same format.) My members have been advised that any PM's they may send in the short term may be lost, but I would like to retrieve them and have the conversion complete.

I may be wrong, but detecting the PM format would be easy wouldnt it? First field a number or a word??

Also, if my assumtions are correct in the fact that the format changes with a YABB upgrade, then this may cause problems for others converting to SMF. If another board has gone through the same upgrade process Yabb1 to Yabb 2.1 to Yabb 2.2 then some of the PM's in their files are likely to have different formats also.

What about the outbox issue being empty? Any ideas there?

Jon

[SleePy]

Well the only problem with detecting the format is knowing what to detect and knowing what order the information is in. I see from your example of another members .msg file, there is no information about using smiles or not. I see the username, I see the ip, subject and body. There is two timestamps but I am not sure which one is the right one to use.

This gets to be the pain really for converting anything from flat file.

I heard btw that Yabb 2.2 supports MySQL? Do they have a converter to go to MySQL? I wonder about maybe using that tool to simply things.

[vfrjon]

I think YABB3 has SQL support, but as its in Alpha, there does not appear to be a download. Package claims it will have a converter tho, but not due for release till later this year.

Looking at a post in both formats of the .msg file, The extra timestamp is only 1 (second? or millisecond?) different, but the current converter actually converts these messages correctly.

This one from the old format,

Liquid|hey mate|1202467792|Hi mate, .....(content edited out).....take care mate..<br /><br />Lance|1202467793|124.185.36.137

and from the new (same message) which works, with the current converter

1202467793|Liquid|jon|||hey mate|1202467792|Hi mate, .....(content edited out).......<br /><br />Lance|1202467793|0|124.185.36.137|s|||

Would it be possible to make a converter that deals with the old format ? I will then put my original .msg file back so they are uniform.

Any idea why none of the sent ones convert? (The ones in the .outbox file)

Jon