Related Topics

[karlbenson]

Following BigMikes comments.

I can confirm the exponentially increased load.
My forum has almost 350,000 posts, 20,000+ topics, 1600+members on a VPS/VDS server.

With this modification on my forum the load increased x300 (not 300%, 30000%)
Topics which took less than 0.1 seconds to be created took 30+ seconds.

I had to remove the modification as it was killing my forum, since then, all page times have gone back to normal

I wouldn't recommend people install this modification on medium/large or open/public forums.

---
Furthermore BigMike, regarding your suggestion to pre-fetch related topics, I do not believe that would help your community as a whole. The problem is the related topics search itself is VERY memory intensive, so it would slow down the pages for everyone else.

Moving forward, I think it may be an idea to reduce the searching capacity, possibly based on searching subject to subject or first posts only.  It may not be as accurate, but it might be the only realistic way to use a related topics mod on large forums - question, how do big forums on other forum software do it much better and faster than this???

[shaiss]

is there an example of this anywhere?

[BigMike]

karlbenson: I was thinking about having a javascripted drop down box, much like the "Additional Options..." field on the reply/post template, in which once clicked on, it will run the search query and then display the results per user in an expanding div or span, whatever SMF uses.

This way, the search requests will only be called upon 1 at a time by real life individual members. This could also be restricted to members only to 1) reduce the amount of those potentially using the feature and 2) an extra incentive for lurking guests to finally make the jump to a member.

As for your suggestions regarding searching just the subject or the opening post, I think that would be a good idea. Is this how the VBB boards are doing it? It must be a huge load for their servers as well and yet they are doing ok as far as I can tell.

Regards,
BigMike

[BigMike]

is there an example of this anywhere?

EDIT: I posted a link to my test forum, but then I remembered that my test forum does not allow guest viewing.

Here is a screen shot of the Related feature in action on a fresh 1.1.3 install:

[BigMike]

I was thinking about having a javascripted drop down box, much like the "Additional Options..." field

Please review the following examples. I have not coded this, I simply modified the HTML for this example to illustrate my idea. The first image is the "Related Topics" field collapsed, the second image is the field expanded with the search in progress, and the third image shows the results of the search.

The idea is that no searching occurs until a user clicks on the "+ Related Topics..." link. Once this link is selected, then the database is searched for any related topics. Finally, after 5 or 10 seconds, or whatever, the results of the database search are displayed for the user to review.

This way hopefully far fewer Related Topic Searches are occurring at any given time.

So what do you guys think? Is this even possible?

[Sakae]

That would be just awesome. I think the major problem of this modification is the bandwidth it "sucks" from the server, for too many topics.

If we could isolate this problem, requesting related topics only in some rare occasions, it would work just fine, I think.

Anyway, I don't know if it's possible because I don't know how to code PHP. But I hope someone hear our voices..

[Guy Verschuere]

Is there a way to download this latest modifications? When will the mod be updated?

[BigMike]

Is there a way to download this latest modifications? When will the mod be updated?

This mod was last updated on May 31st by karlbenson, and is available through the following link:
related_topics_1.1.1 (modified).zip

For installation instructions, I put together a step-by-step procedure on Page 11, Reply #218

[glennk]

So have you found a way of having users search for related topics or some other way of decreasing server load ?

Sounds a good mod, I have similar working in a wordpress news site. but my bandwidth is expensive so dont want heavy server loading.

[BigMike]

I am willing to dive into some coding, but I first would like to know if I can run complex code from the template files? Other wise I'll need to learn how to make function calls across to the source files, which I am sure can't be too hard

[Guy Verschuere]

@BigMike: Thanks, it works.

[Sakae]

I am willing to dive into some coding, but I first would like to know if I can run complex code from the template files? Other wise I'll need to learn how to make function calls across to the source files, which I am sure can't be too hard

Just drown man. I'm pretty sure we can convince the mod's owner to add your update at SMF mods db.

[Farmacija]

ok, i didn't read all 12 pages and maybe someone suggest this already but-it would be much better if it colud be that related topics was just under topic title so when member write topic title , he get the list of similiar topic under box.
see


I think it is worked in ajax

[BigMike]

Yes, that looks very nice.

Perhaps having an option as to where to display the Related Topics would be a good feature.
I personally prefer to have it at the bottom of a thread.

I like the green percentage bars

[Farmacija]

yes, it is from one of biggest serbian forum

[Chriss Cohn]

There is one additional idea from me to speed up things a bit....
Once a user/guest (depends on who you allow it) has recieved the realted topics, it should be stored in some sort of cache (done by File or else) so that the next member/guest inside the same Thread, which will also search for realted topics, retrive that cached results....
A specificable "time to live" in addition to caching would also be nice

There is a Mod out there for mambo/joomla from which you can take some impressions, because file-storing cache works very well with it. The name is "page-cache by IRCmaxell"

I think caching would be the only solution for this ressource-eater.....

Regards, Christian

[niko]

Here's new version of those who want to try it.

New features:

Settings to enable/disable, search method, count of how many related topics to show. Settings will be in Features and Options.

Search method allows you to choose between old fulltext and custom (which also uses fulltext).

Custom should work better as it saves results to database. Which prevents related topics disappearing. Currently there's no way to create custom index. (Replies will make results inaccurate).

Example: http://www.madjoki.com/smftest/index.php?board=1.0

[glennk]

IS the problem solved ? I fear it will zap my bandwidth ?

Also get an error

1.     Extract File     ./Sources/Subs-Related.php     
2.    Execute Modification    ./Sources/ModSettings.php    Test successful
3.    Execute Modification    ./Sources/RemoveTopic.php    Test successful
4.    Execute Modification    ./Sources/Subs-Post.php    Test successful
5.    Execute Modification    ./Sources/Display.php    Test successful
6.    Execute Modification    ./Themes/default/Display.template.php    Test failed
7.    Execute Modification    ./Themes/default/languages/Modifications.english.php    Test successful
8.    Execute Code    related_topics.php

[Chriss Cohn]

Custom should work better as it saves results to database....

So this is some kind of "results-caching" as i mentioned above.......
So when 2.0 with that file-caching comes out, will it cache even that querys the results of related topics?

Regards, Christian

[niko]

In a way, yes.
I have not yet updated related topics to work with 2.0, but there is no caching used apart from mysql table.

I updated this version (1.2 Beta 2) to use new method to use fulltext index (subjects are copied). It provides more accurate results as replies has almost always same subject and if there's some words used too many times, it won't be used.

To use this update to this from previous beta, you must drop table smf_related_topics. After that remove mod and install this. Then go to Features and Options and click "Build index for custom method" (or navigate to http://www.yourdomain.com/forum/index.php?action=related;sa=index;begin)

It will copy subjects to new table & run fulltext query against every topic  and inserts results to smf_related_topics. It will cache only 25 relations as with no limits index was 6 MiB for 2800 topics. (About 150 k rows) Currently it uses 2 queries per topic view but I will reduce it to one.

I will think about caching also

ok, i didn't read all 12 pages and maybe someone suggest this already but-it would be much better if it colud be that related topics was just under topic title so when member write topic title , he get the list of similiar topic under box.
see


I think it is worked in ajax

Good idea.