SMF2's limit on # of boards

[moonjo]

What is the limit on the number of boards for SMF2?
I'm planning to use it as an intranet mini CMS; there will be quite a bit of sub-boards compared to normal forum.
Can it go into hundreds, thousands?
And I'm assuming number posts allowed is high enough.
Thanks.

[Nathaniel]

Theoretically 65535 is the maximum number of boards you can have (because of the board id column in your 'smf_boards' table), but if you have really large numbers of boards then you may have performance issues, because SMF wasn't really designed to cope with hundreds or thousands of boards - especially if you have thousands of topics/messages as well.

You will probably have to use some trial and error to see how well it will work.

[birchy]

I have the same situation. I modified the boards columns from smallInt to bigInt and I've added 73 categories, about 27,000 level 0 boards, and 6 child boards each to bring the total count to almost 173,000 boards, using a php script and mySql, of course. Until I can hack the main board display page to default to collapsed categories, I added 73 records to the collapsed categories table for my user id.

The issue that is insurmountable is that the admin board page is too large to open. I'm willing to replace that page's functions with my own scripts, or mod the page to use collapsible categories also.

In defense of the barrage of beratings I'm sure to receive, my users never need to see all the boards listed. Most of the time they will only see a list of 7 boards at a time. I think if I can hack the default collapsed settings and hack it so that only two categories can be uncollapsed at a time, all should be managable. I don't know what other issues will surface until I can run tests.

Can some kind developer save me a few hours and point me to where I need to go to make those hacks?
Looks like Subs-BoardIndex.php around lines 100+ are involved. I really need the Collapsed Categories table to reverse to be an Expanded Categories table (swapping default from Expanded to Collapsed).
Has anyone else pushed these limits and if so, what other issues did you encounter?
I'm sure that I'll have to convert fields in other tables to bigints also...

[Arantor]

If you haven't touched the other tables, you're already in dire straits, you need to modify all the other places where id_board is used (at the very least, messages, topics, log_mark_read, log_notify, and no doubt a bunch of others) - mediumint would have been big enough though.

There will be a fairly serious performance hit doing this; you're really not meant to have that many boards - I cannot think of a single case where you'd actually need that many boards.

Subs-BoardIndex.php really just fetches the data, ManageBoards.php is more about handling the management with ManageBoards.template.php being the display of it which is where you're looking first, I think.

[birchy]

If you haven't touched the other tables, you're already in dire straits, you need to modify all the other places where id_board is used (at the very least, messages, topics, log_mark_read, log_notify, and no doubt a bunch of others) - mediumint would have been big enough though.

There will be a fairly serious performance hit doing this; you're really not meant to have that many boards - I cannot think of a single case where you'd actually need that many boards.

Subs-BoardIndex.php really just fetches the data, ManageBoards.php is more about handling the management with ManageBoards.template.php being the display of it which is where you're looking first, I think.

The tables that I found were:
boards: id_board, id_parent, board_order => bigInt
log_actions: id_board
log_boards: id_board
log_mark_read: id_board
log_notify: id_board
log_reported: id_board
messages: id_board
message_icons: id_board
moderators: id_board
topics: id_board, id_previous_board

I believe the Board Permission page is another admin page that won't load due to the volume.

As for exceeding the design limits of the development team, it should be no surprise that applications come along that don't fit the mold. I've actually pared my list of boards down from what I would like and what there probably will be in a few years. It's a large scale site, but I have to grow it with little-to-no resources. When ad revenues pick up, I'll hire someone like yourself to rewrite the site the way it should be, probably using smf as the base, instead of relying on a Frankenstein of kludged open-source packages and the half-baked homebrew code, that is the extent of my skill level.

[Arantor]

That looks like the entire list, yes.

As for exceeding the design limits of the development team, it should be no surprise that applications come along that don't fit the mold.

Naturally, that comes with any application, especially as people find ever more creative ways to use the software. I still cannot conceive of a situation where you need 173k boards, unless you're talking about *well* into the millions of posts.

In which case, you're going to have issues; SMF has been proven to scale to 7m posts or so with proper tuning and careful configuration, but it needs a much more drastic overhaul underneath to scale to the number of posts you really need to populate 173k boards. Even with 100 posts per board, that's going to look desolate (even if people can only see a fraction) but 100 posts per board = 17.3m posts, over twice the size of the largest known SMF board at the present time.

I'll hire someone like yourself to rewrite the site the way it should be, probably using smf as the base

I can guarantee it won't be me, but the principle stands. I already know how much work needs to be done to SMF to modernise its code and make it more scalable than it currently is.

instead of relying on a Frankenstein of kludged open-source packages and the half-baked homebrew code

Don't use any SMF mods then, since an unfortunately high number of them are such. It's not the fault of the modders', they're not generally professional level coders, and the fact that the mod system relies on modifying core files to achieve things (though that is changing) means you're essentially working with a system that is kludged open-source packages.

[birchy]

@Arantar: To shed some light on why my site needs so many boards, Forums are one of four major sections. The subject matter is hometowns and the current scope is the US, territories and Canada. That is some 200k localities, about 27K towns with a population of 500 or more. So each town has a main board and the six child boards. I've set the 73 states/provinces/territories as categories. If I can get the board index to default to collapsed categories, the overhead would still be large, but managable. Most users would never see the board index, they would follow links to the parent boards. I would prefer to have the board index page display the general purpose boards and a town lookup form only. Any one user would be interested in only a handful of those boards. Most boards will only have a few posts in them over the next couple of years. The goal is to scale up a successful model for a single-town board that serves a town of less than 10k population that is consistently getting about 6 new registrations per day, and a few hundred posts per day. Other sections include an alumni section (40 million names, 40k schools, 650 visitors per day and I haven't even launched yet), a news/events/obits/weather/classified ads section, and a history/genealogy section. An ambitious project for a shoestring budget. Operating off the Field of Dreams concept: "if you build it, they'll click on your ads"

[Arantor]

So you're still essentially talking about trying to build the single largest SMF forum ever constructed. Good luck, you will need it.

[MrPhil]

Wow! I think you're going to bring any system short of Google's server farm to its knees in pretty short order, once this thing gets going. Let's step back for a moment and think of other ways to do this. Have you thought about a custom directory page (perhaps an image map and/or pulldown lists) to direct visitors to one of 73 separate SMF forums with separate databases? Do the people in one state need to see the posts and such of another state? Once you get big enough, you might even go to a second level of directory pages for counties. That ought to keep your database size under control until you become the new Facebook.

[birchy]

@Arantor: yes, I will. Thanks!
@MrPhil: Yes, a custom directory page is needed. Grateful for any links towards that end.

I've assigned the smf id_board to the Towns table used in the non-forums part of the site. So I will want to use that to SSI?/API? recent topics lists to the related pages of the non-forums pages, or more likely just write code to access the smf topic/posts info myself.

Some users will live in one or two parent boards, some genealogists will visit maybe a few dozen boards, mostly to post and to await replies. The entry for most will be the town portal pages, a town lookup/search form, by drilling down a breadcrumb on the alumni pages, or by chasing links in the surname registry. Most users won't come to the site to use the forums initially, but the goal is to have a certain percentage return and camp out in the forums for their current town or for their hometown, or both. For these heavy use users, the Town Portal page will still be their entry point. There they can see a summary of the hot/recent topics and click through to the forums. (They can also read local and national news, local and national obits, weather, events, but most will click through to the forums from there. They can toggle between "favorite" towns on the portal page, as well. Oh, and there is a wiki, also.

Another way to think of it, is as 27,000 forum sites under one umbrella.

If anyone is curious, the one-town site that is a partial model for the forums and some parts of the portal is: MilesCity.com
My site is: http://www.tree52.com/town.php?ct=11150 (for example)
If you want to see how far I've taken the forums section, go directly to the login page and sign in as "smf", pwd="showme" :
http://tree52.com/forums/index.php?action=login
(If you go to the index page as a guest, the page won't load)

[Road Rash Jr.]

A man after mine own heart LOL I too have an international community but rather than plugging a 230V system into a 110V socket I've taken another route by regionalizing every country with one forum per region. So far I have 25 separate forums. This keeps the work load at a more reasonable rate so that even if one or several forums are under heavy usage, it doesn't drag the others down.
If you are planning on having many users, the biggest downside for them in your setup will be the lag time, and many possible "system unavailable" errors. Too many of these and they will go elsewhere. That and if you don't have your own private server system, you will quickly out live your welcome on any shared server system.

[Road Rash Jr.]

I was just on your forum and assuming I was the only one there, a 10 second page load will put many user to sleep. Add more users on at one time and the lag will be exponential and kill the server.

[birchy]

I was just on your forum and assuming I was the only one there, a 10 second page load will put many user to sleep. Add more users on at one time and the lag will be exponential and kill the server.

I assume that you're referring to the Forum Board Index page, the one I'm trying to modify...
Your point is noted, but as I've explained, that page will hardly ever be seen by users, if ever. At least in this state.

[birchy]

ManageBoards.template.php being the display of it which is where you're looking first, I think.

From what I can see, ManageBoards.template.php (in the default theme) doesn't do anything to reference the collapsed settings. They seem to arrive to that script already loaded in the $context['categories'] container. Any idea where that container is loaded?

It looks to me like all the heavy lifiting is in the query in Subs-BoardIndex.php, about line54.

...LEFT JOIN {db_prefix}collapsed_categories AS cc ON (cc.id_cat = c.id_cat AND cc.id_member = {int:current_member})' : '')) . '...

This is a little too complex to get my head around, so if anyone can show me how to modify this query to set all categories to collapsed for guests, I'd luv ya forever.

At least a ten second load is better than no load at all at this stage... first things first.

[birchy]

I guess this is a "never mind" post. I think I actually hacked a mod that seems to work:

(Subs-BoardIndex.php, about line54)

   $hackStr = $user_info['is_guest'] ? '1 AS is_collapsed,' : 'IFNULL(cc.id_member, 0) AS is_collapsed,' ;
   $result_boards = $smcFunc['db_query']('boardindex_fetch_boards', '
      SELECT' . ($boardIndexOptions['include_categories'] ? '
         c.id_cat, c.name AS cat_name,' : '') . '
         b.id_board, b.name AS board_name, b.description,
         CASE WHEN b.redirect != {string:blank_string} THEN 1 ELSE 0 END AS is_redirect,
         b.num_posts, b.num_topics, b.unapproved_posts, b.unapproved_topics, b.id_parent,
         IFNULL(m.poster_time, 0) AS poster_time, IFNULL(mem.member_name, m.poster_name) AS poster_name,
         m.subject, m.id_topic, IFNULL(mem.real_name, m.poster_name) AS real_name,
         ' . ($user_info['is_guest'] ? ' 1 AS is_read, 0 AS new_from,' : '
         (IFNULL(lb.id_msg, 0) >= b.id_msg_updated) AS is_read, IFNULL(lb.id_msg, -1) + 1 AS new_from,' . ($boardIndexOptions['include_categories'] ? '
         c.can_collapse, '.$hackStr : '')) . '
         IFNULL(mem.id_member, 0) AS id_member, m.id_msg,
         IFNULL(mods_mem.id_member, 0) AS id_moderator, mods_mem.real_name AS mod_real_name


First bold item is new code, second is replaced code.

[Aleksi "Lex" Kilpinen]

Quite honestly, I fear you would need a whole server farm to run something like that, and the way SMF works is really not meant for something like that...

9: Do you really need that many boards? I tend to see this rather often and the answer is almost always no. Too many boards actually reduces your forum's activity, for one, but in the case of optimization here, each and every board takes up resources on the relatively common index load, as well as space for the log_boards and log_mark_read tables, with an amount that grows for each new member you gain.

So in other words, your database will build up to be huge in relativily no time at all, assuming the boards would really become active...

EDIT:
If you are determined to do something like this, have you concidered cutting it down to smaller areas?
Like have a subdomain, and a forum of it's own, for each state for example?

[birchy]

Well, we'll see. I've hit another snag. A few Maintenance tools seem to choke on the size of the boards table, as well as the about three admin pages mentioned before.

[Aleksi "Lex" Kilpinen]

On a sidenote, your bot challenge doesn't work very well LOL - I answered some 10 questions there just now, and they keep on coming - I'll never get past that

[birchy]

No, it's broken, but should only be an issue with non-US/Canadian visitors, so it's down the list of todo's.
I assume that you are in Finland?

[Aleksi "Lex" Kilpinen]

I assume that you are in Finland?

You would assume correctly