Splitting the Forum

[Arantor]

The SMF software supports roughly 10k categories, 100k boards, and 10mil messages are supported without further modification.

Um... no.

In a stock installation the figures are quite a bit out.

Category ids are tinyint(4) unsigned, meaning you get theoretical max 255 out of it.
Board ids are smallint(5) unsigned, meaning you get theoretical max 65535 out of it.
Topic ids are mediumint( unsigned, meaning you get theoretical max 16777215 out of it.
Message ids are int(10) unsigned, meaning you get theoretical max 4294967295 out of it.

Circumstances can mean these figures can be halved, however, but those are the stock limits. If you have modified them in this mod, I would be very, very careful if you have modified everywhere these limits are implicitly applied.

I would note that I know of an SMF forum with currently 70.7 million posts and none of these limits have been hit. Others, however, have such as an implied 65535/per day limit in the stats tracking.

[DenDen60]

The SMF software supports roughly 10k categories, 100k boards, and 10mil messages are supported without further modification.

Um... no.

In a stock installation the figures are quite a bit out.

Category ids are tinyint(4) unsigned, meaning you get theoretical max 255 out of it.
Board ids are smallint(5) unsigned, meaning you get theoretical max 65535 out of it.
Topic ids are mediumint( unsigned, meaning you get theoretical max 16777215 out of it.
Message ids are int(10) unsigned, meaning you get theoretical max 4294967295 out of it.

Thanks for the clarification Arantor.

Now, if I understand what Dougiefresh, it really does not matter since, again, if I understand properly, the mod creates another instance of SMF.

I hope I got this right. Otherwise, I will need to look at how to increase these maximums, especially the categories and the boards.

[Kindred]

I don't think that this mod creates another instance...

AFAIK, it uses the same database tables with additional fields to define what is displayed where...

[Arantor]

This is also my understanding of how the mod works, and trying to create such a monster is going to actually have many, many bottlenecks under load that cannot be fixed.

SMF is not designed to function in the fashion in question, it was simply never designed to scale in that fashion. The correct solution would be to run multiple installations and use something (that currently does not exist) in the fashion of IP.Connect to have one master and many slaves for the purposes of authentication.

[DenDen60]

This is also my understanding of how the mod works, and trying to create such a monster is going to actually have many, many bottlenecks under load that cannot be fixed.

SMF is not designed to function in the fashion in question, it was simply never designed to scale in that fashion. The correct solution would be to run multiple installations and use something (that currently does not exist) in the fashion of IP.Connect to have one master and many slaves for the purposes of authentication.

Ok so what would it take to create this something?

[Kindred]

A massive amout of code and someone with enough time and desire to write it...

[Arantor]

Such a bridge need not be huge, but it is fiendishly complex and is pretty much what I had been advocating from the off with this particular use case, but while I have the skills, it doesn't seem viable for me to do it unless I charge a stupid amount ($75-100 per forum network) to actually make it worth even close to the time for me to do it.

[DenDen60]

Ok, So if you look at the hierarchical structures of my communities, see attachment, what would you recommend? (This is what I am developping as we speak, using th Split Forum mod.)

The top community (the global practice), is the place by which people will join.

[DenDen60]

Such a bridge need not be huge, but it is fiendishly complex and is pretty much what I had been advocating from the off with this particular use case, but while I have the skills, it doesn't seem viable for me to do it unless I charge a stupid amount ($75-100 per forum network) to actually make it worth even close to the time for me to do it.

Arantor,

Considering the hierarchy of my communities, could I start using this MOD and then move to bridges?

[Arantor]

Considering how bridges actually work, the answer would logically appear to be a no.

Then again, given that I see currently one possible customer for such a bridge, there's no incentive for me to do it for anything less than several hundred dollars for my time...

[DenDen60]

Arantor, would there be another way to do this. 7 years ago, we did it by creating a mod that used one database for members and a routine that exchange information between different instances of SMF. Users did not have to registered for each instance per se. The mod was installed on each forum and the members were register automatically.

[DenDen60]

Here are the three mods that were created when I did my first "proof of concept in 2007. The mod links different instances of SMF to one user database.

[Arantor]

That is pretty much the approach I have been advocating this entire time... even before you started with this mod.

[DenDen60]

So would it be complicated to upgrade this mod to function with SMF 2.09?

[Arantor]

Complete rewrite.

[DenDen60]

Complete rewrite.

How much $ ?

[DenDen60]

Let's imagine that I start with the split forum mod, when would the process of adding sub forums really affect the users?

If I start with 25 sub forums and 10 000 members, will it be problematic? What about when there are 50 sub forums and 100 000 users? 100 sub forums and 250 000 users.

Thanks for your help.

[dougiefresh]

@denden60: You might want to read this topic: Share user database for 2 (or more) different forums.  I dunno if it would help you, but it might be worth looking at.....

[DenDen60]

Thanks Dougiefresh. I remember that post and even participated in it. From my understanding of the discussion it seems to me a problematic approach.

[DenDen60]

Now, if I understand what Dougiefresh, it really does not matter since, again, if I understand properly, the mod creates another instance of SMF.

No, this mod doesn't create anything but the APPEARANCE of another instance of SMF.  By restricting what is chosen by the database, I can effectively output multiple forums using the same code....  Keep in mind that it does NOTHING to prevent overloading the server.....

Then the question is at one point, adding sub forums, starts to affects the overall functioning of SMF and impedes users' experience?

Should I pay now to have Arator rewrite the mod or wait until we reach a threshold and then make changes.

[dougiefresh]

Can't answer the first question.....

Regarding second question: If you're asking my opinion, it would make more sense to do the changes now since I assume you're still in the planning stages of your site(s), than to TRY to do them later....  Doing it later can result in unwanted/unexpected/unplanned-for downtime during the "upgrade" process....

[DenDen60]

I prefer to go the second route as well.

Arantor, I you interested?

[Arantor]

Firstly... it might help if you get my name right occasionally. It's a small and petty thing, I know, but programming is all about the little details - and it's very important to get them right.

Secondly... this is where it gets complicated. This is something that doesn't come up all that often, meaning I need to do it once and charge a lot for it (hundreds of dollars, possibly even verging towards 4 figures), or I do it as a generic mod that I charge per sale... but even than because of the low use of it, and the high maintenance factor, it's still going to be in the hundreds of dollars category somewhere along the line. Honestly, even $100 per copy (one copy per network of linked sites) seems a shade too low to me given the time investment and making it work properly with everything else.

It's not that I'm not interested... it's that I *have* to make it worth my time to do. Because honestly I doubt I would even be looking *any* at the code you already have anyway.

[DenDen60]

Sorry about this Arantor. Looking at my preview posts, I seem to do it right 60% of the time. But after corrections, I am now up to 100%. 

What does Arantor mean by the way?

I understand that there is a major difference between SMF 1.1 and SMF 2, so I understand that you might not be able to code I shared with you?  This being said the logic and some of the processes are still valid.

Now I understand that it has to be worth your time. Hope we can reach an agreement of some kind. You can inbox me if you want to pursue this off line.

[Arantor]

Does it have to mean anything? I've just used it for the last 12 years.

As for differences... the table columns in the database changed between the two, as did every single database query had to be altered; SMF 1 uses a thing called db_query while SMF 2 uses a thing called $smcFunc, and while a simple find/replace can sometimes work, it probably won't for something like this.

There's also maintenance issues with respect to updates that need to be handled differently for 2.0 - especially since such a thing should ideally be built with 2.0/2.1 in mind, not just 2.0 (and it's very possible to make that be a thing by way of the integration hooks, in fact, it just needs to be built looking forwards not backwards, which is also where the intention of a rewrite from scratch would come in)

I would also note that the 'logic and some of the processes' being valid is an interesting assessment. How does one assess whether the logic is the same when some of the most fundamental parts of both of those have changed?

[DenDen60]

No of course. I was just curious.

As for the logic, I meant the ideas behind the coding. This being said, maybe they are not pertinent anymore. 

[DenDen60]

So Arantor, if you are interested just inbox me so that we can pursue this off-line

[DenDen60]

Wow I have just realized that I had not updated my profile since 2006

I guest it was time for me to update it. 

[DenDen60]

I am going to ask this question again because it is important for me.

At one point, adding sub forums, starts to affects the overall functioning of SMF and impedes users' experience? (Meaning it become slow)

[Arantor]

You need to define what 'slow' means.

Adding boards slows things done. Period. More boards = slower. Can't be avoided.

It only starts to become a *real* problem on most hardware when you start going into hundreds and hundreds of boards. I also suspect this mod hasn't been tweaked for such configurations (since even SMF itself needs consideration at those levels) and that's pretty much per-install anyway.

[DenDen60]

Thanks for your response Arantor.

My pilot project aims to start around 25 sub forum, each representing a community of practice for citizens,  with let say 7 categories and a few boards and several sub-board each.

At the beginning, I don't expect to have more then 10 000 members ( I already have them)  out of which maybe a few hundred will come occasionally and and 50 committed members participating.

[Arantor]

Members aren't a huge problem; this site has 350k ish visitors.

Boards are where the pain point bites most.

[DenDen60]

I really don't understand how boards can be problematic? I am not saying you are wrong of course, but in my head, a board is a static cell which is associated with a list of discussions.

I see this as a hard drive storing information. 

[Arantor]

The problem is how board access is established.

Board access scales inefficiently, the more boards and the more groups, the slower it gets when querying for that data, and it scales in O(n log 2) time in the worst cases. Meaning that for 10 boards vs 1 board the internal processing time can scale by up to a factor of 100. Or worse.

[DenDen60]

But doesn't today's server with high processing power and SSDs compensate for that inefficiency?

[Arantor]

Not really, no. Because the way higher processing power goes is horizontal - you scale by breaking the workload down and parallelising it. Except this is something you cannot parallelise meaning that the work unit is still running in single thread order, just as it was ten years ago, and computing power is not quite so scaled up when you're talking single core order.

[DenDen60]

Thanks again for your  answers.

When ever you are ready, inbox me so that we can discuss how we can work together.

[DenDen60]

What is the differences between having 25 separates instances of SMF, versus one instance with 25 sub-forums? Do both affect the performance of the server in the same way?

[Kindred]

they will both affect the server... but in different ways.


note -- I have split this discussion out of the split forum mod thread, since it has gone into the real of "build another solution" and was cluttering up the mod thread with the discussion.

[DenDen60]

Thanks Kindred,

Which one will effect the server less or have less an impact on the end user?

[Arantor]

And has a serious number of consequences - and any such kind of bridge will not come cheaply.

I mean, I've been thinking about it and trying to work out what such a thing might cost to build.

Honestly, though, it's not something I want to engage in unless people are prepared to pay serious money for it - because as much as it intrigues me and I'd have fun building, I don't think I could build this just for one person.

I have also thought about making a more complex auth bridge that handles FB, Twitter and stuff like that - as well as cross-forum auth, maybe that would be worth more money. I just don't think I could build it in a fashion for just one person where they would pay the amount of money; I would expect to spend a good 2-3 full weeks on just the multi-forum auth, let alone the other stuff. That especially depends on whether things like avatar sharing occurs (which is more complex since you're talking about pushing content between systems rather than a simple master/slave replication)


As far as the server... it's really simple: the more you have going on, the more the impact is. It's simple math. It's easier to scale multiple instances than it is to scale one megainstance simply because you can do things like proper sharding and separating machines to separate servers.

[DenDen60]

Thanks for your feedback Arantor.

This led me to see this option.

Let us say we have one Master Central Member Database (MCMD) and a multitude of Secondary Member Databases. (SMD)

The first time member registers, they register in the MCMD. At the same time we create a small dashboard for them that contains the list of forums they are members of.

So the first time, there is only one forum on their dashboard.

While their sessions are opened, if they decide to registered on a second forum, it becomes part of their dashboard.

The registration process duplicates their information contained in the MCMD into the SMD of the forum they have just registered.

Would that make it easier to develop?

Thanks again for trying to figure out how to make it work.

Denis
Ps.: Most user will only use four or five. Some will use more, but I don't see anyone using more than 12 to 15.

[Arantor]

Would that make it easier to develop?

Nope, that just adds multiple layers of complexity.

[DenDen60]

Then how would you tackle this?

I was under the impression that the latest option was easier because each forum are then independent.

For example, if I want to send a newsletter to forum "C", since forum "C" has it's own database, I can send the newsletter easily.

The only complicated part is the one where you login and the process by which you first duplicate the login information, avatars signature etc, from the MCMD into the SMD, and when you synch changes made in the MCMD between the MCMD into the SMD.

But this is my understanding.

[Arantor]

That's the thing; your concept of a dashboard etc. massively complicates it by adding things that simply do not need to exist.

It's really simple: user tries to login on any forum. If the forum is a slave, call the master. If the account exists, create a local copy of that account. Then treat it as a mostly local account as far as account existence/integration is concerned (but simply re-defer back to master on the matters of passwords; similarly, when changing passwords from any slave, renotify master of changes)

Similarly with registration, registration on any given slave simply pushes it to the master to deal with.

This whole dashboard thing simply does not need to exist. You tie the systems together internally only and need not trouble the user with trivialities like which forum(s) they are members of.

In all honesty I do not see us being able to work together, unfortunately. The ideas of cross forum bridges have intrigued me for some time but my style simply does not mesh with your intentions.

[DenDen60]

Ok, well thanks for your feedback.

Denis
Ps.: A dashboard is just a page that shows the name of the users and the boards that he belongs to. (These are just included in the master database when he first logs in. )

[DenDen60]

So Arantor,

Let's say we don't do the dashboard, would you be interested?