Suggestion: Beta testing process for SMF Mods

[mytreo]

Hello SMF team and community

I wonder what everyone thinks of the suggestion to have a recognised process for beta testing the SMF mods that are developed in this forum. The contibutions made by the members here are really excellent, but they would undoubtedly be even better if each mod went through a beta period where more experienced members could test it thoroughly and everybody could contribute their suggestions for how the mod could be improved - after all many minds are greater than one.

Those of us who have come from YaBB Gold will be familiar with this type of system as it works exceptionally well at boardmod.org. The result is really high quality mods with excellent functionaility and no bugs when they are released as Final versions.

I can't see any disadvantages to this type of system, it just makes perfect sense to me. What does everyone else think?

Best
Chris

[Daniel D.]

It's a good idea, but I doubt that everyone knows that these are BETA versions/mods, which could make troubles on a live board. Look how many people complains during the beta of SMF!! I think some users won't learn it - Beta is Beta. You must choose you testers carefully.

[mytreo]

I don't think that is any reason not to do it, if anything that is another reason to do it as we should try to get everybody educated to the idea of FINAL and BETA versions. At the moment the distinction is unclear where mods are concerned because to be honest some of them are horribly bug-ridden.

If you are familiar with Boarmod.org then you'll know that their system really does work. They have different boards setup for BETA and FINAL mods and a mod is only placed into the FINAL board (and thus uploaded to the mod server) once it has completed a period in BETA.

Another key advantage to me at least is that I may create a few mods if I could discuss the development in a beta board with others, but I wouldn't even consider releasing a mod that I have made on my own without the input of others and without proper testing.

Chris

[Daniel D.]

I know, but the problem is the beta period. Many users will use the mods as "final" versions in their live boards, even if you tell them hundred times that these are beta versions and won't work as they should. Some will use it on modified boards and they won't work, too.

Perhaps I'm just too pessimistic in some things . It's hard to defy the experience.

[bloc]

It sounds like a good idea. It would perhaps make everyones life easier if the mod work as it should - or as intended.

I cant help but think of the arcade mod in this respect - as far I have seen it works...but there is a lot of people trying to get it to work....

But how about it is beta-tested/approved/etc only with team-members? Or is it too much to ask for..?

[mytreo]

Daniel, I think it would be better that people install BETA mods that don't work or have bugs - at least then we can say "well it's clearly marked as still being a BETA mod". Currenly they just get cheesed off that the mod doesn't work. 

The key thing would be setting out the boards properly so that no confusion can possibly arise. The boards should be clearly marked

FINAL mods for SMF v1.0/v1.0.1

and

BETA mods for SMF v1.0/v1.0.1

with sticky topics explaining that BETA mods are still under development and should not be installed on live forums. It may also be good to ask people not to install BETA mods unless they are prepared to give feedback for bug fixes and improvements. Having the boards seperated like this would also help when the mod goes FINAL as then the thread in the final board would only be for Support (installing and the like) and not full of bug reports and fixes like at present.

The real deal is that people don't want to install buggy mods and currently they have no choice. We must also remember that buggy mods reflect badly on SMF too, a user will soon get cheesed off with mods that don't work or that kill their forum, and they will look elsewhere.

I don't think the beta testing should be restricted to team members. There are lots of us here who are more than happy to test mods, make suggestions, and provide feedback just for the greater good of SMF. The mod developer should decide when to make the mod FINAL after they are happy that the mod has been thoroughly tested and all known bugs are resolved.

Again, I really can't see any argument against adopting this type of system, IMO everybody would benefit and nobody would lose. I really hope SMF will consider these changes

Chris

[Kindred]

Of course they have a choice...

If they don't like, or are having problems with "non-final" mods, then they can write thier own mods!
Or... they could choose to not install the mod.

The problem with your suggestion is this:

SMF is a community of users. Some of the mods come from people on the SMF team... but many come form general users.
How are you going to "enforce" a beta-test?

Here's the deal....     A person posts a mod.
You can generally assume that the mod has been tested on this person's local machine.
(you can also look at the author of the mod;  for example, I have never had any problems with mods by Grudge or Anguz)
Look at the post date of the mod...
Look at the comments of others who have installed the mod...

If you feel comfortable with all this, then install the mod on your live board.

Quite honestly, I think the whole alpha-beta-gamma test period is well and good for full software... but for things like mods, you accept the responsibility of installing it on your own.

Hell, I ran a BBS back in the days of modem-dialup...
Since the source code was released with registration... (first in Basic, then Pasal, Turbo Pascal, C and finally C++)  and the software (WWIV) was used worldwide, there were MANY people releasing mods, hacks and other add-ons for the software.
It was always assumed (and accepted) that it was YOUR responsibility if you chose to install any mod(s). The author might help you if you encountered any problems... but in general, most of the SysOps just picked up their handy C++ book, read up on why the code might have borken and fixed it...   Sometimes we would post our modified comments...  but since each board was heavily and individually modded, it was likely thata fix for one might not be appropriate for another...

Honestly, people today really whine about things way too much and don't take enough personal responsibility.
If you add code to a software without knowing what that code does... you had better be willing to accept the consequences.

[mytreo]

Of course they have a choice...

If they don't like, or are having problems with "non-final" mods, then they can write thier own mods!

Brilliant!!!

[Daniel D.]

Daniel, I think it would be better that people install BETA mods that don't work or have bugs - at least then we can say "well it's clearly marked as still being a BETA mod". Currenly they just get cheesed off that the mod doesn't work.

The key thing would be setting out the boards properly so that no confusion can possibly arise. The boards should be clearly marked

FINAL mods for SMF v1.0/v1.0.1

and

BETA mods for SMF v1.0/v1.0.1

with sticky topics explaining that BETA mods are still under development and should not be installed on live forums. It may also be good to ask people not to install BETA mods unless they are prepared to give feedback for bug fixes and improvements. Having the boards seperated like this would also help when the mod goes FINAL as then the thread in the final board would only be for Support (installing and the like) and not full of bug reports and fixes like at present.

The real deal is that people don't want to install buggy mods and currently they have no choice. We must also remember that buggy mods reflect badly on SMF too, a user will soon get cheesed off with mods that don't work or that kill their forum, and they will look elsewhere.

I don't think the beta testing should be restricted to team members. There are lots of us here who are more than happy to test mods, make suggestions, and provide feedback just for the greater good of SMF. The mod developer should decide when to make the mod FINAL after they are happy that the mod has been thoroughly tested and all known bugs are resolved.

Again, I really can't see any argument against adopting this type of system, IMO everybody would benefit and nobody would lose. I really hope SMF will consider these changes

Chris

You don't understand me 100%, do you?

I think it's a very good idea and I think mods should be tested before the release, too!! But I think there are also many people who are complaining like they did during the beta phase of SMF. You can't prevent it, but I think we've  to live with it.

[mytreo]

Of course they have a choice...

If they don't like, or are having problems with "non-final" mods, then they can write thier own mods!
Or... they could choose to not install the mod.

As I said above, Brilliant!!! Why I never thought of that I don't know?

I think you're missing the whole point here, which is to get mods that DO work. Not everyone can code, and the whole point of packaging something into a mod is so that people don't have too.

The problem with your suggestion is this:

SMF is a community of users. Some of the mods come from people on the SMF team... but many come form general users.
How are you going to "enforce" a beta-test?

That is not a problem at all. Firstly I can't see why any developer would not want to beta test their mods with experienced users before unleashing them onto the general public (Why you seem to think testing something is bad is wholly beyond me). Secondly with the two boards laid out clearly as above if a mod was posted as FINAL  and it was found to be untested and buggy then it could be moved to the BETA board. Easy.

Here's the deal.... A person posts a mod.
You can generally assume that the mod has been tested on this person's local machine.
(you can also look at the author of the mod; for example, I have never had any problems with mods by Grudge or Anguz)
Look at the post date of the mod...
Look at the comments of others who have installed the mod...

If you feel comfortable with all this, then install the mod on your live board.

Yes that's how it currently works but 1) I think even the great mods from grudge and others would have benefitted from feedback before being made final. 2) testing on a local machine in one environment is not testing 3) The current system plainly doesn't work - just look at some of the threads in the mods board they are littered with bugs.

Quite honestly, I think the whole alpha-beta-gamma test period is well and good for full software... but for things like mods, you accept the responsibility of installing it on your own.

This defies logic, as I said above the whole point of a packaged mod is to make it simple and easy for people who don't want to get involved with coding. Of course I can often debug mods myself, but others can't, and I'd rather that my debugging was shared with the benefit of others.

Hell, I ran a BBS back in the days of modem-dialup...
Since the source code was released with registration... (first in Basic, then Pasal, Turbo Pascal, C and finally C++) and the software (WWIV) was used worldwide, there were MANY people releasing mods, hacks and other add-ons for the software.
It was always assumed (and accepted) that it was YOUR responsibility if you chose to install any mod(s). The author might help you if you encountered any problems... but in general, most of the SysOps just picked up their handy C++ book, read up on why the code might have borken and fixed it... Sometimes we would post our modified comments... but since each board was heavily and individually modded, it was likely thata fix for one might not be appropriate for another...

So? This is SMF, now.

Honestly, people today really whine about things way too much and don't take enough personal responsibility.
If you add code to a software without knowing what that code does... you had better be willing to accept the consequences.

Helpful!

Chris

[mytreo]

You don't understand me 100%, do you?

I think it's a very good idea and I think mods should be tested before the release, too!! But I think there are also many people who are complaining like they did during the beta phase of SMF. You can't prevent it, but I think we've to live with it.

Hehehe I'm glad we're agreeing

Yes some people might not "get it" and they may download and install BETA mods without realising they are BETA. All we could is to make it as obvious as possible that if people aren't competent coders then they should stick with the FINAL mods... a few people might not see this or may choose to ignore it but then there is no compensating for that really, just their tough luck! I think the vast majority would be very grateful for well tested FINAL mods that they can install with ease

Chris

[Trekkie101]

Well beta testing is a good idea IMO as you will get to test on various enviroments at times under different stress levels, which can show un-expected bugs. For example, at my local machine php.ini give permission to everything, anything thats maybe a secirity issue is till allowed as its only me where on the web im under a lot of restrictions and have to accomodate for different conditions. Fair enough I dont write mods, minus the one I was asked to make from code that already existed. I would love to beta test as to me its essential but as Peter Duggan keeps saying "Were working on something" or words to that effect or similar. So there will be something coming up soon rather than later.

[Midgard]

Good idea, you're right...

[redone]

A agree with mytreo fully on this one. The point is to set a standard for the coding and approve mod's that work as expected and can be installed/uninstalled without issue.

As it stands right now installing a bunch of mods and attempting to uninstall some or in somecases just one of them is one of the fastest ways to break your forum.

I have said this before I almost went back to phpBB as a newbie because of problems I perceived to be with SMF and the package manager. Only after being patient and getting help from here did I learn it was the mod's that were the issue not the package manager!

Approved mod's are only common sense.

[bloc]

And what about when someone installs 4-5 mods that literally touch each others modifications, and you are not sure which one is the culprit...? Most mods work as expected install/uninstall-wise on the developers machine/test server, but then real life proves otherwise... I sit in a glasshouse on this one, and will take responsibility for that, by getting people with me that CAN take a faulty installation without the end of world happening, before ANY release.


Approving mods are well and nice, but it will lay a burden on the Mod team which perhaps is not wanted. And mods is by nature, experiments with a working stable installation, no matter how you put it. You take a chance when installing one, and the more mods in, the bigger the risk.

A standard for the coding..? I am no expert, but to me it seems every developer has its own style - who's to say its better than this and this..? That it WORKS is almost without saying. Its the glitches that can ruin a seemingly working mod. And thats why we have beta-testing....

(I am sorry if I sound a bit irritated, having a cold might have something to do with it...  )

[mytreo]

I'm glad to see some support for this suggestion. It's beyond me that some people can't see the sense in this!

I'm not suggesting that SMF officially approve mods so there would be no burden on the mod team or anybody else, all I'm suggesting is that we have a recognised difference between BETA mods and FINAL mods. Any mod developer worth their salt won't let a mod go FINAL until they know it has no bugs so there shouldn't be any need for official approval.

What is key is that we have a board where mod developers can release their mods as BETA and gain the feedback from experienced users and those willing to help debug it before it is released and added to the package server. There are loads of us that would help test mods and the end result is that mods would better, and SMF users experience would be greatly improved.

[Col]

Hi,

I remember you, Mytreo, from YaBB and BaordMod. I too think the system at BoardMod works well. I may be wrong, but I believe the mod developer at BM asks for the mod to moved to the 'finals' board. Of course bugs can still pop up, but I think it's useful for all concerned to know what's still being developed, and what's thought to be a finished mod. I find that there is far more testing at BM from far more people than here, because thay really understand that it's a BETA. Very many test there, with very varied abilites, and this really does help.

I think at BM mod developers take the 'finals' board seriously, as it would reflect upon them if they kept releasing buggy mods. - Indeed, very many of the mods never actually make the finals board, either because the mod runs into real difficulty; or there just isn't the demand for it; or the mod developer just gets tired of the project, and the problems. This has to be better than having no distiction between mods at different stages of delelopment.

If a BM type system were here, and with the level of activity of this forum compared to BM, it would work very well.

[redone]

The constant contribution of mod developers are a vital part of the continued success of SMF.

Those that have been running SMF for a while now and understand how mod's work are fine but those that seek the package manager as a fast track solution are often surprised with the results!

I have to say though the growing groups of members joining up with authors to test out beta's before being released here really is a huge step forward

[Col]

I notice that Bloc has removed his TinyPortal files until it's stable. I'm sure that if there were designated BETA areas this would not need to be done. It is too easy at the moment for newbies to jump into using a BETA mod on their live forums without realising the posible consequences. On the other hand, there are many that can, should, and will test the betas out that will be unable to when a BETA is pulled like this. - It has to be everyone's interest that many people are testing new mods.

[bloc]

... On the other hand, there are many that can, should, and will test the betas out that will be unable to when a BETA is pulled like this. - It has to be everyone's interest that many people are testing new mods.

There is actually a possibillity to try out the existing + upcoming versions, but I removed them because of the increasing dismay caused by its beta nature, as you point out. I do have set up a section up on my website limited to those who can and want to try it out properly - and are prepared for errors along the way. I might add that I don't see any problem in doing the same here in some form, if there is any interest.