News:

Want to get involved in developing SMF, then why not lend a hand on our github!

Main Menu

Why does my customization (mod/theme) take so long to be reviewed?

Started by Joshua Dickerson, March 02, 2007, 02:16:59 AM

Previous topic - Next topic

Joshua Dickerson

I want to address this question since it seems to be pretty common. To make it short - time.

The unabridged reason: we are all volunteers and we do this when we can. Some mods are very extensive and they take quite a bit of time to review. A lot of the customizations that I have seen lately don't match the coding guidelines which takes a lot of time to handle. We make it a point to help customizers whenever possible. So, we try to send a PM to the author letting them know why we didn't approve the mod and suggestions as to how to do things better. If your customization doesn't contain any noticeable errors in the package-info.xml file, we attempt to install it. If it doesn't install, we try to figure out why. If it does, we then check to make sure it does what it is supposed to. If it doesn't, we then check to see why it doesn't.

Sometimes, we don't have the time to check all of these steps or we are taking all of our time to look at a larger customization that may have been in the queue for a while. We are really trying to improve the speed at which we do this. Trust me when I say that we really want to see SMF have as many customizations as possible, but it isn't possible to do as fast as people make them.

On another note, the reason that we don't want you to keep asking this question is because it takes us even longer and we don't want to waste time answering this question over and over again. If you have suggestions to make the process faster, by all means, post them. We are happy to hear how we can do things better :)
Come work with me at Promenade Group



Need help? See the wiki. Want to help SMF? See the wiki!

Did you know you can help develop SMF? See us on Github.

How have you bettered the world today?

Sarke

How about 1,000 monkeys working 1,000 computers?


I would suggest making all unapproved mods open-betas, so the community can download and review mods so that most problems will have been fixed by the time a SMF person reviews it.

I'd recommend doing this by creating a thread in a "Beta Mods" board for each newly submitted mod, much like how it's done when the mod is approved.  Keep the way it is now so unapproved mods can't be downloaded, but in the mod thread in "Beta Mods" provide a link to download the mod for beta testing.

This way the community can help speed up the approval process so the number of bug fixing the SMF people have to do is minimal, and regular users won't download unapproved mods by mistake.


The current system simply isn't working, at least IMO.  Mods are submitted faster than you guys can review them all, and as SMF grows so will the queue of unapproved mods.

Just my two cents...


Other than that, keep up the good work.

My MODs          Please don't PM me for support, post in the appropriate topic.

karlbenson

^ yep i agree completely.

When a mod is submitted, having a topic created instantly where people can help to 'refine' the mod, bug fix and generally improve it.  This would also open communication lines more so between mods and authors and other authors.

I really appreciate the hard work of the mod support team in checking the mods.  However in the time it could take for a mod to be approved by the current process, community members could have tested, refined, fixed any problems that would have lead to it being rejected.  This is even more so with large new features.

Chris Curran

Sarke's idea is better than the way the system operates now. I've had a calendar mod up for review since Feb 3 and received zero feedback from the mod team.

cheers

Valodim

"beta mods" sounds like a good idea, it provides earlier availability of mods for beta-happy people, and a platform for sorting out rough errors... definitely worth at least a thought of the mod team :)

NEMINI

all betas should be controlled though.  Proven users capable of providing useful feedback to get it to a final release stage.  Users who will just say it doesn't work and get mad won't help, in fact it makes it worse.  There needs to be a beta team, users capable and willing to test unapproved mods in multiple environments and report back their findings in clear, concise and useful manners.

Chris: last  I looked your mod showed it was updated March 8th (I may be remembering the exact date wrong but I am sure it was in march), does this mean you changed something that the team pointed out or fixed something you found and resubmitted it, thereby starting the process all over again?   
signatures are boring.

Valodim

Quote
Users who will just say it doesn't work and get mad won't help, in fact it makes it worse.

Right, but if those mods are on a seperate board I don't think that would be a problem...

Quote
There needs to be a beta team, users capable and willing to test unapproved mods in multiple environments and report back their findings in clear, concise and useful manners.

I think public voluntary work would work just fine here...

NEMINI

Quote from: Valodim on March 11, 2007, 06:59:20 PM
Quote
Users who will just say it doesn't work and get mad won't help, in fact it makes it worse.

Right, but if those mods are on a seperate board I don't think that would be a problem...

Quote
There needs to be a beta team, users capable and willing to test unapproved mods in multiple environments and report back their findings in clear, concise and useful manners.

I think public voluntary work would work just fine here...
I'll have to respectfully disagree.  How does having a bunch of people who can't/won't provide useful feedback help get a mod approved faster?
signatures are boring.

Sarke

Quote from: NEMINI on March 11, 2007, 07:02:57 PM
I'll have to respectfully disagree.  How does having a bunch of people who can't/won't provide useful feedback help get a mod approved faster?

I think that's a very pessimistic view.  Having it in a board clearly marked "beta" should get through to most people, and if someone does decide to use it and ask for support the mod maker can either help them and find the cause, or simply tell them that the mod is currently in beta.

What DOES help is that there will be people who want to help test mods and they will provide feedback.  Even if it's just a simple "I get this error", it will help the mod maker iron out most bugs by the time it gets to the approval process.  Not everyone will be helpful, but even if just a few are it helps.

If SMF had the manpower to spare we wouldn't be in this situation.

My MODs          Please don't PM me for support, post in the appropriate topic.

NEMINI

perhaps it is pessimistic, but it is also realistic.  There is more to beta testing then installing it and reporting any issues you accidentally come upon.  Beta testing is about trying to break things, its about purposely testing every single function and seeing if you can break them.  It's about using it in different environments, different set-ups.  It's about seeing if you can purposefully replicate bugs others report and reporting back if you can or cannot.
Beta testing is not, install it, hope it works, if it doesn't just uninstall it and wait for the next version.
signatures are boring.

Sarke

We're not talking about professional beta testing, just people trying to help.  No they're not experts, but reporting errors does help.  Not every regular SMF user is an idiot.

It's better than nothing.

My MODs          Please don't PM me for support, post in the appropriate topic.

NEMINI

Is it really though?  Or is it just a way for people who don't want to wait to get their hands on it sooner?  (FYI: those are rhetorical questions.)
signatures are boring.

Sarke

Yes, I'm not just someone waiting to get my hands on a mod sooner, but I'm talking from the experience I've had with my own mods.  Even after my mods were approved your average Joe SMF user found errors that the approval process missed.  Why not find these before it's approved?  It will help.

Why are you so against the idea?

My MODs          Please don't PM me for support, post in the appropriate topic.

NEMINI

I don't believe throwing unchecked code out to just anyone is being very responsible.  If one of these betas causes a bunch of newer webmasters/mistresses to get hacked/sites stop working do you think they are going to care or understand it was beta and not meant for production sites?  You can post all the warnings you want but the majority do not read ahead of time, only after its too late (if you don't believe that just hang around and give support in the forum and see how often the same questions are asked, the same basic information gone over 1000 times, how many times has it been said the package manager only updates the default theme?). 
A closed beta will allow people who understand the risks and have the knowledge to handle such risks, its also keeps the code from being posted publically where some hacker can dig through it to find a weakness faster then the developers can advise on closing them.
signatures are boring.

Valodim

It's not very difficult to find a compromise here: make it "closed" beta, but with very loose requirements.

A simple pm to a team member "I'd like to be a beta tester, I am aware of the risks and so on" and there we go, this easy step definitely eliminates the possibility of someone whining about his broken 100000 posts smf installation.

Quote
perhaps it is pessimistic, but it is also realistic.  There is more to beta testing then installing it and reporting any issues you accidentally come upon.

I disagree. In easily 80% of all cases line number and php error message are sufficient for the developer to find the bug with little effort.  

Joshua Dickerson

Valodim: that is assuming that the error is a parse error. It could be something not working as expected.
Come work with me at Promenade Group



Need help? See the wiki. Want to help SMF? See the wiki!

Did you know you can help develop SMF? See us on Github.

How have you bettered the world today?

Sarke

Could be something else too: empty arrays, undefined indexes, etc.  It all shows up in the error log.  Even a "this link doesn't work" let's the mod maker know what to look at.

My MODs          Please don't PM me for support, post in the appropriate topic.

Hambil

Not matter how carefully you review a mod, bugs will slip through. I've found bugs in many of the mods I've installed from this site. And if you look at almost any mod that's been around for any length of time you'll see a history posted right in the mod of bug fixes.

My point? Although what you are trying to do by reviewing mods before making them available may be noble, it's flawed and un-workable as SMF grows.

Take a page from vBulletin's book and create a separate mod site - so people who want to advance the core product can focus on that and separate themselves from any legal/moral responsibilities for mods.

Or at least go with the open beta idea. What you are doing right now not only doesn't work well, but it is killing the spirit of people who write mods who do it, let's face it, mostly for the positive feedback.

By the time a mod is approved here, I've already written another one.

And, not to slam any particular user, but I saw a mod posted after mine and approved the next day, and it now has 200+ downloads already, while my mod hasn't even been downloaded for review yet. So, someone is playing favorites somehow.

Oh, and I just check and said mod already has a bug fix posted today...

www.catnine.net/smf

SleePy

Even if a bug does slip through it is better to have caught 90% of them than none from not checking.
As well as Duplicate mods to appear in the mod site and those have to be dealt with depending on the mod such. So many checks are done to ensure that the end user does not receive to much trouble is done.
You can let people download betas but updating the mod site can confuse those just wanting to get the mod and get it working and don't want to mess with betas. This is why you should keep it separate.  
Jeremy D ~ Site Team / SMF Developer ~ GitHub Profile ~ Join us on IRC @ Libera.chat/#smf ~ Support the SMF Support team!

Joshua Dickerson

You're right, bugs will always slip through. Though, we can do what little we can to ensure that the newest user to SMF can rest assured that there has been some kind of review before they install a customization. You are also right about scaling the customization system - it will take a lot of work.

A couple of things we are trying to do to improve that is to put out guidelines (I am working on them right now) so authors know exactly what to do. Next is to improve the customization site. Have the system check for XML validation, check for mod parsing errors, check for PHP parsing errors, and check for some other basic things. After that, we will have a system that can almost run itself. As you can see, that will take a lot of work to get all of that done.

If you want something even sooner, you can create your own package server. There is a tutorial on how to do that in the Package SDK. It is actually really simple - just a XML file named packages.xml.

As to playing favorites - I haven't seen any favoritism being done. I couldn't care less who submits a mod. If anything, I scrutinize customizations by the team more than I would a normal user because I want them to set the example.
Come work with me at Promenade Group



Need help? See the wiki. Want to help SMF? See the wiki!

Did you know you can help develop SMF? See us on Github.

How have you bettered the world today?

Advertisement: