News:

Want to get involved in developing SMF, then why not lend a hand on our github!

Main Menu

SMF build tools, CSS & JS optimization, mods as packages & dependency handling

Started by qc, April 30, 2016, 07:31:23 PM

Previous topic - Next topic

qc

Hi SMF developers,

When I started working on a custom theme for our forum running SMF 2.0.11, I noticed a few drawbacks of the way SMF currently handles themes and packages:

  • many individual Javascript, CSS and image files cause many individual HTTP requests to the server
  • Javascript and CSS files must be manually minified
  • LESS, Coffeescript & co must be manually compiled
  • Each mod handles and loads its own dependencies such as jQuery
  • Updating CSS or Javascript doesn't cause the client browser cache to be updated (when caching is active)

As a solution, I started using a build tool (brunch.io, but any other will do the job, too). Such build tools automatically minify and concatenate Javascript and CSS into single 'global' files, automatically resolve dependencies, handle cache busting and more.

The result is a faster forum with reduced page size that can take full advantage of client-side caching and further optimizing techniques such as image maps and HTML minification.

A necessary precondition however was to manually remove all manual Javascript and CSS includes from all theme files and mods. As the build-in SMF package manager has not been made with this build process in mind, a new mod installation now requires to install and modify each mod locally, run the build pipeline and only then deploy to the server.

I would like to hear your opinions on how SMF can be made better suited for such a build process. Do you use any build tools and if yes, how does your build pipeline look like? Do you think a package manager and automatic dependency management via bower, composer or similar can replace or at least complement the build-in SMF package manager in the future?

Best regards,
qc
Playing quizduell? Having quizduell questions? Our german quizduell forum quizcommunity.de is looking for quiz freaks to come and play quizduell with us :)

Arantor

The SMF package manager predates most of these movements, since in 2003 it was a fairly new idea, and predates jQuery being a standard thing.

The other reality is that you'd have to get every single mod to actually follow such a build process, otherwise it doesn't help any, because as soon as any mod doesn't use it, it limits the usefulness of any build process :(

Note that 2.1 does have its own minifier built in though it does not support LESS/SASS. It also bundles jQuery.

live627

Quote
Note that 2.1 does have its own minifier built in
and even then, it only does

       
  • scripts.js
  • theme.js
  • smf_jquery_plugins.js
  • index.css
  • responsive.css
https://github.com/SimpleMachines/SMF2.1/wiki/CSS-&-JavaScript-minifier

Antes

Quote from: live627 on April 30, 2016, 08:54:25 PM
Quote
Note that 2.1 does have its own minifier built in
and even then, it only does

       
  • scripts.js
  • theme.js
  • smf_jquery_plugins.js
  • index.css
  • responsive.css
https://github.com/SimpleMachines/SMF2.1/wiki/CSS-&-JavaScript-minifier

So it can't bundle the output of loadCSS/loadJS  as min.css/js ?

Arantor

No, it doesn't work entirely well either if you have multiple themes, I was trying to add my theme's CSS as an additional file to be added after the main theme CSS in the minified version and it does odd things like putting it first, or omitting it and I have a terrible feeling the minified file ends up in the default theme folder which is not appropriate for a site that intentionally runs multiple themes for users.

qc

The usual building & deployment process of any 'bigger' web application nowadays seems to involve local development and dependency resolution and running all kinds of preprocessors before finally deploying to the server (some even run their node or composer package manager on the production server which I don't recommend).

The SMF deployment process however requires pre-build packages (core, mods & themes) that can easily be installed by anyone on any webserver just running PHP & MySQL. Of course, this requires each package to come bundled with all its dependencies (PHP and JS libraries) and precompiled / minified JS / CSS, since no package author can rely on any dependency being already installed (it seems that with 2.1, you now can rely on a specific jquery being available).

I haven't found a good solution yet that allows both the developer to run his local build tools with dependency resolution etc. and the end-user to still have the option to simply install 'readymade, self-contained' packages via SMF package manager.

A PHP variant of Ruby's asset pipeline might be able to solve the minification, concatenation & caching problem on the fly, but won't help with dependency resolution (e.g. two packages requiring the same library -> load this library only once and in a version that both packages support).

My ideal future SMF would allow professional users to run a build pipeline while still giving the casual user the option to upload & deploy individual precompiled packages on the fly...  :D
Playing quizduell? Having quizduell questions? Our german quizduell forum quizcommunity.de is looking for quiz freaks to come and play quizduell with us :)

Arantor

The problem with the "prebuilt" model you're talking about is the one Discourse and NodeBB have with Docker. It works as badly as SMF's, just in different ways.

qc

Quote from: Arantor on April 30, 2016, 09:44:47 PM
The problem with the "prebuilt" model you're talking about is the one Discourse and NodeBB have with Docker. It works as badly as SMF's, just in different ways.
With 'prebuilt' I meant the way mods and themes are deployed right now in SMF.
Playing quizduell? Having quizduell questions? Our german quizduell forum quizcommunity.de is looking for quiz freaks to come and play quizduell with us :)

Arantor

I wasn't. I was talking about the kind of deployment process you were talking about where there is a true build process and asset pipeline and thus only one sort of dependency manager.

It doesn't work in practice as well as you're thinking it might :( Discourse, NodeBB do this, it doesn't work. XenForo is closer to what you envisage SMF as being capable of, and that too has much the same problem, though it provides better structuring to cope with some of it, but it is by no means a solved problem and the jury is still out as to whether it is even a solvable one.

qc

Quote from: Arantor on April 30, 2016, 10:24:12 PM
I was talking about the kind of deployment process you were talking about where there is a true build process and asset pipeline and thus only one sort of dependency manager.

It doesn't work in practice as well as you're thinking it might :( Discourse, NodeBB do this, it doesn't work.
Hi Arantor, thanks for your clarification. Since you seem to have experience with these systems, could you elaborate a bit on what does not work? Just so we can factor that in to our assessment of the ideal future streamlined SMF building and deployment process™ :)
Playing quizduell? Having quizduell questions? Our german quizduell forum quizcommunity.de is looking for quiz freaks to come and play quizduell with us :)

Arantor

Getting mods to play nicely together, expecting all mod authors to actually follow such things, expecting mod authors to never do things like bundling incompatible versions of things (e.g. the core bundles one version of jQuery but a plugin that a developer wants to use needs either an older or a newer version of jQuery, or different jQuery UI themes)

This is even before we get into the realms of trying to juggle CSS from multiple sources (core system, us current theme, plus whatever CSS a given plugin needs), plus whatever unique fixes or changes a site owner wants, all then with minification or whatever.

There are few rules that actually matter in computer science. One of them is that you can have it performant, or you can have it customisable, both do not exist.

I've probably written more extreme mods than most people and in the last large scale mod I wrote for both 2.0 and 2.1 I deliberately avoided using things like jQuery and did it with bare JS almost everywhere simply because I could rely on that better than trying to juggle the different permutations.

Doing what you're suggesting with proper dependency management is what NodeJS' package manager, npm, is just discovering to be hard.

If it's hard enough at the library level, where dependencies can be sanely tracked, imagine what it becomes like at different levels simultaneously, where you're not just measuring core platform plus all its libraries, but core platform, many optional dependencies, each of which may conflate with each other on multiple levels.

The end result is a hideous mess unless someone sits and scrupulously and ruthlessly maintains the lists of what plays nicely with what. Debian does this, and it's huge work. And the minute you step outside of these processes to have dependency management, it all goes entirely to pieces. And you can bet people won't follow the rules for various reasons, be it time, skill level or other.

Not to mention that any serious changes are SMF 3 territory which, realistically, is years away.

But I invite you to see how other platforms do it, and the realities that it brings. Drupal does it, XenForo does bits of it, Discourse does it. And it makes development harder for little benefit to the user in practise.

qc

Quote from: Arantor on April 30, 2016, 10:45:29 PM
Getting mods to play nicely together, expecting all mod authors to actually follow such things, expecting mod authors to never do things like bundling incompatible versions of things
True, one has to recognize that only a small subset of mod & theme authors will probably have the patience or knowledge to follow such practices. Others would have to learn it, which costs time, which will inevitably reduce the number of new mods & themes.

I personally however would not be unhappy about having fewer mods available, as long as they are of higher quality. I end up fixing and modifying most modifications anyway right now.

Quote from: Arantor on April 30, 2016, 10:45:29 PMThis is even before we get into the realms of trying to juggle CSS from multiple sources (core system, us current theme, plus whatever CSS a given plugin needs)
I am not sure if I can follow you here... it is indeed challenging to get mods and themes to play nicely together, which is probably the reason why only so few themes change the HTML layout and most just introduce simple CSS enhancements. Is that what you mean with juggling CSS?

Quote from: Arantor on April 30, 2016, 10:45:29 PM
If it's hard enough at the library level, where dependencies can be sanely tracked, imagine what it becomes like at different levels simultaneously, where you're not just measuring core platform plus all its libraries, but core platform, many optional dependencies, each of which may conflate with each other on multiple levels.
...hm, you might be overthinking this. So far, my SMF installation has the following dependencies (theme & 3rd-party plugins): bootstrap, jquery, jquery-timeago, JSON.php
That's a pretty standard set of requirements and so far I do not experience any conflict. Might be different for users running more modifications, of course.

Quote from: Arantor on April 30, 2016, 10:45:29 PM
Not to mention that any serious changes are SMF 3 territory which, realistically, is years away.
Our discussion might also inspire more proficient users to just go ahead and try something new, like I did. The more different approaches we have, the better - and who knows, perhaps someone will discover the golden bullet ::)

Quote from: Arantor on April 30, 2016, 10:45:29 PM
[...] it makes development harder for little benefit to the user in practise.
I agree with the first part of this statement. A build process might make SMF less interesting for hobbyist programmers. This might work for some projects, but I am not sure if it might be beneficial for SMF.

I might not be the target user group for SMF, but I definitely benefit from my current setup - but then again, I also face the drawbacks listed in my initial post. Thank you for your feedback, Arantor!
Playing quizduell? Having quizduell questions? Our german quizduell forum quizcommunity.de is looking for quiz freaks to come and play quizduell with us :)

Arantor

Ok, let me explain how the dependency tracking and version issue comes along.

Core system provides, say, jQuery 2.0 and Bootstrap 3.2. A theme comes along and wants to use Bootstrap 4. Any mods now don't necessarily work correctly because they were built for Bootstrap 3.2. Even mods trying to use Bootstrap 3.3 might not work correctly.

So fine, you might say, move things to Bootstrap 3.3... Except that breaks some stuff dependent on how 3.2 works.

For your use case, this is fine. But for the average forum admin who has a custom theme and anywhere up to 25 mods (conservative case, some people are into the hundreds) each with potentially their own requirements this gets hairy fast.

While it's true that a lot of mods will just use whatever the core provides, there are inevitably ones that don't, especially if the core doesn't force mod authors into the treadmill of versions, which is where it also inevitably starts breaking when people want to upgrade to use the newest shiny and forget that everything else might rely on older shiny.

I'm also not sure how to respond to your observation about "more proficient users", as, um, there aren't many more proficient SMF people than me at this point ;) I'm "that guy" who does things that are outlandish and crazy like building entire mods that don't use file changes even when they introduce pretty style URLs to the mod in the process ;) I've been involved in projects where auto minification is a feature, and the reality is that you end up chasing down all kinds of edge cases where things don't work properly solely because of the minification process that never show up in development because the devs will have that turned off for development. Wedge did this and the amount of time it burned in development wasn't even funny...

qc

@Arantor: The version compatibility issues you mention are general problems that you would have with or without dependency management. A dependency manager however might let you know beforehand that you will run into problems.

Quote from: Arantor on May 01, 2016, 07:36:07 AM
For your use case, this is fine. But for the average forum admin who has a custom theme and anywhere up to 25 mods (conservative case, some people are into the hundreds) each with potentially their own requirements this gets hairy fast.
I have a custom theme and about 20 mods, so I would fall into your second category of the 'average forum admin'. Still, even with more mods, the list of dependencies is pretty small, since without proper dependency management, mod authors so far prefer to re-implement common functionality instead of relying on libraries which might or might not be available (and would cause all kinds of namespace and versioning issues if already available). The notable exception is jQuery. In my setup, bootstrap, jquery-timeago and the Like Posts mod all require jQuery to be loaded on the client side. Without build process, I would end up with two or three jQuery libraries being loaded. The Like Posts mod even performs a client-side comparison of the loaded vs. required jQuery library and then dynamically fetches it from ajax.googleapis.com.

I am still interested in some feedback on my first two questions though: how can SMF be made better suited for such a build process? and do you use any build tools and if yes, how does your build pipeline look like?

PS: With "more proficient users" I meant "more proficient than the average SMF user", not "more proficient than you or me" ;)
Playing quizduell? Having quizduell questions? Our german quizduell forum quizcommunity.de is looking for quiz freaks to come and play quizduell with us :)

Arantor

I don't have a build process, I never did because I kept finding that a build process didn't actually help nearly as much as people think.

Minification is great if you're pushing large chunks of code at a client, like these days where pushing 1MB or more of assets for SPAs etc is common. For what SMF uses, even in 2.1, the difference minification makes is actually largely negligible for the most part. People don't, generally, stop to compare the real size served by Apache versus the "effective" length because properly configured servers don't send it without compression and my experience is frequently that the compression works better, not worse, without minification.

Best improvements SMF can make really amount to "deferring all the JS to the footer" and reducing things like gradients and shadows to just trim the amount of CSS needed - don't have a build process by not needing a build process to begin with.

qc

It is a bit off-topic, but...
Quote from: Arantor on May 01, 2016, 11:50:12 AMmy experience is frequently that the compression works better, not worse, without minification.
Would be strange since JS minification can make use of domain specific knowledge whereas gzip can't (and thus has to preserve comments, whitespace etc.). Also, replacing e.g. long variable names with short abbreviations does not change the number of unique strings in a JS file and thus should improve, not reduce gzip performance. According to http://www.gribble.org/techreports/minification/ minification on top of gzip saves another 10% of traffic.

Quote from: Arantor on May 01, 2016, 11:50:12 AMBest improvements SMF can make really amount to "deferring all the JS to the footer" and reducing things like gradients and shadows to just trim the amount of CSS needed
Nowadays you can just add the async and defer attribute to the <script> tag. Some sites like e.g. github include both an async js and the normal synchronous js file for the parts that cannot be loaded asynchronously. Would be nice to have in SMF, too, but it would require to untangle all the mess that comes with the inlined scripts in the default templates. Did you ever try asynchronously loaded javascript in combination with SMF?
Playing quizduell? Having quizduell questions? Our german quizduell forum quizcommunity.de is looking for quiz freaks to come and play quizduell with us :)

Arantor

I used to work with a guy who took SMF, rewrote the code to take all existing inline events and through some voodoo, then deferred them all until after the main code files loaded in the footer. He did some crazy stuff about rewriting code until such times as the gripped version of files ended up smaller. He would spend entire afternoons to shave 10 bytes off his minified gzipped files.

My experience is also that this stuff tends to save small amounts of time that users don't even notice.

Though in reality the best files are the ones that don't even have to be loaded, which is when the build process hurts you even more where you either have to bundle every single CSS/JS thing rather than being modular about it, or you skip out on caching some things at the expense of others. Expires headers are awesome ;)

Advertisement: