News:

SMF 2.1.4 has been released! Take it for a spin! Read more.

Main Menu

Images on S3

Started by ahriman1911, September 13, 2021, 10:37:33 AM

Previous topic - Next topic

ahriman1911

I'd like to request a feature or mod to enable offload of images to S3. I'm willing to pay a recognized SMF contributor or mod developer to implement the feature if you believe no one else would benefit.

I know in the past there were people saying that if you implemented S3, then you'd have to implement Azure, etc. But today S3 is the defacto standard. There are a TON of storage lockers that implement the S3 object storage protocol (i.e. Wasabi,etc)

I have a forum with something like 500 GB of images and it is impossible to host it on any shared hosting anywhere. I have no choice to self administer a dedicated server and I'd really like to have an SMF forum that is easy to manage and can be hosted on a shared hosting somewhere.

Thanks community! I'm looking forward to your feedback.

Arantor

It's actually a really nasty job even if you happen to use any of the obvious connectors (e.g. Flysystem with AWS SDK) because so much of SMF makes some assumptions about files being locally held both for avatars and attachments.

It amounts to a partial rewrite of the entire attachments system since you have to not only fix the actual attachments code, but all the integrity code to go with it.
Holder of controversial views, all of which my own.


ahriman1911

I figured something about SMF made the issue complex because no one has put out a S3 mod when pretty much every other forum has had a mod for this for some time now. I try to stay away from hosting entire websites on AWS, just since it is so complex, but S3 is a great solution to use as a plugin for traditional hosting. I use several wordpress plugins that implement S3 offloading for images and it lets you host otherwise large websites on small hosting accounts. Its really great.

Flysystem is an option, but it always seemed to me using flysystem would be insane, since S3 is designed from the ground up to serve the files directly to end users. Putting the extra load / traffic on the server that way just never seemed like a good option.

I think hosting avatars locally is not really a big deal. They are small and it wouldn't be necessary to offload them. The big issue is attachments since they can run into the hundreds of gigabytes in larger forums. Avatars are not an issue.

Obviously the best and cleanest option would be to modify the attachments code to support it. But if that is too complex, perhaps the mod route with support for "external attachments"... something to that effect that implements the attachment option, but just puts a link to the external S3 URL and doesn't handle any of the validations.

Arantor

Given what Flysystem actually is, it is the correct tool for the job, since it's what XenForo amongst others uses (local mode by default, then it's just a config change to move to S3). The whole point is that it abstracts away the physical storage location from the application.

Also you miss my point, it isn't about whether it's "hassle" to store avatars locally... they are literally part of and served by the attachments subsystem by default meaning that you have to deal with avatars in any case.

Also, also, S3 is *not* "designed from the ground up" (and some vendor implementations absolutely not) to serve directly to users. It is a feature it has but that wasn't its original design goal and I'd assume you'd want to preserve access controls with S3 meaning time sensitive signed URLs. This is not trivial to do though Flysystem plus the S3 connector will help enormously.

Lastly, as far as "a mod route" is concerned, I don't think it would be massively simpler, but then again I'm only former SMF dev team and have implemented parts of this in my own world but that's so not usable for SMF it isn't funny.
Holder of controversial views, all of which my own.


ahriman1911

I concede you know a lot more issue about this than me, but my understanding is that if Flysystem was implemented the files get downloaded to the local server and then served from the server to the client, so you add additional latency, since the transit from AWS > Server has to be added.

With a native S3 implementation you just (1) send the URL location to the user and (2) the user's browser fetches it directly from S3, so no extra latency or bandwidth consumed.

If Flysystem needs to download the file to the local server first you duplicate outbound bandwidth (S3 + local server) and add an additional layer if inbound transit AWS > Server.

Arantor

It's really not like that. Again, I even explicitly mentioned the time sensitive URLs part which is a feature of S3, supported by Flysystem and its S3 connector, to serve the file directly from S3 while preserving permissions of it (namely, you don't request the signed URL unless you have authorised the download from the main application)

And it is a massive job to rearchitect the entire file serving system of SMF to even support the naive version that doesn't cleanly redirect to S3.
Holder of controversial views, all of which my own.


ahriman1911

Gotcha. Anyone interested in putting in the work to add S3 (or S3/Flysystem) support for a fee? I'm happy to pay something to help towards getting this up and running.

shadav

you could post in the Help Wanted (not for support) following the rules stickied in that board on how to post :)
though given what arantor has said I would think it's gonna end up being quite costly for someone to put in the time and effort, so might want to keep that in mind, and then to keep it up to date with each new release of smf and maybe php versions

shadav

I didn't want to take away from your paid request post, so I'll post here
but I saw your concern with file size of images
you can install this mod to help a bit with that
https://custom.simplemachines.org/index.php?mod=4082

ahriman1911

Thanks, I'll have a look.

vbgamer45

Have built this for SMF 2.0.x and SMF 2.1.x as a paid mod that handles attachments and attachment thumbnails as option:
https://www.smfhacks.com/s3systemforsmf.php
Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

Advertisement: