News:

Bored?  Looking to kill some time?  Want to chat with other SMF users?  Join us in IRC chat or Discord

Main Menu

Using Amazon's S3 for attached files

Started by Anguz, December 28, 2009, 11:34:48 PM

Previous topic - Next topic

Anguz

Is there a way to configure the forum (2.0 RC2) to use an Amazon S3 hosting account for attached files?
Cristián Lávaque http://cristianlavaque.com

Arantor

If you can attach an S3 instance as if it were a local path (e.g. /home/myuser/attachments really points to an S3 partition), perhaps, but otherwise not without a substantive reworking of things if I'm not mistaken.

Anguz

Well, it'd be like uploading via FTP, I'd guess. So if S3 supports the login data in the URL as does FTP, then maybe... dunno.

I'd need to look into the S3 documentation to see if there's another way (like anonymous uploading kind of thing).
Cristián Lávaque http://cristianlavaque.com

Arantor

Uploading is the easy part. You then have to serve the files again, which means either having the files available "locally" (i.e. not through URLs or FTP) or modifying the code to do it.

Something like that

The bigger question is why would you want to do it?

Anguz

@Mark

Because S3's rates are waaay cheaper for space and bandwidth and super reliable.

@Arantor

Would it be that hard to configure URLs to be taken from another domain?

Anyway, if needed, the S3 service can be configured under a subdomain of the one where the forum is using CNAMEs. Would that help?
Cristián Lávaque http://cristianlavaque.com

Arantor

Quote from: Anguz on December 29, 2009, 11:19:38 AM
Would it be that hard to configure URLs to be taken from another domain?

Apart from gutting the upload system and making that receive them immediately open an FTP connection (risking PHP timeouts etc), followed by gutting the display template to repost.

Oh, it's doable, just it would have to be a code rewrite of substantial parts of code.

H

If you have a Xen/KVM VPS or a dedicated server you could use the S3 FUSE file system (here) to easily add attachments. You'd have to do additional modification to prevent SMF itself from serving the attachments and decide how you're going to handle restricted attachments
-H
Former Support Team Lead
                              I recommend:
Namecheap (domains)
Fastmail (e-mail)
Linode (VPS)
                             

Something like that

A simpler hack might be to use rsync and have it run every 10 minutes or whatever. Then you can hack SMF's display.php to redirect if the attachment is older than say, half an hour (by checking the file modified time).

If you watch for deals, you can get great deals on bandwith. I get 10TB/mn on a $100/mn server, or $0.01 per GB. S3 is waaaaay more expensive than that. I only use 1/6th of the bandwidth, but $0.06/GB is still pretty decent.

Anguz

Is there a way with Apache to make the attachments dir be an alias that'd point to the S3 server? That way SMF's code would remain intact.
Cristián Lávaque http://cristianlavaque.com

Something like that

As in redirect the URL with Apache? Not directly. SMF 2.0 requires that all attachments have encrypted filenames for security reasons. You might be able to configure display.php to send out an HTTP redirect header to a URL on S3 though, and use that with the rsync/delay solution I was talking about.

Anguz

Thanks Mark. As you could probably tell, I'm not very experienced with web servers :P. I searched what rsync is and looks interesting.

No matter what I do, attaching a file will use at least twice the file size in bandwidth, first to upload it to the server where the forum is hosted, and then when copying it to the S3 server, right? It'd still save the serves to forum users, though.

Cristián Lávaque http://cristianlavaque.com

Anguz

Quote from: Arantor on December 29, 2009, 11:31:19 AM
Oh, it's doable, just it would have to be a code rewrite of substantial parts of code.

Got it.

Quote from: H on December 29, 2009, 11:36:27 AM
If you have a Xen/KVM VPS or a dedicated server you could use the S3 FUSE file system (here) to easily add attachments. You'd have to do additional modification to prevent SMF itself from serving the attachments and decide how you're going to handle restricted attachments

Missed that post, sorry!

What you suggest looks very promising. I have no idea how to go about configuring the thing, but I could learn... Read a bit about that over at FuseOverAmazon, but it's way out of what I know how to do.  ::)
Cristián Lávaque http://cristianlavaque.com

Something like that

What is your monthly budget for hosting, anyway?

Anguz

Very, very low at the moment. :P

Actually, I'm hosting my friend's forum in a reseller account I've had for about 5 years and never upgraded, which already has plenty of websites and I don't have much space or bandwidth left in. That's why I thought I'd use Amazon for the attachments, to save myself from upgrading the account or getting a new webhost.

I guess I will start with attachments disabled, have them upload the vids to YouTube and pictures to Picasa, or something like that, and eventually, if the volume of files justifies it, get a new hosting for it.
Cristián Lávaque http://cristianlavaque.com

Something like that

Have you contacted the place with the reseller account about getting upgraded? If you've been with them 5 years, I'm sure their basic package would include more bandwidth and disk space now. Sometimes it's as simple as asking! I got my bandwidth costs cut by 65% before by simply asking.

Anguz

Thanks for the tip, Mark.

I went over there and checked out their basic package and what I have. The package remains the same, but I do have a higher bandwidth and I remember having upgraded that once when they got better bandwidth prices and offered the option to increase our limits. Still, I have most of it allocated to the existing websites, which are mostly for friends and myself.

What I did see is that I can re-allocate disk space and bandwidth (many of the websites are almost dead or closed) and that'll free up a ton of resources that I can use in the more active projects. :)

Still, having an option to host attached files in a server other than the one the forum is running from, would seem like a good feature for some, but I realize it's not one needed by default.
Cristián Lávaque http://cristianlavaque.com

Something like that

I'm glad you were able to solve the problem an easier way :)

Anguz

Cristián Lávaque http://cristianlavaque.com

Advertisement: