Advertisement:

Author Topic: Using Amazon's S3 for attached files  (Read 4631 times)

Offline Anguz

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 3,430
  • Gender: Male
    • cristianlavaque.com
Using Amazon's S3 for attached files
« on: December 28, 2009, 11:34:48 PM »
Is there a way to configure the forum (2.0 RC2) to use an Amazon S3 hosting account for attached files?
Cristián Lávaque http://cristianlavaque.com

Offline Arantor

  • Resident Overthinker
  • SMF Friend
  • SMF Legend
  • *
  • Posts: 73,098
Re: Using Amazon's S3 for attached files
« Reply #1 on: December 29, 2009, 02:31:28 AM »
If you can attach an S3 instance as if it were a local path (e.g. /home/myuser/attachments really points to an S3 partition), perhaps, but otherwise not without a substantive reworking of things if I'm not mistaken.
No good deed goes unpunished
All helpful urges should be circumvented

Offline Anguz

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 3,430
  • Gender: Male
    • cristianlavaque.com
Re: Using Amazon's S3 for attached files
« Reply #2 on: December 29, 2009, 02:38:46 AM »
Well, it'd be like uploading via FTP, I'd guess. So if S3 supports the login data in the URL as does FTP, then maybe... dunno.

I'd need to look into the S3 documentation to see if there's another way (like anonymous uploading kind of thing).
Cristián Lávaque http://cristianlavaque.com

Offline Arantor

  • Resident Overthinker
  • SMF Friend
  • SMF Legend
  • *
  • Posts: 73,098
Re: Using Amazon's S3 for attached files
« Reply #3 on: December 29, 2009, 02:43:31 AM »
Uploading is the easy part. You then have to serve the files again, which means either having the files available "locally" (i.e. not through URLs or FTP) or modifying the code to do it.
No good deed goes unpunished
All helpful urges should be circumvented

Offline Something like that

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 2,496
  • Gender: Male
  • ]
Re: Using Amazon's S3 for attached files
« Reply #4 on: December 29, 2009, 11:11:24 AM »
The bigger question is why would you want to do it?

Offline Anguz

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 3,430
  • Gender: Male
    • cristianlavaque.com
Re: Using Amazon's S3 for attached files
« Reply #5 on: December 29, 2009, 11:19:38 AM »
@Mark

Because S3's rates are waaay cheaper for space and bandwidth and super reliable.

@Arantor

Would it be that hard to configure URLs to be taken from another domain?

Anyway, if needed, the S3 service can be configured under a subdomain of the one where the forum is using CNAMEs. Would that help?
Cristián Lávaque http://cristianlavaque.com

Offline Arantor

  • Resident Overthinker
  • SMF Friend
  • SMF Legend
  • *
  • Posts: 73,098
Re: Using Amazon's S3 for attached files
« Reply #6 on: December 29, 2009, 11:31:19 AM »
Would it be that hard to configure URLs to be taken from another domain?

Apart from gutting the upload system and making that receive them immediately open an FTP connection (risking PHP timeouts etc), followed by gutting the display template to repost.

Oh, it's doable, just it would have to be a code rewrite of substantial parts of code.
No good deed goes unpunished
All helpful urges should be circumvented

Offline H

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 21,662
  • Gender: Male
Re: Using Amazon's S3 for attached files
« Reply #7 on: December 29, 2009, 11:36:27 AM »
If you have a Xen/KVM VPS or a dedicated server you could use the S3 FUSE file system (here) to easily add attachments. You'd have to do additional modification to prevent SMF itself from serving the attachments and decide how you're going to handle restricted attachments
-H
Former Support Team Lead
                              I recommend:
Namecheap (domains)
Fastmail (e-mail)
Linode (VPS)
                             

Offline Something like that

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 2,496
  • Gender: Male
  • ]
Re: Using Amazon's S3 for attached files
« Reply #8 on: December 29, 2009, 11:38:51 AM »
A simpler hack might be to use rsync and have it run every 10 minutes or whatever. Then you can hack SMF's display.php to redirect if the attachment is older than say, half an hour (by checking the file modified time).

If you watch for deals, you can get great deals on bandwith. I get 10TB/mn on a $100/mn server, or $0.01 per GB. S3 is waaaaay more expensive than that. I only use 1/6th of the bandwidth, but $0.06/GB is still pretty decent.

Offline Anguz

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 3,430
  • Gender: Male
    • cristianlavaque.com
Re: Using Amazon's S3 for attached files
« Reply #9 on: December 29, 2009, 10:18:35 PM »
Is there a way with Apache to make the attachments dir be an alias that'd point to the S3 server? That way SMF's code would remain intact.
Cristián Lávaque http://cristianlavaque.com

Offline Something like that

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 2,496
  • Gender: Male
  • ]
Re: Using Amazon's S3 for attached files
« Reply #10 on: December 29, 2009, 10:22:36 PM »
As in redirect the URL with Apache? Not directly. SMF 2.0 requires that all attachments have encrypted filenames for security reasons. You might be able to configure display.php to send out an HTTP redirect header to a URL on S3 though, and use that with the rsync/delay solution I was talking about.

Offline Anguz

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 3,430
  • Gender: Male
    • cristianlavaque.com
Re: Using Amazon's S3 for attached files
« Reply #11 on: December 29, 2009, 11:49:49 PM »
Thanks Mark. As you could probably tell, I'm not very experienced with web servers :P. I searched what rsync is and looks interesting.

No matter what I do, attaching a file will use at least twice the file size in bandwidth, first to upload it to the server where the forum is hosted, and then when copying it to the S3 server, right? It'd still save the serves to forum users, though.

Cristián Lávaque http://cristianlavaque.com

Offline Anguz

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 3,430
  • Gender: Male
    • cristianlavaque.com
Re: Using Amazon's S3 for attached files
« Reply #12 on: December 29, 2009, 11:57:07 PM »
Oh, it's doable, just it would have to be a code rewrite of substantial parts of code.

Got it.

If you have a Xen/KVM VPS or a dedicated server you could use the S3 FUSE file system (here) to easily add attachments. You'd have to do additional modification to prevent SMF itself from serving the attachments and decide how you're going to handle restricted attachments

Missed that post, sorry!

What you suggest looks very promising. I have no idea how to go about configuring the thing, but I could learn... Read a bit about that over at FuseOverAmazon, but it's way out of what I know how to do.  ::)
Cristián Lávaque http://cristianlavaque.com

Offline Something like that

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 2,496
  • Gender: Male
  • ]
Re: Using Amazon's S3 for attached files
« Reply #13 on: December 30, 2009, 01:58:00 AM »
What is your monthly budget for hosting, anyway?

Offline Anguz

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 3,430
  • Gender: Male
    • cristianlavaque.com
Re: Using Amazon's S3 for attached files
« Reply #14 on: December 30, 2009, 11:05:16 AM »
Very, very low at the moment. :P

Actually, I'm hosting my friend's forum in a reseller account I've had for about 5 years and never upgraded, which already has plenty of websites and I don't have much space or bandwidth left in. That's why I thought I'd use Amazon for the attachments, to save myself from upgrading the account or getting a new webhost.

I guess I will start with attachments disabled, have them upload the vids to YouTube and pictures to Picasa, or something like that, and eventually, if the volume of files justifies it, get a new hosting for it.
Cristián Lávaque http://cristianlavaque.com

Offline Something like that

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 2,496
  • Gender: Male
  • ]
Re: Using Amazon's S3 for attached files
« Reply #15 on: December 30, 2009, 11:21:20 AM »
Have you contacted the place with the reseller account about getting upgraded? If you've been with them 5 years, I'm sure their basic package would include more bandwidth and disk space now. Sometimes it's as simple as asking! I got my bandwidth costs cut by 65% before by simply asking.

Offline Anguz

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 3,430
  • Gender: Male
    • cristianlavaque.com
Re: Using Amazon's S3 for attached files
« Reply #16 on: December 30, 2009, 11:41:29 AM »
Thanks for the tip, Mark.

I went over there and checked out their basic package and what I have. The package remains the same, but I do have a higher bandwidth and I remember having upgraded that once when they got better bandwidth prices and offered the option to increase our limits. Still, I have most of it allocated to the existing websites, which are mostly for friends and myself.

What I did see is that I can re-allocate disk space and bandwidth (many of the websites are almost dead or closed) and that'll free up a ton of resources that I can use in the more active projects. :)

Still, having an option to host attached files in a server other than the one the forum is running from, would seem like a good feature for some, but I realize it's not one needed by default.
Cristián Lávaque http://cristianlavaque.com

Offline Something like that

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 2,496
  • Gender: Male
  • ]
Re: Using Amazon's S3 for attached files
« Reply #17 on: December 30, 2009, 11:47:11 AM »
I'm glad you were able to solve the problem an easier way :)

Offline Anguz

  • SMF Friend
  • SMF Hero
  • *
  • Posts: 3,430
  • Gender: Male
    • cristianlavaque.com
Re: Using Amazon's S3 for attached files
« Reply #18 on: December 30, 2009, 11:54:29 AM »
Me too. :D

Thanks a lot.
Cristián Lávaque http://cristianlavaque.com