Attachments, how to handle old files and growing number of them

[6Dasher]

My site's been running SMF for years now and I've accumulated around 6GB of attachments over the past 8+ years. How do you guys handle attachments, do you delete after a certain age or just let them be?

NB: I have attachments set up over multiple directories, creating a new one generally every 1.5GB to keep them manageable.

[vbgamer45]

6gb doesn't seem like a ton. Is it many files? I would keep them if you have the space otherwise breaks user experience.

[6Dasher]

Around 30.000 files in 4 directories. Don't think this is any kind of issue technically, just wondered if everyone kept their attachments.

[shawnb61]

That's a very good question... 

My host's policies are that it does full backups & incrementals thru your first 10GB, only weekly full backups after 10GB, and **NO** backups if you grow past 20GB.  Although you don't get charged for the extra usage, you lose features. 

We have recently blown past the 10GB point, and I don't like the fact that I've lost my incrementals. 

I don't want to purge old attachments, because many of the oldest attachments at this point aren't available elsewhere (old gear manuals, documentation, in some cases legacy software) at least not all in one place like our forum.  It'd be a shame to purge 'em. 

So...  I've been considering using an online backup feature, such as Amazon Glacier, Backblaze or JungleDisk.  And not relying on my host at all.  Still thinking it thru at the moment. 

You know, my PC I'm sitting at right now has ~2.5TB of storage total...  Kinda ridiculous to be worrying about 10GB...   

[Kindred]

Get a better host?

I have 100gb of storage space on my host

[GL700Wing]

Get a better host?

I have 100gb of storage space on my host

Plenty of hosts offer lots of storage space but not many offer backups to match the allocated storage space.

I use shared hosting and even though my provider does backups their policy is really "best effort - no guarantees" so I backup my websites/forums to an offsite backup location using 'rsync' and a script I wrote. 

I keep a single copy of the attachments folder(s)/images cache on the offsite backup location and never remove any attachments/cache images from the offsite backup so that I always have the option to restore an attachment or, if I ever needed to recreate my forum, to reload the existing images cache.  In addition, I keep a week's worth of synced backups of the rest of the forum folders, a month's worth of database backups and copies of my backup scripts.

[Kindred]

My host backs up my site daily...  of course, I don't depend on that and do my own offsite autobackup... but they do.

[6Dasher]

I have the space and can always get more space if needed, so this is not a technical issue. I also backup all my sites daily on my host as I have the room, might move the backup elsewhere if it gets too big. I think it's more a philosofical question on how other people (you guys) handle attachments.

[vbgamer45]

You can place millions of files in folder.  I would say at your volume is it is ok. The only issue if you do major directory listings on a command line or via ftp then that would be slow.

[Kindred]

I don't worry about it.   I have forums that have been around for a decade or more... 

[GigaWatt]

You know, my PC I'm sitting at right now has ~2.5TB of storage total...  Kinda ridiculous to be worrying about 10GB...   

Yeah . That's why I do my backups locally. Today's drive capacity is so huge, IMO it's impractical to rely on online backup solutions... not to mention that you can compress everything .

[vbgamer45]

Just make sure to back to multiple locations that is the main thing and test the backups.

[GigaWatt]

True . Copy to multiple locations and do a CRC check or the copied files. I use TeraCopy, it does it on the fly, don't have to check them afterwards.

[Sir Osis of Liver]

Some ftp configurations will choke on >10,000 files.

[GigaWatt]

You mean servers or clients?

[Sir Osis of Liver]

Server.  Was doing a salvage job on a forum with 20k files in a single attachment directory, trying to upload them with FileZilla.  After upload completed, FZ only displayed 10k files.  Wasted a lot of time reuploading several gigs of files, contacted host support, they could see all 20k files.  They changed something in server config, then I could see them in FZ.

[EthanR]

Been wondering what to do with my attachment directory. At what point do I create a second directory for the attachments, currently I'm sitting on 10000 files in the first directory. Are there any disadvantages to using more than 1 attachment directory in SMF?

What Sir Osis has come across is what I'm afraid of down the line..

[Sir Osis of Liver]

I wouldn't have more than 10,000 files in one attachment directory.  Costs nothing to add additional directories, and makes backups and file management a lot easier.