Attachments, how to handle old files and growing number of them

Started by 6Dasher, April 23, 2018, 06:34:26 PM

Previous topic - Next topic

6Dasher

My site's been running SMF for years now and I've accumulated around 6GB of attachments over the past 8+ years. How do you guys handle attachments, do you delete after a certain age or just let them be?

NB: I have attachments set up over multiple directories, creating a new one generally every 1.5GB to keep them manageable.
SMF user since 2005.

vbgamer45

6gb doesn't seem like a ton. Is it many files? I would keep them if you have the space otherwise breaks user experience.
Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

6Dasher

Around 30.000 files in 4 directories. Don't think this is any kind of issue technically, just wondered if everyone kept their attachments.
SMF user since 2005.

shawnb61

That's a very good question... 

My host's policies are that it does full backups & incrementals thru your first 10GB, only weekly full backups after 10GB, and **NO** backups if you grow past 20GB.  Although you don't get charged for the extra usage, you lose features. 

We have recently blown past the 10GB point, and I don't like the fact that I've lost my incrementals. 

I don't want to purge old attachments, because many of the oldest attachments at this point aren't available elsewhere (old gear manuals, documentation, in some cases legacy software) at least not all in one place like our forum.  It'd be a shame to purge 'em. 

So...  I've been considering using an online backup feature, such as Amazon Glacier, Backblaze or JungleDisk.  And not relying on my host at all.  Still thinking it thru at the moment. 

You know, my PC I'm sitting at right now has ~2.5TB of storage total...  Kinda ridiculous to be worrying about 10GB...    ::)
Address the process rather than the outcome.  Then, the outcome becomes more likely.   - Fripp

Kindred

Get a better host?

I have 100gb of storage space on my host
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

GL700Wing

Quote from: Kindred on April 23, 2018, 09:58:34 PM
Get a better host?

I have 100gb of storage space on my host
Plenty of hosts offer lots of storage space but not many offer backups to match the allocated storage space.

I use shared hosting and even though my provider does backups their policy is really "best effort - no guarantees" so I backup my websites/forums to an offsite backup location using 'rsync' and a script I wrote. 

I keep a single copy of the attachments folder(s)/images cache on the offsite backup location and never remove any attachments/cache images from the offsite backup so that I always have the option to restore an attachment or, if I ever needed to recreate my forum, to reload the existing images cache.  In addition, I keep a week's worth of synced backups of the rest of the forum folders, a month's worth of database backups and copies of my backup scripts.
Life doesn't have to be perfect to be wonderful ...

Kindred

My host backs up my site daily...  of course, I don't depend on that and do my own offsite autobackup... but they do.
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

6Dasher

I have the space and can always get more space if needed, so this is not a technical issue. I also backup all my sites daily on my host as I have the room, might move the backup elsewhere if it gets too big. I think it's more a philosofical question on how other people (you guys) handle attachments.
SMF user since 2005.

vbgamer45

You can place millions of files in folder.  I would say at your volume is it is ok. The only issue if you do major directory listings on a command line or via ftp then that would be slow.
Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

Kindred

I don't worry about it.   I have forums that have been around for a decade or more... 
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

GigaWatt

Quote from: shawnb61 on April 23, 2018, 09:37:33 PM
You know, my PC I'm sitting at right now has ~2.5TB of storage total...  Kinda ridiculous to be worrying about 10GB...    ::)

Yeah ;). That's why I do my backups locally. Today's drive capacity is so huge, IMO it's impractical to rely on online backup solutions... not to mention that you can compress everything ;).
"This is really a generic concept about human thinking - when faced with large tasks we're naturally inclined to try to break them down into a bunch of smaller tasks that together make up the whole."

"A 500 error loosely translates to the webserver saying, "WTF?"..."

vbgamer45

Just make sure to back to multiple locations that is the main thing and test the backups.
Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

GigaWatt

True ;). Copy to multiple locations and do a CRC check or the copied files. I use TeraCopy, it does it on the fly, don't have to check them afterwards.
"This is really a generic concept about human thinking - when faced with large tasks we're naturally inclined to try to break them down into a bunch of smaller tasks that together make up the whole."

"A 500 error loosely translates to the webserver saying, "WTF?"..."

Sir Osis of Liver

Ashes and diamonds, foe and friend,
 we were all equal in the end.

                                     - R. Waters

GigaWatt

"This is really a generic concept about human thinking - when faced with large tasks we're naturally inclined to try to break them down into a bunch of smaller tasks that together make up the whole."

"A 500 error loosely translates to the webserver saying, "WTF?"..."

Sir Osis of Liver

Server.  Was doing a salvage job on a forum with 20k files in a single attachment directory, trying to upload them with FileZilla.  After upload completed, FZ only displayed 10k files.  Wasted a lot of time reuploading several gigs of files, contacted host support, they could see all 20k files.  They changed something in server config, then I could see them in FZ.
Ashes and diamonds, foe and friend,
 we were all equal in the end.

                                     - R. Waters

EthanR

Been wondering what to do with my attachment directory. At what point do I create a second directory for the attachments, currently I'm sitting on 10000 files in the first directory. Are there any disadvantages to using more than 1 attachment directory in SMF?

What Sir Osis has come across is what I'm afraid of down the line..

Sir Osis of Liver

I wouldn't have more than 10,000 files in one attachment directory.  Costs nothing to add additional directories, and makes backups and file management a lot easier.
Ashes and diamonds, foe and friend,
 we were all equal in the end.

                                     - R. Waters

Advertisement: