Advertisement:
« previous next »
Pages: [1]   Go Down

Author Topic: Attachments, how to handle old files and growing number of them  (Read 4386 times)

Offline 6Dasher

  • Jr. Member
  • **
  • Posts: 109
Attachments, how to handle old files and growing number of them
« on: April 23, 2018, 06:34:26 PM »
My site's been running SMF for years now and I've accumulated around 6GB of attachments over the past 8+ years. How do you guys handle attachments, do you delete after a certain age or just let them be?

NB: I have attachments set up over multiple directories, creating a new one generally every 1.5GB to keep them manageable.
Logged
SMF user since 2005.

Offline vbgamer45

  • Customizer
  • SMF Super Hero
  • *
  • Posts: 23,474
    • smfhacks on Facebook
    • VBGAMER45 on GitHub
    • @createaforum on Twitter
    • SMF For Free
Re: Attachments, how to handle old files and growing number of them
« Reply #1 on: April 23, 2018, 06:49:23 PM »
6gb doesn't seem like a ton. Is it many files? I would keep them if you have the space otherwise breaks user experience.
Logged
Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

Offline 6Dasher

  • Jr. Member
  • **
  • Posts: 109
Re: Attachments, how to handle old files and growing number of them
« Reply #2 on: April 23, 2018, 06:54:43 PM »
Around 30.000 files in 4 directories. Don't think this is any kind of issue technically, just wondered if everyone kept their attachments.
Logged
SMF user since 2005.

Offline shawnb61

  • Dev. Team Mentor
  • Developer
  • SMF Hero
  • *
  • Posts: 3,113
    • sbulen on GitHub
Re: Attachments, how to handle old files and growing number of them
« Reply #3 on: April 23, 2018, 09:37:33 PM »
That's a very good question... 

My host's policies are that it does full backups & incrementals thru your first 10GB, only weekly full backups after 10GB, and **NO** backups if you grow past 20GB.  Although you don't get charged for the extra usage, you lose features. 

We have recently blown past the 10GB point, and I don't like the fact that I've lost my incrementals. 

I don't want to purge old attachments, because many of the oldest attachments at this point aren't available elsewhere (old gear manuals, documentation, in some cases legacy software) at least not all in one place like our forum.  It'd be a shame to purge 'em. 

So...  I've been considering using an online backup feature, such as Amazon Glacier, Backblaze or JungleDisk.  And not relying on my host at all.  Still thinking it thru at the moment. 

You know, my PC I'm sitting at right now has ~2.5TB of storage total...  Kinda ridiculous to be worrying about 10GB...    ::)
Logged
Address the process rather than the outcome.  Then, the outcome becomes more likely.   - Fripp

Offline Kindred

  • The Mean One
  • Support Specialist
  • SMF Legend
  • *
  • Posts: 60,210
  • Gender: Male
    • Kindred-999 on GitHub
Re: Attachments, how to handle old files and growing number of them
« Reply #4 on: April 23, 2018, 09:58:34 PM »
Get a better host?

I have 100gb of storage space on my host
Logged
Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

Offline GL700Wing

  • Full Member
  • ***
  • Posts: 666
  • Gender: Female
Re: Attachments, how to handle old files and growing number of them
« Reply #5 on: April 23, 2018, 10:24:20 PM »
Quote from: Kindred on April 23, 2018, 09:58:34 PM
Get a better host?

I have 100gb of storage space on my host
Plenty of hosts offer lots of storage space but not many offer backups to match the allocated storage space.

I use shared hosting and even though my provider does backups their policy is really "best effort - no guarantees" so I backup my websites/forums to an offsite backup location using 'rsync' and a script I wrote. 

I keep a single copy of the attachments folder(s)/images cache on the offsite backup location and never remove any attachments/cache images from the offsite backup so that I always have the option to restore an attachment or, if I ever needed to recreate my forum, to reload the existing images cache.  In addition, I keep a week's worth of synced backups of the rest of the forum folders, a month's worth of database backups and copies of my backup scripts.
Logged
Life doesn't have to be perfect to be wonderful ...

Offline Kindred

  • The Mean One
  • Support Specialist
  • SMF Legend
  • *
  • Posts: 60,210
  • Gender: Male
    • Kindred-999 on GitHub
Re: Attachments, how to handle old files and growing number of them
« Reply #6 on: April 23, 2018, 10:31:37 PM »
My host backs up my site daily...  of course, I don’t depend on that and do my own offsite autobackup... but they do.
Logged
Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

Offline 6Dasher

  • Jr. Member
  • **
  • Posts: 109
Re: Attachments, how to handle old files and growing number of them
« Reply #7 on: April 23, 2018, 11:38:01 PM »
I have the space and can always get more space if needed, so this is not a technical issue. I also backup all my sites daily on my host as I have the room, might move the backup elsewhere if it gets too big. I think it's more a philosofical question on how other people (you guys) handle attachments.
Logged
SMF user since 2005.

Offline vbgamer45

  • Customizer
  • SMF Super Hero
  • *
  • Posts: 23,474
    • smfhacks on Facebook
    • VBGAMER45 on GitHub
    • @createaforum on Twitter
    • SMF For Free
Re: Attachments, how to handle old files and growing number of them
« Reply #8 on: April 23, 2018, 11:47:36 PM »
You can place millions of files in folder.  I would say at your volume is it is ok. The only issue if you do major directory listings on a command line or via ftp then that would be slow.
Logged
Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

Offline Kindred

  • The Mean One
  • Support Specialist
  • SMF Legend
  • *
  • Posts: 60,210
  • Gender: Male
    • Kindred-999 on GitHub
Re: Attachments, how to handle old files and growing number of them
« Reply #9 on: April 24, 2018, 08:52:07 AM »
I don't worry about it.   I have forums that have been around for a decade or more... 
Logged
Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

Offline GigaWatt

  • The Smiley Guy
  • SMF Friend
  • SMF Hero
  • *
  • Posts: 2,341
  • Gender: Male
    • MechDR on GitHub
    • Macedonian electronics forum
Re: Attachments, how to handle old files and growing number of them
« Reply #10 on: April 24, 2018, 05:04:48 PM »
Quote from: shawnb61 on April 23, 2018, 09:37:33 PM
You know, my PC I'm sitting at right now has ~2.5TB of storage total...  Kinda ridiculous to be worrying about 10GB...    ::)

Yeah ;). That's why I do my backups locally. Today's drive capacity is so huge, IMO it's impractical to rely on online backup solutions... not to mention that you can compress everything ;).
Logged
"This is really a generic concept about human thinking - when faced with large tasks we're naturally inclined to try to break them down into a bunch of smaller tasks that together make up the whole."

"A 500 error loosely translates to the webserver saying, "WTF?"..."

Offline vbgamer45

  • Customizer
  • SMF Super Hero
  • *
  • Posts: 23,474
    • smfhacks on Facebook
    • VBGAMER45 on GitHub
    • @createaforum on Twitter
    • SMF For Free
Re: Attachments, how to handle old files and growing number of them
« Reply #11 on: April 24, 2018, 05:07:18 PM »
Just make sure to back to multiple locations that is the main thing and test the backups.
Logged
Community Suite for SMF - Take your forum to the next level built for SMF, Gallery,Store,Classifieds,Downloads,more!

SMFHacks.com -  Paid Modifications for SMF

Mods:
EzPortal - Portal System for SMF
SMF Gallery Pro
SMF Store SMF Classifieds Ad Seller Pro

Offline GigaWatt

  • The Smiley Guy
  • SMF Friend
  • SMF Hero
  • *
  • Posts: 2,341
  • Gender: Male
    • MechDR on GitHub
    • Macedonian electronics forum
Re: Attachments, how to handle old files and growing number of them
« Reply #12 on: April 24, 2018, 05:38:06 PM »
True ;). Copy to multiple locations and do a CRC check or the copied files. I use TeraCopy, it does it on the fly, don't have to check them afterwards.
Logged
"This is really a generic concept about human thinking - when faced with large tasks we're naturally inclined to try to break them down into a bunch of smaller tasks that together make up the whole."

"A 500 error loosely translates to the webserver saying, "WTF?"..."

Offline Sir Osis of Liver

  • SMF Super Hero
  • *******
  • Posts: 10,940
  • We were all equal in the end.
Re: Attachments, how to handle old files and growing number of them
« Reply #13 on: April 24, 2018, 05:39:58 PM »
Some ftp configurations will choke on >10,000 files.
Logged
“The best laid schemes o' mice an' men / Gang aft a-gley.” - Robert Burns

Offline GigaWatt

  • The Smiley Guy
  • SMF Friend
  • SMF Hero
  • *
  • Posts: 2,341
  • Gender: Male
    • MechDR on GitHub
    • Macedonian electronics forum
Re: Attachments, how to handle old files and growing number of them
« Reply #14 on: April 24, 2018, 05:46:41 PM »
You mean servers or clients?
Logged
"This is really a generic concept about human thinking - when faced with large tasks we're naturally inclined to try to break them down into a bunch of smaller tasks that together make up the whole."

"A 500 error loosely translates to the webserver saying, "WTF?"..."

Offline Sir Osis of Liver

  • SMF Super Hero
  • *******
  • Posts: 10,940
  • We were all equal in the end.
Re: Attachments, how to handle old files and growing number of them
« Reply #15 on: April 24, 2018, 05:55:33 PM »
Server.  Was doing a salvage job on a forum with 20k files in a single attachment directory, trying to upload them with FileZilla.  After upload completed, FZ only displayed 10k files.  Wasted a lot of time reuploading several gigs of files, contacted host support, they could see all 20k files.  They changed something in server config, then I could see them in FZ.
Logged
“The best laid schemes o' mice an' men / Gang aft a-gley.” - Robert Burns

Offline EthanR

  • Semi-Newbie
  • *
  • Posts: 36
Re: Attachments, how to handle old files and growing number of them
« Reply #16 on: September 05, 2018, 08:17:33 PM »
Been wondering what to do with my attachment directory. At what point do I create a second directory for the attachments, currently I'm sitting on 10000 files in the first directory. Are there any disadvantages to using more than 1 attachment directory in SMF?

What Sir Osis has come across is what I'm afraid of down the line..
Logged

Offline Sir Osis of Liver

  • SMF Super Hero
  • *******
  • Posts: 10,940
  • We were all equal in the end.
Re: Attachments, how to handle old files and growing number of them
« Reply #17 on: September 05, 2018, 10:42:05 PM »
I wouldn't have more than 10,000 files in one attachment directory.  Costs nothing to add additional directories, and makes backups and file management a lot easier.
Logged
“The best laid schemes o' mice an' men / Gang aft a-gley.” - Robert Burns
Pages: [1]   Go Up
« previous next »