Huge amount of photos - file management / managing attachments?

Started by ModelBoatMayhem, August 28, 2016, 12:37:22 PM

Previous topic - Next topic

ModelBoatMayhem



Hey everyone!

My Forum is very heavily picture based but this has become a growing issue over the last few years, we now have over 8G's worth of photos!
I can't find any way of finding out but at a guess, over 110,000!

I was hoping that SMF2+ would have multiple directories for attachment handling but it didn't seem to materialize, well not that I can find!

1. Is there a better way of handling thousands of photos in 2+ that I don't know about?
2. What file manager will allow me to see that may and manage that many files?

Thanks, Martin - http://www.modelboIyhem.co.uk/forum/


That's my firm opinion.... but what do I know?!

Illori

SMF 2.0 does support multiple attachment directories

http://wiki.simplemachines.org/smf/SMF2.0:Attachments_and_Avatars#Attachment_Settings

Configure multiple attachment directories - This link will take you to another screen which allows you to configure multiple directories for your attachments. This is particularly useful as your site gets bigger, since there is a limit to the total number of files in (or total size of) a directory.

ModelBoatMayhem

OK, thanks Illori:

Quote
Configure multiple attachment directories - This link will take you to another screen which allows you to configure multiple directories for your attachments.
  This is particularly useful as your site gets bigger, since there is a limit to the total number of files in (or total size of) a directory.

Does this mean I can only set one active Folder?
What happens to old my old attachments in the old folder?

That's my firm opinion.... but what do I know?!

Kindred

They stay... The active directory is the one where NEW content gets added
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

ModelBoatMayhem

That's my firm opinion.... but what do I know?!

a10

^^^ Yes, attachment folder can be split, have done it and all well. Long time ago so don't remember how :O) but made some posts about it, search. Now got many folders, and adding a new folder when reaching approx 5000 files.

Should be no problem, but first backup everything (database with mysql export or mysqldumper, and all files with ftp).
2.0.19, php 8.0.23, MariaDB 10.5.15. Mods: Contact Page, Like Posts, Responsive Curve, Search Focus Dropdown, Add Join Date to Post.

Illori

i am not sure without database changes that you can split up the existing attachments. so if you dont know what you are doing i would not recommend touching the existing attachments.

ModelBoatMayhem

That's my firm opinion.... but what do I know?!

Kindred

Actually, having done it - I am certain that it REQUIRES database changes to the data in order to properly split an existing directory
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

ModelBoatMayhem


If Anyone has instructions oe willing to take it on, I'd be most interested in hearing from them!
That's my firm opinion.... but what do I know?!

a10

Just made a test to recollect what I did long time ago (distributing approx 20.000 attachments from one (too big) folder into 4 folders).

My test: picked a thumbnail + it's original in folder 1, moved them to folder 2 (= pictures 404 in post), then run File Maintenance > Attachment Integrity Check. Fixed (pictures showing in post). See attach.

Very easy, no need for any database work. Test first with a few attachments to get the hang of it. Of course, backup before doing anything.
2.0.19, php 8.0.23, MariaDB 10.5.15. Mods: Contact Page, Like Posts, Responsive Curve, Search Focus Dropdown, Add Join Date to Post.

ModelBoatMayhem

 
2. What file manager will allow me to see that may and manage that many files?
That's my firm opinion.... but what do I know?!

a10

Quote from: ModelBoIyhem on August 30, 2016, 12:10:49 PM
2. What file manager will allow me to see that may and manage that many files?

^^^ Don't know. In ftp could only see 9999 files in the original attachment folder (hosting number limit for displaying, but all files were there, but not listed in ftp), started moving files to att2, then att3, then att4 until the original att folder went down to approx 5000 files, and filled the other folders to around 5000 files.

Did it in many steps over time, moved a good number then run Attachment Integrity Check, repeating until done. Many years since this was done, everything working great.
2.0.19, php 8.0.23, MariaDB 10.5.15. Mods: Contact Page, Like Posts, Responsive Curve, Search Focus Dropdown, Add Join Date to Post.

ModelBoatMayhem

#13

Tried FTP, cPanel & FileZilla, but all 'time out' before opening the directory.

Can I move files within SMF or which file manager will help me?   ???
That's my firm opinion.... but what do I know?!

a10

I'd contact the host for assistance. I remember the ftp listing were very slow here with (too) many files, but always (sometimes had to try\refresh a few times) got the first 9999 showing up so could start cleaning up.

btw... if no working ftp \ cpanel, how are you managing to backup ?? (don't say there's no backups...! :O)
2.0.19, php 8.0.23, MariaDB 10.5.15. Mods: Contact Page, Like Posts, Responsive Curve, Search Focus Dropdown, Add Join Date to Post.

ModelBoatMayhem


Yes, host are running a special script for me tonight.
Splitting one folder to 8 in one go! ..... nervous!   :-X
That's my firm opinion.... but what do I know?!

ModelBoatMayhem


Well a couple of hiccups but were done! ...... phew!

Nine new date related directories and all now easily manageable with any file manager.
If you find yourself in this situation, speak to your hosting company first, they can be very helpful when they want to be!

.... must be near renewal time!   ;)
That's my firm opinion.... but what do I know?!

Advertisement: