News:

SMF 2.1.4 has been released! Take it for a spin! Read more.

Main Menu

maximum files per folder etc

Started by Moerpot, September 26, 2013, 09:44:32 AM

Previous topic - Next topic

Moerpot

I would like to know if there is somehow a method that restrict/or set the maximum files that can be saved in a specified folder?

Let us take a quick peek at this forum's community folder for instance:
Currently there are more than 6510 files in this folder alone.
Next month there will be at least another 2000 files added (for each topic)

(Not many topics done since 2003, but still it's growing at a fast speed)
The more files in one folder, the slower the speed of page transitions.

And also, can some of these files (topics) be moved at a later stage to another physical (archive) folder? {Read as: Delete/Move inactive users and Old topics}

Arantor

Are you talking about files or topics? Topics aren't saved as files anywhere, so I presume you're talking about boards and topics.

When a board is getting full, just change the permissions on it to stop users adding new topics and give them a board for it.

There is a facility in the admin panel that I haven't used in a while for moving topics but I'm not sure it's selective enough for your purposes.

Moerpot

Each topic is saved as a .html file in the folder. Maybe it's also saved inside the database - I believe so. But that led me to another question:
If the topic in the database can be "moved", then the physical file in the folder must also be moved...

This became tricky.

OK. Let me do it a this way:
I will upload a file called indexe582.html to one of my domains, outside of this forum. Then you load it. Maybe it will make more sense to you then.

Here is the link:
http://www.zir.za.net/indexe582.html [nofollow]

Arantor

So you've saved a page from this site to your server... right... I don't understand how that's relevant.

Topics in the forum are not saved as files in the folder - ever. They are in the database and they are retrieved from the database as needed.

Moerpot

I did not "saved" anything from the forum. That file came directly from your server.

However, if you type name directly in the address, it shows an error page, as the topic is in the database list and do not list what is on the hard drive. But still, the file is on the server in that designated folder. (and more than 6510 other files (at least))
www.simplemachines.org/community/indexe582.html

More than this, I cannot give a better understanding.

Unless:
Look at the forum from a DOS or FTP perspective. With other words, see the forum as a folder with sub-folders with files in it.
In the /community/ folder are stored all the topics ever written in this forum.

Do yourself the favor, maybe just to understand better what I try to ask you: Mirror this forum to your hard drive.
Please?

Kindred

#5
that "file" did not come from our sevrer - because there is no such FILE on our server.

All content for the SMF forum is stored in the database - not as flat files.

try to duplicate referential database content as flat files is just plain stupid.
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

Arantor

Quotewww.simplemachines.org/community/indexe582.html

Doesn't exist.

QuoteDo yourself the favor, maybe just to understand better what I try to ask you: Mirror this forum to your hard drive.
Please?

Aside from the small fact of downloading a multi-gigabyte database, I can guarantee you that would be a waste of my time.

When a URL has ? in it, it means it requests the file before the ? and passes information to it from after. Everything goes through index.php - literally everything. The request is passed to index.php with things like the topic as a parameter to it. There aren't separate files for everything.

I could show you the various installations I have if you like where I could show you the files that aren't present.

Moerpot

#7
OK. Here it goes:
This currently all the main folders on your server:
[www.trolling.com.au] <DIR> 2013/09/26 15:46 ----
[feeds.d15.biz] <DIR> 2013/09/26 15:35 ----
[1.bp.blogspot.com] <DIR> 2013/09/26 15:48 ----
[24.media.tumblr.com] <DIR> 2013/09/26 14:13 ----
[avatars.simplemachinesweb.com] <DIR> 2013/09/26 14:07 ----
[cache.www.gametracker.com] <DIR> 2013/09/26 15:35 ----
[chalkcat.com] <DIR> 2013/09/26 15:34 ----
[charlottezweb.com] <DIR> 2013/09/26 15:44 ----
[cripspix.webstarts.com] <DIR> 2013/09/26 15:20 ----
[data.whicdn.com] <DIR> 2013/09/26 15:40 ----
[engrz.com] <DIR> 2013/09/26 15:48 ----
[feeds2.feedburner.com] <DIR> 2013/09/26 15:41 ----
[files4design.com] <DIR> 2013/09/26 14:12 ----
[gametechmods.com] <DIR> 2013/09/26 15:44 ----
[garygadsdon.com] <DIR> 2013/09/26 15:33 ----
[i.imgur.com] <DIR> 2013/09/26 15:49 ----
[i103.photobucket.com] <DIR> 2013/09/26 15:45 ----
[i11.tinypic.com] <DIR> 2013/09/26 15:45 ----
[i1194.photobucket.com] <DIR> 2013/09/26 14:12 ----
[i127.photobucket.com] <DIR> 2013/09/26 15:40 ----
[i164.photobucket.com] <DIR> 2013/09/26 15:40 ----
[i2.tinypic.com] <DIR> 2013/09/26 15:35 ----
[i210.photobucket.com] <DIR> 2013/09/26 15:33 ----
[i239.photobucket.com] <DIR> 2013/09/26 15:42 ----
[i258.photobucket.com] <DIR> 2013/09/26 15:48 ----
[i301.photobucket.com] <DIR> 2013/09/26 14:17 ----
[i34.photobucket.com] <DIR> 2013/09/26 14:14 ----
[i357.photobucket.com] <DIR> 2013/09/26 15:45 ----
[i39.tinypic.com] <DIR> 2013/09/26 15:46 ----
[i50.tinypic.com] <DIR> 2013/09/26 15:44 ----
[i697.photobucket.com] <DIR> 2013/09/26 14:14 ----
[idesign360.com] <DIR> 2013/09/26 15:33 ----
[img.honorshard.com] <DIR> 2013/09/26 15:46 ----
[img.photobucket.com] <DIR> 2013/09/26 15:36 ----
[img16.photobucket.com] <DIR> 2013/09/26 15:39 ----
[jokesaboutcancer.files.wordpress.com] <DIR> 2013/09/26 15:45 ----
[labradoodle-360.com] <DIR> 2013/09/26 15:28 ----
[media.simplemachinesweb.com] <DIR> 2013/09/26 14:03 ----
[missallsunday.com] <DIR> 2013/09/26 15:26 ----
[mugenguild.com] <DIR> 2013/09/26 15:35 ----
[myfolderz.com] <DIR> 2013/09/26 15:44 ----
[norcalcarpclub.com] <DIR> 2013/09/26 15:44 ----
[oi47.tinypic.com] <DIR> 2013/09/26 15:34 ----
[ourcountryhaven.com] <DIR> 2013/09/26 14:14 ----
[outdoorclubjapan.com] <DIR> 2013/09/26 15:42 ----
[pagead2.googlesyndication.com] <DIR> 2013/09/26 14:03 ----
[portamx.com] <DIR> 2013/09/26 14:20 ----
[rockciclopedia.com] <DIR> 2013/09/26 15:42 ----
[smf-media.com] <DIR> 2013/09/26 15:43 ----
[test.xptsp.com] <DIR> 2013/09/26 15:30 ----
[tickers.tickerfactory.com] <DIR> 2013/09/26 14:12 ----
[u.cubeupload.com] <DIR> 2013/09/26 15:48 ----
[www.avatarsdb.com] <DIR> 2013/09/26 14:08 ----
[www.banahosting.com] <DIR> 2013/09/26 15:36 ----
[www.bordeglobal.com] <DIR> 2013/09/26 14:12 ----
[www.charlottezweb.com] <DIR> 2013/09/26 15:45 ----
[www.choppix.com] <DIR> 2013/09/26 15:45 ----
[www.firstpost.com] <DIR> 2013/09/26 14:17 ----
[www.forumscp.com] <DIR> 2013/09/26 15:49 ----
[www.gravure-et-traductions.com] <DIR> 2013/09/26 14:14 ----
[www.idesign360.com] <DIR> 2013/09/26 15:33 ----
[www.katzy.dsl.pipex.com] <DIR> 2013/09/26 15:37 ----
[www.kayssplace.com] <DIR> 2013/09/26 15:11 ----
[www.likeateam.com] <DIR> 2013/09/26 15:36 ----
[www.malasrpskaprodavnica.com] <DIR> 2013/09/26 15:12 ----
[www.mustangv8.com] <DIR> 2013/09/26 15:33 ----
[www.my-safe-haven.com] <DIR> 2013/09/26 15:34 ----
[www.myu2sig.com] <DIR> 2013/09/26 15:36 ----
[www.nettouring.com] <DIR> 2013/09/26 15:41 ----
[www.paypal.com] <DIR> 2013/09/26 14:12 ----
[www.pohrani.com] <DIR> 2013/09/26 15:35 ----
[www.smfarabic.com] <DIR> 2013/09/26 15:40 ----
[www.smileyvault.com] <DIR> 2013/09/26 15:42 ----
[www.symbianize.com] <DIR> 2013/09/26 15:46 ----
[www.thechromesource.com] <DIR> 2013/09/26 15:41 ----
[www.thekrashsite.com] <DIR> 2013/09/26 15:48 ----
[www.xerom-zone.com] <DIR> 2013/09/26 15:39 ----
[zaslike.com] <DIR> 2013/09/26 15:39 ----
[www.smfportal.de] <DIR> 2013/09/26 15:40 ----
[www.mpsoftware.dk] <DIR> 2013/09/26 14:12 ----
[www.aeromodelling.gr] <DIR> 2013/09/26 15:42 ----
[aplicimagens.info] <DIR> 2013/09/26 15:33 ----
[acans.me] <DIR> 2013/09/26 14:08 ----
[i.qkme.me] <DIR> 2013/09/26 15:42 ----
[djomni.julosoft.net] <DIR> 2013/09/26 15:45 ----
[dream-portal.net] <DIR> 2013/09/26 15:44 ----
[mp3quran.net] <DIR> 2013/09/26 15:40 ----
[r21.imgfast.net] <DIR> 2013/09/26 15:44 ----
[sangham.net] <DIR> 2013/09/26 14:14 ----
[us.media1.battle.net] <DIR> 2013/09/26 14:12 ----
[www.atheistthinktank.net] <DIR> 2013/09/26 15:37 ----
[www.glitchpc.net] <DIR> 2013/09/26 15:49 ----
[www.smfpersonal.net] <DIR> 2013/09/26 15:33 ----
[yunitedbalkan.net] <DIR> 2013/09/26 15:39 ----
[zelda-temple.net] <DIR> 2013/09/26 15:45 ----
[www.bkm.cc.nf] <DIR> 2013/09/26 15:37 ----
[dj.omnimaga.org] <DIR> 2013/09/26 15:45 ----
[omnimaga.org] <DIR> 2013/09/26 15:45 ----
[upload.wikimedia.org] <DIR> 2013/09/26 15:34 ----
[www.forum-software.org] <DIR> 2013/09/26 15:48 ----
[www.simplemachines.org] <DIR> 2013/09/26 15:33 ----
[www.tiny-url.org] <DIR> 2013/09/26 15:43 ----
[www.program-tv.ro] <DIR> 2013/09/26 15:45 ----
[www.dodaj.rs] <DIR> 2013/09/26 15:35 ----
[huw.org.uk] <DIR> 2013/09/26 15:33 ----
[www.greenpeace.org.uk] <DIR> 2013/09/26 14:12 ----
[imageshack.us] <DIR> 2013/09/26 14:23 ----
[img135.imageshack.us] <DIR> 2013/09/26 14:14 ----
[img171.imageshack.us] <DIR> 2013/09/26 15:41 ----
[img203.imageshack.us] <DIR> 2013/09/26 15:34 ----
[img253.imageshack.us] <DIR> 2013/09/26 15:34 ----
[img361.imageshack.us] <DIR> 2013/09/26 15:33 ----
[img41.imageshack.us] <DIR> 2013/09/26 15:43 ----
[img503.imageshack.us] <DIR> 2013/09/26 15:36 ----
[img546.imageshack.us] <DIR> 2013/09/26 15:40 ----
[img585.imageshack.us] <DIR> 2013/09/26 15:46 ----
[img833.imageshack.us] <DIR> 2013/09/26 15:34 ----
[img836.imageshack.us] <DIR> 2013/09/26 15:34 ----
[www.oldiesmann.us] <DIR> 2013/09/26 14:12 ----


Sub-folders of the forum alone. Community I will only give the total of files (so far):
[ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+]
index.html 7,278 2013/09/26 14:07 -a--
7 k in 1 files
[about]
_s.html 166 2013/09/26 14:07 -a--
bylaws.html 71,108 2013/09/26 14:07 -a--
contact.html 23,104 2013/09/26 14:07 -a--
features.html 608 2013/09/26 14:02 -a--
index.html 17,181 2013/09/26 14:02 -a--
index-2.html 17,208 2013/09/26 14:02 -a--
license.html 604 2013/09/26 15:33 -a--
members.html 25,984 2013/09/26 14:07 -a--
opensource.html 17,257 2013/09/26 14:03 -a--
privacy.html 29,004 2013/09/26 14:07 -a--
security.html 608 2013/09/26 15:32 -a--
sitemap.html 24,802 2013/09/26 14:07 -a--
trademarks.html 25,542 2013/09/26 14:07 -a--
values.html 19,539 2013/09/26 14:07 -a--
whyfree.html 19,086 2013/09/26 14:07 -a--
284 k in 15 files
[about\smf]
betaagreement.html 17,315 2013/09/26 14:07 -a--
copyright.html 20,620 2013/09/26 14:07 -a--
features.html 35,046 2013/09/26 14:07 -a--
index.html 17,875 2013/09/26 14:07 -a--
license.html 21,502 2013/09/26 14:07 -a--
security.html 28,489 2013/09/26 14:07 -a--
stats.html 18,983 2013/09/26 14:07 -a--
team.html 49,587 2013/09/26 14:02 -a--
teamagreement.html 20,228 2013/09/26 14:07 -a--
translateagreement.html 16,708 2013/09/26 14:07 -a--
240 k in 10 files
[charter]
charter_signup.html 15,649 2013/09/26 15:32 -a--
index.html 18,378 2013/09/26 14:07 -a--
33 k in 2 files
[community]
568,951 k in 9,144 files


Mod edit: Please put big pieces of text in [code][/code] tags.

continue...

Arantor

Um, no, there aren't any such folders there.

What you've done is thrown something like wget at the site and it's downloaded everything, including things on other sites that have nothing to do with this one.

I'm sorry but what you're trying to argue has nothing to do with how SMF works, and I do have a little understanding of how it works.

Moerpot

I can see I speak to the wrong person here :(

NanoSector

Merged your posts. Please don't split a single list in multiple posts, and keep it into code tags.
My Mods / Mod Builder - A tool to easily create mods / Blog
"I've heard from a reliable source that the Answer is 42. But, still no word on what the question is."

Arantor

I've been using SMF for 7 years, I've written tens of thousands of lines of modifications to it. Believe me when I say I know how it works (I wouldn't have a *Developer* badge if I didn't)

The point is you're not using SMF the way any normal user uses it. You're using a tool that mirrors a site which ends up doing all kinds of weird things - not how the software actually works!

What software are you using to generate the lists you are? Because I guarantee you those things do not exist on the server in almost every single case. There are no .html files on this site, it's all PHP.

Moerpot

A person can only paste a maximum characters per post. The forum did not want to post it in full...

Your rules, not mine :(

Burke ♞ Knight

No.

You are not speaking to the wrong person.

You need to read what he says, and stop trying to say you know more of the workings of SMF than someone who not only has worked with it for years, been team member more times than not, and has degrees even I could only dream of.

Moerpot

OK.
Maybe you're the right person. You want to brag? I can too:

I am programmer since 1988. But that is not the point. The point here is: Well, it's all in the 1st post...

Arantor

So you're using a tool to mirror a site and the tool doesn't work the way you expect... sounds to me like you need a different tool if you're going to do that.

Kind of like having a hammer to bang in some nails and complaining about the wall being too tough - means you're using the wrong hammer, not that the wall's the problem. If you want to mirror sites, fine. If you want to mirror sites and have x files per folder, talk to the people who made the tool, not the people whose sites you're stealing the content of.

Moerpot

I ask with a sincere heart. Nothing sarcastic. I want you to ask around and do some homework (even if it's only for me). If what I asked is possible, even if you do not understand what I'm asking.

Please?

If I want to install the forum on my domain, I will have to communicate with you guys. There are only one other alternative, one that I do not want to take at this point.
So please guys and girls.

NanoSector

Lets get this straight.

SMF does NOT save .html files on a server, it gathers all data from the database. No saving to HTML files.
My Mods / Mod Builder - A tool to easily create mods / Blog
"I've heard from a reliable source that the Answer is 42. But, still no word on what the question is."

Moerpot

Quote from: Arantor on September 26, 2013, 11:07:16 AM
So you're using a tool to mirror a site and the tool doesn't work the way you expect... sounds to me like you need a different tool if you're going to do that.

Kind of like having a hammer to bang in some nails and complaining about the wall being too tough - means you're using the wrong hammer, not that the wall's the problem. If you want to mirror sites, fine. If you want to mirror sites and have x files per folder, talk to the people who made the tool, not the people whose sites you're stealing the content of.

I mirror a site to see where there are bottle-nekcks and maybe error that can/will surface in the future.

I just try to do my homework to the fullest.

NanoSector

And why do you need to mirror a site for that?
My Mods / Mod Builder - A tool to easily create mods / Blog
"I've heard from a reliable source that the Answer is 42. But, still no word on what the question is."

Moerpot

...and that is where my 1st question came from.... too many files in one folder

LiroyvH

Quote from: Moerpot on September 26, 2013, 11:13:06 AM
...and that is where my 1st question came from.... too many files in one folder

That is because of the httrack tool that you're using that follows all links and stores them as files... It has nothing to do with SMF at all.
If you want to learn how to limit files in your directories, see the relevant documentation of your operating system.
((U + C + I)x(10 − S)) / 20xAx1 / (1 − sin(F / 10))
President/CEO of Simple Machines - Server Manager
Please do not PM for support - anything else is usually OK.

Arantor

QuoteI want you to ask around and do some homework (even if it's only for me). If what I asked is possible, even if you do not understand what I'm asking.

What you asked for is probably possible. It's also a complete waste of time when dealing with database content because you're not dealing with files.

There are hundreds of thousands of topics on this site. There are not hundreds of thousands of files. The forum data is stored in many fewer files, all grouped together. The bottlenecks will be common to all pages, all the bottlenecks are in index.php and the code called from it.

Mirroring a dynamic site will likely not tell you any of the problems that can occur (and you might even be *creating* problems out of it)

Lastly, but by no means least, you're using a tool that is not produced by Simple Machines and you want to change what that tool does - but it's nothing to do with us.


The bottom line is what you're asking for is not going to happen.

Moerpot

A few years ago I confirmed with vbBullitin. They have the same answer. At that time our server had in the main index folder almost 27 million files. They still have no answer to that problem.

So before it became a problem with my own forum, I want to know how to access it.

Arantor

Seriously... you cannot get to the forum content with a file scraping tool. You need database tools to do it, just like you can't use a hammer to deal with screws, you need a screwdriver.

Moerpot

Well, then I'm just mis-informed.

Sorry for all the troubles.


Kindred

Quote from: Moerpot on September 26, 2013, 11:17:03 AM
Well, then I'm just mis-informed.

You are quite misinformed.

One would assume that, after you got the same befuddled look after asking on both VB and SMF - and then being told by both groups that "Forums do not work that way" that you would try to understand.

Let me repeat, one more time, what I originally said and what Aranotr has been trying to tell you.

SMF (and VB and myBB and phpBB, etc) uses a DATABASE to store **ALL** of the forum content such as users and posts.
This information exists ONLY in the database but is displayed by script when a user requests to view a post or a profile.
It is pulled out of the database.
there is no html file. period.

Trying to display a forum as flat html files is silly, and, as I said in my first repsonse, stupid.
There is a forum using our software that just hit 50 MILLION posts... I can't even imagine trying to figure out how to do that as flat files.

Additionally, your "demonstration" list had files and directories built on content that is not related to Simplemachines.org in any way.
charelotzweb is a host who posts here... but you seem to have pulled their main site and think its part of simplemachines.org because there are links to it in messages.
missallsunday is one of our developers but you seem to have pulled her personal site content into one of your scrapes... see above.
basically, your scraper has taken every single link in any manner displayed (signatures, adverts, etc) and assumed that it was a subdirectory/subsite of simplemachines which is WRONG.
Слaва
Украинi

Please do not PM, IM or Email me with support questions.  You will get better and faster responses in the support boards.  Thank you.

"Loki is not evil, although he is certainly not a force for good. Loki is... complicated."

Advertisement: