Package server chokes for New Feature current amount

Started by Chen Zhen, February 10, 2014, 11:57:33 PM

Previous topic - Next topic

Chen Zhen

Admin -> Package Manager -> Download Packages -> Package Servers -> SMF 3rd Party Mod Site -> Browse -> New Feature

  The way the routine is written in PackageGet.php it eats up a lot of memory for the current list of 274 New Feature mods.
I tested this on several forums I have admin access to to be sure but I knew it would behave the same way because most people on shared hosting are not set up at ie. 512M. The result is php crashing when it exhausts the available memory.

  I attempted to set the limit prior to the $packages loop but this did not solve the issue. I was able to get this to work but I had to set that limit within the loop itself. Imo that routine needs to be handled differently because it should not require that much memory for the task it is performing.

I patched it with the example shown below but as I said I don't like it at all.

file: /Sources/PackageGet.php
find:
            if ($package['count'] > 1)
                $context['list_type'] = 'ol';

add after the above:
            if (round(memory_get_peak_usage()/1048576) > ((int)ini_get('memory_limit') - 16))
                @ini_set('memory_limit', '512M');

... I wrote it like that because I didn't want it to make the setting (repeatedly!) until it almost exhausted the available memory. I suppose one can ask their host to set the limit to 512M but that did not work when I did it prior to the loop.

My SMF Mods & Plug-Ins

WebDev

"Either you repeat the same conventional doctrines everybody is saying, or else you say something true, and it will sound like it's from Neptune." - Noam Chomsky

Arantor

This is already known and things were started to deal with this; you may have noticed that I moved some packages around to try to curb this.

You may also note that the listing of mods shown is out of date. There are far more than 269 "New Features".

Chen Zhen

So I assume it should be showing (currently) 522 mods - whatever is already in the persons Packages folder.  There has got to be a way to do that without using so much memory. What about having some sort of pages system set up? Where the feed only gives it 100 mods at a time per page and this is easily controlled via the url. The user clicks on pages to cycle through the mods.


My SMF Mods & Plug-Ins

WebDev

"Either you repeat the same conventional doctrines everybody is saying, or else you say something true, and it will sound like it's from Neptune." - Noam Chomsky

Arantor

It's supposed to list the 522 mods on the mod site, except it can't because it chokes.

That's why I was pushing for some reorganisation but there's too many mods that don't fit in any actual category in there.

Chen Zhen

  If it only received 100 mods at a time it wouldn't choke. 522 mods is fine but set it up as 5 pages (100 max per page) here on this site. The routine inside that function has the admin cllick on pages to see a max of 100 mods per page.  This way there can be 1500 mods & it won't matter. It's just an idea.

Regards.


My SMF Mods & Plug-Ins

WebDev

"Either you repeat the same conventional doctrines everybody is saying, or else you say something true, and it will sound like it's from Neptune." - Noam Chomsky

Arantor

Doesn't work with the way package servers work, unfortunately.

Chen Zhen


  Hmm.. How about I pitch another 2 ideas out there.

  Can the routine control the order of the list (ie. alphabetical) and only read file names starting with certain letters? This way it can read only files ie. A-F, G-L, M-R, S-X, Y-Z and have links/pages for those? 
   
  Another possible option is to have a script on this site that reads the entire directories and then splits them into pages here. Then the function in the SMF forum package accesses those pages of links.


 

My SMF Mods & Plug-Ins

WebDev

"Either you repeat the same conventional doctrines everybody is saying, or else you say something true, and it will sound like it's from Neptune." - Noam Chomsky

Arantor

Nope. It's simply reading what the server gives it, and the server just gives it the whole file at once. I suppose in theory the server could split it itself, so you'd have to browse to New Features, then New Features A-D, or New Features > New Features E-H or whatever. But that requires a non-trivial change to the architecture here that produces said XML files (since they're statically served, not dynamically generated) in the first place.

It already takes a scary amount of time to prepare in the first place.

And honestly, the team's time would be better spent working on 2.1 and going bug hunting in that (there's enough on Github, still many more on the various places here too)

Chen Zhen


  This seems to be working properly now btw... it is showing 541 mods for New Feature atm using the edit I posted.

My SMF Mods & Plug-Ins

WebDev

"Either you repeat the same conventional doctrines everybody is saying, or else you say something true, and it will sound like it's from Neptune." - Noam Chomsky

Arantor

Which is clever seeing how there aren't 541 mods in that category on the mod site.

Chen Zhen


It's the list of mod types that shows totals that is not correct. When I stated there were 274 mods in it I must have been looking at that list of mod types instead of the actual New Feature list itself. I was really tired last night when I was messing with it & posting here so that doesn't surprise me. When I have some time later this evening I will fix the mod counts in that list. 

My SMF Mods & Plug-Ins

WebDev

"Either you repeat the same conventional doctrines everybody is saying, or else you say something true, and it will sound like it's from Neptune." - Noam Chomsky

Arantor

Or it might be the outdated packages.xml parent file that's served from here that does say 274 because it hasn't been rebuilt in ages. Like I said, I was trying to get them reorganised, which would culminate in a rebuilding of all the files, but there were too many things to be meaningfully categorised.

*shrug* I figure it'll get worked out someday, one way or another.

Chen Zhen


Yes those file counts are being displayed here on this site.
ref. http://custom.simplemachines.org/packages/mods/packages.xml?language=english

  Imo if no one has time to work on any of this prior to a 2.0.8 update, perhaps inserting the edit I provided (or something similar) can be implemented. At least if someone decides to use those links provided on that page, it won't crash their forum due to taxing the memory.

My SMF Mods & Plug-Ins

WebDev

"Either you repeat the same conventional doctrines everybody is saying, or else you say something true, and it will sound like it's from Neptune." - Noam Chomsky

Arantor

I'd argue there are more important things than this for 2.0.8, like fixing 2.0.7, like fixing the board index screwing up on InnoDB or MySQL 5.6/MyISAM tables.

Chen Zhen


Imo might as well tally them up.

re. InnoDB/MyISAM issue:
What are the specifics for this issue you are referring to? Can you provide me a link that has more details about this?

Regards.

My SMF Mods & Plug-Ins

WebDev

"Either you repeat the same conventional doctrines everybody is saying, or else you say something true, and it will sound like it's from Neptune." - Noam Chomsky

Arantor

SMF for years does a slightly odd thing: when you add a board, delete a board or rearrange the boards, it runs an ALTER TABLE to force the boards table to be sorted by the board_order column. The theory being that if you select any subset of the table, or the whole table, you don't have to explicitly order it thereafter. Which from a performance perspective is good because the boards table contains a number of text columns which will force a filesort (== much slower compared to a table scan)

InnoDB ignores that and always orders the table by primary key, to which the suggestions elsewhere were to make board_order the table's physical primary key, and to then make id_board a unique key so everything still works. That's fine, but it needs to be done and taken into account in the installer.

MySQL 5.6/MyISAM apparently disregards the ordering predication, though I've seen suggestions that forcing both categories and boards to be ordered it continues to work (suggesting the problem is related to how 5.6's changes to the query parser are implemented for that query)

Tracked under https://github.com/SimpleMachines/SMF2.1/issues/887 with notes in https://github.com/SimpleMachines/SMF2.1/issues/858 relating to 2.1's changes needing to be split out (since in 2.1, boards can have groups set as moderators, as opposed to just individuals, and this is a performance headache too; but even in 2.0 ideally the moderators query should be split out of the main query and into a separate one - two smaller queries will outperform one vast one)

Chen Zhen

The issue I originally reported here seems to have been mysteriously resolved.
I know I tested this on 3 live and 1 local forums prior to this report and github pr although today this seems to work just fine without any edits.
I can only assume that something in that folder was possibly causing the issue which has since been emended.

I best heed Randall Terry's advice.



My SMF Mods & Plug-Ins

WebDev

"Either you repeat the same conventional doctrines everybody is saying, or else you say something true, and it will sound like it's from Neptune." - Noam Chomsky

Scratchit

Hi

Sorry if I'm posting in the wrong place.

I tried to Browse the Packages, when I click on Themes, BBC or any other category all I get is a blank page. Nothing shows up on any of them.

Any ideas

Cheers

Scratchit

Hi

Being as there is a problem browsing and downloading Mods, if I have the mod number, say 3249. Is there a way to download it using the 'Download as Package by URL' and 'Name of file'
Anybody know what info I would put in the two boxes?

Cheers

emanuele

Quote from: Scratchit on February 23, 2014, 10:27:25 AM
Is there a way to download it using the 'Download as Package by URL' and 'Name of file'
Anybody know what info I would put in the two boxes?
The link you find in the box "Download this mod" and nothing in "name of file".
Otherwise you can just download the package to your desktop (or somewhere else) and then use the "upload" section and upload the file you just downloaded. ;)


Take a peek at what I'm doing! ;D




Hai bisogno di supporto in Italiano?

Aiutateci ad aiutarvi: spiegate bene il vostro problema: no, "non funziona" non è una spiegazione!!
1) Cosa fai,
2) cosa ti aspetti,
3) cosa ottieni.

Advertisement: