Simple Machines Community Forum

Customizing SMF => Modifications and Packages => Topic started by: nend on May 23, 2016, 06:38:02 PM

Title: Image Processing Memory Limit
Post by: nend on May 23, 2016, 06:38:02 PM
Link to Mod (http://custom.simplemachines.org/mods/index.php?mod=4111)

This modification adds a extra option to the modification settings that allows forum administrators to tweak the server memory limit during image manipulation in the SMF thumbnail system.

It also checks to see if the memory requested is given to the image process, if not then the script just returns. This should prevent issues with 500 internal server errors due to memory exhaustion.

Please be aware that file size has nothing to do with the amount of memory you need to create a thumbnail. A image has to be decompressed into memory and each pixel sampled. To figure out memory requirements in MB's, for a particular image please use the formula below.
Height * Width * 4 / 1000000

Requirements
PHP 5.2.0 or lower must be compiled with --enable-memory-limit.

License
Copyright (c) 2016, Russell Najar
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
  list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice,
  this list of conditions and the following disclaimer in the documentation
  and/or other materials provided with the distribution.

* Neither the name of SMF-Smart-Cache nor the names of its
  contributors may be used to endorse or promote products derived from
  this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Title: Re: Image Processing Memory Limit
Post by: -Rock Lee- on May 23, 2016, 08:30:27 PM
his undoubtedly is very useful in certain shared servers or giving me problems when trying to process many images ... I leave the translation to the Latin Spanish 8)

Latin Spanish utf-8
$txt['ipmemlimit'] = 'LĂ­mite de memoria de procesamiento de imagen(es)';

And Latin Spanish
$txt['ipmemlimit'] = 'LÍmite de memoria de procesamiento de imagen(es)';


Regards!
Title: Re: Image Processing Memory Limit
Post by: PPZ-01 on September 08, 2016, 09:50:41 AM
   I installed the mod, but I have to clarify settings.

   If I adjusted photo size of 1024x768px, then it would we 1024x768x4/1000000=3.14?  Right?  Then, in the field I enter say, number 3 or number 4?

   Default was 128 after installing of the mod.
Title: Re: Image Processing Memory Limit
Post by: nend on September 08, 2016, 10:03:02 AM
Go ahead and round up, but 128 should be a safe setting.

This modification is mainly for forum owners that want to up the memory limit for large image thumbnail generation. I wouldn't recommend using it to impose a lower limit as there are better ways to do that. I recommend if that is what you're looking into achieving, then use the Attachment Settings in the admin panel instead.
Title: Re: Image Processing Memory Limit
Post by: nend on November 14, 2016, 10:19:55 PM
New version uploaded with some changes under the hood.
+ Now tries to detect bit depth of image instead of assuming.
+ Considers current memory usage.

Also those wondering what settings they should use you can use this little script to give you an idea.
<?php
// Replace with url of image or path
$file "http://your.image.here/path/to.it";
$sizes getimagesize($file);

// Do we know the bit depth?
if (!empty($sizes['bits'])) {
$bitDepth $sizes['bits'] * $sizes['channels'];
// How much do we need and how much have we used.
$need round(((($sizes[0] * $sizes[1] * $bitDepth 8) + 65536 memory_get_usage()) * 1.8) / 1048576);
echo '<br />The Color Depth of this image is '.$bitDepth.'-bit.<br />You need '.$need.' MB to process this image.<br />
<br /><b>Test Results</b>'
;

$src imagecreatefromjpeg($file);
$dst imagecreatetruecolor(100,100);
imagecopyresampled($dst$src0000,100,100,$sizes[0],$sizes[1]);
echo '<br />Test inicates it took '.round(memory_get_peak_usage() / 1048576).' MB to process this image.';
}else
echo 'Couldn\'t get Bit Depth of image.';

?>
Title: Re: Image Processing Memory Limit
Post by: nend on November 15, 2016, 06:10:42 PM
0.1.1 Released
* Bugfix, forgot to convert bits to bytes.
+ Aded additional logic to get how much memory needed.


Anyone running 0.1.0 is urged to upgrade to 0.1.1 as there is quite a big difference in bits and bytes and the modification will request allot more than what is needed.
Title: Re: Image Processing Memory Limit
Post by: Romanj on January 23, 2017, 11:27:26 AM
nend, thanks for the mod!

My server's defaults:
   memory_limit   256M
   upload_max_filesize 64M
   post_max_size 64M

Without the mod SMF 2.0.13 was able to process (resize) images up to ~17Mpix.
With the mod set to 256, it processes images up to ~26Mpix (6000x4300).

I don't know why with the same memory limit there are different results with images.
Possibly because of string @ini_set('memory_limit', '90M'); in Subs-Graphics.php?

After 26Mpix it goes to HTTP ERROR 500, though I've tried to set limit to 512 (its dedicated server, that amount should be available).
So where is the next bottleneck could be? Is it possible to go beyond 26Mpix?

On the local machine SMF 2.0.13 easily handles 48Mpix (8000x6000) with this mod. (May be more than 48, I did not try that)
So limit 26Mpix should be server-related.
Title: Re: Image Processing Memory Limit
Post by: nend on January 23, 2017, 12:09:07 PM
*edit
I updated the modification on the customization site. Please try that one and see if it works for you.  ;)
*end of edit

It may be that the images that are returning 500 errors are due to the script not being able to get the actual bit depth and defaults to 16bbp. This isn't the worst case scenario as some high quality images are 32bbp.

This line in the code.
$bitDepth = 48; // Probally a bad idea assuming, but 48 bits may be the magic number.

See what happens if you change it from 48 to 96 for 32bbp. If the images aren't taken with a camera then there is a chance they may also have a alpha channel which would bring it up from 96 to 128.

If that doesn't work try changing this line.
$need = round(((($sizes[0] * $sizes[1] * $bitDepth / 8) + 65536 + memory_get_usage()) * 1.8) / 1048576);

Where it says * 1.8, change the 1.8 up a digit or so. This is sensitive and is supposed to account for some overhead during processing.

Worst case scenario with that resolution is that you need at least 400MBs to process.
Quote from: Romanj on January 23, 2017, 11:27:26 AM
I don't know why with the same memory limit there are different results with images.
Possibly because of string @ini_set('memory_limit', '90M'); in Subs-Graphics.php?
That string will be void once we set the memory limit.

I am thinking though, it may be better though just to set the memory limit if all test pass and only return if it could be validated that there will be a 500 if the script continued.
Title: Re: Image Processing Memory Limit
Post by: Romanj on January 23, 2017, 03:20:27 PM
nend, thanks for your help!

I've tried updated modification, then I've tried to replace 96 with 128, then I've tried to change 1.8 with 2.8.
Unfortunately, I've got the same results - 500 errors every time when picture is bigger than ~26Mpix.
All of this with 512 setting for the mod. Also I set php_value memory_limit 512M in .htaccess (because I don't have access to php.ini).

Image is basically the same in all my tests. I just made different resoulutions of it.
With the 6000x4400 (26.4Mpix) sample I'm starting to get 500 error on my working server. Though, it works fine on my test sever on the local machine, up to 48Mpix (and may be more).

I suppose it could be some tricky limitations on a working server. May be because of different versions of PHP or GD or something like that.

Anyway, I'm glad that your mod helped to raise limit of 500 error from 17Mpix to 26Mpix.
26Mpix is almost enough. Not so many cameras are shoting images bigger, not so many people trying to shove them on a forum)
Just interested to find where that bottleneck is.
Title: Re: Image Processing Memory Limit
Post by: nend on January 23, 2017, 06:13:40 PM
Can you send me a PM so I can troubleshoot this further.

If possible, include a link to a image that is causing a 500 error. Something isn't right as the modification is supposed to detect when a image is too big and bail out before a 500 error occurs.
Title: Re: Image Processing Memory Limit
Post by: Romanj on January 24, 2017, 03:46:53 AM
I found the problem. It's not because of your mod, it works beautifully.
Though error 500 still appeared when image was too big. I'll send deatils to PM.

The limitation of 26Mpix caused by this custom code in Post.php:
// Rotate image according to Exif orientation via simplemachines.org/community/index.php?topic=521231.20
                        $imageFile = $dummy;
                        $imageSize = getimagesize($imageFile);

                        if ($imageSize['mime'] == 'image/jpeg')
                        {
                                $imageExif = exif_read_data($imageFile);

                                if (array_key_exists('Orientation', $imageExif))
                                {
                                        $imageOrientation = $imageExif['Orientation'];
                                        $image = imagecreatefromjpeg($imageFile);
                                        $imageRotated = false;

                                        switch($imageOrientation)
                                        {
                                                case 3:
                                                        $image = imagerotate($image, 180, 0);
                                                        $imageRotated = true;
                                                        break;
                                                case 6:
                                                        $image = imagerotate($image, -90, 0);
                                                        $imageRotated = true;
                                                        break;
                                                case 8:
                                                        $image = imagerotate($image, 90, 0);
                                                        $imageRotated = true;
                                                        break;
                                        }

                                        if ($imageRotated)
                                                imagejpeg($image, $imageFile);

                                        imagedestroy($image);
                                }
                        }


It's from this post: http://www.simplemachines.org/community/index.php?topic=521231.msg3865613#msg3865613
It suppose to rotate wrong orientated images. But fails when image is bigger than 26Mpix on my server.

When I deleted it, I was able to upload 48Mpix (8000x6000) photo to my working server with your mod setting to 512M.
Title: Re: Image Processing Memory Limit
Post by: nend on January 24, 2017, 08:27:15 AM
What is the server setting for max_execution_time? I am thinking this is a timeout issue.
Title: Re: Image Processing Memory Limit
Post by: Romanj on January 25, 2017, 03:18:36 AM
max_execution_time   10

Though it feels that processing of 48Mpix image takes more time than cases where I got 500 error. Error 500 comes right after uploading of a file is finished (percents of upload in bottom left corner of Google Chrome). Processing of 48Mpix takes few seconds more after uploading is finished.

Anyway later I can test that more thoroughly if it could be the case.
Title: Re: Image Processing Memory Limit
Post by: Romanj on February 10, 2017, 09:36:36 AM
Small update.
It is not directly related to this great mod, but in case someone interested, it is possible to keep image autorotation by exif-orientation without meeting with server limitations of this action (in my case 26Mpix).
It is easy to do by checking filesize before autorotation. If file is too big then rotate it after resizing. Works fine.
Title: Re: Image Processing Memory Limit
Post by: Rob Lightbody on April 25, 2018, 03:18:59 PM
Wow, this mod appears to have instantly fixed an issue I was having for pretty much any image from a modern camera.

Thanks!
Title: Re: Image Processing Memory Limit
Post by: EthanR on September 04, 2018, 09:53:20 PM
Fixed the issue of large image uploads timing out/white screen. (especially from images from phones as mentioned above)  :)
Title: Re: Image Processing Memory Limit
Post by: Goten22 on September 10, 2018, 11:17:49 AM
It works but big images don't have miniatures generated. Any ideas how to fix it?
Title: Re: Image Processing Memory Limit
Post by: Kindred on September 10, 2018, 01:07:53 PM
get a different server....
Title: Re: Image Processing Memory Limit
Post by: Goten22 on September 10, 2018, 03:13:46 PM
Solved - I increased "Image Processing Memory limit" to 512.
Title: Re: Image Processing Memory Limit
Post by: KittyGalore on February 13, 2021, 05:54:45 PM
Just had to install this mod because i had issues with attachments. This mod needs to be updated it compatible for only 2.0.12 and 2.0.11
Title: Re: Image Processing Memory Limit
Post by: Kindred on February 13, 2021, 10:41:40 PM
Or just use emulate to install the mod?
Title: Re: Image Processing Memory Limit
Post by: shawnb61 on February 13, 2021, 11:16:28 PM
I have used this mod across my prod & test environments for some time.  I have never seen a conflict with SMF patches.  It installs on 2.0.18 with no issues.
Title: Re: Image Processing Memory Limit
Post by: landyvlad on September 10, 2021, 07:42:22 AM
So... I have this mod on my forum and I guess its still needed ?
Title: Re: Image Processing Memory Limit
Post by: GL700Wing on September 10, 2021, 07:46:14 AM
Quote from: landyvlad on September 10, 2021, 07:42:22 AMSo... I have this mod on my forum and I guess its still needed ?
Which version of SMF are you using it with?
Title: Re: Image Processing Memory Limit
Post by: landyvlad on September 10, 2021, 08:53:09 AM
2.0.17 and upgrading to 2.0.18


(2.1 in due course but not soon)
Title: Re: Image Processing Memory Limit
Post by: GL700Wing on September 10, 2021, 07:16:35 PM
Quote from: landyvlad on September 10, 2021, 08:53:09 AM2.0.17 and upgrading to 2.0.18


(2.1 in due course but not soon)
If you're using the Automatic Attachment Rotation (and Resize) (https://custom.simplemachines.org/index.php?mod=4087) mod and you have version 5.10 or later installed you no longer need this mod because I added the option of specifying additional memory for image processing to the Automatic Attachment Rotation (and Resize) (https://custom.simplemachines.org/index.php?mod=4087) mod.