Topic: Image Processing Memory Limit

[nend]

Link to Mod

This modification adds a extra option to the modification settings that allows forum administrators to tweak the server memory limit during image manipulation in the SMF thumbnail system.

It also checks to see if the memory requested is given to the image process, if not then the script just returns. This should prevent issues with 500 internal server errors due to memory exhaustion.

Please be aware that file size has nothing to do with the amount of memory you need to create a thumbnail. A image has to be decompressed into memory and each pixel sampled. To figure out memory requirements in MB's, for a particular image please use the formula below.

Code: [Select]

Height * Width * 4 / 1000000
Requirements
PHP 5.2.0 or lower must be compiled with --enable-memory-limit.

License

Code: [Select]

Copyright (c) 2016, Russell Najar
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
  list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice,
  this list of conditions and the following disclaimer in the documentation
  and/or other materials provided with the distribution.

* Neither the name of SMF-Smart-Cache nor the names of its
  contributors may be used to endorse or promote products derived from
  this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

[Rock Lee]

his undoubtedly is very useful in certain shared servers or giving me problems when trying to process many images ... I leave the translation to the Latin Spanish

Latin Spanish utf-8

Code: [Select]

$txt['ipmemlimit'] = 'Límite de memoria de procesamiento de imagen(es)';
And Latin Spanish

Code: [Select]

$txt['ipmemlimit'] = 'LÍmite de memoria de procesamiento de imagen(es)';

Regards!

[PPZ-01]

   I installed the mod, but I have to clarify settings.

   If I adjusted photo size of 1024x768px, then it would we 1024x768x4/1000000=3.14?  Right?  Then, in the field I enter say, number 3 or number 4?

   Default was 128 after installing of the mod.

[nend]

Go ahead and round up, but 128 should be a safe setting.

This modification is mainly for forum owners that want to up the memory limit for large image thumbnail generation. I wouldn't recommend using it to impose a lower limit as there are better ways to do that. I recommend if that is what you're looking into achieving, then use the Attachment Settings in the admin panel instead.

[nend]

New version uploaded with some changes under the hood.
+ Now tries to detect bit depth of image instead of assuming.
+ Considers current memory usage.

Also those wondering what settings they should use you can use this little script to give you an idea.

Code: [Select]

<?php
// Replace with url of image or path
$file = "http://your.image.here/path/to.it";
$sizes = getimagesize($file);

// Do we know the bit depth?
if (!empty($sizes['bits'])) {
$bitDepth = $sizes['bits'] * $sizes['channels'];
// How much do we need and how much have we used.
$need = round(((($sizes[0] * $sizes[1] * $bitDepth / 8) + 65536 + memory_get_usage()) * 1.8) / 1048576);
echo '<br />The Color Depth of this image is '.$bitDepth.'-bit.<br />You need '.$need.' MB to process this image.<br />
<br /><b>Test Results</b>';

$src = imagecreatefromjpeg($file);
$dst = imagecreatetruecolor(100,100);
imagecopyresampled($dst, $src, 0, 0, 0, 0,100,100,$sizes[0],$sizes[1]);
echo '<br />Test inicates it took '.round(memory_get_peak_usage() / 1048576).' MB to process this image.';
}else
echo 'Couldn\'t get Bit Depth of image.';

?>

[nend]

Code: [Select]

0.1.1 Released
* Bugfix, forgot to convert bits to bytes.
+ Aded additional logic to get how much memory needed.
Anyone running 0.1.0 is urged to upgrade to 0.1.1 as there is quite a big difference in bits and bytes and the modification will request allot more than what is needed.

[Romanj]

nend, thanks for the mod!

My server's defaults:
   memory_limit   256M
   upload_max_filesize 64M
   post_max_size 64M

Without the mod SMF 2.0.13 was able to process (resize) images up to ~17Mpix.
With the mod set to 256, it processes images up to ~26Mpix (6000x4300).

I don't know why with the same memory limit there are different results with images.
Possibly because of string @ini_set('memory_limit', '90M'); in Subs-Graphics.php?

After 26Mpix it goes to HTTP ERROR 500, though I've tried to set limit to 512 (its dedicated server, that amount should be available).
So where is the next bottleneck could be? Is it possible to go beyond 26Mpix?

On the local machine SMF 2.0.13 easily handles 48Mpix (8000x6000) with this mod. (May be more than 48, I did not try that)
So limit 26Mpix should be server-related.

[nend]

*edit
I updated the modification on the customization site. Please try that one and see if it works for you. 
*end of edit

It may be that the images that are returning 500 errors are due to the script not being able to get the actual bit depth and defaults to 16bbp. This isn't the worst case scenario as some high quality images are 32bbp.

This line in the code.

Code: [Select]

$bitDepth = 48; // Probally a bad idea assuming, but 48 bits may be the magic number.
See what happens if you change it from 48 to 96 for 32bbp. If the images aren't taken with a camera then there is a chance they may also have a alpha channel which would bring it up from 96 to 128.

If that doesn't work try changing this line.

Code: [Select]

$need = round(((($sizes[0] * $sizes[1] * $bitDepth / 8) + 65536 + memory_get_usage()) * 1.8) / 1048576);
Where it says * 1.8, change the 1.8 up a digit or so. This is sensitive and is supposed to account for some overhead during processing.

Worst case scenario with that resolution is that you need at least 400MBs to process.

I don't know why with the same memory limit there are different results with images.
Possibly because of string @ini_set('memory_limit', '90M'); in Subs-Graphics.php?

That string will be void once we set the memory limit.

I am thinking though, it may be better though just to set the memory limit if all test pass and only return if it could be validated that there will be a 500 if the script continued.

[Romanj]

nend, thanks for your help!

I've tried updated modification, then I've tried to replace 96 with 128, then I've tried to change 1.8 with 2.8.
Unfortunately, I've got the same results - 500 errors every time when picture is bigger than ~26Mpix.
All of this with 512 setting for the mod. Also I set php_value memory_limit 512M in .htaccess (because I don't have access to php.ini).

Image is basically the same in all my tests. I just made different resoulutions of it.
With the 6000x4400 (26.4Mpix) sample I'm starting to get 500 error on my working server. Though, it works fine on my test sever on the local machine, up to 48Mpix (and may be more).

I suppose it could be some tricky limitations on a working server. May be because of different versions of PHP or GD or something like that.

Anyway, I'm glad that your mod helped to raise limit of 500 error from 17Mpix to 26Mpix.
26Mpix is almost enough. Not so many cameras are shoting images bigger, not so many people trying to shove them on a forum)
Just interested to find where that bottleneck is.

[nend]

Can you send me a PM so I can troubleshoot this further.

If possible, include a link to a image that is causing a 500 error. Something isn't right as the modification is supposed to detect when a image is too big and bail out before a 500 error occurs.

[Romanj]

I found the problem. It's not because of your mod, it works beautifully.
Though error 500 still appeared when image was too big. I'll send deatils to PM.

The limitation of 26Mpix caused by this custom code in Post.php:

Code: [Select]

// Rotate image according to Exif orientation via simplemachines.org/community/index.php?topic=521231.20
                        $imageFile = $dummy;
                        $imageSize = getimagesize($imageFile);

                        if ($imageSize['mime'] == 'image/jpeg')
                        {
                                $imageExif = exif_read_data($imageFile);

                                if (array_key_exists('Orientation', $imageExif))
                                {
                                        $imageOrientation = $imageExif['Orientation'];
                                        $image = imagecreatefromjpeg($imageFile);
                                        $imageRotated = false;

                                        switch($imageOrientation)
                                        {
                                                case 3:
                                                        $image = imagerotate($image, 180, 0);
                                                        $imageRotated = true;
                                                        break;
                                                case 6:
                                                        $image = imagerotate($image, -90, 0);
                                                        $imageRotated = true;
                                                        break;
                                                case 8:
                                                        $image = imagerotate($image, 90, 0);
                                                        $imageRotated = true;
                                                        break;
                                        }

                                        if ($imageRotated)
                                                imagejpeg($image, $imageFile);

                                        imagedestroy($image);
                                }
                        }
It's from this post: http://www.simplemachines.org/community/index.php?topic=521231.msg3865613#msg3865613
It suppose to rotate wrong orientated images. But fails when image is bigger than 26Mpix on my server.

When I deleted it, I was able to upload 48Mpix (8000x6000) photo to my working server with your mod setting to 512M.

[nend]

What is the server setting for max_execution_time? I am thinking this is a timeout issue.

[Romanj]

max_execution_time   10

Though it feels that processing of 48Mpix image takes more time than cases where I got 500 error. Error 500 comes right after uploading of a file is finished (percents of upload in bottom left corner of Google Chrome). Processing of 48Mpix takes few seconds more after uploading is finished.

Anyway later I can test that more thoroughly if it could be the case.

[Romanj]

Small update.
It is not directly related to this great mod, but in case someone interested, it is possible to keep image autorotation by exif-orientation without meeting with server limitations of this action (in my case 26Mpix).
It is easy to do by checking filesize before autorotation. If file is too big then rotate it after resizing. Works fine.

[The QE2 Story Forum]

Wow, this mod appears to have instantly fixed an issue I was having for pretty much any image from a modern camera.

Thanks!

[EthanR]

Fixed the issue of large image uploads timing out/white screen. (especially from images from phones as mentioned above) 

[Goten22]

It works but big images don't have miniatures generated. Any ideas how to fix it?

[Kindred]

get a different server....

[Goten22]

Solved - I increased "Image Processing Memory limit" to 512.