How to cancel a profile export?

[MobileCS]

I'm getting an email from crontab every 6 minutes with this error message :

Code Select Expand

PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 45200197 bytes) in /www/example.com/httpdocs/forum/Sources/Profile-Export.php on line 1777
I do not want to raise my PHP memory limit to 512MB, so how can cancel this profile export?

[MobileCS]

I found it in the background_tasks section in the database.

My next question is why does this feature require so much ram?

The person who used this feature is a new user, with no posts or messages.

Code Select Expand

{"profile":0,"posts":0,"personal_messages":0},"latest":{"profile":1,"posts":null,"personal_messages":null}

[Sesquipedalian]

It certainly shouldn't on a normal SMF installation. Something odd is happening. 

Please post a list of the mods you have installed, and attach a copy of both your ./Sources/Profile-Export.php file and your ./Sources/Tasks/ExportProfileData.php file.

[MobileCS]

Mods installed:

Remove "Last Edit"
Users Online Today
Custom Board Icons
Simple Audio Video Embedder
Stop Forum Spam
Sphinx for SMF

[Sesquipedalian]

Code Select Expand

{"profile":0,"posts":0,"personal_messages":0},"latest":{"profile":1,"posts":null,"personal_messages":null}

Where exactly did you find this string?

[MobileCS]

smf_background_tasks table - in the task_data column

[Sesquipedalian]

Was that the entire content of that column for that task?

[MobileCS]

Code Select Expand

{"format":"XML_XSLT","uid":65221,"lang":"","included":{"profile":{"func":"getXmlProfile","langfile":"Profile"},"posts":{"func":"getXmlPosts","langfile":"Post"},"personal_messages":{"func":"getXmlPMs","langfile":"PersonalMessage"}},"start":{"profile":0,"posts":0,"personal_messages":0},"latest":{"profile":1,"posts":null,"personal_messages":null},"datatype":"profile","format_settings":{"extension":"styled.xml","mime":"text\/xml","description":"Styled XML (human and machine friendly)","per_page":500},"last_page":0,"dlfilename":"Example-Forum_ExampleMember_Profile_Posts_Personal-Messages"}

[Sesquipedalian]

Thanks, but I didn't ask for a different sample of the string, and without context this new sample doesn't help me much.

• Is this latest sample from in the instance of the background task that was producing errors, or does it come from a different instance?
• If this latest sample was not from the instance that was producing errors, were errors also generated by the instance that this sample comes from, or not?

[MobileCS]

I had already deleted the original that was giving me errors (to stop the crontab from sending me emails every 6 minutes), so I started a new one using the same registered username (profile / actions / download profile data).

The one I gave you is the full string, which also gives me the same issues. The only thing I changed was my forum name and the member name.

[Sesquipedalian]

Thank you, that is helpful information. I will need to try to replicate the problem on my test servers in order to figure out what might be going on.

A few more questions as I try to diagnose the issue:

1. Are you using a real cron job to run SMF's cron.php, or are you relying on SMF's default, JavaScript based method?
2. If you are using a real cron job, how often do you have it scheduled to run?
3. What value do you have set in the "Rate at which to process posts & personal messages for export" setting? Although the member in question has no posts or personal messages, this might still be relevant.
4. Do exports of other members' data also produce these errors, or is the problem unique to this member?

[MobileCS]

1) I am using a real cron job to run SMF cron.php - javascript method is disabled
2) The cron is set to run every minute
3) Rate at which to process posts & personal messages = 250
4) The error happens for all member accounts

[MobileCS]

Here is some additional information:

When I change the option "File format for exported data" to "XML (machine friendly)", I do not get the out of memory error, but this instead :

Code Select Expand

file_put_contents(/www/example.com/httpdocs/forum/exports/16ab57bcf4e09fc159fb330f5d6d395237d72fed.xml.tmp): Failed to open stream: Permission denied
The issue now is that "exports" folder does have full owner/group permissions (0775).

To test it, I uploaded this test.php file into that folder and ran it in my web browser (https://example.com/forum/exports/test.php) and it did create the testfile.txt file without issues.

Code Select Expand

<?php
$data = "Testing file_put_contents.";
$file = "testfile.txt";

if (file_put_contents($file, $data) === false) {
   echo "Failed to write to file.";
} else {
   echo "Data successfully written to file.";
}


[Sesquipedalian]

That is very helpful information, thank you. This is definitely helping me to narrow the range of possible causes.

What happens if you temporarily disable the real cron job and reenable the JavaScript method? Do the errors still occur?

Also, did the "permission denied" error message say which line it occurred on?

[MobileCS]

Yes, when I tried a couple times it gave me the error on :

/Sources/tasks/ExportProfileData.php (Line 252)
/Sources/tasks/ExportProfileData.php (Line 333)

I still have the out of memory error when I disable my cron, and enabled the javascript method. The only difference is where the error is shown (was only showing in SMF logs, now only showing in PHP error log file).

[Sesquipedalian]

/Sources/tasks/ExportProfileData.php (Line 333)

Line 333 contains a call to fopen() rather than file_put_contents(), which makes perfectly good sense but also means that you've so far only been sharing samples from your error logs that you figure to be representative. While I appreciate that well-meaning attempt to narrow things down to the most relevant parts, it is actually counterproductive. Can you please share the entire set of error messages for a single run of the background task? Feel free to anonymize the data, but otherwise retain all of it, even repeated messages.

[shawnb61]

Two thoughts here...

If one of these is executing, I believe either the admin or requesting user can cancel it on the user profile export screen.

IIRC, doing so even properly removes the request from the background tasks queue.

Also...  There is a config parameter for export rate.  If dealing with resource issues, try a lower rate.

During testing, I simulated an article-based forum, with thousands of messages that were at the post size limit. The exports were HUGE - many GB... To get that to run, IIRC I had to set the rate all the way down to something like 5.   (Default is 250...)

[MobileCS]

I never realized there was an fopen error in my list, all I saw was (10) file_put_contents errors. I've since cleared those errors, and unable to get fopen to show up again.

Code Select Expand

CLI cron.php
/forum/Sources/tasks/ExportProfileData.php (Line 446)
file_put_contents(/forum/exports/6b3bf37b735cd00d522c69309b772230190c00ee.xml.progress.json): Failed to open stream: Permission denied

CLI cron.php
/forum/Sources/tasks/ExportProfileData.php (Line 252)
file_put_contents(/forum/exports/6b3bf37b735cd00d522c69309b772230190c00ee.xml.tmp): Failed to open stream: Permission denied
Switching "Rate at which to process posts & personal messages for export" from 250 down to 5 changes nothing.

[Sesquipedalian]

Thank you, @MobileCS. That is helpful.

At this point, it seems clear from the error messages that the fundamental cause of the problem is a file permissions error. 

My initial suspicion was that your system's cron process was being run under a different user than your web server, and thus that the *.xml.progress.json and *.xml.tmp files that are initially created in ./Sources/Profile-Export.php (and thus by the web server user) could not be written to during the execution of ./Sources/Tasks/ExportProfileData.php (which is run by the cron process's user). However, since you saw the same errors when running the background task via SMF's JavaScript based method (and thus by the web server user), that means the file permissions problem is something less obvious. I'm honestly stumped about what could be causing this file permission problem. You may need to do some deeper investigation in order to figure out why the files are becoming unwritable after their initial creation.

Meanwhile, the memory exhaustion problem when using the "Styled XML" output format is a separate, secondary problem that I should be able to diagnose and solve once I can reproduce the initial triggering condition. I will try forcibly making the the *.xml.progress.json and *.xml.tmp files unwritable after their initial creation on my test machine. That may be sufficient to trigger the secondary problem, but it also may not be. If you can figure out exactly what is causing the file permissions problem on your server, that will help me to better reproduce the conditions on my test machine and thereby help me to have a better chance of success at diagnosing and solving the problem.

[MobileCS]

Ok, setting cron to run as "www" 50% solved the solved the issue.

When I set "File format for exported data" to "XML (machine friendly)" it works just fine now.

However, I still get the out of memory error when setting it to "Styled XML".