Simple Machines Community Forum

SMF Support => SMF 2.0.x Support => Topic started by: Zirkon on June 07, 2016, 03:37:28 PM

Title: The mail man is on strike
Post by: Zirkon on June 07, 2016, 03:37:28 PM
I have been fighting with a mail problem for some weeks now and getting no where. Cannot seem to find any topics here with any resolves. So here it is.

When sending a newsletter either by PM or Email only early members of the forum receive them. After a ways down the members list they get blank PM's and/or emails.

It's happening with all email providers.
Didn't copy/paste the content with Word.
Get the same results whether sending via the PHP or SMTP options. (SMTP setting have been provided by the host)
Sending by either newsletter PM version only or HTML only has the same issue.
Did a file refresh/install and it still happens.

Sending forum emails and PM's works fine.

Host says no problem on there end. I trust that as all is fine on my other site. All Newsletters and regular mail is readable.


This is just an issue with the Newsletter bulk send only.
Currently using SMF 2.0.11

With the file refresh/install/big upgrade method most of my files have been replaced. That leaves me with the settings files and the database.

Any ideas how to troubleshoot this or what to look for?
Title: Re: The mail man is on strike
Post by: Siirist on June 07, 2016, 10:35:16 PM
Hi,

It may be that the settings are set that they are not getting the announcements (aka newsletters)

See attached screen shots

Be Well,
Siriist
Title: Re: The mail man is on strike
Post by: Zirkon on June 08, 2016, 03:53:02 PM
Quote from: Siirist on June 07, 2016, 10:35:16 PM
Hi,

It may be that the settings are set that they are not getting the announcements (aka newsletters)

See attached screen shots

Be Well,
Siriist

That's not the problem. As I stated they receive blank newsletters by PM or email. The newsletter title is all they can read. The body of the message is blank.

If they aren't enrolled in newsletters they get nothing. That part works fine. The problem is with those that actually want the newsletter get blanks.

In hopes someone has an answer.
Title: Re: The mail man is on strike
Post by: Zirkon on June 09, 2016, 03:44:31 PM
Just an update I looked through the settings files BAK & PHP and cannot find anything in them that would cause this issue. Think it might be database related but have no idea how to figure it out.
Title: Re: The mail man is on strike
Post by: Zirkon on June 10, 2016, 09:51:10 PM
Does anyone know if there is a limit or glitch in SMF for the size/amount of mail in the database? I have some members that use the PM system heavily for chatting. You always see them sending back n forth on Who's Online. The past newsletters have been sent via PM also. Thinking maybe all that mail clogging up the database might be causing this issue. It's a shot in the dark at this point.

I remember reading on here that after a certain amount of posts there was a glitch in SMF that you had to mend. Not sure if that is still a current issue or not. Think it was generating blank posts when it happened but my memory is foggy.

Are there any support people here that specialize in the mail function of SMF?
Title: Re: The mail man is on strike
Post by: Sir Osis of Liver on June 10, 2016, 10:05:40 PM
What are your mail queue settings? 
Title: Re: The mail man is on strike
Post by: Zirkon on June 10, 2016, 10:22:29 PM
These are the current settings.

Maximum emails to send per minute: 5
Maximum amount of emails to send per page load: 2

I have tried it with and without the queue. Never had a problem with the setting before.
Title: Re: The mail man is on strike
Post by: Sir Osis of Liver on June 10, 2016, 10:32:25 PM
How many are you sending?  It's odd that members receive the emails, but they're blank.  Nothing to do with settings, and mail is working if emails are sent.  Don't see how it could have anything to do with database.

Title: Re: The mail man is on strike
Post by: Zirkon on June 10, 2016, 10:35:08 PM
Just over 4000. I'm just guessing it's database related.  None of my mods are mail related.

I'm all eyes at this point. Appreciate your response to this thread. Thank you.
Title: Re: The mail man is on strike
Post by: Sir Osis of Liver on June 10, 2016, 11:02:58 PM
This problem has been reported previously, with no resolution -

http://www.simplemachines.org/community/index.php?topic=542565.msg3855252#msg3855252

I've posted an inquiry on team board, will see if anyone has a clue.
Title: Re: The mail man is on strike
Post by: Zirkon on June 10, 2016, 11:06:15 PM
 8) Thank you very much.
Title: Re: The mail man is on strike
Post by: nend on June 11, 2016, 01:45:36 PM
Hi,

Since this function is handled by JavaScript form re-submission, the problem may be caused by conflicting JavaScript code installed on your website or even by your very own web browser.

I ask that you provide any environmental details below, like what browser your using and any plugins installed such as ad blocking software.

Can you also provide any details of any scripts you may have installed, we are particularly looking for anything JavaScript related.
Examples of these can be
* AJAX, scripts that do things without a page reload.
* Advertising network ad codes.
* DOM modifying scripts that add things to the page.

Any script that does anything to the post window, is especially interesting as this is the same input name used in the JavaScript portion of the Newsletters script.

Also please provide a link to your site, this would help greatly with the troubleshooting process.

Thanks,
Title: Re: The mail man is on strike
Post by: Zirkon on June 11, 2016, 07:04:02 PM
The browser I use is Firefox.
Java on my computer was out of date but not enabled or installed in Firefox. Has been that way for some time.

Adblocker plugins were:
Adblock Edge      2.1.9.1
Disconnect      3.15.3.1

Forum Mods installed at time of issue were:

I also was running BlueImpsAjaxChat It was removed when I refreshed the SMF files.

At the time I was running SMF 2.0.9 now running 2.0.11 still having the newsletter issue.
Tapatalk has been updated to the current version 4.3.9
SMF Theme Changer has not been re-installed
Invitation Message In Your Face (IMIYF) has not been re-installed
More Spiders has not been re-installed

Theme changes and chat is now being handled by Simple Portal

All said and done it still is happening.

Is it okay if I send my sites link via PM? Rather not put it out in the open.
Title: Re: The mail man is on strike
Post by: Zirkon on June 11, 2016, 07:09:32 PM
Looking around it happened to this fellow to somewhat recently http://www.simplemachines.org/community/index.php?topic=543304.0

We share some similar modifications if it helps any.
Title: Re: The mail man is on strike
Post by: nend on June 11, 2016, 10:22:26 PM
Since the newsletters portion is done via JavaScript I would recommend disabling the ad block plugins for your web site. However I don't believe this to be the root of the problem.

Can I have you edit...

In Sources/ManageNews.php
Find
// Save the message and its subject in $context
$context['subject'] = htmlspecialchars($_POST['subject']);
$context['message'] = htmlspecialchars($_POST['message']);


Replace with
// Save the message and its subject in $context
$context['subject'] = $smcFunc['htmlspecialchars']($_POST['subject']);
$context['message'] = $smcFunc['htmlspecialchars']($_POST['message'], ENT_QUOTES);


After the edits go ahead and give it another try and report back if this solved the issue for you.

I have a hunch that this is the problem.
Title: Re: The mail man is on strike
Post by: Sir Osis of Liver on June 11, 2016, 11:30:22 PM
Wouldn't that affect all emails? OP and other poster reported that some emails are complete, others are empty.
Title: Re: The mail man is on strike
Post by: Zirkon on June 11, 2016, 11:34:36 PM
Edited.

Waiting for it to work its way through the queue. Hope this works.

I received the email to my admin account being at the top of the list. All content was there, as has been, it's just when it gets further down the list the problem starts. Will see what happens.
Title: Re: The mail man is on strike
Post by: nend on June 11, 2016, 11:55:26 PM
Quote from: Sir Osis of Liver on June 11, 2016, 11:30:22 PM
Wouldn't that affect all emails? OP and other poster reported that some emails are complete, others are empty.


It only affects emails sent via the Newsletter form in the Administration Center, other email notifications run a bit differently as they are database driven.

Quote from: Zirkon on June 11, 2016, 11:34:36 PM
Edited.

Waiting for it to work its way through the queue. Hope this works.

I received the email to my admin account being at the top of the list. All content was there, as has been, it's just when it gets further down the list the problem starts. Will see what happens.

You should keep the browser sitting on the page in the Administration Panel or the Que will not be processed. If the Que stops before completing then there is something else wrong.

*edit @ Zirkon, disregard what I typed above, must of misread your post.
Title: Re: The mail man is on strike
Post by: Sir Osis of Liver on June 12, 2016, 12:07:54 AM
Quote from: nend on June 11, 2016, 11:55:26 PM
It only affects emails sent via the Newsletter form in the Administration Center, other email notifications run a bit differently as they are database driven.

What I meant was, shouldn't all emails be blank.  Users are reporting that some recipients get the complete email, while others receive an empty email, all in same batch.
Title: Re: The mail man is on strike
Post by: nend on June 12, 2016, 12:18:16 AM
Quote from: Sir Osis of Liver on June 12, 2016, 12:07:54 AM
Quote from: nend on June 11, 2016, 11:55:26 PM
It only affects emails sent via the Newsletter form in the Administration Center, other email notifications run a bit differently as they are database driven.

What I meant was, shouldn't all emails be blank.  Users are reporting that some recipients get the complete email, while others receive an empty email, all in same batch.

If the forum encoding is not UTF-8, when the first batch gets processed, the initial form submit it is submitted with the correct encoding. However since we are not using SMF's function here, if the PHP version is 5.4 or above then when htmlspecialchars gets called it converts the entities to UTF-8 to pass on to the form. Since the form is not UTF-8, from there on you have mismatched encoding which I believe is the cause.

*edit, I couldn't understand what I typed.
Title: Re: The mail man is on strike
Post by: Sir Osis of Liver on June 12, 2016, 12:26:35 AM
Actually, I kind of understand that.  Someone who is experiencing the problem will have to try your fix.  Zirkon?
Title: Re: The mail man is on strike
Post by: Zirkon on June 12, 2016, 12:40:40 AM
I'm lost now. Not sure if I was supposed to do the edit or not. It was done when nend posted the code.

I watched the queue for a while. It was sending. Closed the browser. 40 minutes later (now) I opened it and it was sending the emails while the browser was closed.

Waiting for it to get to my test account which is further down the list to see what it produces.
Title: Re: The mail man is on strike
Post by: Zirkon on June 12, 2016, 12:43:15 AM
I do not have UTF-8 setup on the forum and sending with SMTP.
Title: Re: The mail man is on strike
Post by: Sir Osis of Liver on June 12, 2016, 12:55:29 AM
The problem is not the queue, all emails are being sent, but some are arriving empty.  If nend is correct, first batch is ok, but later in the process they become blank.  That's consistent with what you and other posters have reported.  Converting database to utf-8 may fix the problem, but it's not utf-8 by default.  If the $smcFunc corrects the problem, that's a better solution.
Title: Re: The mail man is on strike
Post by: Zirkon on June 12, 2016, 01:00:21 AM
Thanks for clarifying. I understand now.

Title: Re: The mail man is on strike
Post by: Zirkon on June 12, 2016, 10:13:46 AM
I'm getting reports from both new and long time members that the newsletter was received in full. Newsletter was received fine on my test account too.

I thank you very much for this fix Nend  8)

Still trying to wrap my head around this "$smcFunc" part. I understand it is a database trigger of sorts. Is this something I should be concerned about with my database file or is this a UTF-8 issue? I'm still left to try and figure out why this all of a sudden was a problem when nothing was done to the forum files for a good long time. Host upgrades/patches of the server files? Many of my files were 4 years old or older, file rot possibly?

This is one issue I would not have been able to fix on my own, so thank you again for your support.
Title: Re: The mail man is on strike
Post by: nend on June 12, 2016, 10:47:47 AM
Quote from: Zirkon on June 12, 2016, 10:13:46 AM
I'm getting reports from both new and long time members that the newsletter was received in full. Newsletter was received fine on my test account too.

I thank you very much for this fix Nend  8)

Still trying to wrap my head around this "$smcFunc" part. I understand it is a database trigger of sorts. Is this something I should be concerned about with my database file or is this a UTF-8 issue? I'm still left to try and figure out why this all of a sudden was a problem when nothing was done to the forum files for a good long time. Host upgrades/patches of the server files? Many of my files were 4 years old or older, file rot possibly?

This is one issue I would not have been able to fix on my own, so thank you again for your support.

More than likely your host updated PHP and this caused the issue. PHP changed the default output for htmlspecialchars in PHP 5.4 to UTF-8. Since your forum doesn't use UTF-8 and tells the browser it uses ISO, they can't understand each other when talking. Even though the browser knows UTF-8, the document tells the browser it is a different language so the browser treats UTF-8 as ISO.

It is just the same thing as trying to convert Chinese to English when your told the source is Japanese if that makes any sense.
Title: Re: The mail man is on strike
Post by: Fire & Safetyfora on June 12, 2016, 11:06:18 AM
What is the link to your forum?
Title: Re: The mail man is on strike
Post by: Zirkon on June 12, 2016, 11:18:42 AM
Okay that explains it Nend.  I will inform my host in case the issue pops up again with any of there customers using SMF.

Thank you for the explanation.  ;) I feel better knowing I was somewhat going in the right direction with this bug. Still never would have figured it out on my own.

My other forum doesn't have this problem but is much smaller in size. Only 260 members or so. Do you think I should apply this patch to that site too? Seems like I should as that one depends heavily on the newsletter function every month for meeting locations. Think I might be getting close to where this problem starts if we pick up more members there.

Title: Re: The mail man is on strike
Post by: Zirkon on June 12, 2016, 11:53:23 AM
Quote from: Fire & Safetyfora on June 12, 2016, 11:06:18 AM
What is the link to your forum?

I thought this was an out of the blue question until I looked back into your posts. Looks like you have had similar issues. I did read your thread and the others when trying to research this problem. I won't post my site link here but can send it to you in a PM if you wish. Sort of feel vulnerable doing that here.

It makes me wonder how many forums have this issue and just don't know about it as they may not send newsletters or just gave up on a fix when threads here about the issue have not been resolved.

Do you think maybe SMF needs a new update to 2.0.12, to get it current to standards?

Not trying to cause any problems here. I just know that this has been a massive headache and burden in my life these past weeks that has haunted me to no end until today. The hosts are doing there part by putting these updates to the servers in effect as they should be doing.

With all of the blank sends I have lost a chunk of members on my forum with account deletion requests with every failed attempt. That part stings a bit as their was no other way to troubleshoot this. I have been getting regular beatings from my counterpart on this with every request.

I ask that the SMF staff please look into this matter.

Thank you for your support.
Title: Re: The mail man is on strike
Post by: margarett on June 12, 2016, 12:33:36 PM
It is being looked at. Can't promise it will be fixed in 2.0.12 but it will be fixed ;)
Title: Re: The mail man is on strike
Post by: Zirkon on June 12, 2016, 12:37:58 PM
 8) Thank you Margarett.
Title: Re: The mail man is on strike
Post by: Fire & Safetyfora on June 13, 2016, 10:24:07 AM
Quote from: Zirkon on June 12, 2016, 11:53:23 AM
Quote from: Fire & Safetyfora on June 12, 2016, 11:06:18 AM
What is the link to your forum?

I thought this was an out of the blue question until I looked back into your posts. Looks like you have had similar issues. I did read your thread and the others when trying to research this problem. I won't post my site link here but can send it to you in a PM if you wish. Sort of feel vulnerable doing that here.

It makes me wonder how many forums have this issue and just don't know about it as they may not send newsletters or just gave up on a fix when threads here about the issue have not been resolved.

Do you think maybe SMF needs a new update to 2.0.12, to get it current to standards?

Not trying to cause any problems here. I just know that this has been a massive headache and burden in my life these past weeks that has haunted me to no end until today. The hosts are doing there part by putting these updates to the servers in effect as they should be doing.

With all of the blank sends I have lost a chunk of members on my forum with account deletion requests with every failed attempt. That part stings a bit as their was no other way to troubleshoot this. I have been getting regular beatings from my counterpart on this with every request.

I ask that the SMF staff please look into this matter.

Thank you for your support.

No, not a out of the blue question. Send your website in a PM to me, i think i can help you. ;)
Title: Re: The mail man is on strike
Post by: Kindred on June 13, 2016, 06:35:30 PM
He has already resolved the issue...
Title: Re: The mail man is on strike
Post by: Zirkon on June 13, 2016, 09:33:32 PM
Kindred pretty much answered the question.

No point to it as my site is running very smooth now. Nothing to fix.
Thanks again for the help.