Odd random bug with email

[smc1979]

Hi all.

I have searched the forum and found a lot of things on email. But none of them where like my problem.

I run 2 forums on 2 different hosts, so I have been able to verify it isnt anything with the hosts.

Now email works. I have notification on every board so I know when a user posts a new topic. Plus I have it set to notify me on any thread I post on. I at one point I had email on for activation but had to turn it off because of what is happening.

What is happening on both sites is not all notifications get emailed out. 90% of them do. And this even happened with activation on a new account. Had users send me a email from my site saying they never got an email from the forum to finish registering.

So an example.

I wake up and see I have 6 emails from the forum, some new topics, others are replys, and some even letting me know I have a new PM. I log in and check for other ones and there is always 1 or 2 new topics or PMs that where posted that I never got a notification for.

I currently use php email on both sites and even tried using my gmail account. No matter what email works like it should but there is always a few that never get sent. I tried with all add ons disabled and such.

So it is odd. Email works fine, just every now and then the forums doesn't send them out. On both notifications and email activation. Since I disabled email activation I haven't had any more new member complaints.

Would just be nice to find out why some of them don't go out. And again this is with both new topic notifications and topic replys. Otherwise email works just fine.

Am I missing something?

Shane

[emanuele]

Are you using the mail queue?

[smc1979]

I tried with it on, but it has been off for a while now.

[mrintech]

http://wiki.simplemachines.org/smf/Mail

Maybe you are exceeding your Webhost's Hourly/Daily Email Limit. Configure mail queue by asking your webhost what's their hourly/daily Email limit?, else your webhost might suspend your Email services.

[smc1979]

Nope not a limit problem. My host has 200 emails per hour. Plus when I tried gmail they have a 500 email per day limit. You hit it you cant send emails for 24 hours. Never hit the limit with them either.

Already tested that. Plus when I look at the time of the posts of the missed ones there will be emails after the ones that I didn't get emails for.

I have tested everything I can think of and cant find the problem. It happens daily.

Perfect example is this morning. I woke up had 5 emails from the forum and 2 that didn't get mailed to me. (Total of 7)

The 2 that didn't get emailed to me are times in between the 5 that did get mailed to me. But for some reason these 2 didn't.

I dont think there is an answer to this one. Unless the programmers can replicate my problem they wont find the cause or a fix. I am a programmer as well and when a user submits a bug I have to be able to replicate it in order to fix it.

I just might be out of luck on this one.

Shane

[emanuele]

Can you check the error log and see if there is anything that could be relevant?

[smc1979]

Just checked all logs. Besides the insane amount of bots trying to login there isnt nothing else in the logs.

That is why this one if driving me nuts. I have nothing to point me in the right direction.

I dont know the core of the simple machines code so I cant go digging into the php and such.

I do think it is something in the core though. Something simply missing a beat.

I need to pay attention and see if there is any pattern to it.

I just logged into my other forum and have 5 emails from it and no missed ones. On the 2nd forum it isn't no where as busy as my main forum but it does happen there.

So far all I have to go by is the more busier the forum the more it happens.

Shane

[emanuele]

One thing I would suggest is to re-enable the mail queue and when you don't receive an email go there and see if it is in the queue or not (that *should* help in identify when the mail sending fails).

[smc1979]

Ok I will do that now on both sites and see if it helps locate any info.

Shane

[MrPhil]

I know you're pretty certain that you aren't exceeding host email sending rate limits, but does your host have a "per minute" cap? Maybe you're hitting that once in a while. Can your host provide you with any records of your actual email rate and that you definitely are or are not hitting any limits? Could someone have hijacked your site and is spewing out spam at 180 per hour or so? In that case, if the per-minute cap isn't more than 4 or 5, you might be hitting it fairly often. It would be good to definitely establish how many emails are actually going out from the site (presumably primary domain, add-on domains, subdomains are all together), before spending a lot of time diving through the code and instrumenting it.

[smc1979]

No already confirm it with them. 100 emails an hour. gmail is 500 emails a day. I used both, never hit any limits. If I had hit any limits I wouldn't be able to send any email my my account.

If you hit the limit with my host (dreamhost) they block you from sending for an hour. gmail is 24 hours. And my 2nd forum is on a partners server with no limits since it is his server. The 2nd forum is no where as busy as my main so it doesn't happen as much there but it still does happen.

I have enabled the mail queue to see if anything gets stuck there or not.

What would be a great support tool for something like this is a way, just for debugging, is to have a log of all emails sent out.

Anyways I will see how the next 24 hours go as it normally happens daily. :-)

Shane

[MrPhil]

Yes, an outbound mail log would be a good thing for debugging purposes (at a minimum). How about asking for a mod, or even a new feature?

[smc1979]

So far this morning I only had 1 post reply and got an email for it. But had a person send me a new PM and never got an email that I got it.

I will wait another 24 hours for more and see how it goes. Mail queue was empty.

If it had a log of emails sent out I could see if the forum ever even tried to send the email :-)

Shane

[emanuele]

I suggested the mail queue exactly because if there is an issue when sending the email (I mean the moment the "mail" command is executed) it adds the email back to the queue and will try to send it later.

The only case this would fail is if the script times out while still sending the emails.

So, the other thing you can try is to go to admin > maintenance > mail > settings and reduce the "Maximum amount of emails to send per page load" to something small: 5? (or even 1, but then if the forum doesn't have a lot of traffic you may not be able to keep up with the mail queue, you can try a couple of values and see)

That should completely avoid any timeout issue (hopefully).

[smc1979]

Today's emails did come in. My host must have a backup of email today as it took 2 hours to come in.

But since enabling the mail queue (I have it set to 3 emails per min and 0 for page load) on both forums I haven't had any missed notifications yet.

Today has been slower than normal though. Lets see how the next few days play out :-)

Shane

[emanuele]

For the record: 0 per page load means 5. That's an hard-coded value: if the setting is zero then it is set automatically to 5.

[smc1979]

Good to know, thanks :-)

Shane

[smc1979]

Been a few days and so far no missed notifications :-)

Shane

[smc1979]

OK so today I had 8 emails from the forums and only 1 notification was missed.

Since enabling the mail queue it has cut down how often this happens. Was happening everyday now its every few days. Thats a plus in my book.

Checked the mail queue and nothing was there so nothing I can find why this notification was missed.

Also on my 2nd forum that isn't as busy hasn't had a missed one yet.

If there was some way to log what emails are sent out at least I could verify if the forum ever sent the email in the first place or if it someone skipped the notification and never tried to send it to begin with.

Shane

[emanuele]

And here we are again explaining what's going on with the mail queue enabled:
1) a notification has to be sent,
2) SMF inserts the notification into the database,
3) at some point SMF decides to send a bunch of emails,
4) if gets some mails from the queue and deletes them (don't stop here, go on reading),
5) it takes those emails one by one and sends them logging any failure,
6) if everything is fine: HAPPY! And go sleep for a while,
7) for each and every mail in the batch, if the php command "mail" or the SMF function smtp_mail return false, SMF takes note of that,
8) if any of the emails returned an error while sending (see 7), SMF logs there has been an error (pause: if during the successive round of mails everything goes smooth the log is unset because it's useless) and inserts those mails back into the queue for further processing.

The nice part of that is if there is any error SMF can catch the mails are into the queue and you can see them, otherwise the error is somewhere else and probably difficult to catch (e.g. timeout, memory limit, something else network related, etc.).

Previously I asked about errors in the log because there is a bug that could affect the mail queue, so again: do you have any error in the log? Something like:

Code Select Expand

8: Undefined index: character_set
File: /smf/Sources/Subs-Post.php
Alternatively:
1) are you sure those emails are not in your spam folder? (the one of your email client or the one of your webmail?)
2) since you are keeping track of every notification you miss, is there anything "special" about these that the others (those that arrive) don't have? (posts longer than the average? Special chars somewhere? html? Anything else?)
3) try to reduce slightly the number of emails sent per round to 3 or 4.

[smc1979]

Only errors are ones caused by spam bots


Apply Filter: Only show the error messages of this URL
http://forums.********.com/index.php?action=login2;sa=check;member=23257
Apply Filter: Only show the errors with the same message
8: Undefined index: title
Apply Filter: Only show the errors from this file
File: /home/smcadmin/forums.pcwintech.com/Themes/default/Errors.template.php (body_above sub template - eval?)
Line: 461

1) are you sure those emails are not in your spam folder? (the one of your email client or the one of your webmail?)

Spam is disabled.

2) since you are keeping track of every notification you miss, is there anything "special" about these that the others (those that arrive) don't have? (posts longer than the average? Special chars somewhere? html? Anything else?)

No nothing that I can see as a pattern. All random.

3) try to reduce slightly the number of emails sent per round to 3 or 4.

Is currently already set to 3.

Shane

[smc1979]

Well no luck. Today was higher traffic.

Had only 3 emails for notifications and 4 missed ones.

Checked error logs and nothing what so ever. And mail queue was empty.

I don't know if it has to do that I am set to be notified on nearly every single post and reply that might be the problem, perhaps to many? Or what but since it doesn't seem to be happening to others I am going to have to just chalk it up as bad luck for me lol.

Again, being a programmer myself I know that the only way to fix anything is to be able to reproduce it. But so far I haven't been able to find a cause or pattern and no one else seems to have or notice the same problem.

If there was some sort of log feature to enable to simply record every email that was sent out I could at the very least confirm if it is ever even sent on these missed ones. That would point me in the right direction.

But for now I will just have to live with it as I have till I can get better info for you.

Thanks for all the help.

Shane

[LexusFTW]

Shane, have you been able to figure this out?  I'm having a similar issue.

[smc1979]

Nope it still happens. I just learn to deal with it.

I have it set to email me on every single new post that is made in all the forum sections and since I reply to every single post on both sites my notification list a very large. I think this is where the bug is. But I cant trace it and it is happening on both forums on different hosts.

But since I cant give anyone any way to replicate it, except for having a very high number of threads that are set to give you notification (I am talking 10,000+), then I don't think we will get an answer.

I even changed it to where the emails sent went through my gmail account instead and the problem was still there, and still very random. I have had multiple users (Not a high amount) saying they never get some emails but will get others as well.

Some days I have no missed email notifications, other days I login look for new replies and see that I miss anywhere from 1 to 5 of them or more. There is no constant behavior to it.

The only way to fix a bug is to replicate it. If only someone could make an add on to turn on a detailed log file for all notifications. Then I could track the sucker down :-)

Shane