Developer docs?

[motechman]

I need to get to the bottom of an email issue and am trying to understand the code (I'm an experienced software dev). Are there any docs that describe SMF such as the scheduling algorithm?

At this point on my learning curve ANY suggestions or pointers are welcome, such as conventions, where constants are generally found, important files or DB tables etc.

What is the general flow of processing a page request, and how does that drive the scheduling system? I can see in the ScheduledTasks.php module how emails are processed out of the queue and how daily digests of notifications are processed but it's not so clear how items are added to the mail queue or activity is flagged for notification.

From what I've read in the user documentation, if notification is requested for a board, any posts to topics (including new topics) in that board will be flagged for notification. Further, I gather that the log_notify table plays a central role, with the "sent" column updated to a "1" after the ScheduledTasks sends the notification. When is it then reset for the next cycle of posts?

Lastly I discovered the functions in Subs-Post.php, including "sendNotifications" and "AddMailQueue". I have yet to study those in detail. It looks like the sendNotifications function handles board level notifications, which is called when posts are made. If 100 members request notification on a topic which has a new post, and the system configuration uses the mail queue, how are the notification emails generated and sent thru that mail queue?

Code Select Expand


// Find the members with notification on for this topic.  (AND PRESUMABLY THE BOARD THIS TOPIC IS UNDER)
$members = $smcFunc['db_query']('', '
SELECT
mem.id_member, mem.email_address, mem.notify_regularity, mem.notify_types, mem.notify_send_body, mem.lngfile,
ln.sent, mem.id_group, mem.additional_groups, b.member_groups, mem.id_post_group, t.id_member_started,
ln.id_topic
FROM {db_prefix}log_notify AS ln
INNER JOIN {db_prefix}members AS mem ON (mem.id_member = ln.id_member)
INNER JOIN {db_prefix}topics AS t ON (t.id_topic = ln.id_topic)
INNER JOIN {db_prefix}boards AS b ON (b.id_board = t.id_board)
WHERE ln.id_topic IN ({array_int:topic_list})
AND mem.notify_types < {int:notify_types}
AND mem.notify_regularity < {int:notify_regularity}
AND mem.is_activated = {int:is_activated}
AND ln.id_member != {int:current_member}' .
(empty($members_only) ? '' : ' AND ln.id_member IN ({array_int:members_only})') . '
ORDER BY mem.lngfile',
array(
'current_member' => $user_info['id'],
'topic_list' => $topics,
'notify_types' => $type == 'reply' ? '4' : '3',
'notify_regularity' => 2,
'is_activated' => 1,
'members_only' => is_array($members_only) ? $members_only : array($members_only),
)
);


[Suki]

Hello.

SMF handles almost everything from its main index.php file, that file delegates the process to a Source file depending on which action is been called.

We don't use constants, just one and thats only to define our enviroment.  Any important file is loaded by index.php itself before and pages gets processed.

The scheduled task system gets called on each  request, our scheduled system relies on page requests (you can of course set up an external cron job and call it directly).


Scheduled.php calls its own function ReduceMailQueue() which deals with processing the mail queue, its pretty straight forward process, theres a mail_queue table that has all relevant info, whenever a notification process gets set, that table is updated and thats the info ReduceMailQueue() uses to know how many emails are going to be sent on each page load.

[motechman]

Awesome Suki, appreciate the info.

If as you indicate the scheduledTasks function is called on every request, I could increase the rate at which emails are processed simply by just making a request of index.php (by using a monitoring tool like monit or cron as you suggest).

If I had 60 members to be notified for a given post and10 emails are processed on each request, all emails would be sent after only 6 requests. I can reduce the number of emails processed on each request and make more requests stretched out over a longer period to reduce the load on each request and improve responsiveness.

Thx again Suki, you've been a big help.

[Suki]

Yes, you could determinate the exact amount of mails that gets sent by controlling the amount of requests to index.php.  You can also bypass index.php entirely and just directly call ScheduledTask::AutoTask()  on a separate php file, you will need a couple of SMF files but thats about it.

Just be aware that this is still a PHP process and as such its still subject to things like memory limit, requests time outs, php's mail() limitations,  things like that.

[motechman]

Thx for that update. I'm honing in on the problem, but my understanding is still weak in how board level notifications are processed.

I though when I first spotted this query in Subs-Post.php it would find all members that need to be notified, either board level or topic level:

Code Select Expand


// Find the members with notification on for this topic.
$members = $smcFunc['db_query']('', '
SELECT
mem.id_member, mem.email_address, mem.notify_regularity, mem.notify_types, mem.notify_send_body, mem.lngfile,
ln.sent, mem.id_group, mem.additional_groups, b.member_groups, mem.id_post_group, t.id_member_started,
ln.id_topic
FROM {db_prefix}log_notify AS ln
INNER JOIN {db_prefix}members AS mem ON (mem.id_member = ln.id_member)
INNER JOIN {db_prefix}topics AS t ON (t.id_topic = ln.id_topic)
INNER JOIN {db_prefix}boards AS b ON (b.id_board = t.id_board)
WHERE ln.id_topic IN ({array_int:topic_list})
AND mem.notify_types < {int:notify_types}
AND mem.notify_regularity < {int:notify_regularity}
AND mem.is_activated = {int:is_activated}
AND ln.id_member != {int:current_member}' .
(empty($members_only) ? '' : ' AND ln.id_member IN ({array_int:members_only})') . '
ORDER BY mem.lngfile',
array(
'current_member' => $user_info['id'],
'topic_list' => $topics,
'notify_types' => $type == 'reply' ? '4' : '3',
'notify_regularity' => 2,
'is_activated' => 1,
'members_only' => is_array($members_only) ? $members_only : array($members_only),
)


But not so. When a member ONLY has elected board level notifications and has never posted in the topic, this query will not generate a notification to that user.

I'm specifically trying to understand how board level notifications are triggered when a post is made to a topic. I thought if the contents of the log_notify table were basically a Cartesian join of all the member_ids and all of the board_ids that should notify everyone when a post is made to any (topic under) any board. That assumption is incorrect, but I'm trying to understand why.

I used the following query to fill the log_notify table (after truncating it):

Code Select Expand


insert into smf_log_notify (`id_member`, `id_topic`, `id_board`, `sent`)
select distinct m.id_member, 0, b.id_board, 0
  from smf_boards b
    join smf_members m


This produces a row for each board with every member; rows = number of members * number of boards. In my case that is 2838 rows.

However, to determine which members get notified when a post is made the tables are joined differently:

Code Select Expand


SELECT distinct M.id_member
  FROM smf_log_notify N their
  INNER JOIN smf_members M ON N.id_member = M.id_member
  INNER JOIN smf_topics T ON N.id_board = T.id_board
WHERE T.id_topic = THE TOPIC OF THE NEW POST


That query will produce a list of all members no matter where a post is made. The only reason to do this is that it uses the existing architecture and preserves each user's control over notifications. It makes notifications enabled by default with the option in each user's profile to opt out (every board is listed with a checkbox under "Board Notifications".

Now I could modify the code to send notifications to all members on each post, but I don't like solving the problem via such hacks. Some may consider it a hack to fill the database as I did to achieve a "global notification", but it's not really, as the same thing could be done manually if every member elected to be notified of activity on all boards. All my query does is automate that process. But there's something I'm missing and I don't understand what it is.

Can you shed some light on how board level notifications work and how they are processed?

[motechman]

I am 99% certain board level notifications do not work properly, meaning that when new posts are made to topics in a board having notifications set, no notifications are made to members who have requested board level notifications on such boards. I have not yet systematically performed the steps below but when I return I intend to. My testing has been ad hoc to this point, but I am rather confident non-the-less.

Test criteria:
1 ) SMF 2.11 installation on Linux Ubuntu, mail queue enabled, php mailer, 80 emails / minute, 10 per page load
2 ) patch Sources/ScheduledTasks.php by adding "return false;" right after global references so mail queue is not emptied
3 ) empty log_notify table
4 ) Member #1 & #2 - set profile to: no newsletters etc, no notification on post or reply, no post content in notification email,
     instant for 1st unread reply (no digest), notify of replies and moderation (alternate test condition: allow notification on post or reply, which is default)
5 ) Member #1 - Elect board notification on a single board
6 ) Member #2 - Elect board notification on the same board as Member #1
7 ) Verify log_notify now contains only 2 rows with correct id_member and id_board corresponding to steps 5 and 6 above
8 ) Member #1 posts a reply to an existing topic under the board both members requested to be notified about

The above test should result in an email notification in the mail queue for member #2. Since there are no entries in the log_notify table with a non-zero value for id_topic, the database query used by the sendNotifications() function will not return any rows and thus no notifications will be performed, although activity has transpired in a board marked for notification.

I will reply with my findings after I return.

[Illori]

board notifications only notify users of new TOPICS that have been made, not replies to those topics.

[Kindred]

and, IIRC, only informs them of the FIRST new topic -- it won't send another notification until they visit the board and reset the flag.

(at least that is how the notify of new responses within a thread works, so I kinda assume the same logic was used for notify of new topics in a board)

[Illori]

that depends on the setting you select in your profile for notifications.

[motechman]

board notifications only notify users of new TOPICS that have been made, not replies to those topics.

Well well well, that explains a lot. No wonder users are not being notified as I expected, my expectations were all wrong.

Now I need to engineer a solution to my problem, so it looks like I need to dive into the "mod" rabbit hole to do it right. Bummer, the problem just became much more involved.

Thanks to all who commented here to set me straight.

[motechman]

I can anticipate catching some flak for this post, and I agree these changes are definitely a hack and it would be better to do this through a mod. I have not studied the mod API so I don't know this would be possible using it but I'd be surprised if it wasn't.

Anyway, I have decided to do this for the usual reason of expediency. A poor excuse but it gets the job done for this particular community. It has the bad side affect of having to perform these changes manually with each new SMF version, with no guarantee concerning it's side affects in that version. At least the changes are isolated to a single SQL query in a single php file, plus the manual query to truncate the log_notify table and fill it as described (BTW, I see I left out the function with ScheduledTasks be commented out -  not the main scheduler but ReduceMailQueue).

This hack also has the limitation that the log_notify table must be updated with each new member added. If that is left out any new members added would need to opt in for notifications by going to each board and requesting notification be turned on. This hack should never be used on forums with a large number of boards or members! With 43 boards and 68 members there are just shy of 3000 rows in the log notify table for all members to be notified of a post anywhere on the forum.

When I get some time I will investigate turning this into a mod.

Query to fill {db_prefix}log_notify table:

Code Select Expand


insert into smf_log_notify (`id_member`, `id_topic`, `id_board`, `sent`)
   select distinct m.id_member, 0, b.id_board, 0
      from smf_boards b, smf_members m
   where m.id_member not in (select distinct id_member from smf_log_notify)


The above can be run anytime a new member is added or initially to fill the table. You could truncate the table first so only board level notifications exist there. Topic level notifications are no longer necessary with this hack and are not processed, so there is no need for those entries in the table.

Replacement Query in Subs-Post.php::sendNotifications:

Code Select Expand


        $members = $smcFunc['db_query']('', '
                SELECT
                        mem.id_member, mem.email_address, mem.notify_regularity, mem.notify_types, mem.notify_send_body, mem.ln$
                        ln.sent, mem.id_group, mem.additional_groups, b.member_groups, mem.id_post_group, t.id_member_started,
                        t.id_topic
                FROM {db_prefix}log_notify AS ln
                        INNER JOIN {db_prefix}members AS mem ON (mem.id_member = ln.id_member)
                        INNER JOIN {db_prefix}topics AS t ON (t.id_board = ln.id_board)
                        INNER JOIN {db_prefix}boards AS b ON (b.id_board = ln.id_board)
                WHERE t.id_topic IN ({array_int:topic_list})
                        AND mem.notify_types < {int:notify_types}
                        AND mem.notify_regularity < {int:notify_regularity}
                        AND mem.is_activated = {int:is_activated}
                        AND ln.id_member != {int:current_member}' .
                        (empty($members_only) ? '' : ' AND ln.id_member IN ({array_int:members_only})') . '
                ORDER BY mem.lngfile',


This query will return the list of members who have board level notifications enabled for the parent board containing the post(s). Board level notifications are enabled / disabled in the standard way (they can be disabled in each board or in the member's profile, but may only be enabled in each board; the query that fills the log_notifiy table above "opts in" all members by default)