I want to do something custom - a link in the main menu to a custom page that checks for group membership then forwards the user to one of two different pages. I can handle pretty much all of this - the query to get the group membership, the code for setting the cookie, and the forwarding code - but I'm not sure where's the best place to do it. I know I want to set it at login - but I don't really understand the file structure of SMF so I don't know which file to put the code in.
None of this is necessary.
On your custom page include SSI.php from the forum, observe you now have $user_info for the logged in user, and more importantly $user_info['groups'] where you can check with in_array to see if the user is in the group or groups you care about.
You may need to set subdomain cookies to be on.