Since this post I've had some time to reflect on a few things, namely the stuff I think about that's essentially API connectivity. Might spark a few ideas or discussions, so here we go.
Specifically, there's a bunch of places where things are primarily extensible along a single axis and while you could do much larger changes, the vast focus of such is along those axes. For example, new payment providers for paid subs, new search backends, new autosuggest handlers, new scheduled tasks.
So I thought about it and came up with a solution I think is sensible. Each of those different things in my setup is already a class - each of the autosuggest backends, each of the scheduled tasks, each of the payment providers, they're all separate classes and (now) all of them have parent interfaces to work off. (Some have abstracts to help them but they all have discrete interfaces, Payment\PaymentProvider, Search\Searchable, Helper\Autocomplete\Completable, Task\Schedulable)
Each of these now extends an interface called Discoverable, and this is where things get interesting - the system will on occasion scan everything it can find and log which interfaces extend Discoverable, and any classes that implement *that* interface can be found pretty painlessly by asking the class manager for the classes that implement a specific interface. I didn't do it for all interfaces and all classes, because that seemed overkill, but doing it for a specific subset of interfaces seemed like a sensible idea, and this extends pretty seamlessly to plugins.
This means several things I can now do:
1. I don't have to maintain a list of scheduled tasks any more during the installer. The installer could, during late stage setup, request the list of scheduled tasks, find the ones it doesn't already have and add them. (I already remodelled the scheduled tasks table to not rely on having a function name but only a class name, and classes are responsible for providing their own name/description now). This does imply that the tasks have some idea what their schedule should be as well, but that's OK, most of them run daily and it doesn't take much to create something like the stock install list if you already have a list of class names.
2. I don't need to have hooks or weird detection to implement new things. If I make a plugin that provides, I dunno... Stripe as a payment provider, I make a class that follows the PaymentProvider interface and that's it. No more looking for files of the right name structure or specific comments in their header, I just implement the right interface, run the install and I'm done. Similarly for autosuggest, I don't have to go and create the logic then add a hook, I just define the class with the right interface, run the install (or clear cache), and it's ready to go.
It's also a long-life cache so it does occasionally need to be rebuilt if you add/remove a plugin or something like that, but the price of convenience in general totally seemed worth it to me. Doing it for caching requires more thought though because of the way I do everything else with hooks.
As for overheads, the overhead is really only paid if you take a code path that actually uses one of these; autosuggest, paid subs, search etc. don't need to do this lookup and don't even load the ClassManager if they don't actually need it, and while there is a slight cost of actually having to load the cache/load the extra interface, this is offset very easily by opcache.
Example for what I mean, loading a list of the autocomplete backends where it builds a map of things like 'member' => 'StoryBB\\Helper\\Autocomplete\\Member' for the routing purposes:
$searchTypes = [];
foreach (ClassManager::get_classes_implementing('StoryBB\\Helper\\Autocomplete\\Completable') as $class)
{
$searchTypes[strtolower(substr(strrchr($class, '\\'), 1))] = $class;
}
(where ClassManager has an appropriate use statement)
But it just returns a regular array and lets the caller decide how it wants to use that list. Payment gateways do more for example than just this (though in theory there's no reason why I couldn't boil it down more)
Y'all might just think I'm a bit crazy but crazy is good right??

Anyway, I hope this has been interesting and hopefully it might spark an idea or two.