Hi Jesse
The audience based visibility is an elegant solution once this has been setup.- but not elegant to set up.
Possibly using Multitenancy and adding the users from the extra audience to the tenant and all the other users to another tenant.
Users are only able to see courses in the tenant category that they are a member or participant of.
Regards