I have continued my testing and found the solution.  The activity restriction has a further visibility option next to each restriction.  Toggling the eye will make the restricted activity visible or not if a user does not meet the conditions.