Hi James,

Unfortunately not, user administration permissions give access to all users or none, there is not a way to manage segments of users.

There is a Suggest Features request for this (it talks about organisation based management, but the Tracker item also includes Audiences in the discussion.) I recommend going to that post to +1 and add your input.

Thanks!

Wes