The only other solution is setting skip for every permission (other than the ones you want to be overridden) in that server group.