As mentioned before it's very well possible to do this, you just have to use the correct tools for the correct situation. You wouldn't attempt to use a huge drill bit to drill a tiny hole either.
Server groups by definition (and as the name suggest) provide permissions server wide (ie. across all channels). You however want to limit the permissions on a single channel (or a set of channels) as such a tool that works server wide is not going to work. You will have to use channel groups as they provide permissions on channel level.
Simply create a channel group that grants access to the channel, and give that channel group to those people in the channel(s) that those people should be able to join, remove the join power from the server groups that should not be able to go into all the channels and you're pretty much set.
When sending PMs please make sure to include a reference link to the thread in question in the body of your message.