I personally think that it is not a bad idea in general. The following tweaks could be an answer to your valid concern.
1) Make 'Share server connection with friends list' a client option a user can toggle on or off.
2) Make 'Allow sharing of server connection' a server option a server admin can toggle on or off.