No, you cannot limit who you share Smart Views with.
The Smart Views are shared in a project and are available for all team members.
The Project leader does have the option to block editing rights for certain people. When the 'edit rights' option is disabled, the team member will not be able to add and/or modify shared Smart Views.