I want only the selected people to fill survey. Can I make my survey Invite only?

You can make your survey Private so that only the people who have its password are able to fill in the Survey response. 

Follow these steps to make your Survey Private : 

Login >> Survey >> Click "Manage" on the particular Survey >> In the left panel, click on "Setup" >> Settings >> Visibility : "Private"

Once you select "Private", you will be asked to create a password.