Can I only give a few people access to my personal folder?

Yes.

When you click on the pencil next to your folder, you can decide who can access your folder:

You will see this window:

You can easily add an user or multiple users by clicking on "Add User" and then "Save".