When you select the Broadcast tab, your broadcast will be Public by default. To select who you share your broadcast with, tap "Public" to change the audience.
Once you tab "Public", you will see the following screen showing the Groups you are a part of and a "Search Mutual Followers" option. For a user to be searchable here, you will need to follow them and they will need to follow you. You can select either a Group or mutual followers. Once selected, tap "Done" in the upper right corner.
The Group or mutual followers you select will show up at the top of the screen, and your broadcast will only be shared with Group members or your selected followers.
You can learn more about what happens when you start a private broadcast here.