If you click the options arrow on the chat feed (next to your name/status), there is a "Privacy Settings" option that allows you to specify which circles can chat with you.
fwiw, you can also specify which circles are in "Your circles" (the ones you broadcast to) and which circles you are just following in the G+ settings (https://www.google.com/settings/plus).
fwiw, you can also specify which circles are in "Your circles" (the ones you broadcast to) and which circles you are just following in the G+ settings (https://www.google.com/settings/plus).