Comment by kmundnic

Right, and unfortunately there doesn't seem to be a way to remove the permissions by app (only for all apps by turning off Handoff).