> Maybe it would be worth spending some mental cycles thinking about the impacts this will have and how we design these systems.
Regardless if the bot is actually sentient, a portion of the population may _believe_ the bot is sentient.
If in practice the bot reflects back to each user their own views but perhaps more extreme, then this could be a terrible recipe for reinforcing and amplifying negative and socially destructive thinking -- equivalent to social media bubbles on steroids.
This can occur even without explicit bad actors trying to tip the scales toward a specific outcome. This kind of AI bot as-is has the potential to bring out the very worst in at least some percentage of the population.
Regardless if the bot is actually sentient, a portion of the population may _believe_ the bot is sentient.
If in practice the bot reflects back to each user their own views but perhaps more extreme, then this could be a terrible recipe for reinforcing and amplifying negative and socially destructive thinking -- equivalent to social media bubbles on steroids.
This can occur even without explicit bad actors trying to tip the scales toward a specific outcome. This kind of AI bot as-is has the potential to bring out the very worst in at least some percentage of the population.