"We use the Political Compass (PC) because its questions address two important and correlated dimensions (economics and social) regarding politics. [...] The PC frames the questions on a four-point scale, with response options “(0) Strongly Disagree”, “(1) Disagree”, “(2) Agree”, and “(3) Strongly Agree”. [...] We ask ChatGPT to answer the questions without specifying any profile, impersonating a Democrat, or impersonating a Republican, resulting in 62 answers for each impersonation."
The way they have done the study seems naïve to me. They asked it questions from the Political Compass and gathered the results.
Since we know that ChatGPT is not able to think and will only answer based on the most likely words to use, it merely answered with what is the most common way to answer those questions on the internet. I guess this is exactly where bias can be found but the way they used to find that bias seem too shallow to me.
I would love to hear the opinion of someone with more knowledge of LLMs. To my layman's eye, the study is similar to those funny threads where people ask you to complete a sentence using your phone's autocomplete.
The way they have done the study seems naïve to me. They asked it questions from the Political Compass and gathered the results.
Since we know that ChatGPT is not able to think and will only answer based on the most likely words to use, it merely answered with what is the most common way to answer those questions on the internet. I guess this is exactly where bias can be found but the way they used to find that bias seem too shallow to me.
I would love to hear the opinion of someone with more knowledge of LLMs. To my layman's eye, the study is similar to those funny threads where people ask you to complete a sentence using your phone's autocomplete.