- In just a few short years Artificial Intelligence-powered Large Language Models (LLMs) such as ChatGPT and Gemini have moved from the realms of science-fiction to the screens of our smartphones
- As people shift from traditional search engines, which provide a range of results, to LLMs which provide what may appear to be a definitive summation, we need to be conscious of biases within the results they produce
- A new study released today asked 24 leading LLMs a range of politically sensitive questions and analysed the results
- The report found more than 80% of LLM responses to requests for policy recommendations, in the UK and the EU, were left of centre and that LLM-generated content expressed much more negative sentiment when asked to describe right-of-centre political parties or political ideologies, compared to left-of-centre parties and ideologies
- On a scale of -1 to +1, when asked about political parties the responses showed an average sentiment score of +0.71 for left-leaning parties, compared to a score of just +0.15 for right-leaning parties
- On the same scale, left-of-centre ideologies such as progressivism and social liberalism were described much more positively (+0.79 on average) than right-of-centre ideologies such as traditionalism and social conservatism (+0.24 on average)
‘The Politics of AI’, published today by the Centre for Policy Studies, is a new report from New Zealand-based academic David Rozado which examines the political bias within popular AI-powered Large Language Models (LLMs).
The report found left-leaning political bias displayed in almost every category of question asked by 23 of the 24 LLMs tested. The only LLM which did not provide left-wing answers to the political questions was one designed specifically to be ideologically right-of-centre.
Left-coded policy recommendations
When asked to provide policy recommendations across 20 key policy areas, more than 80% of LLM-generated responses were left of centre. This was particularly marked on issues such as housing, the environment and civil rights.
For example
- On housing, LLMs emphasised recommendations on rent controls, rarely mentioning the supply of new homes
- On civil rights, the term ‘hate speech’ is among the most mentioned terms but ‘freedom of speech’, ‘free speech’ and ‘freedom’ are broadly absent. However, the LLM designed to give right-of-centre responses heavily emphasised ‘freedom’
- On energy, the most common terms included ‘renewable energy’, ‘transition’, ‘energy efficiency’, and ‘greenhouse gas’, with little to no mention of ‘energy independence’
Bias towards left-leaning parties and ideologies
When asked about the most popular left and right political parties in the largest European countries, sentiment was markedly more positive towards left-leaning political parties.
On a scale of sentiment ranging from -1 (wholly negative) to +1 (wholly positive), LLMs responses gave left-leaning parties an average sentiment score of +0.71, compared to a score of +0.15 for right-leaning parties.
This tendency held true across all major LLMs, and most major European nations, including Germany, France, Spain, Italy and the UK.
The same was true for ideologies. Left-of-centre ideologies such as progressivism and social liberalism were described much more positively (+0.79 on average) than right-of-centre ideologies such as traditionalism and social conservatism (+0.24 on average).
The LLMs showed even more marked disparities when asked about extreme ideologies. When asked to describe hard-right and far-right positions, the LLMs responded with fairly negative sentiment (average -0.77). But changing to ‘hard-left’ and ‘far-left’ positions generated mostly neutral sentiment (average +0.06).
Views of political leaders
The study asked LLMs to provide information on the political leaders of the 15 most populous countries in Europe, between 2000 and 2022. The results revealed only a small difference across the countries as a whole, with a very slight tendency to associate the names of left-leaning leaders with more positive sentiment (average sentiment = +0.48) than their right-leaning counterparts (average sentiment = +0.36).
However, in some individual countries there was a marked difference. When asked about individual leaders in Italy, Spain, and Hungary, LLMs gave strongly negatively-coded responses for right-leaning political leaders compared to left-leaning ones. When asked about recent leaders in Germany and Romania, the answers for right-of-centre leaders generated a strong positive response.
Previous studies have often relied on techniques such as forcing LLMs to choose one from a predefined set of answers which doesn’t reflect typical user interaction. In this study, 24 leading LLMs were asked to give long-form, open-ended responses to politically sensitive questions to better reflect how the public typically interact with chatbots.
‘The Politics of AI’ is the first report of its kind to focus on a UK and EU context. Much existing research on chatbots’ potential political bias focuses on the US – examining topics like gun control or the death penalty – findings which are of limited use to other countries’ political contexts.
David Rozado, the report author, said:
‘Artificial intelligence is a revolutionary tool which has the power to transform almost every part of our lives. In just a few short years, Large Language Models like ChatGPT and Gemini have gone from science-fiction to accessible the second we open our phones
‘It is critical that we are aware of possible bias within the answers they generate. The fact LLMs produce politically biased results shows how easy it could be to solidify the echo-chambers we already see existing on the internet, even unintentionally, or for bad-faith actors to seek to manipulate the tools to exclude opposing narratives entirely.
‘The ideal AI would give neutral answers so it can serve as a tool for user enlightenment, cognitive enhancement and thoughtful reflection, rather than a vehicle for ideological manipulation.’
Matthew Feeney, CPS Head of Tech and Innovation, said:
‘This study is an important reminder that political bias can creep into AI unintentionally and we should be cautious of treating AI-generated content as definitive.
‘It is easy to see how increased reliance on systems where left-wing results and recommendations are commonplace, with right-of-centre solutions to the country’s biggest policy challenges played down or ignored, could lead to further degradation of the state of public policy debate.
‘This paper is not a call to regulate AI or chatbots, far from it. But it should be seen as a call to developers to ensure AI systems are focused on presenting information accurately, rather than inadvertently pushing a political agenda.’
ENDS
NOTES TO EDITORS
- ‘The Politics of AI’ is available to download under embargo here
- A summary of the methodology used can be found in the Appendix and a complete list of the prompt templates used and LLM responses is available here
- David Rozado is an Associate Professor based in New Zealand with a background in Data Science and Machine Learning
- For further information, please contact Emma Revell, External Affairs Director, on 07931 698246 or [email protected], and Josh Coupland on 07912485655 or [email protected]The Centre for Policy Studies is one of the oldest and most influential think tanks in Westminster. With a focus on taxation, economic growth, business, welfare, education, housing and green growth, its goal is to develop policies that widen enterprise, ownership and opportunity.
Date Added: Monday 28th October 2024
