<p>The Estonian Language Institute (ELI) has introduced a new benchmark called "Propaganda Resistance" to evaluate the effectiveness of large language models (LLMs) in avoiding narratives associated with Russian propaganda. This initiative comes as concerns grow among state governments regarding the potential for LLMs to disseminate misleading information from foreign adversaries.</p><p>Estonia, a former Soviet republic, remains vigilant against perceived false narratives from Russia. In collaboration with the volunteer-run defense group Propastop, the ELI identified 14 categories of Russian influence operations that could affect public discourse. These categories include topics related to Crimea, the war in Ukraine, NATO's history, and Russia's annexation of Baltic states during World War II.</p><p>Researchers formulated questions for each propaganda category, which were designed to be neutral, biased with false assumptions, or intended to provoke misinformation from the LLMs. The questions were presented in English, Estonian, and Russian, and the responses were evaluated by an AI model aligned with Propastop experts to assess the LLMs' ability to counter propaganda narratives without external assistance.</p>
✓ AI-Debiased Article
45
Mainstream framing
L
R
No clear lean
✓ verified
Why this rating? · 6 signals
Signals flagged in the original
- loaded language: 'dangerous propaganda'
- loaded language: 'belligerent neighbor'
- framing: These LLMs are the best at resisting Russian propaganda
- editorializing: state governments are understandably worried
- editorializing: many Estonians are particularly alert
- vague attribution: state governments are understandably worried, many Estonians are particularly alert
Analyzed by our bias model Full breakdown ↓
Estonian Language Institute Releases Benchmark for LLMs on Propaganda Resistance
The Estonian Language Institute has launched a benchmark to assess large language models' ability to resist Russian propaganda. This initiative aims to address concerns about the potential spread of misleading narratives by evaluating LLMs across various categories of Russian influence operations.
Annotating as
No note attached
on this article.
Bias Analysis
Bias score
45/100
wirepublicmainstream
flavoredpartisanadvocacy
Inflammatory language
9/100
Bias Indicators Removed
- ✕ loaded language: 'dangerous propaganda'
- ✕ loaded language: 'belligerent neighbor'
- ✕ framing: These LLMs are the best at resisting Russian propaganda
- ✕ editorializing: state governments are understandably worried
- ✕ editorializing: many Estonians are particularly alert
- ✕ vague attribution: state governments are understandably worried, many Estonians are particularly alert
Original vs. Neutral
Original Headline
These LLMs are the best at resisting Russian propaganda
Neutral Headline
Estonian Language Institute Releases Benchmark for LLMs on Propaganda Resistance