• Skip to main content
  • Skip to primary sidebar
  • Skip to footer
  • Core Principles of Responsible AI
    • Accountability
    • Fairness
    • Privacy
    • Safety and Security
    • Sustainability
    • Transparency
  • Special Topics
    • AI in Industry
    • Ethical Implications
    • Human-Centered Design
    • Regulatory Landscape
    • Technical Methods
  • Living Dictionary
  • State of AI Ethics
  • AI Ethics Brief
  • 🇫🇷
Montreal AI Ethics Institute

Montreal AI Ethics Institute

Democratizing AI ethics literacy

Attacking Fake News Detectors via Manipulating News Social Engagement

July 1, 2023

🔬 Research Summary by Haoran Wang, a doctoral student at Illinois Institute of Technology, with an interest in building trustworthy AI systems to verify natural language information.

[Original paper by Haoran Wang, Yingtong Dou, Canyu Chen, Lichao Sun, Philip S. Yu, and Kai Shu]


Overview: Although recent works have exploited the vulnerability of text-based misinformation detectors, the robustness of social-context-based detectors has not yet been extensively studied. In light of this, we propose a multi-agent reinforcement learning framework to probe the robustness of existing social-context-based detectors. We offer valuable insights to enhance misinformation detectors’ reliability and trustworthiness by evaluating our method on two real-world misinformation datasets.    


Introduction

The popularity of social media platforms as sources of news consumption, particularly among the younger generation, has led to a significant increase in misinformation. To address this issue, several text- and social context-based fake news detectors have been proposed. However, recent research has started to focus on uncovering the vulnerabilities of these detectors. In this paper, we introduce an adversarial attack framework designed to assess the robustness of Graph Neural Network (GNN)-based fake news detectors. Our approach utilizes a multi-agent reinforcement learning (MARL) framework to simulate the adversarial behavior exhibited by fraudsters on social media platforms. Real-world evidence suggests that fraudsters coordinate their actions to share different news articles, aiming to evade detection by fake news detectors. To capture this behavior, we model our MARL framework as a Markov Game, incorporating bot, cyborg, and crowd worker agents with distinct costs, budgets, and influence levels. Deep Q-learning techniques search for the optimal policy that maximizes rewards in this adversarial setting. Through extensive experimentation on two real-world datasets of fake news propagation, we demonstrate the effectiveness of our proposed framework in sabotaging the performance of GNN-based fake news detectors. The results highlight the vulnerability of these detectors when faced with coordinated adversarial attacks. By presenting this adversarial attack framework and showcasing its impact on GNN-based fake news detectors, our paper provides valuable insights for future research in the field of fake news detection. It underscores the need to develop more robust and resilient detection mechanisms to counter the sophisticated tactics employed by fraudsters in spreading misinformation.

Key Insights

Probing the Robustness of Social Misinformation Detectors via Manipulating News Social Engagement

We draw inspiration from previous research on GNN robustness to analyze the robustness of social-engagement-based misinformation detectors. In our approach, we propose attacking GNN-based detectors by simulating the adversarial behaviors exhibited by fraudsters in real-world misinformation campaigns. 

However, this simulation presents three significant challenges that need to be addressed. The first challenge concerns the evasion tactics employed by malicious actors to promote fake news on social media. These actors typically manipulate controlled user accounts to share various social posts while attempting to evade detection. However, most existing GNN adversarial attack methods assume the ability to perturb all nodes and edges, which is impractical in this scenario. We must develop strategies that account for the limited control over nodes and edges to devise effective attacks. The second challenge stems from many deployed GNN-based fake news detectors being gray-box models with diverse architectures tailored to the heterogeneous user-post graph. As a result, the gradient-based optimization methods used in previous works cannot be directly applied to devise attacks. Alternative approaches must be explored to tackle this challenge and overcome the limitations imposed by the model architecture. The third challenge arises from different types of coordinated malicious actors with varying capabilities, budgets, and risk appetites in real-world misinformation campaigns. Key opinion leaders, for example, possess stronger influence but require more resources to cultivate compared to social bots. It is important to account for this diversity and develop attack strategies that can adapt to different types of malicious actors. 

To address these challenges, we propose a dedicated Multiagent Reinforcement Learning (MARL) framework, distinct from previous GNN robustness research. This MARL framework is designed to simulate the real-world behavior of fraudsters sharing different posts. We utilize deep reinforcement learning techniques to flip the classification results of target news nodes by modifying the connections of users who have shared the post. Our MARL framework is formulated as a Markov Game, where multiple agents work together to flip the classification results collectively. Through extensive experimentation, we have made several observations from our results.

Finding 1: Existing social-context-based detectors are vulnerable to adversarial attacks

Our experiment results on two real-world misinformation datasets show that the proposed method (MARL) is effective on fake news in both datasets. Among the popular GNN-based misinformation detectors that we tested, Graph Convolutional Neural Network (GCN) is the most vulnerable, where MARL can achieve a 92% success rate in attacking fake news in the Politifact dataset. This is likely due to the low breakdown point of GCN’s weighted mean aggregation method.

Finding 2: Pay attention to “seemingly good” users when defending against social attacks

We divide the users into “good” and “bad” groups based on the historical number of real news they have tweeted. Our experiment results show that the seemingly “good” users have more influence on flipping the classification label of fake news during attacks. In reality, social media companies should pay equal attention to all users when building more robust detectors since the seemingly “good” users exhibit greater influence during attacks if they become compromised.

Finding 3: More “viral” news is inherently more robust than the news that receives little or no attention

We categorize news posts based on their degrees, representing how often users retweet them. Our experiment results show that news with higher degrees, meaning the more “viral” news is harder to be attacked than news with lower degrees. This suggests news that receives undersized attention could be easy targets for attackers.

Between the lines

Our experimental findings demonstrate that Multi-Agent Reinforcement Learning (MARL) significantly enhances the overall performance of attacks when compared to our baseline methods, particularly showing its effectiveness against GCN-based detectors. While these results are promising, it is important to acknowledge two major limitations in this paper:

1) This work only employs a simple heuristic to select users for action aggregation. 

2) The search space of the Q network is considerably large and results in a high computational cost on larger datasets like Gossipcop. 

Therefore, there are several interesting directions that need further investigation. The first is to automate the selection of optimal agents for action aggregation. The second one is effectively reducing the deep Q network’s search space. Finally, we used a vanilla MARL framework in this paper. It would be interesting to explore a more complex MARL framework for this task.

Want quick summaries of the latest research & reporting in AI ethics delivered to your inbox? Subscribe to the AI Ethics Brief. We publish bi-weekly.

Primary Sidebar

🔍 SEARCH

Spotlight

AI Policy Corner: Frontier AI Safety Commitments, AI Seoul Summit 2024

AI Policy Corner: The Colorado State Deepfakes Act

Special Edition: Honouring the Legacy of Abhishek Gupta (1992–2024)

AI Policy Corner: The Turkish Artificial Intelligence Law Proposal

From Funding Crisis to AI Misuse: Critical Digital Rights Challenges from RightsCon 2025

related posts

  • Value-based Fast and Slow AI Nudging

    Value-based Fast and Slow AI Nudging

  • Promoting Bright Patterns

    Promoting Bright Patterns

  • Going public: the role of public participation approaches in commercial AI labs

    Going public: the role of public participation approaches in commercial AI labs

  • Defending Against Authorship Identification Attacks

    Defending Against Authorship Identification Attacks

  • The Bias of Harmful Label Associations in Vision-Language Models

    The Bias of Harmful Label Associations in Vision-Language Models

  • The GPTJudge: Justice in a Generative AI World

    The GPTJudge: Justice in a Generative AI World

  • The Ethics Owners — A New Model of Organizational Responsibility in Data-Driven Technology Companies...

    The Ethics Owners — A New Model of Organizational Responsibility in Data-Driven Technology Companies...

  • Top 5 takeaways from our conversation with I2AI on AI in different national contexts

    Top 5 takeaways from our conversation with I2AI on AI in different national contexts

  • Customization is Key: Four Characteristics of Textual Affordances for Accessible Data Visualizatio...

    "Customization is Key": Four Characteristics of Textual Affordances for Accessible Data Visualizatio...

  • Analysis of the “Artificial Intelligence governance principles: towards ethical and trustworthy arti...

    Analysis of the “Artificial Intelligence governance principles: towards ethical and trustworthy arti...

Partners

  •  
    U.S. Artificial Intelligence Safety Institute Consortium (AISIC) at NIST

  • Partnership on AI

  • The LF AI & Data Foundation

  • The AI Alliance

Footer

Categories


• Blog
• Research Summaries
• Columns
• Core Principles of Responsible AI
• Special Topics

Signature Content


• The State Of AI Ethics

• The Living Dictionary

• The AI Ethics Brief

Learn More


• About

• Open Access Policy

• Contributions Policy

• Editorial Stance on AI Tools

• Press

• Donate

• Contact

The AI Ethics Brief (bi-weekly newsletter)

About Us


Founded in 2018, the Montreal AI Ethics Institute (MAIEI) is an international non-profit organization equipping citizens concerned about artificial intelligence and its impact on society to take action.


Archive

  • © MONTREAL AI ETHICS INSTITUTE. All rights reserved 2024.
  • This work is licensed under a Creative Commons Attribution 4.0 International License.
  • Learn more about our open access policy here.
  • Creative Commons License

    Save hours of work and stay on top of Responsible AI research and reporting with our bi-weekly email newsletter.