• Skip to main content
  • Skip to secondary menu
  • Skip to primary sidebar
  • Skip to footer
Montreal AI Ethics Institute

Montreal AI Ethics Institute

Democratizing AI ethics literacy

  • Articles
    • Public Policy
    • Privacy & Security
    • Human Rights
      • Ethics
      • JEDI (Justice, Equity, Diversity, Inclusion
    • Climate
    • Design
      • Emerging Technology
    • Application & Adoption
      • Health
      • Education
      • Government
        • Military
        • Public Works
      • Labour
    • Arts & Culture
      • Film & TV
      • Music
      • Pop Culture
      • Digital Art
  • Columns
    • AI Policy Corner
    • Recess
  • The AI Ethics Brief
  • AI Literacy
    • Research Summaries
    • AI Ethics Living Dictionary
    • Learning Community
  • The State of AI Ethics Report
    • Volume 6 (February 2022)
    • Volume 5 (July 2021)
    • Volume 4 (April 2021)
    • Volume 3 (Jan 2021)
    • Volume 2 (Oct 2020)
    • Volume 1 (June 2020)
  • About
    • Our Contributions Policy
    • Our Open Access Policy
    • Contact
    • Donate

Attacking Fake News Detectors via Manipulating News Social Engagement

July 1, 2023

🔬 Research Summary by Haoran Wang, a doctoral student at Illinois Institute of Technology, with an interest in building trustworthy AI systems to verify natural language information.

[Original paper by Haoran Wang, Yingtong Dou, Canyu Chen, Lichao Sun, Philip S. Yu, and Kai Shu]


Overview: Although recent works have exploited the vulnerability of text-based misinformation detectors, the robustness of social-context-based detectors has not yet been extensively studied. In light of this, we propose a multi-agent reinforcement learning framework to probe the robustness of existing social-context-based detectors. We offer valuable insights to enhance misinformation detectors’ reliability and trustworthiness by evaluating our method on two real-world misinformation datasets.    


Introduction

The popularity of social media platforms as sources of news consumption, particularly among the younger generation, has led to a significant increase in misinformation. To address this issue, several text- and social context-based fake news detectors have been proposed. However, recent research has started to focus on uncovering the vulnerabilities of these detectors. In this paper, we introduce an adversarial attack framework designed to assess the robustness of Graph Neural Network (GNN)-based fake news detectors. Our approach utilizes a multi-agent reinforcement learning (MARL) framework to simulate the adversarial behavior exhibited by fraudsters on social media platforms. Real-world evidence suggests that fraudsters coordinate their actions to share different news articles, aiming to evade detection by fake news detectors. To capture this behavior, we model our MARL framework as a Markov Game, incorporating bot, cyborg, and crowd worker agents with distinct costs, budgets, and influence levels. Deep Q-learning techniques search for the optimal policy that maximizes rewards in this adversarial setting. Through extensive experimentation on two real-world datasets of fake news propagation, we demonstrate the effectiveness of our proposed framework in sabotaging the performance of GNN-based fake news detectors. The results highlight the vulnerability of these detectors when faced with coordinated adversarial attacks. By presenting this adversarial attack framework and showcasing its impact on GNN-based fake news detectors, our paper provides valuable insights for future research in the field of fake news detection. It underscores the need to develop more robust and resilient detection mechanisms to counter the sophisticated tactics employed by fraudsters in spreading misinformation.

Key Insights

Probing the Robustness of Social Misinformation Detectors via Manipulating News Social Engagement

We draw inspiration from previous research on GNN robustness to analyze the robustness of social-engagement-based misinformation detectors. In our approach, we propose attacking GNN-based detectors by simulating the adversarial behaviors exhibited by fraudsters in real-world misinformation campaigns. 

However, this simulation presents three significant challenges that need to be addressed. The first challenge concerns the evasion tactics employed by malicious actors to promote fake news on social media. These actors typically manipulate controlled user accounts to share various social posts while attempting to evade detection. However, most existing GNN adversarial attack methods assume the ability to perturb all nodes and edges, which is impractical in this scenario. We must develop strategies that account for the limited control over nodes and edges to devise effective attacks. The second challenge stems from many deployed GNN-based fake news detectors being gray-box models with diverse architectures tailored to the heterogeneous user-post graph. As a result, the gradient-based optimization methods used in previous works cannot be directly applied to devise attacks. Alternative approaches must be explored to tackle this challenge and overcome the limitations imposed by the model architecture. The third challenge arises from different types of coordinated malicious actors with varying capabilities, budgets, and risk appetites in real-world misinformation campaigns. Key opinion leaders, for example, possess stronger influence but require more resources to cultivate compared to social bots. It is important to account for this diversity and develop attack strategies that can adapt to different types of malicious actors. 

To address these challenges, we propose a dedicated Multiagent Reinforcement Learning (MARL) framework, distinct from previous GNN robustness research. This MARL framework is designed to simulate the real-world behavior of fraudsters sharing different posts. We utilize deep reinforcement learning techniques to flip the classification results of target news nodes by modifying the connections of users who have shared the post. Our MARL framework is formulated as a Markov Game, where multiple agents work together to flip the classification results collectively. Through extensive experimentation, we have made several observations from our results.

Finding 1: Existing social-context-based detectors are vulnerable to adversarial attacks

Our experiment results on two real-world misinformation datasets show that the proposed method (MARL) is effective on fake news in both datasets. Among the popular GNN-based misinformation detectors that we tested, Graph Convolutional Neural Network (GCN) is the most vulnerable, where MARL can achieve a 92% success rate in attacking fake news in the Politifact dataset. This is likely due to the low breakdown point of GCN’s weighted mean aggregation method.

Finding 2: Pay attention to “seemingly good” users when defending against social attacks

We divide the users into “good” and “bad” groups based on the historical number of real news they have tweeted. Our experiment results show that the seemingly “good” users have more influence on flipping the classification label of fake news during attacks. In reality, social media companies should pay equal attention to all users when building more robust detectors since the seemingly “good” users exhibit greater influence during attacks if they become compromised.

Finding 3: More “viral” news is inherently more robust than the news that receives little or no attention

We categorize news posts based on their degrees, representing how often users retweet them. Our experiment results show that news with higher degrees, meaning the more “viral” news is harder to be attacked than news with lower degrees. This suggests news that receives undersized attention could be easy targets for attackers.

Between the lines

Our experimental findings demonstrate that Multi-Agent Reinforcement Learning (MARL) significantly enhances the overall performance of attacks when compared to our baseline methods, particularly showing its effectiveness against GCN-based detectors. While these results are promising, it is important to acknowledge two major limitations in this paper:

1) This work only employs a simple heuristic to select users for action aggregation. 

2) The search space of the Q network is considerably large and results in a high computational cost on larger datasets like Gossipcop. 

Therefore, there are several interesting directions that need further investigation. The first is to automate the selection of optimal agents for action aggregation. The second one is effectively reducing the deep Q network’s search space. Finally, we used a vanilla MARL framework in this paper. It would be interesting to explore a more complex MARL framework for this task.

Want quick summaries of the latest research & reporting in AI ethics delivered to your inbox? Subscribe to the AI Ethics Brief. We publish bi-weekly.

Primary Sidebar

🔍 SEARCH

Spotlight

ALL IN Conference 2025: Four Key Takeaways from Montreal

Beyond Dependency: The Hidden Risk of Social Comparison in Chatbot Companionship

AI Policy Corner: Restriction vs. Regulation: Comparing State Approaches to AI Mental Health Legislation

Beyond Consultation: Building Inclusive AI Governance for Canada’s Democratic Future

AI Policy Corner: U.S. Executive Order on Advancing AI Education for American Youth

related posts

  • 10 Takeaways from the State of AI Ethics in Canada & Spain

    10 Takeaways from the State of AI Ethics in Canada & Spain

  • Research summary: Beyond Near- and Long-Term: Towards a Clearer Account of Research Priorities in AI...

    Research summary: Beyond Near- and Long-Term: Towards a Clearer Account of Research Priorities in AI...

  • Declaration on the ethics of brain-computer interfaces and augment intelligence

    Declaration on the ethics of brain-computer interfaces and augment intelligence

  • Demographic-Reliant Algorithmic Fairness: Characterizing the Risks of Demographic Data Collection an...

    Demographic-Reliant Algorithmic Fairness: Characterizing the Risks of Demographic Data Collection an...

  • Best humans still outperform artificial intelligence in a creative divergent thinking task

    Best humans still outperform artificial intelligence in a creative divergent thinking task

  • Research summary: Robot Rights? Let’s Talk about Human Welfare instead

    Research summary: Robot Rights? Let’s Talk about Human Welfare instead

  • Democratising AI: Multiple Meanings, Goals, and Methods

    Democratising AI: Multiple Meanings, Goals, and Methods

  • Theorizing Femininity in AI: a Framework for Undoing Technology’s Gender Troubles (Research Summary)

    Theorizing Femininity in AI: a Framework for Undoing Technology’s Gender Troubles (Research Summary)

  • Unprofessional Peer Reviews Disproportionately Harm Underrepresented Groups in STEM (Research Summar...

    Unprofessional Peer Reviews Disproportionately Harm Underrepresented Groups in STEM (Research Summar...

  • Discover Weekly: How the Music Platform Spotify Collects and Uses Your Data

    Discover Weekly: How the Music Platform Spotify Collects and Uses Your Data

Partners

  •  
    U.S. Artificial Intelligence Safety Institute Consortium (AISIC) at NIST

  • Partnership on AI

  • The LF AI & Data Foundation

  • The AI Alliance

Footer


Articles

Columns

AI Literacy

The State of AI Ethics Report


 

About Us


Founded in 2018, the Montreal AI Ethics Institute (MAIEI) is an international non-profit organization equipping citizens concerned about artificial intelligence and its impact on society to take action.

Contact

Donate


  • © 2025 MONTREAL AI ETHICS INSTITUTE.
  • This work is licensed under a Creative Commons Attribution 4.0 International License.
  • Learn more about our open access policy here.
  • Creative Commons License

    Save hours of work and stay on top of Responsible AI research and reporting with our bi-weekly email newsletter.