• Skip to main content
  • Skip to primary sidebar
  • Skip to footer
  • Core Principles of Responsible AI
    • Accountability
    • Fairness
    • Privacy
    • Safety and Security
    • Sustainability
    • Transparency
  • Special Topics
    • AI in Industry
    • Ethical Implications
    • Human-Centered Design
    • Regulatory Landscape
    • Technical Methods
  • Living Dictionary
  • State of AI Ethics
  • AI Ethics Brief
  • šŸ‡«šŸ‡·
Montreal AI Ethics Institute

Montreal AI Ethics Institute

Democratizing AI ethics literacy

Breaking Your Neural Network with Adversarial Examples

November 18, 2020

Written by Kenny Song (@helloksong). Co-founder of Citadel AI.


Fundamentally, a machine learning model is just a software program: it takes an input, steps through a series of computations, and produces an output. In fact, all software has bugs and vulnerabilities, and machine learning is no exception.

One prominent bug – and security vulnerability – in current machine learning systems is the existence of adversarial examples. An attacker can carefully craft an input to the system to make it predict anything the attacker wants.

For example, by tweaking a few pixels in the source image, we can make a neural network think this ā€œStopā€ sign is a ā€œ120 km/hrā€ sign, with 99.9% confidence.

Try this demoĀ in your browser!

Beyond misclassifying street signs, attackers could use this to:

  • Impersonate others in facial recognition systems
  • Bypass content moderation and spam filters in social networks
  • Inject adversarial bytes into malware to bypass antivirus systems

This problem is well-known in the academic community, with thousands of published papers. Yet few practitioners invest resources to defend their ML systems against these attacks. This is partially a visibility problem – most of this knowledge is locked inside research literature.

To increase awareness of these risks, I created adversarial.js, a library of adversarial attacks in JavaScript. It has an interactive demo that generates adversarial examples in your browser. No installation, no manual, just open the webpage and start playing.

Hopefully, by showcasing these attacks in an easy-to-understand way, we can help others discover this failure mode of machine learning. In particular, I hope that it motivates practitioners and real-world system owners to consider these risks & defenses.

What are the defenses? There are several proposals, such as adversarial training or admission control. Some are implemented in open-source libraries including CleverHans, Foolbox, or ART. However, no method is universal and many have proven ineffective, so work with an expert to invest in your defenses appropriately.

To learn more about adversarial examples, check out the library FAQ, or get in touch with the author. 

Want quick summaries of the latest research & reporting in AI ethics delivered to your inbox? Subscribe to the AI Ethics Brief. We publish bi-weekly.

Primary Sidebar

šŸ” SEARCH

Spotlight

AI Policy Corner: Texas and New York: Comparing U.S. State-Level AI Laws

What is Sovereign Artificial Intelligence?

AI Policy Corner: The Kenya National AI Strategy

AI Policy Corner: New York City Local Law 144

Canada’s Minister of AI and Digital Innovation is a Historic First. Here’s What We Recommend.

related posts

  • System Cards for AI-Based Decision-Making for Public Policy

    System Cards for AI-Based Decision-Making for Public Policy

  • Reduced, Reused, and Recycled: The Life of a Benchmark in Machine Learning Research

    Reduced, Reused, and Recycled: The Life of a Benchmark in Machine Learning Research

  • Universal and Transferable Adversarial Attacks on Aligned Language Models

    Universal and Transferable Adversarial Attacks on Aligned Language Models

  • Ethics of AI in Education: Towards a Community-wide Framework

    Ethics of AI in Education: Towards a Community-wide Framework

  • Study of Competition Issues in Data-Driven Markets in Canada

    Study of Competition Issues in Data-Driven Markets in Canada

  • The Impact of Recommendation Systems on Opinion Dynamics: Microscopic versus Macroscopic Effects

    The Impact of Recommendation Systems on Opinion Dynamics: Microscopic versus Macroscopic Effects

  • Bridging the Gap: The Case For an ā€˜Incompletely Theorized Agreement’ on AI Policy (Research Summary)

    Bridging the Gap: The Case For an ā€˜Incompletely Theorized Agreement’ on AI Policy (Research Summary)

  • The Role of Arts in Shaping AI Ethics

    The Role of Arts in Shaping AI Ethics

  • On Measuring Fairness in Generative Modelling (NeurIPS 2023)

    On Measuring Fairness in Generative Modelling (NeurIPS 2023)

  • Exchanging Lessons Between Algorithmic Fairness and Domain Generalization (Research Summary)

    Exchanging Lessons Between Algorithmic Fairness and Domain Generalization (Research Summary)

Partners

  • Ā 
    U.S. Artificial Intelligence Safety Institute Consortium (AISIC) at NIST

  • Partnership on AI

  • The LF AI & Data Foundation

  • The AI Alliance

Footer

Categories


• Blog
• Research Summaries
• Columns
• Core Principles of Responsible AI
• Special Topics

Signature Content


• The State Of AI Ethics

• The Living Dictionary

• The AI Ethics Brief

Learn More


• About

• Open Access Policy

• Contributions Policy

• Editorial Stance on AI Tools

• Press

• Donate

• Contact

The AI Ethics Brief (bi-weekly newsletter)

About Us


Founded in 2018, the Montreal AI Ethics Institute (MAIEI) is an international non-profit organization equipping citizens concerned about artificial intelligence and its impact on society to take action.


Archive

  • Ā© MONTREAL AI ETHICS INSTITUTE. All rights reserved 2024.
  • This work is licensed under a Creative Commons Attribution 4.0 International License.
  • Learn more about our open access policy here.
  • Creative Commons License

    Save hours of work and stay on top of Responsible AI research and reporting with our bi-weekly email newsletter.