Summary contributed by Abhishek Gupta (@atg_abhishek), founder of the Montreal AI Ethics Institute.
*Authors of full paper & link at the bottom
Mini-summary: The paper presents a comparative analysis of biases as they arise in humans and machines with an interesting set of examples to boot. Specifically, taking a lens of cognitive biases in humans as a way of better understanding how biases arise in machines and how they might be combatted is essential as AI-enabled systems become more widely deployed. What is particularly interesting about the paper is also how the author takes a simple k-nearest neighbor (kNN) approach to showcase how biases arise in practice in algorithmic systems. Also, tackling the hard problem of proxy variables is done through the use of illustrative examples that eschew the overused example of zip codes as a proxy for race. Taking multiple different iterations on the same running example helps to elucidate how biases can crop up in novel ways even when we have made genuine efforts to remove sensitive and protected attributes and made other attempts to prevent biases from seeping into the dataset. Finally, the paper concludes with a call to action for people to closely examine both human and machine biases in conjunction to create approaches that can more holistically address the issues of harm for people who are disproportionately impacted by these systems.
Full summary:
The paper investigates how there are similarities between human cognitive biases and algorithmic biases and how this might provide some clues in helping us combat these ethical issues in the systems. One of the things highlighted in the paper is the proxy problem, where there are other attributes in the dataset that strongly correlate with sensitive or protected information and even upon removing them, the biases reflected in the proxies are still present and lead to unjust outcomes.
The paper makes the argument that biases exist everywhere around us where there is induction putting them into the following categories: “biases can be cognitive, algorithmic, social, non-social, epistemically reliable, epistemically unreliable, morally reprehensible, or morally innocuous.” The way our thoughts are captured and represented in online datasets form the basis for the biases that get represented in the systems, for example, people searching for things like “Is my son gifted?” more often than “Is my daughter gifted?” gets captured in language models that use training data from such searches.
Statistical regularities when paired with the utilization of even neutral technologies can manifest biases since ultimately they are a way of surfacing both implicit and explicit patterns from the data. The paper utilizes a toy use-case with the k-nearest neighbor algorithm to show how even innocuous conceptions of a use-case can quickly create biases that are wholly unintended by the creators of the system.
One of the places where the paper shines is in giving quite accessible examples that illustrate the problems when people make inferential jumps, be they intended or unintended, that lead to the codification of biases in algorithmic systems. Take the case for the common societal perceptions that the elderly are bad with technology; while it may be true that it is the case for some subset of that demographic, generalizing that assumption to others without verification and other information can lead to patronizing behaviour towards them that is unwarranted. Many other such instances are manifested through the errors that might seep in from those who are labelling the data when they might believe in some of stereotypes and allocate a label that the elderly are bad with technology even when there isn’t evidence to support that. Perhaps, this then raises the question on whether there are unproblematic datasets and if not then by the adage of garbage-in, garbage-out, we’re bound to a fate where such systems will continue to produce problematic outputs. When thinking about implicit bias, in the case of the algorithmic system, one can argue that this is a case of codification, whether through errors or intentionally, and hence the system behaves quite in the same way as humans do with their own cognitive biases.
As mentioned in this paper, given that biases are inevitable when considering induction, achieving a normatively neutral notion of bias where we regard bias as problematic would then make induction impossible and limit the usefulness of some of the more advanced techniques that we employ in predictive systems. The paper provides an interesting example on proxy variables that eschews the common race proxy codified in the zip codes in the US.Specifically, the author mentions a case of how watchers of Fox News are somehow perceived to be bad with technology, building on the previous example, not realizing that even though the person doesn’t have any biases against those who watch Fox News, or the elderly, because the viewership of Fox News skews towards the elderly, there is a risk of transferring and encoding that in the judgements made about them in terms of their ability to use technology.
Another example demonstrates how biases can arise even when the sensitive or protected attributes are stripped from consideration; looking at historical presidential candidates in the US and mapping them onto a 2D space with skin tone on one axis and the propensity to dress in a feminine manner on another, even without codifying the sensitive attributes into the training, someone who doesn’t fit the past patterns will be deemed to be “unfit” to POTUS. To counteract this problem, the author suggests finding counterexamples that beat the expressed stereotypes and embed them into the dataset allowing for a more holistic representation within the data which can then be picked up by the system. While there are large systemic issues that need to be addressed when trying to find such counterexamples, it at the very least bolsters the case for why representation matters.
Yet another example covered by the author talks about hiring in academia and how one might rely on publication records as an indication of a candidate who is worthy. In the field of philosophy, what has been found is that women represent a much lower percentage of published authors than their actual presence in the field. Now, one might argue to remove this attribute from consideration given that it has a negative impact on women in the hiring process but then we run into the issue of being able to find another attribute that can help us make decisions when hiring faculty and it is not clear if there is something else that can substitute for it.
Ultimately, the paper concludes that the study of biases in both humans and machines will lead to a better understanding in both domains and at least make us realize that purely algorithmic interventions to address machine bias will be futile efforts.
Original paper by Gabbrielle M Johnson: http://philsci-archive.pitt.edu/17169/1/Algorithmic%20Bias.pdf