🔬 Research Summary by Michael Feffer, a Societal Computing PhD student at Carnegie Mellon University.
[Original paper by Michael Feffer, Hoda Heidari, and Zachary C. Lipton]
Overview: Given the increased application of AI and ML systems in situations that may involve tradeoffs between undesirable outcomes (e.g., whose lives to prioritize in the face of an autonomous vehicle accident, which loan applicant(s) can receive a life-changing loan), some researchers have turned to stakeholder preference elicitation strategies. By eliciting and aggregating preferences to derive ML models that agree with stakeholders’ input, these researchers hope to create systems that navigate ethical dilemmas in accordance with stakeholders’ beliefs. However, in our paper, we demonstrate via a case study of a popular setup that preference aggregation is a nontrivial problem, and if a proposed aggregation method is not robust, applying it could yield results such as the tyranny of the majority that one may want to avoid.
Introduction
“In the case of an unavoidable accident, should an autonomous vehicle sacrifice its passengers or pedestrians?” People around the world answered this question for different hypothetical situations (e.g., young passengers versus elderly pedestrians) as part of the Moral Machine experiment. A follow-up work by Noothigattu et al. (2018) proposes a method to derive an ML model from everyone’s responses that could quickly navigate such a dilemma if encountered in the real world. The authors argue that the averaged model best reflects the participants’ values by estimating utility models from each individual and then averaging the parameters of these models together.
Our work scrutinizes this method as it has inspired numerous follow-ups. Specifically, we consider a population composed of two groups, a majority and a minority, where members of each group have the same preferences, and we analyze the properties of the averaging aggregation approach. We find that if participants are truthful, concessions to the minority group are sub-proportional in group size (e.g., if the minority is 30% of the population, decisions go their way less than 30% of the time). If participants misreport their preferences, the tyranny of the majority or unstable equilibria may result.
Key Insights
Setup
Population
We consider a population composed of two groups: a majority and a minority. Each has within-group homogeneity such that individuals within a group have the same preferences, characterized by a true preference vector for that group. During elicitation, we assume that each individual can answer an arbitrary number of pairwise comparison questions such that they can directly report a preference vector. All preference vectors and vectorized alternatives (outcomes that a participant can choose during elicitation) are unit-length to prevent undue influence by any one individual or alternative.
Game
We model the interaction between the two groups as a two-player normal-form game where each group is a player. Payoff utilities for each group are dot products between the averaged aggregate vector (a weighted sum by population proportion between the vectors reported by each group) and the group’s true preference vector. Within each group, each individual reports the same vector, but this vector is not necessarily equal to the true preference vector. (See Section 3 of our paper for more details.)
Analysis
Truthful Setting
If each group reports its true preference vectors, we find that the fraction of decisions that go the minority group’s way are sub-proportional in group size. Moreover, this fraction is also affected by the degree to which the majority and minority groups disagree on issues. For instance, if the minority group comprises 30% of the population and the two true preference vectors are nearly the same, concessions will be less than 30% but may be close to it. However, if the two preference vectors are almost diametrically opposed, concessions may be close to 0%.
Strategic Setting
If each group is strategic and misreports their preferences, a pure strategy Nash equilibrium may exist where the majority group can offset the minority group’s report such that the aggregate vector points toward the majority group’s true preference vector. This maximizes the majority group’s utility and yields a tyranny-of-the-majority result over all future decisions. We find that the existence of this equilibrium depends on both the difference in group sizes and the difference between the true preference vectors. Namely, equilibrium exists if and only if the majority group is sufficiently larger than the minority group and the two preference vectors are sufficiently close. (See Section 4 of our paper for theorems.)
Discussion
No Collusion Required
Our analysis depends on all individuals within each group reporting the same preference vector. At first glance, this might appear to happen only with intricate collusion and collaboration within a group. But, we provide a proof sketch arguing that such coordination is not required to obtain these results. The main idea is that if our results were not obtained even without collusion, this would mean that at least one individual was not utility-maximizing. This contradiction assumes that agents are rational, so our results hold even without coordination.
Aggregation Alternatives
In light of averaging aggregation being vulnerable to strategic voting, we explored other parts of the computational social choice and voting theory literature for alternative aggregation methods. Median aggregation is robust in a one-dimensional preference setting, but the concept of a median in more than one dimension is not straightforward. Existing work by El-Mhamdi et al. proves that the geometric median (vector that minimizes Euclidean distance to each vector in the sample) is not strategy-proof and the coordinate-wise (vector where each coordinate is the median of coordinates of each vector in the sample) is strategy-proof but not group strategy-proof. Another aggregation approach, randomized dictatorship (where one vector is chosen randomly from the sample as the aggregate), is strategy-proof and avoids sub-proportional concessions to the minority group. Still, it is arguably undesirable for other reasons (e.g., participants may not feel comfortable being at the mercy of someone they do not know, even if the approach is mathematically fair).
Other Considerations
Prior work by Zeckhauser states, “No voting system that relies on individuals’ self-interested balloting can guarantee a nondictatorial outcome that is Pareto-optimal…perhaps we should not ask despairingly, Where do we go from here?, but rather inquire, Have we been employing the appropriate mindset all along?” For our problem, this may mean a “one-size-fits-all” model that quickly resolves all ethical dilemmas may be untenable and the wrong approach. Other work highlights that participation requires context and locality, both of which may be lost on a system designed to scale. Moreover, resolution by voting also removes avenues for dissensus and deliberation, through which competing values can be reconciled.
Between the lines
The type of work to which we respond with this paper suggests that eliciting stakeholders’ preferences and building models in accordance with these preferences are sufficient to navigate issues in the AI fairness, accountability, transparency, and ethics (FATE) problem space. We demonstrate that even if one could perfectly elicit participant preferences, aggregation via averaging may be undesirable due to differences in population size and strategic voting. These results prompt further investigation of other assumptions made in this area, ranging from rational actor models to even the posed problem itself (e.g., outcomes beyond the two proposed to participants in the Moral Machine, such as alerting pedestrians or deploying an emergency brake, might exist in the real world). Other facets, such as power dynamics between ML practitioners and (typically non-technical) stakeholders, as well as questions of representation in the stakeholder sample and problem setup (e.g., whose values are being represented?), may also warrant further exploration.