debugging coded bias
an argument for algorithmic affirmative action
march 3rd, 2022
I originally wrote a similar version of this piece for CS 497, a Discrimination in Computing course I’m taking at Waterloo this semester. Thank you to edits from our course TA and Waterloo MMath student Liam Hebert (who’s doing really cool research into AI and social networks!) This is the first time in my degree that I’ve had the chance to write critically about algorithms. I found when researching this essay that the topic of algorithmic fairness is hotly debated, so please take this approach to be one of many “hot takes” that are out there. I’m always learning, and would be happy to chat! Enjoy.
***
After the death of George Floyd in summer 2020, an estimated 15 million Americans participated in demonstrations for the #BlackLivesMatter movement, a call for action to address anti-Black racism. [1] At the time, society was also facing a contrasting school of thought: #AllLivesMatter, whose supporters believe in a race-neutral approach to inequity, which has been shown to “mask the power inequalities that result from racial biases.” [2] Civil rights expert and professor at UC Berkeley John A. Powell describes that when “one replaces “Black Lives Matter” with “All Lives Matter,” one may be slipping into a false assumption that we are all similarly situated. We are not.” [3]
The relationship between these two movements parallels the tension in computer science between two schools of AI fairness. On one hand, there is the status quo of anti-classification algorithms, which remove protected variables from algorithms altogether. This is like when a company says “we don’t include race in our algorithms! How could we discriminate based on it?” A newer school of thought is algorithmic affirmative action, which would intentionally use protected variables to benefit disadvantaged groups. This approach has been explored extensively (and with a high degree of controversy) in college admissions processes. While anti-classification seems like the “objective” model (and reflects the pillars of a general desire for objectivity in computer science), the model has inherent flaws that raise a greater question: who defines algorithmic fairness to begin with?
Tech companies have the power to mitigate systemic discrimination in society by intentionally building algorithms that address historic inequalities perpetuated by human biases. This would require them to move past a mindset of ‘blindness’ by considering who benefits (and doesn’t) from blind systems to begin with. In order to disrupt the cycle of oppression that will inevitably be perpetuated by proxy variables in machine learning (ML) algorithms, companies must take the approach of incorporating the sensitive variables they need to achieve algorithmic affirmative action.
Consider a company that follows an anticlassification model of excluding protected characteristics from its ML model, on the basis that this keeps the user’s data safe. That company will still use proxy variables to build compromising data models about the user, leaving them equally vulnerable to data abuse. In 2018, American companies spent more than $19 billion to acquire and analyze user data [4]. At the volume that our data is already collected, including another variable in our data profiles doesn’t significantly increase our risk of exposure. In its instructions for handling sensitive data in ML datasets, Google acknowledges that “information that could allow a data engineer or malicious actor to indirectly deduce the sensitive information is also considered sensitive.” [5] Clearly, today’s seemingly blind ML datasets already contain sensitive information about the user.
Considering the scale at which companies currently aggregate data, the user is at risk the moment they share any form of personal data, whether it is considered sensitive or not. Recently, data from the LGBTQ+ dating app Grindr was used to digitally “out” Monsignor Jeffrey Burrill, a member of the U.S. Conference of Catholic Bishops. A Catholic online news commentary site called The Pillar obtained “commercially available records of app signal data to track a mobile device correlated to Monsignor Burrill that suggested he was at the same time engaged in serial and illicit sexual activity.” Without even using what is traditionally considered ‘sensitive information,’ the newspaper was able to scrape Grindr’s aggregated location data from a third party advertising service, highlighting the specific coordinates of Monsignor Burrill’s church and his private cabin to expose his usage of Grindr. [6] This exemplifies how individuals are already subject to data abuse without the company’s dataset containing their protected characteristics to begin with. Google notes that an aggregate database is already enough to hack encrypted data through frequency analysis attacks, [5] a method of deducing sensitive information through finding frequencies of letters in a text. Today’s datasets are not designed to protect consumers to begin with, which means “blind systems” are already failing to protect us. Including sensitive information would not not tip the scales of consumer privacy towards disrepair: rather, this practice has the potential to be leveraged for positive ethical outcomes.
Anticlassification algorithms are not just vulnerable to data misuse — they also perpetuate the cycle of discrimination that could be effectively adressed by algorithmic affirmative action. Researchers at Georgetown Law found that “achieving the anti classificatory ideal of colorblindness is impossible in the machine-learning context, because we cannot create truly colorblind algorithms. Machines are too effective in identifying proxies.” [7] Although it would be ideal to avoid sensitive variables and the data risks they would introduce altogether, the resulting algorithm would still perpetuate biases; thus, anti-classification fails to mitigate the risk of biased algorithms altogether. In contrast, using sensitive variables has been shown to reduce discrimination. In their study on anti-discrimination laws in fintech, Kelley et al. found that “regimes that prohibit the use of gender (like those in the United States) substantially increase discrimination…. [but] down-sampling the training data to rebalance gender, gender-aware hyperparameter selection, and up-sampling the training data to rebalance gender, all reduce discrimination.” [8] These findings reveal that including protected variables in ML algorithms would have a positive impact on society, while failing to do so ultimately causes further harm to disadvantaged groups. While users may consent to sharing their data with a company in exchange for personalized recommendations, they do not deserve or expect to be unfairly treated by an algorithm. Algorithmic affirmative action would compensate for the inherent biases that disadvantaged groups are inevitably subject to when interacting with ML technologies.
The risk of biased ML models compounds when one considers society’s tendency to believe that computers are more impartial than human decision makers, especially in settings that perpetuate historical inequity. Consider the criminal justice system, where risk assessment instruments (RAI) are used by courts in more than 46 states to determine the risk of convicted criminals. [9] O’Neil notes that one such tool, the LSI-R questionnaire for recidivism programs, is inherently discriminatory, because “if early ‘involvement’ with the police signals recidivism, poor people and racial minorities look far riskier.” [10] Without correcting historic inequities embedded in algorithms, companies end up perpetuating a cycle of oppression, at a much larger scale.
A Columbia University study that found that “using risk-assessment tools, even for progressive ends, is going to significantly aggravate the already unacceptable racial disparities in our criminal justice system.” [11] As society moves towards an increasing reliance on algorithms that we are inclined to believe are impartial, it is critical to ensure that they are fair to begin with. A Propublica investigation found that “in 19 different risk development methodologies used in the United States, validity is only examined in one or two studies…completed by the same people who developed the instrument.” [9] When a judge makes a decision based on an RAI, they inherently believe that it is a more impartial decision-maker than a human. This can lead them to make stronger sentencing decisions in the RAI setting, or to feel more justified in pursuing a sentencing that might otherwise be considered discriminatory. Thus, it becomes critical to make reliable tools for large-scale recommendations.
At the scale that machine learning algorithms are being adopted, the oppression that could result from failing to regulate biased algorithms far outweighs the risk of doing nothing. Users today already experience data privacy issues in the anti-classification model, which also perpetuates oppression. As described by former US President Lyndon B. Johnson in his 1965 commencement address at Howard University, “[y]ou do not take a person who, for years, has been hobbled by chains and liberate him, bring him up to the starting line of a race and then say, ‘You are free to compete with all the others,’ and still justly believe that you have been completely fair.” [12] Ultimately, algorithmic affirmative action would introduce a low amount of risk in comparison to its long-term potential to improve the impacts of algorithmic decision-making as a whole.
***
Thank you as always for making it to the end of this piece. I’m new to the field of algorithmic bias, so please don’t hesitate to share your input or feedback on the technical or societal feasibility of my suggested approach. I’m currently exploring research topics in the field and just getting into learning about AI — so any and all suggestions are appreciated! Reach out to me on Twitter or LinkedIn.
references
[1] Buchanan, Larry, Quoctrung Bui, and Jugal K. Patel. “Black Lives Matter May Be the Largest Movement in U.S. History.” The New York Times. The New York Times, July 3, 2020. https://www.nytimes.com/interactive/2020/07/03/us/george-floyd-protests-crowd-size.html.
[2] Gallagher, Ryan J, Andrew J Reagan, Christopher M Danforth, and Peter Sheridan Dodds. “Divergent Discourse between Protests and Counter-Protests: #BlackLivesMatter and #Alllivesmatter.” PloS one. Public Library of Science, April 18, 2018. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5906010/.
[3] Powell, John A. “When We Fully Claim Black Lives Matter, We Move Closer to All Lives Matter.” When we fully claim Black Lives Matter, we move closer to All Lives Matter | Othering & Belonging Institute. Othering & Belong Institute, July 15, 2015. https://belonging.berkeley.edu/when-we-fully-claim-black-lives-matter-we-move-closer-all-lives -matter.
[4] Matsakis, Louise. “The Wired Guide to Your Personal Data (and Who Is Using It).” Wired. Conde Nast, February 15, 2019. https://www.wired.com/story/wired-guide-personal-data-collection/.
[5] “Considerations for Sensitive Data within Machine Learning Datasets | Cloud Architecture Center | Google Cloud.” Google. Google. Accessed January 25, 2022. https://cloud.google.com/architecture/sensitive-data-and-ml-datasets.
[6] Zac Davis. “Tabloids, Scandal and Spying: The U.S. Catholic Church Has Hit a New, Dangerous Low Point.” America Magazine, July 22, 2021. https://www.americamagazine.org/faith/2021/07/22/catholic-church-spying-pillar-burrill-grindr-u sccb-privacy-241106.
[7] Jason, Bent R. “Is Algorithmic Affirmative Action Legal?” Georgetown Law Journal, June 4, 2020.https://www.law.georgetown.edu/georgetown-law-journal/wp-content/uploads/sites/26/202 0/04/Is-Algorithmic-Affirmative-Action-Legal.pdf.
[8] Kelley, Stephanie, Anton Ovchinnikov, David R. Hardoon, and Adrienne Heinrich. “Anti-Discrimination Laws, AI, and Gender Bias: A Case Study in Non-Mortgage Fintech Lending.” SSRN, November 23, 2020. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3719577.
[9] Julia Angwin, Jeff Larson. “Machine Bias.” ProPublica, May 23, 2016. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.
[10] “Weapons of Math Destruction.” The American Scholar, September 9, 2016. https://theamericanscholar.org/weapons-of-math-destruction/.
[11] Harcourt, Bernard E. “Risk as a Proxy for Race.” SSRN, September 16, 2010. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1677654.
[12] “Commencement Address at Howard University: ‘to Fulfill These Rights.” The American Presidency Project, June 4, 1965. https://www.presidency.ucsb.edu/documents/commencement-address-howard-university-fulfill-t hese-rights.
