Weapons of Math Destruction

A professional mathematician and data scientist, Cathy O’Neil refers to data models/algorithms as “opinions in code”. She writes in-depth about how mathematical data models capture the inherent biases not only of the creators of the algorithm, but also the data upon which the model is trained. For example, if previous successful job applicants happen to share common characteristics such as race and gender, then building a model on this data could result in the exclusion of all future applicants that apply who do not share this particular race and gender.

Worse, in an algorithm like this, there is no feedback, so the model could be wreaking havoc on individuals with virtually no way to determine this is the case. O’Neil argues that in order to prevent this, there must be more feedback built into the system as well as more examination of the data set and algorithm to weed out inherent bias. Since writing this book, O’Neil has founded ORCAA, which performs “Algorithm Audits” which claim to reduce bias.

O’Neil begins by explaining in layman terms what a data model is and how an algorithm is an “opinion in code”. She then outlines her journey from academic mathematician to “quant” to data scientist. After establishing her credentials, she begins to enumerate many examples of how data models are being used against the lower classes. Examples include targeting lower-class individuals by for-profit universities, recidivism models used in the justice system, the injustice of credit reports being used as a proxy of reliability for a variety of life events such as getting a job or even receiving a promotion at work.

O’Neil goes as far as to say that data models and algorithms are a threat to democracy as big tech companies like Facebook and Google control what we see and how we interact in the age of social media. Note that this book was published in 2017, well before the 2020 election where there was so much contention about polarization due to algorithmic segregation of ideas (everyone is within their own thought bubble and we don’t see enough differing opinions due to social media feed procurement).

Overall, this book is a somewhat difficult read in sections but makes valuable insights into the nature of big data and the dangers it presents to society. Watching Ms. O’Neil’s TED talk on the same subject will give a brief overview of the same subject and help determine if reading this book would be of interest.