I think you’d do well to read Cathy O’Neils Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. I saw her read at the Miami Book Fair several months ago and immediately bought a copy. I even got her to sign it which is kinda cool ;)
Cathy’s big idea is that we’re absolutely surrounded by algorithms that inform big decision making. There are lots of good algorithms that help us. Sadly, there are lots of insidiously, dangerous, bad algorithms that do serious damage, and they are lurking all about disguised as good algorithms.
One aspect of a good algorithm is some kind of feedback and correctional system. Early on Cathy points to some advertising algorithms as an example of a healthy algorithm. For example, if an algorithm is in place to recommend a product you should buy, and it does a terrible job at that, it will be tweaked until fixed, thereby correcting what is has set out to do. Moneyball-style algorithms are the same. The data is open. Baseball team managers use algorithms to help recruit for their team and manage how they play. If it isn’t working, it will be tweaked until it does.
A bad algorithm might lack a feedback loop. One of her strongest examples is in the algorithms that rate teachers. There is plenty of evidence that these algorithms are often wrong, ousting teachers that definitely should not have been. And not in a “they tested badly, but have a heart of gold” way, in a “the algorithm was actually just wrong” way. What makes something like this a “weapon of math destruction” (WMD) then, is the fact that it affects a lot of people, screws up, and there is no correction mechanism. There are lots of interesting criteria, though. I’ll let you read more about it.
There is an awful lot of considerations and nuance here, and I think Cathy delivers pretty gracefully on all that. She has an impressive pedigree academically, professionally, and journalistically. There is some pitchfork raising here, but the prongs are made of research, data, and morals.