Choice & harm & context

For my entire adult life, taking an airplane has involved the possibility of being asked to step into a millimetre wave scanner so that any contraband objects I may be carrying on my person can be detected. This step beyond the more traditional metal detector, a step we now take for granted, is a symptom of the post-9/11 world in which liquids in carry-on luggage must be limited, hands must be swabbed for explosives, and shoes must go through the x-ray machine. In the time I've been flying in this security climate (though these days, I do my best to avoid flying if at all possible), I've been able to make a choice about my engagement with the millimetre wave scanner: I can opt out of the scanner as long as I'm willing to be subjected to a manual pat-down by security personnel. This inevitably takes longer than the scanner, generally happens just next to the queue so that all the other passengers can see, and is engineered to be undesirable, compared to the default option. So whenever I fly, I make a choice that I find appropriate and principled, and I opt to take the pat-down instead of going through the scanner. The ability to make this choice is provided by regulations which give me the right to refuse the scanner, combined with my own understanding of how the technology works, and whether or not I find it reasonable to engage with.

Despite routinely making choices like this in my life. I am not a fan of the concept of the rational economic actor. That's the idea, from classical economics, that people (and companies, as well) are in rational pursuit of choices which are optimal for them individually, and that they have sufficient information to be able to make those choices. The rationality of these actors, in aggregate, acts as a force that somehow makes the market they are operating in fairer and more correct. There is little novelty in critiquing the idea of the rational economic actor, and I’m not an economist, so I won’t take a proper stab at going into more detail – many others have done that before, and better.

Suffice it to say, though, that while I don’t subscribe to the idea that people are rational economic actors, I do believe that the concept of “choice” is relevant and interesting in the context of technology use. Crucially, there’s a difference between having some choice and having no choice at all. While one may not be making a totally optimal and rational decision to use Instagram or buy an iPhone, there is an element of individual decision-making in those most of the time, even if it’s not based on total freedom or having complete information about the pros and cons of one's decisions.

The choice one makes to use, for example, WhatsApp, when everyone else is using it, and opting out would mean being cut out of a group, is not a very free choice. It’s instead quite coerced, a choice one might not have made individually, but that they feel obliged to make in order to rub along nicely within their social world. It’s a sort of pseudo-choice. These pseudo-choices are incredibly common and, as with the example of the millimetre wave scanner at the airport, are often designed to heavily favour one option over the other – they are choices, but they are not free choices.

When choice is not available

But there are other technologies we encounter which are chosen without our input, and which can be exceptionally difficult to avoid or opt out of. These are the cameras we encounter when out in the street or at the grocery store, the algorithmic decision-aid systems used by potential employers during pre-screenings, and so many other tools that are used on us, without our meaningful consent. You could decide to not go to the grocery store with the behaviour detection cameras, or not apply to a potential employer who uses an algorithmic screening process, or just not go outside. Some of these could be choices, but they don’t contain opportunities for meaningful consent, and they don’t offer an opt-out beyond “just don’t engage.” Sometimes, even “don’t engage” isn’t an option. When these tools land in government services, or in public school systems, or the only grocery store within a reasonable distance, “don’t engage” becomes nigh-on impossible.

So though I am not a fan of the model of choice laid out in the idea of the rational economic actor, I do find it important to acknowledge the existence of choice, inasmuch as we can use it to draw a line between the technologies we decide to use for ourselves, even when the choices we’re making are not totally free, and technologies we are forced to use, or which are used on us. The potential harm we face is greater when a technology is inescapable, not least of all because we do not have the choice to not engage with it, or to not be touched by it.

The influence of context

I have argued elsewhere that not only can technology not be inherently neutral, but it’s also not inherently and inescapably either good or bad – it’s instead influenced by context. This means that evaluating a technology on its own is often not going to be sufficient in determining what harms it might pose. The societal, interpersonal, legal, procedural, structural, and other wrappers around the technology in a given implementation all also contribute to its capacity for harm, and the specifics of the harm it has the capacity to inflict.

All harm stemming from oppressive or negligent implementations of technology is harm, and none of it should happen, but it’s a couple degrees worse when it’s harm that comes from a system the harmed person can’t avoid. We are seeing this happening more and more frequently. Out of the United States in the last few years, there’s been a stream of stories about wrongful arrests based on false positives in facial recognition systems (with, of course, an over-representation of people of colour in those false positives). In the Netherlands, there was the infamous Childcare Benefits Scandal, in which a system trained on already problematic data compounded government discrimination based on characteristics like citizenship or marital status. Technologies which, within the structures they’re implemented, re-enact and compound existing harms can turbocharge the risk of creating societies which are oppressive and unequal.

If we want to begin to address these problems, we need to get sharper, as societies, on understanding the context of our technologies. We can’t assume that resolving current and potential harms is a goal easily achieved by making changes to only the technologies themselves. If a governmental agency has already baked bias into its decision-making structure, technologies it implements will run a high risk of serving those biases. If policing is already steeped in racial prejudice, adding facial recognition cameras will not make it less racist. If food deserts exist where residents are all captured by one store, with no other options, no one will feel able to opt out of automated behaviour detection by just not shopping at that store. Instead, if we want to reconfigure the way technologies impact wellbeing, we also need systems (social, legal, procedural, organizational, etc.) that offer possibilities to avoid harm and potentially enact good. Technology exists in context, and change cannot happen without attending to and changing that context.