Seeing

This essay is about computer vision (CV). While I’m not a computer scientist and CV is not my area of specialization, I’m closer to the proverbial metal with CV than with many other technologies that fall under the umbrella of “AI.” I’m closer to CV than I am to other systems which use large datasets to help produce inferences. There are things worth saying about the role of CV in society, and there are also things to say about its central metaphor, and about its manner of organizing and presenting information. So this is an essay about CV. It’s also an essay about seeing, how we port a human idea of seeing to machines, and what else we bring along when we do that.

The reasons I’m closer to the metal with CV than with other “AI” technologies are twofold: I find it important to understand something that is quietly pervasive in my technology-mediated life (and those of others who live in similar media and surveillance environments); and I have been busy for the last few months working on ways to help others understand some of the assumptions and choices underlying computer vision. The first reason leads to the second, but the two together have led me to spend an increased amount of time with computer vision systems.

I’ll start with the desire to help others understand the assumptions underlying CV systems. There’s this nice project I’m working on with a colleague. It involves data labelling, and locally running a very small, very bounded CV system. If I had to explain it very simply, I’d say that we’re trying to help people understand that not only is there always a human at the bottom of the judgement coming out of an automated system (even if that's fifty years ago, and now almost totally invisible), but there are baked-in, historically-rooted assumptions about how the world works. That, broadly, is what we’re trying to make visible for a general audience. We are trying to make things visible which are normally valued for their invisibility. We’re doing that by allowing people interacting with our system to create their own labelling schemes for a collection of objects. We are asking the people who use our system to bring their own subjectivities back into computer vision.

WTF is CV?

First, though, let’s get some practicalities out of the way. What do I mean when I talk about computer vision? Computer vision is a subset of the field of artificial intelligence, which is a subset of computer science and cognitive science. Or, from a historical perspective, many of the things we now think of as artificial intelligence are subsets of early work on computer vision and computer sensing. All of these things come out of cybernetics, a field based on the study of systems and their processes, with an eye towards comparing living, biological systems with technical ones (and building technical systems based on learnings from biological ones).

Computer vision is about the processing and use of visual inputs. Early computer vision work was very interested in cognition and visual processing. That’s because early computer vision work was interested in replicating the role of eyes in the human sensory processing system – a very cybernetics kind of attitude. A lot of early work in cybernetics and computer science in general was interested in trying to replicate how human cognition works. So computer vision was about the processing of visual signals into usable information and categorization, and the subsequent use of those categorizations to organize information, or to take action in the world.

So how does computer vision work? As I said above, I’m not a computer scientist, so I’ll keep it basic and conceptual. One of the mechanisms by which computer vision works is comparing pixels to one another. Imagine it in its absolute simplest form: an image has a black area and a white area. No grey, just literally black and white. We want our computer vision system to be able to tell us which portion of the image is black and which portion of the image is white. The CV system needs to be able to identify what black and white are, and indicate which area is black and which area is white. This is easy mode. All we need to do is program it to trace or otherwise highlight the black areas and label them as black, and highlight the white areas and label them as white. Encodings “black” and “white” isn't a very hard problem, and requires little judgement, so this version of the exercise is comparatively easy. But computer vision is generally concerned with things more complex than labelling which area of an image is white and which area is black. On a more complex level, processes like optical character recognition (OCR) are simple forms of computer vision (nothing in CV is actually simple, and OCR as we know it now is breathtakingly complex and usually involves neural networks). If you’ve ever used a tool that takes an image of text and outputs letters, you’ve used OCR. That’s one use of the mechanisms of computer vision.

It’s one thing to label colours. Eventually, though, a descriptive layer comes into the process. If we’re talking about OCR, the description of a shape is going to be what the character is. A thing that looks like the number “1” should be labelled as “1” if we want a useful description of what’s being identified. James E. Dobson’s The Birth of Computer Vision (which I wrote about last week, and which provides an excellent early history of the field) offers the example of a computer vision system designed to discriminate between images of circles and squares. The reality of CV now is much more complex than identifying what is a square and what is a circle. But a key point remains: the purpose of the system is to identify items and discriminate between “this” and “that.” If we are to get accurate labels, though, human discrimination needs to be involved at some point. The categories of "circle" and "square" are provided, and important work in more recent CV efforts has involved the creation of very large datasets with human labour going into the detailed segmentation and labelling of huge sets of images. The differentiation of “this” and “that” is at the heart of CV, is one important vector in the way CV creates specific flattenings of the worlds, and in the last few decades, become pervasive.

CV creep

Cameras are very good now, and are everywhere. This is useful for computer vision. Today, common uses of computer vision include detecting and identifying objects and people in photos, making self-driving cars work, and automated number plate recognition (ANPR), among many other uses. The scan car driving around and giving parking tickets is one very common and everyday application of computer vision that exists quite pervasively here in the Netherlands. Elsewhere, you may be encountering computer vision when a parking garage doesn’t require your payment stub for the gate to open – that’s using ANPR to recognize your car and let you out. If you’re in the US, the infamous Flock cameras used by local police and governments are an increasingly pervasive application of CV. In short, computer vision is everywhere, both in devices you choose to own, and in the world around you.

Because cameras are very general-purpose tools, and because computer vision is predominantly software-based, people who are concerned about computation and privacy tend to be interested in making sure that the common and everyday tools used for computer vision don’t also get used for other applications. An example from the Dutch context: during Covid when there were rules about how many people were allowed to be near each other, there were concerns that scancars (those cars from the previous paragraph – the ones that enforce paid and permitted parking) would be used to police public gatherings. Both ANPR and detecting crowd size are public safety/public order applications of computer vision. The general purpose nature of cameras can cause fear and suspicion that an approved (legally sanctioned, deemed at some level to be proportionate) use of camera-based surveillance with computer vision can be an opportunity to get a foot in the door and introduce other forms of computationally-enabled surveillance.

In your phone’s photo roll, or on social media, computer vision is used to detect objects and people. This is how you can search your photos for a specific thing or person.

This pervasiveness is why knowing a bit more about computer vision matters. Lack of understanding of computer vision systems is a widespread societal issue. In some countries, like the one I live in, the use of computer vision systems touches basically everyone – which is not something you can say about many issues. With CV, the lack of awareness and understanding can have direct effects on literally all sectors of society, because literally everyone is touched by these systems in one way or another.

While the term “AI” is being used for systems that generate media, whether text, video, audio, or whatever, computer vision tends to come into current conversations far less frequently. To me, this is a really unfortunate oversight in the public discourse. Assumptions, categories, flattenings, and indeed biases are baked into computer vision systems, invisibly and quietly. Examples abound, but the shared outcome is often negative consequences for one group or another.