5 min and 20 sec to read, 1332 words

In this recent paper the authors discuss public attitudes to technology as sentient or conscious. The overall take away is well summarized in the abstract:

Future developments in AI capabilities and safety will depend on public opinion and human-AI interaction. To begin to fill this research gap, we present the first nationally representative survey data on the topic of sentient AI: initial results from the Artificial Intelligence, Morality, and Sentience (AIMS) survey, a preregistered and longitudinal study of U.S. public opinion that began in 2021. Across one wave of data collection in 2021 and two in 2023 (total N = 3,500), we found mind perception and moral concern for AI well-being in 2021 were higher than predicted and significantly increased in 2023: for example, 71% agree sentient AI deserve to be treated with respect, and 38% support legal rights. People have become more threatened by AI, and there is widespread opposition to new technologies: 63% support a ban on smarter-thanhuman AI, and 69% support a ban on sentient AI. Expected timelines are surprisingly short and shortening with a median forecast of sentient AI in only five years and artificial general intelligence in only two years.

The findings here are difficult to parse, since what we are discussing is so unclear – but it is still really valuable, as it shows how the design metaphors of predictive technologies create challenges merely by virtue of being metaphors of mind, rather than metaphors of matter.

Any technology comes with a user interface metaphor.¹ See e.g. Neale, D.C. and Carroll, J.M., 1997. “The role of metaphors in user interface design.” In Handbook of human-computer interaction (pp. 441-462). North-Holland. The simplest example of this in computer software may be the operating systems and “windows”. We open and close windows, re-arrange them and work within them. The Internet came with its own metaphors of links, pages, sites etc and we quickly absorbed that new framing as well. With social media things became more troubling since we started to use more intentional metaphorical language such as “likes” for expressing some kind of sentiment about a post or “friends” for social contacts — and here the visibility of the metaphorical frame was radically reduced.

With AI the UI-metaphor is almost entirely built on intentionality and intentional actions such as wanting, thinking, answering, hallucinating etc — and we speak of predictive models as we would of other intentional systems, including ourselves.

This is not a problem in itself, it is a handy set of metaphors and we can use them to make sense of the technology faster than if we had to explain what is happening in a more mechanical way – but it does become a problem when we end up assuming that the metaphorical framing is real, and asking questions that are out of frame.² In a really early note on metaphors in programming, Alan Cooper noted “The biggest problem is that by representing old technology, metaphors firmly nail our conceptual feet to the ground, forever limiting the power of our software. They have a host of other problems as well, including the simple fact that there aren’t enough metaphors to go around, they don’t scale well, and the ability of users to recognize them is questionable.” Cooper, A., 1995. “The myth of metaphor.” Visual Basic Programmer’s Journal, 3, pp.127-128.

If we did this for operating systems everyone would react – asking if a window is clean when you are referring to a window in an operating system makes no sense, and someone who said they would spend Sunday cleaning their operating system windows would essentially just have explained that they did not quite understand the UI metaphor well enough to know that the windows do not need to be cleaned (although the screen may, and there is a point here that we should return to in a later post – metaphors are layered).

The problem here is an interesting one: how do we determine the metaphorical frame and where it breaks? What if I claimed the following:

(i) Saying that we should not build sentient AI is like saying that we need to periodically clean our Microsoft OS windows. It is a metaphorical mistake.

How would you react to that? It would depend on how you think about the technology interface and the underlying technology. If you believe that what is going on in the technology replicates what is going on in the mind, and hence that it should be possible to also build sentience you would say that I have it wrong, and that the question of whether to build sentient AI is a real question we need to resolve (unlike the question of if we need to clean our Microsoft windows).

But if you believe that the technology itself is radically different you would say that is roughly right — and that we need to re-think how we clarify the UI metaphor so that we do not get stuck in questions like this.

It is important to note here, by the way, that there is no argument about the complexity of the technology — we may be able to build something that is to sentience as predictive technologies are to human intelligence, and we do not need to argue that there is a limit or boundary here that cannot be crossed. In fact, the skepticism about our ability to build AGI is a skepticism that operates within the UI metaphor, accepting its extension in a way that obscures the better question of what it is that we are really building. ³ It is a tired observation to note that we would be better off if we did not call it “artificial intelligence” but something more neutral, but I think that observation misses the fact that we are building this with an intention to mirror ourselves, and that is a key property of the technology that results — but to some degree a different name would have helped us not to get stuck so quickly in the UI-metaphor.

The study takes the interesting perspective that this is partly, if not wholly, about perception and what they are interested in is if people believe that we should clean our Microsoft Windows or not — not whether that is a metaphor mistake. That is a useful approach, since it allows us – if we do indeed think it is a metaphor mistake this gives us a chance to think through the extent to which this mistake is made. ⁴ The study, in this respect, could work like one of Kahneman’s and Tversky’s studies charting the kinds of breakdowns of the rational mind that we have evolved to be vulnerable to.

And it seems – from the numbers cited above – that the severity of the UI metaphor confusion is high.

There are probably many reasons for this, including that we have evolved to apply intentional stances and frames to things that behave minimally intentionally since that is, from a cognitive economy standpoint, extremely efficient. Just as we have evolved to predict narratives and not propositions when it comes to probabilities. But we should be careful not to get stuck in the ease of the metaphor as the key criterion for its applicability — if we agree that it is still a metaphor.

If we don’t – and we think that we are building intelligence and that we can build sentience and consciousness – we need to discuss why we believe that the UI metaphor (and some of the design metaphors like neural networks) are really not metaphors at all, but descriptions of the real thing.

This is where things get interesting, and there is a lot more to be said here – not least about the differences between the simulated and the artificial. This is a subject I hope to return to, in a later post or essay.

Read more in “What Do People Think about Sentient AI?” by Jacy Reese Anthis, Janet V.T. Pauketat, Ali Ladak, Aikaterina Manoli .

Footnotes and references

• 1
  See e.g. Neale, D.C. and Carroll, J.M., 1997. “The role of metaphors in user interface design.” In Handbook of human-computer interaction (pp. 441-462). North-Holland.
• 2
  In a really early note on metaphors in programming, Alan Cooper noted “The biggest problem is that by representing old technology, metaphors firmly nail our conceptual feet to the ground, forever limiting the power of our software. They have a host of other problems as well, including the simple fact that there aren’t enough metaphors to go around, they don’t scale well, and the ability of users to recognize them is questionable.” Cooper, A., 1995. “The myth of metaphor.” Visual Basic Programmer’s Journal, 3, pp.127-128.
• 3
  It is a tired observation to note that we would be better off if we did not call it “artificial intelligence” but something more neutral, but I think that observation misses the fact that we are building this with an intention to mirror ourselves, and that is a key property of the technology that results — but to some degree a different name would have helped us not to get stuck so quickly in the UI-metaphor.
• 4
  The study, in this respect, could work like one of Kahneman’s and Tversky’s studies charting the kinds of breakdowns of the rational mind that we have evolved to be vulnerable to.