Category: Spurious investigations

One of the things that generative AI will enable is the summarisation of the growing flows of information that we all live in. This is not surprising to the reader of Herbert Simon, who suggested that with a wealth of information comes a poverty of attention and a need to allocate attention efficiently. Now, what it does help us understand is that attention allocation can be achieved in a multitude of ways. The first is to help us focus attention on the right piece of information – this is essentially what a recommendation algorithm does. The second is to focus attention on the right features, and at the right resolution, in an information set. This is what a summarising algorithm does.

Summaries have been around for ages, most of them produced by people – and so they are not new in themselves. The time it takes to summarise a field has grown, however, and today there are several fields of research and knowledge that are impossible to summarise well before they change in fundamental ways. The limits of summarisation also limit how we can update our knowledge.

Now, if we believe that the quality of our decisions depend on the quality of the information we draw on when we make those decisions, this should worry us. It seems we could make better decisions if we had access to summaries of different fields as they evolve – at least if it is true that these summaries can be made in such a way that they capture the salient features of the evolving information landscape that are relevant for the decisions that we want to make. Are such summaries possible? Is the tendency to hallucinate that generative AI has a fatal flaw in producing such summaries? This is increasingly the focus of research.

The paper “Improving Primary Healthcare Workflow Using Extreme Summarization of Scientific Literature Based on Generative AI” proposes a summarisation approach to address the challenge faced by primary care professionals in keeping up-to-date with the latest scientific literature.

The researchers employed generative artificial intelligence techniques based on large-scale language models to summarise abstracts of scientific papers. The goal was to reduce the cognitive load experienced by practitioners, making it easier for them to stay informed about the latest developments relevant to their work.The study involved 113 university students from Slovenia and the United States, who were divided into three groups. Each group was provided with different types of abstracts (full abstracts, AI-generated short abstracts, or the option to choose between the two) and use cases related to preventive care and behaviour change.

The findings in the paper suggest that AI-generated summaries can significantly reduce the time needed to review scientific literature. However, the accuracy of knowledge extraction was lower in cases where the full abstract was not available, and this is key — we need to have a sense of what the ideal resolution of the data set used to summarise really is. It seems obvious that summaries made from a set of full papers will be more costly and take longer time, than summaries made from some kind of abstracts. This in turn suggests that we should think about a hierarchy of summaries here, and that there may be an argument for forcing longer abstracts on all papers submitted, so that the abstracts are more legible for AI-models that summarise them!

The design of summaries will quickly become a key element in how we learn about things.

In another example the paper titled “Summaries, Highlights, and Action items: Design, implementation and evaluation of an LLM-powered meeting recap system” explores the application of Large Language Models (LLMs) to improve efficiency and effectiveness of online meetings. The researchers designed, implemented, and evaluated a system that uses dialogue summarisation to create meeting recaps, focusing on two types of recap representations: important highlights, and a structured, hierarchical minutes view.

The study involves seven users in the context of their work meetings and finds that the LLM-based dialogue summarisation shows promise. However, it also uncovers limitations, such as the inability of the system to understand what’s personally relevant to participants, a tendency to miss important details, and the potential for mis-attributions that could affect group dynamics – but does that matter?

What is interesting is that there may be a quite low threshold for the quality of summaries for many readers (this will vary of course) and so even summaries that have these limitations could easily be valuable if you miss a meeting and no notes were taken. To preserve privacy, we would also have to think through different kinds of limits on attribution here.

One way to think about summarisation algorithms is to suggest that they compress information – and that raises an interesting question about how much information can be compressed for different purposes. How much can a meeting be compressed? This question quickly turns very funny, since we have all been in meetings that could have been compressed not even to emails, but into the sentence “we don’t know what we are doing, really” – but there is a serious use for it as well: we should look at the information density of our different activities.

One area that is over-ripe for information compression is email. If you look at your inbox in the morning and imagine that you could summarise it into a few items – how much would then actually be lost? I often find different emails on the same subject, and would love the ability to just ask my inbox to summarise the latest on a subject with views and action items. That would allow my a “project view” of my email and I would be able to step back and track work streams. The fact that email is not already the subject of massive efforts of summarisation and compression is somewhat baffling.

You could also allow for different summarisation views – summarise on individual, on project or on a topic. Summarise on emotional content — give me all the angry emails first. There are endless opportunities. One example of this idea – in this case to summarise on topic – is found in the paper titled “An End-to-End Workflow using Topic Segmentation and Text Summarisation Methods for Improved Podcast Comprehension” where authors combine topic segmentation and text summarisation techniques to provide a topic-by-topic breakdown, rather than a general summary of the entire podcast.

Now, you may be tempted to test this by inputting your inbox headlines into a chatbot – but you shouldn’t, remember that your email is probably filled with somewhat sensitive information. But that you may be tempted show something: even just headline summaries would sometimes be helpful – or would highlight how bad we are at writing good subject headings in email.

Summarisation naturally also carries risks – summaries destroy information, and nuance – and you could imagine second and third order effects to a society that consumes summaries rather than the original thing – the lack of nuance could be disastrous for some classes of decision (legal decision making comes to mind). This is true for all shifts and changes in attention allocation, however – since society is made of attention.

Recommendation algorithms and summarisation algorithms are just two dimensions here. How we re-design, necessarily re-design, we should say, attention allocation will change society too.

Are there some commons that we should protect that we are just not aware of? Well, for the longest time the idea that there was a “climate” to protect was not obvious to people – and the overall discussion we now have about climate change required a concept to focus on, and climate ended up being the focal concept.

One way to think about the future is to think about possible focal concepts like this that have not been invented yet. Here are a few possible examples.

• The Human Text – the sum total of all things written by human beings, not generated by AI. The value of this commons is still unclear to us (why is a text written by a human being in some way more worthy of protection than a generated text?) – but we may discover that there are some qualities in The Human Text that are unique for that corpus. This could of course be true for a lot of other forms of creative outputs too.
• Social autonomy. The level of overall autonomy that we retain as a society, and the degree to which we are relying on systems to solve our problems for us. How many decisions are made by human beings and not by systems? This is a difficult one, since we usually look to the quality of the decision rather than its origin, but there may well be reasons to value human decision making not just as an artisanal quaint thing, but as better for different reasons.
• Agency. Closely related – but more about the distribution of agency over systems, perhaps. Evolution creates behavioural complexity through pre-programming (spider’s web) or agency. We now have created a new, artificial agency – post-programming – and human agency may be squeezed in between these two forms of programming. This may make a lot of sense from evolution’s perspective – i.e. adaptive capability may increase when we revert to the ideal state of pre and post programming and avoid messy agency in the middle – but still is interesting to think about the value of agency in the middle. Dennett and others seem to argue that agency in the middle is what underpins almost all of our social institutions, even if it sometimes is a fiction. And maybe this connects to the first one: maybe there is a value in agency-produced text?
• Fact. This is a common that we often are quite blind to – and feel uncomfortable articulating – but the depletion of facts is an interesting and hard problem to work on. Facts do – as Sam Arbesman points out – have a half-life, so there is a natural depletion – but beyond that depletion the rate at which we destroy this particular commons has to do with education, public sphere norms and many, many other social patterns. What if we started to treat fact as a commons? And yes – you could make the same argument for truth, and analyse the evolution of our relationship to truth as a commons problem.
• Attention. While deeply individual, in some sense, attention is actually a commons we draw on to build society and democratic institutions. The way in which we manage this commons is determinative of the quality of democracy that we get.
• Childhood. Do we still have childhoods like we used to? Is childhood a commons that underpins our civilisational growth and prosperity? What about the Global South / North divide in childhoods etc?

I am sure there are many other possible focal concepts as well. These three interest me more as examples than anything else — and the real prediction question is which focal concepts will seem obvious in the future? Animal life quality comes to mind as well, as one that many have guessed at – the idea that we eat animals may seem a lot like insulating with asbestos or torture in the future, they argue. That does not seem wholly unlikely to me, with synthetic meat on the horizon etc. These changes can be relatively swift – how long did it take for climate to emerge as an obvious commons? Arguably this took at most a decade, and then we can debate if it starts as “common heritage of mankind” or with the 1988 establishment of the IPCC, but still.

A lot of these things come back to a more fundamental commons that has remained invisible for a long time – since we believe it is deeply individual: time. As noted by many, the best measure of freedom is probably discretionary time, but that is not quite right. If you are free to use your time alone you may feel individually free – but are you really free in a more fundamental way? If there is not a certain pool or commons of shared discretionary time in a society, it falls apart. This is the argument for what we sloppily call leisure – and we probably need to think much more about this than we are currently.

We are all living in the invisible commons of time.

Let’s say that we want to understand if a certain system or phenomenon is exhibiting human-level intelligence. What would we then look for? There are multitudes of such tests, and they look for a great variety of things. One catalogue of such tests is BIG-bench, a battery of task tests that is aimed at understanding the capabilities of different systems. ¹ Srivastava, Aarohi, et al. “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.” arXiv preprint arXiv:2206.04615 (2022).See here for the paper.

BIG-bench looks at a large number of tasks that are both creative and difficult, and it is a fascinating catalogue – but it does not specifically look at if the system plays. Yet, the ability to play is probably fundamental to human intelligence – not just incidental or a kind of idling mode. ² See for example Fein, Greta G. “Skill and intelligence: The functions of play.” Behavioral and Brain Sciences 5.1 (1982): 163-164. exploring the adaptiveness of play and the setting of intrinsic limitations.

Would it then not make sense to look at if there is a way to test for play? To see if an intelligence can play? Well, you could argue that it seems quite clear that LLMs can play with us – we can ask it to play any game where we give it the rules, and we can even ask it to make up a game about something and it will do so. But that is not quite what we are looking for. What we want to see is if a system starts playing itself – if it can be, in some sense, curious enough to set intrinsic limitations and obstacles for itself and find joy in playing a game that it has invented for itself.

This notion of motivation by curiosity also seems a little trivial, you just need to build a system that has an instruction to play games when it does nothing else, right? Not quite. When we play and how we play are not random behaviours. Play is not the equivalent of a human screensaver. Instead, play is something that is related to building models of the world, planning ahead, remembering systems of rules — it is much more fundamental.

Games may play a much larger role in intelligence than we think, and the role of play is still being explored conceptually in philosophy. ³ See e.g. Nguyen, C. Thi. “Philosophy of games.” Philosophy Compass 12.8 (2017): e12426. and the same author’s subsequent book Nguyen, C. Thi. Games: Agency as art. Oxford University Press, USA, 2020. . Maybe the study of complex systems like models and their behaviour should focus not only on problem solving, but the ways in which agency is exhibited and structured in play and games.

There is an irony here – in a way – since a lot of research has focused on how to play games and to do so until the machine can beat human players. Maybe that was almost right, and the greater question is not how to play, but when and why we play and testing for that in a different way.

There is much more to be explored here – like how our narrow understanding of rationality and intelligence obscures the adaptive value and role of things like play and games, or how games – as mental models – structure the way we learn and understand the world.

Notes:

• 1
  Srivastava, Aarohi, et al. “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.” arXiv preprint arXiv:2206.04615 (2022).See here for the paper.
• 2
  See for example Fein, Greta G. “Skill and intelligence: The functions of play.” Behavioral and Brain Sciences 5.1 (1982): 163-164. exploring the adaptiveness of play and the setting of intrinsic limitations.
• 3
  See e.g. Nguyen, C. Thi. “Philosophy of games.” Philosophy Compass 12.8 (2017): e12426. and the same author’s subsequent book Nguyen, C. Thi. Games: Agency as art. Oxford University Press, USA, 2020.

Here is a common thought model: artificial intelligence will become more and more intelligent, and reach human capability. At that point it becomes something like an artificial general intelligence and some time after that it will use that intelligence to bootstrap itself into an artificial super intelligence. The model then is one-dimensional: [AI-AGI-ASI]

Now, there is a problem with this model, and that is that it anchors on the idea that intelligence is a one-dimensional capability, a bit like height. We imagine intelligence almost like the height of a child, an adolescent and then an adult. But that surely must be a simplification?

Intelligence as a concept is tricky. There are many different definitions ¹ See e.g. Legg, Shane, and Marcus Hutter. “A collection of definitions of intelligence.” Frontiers in Artificial Intelligence and applications 157 (2007): 17. – and the number of definitions have not become fewer in the last 15 years and the usefulness of the concept can be seriously questioned – even if we can hope to find working definitions that can be better than some other definitions ² See e.g. Wang, Pei. On the working definition of intelligence. Technical Report 94, Center for Research on Concepts and Cognition, Indiana University, 1995. .

One approach here is to abandon the concept of intelligence and instead speak about capability or competence. A system can become increasingly capable or competent – and that is what we care about. When a system is as capable as a human being in a domain we have human-equivalent capability. That is enough. We do not need to call it intelligence. This also allows us to escape the metaphorical morass that follows with using the term “intelligence” since it is so closely entwined in a language game that deals with human intentionality.

This is hardly a revolutionary approach – capability language is already a key component in many discussions about artificial intelligence, so why would we make this shift? One reason is that it helps us explore a discussion about the different dimensions at play here. We can agree that systems become more capable, but we should still ask if they also develop in other dimensions.

One well-known such dimension that concerns a lot of researchers is explainability. As the systems become more capable they become less explainable and there seems to be a trade-o ff there. That brings another tricky concept into our midst – explanation – and forces us to discuss exactly what it means to explain something. This is not a trivial question, either, but a live philosophical debate that is largely undecided. We know that there are different stances / levels of explanation that we can explore – from physicalistic to intentional in Dennett’s works for example – and that explanations can be evaluated in a multitude of ways. ³See Dennett, Daniel C. The intentional stance. MIT press, 1989 but also early works like Thagard, Paul, Dawn M. Cohen, and Keith J. Holyoak. “Chemical Analogies: Two Kinds of Explanation.” IJCAI. 1989, where different kinds of explanations are introduced. The philosophical standard works include books like Von Wright, Georg Henrik. Explanation and understanding. Cornell University Press, 2004 (1971) and Apel, Karl-Otto. Understanding and explanation: A transcendental-pragmatic perspective. MIT Press, 1984.

Instead of explainability, then, we could just use another term – and one that is not just about explaining why or what a technical system is doing, but the degree to which we understand it. I would suggest that we could call that quality simply strangeness.

What emerges is a re-phrasing (but I think a useful one) where we can explore how capability and strangeness develop over time in complex systems of the type we are now building.

Now, there are different hypotheses here that we can explore – but the reason this interests me is that I wonder if systems become stranger faster than they become capable. We can then imagine different system paths across this space and discuss if there are general paths that we can expect or that would be interesting to understand. System A and system B in this picture are interesting examples:

System A goes through a period of strangeness before becoming much more capable, and system B is strange to begin with and then becomes very capable. And looking at this we can then play around with concepts like artificial general intelligence and artificial super intelligence and artificial strange intelligence:

This would be a good reminder that as systems become more complex, they do not only become more capable, but they also become more inscrutable and inaccessible to our understanding – but at the same time, a stranger in our midst can be a way for us to see ourselves anew. ⁴ See for example Tuan, Yi-Fu. “Strangers and strangeness.” Geographical Review (1986): 10-19, who notes that there is a kind of grace to strangeness . And the etymology here is also satisfying: “from elsewhere, foreign, unknown, unfamiliar, not belonging to the place where found” – a good reminder that we are dealing not with a replication of the familiar, ourselves, but the uncovering of something truly strange.

Notes

• 1
  See e.g. Legg, Shane, and Marcus Hutter. “A collection of definitions of intelligence.” Frontiers in Artificial Intelligence and applications 157 (2007): 17. – and the number of definitions have not become fewer in the last 15 years
• 2
  See e.g. Wang, Pei. On the working definition of intelligence. Technical Report 94, Center for Research on Concepts and Cognition, Indiana University, 1995.
• 3
  See Dennett, Daniel C. The intentional stance. MIT press, 1989 but also early works like Thagard, Paul, Dawn M. Cohen, and Keith J. Holyoak. “Chemical Analogies: Two Kinds of Explanation.” IJCAI. 1989, where different kinds of explanations are introduced. The philosophical standard works include books like Von Wright, Georg Henrik. Explanation and understanding. Cornell University Press, 2004 (1971) and Apel, Karl-Otto. Understanding and explanation: A transcendental-pragmatic perspective. MIT Press, 1984.
• 4
  See for example Tuan, Yi-Fu. “Strangers and strangeness.” Geographical Review (1986): 10-19, who notes that there is a kind of grace to strangeness

In considering the potential for an ethology of artificial intelligence (AI), we are venturing into uncharted territory. The idea of observing and studying AI behavior as we would an animal’s seems, at first glance, to be an attempt to anthropo- or biomorphize technology. However, as AI systems like large language models become increasingly complex, the need to understand these systems through their behavior becomes more pressing. In fact, the complexity and opaqueness of these systems often necessitate an approach akin to ethology, the study of animal behavior.

This also goes to the heart of the question around explainability. What an explanation really is, is a surprisingly deep question. One fairly clear view in philosophy is that explanations exist at different levels – and ethology is a way to frame a nested hierarchy of explanations into a comprehensive explanation of behavior over all. An ethological explanation is composed of layers of explanations.

One formulation of this is that of Niko Tinbergen, who approaches animal behavior from four primary angles: adaptation, phylogeny, mechanism, and ontogeny. Applying these lenses to the study of AI presents unique challenges, but it also potentially offers us new ways of understanding these complex systems – a nested system of explanations.

The first question of adaptation–why the AI is performing the behavior–might seem straightforward in the case of AI, as its behaviors are generally determined by human-defined goals and objectives. However, as AI becomes more complex and capable of self-learning, this question becomes more intricate. The AI may develop behaviors that optimize its performance in unexpected ways, making the question of why it is behaving in a certain way less straightforward.

Phylogeny, or the evolutionary history of a behavior, is a concept that might seem out of place when applied to AI. Unlike biological entities, AI does not evolve through natural selection over generations. However, AI does have a form of phylogeny if we consider the iterative development and improvement of AI models as a kind of evolution. Understanding the ‘phylogeny’ of an AI system could involve tracing back its development, examining the various iterations it went through, and the changes made at each step.

Mechanism, or what triggers the behavior, is perhaps the easiest to apply to AI, as it involves the inner workings of the system. However, as AI systems grow more complex and their decision-making processes more opaque, understanding these mechanisms becomes more challenging. This is particularly the case with machine learning models where the exact decision-making process is not always easily extractable or understandable.This is where traditional explainability questions often end up.

Finally, ontogeny, or how behavior develops over an individual’s lifetime, presents another intriguing lens through which to view AI. While AIs do not have lifetimes in the biological sense, they do have operational lifespans during which their behavior can change significantly, especially in the case of learning models. Observing how an AI’s behavior changes over time, in response to new data or changes in its operating environment, could provide valuable insights into its functioning.

So, can we develop an ethology of AI? The application of Tinbergen’s questions to AI is certainly challenging, but not impossible. While some aspects of these questions require modification or reinterpretation, they provide a potentially fruitful framework for understanding the behavior of complex AI systems. In fact, they might become necessary tools given the increasing complexity of these systems.

Refusing this approach, we risk confining ourselves solely to mechanistic explanations of AI behavior. Such a narrow view would limit our understanding of AI and potentially blind us to significant aspects of their operation and implications. By embracing an ethological approach, we can gain a more holistic and nuanced understanding of these complex systems, aiding us in their development, management, and integration into society. As we continue to create and interact with increasingly sophisticated AI, the need for such understanding will only grow more urgent.

Now, we can also take this another level and study man-machine complex systems, and the resulting ethology of technology in use here. This will allow us to find even better ways of understanding risks and weaknesses in the systems. Our focus on understanding how the technical systems work is misplaced when the reality is that it is the technologies in human use we need to understand – and when we defer to systems, and how they can be designed to avoid that we defer to them when we – and they – have the least understanding of what is going on. ¹ See this intriguing argument for models stating they do not know when their confidence levels sink below a certain threshold.

This mix of anthropology and ethology could well develop into a key skill in the very near future.

• 1
  See this intriguing argument for models stating they do not know when their confidence levels sink below a certain threshold.

One of the most common, yet unexamined, concepts in technology policy is the idea of a “pace of technical development”. Usually, it is described as “fast” and taken to mean that we need to make interventions in a timely manner. This is all good and well, but what does it mean to say that the pace of innovation or technical development is fast? Relative to what? One way to approach that question is to say that we should look at two kinds of speed, and we can do that by employing a very simple mental model: that of the alphabet. ¹ This notion derives from a talk by Ricardo Hausman on knowledge and economy.

The alphabet is essential in developing textual space – the space of texts that we have actually written and that can be used for different purposes.² There is an important difference between the texts that could be written and the texts we have actually written. When we speak of the pace of textual development (the analogue here of technical development), we immediately realise that it depends on two different things: how many actual letters there are to combine and how many combinations we have made. The invention of a new symbol or letter allows us to create more words, and hence more texts.

The textual space, then, develops through capability and combination.

This is not unlike biological evolution – the idea of speciation speed and the emergence of new biological capabilities captures something like this as well. This is – grossly simplified – the idea behind the Cambrian evolution.

Systems that develop in this way are tricky to study, and we often miss the sum total pace of development because we are only looking at the pace with which new letters are added to the alphabet, and not the number of new combinations that this makes possible or the number of actual combinations that are produced.

As we add new letters the number of possible words grows fast – combinatorially – and the textual space then expands at the pace we try new texts and words and combinations.

There is no reason to think that the pace of technical development is any different. It is also, very likely, a composite speed where capabilities and combinations unfold the technical space. Think about the Internet as an example: it added a whole bucket of letters to our alphabet and suddenly there are a lot of new things we can do to explore the adjacent possible – and a lot of the progress we have seen since has been fuelled by that, but progress in combinations rather than pure capabilities. And look at AI now – the way that the field is growing is Cambrian in nature, for the moment. ³ See here for example.

There are other interesting aspects of this mental model that also are worth thinking more about.

Given a certain alphabet that vastly exceeds our ability to explore the textual space that can be generated by it – how fast are new words and new texts added? It seems likely that we produce more new words today than before, but not per English language speaker. So what does this tell us about the pace of textual development?

[ Alphabet – dictionary – text ] as a model could be translated to capabilities, tools, usages in technical development – but there are surely other mappings as well, so what do they look like?

In order to be able to compare the pace of development of different systems, we can assume a sort of general model where they are all languages, with alphabets, dictionaries and texts. What can we then say about law and technology? Is law developing slower than technology? It would seem that way, but why is that? Is it because of a lack of letters in the alphabet? A lack of words in the dictionary? Or is there something about the production of text in some systems that is harder? This seems to suggest that there is a role in the mental model for writing / producing text and the costs associated with that.

Can systems, equally, be slowed down by having too many words in a dictionary? Or too many letters?

At the very least it seems prudent to think of technical development as a two dimensional process – with capabilities and combinations as a first approximation – alphabets and words. It is probably even better described by some n-dimensional analogy, but economy of description also matters.

Notes

• 1
  This notion derives from a talk by Ricardo Hausman on knowledge and economy.
• 2
  There is an important difference between the texts that could be written and the texts we have actually written.
• 3
  See here for example.