The summarised society

One of the things that generative AI will enable is the summarisation of the growing flows of information that we all live in. This is not surprising to the reader of Herbert Simon, who suggested that with a wealth of information comes a poverty of attention and a need to allocate attention efficiently. Now, what it does help us understand is that attention allocation can be achieved in a multitude of ways. The first is to help us focus attention on the right piece of information – this is essentially what a recommendation algorithm does. The second is to focus attention on the right features, and at the right resolution, in an information set. This is what a summarising algorithm does.

Summaries have been around for ages, most of them produced by people – and so they are not new in themselves. The time it takes to summarise a field has grown, however, and today there are several fields of research and knowledge that are impossible to summarise well before they change in fundamental ways. The limits of summarisation also limit how we can update our knowledge.

Now, if we believe that the quality of our decisions depend on the quality of the information we draw on when we make those decisions, this should worry us. It seems we could make better decisions if we had access to summaries of different fields as they evolve – at least if it is true that these summaries can be made in such a way that they capture the salient features of the evolving information landscape that are relevant for the decisions that we want to make. Are such summaries possible? Is the tendency to hallucinate that generative AI has a fatal flaw in producing such summaries? This is increasingly the focus of research.

The paper “Improving Primary Healthcare Workflow Using Extreme Summarization of Scientific Literature Based on Generative AI” proposes a summarisation approach to address the challenge faced by primary care professionals in keeping up-to-date with the latest scientific literature.

The researchers employed generative artificial intelligence techniques based on large-scale language models to summarise abstracts of scientific papers. The goal was to reduce the cognitive load experienced by practitioners, making it easier for them to stay informed about the latest developments relevant to their work.The study involved 113 university students from Slovenia and the United States, who were divided into three groups. Each group was provided with different types of abstracts (full abstracts, AI-generated short abstracts, or the option to choose between the two) and use cases related to preventive care and behaviour change.

The findings in the paper suggest that AI-generated summaries can significantly reduce the time needed to review scientific literature. However, the accuracy of knowledge extraction was lower in cases where the full abstract was not available, and this is key — we need to have a sense of what the ideal resolution of the data set used to summarise really is. It seems obvious that summaries made from a set of full papers will be more costly and take longer time, than summaries made from some kind of abstracts. This in turn suggests that we should think about a hierarchy of summaries here, and that there may be an argument for forcing longer abstracts on all papers submitted, so that the abstracts are more legible for AI-models that summarise them!

The design of summaries will quickly become a key element in how we learn about things.

In another example the paper titled “Summaries, Highlights, and Action items: Design, implementation and evaluation of an LLM-powered meeting recap system” explores the application of Large Language Models (LLMs) to improve efficiency and effectiveness of online meetings. The researchers designed, implemented, and evaluated a system that uses dialogue summarisation to create meeting recaps, focusing on two types of recap representations: important highlights, and a structured, hierarchical minutes view.

The study involves seven users in the context of their work meetings and finds that the LLM-based dialogue summarisation shows promise. However, it also uncovers limitations, such as the inability of the system to understand what’s personally relevant to participants, a tendency to miss important details, and the potential for mis-attributions that could affect group dynamics – but does that matter?

What is interesting is that there may be a quite low threshold for the quality of summaries for many readers (this will vary of course) and so even summaries that have these limitations could easily be valuable if you miss a meeting and no notes were taken. To preserve privacy, we would also have to think through different kinds of limits on attribution here.

One way to think about summarisation algorithms is to suggest that they compress information – and that raises an interesting question about how much information can be compressed for different purposes. How much can a meeting be compressed? This question quickly turns very funny, since we have all been in meetings that could have been compressed not even to emails, but into the sentence “we don’t know what we are doing, really” – but there is a serious use for it as well: we should look at the information density of our different activities.

One area that is over-ripe for information compression is email. If you look at your inbox in the morning and imagine that you could summarise it into a few items – how much would then actually be lost? I often find different emails on the same subject, and would love the ability to just ask my inbox to summarise the latest on a subject with views and action items. That would allow my a “project view” of my email and I would be able to step back and track work streams. The fact that email is not already the subject of massive efforts of summarisation and compression is somewhat baffling.

You could also allow for different summarisation views – summarise on individual, on project or on a topic. Summarise on emotional content — give me all the angry emails first. There are endless opportunities. One example of this idea – in this case to summarise on topic – is found in the paper titled “An End-to-End Workflow using Topic Segmentation and Text Summarisation Methods for Improved Podcast Comprehension” where authors combine topic segmentation and text summarisation techniques to provide a topic-by-topic breakdown, rather than a general summary of the entire podcast.

Now, you may be tempted to test this by inputting your inbox headlines into a chatbot – but you shouldn’t, remember that your email is probably filled with somewhat sensitive information. But that you may be tempted show something: even just headline summaries would sometimes be helpful – or would highlight how bad we are at writing good subject headings in email.

Summarisation naturally also carries risks – summaries destroy information, and nuance – and you could imagine second and third order effects to a society that consumes summaries rather than the original thing – the lack of nuance could be disastrous for some classes of decision (legal decision making comes to mind). This is true for all shifts and changes in attention allocation, however – since society is made of attention.

Recommendation algorithms and summarisation algorithms are just two dimensions here. How we re-design, necessarily re-design, we should say, attention allocation will change society too.

Leave a Reply