Unpredictable Patterns

Contact

  • Towards a philosophy of prediction III: events or worlds, propositions or narratives?

    +

    +

    +

    +

    +

    +

    What is it we predict when we predict something? The perhaps most common belief seems to be that we predict events, so that we predict event E will occur with some probability P. But how do we divide the world into events? What is an event? It cannot be a single moment in time, isolated from all other moments, since that would mean that we could not predict it in any meaningful way. A prediction relies on a web of causality of some sort, or at least a web of some kind of connections to the event that we are trying to predict — an event is not an isolated moment, it is the product of at least a set of other moments. 1 There is surely some kind of connection with Quine’s web of belief here. See on his holism: https://plato.stanford.edu/entries/quine/ and equally there is a connection to Whitehead’s philosophy of process and event here, see https://philarchive.org/archive/MASRTC-3

    But this presents another problem: how do we decide which events E(1)…E(N) go into the set of relevant preceding events when we want to predict a specific event E (x)? One image here would be that of a landscape – where we are predicting a point in the landscape on the basis of other adjacent points, and even if the entire landscape can contribute to the prediction’s accuracy, it does so only marginally. Predicting our event requires understanding the adjacent event space in some way – and determining that space is really hard (this is sort of what super forecasters do when they turn a prediction into a Fermi-problem with a set of factors that will impact the probability of an event. Identifying the baseline is a bit like identifying the neighbourhood of the event, perhaps).

    In some sense, however, all predictions are predictions of worlds. If I predict Bill Clinton will win the election I predict that we will be in a world W such that Bill Clinton is president of the United States in that world. “World” here is shorthand for the entire state space we are in, among all possible state spaces of the universe (indeed, we should probably also be able to say that we are predicting universes).

    So which is it? Are we predicting events or worlds? Is there a difference? Does it matter? It seems as if the mental image we have when we are predicting is different – predicting an event is like predicting which card will be drawn from a deck, whereas predicting a world seems more like envisioning a future wholesale. But that is not enough to argue that there is a meaningful difference, I think. But there is something about this distinction that still makes sense, if nothing else as a tool for checking our predictions. If we predict an event, it should be compatible with the world we predict overall — and so the prediction of a point in the landscape needs to be compatible with what we know about the space of possible landscapes. If I predict an event E that is not possible in any conceivable world, my prediction is a curious kind of mistake.

    Equally, we may benefit from remembering that adjacency is not determined only by proximity in space and time — some events can be predicted by looking at something that happened a long time ago, far away. Some people’s actions are determined by events and experiences in their childhood, for example, and predicting what they will do next may depend not on what they have just done, but by who they are (identity-based predictions are in a sense world-predictions. A person who loves dogs, cats and other animals may have had a bad encounter with a snake in their childhood and may react badly to a snake even though they could have been thought to love animals based on their recent interactions).

    There is a lot of connections with Nelson Goodman’s philosophy here, I think. His notion of “world versions” seem especially relevant to the philosophy of predictions. In some way we make worlds when we predict, and those worlds are narratively described, and we then predict the narrative rather than the event. And this seems important as we keep exploring this subject: events are atomistic, propositional, and our thinking is not propositional. When we predict something we rarely predict a proposition — even though this is often how prediction is framed.

    A simple example would be the prediction that Bill Clinton would be president. That prediction might be formulated as “the probability P that the proposition “Bill Clinton is president” is true at time t” — but when we predict we predict stories, or narratives, so it is more like “the probability P that the story that ends with Bill Clinton winning the election at time T ends up being true”. This matters, since the unit of prediction matters for understanding how good predictions are composed. 2 This leads us to the questions about rationality. Kahneman’s and Tversky’s bankperson Linda is an interesting example that I wrote more about here. See e.g. here.

    Exploring a philosophy of predictions requires understanding what it is that is predicted – events or worlds, propositions or stories (narratives).3 Or if we should speak about predicting “a web” or “a tree” in some way. See interesting discussion here: https://link.springer.com/chapter/10.1057/9781137472519_8

    These are questions of how we configure a problem. Problem configuration is an undervalued question that would be worth more attention — and it is not just about framing, either — even if the two are closely related.4 Note to self: it would be good to explore the configuration of problems more in detail – and how it relates to epistemology.

    Footnotes and references

    • 1
      There is surely some kind of connection with Quine’s web of belief here. See on his holism: https://plato.stanford.edu/entries/quine/ and equally there is a connection to Whitehead’s philosophy of process and event here, see https://philarchive.org/archive/MASRTC-3
    • 2
      This leads us to the questions about rationality. Kahneman’s and Tversky’s bankperson Linda is an interesting example that I wrote more about here. See e.g. here.
    • 3
      Or if we should speak about predicting “a web” or “a tree” in some way. See interesting discussion here: https://link.springer.com/chapter/10.1057/9781137472519_8
    • 4
      Note to self: it would be good to explore the configuration of problems more in detail – and how it relates to epistemology.

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Towards a philosophy of prediction II: the dimensions of predictions

    +

    +

    +

    +

    +

    +

    Predictions are about the future, but that is about as much as we can be sure about — there seems to be many different dimensions to predictions that create a whole taxonomy of them for us to consider as we think about how to understand their role in society.

    First, there is something we can call resolution. A prediction can be targeted in time and space in such a way that it is highly precise. If I predict you will die at 22.34, at a certain longitude and latitude, on a certain date and in a certain way, I have made a very different prediction than if I predict that one day you will die.

    One way to think about the former kind of prediction is to say that it is not just more precise, but also more complex in that it combines, in itself, several different variables in state space, so it divides state space into a smaller set than the latter one does. The latter prediction merely states that state space is divided into one part where you live and one where you do not, and at one point you will be in the part of state space where you do not live.

    The former prediction, however, sections out a much more precise chunk of state space and states that you will find yourself in that chunk.

    This raises the interesting question of how precise a prediction can be: can a prediction be complete in some sense? If there is such a thing as a completely defined prediction and all others are just partial predictions, then partial and complete predictions are end points in this particular dimensions. So this give is a first candidate dimension:

    (I) Resolution – complete -partial predictions of systems

    Are there other relevant dimensions?

    We can play around with a toy model in which we put three balls into three slots lined up in a single line, and we randomly change the configuration every hour. The balls are blue, black and red, and our job is to predict in what state this system will be in the future. If we only have to predict that there will be a blue ball in the line-up at some point in time, our job is easy – we can do that with high probability of being right. If we have to predict the probability that the balls will appear in some given order, like red / blue / black at some point in time, our task is harder, but the longer the time window, the easier it gets. Predicting the sequence of ball configurations is harder than predicting a single state of the system – and so on.

    But this toy model fails us in thinking about systems that are teleonomic, so here there seems to be a different dimension for us to explore. Systems that are intentional or intentionally configured seem harder to predict because they respond to predictions and change with that response.

    Let’s assume the balls are placed by an opponent who is rewarded when we predict the wrong order of the balls, and that we now are locked in a prediction game — here the nature of the prediction is dependent on the legibility of our actions by an opponent. Or many opponents!

    Predictions can, then, be agonistic. They can be adversarial games where we are trying to predict what someone else predicts and then act accordingly. These kinds of predictions seem different, because they include the prediction as a part of the system predicted, and need to find some kind of depth to make sense. It is easy to imagine a world in which I predict that you predict that I predict…and so on, but time sets natural boundaries to the prediction depth we deal with (although there is a fun short story here about two eternal and omnipotent beings trying to play chess, predicting endlessly what the other would do and so never playing that first move — and this raises the question of if there is an end to such reciprocal prediction loops or if they are truly infinite in some sense (how infinite can an infinite regress really be?).

    This is, of course, a kind of prediction game, and we should recognise that most prediction games will be n-person and not just 2–person games. This adds another layer of complexity that enriches the question of what we need to work with.

    We could say that predictions are exogenous to some systems and endogenous to others – in that predicting some systems change them, but predicting others do not. If I communicate my prediction about the stock market this prediction is absorbed to some degree determined by my credibility etc. But if I predict the weather or the solar system, the predicted system does not change because of my prediction.

    This gives us a second dimension or category to work with:

    (II) Competition – n-person predictions vs single person predictions (akin to game theory / decision theory).

    Predictions can be high or low resolution, they can be games or decisions. Are there other categories here we should explore? We could look at things like temporal dimensions (but that is sort of included in the first version), complexity of the system predicted (hinting at the fact that some systems are chaotic and some deterministic and some in between in a state of uncertainty of some kind) — but for now these two dimensions seem to capture something important.

    The issue of system complexity – and the resulting predictability – is interesting. It raises issues like if the stock market is more or less complex than the weather because of its ability to react to the predictions made about it – or if this actually can make it more stable in some ways?

    So, we would say that self-reflexive systems are more complex in some way than non-reflexive system, but is that necessarily a complexity that make the former harder to predict? Our folk-psychology says no: this is why we have the concept of the self-fulfilling prophecy, and this is why my prediction of other people’s behavior and norms will necessarily make me act in conformance and so further make those norms likely in the overall system.

    Something to get back to.

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Towards a philosophy of prediction I: the logic of predictions as goods

    +

    +

    +

    +

    +

    +

    One way to think about how artificial intelligence changes society is to do what Agrawal et al do in their excellent book Prediction Machines: they assume that the price of predictions, quality held constant, will converge to zero. If we simplify: predictions will become cheaper.

    There is a problem with this view, however, and that is that the idea is that predictions can be thought of as goods that are in some way independent of each-other – but any prediction of any sufficiently complex system is a prediction of others and their predictions, as well.

    So the quality and price of a prediction depends on the number of other predictions and the quality of them as well — this in turn suggests that we need to price predictions according to some other model. But how?

    What is a prediction worth? The simplest answer to this would be to say something that a prediction should be priced to represent the possible profit I can make from acting on that prediction, somehow discounted relative to the probability of the prediction, say.

    If that is the case, however, the price of the prediction is dependent on the availability of that prediction to other actors as well — and the likelihood that I can act before they do. A bit like insider trading, in a way – when the information become common knowledge it is priced into the market and so has no value at all.

    So does this mean that the price of predictions is simply the same thing as the price of any information? That does not seem obviously wrong: information can be priced according to the gains we can make by acting on that information over time — but, of course, such information can also be about the past, right? Or maybe not: for us to be able to act on a piece of information, it needs to say something about a future state of the world where there is some value to having the information about the past.

    A simple example shows this for us: let’s assume that I live in the Roman Empire, and I have received word that our forces have been defeated at a battle, and so access to grain will be cut off. I am the only one that knows this so I can act on it and buy up all the grain available at prices minus this information, and I know that it is highly likely that given this information the price of grain will go up — what I have done then is to use an information asymmetry in order to make a profit.

    This is textbook stuff. Does this mean, then, that predictions really are just information asymmetries and what we should be looking at is how the price dynamics of information asymmetries change with an abundance of predictions made at cheaper prices?

    If nothing else this provides us with a first thread to pull on and explore further.

    So – what we are saying is that there is some volume of information asymmetries in the current market, and that we expect this volume of information asymmetries to change and increase in some way with AI. What does this mean for economies overall?

    A challenge with predictions here is that they can be nested in regresses of different kinds. If I predict a shortage of grain, and you predict that I predict a shortage of grain and then you predict that I predict that you predict — what do we then do? Here it seems to depend on the ability to act that we have — if I predict you predict the same as me, but you have better abilities to act on the prediction, well, then I should turn elsewhere for interesting moves on the economic board (assuming it is not zero-sum).

    But where I predict there is a chance to act on predictions that I know you are aware of as well I should act faster, and more at scale. This suggests that an increase in information asymmetries in an economy should drive some kind of volatility at least.

    And this is an interesting overall observation that may come from all of this preliminary exploration: while we thought that predicting the world might make it more stable, it may in fact make the world harder to predict as it become more volatile. The price of predictions then should go up and we should end up with a U-shaped curve as predictions become more available when it comes to their prices?

    TBC

    +

    +

    +

    +

    +

    +

    + ,

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Life is its own metaphor

    +

    +

    +

    +

    +

    +

    In this recent essay, Philip Ball makes a series of important observations in the service of an argument for ditching mechanical or machine metaphors when describing biology. We misunderstand life if we seek analogies outside of life. One example he cites is our understanding of the genome, and here I really found myself guilty of what he describes: thinking about the genome as a database that is read and then copied and…we have all used that language when we describe genetics, but Ball cites Barbara McClintock1 Her Nobel speech is worth reading in full, here: https://www.nobelprize.org/uploads/2018/06/mcclintock-lecture.pdf , and makes the point crisp and clear:

    So how now should we be speaking about biology? Keller herself tentatively suggested that we might adopt the prescient suggestion of the Nobel laureate biologist Barbara McClintock in recognising that the genome is a responsive, reactive system, not some passive data bank: as McClintock called it, a ‘highly sensitive organ of the cell’.

    There’s virtue in that picture, but I think it points to a wider consideration: that the best narratives and metaphors for thinking about how life works come not from our technologies (machines, computers) but from life itself. Some biologists now argue that we should think of all living systems, from single cells upwards, not as mechanical contraptions but as cognitive agents, capable of sifting and integrating information against the backdrop of their own internal states in order to achieve some self-determined goal. Our biomolecules appear to make decisions not in the manner of on/off switches but in loosely defined committees that obey a combinatorial logic, comparable to the way different combinations of just a few light-sensitive cells or olfactory receptor molecules can generate countless sensations of colour or smell. The ‘organic technology’ of language, where meaning arises through context and cannot be atomised into component parts, is a constantly useful analogy. Life must be its own metaphor.

    This puts agency at the heart of the discussion about not just life, but also intelligence – and so asks some profound questions about the metaphors that govern our research into artificial intelligence and machine learning.

    But do metaphors really matter? Is this not just philosophical woowoo that is spouted by those who cannot build and code? It is easy to dismiss philosophical analysis of technology as an armchair hobby, but I do think that such a dismissal would be a bad mistake. Our choice of metaphor really determines the possible design space for what we do to a much higher degree than we usually recognize, and this is true for the mind as well as for any other phenomenon we want to understand better.

    It is enough to imagine that we were forced to explore the mind in any of the earlier metaphors we used for it – like windmills or electrical circuits or telephone switchboard – to realize that the boundaries of our design space are the boundaries of our metaphors.2 Today’s different metaphors include mind as brain, mind as computer and mind as rhizome – see Schuh, K.L. and Cunningham, D.J., 2004. “Rhizome and the mind: Describing the metaphor.” The idea that we should describe life as agency, and then intelligence as a feature of life – and so hence dependent on agency – is one that is floating around in many different formats right now, and I do think that there is a lot to be said for exploring this perspective more deeply in order to understand how agency fits into the larger understanding of what we call intelligence.

    Ball concludes:

    And shouldn’t we have seen that all along? For what, after all, is extraordinary – and challenging to scientific description – about living matter is not its molecules but its aliveness, its agency. It seems odd to have to say this, but it’s time for a biology that is life-centric.

    Footnotes and references

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Reading Kurzweil: singularities, exponentials and logistic curves

    +

    +

    +

    +

    +

    +

    Having just finished Kurzweil’s latest, interesting , book The Singularity Is Nearer (2024) I am still left with some unease about the argument in it, and I think it has to do with something very simple – baselines. When ever we are asked to predict something, Tetlock teaches us to start out thinking about baselines, the knowledge we have about different kinds of precedents and patterns that we already know. If I am asked to predict if a certain bill will pass in Congress in the US, I should start from looking at what percentage of introduced bills are actually passed, and then adjust as I fermize the problem into a lot of different factors.1 A nice start can be made here https://www.govtrack.us/congress/bills/statistics

    So, if I am asked if I believe that a certain curve that is currently exponential will continue to be exponential or develop into a logistic curve, or an S-curve, I should ask what frequency of exponential curves have turned out to be persistently exponential and what frequency have turned out to be logistic. And this is where it is really hard to find any kind of evidence for exponential curves not turning into logistic ones. So the baseline expectation we should have for the prediction of if a particular exponential curve will turn logistic should be, well, 100%.

    But, wait, you may say — that is not what Kurzweil argues at all — he does not say that the exponential change we see in an individual technology will continue forever. He is arguing that it will generate a new technology, and that will in turn create an exponential change and so on. What he is really trying to do is to predict the pattern of logistic curves of change, suggesting that they themselves map out a kind of macro-curve that is, ultimately, exponential.

    And this curve is unlike the individual development curves for specific technologies, because it is a curve for technology in general. But, then, it is still an exponential curve, and we are still being asked to predict it — and so why should we not argue that even for this aggregated exponential the baseline expectation should be that it turns logistic rather than continues exponentially into the future?

    In some ways, Kurzweil’s argument strikes me as an argument against limits – and this may be how we have to understand it: what is the probability that there are no limits for the overall development of technological capability in society? Here it feels like Vaclav Smil would be exploding, eager to get a word in — and I would love to see a discussion between the two of them – but that may be long coming.2 Smil offers a rebuttal here that is quite interesting https://www.wsj.com/articles/tech-progress-is-slowing-down-b7fcaee0 — and one way to think of Smil is as the philosopher of limits. The notion of the limit is deep and ill-explored in general, and something that we should spend more time on, I suspect.

    All in all there is a lot here, and the book is really well-researched and argued, and I do sympathize with Kurzweil’s insistent reminders of the fact that things are getting better, but the argument feels unfinished.

    Footnotes and references

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • How do we decide how much energy to spend on AI?

    +

    +

    +

    +

    +

    +

    There has been a lot1 See for example: https://www.theverge.com/24066646/ai-electricity-energy-watts-generative-consumption and https://www.weforum.org/agenda/2024/04/how-to-manage-ais-energy-demand-today-tomorrow-and-in-the-future/ and https://theconversation.com/ai-supercharges-data-center-energy-use-straining-the-grid-and-slowing-sustainability-efforts-232697 – rightly – written about how much energy AI will consume and how this affects everything from climate commitments to carbon footprints of the information society. This question is real, and requires some structured thinking – and what I have been missing in the reporting so far is a closer examination of how we should make decisions around resource use for technological development and progress. My purpose here is not to recommend a solution, but rather to try to figure out what different decision criteria we could apply to make decisions about future investments in AI and the resultant energy use.

    The perhaps simplest and most straightforward answer is that we should not produce any technology that increases our carbon footprint at all. Let’s call this the “absolute minimization principle”, where all decisions around the design, development and use of technology are made simply by looking at if the present use increases or decreases climate impact. If a technology replaces another more energy intense technology then we can use it, if it does not we do not use or deploy it – and we check for possible increases in the use of whatever functionality the technology underpins to make sure that we do not fool ourselves by replacing a more energy intense technology with a more efficient one, but instead increasing the overall use in such a way as to – in absolute terms – increase energy use.

    This principle does away with what almost all other decision criteria will have to deal with: the question of if present energy use can reduce future energy needs, and if the sum total impact over a particular time period is good or bad. Under this principle, very little of the AI we are building out today would be allowed at all — since we see increases in energy use from almost every company working on this technology.

    What we have seen, instead, is a second approach where almost all arguments are of the form:

    (i) We should develop and use technology T if the net effect on the climate over time t is positive.

    Let’s say we our current development and use increases the climate impact by E, but we can reduce that impact in 10 years by 2E – well then we should allow for investing in the technology.

    The challenge here is that we cannot say with absolute certainty that we will see this effect, so we have to assess the probability that there will be a net positive effect before we invest, and that is tricky. We also have to figure out exactly how to assess the climate impact we are looking at. Is it carbon footprints? Is it a compound measure looking at possible increased risk from nuclear power and other energy sources? Should we require that we only invest in the technology if the energy used is – by some definition – green and renewable? We could imagine variations on (i) that take all of these different factors into account, and then try to assess the probability and at some expected positive effect on the climate allow for the investment in the technology, and this is the approach that most people seem to center around today, more or less.

    There are also questions about the alternative use of the energy at play; what if there is a need to allocate energy use to either developing technology or, say, provide air conditioning to areas affected by extreme weather? Now, energy is not really fungible in that way (so that you can choose where in the world to use it at any moment in time), but some allocation alternatives always exist, and so the alternative uses also may factor into our discussion. This could expand the principle to something like:

    (ii) We should develop and use technology T at energy use E if the net effect on fairness, climate and other key human values is positive with some probability P over time t.

    We then need to develop some way of assessing the open variables in this equation, and try to figure out what the boundaries should be. Should we allow for smaller probabilities for potentially great effects on the different factors we are checking for? Look at the case of fusion – what if AI helps us solve the problem of fusion and we gain access to a clean, renewable and cheap energy source for the world? Should we, even with a very slight probability of this happening, allow for using enormous amounts of energy?

    This second approach to the decision making is an attempt at assessing costs and benefits and trying to sort out if we can defend the energy use on the basis of predictions about the future. But we could also explore a third approach to this decision, and it could look something like this:

    (iii) We will allocate x percent of energy to experiments with technology that may be transformative, in order to ensure explore the frontiers of knowledge and science.

    Here we set an upper cap on how much energy can be used for different kinds of bets on the future. As long as the overall energy use, and climate impact, is within that cap we allow for it to proceed. Of the amount of competing bets sum up to something above cap we can allocate the energy use here by reverting back to the cost / benefit principle or – if we think our ability to do that assessment is weak at best – we randomize allocation.

    There are many other possible approaches here, but a meta-discussion like this, about how a decision like this should be made, is often a helpful way of thinking through a challenge, and it seems to me that this is what we need in this particular case. The energy use in developing and using AI will be significant, and how we approach the question of if that can be defended needs to be well-structured.

    Finally, if we believe that what we are building is a cognitive infrastructure for the world, we could ask a very different question: how much of a civilization’s overall energy access should be used by such a cognitive infrastructure?

    The human brain consumes 20 percent of the energy we need everyday2See eg. https://qbi.uq.edu.au/brain/discovery-science/how-your-brain-makes-and-uses-energy#:~:text=Your%20brain%20is%20arguably%20the,our%20food%20into%20simple%20sugars. — and if we use the age-old philosophical analogy of the body to the state, well, then made around 20 percent of energy is what we should expect to use for cognition and intelligence? If not, why not? How much energy would you invest in problem solving, science, intelligence, cognition if you were designing a society behind a rawlsian veil of ignorance?

    How much energy should a civilization use for thinking?

    Footnotes and references

    +

    +

    +

    +

    +

    +

    +
    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • On cognitive infrastructure

    +

    +

    +

    +

    +

    +

    Introduction

    Here is an interesting policy exercise: assume that you had to talk about artificial intelligence and machine learning, and how it will change the way we work, do research, learn and so on in society – but you could not use the concept of “artificial intelligence” at all, but had to speak about the coming technological, possible transformation more broadly — what would you do then? This short note suggests that one way of thinking about this is to think about it as the building of a cognitive infrastructure in our society.

    A cognitive stack

    The idea of a cognitive infrastructure as a key element in our society could build on the old distinction between data, information, knowledge and wisdom — but we would shift the last layer here, and call it decisions instead. The reason for this is that this last layer is an activity – a flow – rather than a stock. Data, information and knowledge are all there to help us make decisions and ultimately to learn about the world we are in. And our decisions are a part of an adapative pattern, and allows us to deal with the complexity that we have created in pursuit of progress and welfare.

    Cognitive infrastructure, then, is the kind of infrastructure that underpins this stack. It covers practices for collecting and curating data, structuring and interpreting it, interpreting and learning from information, and making decisions with the resulting knowledge. At all layers in the stack, there are infrastructure investments needed, and an organization or a country will only be as good as its cognitive infrastructure allows it to be.

    Investing in cognitive infrastructure: data

    If we want to make public investments in our cognitive infrastructure, we need to ensure that we do not treat this as something separate to our overall social and economic activity. It becomes important to ensure that we really embed this stack thinking in all of our policy decisions.

    For data that could mean that just as we present consequence analysis for the environment or entrepreneurs with all new policy proposals, we should make sure that all public investments and policy proposals also come with a plan for the collection and curation of data. This will ensure that we build data in from the beginning, designing with the data we may need – and hence also with the kind of information, knowledge and decisions we want to make – from the start. This – learning by design – is a key stance that we all – whether public sector or companies – benefit from taking.

    This is where we figure out what the statistics agencies of the future should look like – and if they should be stand-alone agencies or embedded functions in all public bodies and institutions.

    Investing in cognitive infrastructure: information

    The next layer is information – here we are concerned with the structuring of data into information that can be interpreted and learned from. This is where we need to ensure that there are standards, structures and research into how we best structured data. Benchmarks, tests and long term curation of the data also goes here.

    Traditionally this is where we have had libraries, and where we now need to build next generation libraries and invest in educating librarians, experts in structuring data.

    Investing in cognitive infrastructure: knowledge

    Knowledge is the product of learning, and when we are trying to build out this layer we want to make sure that all our of institutions, organizations and groups have learning practices in place. Actively collecting and distilling what we have learned requires also setting up experiment infrastructures that allow us to test the knowledge we think we have. In many ways this is about building out science to a socially shared and participatory activity.

    Investing in cognitive infrastructure: decision making

    The final layer is one in which we document, follow up and analyze our decisions – and apply all the knowledge produced in the cognitive infrastructure to ensure that we make decisions that are the best possible given the time / data and resources we believe a decision merits. Decision making is in many ways at the heart of our societies and for far too long decision making has been left to hunches and intuition only (even if these continue to play a role in decision making under different kinds of uncertainty.

    Conclusions

    As this quick sketch suggests, the notion of a cognitive infrastructure may help us think more about what we want to use technology for, rather than focus on the technology itself. It can be built out and explored more in detail – of course – but there is a thread here to pull on.

    +

    +

    +

    +

    +

    +

    + ,
    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Using GenAI as a legal tool

    +

    +

    +

    +

    +

    +

    One of the key use cases for generative AI that has been identified by observers and analysts is in legal work. There are many different reasons for this — legal analysis is rule-based, legal texts are often well-structured and there are well-defined corpora of texts that should be taken into account when working on a legal problem of some kind. 1 See for example Stokel-Walker, Chis “Generative AI is coming for the lawyers” Wired 2023.02.21.

    Some have argued that generative AI may fundamentally transform the law as a practice and institution2 See “Generative AI could radically alter the practice of law” The Economist June 2023. and some official bodies like the European Commission for the Efficiency of Justice have started to grapple with the challenges that the use of these tools generate. In the information note “Use of Generative Artificial Intelligence (AI) by judicial professionals in a workrelated context” published this year, we find some pretty solid advice for the use of generative AI overall:

    D. How should it be applied?

    1. Make sure that the tool’s use is authorised and appropriate for the desired purpose.
    2. Bear in mind that it is only a tool and try to understand how it works (be aware of human cognitive biases).
    3. Give preference to systems that have been trained on certified and official data, the list of which is known, to limit the risks of bias, hallucination, and copyright infringement.
    4. Give the tool clear instructions (prompts) about what is expected of it. It is through conversation that the machine will obtain the instructions it needs, so do not hesitate to engage it, unlike a search engine. Asking for clarification or even refining or modifying the request is possible. For example, give the machine a context (country, period of time), define the task (e.g. write a summary in xx words), specify who the output is intended for, how it is to be produced and the tone the tool should adopt, ask for a specific presentation format, check that the instructions have been properly understood (by asking the machine rephrasing them), provide examples of the answers expected for similar questions to enable the tool to imitate their form and style.
    5. Enter only non-sensitive data and information which is already available in the public domain.
    6. Always check the correctness of the answers, even in case references are given (especially check the existence of the reference).
    7. Be transparent and always indicate if an analysis or content was generated by generative AI.
    8. Reformulate the generated text in case it shall feed into official and/or legal documents.
    9. Remain in control of your choice and the decision-making process and take a critical look at the made proposals.

    These recommendations are quite good and thoughtful — the only thing missing here is possibly the use of generative AI to challenge your own views and provide a second and third opinion on what you have written yourself. Adding this would make this list a solid checklist for anyone engaging with these tools in their day-to-day. Even the list of things not to do is quite good:

    E. When should it not be applied?

    1. In case you are not aware of, do not understand or do not agree to the terms and conditions of use.
    2. In case it is forbidden/against your organisational regulations.
    3. In case you cannot assess the result for factual correctness and bias.
    4. In case you would be required to enter and thus disclose personal, confidential, copyright protected or otherwise sensitive data.
    5. In case you must know how your answer was derived.
    6. In case you are expected to produce a genuinely self-derived answer

    The thing that stands out here are the two last recommendations – and I think they are confused. The idea of a genuinely self-derived answer seems close to the medieval epistemological category of “illumination” where knowledge just appears in the writers mind by divine intervention. It is unlikely that there is anything that is genuinely self-derived at all; we all depend on sources and education and references of different kinds. And the trick here is to realize that these tools are not information retrieval tools — they do not derive answers nor do they produce search results in the same way that a classical search engine does — they are more like improvisation tools, suggesting possible variations on a theme, new ideas and holes in your argument. Think about them as a discussion partner of sorts.

    If used in a good way, these tools will significantly increase the robustness and speed of the legal system – but law still needs to be practice by (human) hand. Not least – as shown by Irina Carnat in a recent paper – because of accountability issues: Carnat, in her conclusion, puts it well:3 See Carnat, Irina, Addressing the Risks of Generative AI for the Judiciary: The Accountability Framework(s) Under the EU AI Act. Available at SSRN: https://ssrn.com/abstract=4887438 or http://dx.doi.org/10.2139/ssrn.4887438

    The recent leap in the state of the art of NLP brought about by the development of generative LLMs have an intuitive appeal for the legal profession, including the prospect of using them to assist judicial decision-making. While these advancements present new frontiers for legal automation, the risks of overreliance on the algorithmic output pose significant concerns in terms of accountability.

    The analysis in this paper has highlighted how the accountability gap identified in prior instances of AI-driven judicial decision-making, such as the COMPAS case, persists despite the technological shift towards more advanced generative models. Factors like model opaqueness, their proprietary nature, lack of knowledge of the system’s functioning and accuracy, continue to hinder the human decision-maker’s understanding of the system’s functioning, leading to inappropriate reliance on the AI’s outputs.

    To address these risks, this paper has examined the regulatory framework established by the EU
    Artificial Intelligence Act. The risk-based approach of the regulation, including the specific provisions for high-risk AI systems deployed in the justice domain, provides a foundation for developing an accountability framework. Key elements of this framework include the requirements for human oversight, clear delineation of roles and responsibilities along the AI value chain, and the critical importance of AI literacy among all stakeholders involved in the deployment and use of these systems.

    Ultimately, the successful and responsible integration of generative LLMs in judicial decision-making will require a comprehensive approach that goes beyond technical considerations. It must address the cognitive biases, ensuring that human decision-makers maintain active agency and control over the process, rather than becoming passive overseers of algorithmic outputs. By emphasizing the shared responsibility across the AI value chain and the imperative of AI literacy, the regulatory framework can help mitigate the risks of automation bias and preserve the fundamental principles of due process and the rule of law.

    The one issue I have with Carnat’s conclusion is that she assumes that this accountability is robust today. The real question about the use of any new tool in legal practice should be if it – relative to the existing state of affairs – improves things or not. There are always risks associated with any tool, but are they greater or less than the system as it works today? The ability to use generative AI to challenge early drafts of judicial decisions might well suffer from a deficit in accountability, but what accountability do we have in today’s system?

    This is why proposals such as Orly Lobel’s about a right to automated decision making4 See Lobel’s paper: Lobel, Orly, The Law of AI for Good (January 26, 2023). San Diego Legal Studies Paper No. 23-001, Available at SSRN: https://ssrn.com/abstract=4338862 or http://dx.doi.org/10.2139/ssrn.4338862 ., even in judicial matters, are so intriguing. It is not that the machine will not be biased, make mistakes or sometimes even hallucinate — it is more about the frequency with which it does that, and the comparable frequency of similar mistakes made by today’s system.

    My prediction for this would be that in 30 years, not testing an opinion, a contract or a legal document in an LLM or its descendant AI will be considered malpractice.

    Footnotes and references

    +

    +

    +

    +

    +

    +

    +
    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Metaphor, sentience and technology

    +

    +

    +

    +

    +

    +

    In this recent paper the authors discuss public attitudes to technology as sentient or conscious. The overall take away is well summarized in the abstract:

    Future developments in AI capabilities and safety will depend on public opinion and human-AI interaction. To begin to fill this research gap, we present the first nationally representative survey data on the topic of sentient AI: initial results from the Artificial Intelligence, Morality, and Sentience (AIMS) survey, a preregistered and longitudinal study of U.S. public opinion that began in 2021. Across one wave of data collection in 2021 and two in 2023 (total N = 3,500), we found mind perception and moral concern for AI well-being in 2021 were higher than predicted and significantly increased in 2023: for example, 71% agree sentient AI deserve to be treated with respect, and 38% support legal rights. People have become more threatened by AI, and there is widespread opposition to new technologies: 63% support a ban on smarter-thanhuman AI, and 69% support a ban on sentient AI. Expected timelines are surprisingly short and shortening with a median forecast of sentient AI in only five years and artificial general intelligence in only two years.

    The findings here are difficult to parse, since what we are discussing is so unclear – but it is still really valuable, as it shows how the design metaphors of predictive technologies create challenges merely by virtue of being metaphors of mind, rather than metaphors of matter.

    Any technology comes with a user interface metaphor.1 See e.g. Neale, D.C. and Carroll, J.M., 1997. “The role of metaphors in user interface design.” In Handbook of human-computer interaction (pp. 441-462). North-Holland. The simplest example of this in computer software may be the operating systems and “windows”. We open and close windows, re-arrange them and work within them. The Internet came with its own metaphors of links, pages, sites etc and we quickly absorbed that new framing as well. With social media things became more troubling since we started to use more intentional metaphorical language such as “likes” for expressing some kind of sentiment about a post or “friends” for social contacts — and here the visibility of the metaphorical frame was radically reduced.

    With AI the UI-metaphor is almost entirely built on intentionality and intentional actions such as wanting, thinking, answering, hallucinating etc — and we speak of predictive models as we would of other intentional systems, including ourselves.

    This is not a problem in itself, it is a handy set of metaphors and we can use them to make sense of the technology faster than if we had to explain what is happening in a more mechanical way – but it does become a problem when we end up assuming that the metaphorical framing is real, and asking questions that are out of frame.2 In a really early note on metaphors in programming, Alan Cooper noted “The biggest problem is that by representing old technology, metaphors firmly nail our conceptual feet to the ground, forever limiting the power of our software. They have a host of other problems as well, including the simple fact that there aren’t enough metaphors to go around, they don’t scale well, and the ability of users to recognize them is questionable.” Cooper, A., 1995. “The myth of metaphor.” Visual Basic Programmer’s Journal, 3, pp.127-128.

    If we did this for operating systems everyone would react – asking if a window is clean when you are referring to a window in an operating system makes no sense, and someone who said they would spend Sunday cleaning their operating system windows would essentially just have explained that they did not quite understand the UI metaphor well enough to know that the windows do not need to be cleaned (although the screen may, and there is a point here that we should return to in a later post – metaphors are layered).

    The problem here is an interesting one: how do we determine the metaphorical frame and where it breaks? What if I claimed the following:

    (i) Saying that we should not build sentient AI is like saying that we need to periodically clean our Microsoft OS windows. It is a metaphorical mistake.

    How would you react to that? It would depend on how you think about the technology interface and the underlying technology. If you believe that what is going on in the technology replicates what is going on in the mind, and hence that it should be possible to also build sentience you would say that I have it wrong, and that the question of whether to build sentient AI is a real question we need to resolve (unlike the question of if we need to clean our Microsoft windows).

    But if you believe that the technology itself is radically different you would say that is roughly right — and that we need to re-think how we clarify the UI metaphor so that we do not get stuck in questions like this.

    It is important to note here, by the way, that there is no argument about the complexity of the technology — we may be able to build something that is to sentience as predictive technologies are to human intelligence, and we do not need to argue that there is a limit or boundary here that cannot be crossed. In fact, the skepticism about our ability to build AGI is a skepticism that operates within the UI metaphor, accepting its extension in a way that obscures the better question of what it is that we are really building. 3 It is a tired observation to note that we would be better off if we did not call it “artificial intelligence” but something more neutral, but I think that observation misses the fact that we are building this with an intention to mirror ourselves, and that is a key property of the technology that results — but to some degree a different name would have helped us not to get stuck so quickly in the UI-metaphor.

    The study takes the interesting perspective that this is partly, if not wholly, about perception and what they are interested in is if people believe that we should clean our Microsoft Windows or not — not whether that is a metaphor mistake. That is a useful approach, since it allows us – if we do indeed think it is a metaphor mistake this gives us a chance to think through the extent to which this mistake is made. 4 The study, in this respect, could work like one of Kahneman’s and Tversky’s studies charting the kinds of breakdowns of the rational mind that we have evolved to be vulnerable to.

    And it seems – from the numbers cited above – that the severity of the UI metaphor confusion is high.

    There are probably many reasons for this, including that we have evolved to apply intentional stances and frames to things that behave minimally intentionally since that is, from a cognitive economy standpoint, extremely efficient. Just as we have evolved to predict narratives and not propositions when it comes to probabilities. But we should be careful not to get stuck in the ease of the metaphor as the key criterion for its applicability — if we agree that it is still a metaphor.

    If we don’t – and we think that we are building intelligence and that we can build sentience and consciousness – we need to discuss why we believe that the UI metaphor (and some of the design metaphors like neural networks) are really not metaphors at all, but descriptions of the real thing.

    This is where things get interesting, and there is a lot more to be said here – not least about the differences between the simulated and the artificial. This is a subject I hope to return to, in a later post or essay.

    Read more in “What Do People Think about Sentient AI?” by Jacy Reese Anthis, Janet V.T. Pauketat, Ali Ladak, Aikaterina Manoli .

    Footnotes and references

    • 1
      See e.g. Neale, D.C. and Carroll, J.M., 1997. “The role of metaphors in user interface design.” In Handbook of human-computer interaction (pp. 441-462). North-Holland.
    • 2
      In a really early note on metaphors in programming, Alan Cooper noted “The biggest problem is that by representing old technology, metaphors firmly nail our conceptual feet to the ground, forever limiting the power of our software. They have a host of other problems as well, including the simple fact that there aren’t enough metaphors to go around, they don’t scale well, and the ability of users to recognize them is questionable.” Cooper, A., 1995. “The myth of metaphor.” Visual Basic Programmer’s Journal, 3, pp.127-128.
    • 3
      It is a tired observation to note that we would be better off if we did not call it “artificial intelligence” but something more neutral, but I think that observation misses the fact that we are building this with an intention to mirror ourselves, and that is a key property of the technology that results — but to some degree a different name would have helped us not to get stuck so quickly in the UI-metaphor.
    • 4
      The study, in this respect, could work like one of Kahneman’s and Tversky’s studies charting the kinds of breakdowns of the rational mind that we have evolved to be vulnerable to.

    +

    +

    +

    +

    +

    +

    +
    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

  • Falsifiability in Evaluating Language Models: Insights from Recent Research

    +

    +

    +

    +

    +

    +

    A recent paper seems to have found something interesting — here is the tl;dr:

    Taking inspiration from those problems and aiming for even simpler settings, we arrived at a very simple problem template that can be easily solved using common sense reasoning but is not entirely straightforward, of the following form: “Alice has N brothers and she also has M sisters. How many sisters does Alice’s brother have?”. The problem – we call it here “AIW problem” – has a
    simple common sense solution which assumes all sisters and brothers in the problem setting share the same parents. The correct response C – number of sisters – is easily obtained by calculating M + 1 (Alice and her sisters), which immediately gives the number of sisters Alice’s brother has.


    Initially, we assumed that the AIW problem will pose no challenge for most of the current state-of-the-art LLMs. In our initial experiments, we started by looking at the problem instance with N = 4 and M = 1, with correct the response being C = 2 sisters and we noticed to our surprise that on the contrary, most state-of-the-art models struggle severely to provide correct responses when confronted with the problem. We also noticed already during early experimentation that even slight variations of numbers N and M may cause strong fluctuations of correct response rate.

    This mirrors a research approach that we have seen in many cases with LLMs: find some kind of prompt that gives a response that the model cannot handle, us that to make a claim about the overall capability of models and then suggest better benchmarks. This is not a bad approach as such, as it ensures that we are not blinded by benchmarks, but it raises the question of how we evaluate single prompt approaches overall.

    If the claim is something like:

    (i) There is a prompt P that model M does not handle well, hence model M is not as good as benchmarks B(1)…B(N) suggest.

    We need to think about if we think about the validity of the claim as such. And if we demand more prompts so that there is a class of prompts that a model does not handle well, we need to determine how large such a class needs to be to be considered significant. In a way this is just about regular, basic theory of science questions and if a “counterprompt” can be seen as the same as a falsification of something, and if that is the case, what it is a falsification of. The authors refer to Popper in their conclusion:

    Facing these initial findings, we would like to call upon scientific and technological ML community to work together on providing necessary updates of current LLM benchmarks that obviously fail to discover important weaknesses and differences between the studied models. Such updates might feature sets of problems similar to AIW studied here – simple, to probe specific kind of reasoning deficiency, but customizable, thus offering enough combinatorial variety to provide robustness against potential contamination via memorization. We think that strong, trustful benchmarks should follow Karl Popper’s principle of falsifiability [55] – not trying to confirm and highlight model’s capabilities, which is tempting especially in commercial setting, but in contrast do everything to break model’s function, highlighting its deficits, and thus showing possible ways for model improvement, which is the way of scientific method.

    The claim (i) seems to suggest that a single prompt P, or class of prompts, can falsify all benchmarks in some dimension (or completely, if we take a strong stance). This seems a bit unreasonable, but still interesting to explore. What we seem to be lacking here is a theory of evidence / significance / falsification in relation to evaluations and benchmarks, and this in turn seems to say something about the complexity of assessing general capability in a model. A stronger version of claim (i) could be something more general like:

    (ii) If there is a prompt P such that a model M fails at it AND this problem is easily solved by a human being, this means the model should be considered falsified as a whole.

    This obviously seems to strong, but remains intriguing to think about. What is it that we are falsifying here? Is it the claim of generality? And to what degree is a benchmark, or the use of multiple benchmarks, a claim to generality? I think the authors are right to point out that using multiple benchmarks and tests risks being interepreted as suggesting there is a general capability G that we test alongside the specific capabilities S(1)…S(N) a benchmark focuses on. But this may not be true at all, and raises the question of if there is any benchmark or basket of tests that can test for G here.

    At the heart of this lies the question of what it means to falsify a model, I think. This is where the paper’s reference to Popper and scientific method feels like a tease, not exploring that full theory of model falsification – if indeed there is such a thing.

    Old hat, in a sense, as it connects to the discussion of how you would test for AGI — but still. Interesting. And worthwhile read. See Nezhurina, M., Cipolina-Kun, L., Cherti, M. and Jitsev, J., 2024. Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models. arXiv preprint arXiv:2406.02061.

    +

    +

    +

    +

    +

    +

    +
    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +

    +