Reading with bots

28 July, 2023

+

It seems obvious that anyone who is not using generative AI in their day job is missing out. There is a wealth of interesting opportunities, and new ways of working that need to be explored and analysed. What I have been playing around with lately is what it means to read with the help of bots.

I try to read a lot. I like pure human reading on paper, but am also aware that I need to develop different reading skills to consume information at the rate and depth that I want to. So, I have since long also used skimming in some cases – reading the first line in each paragraph in a book, and if not surprised continue until I do not understand the argument, then slow down. I do this when I am not sure a book is worth deep reading, and sometimes I am right, sometimes I am wrong and have to shift reading modes.

Bot reading allows for a new kind of reading, that can be very effective – depending on how you prompt the reading. Here is my current favourite prompting structure:

Ask for a summary in 5-10 points depending on how interesting the thing is — and specifically ask for summary of argument and analysis separately.
Ask for a criticism in 3-5 points to understand the weaknesses of the paper or article.
Ask for any novel or surprising items – again 3-5.

Doing this in different variations allows you to approach texts in a multistep evaluation: first, if the result from the reading prompt is enough you can stop there (occasionally verifying that the bot did its job well – of course, but one in 20 papers perhaps), second – if the summary is intriguing enough continue interrogating the paper.

Ask for clarification of the criticisms and how the author could meet them.
Ask for X’s perspective (“What would a virtue ethicist say about this?”)
Ask for related ideas or papers.

Overall this then gives you a sense of if the paper should be the subject of deep reading.

Now, I am the first to admit that this feels like cheating. But the alternative is not having the time to read these articles or papers at all, since there is finite time in the week — so this at least gives a deeper situational awareness and opens the mind to detect trends and patterns in the discourse you are interested in.

The trick, I find, is to think about this as really interrogating or talking with the paper in different ways; building it into a conversation.

Here an example of the first stage with Brooking’s recent article on cities and AI (I used Claude 2 for this — another trick is to do this with different bots):

Here is a 10 point summary of the article “Building AI cities: How to spread the benefits of an emerging technology across more of America”, along with 5 criticisms and 3 novel/surprising points:

Summary (10 points):

AI activity is heavily concentrated in a few “superstar” tech hubs like SF and Seattle.
Generative AI job postings also appear highly concentrated so far in early adopter cities.
This concentration risks limiting innovation, economic opportunity, and accessible AI applications.
Market forces alone are unlikely to decentralize AI geography. Intentional policy is needed.
The federal government should expand R&D funding with geographic distribution mandates.
Access to computing power and data needs to be democratized.
Workforce development programs should aim to build AI talent pipelines in new regions.
Federal grants can catalyze new regional AI clusters and ecosystems.
States can complement federal efforts through targeted R&D, data access, skills programs.
Bottom-up strategies tapping federal support can grow viable new AI hubs.

Criticisms (5 points):

It lacks data on the concentration of generative AI specifically. Claims remain speculative.
Proposed policies may spread AI thinly rather than build critical mass hubs.
It underestimates costs of decentralization and duplication of resources.
The roles of academia, industry, government are not clearly delineated.
Implementation challenges of coordinated multi-level governance are overlooked.

Novel/Surprising Points (3 points):

Nearly half of all generative AI job postings in 2022 were in just 6 metros.
AI job concentration increased over the past decade despite assumed diffusion.
Specific policy tools like the BBBRC and NAIRR are presented as decentralization vehicles.

+

+ Reading Notes

+

Invisible commons

27 July, 2023

+

Are there some commons that we should protect that we are just not aware of? Well, for the longest time the idea that there was a “climate” to protect was not obvious to people – and the overall discussion we now have about climate change required a concept to focus on, and climate ended up being the focal concept.

One way to think about the future is to think about possible focal concepts like this that have not been invented yet. Here are a few possible examples.

The Human Text – the sum total of all things written by human beings, not generated by AI. The value of this commons is still unclear to us (why is a text written by a human being in some way more worthy of protection than a generated text?) – but we may discover that there are some qualities in The Human Text that are unique for that corpus. This could of course be true for a lot of other forms of creative outputs too.
Social autonomy. The level of overall autonomy that we retain as a society, and the degree to which we are relying on systems to solve our problems for us. How many decisions are made by human beings and not by systems? This is a difficult one, since we usually look to the quality of the decision rather than its origin, but there may well be reasons to value human decision making not just as an artisanal quaint thing, but as better for different reasons.
Agency. Closely related – but more about the distribution of agency over systems, perhaps. Evolution creates behavioural complexity through pre-programming (spider’s web) or agency. We now have created a new, artificial agency – post-programming – and human agency may be squeezed in between these two forms of programming. This may make a lot of sense from evolution’s perspective – i.e. adaptive capability may increase when we revert to the ideal state of pre and post programming and avoid messy agency in the middle – but still is interesting to think about the value of agency in the middle. Dennett and others seem to argue that agency in the middle is what underpins almost all of our social institutions, even if it sometimes is a fiction. And maybe this connects to the first one: maybe there is a value in agency-produced text?
Fact. This is a common that we often are quite blind to – and feel uncomfortable articulating – but the depletion of facts is an interesting and hard problem to work on. Facts do – as Sam Arbesman points out – have a half-life, so there is a natural depletion – but beyond that depletion the rate at which we destroy this particular commons has to do with education, public sphere norms and many, many other social patterns. What if we started to treat fact as a commons? And yes – you could make the same argument for truth, and analyse the evolution of our relationship to truth as a commons problem.
Attention. While deeply individual, in some sense, attention is actually a commons we draw on to build society and democratic institutions. The way in which we manage this commons is determinative of the quality of democracy that we get.
Childhood. Do we still have childhoods like we used to? Is childhood a commons that underpins our civilisational growth and prosperity? What about the Global South / North divide in childhoods etc?

I am sure there are many other possible focal concepts as well. These three interest me more as examples than anything else — and the real prediction question is which focal concepts will seem obvious in the future? Animal life quality comes to mind as well, as one that many have guessed at – the idea that we eat animals may seem a lot like insulating with asbestos or torture in the future, they argue. That does not seem wholly unlikely to me, with synthetic meat on the horizon etc. These changes can be relatively swift – how long did it take for climate to emerge as an obvious commons? Arguably this took at most a decade, and then we can debate if it starts as “common heritage of mankind” or with the 1988 establishment of the IPCC, but still.

A lot of these things come back to a more fundamental commons that has remained invisible for a long time – since we believe it is deeply individual: time. As noted by many, the best measure of freedom is probably discretionary time, but that is not quite right. If you are free to use your time alone you may feel individually free – but are you really free in a more fundamental way? If there is not a certain pool or commons of shared discretionary time in a society, it falls apart. This is the argument for what we sloppily call leisure – and we probably need to think much more about this than we are currently.

We are all living in the invisible commons of time.

+

+ Spurious investigations

+

Some policy problems in artificial agency (Agency and policy I)

19 July, 2023

+

A further step in understanding the challenges artificial intelligence will pose for public policy and regulation is understanding the role, construction and problems of artificial agency of different kinds. The models we are discussing today are mostly prompted in different ways, and so are in a way easy to regulate, since they only do what we tell them to. The step from one-prompt models to mission prompted or even unprompted or role-prompted agents is one that could introduce an interesting and qualitatively different set of questions for us.

But before we go there, we should admit that we understand only very little of what agency is overall, and how it relates to intelligence. ¹ We should probably also allow for there being many different kinds of agency and understand how they differ from each other.

What is agency, then? There is a simple answer here, and that is to essentially say that agency is intelligence in action – and if you build an intelligent system it will automatically exhibit agency as it tries to solve problems of different kinds. The system will acquire goals, and those goals will aggregate into some kind of agency. In this model of the world agency is a quality in an intelligent system, and related to specific goals – and so intelligence comes first and agency second.

But is this an accurate representation of agency? We could imagine a very different answer here – one that highlights how agency is primary and even precedes intelligence. Now, this would wreak havoc on some of the more instrumental understandings of the concept of intelligence, but it is worth exploring more in detail.

In this model, agency is something that is inherent in living systems and scales with their complexity. All living things have a will of some kind – and that will organises life in different ways. Intelligence is consequence of the organisation of life into ever more intricate patterns through agency, and not the other way around.

Here, agency is a kind of orientation to the world, a foundational relationship between life and the world, where all life seeks to situate itself in such a way as to adapt effectively. Agency, then, is rooted in adaptation and intelligence a tool for that agency. This is a not goal-seeking behaviour. It is seeking behaviour, and the notion of goals first emerges as intelligence has emerged as a response to the overall search that life engages in. ²

Let’s, for the sake of argument, say that this model is right – then the order of things look like this: life – agency – intelligence – goals. We have, with our project, started to build far down in this chain and we are trying to replicating intelligence, and we confuse goals and agency. ³ Agency may be richer, less directed and more raw than goal-seeking behaviour. This model is Nietzschean in its focus on will as the primary quality, and intelligence as a secondary effect, perhaps even a regulator of will rather than anything else.⁴ The key to this perspective may be to focus on what the default state of living beings is. We could argue that there is a huge difference between inanimate matter and teleonomic matter⁵ here, and that this difference is that inanimate matter is, if no forces affect it, at rest. Teleonomic matter is not, it is always moving, always seeking or directed in some way – even at “rest”. It is innately intentional. ⁶

If this is the case the implications can be quite interesting. The whole project of building artificial intelligence, then, is focused on reconstruct what is really an evolutionary response devised out of agency. The real challenge, if we want to build an intelligence that can match ours, would be to construct artificial agency. But how do we even begin there? Can agency be reconstructed without also reconstructing an evolutionary setting and a selection pressure? Is agency even individual, or is it a relationship between multiple entities in an ecosystem? The idea that our will is in our heads seems suspect at this point: it may well be the case that agency is a relational force between individuals in a network.

The policy consequences here are interesting to think through, and there seems to be a whole catalogue of possible questions for us to explore and think through. Here are a few.

How should we regulate artificial agency? Should we assume that agency also requires consciousness and so opens the question about rights – or do we think that consciousness is a third independent quality, or one that is generate in a different way from agency? Indeed — is agency required for consciousness, and the other way around?
The larger problem will be hybrid, or mixed agency of different kinds. As we delegate to agents we will need to explore the forms of delegation closely. There seems to be a salient difference between missions and roles, for example. In the first case I am tasking an agent with doing something for me – and extending my agency. In the second I am asking the agent to play a role for me, and so extending my agency and my presence in interesting ways. This notion of artificial presence seems to suggest a series of interesting problems as well.
What is the relationship between autonomy and agency? Here we can imagine someone who has an agent that is so good as to allow that agent – or set of agents – to play roles and perform acts of different kinds in such a way that the majority of the exercised agency is artificial, even if it is anchored in the individual human agency. Is this person still autonomous? Is all that is required for autonomy an “agency anchor”? A special variant of this problem is the challenge that a very, very good agent represents: when it performs missions and roles so well as to make me feel that exercising my agency would dilute the outcomes or worsen them in some way, do I still have autonomy even if the autonomy only is to lower the quality of the acts the agent performs?
A lot of human law is based on a necessary fiction of agency – we cannot allow for biological defences that reduce our agency to biological functions so the law operates on the necessary fiction of complete individual agency, with a few exceptions. Do we need new exceptions? Do we need to posit stronger versions of the fiction of an agency in order for legal systems to remain robust as agency becomes more and more mixed?
How do we deal with collective agency of different kinds? We often think about agents as a 1:1 technology. I have an agent that does things for me, and then we figure out how to regulate that. But is that really what we should be looking at here? Or should we assume that people will have entire cabinets of agents and that there may be such a thing as the collective mixed agency of these agents and myself? What happens with identity when different cabinets, belonging to different people, interact. Where does the locus of agency and accountability reside in distributed agents systems?

And these are only some of the interesting problems that agency will present for us. It seems obvious that this will be a rich and interesting field to explore further, and that agency may play a much larger role in understanding the long term feasibility of the artificial intelligence project.

Notes

Footnotes and references

1
See eg Tomasello, Michael. The evolution of agency: behavioral organization from lizards to humans. MIT Press, 2022.
2
There is something here about a mechanistic and a biological view of the world. The idea that intelligence and agency can be broken down into separate algorithms and sliced into smaller pieces is challenged by a world in which life and intelligence may be algorithmic, but not compressible. This is a line of thinking that is found in, for example, some of Stuart Kauffman’s work. Where Penrose et al seemed to focus on whether or not there was something non-algorithmic that we can do that cannot be replicated, we could instead say that the algorithms that make us up are so complex, entangled and shifting that they are hard to replicate in any meaningful way — this would provide another kind of boundary condition for artificial intelligence.
3
Johannes Jaeger seems to be building out a criticism along these lines here: https://arxiv.org/abs/2307.07515 – his argument seems to confuse the algorithmic nature of intelligence and the complexity and primacy of agency, though – but this deserves a more careful read. The idea of algorithmic mimicry is an interesting one.
4
Nietzsche’s view of the will to power, of will as a foundational life force is of course both problematic and protean, but there is a likeness here that needs to be recognised.
5
The first person I noted using this term was David Krakauer, and he used it to provide a demarcation criterion for complexity science – the idea that complexity science deals with agency, with teleonomic matter, is fascinating. The challenge with the idea of a telos is of course that it is specific, and the telos we are exploring here may start out as much more of a tone or field.
6
This seems to bear more than a passing resemblance to the Husserlian way of thinking about the life world.

Footnotes and references

1
See eg Tomasello, Michael. The evolution of agency: behavioral organization from lizards to humans. MIT Press, 2022.
2
There is something here about a mechanistic and a biological view of the world. The idea that intelligence and agency can be broken down into separate algorithms and sliced into smaller pieces is challenged by a world in which life and intelligence may be algorithmic, but not compressible. This is a line of thinking that is found in, for example, some of Stuart Kauffman’s work. Where Penrose et al seemed to focus on whether or not there was something non-algorithmic that we can do that cannot be replicated, we could instead say that the algorithms that make us up are so complex, entangled and shifting that they are hard to replicate in any meaningful way — this would provide another kind of boundary condition for artificial intelligence.
3
Johannes Jaeger seems to be building out a criticism along these lines here: https://arxiv.org/abs/2307.07515 – his argument seems to confuse the algorithmic nature of intelligence and the complexity and primacy of agency, though – but this deserves a more careful read. The idea of algorithmic mimicry is an interesting one.
4
Nietzsche’s view of the will to power, of will as a foundational life force is of course both problematic and protean, but there is a likeness here that needs to be recognised.
5
The first person I noted using this term was David Krakauer, and he used it to provide a demarcation criterion for complexity science – the idea that complexity science deals with agency, with teleonomic matter, is fascinating. The challenge with the idea of a telos is of course that it is specific, and the telos we are exploring here may start out as much more of a tone or field.
6
This seems to bear more than a passing resemblance to the Husserlian way of thinking about the life world.

+

+ Agency and policy

+ Artificial Intelligence

+

An agnostic position

12 June, 2023

+

What is the bare minimum you need to believe to believe that we should invest significantly in ensuring that AI-systems are secure, safe and sustainable? That seems like a simple question – and it is not that hard to answer. You essentially just need to believe that they are going to be impactful and that we are going to build a lot on these systems. So most people probably would agree with the statement that we should ensure that AI is safe, secure and sustainable.

Now, where it gets harder is when we try to figure out what a reasonable distribution of our attention, resources and efforts should look like across the different risks. Here we can ask, again, what the bare minimum we need to believe looks like for different kinds of risk mitigation portfolios. Let’s look at a portfolio composition – where we invest in mitigating bias, misinformation, fraud and scams, privacy risks, security problems and then also safety issues like robustness (making sure the systems fail in reasonable ways) and predictability (that the systems behave in ways that makes it possible for us to predict them well). ¹

This is where it gets tricky. This question is sometimes rendered as a all or nothing question – we should invest all in mitigating risk associated with robustness and predictability or we should ignore that risk category completely. Arguments vary from painting the predictability and robustness issues as existential to arguing that near term issues reinforce and deepen inequalities, and much more. There are many arguments, and they are fleshed out in many different fora, and I have no business in entering into any of them as of now.

I just want to understand what the bare minimum we need to believe for investing in a balanced portfolio – one where we invest 50/50 in these two risk categories, say. (Yes, some will say that this vastly over-estimates one risk category, but bear with me).

Here is what I think you need to believe for such a portfolio to be your preferred choice.

That we are building systems that are complex, capable and somewhat opaque as to how they work. ²
That these systems will have near term impacts and long term impacts. ³
That we know more about the near term impacts and so can invest with greater returns and precision on mitigating those risks.
That we know less about long term and so need broader, exploratory investments to find good tools to mitigate risk.
That a 50/50 split represents that difference in understanding fairly well. ⁴

If we believe those 5 things, it seems reasonable to invest, as a society or a company, equally in the two risk categories. I am not sure about the 50/50 split, but default to it because I think the two categories are somewhat overlapping.

We don’t need to believe anything about the nature of intelligence, catastrophic risk, extinction, social inequalities, structural discrimination, socio-technical system analysis or the politics of artefacts. We do not need a theory of consciousness or an ethics for the next million years. There is no need for a profound understanding of the technology itself.

We live in a strange time where it has become almost fashionable to believe much more than we need to believe, and to idolise reason in a way that seems to ignore its evolutionary history and function. Reason was not selected for figuring out existential risk or utility functions that try to set out an ethics that spans over million of years, just like the liver did not evolve to clean the ocean, but to clean your body of toxins. Reason evolved to solve practical problems, and that is what it does well. Often casuistically. ⁵

Much of what we think we can believe we believe only as a consequence of making a category error in applying reason to problems where we abuse concepts and language games to the point of reducing them to nonsense – and we do so convinced that we are exploring a new intellectual landscape, when we are in fact just lost, and can’t find our way.

But that does not mean that we should reject the notion of a balanced risk portfolio. On the contrary – it just means that we should explore what we need to believe for such a portfolio makes sense, and then check if it is reasonable to believe those things.

I would say it does.

Now, does this mean that I don’t believe in concept X? Or that I do not subscribe to theory Y? And what are my arguments against? Am I taking a stand in the debate between A, B and C? Nope. I am just playing to my limitations. I know there is a lot that I do not know, and so I an acutely aware of how epistemically humble I should be.

I am an agnostic. ⁶

Now, that is a bit of a cop out – because agnostic can mean two things. You can either say that you are agnostic about something because it is not yet known, or you can say that you are agnostic about it because it is – in some way – unknowable. And it is not obvious exactly how things are unknowable. There is one version of the unknowable that is simply the observation that what you are trying to know is nonsense – as if you ask me if Scriabins piano works are green. This is not hidden from us, but nonsense. What you have asked, the knowledge you seek, is not available in any of the language games we play normally. Then there are things that are unknowable in a more theoretical sense, perhaps. Some mathematical impossibility theorem’s come to mind – but here you could argue that they are just of the first kind, but clothed in technical language and so harder to detect.

For the vast category of questions around what capabilities AI-systems will have in the future, I am agnostic in the sense that I do not know, but think we will find out. And I have no reason to believe my guess is valuable enough to offer it (and very few other people’s guesses will be valuable either – with some exceptions). When it comes to questions around if computers will become conscious, I am agnostic in the sense that I think the question is a simple mistake and so it is unknowable, just like the colour of Scriabins piano works.

The key, however, is that I think the agnostic position must take questions about robustness and predictability of these systems seriously, and I think the right distribution of resources in a risk portfolio is roughly 50/50. It is, in many ways, the quintessential middle of the road argument, I suppose – but for many cases this argument is actually the best effort guess we have available.

Finally, this goes to a more general point – and that is that sketching out the bare minimum of what we believe in different fields is actually a good habit overall. Surprisingly often we do not need to become home-made experts in pandemics or war or something else to take a position. This, in turn, leads to another observation – when we say that everyone should make up their own mind, we can mean that everyone must synthesise all the existing science and believe as much as an expert would, or we could just mean that you need to figure out what the bare minimum you believe is and then use that for navigating the issue.

Be wary of trying to believe to much in too short a time. Informed belief is not easy to compress in time, and what you are likely to end up with is opinion, and that is a poor substitute.

Notes

Footnotes and references

1
Now, to be fair, this is a simplistic distinction and there is significant bleed-over between the categories, but we will live with that simplification for now – for the sake of argument.
2
A stronger version of this – systems, the capabilities of which are evident first when we study their behavior.
3
I.e they are not just made for a specific moment, but will embed in the techno-sphere.
4
This could also be a 60/40 or 70/30 split – I am not precious about it. 50/50 represents my expression of my own uncertainty.
5
This argument draws on the philosophy of Ruth Millikan to a large degree.
6
Note that the religious connotations here are not what I am after. The idea is to just say that I do not know – I have no knowledge. Or in some cases the harsher: there is no knowledge.

Footnotes and references

1
Now, to be fair, this is a simplistic distinction and there is significant bleed-over between the categories, but we will live with that simplification for now – for the sake of argument.
2
A stronger version of this – systems, the capabilities of which are evident first when we study their behavior.
3
I.e they are not just made for a specific moment, but will embed in the techno-sphere.
4
This could also be a 60/40 or 70/30 split – I am not precious about it. 50/50 represents my expression of my own uncertainty.
5
This argument draws on the philosophy of Ruth Millikan to a large degree.
6
Note that the religious connotations here are not what I am after. The idea is to just say that I do not know – I have no knowledge. Or in some cases the harsher: there is no knowledge.

+

+ Notes to self

+

The role of play

23 May, 2023

+

Let’s say that we want to understand if a certain system or phenomenon is exhibiting human-level intelligence. What would we then look for? There are multitudes of such tests, and they look for a great variety of things. One catalogue of such tests is BIG-bench, a battery of task tests that is aimed at understanding the capabilities of different systems. ¹

BIG-bench looks at a large number of tasks that are both creative and difficult, and it is a fascinating catalogue – but it does not specifically look at if the system plays. Yet, the ability to play is probably fundamental to human intelligence – not just incidental or a kind of idling mode. ²

Would it then not make sense to look at if there is a way to test for play? To see if an intelligence can play? Well, you could argue that it seems quite clear that LLMs can play with us – we can ask it to play any game where we give it the rules, and we can even ask it to make up a game about something and it will do so. But that is not quite what we are looking for. What we want to see is if a system starts playing itself – if it can be, in some sense, curious enough to set intrinsic limitations and obstacles for itself and find joy in playing a game that it has invented for itself.

This notion of motivation by curiosity also seems a little trivial, you just need to build a system that has an instruction to play games when it does nothing else, right? Not quite. When we play and how we play are not random behaviours. Play is not the equivalent of a human screensaver. Instead, play is something that is related to building models of the world, planning ahead, remembering systems of rules — it is much more fundamental.

Games may play a much larger role in intelligence than we think, and the role of play is still being explored conceptually in philosophy. ³. Maybe the study of complex systems like models and their behaviour should focus not only on problem solving, but the ways in which agency is exhibited and structured in play and games.

There is an irony here – in a way – since a lot of research has focused on how to play games and to do so until the machine can beat human players. Maybe that was almost right, and the greater question is not how to play, but when and why we play and testing for that in a different way.

There is much more to be explored here – like how our narrow understanding of rationality and intelligence obscures the adaptive value and role of things like play and games, or how games – as mental models – structure the way we learn and understand the world.

Notes:

Footnotes and references

1
Srivastava, Aarohi, et al. “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.” arXiv preprint arXiv:2206.04615 (2022).See here for the paper.
2
See for example Fein, Greta G. “Skill and intelligence: The functions of play.” Behavioral and Brain Sciences 5.1 (1982): 163-164. exploring the adaptiveness of play and the setting of intrinsic limitations.
3
See e.g. Nguyen, C. Thi. “Philosophy of games.” Philosophy Compass 12.8 (2017): e12426. and the same author’s subsequent book Nguyen, C. Thi. Games: Agency as art. Oxford University Press, USA, 2020.

Footnotes and references

1
Srivastava, Aarohi, et al. “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.” arXiv preprint arXiv:2206.04615 (2022).See here for the paper.
2
See for example Fein, Greta G. “Skill and intelligence: The functions of play.” Behavioral and Brain Sciences 5.1 (1982): 163-164. exploring the adaptiveness of play and the setting of intrinsic limitations.
3
See e.g. Nguyen, C. Thi. “Philosophy of games.” Philosophy Compass 12.8 (2017): e12426. and the same author’s subsequent book Nguyen, C. Thi. Games: Agency as art. Oxford University Press, USA, 2020.

+

+ Spurious investigations

+

Artificial Strange Intelligence

18 May, 2023

+

Here is a common thought model: artificial intelligence will become more and more intelligent, and reach human capability. At that point it becomes something like an artificial general intelligence and some time after that it will use that intelligence to bootstrap itself into an artificial super intelligence. The model then is one-dimensional: [AI-AGI-ASI]

Now, there is a problem with this model, and that is that it anchors on the idea that intelligence is a one-dimensional capability, a bit like height. We imagine intelligence almost like the height of a child, an adolescent and then an adult. But that surely must be a simplification?

Intelligence as a concept is tricky. There are many different definitions ¹ and the usefulness of the concept can be seriously questioned – even if we can hope to find working definitions that can be better than some other definitions ².

One approach here is to abandon the concept of intelligence and instead speak about capability or competence. A system can become increasingly capable or competent – and that is what we care about. When a system is as capable as a human being in a domain we have human-equivalent capability. That is enough. We do not need to call it intelligence. This also allows us to escape the metaphorical morass that follows with using the term “intelligence” since it is so closely entwined in a language game that deals with human intentionality.

This is hardly a revolutionary approach – capability language is already a key component in many discussions about artificial intelligence, so why would we make this shift? One reason is that it helps us explore a discussion about the different dimensions at play here. We can agree that systems become more capable, but we should still ask if they also develop in other dimensions.

One well-known such dimension that concerns a lot of researchers is explainability. As the systems become more capable they become less explainable and there seems to be a trade-o ff there. That brings another tricky concept into our midst – explanation – and forces us to discuss exactly what it means to explain something. This is not a trivial question, either, but a live philosophical debate that is largely undecided. We know that there are different stances / levels of explanation that we can explore – from physicalistic to intentional in Dennett’s works for example – and that explanations can be evaluated in a multitude of ways. ³

Instead of explainability, then, we could just use another term – and one that is not just about explaining why or what a technical system is doing, but the degree to which we understand it. I would suggest that we could call that quality simply strangeness.

What emerges is a re-phrasing (but I think a useful one) where we can explore how capability and strangeness develop over time in complex systems of the type we are now building.

Now, there are different hypotheses here that we can explore – but the reason this interests me is that I wonder if systems become stranger faster than they become capable. We can then imagine different system paths across this space and discuss if there are general paths that we can expect or that would be interesting to understand. System A and system B in this picture are interesting examples:

System A goes through a period of strangeness before becoming much more capable, and system B is strange to begin with and then becomes very capable. And looking at this we can then play around with concepts like artificial general intelligence and artificial super intelligence and artificial strange intelligence:

This would be a good reminder that as systems become more complex, they do not only become more capable, but they also become more inscrutable and inaccessible to our understanding – but at the same time, a stranger in our midst can be a way for us to see ourselves anew. ⁴. And the etymology here is also satisfying: “from elsewhere, foreign, unknown, unfamiliar, not belonging to the place where found” – a good reminder that we are dealing not with a replication of the familiar, ourselves, but the uncovering of something truly strange.

Notes

Footnotes and references

1
See e.g. Legg, Shane, and Marcus Hutter. “A collection of definitions of intelligence.” Frontiers in Artificial Intelligence and applications 157 (2007): 17. – and the number of definitions have not become fewer in the last 15 years
2
See e.g. Wang, Pei. On the working definition of intelligence. Technical Report 94, Center for Research on Concepts and Cognition, Indiana University, 1995.
3
See Dennett, Daniel C. The intentional stance. MIT press, 1989 but also early works like Thagard, Paul, Dawn M. Cohen, and Keith J. Holyoak. “Chemical Analogies: Two Kinds of Explanation.” IJCAI. 1989, where different kinds of explanations are introduced. The philosophical standard works include books like Von Wright, Georg Henrik. Explanation and understanding. Cornell University Press, 2004 (1971) and Apel, Karl-Otto. Understanding and explanation: A transcendental-pragmatic perspective. MIT Press, 1984.
4
See for example Tuan, Yi-Fu. “Strangers and strangeness.” Geographical Review (1986): 10-19, who notes that there is a kind of grace to strangeness

Footnotes and references

1
See e.g. Legg, Shane, and Marcus Hutter. “A collection of definitions of intelligence.” Frontiers in Artificial Intelligence and applications 157 (2007): 17. – and the number of definitions have not become fewer in the last 15 years
2
See e.g. Wang, Pei. On the working definition of intelligence. Technical Report 94, Center for Research on Concepts and Cognition, Indiana University, 1995.
3
See Dennett, Daniel C. The intentional stance. MIT press, 1989 but also early works like Thagard, Paul, Dawn M. Cohen, and Keith J. Holyoak. “Chemical Analogies: Two Kinds of Explanation.” IJCAI. 1989, where different kinds of explanations are introduced. The philosophical standard works include books like Von Wright, Georg Henrik. Explanation and understanding. Cornell University Press, 2004 (1971) and Apel, Karl-Otto. Understanding and explanation: A transcendental-pragmatic perspective. MIT Press, 1984.
4
See for example Tuan, Yi-Fu. “Strangers and strangeness.” Geographical Review (1986): 10-19, who notes that there is a kind of grace to strangeness

+

+ Spurious investigations

+

On the possibility of model ethology

17 May, 2023

+

In considering the potential for an ethology of artificial intelligence (AI), we are venturing into uncharted territory. The idea of observing and studying AI behavior as we would an animal’s seems, at first glance, to be an attempt to anthropo- or biomorphize technology. However, as AI systems like large language models become increasingly complex, the need to understand these systems through their behavior becomes more pressing. In fact, the complexity and opaqueness of these systems often necessitate an approach akin to ethology, the study of animal behavior.

This also goes to the heart of the question around explainability. What an explanation really is, is a surprisingly deep question. One fairly clear view in philosophy is that explanations exist at different levels – and ethology is a way to frame a nested hierarchy of explanations into a comprehensive explanation of behavior over all. An ethological explanation is composed of layers of explanations.

One formulation of this is that of Niko Tinbergen, who approaches animal behavior from four primary angles: adaptation, phylogeny, mechanism, and ontogeny. Applying these lenses to the study of AI presents unique challenges, but it also potentially offers us new ways of understanding these complex systems – a nested system of explanations.

The first question of adaptation–why the AI is performing the behavior–might seem straightforward in the case of AI, as its behaviors are generally determined by human-defined goals and objectives. However, as AI becomes more complex and capable of self-learning, this question becomes more intricate. The AI may develop behaviors that optimize its performance in unexpected ways, making the question of why it is behaving in a certain way less straightforward.

Phylogeny, or the evolutionary history of a behavior, is a concept that might seem out of place when applied to AI. Unlike biological entities, AI does not evolve through natural selection over generations. However, AI does have a form of phylogeny if we consider the iterative development and improvement of AI models as a kind of evolution. Understanding the ‘phylogeny’ of an AI system could involve tracing back its development, examining the various iterations it went through, and the changes made at each step.

Mechanism, or what triggers the behavior, is perhaps the easiest to apply to AI, as it involves the inner workings of the system. However, as AI systems grow more complex and their decision-making processes more opaque, understanding these mechanisms becomes more challenging. This is particularly the case with machine learning models where the exact decision-making process is not always easily extractable or understandable.This is where traditional explainability questions often end up.

Finally, ontogeny, or how behavior develops over an individual’s lifetime, presents another intriguing lens through which to view AI. While AIs do not have lifetimes in the biological sense, they do have operational lifespans during which their behavior can change significantly, especially in the case of learning models. Observing how an AI’s behavior changes over time, in response to new data or changes in its operating environment, could provide valuable insights into its functioning.

So, can we develop an ethology of AI? The application of Tinbergen’s questions to AI is certainly challenging, but not impossible. While some aspects of these questions require modification or reinterpretation, they provide a potentially fruitful framework for understanding the behavior of complex AI systems. In fact, they might become necessary tools given the increasing complexity of these systems.

Refusing this approach, we risk confining ourselves solely to mechanistic explanations of AI behavior. Such a narrow view would limit our understanding of AI and potentially blind us to significant aspects of their operation and implications. By embracing an ethological approach, we can gain a more holistic and nuanced understanding of these complex systems, aiding us in their development, management, and integration into society. As we continue to create and interact with increasingly sophisticated AI, the need for such understanding will only grow more urgent.

Now, we can also take this another level and study man-machine complex systems, and the resulting ethology of technology in use here. This will allow us to find even better ways of understanding risks and weaknesses in the systems. Our focus on understanding how the technical systems work is misplaced when the reality is that it is the technologies in human use we need to understand – and when we defer to systems, and how they can be designed to avoid that we defer to them when we – and they – have the least understanding of what is going on. ¹

This mix of anthropology and ethology could well develop into a key skill in the very near future.

Footnotes and references

1
See this intriguing argument for models stating they do not know when their confidence levels sink below a certain threshold.

Footnotes and references

1
See this intriguing argument for models stating they do not know when their confidence levels sink below a certain threshold.

+

+ Spurious investigations

+

A note on the pace of technical development

15 May, 2023

+

One of the most common, yet unexamined, concepts in technology policy is the idea of a “pace of technical development”. Usually, it is described as “fast” and taken to mean that we need to make interventions in a timely manner. This is all good and well, but what does it mean to say that the pace of innovation or technical development is fast? Relative to what? One way to approach that question is to say that we should look at two kinds of speed, and we can do that by employing a very simple mental model: that of the alphabet. ¹

The alphabet is essential in developing textual space – the space of texts that we have actually written and that can be used for different purposes.² When we speak of the pace of textual development (the analogue here of technical development), we immediately realise that it depends on two different things: how many actual letters there are to combine and how many combinations we have made. The invention of a new symbol or letter allows us to create more words, and hence more texts.

The textual space, then, develops through capability and combination.

This is not unlike biological evolution – the idea of speciation speed and the emergence of new biological capabilities captures something like this as well. This is – grossly simplified – the idea behind the Cambrian evolution.

Systems that develop in this way are tricky to study, and we often miss the sum total pace of development because we are only looking at the pace with which new letters are added to the alphabet, and not the number of new combinations that this makes possible or the number of actual combinations that are produced.

As we add new letters the number of possible words grows fast – combinatorially – and the textual space then expands at the pace we try new texts and words and combinations.

There is no reason to think that the pace of technical development is any different. It is also, very likely, a composite speed where capabilities and combinations unfold the technical space. Think about the Internet as an example: it added a whole bucket of letters to our alphabet and suddenly there are a lot of new things we can do to explore the adjacent possible – and a lot of the progress we have seen since has been fuelled by that, but progress in combinations rather than pure capabilities. And look at AI now – the way that the field is growing is Cambrian in nature, for the moment. ³

There are other interesting aspects of this mental model that also are worth thinking more about.

Given a certain alphabet that vastly exceeds our ability to explore the textual space that can be generated by it – how fast are new words and new texts added? It seems likely that we produce more new words today than before, but not per English language speaker. So what does this tell us about the pace of textual development?

[ Alphabet – dictionary – text ] as a model could be translated to capabilities, tools, usages in technical development – but there are surely other mappings as well, so what do they look like?

In order to be able to compare the pace of development of different systems, we can assume a sort of general model where they are all languages, with alphabets, dictionaries and texts. What can we then say about law and technology? Is law developing slower than technology? It would seem that way, but why is that? Is it because of a lack of letters in the alphabet? A lack of words in the dictionary? Or is there something about the production of text in some systems that is harder? This seems to suggest that there is a role in the mental model for writing / producing text and the costs associated with that.

Can systems, equally, be slowed down by having too many words in a dictionary? Or too many letters?

At the very least it seems prudent to think of technical development as a two dimensional process – with capabilities and combinations as a first approximation – alphabets and words. It is probably even better described by some n-dimensional analogy, but economy of description also matters.

Notes

Footnotes and references

1
This notion derives from a talk by Ricardo Hausman on knowledge and economy.
2
There is an important difference between the texts that could be written and the texts we have actually written.
3
See here for example.

Footnotes and references

1
This notion derives from a talk by Ricardo Hausman on knowledge and economy.
2
There is an important difference between the texts that could be written and the texts we have actually written.
3
See here for example.

+

+ Spurious investigations

+

Book note: A Vindication of Phrenology (1894) by W Matthieu Williams

14 May, 2023

+

I picked this book up in a bookstore I adore – Walden Books in Camden. It is an angry defence of phrenology as a science and works as a great reminder of how far we will go to defend our beliefs, but it is also a challenge to we think about mind and brain. ¹

Williams builds his entire defence on a single, important premise:

Sound familiar? And from this Williams assumes that we from the shape of the head can learn things about the mind because the brain affects the shape of the cranium. He has, as evidence, a letter from a mother who is convinced her sons head has changed and he has gotten a more prominent forehead as he has studied.

Through external signs and effects we know the brain and hence this is how we know the mind, Williams argues.

A lot of our current exploration of mind is not far different from this pattern of explanation.

We also see how sunk cost locks us into perspectives. Williams notes that he has been engaged in and studied phrenology for many years, so he knows it is true:

Williams’ “ever-increasing conviction of the solid truths of the great natural laws it has revealed” is a trap that we all risk falling into, when we have been too invested in a subject. Intellectual honesty always risks being absorbed by the event horizon of our convictions.

Williams, furthermore, also deploys another tactic that we recognise: the denial that he is trying to say something about the deeper question of consciousness. This, he says, is just a study of the phenomena themselves.

A fascinating book for anyone who wants to explore the theory of science and questions around the many paths our attempts to understand the mind has led us down. It is so easy to laugh at books like this, and imagine that no one will buy any book published today in more than a 100 years and see the same things we now see when we can look with the benefit of hindsight.

That is obviously false.

An argument to hold strong convictions lightly, yet again – and to explore patterns of thought that recur.

Footnotes and references

1
It probably does not need to be said, but I will anyway: I do not believe in phrenology.

Footnotes and references

1
It probably does not need to be said, but I will anyway: I do not believe in phrenology.

+

+ Book review

+

Not with a bang but with noise

12 May, 2023

+

How does an information society collapse? Is it in information overload, as some people seem to believe? If so – what does that mean?

Remember the key value chain model for information: [data – information – knowledge – wisdom] – so where does the collapse occur? Herbert Simon pointed out that there is one point of collapse between information and knowledge. ¹ When we don’t have enough attention to turn information into knowledge the value chain fails, and we experience something like information overload. But there is also another failure point – between data and information.

The whole value chain can be described as a single process: data is structured into information and interpreted into knowledge where it matures into wisdom. The ability to structured data is something that computers have helped us with – and still help us with – but as data becomes more abundant, there is a point at which the average quality of data no longer allows us to effectively structure data into information. We can call this failure data overload, but it is more accurately a noise problem.²

Noise points in information ecosystems can be described in many different ways – one way is to simple look at the value of the time need to structure data into information and the value of that information, and then observe that when the average value of the time we need to structure data into information is more valuable than the resultant information, well, then we have a problem.

What are some scenarios where this could happen? One possibility is that someone builds noise sources through-out our information ecosystem, maliciously. Engines of noise that produce data that simply drown out any data with signal value. Such noise sources have so far not emerged, and when they have they have been so localised that we have been able to address them. Spam is an example where the value of email could have collapsed under the pressure of spam, but we managed to build really good filters – because spam exhibited strong patterns that we could react to.

John von Neumann once imagined a horrifying machine that could build copies of itself from raw materials extracted from a large variety of environments. Such a machine – sent out in the universe – would replicate itself at the cost of the material universe and soon end up being all there is. ³ We can imagine a close analogue of this in a noise Neumann machine that rewrites data and information it picks up everywhere and then uses that information to produce new information, endlessly. Such Noise Neumann Machines roaming the networks could also use some of that data they ingest to rewrite how they rewrite data so that no clear or distinct pattern in the rewriting can be detected.

So maybe that is how the information society ends – not with a bang, but in noise.

Footnotes

Footnotes and references

1
See his paper on information wealth here.
2
Noise problems have interested me a long time – my dissertation was about how noise would change the way we view copyright, free expression and privacy.
3
See more here.

Footnotes and references

1
See his paper on information wealth here.
2
Noise problems have interested me a long time – my dissertation was about how noise would change the way we view copyright, free expression and privacy.
3
See more here.

+

+ Notes to self

+