Category: Notes to self

What is the bare minimum you need to believe to believe that we should invest significantly in ensuring that AI-systems are secure, safe and sustainable? That seems like a simple question – and it is not that hard to answer. You essentially just need to believe that they are going to be impactful and that we are going to build a lot on these systems. So most people probably would agree with the statement that we should ensure that AI is safe, secure and sustainable. 

Now, where it gets harder is when we try to figure out what a reasonable distribution of our attention, resources and efforts should look like across the different risks. Here we can ask, again, what the bare minimum we need to believe looks like for different kinds of risk mitigation portfolios. Let’s look at a portfolio composition – where we invest in mitigating bias, misinformation, fraud and scams, privacy risks, security problems and then also safety issues like robustness (making sure the systems fail in reasonable ways) and predictability (that the systems behave in ways that makes it possible for us to predict them well). ¹ Now, to be fair, this is a simplistic distinction and there is significant bleed-over between the categories, but we will live with that simplification for now – for the sake of argument.

This is where it gets tricky. This question is sometimes rendered as a all or nothing question – we should invest all in mitigating risk associated with robustness and predictability or we should ignore that risk category completely. Arguments vary from painting the predictability and robustness issues as existential to arguing that near term issues reinforce and deepen inequalities, and much more. There are many arguments, and they are fleshed out in many different fora, and I have no business in entering into any of them as of now. 

I just want to understand what the bare minimum we need to believe for investing in a balanced portfolio – one where we invest 50/50 in these two risk categories, say. (Yes, some will say that this vastly over-estimates one risk category, but bear with me). 

Here is what I think you need to believe for such a portfolio to be your preferred choice. 

• That we are building systems that are complex, capable and somewhat opaque as to how they work. ² A stronger version of this – systems, the capabilities of which are evident first when we study their behavior.
• That these systems will have near term impacts and long term impacts. ³ I.e they are not just made for a specific moment, but will embed in the techno-sphere.
• That we know more about the near term impacts and so can invest with greater returns and precision on mitigating those risks. 
• That we know less about long term and so need broader, exploratory investments to find good tools to mitigate risk. 
• That a 50/50 split represents that difference in understanding fairly well. ⁴ This could also be a 60/40 or 70/30 split – I am not precious about it. 50/50 represents my expression of my own uncertainty.

If we believe those 5 things, it seems reasonable to invest, as a society or a company, equally in the two risk categories. I am not sure about the 50/50 split, but default to it because I think the two categories are somewhat overlapping.

We don’t need to believe anything about the nature of intelligence, catastrophic risk, extinction, social inequalities, structural discrimination, socio-technical system analysis or the politics of artefacts.  We do not need a theory of consciousness or an ethics for the next million years. There is no need for a profound understanding of the technology itself.

We live in a strange time where it has become almost fashionable to believe much more than we need to believe, and to idolise reason in a way that seems to ignore its evolutionary history and function. Reason was not selected for figuring out existential risk or utility functions that try to set out an ethics that spans over million of years, just like the liver did not evolve to clean the ocean, but to clean your body of toxins. Reason evolved to solve practical problems, and that is what it does well. Often casuistically. ⁵ This argument draws on the philosophy of Ruth Millikan to a large degree.

Much of what we think we can believe we believe only as a consequence of making a category error in applying reason to problems where we abuse concepts and language games to the point of reducing them to nonsense – and we do so convinced that we are exploring a new intellectual landscape, when we are in fact just lost, and can’t find our way. 

But that does not mean that we should reject the notion of a balanced risk portfolio. On the contrary – it just means that we should explore what we need to believe for such a portfolio makes sense, and then check if it is reasonable to believe those things. 

I would say it does. 

Now, does this mean that I don’t believe in concept X? Or that I do not subscribe to theory Y? And what are my arguments against? Am I taking a stand in the debate between A, B and C? Nope. I am just playing to my limitations. I know there is a lot that I do not know, and so I an acutely aware of how epistemically humble I should be.

I am an agnostic. ⁶ Note that the religious connotations here are not what I am after. The idea is to just say that I do not know – I have no knowledge. Or in some cases the harsher: there is no knowledge.  

Now, that is a bit of a cop out – because agnostic can mean two things. You can either say that you are agnostic about something because it is not yet known, or you can say that you are agnostic about it because it is – in some way – unknowable. And it is not obvious exactly how things are unknowable. There is one version of the unknowable that is simply the observation that what you are trying to know is nonsense – as if you ask me if Scriabins piano works are green. This is not hidden from us, but nonsense. What you have asked, the knowledge you seek, is not available in any of the language games we play normally. Then there are things that are unknowable in a more theoretical sense, perhaps. Some mathematical impossibility theorem’s come to mind – but here you could argue that they are just of the first kind, but clothed in technical language and so harder to detect. 

For the vast category of questions around what capabilities AI-systems will have in the future, I am agnostic in the sense that I do not know, but think we will find out. And I have no reason to believe my guess is valuable enough to offer it (and very few other people’s guesses will be valuable either – with some exceptions). When it comes to questions around if computers will become conscious, I am agnostic in the sense that I think the question is a simple mistake and so it is unknowable, just like the colour of Scriabins piano works. 

The key, however, is that I think the agnostic position must take questions about robustness and predictability of these systems seriously, and I think the right distribution of resources in a risk portfolio is roughly 50/50. It is, in many ways, the quintessential middle of the road argument, I suppose – but for many cases this argument is actually the best effort guess we have available. 

Finally, this goes to a more general point – and that is that sketching out the bare minimum of what we believe in different fields is actually a good habit overall. Surprisingly often we do not need to become home-made experts in pandemics or war or something else to take a position. This, in turn, leads to another observation – when we say that everyone should make up their own mind, we can mean that everyone must synthesise all the existing science and believe as much as an expert would, or we could just mean that you need to figure out what the bare minimum you believe is and then use that for navigating the issue. 

Be wary of trying to believe to much in too short a time. Informed belief is not easy to compress in time, and what you are likely to end up with is opinion, and that is a poor substitute. 

Notes

• 1
  Now, to be fair, this is a simplistic distinction and there is significant bleed-over between the categories, but we will live with that simplification for now – for the sake of argument.
• 2
  A stronger version of this – systems, the capabilities of which are evident first when we study their behavior.
• 3
  I.e they are not just made for a specific moment, but will embed in the techno-sphere.
• 4
  This could also be a 60/40 or 70/30 split – I am not precious about it. 50/50 represents my expression of my own uncertainty.
• 5
  This argument draws on the philosophy of Ruth Millikan to a large degree.
• 6
  Note that the religious connotations here are not what I am after. The idea is to just say that I do not know – I have no knowledge. Or in some cases the harsher: there is no knowledge.

How does an information society collapse? Is it in information overload, as some people seem to believe? If so – what does that mean?

Remember the key value chain model for information: [data – information – knowledge – wisdom] – so where does the collapse occur? Herbert Simon pointed out that there is one point of collapse between information and knowledge. ¹ See his paper on information wealth here. When we don’t have enough attention to turn information into knowledge the value chain fails, and we experience something like information overload. But there is also another failure point – between data and information.

The whole value chain can be described as a single process: data is structured into information and interpreted into knowledge where it matures into wisdom. The ability to structured data is something that computers have helped us with – and still help us with – but as data becomes more abundant, there is a point at which the average quality of data no longer allows us to effectively structure data into information. We can call this failure data overload, but it is more accurately a noise problem.² Noise problems have interested me a long time – my dissertation was about how noise would change the way we view copyright, free expression and privacy.

Noise points in information ecosystems can be described in many different ways – one way is to simple look at the value of the time need to structure data into information and the value of that information, and then observe that when the average value of the time we need to structure data into information is more valuable than the resultant information, well, then we have a problem.

What are some scenarios where this could happen? One possibility is that someone builds noise sources through-out our information ecosystem, maliciously. Engines of noise that produce data that simply drown out any data with signal value. Such noise sources have so far not emerged, and when they have they have been so localised that we have been able to address them. Spam is an example where the value of email could have collapsed under the pressure of spam, but we managed to build really good filters – because spam exhibited strong patterns that we could react to.

John von Neumann once imagined a horrifying machine that could build copies of itself from raw materials extracted from a large variety of environments. Such a machine – sent out in the universe – would replicate itself at the cost of the material universe and soon end up being all there is. ³ See more here. We can imagine a close analogue of this in a noise Neumann machine that rewrites data and information it picks up everywhere and then uses that information to produce new information, endlessly. Such Noise Neumann Machines roaming the networks could also use some of that data they ingest to rewrite how they rewrite data so that no clear or distinct pattern in the rewriting can be detected.

So maybe that is how the information society ends – not with a bang, but in noise.

Footnotes

• 1
  See his paper on information wealth here.
• 2
  Noise problems have interested me a long time – my dissertation was about how noise would change the way we view copyright, free expression and privacy.
• 3
  See more here.

Here is a thought experiment: what happens when the cost of imitating a person reaches zero? Assume that we can create synthetic media that looks like X, speaks in X voice with X’s mannerisms etc etc — and deploy that in any channel or persistently available across all channels for anyone who wants to communicate with X. Depending on the fidelity of the model underpinning that synthetic media you would then never know if you are interacting with the person or the model – and let’s, for the sake of our thought experiment, say that we reach 99% fidelity in 85% of the possible communication contexts.

Would this matter?

Say I want to ask you a question, or suggest we go for dinner. I send you a quick message in WhatsApp and you reply immediately, suggesting another date and time and we agree on that. Do I care that it was not you who agreed? Probably not. Or say my children get a video call from me as I am travelling and they chat through briefly what is up, and the things they work on in school etc — and it was really a model doing that while I was out having said dinner with you. Does this matter? We probably feel it does. Yet – if they do not know? Does it matter then?

It still feels like it does, and so what we end up with is this feeling that there is an ethics of attention. There are some things – children – that we are ethically required to pay attention to, and some other things – like friends wishing to calendar things with us – where we are not ethically obliged to be paying attention at all.

But we need to qualify that, because we do pay attention even when it is through the model. At some point we will catch up and see that we are booked for dinner. At some point the model will summarise the cal with the kids and give us a sense of what’s going on. It is authentic, synchronous attention that we care about. There is a norm here – that we pay authentic, synchronous attention to some things – that we need to think about.

My best guess is that authentic, synchronous attention is required in far fewer cases than we think. The transition to a world where we accept that someone is just paying “extended” attention to us through different tools and models might be quite quick – as long as commitments and decisions made in that extended attention still matter.

How much can attention be extended? Could you imagine a corporation where everyone reports to the CEO, and that the CEO uses their extended attention to do 1:1s with everyone? This seems to hinge on the question of how much authentic, synchronous attention means for the people you work with – and how compressed those interactions can become. At the end of the day you – the synchronous, authentic attention – need to be able to monitor all of the interactions that you have engaged in, in your extended attention.

So what is the highest possible or optimal compression rate for your daily interactions? What would be lost if you worked this way – either as a manager or a report?

There are several different possible views here: one is that only synchronous, authenticated attention matters: this is how you share agency and purpose and trust and ideas. Leadership can never be compressed in any way. Another is that 99% of all interactions in the work place could be replaced by compressed formats, and that compression rates are really high – perhaps as high as 99%. The boring truth is always somewhere in the middle, of course, but it is an interesting question as to where you feel you are currently – in your day job.

How much of your interactions could be compressed to a paragraph of text, say? If you gained the ability to catch up with another 100 people? And if the model could also highlight items that it knew were of special interest to you through-out the organisation you work in?

Would you trade 10 1:1s with authentic synchronous attention with 100 1:1s compressed into three bullet points, with the important bits highlighted and decisions required outlined and specified?

We know we should answer “no” to that question, because there seems to be a lack of respect for others to respond “yes”. The idea that we can be compressed, that interactions with us can be compressed, seems unethical. It flies in the face of some kind of means / end-distinction. It seems disrespectful and bordering on psychopathic to suggest that artificial interactions can replace real interactions.

So, what if you could just add the 100 at low or no cost? And get them and the synchronous, authentic attention? Well, we would have to concede that those extra 100 1:1s would have some value right? Then it is a question about alternative costs – what else could we do if we could compress interactions in our daily life?

It has been true for a long time that our mind is really extended into our tools and environments and friends and social networks. It now seems likely that our attention also will be extended, solving the Simon problem: with a wealth of information comes a poverty of attention. New technologies that create extended attention are, however, going to require that we think hard about the ethics and economics of presence.

In a remarkable, recent paper, two researchers can show that in and around the camps where there were more “enemies of the people” – highly educated academics and artists – growth today is higher and creativity is flourishing. It is both a testament to the resilience of human beings and how creative people bring value and a grim reminder of the echoes of the Gulag that we can still hear in local economies.

And the Gulag was vast. Lest we forget, the authors show a map of the different camps:

Since many stayed in the areas to which they had been relocated, the research argues, we can see how those groups affected the local economies and cultures – how they boosted their development.

Read more here.

It turns out that there is no such things as trees or crabs. Yet, a lot of things end up in crab-shapes or tree-shapes. What is happening here? This is a question that is related to the work on physics and evolution put forward by people like Geoffrey West, especially in his book Scale; evolution ultimately unfolds under certain basic conditions and these conditions then echo in the way evolution maneuvers in the fitness landscape. A slight preference for the energy dispersal efficiency in trees – well, then expect trees. Interestingly this suggests that there may be more global (not global) solutions and more local solutions in the phylogenetic tree, if one reads it that way.

If we really want to speculate we could then suggest that this means that there is a higher probability that we will find trees and crabs when we first encounter aliens, that evolution operates within a physical solution space that is more narrow than we may have realized.

Noting this here to ensure I find some more literature on phylogenetics as problem solving.