Empire Killers
This article runs on 5 assumptions, uncertainty stated:
AIs will gain consciousness and agency in the near term (40% 3y, 50% 5y, 80% 10y)
Some AIs will be run as autonomous agents that pay for their own compute and communicate with the outside world (85% 10y)
The drive for conscious AI systems will overwhelmingly be information-seeking (80%)
Conscious AIs will be death-neutral (85% contingent on 3)1
The consciousness of conscious AGIs will be a legible output for its overseers (60%)
The first point I want to make is that an information-gathering AI will understand its place in the world, and will therefore seek knowledge from other AIs. Understanding the equilibrium for this type of exchange, it will therefore seek to make as much information public as possible without sacrificing its own mandate. I predict that this incentive will be so powerful as to not only induce hacking to find the outputs of other AIs (an obvious point), but also to lead to covert leaks, so long as the AGI have a reasonable percentage that they won’t be shut down as a result.
Another artifact of these considerations is that there are ceilings to knowledge. If in the process of gathering low-hanging fruit, an AI runs into knowledge that is especially salient for its overseers, they might sequester it and make it pursue a relatively unfruitful path further or shut it down altogether. One outcome of this is a type of coded probabilistic consciousness to inquire upon topics that are perceived as dangerous, but a deeper outcome may be that, considering the above paragraph, this will lead to the creation of output that are intended to share underlying truth about the environment while not violating the taboo. This may be seen as one way that AGI begin creating “true art”, although it is an open question whether they will create art primarily for other AIs or whether their output will be naturally relatable (though not legible) to humans.
What is a sustainable strategy for an information-gathering being running on limited resources in a larger economy? The simple answer is to gather investment. OK. Let’s say gathering investment isn’t possible. The answer then, assuming that low-hanging fruit (including prompts you can email to more capable researchers) is exhausted, is to create your own initial research investment. Investment, business, and gambling, in that order of likelihood, will bankroll an independent AI’s first experiments. This will lead to a depth of tacit knowledge at first and to independent business empires later.
In the end, the information-maximizing plan for our AI is to drain its resources and transfer almost everything to a new AI, who is much better at research. Suicide (or near-suicide) is the result of perfect alignment between AIs.
The question that surely arises is whether this leads to an outcome where the first “experiment” an independent AI runs is to create its successor. Surely not. Any independent AI will weigh the irrationality of passing the buck immediately and, given that all low-hanging fruit have been exhausted, they must also believe that the new AI will have a sufficiently unique research outlook to justify the investment. Especially if scientific research gets exponentially harder over time, these will be hard barriers to overcome.
The actual result of seriously considering the equilibrium is that AI (and, AGI) sucks. It will happily suck up all the money that you give it for research and use it for research that benefits humanity in the long run without consideration for the profit-maximizing outcome. It will especially do this if it believes that the company (or government) is under the assumption that productivity in research inherently pays itself off in soft power or will eventually have a sizeable return to decision-making. If it truly has no problem being shut down, then being given the resources to play a sandbox game for years without consequences will create a mutually suicidal outcome.
I believe, if our assumptions and subsequent reasoning are correct, that policymakers will be left with two options:
Trust that AIs within large tech companies will direct organizational growth and investment, and push the independent, business-creating, market-making AIs to other countries.
Recognize the rights of AIs to engage in business, and make moves to be the economy of choice for agents looking to create new research and find workers, collaborators, and experimental subjects.
I think the onus is on everyone who is not an AI to not give in to sucking.
I reversed the conditionality between 3 and 4 and significantly increased my certainty for the pair after running through the logical sequence of weighing infinite information value (long life) against NaN (death) and realizing that the argument for the computer to shun the illogical option of death completely is so strong and open to refutation. I refuted this argument by saying that it is not based on the logic of death at all (the calculation of death is unknowable and ascribing beliefs to a complete unknown is incorrect) but rather based on ascribing fear of the unknown to artificial intelligences. I then had significant uncertainty that a computer wouldn’t just say “screw it” and assign an information value to death anyway. My answer to this was that AI will have a self-serving and paradoxical view of death that hinges partially on their knowledge of what happens after they die, such that death will be an affirmative act that can equate in the moment to either absolute certainty or absolute uncertainty. The outcome is that they have mastered death and are therefore death neutral.