This is part 2 of a 5-part sequence:

Part 1: summary of Bostrom's argument

Part 2: arguments against a fast takeoff

Part 3: cosmic expansion and AI motivation

Part 4: tractability of AI alignment

Part 5: expected value arguments

Premise 1: Superintelligence is coming soon

I have very little to say about this premise, since I am in broad agreement with Bostrom that even if it takes decades or a century, super-human artificial intelligence is quite likely to be developed. I find Bostrom's appeals to surveys of AI researchers regarding how long it is likely to be until human level AI is developed fairly unpersuasive, given both the poor track record of such predictions and also the fact that experts on AI research are not necessarily experts on extrapolating the rate of technological and scientific progress (even in their own field). Bostrom, however, does note some of these limitations, and I do not think his argument is particularly dependent upon these sorts of appeals. I therefore will pass over premise 1 and move on to what I consider to be the more important issues.

Premise 2: Arguments against a fast takeoff

Bostrom’s major argument in favour of the contention that a superintelligence would be able to gain a decisive strategic advantage is that the ‘takeoff’ for such an intelligence would likely be very rapid. By a ‘fast takeoff’, Bostrom means that the time between when the superintelligence first approaches human-level cognition and when it achieves dramatically superhuman intelligence would be small, on the order of days or even hours. This is critical because if takeoff is as rapid as this, there will be effectively no time for any existing technologies or institutions to impede the growth of the superintelligence or check it in any meaningful way. Its rate of development would be so rapid that it would readily be able to out-think and out-maneuver all possible obstacles, and rapidly obtain a decisive strategic advantage. Once in this position, the superintelligence would possess an overwhelming advantage in technology and resources, and would therefore be effectively impossible to displace.

The main problem with all of Bostrom’s arguments for the plausibility of a fast takeoff is that they are fundamentally circular, in that the scenario or consideration they propose is only plausible or relevant under the assumption that the takeoff (or some key aspect of it) is fast. The arguments he presents are as follows:

• Two subsystems argument: if an AI consists of two or more subsystems with one improving rapidly, but only contributing to the ability of the overall system after a certain threshold is reached, then the rate of increase in the performance of the overall system could drastically increase once that initial threshold is passed. This argument assumes what it is trying to prove, namely that the rate of progress in a critical rate-limiting subsystem could be very rapid, experiencing substantial gains on the order of days or even hours. It is hard to see what Bostrom’s scenario really adds here; all he has done is redescribed the fast takeoff scenario in a slightly more specific way. He has not given any reason for thinking that it is at all probable that progress on such a critical rate-limiting subsystem would occur at the extremely rapid pace characteristic of a fast takeoff.
• Intelligence spectrum argument: Bostrom argues that the intelligence gap between ‘infra-idiot’ and ‘ultra-Einstein’, while appearing very large to us, may actually be quite small in the overall scheme of the spectrum of possible levels of intelligence, and as such the time taken to improve an AI through and beyond this level may be much less than it originally seems. However, even if it is the case that the range of the intelligence spectrum within which all humans fall is fairly narrow in the grand scheme of things, it does not follow that the time taken to traverse it in terms of AI development is likely be on the order of days or weeks. Bostrom is simply making an assumption that such rapid rates of progress could occur. His intelligence spectrum argument can only ever show that the relative distance in intelligence space is small; it is silent with respect to likely development timespans.
• Content overhang argument: an artificial intelligence could be developed with high capabilities but with little raw data or content to work with. If large quantities of raw data could be processed quickly, such an AI could rapidly expand its capabilities. The problem with this argument is that what is most important is not how long it takes a given AI to absorb some quantity of data, but rather the length of time between producing one version of the AI and the next, more capable version. This is because the key problem is that we currently don’t know how to build a superintelligence. Bostrom is arguing that if we did build a nascent superintelligence that simply needed to process lots of data to manifest its capabilities, then this learning phase could occur quickly. He gives no reason, however, to think that the rate at which we can learn how to build that nascent superintelligence (in other words, the overall rate of progress in AI research) will be anything like as fast as the rate an existing nascent superintelligence would be able to process data. Only if we assume rapid breakthroughs in AI design itself does the ability of AIs to rapidly assimilate large quantities of data become relevant.
• Hardware overhang argument: it may be possible to increase the capabilities of a nascent superintelligence dramatically and very quickly by rapidly increasing the scale and performance of the hardware it had access to. While theoretically possible, this is an implausible scenario since any artificial intelligence showing promise would likely be operating near the peak of plausible hardware provision. This means that testing, parameter optimisation, and other such tasks will take considerable time, as hardware will be a limiting factor. Bostrom’s concept of a ‘hardware overhang’ amounts to thinking that AI researchers would be content to ‘leave money on the table’, in the sense of not making use of what hardware resources are available to them for extended periods of development. This is especially implausible in the event of groundbreaking research involving AI architectures showing substantial promise. Such systems would hardly be likely to spend years being developed on relatively primitive hardware only to be suddenly and very rapidly dramatically scaled up at the precise moment when practically no further development is necessary, and they are already effectively ready to achieve superhuman intelligence.
• ‘One key insight’ argument: Bostrom argues that ‘if human level AI is delayed because one key insight long eludes programmers, then when the final breakthrough occurs, the AI might leapfrog from below to radically above human level’. Assuming that ‘one key insight’ would be all it would take to crack the problem of superhuman intelligence is, to my mind, grossly implausible, and not consistent either with the slow but steady rate of progress in artificial intelligence research over the past 60 years, or with the immensely complex and multifaceted phenomenon that is human intelligence.

Additional positive arguments against the plausibility of a fast takeoff include the following:

• Speed of science: Bostrom’s assertion that artificial intelligence research could develop from clearly sub-human to obviously super-human levels of intelligence in a matter of days or hours is simply absurd. Scientific and engineering projects simply do not work over timescales that short. Perhaps to some degree this could be altered in the future if (for example) human-level intelligence could be emulated on a computer and then the simulation run at much faster than real-time. But Bostrom’s argument is that machine intelligence is likely to precede emulation, and as such all we will have to work with at least up to the point of human/machine parity being reached is human levels of cognitive ability. As such it seems patently absurd to argue that developments of this magnitude could be made on the timespan of days or weeks. We simply see no examples of anything like this from history, and Bostrom cannot argue that the existence of superintelligence would make historical parallels irrelevant, since we are precisely talking about the development of superintelligence in the context of it not already being in existence.
• Subsystems argument: any superintelligent agent will doubtlessly require many interacting and interconnected subsystems specialised for different tasks. This is the way even much narrower AIs work, and it is certainly how human cognition works. Ensuring that all these subsystems or processes interact efficiently, without one inappropriately dominating or slowing up overall cognition, or without bottlenecks of information transfer or decision making, is likely to be something that requires a great deal of experimentation and trial-and-error. This in turn will take extensive empirical experiments, tinkering, and much clever work. All this takes time.
• Parallelisation problems: many algorithms cannot be sped up considerably by simply adding more computational power unless an efficient way can be found to parallelise them, meaning that they can be broken down into smaller steps which can be performed in parallel across many processors at once. This is much easier to do for some types of algorithms and computations than others. It is not at all clear that the key algorithms used by a superintelligence would be susceptible to parallelisation. Even if they were, developing efficient parallelised forms of the relevant algorithms would itself be a prolonged process. The superintelligence itself would only be able to help in this development to the degree permitted by its initially limited hardware endowment. We therefore would expect to observe gradual improvement of algorithmic efficiency in parallelisation, thereby enabling more hardware to be added, thereby enabling further refinements to the algorithms used, and so on. It is therefore not at all clear that a superintelligence could be rapidly augmented simply by ‘adding more hardware’.
• Need for experimentation: even if a superintelligence came into existence quite rapidly, it would still not be able to achieve a decisive strategic advantage in similarly short time. This is because such an advantage would almost certainly require development of new technologies (at least the examples Bostrom gives almost invariably involve the AI using technologies currently unavailable to humans), which would in turn require scientific research. Scientific research is a complex activity that requires far more than skill at ‘prediction and means-end reasoning’. In particular, it also generally requires experimental research and (if engineering of new products is involved) producing and testing of prototypes. All of this will take time, and crucially is not susceptible to computational speedup, since the experiments would need to be performed with real physical systems (mechanical, biological, chemical, or even social). The idea that all (or even most) such testing and experimentation could be replaced by computer simulation of the relevant system is absurd, since most such simulations are completely computationally intractable, and likely to remain so for the foreseeable future (in many cases possibly forever). Therefore in the development of new technologies and scientific knowledge, the superintelligence is still fundamentally limited by the rate at which real-world tests and experiments can be performed.
• The infrastructure problem: in addition to the issue of developing new technologies, there is the further problem of the infrastructure required to develop such technologies, or even just to carry out the core objectives of the superintelligence. In order to acquire a decisive strategic advantage, a superintelligence will require vast computational resources, energy sources to supply them, real-world maintenance of these facilities, sources of raw materials, and vast manufacturing centres to produce any physical manipulators or other devices it requires. If it needs humans to perform various tasks for it, it will likely also require training facilities and programs for its employees, as well as teams of lawyers to acquire all the needed permits and permissions, write up contracts, and lobby governments. All of this physical and social infrastructure cannot be built in the matter of an afternoon, and more realistically would take many years or even decades to put in place. No amount of superintelligence can overcome physical limitations of the time required to produce and transform large quantities of matter and energy into desired forms. One might argue that improved technology certainly can reduce the time taken to move matter and energy, but the point is that it can only do so after the technology has been embodied in physical forms. The superintelligence would not have access to such hypothetical super-advanced transportation, computation, or construction technologies until it had built the factories needed to produce the machine tools with are needed to precisely refine the raw materials needed for parts in the construction of the nanofactory... and so on for many other similar examples. Nor can even vast amounts of money and intelligence allow any agent to simply brush aside the impediments of the legal system and government bureaucracy in an afternoon. A superintelligence would not simply be able to ignore such social restrictions on its actions until after it had gained enough power to act in defiance of world governments, which it would not be able to do until it had already acquired considerable military capabilities. All of this would take considerable time, precluding a fast takeoff.