To those who don’t know Bostrom, he’s a prof of filosofy at Oxford. His book concerns superintelligence which is defined “any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest“, even right-tail performance. The book is basically a series of comments about how to best go on about developing this and the dangers it entails.
Perhaps the most interesting part of the book was this:
Box 9 Strange solutions from blind search
Even simple evolutionary search processes sometimes produce highly unexpected results, solutions that satisfy a formal user-defined criterion in a very different way than the user expected or intended.
The field of evolvable hardware offers many illustrations of this phenomenon. In this field, an evolutionary algorithm searches the space of hardware designs, testing the fitness of each design by instantiating it physically on a rapidly reconfigurable array or motherboard. The evolved designs often show remarkable economy. For instance, one search discovered a frequency discrimination circuit that functioned without a clock—a component normally considered necessary for this function. The researchers estimated that the evolved circuit was between one and two orders of magnitude smaller than what a human engineer would have required for the task. The circuit exploited the physical properties of its components in unorthodox ways; some active, necessary components were not even connected to the input or output pins! These components instead participated via what would normally be considered nuisance side effects, such as electromagnetic coupling or power-supply loading.
Another search process, tasked with creating an oscillator, was deprived of a seemingly even more indispensible component, the capacitor. When the algorithm presented its successful solution, the researchers examined it and at first concluded that it “should not work.” Upon more careful examination, they discovered that the algorithm had, MacGyver-like, reconfigured its sensor-less motherboard into a makeshift radio receiver, using the printed circuit board tracks as an aerial to pick up signals generated by personal computers that happened to be situated nearby in the laboratory. The circuit amplified this signal to produce the desired oscillating output.16
In other experiments, evolutionary algorithms designed circuits that sensed whether the motherboard was being monitored with an oscilloscope or whether a soldering iron was connected to the lab’s common power supply. These examples illustrate how an open-ended search process can repurpose the materials accessible to it in order to devise completely unexpected sensory capabilities, by means that conventional human design-thinking is poorly equipped to exploit or even account for in retrospect.
The tendency for evolutionary search to “cheat” or find counterintuitive ways of achieving a given end is on display in nature too, though it is perhaps less obvious to us there because of our already being somewhat familiar with the look and feel of biology, and thus being prone to regarding the actual outcomes of natural evolutionary processes as normal—even if we would not have expected them ex ante. But it is possible to set up experiments in artificial selection where one can see the evolutionary process in action outside its familiar context. In such experiments, researchers can create conditions that rarely obtain in nature, and observe the results.