ABSTRACT
Friendly Artificial Intelligence (FAI) is the project of ensuring that the world's superintelligences are safe and useful for humans. Bostrom observed that the FAI problem is less difficult than creating the first artificial general Intelligence (AGI). Brain–computer interfaces have made impressive advances, and a world in which humans become the superintelligences is surely prefera ble to one in which humans might be enslaved or killed by machine superintelligences. Cynics have sneered that the solution to a rogue superintelligence is simple: just flick the off-switch. The reason why relinquishment is hard is that the incentive for developing better and better Artificial Intelligence is too strong. The hope that intelligence augmentation can forestall AGI and artificial superintelligence is probably a forlorn one. Linking computers and human brains to enable simple tasks such as manoeuvring a robot arm is one thing. The most hopeful version of the indirect approach to controlling superintelligence’s actions is to add another level of indirectness.
