Before artificial general intelligence existed, before a superintelligence was created, some clever people observed that if we succeeded in creating machines smarter than we were that humans would have no way of determining what would happen next. A superintelligence would lack the ability even to describe to us what it was doing and why it was doing it. It would be in the same situation as a human trying to describe to a dog why they were writing a technical manual. Not only would the dog not understand what a technical manual was, but what writing was or the book’s subject! Those same people also observed that a superintelligence might learn to whistle in ways that would make humans heel.
– Professor Holly Wu Continue reading The Memphis Project III