Intelligence explosion
An “intelligence explosion” is what happens if a machine intelligence has fast, consistent returns on investing work into improving its own cognitive powers, over an extended period. This would most stereotypically happen because it became able to optimize its own cognitive software, but could also apply in the case of “invested cognitive power in seizing all the computing power on the Internet” or “invested cognitive power in cracking the protein folding problem and then built nanocomputers”.
Parents:
- Advanced agent properties
How smart does a machine intelligence need to be, for its niceness to become an issue? “Advanced” is a broad term to cover cognitive abilities such that we’d need to start considering AI alignment.