Rich domain
A domain is ‘rich’, relative to our own intelligence, to the extent that (1) its search space is too large and irregular for all of the best strategies to be searched by our own intelligence, and (2) the known mechanics of the domain do not permit us to easily place absolute bounds against some event occurring or some goal of interest being strategically obtainable.
For the pragmatic implications, see Cognitive uncontainability and Almost all real-world domains are rich.
spectrum: Logical tic-tac-toe, logical chess, logical go, real-world tic-tac-toe, human brain, Internet, real world
since this is about advanced safety, we have to assume that there’s a smarter-than-us intelligence that could seek out and exploit the tiniest loophole in our reasoning about how safe we are, and accordingly be safe and conservative in what we think is a ‘narrow’ domain, and flag every time we rely on the assumption that the agent’s strategy space is narrow and therefore we can cognitively contain something smarter than us
talk about that old example of the evolutionary algorithm that used transistors to form a radio to form a clock circuit; gravity means that everything interacts with everything
paradigmatic example: That Alien Message
<div>
Children:
- Logical game
Game’s mathematical structure at its purest form.
- Almost all real-world domains are rich
Anything you’re trying to accomplish in the real world can potentially be accomplished in a lot of different ways.
Parents:
- Cognitive uncontainability
‘Cognitive uncontainability’ is when we can’t hold all of an agent’s possibilities inside our own minds.