Standard agent properties

Boundedly rational agents

• Have probabilistic models of the world.

• Update those models in response to sensory information.

• The ideal algorithm for updating is Bayesian inference, but this requires too much computing power and a bounded agent must use some bounded alternative.

• Implicitly, we assume the agent has some equivalent of a complexity-penalizing prior or Occam’s Razor. Without this, specifying Bayesian inference does not much constrain the end results of epistemic reasoning.

• Have preferences over events or states of the world, quantifiable by a utility function that maps those events or states onto a scalar field.

• These preferences must be quantitative, not just ordered, in order to combine with epistemic states of uncertainty (probabilities).

• Are consequentialist: they evaluate the expected consequences of actions and choose among actions based on preference among their expected consequences.

• Bounded agents cannot evaluate all possible actions and hence cannot obtain literal maximums of expected utility except in very simple cases.

• Act in real time in a noisy, uncertain environment.

For the arguments that sufficiently intelligent agents will appear to us as boundedly rational agents in some sense, see:

• Relevant powerful agents will be highly optimized

• Sufficiently optimized agents appear coherent

Economic agents

• Achieve their goals by efficiently allocating limited resources, including, e.g., time, money, or negentropy;

• Try to find new paths that route around obstacles to goal achievement;

• Predict the actions of other agents;

• Try to coordinate with, manipulate, or hinder other agents (in accordance with the agent’s own goals or utilities);

• Respond to both negative incentives (penalties) and positive incentives (rewards) by planning accordingly, and may also consider strategies to avoid penalties or gain rewards that were unforeseen by the creators of the incentive framework.

Naturalistic agents

• Naturalistic agents are embedded in a larger universe and are made of the same material as other things in the universe (wavefunction, on our current beliefs about physics).

• A naturalistic agent’s uncertainty about the environment is uncertainty about which natural universe embeds them (what material structure underlies their available sensory and introspective data).

• Some of the actions available to naturalistic agents potentially alter their sensors, actuators, or computing substrate.

• Sufficiently powerful naturalistic agents may construct other agents out of resources available to them internally or in their environment, or extend their intelligence into outside computing resources.

• A naturalistic agent’s sensing, cognitive, and decision/​action capabilities may be distributed over space, time, and multiple substrates; the applicability of the ‘agent’ concept does not require a small local robot body.