In the con­text of value al­ign­ment the­ory, ‘Utility’ always refers to a goal held by an ar­tifi­cial agent. It fur­ther im­plies that the agent is a con­se­quen­tial­ist; that the agent has prob­a­bil­is­tic be­liefs about the con­se­quences of its ac­tions; that the agent has a quan­ti­ta­tive no­tion of “how much bet­ter” one out­come is than an­other and the rel­a­tive size of differ­ent in­ter­vals of bet­ter­ness; and that the agent can there­fore, e.g., trade off large prob­a­bil­ities of a small util­ity gain against small prob­a­bil­ities of a large util­ity loss.

True co­her­ence in the sense of a von-Neu­mann Mor­gen­stern util­ity func­tion may be out of reach for bounded agents, but the term ‘util­ity’ may also be used for the bounded analogues of such de­ci­sion-mak­ing, pro­vided that quan­ti­ta­tive rel­a­tive in­ter­vals of prefer­abil­ity are be­ing com­bined with quan­ti­ta­tive de­grees of be­lief to yield de­ci­sions.

Utility is ex­plic­itly not as­sumed to be nor­ma­tive. E.g., if speak­ing of a pa­per­clip max­i­mizer, we will say that an out­come has higher util­ity iff it con­tains more pa­per­clips.

Hu­mans should not be said (with­out fur­ther jus­tifi­ca­tion) to have util­ities over com­pli­cated out­comes. On the main­stream view from psy­chol­ogy, hu­mans are in­con­sis­tent enough that it would take ad­di­tional as­sump­tions to trans­late our psy­chol­ogy into a co­her­ent util­ity func­tion. E.g., we may differ­ently value the in­ter­val be­tween two out­comes de­pend­ing on whether the in­ter­val is framed as a ‘gain’ or a ‘loss’. For the things hu­mans do or should want, see the spe­cial use of the word ‘value’. For a gen­eral dis­am­bigua­tion page on words used to talk about hu­man and AI wants, see Lin­guis­tic con­ven­tions in value al­ign­ment.

On some con­stru­als of value, e.g. re­flec­tive equil­ibrium, this con­strual may im­ply that the true val­ues form a co­her­ent util­ity func­tion. Nonethe­less, by con­ven­tion, we will not speak of value as a util­ity un­less it has been spel­led out that, e.g., the value in ques­tion has been as­sumed to be a re­flec­tive equil­ibrium.

Mul­ti­ple agents with differ­ent util­ity func­tions should not be said (with­out fur­ther ex­po­si­tion) to have a col­lec­tive util­ity func­tion over out­comes, since at pre­sent, there is no ac­cepted canon­i­cal way to ag­gre­gate util­ity func­tion­slink loud­ness prob­lem.