Ideal target

The ‘ideal tar­get’ of a meta-util­ity func­tion \(\Delta U\) which be­haves as if a ground-level util­ity func­tion \(U\) is tak­ing on differ­ent val­ues in differ­ent pos­si­ble wor­lds, is the value of \(U\) in the ac­tual world; or the ex­pected value of \(U\) af­ter up­dat­ing on all pos­si­ble ac­cessible ev­i­dence. If choco­late has €8 util­ity in wor­lds where the sky is blue, and €5 util­ity in wor­lds where the sky is not blue, then in the AI’s ‘ideal tar­get’ util­ity func­tion, the util­ity of choco­late is €8.

Parents:

  • Moral uncertainty

    A meta-util­ity func­tion in which the util­ity func­tion as usu­ally con­sid­ered, takes on differ­ent val­ues in differ­ent pos­si­ble wor­lds, po­ten­tially dis­t­in­guish­able by ev­i­dence.