The ‘pro­gram­mers’ are the hu­man be­ings cre­at­ing what­ever ad­vanced agent is be­ing talked about.

The word ‘pro­gram­mer’ isn’t meant to im­ply that the AI is hard­coded as a fixed al­gorithm and then run; it’s just the com­pact word we use to re­fer to the AI’s cre­ators. (E.g., the cre­ation pro­cess could in­clude ex­plic­itly cod­ing al­gorithms, teach­ing the agent, ex­pos­ing it to par­tic­u­lar ex­pe­riences, la­bel­ing its ex­pe­riences for pur­poses of su­per­vised learn­ing, or any num­ber of other nur­tur­ing pro­cesses ex­tend­ing over years.)

Many pro­pos­als for an AI’s prefer­ence frame­work, or solv­ing the value iden­ti­fi­ca­tion prob­lem, re­quire that the AI ex­plic­itly model the pro­gram­mers and, even be­fore then, figure out which ob­jects in­side the AI’s be­liefs about the en­vi­ron­ment are the pro­gram­mers. Th­ese are re­spec­tively the pro­gram­mer mod­el­ing and pro­gram­mer iden­ti­fi­ca­tion prob­lems.


  • AI alignment

    The great civ­i­liza­tional prob­lem of cre­at­ing ar­tifi­cially in­tel­li­gent com­puter sys­tems such that run­ning them is a good idea.