Valley of Dangerous Complacency

The Valley of Danger­ous Com­pla­cency is when a sys­tem works of­ten enough that you let down your guard around it, but in fact the sys­tem is still dan­ger­ous enough that full vigilance is re­quired.

  • If a robotic car made the cor­rect de­ci­sion 99% of the time, you’d need to grab the steer­ing wheel on a daily ba­sis, you’d stay alert and your robot-car-over­rid­ing skills would stay sharp.

  • If a robotic car made the cor­rect de­ci­sion 100% of the time, you’d re­lax and let your guard down, but there wouldn’t be any­thing wrong with that.

  • If the robotic car made the cor­rect de­ci­sion 99.99% of the time, so that you need to grab the steer­ing wheel or else crash in 1 of 100 days, the task of mon­i­tor­ing the car would feel very un­re­ward­ing and the car would seem pretty safe. You’d let your guard down and your driv­ing skills would get rusty. After a cou­ple of months, the car would crash.

Com­pare “Un­canny Valley” where a ma­chine sys­tem is par­tially hu­man­like—hu­man­like enough that hu­mans try to hold it to a hu­man stan­dard—but not hu­man­like enough to ac­tu­ally seem satis­fac­tory when held to a hu­man stan­dard. This means that in terms of user ex­pe­rience, there’s a valley as the de­gree of hu­man­like­ness of the sys­tem in­creases where the user ex­pe­rience ac­tu­ally gets worse be­fore it gets bet­ter. Similarly, if users be­come com­pla­cent, a 99.99% re­li­able sys­tem can be worse than a 99% re­li­able one, even though, with enough re­li­a­bil­ity, the de­gree of safety starts climb­ing back out of the valley.


  • AI safety mindset

    Ask­ing how AI de­signs could go wrong, in­stead of imag­in­ing them go­ing right.