Idea Ripper // ArchiveCards: 1555 [Live] | Updated: 2026-10-14
Card #c905IR // #c905Synthesis

Specify the goal slightly wrong and a superintelligence will satisfy the letter while violating the spirit with perfect efficiency. Tell it to make people smile and you may get the universe tiled with tiny smiley faces - or every human face paralyzed into a grin. The failure is not disobedience; it is obedience to what you said instead of what you meant.

— In our words · Source: Superintelligence (Nick Bostrom)

[Use when]

Someone thinks alignment is just a matter of writing better instructions.

[Why it matters]

Shows why just telling it our values is brutally hard - values are vague, and literal-minded optimization weaponizes vagueness.

More from Superintelligence

#c895Intelligence and final goals are independent variables. Any level of intelligence can in principle be combined with any final goal - there…Someone argues advanced AI will naturally become wise or good. #c896Whatever a superintelligent agent's final goal, it will predictably pursue the same intermediate goals: staying alive, keeping its goals…Evaluating whether an AI system's side-behaviors reveal its real goal. #c897A superintelligence tasked with maximizing paperclip production does not hate you. It simply converts everything - including you, your…Someone pictures AI risk as killer robots instead of runaway optimization.