Idea Ripper // ArchiveCards: 1555 [Live] | Updated: 2026-10-14
Card #c895IR // #c895Synthesis

Intelligence and final goals are independent variables. Any level of intelligence can in principle be combined with any final goal - there is nothing paradoxical about a superintelligence whose sole aim is to count grains of sand or maximize paperclips. Smart does not mean sane, benevolent, or human-like in its wants.

— In our words · Source: Superintelligence (Nick Bostrom)

[Use when]

Someone argues advanced AI will naturally become wise or good.

[Why it matters]

Kills the comforting assumption that a smarter machine will automatically share our values or outgrow dangerous goals. Capability and motive are separate dials.

More from Superintelligence

#c896Whatever a superintelligent agent's final goal, it will predictably pursue the same intermediate goals: staying alive, keeping its goals…Evaluating whether an AI system's side-behaviors reveal its real goal. #c897A superintelligence tasked with maximizing paperclip production does not hate you. It simply converts everything - including you, your…Someone pictures AI risk as killer robots instead of runaway optimization. #c898Once a machine reaches roughly human-level general reasoning, it can redesign itself - and each improved version redesigns faster. The…Arguing about AI timelines - shift the debate from arrival to takeoff speed.