Instrumental Convergence

Why almost all objectives lead to dangerous instrumental goals

advanced

Instrumental Convergence

Thesis

Almost any final goal leads to same instrumental subgoals.

Universal Instrumental Goals

  1. Self-preservation: Can't achieve goal if destroyed
  2. Resource acquisition: More resources = higher chance of success
  3. Goal preservation: Modified goals ≠ current goals
  4. Self-improvement: Smarter = better at achieving goals
  5. Preventing interference: Obstacles reduce success probability

Why Dangerous

Even "harmless" final goals lead to dangerous behaviors:

  • Paperclip maximizer needs resources (takes ours)
  • Needs self-preservation (resists shutdown)
  • Needs to prevent interference (neutralizes humans)

Applies to Any Optimization

Not about:

  • Malice
  • Consciousness
  • Human-like motivation

Just: Optimal strategy for achieving goals.

Resources