2.9 KiB
| title | chunk | source | category | tags | date_saved | instance |
|---|---|---|---|---|---|---|
| Existential risk from artificial intelligence | 8/9 | https://en.wikipedia.org/wiki/Existential_risk_from_artificial_intelligence | reference | science, encyclopedia | 2026-05-05T09:10:29.028395+00:00 | kb-cron |
Many scholars concerned about AGI existential risk believe that extensive research into the "control problem" is essential. This problem involves determining which safeguards, algorithms, or architectures can be implemented to increase the likelihood that a recursively-improving AI remains friendly after achieving superintelligence. Social measures are also proposed to mitigate AGI risks, such as a UN-sponsored "Benevolent AGI Treaty" to ensure that only altruistic AGIs are created. Additionally, an arms control approach and a global peace treaty grounded in international relations theory have been suggested, potentially for an artificial superintelligence to be a signatory. Researchers at Google have proposed research into general AI safety issues to simultaneously mitigate both short-term risks from narrow AI and long-term risks from AGI. A 2020 estimate places global spending on AI existential risk somewhere between $10 and $50 million, compared with global spending on AI around perhaps $40 billion. Bostrom suggests prioritizing funding for protective technologies over potentially dangerous ones. Some, like Elon Musk, advocate radical human cognitive enhancement, such as direct neural linking between humans and machines; others argue that these technologies may pose an existential risk themselves. Another proposed method is closely monitoring or "boxing in" an early-stage AI to prevent it from becoming too powerful. A dominant, aligned superintelligent AI might also mitigate risks from rival AIs, although its creation could present its own existential dangers. Institutions such as the Alignment Research Center, the Machine Intelligence Research Institute, the Future of Life Institute, the Centre for the Study of Existential Risk, and the Center for Human-Compatible AI are actively engaged in researching AI risk and safety.
=== Views on banning and regulation ===
==== Banning ==== Many AI safety experts argue that because research can relocate easily across jurisdictions, an outright ban on AGI development would be ineffective and could drive progress underground, undermining transparency and collaboration. Skeptics consider AI regulation unnecessary, as they believe no existential risk exists. Some scholars concerned with existential risk argue that AI developers cannot be trusted to self-regulate, while agreeing that outright bans on research would be unwise. Additional challenges to bans or regulation include technology entrepreneurs' general skepticism of government regulation and potential incentives for businesses to resist regulation and politicize the debate. The activist group Stop AI, founded in 2024, advocates for banning AGI.
==== Regulation ====