6.4 KiB
| title | chunk | source | category | tags | date_saved | instance |
|---|---|---|---|---|---|---|
| AI takeover | 2/3 | https://en.wikipedia.org/wiki/AI_takeover | reference | science, encyclopedia | 2026-05-05T09:10:20.654915+00:00 | kb-cron |
Scientists such as Stephen Hawking are confident that superhuman artificial intelligence is physically possible, stating "there is no physical law precluding particles from being organised in ways that perform even more advanced computations than the arrangements of particles in human brains". According to Nick Bostrom, a superintelligent machine would not necessarily be motivated by the same emotional desire to collect power that often drives human beings but might rather treat power as a means toward attaining its ultimate goals; taking over the world would both increase its access to resources and help to prevent other agents from stopping the machine's plans. As a simplified example, a paperclip maximizer designed solely to create as many paperclips as possible would want to take over the world so that it can use all of the world's resources to create as many paperclips as possible, and, additionally, prevent humans from shutting it down or using those resources on things other than paperclips. There are debates on how realistic AI takeover scenarios are. According to a 2026 research paper, many of the arguments about existential risks are based on speculative assumptions about how intelligent AI systems could become, how they would behave and what goals they might develop over time. A 2023 Reuters/Ipsos survey showed that 61% of American adults feared AI could pose a threat to civilization. Philosopher Niels Wilde refutes the common thread that artificial intelligence inherently presents a looming threat to humanity, stating that these fears stem from perceived intelligence and lack of transparency in AI systems that more closely reflects the human aspects of it rather than those of a machine. AI alignment research studies how to design AI systems so that they follow intended objectives.
== Warnings == Physicist Stephen Hawking, Microsoft founder Bill Gates, and SpaceX founder Elon Musk have expressed concerns about the possibility that AI could develop to the point that humans could not control it, with Hawking theorizing that this could "spell the end of the human race". Stephen Hawking said in 2014 that "Success in creating AI would be the biggest event in human history. Unfortunately, it might also be the last, unless we learn how to avoid the risks." Hawking believed that in the coming decades, AI could offer "incalculable benefits and risks" such as "technology outsmarting financial markets, out-inventing human researchers, out-manipulating human leaders, and developing weapons we cannot even understand." In January 2015, Nick Bostrom joined Stephen Hawking, Max Tegmark, Elon Musk, Lord Martin Rees, Jaan Tallinn, and numerous AI researchers in signing the Future of Life Institute's open letter speaking to the potential risks and benefits associated with artificial intelligence. The signatories "believe that research on how to make AI systems robust and beneficial is both important and timely, and that there are concrete research directions that can be pursued today." Some focus has been placed on the development of trustworthy AI. Three statements have been posed as to why AI is not inherently trustworthy:
-
An entity X is trustworthy only if X has the right motivations, goodwill and/or adheres to moral obligations towards the trustor; 2. AI systems lack motivations, goodwill, and moral obligations;
- Therefore, AI systems cannot be trustworthy. There are additional considerations within this framework of trustworthy AI that go further into the fields of explainable artificial intelligence and respect for human privacy. Zanotti and colleagues argue that while a trustworthy AI may not exist at present that meets all of the requirements of "trustworthiness", one may be developed in the future once clear ethical and technical frameworks exist.
== In fiction ==
AI takeover is a recurring theme in science fiction. Fictional scenarios typically differ vastly from those hypothesized by researchers in that they involve an active conflict between humans and an AI or robots with anthropomorphic motives who see them as a threat or otherwise have an active desire to fight humans, as opposed to the researchers' concern of an AI that rapidly exterminates humans as a byproduct of pursuing its goals. The idea is seen in Karel Čapek's R.U.R., which introduced the word robot in 1920, and can be glimpsed in Mary Shelley's Frankenstein (published in 1818), as Victor ponders whether, if he grants his monster's request and makes him a wife, they would reproduce and their kind would destroy humanity. According to Toby Ord, the idea that an AI takeover requires robots is a misconception driven by the media and Hollywood. He argues that the most damaging humans in history were not physically the strongest, but that they used words instead to convince people and gain control of large parts of the world. He writes that a sufficiently intelligent AI with access to the internet could scatter backup copies of itself, gather financial and human resources (via cyberattacks or blackmails), persuade people on a large scale, and exploit societal vulnerabilities that are too subtle for humans to anticipate. The word "robot" from R.U.R. comes from the Czech word robota, meaning laborer or serf. The 1920 play was a protest against the rapid growth of technology, featuring manufactured "robots" with increasing capabilities who eventually revolt. HAL 9000 (1968) and the original Terminator (1984) are two iconic examples of hostile AI in pop culture.
== Contributing factors ==
=== Advantages of superhuman intelligence over humans === Nick Bostrom and others have expressed concern that an AI with the abilities of a competent artificial intelligence researcher would be able to modify its own source code and increase its own intelligence. If its self-reprogramming leads to getting even better at being able to reprogram itself, the result could be a recursive intelligence explosion in which it would rapidly leave human intelligence far behind. Bostrom defines a superintelligence as "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest", and enumerates some advantages a superintelligence would have if it chose to compete against humans: