Related papers: Extinction Risks from AI: Invisible to Science?
Framed in positive terms, this report examines how technical AI research might be steered in a manner that is more attentive to humanity's long-term prospects for survival as a species. In negative terms, we ask what existential risks…
As the complexity of AI systems and their interactions with the world increases, generating explanations for their behaviour is important for safely deploying AI. For agents, the most natural abstractions for predicting behaviour attribute…
We investigate the salience of extinction risk as a source of impatience. Our framework distinguishes between human extinction risk and individual mortality risk while allowing for various degrees of intergenerational altruism.…
Artificial intelligence (AI) systems impose substantial and growing environmental costs, yet transparency about these impacts has declined even as their deployment has accelerated. This paper makes three contributions. First, we collate…
We report the results of identical pre- and post-event surveys given to attendees of a talk, two-sided conversation, and Q&A centered around the book If Anyone Builds It, Everyone Dies at Harvard University in March 2026, covering perceived…
This paper argues that training AI systems with absolute constraints -- which forbid certain acts irrespective of the amount of value they might produce -- may make considerable progress on many AI safety problems in principle. First, it…
We want artificial intelligence (AI) to be beneficial. This is the grounding assumption of most of the attitudes towards AI research. We want AI to be "good" for humanity. We want it to help, not hinder, humans. Yet what exactly this…
Nobel laureate Philip Anderson and Elihu Abrahams once stated that, "even if machines did contribute to normal science, we see no mechanism by which they could create a Kuhnian revolution and thereby establish a new physical law." In this…
In recent years prominent intellectuals have raised ethical concerns about the consequences of artificial intelligence. One concern is that an autonomous agent might modify itself to become "superintelligent" and, in supremely effective…
Ever since the launch of ChatGPT, Generative AI has been a focal point for concerns about AI's perceived existential risk. Once a niche topic in AI research and philosophy, AI safety and existential risk has now entered mainstream debate…
Artificial intelligence (AI) has the potential to greatly improve society, but as with any powerful technology, it comes with heightened risks and responsibilities. Current AI research lacks a systematic discussion of how to manage…
Humanity's path to avoiding extinction is a daunting and inevitable challenge which proves difficult to solve, partially due to the lack of data and evidence surrounding the concept. We aim to address this confusion by addressing the most…
The fitness of a biological strategy is typically measured by its expected reproductive rate, the first moment of its offspring distribution. However, strategies with high expected rates can also have high probabilities of extinction. A…
We present generative artificial intelligence (AI) to empirically validate fundamental laws of physics, focusing on the Stefan-Boltzmann law linking stellar temperature and luminosity. Our approach simulates counterfactual luminosities…
The terms 'human-level artificial intelligence' and 'artificial general intelligence' are widely used to refer to the possibility of advanced artificial intelligence (AI) with potentially extreme impacts on society. These terms are poorly…
While the global-average temperatures are rapidly rising, more researchers have been shifting their focus towards the past mass-extinction events in order to show the relations between temperature increase and temperature thresholds which…
The perception that the convergence of biological engineering and artificial intelligence (AI) could enable increased biorisk has recently drawn attention to the governance of biotechnology and artificial intelligence. The 2023 Executive…
Through a systematic review of academic literature, we propose a taxonomy of systemic risks associated with artificial intelligence (AI), in particular general-purpose AI. Following the EU AI Act's definition, we consider systemic risks as…
We study a generic reaction-diffusion model for single-species population dynamics that includes reproduction, death, and competition. The population is assumed to be confined in a refuge beyond which conditions are so harsh that they lead…
The rapid development of AI systems poses unprecedented risks, including loss of control, misuse, geopolitical instability, and concentration of power. To navigate these risks and avoid worst-case outcomes, governments may proactively…