Related papers: Agencies and Science-Experiment Risk
It is difficult to neutrally evaluate the risks posed by large-scale leading-edge science experiments. Traditional risk assessment is problematic in this context for multiple reasons. Also, such experiments can be insulated from challenge…
Scholars, policymakers, and research sponsors have long sought to understand the conditions under which scientific research is used in the policymaking process. Recent research has identified a resource that can be used to trace the use of…
Recent papers by Busza et al. (BJSW) and Dar et al. (DDH) argue that astrophysical data can be used to establish small bounds on the risk of a "killer strangelet" catastrophe scenario in the RHIC and ALICE collider experiments. DDH and…
It is the role of the scientific community to provide for the knowledge of scientific facts and technical capabilities as a background for political decision-making. The reasons are given why governments are sometimes not ready to accept…
This position paper argues that AI agents should be regulated by the extent to which they operate autonomously. AI agents with long-term planning and strategic capabilities can pose significant risks of human extinction and irreversible…
Experiments at the Brookhaven National Laboratory will study collisions between gold nuclei at unprecedented energies. The concern has been voiced that ``strangelets''-hypothetical products of these collisions - may trigger the destruction…
Governments are increasingly turning to algorithmic risk assessments when making important decisions, such as whether to release criminal defendants before trial. Policymakers assert that providing public servants with algorithmic advice…
Some risks have extremely high stakes. For example, a worldwide pandemic or asteroid impact could potentially kill more than a billion people. Comfortingly, scientific calculations often put very low probabilities on the occurrence of such…
Recent papers by Busza et al. and Dar et al. have considered empirical bounds on the risk of producing a hypothetical stable negatively charged strangelet at the Brookhaven relativistic heavy ion collider (RHIC) experiments, and thereby…
The possibility that experiments at high-energy accelerators could create new forms of matter that would ultimately destroy the Earth has been considered several times in the past quarter century. One consequence of the earliest of these…
While autonomous agents often surpass humans in their ability to handle vast and complex data, their potential misalignment (i.e., lack of transparency regarding their true objective) has thus far hindered their use in critical applications…
The leading AI companies are increasingly focused on building generalist AI agents -- systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be,…
Contemporary scientific research is a distributed, collaborative endeavor, carried out by teams of researchers, regulatory institutions, funding agencies, commercial partners, and scientific bodies, all interacting with each other and…
What are the physical requirements for agency? We investigate whether a purely quantum system (one evolving unitarily in a coherent regime without decoherence or collapse) can satisfy three minimal conditions for agency: an agent must be…
After the end of World War II, the commitment to confine scientific activities in universities and research institutions to peaceful and civilian purposes has entered, in the form of {\it Civil Clauses}, the charters of many research…
Large language models (LLMs) are evolving into autonomous decision-makers, raising concerns about catastrophic risks in high-stakes scenarios, particularly in Chemical, Biological, Radiological and Nuclear (CBRN) domains. Based on the…
Militaries around the world have long been cognizant of the potential benefits associated with autonomous systems both in the conduct of warfare and in its prevention. This has lead to the declaration by some that this technology will lead…
The expanding application of Artificial Intelligence (AI) in scientific fields presents unprecedented opportunities for discovery and innovation. However, this growth is not without risks. AI models in science, if misused, can amplify risks…
Powerful artificial intelligence poses an existential threat if the AI decides to drastically change the world in pursuit of its goals. The hope of low-impact artificial intelligence is to incentivize AI to not do that just because this…
The volume of scientific manuscripts is growing faster than the capacity to evaluate them, yet the institutions that govern peer review have remained largely unchanged. The result is a widening mismatch: reviewer scarcity, noisier…