{"id":16075,"date":"2024-12-06T14:10:05","date_gmt":"2024-12-06T17:10:05","guid":{"rendered":"https:\/\/www.fie.undef.edu.ar\/ceptm\/?p=16075"},"modified":"2024-12-06T14:10:05","modified_gmt":"2024-12-06T17:10:05","slug":"riesgos-de-la-inteligencia-artificial-en-los-sistemas-de-comando-y-control-de-armas-nucleares","status":"publish","type":"post","link":"https:\/\/www.fie.undef.edu.ar\/ceptm\/?p=16075","title":{"rendered":"Riesgos de la inteligencia artificial en los sistemas de comando y control de armas nucleares"},"content":{"rendered":"

En el marco del Asia- Pacific Cooperation Economic Summit, el\u00a016Nov24 los l\u00edderes de EEUU y China\u00a0se reunieron en Lima (Per\u00fa), donde acordaron de manera conjunta, \u201cla necesidad de mantener el control del operador humano, sobre las decisiones relacionadas con el uso de armas nucleares<\/em>\u201d. Esta declaraci\u00f3n complementa el documento firmado en 2022 por USA, Francia y RUGB (UK), durante el proceso de revisi\u00f3n del \u201cNuclear Proliferation Treaty\u201d vigente a la fecha. Los pa\u00edses seguir\u00e1n priorizando desarrollos relacionados con la aplicaci\u00f3n de la IA en los sistemas de armas militares, lo que incluir\u00e1 asimismo a los programas de modernizaci\u00f3n de las armas nucleares. Por lo expresado, se alerta acerca de la necesidad de una estricta regulaci\u00f3n sobre el \u00e1rea mencionada.<\/p>\n

\n

On Nov. 16, U.S. and Chinese leaders met on the margins of the Asia-Pacific Economic Cooperation summit in Lima, Peru,\u00a0jointly<\/a>\u00a0affirming<\/a>\u00a0\u201cthe need to maintain human control over the decision to use nuclear weapons.\u201d This declaration echoes a joint\u00a0document<\/a>\u00a0submitted by France, the United Kingdom, and the United States during the Nuclear Nonproliferation Treaty review process in 2022.<\/p>\n

With countries increasingly prioritizing military applications of AI, integrating AI into nuclear weapons systems is becoming a distinct possibility, especially as nuclear arsenals undergo modernization. While some nuclear-weapon states have emphasized the importance of maintaining human oversight and control over decisions to employ nuclear weapons, it is too early to take a victory lap.\u00a0Avoiding a \u201cSkynet<\/a>\u201d scenario, where AI takes independent control of nuclear weapons, does little to reduce the real risks of unintended nuclear launches.<\/p>\n

AI holds the promise of enhancing the performance and capabilities of\u00a0nuclear command, control, and communications systems<\/a>, which form the backbone of nuclear decision-making. However, if integrated with haste and without adequate risk assessment, safeguards, and redundancies, such integration could dramatically heighten the risk of unintended nuclear escalation. Escalation risks can arise from altered decision-making dynamics, accelerated processing speeds that outpace human supervision, or insidious errors that can propagate undetected through complex systems \u2014 regardless of whether humans remain in the decision-making loop.<\/p>\n

To prevent nuclear calamity and ensure the responsible use of AI in nuclear command-and-control, states should move beyond mere prescriptive commitments to human oversight. Reducing the risk of unintended nuclear escalation requires a governance framework that establishes a quantitative threshold for the maximum acceptable probability of an accidental nuclear launch as a uniform safety benchmark.\u00a0Valuable governance lessons can be drawn from civil nuclear safety regulation, in particular what regulators refer to as the \u201crisk-informed\u201d and \u201cperformance-based\u201d safety governance approach<\/a>. Applying these principles to nuclear command-and-control systems requires moving beyond the simplistic human-in-the-loop prescription to focus on assessing the system\u2019s safety performance. The objective is to assess the quantitative likelihood of an accidental nuclear launch with a particular configuration of AI and non-AI subsystems and to ensure that that likelihood remains securely below an acceptable threshold.<\/p>\n

AI\u2019s Impact on Nuclear Risks<\/b><\/p>\n

Assessing how AI can impact the nuclear domain and contribute to unintended escalation is no easy task. The current limited understanding of the behavior of AI models, their rapid and unpredictable advancement, and the complexity and opacity of nuclear systems and subsystems that feed into the decision-making process make this discussion largely speculative. Despite this, it is still possible to foresee how states\u00a0might consider<\/a>\u00a0implementing AI as part of broader efforts to modernize aging nuclear arsenals based on existing nuclear postures and states\u2019 desire to gain a strategic advantage.<\/p>\n

For instance, Gen. Anthony J. Cotton, commander of U.S. Strategic Command, has\u00a0pointed to<\/a>\u00a0AI\u2019s potential to automate data collection, streamline processing, and accelerate data sharing with allies.\u00a0Similarly,\u00a0official<\/a>\u00a0statements<\/a>\u00a0and\u00a0documents<\/a>\u00a0from other\u00a0nuclear<\/a>\u00a0powers often frame AI as a tool to assist human decision-makers to make faster and more informed decisions, beyond the nuclear domain.<\/p>\n

In principle, AI\u2019s ability to analyze vast amounts of data from diverse sources is well-suited to identify threats quickly, analyze sensor data, automate the identification of objects, and evaluate potential courses of action. However, AI introduces a number of significant risks due to the\u00a0inherent limitations<\/a>\u00a0of today\u2019s advanced AI models.<\/p>\n

First, AI is unreliable. Today\u2019s AI can confidently generate false information that can lead to flawed predictions and recommendations, ultimately skewing decision-making. This phenomenon is termed \u201challucinations.\u201d Examples include a large language model generating incorrect facts about historical events, or a vision model \u201cseeing\u201d objects that are not there. Second, the opacity of AI systems \u2014 known as the \u201cblack box\u201d problem \u2014 makes it difficult to fully understand how an AI system reaches its conclusions. This lack of transparency undermines trust and reduces the utility of AI in high-stakes environments like nuclear decision-making, where transparency is crucial. Third, AI systems are susceptible to cyberattacks, creating opportunities for adversaries to compromise the integrity of nuclear command-and-control systems. Finally, current AI models struggle to align outputs with human goals and values, potentially deviating from strategic objectives. The high-pressure environment of nuclear decision-making, combined with limited response time, exacerbates these dangers, as decisions may rely on inaccurate, opaque, compromised, or misaligned information.<\/p>\n

Despite the declarations of some nuclear-armed states to maintain human control in nuclear decision-making, not all of them have explicitly committed to this, leaving room for grave consequences due to misunderstandings or misinterpretations of countries\u2019 intent. But even if all nuclear states made similar declarations, there is no simple way to verify these commitments. Moreover,\u00a0human\u2013machine interaction<\/a>\u00a0itself can introduce severe risks. Operators may place excessive trust in an AI system, relying on its outputs without sufficient scrutiny, or they may distrust it entirely, hesitating to act when speed is critical. Both situations can skew decision-making processes even when AI systems function as intended.\u00a0All of these limitations persist even when states maintain human oversight.<\/p>\n

Further compounding these risks is the uncertainty surrounding AI\u2019s future advancements. While current limitations may eventually be resolved, new risks could also emerge that remain unpredictable at this stage.<\/p>\n

The Precedent of Civil Nuclear Safety Regulation<\/b><\/p>\n

While the risks of AI-integrated command-and-control may seem novel,\u00a0the management of nuclear risks with severe consequences for public health and safety is not a new challenge for governments. Indeed, the principles of risk-informed, performance-based, and technology-neutral regulation \u2014 drawn from the governance of civil nuclear safety \u2014 may usefully apply to the nexus of AI and nuclear command-and-control.<\/p>\n

In the United States, the process of \u201crisk-informing\u201d the regulation of nuclear safety began with the 1975 Reactor Safety Study. This\u00a0quantified<\/a>\u00a0the risks of accidents and radioactive releases associated with nuclear power generation using probabilistic-risk-assessment techniques such as event trees and fault trees. Simply put, these techniques map out the various sequences of cascading events, including system failures, that could ultimately lead to an accident, allowing the probabilities of various consequences to be quantified.<\/p>\n

Prior to the quantification of risks, regulations were based primarily on prescriptive and deterministic requirements. For instance, regulators\u00a0prescribed<\/a>\u00a0multiple redundant safety features to prevent certain foreseen accidents without explicitly considering the likelihood of any given accident sequence. After the 1979 Three Mile Island accident, the Nuclear Regulatory Commission\u00a0expanded<\/a>\u00a0its research into the more extensive application of probabilistic-risk-assessment techniques. This was recommended by investigations after the accident,\u00a0culminating<\/a>\u00a0in a 1995 policy statement and subsequent plans to \u201crisk-inform\u201d the commission\u2019s safety regulation.<\/p>\n

Meanwhile, industry pushed for the more extensive use of performance-based regulation giving the licensee greater flexibility in determining how to accomplish a defined safety goal. Rather than specifying what safety features must be included in the reactor design, a performance-based regulatory requirement would simply\u00a0establish<\/a>\u00a0a quantifiable safety outcome. In its public communication, the Nuclear Regulatory Commission\u00a0illustrates<\/a>\u00a0its performance-based approach using a skydiving example. In this case, the regulator would institute a \u201cperformance requirement\u201d that \u201cthe parachute must open above an altitude of 5,000 feet\u201d without specifying whether that outcome should be ensured with a rip-cord or an automatic activation device.<\/p>\n

Guided by the qualitative safety goal that nuclear power plant operation should not contribute significantly to individual and societal risks, by 1986 the Nuclear Regulatory Commission had\u00a0defined<\/a>\u00a0a measurable benchmark that \u201cthe overall mean frequency of a large release of radioactive materials to the environment from a reactor accident should be less than 1 in 1,000,000 per year of reactor operation.\u201d That benchmark has since been\u00a0refined<\/a>\u00a0into more operationalizable standards.<\/p>\n

In recent years, as diverse and novel reactor concepts emerged, it became clear that many safety features prescribed for traditional reactors were no longer applicable. Regulators have therefore\u00a0prioritized<\/a> the development of technology-neutral regulations allowing greater flexibility in how reactor designs could satisfy safety performance benchmarks. In this context, the probabilistic-risk-assessment techniques and performance-based regulatory approach developed over the decades have proven critical for ensuring the adaptation of safety governance to technological advancement.<\/p>\n

Applying Lessons from Civil Nuclear Safety to Nuclear Command, Control, and Communications<\/b><\/p>\n

As Gen. Cotton\u00a0admitted<\/a>: \u201c[W]e need to direct research efforts to understand the risks of cascading effects of AI models, emergent and unexpected behaviors, and indirect integration of AI into nuclear decision-making processes.\u201d Indeed, the rapid evolution of AI is outpacing research efforts, leaving significant gaps in our understanding of how AI-integrated functions supporting nuclear decision-making might inadvertently lead to escalation.<\/p>\n

Ongoing multilateral discussions on responsible AI integration in the military domain have yet to define what constitutes \u201csafe\u201d AI integration in nuclear command-and-control and adjacent systems, particularly given the high-stakes consequences that even a single error could trigger. To complicate matters further, nuclear-armed states are likely to integrate AI in different ways, driven by their unique doctrines, capabilities, and threat perceptions. For instance, states that perceive themselves as being at a strategic disadvantage may be willing to accept higher risks associated with AI integration if it offers strategic advantages such as faster decision-making and strategic parity.<\/p>\n

Establishing quantifiable risk thresholds for AI integration is therefore essential. Risk assessment frameworks can help policymakers distinguish between high-risk and acceptable AI applications. To ensure that the risks of inadvertent escalation do not exceed established thresholds, they would analyze how specific AI models interact with nuclear command-and-control and adjacent systems and identify cascading failure points as well as their potential consequences.<\/p>\n

This is where civil nuclear safety regulation can provide useful lessons. The management of AI risks in nuclear command-and-control should integrate probabilistic-risk-assessment techniques and adopt performance-based rather than prescriptive benchmarks, where performance refers to the reliability of AI systems, their ability to generate accurate and well-aligned outputs as well as the effectiveness of the system\u2019s safety guardrails. Probabilistic-risk-assessment techniques are necessary because black-box systems are inherently resistant to deterministic fault analysis, and complex accident sequences require systematic risk quantification.<\/p>\n

Additionally, technology-neutral safety governance demands a risk-informed, performance-based approach.\u202fWhile probabilistic-risk-assessment must take technology into account, bilateral and multilateral safety commitments must be applicable to a variety of technologies given the divergent ways in which states are likely to integrate AI into their command-and-control systems. Moreover, the rapid advancement of AI systems will give rise to novel failure modes to which prescriptive guardrails cannot always catch up. As such, rather than rigid prescriptions, which would in any case be exceedingly difficult to verify with an intangible technology like AI, it is far more practical for countries to agree on a set of broad safety goals. Countries could commit, for instance, to the overarching qualitative safety goal that AI systems integrated into nuclear command-and-control should not increase the risk of nuclear weapon use and on that basis develop measurable safety objectives \u2014 such as keeping the risk of an accidental nuclear launch under 1 in 10,000,000 per year. Probabilistic-risk-assessment techniques could then be used to evaluate whether a particular configuration of AI (or non-AI) systems will meet these objectives.<\/p>\n

As an example, it is possible to plot an event tree to assess the probability that the hallucination of threat data as an initiating event may lead to accidental escalation. One branch of the tree may represent the probability of redundant systems correcting the data automatically. Another may represent the probability of human operators double-checking the source data from the early warning system. Yet another, associated with the probability that redundancy and human oversight both fail, may represent the threat data being transmitted onward and strike recommendations being formulated on that basis. The risk of an accidental escalation to nuclear war given this particular initiating event amounts to the probability of all guardrails failing \u2014 a risk that can be assessed quantitatively. If the risk across all initiating events exceeds the defined quantitative threshold, then the configuration of systems must adjust to either eliminate certain high-risk integrations or increase the effectiveness of guardrails.<\/p>\n

When it comes to AI systems, risk assessment of this kind would consider both technological risks and integration risks. Technological risks have to do with a model\u2019s reliability, transparency, and performance. Integration risks, on the other hand, focus on how and where AI is used \u2014 ranging from low-stakes tasks like improving communication efficiency to high-stakes functions such as formulating strike recommendations. The design and built-in redundancies of the system are also crucial factors within the assessment.<\/p>\n

Prescriptive commitments \u2014 such as to the principle of human-in-the-loop or the exclusion of certain types of frontier AI systems \u2014 may seem categorical, but they are neither technology-neutral nor guaranteed to lower the risk of accidental nuclear use below a quantifiable threshold. Indeed, they create a false sense of security and foster the illusion that nuclear weapon possessors can meet their risk reduction obligations by following a set of prescriptions that do not evolve with time and are not guaranteed to keep accident risks below a defined order of magnitude.<\/p>\n

To be sure, objective performance criteria cannot always be defined, which is why civilian nuclear safety regulators have\u00a0retained<\/a>\u00a0some prescriptive requirements. Probabilistic-risk-assessment techniques also have their limitations, particularly in\u00a0assessing<\/a>\u00a0risk contributions from human, organizational, and safety culture factors. Therefore, even as the U.S. Nuclear Regulatory Commission works to risk-inform its safety regulation, it has\u00a0maintained<\/a>\u00a0its commitment to the principle of defense-in-depth, which refers to the practice of layering redundant safety systems that are highly unlikely to fail simultaneously. The same principle should be applied in the context of AI and nuclear command-and-control, but in a way that takes risk insights into account. Lessons from early civil nuclear safety regulation that relied exclusively on redundant safety systems\u00a0showed<\/a>\u00a0that the defense-in-depth approach alone was suboptimal.<\/p>\n

Ultimately, the responsibility to prevent accidental or inadvertent escalation rests with nuclear weapon possessors, regardless of whether their command-and-control systems rely on\u00a0floppy disks<\/a>\u00a0or frontier AI. The safety outcome is what matters, and the approach to AI-nuclear risk reduction must align with its fundamentally performance-based logic.<\/p>\n

Recommendations\u00a0<\/b><\/p>\n

Moving forward, the United States and China should build on their prescriptive commitment to human control and agree on a set of quantifiable safety objectives under the qualitative safety goal that the use of AI should not contribute to an increase in the risk of nuclear war, regardless of whether humans are in the proverbial loop. They should also take the lead in researching probabilistic-risk-assessment techniques that may be used to quantify the accident frequencies of AI-integrated nuclear command-and-control systems. That may include an endeavor to understand the failure modes of various AI systems and develop the appropriate AI safety performance evaluation frameworks. Perhaps most importantly, research should identify the limitations of the various techniques for evaluating AI risks when it comes to nuclear command-and-control applications. The diplomatic process involving the five permanent members of the U.N. Security Council and the Nuclear Nonproliferation Treaty review process can offer opportunities to bring other nuclear- and non-nuclear-weapon states to the table once Washington and Beijing have reached preliminary agreement. The \u201cResponsible AI in the Military\u201d domain summits may serve to involve more diverse stakeholders in the risk management discussion.<\/p>\n

In the end, countries may well discover that it is infeasible to confidently quantify the risks of AI-integrated nuclear command-and-control, or that even the lowest reasonably achievable probability of an accident \u2014 whether 1 in 1,000,000 or 1 in 10,000,000 \u2014 still presents an unacceptably high risk to humanity. If so, the effort to quantify AI risks in nuclear command-and-control systems and to limit those risks below a quantitative threshold will have been worthwhile nevertheless, as it will have revealed the inadequacy of mere prescriptive commitments such as human control. Unless each nuclear weapon state can configure its command-and-control system to ensure that the likelihood of an inadvertent nuclear launch remains below a quantitative threshold, commitments to keep humans in the loop will provide little more than an illusion of safety.<\/p>\n

Fuente:<\/strong> https:\/\/warontherocks.com<\/em><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

En el marco del Asia- Pacific Cooperation Economic Summit, el\u00a016Nov24 los l\u00edderes de EEUU y China\u00a0se reunieron en Lima (Per\u00fa), donde acordaron de manera conjunta,… <\/p>\n","protected":false},"author":1,"featured_media":16076,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[18,24],"tags":[],"_links":{"self":[{"href":"https:\/\/www.fie.undef.edu.ar\/ceptm\/index.php?rest_route=\/wp\/v2\/posts\/16075"}],"collection":[{"href":"https:\/\/www.fie.undef.edu.ar\/ceptm\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fie.undef.edu.ar\/ceptm\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fie.undef.edu.ar\/ceptm\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fie.undef.edu.ar\/ceptm\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=16075"}],"version-history":[{"count":1,"href":"https:\/\/www.fie.undef.edu.ar\/ceptm\/index.php?rest_route=\/wp\/v2\/posts\/16075\/revisions"}],"predecessor-version":[{"id":16077,"href":"https:\/\/www.fie.undef.edu.ar\/ceptm\/index.php?rest_route=\/wp\/v2\/posts\/16075\/revisions\/16077"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.fie.undef.edu.ar\/ceptm\/index.php?rest_route=\/wp\/v2\/media\/16076"}],"wp:attachment":[{"href":"https:\/\/www.fie.undef.edu.ar\/ceptm\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=16075"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fie.undef.edu.ar\/ceptm\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=16075"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fie.undef.edu.ar\/ceptm\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=16075"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}