Causal Power

Introduction

There is a growing list of scenarios that map AI futures: Situational Awareness, AI 2027, AI as Normal Technology, AI-enabled Coups, Gradual Disempowerment, the Intelligence Curse, and others. But we haven’t seen models of what the world looks like if multiple threat models and scenarios interact with each other: what happens to the decisive strategic advantage assumption of AI 2027 if adoption and diffusion lag behind capabilities? What about AI as Normal Technology, if the labs are increasingly “soft” nationalized? Or AI-enabled Coups, if the bubble bursts and there’s a public-sector bailout? In a world with defensive acceleration that keeps pace with capabilities, to what extent does gradual disempowerment occur?

Each of these scenarios, often presented in a series of long-form essays, combines claims—about historical trends, the current state of the world, models of political and economic incentives, and assumptions about ownership and institutional response—to forecast a causal pathway of possible outcomes. They are each read as standalone narratives, which makes it difficult to locate the source of a disagreement or divergence in assumptions across world models.

Every proposed intervention to mitigate the risks of these scenarios also implicitly assumes a threat model and a set of premises. A “hard nationalization” proposal assumes that frontier capabilities and infrastructure are concentrated enough for control to be transferred. A monitoring regime for gradual disempowerment assumes that declining human agency appears in measurable institutional changes before it becomes entrenched.

Forecasters call this process of identifying divergent premises “crux discovery.” Two scenarios may reach different endpoints because of one or more assumptions several steps upstream. They may also predict similar endpoints through different mechanisms. If these premises fail, the intervention might miss the causal pathway it was intended to interrupt or enable. An intervention could also work productively under one threat model but cause harm in another. Comparing the underlying assumptions must be a key part of intervention design and evaluation.

Written scenarios usually follow one path through the future. This can obscure which details are uncertain and can anchor readers on a single trajectory. AI Futures calls the exercise of spelling out a detailed, plausible path from proposal to outcome “scenario scrutiny.” An intervention or policy proposal that survives scrutiny under one scenario may simply rely on that scenario’s assumptions. Thus, applying scenario scrutiny across several world models helps identify those dependencies.

We think it is important to systematically map these threat models: their assumptions, causal pathways, and the measurement systems needed to track them. We draw on a longer history of epistemic tools for making sense of complex systems, which we return to at the end of this post.

Guiding Principles

This technique has a few guiding principles:

• Break statements down into atomic claims1. For example, “humans remain in the loop” can be broken down into: a human occupies a position in the decision chain, that human receives the information needed to evaluate the decision, the human has time to act before the decision executes, the human has the authority to refuse, and that refusal is consequential. We will write more in the future on the risks of over-atomization and what characterizes a terminal claim (disputability).
• Make claims resolvable. Our friends at Metaculus have a good guide on this. They recommend having a tight resolution criteria and a good definition. Under this principle, “AI can replace most human workers” is now replaced with, for example, “will an AI system complete, unassisted, > 80% of tasks in [named task suite] that take human experts more than one week, before 2030?”
• Build together. We should build world models collaboratively with experts, including but not limited to the authors of major threat models (e.g. AI 2027/AI 2040, Gradual Disempowerment, AI-enabled coups), professional forecasters, and subject-matter experts. Our world models will also be built in public. We believe that epistemic tooling benefits from mass distributed discourse, for causal pathway contestation, new pathway discovery, and rapid iteration. This process orientation is in our principles as an organization.

We know that any world model will be necessarily incomplete, and a clean set of causal pathways can imply more certainty than evidence supports. Some causal relationships will be missing; others will be drawn at the wrong level of abstraction. Atomization introduces another judgment call about where to stop decomposing a claim and how to normalize claims so they can be compared. We treat these mappings as contestable working models rather than finished representations of the world.

Hypothesis

We hypothesize that this methodology (and tools to support it) could help:

• Monitoring and forecasting organizations connect individual claims and pathways to measurable proxies and resolvable questions, then update which parts of a world model look more likely as evidence changes. The Metaculus team has built Radiant, which maps complex relationships among forecasts to support better decision-making. We think it could support an important layer within a broader world model.
• Policy teams test an intervention under several threat models and identify the assumptions under which its effects strengthen, weaken or reverse
• Funders and grantmakers see which causal pathways already have well-scoped and funded interventions, where gaps remain, and which uncertain claims or points of divergence have the largest downstream impact
• Researchers, advocacy orgs, and conveners working not just in AI safety (fieldbuilding) but also across fields (fieldbridging), bringing in groups with relevant expertise who may not already be thinking about these risks (pro democracy orgs, comparative politics, legal scholars, etc)
• Groups focused on designing positive futures specify the institutions and distributions of power that enable this, then backchaining to the conditions required and interventions likely to create them

Fig. 1

We see two initial use cases for this methodology being applied to current world models.

1. Crux discovery. From which assumptions do currently proposed world models start diverging from one another? Visualizing cruxes using graph-based modeling is useful for determining the most probable set of pathways, what we call scenarios2, across and within3 models. It is also useful for a principle we are exploring at Compact called adaptive governance. How can policy adapt to rapid changes in both AI capabilities and the state of the world? Here, we reference the Milton Friedman aphorism: “Only a crisis—actual or perceived—produces real change. When that crisis occurs, the actions that are taken depend on the ideas that are lying around.” We believe that developing these critical “ideas lying around” ready to implement in a policy window requires a proper state space search of the ways the world can turn out weirdly, and which interventions are well-matched here.
2. Tracking. For each world model, we will link claims, or create the structure of causal pathways, attach appropriate proxies, or measurements for real-time monitoring, and document and propose interventions, or actions which sit between claims and break or reduce the magnitude of causality. We’ll also attach fields which can contribute usefully to the model development. We believe that within the AI safety and policy spaces, there is often an emphasis on field building over field bridging; this is a topic we will write more about in the near future. For example, claims about workforce displacement could benefit tremendously from conversations with labor sociology scholars and union organizers.

There are additional use-cases for this methodology, such as:

3. Identifying cross-cutting interventions. After constructing several world models, we can begin analyzing which interventions appear across multiple. This then becomes useful for directing limited resources (e.g. technical and political capital, public attention) to the most promising interventions.
4. Temporal mapping. Each world model can play out over different time scales. We can visualize these models temporally – imagine the graph transitioning into a multidimensional force directed graph – and also (attempt to) deduce probabilities with the right proxies.

Fig. 2

Worked Examples

For the sake of illustration, we’ll focus on crux discovery and tracking first.

Fig. 3Crux discovery. AI as Normal Technology and Gradual Disempowerment begin to split off at the claim “Defensive AI becomes stronger than offensive AI over time.” The former claims AI adoption and diffusion do not keep pace with innovation; the latter claims that competitive pressures on the state accelerates AI adoption and diffusion.

Fig. 4Crux discovery. Here we demonstrate another crux between the AI 2027 and the AI bubble models. Furthermore, we want to illustrate how a single model can also contain several cruxes. In AI 2027, for example, whether a deal between the US and China is struck determines frontier training caps.

Fig. 5Tracking. This is a simplified map of the model of authoritarian capture, formalized by Forethought’s AI enabled coups. We call these narrative or conceptual maps.

Fig. 6Tracking. Here we show a more granular, atomic map of one specific dynamic that plays out in AI-enabled coups. Whereas the conceptual map may capture military procurement and model loyalty in a small handful of nodes, we can further decompose by our principle of atomicity. This differentiation between atomic and composite claims suggests building a hierarchical feature into a tool that might help support these mapping exercises.

Next Steps

Fig. 7

If you’ve made it here, thank you for participating in the first installation of Causal Power, our first commitment to the principle of building together and in public. Here are our asks, to structure feedback and collaboration:

• If you are a forecaster, how would you approach our methodology, specifically around atomization and resolvable proxy development?
• If you’re an author of a world model, we’d love to co-develop a map.
• If you are a part of any of the above groups (listed in our hypothesis—like forecasters, funders, policymakers, researchers, fieldbuilders, and others), we’d appreciate a quick call to test whether this approach is useful for you.
• If you are a funder, or know of funders, interested in an epistemic tool that could support this methodology, either for your own allocation or for macro-strategy, please reach out. We are looking for support.
• If you are a policymaker interested in a specific intervention or set of interventions, we can assist in contextualizing that intervention systematically.

Please reach out here with the subject line: “[Causal Power]”

In the 1970s, the transdisciplinary study of cybernetics tried to model complex systems using the (then cutting-edge) principle of feedback. These scholars were using Fortran computers and attempting to model complicated dynamics in supply chains, population growth, pollution, and even an economic simulator. Remarkably, in discussions related to this work, you can find the origins of Artificial Intelligence (see: the Macy conferences). Our work here calls back to the origins of the field of systems dynamics. Ambitiously, we build on this history by applying the technologies of our age, including AI (we plan to write more about what being an AI-native policy organization looks like) and using better processes—building with experts and in public. We believe the technique proposed here is fundamental to addressing power concentration risks head on: we must rigorously understand the problem, make it legible, and coordinate.