When does a falling failure probability still increase catastrophic risk?

Formal object

Let $z$ denote the normal-state consumption equivalent, $c_n$ the catastrophic-state consumption equivalent, and $p_n$ the probability of that state along a sequence of systems indexed by scale $n$. Define the utility gap and catastrophic tail pressure as

$$\Delta_n = u(z)-u(c_n), \qquad T_n=p_n\Delta_n.$$

Expected utility can then be written without approximation:

$$\mathbb E[u(C_n)] = u(z)-T_n, \qquad CE_n=u^{-1}\!\left(u(z)-T_n\right).$$

This decomposition separates two objects that are often compressed into a single probability: the event rate $p_n$ and the welfare exposure $\Delta_n$. A falling $p_n$ is sufficient for declining risk only when the product falls. The certainty equivalent inherits three pathwise regimes:

The middle regime matters. It rules out the false dichotomy in which a vanishing probability must become either irrelevant or infinitely dominant.

Limit classification

Suppose probability and utility loss are regularly varying as catastrophic consumption approaches zero:

$$p(c)=c^{\beta}L_p(1/c), \qquad \Delta(c)=c^{-\rho}L_\Delta(1/c),$$

where $L_p$ and $L_\Delta$ are slowly varying. Tail pressure is therefore

$$T(c)=c^{\beta-\rho}L_p(1/c)L_\Delta(1/c).$$

The leading exponents classify the limit. Probability decay wins when $\beta>\rho$; welfare severity wins when $\rho>\beta$; slowly varying terms decide the boundary $\beta=\rho$. Under CRRA utility with curvature $\gamma>1$, $\Delta(c)$ is asymptotically proportional to $c^{1-\gamma}$. For the path $c_n=\exp(-gn)$, the welfare burden grows at exponent $g(\gamma-1)$.

Architecture exponent

Component reliability is not a systemic model. Let $Z_n$ be a common state generated by architecture $s$, and let local components fail conditionally with probability $q_s(Z_n)$. For a systemic threshold $\alpha$, the probability that at least $\alpha n$ components fail depends on both an adverse common state and conditional concentration.

If $Z_n$ satisfies a large-deviation principle with rate function $J_s$, the systemic-safety exponent is

$$I_s(\alpha)=\inf_z\left\{J_s(z)+d_\alpha\!\left(q_s(z)\right)\right\},$$

with binary relative entropy

$$d_\alpha(q)= \alpha\log\frac{\alpha}{q} +(1-\alpha)\log\frac{1-\alpha}{1-q}.$$

The infimum identifies the least unlikely route to joint failure: an unusually adverse shared state, an unusual concentration of local failures, or a cheaper combination of both. Shared foundation models, cloud control planes, identity systems, data pipelines, and monitoring assumptions enter through $J_s$ and $q_s$; adding nominally independent endpoints does not remove those common modes.

Ambiguity envelope

Assume the nominal systemic probability behaves as $P_{n,s}\asymp\exp[-nI_s(\alpha)]$. A robust decision-maker does not know that nominal law exactly and evaluates probabilities over an ambiguity set that contracts with evidence. If the defensible set contracts at exponent $R$, the worst-case probability has usable exponent

$$I_{\mathrm{robust}}=\min\{I_s(\alpha),R\}.$$

This minimum is the technical reason a safer design can improve faster than the evidence supporting it. Architecture governs the nominal rate. Identification and validation govern how quickly nearby models can be rejected. The combined phase boundary is

$$\boxed{\min\{I_s(\alpha),R\}\gtrless g(\gamma-1)}.$$

Robust tail pressure falls only when the smaller of architecture progress and ambiguity contraction exceeds welfare-exposure growth in utility units. Equality is not a knife-edge answer: finite-system corrections and lower-order terms become decisive.

Finite-system check

The asymptotic result must be connected back to systems of finite size. For the paper's Gaussian-logistic primitive at $n=640$, the direct finite exponent is close to the asymptotic prediction:

This is a numerical implementation check, not an empirical estimate of catastrophic AI risk. It establishes that the stated finite primitive approaches the theorem at the reported scale. It does not establish that deployed systems follow the primitive.

Identification boundary

Even the favorable independent and stationary benchmark imposes a severe validation burden. With zero observed failures in $E$ trials, a one-sided $(1-\delta)$ upper confidence bound is

$$\bar p(E,\delta)=1-\delta^{1/E}.$$

To certify $p\le\tau$, the required exposure is approximately

$$E\ge \frac{\log(1/\delta)}{\tau}.$$

At $\tau=10^{-6}$ and $\delta=0.05$, this is approximately $2.996$ million zero-failure trials. Frontier systems weaken every convenient assumption behind that number: model versions change, exposure is selected, outcomes are dependent, and deployment may move the operating regime. Routine success therefore identifies routine reliability, not automatically an extreme tail.

The empirical program should estimate separate margins: capability and access; critical-function exposure; incident denominators; recovery and rollback time; common dependencies; cascade reproduction; and ambiguity contraction. A single synthetic probability conceals which margin produced the result and which observation could falsify it.

Welfare boundary

Consumption collapse, mortality, loss of agency, and extinction cannot be folded into CRRA curvature without an additional ethical argument. A transparent accounting identity is

$$\Delta=\Delta_C+\Delta_M+\Delta_A+\Delta_F,$$

where the increments represent consumption, mortality, agency, and continuation losses under mutually exclusive counterfactuals. The decomposition is useful only when those counterfactuals prevent double counting. The paper deliberately does not infer a population ethic from ordinary risk aversion.

Policy margins

The model does not imply one universal instrument. It maps instruments to rates. Architecture standards, independent fallbacks, interoperability, and recovery exercises target $I_s(\alpha)$. Prespecified evaluations, incident denominators, disclosure, and adversarial testing target $R$. Staged access, reversible deployment, and critical-function limits target $g$. Liability, bonds, capital requirements, and insurance alter private incentives when firms internalize only part of continuation loss.

The practical test is therefore sharper than “Is the system safe?” A safety case should state which exponent it claims to improve, what evidence identifies that change, which common modes remain, and how much welfare exposure is accumulating while the evidence arrives.