FORMAL KORERO

The Framework employs precautionary satisficing under radical uncertainty, not categorical rejection of probability. Three pillars support this approach:

1. Bounded rationality (Herbert Simon): When cognitive limits prevent accurate probability assignment to novel threats, satisfice rather than optimize
2. Maximin under uncertainty (Rawls): When genuine uncertainty (not just unknown probabilities) meets irreversible stakes, maximin is rational
3. Strong precautionary principle: Appropriate when irreversibility + high uncertainty + public goods all present

Nuclear safety uses probabilities because we have 80+ years of operational data. We have zero for superintelligent AI. The situations are epistemologically distinct.

Four clarifications address this critique:

1. "Necessary" means "required in complete solution" not "uniquely necessary" — the Framework presents five layers where all are needed
2. Air-gapping is Layer 1 (capability constraint), not a replacement — you still need governance within the air gap
3. Current API controls prove the pattern — rate limits, scoping, HITL workflows ARE inference-time architectural constraints; the Framework systematizes informal practices
4. Complementarity, not exclusivity — explicitly stated in Section 2.3 of the original document

Accept the vulnerability while arguing continued value:

1. Honest acknowledgment is superior to proposals that ignore the deception problem
2. Layered defense provides value even with compromised layers — gates create detection opportunities, force complex deception, generate audit trails, buy time
3. The alternative is worse — training-time alignment also vulnerable to deception; capability limits prevent beneficial use; no development forgoes benefits
4. Interpretability progress (70% at scale via sparse autoencoders) may enable future deception detection integrated with gates

The Wittgensteinian framing is about current practical limits justifying architectural rather than introspective verification:

1. Present-tense claim about current capabilities, not eternal impossibility
2. 70% ≠ sufficient for real-time safety verification (30% opaque is enough for hidden capabilities)
3. Interpretability and architecture are complementary — gates create structured checkpoints where interpretability tools apply

Absence of validation is the problem being solved, not a flaw:

1. No learning from existential failures — aviation/nuclear iterate based on accidents; existential risk permits no iteration
2. Honest gap assessment — Table 4.3 IS the empirical assessment showing we lack validated solutions
3. Backwards demand — requiring empirical validation before deploying existential-risk containment means waiting for catastrophe
4. Can borrow validation methodologies: red-team testing, containment metrics, near-miss analysis, analogous domain failures

Conflates bad regulation with regulation per se:

1. Market failures justify intervention for existential risk (externalities, public goods, time horizon mismatches, coordination failures)
2. Alternative is unaccountable private governance by frontier labs with no democratic input
3. Design matters — application-layer regulation (outcomes, not compute thresholds), performance standards, independent oversight, anti-capture mechanisms
4. Empirical success in other existential risks (NPT for nuclear, Montreal Protocol for ozone)

All governance encodes values; transparency is the virtue:

1. Explicit acknowledgment in Section 5 superior to claiming neutrality
2. Bounded pluralism enables community variation within safety constraints (analogous to federalism)
3. Complexity solvable through UX design: sensible defaults, delegation, attention-aware presentation, tiered engagement (apply Christopher Alexander's pattern language methodology)
4. Alternatives are worse (global monoculture, no constraints, race to bottom)

Creates complementarity, not irrelevance:

1. Different risks require different layers — existential risk needs upstream controls (compute governance); operational risk needs application-layer governance
2. Proof-of-concept for eventual foundation model integration — demonstrates pattern for upstream adoption
3. Not all risk from frontier models — fine-tuned, open-source, edge deployments need governance too
4. Sovereignty requires application control — different communities need different policies even with aligned foundation models

Staged safety for staged capability:

1. Appropriate for stages 1-3 (current through advanced narrow AI), not claiming to solve stage 4 (superintelligence)
2. Infrastructure for detecting assumption breaks — explicit monitoring enables escalation before catastrophic failure
3. Continuous risk matters — preventing civilizational collapse (99% → 0.01% risk) has enormous value even if not preventing literal extinction
4. Enables practical middle path — deploy with best-available containment while researching harder problems, vs. premature halt or uncontained deployment

Mechanisms are real and deployable with detail:

1. Metric rotation: Maintain suite of 10-15 metrics, rotate emphasis quarterly, systems can't predict which emphasized next
2. Multi-horizon evaluation: Immediate + short + medium + long-term assessment prevents gaming immediate metrics
3. Holdout evaluation + red-teaming: Standard ML practice formalized in governance
4. Multiple perspectives: Natural tension (user vs. community vs. moderator) forces genuine solutions over gaming
5. Qualitative integration: Narrative feedback resists quantification