Pi Army Papers

Paper

Who’s Scared of AGI?

Fear, Power, and the Regulatory-Capture Playbook

AI risk may be real. That does not mean the companies asking to control the response should be trusted to define the risk, write the rules, and administer the system without independent oversight.

You have seen the headlines:

  • “Existential risk.”
  • “Humanity’s last invention.”
  • “We need emergency action before it is too late.”
  • “The next decade will determine whether we survive.”

Some of the people making these warnings are serious researchers. Some are genuinely frightened. Some have spent years studying model behavior, cybersecurity, autonomy, and alignment. The danger should not be dismissed merely because the warnings come from people with commercial interests.

But the commercial interests should not be dismissed either.

The same companies that are warning the public about dangerous advanced AI are also asking governments to establish licensing systems, capability thresholds, safety boards, compute controls, and certification regimes. Those rules could give the largest labs an enormous advantage over smaller companies, open-source communities, academic researchers, and individuals.

That does not prove the warnings are false. It establishes a conflict of interest.

The central question is not simply:

“Could advanced AI be dangerous?”

The more important policy question is:

“Who gets to define the danger, who verifies the evidence, who writes the rules, and who benefits from enforcing them?”

That is the question of regulatory capture.

This is not an argument that AI risk is imaginary

There are several legitimate categories of AI risk:

  • cyberattacks and automated vulnerability exploitation;
  • fraud, impersonation, and large-scale manipulation;
  • unsafe autonomous systems;
  • concentration of economic and political power;
  • military escalation;
  • labor displacement and social instability;
  • misuse of biological or chemical knowledge;
  • loss of control over highly capable systems.

Some of these risks already exist. Others depend on future capabilities that remain uncertain.

The correct response to uncertainty is not complacency. It is disciplined investigation: transparent evaluations, independent replication, carefully scoped restrictions, incident reporting, secure deployment, and rules that apply to powerful institutions as well as ordinary users.

The mistake is treating uncertainty as a blank check for whoever claims to be the safest custodian.

A sincere warning can still support a bad policy. A researcher can honestly believe that advanced AI is dangerous while also benefiting from rules that restrict competitors. Intentions do not eliminate incentives.

The nuclear analogy needs care

The Manhattan Project is often used as an analogy for AI. The comparison is useful in one respect: both technologies raise questions about weapons, secrecy, national security, and control.

But the analogy also has an important limit.

Nuclear weapons had a directly demonstrated physical mechanism. Once nuclear fission and the chain reaction were understood, the destructive potential could be measured and reproduced. The danger did not depend on a long chain of assumptions about future software behavior, training dynamics, deployment conditions, and institutional control.

AGI risk is different. It may become severe. It may even become catastrophic. But the most extreme claims depend on uncertain assumptions about scaling, autonomy, planning, alignment, access to tools, and the ability of a system to operate outside its intended environment.

That difference does not refute the risk. It means the burden of evidence should be different.

A stronger standard would be:

The more authority a policy would transfer to a small group of companies, the stronger and more independent the evidence should be.

The public should not be asked to accept extraordinary restrictions merely because the people requesting them use extraordinary language.

Who is actually in the room?

The word “humanity” does a great deal of work in AI-safety rhetoric. Companies and policy networks speak about protecting humanity, but the people shaping the proposed response are often concentrated in a small number of countries, institutions, companies, and funding networks.

That matters because the rest of the world does not necessarily describe the AI problem in the same way.

Different governments and communities emphasize different risks:

  • national security and military competition;
  • censorship and political control;
  • fraud and consumer protection;
  • labor displacement;
  • access to technology;
  • privacy and data rights;
  • monopolization of infrastructure;
  • catastrophic loss of control.

The absence of one particular framing in another country does not prove that its researchers privately reject the risk. States have strategic reasons to emphasize some dangers and ignore others. But the difference does show that “humanity has reached a universal consensus” is usually an exaggeration.

A genuine global process would include researchers, governments, workers, businesses, and civil society from many regions—not merely the companies with the most advanced models and the largest lobbying budgets.

The regulatory-capture pattern

Regulatory capture is not always a secret conspiracy. More often, it is a predictable result of incentives.

The recurring pattern looks like this:

  1. Scare the public: emphasize chaos, danger, unfairness, or catastrophe.
  2. Write the rules: allow the affected industry substantial influence over legislation and standards.
  3. Gatekeep: create licenses, certifications, approvals, or compliance systems that only large incumbents can afford.
  4. Rent-seek: force everyone else to pay the approved providers or exit the market.

A regulation can produce real public benefits and still be captured. It does not need to be entirely fraudulent. It only needs to impose costs asymmetrically and give incumbents disproportionate influence over the remedy.

History provides many examples of this tension:

  • Railroad regulation was presented as protection against rate abuse, but major railroads helped shape the rules and smaller competitors suffered.
  • Airline regulation controlled fares and routes for decades, limiting competition until deregulation opened the market.
  • Auto-safety rules addressed genuine dangers but imposed compliance costs that smaller manufacturers found harder to absorb.
  • Car-dealer franchise laws were defended as consumer and local-business protections while also preserving legally mandated middlemen.
  • Pharmaceutical approval fees and testing requirements can protect patients while favoring firms capable of financing enormous compliance programs.
  • Hospital Certificate of Need laws were intended to control costs but can allow incumbent hospitals to veto new competitors.
  • Taxi medallion systems promised orderly service but became scarce, tradeable licenses whose owners benefited from blocking alternatives.
  • Occupational licensing often protects consumers in legitimate professions, but in some occupations licensing boards have been used to restrict entry by potential competitors.
  • Banking reforms can improve stability while also imposing fixed compliance costs that smaller institutions cannot absorb as easily as large ones.

The lesson is not that every regulation is bad. The lesson is that public-interest language does not guarantee public-interest results.

AI is especially vulnerable to this pattern

AI has several characteristics that make regulatory capture unusually easy:

  • the technology is technically complex;
  • the public cannot easily evaluate competing claims;
  • the largest companies possess the most data and expertise;
  • policymakers depend on those companies for technical advice;
  • compute is concentrated in a small number of providers;
  • safety claims are difficult to test independently;
  • model capabilities change faster than legislation;
  • compliance systems can be designed around corporate-scale budgets.

The AI version of the playbook could look like this:

The scare

“Unaligned AI will destroy humanity.”

This may be a sincere warning. It may also be a highly effective political narrative. Fear creates urgency, and urgency reduces scrutiny of who is asking for authority.

The write

Licensing proposals, safety standards, capability thresholds, evaluation requirements, and incident-reporting regimes are drafted with heavy input from the companies being regulated.

Industry expertise is necessary. Industry control is not.

The gate

Only companies with enormous compute budgets, legal departments, security teams, and lobbying operations can satisfy the requirements. Small laboratories, independent developers, and open-source communities are excluded—not because they are demonstrably unsafe, but because the compliance system was designed for incumbents.

The rent

Individuals and businesses no longer run models they control. They access intelligence through a small number of approved APIs, hosted platforms, or certified providers. The companies that raised the alarm become the licensed landlords of computation.

This is the nightmare version of AI governance: not a ban announced in one dramatic law, but a maze of requirements that makes independent access economically or legally impossible.

Open weights are the fault line

The distinction between open-source software and open-weight models is important.

Open-source software generally refers to source code made available under a license. Open-weight models make trained parameters available for download and modification. A model can be open-weight without providing fully reproducible training data or training procedures.

Open weights create legitimate safety concerns. Once released, they cannot easily be recalled. Users can remove safeguards, fine-tune them, connect them to tools, or deploy them without the original developer’s permission.

But open weights also provide important public benefits:

  • independent inspection;
  • academic research;
  • local privacy;
  • competition;
  • resilience against service outages;
  • access for smaller countries and organizations;
  • freedom from unilateral platform rules;
  • the ability to audit and improve systems without trusting one provider.

The alternative—concentrating advanced intelligence inside a few private companies—creates its own risks: monopoly, surveillance, unaccountable power, single points of failure, and the inability to verify what closed systems are doing.

The policy question should therefore not be “open or closed?” in the abstract. It should be:

Which capabilities create which risks, and what is the least restrictive intervention that addresses those risks without making independent research impossible?

A small local model used for document search should not be regulated like a frontier model with autonomous access to critical infrastructure. A broad rule that treats them identically is not sophisticated safety policy. It is a moat.

Who audits the auditors?

If AI risks are serious enough to justify extraordinary authority, then the evidence supporting that authority should be independently verified.

That means asking:

  • Who selected the evaluators?
  • Who paid them?
  • Who defined the evaluation scope?
  • Who supplied the logs and evidence?
  • What incidents were excluded?
  • Can independent groups reproduce the findings?
  • Are the methods public?
  • Do the same requirements apply to the largest labs?
  • Is there a formal appeals process?
  • Who audits the regulator?

A company-selected review can be useful. It is not equivalent to an investigation conducted by an institution with compulsory powers, transparent methods, and no commercial dependence on the subject.

The standard should rise with the proposed authority. If a finding is being used to justify controlling access to advanced intelligence, it should not rest entirely on private evaluations commissioned by the organizations that would benefit from the resulting restrictions.

The compute question

There are two competing visions of the AI future.

In one, compute remains scarce and centralized. Frontier models require enormous infrastructure, inference is metered, and access is controlled through subscriptions, cloud contracts, export rules, and licensing.

In the other, compute becomes increasingly abundant and efficient. Quantization, distillation, sparse architectures, custom silicon, local hardware, and open models make useful intelligence available to individuals and small organizations.

Both futures are possible. The political incentive is obvious: the more dangerous and exceptional AI is portrayed as being, the easier it becomes to justify restricting access.

“This is too powerful for the public” can be a genuine safety argument. It can also be a market-protection argument. Both produce the same result if the policy is poorly designed: fewer independent builders, fewer competitors, and more power for the companies already controlling the infrastructure.

The answer is not to pretend that local models are harmless. The answer is to evaluate actual capability and actual use, rather than treating every downloadable model as a proxy for the most extreme hypothetical future system.

What a better policy would look like

A serious safety framework should contain safeguards against capture as well as safeguards against misuse.

It should include:

  • independent evaluations with published methods;
  • multiple competing auditors;
  • transparent capability thresholds;
  • proportional requirements based on actual risk;
  • exemptions for research and low-risk models;
  • protections for open scientific inquiry;
  • clear liability rules rather than vague discretionary power;
  • sunset clauses and mandatory review;
  • public reporting of incidents and enforcement decisions;
  • equal treatment of incumbents and new entrants;
  • independent technical and civil-society representation;
  • support for local and open alternatives;
  • strong privacy and security requirements for closed providers;
  • scrutiny of concentration in compute, data, and model access.

The goal should be to reduce measurable harm without establishing a permanent private licensing system for intelligence.

What I am not saying

I am not saying AI cannot cause catastrophic harm.

I am not saying every AI-safety researcher is dishonest.

I am not saying regulation is unnecessary.

I am not saying open-weight models are automatically safe.

I am not saying that other countries have solved AI governance.

I am not saying every compliance requirement is a corporate conspiracy.

I am saying that risk and authority must be examined separately.

A person can believe that advanced AI has a meaningful chance of causing catastrophic harm and still oppose giving the largest AI companies control over the definition, evidence, and enforcement of the solution.

A person can support safety research while opposing rules that make independent model development impossible.

A person can favor temporary, targeted restrictions while rejecting permanent gatekeeping.

Those are not contradictions. They are basic institutional hygiene.

The practical response: preserve optionality

The future may bring cheaper inference, more efficient architectures, abundant supply, or severe compute scarcity. The most robust preparation works in every scenario.

Learn how to use local models. Learn agent workflows. Learn automation. Understand model licenses. Keep copies of legally obtained models and the tools required to run them. Preserve configuration files, quantizations, tokenizers, adapters, hashes, and runtime documentation.

A model archive without a compatible runtime can become a digital fossil. A hardware collection without documentation can become an expensive box of parts. Resilience requires both preservation and skill.

This is not about preparing for an apocalypse. It is about maintaining the ability to experiment, learn, and build without depending entirely on one company, one platform, one cloud provider, or one regulatory interpretation.

The two questions that matter

If the danger is universal, why should the institutions asking for extraordinary authority be the sole institutions trusted to define it?

If the proposed rules are about safety rather than market control, why do they so often produce barriers that only the largest companies can afford?

These questions do not prove that AGI is safe. They do not prove that the warnings are fraudulent. They force the people making extraordinary claims to show their work to institutions that do not also benefit from the outcome.

The honest position is not “AI will definitely destroy us” or “AI can never be dangerous.” It is:

We do not yet know the probability or shape of the worst outcomes. We do know that concentrated authority creates its own risks, and that the people asking for control should not be allowed to become the only people who define the emergency.

AI safety may be necessary. So are competition, openness, independent verification, and public accountability.

A safe future should not require the public to rent intelligence from a handful of private gatekeepers.