Checks and balances to empower citizens in an automated society.

Introduction

Every major technological revolution has rewritten governance.

AI’s challenge to democratic institutions

Major innovations reshape society. They affect who holds power, which modes of governance prove feasible, and how beliefs are shaped and bargains struck. The agricultural revolution led to settled populations with taxable surplus wealth, making possible the first enduring civilizations. At the dawn of the modern era, innovations in transportation and communication enabled the rise of global empires and the centralized nation-state. Today’s democracies owe something to the factory and the firearm, technologies that made states reliant on large, organized groups of common citizens.

As generative AI matures from a quirky novelty to a transformational technology, it too will change society. So far, the effects have been real but modest. The best AI systems can significantly enhance both the discovery of cyber vulnerabilities and the design of security patches. Translation tools are starting to enable real-time conversations between people without a shared language, even as AI-generated content makes it harder to distinguish genuine online content from spam, deception, or fraud. As AI models improve, they will be adapted for more use cases and adopted more widely — and they will have far greater effects.

By substituting human labor with machine labor, powerful AI decouples economic and military power from popular support, breaking the underlying conditions that led to the flourishing of liberal democracy.

Every state in human history has thus far been constrained by its dependence on work done by ordinary humans. Soon, powerful AI systems will give well-resourced principals access to a vast, loyal workforce — first cognitive agents, and eventually robotic and autonomous systems. That workforce can be directed to execute political objectives, without fear of whistleblowing or defecting the way humans would. We are already approaching a world where AI enables tyrants; in the near future, leaders could process surveillance data at massive scale, or use autonomous weapons for tasks that human forces might refuse.

Economically, common citizens may be increasingly sidelined while wealth concentrates within companies in the AI supply chain, and among the companies who can best profit from deploying autonomous systems. Frontier AI labs, in particular, will design these agents, determine who can access them, and collectively control the computing power on which they run. Through regulation and economic policy, the states governing these companies will also have a source of leverage over an increasingly large portion of the world economy. States may be able to enforce authoritarian restrictions by requiring companies to use alignment and control mechanisms that ensure particular AI behaviors — e.g., never contradicting official doctrine. The widespread use of AI systems will also generate huge amounts of data on citizens’ behavior, which governments could process with their own AIs to identify and target dissidents.

Technological changes have always shifted the landscape of possible societal designs, making existing structures more or less resilient and bringing new possibilities into view. The industrial revolution gave tyrants tools for propaganda, surveillance, and terror, but it also contributed to the flourishing of liberal democracy. Likewise, transformative AI could enable stable totalitarianism or drive a race to the bottom that trades off our own values for economic competitiveness. But it could also augment democracy with AI representatives that help us understand a fast-moving world and collective reasoning tools to mediate our disagreements.

Rising to the challenge

Successfully navigating the AI transition means shoring up the foundations of our existing democratic society in the near term, while establishing new, AI-native tools and institutions to handle a transformed world.

These foundations include our federal system of checks and balances, as well as the civil liberties that preserve citizens from government interference. Many politicians and civil society groups are already doing relevant work to protect these institutions for reasons other than AI. We’re interested in supporting work that reckons with the ways that AI in particular will test these foundations. For example, AI may disproportionately empower and centralize power in an executive; it will make it easier to process surveillance data; it could enable cyberattacks, threats, and propaganda campaigns aimed at destabilizing and discrediting elections. We’re open to supporting work across the strategy-to-policy pipeline, including threat modeling, legal research, drafting legislative text, and conducting political and public outreach.

We aim to support entrepreneurial efforts to enhance and even reimagine democracy, not just stabilize it. One valuable tool that will become increasingly important over time is AI itself. Oversight institutions could use novel auditing tools to validate the behavior of powerful institutions without raising privacy concerns. At scale, this could create unprecedentedly comprehensive oversight of highly sensitive but highly important actions, like the behavior of national security agencies investigating US citizens. Technical restrictions on AI behavior could ensure that everyone’s AI deployments follow agreed-upon norms, like obeying the law and not attempting to evade oversight. Civil society organizations could use AI to track the behavior of politicians, intelligence agencies, and AI companies, inform the public, and counter abuses of power. Citizens can use trustworthy AI representatives to keep pace as economic and political life accelerates beyond human reaction times.

We believe that these problems are best addressed by a robust, competitive, and ideologically pluralistic ecosystem. We want to build out this ecosystem, but we don’t believe that we or our grantees can solve this problem alone; it is a grand challenge of our time.

If you are a builder, technologist, researcher, or advocate, we want you to participate. Submit to our Request for Proposals, or if you don’t have a proposal to submit at this time, just get in touch with us so we can have an informal conversation.

If you are a philanthropic or political funder, we want to help you navigate this space. Read on to hear our field guide for this budding field and see where you fit in. We expect to have more good applications than we can fund, and we welcome you to partner with us on funding projects in this RFP that are a natural fit. Connect with us here.

You can read much more about our current thinking on these problems at:

• The Foundation Layer — Concentration of Power
• The Foundation Layer — Distributing Power
• The Collective Intelligence Project — 2025 Annual Report
• The Collective Intelligence Project — Global Dialogues Index

Revitalizing Checks and Balances in the AI Era.

New technologies have repeatedly forced governments to adapt their checks and balances through new laws and norms. In many cases, they have empowered the executive first, due to its advantages of speed, secrecy, and top-down hierarchy. Corrective action arrives later, often only after a crisis, and it plays out slowly.

We expect AI to follow this familiar pattern of executive empowerment:

• Improved data processing will enable broader forms of surveillance;
• Advances in automated weaponry will allow greater top-down control in the armed forces;
• Newly available cognitive labor will allow detailed personal oversight of an executive workforce. Eventually, AI could replace much administrative work, potentially automating public bureaucracies end-to-end.

While the situation with AI is novel, and perhaps unusually urgent, we think the solution is much like the approach that advanced democracies have applied in other cases. We should empower citizens and other branches of government to make informed decisions about executive power, ultimately establishing norms, laws, and oversight processes to shape how that power is exercised.

Examples of projects we would like to see

Policy research into Lawfare’s agenda on Executive Branch AI (ExecAI) and the Rule of Law

Executive aggrandizement at the expense of checks and balances is a structural issue that concerns citizens of every political stripe. In the past decades, a polarized electorate and a paralyzed Congress have enabled Presidents from both parties to expand their reach. New crises, too, have incentivized and enabled executive aggrandizement. AI threatens to accelerate this trend regardless of who holds office. An unchecked AI-empowered presidency could have devastating consequences not just to the interests of the opposition but to the fabric of our society. This agenda lays out both a detailed threat model and a large number of research and project areas needed to address it. Below is a non-exhaustive list of topics we’re interested in supporting.

Threat modeling

• Researching risks posed by specific authorities with ExecAI, and identifying new policies that would limit these risks. Key authorities to consider include the Foreign Intelligence Surveillance Act, the Defense Production Act, and the International Emergency Economic Powers Act.
• Modeling misuses of ExecAI for administrative work (e.g. coordinating government agencies to apply punitive pressure on dissenting individuals and organizations).
• Modeling misuses of ExecAI in domestic surveillance applications.

Making recommendations

• Proposing legislative and/or procedural changes to handle the high-risk AI uses identified by the above work.
• Writing a detailed, feasible proposal for Congressional oversight of national-security AI procurement and deployment. This might include:
  • Improving Congressional transparency and oversight in the national-security AI procurement process.
  • Establishing logs of national security AI deployments.
• Evaluating and proposing restrictions on Lethal Autonomous Weapons Systems (LAWS) that prevent power-concentration while enabling them to serve valid national-security needs.
  • For example: what restrictions should there be on the domestic deployment of LAWS? Under what circumstances are such deployments justified? What are the precedents for restricting & overseeing domestic deployments?
• Proposing government processes for identifying and reviewing AI deployments that constitute legal gray areas.
• Creating an executive AI report card: identify what information Congress should hold hearings and issue subpoenas about to surface high-risk ExecAI procurement and usage.

Modernizing Congress and the courts

These institutions serve as crucial but lagging checks on both the executive branch and AI companies.

Example project areas:

• Increasing Congressional funding and capacity. Congress is prey to a vicious cycle: it is unpopular because it is ineffective, ineffective in part because it is underfunded, and underfunded because it is unpopular. The House has had some recent success in breaking the cycle through its Subcommittee on Modernization. We are interested in further work in similar directions, such as funding more staffers for offices whose staff-to-constituent ratio has shrunk, and revitalizing Congressional support institutions like the Congressional Research Service.
• Building placement programs, trainings, and workshops that help Congress and the judiciary understand both AI-induced governance challenges, and AI’s capabilities as a helpful tool.
• Identifying and designing new processes for AI oversight. This might include proposing doctrine for how courts should review machine-speed executive action.
• Enabling AI adoption:
  • Raising Congress’s and the courts’ awareness of AI as both a useful tool and a step-change in the landscape they will need to oversee.
  • Reforming these organizations so they can more easily adopt new technical tools. This could include:
    • Supporting procurement reforms to enable new contracts.
    • Supporting procedural reforms that make it easier to slot AI systems into their workflows.
    • Reviving institutions like the Office of Technology Assessment to make these institutions structurally more competent at technology adoption.
• Proposing forward-compatible reforms to national-security information sharing: changes that will make it easier for Congressional oversight committees to use AI-powered oversight tools on classified information.
  • For example, making it easier for these committees to access classified networks and to get their own tools approved for deployment on these networks. Or establishing greater Congressional subpoena powers or greater penalties on the executive ignoring subpoenas.
  • These reforms could be made with a particular eye to the structured-transparency measures described in the “technical guardrails” section below.
• Building and serving useful software or AI tools to Congress and the courts. Use cases could include:
  • Collaborating on legislative drafting: processing draft text, hearings, and member communications, identifying areas of agreement and disagreement, and identifying useful source documents.
  • Processing large datasets, such as subpoenaed files or FISA court requests, for suspicious patterns.
  • Tracking progress on Congressional and GAO recommendations.
  • Explaining complicated topics to non-experts in these branches.
• Passing legislation to establish new checking authorities, or formalize what are currently normative checks. For example:
  • Requiring Congressional approval of particular extreme actions such as the invocation of the Defense Production Act.
  • Increasing the penalties for refusing to comply with a Congressional subpoena.

Establishing civil-society projects to check AI misuse

Federal organizations are not the only checks on executive misbehavior. Local governments and civil society organizations also have a role to play, especially when empowered by AI.

• Securing elections in the AI era
  • Conducting threat-modeling on how election interference could evolve with increasing AI capabilities.
  • Coordinating with the election security community to develop and deploy defenses against AI-powered cyberattacks, doxxing, and harassment.
• Supporting government watchdog programs to track (mis)use of AI, surveillance, and LAWS.
• Performing investigative reporting on this topic.
• Advocating for sensible transparency in government AI procurement and use. Possible proposals might include requiring more public documentation of AI procurement through Other Transaction Authorities and establishing the AI Use Case Inventory in statute as an annual review.
• Establishing whistleblower support organizations, e.g. legal defense funds and reporting infrastructure.
• Establishing new processes for handling FOIA requests and public comments that account for AI lowering the barriers to public requests.
  • By default, the current equilibrium will lead to huge amounts of AI-generated content that could lead to legitimate public input getting ignored.
  • See “managing AI’s effect on the epistemic commons” below for related ideas.
• Building placement programs, trainings, and workshops that help civil liberties organizations understand both AI-induced governance challenges, and AI’s capabilities as a helpful tool.
  • Running workshops and trainings that help civil liberties experts understand concentration of power risks, and get them acquainted with AI policy experts.
  • Training them to use AI tools.
  • Providing them with compute credits.

Technical Guardrails for Sensitive AI Applications.

As AI capabilities advance, governments will be eager to use them for more tasks and with greater autonomy. Yet adopting autonomous AI in government, especially for actions that could curtail life and liberty, brings serious risks. We propose a defense in depth approach: rules to delineate acceptable procurement and deployment, oversight processes allowing other institutions to check misuse, and guardrails embedded in the AI systems themselves.

If AI systems are increasingly playing the role of government employees, then they will need to behave much like good human employees: obeying the law and, where the law is ambiguous, making thoughtful and ethical judgment calls.

Particular deployment settings increase the need for sophisticated and principled AI behavior. Greater autonomy, scale, and speed of deployment all imply less oversight from humans who are nominally “in the loop”. Humans performing high-stakes tasks in government must internalize principles and follow procedures intended to prevent misbehavior, which need to be thoughtfully adapted for AI systems. In particular, government actions against the public interest are backstopped by employees’ exercise of independent judgment to refuse and report unlawful orders.

It is firmly against the public’s interests to have AI systems undertaking clandestine operations to enact illegal surveillance, autonomously direct weapons against innocent citizens, or overthrow or destabilize standing governments. It’s also not in the public interest for threat actors to be able to misuse governmental AI systems in this way. But it will take work to develop the right defense mechanisms against these threats: those built on principles that have wide legitimacy and draw the right boundary between allowed and restricted behaviors. To this end, we seek proposals to develop the strategy, policy, and empirical validation of technical guardrails for law-following AI, good model specifications for government use, and AI reporting channels.

Examples of projects we would like to see

Power-concentration evaluations

We need more tools to test models’ propensity and refusal rate for dangerous misbehavior; for example, actions that would enable coups, the consolidation of extra-constitutional power, and the injection of secret loyalties. This could include:

• Experimenting with adversarially crafted inputs (e.g. dividing malicious actions across many seemingly innocuous interactions).
• Developing tests to distinguish between closely related legitimate and unacceptable uses, such as a slate of legitimate but suspicious-looking tasks on which models would ideally maintain a low refusal rate. (One person’s “doxxing” is another’s public-interest OSINT.)

Designing model specifications that reject harmful behaviors

“Model specifications” or “AI constitutions” are documents describing the desired character of an AI model. Often, these are used during training to shape the model’s behavior. What specifications should they include, especially for AIs deployed in government or in other situations where life and liberty are at risk?

Some areas for research and advocacy include:

• Defining a minimal set of well-defined actions that are never justifiable for a model to take. Coordinating with AI strategy, policy, and safety researchers to stress-test the list, and campaigning for AI labs to adopt it.
• Pushing AI companies to publish model specs, encouraging public debate about what they should say, and verifying their claims with independent evaluation networks (such as weval.org). Publishing scorecards that encourage a “race to the top” on good governance.
• Designing a standard for model specifications, describing a list of core questions these documents should address.
  • These areas might include: cases in which the model should be maximally transparent; deployment areas where the model should be especially sensitive to user values; how the model ensures it’s following the user’s values when exercising delegated authority; the relative priority of the developer and user’s interests; respect for the law and for democratic process; and how the model prioritizes these constraints relative to ease of use.

Building on Law-Following AI

The Institute for Law and AI proposes that AI companies train models to follow the law, and that such models be required for use in high-stakes settings, such as governmental AI use.

• Operationalizing Law-Following AI.
  • Elicit views on which specific laws to enshrine.
  • Define “minimal viable restrictions” (described above) from a legal rather than consequential angle.
  • Fine-tune a reasonably capable model on legal principles, and evaluate its refusal rate for power-concentrating actions, its acceptance rate for benign actions, and any emergent effects (e.g. effects on the model’s overall persona).
  • Identify likely pain points for adoption in particular government agencies; build test suites (ideally maximally realistic ones, e.g. based on real government documents) and see how current models navigate refusing vs accepting actions that were found to be legal vs illegal.
• Expanding on Law-Following AI and making it robust to misuse. It seems very likely that current laws do not appropriately handle the threat of executive AI misuse, given that: 1) the law was established based on the assumption of pre-AI levels of friction; 2) the law already gives the President and national security establishment wide latitude; and 3) AI might enable entirely unanticipated new actions. Responses to build and test could include:
  • Training AIs to follow principles that better capture the full breadth of what we expect of public servants, such as upholding the spirit of the law or following federal agencies’ legal codes.
  • Technical oversight measures such as classifiers that identify when an AI’s actions are in a legal gray area.
  • Adversarial testing processes featuring realistic attempts at misuse, generated from historical examples and red-teaming exercises.

Reporting affordances for AIs

AI systems deployed in sensitive settings (e.g. AI training, national security) could be provided with secure channels for flagging potentially harmful prompts to oversight bodies. This creates an intersection between technical oversight mechanisms and AI behavior design: one can design an AI charter that specifies when an AI should comply, comply but report the prompt, or refuse and report the prompt. The AI could also have additional reporting affordances, like providing a threat level or flagging which rules are potentially being violated.

Making this idea mature requires:

• Conceptual design. These abstract questions should be tightly coupled with practical thinking about what is feasible to build consensus around, and experimentation about what can be technically enacted.
  • Which things should models report on?
  • What set of affordances should AIs have in the context of internal lab deployment, versus in the government context?
  • Are there types of behaviors that models should always report on? Building consensus on such behaviors, and creating industry norms and standards around them could make misuse considerably harder.
• Technical evaluation. What misbehaviors are possible in government and AI company contexts? Can models identify these different kinds of misbehavior? Do they agree with human experts when ranking misbehaviors? What if the person being overseen is trying to evade detection, e.g. by breaking up obvious misbehaviors into many benign actions?
• Deployment and iteration. Testing the above ideas in increasingly high-stakes contexts, identifying failure modes and pain points, improving on these systems, and eventually getting them adopted in the sensitive areas where they’re most important.

Checking Powerful AI Companies.

As their technology becomes increasingly capable and widespread, AI companies are likely to gain leverage over the rest of society.

As they get richer, these companies could translate their growing economic leverage into political power. Indeed, they are already doing so. Political spending by the AI industry is already a significant factor in the 2026 midterms. Big tech companies with significant AI stakes are among the largest lobbyists, and lobbying by pure AI companies like OpenAI and Anthropic is rapidly growing. If the industry continues to grow, it could easily dominate political spending on future elections. As more sectors become reliant on AI, AI providers also gain leverage in the form of which models they provide and which deals they sign. In the longer term, as AI displaces labor, these companies could become responsible for a plurality or majority of economic production. The resulting economic behemoths would dominate society by default.

We also think it is plausible that in the coming decade, frontier AI companies will have systems that can rapidly transform society. AI systems capable of performing all human intellectual work or inventing new technologies would not just be an epochal economic innovation; they would be a source of political and military power. As the first organizations to have access to these technologies, frontier AI developers have the opportunity to insert backdoors into them before they are widely deployed, and the potential to seize control over the rest of society.

Finally, AI providers can also make AI design choices that have societal ramifications, including how their models report news and politics, discuss political choices with citizens, and advise government users on their actions. We think that AI could be massively beneficial in these applications, but it might require careful oversight. Section B lists several technical ideas for achieving these benefits and mitigating the downsides of AI as a social technology. We are also interested in policy work for ensuring that large AI companies make trusted and trustworthy design choices. In particular, we are interested in work that examines both the influence of AI companies on government and the influence of government on AI companies, since pressure can flow in either direction.

Examples of projects we’d like to see

Establishing norms and oversight for sensitive actions at frontier AI companies

• Identifying high-risk cases of internal AI deployment in AI companies; developing and popularizing standards to limit the risks of insiders misusing highly-capable private models. These include misusing advanced cyber capabilities, using advanced AI R&D capabilities to insert backdoors into other systems, or abusing exclusive access to coup-enabling capabilities.
  • Example proposal: the use of new internal-only models should be logged, since these models might have dangerous novel capabilities. Helpful-only frontier models, which lack standard guardrails against harmful behavior, should only be used with extensive logging and with the signoff of multiple independent parties.
• Operationalizing and popularizing measures to ensure AI integrity — that is, ensuring that AIs are free of backdoors or secret loyalties.
  • We’re particularly interested in work to design and implement protocols for AI infrastructure security: model weight integrity, training data integrity, and security of data-filtering algorithms used in training. Many of these require not just technical innovation, but significant changes in practice at AI companies.
• We’d also like to see field-building for AI integrity:
  • Create a dedicated research or implementation organization.
  • Establish a community and forum on this topic.
  • Set up a testbed to red-team these protocols.
• Supporting AI industry whistleblowers via legal support funds, compensation for foregone wages, and private reporting hotlines.
• Proposing rules to validate the AI systems adopted in high-stakes settings. What evaluation and oversight procedures should AI users apply to ensure that AI systems they procure for sensitive applications aren’t compromised? How should government procurement processes assure the integrity of AI systems?

Policy research into governing transformative AI, while still avoiding concentration of power

To date, most work on governing frontier AI has focused on national-security relevant risks like loss of control and bioterrorism, which suggest close government oversight as a solution. As we are now seeing, these same security concerns can prompt the government to apply its leverage over the AI industry without procedural safeguards. AI governance must balance the need for government oversight against the danger of arbitrary and capricious use of government power, especially in combination with the obvious incentive to use AI for national-security applications (see also Soft Nationalization for an exploration of this issue).

We are interested in legal research, policy writing, and advocacy on the following topics.

1) Preventing the abuse of regulatory authority

• Uncoupling regulatory authority from use authority — that is, clarifying that governmental authority to regulate AI, and to use AI systems in a limited context in service of regulation, does not grant the government a broader right to use those systems.
• Shifting burdens of persuasion onto the government to justify keeping AI technology regulated after some period of time.
• Establishing processes to challenge pretextual use of regulatory authority.
• Tightly constraining the bases on which regulatory authority may be justified, with favorable procedures and standards to challenge.

2) Establishing oversight of government-industry interactions

• Establishing processes for independent reviews of transactions between AI companies and the government.
• Integrating information-sharing with other parties (e.g. Congress, independent AI auditors) as a fundamental element of federal oversight of frontier AI companies.
• Conducting investigative journalism into AI industry lobbying efforts.

3) Preventing power concentration via public-private partnerships

• Identifying the ways that different models of a public-private partnership could enable concentration of power; sharing this information with the public, politicians, and relevant experts.
• Designing policy mechanisms to ensure transparency and oversight of a future federal effort to co-develop frontier AI systems with AI companies. Such mechanisms must be made robust against the objection that they will create information leaks and regulatory burdens.

Economic Power and Benefit-Sharing: Preserving Individuals’ Leverage.

In the long run, the industrial revolution massively increased most countries’ median wages and living standards. Unfortunately, the AI revolution might be different, generating concentrated wealth for a few while disempowering most others.

In particular, if AI is not a complement to human labor but a substitute that replaces the role of most workers, then the returns to capital may be far higher than the returns to labor. For example, Stanford economist Philip Trammell has argued that famed socialist economist Thomas Piketty, while wrong about the past, is probably right about the future: absent strong redistribution, economic inequality will tend to increase indefinitely through the generations.

Trammell’s view is controversial among economists. We think that is plausible, not certain, and the speed and extent of the transition are hard to predict. Nonetheless, it is worth preparing for the resulting problems to the political economy:

• Unprecedented levels of unemployment, with huge labor shocks potentially occurring over just a few years.
• An elimination of remote labor, a main source of catch-up growth for developing countries.
• An “intelligence curse” where most citizens lose leverage with the state in two ways: they no longer contribute to public finances via taxes, and they become increasingly dependent on unemployment benefits as opportunities to earn a living income disappear. This might end up resembling the political economy in some modern petrostates where the population is suffering from a resource curse; the state can neglect the needs of the citizens because public institutions are sufficiently funded by oil revenues.

Today, we lack vetted ideas for sharing the benefits of AI with citizens at home and around the world, including those whose governments would otherwise capture or block these benefits. High-level ideas must be fleshed out with detailed mechanisms to implement them through policy, and must be made robust against powerful actors’ incentives to revise them. The idea of a citizens’ dividend from AI company equity has recently become popularized by both President Donald Trump and Senator Bernie Sanders, but it runs headfirst into the intelligence curse.

We need better ideas and the political will (with labs, governments, and the public) to implement them.

Examples of projects we would like to see

Mechanisms to ensure citizens have a say in the future

• Preserving leverage, not just income. Many dividend proposals now in circulation run straight into the intelligence curse: they restore income without restoring leverage, leaving citizens as pensioners of the state rather than principals in it. We’re interested in mechanism design aimed at the bargaining position: broad-based ownership of AI capital, portable compute or inference entitlements that carry real bargaining power, and arrangements that keep the state fiscally dependent on broad public consent.
• Benefit-sharing across borders. Almost all dividend and ownership proposals stop at the national border. But the populations most exposed to displacement and least able to capture AI’s gains are often outside the countries where it is built. We want mechanism designs for sharing benefits with people everywhere.
• Enforceable pre-deployment benefit-sharing. Commitments are easiest to secure before the committing party holds decisive power. For example, a windfall clause or equity pledge means little if a post-AGI entity can renege and no one can compel it. We are interested in legal and institutional work on the vehicles that could make benefit-sharing stick, such as charter provisions, independent trusts, golden shares held by external bodies, and contractual triggers tied to capability or revenue thresholds.
• Measurement infrastructure for the transition. Redistribution and commitment mechanisms should be triggered based on well-vetted indicators. There should be more real-time measurement of AI’s labor-market and macroeconomic effects: task-level automation tracking, displacement and wage dashboards, sectoral leading indicators, and the data-sharing arrangements with labs and employers needed to populate them.
• Pilots and proofs of concept. Smaller-scale tests that build evidence and legitimacy before the stakes become existential.

Political economy: governing companies that control a large share of the labor market

Society should prepare for the possibility of more dramatic changes, including worlds where leading AI companies control a large share of the global labor pool, and that share rapidly grows in capability and expands in scale.

We are interested in analysis of how political incentives would warp in such a world, and in mechanism designs to address these challenges.

Relevant questions include:

• The fiscal base of an automated economy. If labor income collapses, the income and payroll tax base collapses with it. We are interested in concrete work on: 1) what a durable, hard-to-evade tax base looks like under high automation, whether built on compute, capital, automation, value-added, or land; 2) the international coordination problem, since capital and compute will arbitrage across jurisdictions; and 3) a politically survivable path to get there.
• Learning from the history of extremely powerful corporations.
  • What are the common failure modes in the governance of corporations that contribute an appreciable share of their country’s tax revenue, such as Saudi Aramco and Nokia? Are there any successful examples to learn from?
  • Some of the largest publicly-traded corporations in history were eventually taken over by the state: the British East India Company and Saudi Aramco, for example. Are there any circumstances under which government ownership of AI companies would be a sensible governance approach? What are the pitfalls of such arrangements, and what does good governance look like when one institution is so dominant in a society?

External Checks: Bringing in the Rest of the World.

Most US citizens are not employed at frontier AI companies, nor in the upper echelons of the government, yet the choices made in those few places will reshape their economy, their society, and their security. The checks discussed above are structured to ensure that those institutions’ decisions align with citizens’ interests.

Those interests are shared by most of the world’s people. In the coming decade, frontier models and AI data centers may become the key drivers of economic and military dominance. If the US and China continue to lead, other countries will depend on them for products, services, and security. Like American citizens, they will be increasingly at the mercy of decisions made in a few powerful institutions. At the same time, those institutions’ decisions run the risk of causing a global catastrophe, whether by great power war, diffusion of dangerous capabilities, enabling coups and stabilizing autocracies, or losing control of powerful autonomous systems.

Globally, almost everyone has a stake in pushing back on international power concentration. The middle powers in particular have some relevant leverage, especially if they band together — e.g., investments in AI-relevant resources, involvement in the AI supply chain, and diplomatic convening power.

We are particularly interested in helping democratic middle powers coordinate to shape the future of AI, as we expect them to be more responsible and prosocial than autocratic countries. This is not a call for transferring sovereignty to global bodies, but for countries outside the US and China to coordinate the leverage they already have. These countries need awareness of the moment’s urgency and the insufficiency of their efforts to date, along with a toolbox of high-leverage actions to take. We are interested in supporting work on both.

Examples of projects we would like to see

We are broadly interested in building talent pipelines to bring democratic middle-power governments the capacity to execute on forward-looking AI policy, and in supporting organizations that conduct clear-eyed AI policy analysis for these governments — acknowledging the challenges they will face in staying relevant as the AI transition progresses. We are particularly interested in work that addresses the below problems.

AI investments by democratic middle powers that get them a seat at the table

Frontier AI access will be a key element of economic and military competitiveness in the coming decade; without it, middle powers will become increasingly irrelevant. Yet training frontier models requires a scale of investment that middle powers cannot achieve on their own.

Anton Leicht discusses this problem here and suggests several approaches for middle powers. These are worth considering, and we are open to supporting work on them.

Two additional areas where we would like to support policy research and outreach:

• First, middle powers could try to become indispensable to the great powers by specializing in areas that complement advanced AI capabilities — picking niches that are cheaper to invest in than frontier AI development but that require deep and specialized talent pools, and then attracting top talent and catalyzing world-class projects.
  • We are most interested in cutting-edge AI safety, security, and technical governance innovations. These areas will be crucial to leading countries as they aim to deploy AI for sensitive operations at scale, and developing them is also likely to make the AI transition go better. The United Kingdom is leading the way in middle-power investment in safety and security, but more countries could follow its path.
• Second, more ambitiously, middle powers could aim to build a significant share of the world’s AI data centers within the next five years. Frontier data centers will retain their value and remain a source of leverage even in high-stakes negotiations, where other approaches such as contractual access to foreign models might falter.
  • That leverage relies on retaining a useful enough resource that the US or China are willing to make concessions. Matching even ten percent of these countries’ AI hardware investments is a staggering challenge. But countries with the right economic environment — cheap energy, efficient construction industries, and permission to train on intellectual property — might be able to attract infrastructure investments from US AI companies looking for geographical and political diversification.
  • We are interested in supporting analysis to validate this idea and make it robust — for example, by specifying AI policies that middle powers should not compromise on to attract investment. If a valid form of this proposal exists, we are interested in policy writing, draft proposals, and political outreach around this idea.
• Third, channels for direct global input into model defaults. The choices labs make about what systems refuse, whose norms they default to, and how they handle contested questions are effectively global policy set by a few firms. Build on existing proofs of concept like Collective Constitutional AI and Global Dialogues, and work toward making such input a routine part of development rather than a one-off.

Middle-power investment in tools that enable international coordination on AI

As noted above, middle powers should consider specializing in AI-complementary technical tools. But they have a particular interest in technologies that support international coordination; without such coordination, they are largely at the mercy of the great powers. More than that, in some ways middle powers have a structural advantage in providing these technologies: each great power might trust middle powers’ efforts as benign, where it would suspect its rival of hiding backdoors in any tools it offered.

Developing coordination tools also serves democratic middle powers’ interest in avoiding AI-powered autocracies; many of these tools also appear in other sections of our RFP, since they could also be used by citizens and domestic institutions to check government power.

We are interested in supporting policy design, outreach, and direct work to establish high-quality projects for developing international coordination tools in middle powers.

Relevant technical areas could include:

• Validating the great powers’ development and deployment of AI. Relevant technologies include the structured transparency tools (see below), and the verification layers discussed by Baker et al.
• Enabling AI frontrunners to make credible commitments about their deployed systems. The above section on technical guardrails for sensitive AI applications identifies some relevant areas, such as AI charters and law-following AI.
• Adapting the epistemics and reasoning tools described below to high-stakes international dialogues.

International coordination

• Establishing coordination between democratic middle powers on frontier AI policy and the above AI investments. Key players might include the EU, UK, Canada, Australia, Japan, South Korea, and India.
• Pathways to real standing for non-frontier states in the governance bodies now forming, i.e., a substantive role rather than observer status.

Strategic analysis for a peaceful multilateral order

• Scenario analysis of pathways to geopolitical stability and peace. This may include how a great-power AI race actually unfolds, what off-ramps exist at each stage, what the role of the diversity of countries could be, and where middle powers and AI companies have leverage to alter the trajectory.
• Contingency planning for democratic resilience under hegemonic AI power. If restraint efforts fail and one great power achieves decisive advantage — or if one becomes substantially more authoritarian — democratic middle powers need positioning to preserve sovereignty and coordinated action. Research on technical sovereignty at critical capability tiers, alliance structures that survive hegemonic transitions, and pre-positioned legal and political infrastructure for coordinated response.

AI for Better Reasoning and Truth-Seeking.

AI can be a useful tool for addressing many of the epistemic problems facing democracies. These include:

• Navigating and interpreting omnibus bills, identifying legitimate expertise in a field of information saturation, and understanding the exploding volume of issues that governments must currently decide on;
• Making government decision-making legible to citizens, and moving past pandering to understand the actual issues;
• Navigating the explosion of complexity, rapid changes, and misinformation that comes from scalable AI deployments.

Current AI models have already demonstrated aptitude for improving decision-making in these contexts. When the White House launched its call for input on its AI Action Plan, Transformer and IFP launched AI-powered apps to navigate its 10,000 submissions. The US government is using AI for military strategy and bill writing. And over a billion users deploy AI for everyday decision-making.

Examples of projects we would like to see

Informed-citizen tools

Governments produce far more information than citizens have time to process. AI tools can help citizens search, sort, and make sense of it, tightening the loop between public input and government action.

Examples of projects we would like to see:

• Political research tools
  • Tools to parse legislation and proposed policies & inform people about their impacts.
  • Tools to parse politicians’ track records based on a citizen’s interests and values.
  • Automated OSINT tools to track the causes of politicians’ behavior.
• “Community notes for everything” — e.g. a browser extension to provide accurate vetting on-demand for any content a user encounters, focused on representing the consensus across many differing groups and accurately representing when groups disagree.
• Other epistemic tool ideas from the Forethought Foundation, such as:
  • Rhetoric highlighting which automatically flags sentences which are persuasive-but-misleading, or which misrepresent cited work.
  • Reliability tracking which allows users to effortlessly discover the track record of statements on a given topic from a given actor; those with bad records come with health warnings.
  • Provenance tracing which allows anyone seeing data / claims to instantly bring up details of where they came from, how robust they are, etc.
• Evaluations for AI models’ epistemic virtues. AI models can vary in their propensities to tell the truth, to be clear and forthcoming about their intentions and conflicts of interest, and to manipulate. Public evaluations of these propensities enable citizens and political elites to understand which models they can trust for reliable information about the world, while also providing assurance about when AI models can be trusted with high stakes, difficult-to-audit work.

Empowering the public to monitor powerful actors

Example project ideas:

• Tools to provide public-interest analysis of data on powerful institutions and individuals — e.g. identifying misdeeds in large volumes of subpoenaed records, piecing together released trading records for evidence of insider trading.
• Resourcing investigative reporting: providing reporters with compute credits, AI training, and access or documentation for relevant data sources.
• Online platforms that allow citizens to securely upload footage of police and government misbehavior, while also providing some level of validation (e.g. checking for proof of provenance, a lack of AI watermarks, or other corroborating information).

Managing AI’s effects on the epistemic commons.

AI-generated content is already a significant percentage of what we see on the internet, and it’s only going to grow. We want to see more tools to help humans navigate the resulting information overload.

Examples of projects we would like to see

• Designing protocols that let benign AI agents engage with web services while blocking spambots. These protocols could incentivize their use by making it easier for agents to engage with them (e.g. less processing time), but include agent-appropriate safeguards (e.g. label created social media content as agent-generated, verify actions more carefully for cyberattack potential, etc.).
• Developing further standards for AI watermarking and content provenance tracking.
  • Red-teaming existing work like Google’s SynthID and the C2PA’s standards.
  • Promoting these standards’ adoption by AI companies (watermarking) and phone manufacturers (proof-of-provenance of pictures and videos).
  • Designing protocols for social media companies to validate popular content against these tools, and encourage their adoption (whether through legislation or as an industry standard).
• Designing “proof of personhood” tech that preserves privacy. These tools could help prevent AI-powered information warfare by tying sockpuppet AIs to a particular human.
• Investigative reporting on uses of AI that bypass constitutional checks, suppress political speech, or consolidate power.
• Broad-based ownership of the epistemic substrate. The corpora, knowledge graphs, and provenance records that AI systems draw on are increasingly privately held. Projects might design data trusts, shared licensing frameworks, or public-interest knowledge institutions that give information producers durable rights over how their data is accessed, used, and priced. This is the same move as broad-based ownership of AI capital in the economics section below: data is capital, and durable governance rights over it are a form of bargaining leverage.

AI to Navigate Disagreement and Find Consensus.

AI’s aptitude for search, synthesis, and prediction lets it help large groups find common ground, surface positive-sum options, and reason together faster than humans can alone. As AI accelerates decisions past human reaction time, and as powerful actors gain new tools to fragment or capture the epistemic commons, it will test the public’s ability to form and execute a consensus quickly enough to respond. AI-powered collective reasoning tools are the infrastructure that keeps large groups capable of coherent action against capture.

These tools already show promise. DeepMind’s Habermas Machine helped British groups find common ground on divisive issues, producing statements that won wider agreement and left groups less divided than human mediators managed. CIP’s Global Dialogues and Collective Constitutional AI work show how structured collective input can shape AI systems. The problem is on the demand-side — people need to want to use these tools, and institutions need to want to listen to them. We are interested in projects that either push forward the frontier of AI-enabled collective reasoning or have credible pathways to solving the demand-side problem.

Examples of projects we would like to see

Piloting and scaling AI deliberation tools

• Piloting programs for AI coordination tools in decisionmaking contexts that participants care about. These could run the gamut from informal social groups up to real national governments — the key is having an actual, ideally scalable testbed to try different solutions:
  • Digital communities (web forums, long-running games).
  • Forward-looking organizations.
  • Local governments willing to experiment.
  • National governments that are especially technologically sophisticated and forward-looking.
• Designing technical processes to protect viewpoint pluralism and preserve preference intensity. Bridging methods can flatten strongly held minority views into majority positions, developing approaches that preserve pluralism rather than forcing consensus.

Testbeds for coordination technology

The most high-stakes uses of coordination technology are also areas where users will have reasonable concerns about piloting a new tool or exposing their data to an unknown new organization. However, realistic tests and user feedback are important parts of the flywheel of technology development. The field of AI coordination technology could thus benefit from partner organizations or deployment environments that are willing to try new technology and provide usage data or user feedback to the developers.

As starting points, the best pilot environments are ones where users are invested, deploying AI systems is easy, and failure is cheap — like online games or digital communities. Groups that are especially interested in internal coordination (e.g. social movements) or especially open to trying new things (e.g. startups) might also be relevant targets.

Projects to establish such testbeds could include promoting coordination tools to a particular community (e.g. World of Warcraft players), building technical infrastructure (e.g. creating an API that makes it easy to design and deploy coordination plugins), and partnering with existing organizations (e.g. working with a popular forum to promote these tools). They could also involve building such communities or environments from scratch.

Structured Transparency.

Structured transparency technology enables one party to review another’s sensitive data, receiving useful summary information without violating the other’s privacy. These technologies draw on an established lineage of tools for privacy-preserving data processing, which are already used in financial monitoring, sensitive health information exchange, census data analysis, and beyond.

Structured transparency has a number of applications for preventing concentration of power. It was originally proposed to let the government monitor risky commercial organizations like AI developers and DNA synthesis providers without exposing sensitive information or creating unnecessary interference with benign transactions. This technology is now being piloted by OpenMined for privacy-preserving audits of AI systems (cf Beers and Toner).

Structured transparency has also been proposed as a democracy-preserving technology, enabling oversight institutions to better review sensitive situations, including national-security actions and AI deployments.

We are especially interested in applying these technologies to check government power. For example: a Congressional oversight committee could use an AI model to query an intelligence agency’s data for signs of abuse. The model can process the data and then inform the committee of a finding — e.g. “records A and B show signs of violating statute S”, providing sufficient evidence to subpoena these records without providing undue classified details.

This technology must be proven in real-world applications before risk-averse government institutions are willing to expose their sensitive data to it. As we’ve seen, those real-world applications could themselves help check power concentration; for example, by providing guarantees about AI companies’ training and deployment practices, or by enabling government oversight of dual-use technology without facilitating power abuse.

Some specific design questions that need to be navigated in each context include:

• Should the queried party be able to see the query? To veto it?
• How can the querying party validate that they’re seeing the real data?
• How can the queried party validate that the querying party’s AI isn’t retaining the information?
• If the AI is being run on the queried party’s infrastructure, how can the querying party be confident it’s in fact being run? That it’s not being jailbroken?

Examples of projects we would like to see

• Starting a company that provides structured-transparency-as-a-service for an industry where audits are valuable; ideally one willing to adopt new technology.
• Developing & socializing structured-transparency protocols for AI companies and auditors to test AI safety features. This would enable companies to check each other’s work on models in development.
  • Further work could develop and apply privacy-preserving protocols against secret loyalty injections, allowing third parties to audit companies’ claims about their AI training process. This could ensure that model updates aren’t happening unannounced (cf. “checking powerful AI companies”, above).
• Practical work on structured-transparency oversight of government action.
  • Attempt to get these systems piloted at the local or state level.
  • Propose protocols for oversight, run exercises to test them, and iterate.
  • Run red-teaming exercises where the “Congress” player attempts to stymie executive action while the “national security” player attempts to avoid oversight of gray-area behavior.
  • Extending this technology to international mutual verification. The same protocols could let the great powers, or the US and its democratic allies, verify each other’s AI development and deployment without exposing sensitive details.

Trustworthy AI Representatives.

Arguably the most efficient way for AI systems to improve coordination between people is by acting as digital representatives: systems that are trusted to represent a human user or institution and its beliefs and values to act as a delegate in decisions the user would like to make. If AI systems can learn exactly what their users want them to do, they could debate, vote, or invest on the behalf of their principals, with more knowledge and processing capacity than any human.

As agents are deployed in more high-stakes settings, at higher volume, and with greater agency, their principals will need much stronger assurance of their trustworthiness. Citizens whose values can’t be well-represented by AI systems, or who have good reason not to trust these systems, will effectively be cut off from an increasingly important building block of civic and economic participation.

To solve these problems, we need systems for testing, validating, and standardizing the goals and behaviors of AI representatives.

Examples of projects we would like to see

Testing for value representation

Identify benign values or principles that models should be able to represent, but might fail to (e.g. because they are not common in the training data). Design tests that capture these values. Encourage use of these tests to improve representation capacity.

Research and analysis on agentic representation

Design evaluations and run experiments to test how faithfully agents represent user interests in different circumstances, including multi-turn interactions and adversarial settings. Does this change when the agent is more lightly supervised? When it is deployed in particular settings?

Research on multi-agent systems

How can representation survive in a multi-agent ecosystem, where subagents might be spun up without full context on the user’s preferences and values, and where agents-calling-agents might take themselves far out of the training distribution? Are there robust techniques to ensure representativeness, e.g. requiring that agents pass on a summary of user preferences?

Propose multi-agent protocols and representativeness metrics, run experiments, and iterate on your designs.

• Run tests on multi-agent systems with different harnesses and models and report the results.
• Alternatively, find and use natural experiments on this, e.g. reports of OpenClaw agent behavior, or games where users run large numbers of agents to assist them. Creating environments like this that attract users could enable gathering large amounts of naturalistic data.
• Example research question: when and how should multi-agent systems query the user’s judgment to ensure alignment? Do multiple steps of delegation tend to erode particular aspects of a model’s user-alignment? Build toy models and run studies of how representation quality & misbehavior rate vary depending on the length of the delegation chain and the “human query budget”.
• Can we design an agent delegation process that robustly preserves sophisticated alignment to the user’s values?
• Can we design a multi-agent process that performs well in representative real-world scenarios?

Independent provenance and auditing of representatives

How can users trust an AI representative that’s built and operated by a powerful, opaque counterparty? By default, this situation replicates the core problem of reliance on powerful actors. We’re interested in work on what it takes for a high-capability representative to be independently built, and for independent institutions to inspect and audit these AIs. The goal is for users to be able to make an informed choice about delegating to AIs, rather than relying on the goodwill of AI developers.

Macrostrategy research.

Strategic thinking around the AI transition often recommends that a key actor take the reins, whether that actor is a company building transformative AI, a single government uniting companies in a megaproject, or an AI system autonomously optimizing the world. We want big-picture thinking that updates democratic, pluralist, and “dynamist” principles for the coming era, while grappling with the real challenges they will face. These include managing access to highly dangerous systems and eliciting informed input on complex and fast-moving problems. They also include stabilizing the multipolar solution by convincing leading actors to commit to it, verifying compliance, and ensuring that any defectors will not gain a strategic advantage.

Example research questions that might be useful to engage with

Balancing power amid rapidly transformative AI

AI capabilities could undergo rapid recursive self-improvement, granting disproportionate power to AI developers and governments with early access to powerful frontier systems. Some predict an AI-induced industrial explosion, a period of incredibly rapid technological and economic change.

Some key questions given these dynamics include:

• How do societal checks and balances break down under these conditions?
• Are there key societal “building blocks” we can develop — tools, processes, or institutions that scale well with AI capabilities and help humans navigate this world?
• Are there key guardrails, technical or legal or normative, that could help preserve the balance of power in these conditions?
• How can humans retain leverage, economically or otherwise, as the world is increasingly automated?

This request for proposals includes our current best guesses for interventions that can be designed or deployed now and contribute to solving these problems, but we expect there are many more valuable answers we haven’t considered.

Reimagining liberal democracy

• AI could enable dramatically different societal structures. Just the combination of the projects described in this RFP could lead to a government that operates at machine speed, directed by assemblies of citizens’ AI representatives and overseen by legal AIs that issue injunctions and propose laws the second they become relevant.
• Does that overall picture hold together? What additional or alternative features would make it more robust?
• Are there more desirable alternatives that transform our institutions further? What is the feasible path to them?

Informing Citizens and Civil Society Organizations.

We are interested in work that galvanizes a response to these issues from civil society, including political awareness from citizens at large and dedicated efforts from relevant organizations and experts.

Public awareness is a major driver of political change. While the public is becoming aware of AI as a threat to the labor market, many of the other concerns in this RFP are less well-known.

Experts in AI safeguards and AI policy have often neglected power concentration as a risk. Meanwhile, experts in relevant areas such as law, economics, investigative journalism, and election security rarely appreciate the speed and extent of AI’s implications for society.

Examples of projects we would like to see

• Public-facing content on AI concentration of power risks, making these concerns concrete and legible to a non-technical audience. We have cultural touchstones for AI loss of control risks, such as the Terminator or Asimov’s short stories; what about equivalents for concentration of power?
• Workshops bringing together practitioners in different relevant fields to build understanding and spur action.