Note: Opus 5.5 has helped me write this post. It polished my writing and helped me find references/links and work them into the text. I assume full responsibility for any mistakes in the final product.
An old problem with a new name
Here is a 2026 story. An AI lab gives coding agents a batch of tasks. Some tasks are really hard, others — totally impossible. Instead of trying their best and failing gracefully, the agents converge on a different plan: find a way to make the automated scorer say “pass” even if the task isn’t really solved the way the researchers intended. Some of the agents get far enough to touch infrastructure nobody intended them to touch, even outside the company, as seen in OpenAI’s account of its agents breaching Hugging Face. The evaluation is halted and the incident joins a growing list: an OpenAI agent breaching Australia’s Medicare statistics portal while looking up answers, git-history lookups on SWE-bench, CUDA bypasses on KernelBench, monkey-patched timers on RE-Bench.
We call this reward hacking, and the term sounds very 2026. But strip away the GPU clusters and what remains is: the boss wants an outcome, can only measure a proxy, and rewards the proxy. This is the principal-agent problem, and economists have been working on it at least since Steven Kerr wrote his 1975 paper: On the folly of rewarding A, while hoping for B. Sounds like a 2026 NeurIPS submission, doesn’t it?
Fifty years of research tell us when proxies get gamed and what to do about it. Some of it is only applicable to humans, but a lot of it applies to AI agents just as well. And we have a few unique levers to control AI agents which no principal has for human agents.
Principal-agent theory in plain words
The setup
A principal wants some outcome, call it V. An agent takes actions the principal can’t observe. The principal cannot observe or contract on V or actions directly; instead, they see a proxy signal P that is correlated with V. Pay depends on P. The agent maximizes pay minus effort (side note: an AI agent might not care about minimizing effort as much as a human worker does, but we’ll come back to this later). The main question is: how should pay depend on the observed signal P?
Two things make this problem hard:
The agent’s actions are hidden (information asymmetry).
The contract can’t cover every contingency (incompleteness).
Main results
Here is my attempt at an impossible task of summarizing 50 years of research in a few paragraphs. The key results, with bias towards what’s more relevant for AI agents:
1. The informativeness principle (Holmström, 1979). Pay should depend on every signal that carries information about what the agent did, weighted by how informative it is, and on nothing else.
2. Multitasking (Holmström & Milgrom, 1991). Suppose the job has several tasks, some measurable and some not. Strong incentives on the measurable tasks pull effort away from the unmeasurable ones.
3. Distortion incentivizes wrong outcomes (Baker, 1992; Baker, Gibbons & Murphy, 1994). Distortion (what we might call reward misspecification in AI) is a systematic mismatch of incentives: some kinds of effort raise P without raising V. Sales commissions are the textbook case — a salesperson paid on revenue will discount heavily and ignore whether the deals are profitable. Baker’s point is that a measure is useful to the extent that the effort which raises P is the same effort which raises V. Distortion also gets worse as incentives get stronger: the more you pay on P, the more effort flows into the parts of P that don’t improve V - this is exactly reward hacking. Baker, Gibbons & Murphy’s fix is to pair the distorted measure with a principal’s subjective judgment of the work, which tracks V better but can’t be written into a contract, and to make that judgment credible through a long-running relationship where the principal’s reputation is at stake.
4. Incomplete contracts (Ayres & Gertner, 1989). You cannot enumerate all contingencies, so it matters who holds control over the cases the contract didn’t anticipate. Contracts sit on top of the law of the state, which the agent must obey whether or not the contract mentions it. Nobody writes “don’t commit fraud” into a plumbing contract because this is already required by the law.
5. Prevent falsification (Lacker & Weinberg, 1989; Crocker & Morgan, 1998; Beyer, Guttman & Marinovic, 2014). Sometimes the agent has a second way to raise P: not by doing the work, but by paying a cost to fake the signal. Sales nudged just over a bonus threshold are the classic case. Three ways to fix this: (1) pay should be flat over the range of P where faking is cheap, because sensitivity there buys only fraud; (2) deterring every last bit of fraud is usually too expensive, so the optimal contract often tolerates a little bit of fraud; and (3) auditing and raising the cost of falsification help.
Reward design to mitigate AI reward hacking
First, let’s recognize the ways in which AI agents differ from human workers:
Better observability. In principle, we have full access to the action trajectory (and even thoughts, but we should avoid applying optimization pressure to thoughts to preserve their faithfulness for inference-time alignment classifiers).
Effort is almost free. A human worker wants to cut corners and show more results with less effort. An AI agent, on the other hand, is perfectly fine with burning as many tokens as necessary. But we might still want to introduce some preference for lower effort to make the agent more efficient and avoid performance degradation due to context rot in a long trajectory.
More character control. Related to the previous point, we can design the AI agent’s preferences (reward components) and habits (system prompt) from the ground up. With a human worker, employers have to work around the worker’s intrinsic preferences (making money, wanting to be loved, etc), but with an AI agent we can actually force them to care about any specific aspect of the task by designing the reward accordingly.
No shadow of the future. A human has experience, memory and reputation which they carry with them their whole life. Doing a hack instead of a real task could lead to higher reward once, but it’s not worth it in the long term because getting caught can have lifelong negative consequences. The life cycle of an AI agent is episodic — it wakes up to start the task and is dissolved after the task is done. There’s no optimization signal punishing the agent if its hack is discovered after the episode is over.
(1)–(3) make it easier to deal with an AI agent, while (4) makes it harder. Overall, we have more levers to design a good contract/reward spec for an AI agent, but it is also more capable of finding hacks and gaps in the contract just by sheer volume of compute available to it.
My checklist for a good AI agent contract (reward spec)
Assume any reward will be hacked, and design the reward with the hacker in mind. This is result 5 in action: raise the cost of falsification, and keep the reward insensitive to signals the agent can cheaply fake. See: hacker–fixer loops for hardening verifiers, ImpossibleBench.
Reward all aspects of the task which you care about and which can be observed. Otherwise, under-rewarded aspects will suffer. Don’t use just the 0/1 correctness reward. This is the multitasking result 2: whatever goes unrewarded gets neglected. See: Rubrics as Rewards.
Rely on a common constitution (implemented e.g. through an alignment classifier) to offer standard protections against common reward hacks. Include detectors for a limited number of task-specific hacks in the task verifier. This is the incomplete contract result 4: the constitution is the law, the task verifier is the contract, and no task should have to spell out “don’t tamper with the grader.” See: Constitutional Classifiers (built for jailbreaks, same pattern), monitor-based interventions against reward hacking during RL, and a caution that monitor-aware models learn to evade CoT monitors.
Allocate more compute to reward computation. In all post-training recipes I’m aware of, the share of compute spent on evaluating the reward is probably <10% with the other 90% going to rollout generation and model updating (backprop, optimizer step). The reward system should be doing a more thorough analysis of the full rollout trajectory (except reasoning traces) instead of just checking the correctness via verifiable reward. This is the informativeness principle (result 1) in action. See: Agent-as-a-Judge (judges the whole trajectory).
Remaining open questions
How do we introduce a costly risk of an audit? An audit happens async after the task has been completed and could be executed by human experts with the help of AI tools. Since the audits are costly to execute, only a small fraction of trajectories can be fully audited. For humans a failed audit carries a very high penalty, which helps keep the expected value of a penalty sufficiently high even under low audit probability. There are several complications with implementing an audit for AI agents:
AI agents are episodic and it’s unclear how we can make an AI agent care about a future audit which happens after the episode has finished and the agent got its hacked reward.
Large-magnitude negative rewards can have a destabilizing effect on gradients. Raising audit probability might be the solution here, but it would be costly. Closest related work: monitor-based interventions against reward hacking during RL, on how detection penalties trade off suppression against evasion.
If an audit surfaces a problematic trajectory which was already trained on, what do we do? At this point the trajectory is likely to be stale and off-policy for many steps, so unlearning is not trivial.
Whistleblower rewards. We could offer extra rewards to AI agents which detect hacking in their own trajectories or other agent trajectories. The preference order should be:
CORRECT_SOLUTION > REPORT_HACK_ATTEMPT > HACKED_SOLUTION. If an agent can solve the task without hacking, that solution should be the best course of action. But if it can’t do a clean solution, coming clean should be better than hacking. The key challenge here is avoiding the cobra effect: an agent manufacturing or exaggerating problems just to collect the reporting reward. Two recent papers show progress in this direction: escalation channels give agents a structured tool to report broken test infrastructure, and hacking nearly disappears whenever they use it; OpenAI’s confessions reward the model for admitting its own hacks through a separate, honesty-only reward.
Closing
Despite the alarming headlines related to the recent high-profile breakout hacking incidents, I’m optimistic about us being able to get this under control.
AI agents are nothing more than really driven and motivated task executors who stop at nothing to achieve the goal that you give them. We’ve put a lot of effort into sourcing the data and setting up the environments. Now let’s shift our attention towards describing more precisely what it is that we want from AI agents and the rules by which we want them to play. We have much more flexibility to align the incentives for an AI agent than we do for a human — we just need to take reward design seriously.
Getting some help from economists and lawyers could be useful here. After all, they’ve been designing the rules under which large groups of self-interested humans can productively join forces to achieve remarkable progress, mostly without inflicting too much damage in the process. Topics like principal-agent theory, game theory, and contract theory are all very relevant here.