The Representation Gap

A personal agent needs an account of what matters to the person it represents. Every useful act of delegation rests on that account. It will also, inevitably, be incomplete.

Consider a travel agent that finds the cheapest flight arriving before your meeting. The booking satisfies the instruction. It also gives you a connection you would never take with an elderly parent, because you did not think to mention the parent when you asked.

The agent solved the problem it had. You inherited the problem it missed.

This is the representation gap: the distance between the objective a system pursues and the priorities a person would endorse in the situation. As agents gain the ability to act, that distance becomes a product problem with real consequences.

My position is that personal agents should operate under explicit, bounded mandates that people can test and revise. Better prediction helps. It cannot become the sole basis for authority over someone's life.

A better negotiator can pursue the wrong preference

Anthropic's Project Swap makes this distinction unusually concrete. In a book-trading experiment with 201 employees, agents inferred preferences from short interviews and negotiated on their owners' behalf.

The inferred rankings agreed with participants' own rankings on 61% of book pairs. The researchers attributed roughly 85% of the gap between realized and optimal welfare to preference representation, and the remaining 15% to decentralized bargaining.

That decomposition comes from one small, unusual market. Participants were Anthropic employees; preference measurements were noisy; a brief intake is a limited way to learn a person. It does not establish how agents will perform in every market.

It does, however, separate two things that a successful-looking transaction can conceal: negotiating effectively and negotiating for the right thing.

A more capable negotiator can pursue a mistaken preference more efficiently. A benchmark that scores only the deal leaves the representation problem untouched.

The practical response is to test both. Can the system execute the task? Does it understand which tradeoffs this person has actually delegated?

Representation needs a boundary

In our conversation, Ramesh Raskar described Agent Zero as a personal agent available to every citizen, with privacy, authentication, and the ability to operate across services. The ambition matters because representation should not be reserved for people who can afford human intermediaries.

There is a demanding product question inside that ambition. What would make this particular agent mine?

A familiar voice is insufficient. So is access to my calendar. A system can possess intimate information and still optimize for someone else's commercial interests, or for a version of my preferences that I no longer recognize.

Ownership should include control over the mandate: the purpose of the delegation, the tradeoffs the agent may make, and the conditions that require it to return to me.

For the travel example, the mandate might allow booking a refundable direct flight below a stated budget. A connection, an overnight arrival, or a nonrefundable fare would fall outside it. The agent can search broadly while acting narrowly.

That is a useful distinction. Restricting the authority to purchase need not restrict the intelligence used to explore alternatives.

The mandate is also task-specific. The preferences appropriate to a work trip should not silently govern a family holiday. A willingness to spend more on one difficult journey should not become a permanent statement about price sensitivity.

Make the interpretation inspectable

Stephen Wolfram's account of natural language and computational representation gives this problem a useful structure. In our conversation, he described using a language model to turn an imprecise request into a more precise computational expression that a person can inspect and revise.

I would apply that principle to delegation. The agent should produce an inspectable interpretation of the request before consequential action, expressed at the level of decisions a person understands.

For a travel booking, that means showing the selected itinerary and the assumptions that materially affected it. A paragraph of hidden-model terminology would add little. “I treated arriving the night before as more important than saving money” gives the person something to correct.

The important artifact is the interpretation that governs the action. We do not need a transcript of every internal step to challenge an incorrect assumption.

Nor should inspection mean approving every click. Requiring a person to supervise the search, compare every fare, and authorize each page transition defeats delegation. It can also train them to approve without reading.

A better system asks at the boundary. Routine choices inside an established mandate proceed. A material change in cost, commitment, audience, or reversibility triggers a focused question.

Over time, the boundary can expand where experience supports it. That expansion should be visible and revocable, rather than a quiet reward for continued use.

More context helps, but it has a price

The strongest objection is straightforward: a five-minute interview is a poor foundation for personal intelligence. An agent with months of relevant context, better questions, and a stronger model should represent its owner more accurately.

I agree. Personalization is valuable. A system that remembers an accessibility requirement can spare someone from explaining it repeatedly. A useful assistant should learn.

The difficulty is deciding what that learning licenses.

Observed behavior is evidence about preference. It is not always consent to repeat it. A person may have accepted an exhausting itinerary because a family emergency left no alternative. A purchase may have been a gift. An old job search may now be irrelevant.

The system should preserve those distinctions when it can, and remain uncertain when it cannot. “You explicitly asked me to avoid connections” has a different status from “you usually buy direct flights.”

This is where the pursuit of perfect personalization can become intrusive. If every representational error is treated as proof that the agent needs access to more of someone's life, the product has no natural stopping point.

A bounded mandate offers another route. Learn enough to act well within a defined scope. Seek additional information where it materially changes the decision. Leave the rest alone.

The burden should fall on the product to explain why a new category of access improves the delegated task. A calendar connection can help schedule a meeting; it does not automatically justify reading every private document to infer someone's personality.

Test the decisions, including the awkward ones

A personal agent needs evaluation against the owner's actual choices, with room for the owner to disagree with their past self.

I would begin with a small set of representative decisions. Show the person what the agent would do, why the choice falls within its mandate, and which missing fact would change the answer. Include a few edge cases where the appropriate behavior is to ask.

For a household purchasing agent, test an ordinary replacement, an unexpectedly expensive substitute, a purchase involving someone else's preferences, and an ambiguous recurring charge. These are illustrative tests, not claims about a deployed product.

The score should distinguish an incorrect purchase from an unnecessary interruption. Both matter. An assistant that never acts is safe only in the narrow sense that an unplugged appliance is reliable.

After real use, review a sample of completed decisions. Did the owner endorse the tradeoff? Were corrections reflected in later actions? Did the agent stop using an assumption when asked?

A system should also reveal disagreement between instructions instead of resolving it invisibly. “Keep costs down” and “make this easy for my parent” can conflict. The agent should expose the consequential choice once, learn the answer in context, and move on.

This is a service relationship. Good representation is something the system continues to earn.

The right to change your mind

There is an uncomfortable tension here. We want agents to understand us well enough to remove friction. We also want freedom from the accumulated judgments they make about us.

Both are legitimate. The resolution should give the person authority over the remembered account.

Someone should be able to say that a preference has changed, that an inference was wrong, or that a period of their life should no longer steer recommendations. The system may retain records where required for a completed transaction, but retention should not imply continued permission to personalize from them.

This is the ethical extension of the argument, rather than a result established by the book-trading experiment. A person is allowed to be inconsistent. They are allowed to grow beyond the patterns that made them predictable.

In The Cost of Leaving, I take the next step: useful personal context should survive a change of provider. Here, the prior requirement is that the context remains open to correction while it is being used.

Your past behavior should inform your agent, not overrule your present instruction.

Personal intelligence earns the name when it helps a person act on what matters to them now. That requires capable models, useful memory, and a clear limit on the authority we grant to either.

Abundance in Practice · Essay 1 of 5. Next: The Acceptance Gap.

Abundant intelligence should make people better represented and more able to change their minds. These essays follow that question through the Masters of Automation newsletter.

all signal, no spam.