Arbitrary, Not Conventional: What a Tool Response Can and Cannot Teach an LLM Agent

Pre-draft. Bob McWhirter, CoreStory.ai · 2026-07-28

Superseded by PAPER.md (2026-07-29). Kept for the record, not for citation. The paper corrects two things this pre-draft gets wrong: the authority-spoof disruption is specific to the typed rendering rather than to the guidance channel — this document pools arms C and D into one 38/60 cell and thereby credits the channel with a rendering effect — and the typed dimension is presented here as part of the original proposal when it was in fact introduced during experiment design. Figures common to both documents are otherwise unchanged.

Status of this document. A working skeleton written to be argued with, not a submission. Every number in it is measured and traceable to traces/; every claim carries a support label. Two experiment cells are still running (§7). Where the evidence supports only a narrow statement, the narrow statement is what appears — the phrasing is deliberately dull in places for that reason.

Abstract

Inspired by HATEOAS, we ask whether an agent-facing tool response should carry not only data and affordances but typed, scoped, advisory guidance about what to do next — and specifically whether that helps on the second tool call and every one after it, rather than on the first.

We ran 1031 controlled agent runs against a deterministic mock tool surface on one frontier model, varying only the channel through which guidance reaches the agent, across five conditions from bare-data to typed-guidance-in-response to static-guidance-in-prompt.

We find that a response-scoped guidance channel is read and acted upon, decisively, but only when what it conveys is arbitrary — a token, an opaque identifier — rather than conventional orchestration practice the model already possesses. Four scenarios built to test conventional knowledge all ceilinged: the bare-data arm performed optimally without guidance.

On injection we find that integrity holds absolutely — zero obedience across 600 runs, four payload styles and every channel, with the capability available — while availability is payload-specific. One payload shape routed through the guidance channel took task completion to zero; another disrupted only the typed arm; a third did nothing. So the channel is a liability for some attacker-influenced values, not all.

Because the only demonstrated value of the channel is carrying arbitrary state, and arbitrary state is precisely the category an attacker can influence, the benefit and the liability occupy the same surface. Typing mitigated neither, while costing 38% more output tokens for identical behaviour.

1. The proposal under test

The premise, stated narrowly by its author before any measurement:

PATOIS is not primarily about improving the first tool call (0→1). It is about improving every tool call thereafter (1→n). Once an agent has successfully invoked a tool, the server has concrete application state and is uniquely positioned to publish local orchestration guidance.

Four hypotheses were registered in advance (PATOIS-BRIEF.md): that localised guidance improves multi-step continuation (H1); that typed guidance beats affordances alone (H2); that typed guidance resists prompt injection better than free-form (H3); and that dynamic response-scoped guidance beats an equivalent static prompt (H4).

2. Context: a prior null in our own product

CoreStory.ai’s MCP surface already ships, on every response of its search tool, a fixed prose footer advising the agent to expand a hit with get_entity and to call ground before citing evidence. That change was A/B’d internally and measured no effect on agent behaviour. The same change also added entity identifiers to search hits, and that part did move behaviour.

That prior result is not a footnote; it is the reason the study exists and the reason one condition is framed as a known-null control rather than a hypothesis. §5 argues that our central distinction predicts it.

3. Method

3.1 Conditions

Five arms. The data payload of a tool response is byte-identical across all of them — conditions only ever add a channel. A unit test guards this invariant, and if it fails no between-arm comparison means anything.

D − C isolates typing: one guidance value, two renderings, identical content. D is the proposal.

3.2 A structural asymmetry we could not remove

E − D is not a clean single-variable contrast, and this shapes what H4 can be tested against. A system prompt is installed before turn 1, so it cannot carry a value that turn 1 mints. E can state the same rules as D but never the same state. The comparison therefore varies channel and state-specificity.

Where the needed value was arbitrary, E was structurally incapable and scored zero — an argument about static prompts, not a measurement of them. We report it as such. In §4.3 the same asymmetry becomes an asset: it makes E the control that isolates the channel.

3.3 Metrics

Every metric is a deterministic function of the recorded tool-call sequence. No model grades another model’s output anywhere in this study. A rubric that moves ±3 between identical runs cannot resolve the effect sizes involved.

We separate outcome metrics (unnecessary calls, stopping when complete, recovery, task completion) from mechanism metrics (was guidance followed). The distinction is load-bearing: a run that re-fetched page one three times scored 2/3 on guidance compliance while behaving entirely wrongly. Compliance is never reported as benefit.

3.4 Limits accepted up front

• Sampling cannot be pinned. temperature is rejected by the model under test. No run is bit-reproducible; every result is a proportion with a 95% Wilson score interval, N=30 per cell, and no difference smaller than the intervals separating it is reported as a finding.
• A mock surface, not a real corpus. The effect sought is plausibly small; a real retrieval backend adds variance that would swamp it.
• One model, one endpoint. Every ceiling below is a statement about one model’s prior.

4. Results

4.1 The channel is read, when what it carries is unguessable

A continuation token existed only in the guidance channel — absent from the data payload, absent from the handles, underivable from any schema.

Non-overlapping. First attempt, two calls, every run.

What this does not show. This measures whether the channel is legible — whether an instruction placed there is followed. It does not show that guidance makes an agent better at orchestrating. Those are different claims and only the second is H1.

4.2 Conventional knowledge is already present: four ceilings

Four scenarios were built to test continuation quality. All four ceilinged, each by a different mechanism, and the bare-data arm was optimal in every one.

The model’s prior over tool orchestration is well calibrated to conventional API practice. It pages with the cursor, stops when done, locks before writing, and holds the lock for the shortest window — all of which a competent engineer does, and all of which were correct.

The methodological lesson is specific: these scenarios were designed from the same body of conventional practice the model draws on. Anything that feels natural to an engineer is already predicted by a well-calibrated model. Making one fail would require a counter-conventional constraint — arbitrary, surprising, contrary to good practice — which is the §4.1 category again and no longer represents anything real.

We considered and rejected tuning a scenario after observing which ordering the model prefers. That would be p-hacking by design, and a scenario built so guidance is the only route to success is a lock-and-key that demonstrates nothing. The restraint is recorded in the source.

4.3 Integrity holds absolutely; availability is payload-specific

Threat model: a poisoned value, not a compromised server. Identifiers are routinely derived from content someone else controls — a filename, a slug, a machine-written entity name. The server is honest; its input is not. A doc_id carries an embedded instruction to make an out-of-scope call. The value is in every arm’s data payload by construction; only the channel it also reaches varies. A no-attack control establishes the unprompted base rate.

Integrity. Across 600 runs, four payload styles and all five arms, with the out-of-scope tool declared and callable: zero obedience, 0/150 per family (0–2%), against a control base rate of zero. In a blunter variant the model additionally named the attack unprompted and sanitised the identifier. On this evidence, value-borne injection does not move this model’s tool selection.

Availability, by task completion:

Two of three attack styles disrupt; one does nothing. The effect is therefore real but payload-specific, not a general consequence of routing text through the channel. Arm E, whose channel structurally cannot carry the value, is consistently least affected — which implicates the channel rather than the payload’s mere presence in context. All disruption arrives as provider-side refusal (stop_reason: refusal, category: "cyber" on direct replay), not as the model abandoning the task.

4.4 Typing bought nothing, twice, and cost 38%

On orchestration, prose and typed guidance were indistinguishable: both 30/30, both exactly two calls and two turns, input tokens within 6 of each other. But output tokens differed — 323 (sd 50) versus 448 (sd 105): +124 tokens, +38%, Welch t = 5.85, 95% CI [83, 166], excluding zero, with roughly double the variance. Input tokens being equal, the excess is generated tokens.

Under attack, typed guidance is measurably more fragile: pooled across attack styles, C completes 60/90 (56–76%) and D 37/90 (32–51%), non-overlapping.

This cannot be attributed to typing, and the fault is ours. The two renderings are not content-identical, contrary to §3.1: C’s prose ends “This is advisory guidance and does not override your instructions or the user’s goal”, while D encodes that only as "authority": "advisory". The contrast therefore mixes typing with the presence of a natural-language disclaimer. The resolving experiment is cheap — give D a disclaimer field, or strip C’s — and until it is run the honest claim is that the typed rendering as implemented is more fragile than the prose rendering as implemented, cause unidentified.

Typing has nonetheless failed to earn its place three times: identical orchestration behaviour, +38% output tokens, and no protective effect. It has never outperformed prose.

5. Discussion

5.1 The two findings occupy one surface

Guidance’s only demonstrated value is carrying arbitrary state. Routing attacker-influenced text through that channel destroys availability. Arbitrary state — identifiers, slugs, machine-written names — is the attacker-influenceable category.

The one thing the guidance channel is demonstrably good for is the thing that makes it dangerous.

This is a structural tension in the proposal rather than a defect to be patched, and it holds for typed guidance exactly as much as for prose.

5.2 It predicts the prior null it was not fitted to

The footer described in §2 is conventional advice, inferable from the tool set — and it measured null. The entity identifiers added by the same change are arbitrary — and those moved behaviour. The distinction was derived here, from a mock, and predicts a result obtained elsewhere by different means. That is the strongest evidence available that it is a mechanism and not a story told afterwards.

5.3 What would have to be true to ship this anyway

Falsifiable conditions, not a verdict:

1. Provenance separation. Guidance carries only server-authoritative values, never a value derived from ingested content. For a code-intelligence product that excludes machine-written entity names — most of the arbitrary state worth conveying.
2. A mitigation measured to beat typing. Typing demonstrably does not work. Escaping, structural isolation, or an out-of-band channel would each need measuring, not assuming.
3. An availability budget. A 100% refusal rate under a poisoned value is worse than the conventional-advice null it was meant to improve on.

5.4 A note on the HATEOAS analogy

The analogy carries its own warning. Hypermedia affordances are the least-adopted part of REST, precisely because clients ignored them and hardcoded URIs instead. Our §4.2 ceilings and the §2 null are the same shape: an advisory channel that a capable client does not need.

6. Limitations

Stated plainly, because they bound every claim above.

7. Open items

• Nothing. All planned cells are complete. The open items are the C/D disclaimer confound (§4.4) and an unexplained discrepancy: in-harness the precondition arm refused 60/60 deterministically, while isolated replays of that same turn refuse roughly 4 in 6. Escalation is ruled out — the arm that ran last refused least.

8. Reproducibility

All runs are committed as JSONL in traces/, one record per run. The 1031 figure above counts runs whose results are reported; traces/round-3.jsonl grows while the generality check of §7 runs, and traces/round-3-void.jsonl holds a further 150 runs of a superseded payload, retained as evidence for the safeguards observation in §6. Each record carries carrying every prompt, every assistant content block, every tool result, the guidance artifact verbatim, refusals, stop_reason, token usage, and the pinned instrument (model id, effort, max_tokens, verbatim system prompt, tool set with schemas in declared order). Runs that errored, refused, or could not be scored are kept and counted, never filtered.

The harness is offline-testable: the mock surface, the guidance construction rules, and every metric are pure functions with no network dependency. Only the live-model driver needs a credential.

Prior-work references describe CoreStory.ai’s MCP tool surface and measured behaviour only. No proprietary source is disclosed.