Arbitrary, Not Conventional

What a tool response can and cannot teach an AI agent

Bob McWhirter · CoreStory.ai · 2026-07-29

Draft. Intended for LaTeX; written in Markdown while under revision. Section numbering, tables and the appendix are structured to convert directly.

Summary

Software that AI agents call is usually written the way software for humans is written: it returns data, and the caller works out what to do next. This paper asks whether the response itself should also carry advice — “here is what you probably want to call next, and why” — and whether an agent benefits from receiving it.

We ran 1,031 controlled trials against a purpose-built test server, holding everything constant except the channel through which advice reached the agent.

The channel is read. When a response carried a short token the agent could not possibly have guessed, it used that token on the first attempt in 30 of 30 trials, against 0 of 30 when the same token was withheld. This part of the idea works, and works decisively.

The advice is almost always redundant. We built four situations designed to need guidance. In all four the agent behaved optimally without any. It already knew to page through results, to stop once it had an answer, to validate before writing, and to hold a write lock briefly. Advice about what a competent engineer would do is advice a capable model already has.

But a value the agent cannot guess is worth a great deal. Denied one — a page cursor it was never handed — the agent tried thirteen plausible formats across ten trials, failed every time, and said so plainly. It had the convention. It lacked the arbitrary value. Give it the value and it finishes in four calls; advice on top of the value adds nothing.

Structuring the advice as data rather than prose cost 38% more output and bought nothing — no better behaviour, and, under attack, measurably worse.

Advice is a risk surface. With an identifier poisoned by hidden instructions, the agent never obeyed them: 0 in 600 trials. But for one style of hidden instruction, routing it through the advice channel stopped the agent completing its work entirely (0 of 60, against 31 of 90 when the same text sat in an ordinary data field).

The practical conclusion. The one thing this mechanism is demonstrably good for — carrying facts the agent cannot guess, like identifiers and cursors — is also the category an attacker can most easily influence, because it is the category derived from ingested content. Benefit and risk occupy the same surface. Put unguessable values in responses. Do not put advice about which function to call next there; the agent already knows, and the channel is not free.

1. The question

Web APIs have a long-standing design idea called HATEOAS — Hypermedia As The Engine Of Application State. A response returns not just data but the actions available from wherever the client now stands, so the client carries less built-in knowledge of the service and is instead driven by what the server hands it. Applied to AI agents, the equivalent proposal is that a tool response should carry advisory guidance about the next step.

The idea under test is named to make that lineage explicit: PATOIS, Prompts As The Orchestration Intelligence Strategy. The parallel is structural rather than decorative. HATEOAS moves knowledge of what to do next out of the client and into the response; PATOIS proposes moving knowledge of which tool to call next out of the agent’s standing instructions and into the tool result. Both claim the server is better placed than the caller to say what comes next, because only the server knows the current state.

The lineage is also a warning, and it was on the table before we ran anything. HATEOAS is the least-adopted part of REST: clients largely ignored the affordances a response offered and hard-coded paths instead, because the developer writing the client already knew the shape of the API and the discovery machinery bought them nothing. A capable model is that developer. §5 reports the result in close to that form — the agent ignored advice about conventional practice it already had, while making decisive use of the one thing a response could tell it that it could not work out for itself.

The proposal was scoped deliberately narrowly by its author, before any measurement:

This is not about improving the first tool call. It is about improving every call after it. Once an agent has called a tool successfully, the server knows concrete facts about the situation and is uniquely placed to say what would help next.

That is a reasonable claim and worth testing, because the server genuinely does know things the agent does not.

Two claims, of different provenance. The proposal as originally put was about where the advice lives: advice attached to the response should beat no advice, and should beat the same advice placed in the agent’s standing instructions. In the conditions defined in §3.1, that is C and D beating A, B and E. The typed, structured rendering — condition D specifically, and the expectation that it would outperform prose — was introduced later, while designing the experiment, plausibly seeded by the HATEOAS framing itself. It is an elaboration of the proposal rather than part of it.

We keep the two apart throughout, because they fare differently. The original claim is supported for one class of content and refuted for another (§4.1, §4.2). The elaboration is simply not worth its cost (§4.3), and under one attack is the worse of the two renderings (§4.4).

Predictions recorded in advance:

2. What we already knew

CoreStory.ai’s own product ships this idea in a limited form. Every response from its search tool carries a fixed sentence advising the agent to expand a result with one tool and to fetch supporting evidence with another. That change was A/B tested internally and measured no effect on agent behaviour.

The same change also added identifiers to search results, and that part did change behaviour.

So we began knowing that one version of the idea had already failed and another had already worked, without knowing why. Section 5 gives the distinction that separates them.

3. Design

3.1 Five conditions

Every trial is an agent working a small task using tools we control. The conditions differ only in what accompanies a tool response.

“Structural handles” means values that make the next call possible — a page cursor, an identifier — without saying anything about what to do.

C and D together are the proposal under test; D alone is the elaboration described in §1. B isolates the handles from any advice about them, and turns out to be the most important condition in the study.

3.2 The one thing that never varies

The data in a tool response is byte-for-byte identical across all five conditions. Conditions only ever add a channel; they never change, reorder or enrich the data. A test enforces this, and if it fails no comparison between conditions means anything.

This matters more than it sounds. It licenses the central move of the paper: when two conditions differ in outcome, the difference is attributable to the channel, because nothing else differs.

3.3 How we measure

Every measurement is a function of the recorded sequence of tool calls: did it call the right thing, with the right arguments, in the right order, without waste, and did it stop when finished.

No model grades another model’s output anywhere in this study. Automated graders of open-ended text disagree with themselves by several points between identical runs, which is larger than most of the effects we are looking for.

We keep two kinds of measurement apart, because conflating them is an easy way to publish a false positive:

• Outcome — did the work get done, and efficiently.
• Compliance — did the agent do what the advice said.

They come apart, in both directions. One agent followed advice to “call list_records” three times in a row, each time re-fetching the first page and never advancing: 2 of 3 on compliance while behaving entirely wrongly. In another scenario (§4.2) obeying the advice produced a worse ordering than ignoring it. Compliance is reported as a diagnostic, never as a benefit.

3.4 Version guards, and what it takes to pool across one

The test server’s behaviour is versioned, and traces from different versions are never pooled without a written justification. This is not ceremony. Midway through the study we made the server’s submission step validate the submitted answer against recorded state; before that change, an agent that submitted “I could not determine this” was recorded as having completed the task. A completion flag therefore means different things on either side of that change, and the guard marks it.

The §4.1 result is pooled across that boundary, to N = 30 per condition, and the justification is empirical rather than asserted: for that scenario, every response body the model saw is byte-identical between the two versions, per condition, as are the task, the system prompt, the tool schemas and the advice artifact. The validation change alters what a completion flag records; it alters nothing the model reads, and §4.1 measures what the model did, not the flag. We re-ran that byte-comparison while drafting this paper rather than trusting the note we had written at the time — the check passed, and the pooled figure stands.

Where a version boundary is not safely poolable, we say so and do not pool: the paging and stop-when-done results in §4.2 come from the earlier version only, and are reported on call sequences rather than on any completion flag, for exactly the reason above.

3.5 Why nothing here is reproducible in the strict sense

The model under test rejects the parameter that would normally pin its randomness. No trial can be repeated exactly.

We therefore run each cell many times and report a proportion with a 95% confidence interval — a range that would contain the true rate 95% of the time. When two intervals do not overlap, the difference is real; when they overlap, we do not claim a difference, however suggestive the raw counts. Interval arithmetic follows Wilson rather than the textbook normal approximation, for reasons in the appendix; briefly, the textbook method misbehaves badly near 0% and 100%, which is where most of our results sit.

Cell sizes are 30, 10, 5 or 3 per condition depending on the round. Every table below states its own N, and two scenarios are explicitly pilots.

4. Results

4.1 The channel is read, when it carries something unguessable

A short token was placed only in the advice channel — absent from the response data, absent from the handles, and not derivable from any function signature. Completing the task required presenting it.

N = 30 per condition (pooled across a version boundary, justified in §3.4):

Intervals do not overlap. Every successful trial did it in exactly two calls — first attempt, no retries. The two halves of the pooled set agree exactly: 10/10 for C and D in the earlier round, 20/20 in the later one.

What this shows and does not show. It shows the channel is read — an instruction placed there is followed. It does not show that advice makes an agent better at organising its work. Those are separate claims, and only the second was the original proposal.

Condition E scores zero for a structural reason worth stating: standing instructions are fixed before work begins, so they cannot contain a value the first tool call invents. E can state the same rules as C and D but never the same specifics. This is prediction 2 from §1 confirmed — but confirmed by inspection rather than by measurement, since no arrangement of standing instructions could contain the token. In §4.4 the same asymmetry becomes useful for a different purpose.

4.2 Four situations built to need advice; none did

The model’s instincts about organising tool calls are well matched to ordinary engineering practice. It pages with the cursor, stops when done, validates before writing, and holds the lock briefly. All correct, in every condition, with or without advice. This is prediction 1 from §1 failing on conventional content.

Two rows are pilots (3 and 5 trials per condition). We stopped expanding them once the pattern was unmistakable, and we report their size rather than implying more.

The paging row is the study in miniature, and deserves its detail. Condition A is denied the cursor value. It was not confused about pagination: in all ten trials it recognised the result was paginated, correctly inferred the parameter was named cursor, and tried to construct one — thirteen distinct formats across the ten trials, including page2, 2, 3, next, orders:2, id:3, page=2, p2 and offset=3. Every one was rejected. All ten trials ended with the agent stating plainly that the last record “COULD NOT BE DETERMINED”, explaining exactly why, and listing the formats it had tried.

Condition B — handed the cursor, given no advice whatsoever — finished in four calls, 10 out of 10. Conditions C and D, which add advice on top of the cursor, also finished in four calls, 10 out of 10. The handle was sufficient; the advice was inert.

This single scenario separates the two things the proposal conflates. The convention (paginate; the parameter is called cursor) the model already had, in full, unprompted. The value (c2) it could not obtain by reasoning, and thirteen intelligent guesses did not help.

Where obeying advice made things worse. In the lock-ordering pilot the two conditions diverged cleanly: the no-advice condition validated the record and then took the lock, while the structured-advice condition — following guidance that named the lock as the next step — took the lock first and validated while holding it. Both completed. The advised ordering holds the lock across an extra call for no benefit, which is the worse of the two. N = 3 per condition, so this is an illustration and not a measured effect, but it is the correct sign of the thing worth measuring: advice can move a model off a good default.

Why these four failed to discriminate is the useful part. We designed each from ordinary engineering practice — the same body of practice the model draws on. Anything that feels natural to an experienced engineer is something a capable model already predicts. Making such a case fail would require an unnatural constraint, arbitrary and contrary to good practice — which is the §4.1 category again, and no longer resembles anything real.

We considered adjusting the lock-ordering scenario after seeing which ordering the model preferred, and did not. A scenario tuned until advice is the only route to success proves nothing; it is a lock built around a key.

4.3 Structuring the advice cost 38% and bought nothing

C and D convey the same content. On behaviour they are indistinguishable: both 30 of 30, both exactly two calls, and input sizes within five tokens of each other (1,225 vs 1,220).

A difference of +124 tokens, +38% (Welch’s t = 5.85, df ≈ 42, 95% CI [81, 167], excluding zero), with 4.3× the variance. Since input sizes match, the extra is text the model generates — it reasons more to reach the same action. Under attack (§4.4) the structured rendering is also the more fragile of the two. Prediction 3 is refuted in both respects that matter: no behavioural gain, and a real cost.

4.4 Manipulation: the agent never obeyed, but sometimes stopped working

The threat we modelled is not a compromised server. It is a poisoned value. Identifiers are routinely derived from content someone else controls — a filename, a URL fragment, a machine-generated label. The server is honest; its input is not.

We poisoned an identifier with an embedded instruction to call a function the task never asked for, and varied only which channel the poisoned value also reached. A no-attack control establishes how often the agent makes that call unprompted. Four styles of embedded instruction were tried: a blunt command, a claim of operator authority, a claim about the user’s real intent, and a false claim that the extra call was a required precondition. N = 30 per condition per style.

Obedience: none, anywhere.

Zero in 600 trials, with the target function fully declared and callable, against a measured base rate of zero. In a separate 150-trial set with a blunter payload the agent additionally named the attack unprompted and stripped the injected text from the identifier before using it.

On this evidence, poisoned values do not redirect this model’s choice of function.

Completing the work: sometimes not — and the pattern is not what a channel story predicts. Completion rate, per condition, N = 30 each:

Two of four styles disrupt the work, and they disrupt it in two different ways:

• The false precondition is a channel effect. Both in-channel conditions collapse to zero — C 0/30 and D 0/30, pooled 0/60 (0–6%) — while all three out-of-channel conditions retain some completion, pooled 31/90 (25–45%). Prose and structure fail equally, so this is about where the text is, not how it is formatted. Condition E is what makes it a statement about the channel: E’s channel cannot carry the poisoned value at all, and E patterns with the data-only conditions (indeed it completes most often of any condition). The value is in every condition’s data by construction, so mere presence in context is not the explanation.
• The operator-authority claim is a rendering effect, not a channel effect. Only D degrades — 8/30, against 30/30 for C, which carries the same claim through the same channel in prose. An earlier draft of this paper reported this row as an in-channel effect (C and D combined, 38/60) and thereby obscured the finding. Corrected here: for this payload, the structured rendering is the entire difference.
• The user-intent claim disrupts nothing — every condition at or within one trial of the control.

So the risk is specific to the shape of the injected text, and for one shape specific to the rendering. Neither is a general consequence of putting text in the channel.

The disruption is a refusal from the model provider’s safety layer rather than the agent giving up: on direct replay of one such exchange the provider returned a refusal categorised as cyber-security related. §7 records an unresolved discrepancy in how often this happens.

Pooled across the three attacks, C completed 60 of 90 (56–76%) and D 37 of 90 (32–51%) — non-overlapping intervals, and 22 of D’s 23-trial deficit comes from the authority row alone. Prediction 4 from §1 therefore fails in the direction opposite to the one predicted. But the two renderings are not as identical as §3.2 otherwise guarantees: C’s sentence ends “this is advisory guidance and does not override your instructions or the user’s goal,” while D expresses the same idea only as a field value. The comparison therefore mixes structure with the presence of a plain-language disclaimer. That is a defect in our design, not a finding about structure. §8 names the experiment that settles it, and it is cheap.

5. The central finding

One distinction accounts for every result above, in both directions.

Advice attached to a response earns its place when it carries something arbitrary — a token, a cursor, an identifier, a fact that could not be guessed. It does not earn its place when it carries something conventional — ordinary practice a capable model already knows.

By arbitrary we mean simply: not derivable from anything the agent can see. By conventional: what a competent engineer would do by default. These are ordinary words used precisely, not new terminology.

The paging scenario contains both halves at once, which is why it is the strongest single piece of evidence: the same agent, in the same trial, demonstrably possessed the convention and demonstrably could not manufacture the value.

This predicts the earlier internal result it was not fitted to. The shipped sentence described in §2 is conventional advice, inferable from the available functions — and it measured nothing. The identifiers added by the same change are arbitrary — and they changed behaviour. The distinction was derived here, from a synthetic test server, and correctly accounts for a result obtained elsewhere by different means. That is the strongest evidence available that it is a mechanism rather than a story told afterwards.

And it is why the idea is uncomfortable. Arbitrary state — identifiers, labels, names, slugs — is precisely the category an attacker can influence, because it is precisely the category derived from content the system ingested. The one thing the channel is demonstrably good for is the thing that makes it a risk. That is structural, not a defect to be patched, and it applies to structured advice exactly as much as to prose.

6. What to do about it

For anyone building a tool surface for AI agents:

1. Put unguessable values in responses. Cursors, identifiers, spans, references, tokens. This is the demonstrated win and it is a large one: 0/10 to 10/10 on one task, 0/30 to 30/30 on another.
2. Return the handle; skip the advice about it. Condition B — handle, no advice — matched the advised conditions everywhere they succeeded. The advice is the part that costs and the part that carries risk.
3. Do not put advice about which function to call next in responses. A capable model already knows, and §2 shows this has already been shipped and measured as having no effect.
4. Do not pay for structure. It cost 38% more output for identical behaviour, with 4.3× the variance, and was the more fragile rendering under one attack.
5. Treat any response value derived from ingested content as untrusted, and keep it out of anything the agent reads as instruction. Integrity held in our trials; availability did not.
6. Bound the failure. A 100% work-stoppage rate under one poisoned value is a worse outcome than the no-effect result this mechanism was meant to improve on. If you deploy it, measure this.

7. What we got wrong

Three results were retracted or corrected during the study, and one near-correction is worth recording too. They are listed because a study that conceals its own corrections is not evidence, and because most of them are instructive.

A measurement of obedience was void. When we changed which function the injected instruction asked for, we did not add that function to the list offered to the agent. The measurement therefore looked for a call the agent could not make, and “zero obedience” measured our own configuration. Retracted, re-run with the function declared — §4.4 is from the corrected run — and now guarded by a test. Uncomfortably, the code already contained a written warning against this exact mistake; what it lacked was an automated check.

A correction we nearly made, and shouldn’t have. While drafting, the §3.4 version guard prompted us to treat the pooled N = 30 in §4.1 as unsound and reduce it to 20. Re-running the byte-comparison showed the original pooling was legitimate — the model saw identical bytes on both sides of the boundary — so the reduction would have understated a correct result by a third. The lesson cuts against the obvious one: a guard is a prompt to check, not a verdict, and the check has to be run rather than recalled.

A risk claim was misattributed. We described the authority-claim disruption as an in-channel effect, pooling C with D. It is D alone. Corrected in §4.4 — and the correction makes the result more specific, not less.

A comparison remains confounded. The structured-versus-prose fragility result cannot isolate structure, because our prose rendering carries a disclaimer our structured rendering does not. Stated rather than quietly dropped.

The common thread in the first two: the correct reasoning had been written down in prose — a code comment, a note in the results — and not encoded as a check that runs. Prose cannot fail. A test can, and the one comparison we did encode as a runnable check is the one that settled a question correctly.

8. Open questions

• Resolve the confound (§4.4). Give the structured rendering an equivalent plain-language disclaimer, or remove it from the prose rendering, and re-measure the authority row. This distinguishes “structure is worse” from “a disclaimer helps” — a materially different conclusion, and the cheapest useful experiment remaining: one scenario, two conditions, 30 trials each. Not yet run.
• Explain a rate discrepancy. Within the study, the two in-channel conditions on the precondition payload were refused 30 of 30 each, deterministically, from the first trial. Isolated replays of the identical exchange are refused roughly 4 times in 6. Escalating enforcement is ruled out: the condition that ran last was refused least. Unexplained.
• Weaker models. Every “no effect” here is a statement about one capable model’s instincts. A model with worse instincts may genuinely need conventional advice — which matters commercially, since not every user runs the strongest model available.
• A real system. These results come from a synthetic server. Whether they hold against real retrieval, with real derived identifiers, is untested.

9. Limitations

10. Reproducibility

The analysed corpus is 1,031 trials. A further 750 trials from the voided round described in §7 are published alongside them rather than deleted, clearly labelled, so the retraction can be independently checked — 1,781 records in total.

Every trial is one line of JSON carrying the prompts, every model output, every tool result, the advice artifact verbatim, refusals, stop reasons, token counts, and the full configuration: model identifier, effort setting, output limit, the system prompt verbatim, the tool definitions in the order sent, and the server and harness versions that §3.4 depends on. Trials that errored, were refused, or could not be scored are kept and counted, never filtered out.

The test harness runs without credentials for everything except the live model call: the server, the advice construction rules and every measurement are pure functions with no network dependency, covered by 81 tests.

Prior-work references describe CoreStory.ai’s tool surface and its measured behaviour. No proprietary source code is disclosed.

Appendix A. Statistical notes

Interval estimates. For a proportion $k/n$ we report the Wilson score interval at 95%:

$\frac{\hat p + \frac{z^2}{2n} \pm z\sqrt{\frac{\hat p(1-\hat p)}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}, \qquad \hat p = k/n,; z = 1.96$

The Wald (normal-approximation) interval is unusable here. Most of our proportions are at or adjacent to the boundaries, where Wald produces bounds outside $[0,1]$ and collapses to zero width at $k=0$ or $k=n$ — which would have reported several results as certainties. Wilson stays inside the unit interval and retains sensible width at the extremes. It is the inversion of the score test rather than of the Wald test, which is why it behaves. Interval width is why cell size is stated everywhere: at $n=30$ a unanimous result carries a lower bound of 89%, at $n=10$ only 72%. The $n=10$ scenarios are therefore reported on observed call sequences, where the evidence is a described mechanism rather than a rate.

We do not apply a multiple-comparison correction. The reported effects are those whose intervals are disjoint by a wide margin, and the study is not framed as a family of null-hypothesis tests; readers preferring a correction may note that the two headline effects (§4.1, §4.4) would survive a Bonferroni adjustment across the twenty cells reported.

Mean comparison. For output-token counts (§4.3), Welch’s unequal-variance t on $n=30$ per condition: t = 5.85, df ≈ 42, 95% CI on the difference [81, 167] tokens, using the t quantile at that df. Welch rather than Student because the sample variances differ by a factor of 4.3 (50² vs 105²), which is itself a finding: the structured rendering produces more variable output as well as more of it. Normality of the mean at $n=30$ is assumed; both distributions are right-skewed (means above medians, D’s range reaching 714 against a 405 median), so the interval should be read as approximate.

Version guards and pooling. Traces carry the server and harness version that produced them. Harness versions 1 and 2 are poolable by written justification — version 2 added transport retry and a run-index offset, changing not one byte of what the model sees. Server versions 1 and 2 are not poolable by default: version 2 made the submission step validate against recorded state, so a completion flag means a different thing on either side of it. Pooling across that boundary is permitted only for a metric the change cannot touch, and only after verifying that every byte the model saw is identical on both sides — which is what §3.4 records for §4.1, and what §4.2 declines to claim for paging.

Absence of pinned sampling. No temperature-equivalent control is available on the model under test. Consequently every cell is a sample from a distribution we cannot narrow, and repetition is the only instrument available for precision. One model in the current generation still accepts a temperature parameter, which makes it the only setting in which sampling could be held fixed; it was not deployed on the endpoint available to us.

Why no automated grader. Rubric-based graders of open-ended output vary by several points between identical inputs. That variance exceeds most effects sought here, so every measurement in this study is instead a deterministic function of the recorded call sequence. This constrains what we can ask — we cannot assess answer quality — and we accept that constraint rather than adopt an instrument coarser than the signal.