
AI Agent Token Optimization: Paging Large JSON Tool Responses

By Erin Harris, AI Strategy & Business Transformation Consultant
Agents processing large structured tool results should not inject entire JSON payloads into every model turn. Store full artifacts behind authorized references, return a compact description, and fetch only the required records on demand. Benchmark answer correctness as well as token and latency reductions.
What the Flightscape case shows
On November 17, AWS and Flightscape documented a training-schedule optimization agent with more than 40 tools and multi-megabyte JSON inputs. The team reported approximately 71 percent lower token consumption and nearly 80 percent lower average latency over a five-question run using a representative 23 MB payload. Across 100 questions, token use fell 64 percent and average latency 69 percent. These are reported workload-specific measurements, not a general price reduction. Source: https://aws.amazon.com/blogs/machine-learning/reducing-token-costs-and-latency-in-amazon-bedrock-agentcore-json-workloads/
The failure mode
An agent asks a tool for a result list. The tool returns thousands of records. The entire response enters model context even when the question concerns one subset. On later turns, context grows again. Cost and latency rise while the model may become less reliable because relevant records are buried.
The three-part pattern
First, offload heavy request, response and validation artifacts to an authorized object store, retaining stable references in conversational state. Second, return a count, a small sample and a handle for oversized lists, with a paging tool for additional slices. Third, represent oversized dictionaries through field names, types and sizes, allowing targeted retrieval of nested content. The AWS example uses Amazon S3, AgentCore memory and LangGraph; this is an architecture pattern, not a newly announced universal feature.
Hypothetical example
An operations assistant receives 25,000 inventory events. A manager asks about exceptions at one warehouse yesterday. Instead of copying all 25,000 records into the prompt, the tool returns the matching count and a small authorized sample. The assistant requests the next relevant page if needed. This example illustrates a design choice; no performance outcome is assumed.
The hidden security work
Artifact handles must not become bypass keys. Check user authorization every time a page is retrieved. Apply retention policies, protect sensitive records in storage and logs, and define behavior when an artifact expires or a page request fails. Compaction can omit an exception that matters, so evaluation must include questions where the answer is in a rarely accessed portion of the data.
How to benchmark
Choose a fixed set of representative questions and large payloads. Record token input and output, p50 and p95 latency, task accuracy, storage and retrieval costs, and failure recovery. Compare full-payload and paged implementations using the same questions. Include multi-turn follow-ups and permission-negative tests.
Business decision
This architecture is most relevant when tool responses are large enough to dominate model context. A small CRM lookup returning a few fields may not justify the added complexity. EH Synergy's AI Opportunity & Transformation Assessment can examine whether context management is a meaningful bottleneck before committing to a more elaborate agent stack.
Key Takeaways
- Profile tool outputs before optimizing model selection.
- Return references and pages instead of full payloads by default.
- Protect every artifact retrieval with authorization.
- Measure task accuracy and total infrastructure cost.

