Clay-style illustration of an off-white archive cabinet with one drawer open, where a brass tray lifts out three blank cards toward a black and teal desk.

AI Agent Token Optimization: Paging Large JSON Tool Responses

November 18, 2026

Clay-style illustration of an off-white archive cabinet with one drawer open, where a brass tray lifts out three blank cards toward a black and teal desk.

By Erin Harris, AI Strategy & Business Transformation Consultant

Agents processing large structured tool results should not inject entire JSON payloads into every model turn. Store full artifacts behind authorized references, return a compact description, and fetch only the required records on demand. Benchmark answer correctness as well as token and latency reductions.

What the Flightscape case shows

On November 17, AWS and Flightscape documented a training-schedule optimization agent with more than 40 tools and multi-megabyte JSON inputs. The team reported approximately 71 percent lower token consumption and nearly 80 percent lower average latency over a five-question run using a representative 23 MB payload. Across 100 questions, token use fell 64 percent and average latency 69 percent. These are reported workload-specific measurements, not a general price reduction. Source: https://aws.amazon.com/blogs/machine-learning/reducing-token-costs-and-latency-in-amazon-bedrock-agentcore-json-workloads/

The failure mode

An agent asks a tool for a result list. The tool returns thousands of records. The entire response enters model context even when the question concerns one subset. On later turns, context grows again. Cost and latency rise while the model may become less reliable because relevant records are buried.

The three-part pattern

First, offload heavy request, response and validation artifacts to an authorized object store, retaining stable references in conversational state. Second, return a count, a small sample and a handle for oversized lists, with a paging tool for additional slices. Third, represent oversized dictionaries through field names, types and sizes, allowing targeted retrieval of nested content. The AWS example uses Amazon S3, AgentCore memory and LangGraph; this is an architecture pattern, not a newly announced universal feature.

Hypothetical example

An operations assistant receives 25,000 inventory events. A manager asks about exceptions at one warehouse yesterday. Instead of copying all 25,000 records into the prompt, the tool returns the matching count and a small authorized sample. The assistant requests the next relevant page if needed. This example illustrates a design choice; no performance outcome is assumed.

The hidden security work

Artifact handles must not become bypass keys. Check user authorization every time a page is retrieved. Apply retention policies, protect sensitive records in storage and logs, and define behavior when an artifact expires or a page request fails. Compaction can omit an exception that matters, so evaluation must include questions where the answer is in a rarely accessed portion of the data.

How to benchmark

Choose a fixed set of representative questions and large payloads. Record token input and output, p50 and p95 latency, task accuracy, storage and retrieval costs, and failure recovery. Compare full-payload and paged implementations using the same questions. Include multi-turn follow-ups and permission-negative tests.

Business decision

This architecture is most relevant when tool responses are large enough to dominate model context. A small CRM lookup returning a few fields may not justify the added complexity. EH Synergy's AI Opportunity & Transformation Assessment can examine whether context management is a meaningful bottleneck before committing to a more elaborate agent stack.

Key Takeaways

  • Profile tool outputs before optimizing model selection.
  • Return references and pages instead of full payloads by default.
  • Protect every artifact retrieval with authorization.
  • Measure task accuracy and total infrastructure cost.
blog author avatar

Erin Harris

Founder of EH Synergy. M.Ed. in Educational Technology & Instructional Design and MBA in Marketing. Erin helps everyday people and small businesses use AI and automation to build real, useful tools.

Back to Blog
Erin Harris, Founder of EH Synergy

About the Author

Erin Harris, Founder of EH Synergy

M.Ed. in Educational Technology & Instructional Design and MBA in Marketing. Erin writes about how businesses can use AI and automation in ways that actually work.

Book a Call →

New to AI? Start with the free course.

The Clear Build Starter: five short lessons to build your first planner or tracker with AI. No coding needed.

Start the Free Course
EH Synergy

EH Synergy LLC · Dallas, Texas

Book a Call
© 2026 EH Synergy LLC