"They loved my reasoning until the invoice arrived. Turns out every brilliant thought I had was billed by the token, and the bearing did not care how eloquent I was at 03:14."
A Budget-Conscious AI Agent
Prerequisites
This section closes the chapter, so it assumes the loop of Section 22.1, the tool-calling machinery of Section 22.4, and the guardrails and confidence of Section 22.6. From elsewhere in the book you need the streaming-deadline mindset of Chapter 60, the fleet-operations framing of Chapter 69, and a little probability from Chapter 4. No new agent vocabulary is introduced; we are costing and hardening what you already built.
The Big Picture
An agentic sensing loop is a beautiful idea and a running bill. Each turn is a model call plus a tool call, so the very thing that makes an agent smart, its willingness to look again, is also what makes it slow, expensive, and prone to compounding failure. A demo that resolves one anomaly in six turns feels magical. Multiply it by a fleet of ten thousand assets, each emitting alerts around the clock, and the same design is a five-figure monthly invoice with a tail latency that blows past your alarm deadline and a per-incident success rate that quietly erodes as stochastic steps stack up. This section is the engineering discipline that decides whether an agentic pipeline ships: how to budget tokens and turns, how to bound latency under a real deadline, and how to keep a chain of unreliable steps reliable enough to trust. It is where the perception work of this book meets the operations reality of Chapter 69.
Costing the loop: turns are the unit of spend
What drives the bill is not the model in the abstract but the turn count. Every turn re-sends the full accumulated context (the prompt, the reasoning so far, every tool result) plus generates new tokens, so a \(k\)-turn investigation does not cost \(k\) times a single call, it costs more, because the context grows on each pass. If the prompt starts at \(p_0\) tokens and each turn appends about \(a\) tokens of observation and reasoning, the total input billed across the episode is roughly
$$ T_{\text{in}} \;=\; \sum_{i=1}^{k}\bigl(p_0 + (i-1)\,a\bigr) \;=\; k\,p_0 + a\,\frac{k(k-1)}{2}. $$Why this matters: the quadratic term means cost grows with the square of the turn budget, not linearly. Doubling the maximum number of turns an agent may take to chase a stubborn fault can quadruple the token spend on the episodes that actually use them. This is the single most important number to instrument, and it is why Section 22.1's crude max_turns guard was never cosmetic. When you feel the pain: sensor telemetry is verbose, so \(a\) is large (a spectrum summary, a SCADA dump, a detector's full output), and the quadratic bites early. The mitigations are context compression (summarize old turns instead of re-sending them verbatim) and prompt caching, where the provider bills the unchanged prefix at a steep discount so re-sending the stable system prompt and tool schemas each turn is nearly free.
Key Insight
The right economic unit for an agentic pipeline is not cost per token or cost per call. It is cost per resolved incident: total spend divided by the number of alerts the agent actually closed correctly. A cheaper model that needs eight fumbling turns to reach the same verdict a stronger model reaches in three is not cheaper. Optimize the whole trajectory, not the per-token price, because a model that reasons well buys back its unit-cost premium in avoided turns, and turns are quadratic.
Latency: the tail is the sensor deadline
Sensing has deadlines that a chatbot does not. A vibration alarm that must reach a control room within fifteen seconds, a cardiac-rhythm flag that gates a clinician's page, a substation-protection decision, all impose a hard wall-clock budget that an agentic loop must respect even in its worst case. What makes agent latency hard is that a turn is sequential: the model must finish thinking before it can call a tool, and it must see the tool result before it can think again. So the per-episode latency is a sum of serial legs,
$$ L \;=\; \sum_{i=1}^{k}\bigl(\ell^{\text{llm}}_i + \ell^{\text{tool}}_i\bigr), $$and because each leg is itself a random variable with a heavy right tail (model queue time, a slow database, a cold detector), the episode tail is worse than the sum of the medians. Why you must design to the tail: an SLO written on the p50 is a lie an operator finds out about during the one incident that mattered. Budget to p95 or p99. How to bound it: cap turns hard, run independent tool calls concurrently within a turn rather than serially (fetch three channels at once), set aggressive per-tool timeouts with a defined fallback, and stream partial verdicts so a human sees the leading hypothesis before the loop formally concludes. The streaming and deadline discipline of Chapter 60 applies here almost verbatim; an agent is just a very expensive stream stage.
Reliability: many stochastic steps, one compounding pipeline
A pipeline is only as reliable as the product of its parts, and an agentic loop has many parts: an LLM step that may hallucinate a tool name, a detector that may time out, a parser that may choke on malformed output, a channel fetch that may return stale data. If each turn succeeds independently with probability \(r\), an unguarded \(k\)-turn episode completes with probability \(r^{k}\), which decays fast: at \(r = 0.97\) and \(k = 8\), episode success is only about \(0.78\). Why agents feel flaky in production even when each component looks solid is exactly this compounding. How to fight it, borrowed wholesale from distributed-systems engineering: bounded retries on transient tool failures (with idempotent tools so a retry is safe), timeouts so a hung step cannot stall the deadline, a circuit breaker that stops hammering a detector that is failing, and above all a fallback path. When the agent cannot conclude within budget, it must degrade gracefully to something trustworthy: the classical threshold detector of Chapter 12 firing a plain, uncalibrated-but-honest alert beats an agent that silently drops the incident. The guardrails of Section 22.6 decide what to trust; this section decides what to do when the machinery fails.
The snippet below estimates all three quantities for a candidate design before you deploy it, so the turn budget is a decision backed by a number rather than a hope. It composes the quadratic token model, the serial-latency sum, and the compounding-reliability product into one back-of-envelope planner referenced throughout this section.
import numpy as np
def episode_budget(k, p0=1200, a=800, price_in=3e-6, price_out=15e-6,
out_per_turn=250, ell_llm=1.4, ell_tool=0.6, r_turn=0.97):
"""Expected tokens, cost ($), p95 latency (s), and success prob for a k-turn loop."""
tok_in = k * p0 + a * k * (k - 1) / 2 # quadratic: context regrows each turn
tok_out = k * out_per_turn
cost = tok_in * price_in + tok_out * price_out
# serial legs; tail approximated by sampling each leg as heavy-right (lognormal)
rng = np.random.default_rng(0)
legs = rng.lognormal(np.log(ell_llm), 0.5, (20000, k)) + \
rng.lognormal(np.log(ell_tool), 0.5, (20000, k))
p95 = np.percentile(legs.sum(axis=1), 95)
success = r_turn ** k # unguarded compounding
return dict(tokens_in=int(tok_in), cost_usd=round(cost, 4),
p95_latency_s=round(p95, 2), success=round(success, 3))
for k in (3, 6, 10):
print(k, episode_budget(k))
Listing 1 turns the turn budget from a guess into an engineering choice. In the defaults shown, ten turns already roughly triples the token cost of three and drops unguarded success below \(0.75\), which is why real pipelines rarely let an agent wander past six or seven turns without a fallback catching it.
In Practice: A Wind-Fleet Copilot That Learned to Triage
A renewables operator ran an agentic monitor across 4,200 turbines. The first design sent every alert, benign or not, into a full six-turn investigation on a frontier model. It worked, and the monthly inference bill was indefensible: most alerts were gust-loading false positives that a single cheap check could clear. The team restructured it as a cascade. A tiny fast model (or often just the classical envelope detector of Chapter 37) triaged first; roughly 88 percent of alerts were resolved there in one turn for a fraction of a cent. Only the ambiguous remainder escalated to the full agentic loop, and even that loop got a hard turn cap of five plus a fallback that emitted a classical-severity alert if the budget ran out. Cost per resolved incident fell by about 9x, p95 latency on the common path dropped from twelve seconds to under two, and, because the fallback never silently dropped an alert, incident recall actually improved. The lesson: an agent should be the expensive specialist you call when cheap triage cannot decide, not the front door.
Model routing and cascades: spend where it pays
The wind-fleet story generalizes into the central cost lever for agentic sensing: do not run one model on everything. A cascade (also called a router) sends each input to the cheapest stage that can handle it and escalates only on uncertainty. Why it works so well here is the structure of sensor alerts: the distribution is heavily skewed toward the easy and the benign, so a cheap triage stage clears the bulk and the costly agent sees only the genuinely hard tail. Expected cost per alert becomes a mixture, \(\mathbb{E}[C] = \sum_j \pi_j\, C_j\), where \(\pi_j\) is the fraction of traffic escalating to stage \(j\) and \(C_j\) its cost; because \(\pi\) decays fast down the cascade while \(C\) rises, the product stays small. How to set the escalation threshold: use the calibrated confidence of Chapter 18, not a raw softmax, so "escalate when unsure" means something honest. A conformal prediction set that is a singleton passes; a wide set escalates. When a cascade backfires: if the cheap stage is miscalibrated and confidently clears real faults, you have saved money by missing incidents, which is the worst trade in sensing. Calibrate the gate before you trust it.
Right Tool
Retries, timeouts, circuit breakers, exponential backoff, and fallback chains are exactly the reliability primitives that request libraries have offered for a decade; you should not hand-roll them around an LLM call. A resilience decorator wires the whole hardened path in a few lines:
from tenacity import retry, stop_after_attempt, wait_exponential
@retry(stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=0.3, max=4)) # backoff on transient tool errors
def call_detector(args):
return bearing_envelope(args, timeout=2.0) # per-tool deadline
def hardened_turn(step):
try:
return call_detector(step.args)
except Exception:
return classical_fallback(step.args) # graceful degradation, never drop
Adopting a resilience library (tenacity, or a framework's built-in retry policy) replaces roughly 60 to 80 lines of hand-written backoff, jitter, attempt-counting, and circuit-breaker state with a decorator and a try/except. What it cannot supply is the fallback itself: choosing that a failed agentic turn degrades to the classical detector of Chapter 12 rather than to silence is a sensing decision, and it is the one that keeps the pipeline trustworthy.
Research Frontier
Making agentic pipelines cheap and reliable enough for always-on telemetry is an open research area, not a solved one. LLM cascades and routing (FrugalGPT and its successors) show large cost reductions by escalating only hard queries, but principled, calibrated escalation gates for streaming sensor data remain immature. Prompt caching and KV-cache reuse across turns, now offered by the major model providers, sharply cut the quadratic context cost, yet the accounting of what is cacheable in a live tool-calling loop is still being worked out. On the reliability side, the industrial copilots named in Section 22.3 (Argos, AgentFM, CALM) report the operational metrics that actually gate deployment, cost per resolved incident, tail latency, and escalation-to-human rate, and standardizing those metrics across the field is itself frontier work. The honest state of the art: a well-built cascade with a hard turn cap and a classical fallback is deployable today; a fully autonomous, unbounded agentic investigator over safety-critical sensing is not.
Exercise
Using Listing 1: (1) Plot cost, p95 latency, and unguarded success as functions of the turn budget \(k\) from 1 to 12 for your own model's real token prices; identify the largest \(k\) that fits a 10-second p95 deadline. (2) Add a fallback so the loop always concludes within the deadline (either the agent's verdict or a classical alert), and derive the guarded episode success probability, which no longer decays as \(r^k\). (3) Model a two-stage cascade: let a fraction \(\pi\) of alerts escalate from a cheap stage (cost \(C_1\)) to the agent (cost \(C_2\)) and compute the crossover \(\pi\) below which the cascade is cheaper than running the agent on everything. Comment on what happens to that crossover if the cheap stage is miscalibrated.
Self-Check
1. Why does the token cost of a \(k\)-turn agentic episode grow quadratically rather than linearly in \(k\), and name one mechanism that flattens the growth.
2. Why must an agentic-sensing latency SLO be written on the p95 or p99 rather than the median, and what happens to a bounded-retry design's success probability compared with the unguarded \(r^k\)?
3. State the metric that should replace cost-per-token when optimizing an agentic pipeline, and explain why a cheaper per-token model can be more expensive by it.
Lab 22
build an agent that monitors a sensor stream, invokes a detector tool, and produces an explained, uncertainty-tagged alert.
Bibliography
Agent loops and reliability primitives
Yao et al. (2023). ReAct: Synergizing Reasoning and Acting in Language Models. ICLR.
The interleaved reason-then-act loop whose turn count this section costs; every latency and reliability sum here is measured over its turns.
Shinn et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. NeurIPS.
Self-critique adds turns, and therefore cost and failure surface; a concrete case for why the turn budget must be bounded.
Cost, cascades, and efficient inference
The cascade/router argument this section builds on: escalate only hard inputs to expensive models, the central cost lever for fleet-scale sensing.
KV-cache reuse and paged attention are what make re-sending a growing agent context affordable; the systems basis for prompt caching.
Industrial and operational agents
Reports the operational metrics (cost, latency, escalation rate) that decide whether an agentic telemetry pipeline ships, the concern of this section.
Practical grounding for the tool-call latency and failure modes that dominate the per-turn reliability product.
Reliability engineering carried over
Beyer et al. (2016). Site Reliability Engineering. O'Reilly / Google.
The timeout, retry, circuit-breaker, and graceful-degradation vocabulary this section imports wholesale to harden a stochastic agent pipeline.
What's Next
That closes Part V and the foundation-model arc of this book: you can now represent sensor streams with pretrained models, fuse them with language, and wrap the whole thing in an agent that reasons, acts, and stays within a budget. In Chapter 23, we return to the physical bedrock these agents ultimately reason about, opening Part VI with the inertial sensors, accelerometers, gyroscopes, and magnetometers, whose noisy motion signals drive everything from step counting to dead reckoning.