A successful AI agent pilot can give a team dangerous confidence.
Twenty clean cases go through. The model responds quickly. The connector works. A reviewer is sitting nearby. Everyone sees the expected result, and the project gets labeled ready for production.
Then real demand arrives. Requests overlap. An API starts rejecting calls. The model slows down. The approval queue grows. One downstream system accepts five writes per second while the agent creates twenty. Retries multiply the load. The workflow technically remains online while the business waits.
The pilot answered whether the agent could do the work. Before expansion, you need to know whether the whole system can do enough of the work, fast enough, without losing control or destroying the economics.
Test the workflow, not the demo
An agent is one component inside a chain. The trigger, identity service, model, retrieval layer, tools, APIs, databases, approval steps, notifications, and downstream writes each have their own capacity and failure behavior.
The slowest component sets the workflow's real throughput. A faster model does not help if a billing API allows fewer requests, a reviewer can clear only twelve exceptions an hour, or a CRM job locks records during a nightly sync.
Draw the path from arrival to final business outcome. For each step, record the expected rate, tested rate, maximum allowed concurrency, timeout, retry rule, queue limit, and owner. If those numbers are unknown, the production design is still based on hope.
Build a demand profile from the business
Do not begin with a vendor's maximum transactions per minute. Begin with the workflow's actual demand.
Measure normal volume, peak volume, burst size, seasonal events, coverage hours, item complexity, and acceptable completion time. Separate a steady stream from a sudden batch. Processing 10,000 items over a day is a different problem from receiving 2,000 items at 9:00 a.m.
Include the ugly days. Month end, a campaign launch, storm calls, enrollment deadlines, security incidents, Monday ticket spikes, and recovery after planned maintenance are the days that expose the design. An average taken across a quiet week will hide them.
Then choose a realistic growth case. You do not need to test fantasy volume. You do need evidence for the demand you plan to authorize during the next release period.
Run five capacity tests before expansion
| Test | What to simulate | What to measure |
|---|---|---|
| Expected load | Normal production mix for a full operating window | Completed outcomes, latency, errors, review work, and unit cost |
| Peak burst | Highest credible arrival rate and batch size | Queue growth, rejected calls, throttling, and time to clear |
| Sustained load | Elevated demand long enough to expose leaks and drift | Latency trend, memory, connection use, cost, and degradation |
| Dependency slowdown | A model, API, database, or reviewer operating below normal capacity | Backpressure, timeout behavior, routing, and user impact |
| Recovery load | A paused queue released after the dependency returns | Duplicate prevention, ordering, reconciliation, and safe drain rate |
Use production-like records with sensitive data removed or protected according to your test policy. Keep the same permission boundaries, workflow logic, tool sequence, and review rules planned for release. A lightweight mock can prove basic orchestration, but it may hide real API behavior and downstream locking.
The NIST Generative AI Profile recommends evaluating system performance in real-world scenarios because practical environments can reveal issues that controlled, optimized tests miss. It also advises against extrapolating performance from narrow, anecdotal assessments. That is a good description of why twenty handpicked pilot cases are not a capacity test.
Rate limits need an operating rule
Every external service can constrain the workflow. Published quotas are only the starting point. Limits may vary by account, model, endpoint, time window, region, or service tier. Some systems reject excess requests. Others slow down, queue work, or return a response that invites a retry.
Ask each vendor for the limits that apply to your exact configuration. Then test them. Confirm whether limits are shared across teams or workloads, how usage is reported, how increases are approved, and what happens when the ceiling is reached.
Your own design needs priorities. A backlog of low-value enrichment work should not block a customer escalation or an access-revocation task. Reserve capacity where the business consequence justifies it. Define which work waits, which work falls back to a person, and which work stops.
Control retries before they become new demand
A timeout does not always mean the action failed. The downstream system may have completed the write while the agent stopped waiting. A blind retry can create a duplicate order, ticket, notification, or account change.
Retries also add load during the exact moment a dependency is already struggling. One failed request becomes three. Then ten concurrent jobs do the same thing. The recovery logic turns a slowdown into a larger incident.
Use limited retries, increasing wait intervals, idempotency controls, and a circuit breaker that stops calls when a dependency remains unhealthy. Preserve the transaction state so the team can reconcile uncertain outcomes. The AI agent failure recovery plan should cover overload, not only model errors.
Count human capacity in the load test
If people approve exceptions, complete fallbacks, or investigate uncertain writes, they are part of system capacity.
Run peak volume through the real approval workflow. Measure how many items route to review, the time needed per decision, queue age, rejection rate, escalation volume, and available coverage. Watch what happens when the reviewer is in a meeting, offline after hours, or handling another incident.
A model response measured in seconds can still produce a business outcome measured in days. Do not call the workflow fast while its exception queue is aging.
Set a backlog budget
Queues are useful because they absorb short bursts. They are dangerous when nobody has defined how large they may become.
Set a maximum queue depth and maximum item age for each work class. Define alerts before those limits are reached. Decide when to reduce intake, switch to a simpler process, route work manually, or stop the agent.
Measure time to drain after a spike. A system that handles 100 items per hour against an average arrival of 90 looks healthy until 500 delayed items enter the queue. With only ten units of spare capacity, recovery takes fifty hours unless you add capacity or change priorities.
That recovery period affects customers and staff. Put it in the acceptance decision.
Measure cost under pressure
Production volume can change the economics even when the workflow stays reliable. Longer context, repeated retrieval, tool calls, retries, logging, storage, reviewer time, and higher service tiers all add cost.
Track cost per completed business outcome at expected, peak, and recovery load. Failed attempts do not disappear from the bill. Neither does the labor required to investigate them. Add the findings to the full integration cost model, not a separate technical appendix nobody uses in the funding decision.
If the unit cost rises sharply under normal peak demand, the pilot business case is incomplete.
Define the release decision before the test
A load test is useless if every result becomes acceptable after the fact. Write the thresholds first.
Set the minimum completed outcomes per hour, maximum end-to-end latency, maximum error and duplicate rates, queue-age limit, recovery time, human review load, and cost per outcome. Include a stop condition for consequential failures even when the average looks good.
Record the tested workflow version, model, prompts, connectors, limits, data mix, and infrastructure. A material change can invalidate the result. Feed those triggers into the AI agent release test.
Pass does not have to mean unlimited rollout. It can authorize one department, one volume band, or one action class. Increase authority when the evidence supports it, then test the next boundary.
Make the vendor show you the bottleneck
Do not ask whether the platform scales. The answer will be yes.
Ask the vendor to run your expected and peak workflow mix with your integrations and approval rules. Require transaction-level results for latency, throttling, retries, queue depth, errors, cost, and recovery. Ask which limits belong to the vendor, which belong to connected systems, and which you must monitor.
A pilot that completes the right task is worth continuing. It is not permission to skip production evidence.
Test the volume you intend to authorize. Test the bottlenecks you already know about. Then slow something down and watch the workflow recover. That is how you find out whether you built useful capacity or another queue the business will eventually have to rescue.