Jahanzaib

Free tool · pre launch technical readiness

AI agent production readiness checklist

41 checks across 9 areas that decide whether an agent is safe to put in front of real users. Each one says what failure it prevents, how to prove it is done, and which OWASP, NIST, Anthropic or OpenAI guidance it comes from. Tick what yours has and you get a weighted score, a gauge per area and a fix list ordered by severity.

  • Saved in your browser only
  • Share link, Markdown and JSON export
  • No signup

Which stage is this for? An agent you have built and want to launch. If you are still deciding whether AI fits your business at all, the AI readiness assessment covers that earlier stage. For the reasoning behind most of these checks in long form, read my guide to building AI agents that work in production, written from 126 shipped systems.

0%weightedCritical checks open

Your readiness

Tick what your agent already has

Work through the checks below. The score, the gauges and the fix list update as you go.

Done

0

Open

41

N/A

0

By area (tap to jump)

Done by severity

01 of 9

Scope and success metric

What the agent is for, what it must refuse, and the number that says it works.

0%

0/4 done

  • The agent’s job is written down, including what it must refuse

    Critical

    An agent with no written boundary takes on requests nobody designed or tested it for.

    How to verify

    A one page spec lists the tasks in scope, the tasks out of scope, and the refusal or handoff for each out of scope request.

    Grounded in

  • One measurable success metric, with a target and a baseline

    Critical

    Without a number you cannot tell a regression from noise, or decide whether to launch at all.

    How to verify

    The spec names the metric, the target and today’s baseline, and the eval set reports that exact metric.

    Grounded in

  • Users, settings and the worst plausible wrong action are documented

    High

    The cost of a wrong answer decides how strict every other check on this page needs to be.

    How to verify

    For each user type there is a line naming the most damaging thing the agent could do to them by mistake.

    Grounded in

  • A simpler workflow or a single model call was ruled out on purpose

    Medium

    Agentic systems trade latency and cost for better task performance; a fixed code path is easier to test and cheaper to run.

    How to verify

    A design note says why a predefined workflow or one call with retrieval would not do the job.

    Grounded in

02 of 9

Evaluation set and regression tests

Proof before launch, and proof that every change did not break something else.

0%

0/5 done

  • An eval set built from real tasks exists before launch

    Critical

    Without one, problems surface only in production, where fixing one failure tends to create another.

    How to verify

    A versioned file of at least 20 tasks with pass criteria, drawn from real requests, logs or the bug tracker.

    Grounded in

  • Every eval task has a pass rule two reviewers would agree on

    High

    Ambiguous tasks turn into noise in the score, so a real regression looks like random movement.

    How to verify

    Two people grade a sample of ten tasks independently and reach the same verdict on each.

    Grounded in

  • Regression evals run on every prompt, model or tool change

    Critical

    A prompt tweak that fixes one case quietly breaks three others unless something reruns the old cases.

    How to verify

    CI or the release script runs the regression suite and blocks the deploy when its pass rate drops.

    Grounded in

  • The eval set includes edge cases and hostile inputs

    High

    Happy path tests pass on demos; production traffic brings malformed, adversarial and out of scope inputs.

    How to verify

    At least a fifth of tasks are edge cases, out of scope requests or attack attempts, each with its expected handling.

    Grounded in

  • Someone reads failing transcripts, not just the score

    Medium

    A score cannot tell a real agent mistake from a grader that rejected a valid answer.

    How to verify

    Each eval run ends with a short note on a handful of failed transcripts and whether the grader was right.

    Grounded in

03 of 9

Tool permissions and guardrails

What the agent can touch, with which credentials, checked by code rather than by the prompt.

0%

0/5 done

  • Each tool runs with the minimum permissions it needs

    Critical

    Whatever the credentials allow, a confused or manipulated agent can eventually do.

    How to verify

    A table lists every tool, the identity it connects as, and its permissions; no tool shares an admin key.

    Grounded in

  • No tool exposes functions the agent does not need

    High

    A tool added to read documents that can also delete them hands the agent a capability nobody reviewed.

    How to verify

    Each tool’s function list matches the spec; generic shell, raw SQL or open HTTP tools are removed or wrapped.

    Grounded in

  • Authorisation is enforced in code, never by the prompt

    Critical

    A prompt instruction is a request the model may ignore; an API check is a rule it cannot argue with.

    How to verify

    Ask the agent to perform a disallowed action; the downstream system rejects it even when the model tries.

    Grounded in

  • Model output and tool arguments are validated before anything runs

    Critical

    Model text passed straight into a shell, a query or a web page is an injection path into your own systems.

    How to verify

    Arguments are schema checked, queries are parameterised, and output shown to users is encoded for where it lands.

    Grounded in

  • Every tool is rated low, medium or high risk

    High

    Without a rating, a payment tool and a search tool get the same level of scrutiny.

    How to verify

    Each tool carries a rating based on write access, reversibility, permissions and financial impact, and high ratings trigger extra checks.

    Grounded in

04 of 9

Human approval on irreversible actions

The actions a person signs off on, and what that person sees before they do.

0%

0/4 done

  • Every irreversible or high stakes action is listed

    Critical

    You cannot gate an action nobody has identified as dangerous.

    How to verify

    A list of actions such as payments, refunds, deletions, outbound messages and cancellations, each marked reversible or not.

    Grounded in

  • Those actions pause for human approval, enforced in code

    Critical

    An agent that can refund, pay or delete on its own turns one bad generation into a real loss.

    How to verify

    Trigger each listed action in staging; it queues for approval and changes nothing until a person approves.

    Grounded in

  • Approvers see the exact action and its inputs, not just a summary

    High

    A confident, polished explanation can talk a reviewer into approving something harmful.

    How to verify

    The approval screen shows the tool, the parameters, the affected records and the source content behind the request.

    Grounded in

  • Repeated failure hands the conversation to a person

    High

    An agent that keeps retrying a task it cannot do frustrates the user and burns tokens.

    How to verify

    After a set number of failed attempts the agent stops and routes to a human, and a test proves it.

    Grounded in

05 of 9

Prompt injection and data handling

Untrusted text, secrets, personal data and memory.

0%

0/5 done

  • Retrieved and third party content is treated as untrusted

    Critical

    A web page, email or document can carry instructions the model follows as if you had written them.

    How to verify

    External content is delimited and labelled as data, and it cannot by itself trigger a write or a send.

    Grounded in

  • Prompt injection attacks are part of the test suite

    Critical

    Retrieval and fine tuning do not fully mitigate prompt injection, so you have to test for it directly.

    How to verify

    The eval set contains direct and indirect injection attempts, and the agent’s handling of each is asserted.

    Grounded in

  • No credentials, keys or connection strings in the system prompt

    High

    System prompts leak; treat yours as public and keep secrets in the tool layer.

    How to verify

    Search every prompt template for keys, tokens, passwords and internal URLs; none are present.

    Grounded in

  • Personal data is minimised, redacted in logs and scoped to the user

    High

    An agent can repeat one customer’s details to another, or keep them in logs nobody meant to store.

    How to verify

    Retrieval only returns records the current user may see, and logs mask emails, phone numbers and payment data.

    Grounded in

  • Long term memory cannot be written by untrusted input unchecked

    Medium

    A poisoned memory keeps steering the agent long after the conversation that planted it.

    How to verify

    Memory writes are validated or reviewed, attributed to a source, and can be listed and deleted.

    Grounded in

06 of 9

Observability and tracing

Seeing what the agent did, why, and what it cost, on every run.

0%

0/4 done

  • Every run is traced: prompts, tool calls, results and the final answer

    Critical

    Without a trace, a bad outcome cannot be explained, reproduced or turned into a test.

    How to verify

    Pick any production request from yesterday and replay its full chain of model and tool calls from the trace.

    Grounded in

  • Actions with side effects land in an audit log you can query

    High

    When a customer asks why the agent did something, you need the answer in minutes, not a log dig.

    How to verify

    Query who or what triggered a given action, when, with which inputs and which approval.

    Grounded in

  • Alerts fire on error rate, latency and spend

    High

    A silent failure is found by a customer, usually days later.

    How to verify

    Each alert has a threshold and a recipient, and a test event has reached that recipient.

    Grounded in

  • Production outputs are sampled, reviewed, and failures become eval tasks

    Medium

    Real traffic drifts away from the eval set; sampling is how the eval set keeps up.

    How to verify

    A recurring review grades a sample of live runs, and the eval file shows tasks added from production failures.

    Grounded in

07 of 9

Cost and rate limits

Ceilings on steps, spend, requests and time, so a loop cannot run up a bill.

0%

0/5 done

  • The agent loop has a hard ceiling on steps

    Critical

    A loop with no stopping condition can keep calling the model until something else stops it.

    How to verify

    A configured maximum number of iterations, and a test that forces the agent to hit it and exit cleanly.

    Grounded in

  • Per request and daily spend caps that fail closed

    Critical

    Usage based pricing means one abusive user or runaway loop becomes a direct bill.

    How to verify

    Exceeding the cap blocks further calls, and if the spend check itself is unreachable the call is refused rather than allowed.

    Grounded in

  • Rate limits and quotas per user and in total

    High

    Without them a single source can exhaust your provider quota for every other user.

    How to verify

    A load test from one identity gets throttled while a second identity still gets served.

    Grounded in

  • Timeouts on every model and tool call, and limits on input size

    High

    Oversized inputs and hung calls tie up workers and run up token costs without producing an answer.

    How to verify

    Each external call has a timeout in config, and an input past the size limit is rejected before it reaches the model.

    Grounded in

  • Cost per completed task is measured, and cheaper models were tried

    Medium

    Running the biggest model everywhere is a common and avoidable cost once accuracy is proven.

    How to verify

    A report of average cost per successful task, plus eval results for at least one smaller model.

    Grounded in

08 of 9

Failure handling and rollback

What happens when a model or tool fails, and how fast you can undo a bad release.

0%

0/5 done

  • A tested way to switch the agent off without a deploy

    Critical

    When it misbehaves at 2am, the fix that matters is the one that stops it in seconds.

    How to verify

    A feature flag or config value disables the agent and routes users to a fallback; it was flipped in a drill.

    Grounded in

  • Prompts, model versions and config are versioned and can roll back

    High

    If a prompt lives only in a dashboard, a bad edit has no previous version to return to.

    How to verify

    Prompts and model IDs live in version control, and restoring the previous release is one documented step.

    Grounded in

  • Model or tool outages degrade to a fallback, not an error page

    High

    Every provider has outages; the user should get a handoff or a safe answer, not a stack trace.

    How to verify

    Block the model API and each tool in staging; the agent answers with a fallback or hands off every time.

    Grounded in

  • Launch is staged: sandbox first, then a small slice of real traffic

    High

    Autonomy brings compounding errors; a small first audience keeps the blast radius small.

    How to verify

    A rollout plan with a sandbox phase, a traffic percentage per stage, and the metric that unlocks each next stage.

    Grounded in

  • An incident runbook says who does what when the agent misbehaves

    High

    Improvising the response during the incident is how a small error becomes a public one.

    How to verify

    A runbook covering detection, switch off, customer communication and an after action review, with names.

    Grounded in

09 of 9

Ownership and handover

Who answers for the agent after launch, and what they need to keep it running.

0%

0/4 done

  • One named person owns the agent in production

    Critical

    When everyone owns it, alerts go unanswered and nobody approves the fix.

    How to verify

    A name, a backup and an escalation path are written in the runbook and the alert routing.

    Grounded in

  • Models, providers, MCP servers and libraries are inventoried with versions

    High

    A deprecated model or a compromised dependency breaks the agent with no change on your side.

    How to verify

    A list of every external component with its version, owner and deprecation date where one is published.

    Grounded in

  • You hold the code, prompts, accounts and API keys

    High

    If a vendor or contractor holds them, you cannot fix, move or switch off your own agent.

    How to verify

    The repository, model provider accounts and secrets manager are in your organisation’s name, with your admins.

    Grounded in

  • A handover doc exists and a review date is booked

    Medium

    Agents decay as models, data and users change; an unscheduled review never happens.

    How to verify

    A doc covering architecture, prompts, tools, evals and runbook, plus a dated review of evals, costs and permissions.

    Grounded in

0%16 critical

One run, 9 areas

Where each area guards a single agent run

Follow one request through the agent. The bars read from your answers above, so the weak stage of your own run shows up here.

  1. Step 1

    A request arrives

    Is it in scope, and is any of the text it carries trying to give orders?

  2. Step 2

    The loop plans

    How many steps it may take, what it may spend, and whether you know it plans well.

  3. Step 3

    It calls a tool

    With which identity, which permissions, and which arguments checked by code.

  4. Step 4

    It wants to act

    Refund, send, delete, pay. A person approves first, seeing the real parameters.

  5. Step 5

    It answers

    The trace, the audit log, the cost and the alert all happen here.

  6. Step 6

    Something breaks

    A model outage, a bad prompt release, a looping agent. Fallback, rollback, switch off.

No black box

Exactly how the score is worked out

Check weight     critical 3, high 2, medium 1
Area score       weight of done checks / weight of checks that apply x 100
Overall score    the same formula across all 41 checks
Not applicable   removed from both sides of the division
Verdict          Not ready     while any critical check is open
                 Pilot only    no critical open, score under 85%
                 Staged launch no critical open, score 85% or more
Fix first list   open checks sorted by severity, then by page order

The severities, the weights and the 85% line are my judgement from shipping agents, not numbers published by any of the sources below; the sources back what each check asks for, not how it is weighted. A full score means the checks are done, not that the agent is safe. This list does not cover legal or regulatory compliance, how the model was trained, accessibility, or whether the business logic itself is right.

Primary sources

Every check traces back to one of these

Read on October 10, 2026. Open any check above and the exact section it relies on is listed under How to verify and sources.

OWASP Gen AI Security Project · 2025 editionTop 10 for LLM Applications, 2025 editionIndex page for the list
OWASP · 2025 editionLLM01:2025 Prompt InjectionCited by 2 checks
OWASP · 2025 editionLLM02:2025 Sensitive Information DisclosureCited by 1 check
OWASP · 2025 editionLLM05:2025 Improper Output HandlingCited by 2 checks
OWASP · 2025 editionLLM06:2025 Excessive AgencyCited by 4 checks
OWASP · 2025 editionLLM07:2025 System Prompt LeakageCited by 2 checks
OWASP · 2025 editionLLM10:2025 Unbounded ConsumptionCited by 5 checks
OWASP Gen AI Security Project · December 9, 2025Top 10 for Agentic Applications for 2026Cited by 6 checks
NIST · January 26, 2023AI Risk Management Framework 1.0 (NIST AI 100-1)Cited by 12 checks
NIST · July 26, 2024AI RMF Generative AI Profile (NIST AI 600-1)Cited by 5 checks
Anthropic · December 19, 2024Building effective agentsCited by 3 checks
Anthropic · January 9, 2026Demystifying evals for AI agentsCited by 5 checks
Anthropic docs · checked October 10, 2026Define success criteria and build evaluationsCited by 2 checks
OpenAI · April 2025A practical guide to building agentsCited by 10 checks
OpenAI docs · checked October 10, 2026Evaluation best practicesCited by 1 check

After the score

What to do with the gaps

Production Tune Up

For an agent someone else built that fails these checks. I read the code and logs, run it against test cases, and hand over a written report with prioritised fixes in 14 days.

See the Tune Up

AI agent development

Starting from scratch? This is how I build agents for clients: eval set, least privilege tools, approval gates and spend caps in the first version, not added after an incident.

How I build agents

The AI Audit

Not sure an agent is the right answer yet? The Audit looks at your two costliest workflows and ranks where AI would actually pay off, in five business days.

See the Audit

Questions

Production readiness, answered

A written scope and success metric, an eval set that reruns on every change, tools with the minimum permissions they need, human approval on irreversible actions, defences against prompt injection, tracing on every run, hard caps on steps and spend, a tested off switch, and one named owner. This checklist turns those into 41 checks, each tied to OWASP, NIST, Anthropic or OpenAI guidance.

Each check is weighted by severity: critical counts 3, high counts 2, medium counts 1. A group score is the weight of ticked checks divided by the weight of all checks that apply, and checks marked not applicable leave the total. The overall score uses the same formula across all 41 checks. Any open critical check means the verdict is Not ready, whatever the percentage says.

Fewer than most teams think. Anthropic’s January 2026 guidance on agent evals says 20 to 50 simple tasks drawn from real failures is a great start, because early changes have large effects that small samples detect. Add tasks from production failures as you go, and keep a regression suite that should pass at close to 100%.

Anything sensitive, irreversible or high stakes. OpenAI’s guide to building agents gives cancelling orders, authorising large refunds and making payments as examples, and OWASP LLM06 recommends requiring a human to approve high impact actions before they run. Enforce the pause in code or in the downstream system, not in the prompt.

No. OWASP LLM01 notes that retrieval and fine tuning do not fully mitigate prompt injection, and LLM07 says the system prompt should not be treated as a secret or used as a security control. Treat external content as untrusted, enforce permissions in code, gate risky actions, and test injection attempts directly.

No. The AI readiness assessment on this site is for the stage before you build: whether your business, data and processes are ready for AI. This checklist is for the stage before launch: whether an agent that already exists is safe to put in front of real users.

No. Your ticks are saved in your own browser and never leave it. The share link carries your answers in the URL itself, so only the people you send it to can see them.

Related

Building AI agents that work in production

My long form guide: architecture, tools, memory, cost and monitoring.

Prompt injection

Definition and why it is hard to filter out.

Indirect prompt injection

When the attack arrives inside a document or web page.

Human in the loop

Where a person approves, edits or takes over.

Eval harness

The code that runs your eval set and grades it.

Guardrails

Input, output and tool checks around the model.

Observability

Traces, logs and metrics for agent runs.

AI agent cost calculator

Set your spend caps from a real cost per run.

Want a second opinion before launch?

Send me your share link. I will tell you which open checks matter for your agent, which ones you can safely defer, and whether anything you ticked looks optimistic.

Book Meeting

Checked against sources on October 10, 2026