A founder watches their team demo an AI agent on Friday afternoon. The agent reads a customer email, looks up the customer record, calls a pricing tool, drafts a response. It works. The founder asks the obvious question: “Can we ship this Monday?” The honest answer is no, and the follow-up is “here is what’s between Friday and Monday.” This post is that list. Five blockers we see on every demo-to-deploy conversation, what each looks like in the wild, and what to ship instead.
The demo-to-deploy gap is where most agent projects die. The team has 80% of the work done from the engineer’s perspective and 0% from the operations perspective. The blockers below are how the remaining 20% of perspective becomes 80% of the timeline.
The demo works because the data is faked. The customer record is a 4-row sample in mock_customers.py. The pricing tool returns a constant. The email sender prints to stdout. The agent does not know it is on a stage set. The demo is convincing because the founder is not looking under the floorboards.
What it looks like in code:
# demo.py
MOCK_CUSTOMERS = {
"acme": {"plan": "pro", "credits": 240, "owner": "[email protected]"},
"bigco": {"plan": "ent", "credits": 4800, "owner": "[email protected]"},
}
def lookup_customer(name: str) -> dict:
return MOCK_CUSTOMERS.get(name.lower(), {"plan": "free", "credits": 0})
def send_email(to: str, body: str) -> dict:
print(f"[mock send] to={to}n{body}")
return {"sent": True, "message_id": "demo_msg_001"}
What it looks like in production:
The shortcut some teams take, “we’ll wire the real backends later,” misallocates work. The real backends are the work. The agent’s logic depends on the shape and failure modes of real data; a backend swap at the end is a rewrite of every tool.
What to ship instead: real backends from day one, with a feature flag that lets the agent run in shadow mode (reads happen, writes stub to log-only) until you trust it. Shadow mode lets you exercise the agent against production data without risking production state. It is the cheapest reliability win on the list.
The demo had three customer emails. A polite English renewal question. A polite refund request. An upgrade question. All three from a slide deck the team prepared the day before. The agent handled all three.
Production has every English question, plus emails in Spanish and Vietnamese, customers who reply to a five-message thread by quoting the original at the top, the angry customer who writes in all caps, the bot that sends “unsubscribe” to every reply-to address, the customer whose signature includes “Sent from my iPhone” three times because of a forwarding loop. The agent that handled three handpicked cases is in a different distribution.
This is the eval-harness blocker. Anthropic’s production guidance is direct: “extensive testing in sandboxed environments, along with the appropriate guardrails” before deployment. (Anthropic, “Building Effective Agents”) Three demo cases is not extensive.
What to ship instead: a golden-test suite of 30 to 60 cases drawn from real production data (anonymized, matching live distribution of language, length, sentiment, and edge characteristics), plus per-step scoring functions so a regression on one capability tells you where the failure is. We cover the harness in the eval harness spoke. The architecture move is to make the eval suite the gate the agent must pass before production traffic, not after.
Two rules. First, build the suite before rollout, not after; retroactive evals cost more because you are reverse-engineering what the agent already does. Second, source cases from real traffic; LLM-generated synthetic cases over-represent the distribution that LLM produces, not the one real users produce.
This is Blocker 2’s bigger sibling. A team can have 60 real test cases in a spreadsheet and still ship without a regression gate. The cases exist; nobody runs them on every change. The first prompt tweak after launch silently regresses a behavior the team had reliable on Friday, and the team only notices when a customer complains three weeks later.
“We’ll add evals after launch” is a trade most teams do not understand. The cost curve goes up after launch, not down. Before launch you are capturing the behaviors you want. After launch you are reverse-engineering the behaviors the agent currently has, including the ones nobody consciously decided to ship.
What to ship instead: a CI workflow that runs the golden suite on every PR touching agent code, prompts, or tools. The suite fails the build if any metric drops below threshold. Thresholds live in a config file; changes are deliberate diffs reviewed by humans. The whole rig is in the eval harness spoke and costs roughly five engineer-days. The cost of skipping it is a regression you find weeks late, plus the political cost of “the agent stopped working and we do not know why.”
The demo failed once during the Friday rehearsal. The engineer added a print statement, saw a malformed tool call, fixed it, and the demo passed. In production, the print statement is missing, the malformed call surfaces as a 500 error, and the on-call engineer at 2:14 AM has no record of what the agent saw, which tool it called, what arguments it passed, or which step failed. The post-mortem becomes a guess.
Demos run for ten minutes with one user and one developer staring at the terminal. Production runs for years with thousands of turns per day across hundreds of tenants. The only mechanism that scales to that load is structured, queryable logs on every span of every turn. We cover what to log per span in the observability spoke and how the layer fits in the cornerstone post.
The pattern that costs the most to skip: per-tenant attribution. A demo agent without tenant attribution is fine. A production agent without it cannot answer “did this regression happen for one customer or all customers” and cannot debug “this customer says the agent moved their data at 9:14 AM, what did it actually do.” Adding tenant attribution later is a schema migration; adding it on day one is one extra attribute.
We have watched a team without per-tenant attribution spend eleven days trying to reproduce a customer-reported issue before concluding the agent had not touched the field; another integration had. Eleven days of two engineers to answer a question that one well-formed log query would have answered in 90 seconds.
What to ship instead: OpenTelemetry-shaped spans with the GenAI semantic conventions, redaction at the logging boundary, per-tenant attribution on every span, correlation IDs across distributed calls, and a dashboard the on-call engineer trusts. The OpenTelemetry GenAI conventions are stabilizing as the cross-vendor floor. (OpenTelemetry, “Semantic conventions for generative AI systems”) Pick the standard; do not invent your own schema.
The Friday demo deploy is a git pull, a service restart, and a coffee. The Monday production deploy needs a way to ship the change to one percent of traffic, watch the eval metrics and error rate, and ship to the rest only if the signal is clean. It needs a kill switch in case the model provider has a degradation that lifts your error rate by ten points. It needs a rollback path that does not require a 30-minute build.
Agent teams often deprioritize this work because the agent feels like an inference call, not a backend. It is a backend.
The rollout pattern:
For agents that depend on a specific model version, also pin the model. A provider model update that ships overnight changes behavior on every tool call. Upgrade deliberately with an eval-gated rollout, not implicitly with whatever the provider’s default endpoint resolves to today.
The five blockers above are the work between Friday and a credible production launch. The ship plan we run with clients who have a working demo:
Four weeks is the floor for a team that has done it before. Six to eight is the typical timeline. Twelve weeks is what we see when a team skips Week 2; the regressions surface in Weeks 6 to 12 and the eval work happens under incident pressure instead of as planned engineering.
Our typical end-to-end production agent build runs 16 to 24 weeks. Teams that compress to 8 to 12 weeks have either done it before or are accepting the production risk of skipping layers. Teams that quote 4 weeks are quoting the demo. From Prompt to Production is the work between those two estimates.
The five blockers are the operational view of the five layers in the cornerstone production AI agent architecture:
Layer 5 (compliance integration) is not a blocker in the same operational sense; it informs every decision in the other four. If your agent operates in a regulated environment (HIPAA, SOC 2, PCI-DSS, model-risk review), Layer 5 is a parallel workstream that starts on day one. We cover the HealthTech case in the HIPAA architecture spoke.
Why do AI agent demos work but production deploys fail?
The demo runs against fake data with a curated input set, no real backends, no real users, no error rates, and no observability. The production deploy adds real backends with their failure modes, the real input distribution, real concurrent users, real edge cases, and the requirement to debug what went wrong at 2 AM. The agent logic is the same; the operational surface area is an order of magnitude larger.
How long does it take to move an AI agent from demo to production?
Four weeks is the floor for a team that has done it before. Six to eight weeks is the typical timeline. Twelve weeks or more if you are skipping the eval harness and adding it after launch. The duration tracks the five blockers: real backends, golden test suite, regression gate, observability, and deploy mechanics.
What is shadow mode for an AI agent deploy?
A feature-flag mode where the agent runs against real production data but takes no user-visible action. Tool reads happen; tool writes are stubbed to log-only. Shadow mode lets you exercise the agent against production-shaped inputs and observe behavior without risk to production state. It is the cheapest reliability win in an agent rollout.
What is a canary deploy for an AI agent?
A staged rollout where the new agent version takes 1% of traffic first, then 10%, then 50%, then 100%. Each step gates on error rate, eval-suite pass rate replayed against recent traffic, and cost-per-turn deviation. A failure at any step rolls back to the prior step. Canary deploys catch the regressions that the eval suite did not anticipate.
What is a kill switch for an AI agent?
A single feature flag or environment variable that turns the agent off across the fleet without a redeploy. Used when a third-party model provider has a degradation, when a regression escapes canary, or when an incident requires immediate containment. The on-call engineer must be able to flip it without paging anyone else.
Do you need all five demo-to-deploy blockers for an internal-only AI agent?
Internal agents used by a single team need real backends and basic observability. The golden test suite, regression gate, canary deploy, and kill switch matter more as the user count and the cost of a wrong answer grow. An internal-only agent with 5 users on a low-stakes workflow can ship lighter; an internal agent that touches money, customer records, or production systems needs the full set.
If your team has a working agent demo and a production launch date, Mirgen and the Motomtech team have shipped this layer for HealthTech, FinTech, and field-operations clients. Book a 15-min discovery call to walk through which of the five blockers your current build is stuck on: https://cal.com/mirgen-motomtech/quick-intro.
Mirgen Hoxha, CEO, Motomtech.
