Every custom software vendor’s deck now has an “AI-augmented engineering” slide. Most of them mean different things by it, and a handful mean nothing at all. The six questions below are how you tell which is which before you sign the SOW.
These aren’t trick questions. They’re the things a vendor who has actually institutionalized AI tooling in their SDLC can answer in two sentences each. A vendor who has put “AI-accelerated” on the deck because it tested well in sales conversations will hedge, generalize, or pivot to a case study about something else.
The market has run ahead of the buyer’s ability to evaluate. In 2024, AI-augmented dev was a differentiator on a handful of decks. As of the 2025 Stack Overflow Developer Survey, 84% of developers were using or planning to use AI tools, and DORA’s 2025 State of AI-assisted Software Development report put adoption at 90% with more than 80% reporting productivity gains. AI tooling is now table stakes at the individual developer level. The differentiator has moved to the team and process level.
That means the vendor’s claim of “we use AI” is approximately as meaningful as “we use Git.” The real question is whether the workflow around the tooling is institutionalized, reviewed, and measurable. The questions below get at that.
The answer you want: every PR. No exceptions for “small fixes,” no LGTM bot stamps on AI-generated code, no late-Friday merges without a second pair of eyes. A senior engineer reads every diff before it lands on main.
The reason this matters more than it used to: AI-generated code is confidently wrong in a way human-written code usually isn’t. A junior engineer writing a buggy auth check tends to write code that looks buggy. Claude or GitHub Copilot writing a buggy auth check writes code that looks correct, because the surface-level patterns match production code the model has seen. The fix is human review at the diff level, every time.
A vendor who says “we review most PRs” or “small refactors can ship without review” is describing a process where AI hallucinations leak into production. A vendor who says “every PR gets a senior review, and we measure review latency to make sure it’s not the bottleneck” has done the engineering work to make per-PR review compatible with velocity.
Follow-up to ask: what is your median PR review latency? A team that reviews every PR but takes three days to get to it is bottlenecked. A team that reviews every PR within four working hours has built the workflow around it.
The answer you want: a specific list with specific use cases per tool. Claude or Cursor for IDE-level coding and refactoring. GitHub Copilot for autocomplete and inline suggestions. Aider or Continue.dev for repo-scale agentic edits if those are in the mix. Maybe an internal eval harness on top.
What you’re listening for is whether the answer is a list of categories or a list of judgments. “We use Claude for refactoring legacy modules because the long-context window holds the whole file plus tests, and Copilot for inline suggestions because the latency is lower for typing-speed completions” is a judgment-bearing answer. “We use AI tools throughout our workflow” is a hand-wave.
You’re also listening for whether the engineers using the tools have the discretion to choose. A vendor that mandates one specific tool across all engineers regardless of task is buying licenses rather than running a workflow. A vendor whose engineers route the right task to the right tool, and can defend the routing, has internalized it.
Vendors who can’t name their tools, or who name them but can’t explain when they reach for one vs. another, are probably using AI tools sometimes and shipping the same way they shipped in 2023. That’s not a fatal flaw, but it isn’t AI-accelerated either.
The answer you want: 80% or higher on business logic, with coverage measured per changed line rather than per repo. A team that runs 85% coverage repo-wide but ships AI-generated features with 30% line coverage on the new code has a measurement that hides the risk.
The reason this is the right number: AI-generated tests can be shallow. Claude will write a test that calls the function and asserts it doesn’t throw. That counts toward coverage but doesn’t verify behavior. The discipline that catches this is reviewing AI-generated tests with the same skepticism as AI-generated code, and measuring coverage in a way that includes branch and assertion depth, not just lines executed.
A vendor who can show you their coverage dashboard, point at the threshold they enforce, and tell you how often they let PRs ship under threshold (the answer should be “rarely, with sign-off”) has done the work. A vendor who says “we have good test coverage” without a number is hand-waving.
The other thing to ask: do they measure mutation coverage on critical paths? Mutation testing catches the shallow-test problem by mutating the production code and verifying the tests fail. Not every team runs it. The ones that do on auth, billing, and core business logic are the ones who have thought about AI-generated test depth.
The answer you want: you do. Source code in your repo, your infrastructure accounts, your secrets, your CI/CD. The vendor’s role ends when the engagement ends. No retained-IP clause, no proprietary internal tooling that you can’t run without paying them, no infrastructure dependencies on their AWS account.
This question is more important in the AI-accelerated era because the failure mode has shifted. Just two years ago, vendor lock-in was about code complexity and undocumented patterns. Today, it can be about agent-driven build pipelines, prompts-as-code, and internal eval harnesses that the vendor treats as proprietary. If their AI-augmented workflow includes tooling you can’t take with you, the moment you offboard them, your team inherits a codebase they can’t iterate on at the velocity the vendor was running.
Ask explicitly: if we offboard you in six months, can our team continue shipping at a comparable cadence using their own AI tooling? The answer should be yes, with the caveat that your team will need to do their own onboarding. If the answer involves keeping the vendor on a retainer to maintain the workflow, the velocity gain belongs to them, not to you.
The answer you want: the same as for human-written code, with no exceptions. No “AI-generated, fast-tracked,” no separate queue, no relaxed standards because “the LLM probably got it right.”
The reason this matters: the moment an organization carves out a fast path for AI-generated code, the volume of AI-generated code rises to fill the path. Engineers learn that the easiest way to ship is to label a PR as AI-augmented, and the review queue degrades silently over a quarter. The teams that hold the line treat AI-generated code as a normal PR, with the same gates, the same coverage requirements, and the same review burden.
A subtler version of the same answer: the vendor distinguishes between AI-assisted code (the engineer drove, the AI helped) and AI-generated code (the AI drove, the engineer reviewed). The first gets normal review. The second gets the same review plus a check that the engineer can defend the architectural choices in the diff. If the engineer can’t explain why the function is structured the way it is, the PR goes back.
This is the question where vendors most often pivot to a case study instead of answering. If the answer drifts toward “we shipped feature X in record time,” redirect: I asked about the review process, not the speed.
The answer you want: a specific artifact. A regression suite that runs against AI-generated PRs and flags hallucinations. A monthly internal report on AI-tooling productivity broken down by phase (implementation, refactoring, documentation). A coverage trendline that shows what happened when they started routing more work through Claude in Q4.
This is the question that separates the vendors who have institutionalized AI workflow from the vendors who put it on a slide. Running internal evals on AI-generated code is the work that establishes whether the productivity claim is real. A team that runs no evals is claiming velocity based on developer self-report, which the DORA 2025 report explicitly cautions against: self-reported AI productivity gains are real but get conflated with task-level gains that don’t compound at the team level.
If a vendor can show you the eval, even at a high level, the claim has a foundation. If they can’t, the 50% faster number on their deck is the same kind of number every other vendor’s deck has: it tested well in sales conversations and was never validated.
A reasonable answer: “We measure phase-level velocity quarterly and share an internal report. I can’t share the artifact, but I can walk you through the methodology.” That’s defensible. The non-answer is “trust us, our engineers ship faster.”
You’re not grading on a 6-out-of-6 rubric. A vendor that can answer four of these clearly and concretely, and acknowledges the gaps on the other two, is being honest about where they are. A vendor that can answer all six in detail has either done the work or has a very polished sales process; either way, the follow-up is to ask for a paid pilot before the full engagement so the answers get tested against delivery.
A vendor that can’t answer any of these without pivoting to case studies has put “AI-accelerated” on the deck because their competitors did. Their pricing might still be competitive, and their engineering team might still ship reasonable software at traditional velocity. What you won’t get is the timeline compression the claim implies.
The honest version of the velocity math is documented in the cornerstone post on AI-accelerated custom software development: traditional offshore quotes 16 to 24 weeks for a typical SaaS MVP, AI-accelerated teams compress that to 8 to 12. That delta is real when the workflow is real. It evaporates when the workflow is just licenses for Cursor distributed to engineers who use them inconsistently.
Q: How do you evaluate an AI-accelerated software development vendor?
Six concrete questions separate vendors who have institutionalized AI tooling from vendors who put it on a slide: senior-engineer review rate per PR, named AI tools in the stack with use cases per tool, test coverage threshold at delivery, codebase ownership after engagement, review process for AI-generated code, and a recent internal evaluation the team ran on AI productivity. Vendors who have institutionalized the workflow answer each in two sentences. Vendors who haven’t pivot to case studies or generalize.
Q: What is a realistic test coverage threshold for AI-generated code?
Aim for 80% or higher on business logic, measured per changed line rather than repo-wide. The reason: AI-generated tests can be shallow (calling the function without asserting on behavior), so coverage on the new code matters more than aggregate coverage. Teams that run mutation testing on critical paths catch the shallow-test problem by verifying the tests actually fail when production logic mutates.
Q: Should AI-generated code get a different review process than human-written code?
No. The same review standards, the same coverage gates, the same reviewer burden. The moment an organization carves out a fast path for AI-generated PRs, the volume rises to fill the path and review quality degrades. Teams that hold the line treat AI-generated code as a normal PR, with the additional check that the submitting engineer can defend the architectural choices in the diff.
Q: What does AI-accelerated vendor lock-in look like in 2026?
It looks like an AI-augmented workflow you can’t take with you. Proprietary internal tooling, prompts-as-code that the vendor treats as IP, agent-driven build pipelines that depend on the vendor’s accounts, eval harnesses that require their licenses. Ask the offboarding question explicitly: if we end the engagement in six months, can our team continue shipping at a comparable cadence on their own AI tooling? The answer should be yes.
Q: How can I verify a vendor’s 50% faster claim before signing the SOW?
Ask for a recent internal evaluation or productivity metric, broken down by phase (implementation, refactoring, documentation). If they can’t share the artifact, ask them to walk through the methodology. If neither is available, the claim is self-reported velocity, which conflates task-level gains with team-level outcomes. A paid two-week pilot, scoped to a single feature, is the most reliable way to test the claim against delivery.
Motomtech runs the six-question checklist on our own SDLC every quarter and reports against it internally. If you’re evaluating vendors and want a second pair of eyes on the proposals you’re reading, book a 15-min discovery call and we’ll walk through the answers your shortlist is giving you. From Prompt to Production means the workflow has to be real before the timeline gets shorter.
Mirgen Hoxha, CEO, Motomtech.
