RSI TECH
Contact

Insights / Production decision

Is your AI prototype ready for production?

A technical buyer should be able to inspect evidence across eight system dimensions and choose one of four actions: build, repair, constrain or stop. A polished demonstration is not the decision record.

Decision method

Inspect the operating boundary, not the prototype screen.

Define the users, data, decision, downstream action and failure cost first. Evidence is useful only when it represents that boundary and can be reproduced after a prompt, model, source or integration changes.

Read every dimension before selecting an action. A strong model score cannot compensate for unauthorized data, an unowned incident path or an integration that repeats consequential work.

Eight dimensions

Require observable evidence for each production responsibility.

  1. 01

    Data and permissions

    Observable evidence
    A named inventory identifies every source, owner, lawful use, access rule, retention period and deletion path. Representative inputs can be reproduced without copying production secrets into prompts, logs or evaluation fixtures.
    Failure signal
    The team cannot say which source is authoritative, who may use it, how revoked access reaches derived data, or how a user deletion reaches caches and indexes.
    Decision implication
    Repair the data contract before expanding access. Stop the use case when required data has no lawful or technically enforceable path.

  2. 02

    Retrieval

    Observable evidence
    Versioned queries show whether required evidence is found, where it ranks, whether forbidden material stays out and whether citations resolve to the source version used for the answer.
    Failure signal
    A fluent answer can hide empty, stale, incomplete or unauthorized context, and operators cannot separate retrieval failure from generation failure.
    Decision implication
    Repair ingestion, filtering, ranking and source lifecycle independently. Constrain the release to direct search or a smaller source set if generated answers remain unsupported.

  3. 03

    Model behavior

    Observable evidence
    Representative tasks record supported answers, uncertainty, refusal, format adherence and behavior under missing or conflicting context. Provider and prompt versions are traceable for each evaluated release.
    Failure signal
    The system produces confident unsupported output, breaks downstream formats, follows untrusted instructions or changes materially when a model or prompt changes.
    Decision implication
    Repair task constraints and fallback behavior, or constrain the model to a draft or recommendation role with human confirmation before consequential action.

  4. 04

    Evaluation

    Observable evidence
    A versioned dataset represents common work, costly failures and known edge cases. Metrics, thresholds and review rules connect directly to a release decision and are rerun after material changes.
    Failure signal
    Quality is demonstrated through selected examples or one average score, with no regression signal for the decisions users actually make.
    Decision implication
    Do not scale from demonstration traffic. Repair the evaluation contract first, then build only when the release threshold reflects the operating risk.

  5. 05

    Reliability

    Observable evidence
    Timeouts, retries, rate limits, dependency failures and degraded modes are exercised. Alerts identify the affected boundary, and a tested recovery or rollback path returns the service to a known state.
    Failure signal
    Provider or data failure becomes an indefinite spinner, duplicate action, silent partial result or incident that no named operator can diagnose and recover.
    Decision implication
    Repair failure handling before wider use. Constrain traffic or disable consequential actions until retries, idempotency, fallback and recovery are observable.

  6. 06

    Integration

    Observable evidence
    Interfaces validate identity, permissions, product state and boundary input. Downstream actions are idempotent, partial completion is visible, and users receive a specific recovery path.
    Failure signal
    The prototype depends on manual copy and paste, bypasses real authorization, loses state between systems or can repeat a consequential action after retry.
    Decision implication
    Repair the product and system boundary rather than hiding it behind a larger prompt. Build when the workflow behaves correctly under real state and access rules.

  7. 07

    Cost

    Observable evidence
    Expected demand is translated into separate budgets for ingestion, retrieval, reranking, inference, storage and operator review. Load tests expose latency and unit cost at the intended boundary.
    Failure signal
    The team knows a demo request cost but cannot estimate peak latency, retry amplification, reindexing cost or the human work required to correct low-quality output.
    Decision implication
    Constrain the workflow, model, context or traffic when economics are uncertain. Stop when a realistic quality threshold cannot fit the available operating budget.

  8. 08

    Ownership

    Observable evidence
    Named people own source changes, evaluation data, release approval, incident response, spend and user recovery. Runbooks, source, tests and decision records are available to the receiving team.
    Failure signal
    A vendor, researcher and product team each assume another party owns quality, access, incidents or the authority to pause the system.
    Decision implication
    Repair the operating model before launch. Stop a consequential release when nobody has both the evidence and authority to make or reverse the decision.

Four action outcomes

Make the next decision explicit.

  1. A01

    Build

    The intended boundary has representative evidence, enforceable access, a release threshold, controlled failure, viable cost and named owners. Proceed with the smallest production slice that preserves those conditions.

    Production AI and LLM systems →

  2. A02

    Repair

    The use case remains valid, but one or more production controls are missing. Sequence data, retrieval, evaluation, integration or operating repairs before adding users or new model behavior.

    AI prototype to production →

  3. A03

    Constrain

    Evidence supports a narrower release. Reduce users, sources, actions, traffic or autonomy, then state the boundary in product behavior and monitoring rather than relying on policy text alone.

    RAG and retrieval systems →

  4. A04

    Stop

    Required data cannot be used safely, the decision cannot be evaluated, realistic economics do not work, or nobody can own the risk. Record the blocker and the evidence needed before reconsidering.

    AI/LLM production readiness audit →

Decision notes

Questions technical buyers ask

Does one failed dimension always stop a release?
No. The failure may support a repair or a narrower boundary. A stop is appropriate when the missing control invalidates the use case, such as unlawful data access or no owner for a consequential action.
Can the review use read-only production access?
Usually. Architecture, representative data, evaluation output, telemetry and configuration can often be inspected without write access. Any unverified boundary stays visible in the decision.
When should a readiness audit precede implementation?
Use the audit when blockers, evidence quality or repair order are uncertain enough to change the implementation scope. If the boundary and gaps are already explicit, start with the relevant production or hardening service.

About this guide

RSI Tech production engineering practice

RSI Tech is Rafal's independent applied-AI engineering studio. The guide reflects the system boundaries used across architecture, implementation, evaluation, operations and handover work.

About Rafal and RSI Tech →