Production LLM Copilot Reference Architecture
How we engineer enterprise copilots that survive security review, evaluation gates, and operations handover.
Enterprise copilots fail in predictable ways: retrieval without authorization, prompts without versioning, features without evaluation, and launches without rollback. This brief documents the architecture Zestlan uses when AI must reach production — not stay in a pilot sandbox.
It is written for CTOs, platform leads, and security architects evaluating whether their organization can operate an LLM feature after vendors leave.
Architecture layers
A production copilot is not a chat UI on top of an API call. It is a system of ingestion, retrieval, orchestration, policy enforcement, evaluation, and monitoring — each with explicit ownership.
- Ingestion: document pipelines with metadata, refresh schedules, and source-of-truth tagging
- Chunking & embedding: strategy matched to document type — policies, tickets, manuals behave differently
- Vector store with ACL: document-level access enforced at query time, not filtered after retrieval
- Orchestration: tool routing, context assembly, and response formatting with policy checks
- Evaluation harness: regression suites run before promotion — accuracy, safety, latency, cost
- Observability: traces, feedback, drift signals, and incident playbooks
Retrieval with authorization
The most common enterprise copilot failure is retrieving content the user should not see. Filtering results in application code after vector search is insufficient — access rules must be part of the retrieval query itself.
We model documents with tenant, department, and sensitivity attributes indexed alongside embeddings. The retrieval layer applies the same RBAC rules as the source systems wherever possible.
Evaluation before release
Every copilot feature ships with a versioned evaluation dataset representing real user questions — including edge cases, policy-boundaries, and known failure modes from pilot usage.
Promotion criteria include minimum accuracy on the eval set, maximum latency at P95, zero critical safety failures, and documented rollback steps.
- Golden Q&A sets maintained by domain owners — not only ML engineers
- Automated regression on every prompt or model change
- Human review queue for low-confidence responses in high-risk domains
- Cost per successful answer tracked — not just token volume
Operations handover
Operations teams need runbooks: how to disable a feature flag, roll back a prompt version, re-index a corrupted source, and interpret monitoring dashboards. If the vendor is the only person who can operate the copilot, it is not production-ready.
Production checklist
- Document-level access control enforced at retrieval
- Evaluation dataset approved by domain owners before launch
- Prompt and model versions pinned with one-click rollback
- PII handling policy documented and tested
- Human escalation path for high-risk or low-confidence responses
- Production tracing and user feedback capture enabled
- Cost and latency budgets defined with alerts
- Runbook reviewed by client operations team before go-live
Summary
Enterprise copilots are platform features — not demos. The architecture above is how we keep them defensible under security review and useful after launch.
If you are planning a copilot program, start with evaluation criteria and access control design before choosing a model or UI framework.
Planning an enterprise copilot?
Zestlan engineers copilots, RAG systems, and AI agents with evaluation harnesses and production observability.