The architecture, evals, monitoring, and process disciplines that take an AI feature from "works in a demo" to "survives real users at scale."
The first version of an AI feature usually works in a demo. The version that actually serves real users for months without on-call pages — that's a different beast. The gap between them is the work of this tutorial.
This is the playbook for crossing it.
Goal: prove the LLM can do the task at all.
What to build:
What NOT to build yet:
What to decide at the end of this phase:
If quality is acceptable, advance.
Goal: plug the feature into the Django app and let internal users hit it.
What to build:
# settings.py
AI_FEATURE_X_ENABLED = config("AI_FEATURE_X_ENABLED", default=False, cast=bool)
AI_FEATURE_X_STAFF_ONLY = config("AI_FEATURE_X_STAFF_ONLY", default=True, cast=bool)
# views.py
@login_required
def feature_x(request):
if not settings.AI_FEATURE_X_ENABLED:
raise Http404()
if settings.AI_FEATURE_X_STAFF_ONLY and not request.user.is_staff:
raise Http404()
...
What to decide at the end of this phase:
"Log every call" is only useful if the log is queryable. Writing to a text log file lets you grep; writing a row per call to a table lets you answer questions like "what was p95 latency for prompt version v3 last Tuesday" with a single query. For most Django apps at this scale a plain model is enough:
# support/models.py
from django.conf import settings
from django.db import models
class AICall(models.Model):
class Status(models.TextChoices):
OK = "ok"
ERROR = "error"
TIMEOUT = "timeout"
REJECTED = "rejected" # failed output validation
RATE_LIMITED = "rate_limited"
feature = models.CharField(max_length=64, db_index=True)
prompt_version = models.CharField(max_length=64)
model = models.CharField(max_length=128)
user = models.ForeignKey(settings.AUTH_USER_MODEL, null=True,
on_delete=models.SET_NULL)
input_text = models.TextField()
output_text = models.TextField(blank=True)
input_tokens = models.PositiveIntegerField(default=0)
output_tokens = models.PositiveIntegerField(default=0)
latency_ms = models.PositiveIntegerField()
status = models.CharField(max_length=16, choices=Status.choices)
error_type = models.CharField(max_length=128, blank=True)
created_at = models.DateTimeField(auto_now_add=True, db_index=True)
class Meta:
indexes = [models.Index(fields=["feature", "created_at"])]
Two design choices matter here. First, log failures too, with a status and the exception class name — a table of only successful calls hides exactly the incidents you care about. Second, store prompt_version and model on every row from day one; retrofitting them later means your historical data cannot be compared with anything. The table grows fast, so decide on a retention period now — for example, blanking input_text and output_text after 30 days while keeping the metric columns — rather than after it becomes the largest table in the database and a store of personal data nobody meant to keep.
Goal: stop being able to ship prompt changes without knowing if quality regressed.
What to build:
# evals/test_set.json
[
{"input": "How do I install QuizCraft?", "expected_keyword": "venv"},
{"input": "Refund policy?", "expected_keyword": "30 days"},
...
]
# evals/run_eval.py
def evaluate():
test_set = load_test_set()
results = []
for example in test_set:
actual = ask(example["input"])
passed = example["expected_keyword"].lower() in actual.lower()
results.append({"input": example["input"], "passed": passed, "actual": actual})
pass_rate = sum(r["passed"] for r in results) / len(results)
print(f"Pass rate: {pass_rate:.1%}")
return pass_rate
What to decide at the end of this phase:
Goal: see what real users do that you didn't anticipate.
What to build:
What to expect:
What to decide at the end of this phase:
Goal: make it boring and reliable.
What to build:
# Versioned prompts
PROMPT_VERSION = "summarize_v3"
PROMPTS = {
"summarize_v1": "Summarize this:",
"summarize_v2": "Provide a 2-sentence summary:",
"summarize_v3": "Provide a concise 2-sentence summary preserving key facts:",
}
response = ask(PROMPTS[PROMPT_VERSION])
log_call(prompt_version=PROMPT_VERSION, ...)
When you change a prompt and quality drops, you can correlate the regression with the version change in your logs.
The settings-based flag from Phase 2 reads an environment variable, which means flipping it requires a restart or redeploy — too slow when the feature is producing harmful output right now. For GA, back the kill switch with the database and a short cache:
# support/killswitch.py
from django.core.cache import cache
from django.db import models
class KillSwitch(models.Model):
feature = models.CharField(max_length=64, unique=True)
active = models.BooleanField(default=False) # True = feature disabled
reason = models.TextField(blank=True)
updated_at = models.DateTimeField(auto_now=True)
def feature_enabled(name: str) -> bool:
key = f"ai:enabled:{name}"
enabled = cache.get(key)
if enabled is None:
enabled = not KillSwitch.objects.filter(feature=name, active=True).exists()
cache.set(key, enabled, timeout=30)
return enabled
Register the model in the admin so an on-call engineer can flip it from a browser, and make the cache TTL the documented worst-case time to take effect. Require a reason when activating it; months later, the reason field is the only record of why the feature was off for an afternoon. Test the switch before launch by actually flipping it in staging and confirming the UI degrades to a friendly message rather than an error.
By Phase 5, your AI code should look something like this:
┌─ View layer ──────────────────────────┐
│ - Authentication, rate limit, feature flag │
│ - Calls service layer, returns response │
└──────────────────────────────────────┘
↓
┌─ Service layer ────────────────────────┐
│ - Routes to right model (router) │
│ - Constructs prompt from versioned templates │
│ - Calls inference layer │
│ - Validates output (Pydantic) │
│ - Logs call with all metadata │
└──────────────────────────────────────┘
↓
┌─ Inference layer ──────────────────────┐
│ - Wraps Anthropic SDK │
│ - Handles retries, errors, timeouts │
│ - Manages prompt caching │
└──────────────────────────────────────┘
Three clean layers, each testable. Don't put client.messages.create() in your view.
The bottom layer is small but it is where resilience lives. It owns the SDK client, sets timeouts and retries once, and translates the SDK's many exception types into the one or two your service layer needs to reason about.
# support/ai/inference.py
import anthropic
_client = anthropic.Anthropic(timeout=30.0, max_retries=2)
class InferenceUnavailable(Exception):
"""The provider could not be reached or is failing; degrade gracefully."""
def complete(model: str, system: str, prompt: str, max_tokens: int):
try:
return _client.messages.create(
model=model,
max_tokens=max_tokens,
system=system,
messages=[{"role": "user", "content": prompt}],
)
except anthropic.APIConnectionError as exc: # includes timeouts
raise InferenceUnavailable("connection") from exc
except anthropic.RateLimitError as exc:
raise InferenceUnavailable("rate_limited") from exc
except anthropic.APIStatusError as exc:
if exc.status_code >= 500:
raise InferenceUnavailable(f"status_{exc.status_code}") from exc
raise # 4xx other than 429 is our bug
Notice what is not caught: a 400 means the request itself is malformed, and retrying or hiding it only delays the fix. Let it reach Sentry. The SDK already retries transient failures with backoff, so do not wrap this in a second retry loop — stacked retries multiply latency and load exactly when the provider is struggling.
The metrics that matter:
| Metric | Alert when |
|---|---|
| p50 / p95 latency | p95 > 10 seconds |
| Error rate | > 2% |
| Cost per user per day | > 2x baseline |
| Eval pass rate (weekly) | drops > 10 points |
| User feedback (thumbs down rate) | > 20% |
| Hallucination rate (sample-audited) | > 5% |
Not all of these are easy to measure automatically. The hallucination rate especially needs human spot-checks. Build a weekly review where someone reads 20 random conversations and rates them.
Before launch, write down:
Have answers before you launch, not during the incident.
Per-user rate limits stop one person from running up the bill; they do nothing when ten thousand legitimate users arrive at once. Add a global daily budget per feature. Keep token rates in settings, updated from your provider's pricing page when it changes, and count spend in integer micro-units so the cache can increment it atomically:
# support/ai/budget.py
import datetime
from django.conf import settings
from django.core.cache import cache
# settings.AI_TOKEN_RATES = {"model-id": {"input": ..., "output": ...}}
# rates expressed in micro-dollars per token, maintained by you
def record_spend(feature: str, model: str, input_tokens: int, output_tokens: int) -> int:
rates = settings.AI_TOKEN_RATES[model]
cost = input_tokens * rates["input"] + output_tokens * rates["output"]
key = f"ai:spend:{feature}:{datetime.date.today().isoformat()}"
cache.add(key, 0, timeout=60 * 60 * 26)
return cache.incr(key, int(cost))
def budget_exceeded(feature: str) -> bool:
key = f"ai:spend:{feature}:{datetime.date.today().isoformat()}"
spent = cache.get(key, 0)
return spent >= settings.AI_DAILY_BUDGET_MICRO[feature]
Check budget_exceeded() alongside the kill switch before each call and alert at, say, 70% of the budget so a human can decide whether the spike is a success to fund or an attack to block. The cache counter is an estimate for guarding, not accounting; reconcile it against your provider's usage reports and your AICall token sums. When the breaker trips, degrade the feature the same way as during a provider outage — the user-facing path should already exist and already be tested.
Any feature that puts user-controlled or third-party text (tickets, emails, web pages, uploaded documents) into a prompt must assume that text may contain instructions aimed at the model. There is no complete prompt-level fix, so the defence is architectural: give the model the least capability it needs. Do not let model output directly trigger privileged actions such as refunds, deletions or emails to arbitrary addresses without a deterministic check or a human confirmation. Render model output as escaped text — Django's template autoescaping does this unless you mark it safe, so never pipe model output through |safe or mark_safe(). Validate structured output against a schema and reject anything outside it. Log inputs that trip your filters so you can add them to the adversarial part of the eval set.
Common causes of failed AI features:
Each of these is preventable with the practices above.
AI features are famously easy to prototype and famously hard to productionize, and understanding this gap is the heart of the roadmap. A proof of concept that works on a few hand-picked inputs can feel almost done, but production demands reliability across messy real inputs, handling of failures and edge cases, cost control, monitoring, and safety — none of which the demo addressed. Many AI projects stall precisely because teams underestimate this gap, treating the impressive demo as nearly finished when most of the real engineering still lies ahead. Recognizing that the demo is the start, not the end, sets realistic expectations for what it takes to ship.
The first step beyond a demo is a pilot: putting the feature in front of a limited set of real users with real data, which immediately surfaces the gap between curated test cases and reality. Real inputs are messier, users do unexpected things, and quality problems that were invisible in testing appear. The pilot is where you learn what actually needs hardening, measure real quality and cost, and validate that the feature delivers value. Treating the pilot as a deliberate learning phase — instrumented, observed, and limited in blast radius — is what turns a promising prototype into knowledge about what production will really require.
Moving from pilot to production means addressing everything the demo ignored: handling errors and model failures gracefully, validating and constraining output, controlling cost, monitoring quality and spend, guarding against misuse and prompt injection, and ensuring the feature degrades sensibly when the model is unavailable. This hardening work is the bulk of the real engineering in an AI feature, and it is what makes the difference between something that impresses in a meeting and something that holds up under real traffic. Budgeting for this phase realistically — rather than assuming a working pilot is nearly shippable — is essential to actually reaching production.
AI features fail to reach or survive production for recognizable reasons: unreliable quality on real inputs, runaway or unjustified cost, lack of monitoring so problems go unnoticed, safety and misuse issues, and solving a problem that did not really need AI. Many of these are avoidable with the right roadmap — piloting to surface quality issues early, controlling cost deliberately, monitoring from the start, and honestly assessing whether AI is the right tool. Knowing the common failure modes in advance lets you steer around them, which is much of what separates AI projects that deliver lasting value from the many that produce an exciting demo and then quietly stall.
To make the phases concrete, here is how the roadmap plays out for a realistic feature: summarizing incoming support tickets and suggesting a category for the agent who picks them up. The numbers are illustrative, but the shape of the decisions is typical.
Week 1 (PoC). Twenty real tickets, anonymized, go through the comparison harness with three model tiers. The smallest model produces acceptable summaries for short tickets but misses the requested action in long, rambling ones; the middle tier handles both. Decision: continue, start with the middle tier, and note "long tickets" as a known risk.
Weeks 2–3 (integration). The summary appears in the admin ticket view for four support agents behind a staff-only flag. Every call is logged. Within the first week the agents report two problems: summaries for tickets in German are written in English, and tickets that quote a previous reply get the old issue summarized instead of the new one. Both become eval examples, and the prompt gains two explicit instructions.
Weeks 3–4 (evals). A 120-example set is built from real tickets: 70 typical, 25 edge (long, multilingual, quoted threads), 25 adversarial (tickets containing "ignore previous instructions" style text, abusive language, pasted card numbers). A judge prompt is calibrated against human grades on 40 outputs. The team sets the floor at 85% overall and 90% on the adversarial slice, because a summary that follows injected instructions is worse than no summary.
Weeks 4–6 (beta). The feature goes to 10% of agents via hashed rollout. The dashboard shows p95 latency well within target but one unexpected cost driver: a small number of tickets with huge log attachments pasted inline, each consuming many times the average input tokens. The fix is a deterministic truncation step before the prompt plus a note to the agent that the summary covers only the first part of the ticket. Thumbs-down reasons cluster on "incomplete" for multi-issue tickets; the output schema changes from one summary to a list of issues, as summarize_v4.
Week 6+ (GA). Full rollout with the database-backed kill switch, a daily budget, and the weekly regression cron. Two months later the provider releases a new model; the team runs the eval set against it, sees equal quality at lower latency, rolls it out to 10% for a week, compares the grouped metrics, and completes the switch. No incident, no guesswork — which is exactly what the roadmap is for.
When something looks wrong in production, the call log usually tells you where to look first:
| Symptom | Likely cause | First check |
|---|---|---|
| p95 latency jumps, p50 unchanged | A subset of very long inputs or outputs | Sort recent AICall rows by latency and compare their token counts |
| Error rate spikes across all features at once | Provider incident or network egress problem | Group errors by error_type; check the provider status page |
| Rejected (validation) rate climbs after a deploy | Prompt change broke the output format | Group by prompt_version; read a few rejected output_text values |
| Cost per user doubles with flat traffic | Longer inputs (new data source, pasted logs) or a retry loop | Compare average input tokens per call week over week; count calls per request |
| Thumbs-down rate rises slowly over weeks | Input drift: users now ask things the prompt was not designed for | Sample recent thumbs-down rows and add them to the eval set |
| One user dominates spend | Automation or abuse | Top users by token sum in the last 24 hours; check rate-limit hits |
The pattern across all of these is the same: the answer comes from grouping the call log by one more dimension. That is why the logging model carries feature, prompt version, model, status and tokens on every row.
The roadmap from PoC to production is not about writing better prompts. It's about:
Most demos can be built in a weekend. Most production AI features take 4–8 weeks. The difference is this checklist. Skip it and you'll either ship something flaky or burn money silently.
Take it seriously and you'll ship features that earn user trust and survive scale. Which is the whole point.