AI Engineering: Deploying AI Systems

What you will learn

How to take an AI-powered feature from a notebook to production: environment configuration, rate limits, logging, fallbacks, monitoring, and human-in-the-loop.

Environment configuration

# .env (NEVER commit this — .gitignore it)
OPENAI_API_KEY=sk-...
OPENAI_BASE_URL=https://api.openai.com/v1
AI_MODEL=gpt-4o
AI_TEMPERATURE=0.3
AI_MAX_TOKENS=2048
RATE_LIMIT_RPM=100
LOG_LEVEL=INFO

Load at startup with environment variables or a .env file using python-dotenv or similar.

Rate limits

Every API has rate limits (requests per minute, tokens per minute). Exceeding them returns HTTP 429:

import time
import random

def call_with_retry(client, messages, max_retries=3):
    for attempt in range(max_retries):
        try:
            resp = client.chat.completions.create(
                model="deepseek-v4-flash", messages=messages
            )
            return resp.choices[0].message.content
        except Exception as e:
            if attempt == max_retries - 1:
                raise
            time.sleep(2 ** attempt + random.uniform(0, 1))
    return None

Logging everything

Log every request and response (but NOT full content if it contains PII!):

{
  "timestamp": "2026-07-08T15:30:00Z",
  "request_id": "req_abc123",
  "model": "deepseek-v4-flash",
  "prompt_tokens": 450,
  "completion_tokens": 120,
  "latency_ms": 850,
  "status": "success"
}

Use structured logging (JSON) so you can query logs: "Show me all 500 errors in the last hour."

Fallbacks

No AI provider is 100% reliable. Plan for failure:

1. Primary model fails → fallback model (different provider) → cached response → graceful error to user
2. Cache common responses so you don't call the API for the same question twice
3. Circuit breaker: if the API fails 5 times in a row, stop trying for 60 seconds

Human-in-the-loop (HITL)

For high-stakes use cases, don't let the AI act autonomously:

• AI suggests, human approves — email drafts, content moderation, financial advice
• AI automates, human reviews — non-critical with periodic audit
• AI only — low-risk, well-tested, with clear fallbacks

Monitoring

Track these metrics in your dashboard:

Common mistakes

• No rate-limit handling — your app crashes on the first spike.
• No monitoring — you find out the API has been down for hours from a user email.
• No fallback model — if OpenAI is down, your app is down.
• Logging raw prompts and completions without redacting PII — legal risk.
• Hardcoding model names — use environment variables so you can swap models without deploying code.

Quick check below!