Log inStart free
← Back to catalog

LLMOps and production

Cost, cache, monitoring, evals and safety of AI systems in production.

Updated: 26 July 2026Editor: Skilvy editorial

22 lessons · 9 h read · 2 free · 4 AI-graded projects

About this course

A course on running AI systems in production: why green tests don't save you from silent degradation, and what to do when the model, prompt, or data shift under you. It covers versioning, rollout, offline and online evaluation, observability, incidents, and spend tracking.

Five modules move from what breaks in production through versions and environments, quality measurement, observability and incidents, to money, security, and roadmap. In "Versions and environments" you write a one-to-two-page change policy for an on-call engineer. In "Measuring quality" you build an evaluation system with a categorized dataset composition. In "Observability and incidents" you assemble an operational package: logging, tracing, alerts, and a postmortem.

For people whose AI feature is already live or nearly ready to ship and needs to survive its first on-call rotation. Requires at least basic software operations experience and familiarity with calling LLM APIs. This isn't an intro prompting course — it's about what happens after deploy.

What you'll learn

  • Map changes across four moving parts: code, prompt, model, data
  • Assess an AI system's maturity level, from script to on-call-ready
  • Version prompts, configs, and evaluation sets
  • Run rollouts through canaries, shadow traffic, and rollback
  • Build an offline evaluation set and read online signals from live traffic
  • Set up logging and tracing that can reproduce a failure
  • Configure alerts and get through the first hour of an incident without panic
  • Track spend by category and cut cost without losing quality

What you'll end up with

  • A one-to-two-page change policy for the AI system
  • A quality measurement system with a categorized evaluation dataset
  • An operational package: logging schema, tracing, alerts, incident postmortem
  • A quarter-long operations plan (paid tiers include reviewer feedback and a certificate)

Not for you if there's no AI feature yet and you're still picking a model — this course starts where the system is already answering users.

Course program

Module 1 · What breaks in production
1Operating an AI system: what is different hereFree2Green tests and silent degradationFree3Map of changes: four moving parts4Maturity levels: from a script to an on-call rota
Module 2 · Versions and environmentsproject
5Versioning prompts, configs and eval sets6Environments: where to try things without real traffic7Rollout: canary, shadow traffic, rollback8Reproducibility: how to repeat yesterday's failure
Module 3 · Measuring qualityproject
9Offline evaluation on a set and its limits10Online signals: what live traffic tells you11Human review: how much, by whom and how often12Quality and product: where the metric meets money
Module 4 · Observability and incidentsproject
13What to log and what must never be logged14Tracing: the path of a single request15Alerts: what to set them on and how not to drown in noise16On-call and the first hour of an incident17Incident review and what becomes a check
Module 5 · Money, security, evolutionproject
18Cost attribution: who spends and on what19Optimising cost without losing quality20Limits, quotas and protection from ruinous traffic21Security in operation: access, data, injections22Evolution: how to decide what to do next
$89
one-time purchase · lifetime access
Or included in Ultra
Start freeGet a subscription
Practical projects
0 / 4 passed
Certificate progress →

What's included

  • Lifetime access
  • Tasks and quizzes
  • Certificate — on a paid plan
  • Updates forever