AIP-C01 lesson on GenAI evaluation frameworks, RAG quality tests, agent evaluations, LLM-as-a-judge, quality gates, and release validation.
AWS wants evaluation systems that compare meaningful alternatives, not vague “looks better” judgments. If you cannot tell whether the model, prompt, retrieval layer, tool workflow, or release configuration changed the result, your evaluation design is too weak for AIP-C01.
The current AIP-C01 domain page frames this task as implementing evaluation systems for GenAI. The recurring exam patterns include:
That means the exam is not only asking “how would you test this once?” It is asking whether the team can repeatedly compare alternatives and decide which one should ship.
| Requirement | Strongest first fit | Why |
|---|---|---|
| Need to compare candidate models or prompts objectively | Side-by-side evaluation set with fixed metrics and rubrics | Prevents subjective “this feels better” decisions |
| Need to test a RAG change | Retrieval-aware evaluation with relevance and factual-grounding checks | FM output quality depends on retrieval quality too |
| Need to compare production safety of alternatives | Include harmful-output and policy-violation checks in the eval set | Fast answers that fail safety are not release-ready |
| Need to know whether users actually prefer the new behavior | User feedback loop, ratings, or structured human review | Technical metrics alone may miss real UX problems |
| Need low-risk rollout after a promising result | A/B or canary release with rollback path | The best lab result can still fail in production |
If an answer says “evaluate the model” but never says against what, it is incomplete.
AWS explicitly treats FM evaluation as a broader problem than traditional single-label ML metrics. Strong evaluation design often includes:
That is why a single “accuracy” number is often too shallow for AIP-C01.
Many AIP-C01 failures happen outside the FM itself:
If the app is RAG- or agent-based, the strongest evaluation system usually measures the whole workflow rather than only the model output in isolation.
The AWS domain page explicitly includes user-centered evaluation. That should make you think about:
This is important when two candidates both “work” technically but one is clearly better in practice.
Before promotion, the system should show:
That last point matters. A release is not really ready if the team cannot revert it cleanly.
When the question is how to prove the new version is better, ask: compared to what, by which metrics, with which dataset, under which release controls?