Study Databricks GENAI-ASSOC Retrieval Quality and Reranking: key concepts, common traps, and exam decision cues.
The exam expects you to evaluate retrieval as its own subsystem. Good answers separate “the right evidence was never eligible” from “good candidates were retrieved but ordered badly” and from “the model misused the right evidence.”
| Failure signal | Better first explanation |
|---|---|
| wrong documents appear | poor chunking, embeddings, or filters |
| right documents exist but rank too low | reranking or ranking logic issue |
| too many near-duplicates | overlap or chunk design may be wasteful |
| too many embeddings for the vector store limit | chunk size and overlap may need adjustment |
| Step | Role |
|---|---|
| initial retrieval | fetches candidate documents |
| reranking | improves ordering among the retrieved candidates |
| Symptom | Better first layer to inspect |
|---|---|
| the answering evidence never appears | source set, chunking, embeddings, or filters |
| good evidence appears but too low | reranking or ranking logic |
| duplicate evidence floods the result set | overlap and chunk design |
| vector index is too large or expensive | chunk size, overlap, and record count |
| Trap | Better rule |
|---|---|
| bigger top-k fixes everything | more candidates can hurt cost and focus |
| reranking solves missing-source problems | reranking can only improve the order of retrieved candidates |
| evaluation only after deployment | retrieval metrics belong in development too |
A RAG app retrieves the right policy chunk, but it often ranks behind near-duplicate chunks that are less specific. Which subsystem should you inspect first?
Correct answer: A. The right evidence is already in the candidate set, so the first suspect is ranking order rather than source absence.
Retrieval-quality questions usually hinge on where the failure occurs. If the right candidates were found but ordered badly, think reranking. If you created too many embeddings, revisit chunk size and overlap. If the right documents never entered the candidate set, reranking cannot save you. The exam usually rewards fixing the earliest broken layer.