A retrieval-augmented generation (RAG) golden dataset is the evidence contract behind an evaluation run. It says which question was asked, what answer is acceptable, which chunks can support that answer, and exactly which corpus revision made those judgments meaningful. Without that contract, a result can look repeatable while comparing a new retriever against data that no longer exists.
I think of the set as a small, versioned product. It is not a bag of prompts. It carries human decisions, source lineage, and rules for when those decisions must be revisited. That makes it useful when you change chunking, re-index a handbook, replace a source revision, or discover that real questions do not resemble the questions your team wrote in a workshop.
The RAG architecture belongs around this work, but it is separate from it. My RAG with Semantic Kernel guide covers an implementation path. This article stays focused on the data that lets you judge any implementation fairly.
Build a RAG golden dataset as a versioned set of judgments
The 2024 Evaluation of Retrieval-Augmented Generation survey describes RAG evaluation as difficult partly because a RAG system combines retrieval and generation while depending on dynamic knowledge sources. That dynamic part is the operational detail that deserves attention. A judgment about a chunk is only valid relative to the corpus and chunking scheme that produced it.
Start with one case per meaningful user need. A case should contain the user query, an expected answer, the IDs of chunks that are relevant evidence, provenance for those chunks, and an immutable corpus revision. The expected answer is not a script that the system must repeat word for word. It is the factual outcome a reviewer expects when the cited evidence is available.
That distinction matters. An answer can use different wording and still be correct. Conversely, a fluent answer can include an unsupported detail even when it resembles a reference answer. The golden set therefore records both the answer expectation and the retrieval evidence that makes the expectation defensible.
Here is a provider-neutral C# representation. It uses only the .NET base class library and can be pasted into a new .NET 8 console project as Program.cs.
using System;
using System.Collections.Immutable;
using System.Text.Json;
namespace GoldenDatasetExamples;
public sealed record ChunkProvenance(
string ChunkId,
string DocumentId,
string SourceUri,
string SourceRevision,
string ContentHash);
public enum ReviewStatus
{
Pending,
Approved,
Rejected
}
public sealed record ReviewDecision(
ReviewStatus Status,
string ReviewedBy,
DateTimeOffset ReviewedAt,
string Reason);
public sealed record GoldenCase(
string Id,
string Query,
string ExpectedAnswer,
ImmutableArray<string> RelevantChunkIds,
ImmutableArray<ChunkProvenance> Provenance,
string CorpusRevision,
ReviewDecision Review);
public static class Program
{
public static void Main()
{
var caseToSerialize = new GoldenCase(
Id: "refund-policy-001",
Query: "When can a customer request a refund?",
ExpectedAnswer: "A customer can request a refund within 30 days of purchase.",
RelevantChunkIds: ImmutableArray.Create("refund-policy-v4:12"),
Provenance: ImmutableArray.Create(
new ChunkProvenance(
"refund-policy-v4:12",
"refund-policy-v4",
"https://docs.example.test/refunds",
"v4",
"4D1C")),
CorpusRevision: "corpus-2026-09-01",
Review: new ReviewDecision(
ReviewStatus.Approved,
"Avery",
new DateTimeOffset(2026, 9, 1, 0, 0, 0, TimeSpan.Zero),
"The source revision supports the expected answer."));
Console.WriteLine(JsonSerializer.Serialize(
caseToSerialize,
new JsonSerializerOptions { WriteIndented = true }));
}
}
The important fields are the ones teams are tempted to omit. A document title is useful for a reviewer, but a chunk ID tells the retrieval layer what was judged. A source URL helps somebody inspect the document, while a source revision and hash distinguish one policy revision from another. CorpusRevision joins those individual facts into a testable snapshot of the whole retrieval corpus.
If your corpus uses multiple source types, keep the provenance shape consistent rather than putting file paths in one case, URLs in another, and informal notes elsewhere. An evaluator should be able to answer "where did this expected answer come from?" without guessing which metadata convention applied that month.
For a practical retrieval implementation, the document Q&A example with Semantic Kernel is useful context. The point here is more fundamental: persist the evidence identity before deciding how to score a retrieved result.
Validate each RAG golden dataset case before evaluating a system
JSON makes a golden set portable, not trustworthy. Validate it when a case is authored and again when it is loaded for an evaluation run. The validation should reject ambiguous cases early: blank queries, blank answer expectations, duplicate relevant chunk IDs, missing provenance, and a provenance entry that does not correspond to an identified chunk.
This small validator does not decide whether an answer is factually good. That is a human judgment. It does ensure that a case has enough structure to be reviewed and replayed.
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
namespace GoldenDatasetExamples;
public sealed record ChunkProvenance(
string ChunkId,
string DocumentId,
string SourceUri,
string SourceRevision,
string ContentHash);
public enum ReviewStatus
{
Pending,
Approved,
Rejected
}
public sealed record ReviewDecision(
ReviewStatus Status,
string ReviewedBy,
DateTimeOffset ReviewedAt,
string Reason);
public sealed record GoldenCase(
string Id,
string Query,
string ExpectedAnswer,
ImmutableArray<string> RelevantChunkIds,
ImmutableArray<ChunkProvenance> Provenance,
string CorpusRevision,
ReviewDecision Review);
public static class GoldenCaseValidator
{
public static ImmutableArray<string> Validate(GoldenCase testCase)
{
var errors = ImmutableArray.CreateBuilder<string>();
if (string.IsNullOrWhiteSpace(testCase.Query))
{
errors.Add("Query is required.");
}
if (string.IsNullOrWhiteSpace(testCase.ExpectedAnswer))
{
errors.Add("Expected answer is required.");
}
if (string.IsNullOrWhiteSpace(testCase.CorpusRevision))
{
errors.Add("Corpus revision is required.");
}
if (testCase.RelevantChunkIds.IsDefaultOrEmpty)
{
errors.Add("At least one relevant chunk is required.");
}
if (testCase.RelevantChunkIds.Distinct(StringComparer.Ordinal).Count()
!= testCase.RelevantChunkIds.Length)
{
errors.Add("Relevant chunk IDs must be unique.");
}
if (testCase.RelevantChunkIds.Any(
id => testCase.Provenance.Count(provenance =>
string.Equals(provenance.ChunkId, id, StringComparison.Ordinal)) != 1))
{
errors.Add("Every relevant chunk must have exactly one provenance entry.");
}
if (testCase.Review.Status == ReviewStatus.Approved
&& (string.IsNullOrWhiteSpace(testCase.Review.ReviewedBy)
|| string.IsNullOrWhiteSpace(testCase.Review.Reason)))
{
errors.Add("Approved cases require reviewer and decision metadata.");
}
return errors.ToImmutable();
}
}
public static class Program
{
public static void Main()
{
var testCase = new GoldenCase(
"refund-policy-001",
"When can a customer request a refund?",
"A customer can request a refund within 30 days of purchase.",
ImmutableArray.Create("refund-policy-v4:12"),
ImmutableArray.Create(
new ChunkProvenance(
"refund-policy-v4:12",
"refund-policy-v4",
"https://docs.example.test/refunds",
"v4",
"4D1C")),
"corpus-2026-09-01",
new ReviewDecision(
ReviewStatus.Approved,
"Avery",
new DateTimeOffset(2026, 9, 1, 0, 0, 0, TimeSpan.Zero),
"The source revision supports the expected answer."));
Console.WriteLine(GoldenCaseValidator.Validate(testCase).Length);
}
}
This is deliberately strict. A case with no relevant chunks may still be a valuable abstention or no-answer scenario, but it needs an explicit case type rather than an accidental hole in the schema. Make that choice visible in the data model. Otherwise, a missing judgment can be mistaken for a test that expects nothing to be retrieved.
The same discipline applies when chunks are regenerated. If a new splitter changes refund-policy-v4:12 into three chunks, do not silently map the old ID to the first new chunk. Review the new evidence against the query, assign a new corpus revision, and preserve the historical case if you need to compare revisions.
Generate synthetic cases as candidates, not truth
Synthetic question and answer generation can quickly create a useful starting pool. It is especially helpful when a new corpus has little query history. Give a generator a bounded source excerpt, require it to return the source chunk ID, and ask it for a question that a real reader might ask. Then put the output in a review queue.
The boundary is essential. A synthetic RAG golden dataset entry is a hypothesis about a good test case, not ground truth: the Ragas paper evaluates generated test data against manually annotated ground truth. Require a reviewer to confirm the query, expected answer, source revision, and relevant chunks before approving a candidate.
The 2025 RAG evaluation survey catalogs RAG-specific datasets and evaluation frameworks for changing knowledge sources. The separate survey of LLM-based RAG evaluation reviews automated approaches alongside human judgment. Treat automation as case creation support, then record the human judgment that establishes which facts and chunks an approved case represents.
I use synthetic cases in three lanes:
- Seed a review backlog with candidates tied to explicit source chunks.
- Broaden phrasing around a human-authored scenario without replacing the human scenario.
- Explore newly added documents while clearly marking every case as uncalibrated.
I do not promote a generated case because it has clean JSON. A reviewer must confirm that the query is plausible, the expected answer says no more than the source supports, and the listed chunks are actually sufficient. If any part fails, edit or discard it. Keeping rejected candidates can be useful for improving a generation prompt, but they must never enter the approved set by accident.
That separation keeps generated cases from silently becoming the evidence standard. Add human-authored cases for the language, incomplete questions, and abstention scenarios that matter in your domain, then retain the review decision with each approved case.
Calibrate a human sentinel set
A full human-reviewed suite can be expensive. A small RAG golden dataset of human sentinel cases is a practical complement: a deliberately maintained group of cases used to compare synthetic and automated evaluation with informed review. The LLM-based RAG evaluation survey distinguishes automated approaches from human judgment, which is why this article retains an explicit review decision rather than inferring approval from generated data.
Choose sentinels for their decision value, not because they are the most convenient examples. Include common queries, ambiguous wording, high-consequence policy questions, queries that require joining two sources, and questions whose correct outcome is uncertainty. The set should also include cases that tend to break when a document is revised or a chunk boundary moves.
For a sentinel record, have two reviewers independently inspect the query, expected answer, relevant chunks, and corpus revision, then record the resolution and its reason. This is a review policy that preserves the human-judgment boundary discussed in the LLM-based RAG evaluation survey. The resulting RAG golden dataset record makes the decision inspectable when the same case is reconsidered later.
using System;
using System.Collections.Immutable;
using System.Linq;
namespace GoldenDatasetExamples;
public sealed record SentinelReview(
string CaseId,
string Reviewer,
ImmutableArray<string> RelevantChunkIds,
bool ExpectedAnswerSupported);
public sealed record CalibrationResult(
string CaseId,
bool IsCalibrated,
ImmutableArray<string> Differences);
public static class SentinelCalibrator
{
public static CalibrationResult Compare(
SentinelReview first,
SentinelReview second)
{
var differences = ImmutableArray.CreateBuilder<string>();
if (!string.Equals(first.CaseId, second.CaseId, StringComparison.Ordinal))
{
differences.Add("Reviews refer to different cases.");
}
if (first.ExpectedAnswerSupported != second.ExpectedAnswerSupported)
{
differences.Add("Reviewers disagree on answer support.");
}
if (!first.RelevantChunkIds.Order().SequenceEqual(second.RelevantChunkIds.Order()))
{
differences.Add("Reviewers selected different relevant chunks.");
}
return new CalibrationResult(
first.CaseId,
differences.Count == 0,
differences.ToImmutable());
}
}
public static class Program
{
public static void Main()
{
var first = new SentinelReview(
"refund-policy-001",
"Avery",
ImmutableArray.Create("refund-policy-v4:12"),
true);
var second = first with { Reviewer = "Jordan" };
Console.WriteLine(SentinelCalibrator.Compare(first, second).IsCalibrated);
}
}
The code checks agreement; it cannot create agreement. When reviewers disagree, re-open the source material, clarify the expected answer or query scope, and record the resolution instead of averaging their chunk lists.
This approach also helps keep evaluation data independent from implementation enthusiasm. A team can change its retriever, prompt, or search approach, but it should not casually edit the sentinel's expected answers to make the change look successful. For background on how the retrieval corpus gets its units of evidence, see the existing chunking strategies for RAG article.
Refresh when the evidence changes
A RAG golden dataset needs a refresh policy because its claims are tied to a corpus revision. The 2024 RAG evaluation survey identifies dynamic knowledge sources as part of the evaluation problem, so refresh is a deliberate review triggered by a change that could invalidate a judgment, not a scheduled rewrite of the whole set.
Use a changed source revision, deleted or replaced chunks, a new chunking strategy, an embedding migration that requires re-indexing, or a changed retrieval filter as review triggers because each can change the evidence a case identifies. When telemetry exposes a recurring real-query pattern that is absent from the approved set, use the OpenTelemetry observability guide to make that observation queryable before proposing a new case. A refresh may confirm that a case remains valid, revise its relevant chunk IDs, or retire it because the business rule it tested no longer exists.
Make the RAG golden dataset review scope proportional to the evidence change. Review each case whose answer or evidence relies on a revised policy; review more broadly when a chunking configuration changes the identifiers and local context across the corpus. Record the trigger alongside the decision so a later evaluator can distinguish routine maintenance from a new judgment about correctness.
Keep the decision explicit. The following example compares the case revision to the current corpus revision and returns a refresh request with a reason. A production application can store the request wherever it manages evaluation data; it does not need a vendor SDK to represent the decision.
using System;
using System.Collections.Immutable;
namespace GoldenDatasetExamples;
public sealed record GoldenCase(string Id, string CorpusRevision);
public sealed record RefreshRequest(
string CaseId,
string PreviousCorpusRevision,
string CurrentCorpusRevision,
ImmutableArray<string> Reasons);
public static class GoldenSetRefresh
{
public static RefreshRequest? CreateRequest(
GoldenCase testCase,
string currentCorpusRevision,
bool sourceChanged,
bool newQueryPatternObserved)
{
var reasons = ImmutableArray.CreateBuilder<string>();
if (!string.Equals(
testCase.CorpusRevision,
currentCorpusRevision,
StringComparison.Ordinal))
{
reasons.Add("The corpus revision changed.");
}
if (sourceChanged)
{
reasons.Add("A source behind the case changed.");
}
if (newQueryPatternObserved)
{
reasons.Add("A reviewed query pattern is not represented.");
}
return reasons.Count == 0
? null
: new RefreshRequest(
testCase.Id,
testCase.CorpusRevision,
currentCorpusRevision,
reasons.ToImmutable());
}
}
public static class Program
{
public static void Main()
{
var request = GoldenSetRefresh.CreateRequest(
new GoldenCase("refund-policy-001", "corpus-2026-09-01"),
"corpus-2026-09-08",
sourceChanged: true,
newQueryPatternObserved: false);
Console.WriteLine(request is not null);
}
}
Treat the refresh as a review gate, not an automatic rewrite. A changed corpus revision tells you the old evidence might be stale. It does not tell you which replacement chunk is relevant, whether the expected answer changed, or whether the case should now be an abstention. Those decisions still need source inspection and, for sentinel cases, human calibration.
The semantic search engine walkthrough can help connect this data contract to a working search surface. Keep the boundary intact, though: a search implementation retrieves a case's evidence; the golden set records why that evidence is expected.
Keep the set small enough to believe
RAG golden dataset coverage matters, but a large unreviewed collection is less valuable than a smaller set with clear provenance. Start with representative, human-understood cases. Add reviewed synthetic candidates where they expose gaps. Maintain sentinels across corpus revisions. Retire cases that no longer represent a valid user need.
That produces a RAG golden dataset that changes for visible reasons. It also makes evaluation conversations much sharper. Instead of debating whether a number moved, a team can inspect the query, expected answer, chunks, source revision, and reviewer decision behind the result.
Frequently asked questions
What belongs in a RAG golden dataset?
Include the query, an expected answer, relevant chunk IDs, chunk provenance, a corpus revision, and review metadata; this evidence structure addresses the dynamic-source evaluation context identified by the 2024 RAG evaluation survey. Add an explicit case type when a question is expected to produce an abstention or no answer.
Is a generated question automatically valid evaluation data?
No. A generated question is a candidate until a reviewer verifies that it is plausible, that its expected answer is supported by the cited source revision, and that its relevant chunks are sufficient; Ragas evaluates generated test data against manually annotated ground truth.
Why store both chunk IDs and provenance?
Chunk IDs identify the retrieved evidence in a particular index. Provenance explains the source document, revision, and content identity behind those IDs. Keep both so a judgment can be replayed and reviewed after the corpus changes, an evaluation concern described in the 2024 RAG evaluation survey.
How large should a human sentinel set be?
Use enough cases to represent common, ambiguous, high-consequence, multi-source, and abstention scenarios in your domain. Their value comes from recorded human calibration and maintenance, rather than a universal target count; the LLM-based RAG evaluation survey reviews human judgment alongside automated approaches.
When should a RAG golden dataset be refreshed?
Refresh it when source or corpus revisions change, chunks are regenerated, retrieval assumptions change, or reviewed real questions reveal an unrepresented pattern. Review the affected evidence rather than automatically accepting replacements because RAG evaluation depends on dynamic knowledge sources, as the 2024 survey explains.
Can a RAG golden dataset contain more than one acceptable answer?
Yes. Store an answer expectation that describes the supported outcome, or model multiple accepted answer statements. Keep every accepted outcome tied to relevant evidence in the recorded corpus revision so the judgment remains reviewable as knowledge sources change, consistent with the 2025 RAG evaluation survey.