RAG Evaluation Metrics in .NET: Measure Retrieval and Grounding

RAG evaluation metrics in .NET are most useful when I evaluate retrieval-augmented generation as separate retrieval and answer-quality questions. Did the system retrieve useful evidence? Did the answer stay supported by that evidence? Did it actually address the question? A fluent response can fail any of those checks, so one overall score does not tell me what to fix.

This article covers an offline evaluation run with deterministic retrieval measurements and distinct answer-grounding measurements, which the Evaluation of Retrieval-Augmented Generation survey treats as separate RAG evaluation components. It does not prescribe how to construct evaluation data, instrument production traces, test security, or adopt an evaluation package. Those are separate concerns. For a framework-specific walkthrough of the pipeline being measured, see RAG with Semantic Kernel in C#.

RAG evaluation metrics in .NET should separate failure layers

Retrieval is a ranked-list problem. For a query, the retriever returns chunks in an order, and the evaluation run compares that order with relevance judgments. Precision@K, Recall@K, MRR, and nDCG describe different properties of that list.

Generation is an answer-and-context problem. Groundedness asks whether the answer's factual claims are supported by the retrieved context, while answer relevance asks whether the answer responds to the user's question; RAG evaluation research treats retrieval and generation as distinct components. Neither measurement is a substitute for the other, and neither can repair a retrieval miss.

That separation matters when a RAG application appears wrong:

1. Low Recall@K means the needed evidence did not reach the prompt, so prompt changes cannot make absent evidence available.
2. Low Precision@K means the context includes too much irrelevant material, giving the model more noise to sort through.
3. Strong retrieval with low groundedness means the answer added or distorted claims beyond the supplied context.
4. Strong groundedness with low answer relevance means the response may be well supported but still evade the question.

The Evaluation of Retrieval-Augmented Generation survey frames RAG evaluation around its retrieval and generation components. That is a practical starting point for RAG evaluation metrics in .NET: record observations at the layer that can explain the failure before combining them into a release decision.

This is also why a semantic-search result score is not an evaluation metric by itself. A store-specific similarity score is useful for ranking within that store, but it is neither a relevance judgment nor evidence that an answer is grounded. If you are refining the retrieval layer, the existing semantic search engine guide is useful implementation context. The evaluation contract remains provider-neutral.

RAG evaluation metrics in .NET: Measure the ranked retrieval list

Precision@K is the number of relevant items among the first K results divided by K (Introduction to Information Retrieval). It answers, "How much of the context budget is useful at this cutoff?" A result list with three relevant chunks in its first five positions has Precision@5 of 0.6.

Recall@K is the number of relevant items found in the first K results divided by the total number of relevant items for that query (Introduction to Information Retrieval). It answers, "How much of the available relevant evidence did the retriever find?" If the query has four judged-relevant chunks and the first five results contain three of them, Recall@5 is 0.75.

MRR, mean reciprocal rank, focuses on the first relevant result (TREC-8 Question Answering Track report). For one query, the reciprocal rank is 1 / rank of that first result; MRR averages that value over queries. It is informative when a reader needs one authoritative passage quickly. It deliberately says little about the remaining relevant passages, so it should not stand in for recall.

nDCG, normalized discounted cumulative gain, uses graded relevance and discounts lower-ranked results (Introduction to Information Retrieval). It rewards a highly useful chunk near the top more than the same chunk near the bottom. Normalizing by the ideal ranking makes the value comparable across queries with different judgment distributions.

When implementing RAG evaluation metrics in .NET, write the cutoff next to every metric name. A change from Precision@5 to Precision@10 changes the question being answered, even when the retrieval system is unchanged.

The following provider-neutral program calculates all four measures from a unique ranking and a chunk-ID-to-grade judgment map. It uses only the .NET base class library, so no RAG package is implied. It is validated for .NET 7 with C# 10: OrderDescending is available from .NET 7, and the language version supports file-scoped namespaces and records. Enumerable.OrderDescending The namespace keyword Configure language version The example's values are deliberately small enough to inspect by hand.

using System;
using System.Collections.Generic;
using System.Linq;

namespace RetrievalMetrics;

public sealed record JudgedRanking(
    IReadOnlyList<string> ChunkIds,
    IReadOnlyDictionary<string, int> GradesByChunkId);

public static class Program
{
    public static void Main()
    {
        var ranking = new List<string>
        {
            "chunk-7",
            "chunk-3",
            "chunk-9",
            "chunk-1",
            "chunk-5",
        };

        var gradesByChunkId = new Dictionary<string, int>
        {
            ["chunk-1"] = 0,
            ["chunk-3"] = 3,
            ["chunk-5"] = 2,
            ["chunk-7"] = 0,
            ["chunk-9"] = 1,
        };
        var rankings = new List<JudgedRanking>
        {
            new(ranking, gradesByChunkId),
        };

        Console.WriteLine($"Precision@3: {Metrics.PrecisionAtK(ranking, gradesByChunkId, 3):F3}");
        Console.WriteLine($"Recall@3: {Metrics.RecallAtK(ranking, gradesByChunkId, 3):F3}");
        Console.WriteLine($"MRR: {Metrics.MeanReciprocalRank(rankings):F3}");
        Console.WriteLine($"nDCG@3: {Metrics.NdcgAtK(ranking, gradesByChunkId, 3):F3}");
    }
}

public static class Metrics
{
    public static double PrecisionAtK(
        IReadOnlyList<string> ranking,
        IReadOnlyDictionary<string, int> gradesByChunkId,
        int k)
    {
        ValidateInputs(ranking, gradesByChunkId, k);

        return ranking.Take(k).Count(id => gradesByChunkId[id] > 0) / (double)k;
    }

    public static double RecallAtK(
        IReadOnlyList<string> ranking,
        IReadOnlyDictionary<string, int> gradesByChunkId,
        int k)
    {
        ValidateInputs(ranking, gradesByChunkId, k);

        var relevantCount = gradesByChunkId.Values.Count(grade => grade > 0);
        if (relevantCount == 0)
        {
            return 0;
        }

        return ranking.Take(k).Count(id => gradesByChunkId[id] > 0)
            / (double)relevantCount;
    }

    public static double MeanReciprocalRank(
        IReadOnlyList<JudgedRanking> rankings)
    {
        if (rankings.Count == 0)
        {
            return 0;
        }

        return rankings.Average(ranking =>
        {
            ValidateInputs(ranking.ChunkIds, ranking.GradesByChunkId, 1);
            var firstRelevant = ranking.ChunkIds
                .Select((id, index) => (id, index))
                .FirstOrDefault(item => ranking.GradesByChunkId[item.id] > 0);

            return firstRelevant.id is null
                ? 0
                : 1d / (firstRelevant.index + 1);
        });
    }

    public static double NdcgAtK(
        IReadOnlyList<string> ranking,
        IReadOnlyDictionary<string, int> gradesByChunkId,
        int k)
    {
        ValidateInputs(ranking, gradesByChunkId, k);

        var actual = DiscountedCumulativeGain(ranking.Take(k).Select(id => gradesByChunkId[id]));
        var ideal = DiscountedCumulativeGain(gradesByChunkId.Values.OrderDescending().Take(k));

        return ideal == 0 ? 0 : actual / ideal;
    }

    private static void ValidateInputs(
        IReadOnlyList<string> ranking,
        IReadOnlyDictionary<string, int> gradesByChunkId,
        int k)
    {
        if (k <= 0)
        {
            throw new ArgumentOutOfRangeException(nameof(k), "K must be positive.");
        }

        if (ranking.Count != ranking.Distinct(StringComparer.Ordinal).Count())
        {
            throw new ArgumentException("A ranking must not contain duplicate chunk IDs.", nameof(ranking));
        }

        if (gradesByChunkId.Values.Any(grade => grade < 0))
        {
            throw new ArgumentException("Relevance grades must be non-negative.", nameof(gradesByChunkId));
        }

        if (ranking.Any(id => string.IsNullOrWhiteSpace(id) || !gradesByChunkId.ContainsKey(id)))
        {
            throw new ArgumentException(
                "Every ranking entry must have a stable ID and a corresponding relevance grade.",
                nameof(ranking));
        }
    }

    private static double DiscountedCumulativeGain(IEnumerable<int> grades)
    {
        return grades.Select((grade, index) =>
            (Math.Pow(2, grade) - 1) / Math.Log2(index + 2)).Sum();
    }
}

The output is deterministic because relevance is input data, not inferred at scoring time. The scorer requires a unique ranking, non-negative grades, and a grade for every ranked chunk; its ideal list is derived from that same judgment map. In a real RAG evaluation metrics in .NET run, use the same relevance rules for every compared configuration, keep K explicit, and preserve the corpus revision used for retrieval. This article assumes those judgments already exist rather than covering their construction or maintenance.

One implementation detail is worth calling out. The sample treats zero as not relevant and positive grades as relevant for Precision@K, Recall@K, and MRR. nDCG retains the grades. That lets a run distinguish a marginally useful chunk from a highly useful one without pretending that every metric answers the same question.

Use metric combinations instead of a winner

No retrieval metric declares a retriever universally better. Each makes a different tradeoff visible.

For example, a configuration can raise Recall@10 by retrieving broader context while lowering Precision@5. That is not automatically a regression or an improvement. It is evidence that the candidate set changed. The sensible next question is whether the generation layer benefits from that extra evidence under the same context budget.

The RAG evaluation survey cautions that conventional information-retrieval metrics do not necessarily predict end-to-end answer quality. A chunk can be topically relevant yet fail to contain the detail needed to answer. Conversely, a lower-ranked passage might contain the decisive constraint. Read retrieval scores as diagnostic signals, not as proof that the answer is correct.

That distinction is important for document Q&A. The document Q&A RAG example shows the kind of retrieval-to-answer path that these measurements inspect. This article intentionally stops before recommending a retriever, model, or framework.

Groundedness and answer relevance measure different answer failures

Groundedness is a claim-support check (Evaluation of Retrieval-Augmented Generation). Break an answer into factual claims and ask whether each claim is supported by the retrieved context supplied to the generator. An answer can be factually plausible in the outside world and still be ungrounded in this measurement when the supplied context does not support it. That boundary is useful because it tests whether the system stayed within its evidence.

Answer relevance is a question-answer fit check (Evaluation of Retrieval-Augmented Generation). It asks whether the response addresses the user's request. A grounded answer can fail this test by accurately summarizing an unrelated retrieved chunk. An answer that directly addresses the question can still fail groundedness by inventing a detail.

RAG evaluation metrics in .NET work becomes clearer when those two values sit beside, rather than inside, the retrieval table. The first identifies unsupported claims; the second identifies an answer that missed the user's intent.

The next program keeps those results as separate records. It does not call an LLM. A reviewer or a separately calibrated evaluator supplies the per-claim support and question-fit inputs; the program only calculates and reports the measurements.

using System;
using System.Collections.Generic;
using System.Linq;

namespace AnswerMeasurements;

public sealed record ClaimAssessment(string Claim, bool IsSupportedByRetrievedContext);

public sealed record AnswerAssessment(
    string QueryId,
    IReadOnlyList<ClaimAssessment> Claims,
    bool AddressesQuestion)
{
    public double Groundedness =>
        Claims.Count == 0
            ? 0
            : Claims.Count(claim => claim.IsSupportedByRetrievedContext)
                / (double)Claims.Count;

    public double AnswerRelevance => AddressesQuestion ? 1 : 0;
}

public static class Program
{
    public static void Main()
    {
        var assessment = new AnswerAssessment(
            "policy-42",
            new List<ClaimAssessment>
            {
                new("The policy expires at the end of the calendar year.", true),
                new("The policy applies to contractors.", false),
            },
            AddressesQuestion: true);

        Console.WriteLine($"Groundedness: {assessment.Groundedness:F3}");
        Console.WriteLine($"Answer relevance: {assessment.AnswerRelevance:F3}");
    }
}

Binary labels make the distinction easy to see. A team can use an ordinal scale when its written rubric defines what each value means, but it should not blend the values until the interpretation is clear. For RAG evaluation metrics in .NET, the valuable result is not a polished composite number. It is knowing whether unsupported claims, question mismatch, or missing retrieval evidence caused the problem.

Treat LLM judges as fallible measurement instruments

An LLM can assess support or question fit when prompted with an answer and its retrieved context (Can LLMs Be Trusted for Evaluating RAG Systems?). That can speed up repeated offline comparisons, especially when every response receives the same rubric and input shape. It does not make the resulting score objective.

For RAG evaluation metrics in .NET, a judge score is an observation produced by a configured process, not a ground-truth fact (Can LLMs Be Trusted for Evaluating RAG Systems?). Its rubric and inputs are part of the measurement.

The survey of LLM-based RAG evaluation reviews automated approaches alongside human judgment and their limitations. Known risks include a judge favoring longer answers, preferring a presented ordering, or sharing blind spots with the model that produced the answer. A judge can also be misled when context is incomplete or when a rubric does not define the boundary between reasonable inference and unsupported invention.

Use that evidence to set a modest role for an LLM judge:

1. Give it the exact query, answer, and retrieved context for the layer being scored.
2. Ask a narrowly defined question, such as whether a particular claim is supported by that context.
3. Store the rubric, judge identity, and raw decision with the result so a later run is comparable.
4. Compare a sample of decisions with human review before relying on the score for an important decision.

This is not a recommendation for a particular package or hosted evaluator. The evaluator boundary can remain an interface in an application until a specific implementation has been independently validated. The goal is to avoid turning a convenient judge score into a claim that the RAG system is correct.

Shape an offline evaluation run

An offline run should make it possible to compare two configurations on the same inputs without hiding the layers. The run takes an existing set of query judgments, executes a named configuration against one corpus revision, scores retrieval, then records answer measurements separately. This separation follows the RAG evaluation survey's component-oriented treatment and remains useful without production instrumentation.

That makes RAG evaluation metrics in .NET a repeatable comparison instead of an anecdotal prompt test. The run should retain the same query IDs, judgments, corpus revision, configuration identity, and scoring rules for every candidate.

This small program models the aggregation boundary. It receives already-calculated per-query metrics and emits separate retrieval and answer summaries. A real runner can populate these records from any provider, as long as it preserves the same identifiers and evaluation rules.

using System;
using System.Collections.Generic;
using System.Linq;

namespace OfflineEvaluationRun;

public sealed record QueryResult(
    string QueryId,
    double PrecisionAt5,
    double RecallAt5,
    double ReciprocalRank,
    double NdcgAt5,
    double Groundedness,
    double AnswerRelevance);

public sealed record EvaluationSummary(
    double MeanPrecisionAt5,
    double MeanRecallAt5,
    double MeanReciprocalRank,
    double MeanNdcgAt5,
    double MeanGroundedness,
    double MeanAnswerRelevance);

public static class Program
{
    public static void Main()
    {
        var results = new List<QueryResult>
        {
            new("q-1", 0.80, 0.67, 1.00, 0.92, 1.00, 1.00),
            new("q-2", 0.40, 0.50, 0.50, 0.61, 0.50, 1.00),
        };

        var summary = Summarize(results);
        Console.WriteLine(summary);
    }

    public static EvaluationSummary Summarize(IReadOnlyList<QueryResult> results)
    {
        if (results.Count == 0)
        {
            throw new ArgumentException("At least one query result is required.", nameof(results));
        }

        return new EvaluationSummary(
            results.Average(result => result.PrecisionAt5),
            results.Average(result => result.RecallAt5),
            results.Average(result => result.ReciprocalRank),
            results.Average(result => result.NdcgAt5),
            results.Average(result => result.Groundedness),
            results.Average(result => result.AnswerRelevance));
    }
}

Name the configuration and corpus revision outside this compact example, then retain them with every result. Otherwise a metric change is ambiguous: it could come from an embedding change, a retrieval change, a corpus change, a prompt change, or a judge change. Keeping the layers separate makes a difference explainable.

Offline scoring is different from operational instrumentation. If you need to examine pipeline tracing and telemetry after the metrics identify a concern, OpenTelemetry and observability in Microsoft Agent Framework covers that distinct topic. If application composition needs cross-cutting logging or caching around the evaluation runner, the Microsoft Agent Framework middleware guide covers that supplementary implementation concern. Here, the focus stays on repeatable retrieval and answer measurements.

A practical reading of the results

Start a RAG evaluation metrics in .NET review with the retrieval table, then read groundedness and answer relevance beside it. Avoid declaring success from a single higher average.

If Recall@K falls, inspect whether relevant chunks are absent. If nDCG falls while recall holds, inspect whether highly useful chunks moved down the ranking. If retrieval remains steady but groundedness falls, inspect how the answer used the same context. If groundedness remains high but answer relevance falls, inspect the question interpretation or answer instruction. Each pattern points to a different investigation.

The supporting code is intentionally plain C#. It demonstrates where deterministic math ends and evaluation judgment begins. That boundary is more durable than a package-specific convenience API, and it keeps the next experiment honest about what its score means.

Frequently asked questions

What is the difference between Precision@K and Recall@K in RAG?

Precision@K measures how much of the first K retrieved context is relevant (Introduction to Information Retrieval). Recall@K measures how much of the judged-relevant evidence appears in those results. Use both because a concise context can have high precision while still omitting necessary evidence.

When is MRR more useful than nDCG?

MRR is useful when the first relevant result is the main concern (TREC-8 Question Answering Track report). nDCG is more useful when multiple results have different degrees of usefulness and their ordering matters (Introduction to Information Retrieval). They answer different ranking questions rather than competing for one universal score.

Can high retrieval metrics prove that a RAG answer is correct?

No. Retrieval metrics describe the ranked evidence list, not how the generator used it (Evaluation of Retrieval-Augmented Generation). Check groundedness for support by supplied context and answer relevance for response to the question.

Is groundedness the same as answer relevance?

No. Groundedness checks support for answer claims in retrieved context (Evaluation of Retrieval-Augmented Generation). Answer relevance checks whether the response addresses the question. A response can pass one and fail the other.

Should an LLM judge be the only evaluator for RAG?

No. An LLM judge can provide repeatable automated observations, but its decisions can reflect prompt, position, verbosity, and model-family biases (Can LLMs Be Trusted for Evaluating RAG Systems?). Keep the rubric and judge configuration explicit, and compare decisions with human review where the consequences matter.

Why keep retrieval and answer measurements separate?

Separate measurements localize failure. A low recall result calls for retrieval investigation, while an ungrounded answer with strong retrieval calls for generation or answer-contract investigation. Combining them too early hides that distinction (Evaluation of Retrieval-Augmented Generation).