Observing .NET RAG Systems: OpenTelemetry, Latency, and Cost

An answer can be wrong for many reasons in a retrieval-augmented generation pipeline. The relevant document might not have been indexed. Search might have returned weak evidence. A reranker, a component that reorders retrieved candidates by relevance, might have discarded the useful chunk. The prompt might have become too large, or generation might have failed after retrieval succeeded. OpenTelemetry RAG .NET instrumentation makes those stages visible as one operation instead of a single opaque “AI request.”

That distinction matters before a team starts changing models or search settings. A slow response is not necessarily a slow model. A poor answer is not necessarily a generation failure. Traces provide the timeline and parent-child relationships needed to identify which boundary produced the result. For framework-specific context, see RAG with Semantic Kernel in C#. This article stays focused on a provider-neutral RAG pipeline and the telemetry contract around it.

OpenTelemetry RAG .NET starts with one end-to-end story

OpenTelemetry defines a trace as a tree of spans. A span represents one operation, and nested spans preserve the relationship between a request and its work. The OpenTelemetry Tracing API specification also defines immutable trace and span context that can propagate across process boundaries. That is the foundation OpenTelemetry RAG .NET systems need: one trace for an answer request, with child spans for the stages that contributed to it.

Before optimizing an exporter or dashboard, define the diagnostic questions and approved telemetry boundary; exporter and retention choices may need to be decided in parallel. Ask what a trace must answer:

• Did the request reach retrieval, and how long did each stage take?
• How many candidates entered search, reranking, and prompt assembly?
• Did the pipeline return evidence, generate a response, fail, or time out?
• Which corpus and embedding revisions were involved, using non-sensitive identifiers?

Those questions lead to durable span names and a small attribute vocabulary. In an OpenTelemetry RAG .NET design, the intent is not to reconstruct a conversation from telemetry. It is to explain the path through the system with counts, revisions, outcomes, and durations.

Use a stable, application-owned ActivitySource rather than creating sources dynamically. An OpenTelemetry RAG .NET source name gives host configuration one reliable instrumentation scope to subscribe to, while child activities automatically inherit the active request context.

using System.Diagnostics;

namespace RagObservability;

public static class RagTelemetry
{
    public const string SourceName = "DevLeader.Rag";

    public static readonly ActivitySource Source = new(SourceName);
}

public sealed record RagRequestMetadata(
    int RequestedResultCount,
    string CorpusRevision,
    string RetrievalMode);

public static class RagTracing
{
    public static Activity? StartRequest(RagRequestMetadata metadata)
    {
        Activity? activity = RagTelemetry.Source.StartActivity(
            "rag.request",
            ActivityKind.Internal);

        activity?.SetTag("rag.requested_result_count", metadata.RequestedResultCount);
        activity?.SetTag("rag.corpus_revision", metadata.CorpusRevision);
        activity?.SetTag("rag.retrieval_mode", metadata.RetrievalMode);

        return activity;
    }
}

Activity.Current flows the active operation across async calls, and ActivitySource.StartActivity creates this internal application span in that active context. Framework instrumentation owns the inbound ActivityKind.Server span; when it has created one, rag.request becomes its child. For OpenTelemetry RAG .NET, if a queue message or another service starts the work, propagate the trace context through that transport using the host's OpenTelemetry instrumentation rather than inventing a separate correlation ID for every component.

Trace ingestion and answer paths separately

An ingestion trace and an answer trace share a corpus, but they answer different operational questions. Mixing them into a generic rag.pipeline span makes failures harder to classify.

For ingestion, a useful root span is rag.ingest. Its child spans can represent extraction, chunking, embedding generation, and index write. In OpenTelemetry RAG .NET, attributes should identify a source revision or a batch count, not the source body, file name, or uploader identity. An index write failure then remains connected to the derived corpus revision without putting document content into the telemetry store.

For an answer request, use an application-owned rag.request span beneath the host's inbound server span when one exists, with child spans such as:

1. rag.embed_query
2. rag.search
3. rag.rerank
4. rag.build_prompt
5. rag.generate

The rag.* names and attributes in this article are application-private conventions, not OpenTelemetry semantic conventions. Define their contract before emitting them:

The Microsoft.Extensions.AI 10.7.0 OpenTelemetryEmbeddingGenerator implements GenAI semantic conventions v1.41, which its pinned source marks experimental and subject to change. Reranking is optional. When it is not enabled, omit that span instead of emitting a pretend success with a zero duration. The trace should describe work that happened. This separation also complements an existing Semantic Kernel document Q&A application guide, without prescribing its orchestration approach.

The search span is a good example of a narrow telemetry boundary. An OpenTelemetry RAG .NET search span records candidate and result counts plus the result state, but it does not record the query, filter expression, document IDs, or retrieved text.

using System.Diagnostics;
using OpenTelemetry.Trace;

namespace RagObservability;

public sealed record SearchOutcome(
    int CandidateCount,
    int ReturnedCount,
    bool TimedOut);

public static class SearchTracing
{
    public static SearchOutcome TraceSearch(
        Func<SearchOutcome> search,
        int requestedResultCount)
    {
        using Activity? activity = RagTelemetry.Source.StartActivity(
            "rag.search",
            ActivityKind.Internal);

        activity?.SetTag("rag.requested_result_count", requestedResultCount);

        try
        {
            SearchOutcome outcome = search();

            activity?.SetTag("rag.candidate_count", outcome.CandidateCount);
            activity?.SetTag("rag.returned_count", outcome.ReturnedCount);
            activity?.SetTag("rag.timed_out", outcome.TimedOut);
            activity?.SetStatus(
                outcome.TimedOut ? ActivityStatusCode.Error : ActivityStatusCode.Ok);

            return outcome;
        }
        catch (Exception exception)
        {
            activity?.SetStatus(ActivityStatusCode.Error);
            activity?.RecordException(exception);
            throw;
        }
    }
}

This example treats TimedOut as a dependency deadline failure, so it sets an error status. If an application instead defines it as an expected outcome, set an Ok status and emit a separate controlled outcome value. A normal empty result is successful when the corpus has no applicable source. Record exceptions through the selected OpenTelemetry mechanism only after confirming that exception data follows the same telemetry policy.

The returned counts are diagnostic signals, not quality scores. A request can retrieve eight chunks and still fail an offline relevance evaluation. Conversely, zero results might be an expected outcome for a corpus with no applicable source. OpenTelemetry RAG .NET traces explain what happened on an individual path; evaluation determines whether the behavior was useful.

Instrument the embedding boundary with Microsoft.Extensions.AI

Embedding calls occur during both ingestion and request handling. They deserve a consistent span because their latency, errors, and batching behavior can affect the entire trace. For OpenTelemetry RAG .NET, the Microsoft.Extensions.AI embedding documentation and v10.7.0 source, verified 2026-08-10, show EmbeddingGeneratorBuilder and its UseOpenTelemetry decorator for an IEmbeddingGenerator<string, Embedding<float>>.

The following composition uses that documented surface. It deliberately accepts an existing generator rather than selecting a provider. This OpenTelemetry RAG .NET boundary keeps trace policy independent of the embedding service and avoids treating a provider adapter as RAG architecture.

using Microsoft.Extensions.AI;

namespace RagObservability;

public static class EmbeddingInstrumentation
{
    public static IEmbeddingGenerator<string, Embedding<float>> AddTracing(
        IEmbeddingGenerator<string, Embedding<float>> innerGenerator)
    {
        return new EmbeddingGeneratorBuilder<string, Embedding<float>>(
                innerGenerator)
            .UseOpenTelemetry(
                sourceName: RagTelemetry.SourceName,
                configure: static generator => generator.EnableSensitiveData = false)
            .Build();
    }
}

Warning: Keep EnableSensitiveData false and do not set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true for ordinary production tracing. The v10.7.0 source shows that the environment variable otherwise enables raw additional request and response properties. Treat any exception as a restricted diagnostic configuration.

For this validated example, reference Microsoft.Extensions.AI 10.7.0 directly; it already depends on Microsoft.Extensions.AI.Abstractions 10.7.0. The decorator emits embedding telemetry under the supplied source name (Microsoft Learn). Pass the source name so the tracer provider subscribes to both application and embedding instrumentation. The active Activity.Current context establishes the embedding span's parent-child relationship.

This is one point where decorator ordering matters. If a cache wraps the traced generator, a cache hit might not produce a downstream embedding span. If tracing wraps the cache, the trace can show the cache operation as part of the embedding boundary. An OpenTelemetry RAG .NET contract should choose the meaning it needs, document it, and use the same order for ingestion and query embeddings so their traces remain comparable.

OpenTelemetry RAG .NET latency boundaries make waits actionable

End-to-end duration is essential, but it is only the sum of distinct waits. OpenTelemetry RAG .NET should retain spans at the boundaries where the pipeline delegates work: embedding, retrieval, reranking, prompt construction, and generation. Each span duration lets an investigation distinguish local computation from a dependency call and identify whether a regression starts before or after evidence is selected.

Prompt construction deserves its own internal span even when it is fast. It is the boundary where retrieved chunks become model input. In OpenTelemetry RAG .NET, record bounded structural data such as rag.context_chunk_count and rag.context_character_count. Do not attach the assembled prompt to ordinary traces. If content-level diagnostics are necessary, use a separately approved, access-restricted, short-retention workflow. A trace can explain that the context grew from two chunks to twelve without storing a user's question or a document excerpt.

Generation should record an outcome and usage values only when the provider response makes them available. Avoid deriving token counts from captured prompt text. A generic usage record makes the cost boundary explicit without pretending that a provider's price schedule is universal.

namespace RagObservability;

public sealed record ModelUsage(int InputTokens, int OutputTokens);

public sealed record UsageRates(
    string RateRecordId,
    string ModelOrRoutingKey,
    string CurrencyCode,
    int TokensPerRateUnit,
    decimal InputCostPerRateUnit,
    decimal OutputCostPerRateUnit,
    DateOnly EffectiveDate);

public static class UsageCost
{
    public static decimal Estimate(ModelUsage usage, UsageRates rates)
    {
        if (rates.TokensPerRateUnit <= 0)
        {
            throw new ArgumentOutOfRangeException(
                nameof(rates.TokensPerRateUnit));
        }

        decimal input =
            usage.InputTokens / (decimal)rates.TokensPerRateUnit * rates.InputCostPerRateUnit;
        decimal output =
            usage.OutputTokens / (decimal)rates.TokensPerRateUnit * rates.OutputCostPerRateUnit;

        return input + output;
    }
}

Treat the result as an estimate tied to an effective-dated rate record, model or routing key, currency, and token rate unit outside the trace. An OpenTelemetry RAG .NET trace can carry input and output token counts, the rate-record ID, and the estimate. That supports cost attribution and change detection without hardcoding provider prices in application code. It also prevents a change in pricing or model routing from silently changing the meaning of a historical cost series. Cached, batch, regional, and provider-specific billing rules can require separate rate records or calculations.

Minimize telemetry before it leaves the process

The most useful RAG telemetry is often metadata, not content. Query text, raw prompts, retrieved passages, user identifiers, access filters, and generated answers can contain sensitive data (OpenTelemetry sensitive-data guidance). An OpenTelemetry RAG .NET implementation that records them by default turns its observability system into another corpus that must be governed, retained, and deleted correctly.

For OpenTelemetry RAG .NET, prefer these boundaries:

• Use opaque, rotating request or session references only when a separate controlled system can resolve them.
• Record revision identifiers, counts, booleans, durations, outcome codes, and bounded size measurements.
• Keep document and chunk identifiers out of broad telemetry when they reveal internal structure. If incident response requires them, use a restricted diagnostic channel with a short retention policy rather than ordinary traces.
• Configure allow-listing, filtering, or redaction before export as defense in depth, and review sampled traces as carefully as application logs (OpenTelemetry sensitive-data guidance). Those processors are not permission to capture raw content in ordinary traces.
• Measure a privacy-preserving online signal, such as a coarse feedback outcome or an escalation count, separately from content needed for offline labeling.

This is data minimization, not a security-defense tutorial. It gives operators enough evidence to see a retrieval miss, an empty result, or a generation timeout while avoiding a raw transcript. The same discipline applies to traces produced by orchestration frameworks. My earlier OpenTelemetry and observability in Microsoft Agent Framework article covers that separate framework context; it does not replace an application-owned RAG telemetry contract.

Keep offline evaluation and online observation distinct

Offline evaluation asks whether a known query retrieved useful evidence and produced a grounded answer against a labeled dataset. It needs retained judgments, relevant chunk identifiers, corpus revision, and carefully controlled evaluator inputs. OpenTelemetry RAG .NET online observation asks what happened in production: durations, errors, empty retrievals, bounded counts, and sampled feedback.

These systems should correlate through safe version metadata, not through an attempt to export every production prompt into an evaluation dataset. For OpenTelemetry RAG .NET, tag both systems with a retrieval-policy version, corpus revision, embedding revision, and prompt-template revision. A rise in online empty results after a corpus revision is then a hypothesis to investigate with offline evaluation. It is not proof that a model regressed.

That separation preserves the focus of RAG evaluation only in spirit: this article does not define Precision@K, groundedness, or judge scoring. Instead, OpenTelemetry RAG .NET provides the operational evidence that tells you where to look when those measurements move.

Use a small, explicit revision envelope for both paths:

namespace RagObservability;

public sealed record RagRevision(
    string CorpusRevision,
    string EmbeddingRevision,
    string RetrievalPolicyRevision,
    string PromptTemplateRevision);

public static class RevisionTags
{
    public static void AddTo(Activity? activity, RagRevision revision)
    {
        activity?.SetTag("rag.corpus_revision", revision.CorpusRevision);
        activity?.SetTag("rag.embedding_revision", revision.EmbeddingRevision);
        activity?.SetTag("rag.retrieval_policy_revision", revision.RetrievalPolicyRevision);
        activity?.SetTag("rag.prompt_template_revision", revision.PromptTemplateRevision);
    }
}

Revision tags make an observed change testable. They do not decide whether a change is good. An OpenTelemetry RAG .NET workflow compares a controlled offline run before promotion, then observes production behavior using the same version identifiers and privacy-minimizing fields.

A practical tracing contract for a RAG pipeline

Before adding more attributes, write down the contract. An OpenTelemetry RAG .NET ingestion trace should show a batch moving through extract, chunk, embed, and index. A request trace should show query embed, search, optional rerank, prompt construction, and generation. Both should carry only the revision fields and bounded counts needed to connect an outcome to a deployed configuration.

This approach is more useful than a single “RAG latency” number. It can show whether a generation slowdown is actually a retrieval timeout, whether a new corpus revision increased empty results, and whether a prompt-policy change increased input-token usage. It also stays educational rather than becoming a dashboard-specific procedure. The middleware and custom-pipeline overview is a useful related read for cross-cutting pipeline concerns.

The goal of OpenTelemetry RAG .NET is not maximal telemetry. It is a trace that lets a developer explain the request path, a cost estimate that has a stated boundary, and enough safe version data to connect production observations to deliberate offline evaluation.

Frequently asked questions

Should every RAG stage have its own span?

Create spans for meaningful boundaries where work can fail, wait, or change independently: embedding, search, reranking, prompt construction, and generation. Avoid spans for trivial property assignment or every loop iteration. The trace should make the pipeline understandable, not create a high-volume event stream with no diagnostic value.

How should OpenTelemetry RAG .NET correlate ingestion with requests?

Do not force an ingestion trace to be the parent of a future request. They can occur days apart. Correlate them with a non-sensitive corpus revision and embedding revision. A request trace can then state which derived corpus it used without exposing the ingested source.

Can I put the user query in a span attribute?

Avoid that default. Queries can contain personal data, proprietary terms, or secrets. Use lengths, coarse categories you control, outcome values, and revision identifiers for broad operational tracing. Handle content-level diagnostics through a deliberately restricted process when it is genuinely required.

Where do token counts belong in a trace?

Attach input and output token counts to the generation boundary when the response reports them. Keep the rate record and billing calculation separate so a cost estimate remains reproducible when a price schedule changes. Do not infer counts by retaining raw prompts.

Is a high retrieval count proof that the answer was grounded?

No. Candidate and returned counts describe pipeline behavior, not answer quality. Use offline evaluations with labeled evidence and groundedness checks to assess quality. Use the trace to identify the stage that warrants investigation when those evaluations change.

Does the Microsoft.Extensions.AI decorator replace application spans?

No. UseOpenTelemetry instruments the embedding-generator boundary. Application-owned ActivitySource spans still describe search, reranking, prompt construction, generation, and the end-to-end RAG request. Together, they create one coherent trace when both sources are subscribed and an active context is propagated.