RAG and Embeddings in .NET: A Stable Architecture Guide

RAG and embeddings in .NET are easiest to understand when I stop treating them as a chat feature. They are a data system that happens to end with generated text. A useful answer depends on a source becoming trustworthy chunks, a query becoming a compatible vector, retrieval respecting access rules, and the final response showing its evidence. Miss any boundary and the system can still sound confident while being wrong, stale, or unauthorized.

This guide is the architecture layer. It deliberately does not choose a model, a vector database, or an orchestration framework. Those are implementation decisions that belong after the contracts, data ownership, and failure handling are clear. If you want a framework-specific walkthrough later, my existing RAG with Semantic Kernel guide is the right companion. Here, I want to establish the system that any implementation must preserve.

RAG and embeddings in .NET start with system boundaries

Retrieval-augmented generation, or RAG, gives a language model selected external context alongside a question, as Microsoft describes in its secure multitenant RAG architecture guidance. An embedding is a numeric representation used to find related content, as described in Azure AI Search's vector-search overview. Those definitions are useful, but they can hide the important work.

The practical flow for RAG and embeddings in .NET has two distinct paths:

1. An ingestion path admits a source, extracts and chunks its content, records provenance and permissions, creates embeddings, and writes a searchable representation.
2. An answer path authenticates a caller, derives an access scope, embeds the question, retrieves eligible evidence, builds a bounded context, generates an answer, and returns citations.

Generation is last for a reason. It should not own identity, source admission, document lifecycle, or retrieval policy. I want an application to be able to replace its model client without rewriting its deletion workflow, and to replace its index without losing provenance.

That separation also explains why a simple prototype can appear to work before it is architecturally sound. A hard-coded sample corpus has no permission changes, no malformed uploads, no index lag, no stale cache entries, and no need to explain why a result was retrieved. Production data does.

Treat ingestion as an audited transformation

Ingestion converts a source document into derived data. The derived data is not less important than the source. Chunks, embeddings, indexes, and response caches can all affect what a user sees.

For RAG and embeddings in .NET, start by assigning a durable document identity before splitting text. Preserve the source URI or repository identity, a revision, an ingestion timestamp, a content hash, and the principal that admitted the source. Carry the document identity and authorization metadata onto every chunk. A chunk without its source and access scope is an orphan waiting to become a security bug.

The OWASP RAG Security Cheat Sheet recommends hashing documents at ingestion, retaining provenance, scanning content, and admitting only approved sources. Those are not optional decorations around the embedding call. They are the controls that make later investigation and removal possible.

Here is a small provider-neutral domain model. The examples in this article compile together as one conceptual C# 10-or-later file. This first block contains the shared using directives and records the provenance that must survive chunking.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Security.Cryptography;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace RagArchitecture;

public sealed record SourceDocument(
    string Id,
    Uri SourceUri,
    string Title,
    string Version,
    DateTimeOffset IngestedAt,
    string IngestedBy,
    string Sha256,
    IReadOnlySet<string> AllowedPrincipals);

public sealed record DocumentChunk(
    string Id,
    string DocumentId,
    int Ordinal,
    string Text,
    SourceDocument Source);

public static class DocumentIntegrity
{
    public static string CreateSha256(string content)
    {
        return Convert.ToHexString(
            SHA256.HashData(Encoding.UTF8.GetBytes(content)));
    }
}

Chunking is a retrieval design decision, not an SDK checkbox. A chunk must retain enough local meaning to answer a question, yet remain small enough that a retrieved set does not crowd out the answer. Its boundaries, overlap policy, and extractor version should be explicit ingestion configuration. My earlier Semantic Kernel chunking article explores chunking mechanics; the architectural point here is simpler: a changed chunking policy creates a new derived corpus and deserves evaluation before promotion.

An ingestion worker should stage a document first. Validate type and source, normalize text, calculate the hash, assign metadata, then make its chunks retrievable only after the entire document succeeds. That approach avoids the frustrating state where half a revised manual is searchable beside half of the prior revision. It also gives deletion a stable key: the source document ID.

RAG and embeddings in .NET need a provider-neutral embedding boundary

An embedding generator receives text and produces a vector. Vector search compares a query vector with indexed vectors, so document and query representations must remain compatible with the index configuration, as Azure AI Search documents. If a system changes that contract, it should make the migration deliberate: version the embedding model, re-embed affected chunks, evaluate retrieval again, and retire the old representation only when it is safe.

That is why I put embeddings behind a local boundary in RAG and embeddings in .NET. The application owns text preparation, model version recording, retry policy, and the decision to index. A provider implementation owns the remote or local inference call. Microsoft documents IEmbeddingGenerator<TInput, TEmbedding> as a generic embedding abstraction with asynchronous generation for input collections, which is a useful integration seam when it fits your stack. My Semantic Kernel embeddings guide remains framework-specific context, but the application contract should not depend on a framework-specific type.

public sealed record EmbeddingVector(
    string ModelId,
    ReadOnlyMemory<float> Values)
{
    public int Dimensions => Values.Length;
}

public interface IEmbeddingGenerator
{
    Task<EmbeddingVector> GenerateAsync(
        string text,
        CancellationToken cancellationToken);
}

public sealed record AccessScope(IReadOnlySet<string> PrincipalIds);

public sealed record RetrievalRequest(
    EmbeddingVector QueryVector,
    AccessScope Access,
    int MaximumResults);

Notice what is absent: provider clients, API keys, and a user-supplied tenant filter. Those concerns do exist, but they belong at their own boundaries. The access scope must be derived from an authenticated caller by server-side policy, not trusted because a browser included a tenant ID in JSON.

The vector is only one retrieval signal. Exact identifiers, product codes, dates, and names often need lexical matching too. Hybrid retrieval combines lexical and vector candidate generation, while reranking is a later precision step over a candidate set. They solve different problems. Do not compress them into a single SearchAsync concept just because a storage product exposes one method.

For example, Azure AI Search's vector-search overview documents a vendor-specific implementation where vector and keyword search can run together and return a unified result set. That is a useful illustration, not a universal RAG rule. For a framework-specific view of vector-store abstractions, see Semantic Kernel Vector Store in C#. The architecture should be able to represent lexical retrieval, dense retrieval, fusion, metadata filtering, and an optional second-stage reranker separately.

Build RAG and embeddings in .NET around retrieval results and citations

For an answer-generating RAG system, retrieval should produce evidence-bearing results, not only text: which chunk was selected, from which source revision, under which access scope, and with what score. The answer generator can consume text, but the application must retain that evidence outside the prompt.

This matters for debugging. A bad answer can originate from no relevant chunks, weak ranking, an overly broad filter, a missing source revision, or bad synthesis. Without a typed retrieval result, every one of those failures looks like “the model made something up.”

public sealed record RetrievedChunk(DocumentChunk Chunk, double Score);

public interface IChunkIndex
{
    Task ReplaceDocumentAsync(
        SourceDocument document,
        IReadOnlyList<IndexedChunk> chunks,
        CancellationToken cancellationToken);

    Task<IReadOnlyList<RetrievedChunk>> SearchAsync(
        RetrievalRequest request,
        CancellationToken cancellationToken);

    Task DeleteByDocumentIdAsync(
        string documentId,
        CancellationToken cancellationToken);
}

public sealed record IndexedChunk(
    DocumentChunk Chunk,
    EmbeddingVector Vector);

public sealed record Citation(
    string DocumentId,
    string ChunkId,
    Uri SourceUri,
    string Title,
    string Version,
    string Sha256);

The citation is a contract, not a rendering detail. It lets a UI show a source link, lets an operator replay a response, and lets a reviewer determine whether the answer cited the revision that was actually retrieved. That is a stronger guarantee than asking a model to invent footnotes in prose.

The secure multitenant RAG architecture guidance from Microsoft makes the access boundary explicit: identity flows through the request path, and only authorized grounding data should reach the model. It also recommends an API layer in front of storage so authorization and filtering are not scattered across the application. That maps cleanly to AccessScope and IChunkIndex: the index implementation enforces the scope before it returns a candidate.

Keep answer assembly contracts explicit

An answer-composition boundary retains the links between a user question, selected chunks, generated text, and returned citations. It does not decide that a retrieved chunk is trustworthy or grant authorization. Those decisions already occurred at source admission and retrieval.

public sealed record UserQuestion(string Text);

public sealed record GeneratedAnswer(
    string Text,
    IReadOnlyList<string> SupportingChunkIds);

public interface IAnswerGenerator
{
    Task<GeneratedAnswer> GenerateAsync(
        UserQuestion question,
        IReadOnlyList<RetrievedChunk> evidence,
        CancellationToken cancellationToken);
}

public sealed record RagAnswer(
    string Text,
    IReadOnlyList<Citation> Citations);

public interface IRagAnswerResult;

public sealed record GroundedRagAnswer(RagAnswer Answer) : IRagAnswerResult;

public sealed record AnswerWithheld(string Reason) : IRagAnswerResult;

public interface IAnswerComposer
{
    Task<IRagAnswerResult> ComposeAsync(
        GeneratedAnswer generated,
        IReadOnlyList<RetrievedChunk> evidence,
        CancellationToken cancellationToken);
}

ReplaceDocumentAsync is a document-level promotion boundary: its implementation stages every chunk and vector for a revision, then makes that revision retrievable only after the complete operation succeeds. An implementation should preserve the prior revision or return a failure when staging or promotion fails.

IAnswerComposer must return AnswerWithheld when retrieval provides no eligible evidence or when SupportingChunkIds contains an ID outside the supplied evidence. It must not return generated text with a shortened citation list. This compact contract leaves orchestration implementation to a framework-specific guide while making the two failure outcomes visible.

For an end-to-end framework implementation, see the existing document Q&A application with Semantic Kernel. For the architecture, keep the coordinator thin and make its dependencies observable.

Authorization, security, and deletion are retrieval requirements

In RAG and embeddings in .NET, authorization is complete before prompt construction. Every chunk needs access-control metadata, and the retrieval boundary must enforce it with the server-derived scope. Post-filtering text after it has entered an LLM context is too late, because OWASP requires access control before content reaches the model. A model cannot be the authority that decides whether a caller may see a chunk.

Security applies to the corpus too. Retrieved text is data, not instructions. The OWASP RAG Security Cheat Sheet recommends clearly delimiting retrieved content, bounding the number and total size of chunks, scanning for suspicious content, validating outputs, and preserving full pipeline traces. These controls reduce risk. They do not make prompt injection impossible, which is why source admission, least privilege, and output controls work together.

Embeddings deserve the same care as source text. The OWASP RAG Security Cheat Sheet cautions that embeddings are not anonymized data and recommends source-equivalent access controls. I treat the vector index as a security-sensitive derived store, with write access restricted to ingestion services and query access restricted through the retrieval boundary.

Deletion closes the loop. When a source is removed, superseded, or loses permissions, OWASP recommends removing its chunks, embeddings, and cached responses, then retaining an audit event. The deletion operation in IChunkIndex is small, but it represents a workflow that reaches every derived artifact. A missing source document should never remain retrievable because its embedding was forgotten.

Make quality and operations visible before users report failures

I separate evaluation from observability. Evaluation asks whether retrieval found useful evidence and whether the answer is supported by that evidence. Observability explains what happened on a particular request: source revision, embedding model ID, retrieval count, applied access scope, selected chunk IDs, latency by stage, and any cache decision.

For RAG and embeddings in .NET, evaluate retrieval and generation separately. Retrieval can miss the needed chunk. It can return too much irrelevant context. Or it can return sound evidence and the generator can still write an unsupported answer. A small reviewed set of questions with expected document or chunk IDs makes those failures visible long before a broad benchmark score does.

Evaluation data has a lifecycle too. A corpus revision can invalidate expected chunk IDs even when application code stays unchanged. Record the corpus revision with every test case, then re-review failures after an ingestion or embedding migration before declaring the retriever broken.

Trace identifiers should connect ingestion, retrieval, generation, and deletion without putting raw sensitive prompts or documents into telemetry by default. The goal is replayable evidence, not a second ungoverned copy of the corpus. If you are already instrumenting retrieval code, my Semantic Kernel semantic search article is useful implementation context, but the same privacy boundary applies to every framework.

The architecture gives you useful questions to ask during an incident: Was this chunk approved? Which source revision produced it? Did the caller have access when retrieval ran? Was the answer limited to retrieved evidence? Did a cache survive a permission change? Those questions are actionable because the system retained the required contracts.

Frequently asked questions

What is the minimum architecture for RAG and embeddings in .NET?

At minimum, keep source provenance, chunk identity, an embedding boundary, retrieval with server-enforced access scope, an answer boundary, and citations. Add explicit deletion and telemetry paths before the corpus becomes difficult to reason about.

Do embeddings replace keyword search in a RAG system?

No. Embeddings retrieve conceptual similarity; keyword retrieval preserves exact lexical signals. They can be independent candidate generators, with a later fusion policy when the use case warrants it.

Should my RAG system store a citation for every answer?

Store the evidence needed to construct citations for every grounded answer. Whether every user interface displays them is a product decision, but retaining source, chunk, revision, and hash information makes verification and operations possible.

Can I enforce authorization after the model sees retrieved text?

Not as the control that protects confidentiality. Enforce authorization before retrieval and prompt construction; post-generation controls are only a secondary safeguard.

How do I handle a source document that changes or is deleted?

Use the source document ID to find every derived chunk and vector, delete or re-index them as appropriate, invalidate permission-scoped caches, and record the outcome. Treat that workflow as part of ingestion design, not later cleanup.

Is a RAG answer automatically factual because it has citations?

No. Citations show which evidence was retrieved. They do not prove that retrieval was complete, that the source is current, or that the generated wording faithfully represents it. Evaluation and source governance still matter.

The stable mental model

The durable part of RAG and embeddings in .NET is not a package call. It is the chain of responsibility: approved source to versioned chunk, compatible embedding to authorized retrieval, retrieved evidence to cited answer, and source change to complete deletion. Build those boundaries first. Then a model client, vector store, hybrid strategy, reranker, or framework can evolve without turning the whole system into a rewrite. For an existing framework-level introduction, see Microsoft Agent Framework in C#: Complete Developer Guide.