Retrieval-Augmented Generation (RAG) has become the standard pattern for grounding LLM outputs in proprietary data. Instead of fine-tuning models on your documents — an expensive and brittle approach — RAG retrieves relevant chunks at query time and injects them into the prompt. This post walks through a production-ready RAG implementation using Azure AI Search and Semantic Kernel in .NET.

Why Azure AI Search + Semantic Kernel

Azure AI Search provides managed vector indexing with hybrid retrieval (keyword + semantic + vector), automatic chunking through built-in skillsets, and enterprise-grade security with managed identities. Semantic Kernel abstracts the orchestration layer — it handles prompt templating, function calling, memory plugins, and multi-step reasoning while keeping you in the .NET ecosystem.

Together they eliminate the infrastructure tax of self-hosted vector databases like Pinecone or Weaviate and let you focus on retrieval quality.

Start by creating an Azure AI Search resource with semantic ranker enabled. You’ll need a 2024-07-01 or later API version for integrated vectorization.

var searchClient = new SearchClient(
    new Uri("https://your-service.search.windows.net"),
    "your-index",
    new DefaultAzureCredential());

Define your index schema with a vector field for embeddings. Azure AI Search supports up to 4096 dimensions per vector field.

Ingesting Documents with Integrated Vectorization

Instead of calling an embedding model separately, configure an indexer with a skillset that chunks and vectorizes automatically.

var index = new SearchIndex("docs")
{
    Fields =
    {
        new SearchField("id", SearchFieldDataType.String) { IsKey = true },
        new SearchField("content", SearchFieldDataType.String)
            { IsSearchable = true, AnalyzerName = "en.microsoft" },
        new SearchField("contentVector", SearchFieldDataType.Collection(SearchFieldDataType.Single))
        {
            IsSearchable = true,
            VectorSearchDimensions = 1536,
            VectorSearchProfileName = "default"
        }
    },
    VectorSearch = new VectorSearch
    {
        Profiles = { new VectorSearchProfile("default", "hnsw") },
        Algorithms = { new HnswAlgorithmConfiguration("hnsw") }
    }
};

The skillset uses Azure OpenAI to generate text-embedding-ada-002 vectors inline — no external pipeline required.

Orchestrating RAG with Semantic Kernel

Semantic Kernel’s plugin architecture lets you wire the retrieval step as a native function and compose it with the LLM call.

var builder = Kernel.CreateBuilder();
builder.AddAzureOpenAIChatCompletion("gpt-4o", endpoint, credential);

var kernel = builder.Build();

kernel.ImportPluginFromObject(new RetrieveDocumentsPlugin(searchClient), "Search");

var prompt = """
    You answer questions using only the provided context.
    If the context doesn't contain the answer, say you don't know.

    Context:
    {{$documents}}

    Question: {{$question}}
    """;

var function = kernel.CreateFunctionFromPrompt(prompt);

var arguments = new KernelArguments
{
    ["question"] = "What is our refund policy?",
    ["documents"] = await kernel.InvokeAsync("Search", "Retrieve", new() { ["query"] = "refund policy" })
};

var answer = await kernel.InvokeAsync(function, arguments);

Hybrid Search with Semantic Ranking

The retrieval plugin calls Azure AI Search with hybrid mode — combining full-text BM25, vector similarity, and semantic reranking for the best results.

var options = new SearchOptions
{
    QueryType = SearchQueryType.Semantic,
    SemanticSearch = new SemanticSearchOptions
    {
        SemanticConfigurationName = "default",
        QueryCaption = new QueryCaption(QueryCaptionType.Extractive)
    },
    VectorSearch = new VectorSearchOptions
    {
        Queries = { new VectorizedQuery(embedding) { KNearestNeighborsCount = 10, Fields = { "contentVector" } } }
    }
};

var results = await searchClient.SearchAsync<SearchDocument>(query, options);

The semantic ranker reorders the top 50 BM25 + vector candidates using a transformer model trained on MS MARCO, producing L2-ranked final results.

Handling Multi-Turn Conversations

For chat scenarios, maintain a sliding window of conversation history and use Semantic Kernel’s ChatHistory object to preserve context.

var chat = new ChatHistory("You are a helpful assistant with access to internal documents.");

while (true)
{
    var userMessage = Console.ReadLine();
    chat.AddUserMessage(userMessage);

    var documents = await RetrieveRelevantChunks(userMessage, chat);
    var augmentedPrompt = BuildAugmentedPrompt(chat, documents);

    var response = await kernel.InvokePromptAsync(augmentedPrompt);
    chat.AddAssistantMessage(response.ToString());
    Console.WriteLine(response);
}

Performance Considerations

  1. Batch indexing — use IndexDocumentsBatch to push multiple documents per request; aim for 100–1000 documents per batch depending on size.
  2. Warm semantic ranker — semantic ranking has cold-start latency; the first query after index creation can take 2–3 seconds.
  3. Chunk sizing — 512–1024 tokens per chunk with 25% overlap works well for most document types. Use the text split skill with maximumPageLength: 1024.
  4. Caching embeddings — for frequently queried documents, cache OpenAI embeddings locally to avoid redundant API calls.

RAG with Azure AI Search and Semantic Kernel gives you a fully managed, production-grade retrieval pipeline entirely within the .NET ecosystem. The combination of hybrid search, semantic reranking, and Semantic Kernel’s orchestration makes it the most pragmatic path from prototype to production for document-grounded AI applications.