# The AI N+1 Problem in .NET

Most .NET developers recognise an N+1 database query when they see one. AI features can recreate the same failure at a more expensive boundary. An application loads 300 support tickets, documents or product descriptions, maps over the collection and calls a language model once for every item. The database query count looks fine. The code is asynchronous. A local test finishes quickly. Yet one logical operation has become 300 remote inference requests, each repeating the same instructions, competing for the same quota and creating another opportunity for throttling or partial failure. The usual response is to add `Task.WhenAll` and celebrate the lower elapsed time. That only makes the requests concurrent. It doesn't reduce their number, remove duplicated input tokens or define what should happen when 287 calls succeed and 13 fail. This is the N+1 model call problem. Solving it requires more than a faster loop.

## N+1 has moved beyond the database

Imagine a service that classifies pending submissions. The first query is efficient and projects only the fields the classifier needs. The problem begins after the data leaves Entity Framework Core.

```csharp
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.AI;

public sealed class SubmissionClassifier(
    SubmissionDbContext dbContext,
    IChatClient chatClient)
{
    public async Task<IReadOnlyList<ClassifiedSubmission>> ClassifyPendingAsync(
        CancellationToken stopToken)
    {
        var submissions = await dbContext.Submissions
            .Where(x => x.Status == SubmissionStatus.Pending)
            .Select(x => new ClassificationInput(x.Id, x.Description))
            .ToListAsync(stopToken);

        var calls = submissions.Select(async submission =>
        {
            var response = await chatClient.GetResponseAsync(
                [
                    new ChatMessage(
                        ChatRole.System,
                        "Classify the submission as Billing, Technical or Other."),
                    new ChatMessage(ChatRole.User, submission.Text)
                ],
                cancellationToken: stopToken);

            return new ClassifiedSubmission(
                submission.Id,
                response.Text);
        });

        return await Task.WhenAll(calls);
    }
}
```

There is one database query followed by one model call for every submission. If the query returns 300 rows, the application sends 300 requests. Each request carries the same system instruction. Each one consumes a request from the provider's quota, opens another failure path and produces a separately variable response. `Task.WhenAll` can reduce wall clock time when the provider and network allow enough parallelism, but the amount of external work remains unchanged. It can also make the burst considerably sharper. As `Task.WhenAll` enumerates the sequence, each asynchronous lambda starts and reaches its first incomplete `await`. Without another control, hundreds of calls can be in flight before the first one completes. The shape is easy to miss because the fan out appears after a sensible database operation.

![](https://cdn.hashnode.com/uploads/covers/67c36038c69a4b7143c5fc49/37e8a795-3e9d-4882-8914-30f63aeb8bf5.png align="center")

A model call is also a less predictable unit of work than a typical database query. Latency varies with model load and output length. Providers commonly enforce both request based and token based limits. Responses can be syntactically successful while still omitting an item or returning an invalid classification. Retries may produce a different answer from the original attempt. For model backed code, call count belongs in the architectural design alongside token volume, batch size, concurrency and result semantics.

## Concurrency doesn't remove duplicated work

Suppose every request contains a fixed system prompt of `S` tokens and one item averaging `I` input tokens. With `N` separate calls, the approximate input volume is:

\[ N(S + I) \]

If the work can be expressed as one batch, the approximate input volume becomes:

\[ S + NI + B \]

`B` represents the JSON structure, identifiers and separators required to frame the batch. The item content still has to be sent, but the fixed instructions and schema are no longer repeated `N` times. Real billing is provider and model specific, so this isn't a price calculator. It shows where the avoidable work comes from. Long instructions, examples, tool descriptions and output schemas make the repeated component much larger than a six word system message. Request count has a separate effect. Many hosted services constrain requests per minute as well as tokens per minute. Azure OpenAI, for example, documents both request rate and token rate quota concepts. A workload can therefore remain below its token allocation and still be throttled because it was split into too many small calls. The current limits and allocation rules are described in [Azure OpenAI quotas and limits](https://learn.microsoft.com/en-us/azure/foundry/openai/quotas-limits).

Parallelism helps throughput only until one of those limits becomes the bottleneck. Past that point, it creates queueing, `429` responses and retry traffic. A concurrency setting of 50 doesn't mean the system has capacity for 50 calls. It means the caller is willing to create up to 50 simultaneous demands on a capacity owned somewhere else.

## Batch the business operation

Classification, extraction, moderation and scoring often support batching because the same instruction is applied independently to many inputs. The important word is independently. A batch should preserve the identity of every item and make that independence explicit in the prompt.

The contract can remain ordinary C#.

```csharp
public sealed record ClassificationInput(
    Guid Id,
    string Text);

public sealed record ClassificationResult(
    Guid Id,
    string Category,
    string? Reason);

public sealed record ClassificationBatchResponse(
    IReadOnlyList<ClassificationResult> Results);
```

The identifier is part of the model contract rather than metadata held only by the caller. Position alone is too weak. Models can reorder results, skip an item or produce an extra entry. If the application assumes that output element 17 corresponds to input element 17, a missing result can attach every subsequent answer to the wrong record. The prompt should request structured output and state that each item must be evaluated independently. The provider adapter can enforce a JSON schema when its model supports structured output. The application should still validate the deserialised response, because valid JSON doesn't prove that the model returned the expected IDs.

```csharp
using System.Text.Json;
using System.Text.Json.Serialization;
using Microsoft.Extensions.AI;

public sealed class ClassificationClient(IChatClient chatClient)
{
    private const string SystemPrompt = """
        Classify each input independently as Billing, Technical or Other.
        Return one result for every supplied id.
        Copy each id exactly and never combine information between inputs.
        Return a JSON object with a results array. Each result must contain
        id, category and reason properties.
        """;

    public async Task<IReadOnlyList<ClassificationResult>> ClassifyAsync(
        IReadOnlyList<ClassificationInput> inputs,
        CancellationToken stopToken)
    {
        var payload = JsonSerializer.Serialize(
            inputs,
            AppJsonSerializerContext.Default.ClassificationInputArray);

        var options = new ChatOptions
        {
            ResponseFormat = ChatResponseFormat.Json,
            Temperature = 0
        };

        var response = await chatClient.GetResponseAsync(
            [
                new ChatMessage(ChatRole.System, SystemPrompt),
                new ChatMessage(ChatRole.User, payload)
            ],
            options,
            cancellationToken: stopToken);

        var batch = JsonSerializer.Deserialize(
            response.Text,
            AppJsonSerializerContext.Default.ClassificationBatchResponse)
            ?? throw new InvalidOperationException(
                "The model returned an empty batch response.");

        return ValidateAndOrder(inputs, batch.Results);
    }

    private static IReadOnlyList<ClassificationResult> ValidateAndOrder(
        IReadOnlyList<ClassificationInput> inputs,
        IReadOnlyList<ClassificationResult> results)
    {
        var expectedIds = inputs
            .Select(x => x.Id)
            .ToHashSet();

        var resultsById = new Dictionary<Guid, ClassificationResult>();

        foreach (var result in results)
        {
            if (!expectedIds.Contains(result.Id))
            {
                throw new InvalidOperationException(
                    $"The model returned unknown id {result.Id}.");
            }

            if (!resultsById.TryAdd(result.Id, result))
            {
                throw new InvalidOperationException(
                    $"The model returned duplicate id {result.Id}.");
            }
        }

        var missingIds = expectedIds
            .Except(resultsById.Keys)
            .ToArray();

        if (missingIds.Length > 0)
        {
            throw new IncompleteModelResponseException(missingIds);
        }

        return inputs
            .Select(x => resultsById[x.Id])
            .ToArray();
    }
}

[JsonSourceGenerationOptions(
    PropertyNamingPolicy = JsonKnownNamingPolicy.CamelCase)]
[JsonSerializable(typeof(ClassificationInput[]))]
[JsonSerializable(typeof(ClassificationBatchResponse))]
internal partial class AppJsonSerializerContext : JsonSerializerContext;
```

This example uses the provider neutral `IChatClient` abstraction from `Microsoft.Extensions.AI`. The interface supports regular and streaming responses and can be composed with telemetry, caching, function invocation and custom middleware. Microsoft's current overview is available in the [`IChatClient` documentation](https://learn.microsoft.com/en-us/dotnet/ai/ichatclient). `ChatResponseFormat.Json` requests structured JSON without prescribing a schema. Where the provider and model support schema constrained output, the application can supply a `ChatResponseFormatJson` containing the actual schema instead. Provider support still needs to be verified, and application validation remains necessary. The domain service owns the expected result, the adapter owns the wire mechanism used to request it.

## Batch size is a token budget decision

Replacing 300 calls with one enormous call exchanges one failure mode for another. A large batch can exceed the model's context window, leave too little room for output or produce a response too large to validate and retry economically. A fixed item count is a useful safety limit, but it isn't enough on its own. Ten short ticket subjects and ten pasted log files have completely different token footprints. The batch builder should consider both the number of items and their estimated token cost. Token counts depend on the tokenizer used by the selected model. Character or word counts are rough admission estimates and should include a conservative margin if an exact compatible tokenizer isn't available. [`Microsoft.ML.Tokenizers`](https://learn.microsoft.com/en-us/dotnet/ai/how-to/use-tokenizers) provides tokenisation components and token counting APIs for .NET, but the concrete tokenizer still needs to match the model closely enough for the limit being enforced. Keeping token measurement behind a small application abstraction avoids coupling the batching policy to one model family.

```csharp
public interface ITokenCounter
{
    int Count(string text);
}

public sealed class TokenBatcher(
    ITokenCounter tokenCounter,
    int maxPromptTokens,
    int maxItems,
    int fixedPromptTokens)
{
    public IEnumerable<IReadOnlyList<ClassificationInput>> Create(
        IEnumerable<ClassificationInput> inputs)
    {
        var batch = new List<ClassificationInput>();
        var usedTokens = fixedPromptTokens;

        foreach (var input in inputs)
        {
            var serialised = JsonSerializer.Serialize(input);
            var itemTokens = tokenCounter.Count(serialised) + 8;

            if (fixedPromptTokens + itemTokens > maxPromptTokens)
            {
                throw new InputExceedsModelBudgetException(input.Id);
            }

            var batchIsFull = batch.Count >= maxItems;
            var tokenBudgetIsFull = usedTokens + itemTokens > maxPromptTokens;

            if (batch.Count > 0 && (batchIsFull || tokenBudgetIsFull))
            {
                yield return batch.ToArray();
                batch.Clear();
                usedTokens = fixedPromptTokens;
            }

            batch.Add(input);
            usedTokens += itemTokens;
        }

        if (batch.Count > 0)
        {
            yield return batch.ToArray();
        }
    }
}
```

The `maxPromptTokens` value should already reserve capacity for the expected response. If each classification can return a category and a short reason, the output allowance grows with batch size. A production policy can estimate that output separately and reduce the prompt budget accordingly. The unexplained `8` in this sample represents JSON framing and a safety allowance. In production it should be a named, measured option. Token estimates should be compared with provider reported usage so the margin can be corrected over time. A batcher that has never been checked against real requests is only expressing confidence, not enforcing a limit.

Oversized individual items need an explicit route. Truncating them silently can change the classification. Depending on the domain, the application may reject them, summarise them through a separate controlled workflow, split them into meaningful sections or send them to a model with a larger context window. That choice belongs to the feature, because it changes which evidence the model sees.

## A batch changes model behaviour

Database batching is usually a transport optimisation. Model batching can change the answer. When several inputs share one context, the model can compare them even when the application didn't ask it to. A category used for an early item may influence a later item. A long or unusually phrased input can pull attention away from shorter neighbours. If one item contains hostile instructions, those instructions are now in the same prompt as other customers' data. Structured framing helps. Each item should be represented as data with an opaque identifier, and the system instruction should state that item content is untrusted and must be evaluated independently. This reduces ambiguity but doesn't create the hard isolation provided by separate requests.

That distinction is especially important for multi tenant or security sensitive workloads. Combining inputs from different trust domains can expand the effect of prompt injection and complicate data residency, logging and authorisation. Some workloads should be batched only within one tenant, one policy boundary or one sensitivity class. Others shouldn't share a model context at all. Batch size therefore has a quality dimension alongside cost and throughput. Evaluation should compare per item and batched results using realistic distributions, including a deliberately adversarial item placed beside ordinary ones. The largest batch that fits the context window is rarely the batch size with the best operational and behavioural properties.

## When calls must remain separate

Some operations genuinely need one request per item. An input may consume most of the context window. Each call may use different tools, permissions or response schemas. Results may have strict latency requirements and need to complete independently. Security policy may prohibit multiple users' content from sharing one inference context. In those cases, the N calls may be intentional, but unbounded fan-out still isn't. `Parallel.ForEachAsync` provides a clear local concurrency boundary. The following version allows four model calls at a time and retains correlation by ID.

```csharp
using System.Collections.Concurrent;

public sealed class IndividualClassificationRunner(
    ClassificationClient classificationClient)
{
    public async Task<IReadOnlyList<ClassificationResult>> ClassifyAsync(
        IReadOnlyList<ClassificationInput> inputs,
        CancellationToken stopToken)
    {
        var results = new ConcurrentDictionary<Guid, ClassificationResult>();

        await Parallel.ForEachAsync(
            inputs,
            new ParallelOptions
            {
                MaxDegreeOfParallelism = 4,
                CancellationToken = stopToken
            },
            async (input, itemStopToken) =>
            {
                var batch = await classificationClient.ClassifyAsync(
                    [input],
                    itemStopToken);

                if (!results.TryAdd(input.Id, batch[0]))
                {
                    throw new InvalidOperationException(
                        $"Duplicate result for {input.Id}.");
                }
            });

        return inputs
            .Select(x => results[x.Id])
            .ToArray();
    }
}
```

The concurrency value shouldn't be copied from a blog post, including this one. It has to reflect provider quotas, average token weight, desired latency and the number of application instances. Four concurrent calls in one process become 80 when 20 replicas run the same code. The [`Parallel.ForEachAsync` API](https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.parallel.foreachasync?view=net-10.0) controls one operation in one process. A limiter in the `IChatClient` pipeline protects all code paths using that client instance. Microsoft demonstrates this composition using `System.Threading.RateLimiting` in its `IChatClient` guidance. The two controls solve different scopes and are often useful together. A local loop prevents one bulk operation from consuming every permit. A shared client limiter stops unrelated features from collectively overwhelming the provider. In a scaled deployment, a provider aware gateway or distributed admission mechanism may be needed because an in memory semaphore cannot see calls made by other replicas.

Request concurrency is also an incomplete approximation when requests vary greatly in size. A 200 token classification and a 60,000 token analysis each occupy one permit but create very different quota pressure. Where token per minute limits dominate, admission should be weighted by estimated input and output tokens, then corrected using actual usage returned by the provider.

## Backpressure begins before the model client

Bulk AI work shouldnt normally remain attached to an HTTP request while hundreds of items are processed. Client disconnects, reverse proxy timeouts and deployment restarts make that lifecycle too fragile. A durable work record gives the application somewhere to store ownership, attempts and partial progress. A bounded `Channel<T>` can be useful inside a single process because producers wait when its capacity is full. The [current .NET channel documentation](https://learn.microsoft.com/en-us/dotnet/core/extensions/channels) describes this backpressure behaviour. A channel doesnt provide durability across restarts, so workloads that must survive process loss need a durable queue or database backed scheduler instead.

The important property is bounded admission. If the upstream system can enqueue work without limit while the model can process only a fixed token volume, the queue has merely moved into memory, a broker or a database table. Queue age will rise until the result is no longer useful. Capacity should be expressed in terms the business can observe. A system might accept 10,000 pending items while keeping the oldest item below a five minute service target. Once it can't meet that target, it can reject new bulk jobs, reduce their size, defer low priority work or route to a cheaper model. These are clearer behaviours than accepting everything and discovering the backlog from a billing alert.

## A successful response can still be partially failed

HTTP success says that the provider returned a response. It doesn't say that every business item completed. A batch of 40 inputs may contain 38 valid results, omit one ID and duplicate another. The JSON can be syntactically valid and the model call can report a successful finish reason. Treating the batch as a single Boolean success either discards 38 useful results or allows corrupted correlation into the database. The execution model should record outcomes per item. Valid, known IDs can be accepted. Unknown and duplicate IDs should be quarantined as response contract violations. Missing IDs can be retried in a smaller batch. An invalid category may be permanently rejected or sent through a repair prompt, depending on the domain.

Retries should operate on the unresolved set rather than replaying every original item. Replaying the whole batch consumes extra tokens and can produce different answers for records that already succeeded. If the application does retry a completed item, it needs a policy for whether the newer answer replaces the earlier one. This makes an attempt identifier useful. Persist the model, prompt version, batch identifier, item identifier, provider request identifier where available, token usage and validation outcome. Those fields allow an operator to explain which invocation produced a stored classification and distinguish a transport retry from a deliberate re-evaluation. The retry batch should usually shrink. If a 50 item response repeatedly omits entries near its end, sending the same 50 item prompt again repeats the conditions that caused the failure. Retrying only the missing items reduces output pressure and isolates malformed content.

## Keep database transactions away from inference

A model request shouldnt run inside a database transaction. Inference latency is variable, providers can throttle, and a retry policy can extend the call far beyond its usual duration. Holding locks or database connections throughout that period couples database health to external model capacity. A safer workflow claims a set of records in a short transaction, records the attempt and ownership version, then commits. The model call runs after the transaction closes. A second short transaction writes each result only if the record is still owned by that attempt and remains in the expected state.

This protects the application from late results. If an operator cancels and requeues the work, or another worker legitimately takes ownership after a lease expires, the earlier model response shouldn't overwrite the newer outcome. Optimistic concurrency, an attempt version or a fencing token can enforce that condition at the write boundary.

The same rule applies to cancellations. Cancelling `stopToken` stops waiting for cooperative operations, but it cannot prove that the remote provider performed no work. A request may finish after the caller has given up. Persisted state must decide whether a late result still belongs to the active attempt.

## Caching needs semantic keys

Batching changes how caching should be approached. Caching the raw batch response by the entire prompt gives poor reuse because a different item order or one changed record creates a new key. It can also make a partial response appear authoritative on the next attempt. Per item caching can be valuable for deterministic extraction or classification, but the key needs more than the input text. The model identifier, prompt version, response schema, relevant options and policy context all influence the meaning of the result. If any of them changes, an old result may no longer be valid. `Microsoft.Extensions.AI` includes caching middleware in the `IChatClient` ecosystem, but cache policy remains an application decision. Highly creative outputs, security sensitive evaluations and decisions based on changing external tools may have little safe reuse. A cache hit is only useful when equivalence is defined precisely enough for the domain.

## Measure cost per accepted result

Provider latency alone doesnt reveal an N+1 problem. A dashboard can show healthy 400 millisecond calls while one user action quietly creates 500 of them. Telemetry should connect model activity to the logical operation. For each job, record the number of source items, batches, model calls, retries and accepted results. Record batch size and token distributions, not just averages. Averages hide the single oversized input that causes most failures.

Cost per accepted result is more informative than cost per call. A larger batch may reduce request count but produce more omissions, driving repair calls and manual review. A smaller batch may cost slightly more in repeated instructions while delivering a higher valid result rate. The effective cost includes the recovery path. Throttle wait, queue age and permit utilisation reveal whether the application is applying backpressure before the provider rejects requests. Missing, unknown and duplicate output IDs should be explicit counters. They are contract failures, not log messages to be discovered during an incident. The OpenTelemetry integration described in the `IChatClient` documentation can capture the model client layer. Application metrics still need to describe item and batch semantics. Be careful with prompt and response capture: full model content can contain personal data, secrets or customer material, and high cardinality item IDs don't belong on metric labels.

## Test distributions rather than happy-path batches

A unit test with three ten word inputs proves very little about a production batcher. Its difficult behaviour appears at boundaries: the item that exactly fills the token budget, the next item that starts a new batch, and the single input that exceeds the budget by itself. The classification client needs contract tests for missing, duplicate and unknown IDs. It should also be tested when results arrive in a different order, because ordering by response position must never become an accidental dependency. Cancellation should be exercised while batches remain outstanding, followed by a late completion to verify that the persistence boundary rejects obsolete work.

Evaluation data should contain the size and language distribution seen in production. Token density varies between inputs, as does expected output length. Include empty values, large pasted documents, malformed Unicode, embedded JSON, markup and text that attempts to instruct the model. Place the hostile sample beside ordinary inputs to detect cross item influence. Load tests need a representative number of application replicas. An in process concurrency limit can perform perfectly on one instance and exceed provider capacity immediately after horizontal scaling. The useful assertions concern total call rate, token admission, queue age and recovery from throttling, rather than raw requests per second alone.

## Decide at the operation boundary

The easiest place to prevent N+1 model calls is the API presented to application code. If the domain operation classifies a collection, expose `ClassifyAsync(IReadOnlyList<ClassificationInput>)` rather than teaching every caller to loop over `ClassifyAsync(ClassificationInput)`. The collection shaped API gives the implementation room to batch, split by tokens, apply shared limits and report partial outcomes. An item shaped interface almost guarantees that batching will be rebuilt awkwardly above it. By the time a decorator sees individual calls, the logical collection, shared deadline and desired result policy may already have been lost.

The application should first decide whether inputs may share a model context. If they can, build batches using item and token limits, enforce identity in the response and measure quality as batch size changes. If they cannot, keep calls separate but apply bounded concurrency and shared admission control. In either case, persist enough attempt state to make retries and late results safe. The database version of N+1 taught developers to inspect what apparently innocent navigation property access does at the storage boundary. AI integration needs the same instinct. Whenever model invocation appears inside `Select`, `foreach` or a per row handler, count the calls created by the surrounding operation. Asynchronous code can hide the waiting. It cannot hide the bill, the quota or the recovery work.