Building an AI Circuit Breaker in .NET

Circuit breakers in .NET are good at recognising technical failure. They can stop calls when a service begins returning HTTP 500 responses, timing out or refusing connections. This works because conventional service failures normally produce observable signals. AI failures are less cooperative, a model can return HTTP 200, valid JSON and every property required by the response schema, yet still produce an answer the application cannot safely use. It may contradict the supplied evidence, invent a customer identifier, omit a critical exclusion or return a plausible value that violates a business rule.

From the HTTP client's perspective, the call succeeded. From the application's perspective, it failed. Once AI output participates in business processes, technical availability is only part of the reliability problem. The application needs to recognise semantic failure and stop sending work through a route that is producing unusable answers. That calls for a circuit breaker that understands the result, not only the request.

Where traditional circuit breakers stop

A conventional circuit breaker observes exceptions and status codes. If enough operations fail within a sampling period, it opens the circuit and temporarily prevents further calls.

The state transition is familiar:

This is effective when failure is visible at the transport boundary. If an Azure OpenAI deployment becomes unavailable or begins timing out, an ordinary resilience pipeline can react.

Look at an extraction response from an insurance submission:

{
  "insuredName": "Northwind Aviation",
  "currency": "EUR",
  "limit": 5000000,
  "country": "Ireland",
  "evidence": []
}

The response is valid JSON. It satisfies the schema. It can be deserialised into a C# record without error. It may still be unusable. Perhaps the source document says the limit is USD 50,000,000. Perhaps the country was inferred from the broker's address rather than the insured's location. Perhaps the absence of evidence violates the application's requirement that every extracted value must be traceable to the source. None of these failures will trip a normal HTTP circuit breaker. Retries don't solve this reliably either. Repeating the same prompt against the same model can produce a different answer, but variation should not be mistaken for recovery. If a prompt deployment, model route or retrieval source is systematically producing poor output, immediate retries can amplify the problem while increasing cost and latency.

Define success at the application boundary

An AI operation should only be treated as successful after its output crosses the application's acceptance boundary. For an extraction workflow, acceptance could require a deserialisable response, complete required fields, valid domain values and supporting evidence. A decision making workflow may also require agreement with deterministic calculations or confirmation that the model used the current version of the source data. This validation belongs outside the model client. The client is responsible for communicating with the provider. The application is responsible for deciding whether the result is usable.

A small outcome type gives the circuit breaker something explicit to observe:

public enum AiOutcomeKind
{
    Accepted,
    TransportFailure,
    InvalidContract,
    BusinessRuleFailure,
    MissingEvidence,
    ContradictoryEvidence,
    PolicyRejection
}

public sealed record AiOutcome(
    AiOutcomeKind Kind,
    string? Reason = null)
{
    public bool IsAccepted => Kind == AiOutcomeKind.Accepted;

    public static AiOutcome Accepted() =>
        new(AiOutcomeKind.Accepted);

    public static AiOutcome Rejected(
        AiOutcomeKind kind,
        string reason) =>
        new(kind, reason);
}

This type stays at the AI integration boundary. It does not need to become a universal result pattern passed through every layer of the application. The important detail is that a completed inference and an accepted inference are recorded separately. The provider may have delivered a perfectly valid response while the application rejected its contents.

Evaluate the answer before recording success

Suppose the application extracts a submission summary using a structured model response:

public sealed record SubmissionExtraction(
    string? InsuredName,
    string? Currency,
    decimal? Limit,
    string? Country,
    IReadOnlyList<FieldEvidence> Evidence);

public sealed record FieldEvidence(
    string Field,
    string Source,
    string Text);

A semantic evaluator can apply deterministic checks after deserialisation:

public interface IAiResponseEvaluator<in T>
{
    ValueTask<AiOutcome> EvaluateAsync(
        T response,
        CancellationToken stopToken);
}

public sealed class SubmissionExtractionEvaluator(
    ICurrencyReferenceData currencies,
    ISubmissionEvidenceVerifier evidenceVerifier)
    : IAiResponseEvaluator<SubmissionExtraction>
{
    public async ValueTask<AiOutcome> EvaluateAsync(
        SubmissionExtraction response,
        CancellationToken stopToken)
    {
        if (string.IsNullOrWhiteSpace(response.InsuredName))
        {
            return AiOutcome.Rejected(
                AiOutcomeKind.BusinessRuleFailure,
                "The insured name was not extracted.");
        }

        if (response.Limit is null or <= 0)
        {
            return AiOutcome.Rejected(
                AiOutcomeKind.BusinessRuleFailure,
                "The limit must be greater than zero.");
        }

        if (string.IsNullOrWhiteSpace(response.Currency) ||
            !await currencies.ExistsAsync(response.Currency, stopToken))
        {
            return AiOutcome.Rejected(
                AiOutcomeKind.BusinessRuleFailure,
                "The currency is not recognised.");
        }

        if (response.Evidence.Count == 0)
        {
            return AiOutcome.Rejected(
                AiOutcomeKind.MissingEvidence,
                "The extraction did not contain supporting evidence.");
        }

        var evidenceResult = await evidenceVerifier.VerifyAsync(
            response,
            stopToken);

        if (!evidenceResult.IsSupported)
        {
            return AiOutcome.Rejected(
                AiOutcomeKind.ContradictoryEvidence,
                evidenceResult.Reason);
        }

        return AiOutcome.Accepted();
    }
}

The evaluator relies on deterministic code wherever possible. Currency validation should use reference data. Numeric constraints should use domain rules. Evidence verification should confirm that cited text exists in the supplied material. Another model can assist with evaluation when a deterministic check is impossible, but model based judging should be treated as an additional signal. Otherwise, the reliability of one probabilistic component becomes dependent on another probabilistic component that may share the same weakness. The evaluator also produces a reason rather than a single confidence score. A value such as 0.71 is difficult to act upon without knowing what it represents. A missing evidence citation and a domain rule violation may have the same numerical score while requiring completely different recovery paths.

Let semantic failures drive the circuit

The circuit breaker needs to observe the final AiOutcome. It should count both technical errors and selected semantic rejections. Not every rejected response should carry equal weight. A malformed response suggests a problem with the model, schema or prompt. A policy rejection may simply mean the model correctly detected content that requires human review. Opening the circuit for the second case could stop a healthy system.

The failure policy therefore needs to remain explicit:

public interface ISemanticFailurePolicy
{
    bool ShouldCountAsCircuitFailure(AiOutcome outcome);
}

public sealed class ExtractionFailurePolicy
    : ISemanticFailurePolicy
{
    public bool ShouldCountAsCircuitFailure(AiOutcome outcome) =>
        outcome.Kind is
            AiOutcomeKind.TransportFailure or
            AiOutcomeKind.InvalidContract or
            AiOutcomeKind.BusinessRuleFailure or
            AiOutcomeKind.MissingEvidence or
            AiOutcomeKind.ContradictoryEvidence;
}

This is where the application defines what a degraded AI route means. The definition can vary by use case even when several use cases share the same provider and model. A summarisation feature may tolerate missing optional details. A financial decision workflow may reject any output without complete evidence. Combining their results into one global circuit would hide those differences.

A semantic circuit breaker in .NET

The following implementation keeps the mechanism deliberately focused. It records a rolling sequence of outcomes, opens after the configured minimum throughput and failure ratio are reached, and allows a probe after the break duration.

public enum SemanticCircuitState
{
    Closed,
    Open,
    HalfOpen
}

public sealed record SemanticCircuitOptions
{
    public required int MinimumThroughput { get; init; }
    public required double FailureRatio { get; init; }
    public required TimeSpan SamplingDuration { get; init; }
    public required TimeSpan BreakDuration { get; init; }
}

internal sealed record RecordedOutcome(
    DateTimeOffset RecordedAt,
    bool Failed);

internal sealed record CircuitSnapshot(
    SemanticCircuitState State,
    DateTimeOffset? OpenedAt,
    bool ProbeInProgress,
    IReadOnlyList<RecordedOutcome> Outcomes)
{
    public static CircuitSnapshot Empty { get; } = new(
        SemanticCircuitState.Closed,
        null,
        false,
        []);
}

The circuit state is abstracted because an in-memory implementation only coordinates requests inside one process:

public interface ISemanticCircuitStateStore
{
    ValueTask<CircuitSnapshot> GetAsync(
        string circuitName,
        CancellationToken stopToken);

    ValueTask<bool> TryUpdateAsync(
        string circuitName,
        CircuitSnapshot expected,
        CircuitSnapshot updated,
        CancellationToken stopToken);
}

The breaker uses optimistic updates so concurrent requests do not silently overwrite one another's outcomes:

public sealed class SemanticCircuitBreaker(
    ISemanticCircuitStateStore stateStore,
    IOptions<SemanticCircuitOptions> options,
    TimeProvider timeProvider)
{
    private readonly SemanticCircuitOptions _options = options.Value;

    public async ValueTask<SemanticCircuitLease> EnterAsync(
        string circuitName,
        CancellationToken stopToken)
    {
        while (true)
        {
            var now = timeProvider.GetUtcNow();
            var current = await stateStore.GetAsync(
                circuitName,
                stopToken);

            var refreshed = RemoveExpiredOutcomes(current, now);

            if (refreshed.State == SemanticCircuitState.Open)
            {
                var canProbe =
                    refreshed.OpenedAt is not null &&
                    now - refreshed.OpenedAt >= _options.BreakDuration;

                if (!canProbe)
                {
                    throw new SemanticCircuitOpenException(
                        circuitName,
                        refreshed.OpenedAt!.Value);
                }

                if (refreshed.ProbeInProgress)
                {
                    throw new SemanticCircuitOpenException(
                        circuitName,
                        refreshed.OpenedAt!.Value);
                }

                var halfOpen = refreshed with
                {
                    State = SemanticCircuitState.HalfOpen,
                    ProbeInProgress = true
                };

                if (await stateStore.TryUpdateAsync(
                    circuitName,
                    current,
                    halfOpen,
                    stopToken))
                {
                    return new SemanticCircuitLease(
                        circuitName,
                        true);
                }

                continue;
            }

            if (refreshed.State == SemanticCircuitState.HalfOpen)
            {
                throw new SemanticCircuitOpenException(
                    circuitName,
                    refreshed.OpenedAt ?? now);
            }

            if (!ReferenceEquals(refreshed, current))
            {
                if (!await stateStore.TryUpdateAsync(
                    circuitName,
                    current,
                    refreshed,
                    stopToken))
                {
                    continue;
                }
            }

            return new SemanticCircuitLease(
                circuitName,
                false);
        }
    }

    public async ValueTask RecordAsync(
        SemanticCircuitLease lease,
        bool failed,
        CancellationToken stopToken)
    {
        while (true)
        {
            var now = timeProvider.GetUtcNow();
            var current = await stateStore.GetAsync(
                lease.CircuitName,
                stopToken);

            CircuitSnapshot updated;

            if (lease.IsProbe)
            {
                updated = failed
                    ? current with
                    {
                        State = SemanticCircuitState.Open,
                        OpenedAt = now,
                        ProbeInProgress = false
                    }
                    : CircuitSnapshot.Empty;
            }
            else
            {
                var recent = RemoveExpiredOutcomes(
                    current,
                    now);

                var outcomes = recent.Outcomes
                    .Append(new RecordedOutcome(now, failed))
                    .ToArray();

                var failureRatio = outcomes.Length == 0
                    ? 0
                    : outcomes.Count(x => x.Failed) /
                      (double)outcomes.Length;

                var shouldOpen =
                    outcomes.Length >= _options.MinimumThroughput &&
                    failureRatio >= _options.FailureRatio;

                updated = recent with
                {
                    State = shouldOpen
                        ? SemanticCircuitState.Open
                        : SemanticCircuitState.Closed,
                    OpenedAt = shouldOpen ? now : null,
                    Outcomes = outcomes
                };
            }

            if (await stateStore.TryUpdateAsync(
                lease.CircuitName,
                current,
                updated,
                stopToken))
            {
                return;
            }
        }
    }

    private CircuitSnapshot RemoveExpiredOutcomes(
        CircuitSnapshot snapshot,
        DateTimeOffset now)
    {
        var cutoff = now - _options.SamplingDuration;
        var recent = snapshot.Outcomes
            .Where(x => x.RecordedAt >= cutoff)
            .ToArray();

        return recent.Length == snapshot.Outcomes.Count
            ? snapshot
            : snapshot with { Outcomes = recent };
    }
}

public sealed record SemanticCircuitLease(
    string CircuitName,
    bool IsProbe);

public sealed class SemanticCircuitOpenException(
    string circuitName,
    DateTimeOffset openedAt)
    : Exception(
        $"Semantic circuit '{circuitName}' has been open since {openedAt:O}.");

The compare and swap behaviour behind TryUpdateAsync is important. Several responses can complete together, particularly when an ingestion workflow fans out across many documents. Losing updates would make the circuit underestimate the failure rate at precisely the point where a deployment is degrading.

Connect inference, evaluation and circuit state

The application service can now execute the model call, evaluate the response and report the semantic outcome:

public sealed class SubmissionExtractionService(
    IAiExtractionClient extractionClient,
    IAiResponseEvaluator<SubmissionExtraction> evaluator,
    ISemanticFailurePolicy failurePolicy,
    SemanticCircuitBreaker circuitBreaker,
    ILogger<SubmissionExtractionService> logger)
{
    public async Task<SubmissionExtraction> ExtractAsync(
        SubmissionDocument document,
        CancellationToken stopToken)
    {
        const string circuitName =
            "submission-extraction:gpt-primary:v4";

        var lease = await circuitBreaker.EnterAsync(
            circuitName,
            stopToken);

        try
        {
            var extraction = await extractionClient.ExtractAsync(
                document,
                stopToken);

            var outcome = await evaluator.EvaluateAsync(
                extraction,
                stopToken);

            var failed =
                failurePolicy.ShouldCountAsCircuitFailure(outcome);

            await circuitBreaker.RecordAsync(
                lease,
                failed,
                stopToken);

            if (!outcome.IsAccepted)
            {
                throw new AiResponseRejectedException(outcome);
            }

            return extraction;
        }
        catch (OperationCanceledException)
            when (stopToken.IsCancellationRequested)
        {
            throw;
        }
        catch (AiResponseRejectedException)
        {
            throw;
        }
        catch (Exception exception)
        {
            await circuitBreaker.RecordAsync(
                lease,
                failed: true,
                stopToken);

            logger.LogWarning(
                exception,
                "AI extraction failed for document {DocumentId}",
                document.Id);

            throw;
        }
    }
}

public sealed class AiResponseRejectedException(
    AiOutcome outcome)
    : Exception(outcome.Reason)
{
    public AiOutcome Outcome { get; } = outcome;
}

Cancellation caused by the caller is allowed to propagate without being counted as a model failure. Otherwise, a deployment could open its AI circuit because users navigated away, upstream requests expired or the application was shutting down. The circuit name also includes the use case and route version. A single circuit named azure-openai would combine unrelated behaviour. A healthy summarisation prompt could conceal a failing extraction prompt, while failures from one experimental deployment could disable every AI feature in the application.

Useful circuit boundaries normally follow the combination of use case, prompt or inference version, model route and sometimes tenant. The exact granularity depends on traffic. Circuits that are too broad mix unrelated outcomes. Circuits that are too narrow may never receive enough traffic to reach their minimum throughput.

Opening the circuit should change the route

Rejecting requests quickly protects the system, but an AI workflow usually needs a deliberate continuation path. An extraction service may route to a secondary model. It may use a previous prompt version, switch to deterministic extraction, queue the document for later processing or send it for human review. The appropriate response depends on the reason the primary route became unhealthy.

A fallback router can keep this decision outside the circuit itself:

public sealed class ResilientSubmissionExtractor(
    SubmissionExtractionService primaryExtractor,
    IFallbackSubmissionExtractor fallbackExtractor,
    ILogger<ResilientSubmissionExtractor> logger)
{
    public async Task<SubmissionExtraction> ExtractAsync(
        SubmissionDocument document,
        CancellationToken stopToken)
    {
        try
        {
            return await primaryExtractor.ExtractAsync(
                document,
                stopToken);
        }
        catch (SemanticCircuitOpenException exception)
        {
            logger.LogWarning(
                exception,
                "Primary semantic circuit is open for document {DocumentId}",
                document.Id);

            return await fallbackExtractor.ExtractAsync(
                document,
                stopToken);
        }
    }
}

The fallback should have its own acceptance checks and circuit state. Sending rejected output to a second model and accepting whatever comes back only moves the reliability problem. Fallbacks can also create unexpected cost. If a cheaper model degrades and every request shifts to a more capable model, the system may remain available while its inference spend rises sharply. Cost limits therefore belong alongside routing and reliability policies.

Semantic circuits need richer telemetry

A circuit that records only success and failure loses much of its diagnostic value. The application should retain the outcome kind, use case, inference version, model deployment, latency, token usage and fallback route. It should also retain a safe reference to the evaluation evidence. Raw prompts and documents may contain sensitive information, so observability data should not casually copy their contents.

With OpenTelemetry, an inference span can include bounded attributes:

activity?.SetTag("ai.use_case", "submission-extraction");
activity?.SetTag("ai.inference_version", "v4");
activity?.SetTag("ai.model_route", "gpt-primary");
activity?.SetTag("ai.outcome", outcome.Kind.ToString());
activity?.SetTag("ai.accepted", outcome.IsAccepted);
activity?.SetTag("ai.circuit", circuitName);

The resulting metrics should distinguish provider availability from semantic acceptance. A service reporting 99.9% technical success and 72% accepted output has a serious reliability problem that ordinary availability dashboards will miss. Acceptance rate also needs context. A sudden increase in MissingEvidence may point to a prompt regression. A rise in InvalidContract may follow a model or schema change. ContradictoryEvidence could indicate poor retrieval, stale source data or an overly permissive prompt. Those signals should be visible independently before they are condensed into the circuit's failure decision.

Avoid one threshold for every failure

A simple failure ratio works for an initial implementation, but semantic failures often have different severity. Ten responses missing optional evidence may justify degradation monitoring. A single response inventing an account number and passing it to a payment tool may require immediate isolation. The circuit policy can account for this by assigning weights or by maintaining separate circuits for distinct failure categories. High severity failures can open a route immediately, while lower severity failures use a rolling threshold. Care is needed when introducing weighted scores. A sophisticated formula can become difficult to explain during an incident. Operators should be able to see why the circuit opened and which observations contributed to the decision. Explicit policies tend to age better than a single opaque quality score.

Store circuit state where the work runs

The sample uses a state store abstraction because local memory is rarely sufficient once an AI service scales out. If five application instances each observe two failures, no individual instance has seen ten failures. A local circuit may remain closed everywhere even though the route is failing across the service. Requests can also move between instances, producing inconsistent open and half open behaviour. A distributed implementation can store the snapshot in Redis, a database or another central coordination store. Updates need atomic comparison so concurrent observations are preserved. The half open probe also needs exclusive ownership; otherwise every instance may decide that it is responsible for testing the recovering route. Durable workflows introduce another complication. An orchestration may pause for minutes or hours before continuing. It should re-evaluate circuit state when it resumes rather than assuming that the route selected at the beginning of the workflow is still healthy. Long running activities should also record the inference version they used. If the prompt or model route changes while work is in flight, late results from the previous version should not be mixed into the new circuit without an explicit policy.

Test failures that still return HTTP 200

Most AI integration tests concentrate on timeouts, invalid JSON and thrown exceptions. Semantic circuit testing needs successful provider responses containing unacceptable answers. The evaluator should be tested with missing evidence, unsupported values, contradictory source text and domain rule violations. The breaker should be tested around its threshold, sampling window, break duration and half open probe. TimeProvider allows these tests to advance time without real delays. The most valuable integration test sends a technically successful completion through the whole pipeline and confirms that it contributes to opening the circuit:

[Fact]
public async Task Opens_circuit_after_semantic_failure_threshold()
{
    var timeProvider = new FakeTimeProvider(
        new DateTimeOffset(2026, 9, 14, 12, 0, 0, TimeSpan.Zero));

    var options = Options.Create(new SemanticCircuitOptions
    {
        MinimumThroughput = 4,
        FailureRatio = 0.75,
        SamplingDuration = TimeSpan.FromMinutes(1),
        BreakDuration = TimeSpan.FromMinutes(2)
    });

    var store = new InMemorySemanticCircuitStateStore();
    var breaker = new SemanticCircuitBreaker(
        store,
        options,
        timeProvider);

    for (var index = 0; index < 3; index++)
    {
        var lease = await breaker.EnterAsync(
            "submission-extraction:v4",
            TestContext.Current.CancellationToken);

        await breaker.RecordAsync(
            lease,
            failed: true,
            TestContext.Current.CancellationToken);
    }

    var successfulLease = await breaker.EnterAsync(
        "submission-extraction:v4",
        TestContext.Current.CancellationToken);

    await breaker.RecordAsync(
        successfulLease,
        failed: false,
        TestContext.Current.CancellationToken);

    await Assert.ThrowsAsync<SemanticCircuitOpenException>(
        async () => await breaker.EnterAsync(
            "submission-extraction:v4",
            TestContext.Current.CancellationToken));
}

Testing against recorded model responses is particularly useful. It allows the team to replay known failure cases without depending on live inference, model availability or nondeterministic output. A prompt or model change can then run against the same evaluation suite before deployment. The tests won't prove that every future answer is safe, but they can detect the return of failure patterns the system already understands.

The circuit can reveal problems outside the model

A semantic circuit opening does not automatically prove that the model is at fault. The retrieved documents may be incomplete. Reference data may be stale. An OCR service may have dropped a page. A schema change may have introduced a required field without updating the prompt. The evaluator itself may contain a regression. This is another reason to preserve structured outcome categories and evidence. The circuit is responding to the health of an inference route as experienced by the application. That route includes context assembly, retrieval, model execution, parsing and validation. Treating the entire pipeline as one opaque model call makes diagnosis harder. Each stage should emit enough information to identify where accepted output began to decline.

Reliability begins after HTTP 200

AI providers can tell us whether an inference request was processed. They cannot decide whether its output is correct for a particular application. That decision belongs at the application boundary, where generated output meets source evidence, domain rules, permissions and current business state. Once acceptance is explicit, existing resilience ideas become useful again. Failure ratios can be measured. Circuits can open. Recovering routes can be probed. Work can move to a fallback or human review. Developers can observe semantic reliability alongside latency and availability. The result is an AI integration that responds to plausible but unusable answers with the same discipline that distributed systems already apply to timeouts and unavailable services. A green HTTP status is only the beginning of the decision.