Most engineers start in the same place. You normalise the schema, remove duplication, keep each fact in one place, and rely on joins to reconstruct the answer when the application needs it. That is still the right default for a transactional system. The problem is that many people quietly stretch that rule too far. They end up assuming that a data model which is logically clean must also be operationally fast. In production, that falls apart quickly.

The real cost of a heavily normalised model is not usually visible in one query. It shows up in repetition. The same joins run over and over. The same aggregates are recalculated on every request. The same object graph is rebuilt for every page, every API call, every dashboard tile, and every export. The database becomes a reconstruction engine. The application becomes a shaping engine. Both work hard, not because the business needs new information, but because the model forces them to keep rebuilding information the system already knows.

This is where denormalisation comes in.

Denormalisation is a deliberate decision to move work away from the read path and into the write path, or into a background projection step, because doing that once is cheaper than doing the same work thousands of times on demand. In a modern C# system, especially one built with ASP.NET Core, EF Core, background workers, queues, Redis, and event driven processing, that trade can transform performance.

The gains are not abstract. You see them in lower latency, lower database CPU, fewer allocations, more stable p95 and p99 response times, better concurrency, and less fragile query behaviour. You also get a cleaner separation between the source of truth and the shape that the application actually needs at the edge.

This is important mainly in systems with heavy read traffic, complex dashboards, queue screens, search pages, configuration endpoints, reporting APIs, and integration surfaces that repeatedly ask for the same shaped view of the data. If you treat every one of those reads as a fresh act of discovery, the system wastes time. If you precompute and persist the shape once, the request becomes a cheap lookup.

The key idea is simple. Normalisation optimises storage and correctness. Denormalisation optimises access. Mature systems usually need both.

The hidden cost of clean relational models

A normalised schema protects integrity. That is its job. It makes writes understandable and it keeps the domain tidy. The trouble starts when the application’s hot paths are read heavy and shape heavy.

Imagine a common business screen. You need to show a case list with case number, customer name, policy type, current stage, outstanding balance, number of open actions, last correspondence date, assigned handler, SLA status, and a search summary. In a fully normalised design, those values may come from six or seven tables, plus a few aggregate queries, plus a handful of rules in application code. Nothing about that is inherently wrong. The problem is frequency. If that same shape is requested constantly, the system is paying the cost of reconstruction on every request.

The request path starts to look like this.

That path is often acceptable in development. Small data volumes hide the cost. A local database hides the latency. The ORM hides the SQL. Then production arrives, concurrency rises, the dataset grows, and the endpoint that felt harmless becomes one of the hottest parts of the estate.

The first thing people usually try is index tuning. That helps. Then they add projections in LINQ. That helps a bit more. Then they introduce caching. That can help a lot, but it often hides the problem rather than fixing it. A cache miss still falls back to the same expensive reconstruction path. Once the underlying shape is wrong for the read pattern, you are tuning around the problem rather than changing it.

That is why denormalisation is so powerful. It does not ask how to run the same expensive query a little faster. It asks whether the query should exist in that form at all.

What denormalisation really means in a C# system

In practice, denormalisation in a C# system usually takes one of a few forms.

You store precomputed aggregates directly on a parent row, such as current balance, open item count, or last activity date.

You build a dedicated read model that already matches a page or API response.

You snapshot descriptive values, such as broker name or product name, onto a transactional record so you do not join to reference tables on every read.

You persist flags and classifications, such as IsUrgent, RiskBand, or HasOpenTasks, instead of recalculating them repeatedly.

You compile expensive response payloads into a cached or persisted format, often as JSON, so the request path can serve them directly.

Those are all forms of the same idea. You take work that would otherwise happen every time the application reads data and you pay for it once when the data changes.

That changes the economics of the system.

If a value changes once an hour but is read ten thousand times in that hour, it is usually madness to compute it ten thousand times. Store it. Keep it fresh. Read it cheaply.

Where the performance gains come from

The gains from denormalisation are easy to hand wave, but the useful part is knowing where they show up.

The first gain is query simplification. A query that previously needed multiple joins, aggregates, and conditional expressions can become a simple index seek against a flat row. That cuts database CPU, logical reads, memory pressure, and plan complexity.

The second gain is lower application overhead. Even if the database work is acceptable, materialising nested EF Core graphs and then reshaping them into DTOs still costs CPU and memory in the application. A flat read model avoids much of that.

The third gain is better tail performance. Complex queries are far more likely to show unstable p95 and p99 latencies, especially under concurrency or when parameter values vary. Simple denormalised queries are usually more predictable.

The fourth gain is improved cache behaviour. A denormalised row or payload already matches the response shape, so a cache miss is not painful. You do not have to rebuild the world before you can refill the cache.

The fifth gain is fewer cross service dependencies. If you snapshot small pieces of descriptive data, one service no longer needs to ask another service the same question on every request. That is often a bigger win than any SQL optimisation.

The sixth gain is better scaling behaviour. Once each request does less work, every instance can serve more traffic with more stable latency. Horizontal scaling starts to work properly because the units of work are cheaper and more predictable.

The right way to picture this is as two separate paths.

The write path gets slightly heavier. The read path becomes dramatically cheaper. In read heavy systems that is exactly the trade you want.

Technique one, precomputed aggregates

This is the most obvious denormalisation technique and still one of the most effective. If an aggregate is read often and changes comparatively rarely, store it.

Think about balances, counts, totals, last updated timestamps, most recent activity, most recent payment date, open task count, total claim value, or number of outstanding documents. Engineers often recalculate these on every request because the database can do it. That does not mean it should.

A typical normalised query might look like this.

public sealed class AccountSummaryService
{
    private readonly FinanceDbContext _db;

    public AccountSummaryService(FinanceDbContext db)
    {
        _db = db;
    }

    public async Task<AccountSummaryDto?> GetAsync(Guid accountId, CancellationToken stopToken)
    {
        return await _db.Accounts
            .Where(x => x.Id == accountId)
            .Select(x => new AccountSummaryDto
            {
                AccountId = x.Id,
                CustomerName = x.Customer.Name,
                CurrentBalance = x.Transactions.Sum(t => t.Amount),
                OpenInvoiceCount = x.Invoices.Count(i => !i.IsPaid),
                LastPaymentUtc = x.Payments
                    .OrderByDescending(p => p.PaidAtUtc)
                    .Select(p => (DateTime?)p.PaidAtUtc)
                    .FirstOrDefault()
            })
            .SingleOrDefaultAsync(stopToken);
    }
}

This is tidy. It is also doing real work every time the endpoint is called. The sum is recomputed. The count is recomputed. The payment ordering is revisited. The joins are rebuilt. If that account page is busy, you are paying that cost repeatedly for no gain in truth.

A denormalised design moves those values onto the account row or onto a dedicated account summary table.

public sealed class Account
{
    public Guid Id { get; set; }
    public Guid CustomerId { get; set; }
    public decimal CurrentBalance { get; set; }
    public int OpenInvoiceCount { get; set; }
    public DateTime? LastPaymentUtc { get; set; }
}

The read path becomes much simpler.

public sealed class AccountSummaryService
{
    private readonly FinanceDbContext _db;

    public AccountSummaryService(FinanceDbContext db)
    {
        _db = db;
    }

    public async Task<AccountSummaryDto?> GetAsync(Guid accountId, CancellationToken stopToken)
    {
        return await _db.Accounts
            .AsNoTracking()
            .Where(x => x.Id == accountId)
            .Select(x => new AccountSummaryDto
            {
                AccountId = x.Id,
                CustomerName = x.Customer.Name,
                CurrentBalance = x.CurrentBalance,
                OpenInvoiceCount = x.OpenInvoiceCount,
                LastPaymentUtc = x.LastPaymentUtc
            })
            .SingleOrDefaultAsync(stopToken);
    }
}

The gain here is not subtle. You have shifted work from every read to only the writes that actually change the values.

There are two good ways to maintain these fields. If the value is part of a hard business invariant, update it in the same transaction as the canonical write. If the value is mainly for display or read optimisation, project it asynchronously through an outbox driven worker.

Here is a synchronous example.

public sealed class PaymentService
{
    private readonly FinanceDbContext _db;

    public PaymentService(FinanceDbContext db)
    {
        _db = db;
    }

    public async Task RecordPaymentAsync(Guid accountId, decimal amount, DateTime paidAtUtc, CancellationToken stopToken)
    {
        var account = await _db.Accounts.SingleAsync(x => x.Id == accountId, stopToken);

        _db.Payments.Add(new Payment
        {
            Id = Guid.NewGuid(),
            AccountId = accountId,
            Amount = amount,
            PaidAtUtc = paidAtUtc
        });

        account.CurrentBalance -= amount;
        account.LastPaymentUtc = paidAtUtc;

        await _db.SaveChangesAsync(stopToken);
    }
}

That looks almost boring, which is exactly the point. Good denormalisation is often simple. It gives you a cheap read path because the system has already done the work.

Technique two, dedicated read models

This is where denormalisation starts to move from a tactical optimisation into a strategic design choice. A read model is a table or document that exists purely because a specific screen, endpoint, or integration needs data in a specific shape.

Suppose you have an underwriting queue page. The page needs submission number, insured name, broker name, product class, status, risk rating, attachment count, assigned underwriter, created date, and an urgency flag. In a fully normalised schema those values may be scattered across several tables and some of them may need to be computed. You can absolutely query them on demand. You will just keep paying for it.

A denormalised read model lets you store exactly what the queue needs.

public sealed class SubmissionReviewQueueItem
{
    public Guid SubmissionId { get; set; }
    public string SubmissionNumber { get; set; } = string.Empty;
    public string InsuredName { get; set; } = string.Empty;
    public string BrokerName { get; set; } = string.Empty;
    public string ProductClass { get; set; } = string.Empty;
    public string Status { get; set; } = string.Empty;
    public string RiskRating { get; set; } = string.Empty;
    public int AttachmentCount { get; set; }
    public string AssignedUnderwriterName { get; set; } = string.Empty;
    public bool IsUrgent { get; set; }
    public DateTime CreatedUtc { get; set; }
}

The query then becomes a simple paged lookup.

public sealed class ReviewQueueService
{
    private readonly UnderwritingDbContext _db;

    public ReviewQueueService(UnderwritingDbContext db)
    {
        _db = db;
    }

    public async Task<IReadOnlyList<ReviewQueueItemDto>> GetPageAsync(int page, int pageSize, CancellationToken stopToken)
    {
        return await _db.SubmissionReviewQueueItems
            .AsNoTracking()
            .OrderByDescending(x => x.IsUrgent)
            .ThenBy(x => x.CreatedUtc)
            .Skip((page - 1) * pageSize)
            .Take(pageSize)
            .Select(x => new ReviewQueueItemDto
            {
                SubmissionId = x.SubmissionId,
                SubmissionNumber = x.SubmissionNumber,
                InsuredName = x.InsuredName,
                BrokerName = x.BrokerName,
                ProductClass = x.ProductClass,
                Status = x.Status,
                RiskRating = x.RiskRating,
                AttachmentCount = x.AttachmentCount,
                AssignedUnderwriterName = x.AssignedUnderwriterName,
                IsUrgent = x.IsUrgent,
                CreatedUtc = x.CreatedUtc
            })
            .ToListAsync(stopToken);
    }
}

This is the kind of change that can take an endpoint from unpredictable and expensive to stable and cheap. It also changes how you index. Instead of trying to satisfy a messy query against the whole domain model, you can build indexes specifically for the queue.

The strongest way to maintain a read model is with a projection pipeline. The domain write commits. An outbox message is recorded in the same transaction. A background worker consumes the outbox and updates the read model. The read path never has to rebuild the queue shape on demand.

That pattern gives you reliability, observability, and a clean failure model. If a projection fails, you can retry it. If you need to rebuild, you can replay events or recalculate from the source of truth.

Technique three, snapshot descriptive data

A huge amount of hidden query cost comes from descriptive joins. Broker name. Product name. Region name. Handler display name. Organisation name. Status description. These are often joined into hot queries simply because they are stored elsewhere. That keeps the schema pure. It also keeps the read path unnecessarily busy.

Snapshotting descriptive values means copying them at the point where they matter. That often improves performance, and in many domains it also improves auditability because it preserves the value as it was when the transaction occurred.

Here is a simple example.

public sealed class Submission
{
    public Guid Id { get; set; }
    public Guid BrokerId { get; set; }
    public string BrokerNameSnapshot { get; set; } = string.Empty;
    public string ProductNameSnapshot { get; set; } = string.Empty;
    public string InsuredName { get; set; } = string.Empty;
    public DateTime CreatedUtc { get; set; }
}

When you create the submission, you take the snapshot.

public sealed class SubmissionService
{
    private readonly UnderwritingDbContext _db;

    public SubmissionService(UnderwritingDbContext db)
    {
        _db = db;
    }

    public async Task<Guid> CreateAsync(CreateSubmissionCommand command, CancellationToken stopToken)
    {
        var broker = await _db.Brokers.SingleAsync(x => x.Id == command.BrokerId, stopToken);
        var product = await _db.Products.SingleAsync(x => x.Id == command.ProductId, stopToken);

        var submission = new Submission
        {
            Id = Guid.NewGuid(),
            BrokerId = broker.Id,
            BrokerNameSnapshot = broker.DisplayName,
            ProductNameSnapshot = product.Name,
            InsuredName = command.InsuredName,
            CreatedUtc = DateTime.UtcNow
        };

        _db.Submissions.Add(submission);
        await _db.SaveChangesAsync(stopToken);

        return submission.Id;
    }
}

Now every queue, dashboard, export, and search result that needs the broker or product name can read it directly from the submission or from a projection row built from it. That removes joins from the hot path and makes results historically accurate. If the broker later changes their display name, older submissions do not silently rewrite history.

This is one of the most underused denormalisation techniques because people dismiss it as duplication. In reality it is often one of the cleanest improvements you can make.

Technique four, persist flags and classifications

A lot of expensive read logic is not about fetching data at all. It is about classifying it. Is this item urgent. Is this account over limit. Is this submission nearing SLA breach. Does this customer require manual review. Is this case ready for escalation. Those rules often combine dates, counts, statuses, and related rows. If they sit on the hot read path, they get recalculated constantly.

If the answer is needed often, persist it.

public sealed class SubmissionReviewQueueItem
{
    public Guid SubmissionId { get; set; }
    public bool IsUrgent { get; set; }
    public string RiskBand { get; set; } = string.Empty;
    public DateTime? EscalationDueUtc { get; set; }
}

The projector decides the values once.

public static class SubmissionClassification
{
    public static bool CalculateUrgency(DateTime createdUtc, string status, int attachmentCount)
    {
        if (status == "Completed")
        {
            return false;
        }

        if (attachmentCount == 0)
        {
            return true;
        }

        return DateTime.UtcNow - createdUtc > TimeSpan.FromHours(24);
    }

    public static string CalculateRiskBand(decimal score)
    {
        if (score >= 80m)
        {
            return "High";
        }

        if (score >= 50m)
        {
            return "Medium";
        }

        return "Low";
    }
}

Once you store these values, the database can index them directly. That is the real shift. Instead of asking the engine to compute urgency on every candidate row, you let it seek on IsUrgent or RiskBand. That changes both performance and plan quality.

Technique five, flattened search columns

Search is a classic source of accidental complexity. Users want one box. They expect it to match case number, broker name, customer name, postcode, product, maybe even a phone number or note. A normalised model turns that into a wide OR condition with several joins, or pushes the team into adding a separate search engine earlier than they really need one.

A useful middle ground is to denormalise search into a dedicated row with flattened searchable fields.

public sealed class SubmissionSearchRow
{
    public Guid SubmissionId { get; set; }
    public string SubmissionNumber { get; set; } = string.Empty;
    public string BrokerName { get; set; } = string.Empty;
    public string InsuredName { get; set; } = string.Empty;
    public string Postcode { get; set; } = string.Empty;
    public string ProductName { get; set; } = string.Empty;
    public string SearchText { get; set; } = string.Empty;
    public DateTime CreatedUtc { get; set; }
}

A projector builds the flattened text.

public sealed class SubmissionSearchProjector
{
    private readonly UnderwritingDbContext _db;

    public SubmissionSearchProjector(UnderwritingDbContext db)
    {
        _db = db;
    }

    public async Task RebuildAsync(Guid submissionId, CancellationToken stopToken)
    {
        var source = await _db.Submissions
            .Where(x => x.Id == submissionId)
            .Select(x => new
            {
                x.Id,
                x.SubmissionNumber,
                x.BrokerNameSnapshot,
                x.InsuredName,
                x.Postcode,
                x.ProductNameSnapshot,
                x.CreatedUtc
            })
            .SingleAsync(stopToken);

        var searchText = string.Join(' ',
            source.SubmissionNumber,
            source.BrokerNameSnapshot,
            source.InsuredName,
            source.Postcode,
            source.ProductNameSnapshot)
            .ToLowerInvariant();

        var row = await _db.SubmissionSearchRows.FindAsync(new object[] { submissionId }, stopToken);

        if (row is null)
        {
            row = new SubmissionSearchRow { SubmissionId = submissionId };
            _db.SubmissionSearchRows.Add(row);
        }

        row.SubmissionNumber = source.SubmissionNumber;
        row.BrokerName = source.BrokerNameSnapshot;
        row.InsuredName = source.InsuredName;
        row.Postcode = source.Postcode;
        row.ProductName = source.ProductNameSnapshot;
        row.CreatedUtc = source.CreatedUtc;
        row.SearchText = searchText;

        await _db.SaveChangesAsync(stopToken);
    }
}

This is not a replacement for a true search platform in every case. It is a practical step that often solves internal search needs very well and removes painful joins from the request path.

Technique six, compiled JSON payloads

Some read paths are expensive not because the data is hard to query, but because the response is expensive to build. Configuration payloads, product catalogues, pricing rules, feature flag definitions, and reference datasets often fall into this category. The source data may be split across multiple tables and the application may have to turn it into a nested object model before serialising it.

If the payload changes relatively rarely and is read heavily, compile it once and store the result.

public sealed class CompiledProductConfig
{
    public Guid ProductId { get; set; }
    public string Version { get; set; } = string.Empty;
    public string JsonPayload { get; set; } = string.Empty;
    public DateTime CompiledUtc { get; set; }
}

A compiler service generates the payload after changes.

public sealed class ProductConfigCompiler
{
    private readonly ProductDbContext _db;

    public ProductConfigCompiler(ProductDbContext db)
    {
        _db = db;
    }

    public async Task CompileAsync(Guid productId, CancellationToken stopToken)
    {
        var product = await _db.Products
            .Where(x => x.Id == productId)
            .Select(x => new
            {
                x.Id,
                x.Name,
                Rules = x.Rules
                    .OrderBy(r => r.Priority)
                    .Select(r => new
                    {
                        r.Key,
                        r.Operator,
                        r.Value
                    })
                    .ToList()
            })
            .SingleAsync(stopToken);

        var payload = JsonSerializer.Serialize(product);

        var row = await _db.CompiledProductConfigs.FindAsync(new object[] { productId }, stopToken);

        if (row is null)
        {
            row = new CompiledProductConfig { ProductId = productId };
            _db.CompiledProductConfigs.Add(row);
        }

        row.Version = Guid.NewGuid().ToString("N");
        row.JsonPayload = payload;
        row.CompiledUtc = DateTime.UtcNow;

        await _db.SaveChangesAsync(stopToken);
    }
}

The request path then becomes almost trivial.

public sealed class ProductConfigService
{
    private readonly ProductDbContext _db;

    public ProductConfigService(ProductDbContext db)
    {
        _db = db;
    }

    public async Task<string?> GetCompiledJsonAsync(Guid productId, CancellationToken stopToken)
    {
        return await _db.CompiledProductConfigs
            .AsNoTracking()
            .Where(x => x.ProductId == productId)
            .Select(x => x.JsonPayload)
            .SingleOrDefaultAsync(stopToken);
    }
}

This is denormalisation at the payload level. It is blunt, and when used in the right place it is extremely effective. You remove query assembly and serialisation work from the hot path entirely.

How to implement denormalisation cleanly in .NET

The biggest risk with denormalisation is not duplication. It is ambiguity. If nobody can tell which model is canonical, which values are derived, how freshness works, and how to repair drift, the design will decay.

A clean .NET implementation usually has four parts.

You keep the canonical write model explicit. This is the model the business truly owns.

You emit an outbox event or domain event when the source data changes.

You project that event into one or more read models in a background process.

You make the read endpoints talk to the read models directly, without trying to reconstruct the domain again.

A simple hosted service can handle the projection side. In larger systems that may be a separate worker or Azure Function. The important part is not the hosting model. The important part is that the projection is idempotent and observable.

public sealed class SubmissionProjectionWorker : BackgroundService
{
    private readonly IServiceScopeFactory _scopeFactory;
    private readonly ILogger<SubmissionProjectionWorker> _logger;

    public SubmissionProjectionWorker(
        IServiceScopeFactory scopeFactory,
        ILogger<SubmissionProjectionWorker> logger)
    {
        _scopeFactory = scopeFactory;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stopToken)
    {
        while (!stopToken.IsCancellationRequested)
        {
            using var scope = _scopeFactory.CreateScope();
            var db = scope.ServiceProvider.GetRequiredService<UnderwritingDbContext>();
            var projector = scope.ServiceProvider.GetRequiredService<SubmissionProjector>();

            var batch = await db.OutboxMessages
                .Where(x => x.ProcessedUtc == null && x.Type == "SubmissionChanged")
                .OrderBy(x => x.OccurredUtc)
                .Take(100)
                .ToListAsync(stopToken);

            if (batch.Count == 0)
            {
                await Task.Delay(TimeSpan.FromSeconds(1), stopToken);
                continue;
            }

            foreach (var message in batch)
            {
                try
                {
                    await projector.ProjectAsync(message.AggregateId, stopToken);
                    message.ProcessedUtc = DateTime.UtcNow;
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex, "Failed to project submission {SubmissionId}", message.AggregateId);
                }
            }

            await db.SaveChangesAsync(stopToken);
        }
    }
}

That worker does not need to be clever. It needs to be reliable. Projection code should be deterministic, repeatable, and easy to rebuild.

This pattern works because it gives you separation. Writes preserve truth. Projections shape data for speed. Reads stay lean.

Measuring the gains properly

If you denormalise without measuring, you are guessing. Sometimes the guess is right, but expert engineering means proving it.

Do not just measure average response time. That hides a lot. You want median, p95, and p99. You want database CPU and logical reads. You want application allocations. You want throughput under concurrency. You want to know whether you made writes slightly heavier and whether that matters.

The pattern to look for is simple. If the old design has acceptable median latency but poor p95 and p99, denormalisation usually helps a lot because it simplifies the work and stabilises the plan. If the old design burns database CPU and application allocations on every request, denormalisation usually helps there too. In C# terms, you should benchmark at three levels. Measure the database query cost. Measure the endpoint under realistic concurrency. Measure the shaping cost in process if serialisation or object mapping is part of the problem. A single stopwatch around an API call is not enough. Its common to see a read endpoint go from well over one hundred milliseconds to below twenty once it moves from reconstruction to direct lookup. More importantly, the tail often tightens dramatically. The endpoint stops having bad days.

That is a stronger win than a modest median improvement because production pain lives in the tail.

The trade offs you must own

Denormalisation works because it changes where the work happens. That means the cost does not disappear. It moves.

Writes may become heavier because you now update projections or summary fields.

You may accept eventual consistency if projections run asynchronously.

You add more moving parts, especially if you use outbox processing and background workers.

You need rebuild and reconciliation tooling because projections can drift if there is a bug.

None of those are reasons to avoid denormalisation. They are reasons to design it properly.

The important discipline is to be explicit. Name read models as read models. Keep projection logic out of the domain core where possible. Decide which values must be transactionally current and which values can lag slightly. Build replays or rebuild jobs so you can recover from bad logic. Make it obvious to every engineer which table tells the truth and which table exists for speed.

If you fail to do that, denormalisation becomes accidental duplication. That is where teams get burned.

When not to denormalise

Do not denormalise because a query feels ugly. Ugly code is not always expensive code.

Do not denormalise values you cannot clearly derive and refresh.

Do not duplicate fields with no owner and no repair story.

Do not turn projections into hidden sources of truth.

Do not assume eventual consistency is always harmless. A stale dashboard count is one thing. A stale available credit decision is another.

Do not denormalise everything. Most systems only need it in a few hot places. If you apply it everywhere, you increase complexity without improving the parts that matter.

The expert judgement is knowing where the hot paths really are and shaping only those.

Denormalisation is one of the few performance techniques that changes the shape of the problem instead of merely tuning around it. Indexes, cache layers, and ORM tweaks all matter, but they mostly help you execute the same work more efficiently. Denormalisation asks a better question. Should the system be doing this work on every read at all.

In many serious C# systems, the honest answer is no.

If the application already knows a balance, a count, a risk band, a queue shape, a search row, or a compiled payload, and if that value is read far more often than it changes, storing it in the form the read path needs is not a compromise. It is good engineering.

The strongest systems keep their transactional core clean and truthful. Then they build denormalised shapes around that core for speed. They measure the gains. They own the trade offs. They keep projections rebuildable. They keep the source of truth clear. That is how you get fast systems without losing control of the design. If you want real performance gains from denormalisation, do not think of it as breaking the rules. Think of it as moving work to the cheapest place in the system. When you do that deliberately, your database stops reconstructing the obvious, your APIs stop carrying unnecessary weight, and your read path starts behaving like it was designed for production rather than for a whiteboard.

In real systems, compression is a transport concern with application-level consequences. It affects bandwidth, CPU, latency, caching behaviour, security posture, and even how you shape your contracts. ASP.NET Core gives you built-in middleware for response compression and request decompression, but the framework does not make the architectural decisions for you. You still need to decide what to compress, where to compress it, when to reject it, and how to avoid using compression as a bandage for bad API design. The current ASP.NET Core guidance is still clear on the fundamentals, use compression to reduce payload size, prefer server or proxy compression where available, and use the built-in middleware when Kestrel or HTTP.sys are serving the app directly because they do not provide built-in compression themselves.

The first mistake Developers make is thinking compression is mainly about speed. It is really about trade-offs. Compression reduces bytes on the wire, which often improves responsiveness, especially for JSON and other text-heavy payloads. At the same time, it costs CPU to compress and decompress data. That trade-off is usually favourable for medium and large JSON responses over public networks, but not always favourable for tiny payloads or already-compressed binary content. That is why serious API design starts with payload shape first and compression second. If your endpoint returns bloated documents with duplicated fields, unnecessary nesting, and data the caller never asked for, compression will help, but only after you already lost the bigger battle. Microsoft’s guidance frames compression as a way to reduce response size and improve responsiveness, not as a replacement for lean responses.

A useful mental model is to separate outbound compression from inbound decompression. Outbound compression is the default case. Your API produces JSON, problem details, text, CSV, or other compressible formats, and the client advertises supported encodings through Accept-Encoding. The response compression middleware examines the request and response, selects a provider such as Brotli or gzip, and writes the compressed payload if the response type is eligible. Inbound decompression is different. There, the client sends a compressed request body and marks it with Content-Encoding, and the request decompression middleware unwraps it before model binding or request body reading happens. ASP.NET Core supports both directions, but they solve different problems and they should not be enabled with the same level of enthusiasm. Response compression is broadly useful. Request decompression is useful only when clients are actually sending large compressed payloads, typically large JSON, text, or similar upload bodies.

In practice, the best default for a modern ASP.NET Core API is straightforward. Compress responses that are actually compressible. Prefer Brotli when the client supports it. Fall back to gzip for compatibility. Leave already-compressed formats alone. If you are running behind IIS, Apache, or Nginx, prefer server-based compression because Microsoft explicitly notes that server modules generally outperform the ASP.NET Core middleware. If you are serving directly from Kestrel or HTTP.sys, use the middleware because those servers do not currently offer built-in compression support.

The second mistake Developers make is compressing everything indiscriminately. Compression is not magic. It works best on text-heavy formats because they contain repeating structure. JSON is the classic win because property names, quotes, punctuation, and repeated values compress well. XML, HTML, CSS, JavaScript, CSV, plain text, and problem details are all strong candidates. JPEG, PNG, MP4, ZIP, and many other binary formats are not. Recompressing data that is already compressed often gives you negligible size reduction and unnecessary CPU overhead. This is exactly why the ASP.NET Core response compression middleware is configured around MIME types. You tell it what content types are eligible instead of asking it to blindly compress whatever leaves the process.

Here is a production-friendly baseline for a .NET API using minimal APIs. It enables response compression, explicitly adds Brotli and gzip, includes JSON-related MIME types, and sets both providers to Fastest because API latency usually matters more than squeezing out the very last percentage point of compression ratio.

using Microsoft.AspNetCore.RequestDecompression;
using Microsoft.AspNetCore.ResponseCompression;
using System.IO.Compression;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddResponseCompression(options =>
{
    options.EnableForHttps = true;

    options.Providers.Add<BrotliCompressionProvider>();
    options.Providers.Add<GzipCompressionProvider>();

    options.MimeTypes = ResponseCompressionDefaults.MimeTypes.Concat(new[]
    {
        "application/json",
        "application/problem+json",
        "text/plain",
        "text/csv"
    });
});

builder.Services.Configure<BrotliCompressionProviderOptions>(options =>
{
    options.Level = CompressionLevel.Fastest;
});

builder.Services.Configure<GzipCompressionProviderOptions>(options =>
{
    options.Level = CompressionLevel.Fastest;
});

builder.Services.AddRequestDecompression();

var app = builder.Build();

app.UseRequestDecompression();
app.UseResponseCompression();

app.MapGet("/api/orders/{id:int}", (int id) =>
{
    var response = new
    {
        Id = id,
        Customer = "ACME Insurance",
        Lines = Enumerable.Range(1, 250).Select(i => new
        {
            LineNumber = i,
            Sku = $"SKU-{i:0000}",
            Quantity = i % 5 + 1,
            Price = 49.99m + i
        })
    };

    return Results.Json(response);
});

app.Run();

This gives you the right starting point, but serious systems usually need more discipline than a baseline setup. One example is compression level. Many developers instinctively choose Optimal, assuming it must be better because the name sounds better. That is too simplistic. In APIs, especially low-latency APIs, Fastest is often the better operational choice because it cuts CPU cost and still captures most of the size reduction on JSON. Optimal can make sense for larger batch-style responses or download scenarios where throughput matters more than raw request latency. The right answer is not theoretical. Benchmark it with your own payloads and concurrency profile.

A useful way to think about the pipeline is this:

Another place where Devlopers get sloppy is HTTPS compression. ASP.NET Core exposes EnableForHttps, and the documented default is false. Microsoft also warns that enabling compression for HTTPS responses containing remotely manipulable content may expose security problems. That warning exists because compression can become part of a side-channel when attacker-controlled input and secret-bearing content share the same compressed response. In normal internal or line-of-business APIs, many people still enable HTTPS compression because the benefits are real and the attack surface may be limited, but that decision should be deliberate. If you reflect attacker-supplied content into a response that also carries secrets, tokens, or sensitive dynamic values, do not just enable HTTPS compression and forget about it. Understand what is actually in those responses.

Request decompression deserves even more caution. The feature is real and useful, but it is not something to switch on simply because the middleware exists. The request decompression middleware automatically inspects Content-Encoding and decompresses supported request bodies, which saves you from writing custom request-body handling code. That part is good. The hard part is operational safety. Inbound compressed payloads shift CPU work onto your servers and can amplify resource consumption if abused. If you accept large compressed uploads, you should pair that with request size limits, timeout controls, careful endpoint scoping, and monitoring. The middleware also needs to run before anything reads the body, otherwise you are too late.

A targeted inbound example looks like this:

app.UseRequestDecompression();

app.MapPost("/api/import/products", async (HttpContext httpContext) =>
{
    httpContext.Features.Get<IHttpMaxRequestBodySizeFeature>()?.DisableMaxRequestBodySize();

    using var reader = new StreamReader(httpContext.Request.Body);
    var json = await reader.ReadToEndAsync();

    return Results.Ok(new
    {
        Message = "Compressed request accepted",
        Characters = json.Length
    });
});

app.Run();

That sample shows the mechanics, but the operational point matters more than the syntax. You should not enable inbound decompression across every endpoint unless the endpoints really need it. A typical CRUD API that accepts small POST and PUT bodies gets little benefit from compressed requests. A bulk import endpoint that accepts a multi-megabyte JSON document might benefit a lot.

You should also think about where compression belongs in a broader deployment. If you are behind Nginx or IIS, server-side compression at the edge is often the better place for outbound response compression because it takes work off the app and can be tuned centrally. Microsoft’s guidance says exactly that, noting that the performance of the ASP.NET Core middleware probably will not match dedicated server modules. That does not make middleware wrong. It just means you should not ignore the reverse proxy when you have one. If you already terminate traffic behind a capable gateway, that is often the best place to handle compression consistently.

Caching behaviour is another area where compression changes system behaviour more than people expect. Once you serve multiple encoded versions of the same representation, the cache key must vary by encoding. That is why compressed responses are tied to Accept-Encoding, and why intermediaries need to treat the compressed and uncompressed versions as distinct representations. If you run API caching, CDN caching, or reverse-proxy caching, compression is no longer just a transport tweak. It becomes part of representation management. That matters even more if you also use ETags. In a well-behaved system, you need consistency in how representations are generated and validated, especially if compression is handled at the proxy layer instead of the app layer.

Another point is that compression interacts with streaming. If your endpoint sends buffered JSON in one shot, compression is easy. If you are sending data progressively, such as large streamed responses, NDJSON, SSE-style traffic, or anything latency-sensitive where flushing behavior matters, compression may introduce buffering or delivery characteristics that work against the protocol. In those cases, the question is not just "can I compress this?" but "does compression preserve the delivery behaviour I actually want?" For real-time or progressive-delivery endpoints, the right answer is often endpoint-specific rather than global.

The security and resilience side should not be ignored either. Kestrel exposes minimum request and response data rate limits, and the documented defaults are 240 bytes per second with a 5 second grace period. That matters because slow clients, large request bodies, and decompression work can combine into unpleasant failure modes if you do not have sane guardrails. Kestrel also supports request header timeouts and request body size limits, and those settings should be part of your overall posture when you accept uploads or large bodies, compressed or otherwise. Compression is not an isolated tuning knob. It sits inside your wider transport hardening model.

Here is a fuller example with Kestrel limits and explicit compression setup:

using Microsoft.AspNetCore.ResponseCompression;
using System.IO.Compression;

var builder = WebApplication.CreateBuilder(args);

builder.WebHost.ConfigureKestrel(options =>
{
    options.Limits.RequestHeadersTimeout = TimeSpan.FromSeconds(30);
    options.Limits.MinRequestBodyDataRate = new(bytesPerSecond: 240, gracePeriod: TimeSpan.FromSeconds(5));
    options.Limits.MinResponseDataRate = new(bytesPerSecond: 240, gracePeriod: TimeSpan.FromSeconds(5));
    options.Limits.MaxRequestBodySize = 20 * 1024 * 1024; // 20 MB
});

builder.Services.AddResponseCompression(options =>
{
    options.EnableForHttps = true;
    options.Providers.Add<BrotliCompressionProvider>();
    options.Providers.Add<GzipCompressionProvider>();

    options.MimeTypes = ResponseCompressionDefaults.MimeTypes.Concat(new[]
    {
        "application/json",
        "application/problem+json"
    });
});

builder.Services.Configure<BrotliCompressionProviderOptions>(options =>
{
    options.Level = CompressionLevel.Fastest;
});

builder.Services.Configure<GzipCompressionProviderOptions>(options =>
{
    options.Level = CompressionLevel.Fastest;
});

var app = builder.Build();

app.UseResponseCompression();

app.MapGet("/api/report", () =>
{
    var report = Enumerable.Range(1, 10_000).Select(i => new
    {
        Id = i,
        Name = $"Item {i}",
        Status = i % 3 == 0 ? "Pending" : "Complete",
        Timestamp = DateTime.UtcNow.AddMinutes(-i)
    });

    return Results.Json(report);
});

app.Run();

That is the kind of configuration that belongs in a serious service. It does not just say "turn compression on." It defines the transport assumptions that go with it.

There is also a design lesson here for internal APIs and service-to-service calls. Developers sometimes assume compression matters only for internet-facing traffic. That is not always true. In cloud environments, especially across regions, VNets, or heavily loaded east-west traffic paths, payload size still matters. Compressing large JSON documents between services can reduce network cost and improve throughput. The catch is that the CPU trade-off now happens on your own estate at scale. If a service is already CPU-bound, compression can make it worse. If the network is the bottleneck, compression can help a lot. Again, this is why you benchmark real workloads instead of arguing from instinct.

If you want a clean set of rules that hold up in practice, they are these. Shape payloads properly first. Compress text-heavy responses by default. Prefer Brotli with gzip fallback. Leave already-compressed binaries alone. Use request decompression only for endpoints that genuinely need it. Prefer edge or proxy compression when your hosting stack supports it well. Treat HTTPS compression as a conscious security decision, not a default checkbox. Add limits and timeouts when you accept large request bodies. Measure the CPU and latency profile under realistic traffic before you call the job done. Those rules are not glamorous, but they are what separate a neat code sample from a production-grade API.

The big point is simple. Payload compression in ASP.NET Core is not a trick. It is part of transport engineering. When you treat it that way, the implementation becomes clearer. You stop asking whether you should "turn on gzip" and start asking the questions that actually matter: where should compression happen, which representations benefit, what security caveats apply, what limits protect the server, and whether your payloads deserved to be that large in the first place.

That's what serious API payload compression looks like in modern .NET. It's not complicated, but it does require intent.

https://learn.microsoft.com/en-us/aspnet/core/performance/response-compression?view=aspnetcore-10.0&utm_source=chatgpt.com

https://learn.microsoft.com/en-us/aspnet/core/fundamentals/middleware/request-decompression?view=aspnetcore-10.0&utm_source=chatgpt.com

This is the part many Developers underestimate. They can design a clean domain model, expose a tidy API, and even get the happy path flowing nicely in development. Then integration begins. Payments have retry semantics you do not control. Fraud services throttle under burst load. ERP platforms respond eventually, but only after translating your request into formats that feel like they were invented in another decade. Humans approve things late, suppliers call back twice, webhooks arrive out of order, and support teams need answers while the workflow is still in flight. None of those problems are unusual. They are the normal operating environment of serious enterprise systems.

That reality creates what is best described as an integration tax. Every dependency adds latency, risk, state mismatch, and behavioural quirks. Every new handoff expands the number of ways a process can stall or become inconsistent. This tax cannot be avoided. If your system has to interact with payment providers, CRM tools, ERP platforms, shipping carriers, external risk engines, old databases, partner APIs, or human approvers, then complexity is already part of the deal. The real question is whether that complexity is handled intentionally or left to leak through the architecture.

The good news is that the same failure shapes show up again and again. Systems struggle with overload, duplicate work, half-completed transactions, tight coupling, invisible state, and awkward coexistence between new platforms and old ones. Once you see those patterns clearly, the architecture becomes much easier to reason about. Durable orchestration platforms such as Azure Durable Functions are especially useful here because they provide a strong set of building blocks for stateful workflows, retries, timers, external events, and long-running coordination. But the bigger lesson is not tied to one platform. The patterns in this article apply whether you are orchestrating with Durable Functions, Temporal, Step Functions, Camunda, MassTransit sagas, or even a carefully designed internal workflow engine.

This article takes those recurring resilience and integration problems and turns them into a practical operating model. We will look at circuit breakers, idempotency, compensation, event-driven handoffs, workflow status, hybrid architecture, and resilience-first design. The goal is not to repeat a chapter from a book. The goal is to turn those ideas into a standalone guide for engineers and architects building systems that need to survive the real world.

Why Integration Gets Harder at Scale

A single integration in a low-volume system is often manageable with little more than an HTTP client, a timeout, and a retry policy. That is why many systems look fine during the first release. The real trouble appears later, once transaction volume grows, external dependencies multiply, and the business starts relying on workflows that stretch across multiple bounded contexts.

At that point, latency stops being an isolated technical concern and starts shaping business outcomes. A fraud service that takes three seconds instead of two might not sound catastrophic, but if that call sits in the middle of a checkout flow, the extra second now becomes customer friction. Multiply that by retries, duplicate callbacks, rate limiting, and a few downstream dependencies, and what looked like a simple workflow becomes a slow-motion queueing problem. Enterprise systems rarely collapse in one dramatic moment. More often, they drown gradually in coordination overhead.

Another issue is failure diversity. Internal services often fail in relatively predictable ways because the same teams own the deployment model, monitoring stack, and operational practices. External systems are different. One dependency might fail fast with clear error codes. Another might hang without responding. Another might accept the request but finish it later. Another might partially succeed and provide no clean rollback. Legacy platforms are especially problematic because they often expose interfaces that were never designed for modern reliability expectations, yet still sit on the critical path of important business processes.

Human interaction adds another layer of uncertainty. Approvals, escalations, document review, manual intervention, and exception handling all introduce variable time windows that cannot be compressed by throwing more CPU at the problem. A workflow might be technically healthy but still paused for six hours waiting on someone in a different department. If the system does not model that state explicitly, operators end up guessing whether it is broken or simply waiting.

This is why mature integration architecture is less about making every dependency perfect and more about building a workflow that can absorb imperfect behaviour. You are not designing for a world where all systems are reliable. You are designing for a world where some systems are slow, some are inconsistent, some are overloaded, and some are still useful enough that the business cannot function without them.

The Core Idea: Resilience Is a Workflow Concern

Developers think about resilience at the level of individual service calls. They add retries to HTTP clients, configure exponential backoff, maybe wrap a few dependencies in a circuit breaker, and consider the job largely done. That helps, but it is not enough. In distributed systems, resilience is rarely just a call-level concern. It is a workflow concern.

A payment retry is not just a payment retry. It is part of a broader transaction that may also reserve inventory, create an order record, notify a customer, update a loyalty profile, and send data into finance systems. A supplier callback is not just an inbound event. It affects which timer should be cancelled, what status should be shown to support, and whether the workflow can proceed to the next stage. A human approval is not just a pause in processing. It changes how you monitor state, set expectations, and decide when intervention is needed.

This is where orchestration platforms earn their keep. They provide a durable memory of the workflow so that retries, waiting, state transitions, and external signals are modelled as first-class behaviour instead of being spread across controller methods, background jobs, and database flags. That durable state is not just useful for implementation. It also creates a place where resilience patterns can be applied consistently.

The rest of this article focuses on those patterns.

Pattern 1: Circuit Breakers Prevent a Bad Dependency from Taking the Workflow Down with It

One of the most common mistakes in integration-heavy systems is treating every failure as a reason to retry harder. That instinct is understandable. Retries solve a lot of transient faults, especially network blips, brief throttling, and short-lived platform issues. The problem is that retries are not free. When a downstream service is genuinely unhealthy, repeated retries can amplify the damage by increasing traffic against a struggling dependency and consuming resources in your own system while little useful work gets done.

That is why circuit breakers matter. A circuit breaker watches failure behaviour over time. If failures cross a threshold, the breaker opens and temporarily blocks new requests to the dependency. Rather than continuing to hammer a service that is already in trouble, the workflow fails fast or routes into a fallback path. After a cooldown period, the breaker can move into a half-open state and allow limited traffic to test whether the downstream system has recovered.

In a long-running workflow, this pattern is especially valuable because it prevents an unhealthy external service from dragging large volumes of orchestration instances into pointless retry loops. Imagine an order pipeline that calls an external fraud scoring API before taking payment. If that provider is returning 500 errors for ten minutes, the wrong response is to let every new order attempt the call repeatedly until the orchestration backlog expands and customer-facing latency spikes. A better response is to trip the breaker, fail new attempts quickly with a clear status, and alert operators that the fraud dependency is down.

A simplified view looks like this:

In Durable Functions, one practical implementation is to use a Durable Entity to hold breaker state for a dependency. The entity can track consecutive failures, the time the breaker opened, and whether a call is allowed. Each orchestration or activity checks the entity before making the dependency call. That gives you a central, durable place to enforce the breaker rather than leaving each workflow instance to make its own isolated decision.

A stripped-back example might look like this in C#:

public record CircuitBreakerState(
    int ConsecutiveFailures,
    DateTime? OpenedAtUtc,
    bool IsOpen);

public class FraudServiceBreakerEntity
{
        public CircuitBreakerState State { get; set; } = new(0, null, false);
        public bool CanExecute(DateTime nowUtc)
        {
            if (!State.IsOpen)
                return true; 

            var cooldown = TimeSpan.FromMinutes(2);

            return State.OpenedAtUtc is { } openedAt && nowUtc - openedAt >= cooldown;

         }

        public void RecordSuccess()
        {
            State = new CircuitBreakerState(0, null, false);
        }

        public void RecordFailure(DateTime nowUtc)
        {
            var failures = State.ConsecutiveFailures + 1;
            if (failures >= 5)
            {
                State = new CircuitBreakerState(failures, nowUtc, true);
                return;
            }

        State = new CircuitBreakerState(failures, State.OpenedAtUtc, false);

    }

}

The important part is not the code. It is the operational behaviour. Once the breaker opens, you stop turning a bad dependency into a system-wide slowdown. You make the failure explicit, measurable, and bounded.

That said, circuit breakers are not magic. They must be tuned carefully. Thresholds that are too aggressive can block useful traffic. Cooldowns that are too long can delay recovery. Breakers also need observability. If the team cannot see when they open, why they opened, and how often they are being exercised, they become another hidden state machine nobody trusts during an incident.

Pattern 2: Idempotency Turns Retries from a Risk into a Safety Net

If you work on distributed systems long enough, you stop asking whether duplicate requests will happen and start asking where they will happen first. Retries from clients, retries from orchestrators, webhook replays, queue redelivery, supplier callbacks, double clicks from users, and timeouts that hide already-completed work all create duplicate execution paths. If your system is not designed for that, it will eventually perform the same side effect twice.

That is where idempotency becomes non-negotiable. An idempotent operation can be executed multiple times with the same logical input and still produce the same final outcome. This does not mean every call is naturally idempotent. It means the system is built so that repeated attempts are recognised and handled safely.

Payment flows are the classic example. If a payment service receives the same charge request twice because the first response timed out, the customer must not be charged twice. The standard approach is to send an idempotency key, often the order ID or payment request ID, with the outbound call. The payment provider stores the first result for that key and returns the same outcome for later retries instead of executing a second charge.

But idempotency belongs far beyond payments. ERP submission endpoints should reject duplicate order registration for the same business reference. Customer reward updates should not apply points twice. Shipping requests should not create duplicate consignments. Inventory allocation should not reserve the same units repeatedly because a callback was delivered more than once.

Here is the shape of the idea:

In your own services, the idempotency mechanism often comes down to a durable write model. You persist a unique business operation key before or alongside the side effect, and later requests with the same key return the stored outcome. Sometimes that means a dedicated idempotency table. Sometimes it means a natural domain guard such as a unique constraint on an external reference. Sometimes it means tracking processed event IDs in an entity or aggregate.

A simple service-side pattern in C# could look like this:

public sealed class ProcessedRequest
{
    public string RequestId { get; init; } = default!;
    public string ResultJson { get; init; } = default!;
    public DateTime ProcessedAtUtc { get; init; }
}

public async Task<PaymentResult> ChargeAsync(
    string requestId, decimal amount, cancellationToken stopToken)
{
    var existing = await db.ProcessedRequests
 .SingleOrDefaultAsync(x => x.RequestId == requestId, stopToken);

    if (existing is not null)
        return JsonSerializer.Deserialize<PaymentResult>(existing.ResultJson)!;

    var result = await gateway.ChargeAsync(amount, stopToken);

    db.ProcessedRequests.Add(new ProcessedRequest
    {
        RequestId = requestId,
        ResultJson = JsonSerializer.Serialize(result),
        ProcessedAtUtc = DateTime.UtcNow
    });

    await db.SaveChangesAsync(stopToken);
    return result;
}

The hard part is deciding the correct scope of the idempotency key. If it is too broad, distinct operations can accidentally collapse into one. If it is too narrow, duplicates slip through. Good idempotency design requires a clear understanding of the business operation, not just the transport request.

It is also worth being blunt about this: retries without idempotency are reckless. They create the appearance of resilience while quietly shifting the cost onto customers, finance teams, and support operations. Once you understand that, idempotency stops feeling like a technical detail and starts feeling like table stakes.

Pattern 3: Compensation Is How You Survive Without Distributed Transactions

Enterprise workflows almost always cross boundaries where a single atomic transaction is impossible. You might charge a card in one system, reserve inventory in another, create a shipment in a third, and register the order in an ERP platform that still thinks SOAP is modern. No transaction coordinator is going to make all of that commit or roll back as one neat unit. Even if it could, you probably would not want the coupling and latency that came with it.

So what happens when part of the workflow succeeds and a later step fails? That is where compensation comes in. Compensation is the deliberate reversal of already-completed actions so that the broader workflow returns to a consistent business state.

Suppose a checkout flow successfully charges the customer, then later fails to allocate stock. Without compensation, the system has taken money for an order it cannot fulfil. That is not a mere technical defect. It is a business failure. The workflow needs a compensating action, such as issuing a refund, releasing provisional customer benefits, and notifying operations if manual review is required.

The same applies in other domains. If a claims workflow opens a financial reserve and later discovers a validation failure, the reserve may need reversing. If an onboarding workflow provisions downstream access and then fails a compliance check, those accounts may need disabling. If a shipping request is accepted and the ERP later rejects the order, logistics and customer communication may both need corrective action.

A compensation flow often looks like this:

Compensation is frequently misunderstood as just calling an undo API. Sometimes that is possible, but often it is not. Real compensations can be asynchronous, partial, or manual. A refund might take time. A shipment might be cancellable only before handoff to the carrier. A legacy platform might support reversal only through an overnight batch. That means compensation needs its own design, status tracking, and operational visibility.

In orchestrated systems, a common pattern is to record which forward steps have completed, then execute compensations in reverse order if the workflow later fails. Durable Functions makes this practical because orchestration state can keep track of what has happened so far.

A simplified orchestration sketch might look like this:

[Function(nameof(ProcessOrderOrchestrator))]

public static async Task Run(
 [OrchestrationTrigger] TaskOrchestrationContext context)
{
    var order = context.GetInput<OrderRequest>();
    var completedSteps = new List<string>();

    try
    {
        await context.CallActivityAsync(nameof(ReserveInventoryActivity), order);

        completedSteps.Add("inventory");
        await context.CallActivityAsync(nameof(ChargePaymentActivity), order);

        completedSteps.Add("payment");
        await context.CallActivityAsync(nameof(RegisterOrderInErpActivity), order);

        completedSteps.Add("erp");
    }
    catch (Exception)
    {
        if (completedSteps.Contains("payment"))
        {
            await context.CallActivityAsync(nameof(RefundPaymentActivity), order);
        }

        if (completedSteps.Contains("inventory"))
        {
            await context.CallActivityAsync(nameof(ReleaseInventoryActivity), order);
        }

        await context.CallActivityAsync(nameof(RaiseOpsAlertActivity), order.OrderId);

        throw;
     }
}

This is deliberately simple, but it makes the main point. Compensation is not an optional extra you add later. It is part of the workflow contract. If a business process can leave the world half-changed, then it also needs a defined path to recover from that condition.

There is another important truth here. Compensation is rarely perfect. You should not promise exact rollback semantics where the domain does not support them. Some workflows are better described as eventually corrected rather than fully undone. That is fine, provided the state transitions are explicit and visible. False certainty is more dangerous than honest eventual consistency.

Pattern 4: Event-Driven Integration Reduces Coupling and Preserves Flow

One of the easiest ways to make orchestration brittle is to let the central workflow call every downstream system directly. It feels simple at first because all the logic is in one place. The orchestrator confirms the order, then calls the ERP, then calls analytics, then calls CRM, then calls some downstream fulfilment component, then maybe calls a notification service. The problem is that each of those direct calls adds latency and dependency pressure to the core flow.

A better option in many cases is to separate the business milestone from the downstream reactions. Once the workflow reaches a meaningful state, such as order confirmed, claim submitted, policy approved, or onboarding completed, it can publish an event. Other systems subscribe independently and handle their own processing. That removes non-essential side effects from the critical path and reduces direct coupling between the orchestrator and every consumer.

Here is the contrast:

This shift matters for several reasons. First, it shortens the synchronous path of the core workflow. Second, it allows new consumers to be added later without modifying the orchestrator. Third, it isolates failure. If analytics is down, that should not usually block order confirmation. If CRM processing is delayed, the business milestone may still be valid.

That does not mean direct calls disappear entirely. Some steps remain essential to the transaction outcome and must stay in the workflow. Payment authorisation is usually not optional. Inventory reservation is often not optional. But secondary reactions are usually better handled as event subscribers.

In Azure, this might mean an orchestration step publishes an OrderConfirmed event into Event Grid or a queue topic after core invariants are satisfied. Separate Functions then react to that event and perform ERP synchronisation, customer communications, and reporting updates. In other stacks, the same pattern could use Kafka, RabbitMQ, SNS/SQS, NATS, or any eventing platform with durable delivery.

A typical event contract should be boring and explicit. That is a good thing. It might include a business ID, event type, timestamp, correlation ID, schema version, and only the data consumers genuinely need. Resist the urge to publish an anemic dump of internal objects. Events are integration contracts, not convenient serialisation shortcuts.

A simple event model could look like this:

public sealed record OrderConfirmedEvent(
    string OrderId,
    string CustomerId,
    decimal Total,
    DateTime ConfirmedAtUtc,
    string CorrelationId,
    int SchemaVersion);

There is a trade-off, of course. Event-driven systems push you toward eventual consistency. Consumers may process at different times. Delivery may be at least once, not exactly once. That takes us right back to idempotency and observability. Event-driven integration works well when paired with those patterns, not when treated as a shortcut that somehow removes the need for them.

Pattern 5: Custom Status and Observability Keep Workflows from Becoming Black Boxes

Many operational incidents are not caused by the workflow being broken. They are caused by nobody being able to tell what the workflow is doing. A long-running integration process can be perfectly healthy while waiting on a supplier response, a human approval, or an overnight ERP batch. Without good status signals, support teams often interpret waiting as failure and failure as waiting. That confusion creates noise, escalations, and manual work that should never have existed.

The fix is simple in principle and often neglected in practice. Long-running workflows need explicit, queryable status. Not vague technical state, but business-meaningful status. A fraud check should not just be running. It should be FraudCheckPending or FraudCheckFailed. An ERP handoff should be ErpSubmissionPending, ErpRegistered, or ErpRejected. A supplier callback stage should be AwaitingSupplierApproval. A manual review should be PendingHumanDecision.

Durable Functions supports custom orchestration status, which is a powerful way to surface this information directly from the workflow runtime. But the same idea applies on any platform. You need a state model that answers the basic operational question: where is this process now, and why is it there?

A practical lifecycle might look like this:

In code, that might be as straightforward as setting custom status at each meaningful stage:

context.SetCustomStatus(new
    {
        orderId = order.OrderId,
        stage = "FraudCheckPending",
        updatedAtUtc = context.CurrentUtcDateTime
    });

That single line is more valuable than many teams realise. Once status is queryable, you can power dashboards, operator portals, support tooling, and incident triage without reverse engineering workflow behaviour from logs.

Observability also needs more than status labels. Correlation IDs must flow through the entire chain, from inbound request to orchestration instance to activity calls to outbound dependency calls and published events. Logs need consistent structured fields. Metrics should cover latency, retries, breaker state, queue depth, timeout counts, compensation frequency, and downstream failure rates. Tracing should allow engineers to follow a transaction through multiple services without playing archaeology across disconnected log stores.

Here is the ugly truth. If your workflow depends on several systems and you do not have proper correlation and state visibility, you do not have an operable architecture. You have a hope-based architecture.

Pattern 6: Hybrid Integration Accepts Reality Instead of Demanding a Rewrite

A lot of technical content on serverless and orchestration quietly assumes the organisation has the freedom to build a clean greenfield system. That is rarely how enterprise work actually looks. Most teams are not replacing everything. They are inserting modern capability into an environment where a mixture of old and new already exists.

That is why hybrid integration matters. Serverless does not have to replace the ERP. It can orchestrate around it. Durable workflows do not have to own every business rule. They can coordinate specialised services that already exist. Modern data stores can support fast projections and reporting while a different platform remains the canonical system of record. New cloud-native capabilities can coexist with legacy systems provided the architectural boundaries are clear.

A realistic enterprise shape often looks something like this:

This hybrid model is often the most pragmatic route to value. The orchestration layer becomes the coordinator of the business process. Compliance-sensitive payment logic can remain in a dedicated service. A legacy ERP can continue as the source of truth for certain financial or operational records. Cloud-native projections can power responsive read models and dashboards without forcing the organisation to migrate everything at once.

That also means architects need discipline around ownership. The orchestration engine should coordinate process state, not become the dumping ground for every piece of business logic in the company. The ERP should retain the responsibilities it is still good at, not be called for every trivial lookup. Projection stores should serve read performance and user experience, not quietly evolve into shadow systems with ambiguous truth boundaries.

The big win in hybrid architecture is incremental progress. You do not need a grand rewrite to improve resilience, observability, and flow control. You can wrap brittle integrations with better orchestration. You can isolate long-running handoffs. You can publish cleaner events. You can add compensations and workflow visibility around systems that were never built with those ideas in mind.

That is usually how real transformation succeeds, not through replacement fantasies but through carefully chosen seams.

Pattern 7: Resilience by Design Means Assuming the System Will Be Incomplete, Slow, and Wrong Sometimes

The strongest systems are not the ones that assume everything will go right. They are the ones that assume at least some parts will go wrong and still define how the workflow should behave. That mindset is what resilience by design really means.

It means assuming partial failure is normal. A dependency might succeed after a retry, fail permanently, or accept work and complete later. A callback might arrive twice. A timer might expire before a human responds. An event consumer might process late. An external system might hold the canonical answer even though your local projection says otherwise. These are not edge cases. They are part of the design space.

Resilience by design also means being honest about consistency. Many distributed workflows are eventually consistent, and pretending otherwise helps nobody. The real architectural task is to define where temporary inconsistency is acceptable, how it is reconciled, and what the user or operator sees while it exists. Good systems make the transition states explicit instead of hiding them behind vague processing messages.

It also means measuring the behaviour that matters. You should know which dependencies are slowest, which steps retry most often, which compensations are frequent, how long workflows remain in waiting states, and which manual interventions are recurring. Teams that do not measure this tend to rediscover the same operational pain every quarter and act surprised each time.

Finally, resilience by design means accepting that supportability is part of architecture. A workflow is not finished when it compiles and passes tests. It is finished when operators can understand it, support teams can explain it, incidents can be triaged quickly, and business stakeholders can trust that failures are bounded and recoverable.

A Concrete End-to-End Example

Let us pull these patterns together in a single scenario. Imagine a large B2B order workflow. An order enters the system through an API. The orchestration starts and immediately assigns a correlation ID that follows the transaction everywhere. The workflow sets its status to Received. It then checks whether the fraud provider breaker is open. If it is, the workflow fails fast with a visible dependency-unavailable status rather than quietly piling into retries.

If the breaker allows execution, the workflow sends a fraud request with a request ID that can be used for deduplication if the provider supports it. Once fraud is approved, payment is attempted with an idempotency key derived from the order ID. That ensures retries cannot double-charge the customer. After payment succeeds, the workflow publishes an OrderConfirmed event so downstream analytics and CRM updates can proceed independently instead of extending the critical path.

Next, the workflow submits the order to a legacy ERP. The ERP is slow and sometimes responds asynchronously, so the orchestration switches status to ErpSubmissionPending and waits for either an external callback or a timeout. If the callback arrives with success, the workflow completes. If the ERP rejects the order, the orchestration enters CompensationInProgress, triggers a refund, releases any provisional inventory state, raises an operational alert, and finally moves the order into a failed terminal state with a reason that support can actually understand.

That end-to-end shape looks like this:

Nothing in that flow is exotic. That is exactly the point. Most resilient architectures are not built from obscure theory. They are built from boring patterns applied consistently and early enough that the system does not rot under growth.

What Developers Usually Get Wrong

The first common mistake is over-centralising the orchestration. Developers discover a workflow engine and start putting every rule, integration, and transformation into the orchestrator itself. That turns the orchestrator into a giant god-process that becomes hard to change and impossible to reason about. The workflow should coordinate. It should not absorb every responsibility.

The second mistake is believing retries are a resilience strategy on their own. They are not. Retries without idempotency, compensation, status visibility, and bounded dependency behaviour are just a way of failing repeatedly.

The third mistake is underestimating operational visibility. Teams often spend far more time designing the happy path than designing the support path. Then the first real incident happens and nobody can answer the obvious questions. Which stage is this order at. Did payment happen already. Has ERP seen it. Is this waiting for a callback or stuck in a retry loop. Those questions should not require an engineer to grep logs across five systems.

The fourth mistake is assuming greenfield purity is required before improvement is possible. It is not. Some of the best resilience gains come from putting orchestration, status modelling, idempotency, and compensations around existing systems rather than replacing them.

The fifth mistake is treating eventual consistency as a flaw to be hidden instead of a reality to be designed for. Users and operators can cope with transition states if those states are honest and understandable. What they cannot cope with is silent ambiguity.

How to Apply These Patterns in Practice

If you are building or modernising an integration-heavy workflow, start by identifying the true business milestones rather than the raw API calls. Ask where side effects happen, which ones must be synchronous, which ones can be event-driven, and which ones need compensation if a later step fails. That alone will usually reveal whether your current workflow is too tightly coupled.

Then look at duplicate execution risk. Anywhere you have retries, redelivery, callbacks, or human re-submission, you need a defined idempotency strategy. Be precise about the operation key and where the result is recorded. Vague assurances that the provider should handle duplicates are not enough. Next, inspect dependency behaviour. Which integrations deserve a circuit breaker. Which ones should fail fast. Which ones should shift into async wait mode with timers and external events. Which ones are important enough to stay on the critical path and which ones should react to events later.

After that, design your status model. Not your log messages, your status model. What are the meaningful states of the workflow from an operator and business perspective. How are those states exposed. Where do correlation IDs flow. What metrics would tell you this process is degrading before customers notice.

Finally, decide how the new workflow lives alongside existing systems. Be explicit about what remains the source of truth, what becomes a projection, and what the orchestrator does and does not own. Hybrid architecture becomes dangerous only when ownership is vague.

Resilience at scale is not about making distributed systems behave like a single local transaction. That fantasy does not survive contact with real dependencies, real organisations, or real time. The job is to build workflows that remain understandable and recoverable when the surrounding systems behave imperfectly.

That is why these patterns are useful. Circuit breakers keep one bad dependency from turning into systemic slowdown. Idempotency makes retries safe. Compensation gives workflows a path back from partial success. Event-driven integration reduces unnecessary coupling. Custom status and observability make the process operable. Hybrid architecture accepts the systems you actually have. Resilience by design ties all of it together into a mindset rather than a patchwork of technical tricks.

Once you start thinking this way, integration architecture changes. You stop asking how to make the happy path pass one more test and start asking how the workflow behaves when the world around it is late, duplicated, unavailable, or inconsistent. That is the right question. It is also the one that separates systems that merely work from systems that keep working.

Most engineers learning modular monoliths fall into two traps. The first group collapses boundaries by sharing DbContexts, repositories, and entities across modules. The second group overcompensates by enforcing microservice-style communication within the monolith, introducing HTTP calls or message buses between modules. Both patterns undermine the purpose of a modular monolith. The correct approach maintains module isolation while enabling fast in-process communication through contracts and events. Vertical Slice architecture alters how we structure these modules.

The Architecture We Are Targeting

We are building a modular monolith with:

• Vertical Slice architecture

• CQRS

• Minimal APIs

• Separate module databases

• No HTTP between modules

• No message bus inside the process

Example modules:

• Users

• Claims

Each module owns its data and exposes capabilities, not services.

Instead of layers like Application, Domain, Infrastructure, the module is organised by features.

src
 ├ Users
 │   ├ Contracts
 │   │   └ UserQueries.cs
 │   │
 │   ├ CreateUser
 │   │   ├ Endpoint.cs
 │   │   ├ Command.cs
 │   │   ├ Handler.cs
 │   │   └ Validator.cs
 │   │
 │   ├ GetUser
 │   │   ├ Query.cs
 │   │   └ Handler.cs
 │   │
 │   └ DeleteUser
 │       ├ Command.cs
 │       └ Handler.cs
 │
 └ Claims
     ├ Contracts
     │   └ ClaimQueries.cs
     │
     ├ CreateClaim
     │   ├ Endpoint.cs
     │   ├ Command.cs
     │   └ Handler.cs
     │
     └ ApproveClaim
         ├ Command.cs
         └ Handler.cs

Each folder is a slice.

The slice contains everything needed for that use case.

Dependency Direction Between Modules

Modules reference contracts only, never implementation slices.

This rule is the cornerstone that keeps a modular monolith genuinely modular instead of slowly degrading into a tangled codebase. The Claims module is allowed to reference Users.Contracts because contracts represent the public capability surface of the Users module. They define what the Users module is willing to expose to the rest of the system in a controlled, stable way. Contracts typically contain simple request objects, response DTOs, and interfaces that describe operations such as queries or commands. Importantly, they contain no business logic, persistence concerns, or internal implementation details. By depending only on this contract layer, the Claims module interacts with the Users module in the same way an external client would, through clearly defined capabilities rather than through direct knowledge of how the module works internally. What the Claims module must never do is reference internal slices like Users.CreateUser, Users.GetUser, or Users.DeleteUser, because those folders represent implementation details of individual vertical slices inside the Users module. Allowing other modules to depend on those slices would immediately couple them to internal design decisions, meaning a simple refactor of a handler or slice could ripple across the entire system. The same rule applies even more strongly to infrastructure concerns such as UsersDbContext. Once another module starts using a different module’s DbContext, it is effectively reaching directly into that module’s database and bypassing its business rules, validations, and invariants. That instantly destroys the boundary between modules and turns the architecture into a shared persistence layer disguised as modules. By restricting dependencies strictly to Users.Contracts, the Users module remains free to reorganise its slices, change its database schema, refactor its handlers, or even split into a separate service later without breaking other modules. The Claims module only knows what operations are available, not how they are implemented, which is exactly the level of coupling a modular monolith is designed to enforce.The Three Communication Mechanisms

Vertical slice modular monoliths usually communicate through:

1. Query contracts

2. Command contracts

3. Domain events

These are not HTTP calls and not service bus messages. They are simple in-process calls.

Pattern 1 - Query Contracts

Suppose the CreateClaim slice needs to verify that the user associated with the claim actually exists and is allowed to submit a claim. At first glance it may seem natural for the Claims module to simply query the Users table directly, especially since everything runs inside the same application and the Users database is technically accessible. However, doing so would immediately violate the boundary between modules because the Claims module would now be coupled to the Users module’s persistence model and database schema. Any change to the Users table structure, indexes, or entity model could silently break the Claims module, and worse, the Claims module would be bypassing any business rules or invariants that the Users module is responsible for enforcing. In a modular monolith, each module owns its data and must be the only component allowed to access that data directly. Instead of reading the Users database, the Claims module should request the information it needs through a query contract exposed by the Users module. This contract defines a simple, explicit capability such as "retrieve a summary of a user by ID." The Claims module then calls that query through an interface defined in Users.Contracts, allowing the Users module to remain the sole authority over how user data is stored, retrieved, and validated. The Claims module gets exactly the information it needs to perform its operation, while the internal implementation of the Users module remains completely hidden behind the contract boundary.

Users.Contracts

public record GetUserSummaryQuery(Guid UserId);

public record UserSummaryDto(
    Guid Id,
    string Email,
    bool IsActive);

public interface IUserQueries
{
    Task<UserSummaryDto?> GetUserSummary(
        GetUserSummaryQuery query,
        CancellationToken stopToken);
}

The contract lives inside Users.Contracts.

No EF. No implementation.

Implementing the Query Slice

Inside the Users module.

Users/GetUserSummary

internal sealed class Handler : IUserQueries
{
    private readonly UsersDbContext db;

    public Handler(UsersDbContext db)
    {
        this.db = db;
    }

    public async Task<UserSummaryDto?> GetUserSummary(
        GetUserSummaryQuery query,
        CancellationToken stopToken)
    {
        return await db.Users
            .Where(x => x.Id == query.UserId)
            .Select(x => new UserSummaryDto(
                x.Id,
                x.Email,
                x.IsActive))
            .FirstOrDefaultAsync(stopToken);
    }
}

Register the slice handler.

services.AddScoped<IUserQueries, Handler>();

Using the Query in a Claims Slice

Inside the CreateClaim slice.

public sealed class Handler
{
    private readonly IUserQueries users;
    private readonly ClaimsDbContext db;

    public Handler(
        IUserQueries users,
        ClaimsDbContext db)
    {
        this.users = users;
        this.db = db;
    }

    public async Task<Guid> Handle(
        Command cmd,
        CancellationToken stopToken)
    {
        var user = await users.GetUserSummary(
            new GetUserSummaryQuery(cmd.UserId),
            stopToken);

        if (user is null)
            throw new Exception("User not found");

        var claim = Claim.Create(cmd.UserId);

        db.Claims.Add(claim);
        await db.SaveChangesAsync(stopToken);

        return claim.Id;
    }
}

The call remains fully in-process.

Pattern 2 - Command Contracts

Sometimes a module does not just need to read information from another module, it needs that module to actually perform some work on its behalf. A good example is when a claim is approved in the Claims module and the user should be notified that their claim has been accepted. It might be tempting for the Claims module to send an email directly or call some notification service itself, but that would be a mistake because notification behaviour belongs to the Users module’s responsibility. The Claims module should not need to know whether notifications are sent via email, SMS, push notification, or some future system that has not even been introduced yet. If Claims implements that logic, it becomes tightly coupled to infrastructure details that are outside its domain. Instead, the Claims module should simply express the intent of the action by issuing a command to the Users module through a contract. That command represents a capability such as "notify this user with this message." The Users module then decides how that notification is handled internally. By structuring the interaction this way, the Claims module remains focused purely on claims-related business logic while the Users module retains full ownership of notification behaviour and the infrastructure required to deliver it. This keeps responsibilities clearly separated and prevents implementation details from leaking across module boundaries.

Users.Contracts

public record NotifyUserCommand(
    Guid UserId,
    string Message);

public interface IUserCommands
{
    Task NotifyUser(
        NotifyUserCommand command,
        CancellationToken stopToken);
}

Users Slice Implementation

Users/NotifyUser

internal sealed class Handler : IUserCommands
{
    public Task NotifyUser(
        NotifyUserCommand command,
        CancellationToken stopToken)
    {
        // send email, SMS etc
        return Task.CompletedTask;
    }
}

Claims Slice Triggering the Command

Claims/ApproveClaim

public sealed class Handler
{
    private readonly IUserCommands users;

    public Handler(IUserCommands users)
    {
        this.users = users;
    }

    public async Task Handle(
        Command cmd,
        CancellationToken stopToken)
    {
        await users.NotifyUser(
            new NotifyUserCommand(
                cmd.UserId,
                "Claim approved"),
            stopToken);
    }
}

Again, no HTTP, no message broker.

Pattern 3 - Domain Events

Commands are appropriate when one module explicitly knows that another module must perform a specific action. In those cases the calling module intentionally invokes a capability exposed by the other module through a contract. Events serve a different purpose. Events are used when a module should not know which other modules might care about something that has happened. Instead of directing another module to do something, the module simply announces that a significant domain event occurred. A good example is user deletion. When the Users module deletes a user, it should not contain logic that checks whether the Claims module exists or whether it needs to clean up claims data. That would tightly couple the Users module to the rest of the system and force it to understand responsibilities that belong to other domains. Instead, the Users module publishes a UserDeleted event indicating that the user has been removed. Other modules that care about that event can react independently. The Claims module might close open claims for that user, an auditing module might archive historical data, and a reporting module might update statistics. None of those reactions are the Users module’s responsibility. By publishing an event rather than issuing direct commands, the Users module remains completely unaware of which modules subscribe to that event, preserving loose coupling and allowing new behavior to be added later without modifying the Users module itself.

public record UserDeletedEvent(Guid UserId);

Event Flow

Users publishes:

await dispatcher.Publish(
    new UserDeletedEvent(userId),
    stopToken);

Claims reacts:

public sealed class Handler
{
    private readonly ClaimsDbContext db;

    public async Task Handle(
        UserDeletedEvent evt,
        CancellationToken stopToken)
    {
        var claims = await db.Claims
            .Where(x => x.UserId == evt.UserId)
            .ToListAsync(stopToken);

        foreach (var claim in claims)
        {
            claim.MarkUserDeleted();
        }

        await db.SaveChangesAsync(stopToken);
    }
}

Users never references Claims.

In-Process Event Dispatcher

Because everything runs in one process, the dispatcher is trivial.

public class EventDispatcher
{
    private readonly IServiceProvider services;

    public EventDispatcher(IServiceProvider services)
    {
        this.services = services;
    }

    public async Task Publish<T>(
        T domainEvent,
        CancellationToken stopToken)
    {
        var handlers =
            services.GetServices<IEventHandler<T>>();

        foreach (var handler in handlers)
        {
            await handler.Handle(domainEvent, stopToken);
        }
    }
}

Minimal API Integration

Endpoints live inside the slice.

Example:

Claims/CreateClaim/Endpoint.cs

app.MapPost("/claims",
    async (
        Command cmd,
        Handler handler,
        CancellationToken stopToken) =>
{
    var id = await handler.Handle(cmd, stopToken);
    return Results.Ok(id);
});

The endpoint talks only to its slice handler.

Why This Works

This architecture preserves:

• strict module boundaries

• independent databases

• vertical slice isolation

• extremely fast in-process calls

The system remains loosely coupled because modules depend only on contracts.

Yet communication remains extremely simple.

The Performance Advantage

In-process contract calls are dramatically faster than external communication.

For high-throughput systems, that difference matters.

A modular monolith using Vertical Slice + CQRS + Minimal APIs should not resemble either a layered monolith or a microservice system.

Slices contain the behaviour. Modules own the data. Contracts define the boundaries.

Queries read across modules. Commands trigger behaviour. Domain events propagate changes.

The result is an architecture that is simple, fast, and strongly modular without introducing the complexity of distributed systems.

This happens because architecture is usually treated as documentation rather than enforcement. You define layering rules, module boundaries, and dependency directions, but those rules exist only as diagrams and wiki pages. They rely on discipline, code reviews, and team knowledge to maintain consistency. As you grow and deadlines tighten, these human safeguards become insufficient.

This is where architectural testing changes everything.

NetArchTest is a lightweight yet powerful library that allows you to express architectural rules as executable tests. Instead of hoping developers respect boundaries, you verify them automatically. Instead of discovering violations during late reviews, you detect them immediately during CI builds. Instead of architecture drifting silently, you make it impossible for violations to enter the codebase unnoticed.

I briefly covered using this tool back in my Modular Monolith walkthrough but decided it deserves its own post so this article explores how NetArchTest works, how to integrate it into a real .NET solution, and how to use it to enforce architectural discipline in modular monoliths, layered systems, and large enterprise applications.

The Problem with Traditional Architectural Governance

Before understanding NetArchTest, it is important to understand why architectural governance often fails.

You define architecture using documentation. Draw diagrams showing for example, layers such as Presentation, Application, Domain, and Infrastructure. They specify that dependencies must flow inward toward the domain. They state that certain modules must not reference each other.

The problem is that documentation does not enforce anything.

Developers under time pressure might introduce shortcuts. A controller may reference a repository directly. An infrastructure class might begin calling domain logic. A shared project might gradually accumulate unrelated responsibilities. These violations often seem small at first, but they compound over time.

Code reviews attempt to catch these issues, but reviewers focus primarily on correctness and functionality. Architectural violations are subtle and easily missed. Furthermore, as systems grow, no single reviewer understands the entire architecture well enough to detect every violation.

Static analysis tools provide some assistance, but they typically focus on style, complexity, or language-level issues. They do not understand business-level architectural rules such as module boundaries or dependency directions.

NetArchTest fills this gap by allowing you to define architecture rules directly in code.

What NetArchTest Actually Does

NetArchTest enables developers to inspect compiled assemblies and verify structural properties. It operates at the level of types, namespaces, dependencies, and inheritance relationships.

Instead of testing behaviour, it tests structure.

You can ask questions such as:

Does any class in the Domain layer reference Infrastructure?

Are all service classes named correctly?

Are certain types sealed or abstract?

Do controllers only depend on application layer interfaces?

Are modules independent from each other?

NetArchTest answers these questions by analysing metadata from compiled assemblies. Because it runs against compiled code, it is extremely fast and integrates seamlessly into standard unit test pipelines.

This approach transforms architecture from a set of conventions into a set of enforceable guarantees.

Installing NetArchTest

Adding NetArchTest to a solution is straightforward. It is typically installed in a dedicated test project responsible for architectural verification.

You add the package via NuGet:

dotnet add package NetArchTest.Rules

Most Developers create a test project named something like:

ArchitectureTests

This project references the assemblies you want to analyse. It does not contain business logic tests. Its sole responsibility is enforcing architectural rules.

This separation keeps architecture validation distinct from functional testing.

Understanding the Core Concepts

NetArchTest revolves around a few core abstractions.

The first is Types. This represents a collection of types extracted from an assembly. You start every test by selecting which assembly or namespace you want to analyse.

The second is Rules. These define conditions that types must satisfy. Rules can check naming conventions, inheritance relationships, dependency restrictions, and more.

The third is Conditions. Conditions specify whether rules must pass or fail, such as ensuring a type should or should not have certain dependencies.

Finally, there are Results. After applying rules, NetArchTest returns a result indicating whether the architecture rule passed and which types violated it.

This model makes architectural testing expressive and readable.

A Simple Example

Take a standard layered architecture with separate projects for Domain, Application, and Infrastructure.

A fundamental rule is that the Domain layer must not depend on Infrastructure.

You can express this rule in NetArchTest as follows:

using NetArchTest.Rules;

public class DomainDependencyTests
{
    [Fact]
    public void Domain_Should_Not_Depend_On_Infrastructure()
    {
        var result = Types
            .InAssembly(typeof(DomainAssemblyMarker).Assembly)
            .ShouldNot()
            .HaveDependencyOn("MyApp.Infrastructure")
            .GetResult();

        Assert.True(result.IsSuccessful);
    }
}

This test inspects all types in the Domain assembly and ensures none reference the Infrastructure namespace.

If a developer accidentally introduces a dependency, this test fails immediately during CI.

Architecture is now enforced automatically.

Enforcing Naming Conventions

Naming conventions are often overlooked, but they play a crucial role in maintainability. NetArchTest can enforce these rules consistently.

Suppose all command handlers must end with the suffix "Handler".

You can enforce this with:

var result = Types
    .InNamespace("MyApp.Application.Commands")
    .Should()
    .HaveNameEndingWith("Handler")
    .GetResult();

This ensures consistent naming across the application layer.

Such rules prevent gradual erosion of structure as new developers join the project.

Enforcing Layered Dependency Direction

One of the most powerful uses of NetArchTest is enforcing dependency direction in layered architectures.

For example, in Clean Architecture, dependencies must flow inward toward the Domain layer. Infrastructure can depend on Application and Domain, but not the other way around.

You can enforce this rule like this:

var result = Types
    .InAssembly(typeof(ApplicationAssemblyMarker).Assembly)
    .ShouldNot()
    .HaveDependencyOn("MyApp.Infrastructure")
    .GetResult();

This prevents infrastructure concerns from leaking into application logic.

Over time, this rule preserves architectural purity.

Enforcing Module Boundaries in Modular Monoliths

For modular monoliths, architectural enforcement becomes even more critical.

Each module should be independent. Modules should communicate through well-defined interfaces rather than direct references.

NetArchTest allows you to enforce module isolation.

For example, suppose you have modules named Users and Billing.

You can ensure the Users module does not directly depend on Billing:

var result = Types
    .InNamespace("MyApp.Modules.Users")
    .ShouldNot()
    .HaveDependencyOn("MyApp.Modules.Billing")
    .GetResult();

This protects module boundaries and prevents tight coupling.

Without such enforcement, modular monoliths often degrade into distributed spaghetti.

Testing for Layer Violations via Inheritance

Another important use case is ensuring that certain classes inherit from specific base types.

For example, all domain entities might be required to inherit from a base Entity class.

You can enforce this rule:

var result = Types
    .InNamespace("MyApp.Domain.Entities")
    .Should()
    .Inherit(typeof(Entity))
    .GetResult();

This ensures consistency in domain modelling practices.

Enforcing Interface Usage

A common architectural rule is that higher layers should depend only on interfaces, not concrete implementations.

For example, controllers should depend only on application interfaces.

NetArchTest can verify this:

var result = Types
    .InNamespace("MyApp.Api.Controllers")
    .ShouldNot()
    .HaveDependencyOn("MyApp.Infrastructure")
    .GetResult();

This guarantees proper abstraction boundaries.

Organising Architecture Tests

As systems grow, architecture tests should be organised clearly.

Most teams structure them by architectural concern.

You might have test classes such as:

LayerDependencyTests

NamingConventionTests

ModuleIsolationTests

DomainIntegrityTests

This organisation makes the intent of each rule clear and ensures the test suite remains maintainable.

Integrating NetArchTest into CI/CD

Architecture tests should run automatically during CI builds. NetArchTest, being lightweight, adds minimal overhead to build times. When a rule is violated, the build fails immediately, providing developers with clear feedback on which types caused the failure. This creates a strong feedback loop that prevents architectural drift. Over time, this automated enforcement becomes one of the most valuable safeguards in a mature codebase.

Combining NetArchTest with Other Tools

These tests work best when combined with other architectural enforcement strategies. You can pair them with Roslyn analysers for compile time checks, solution reference restrictions to prevent project dependencies, code review guidelines for contextual validation, and documentation to explain architectural intent. Together, these tools create a multi-layered defence against architectural decay.

Practical Tips for Real World Use

When introducing NetArchTest into an existing system, start small by enforcing only the most critical rules, such as preventing domain dependencies on infrastructure. Gradually expand coverage to include module boundaries, naming conventions, and inheritance requirements. Avoid creating overly rigid rules that hinder development flexibility, architecture enforcement should support productivity rather than obstruct it. Keep rules readable and well-documented so future developers understand their purpose.

Why Architectural Testing Changes Team Behaviour

Combining NetArchTest with other architectural enforcement strategies, such as Roslyn analysers for compile time checks, solution reference restrictions to prevent project dependencies, code review guidelines for contextual validation, and documentation to explain architectural intent, creates a multi-layered defence against architectural decay. When introducing NetArchTest into an existing system, start small by enforcing only the most critical rules, like preventing domain dependencies on infrastructure, and gradually expand to include module boundaries, naming conventions, and inheritance requirements. Avoid overly rigid rules that hinder development flexibility, architecture enforcement should support productivity rather than obstruct it. Keep rules readable and well-documented for future developers. When developers know architectural rules are enforced automatically, they design solutions more carefully and avoid shortcuts, as violations will be detected immediately. Architecture becomes a living, enforceable contract rather than an aspirational guideline, significantly improving long-term system maintainability.

The Long Term Impact

Over months and years, architectural testing prevents the slow decay that affects most large systems.

Boundaries remain intact. Dependencies remain predictable. Modules stay independent.

Teams can refactor with confidence because structural guarantees are always validated.

NetArchTest transforms architecture from a fragile concept into a durable, enforceable reality.

Architecture is easy to design but difficult to maintain. Without enforcement, even the best designs gradually erode under the pressure of deadlines and evolving requirements.

This provides a practical solution by allowing developers to encode architectural rules as executable tests. These tests run automatically, detect violations instantly, and prevent structural decay.

By integrating NetArchTest into your .NET solution, you transform architecture from documentation into enforcement. You create a system where boundaries remain intact, dependencies remain controlled, and long-term maintainability becomes achievable.

For teams building large enterprise systems, modular monoliths, or long lived platforms, this capability is essential.

This article looks at a different problem entirely. Instead of asking how to run a model locally on a workstation, we are going to look at what it takes to design and operate a local-feeling LLM platform on Azure itself. Not a public API call wrapped in configuration, and not a developer experiment, but a properly isolated, private, identity-secured LLM that lives inside your virtual network and behaves like an internal service. The focus here is not provider abstraction, but architecture, private endpoints, managed identity, governance, and operational discipline. In other words, how you move from local models in a .NET app to LLMs as internal infrastructure

Building a Truly Local LLM Platform on Azure

You might start experimenting with LLMs by calling a public API and wiring the response into an application. That phase is useful, but it ends quickly. As soon as the model touches real data, supports real users, or sits behind a regulated workflow, the questions change. Where does the data go? Who can access the model? How do we control behaviour over time? How do we prevent one bad prompt from blowing the budget?

This is where the idea of a local LLM becomes important.

Local does not mean on your laptop. It means the model is treated as infrastructure. It lives inside your security boundary, participates in your identity model, respects your network topology, and is observable and governable like any other internal service. Azure gives you the primitives to do this, but it does not assemble them for you. That is your job.

This article walks through that assembly in detail.

Why Azure Is a Good Fit for Local Style LLMs

Azure is unusually strong in this space because its AI offerings inherit the same enterprise primitives as storage, databases, and messaging. Azure OpenAI Service is not just a hosted model endpoint. It is a first-class Azure resource that supports private networking, Entra ID authentication, regional isolation, and resource level RBAC. That combination is rare. Many managed LLM platforms stop at API keys and IP allow lists. Azure lets you go further and treat inference as a zero trust internal dependency.

The consequence is architectural. You can design your system so that no developer, no CI pipeline, and no external system can talk to the model unless they are explicitly granted permission and are physically inside your network boundary.

That is what local really means in the cloud.

The Network Is the Product Boundary

The most important decision you will make is to disable public access to the model endpoint and rely exclusively on a private endpoint. Everything else builds on this.

When you enable a private endpoint for Azure OpenAI, Azure assigns a private IP address inside your virtual network and publishes a private DNS record that overrides the public hostname. From that moment on, name resolution itself enforces isolation. Calls from outside your VNet do not resolve.

This is stronger than firewall rules. There is no public surface to attack.

In practice, this means your application workloads must also live inside the VNet, or be integrated into it. Azure Container Apps with VNet injection, AKS, and App Service with regional VNet integration all work. What matters is that the call path from application to model never leaves the private network.

If you get this wrong, everything else is compromised.

Identity Is Not Optional

Using API keys for LLM access is the AI equivalent of storing database passwords in config files. It works, but its not safe.

Azure OpenAI supports Entra ID authentication, and you should consider this mandatory. With managed identity, your application authenticates using its own identity, not a shared secret. Access can be granted, audited, and revoked using the same mechanisms you already trust for storage accounts and message brokers. This has a subtle but powerful effect on system design. Instead of thinking in terms of “who has the key”, you start thinking in terms of “which workload is allowed to perform inference”. That aligns naturally with least privilege design. It also enables something important later. You can run multiple internal LLM facing services, each with different permissions, quotas, or even different model deployments, without duplicating secrets or configuration.

Choosing Models

A common early mistake is to pick the largest, most capable model and build everything around it. This creates two problems. Cost becomes unpredictable, and latency becomes variable. In a local platform setup, you should think in terms of tiers. A fast, smaller model for classification, routing, extraction, and guardrail checks. A stronger model for reasoning heavy tasks. Possibly a specialised model for summarisation or transformation. Azure OpenAI allows you to deploy multiple models side by side. The key is to treat model selection as an architectural decision, not a prompt detail hidden in application code.

When you later introduce routing or fallback logic, this separation becomes invaluable.

The LLM Should Never Be Called Directly

If every feature team writes their own prompt and calls the model directly, you lose control almost immediately. Prompts drift. Behaviour changes subtly. Costs explode in strange places. Nobody owns the system anymore.

Instead, you should introduce an internal LLM gateway service. This is not a proxy in the networking sense. It is a domain aware service that exposes operations like “summarise underwriting notes”, “extract risk factors”, or “draft internal correspondence”.

Internally, this service owns prompt templates, system instructions, token limits, retry policies, and output validation. Externally, it exposes stable, versioned contracts. This mirrors how Developers treat email, payments, or document rendering. The LLM is powerful, but it is not free form.

Once you centralise LLM access, prompt engineering stops being an ad hoc activity and becomes part of your delivery lifecycle. Prompts should be versioned. Changes should be reviewed. Behavioural differences should be tested against known inputs. In regulated environments, you may even need an approval process for prompt updates.

One effective pattern is to store prompts as structured templates with explicit inputs and outputs, rather than raw text blobs. This makes it easier to reason about what is allowed to change and what is not. Over time, this discipline is what keeps your system stable as models evolve.

Retrieval Augmented Generation Done Properly

Most discussions of RAG stop at embeddings and vector search. That is only half the story. In a production system, the retrieval pipeline is just as important as the model. You need to decide what content is eligible for retrieval, how it is chunked, how it is updated, and how relevance is enforced.

Azure gives you several options here, but Azure AI Search integrates particularly well with Azure OpenAI. It supports vector search, hybrid scoring, private endpoints, and managed identity. The critical point is that retrieval should happen before the model sees anything. The model should never be trusted to “remember” or “decide” what context it needs. You provide it with a constrained, curated slice of data and ask it to operate within that boundary. This is how you avoid hallucinations becoming system behaviour.

Failure Modes You Must Design For

LLMs do not fail like databases, message queues or HTTP APIs. If you treat them as just another REST dependency, you will build something that looks stable in testing and becomes unpredictable in production.

The first class of failure is load-related degradation, not hard outages. LLM services tend to fail slowly. As concurrency increases, token queues grow, response times stretch, and eventually requests start timing out upstream. From the caller’s perspective this does not look like a clean failure. It looks like sporadic latency spikes, partial responses, and workflows that suddenly take seconds or minutes longer than expected. If you do not impose strict per-request timeouts and concurrency limits, a small surge in usage can cascade through your system and block unrelated work.

The second failure mode is cost-amplified failure. Traditional services usually fail cheap. An LLM can fail expensively. A subtle prompt change, a retrieval bug, or an unbounded user input can multiply token usage overnight. Nothing crashes, nothing throws an exception, but your bill explodes. This is one of the most dangerous characteristics of LLM systems because it bypasses most operational alarms. Architecturally, this means you must treat token usage as a first-class resource. Hard caps, per-operation budgets, and enforced truncation are not optimisations, they are safety rails.

The third class of failure is semantic drift. LLMs can be up, healthy, and responding successfully while still being wrong in a way that breaks your system. A model version update, a backend optimisation, or even a temperature tweak can subtly change behaviour. Summaries become less precise. Classifications start misfiring. Edge cases creep in. Unlike traditional regressions, these failures do not show up as errors. They show up as business logic slowly becoming unreliable. This is why prompt versioning, golden test cases, and behavioural monitoring matter. You are not just testing availability, you are testing meaning.

Another critical failure mode is partial unavailability. LLMs often fail unevenly. Streaming might work while full completions time out. Small prompts succeed while larger ones fail. One deployment remains responsive while another degrades. Your architecture needs to handle this without collapsing. That usually means isolating LLM calls behind a boundary that can apply routing, fallback models, or degraded modes without forcing every caller to understand those details.

Then there is dependency amplification. Many LLM calls depend on other systems before the model is even invoked. Retrieval pipelines, embedding generation, vector search, prompt construction, and policy checks all sit upstream. A failure in any one of these can make the LLM appear unreliable, even though inference itself is healthy. This is why treating the LLM as a single black box is a mistake. You need visibility and control over each stage of the pipeline, and the ability to short-circuit or degrade when part of that pipeline fails.

All of this leads to a hard architectural question, what happens when the LLM is not available, not fast enough, or not trustworthy enough? There is no universal answer. In some systems, you return cached or last-known-good output. In others, you fall back to a simpler rules based path. In some workflows, you block progress and surface a clear error because proceeding would be worse. The key point is that this decision must be made explicitly, per capability, not implicitly by letting timeouts bubble up. Circuit breakers are essential, but they are not enough on their own. A breaker that simply stops calls does not solve the user experience problem. You need to define degraded behaviour that still makes sense in your domain. That is why these decisions belong at the architectural level. They define system behaviour under stress, not just how code handles exceptions.

The final mistake is assuming that retries are always helpful. With LLMs, retries often amplify the problem. If the model is slow due to load, retrying increases load. If the failure is semantic, retrying produces the same wrong answer. Retries should be rare, bounded, and context-aware. Blind retries are a liability.

I guess the takeaway here is simple but uncomfortable. An LLM is not an optional enhancement once it sits on a critical path. It is a core dependency with unique failure characteristics. You must design for those characteristics deliberately, or the system will design itself under pressure.

Treat the LLM like any other critical dependency, because operationally and financially thats exactly what it is.

Observability Beyond Token Counts

The first thing you need is latency distribution, not average latency. LLM performance degrades unevenly. Median latency may look fine while tail latency quietly doubles. This usually happens under load, when token queues back up or streaming responses stall. If you only watch averages, you will miss the early warning signs. P95 and P99 latency per operation tell you when the system is becoming unreliable long before it outright fails. This is especially important if LLM calls sit on synchronous user-facing paths.

Token usage must be tracked per capability, not per service. “Total tokens per day” tells you very little. What you actually need to know is which operation consumed them. Summarisation, classification, extraction, drafting, routing, each of these has a very different expected token profile. When one of them spikes, it is almost always a design regression, a prompt change, or a retrieval issue. Without per-operation attribution, cost overruns appear mysterious and uncontrollable.

Another critical dimension is prompt version awareness. LLM failures are often self-inflicted. A well-intentioned prompt tweak increases verbosity, weakens constraints, or causes the model to echo input. Nothing crashes. Token usage rises. Latency increases. Behaviour subtly shifts. If you cannot correlate requests and outputs to a specific prompt version, you cannot diagnose this class of failure. Prompt versions are effectively code. They must be observable as such.

Error rates also need reinterpretation. LLM systems do not fail with clean 500s. Many failures surface as timeouts, partial responses, truncated output, or invalid structured responses. You need to classify errors semantically. Was the response incomplete. Did it violate schema. Did it exceed budget. Did it arrive too late to be useful. These distinctions matter more than HTTP status codes.

One of the most overlooked signals is downstream impact. An LLM can appear healthy while degrading everything around it. Slower responses increase queue depth. Larger outputs stress storage or messaging systems. Poor classifications send workflows down the wrong path. Observability needs to extend beyond the model call itself and into what happens next. If LLM output feeds automated decisions, you should monitor reversal rates, human corrections, or escalation frequency. These are behavioural metrics, not technical ones, and they are often the first indication that something has gone wrong.

Logging deserves special care. Logging full prompts and responses is tempting, and in early experiments it can be useful. In production systems, it is often a liability. Prompts may contain sensitive data. Outputs may contain inferred or derived information that creates compliance risk. A safer pattern is to log metadata instead of content. Token counts, input and output sizes, hashes of prompts, prompt identifiers, model version, latency, and validation outcomes usually provide enough signal to diagnose issues without storing raw text. When deeper inspection is required, targeted sampling with strict access controls is safer than blanket logging.

All of this becomes dramatically easier when LLM access is centralised. If every team calls the model directly with their own prompts, observability fragments immediately. Metrics lose consistency. Prompt versions are unknown. Cost attribution becomes political. Centralisation does not mean slowing teams down. It means giving them a stable, observable platform instead of a shared bill. The uncomfortable truth is that LLM observability is closer to monitoring business logic than infrastructure. You are not just watching whether the system is up. You are watching whether it is behaving as intended. Token counts are a starting point. Serious systems go much further.

When You Really Need to Self Host Models

There are legitimate cases where managed inference is not enough. Offline environments, extreme data sovereignty requirements, or deep fine-tuning at the weight level may push you toward self hosted models on AKS with GPU nodes. The important thing is that this is an intentional step, not an accident. The same principles still apply. Private networking. Managed identity where possible. Centralised access. Clear ownership. The mistake is thinking that self hosting is more local by default. Without discipline, it is often less secure and less predictable than a managed service.

The End State You Should Aim For

A successful local LLM on Azure does not feel like AI.

It has an endpoint name that looks like an internal service. It has dashboards. It has quotas. It has owners. Changes are deliberate. Behaviour is predictable.

When you reach that point, the model itself becomes interchangeable. You can swap versions, introduce new deployments, or even change inference backends without rewriting your application.

That is the real value of doing this properly.

If you build it this way from the start, you are not experimenting with LLMs. You are operating them.

If your system has multiple Azure workloads calling each other with Managed Identity, Event Grid, Service Bus, or Durable Functions, you can introduce confused deputy behaviour without realising it. The system stays secure by conventional checks, while still letting the wrong caller trigger the wrong privileged action.

A confused deputy attack happens when a highly privileged service performs an action on behalf of a less privileged caller without properly validating whether the request should be honoured. The key detail is that the deputy is not compromised. It is authenticated, authorised, and behaving as designed. The failure is that it is acting with its own authority on a request it should not have accepted. Azure makes this easy to miss because Managed Identity answers only one question: who am I. It does not answer who asked me to do this, why they asked, whether they are the right workload for this operation, or whether the operation is legitimate within your business rules. If you treat “valid token for my API” as sufficient proof of legitimacy, you have created a deputy that will do anything it is capable of doing for anyone who can obtain the right kind of token.

Take a concrete internal API example. You have an Invoices API that can write invoices to Azure SQL. A Billing worker calls it during a nightly workflow. Both use Managed Identity. Everything feels internal, so you add an endpoint like POST /internal/invoices/generate and protect it with normal JWT validation. You have now created a target where any Azure workload that can obtain a token for that API can trigger invoice generation. The call is authenticated, but the intent is unauthorised. That is a confused deputy.

Here is the failure mode in one sequence. The attacker does not steal a token. They simply obtain one legitimately from Entra ID for the right audience, then use your privileged service as the execution engine.

Managed Identity represents authority, not permission. It says this workload can do powerful things. It does not mean the caller is allowed to ask for those powerful things to be done. If you treat Managed Identity as permission, you are implicitly saying that any caller who can reach your API is allowed to ask it to do anything the API can do. That assumption collapses the moment you have multiple services, multiple teams, multiple workflows, or any pathway where untrusted input can influence a privileged operation. Event-driven systems amplify the problem. Event Grid, Service Bus, and Durable Functions often run without user context and execute with full Managed Identity authority. If your handler processes messages simply because they arrived, it becomes a deputy by default. The attacker no longer needs to reach your HTTP boundary. They only need to get the right event into the right pipeline.

The core defence is intent binding. Every privileged operation needs to be bound to an explicit caller, an explicit target, and an explicit reason. In practical .NET terms, this means you cannot rely solely on ClaimsPrincipal and you cannot rely solely on Managed Identity. You must carry the context that explains why the operation is legitimate, and you must validate that context at the point where the privileged action occurs. One of the simplest and most effective hardening steps is to enforce strict audience validation and explicitly validate the calling workload identity. Many internal APIs accept any token with a valid issuer and broadly matching audience. That is not enough. You want exact audience matching and you want to reject calls from workloads that are not allowed to invoke this API, even if they can technically obtain a token.

Here is a practical example for ASP.NET using AddJwtBearer. This does not solve intent binding by itself, but it closes a major hole where any internal workload can call any internal API.

using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.IdentityModel.Tokens;

builder.Services
    .AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(options =>
    {
        options.Authority = $"https://login.microsoftonline.com/{tenantId}/v2.0";
        options.TokenValidationParameters = new TokenValidationParameters
        {
            ValidAudience = "api://invoices",
            ValidateAudience = true,
            ValidateIssuer = true,
            ValidateLifetime = true,
            ValidateIssuerSigningKey = true,
            ClockSkew = TimeSpan.FromMinutes(1)
        };

        options.Events = new JwtBearerEvents
        {
            OnTokenValidated = ctx =>
            {
                var callerAppId = ctx.Principal?.FindFirst("azp")?.Value
                    ?? ctx.Principal?.FindFirst("appid")?.Value;

                if (callerAppId is null || !AllowedCallers.Contains(callerAppId))
                {
                    ctx.Fail("Caller application is not allowed.");
                }

                return Task.CompletedTask;
            }
        };
    });

static readonly HashSet<string> AllowedCallers = new(StringComparer.OrdinalIgnoreCase)
{
    "00000000-0000-0000-0000-000000000001"
};

Strict caller validation prevents random internal workloads from calling the API, but it still does not guarantee the call is legitimate in business terms. That is where intent binding comes in. The service must refuse to perform privileged work unless the request carries verifiable provenance and the operation is authorised against a rule you control.

A clean way to do this in ASP.NET is resource-based authorisation. Instead of authorising the principal in isolation, you authorise a principal acting on a specific resource context that includes caller intent.

using Microsoft.AspNetCore.Authorization;

app.MapPost("/internal/invoices/generate",
    [Authorize] async (
        string callerService,
        string correlationId,
        GenerateInvoiceRequest request,
        IAuthorizationService authorization,
        ClaimsPrincipal principal,
        CancellationToken stopToken) =>
    {
        var resource = new InvoiceGenerationResource(
            callerService,
            correlationId,
            request.AccountId);

        var result = await authorization.AuthorizeAsync(principal, resource, "InvoiceGeneration");
        if (!result.Succeeded)
            return Results.Forbid();

        await GenerateInvoicesAsync(request, stopToken);
        return Results.Accepted();
    });

At this point the deputy is no longer “just checking a token”. It is checking whether this caller is allowed to request this operation, with this declared purpose, against this target. That logic lives in an authorisation handler, where you can combine workload identity from the token with declared intent from the request.

using Microsoft.AspNetCore.Authorization;

public sealed class InvoiceGenerationRequirement : IAuthorizationRequirement;

public sealed record InvoiceGenerationResource(
    string CallerService,
    string CorrelationId,
    Guid AccountId);

public sealed class InvoiceGenerationHandler
    : AuthorizationHandler<InvoiceGenerationRequirement, InvoiceGenerationResource>
{
    protected override Task HandleRequirementAsync(
        AuthorizationHandlerContext context,
        InvoiceGenerationRequirement requirement,
        InvoiceGenerationResource resource)
    {
        var callerAppId = context.User.FindFirst("azp")?.Value
            ?? context.User.FindFirst("appid")?.Value;

        if (callerAppId != "00000000-0000-0000-0000-000000000001")
            return Task.CompletedTask;

        if (!string.Equals(resource.CallerService, "Billing.Worker", StringComparison.Ordinal))
            return Task.CompletedTask;

        if (resource.AccountId == Guid.Empty)
            return Task.CompletedTask;

        context.Succeed(requirement);
        return Task.CompletedTask;
    }
}

The same principle applies in event-driven paths, except the intent must be carried in the message itself. If you treat messages as implicitly trusted because they arrive on your subscription, you are delegating authority to any actor that can inject a message. A good handler validates provenance and schema before it performs privileged work, and it fails closed when the metadata is missing.

Here is an Azure Function example using Service Bus where intent is enforced before invoice generation is executed.

using Azure.Messaging.ServiceBus;
using Microsoft.Azure.Functions.Worker;
using System.Text.Json;

public sealed class InvoiceMessages
{
    [Function(nameof(HandleInvoiceRequested))]
    public async Task HandleInvoiceRequested(
        [ServiceBusTrigger("invoice-requests", Connection = "ServiceBusConnection")]
        ServiceBusReceivedMessage msg,
        CancellationToken stopToken)
    {
        if (!msg.ApplicationProperties.TryGetValue("producer", out var producer)
            || !Equals(producer, "Billing.Worker"))
            throw new UnauthorizedAccessException("Untrusted producer.");

        if (!msg.ApplicationProperties.TryGetValue("schema", out var schema)
            || !Equals(schema, "invoice-requested:v1"))
            throw new UnauthorizedAccessException("Unexpected schema.");

        var payload = JsonSerializer.Deserialize<InvoiceRequested>(msg.Body);
        if (payload is null || payload.AccountId == Guid.Empty)
            throw new InvalidOperationException("Invalid payload.");

        await GenerateInvoicesAsync(payload, stopToken);
    }
}

public sealed record InvoiceRequested(
    Guid AccountId,
    string CorrelationId,
    DateOnly PeriodStart,
    DateOnly PeriodEnd);

This is the heart of it. Confused deputy attacks do not happen because your tokens are invalid. They happen because your system accepts valid identity as sufficient proof of legitimate intent. The fix is not a new Azure feature. The fix is boundaries, explicit intent, and validation at the moment privileged work is executed.

Audit logs matter here as well, because they are how you catch deputies in the real world. When something goes wrong, you want logs that tell you who requested the action, which service executed it, and why it was allowed. If your logs only say “operation succeeded”, confused deputy behaviour disappears into the noise.

Before shipping a service, it is worth asking one uncomfortable question. If another internal workload can obtain a token for this API, can it trick this service into doing something dangerous. If the answer is not clearly “no” with a reason you can point at in code, you probably have a deputy waiting to be confused.

Confused deputy attacks already exist in many Azure systems, hidden behind valid tokens and successful authentication. The good news is that as a .NET developer you have the tools to shut them down. You control the authorisation pipeline, middleware, message handlers, and execution context. Use them deliberately, and your services stop being deputies that trust too much.

And yet, every change feels heavier than it should.

The mistake many people make at this point is assuming that architecture has to be fixed in one decisive moment. A rewrite. A migration project. A “phase two”. That thinking kills momentum, because the system doesn’t stop needing features just because you want to improve it.

A successful migration does not start with structure. It starts with pressure.

The pressure usually shows up in one place first. A part of the system that changes often. A domain that attracts bugs. A feature area that nobody enjoys touching because it breaks unrelated things.

That area is not your problem child. It’s your opportunity.

You don’t modularise the whole system. You carve out one real boundary and make it honest.

At the start, a typical layered system looks something like this.

In a typical layered system, everything flows through the same pipes. Ownership is implied rather than enforced, and data is shared by default. The architecture relies on discipline and convention instead of hard boundaries.

Trying to “add modules” on top of that usually just adds folders. The underlying model does not change. The coupling is still there, just hidden a little better. That is not the move. The first real step is to identify a business capability that can stand on its own. Not technically, but conceptually. Something you can describe clearly without mentioning how the rest of the system works. Users. Billing. Reporting. Claims. Pick one.

Then you do something that feels small but is actually profound. You stop letting other parts of the system reach into its data. You do not refactor everything at once. You draw a line, and from that point on, the rules change.

That line usually starts as a namespace and ends as an assembly. You extract the code related to that capability into a new project. Not because projects are magical, but because they give you enforcement. Suddenly, references are explicit. Internals are important. The compiler starts helping you.

At first, it will feel awkward. The rest of the system still expects to call services and repositories directly. You don’t fight that immediately. You introduce a thin contract layer and adapt.

Conceptually, the system now looks like this.

The world hasn’t changed. But one boundary is now visible.

Inside that new module, you resist the urge to replicate the layered structure. This is where vertical slices matter most. You organise by behaviour, not by technical concern. CreateUser lives in one place. GetUser lives in one place. Each slice owns its own logic end to end.

This is the moment where migration and design intersect. You’re not just moving code. You’re changing how it’s shaped.

The shared DbContext is usually the next sticking point. This is where many migrations stall, because people try to solve everything at once. You don’t.

You let the module introduce its own DbContext while the rest of the system continues using the shared one. For a while, both exist.

That coexistence is not a failure. It’s a bridge.

During this phase, the goal is not purity. It’s containment. New behaviour goes through the module’s DbContext. Old behaviour stays where it is. Over time, the shared DbContext shrinks instead of growing.

That direction matters more than speed.

Inter-module communication follows the same pattern. At first, the legacy system may still call into the module synchronously. You accept that. But inside the module, you design as if those calls were remote. Contracts are explicit. Events represent facts. Failures are handled locally. You are laying tracks in the direction you want the system to move.

One of the hardest parts of this migration is psychological, not technical. You have to be comfortable with the system being temporarily uneven. Some parts clean. Some parts messy. Some parts modern. Some parts legacy. Trying to make everything consistent too early is how migrations die.

Old code can stay ugly for a while. That’s not debt. That’s triage.

Testing often improves naturally during this process. The first module you extract becomes the first place where slice tests, integration tests, and architecture tests feel obvious instead of forced. That contrast is useful. It gives the team a reference point. People don’t need to be convinced with diagrams. They feel the difference when changes stop breaking unrelated things.

Over time, something interesting happens. The module stops feeling like “the new thing” and starts feeling normal. Meanwhile, the legacy area starts to feel increasingly uncomfortable by comparison. That discomfort is not a problem. It’s information. It tells you where to go next.

Eventually, the system’s shape changes from this.

To something more like this.

And then, slowly, Legacy gets smaller.

Not because you planned a rewrite, but because you stopped feeding it.

The most important thing to understand about this kind of migration is that it is not linear. You will pause. You will adapt. You will make compromises. That’s not failure. That’s working in a live system. The success criteria is simple. Each month, is the system easier to change than it was the month before?

If the answer is yes, you’re doing it right.

This is where the series comes full circle. A modular monolith is not a destination you reach. It’s a direction you commit to. A way of making trade-offs explicit instead of accidental.

You don’t need permission to start. You need one boundary, one module, and the discipline to protect it.

Everything else follows.

If someone reads this series end to end and only takes one thing away, I hope it’s this, architecture is not about being clever early. It’s about staying honest over time.

That’s what actually scales.

“Which part of the system is actually broken?”

If your modular monolith can’t answer that clearly, then at runtime it isn’t really modular at all.

In development, modules are boundaries in code. In production, modules must be boundaries in signal. Logs, metrics, health, failures, and alerts must respect the same lines your architecture does. If they don’t, all the care you took earlier collapses into a single operational blob.

The runtime shape of a healthy modular monolith looks something like this.

One process. One deployment. Multiple, clearly attributable sources of behaviour.

If your logs don’t already look like this conceptually, diagnosing problems will always be slower than it needs to be.

The same idea applies to metrics. A single latency number for the entire application tells you very little once the system grows beyond trivial size. What you actually need to know is whether a slowdown is systemic or local.

At runtime, the system should feel more like this than a flat line.

This doesn’t mean every module needs bespoke dashboards for everything. It means the possibility exists. When an alert fires, you can immediately tell whether one module is misbehaving or whether the entire application is under stress.

That distinction is the difference between a calm response and a panicked one.

Health checks are where many modular monoliths accidentally lie to their operators. A single “healthy or unhealthy” signal flattens the system into something it no longer is. It forces binary decisions in a world that is not binary.

A more honest mental model looks like this.

In this model, the system can express partial degradation. Users might be healthy. Reporting might be lagging. Billing might be failing to reach an external dependency. The application is still running, but not everything is equal. Failure handling is where the difference between architectural intent and operational reality becomes obvious. If an exception in one module can crash unrelated behaviour, then isolation exists only in your head.

At runtime, you want failure to look like this.

Billing failing to react to an event should not destabilise Users. If it does, you’ve accidentally recreated a distributed transaction, just without the tooling to see it.

This is one of the reasons earlier parts of the series pushed so hard on honest communication and local failure. Operations is where dishonesty is punished.

Database operations are another place where modularity often collapses under pressure. When all schema changes are treated as one global concern, deployments become coupled even if the code is not.

The operationally healthy shape looks like this.

Even if all schemas live in the same physical database, ownership is clear. Migration failures are attributable. Rollbacks are targeted. People stop fearing deployments because not every change threatens everything else.

One of the most telling operational signals is how incidents are described. In unhealthy systems, everything is “the system”. In healthy modular monoliths, incidents are scoped naturally.

“We’re seeing elevated latency in Billing.”
“Users is healthy, but Reporting is behind.”

Those sentences only make sense if the runtime architecture supports them.

There’s also a human dimension here that’s easy to underestimate. Systems that are hard to observe become systems people are afraid to touch. Over time, that fear turns into conservatism, then stagnation. Not because change is dangerous, but because the feedback loop is too vague to trust.

Good operational boundaries don’t just help machines. They help people stay confident.

One of the quiet benefits of doing this work inside a modular monolith is that it prepares you for the future without forcing it. If one module eventually needs to be extracted, the operational model already exists.

The diagrams don’t change much. The lines just move.

If this looks familiar, that’s intentional. Extraction should feel like relocation, not reinvention.

Operational clarity is not a nice-to-have. It’s the difference between a system that survives contact with reality and one that slowly erodes trust.

If Parts 1 through 8 were about making the code honest, Part 9 is about making the system honest while it’s running.

That leaves one final problem to address. Not how to design from scratch, and not how to operate once things are clean, but how to get from where most teams actually are to where this series has been pointing all along.

Part 10 is about migrating a legacy layered .NET application to a modular monolith, without stopping delivery or pretending you have infinite time.

That assumption holds only while change is cheap and coordinated. The moment different parts of the system start evolving at different speeds, the absence of a versioning strategy turns every change into a negotiation.

This is not a microservices problem. It’s a time problem.

In a real modular monolith, modules do not mature evenly. Some stabilise quickly and barely change. Others sit close to the business edge and churn constantly. Treating them as if they all move together is how you end up with artificial constraints and unnecessary coupling.

The first thing to be clear about is what we mean by “versioning” in this context. We are not talking about NuGet packages or semantic version numbers published to the world. We are talking about the ability for one module to change its behaviour or contracts without forcing immediate changes in every consumer.

That sounds abstract until you see the alternative.

Imagine a Users module that exposes a query returning user details. At first, it looks like this.

public sealed record UserSummary(Guid Id, string Email);

Billing consumes it. Reporting consumes it. Everything is fine.

Then the Users module evolves. The business now distinguishes between primary and secondary email addresses. The model changes.

public sealed record UserSummary(
    Guid Id,
    string PrimaryEmail,
    IReadOnlyList<string> SecondaryEmails
);

If this change ripples through the entire system immediately, you don’t have independent evolution. You have coordinated refactoring. In a small codebase that might be acceptable. In a system that’s growing, it becomes a tax you pay over and over again.

The simplest and most reliable way to version inside a monolith is not through numbers, but through coexistence. Old behaviour stays available while new behaviour is introduced alongside it.

Nothing forces every consumer to move at once. Migration happens incrementally, with intent.

In practice, this usually means versioning contracts, not modules.

The Users module still deploys as one unit. It still owns its data. But it exposes more than one contract shape for a period of time.

That might look like this in code.

public interface IUserQueriesV1
{
    Task<UserSummaryV1> GetAsync(Guid userId);
}

public interface IUserQueriesV2
{
    Task<UserSummaryV2> GetAsync(Guid userId);
}

Both are implemented internally by the Users module. Both are valid. One is newer. This feels slightly uncomfortable at first, because it introduces duplication. That discomfort is the cost of time. You’re paying it explicitly instead of smearing it across the system.

What matters is that versioning lives at the boundary, not in the core. Internally, the module should have one clear model and one clear understanding of the business. The translation to older or newer shapes happens at the edge. That keeps the complexity contained.

Feature flags are often suggested as an alternative to versioning, and they do have a place here, but only if you’re honest about what they are doing. A feature flag that permanently changes behaviour for different consumers is not a flag. It’s a version switch disguised as configuration.

Used carefully, flags help you stage a change. Used carelessly, they become a second, hidden versioning system that nobody fully understands.

The rule I follow is simple. Flags are temporary. Versions are explicit. If a conditional is going to live longer than a release cycle or two, it should probably be a versioned contract instead.

While we’re talking about feature flags, it’s probably worth being transparent about something. A lot of the opinions in this section come from having been burned by flag sprawl more than once, which is why I’ve been building a small feature flag platform called Flagmesh on the side.

The goal isn’t to turn flags into a second versioning system or bury business logic behind configuration, but to make it obvious what flags exist, why they exist, and when they should be removed. If you ever find yourself needing feature flags and want something deliberately opinionated about scope and cleanup, it’s there. If not, the principles still stand regardless of tooling.

Another place versioning shows up is in events. This is where people often get caught out, because events feel immutable until they aren’t. An event is a promise. Once it’s published, consumers will build assumptions around it. Changing it silently is one of the fastest ways to break trust between modules.

When an event needs to evolve, the safest approach is the most boring one, publish a new event type.

public sealed record UserCreatedV1(Guid UserId, string Email);
public sealed record UserCreatedV2(
    Guid UserId,
    string PrimaryEmail,
    IReadOnlyList<string> SecondaryEmails
);

Both can coexist. Producers can emit both for a while if needed. Consumers can migrate on their own schedule. This is not wasteful. It is respectful of time and autonomy. Inside a single deployment, this approach has an important benefit. You can see who is still using the old contract. You can log it. You can measure it. You can remove it deliberately when the time is right. Nothing is implicit. Nothing breaks “by accident”.

One subtle but important point is that independent versioning only works if your modules are already decoupled in the ways described earlier in the series. If consumers are reaching into internals, or sharing DbContexts, or coordinating synchronously, versioning becomes performative. You can put a V2 suffix on things, but the coupling is still there. This is why versioning belongs after boundaries, slices, data ownership, and communication patterns. It’s not a starting point. It’s a capability you earn.

A modular monolith that supports independent versioning is a system that understands it will live longer than its first set of assumptions. It doesn’t cling to the idea that everything must always move together. It accepts that change is uneven and builds around that reality.

That acceptance is what keeps the system supple instead of brittle.

In the next part of the series, the focus shifts from evolution to operation. Once modules evolve independently, you need to be able to see, diagnose, and manage them independently as well. Logging, metrics, health, and failure handling become the difference between confidence and guesswork.

What matters is learning to distinguish between structural readiness and organisational impatience.

A modular monolith that is genuinely ready to split already behaves like a distributed system in all the places that matter. The boundary you are considering extracting is already autonomous in practice. It owns its data. It commits independently. It communicates through contracts rather than shared state. Failures inside it are visible and tolerated rather than catastrophic.

When that is true, extraction does not introduce new concepts. It changes where things live.

You can often see this clearly by looking at the execution flow.

If this diagram already represents reality inside your monolith, then moving Billing out of process does not change the mental model. It changes transport, deployment, and observability, but not behaviour.

That’s the first and most important signal.

Contrast that with a system where the flow actually looks like this, even if nobody admits it out loud.

This is not a candidate for extraction. This is a warning. What you really have here is a single consistency boundary pretending to be modular. Pulling it apart will not create independence, it will just expose coupling that was previously hidden by process boundaries.

Another strong signal lives in your codebase, not your diagrams. If you look at a module and its public surface area is small, stable, and boring, that’s a good sign. A module that is ready to be split does not need a rich internal API exposed to the rest of the system. It needs a narrow set of contracts that have already proven themselves under change.

That usually looks something like this.

public interface IBillingEvents
{
    Task Handle(UserCreated @event, CancellationToken stopToken);
}

Notice what isn’t there. There is no shared DbContext. There is no orchestration logic. There is no dependency on Users’ internals. Billing reacts to a fact, not a request to coordinate behaviour.

When a module’s public API starts looking like this naturally, without effort or policing, it’s a sign that the boundary is real.

Operational behaviour is another place where readiness becomes obvious. In a healthy modular monolith, a failure inside one module is already treated as local. Logs are scoped. Metrics are scoped. Alerts are scoped. People don’t panic when Billing misbehaves, because Users continues to function.

If your on-call response already distinguishes between “the system is down” and “that module is down”, then extraction won’t change how incidents are reasoned about. It will only change how they are mitigated.

If, on the other hand, every failure is treated as a system-wide emergency because everything is still tightly coupled, splitting will multiply that pain, not reduce it.

There is also a very practical, almost boring test that I’ve learned to trust. Ask yourself how hard it would be to introduce a network boundary tomorrow. Not actually do it, just introduce it. If replacing an in-process event bus with a real message broker feels like a mostly mechanical change, you are close. If it feels like a redesign that touches business logic, persistence, and error handling all at once, you are not.

That distinction matters more than any architectural diagram.

It’s also worth being explicit about what doesn’t justify splitting. Performance alone rarely does. Scale alone rarely does. The phrase “we’ll need microservices eventually” almost never does. Those are projections, not pressures. The pressures that matter are present tense. Teams blocking each other. Deployment risk that forces coordination meetings. Modules with incompatible uptime or scaling needs that are being artificially flattened by a shared runtime. When those pressures exist and the boundaries are already clean, extraction becomes an act of alignment, not rescue.

One of the most overlooked aspects of this decision is cognitive load. A modular monolith done well reduces it. A distributed system increases it. If your current system already feels heavy to reason about, adding network boundaries will not lighten that load. It will distribute it across logs, dashboards, retries, and failure modes.

If, however, your modular monolith feels calm, predictable, and honest, then splitting a module can actually preserve that calm by preventing future coupling from creeping back in.

From experience, the best extractions I’ve seen were almost boring from a code perspective. The module was already designed as if it were remote. The code barely changed. Most of the work happened in CI pipelines, infrastructure, and observability. That’s exactly how it should be. The worst extractions were dramatic. They required heroics. They broke things in surprising ways. In hindsight, they were not premature because microservices are bad. They were premature because modularity was incomplete.

The point of a modular monolith is not to delay microservices. It’s to make them optional. To give you the ability to say yes or no based on reality, not fashion or fear. If you reach the point where a module can leave without the rest of the system noticing much beyond a configuration change, then you’ve succeeded, regardless of whether you actually do it.

Testing is where those claims get audited. Not by diagrams, and not by good intentions, but by code that either passes cleanly or fails in ways you did not expect. Tests have a way of surfacing the truth about a design, because they force the architecture to be exercised rather than explained. This is the part many Developers quietly dread. It exposes the gap between architecture that sounds good in conversation and architecture that actually holds up under pressure. When tests are aligned with real boundaries, they reinforce the design. When they are not, they reveal exactly where those promises start to break down.

Why Traditional Testing Advice Breaks Down Here

Most traditional testing advice assumes one of two worlds. Either you are building a small application where unit tests are sufficient, or you are working in a distributed system where end-to-end tests are unavoidable. A modular monolith sits in an awkward middle ground that neither model fits particularly well.

If you only write unit tests, you miss important classes of failure. Broken wiring goes unnoticed. Boundary violations slip through. Configuration errors hide until runtime. The system looks healthy in isolation, but the pieces do not actually fit together the way you think they do.

At the other extreme, relying heavily on end-to-end tests creates a different set of problems. They are slow to run, brittle to maintain, and hard to diagnose when something fails. Over time, people stop trusting them, and once trust is gone, the tests stop doing their job.The mistake is treating testing as a ladder, moving neatly from unit tests to integration tests to end-to-end tests. In a modular monolith, testing works better as layers of confidence, with each layer answering a different question about the system.

The Question Your Tests Should Answer

Before writing any test, I ask one thing:

What architectural promise am I trying to protect?

If you can’t answer that, the test probably doesn’t matter.

Level 1: Slice Tests (Your Primary Workhorse)

Vertical slices change what “unit testing” really means. The unit is no longer a method, a class, or a service. The unit is the use case itself, the thing the system actually does.

A slice test exercises the full behaviour of that use case. It runs the handler, applies validation, touches persistence, and enforces business rules. What it deliberately avoids is anything outside the slice’s responsibility. There is no HTTP layer involved. There is no serialisation. There is no real infrastructure beyond what the module owns. The key shift is where mocking happens. Mocks belong at the module boundary, not inside the slice. Inside a module, you want reality. You want real code paths and real interactions, because that is how you gain confidence that the module actually works.

A CreateUser slice test, for example, should use a real UsersDbContext backed by an in-memory or test database. It should not mock repositories, and it should not mock EF. If the slice cannot work with its own real persistence model in a test, that is a signal worth paying attention to.

You want to know:

Does this behaviour actually work?

Not:

Can I make this method return the value I expect?

Why Mocking Internals Destroys Confidence

Mocking internals buys you speed, but it does so at the cost of truth. A test that mocks repositories, DbContexts, or domain behaviour is not really testing the system. It is testing your expectations about how the system should behave.

In a modular monolith, most bugs do not live in pure logic. They live in mapping, configuration, wiring, and persistence assumptions. Those are exactly the areas that heavy mocking conveniently hides, which is why everything looks fine in tests. When a slice test fails, you want it to fail because the system is wrong. You do not want it to fail because the test was overly clever or made assumptions that no longer hold. Tests that lean toward reality may be a little slower, but they give you something far more valuable: confidence that the behaviour you see in production is the behaviour you exercised in tests.

Level 2: Module Integration Tests (Boundaries Under Load)

Slice tests tell you that a feature works in isolation. They do not tell you whether modules cooperate correctly once they start talking to each other. That gap is where module integration tests earn their keep.

A module integration test boots a module in a way that is close to reality. It uses real persistence. It uses real in-process messaging. And it exercises multiple slices together, allowing behaviour to flow across a boundary rather than stopping at it.

For example, creating a user publishes an event. The Billing module reacts to that event. A billing profile is created. There is no HTTP involved and no UI in the way, just behaviour moving from one module to another exactly as it would at runtime.

This level of testing answers a different question: do my modules interact the way I believe they do? When these tests fail, the cause is usually not a bug in business logic. It is an architectural failure. An event contract changed. A handler was not registered. A transaction boundary moved. A dependency leaked across a boundary.

That distinction matters, because architectural failures require architectural fixes. Treating them like logic bugs only papers over the real problem.

Level 3: Contract Tests (Protecting Module Agreements)

If modules communicate through contracts, those contracts deserve tests of their own. A contract test does not care how something is implemented. It cares about shape and meaning, and whether both sides still agree on what is being exchanged.

For example, an event is expected to contain specific fields. A query should return data in a particular form. A command should accept a defined set of inputs. None of this is about behaviour inside a module. It is about the agreement between modules.

These tests protect you from one of the most dangerous changes in a modular system: “I just renamed this property, nothing else uses it.” Something always uses it. The damage just happens quietly if nothing is watching. Contract tests live close to the contract itself. They run fast. They fail loudly. And when they fail, they fail for a good reason. They are cheap insurance against silent breakage in places where the system relies on trust between modules. If modules communicate through contracts, those contracts need to be treated as first-class citizens. A contract test exists for one purpose only, to protect the agreement between modules. It does not care how something is implemented internally. It cares about shape, semantics, and intent.

An event must contain specific fields with specific meaning. A query must return data in an expected form. A command must accept a defined set of inputs and reject anything else. These tests are not about business behaviour, they are about ensuring that two independently evolving pieces of the system still understand each other. They protect you from one of the most dangerous changes in modular systems, “I just renamed this property, nothing else uses it”. Something always uses it. The problem is that without contract tests, you do not find out until much later, and usually in a place far removed from the change.

Good contract tests live close to the contract. They run fast. They fail loudly. And when they fail, they tell you exactly what agreement was broken. They are cheap insurance against silent, slow-burn failures that otherwise erode confidence in the architecture over time.

Level 4: Architecture Tests (Your Early Warning System)

Architecture tests sit at the top of the stack, and they do not test behaviour at all. They test structure. That is precisely why they are so effective, and often so uncomfortable. These tests ask blunt questions. Does Billing depend on Users.Infrastructure? Does any module reference another module’s DbContext? Are internals leaking across assembly boundaries? None of these questions are about whether the application appears to work today.

They do not care if the app runs. They care if the architecture is honest. When an architecture test fails, it is usually because someone made a small, reasonable-looking change that would have turned into a serious problem six months later.

Architecture Tests Need Tooling Support

Architecture tests don’t work on intent alone. They need enforcement at build time.

In .NET, that usually means using a structural testing framework that can inspect assemblies and dependencies directly. The two I see used most often in real systems are:

• NetArchTest – simple, focused, and very effective for dependency rules

• ArchUnitNET – more expressive, closer to formal architectural modelling

The specific tool matters less than what you do with it.

A good architecture test doesn’t try to prove the system works. It tries to prove the system can’t cheat.

For example, this single test prevents one of the most damaging boundary violations you can make:

Types.InAssembly(typeof(BillingRoot).Assembly)
    .ShouldNot()
    .HaveDependencyOn("Users.Infrastructure")
    .GetResult()
    .IsSuccessful
    .Should()
    .BeTrue();

This test doesn’t care about behaviour. It doesn’t care about data. It cares about honesty.

When it fails, it fails because someone crossed a boundary they weren’t meant to cross. That’s exactly when you want the build to break.

If you only ever write one kind of architecture test, write this kind.

Where End-to-End Tests Actually Fit

End-to-end tests are not useless. They are just overused. In a modular monolith, their role is much narrower than many people expect.

They should be few in number and focused only on critical paths. Their job is not to validate business rules or edge cases. It is to confirm that the system is wired together correctly at the highest level. Authentication works. Routing is correct. Major workflows do not explode when exercised end to end.

If you find yourself relying on end-to-end tests to validate business logic, that is usually a sign that something is missing at a lower level. You are compensating for gaps in slice tests, module integration tests, or contract tests. That is a smell. End-to-end tests are expensive and blunt instruments. Used sparingly, they provide confidence. Used as a safety net for everything else, they slow teams down and hide the real problems.

The Testing Pyramid Rewritten

In practice, the balance looks very different from the traditional testing pyramid. In a modular monolith, the weight shifts toward the places where behaviour and boundaries actually live.

You want many slice tests, because they validate real use cases in isolation. You want a healthy number of module integration tests, because they confirm that modules cooperate the way you think they do. On top of that, you add a thin but deliberate layer of contract and architecture tests to protect agreements and structural integrity. End-to-end tests should exist, but only in very small numbers.

If your test suite is inverted, with lots of end-to-end tests propping up a weak foundation, something upstream is wrong. Tests are compensating for missing confidence elsewhere, and that is usually a sign that the architecture itself needs attention.

Tests as Architectural Pressure

Here’s something that doesn’t get said often enough:

If something is hard to test, it’s probably badly designed.

Modular monoliths surface this fast.

If you find yourself struggling to test:

• A slice

• A module boundary

• A permission rule

That struggle is feedback.

Ignore it and you’ll pay later.
Listen to it and the design improves.

Where We Are Now

At this point in the series, you have:

• Real module boundaries

• Feature-centric design

• Isolated data

• Honest communication

• Localised authorisation

• A testing strategy that reinforces all of it

What’s left is the question everyone eventually asks.

How do you know when a modular monolith is ready to split?

That’s not a technical question. It’s a judgment call

You’ve enforced real module boundaries.
You’ve organised behaviour into vertical slices.
You’ve isolated data with separate DbContexts.
You’ve stopped modules from chatting like overfriendly neighbours.

And then you hit authorisation.

Suddenly, every nice boundary you worked so hard to protect feels under threat.

Permissions need to be checked everywhere.
Identity is shared.
Policies feel global.
This is where most modular monoliths start leaking.

Why Authorisation Is So Dangerous Architecturally

Authorisation is one of the most deceptive parts of a system from an architectural point of view. It looks like a technical concern, something you can solve once and move on from, but in reality it is deeply business-driven. Every permission encodes a rule about who is allowed to do what, when, and why, and those rules evolve as the business evolves.

Because of that, authorisation cuts across modules. It touches almost every request. It changes over time. And it has a habit of attracting shortcuts, especially under pressure. Those shortcuts often sound reasonable in the moment. Let’s just inject a permission service everywhere. We will centralise all checks in one place. It is fine if modules know about roles.

That is how you end up with a system where every module depends on a shared authorisation core. Permission logic gets smeared across handlers and services. Changing a single rule means touching half the codebase. The modules may still compile separately, but architecturally they are glued together, and the boundaries you thought you had are no longer doing any real work.

The First Principle: Authorisation Is Not Ownership

The biggest mental shift that makes this work is understanding that a module does not own authorisation. A module owns its rules. That distinction is subtle, but it is foundational.

Identity, tokens, and enforcement are cross-cutting concerns. They are about how a decision is applied and verified. Meaning is not cross-cutting. Meaning lives with the business capability that understands it. A Billing module knows what it means to issue an invoice. A Users module knows what it means to deactivate a user. A Claims module knows what it means to approve a claim.

If you centralise the meaning of permissions, you lose modularity instantly. The moment a single place decides what actions really mean across the system, every module starts to depend on that shared interpretation. Boundaries blur, ownership weakens, and what looked like a clean separation turns into another hidden point of coupling.

The Wrong Way (And Why It’s So Tempting)

Here’s the classic mistake.

You create something like:

public static class Permissions
{
    public const string CreateUser = "users:create";
    public const string DeleteUser = "users:delete";
    public const string IssueInvoice = "billing:issue";
}

This pattern shows up quickly in otherwise well-structured systems. Someone introduces a shared permissions class with a set of constants, and every module references it. At first glance, it feels clean, centralised, and consistent. Everyone uses the same strings. There is one obvious place to look. It looks like order. In reality, every module now depends on a shared permissions model. The meaning of those permissions is no longer owned by the modules that enforce them, but by a central construct that everyone must agree on. When the semantics of a permission change, that change ripples outward across the entire system, whether the affected modules care about it or not.

What you have really created is a hidden coupling hub. The dependency may be subtle, but it is powerful. Modules that should be independent are now tied together through shared meaning and shared evolution. Architecturally, this is no better than a shared DbContext. It is just harder to see, and therefore easier to justify until the damage is already done.

Cross-Cutting

Cross-cutting does not mean globally defined. That is where a lot of designs quietly go wrong. Treating something as cross-cutting does not give it permission to own meaning for the entire system. In this context, cross-cutting means that enforcement is consistent. The mechanics of checking permissions, validating tokens, and rejecting unauthorised requests can and should be shared. Infrastructure can be reused. The plumbing does not need to be reinvented in every module.

Meaning, however, must stay local. Each module decides what a permission actually represents in terms of business behaviour. That distinction matters. When enforcement is shared but meaning is local, you get consistency without coupling, and that is the balance modular systems depend on.

The Shape of a Boundary-Respecting Authorization Model

Here’s the structure that holds up long-term:

In a modular system, each module declares its own permissions. It knows what those permissions mean, how they map to business rules, and when they should apply. Those permissions are used internally, close to the behaviour they protect, where their meaning is clear and unlikely to be misinterpreted.

The platform plays a different role. It authenticates users, resolves claims, and enforces checks in a consistent way. It provides the mechanism, not the meaning. No single module owns the whole picture, and thats by design. Meaning stays local to the module that understands it, while enforcement remains shared and predictable. That separation is what keeps authorisation powerful without turning it into another hidden source of coupling.

Permissions Are Part of the Use Case

A permission is not a generic rule. It’s part of a specific behaviour.

That means it belongs with the slice.

For example, in a Users module:

internal static class UserPermissions
{
    public const string Create = "users:create";
    public const string Deactivate = "users:deactivate";
}

Used directly in the slice that cares about it.

[Authorize(Policy = UserPermissions.Create)]
internal sealed class CreateUserEndpoint : IEndpoint
{
    // endpoint mapping
}

This permission:

• Is declared by Users

• Is used by Users

• Is meaningful only in Users

Billing doesn’t care. It shouldn’t even know it exists.

Policies Are Infrastructure, Not Business Logic

This is another common source of leakage.

Policies often become a dumping ground for logic:

services.AddAuthorization(options =>
{
    options.AddPolicy("CanCreateUser", policy =>
        policy.RequireClaim("role", "Admin"));
});

This approach looks innocent at first. Then roles change. Rules evolve. Context starts to matter. Before long, business logic is no longer living in the modules where it belongs, it is living in startup configuration and policy wiring, far away from the actual use cases that give it meaning. The better approach is a clean separation of responsibility. Policies should be mechanical. They exist to enforce checks, not to encode business rules. Meaning should live inside the modules, close to the behaviour it governs. Claims should represent capabilities, not roles or organisational structures.

A policy should answer a simple question:”Does this principal have permission X?” It should not be trying to answer something like “Is this user an admin in department Y during business hours?” That kind of logic depends on context, intent, and business rules, and it belongs near the use case where that context actually exists, not buried inside infrastructure configuration.

Claims as Capabilities, Not Roles

Roles are blunt instruments. They don’t scale well once systems grow.

Capabilities do.

Instead of:

Role: Admin

Think:

Permission: users:create
Permission: billing:issue
Permission: claims:approve

Your identity system emits capabilities.
Your modules decide what those capabilities mean.

This keeps the identity layer dumb and the domain layer expressive.

Avoiding the “Authorisation Service” Anti-Pattern

Another trap is the so-called “Authorisation Service”.

A shared service with methods like:

bool CanCreateUser(User user);
bool CanIssueInvoice(User user);

This kind of design often looks reusable at first glance. In reality, it is catastrophic for modularity. The problem is not the intent, it is the effect.

It centralises business rules that should belong to individual modules. It forces every module to depend on that central logic. And over time, it becomes a change bottleneck, because even small rule adjustments ripple through the entire system.

If a rule is about Users, it belongs in the Users module. If a rule is about Billing, it belongs in the Billing module. That alignment keeps ownership clear and change local.

Cross-cutting enforcement does not mean cross-cutting logic. Confusing the two is how systems quietly lose their modularity while still appearing well-structured on the surface.

Where Authorisation Checks Actually Belong

In a vertical-slice system, authorisation checks belong at the boundary of the slice. That is the point where intent is clearest, where you can say, “this operation is about to happen”, and make a deliberate decision about whether it should be allowed.

They should not be buried in repositories, where they become implicit and easy to bypass. They should not be hidden inside services, where they turn into invisible rules that only exist if you remember to call the right method. And they should not be scattered across helper utilities, where consistency depends on developer discipline. Put the check where the request enters the slice, at the point where the behaviour is named and explicit. That is where the system can tell the truth about what it is doing, and where authorisation can be enforced predictably without leaking into the wrong places.

public async Task<Result> Handle(
    CreateUserCommand command,
    ClaimsPrincipal principal,
    CancellationToken stopToken)
{
    if (!principal.HasPermission(UserPermissions.Create))
        return Result.Forbidden();

    // behaviour
}

That’s explicit. Honest. Local.

If the rule changes, you know exactly where to look.

Handling Shared Identity Without Shared Coupling

Identity is shared, and that part is not controversial. There is a single logged-in user and a single authentication story. But shared identity does not require shared interpretation. Each module maps that identity to meaning on its own terms, based on the business rules it owns. This is the difference between asking “who is this user?” and asking “what are they allowed to do here?” The first question is global and infrastructural. The second is local and contextual. Confusing the two is how modules start to leak meaning into places where it does not belong.

Authorisation mistakes rarely show up as obvious bugs. Instead, they surface as fear of change, overly defensive coding, centralised gatekeepers, and endless regressions. Teams slow down, not because the system is broken, but because nobody is confident that a change will not have unintended side effects. Systems that age well avoid that trap. They keep permission logic obvious, local, and boring. When authorisation is easy to understand and easy to change in one place, the system stays flexible, and the team keeps its momentum.

Up Next in the Series

The next inevitable question is:

How do you test all of this without writing brittle, end-to-end monsters or meaningless unit tests?

Testing modular monoliths requires a different mental model.

PARTS 6 - 10 HERE

Modules are now properly isolated. They own their data. They commit independently. They even fail independently.

And then you hit the next question:

How do these things actually talk to each other?

This is the point where a lot of otherwise solid modular monoliths quietly fall apart. Not all at once. Slowly. Politely. One innocent decision at a time.

A direct method call here.
A shared DTO there.
A synchronous dependency that “should be fine”.

Before long, you haven’t broken any rules explicitly, but you’ve recreated the very thing you were trying to escape.

The Trap: Treating Modules Like Classes

The most common mistake I see is treating modules like they are just big classes. The reasoning usually sounds harmless enough. They are in the same process, so why not just call across and get the answer directly? From a technical point of view, that is correct. From an architectural point of view, it is disastrous. The moment one module calls another synchronously and expects an immediate answer, a chain of coupling is introduced, whether you intended it or not. Execution order becomes fixed. Failure modes bleed across boundaries. Time suddenly matters in ways you did not plan for, and assumptions about internal behaviour start to leak out. What looked like a clean boundary turns into a thin veil. The module may still exist on the filesystem, but its autonomy is gone.

What You’re Actually Trying to Preserve

Before choosing any communication mechanism, it helps to be clear about what you are actually trying to preserve. The goal is not elegance or convenience in the short term. It is about protecting a set of properties that make the system resilient over time. You are trying to preserve autonomy, so each module can make its own decisions without being dragged into someone else’s execution flow. You are trying to preserve replaceability, so modules can evolve, change shape, or even be rewritten without forcing a cascade of changes elsewhere. You are trying to preserve honest failure, where problems surface clearly instead of being buried inside a larger operation. And you are trying to preserve extractability, so future you still has real options when the system needs to change.

If a communication style undermines any of those, it is the wrong choice, no matter how convenient or familiar it feels today. Convenience fades quickly. The consequences of broken boundaries tend to stick around much longer.

Three Ways Modules Can Communicate

In a modular monolith, there are really only three legitimate ways modules should talk to each other.

Everything else is a variation or a mistake.

1. Asking a Question (Synchronous, Contract-Only)

Sometimes a module genuinely needs to ask another module a question. Not to delegate work, and not to coordinate behaviour, but simply to retrieve information that the other module owns. This is the narrowest and safest form of synchronous communication you can allow. In cases like this, the intent is clear. Billing might need to know whether a user exists. An authorisation module might need to check permissions. A reporting feature might need a snapshot of reference data. In each case, the caller is asking for information, not trying to drive the other module’s behaviour.

The constraints around this kind of interaction are non-negotiable. You depend on a contract, never an implementation. You accept that the call can fail and design accordingly. And you do not embed business flow or orchestration logic into the response. When those rules are respected, synchronous queries can exist without eroding the autonomy of the modules involved.

This is not orchestration. It’s lookup.

If the answer disappears tomorrow and you have to replace it with a cache or a projection, nothing fundamental breaks.

2. Announcing Something Happened (Asynchronous, Event-Driven)

This is the most important pattern in the entire series.

When a module completes work it owns, it announces a fact, not an instruction.

“A user was created.”
“An invoice was issued.”
“A policy was cancelled.”

It does not care who reacts. It does not wait. It does not coordinate.

This preserves autonomy better than anything else you can do.

Each module:

• Decides if it cares

• Handles the event in its own time

• Fails independently

If Billing is down, Users still works.
If Notifications breaks, Billing doesn’t care.

That’s not a compromise. That’s the design doing its job.

3. Issuing a Command (Rare, Explicit, Dangerous)

This one should make you uncomfortable, and that discomfort is intentional. A command is not a notification and it is not a question. It is one module telling another module to do something. Not “something happened”, but “you must act”.

There are situations where this is genuinely necessary, but they should be rare. A command carries weight because it explicitly coordinates behaviour across boundaries. When you issue one, you are saying that your operation is not complete unless another module performs work on your behalf. That is a strong form of coupling, even when it is wrapped in a clean interface.

If you find yourself doing this often, your boundaries are lying to you. Either the modules are not as independent as you think, or the responsibility is split in the wrong place.

When commands are unavoidable, treat them with care. Make them explicit so their intent is obvious. Make them intentional so they are not introduced casually. And above all, make them rare, because every command chips away at the autonomy you are trying to preserve.

If this starts to feel like a workflow engine, that’s because it is. At that point, you should acknowledge it and design accordingly, not pretend it’s “just a call”.

The Illusion of Safety in Synchronous Chains

Here’s the pattern that causes the most damage:

On a diagram, this kind of flow looks neat. It is linear, predictable, and easy to explain. One box calls the next, work flows left to right, and everything appears nicely ordered.

In reality, a very different set of behaviours emerges. Latency stacks as each call waits on the next. Failures cascade across boundaries. Retries multiply in unexpected ways. Partial success becomes invisible because everything is hidden behind synchronous calls and assumptions of immediacy.

What you have really created is a distributed transaction in disguise, but without any of the tooling, visibility, or honesty that distributed systems demand. And worse, you have done it inside a monolith, where nobody is watching for those failure modes because the architecture claims they do not exist.

Why Events Feel Uncomfortable at First

Events tend to feel uncomfortable at first because most developers are trained to think in terms of control flow. You call this, then you call that, and if something goes wrong you roll everything back. The path is explicit, linear, and easy to follow in your head.

Events break that mental model. When you publish an event, you do not know who will react to it, when they will react, or in what order. That loss of immediate control can feel dangerous, especially if you are used to relying on transactions to keep everything tidy.

What actually changes is where correctness lives. Instead of being enforced by a single transactional boundary, correctness is enforced through idempotency, retries, observability, and explicit state management. These mechanisms are harder to fake and harder to ignore. They force you to deal with reality rather than hiding it behind rollback semantics, and that honesty is exactly what makes event-driven designs robust over time.

In-Process Messaging Is Not a Shortcut

In-process messaging is often misunderstood as a shortcut. Developers reach for a messaging library and tell themselves that it is just method calls with a bus in the middle. That assumption is where the trouble starts.

The bus is not an implementation detail. The bus is the boundary. If you treat it as invisible, you will design handlers that quietly rely on immediate execution, guaranteed ordering, and the absence of failure. Those assumptions hold only as long as everything stays exactly where it is.

The moment you move that bus out of process, all of those hidden assumptions surface at once, and things start breaking in surprising ways. The system was never designed for the realities it now has to face.

The right mental model is to design your in-process messaging as if it were already remote. Assume latency. Assume failure. Assume retries and reordering. If you do that, extracting the messaging infrastructure later becomes a mechanical change rather than a fundamental redesign, and the boundaries you established early continue to hold.

A Rule That Saves You Years Later

Heres a rule I have learned to trust over time. If a module needs to wait for another module to finish work, you probably have the wrong boundary. Not always, but often enough that it is worth pausing and questioning the design when it happens. Waiting implies coordination. Coordination implies shared responsibility. And shared responsibility implies coupling. Each step pulls the modules closer together, even if the code still looks clean on the surface.

When you notice this pattern, it is a signal to slow down and reassess. Either the work truly belongs in the same module, or the interaction should be reshaped so that one module can proceed independently. Catching this early can save you years of friction later on.

Living With Partial Failure

This is the part most architectures try hard to avoid. Partial failure feels messy and uncomfortable, so the instinct is to design it away. In reality, partial failure is normal. One module succeeds, another fails, and the system continues to run. When that happens, the response should be deliberate. You log what happened. You retry where it makes sense. You compensate if the business process requires it. And, crucially, you observe it so you can understand how often it occurs and why. What you do not do is hide that failure inside a transaction and pretend it never happened. That illusion might hold for a while, but it always leaks eventually, and it almost always leaks at the worst possible time, when the system is under pressure and the cost of surprises is highest.

Why

I dont want systems where I have to remember invisible rules. I do not want to rely on assumptions like “this always happens before that” when there is nothing in the system actually enforcing it. When I come back to a codebase after a break, or late at night when my brain is half-fried, I want the behaviour to be explicit. I want communication patterns that tell the truth about dependencies instead of hiding them behind convention or tribal knowledge. Modules that announce facts and react independently are easier to reason about. They are easier to monitor, easier to debug, and easier to evolve without fear. And they age better, which is something that matters far more than most people like to admit when the system is still young.

Bringing It All Together

So far in the series, we’ve established that:

• Boundaries must be enforced, not documented

• Behaviour belongs in vertical slices

• Data must be owned per module

• Communication must preserve autonomy

If you skip any one of these, the others weaken.

Get them all roughly right, and you end up with something rare, a monolith that doesn’t rot as it grows.

Up Next in the Series

Now that modules can talk safely, the next problem shows up immediately:

How do you apply cross-cutting concerns like authorisation without blowing holes through your boundaries?

Permissions, policies, and identity have a habit of leaking everywhere if you’re not careful.

If modules are truly independent, why are they still sharing a DbContext?

This is where most modular monoliths quietly cheat.

They talk about ownership and boundaries, but behind the scenes everything funnels through a single EF Core DbContext, backed by a single schema, and stitched together with navigation properties that bleed across modules.

The Shared DbContext Lie

Let’s call it what it is. A shared DbContext across modules is a lie, or at least a very convincing one. It claims that modules are independent while quietly giving them direct and implicit access to each other’s persistence concerns. On the surface everything looks clean, but underneath, the boundaries are already compromised. Once a shared DbContext exists, a familiar pattern starts to emerge. Someone adds a cross-module join “just this once” to save time. Navigation properties begin to grow tentacles, reaching into parts of the system they were never meant to know about. Performance tuning stops being a local concern and turns into a global exercise, because a change for one feature can ripple unpredictably across others.

The longer this goes on, the harder it becomes to reverse. When you eventually want to extract a module, whether into its own service or simply into a cleaner internal boundary, you discover that everything is entangled at the data level. What looked like a modular design turns out to be tightly coupled where it matters most.

At that point, you are not really building a modular monolith. You are building a monolith with folders. Its the architectural equivalent of what my Spanish friend says when I drift too far from his paella recipe. Once you start throwing in whatever is convenient, it stops being paella and becomes “arroz con cosas”, rice with things. It might still be edible, but it is no longer the thing you set out to make.

That is exactly what happens with a shared DbContext. The intent was modularity, but convenience takes over. What you end up with may still work, but it has quietly lost its identity, and undoing that damage later is far harder than getting it right up front.

What is Data Ownership?

Data ownership is another area where the language sounds clear but the reality often gets blurred. If a module owns a business concept, then it must own everything that makes that concept real in the system. That includes the schema, the mappings, the persistence rules, and the full lifecycle of the data from creation to deletion. Anything less than that is shared ownership, and shared ownership is where boundaries quietly fall apart. There is no such thing as “mostly owns” or “owns it except for reporting”. The moment another module can shape, query, or optimise that data on its own terms, ownership is already compromised. The module may still be responsible for the concept in theory, but in practice the data has become a shared resource. The architectural consequence of real ownership is simple and uncomfortable for some teams, one DbContext per module. Not one per feature, and not multiple bounded contexts hiding inside the same module. One module, one persistence boundary. That is what makes ownership explicit, enforceable, and durable as the system grows.

The Immediate Pushback

The pushback comes immediately, and to be fair, it is justified. The moment you say “multiple DbContexts”, the same questions surface almost every time. What about transactions? What if a single operation needs to update two modules? Isn’t this just distributed systems inside a single process?

These are good questions, and they are honest ones. They usually come from people who have been burned by data inconsistency or partial failures before, and who are rightly cautious about introducing new failure modes.

The mistake is not in asking those questions. The mistake is in trying to answer them with the wrong tools.

What a Multi-DbContext Modular Monolith Looks Like

At a structural level, it’s simple.

Users.Module
  UsersDbContext

Billing.Module
  BillingDbContext

Each DbContext:

• Lives inside its module

• Is internal to that module

• Only knows about its own aggregates

No shared base DbContext.
No shared migrations.
No shared entity configurations.

Here’s the mental model:

Same physical database if you want. Different schemas. Different contexts. Different ownership.

“But I Need a Transaction Across Modules”

This is the point where most people reach for the wrong answer. Faced with the idea of changing state in more than one module, the instinct is to try to stretch a transaction across the boundary and make the problem go away. The tooling makes this feel tempting, even safe, and there is always some variation of “the framework can handle it if we’re careful enough”.

And technically, some of that does work. You can coordinate multiple persistence operations, get everything to commit or roll back together, and walk away with a sense of consistency. On the surface, it looks like the cleanest solution.

Architecturally, it is a trap. The moment you rely on a shared transactional boundary, you have effectively collapsed the modules back into one. The boundary still exists in name, but not in behaviour. What you gain in short-term convenience, you lose in long-term modularity, flexibility, and the ability to evolve the system without fear.

Why Cross-Module Transactions Are a Smell

Here’s the uncomfortable truth. If two modules must commit atomically, then they are not independent. No amount of layering or careful naming changes that reality. Atomic consistency is a stronger signal than any diagram you can draw. That does not automatically mean your design is bad. It means the boundary is wrong. What you have separated conceptually does not match how the business actually needs the system to behave. The architecture is telling you something, and it is usually worth listening.

In practice, there are only two real possibilities. Either the modules are genuinely part of the same consistency boundary and should be treated as such, or the operation does not truly require atomic consistency and you are reaching for it out of convenience. Most systems blur this line, choosing the comfort of transactions instead of questioning whether the boundary itself makes sense.

Strong Consistency Is Rarely Needed

Imagine a Users module and a Billing module. You create a user and, as part of that flow, you also create a billing profile. The reflex is to assume that both of those actions must succeed or fail together, wrapped in a single atomic transaction. But step back and ask what actually matters. The user needs to exist. Billing needs to know about that user. That is the real requirement. If billing finds out about the new user 50 milliseconds later, nothing meaningful breaks. There is no business catastrophe hiding in that gap.

This is where strong consistency quietly reveals itself as optional rather than mandatory. Inside a monolith, eventual consistency is not a compromise or a failure of design. In many cases, it is the more honest reflection of the business process. By allowing modules to communicate asynchronously and converge on consistency over time, you preserve boundaries, reduce coupling, and end up with a system that is easier to reason about as it grows.

The Correct Pattern: Local Transactions + Events

The correct pattern is much simpler than it first appears, local transactions combined with events. Inside a module, nothing exotic is required. You use a normal EF Core transaction, make your changes, and commit them. The module stays fully in control of its own data and its own consistency. Once that work is complete, the module publishes an event to say that something meaningful has happened. That event is not an implementation detail, it is a deliberate signal to the rest of the system. It represents the only sanctioned way for other modules to react.

That event becomes the boundary crossing point. Other modules can listen, respond, and update their own state in their own time, using their own transactions. Consistency is achieved through coordination rather than coupling, and the integrity of each module’s boundary remains intact.

No shared DbContext.
No distributed transaction.
No lies.

“But What If Billing Fails?”

Good. Now we’re talking about reality. Failure is not an edge case, it is the normal state of complex systems. This is exactly the point where layered monoliths tend to hide that reality by forcing everything through a single transaction. It feels safe because nothing appears to fail, but it is also brittle, because all failure is collapsed into one silent rollback.

With a modular approach, failure becomes explicit. When modules communicate through events, you do not pretend that everything always succeeds. You design for the fact that things can and will fail. Recovery becomes intentional rather than accidental, and system state becomes observable instead of being hidden behind a transaction boundary. If Billing fails, the user still exists. The event that announced the new user can be retried. The failure is visible, traceable, and something you can respond to. That is not a step backwards. It is an honest representation of how the system actually behaves, and honesty is what gives you control when things go wrong.

The Outbox Pattern

Inside the Users module:

using var tx = await db.Database.BeginTransactionAsync(stopToken);

db.Users.Add(user);
db.Outbox.Add(new OutboxMessage(
    "UserCreated",
    new UserCreatedEvent(user.Id)));

await db.SaveChangesAsync(stopToken);
await tx.CommitAsync(stopToken);

Billing never touches Users’ DbContext.
It just reacts to what Users emits.

This is the same pattern that lets you split later without rewriting everything.

What About Read Models?

Another common objection shows up quickly. “But I need to join Users and Billing for queries”. This is usually framed as a hard requirement, but it is really a question about reads, not ownership. Reads do not define ownership. Writes do. The fact that you want to view data together does not mean it should be stored or managed together. Conflating the two is how boundaries get eroded under the guise of convenience. If you genuinely need a combined view, the answer is a read model. You build a projection that listens to the relevant events, materialise a model that is shaped for querying, and optimise it for the questions you actually need to answer. That model can live wherever it makes the most sense, without punching holes through module boundaries. Yes, this means duplication. And yes, that duplication is intentional. Data is copied because it serves a different purpose. It is fine. In fact, it is often the cleanest way to keep writes honest while still giving reads the flexibility they need.

Up Next in the Series

Now that we’ve:

1. Enforced boundaries

2. Structured behaviour

3. Isolated data

The next problem shows up immediately:

How do modules talk to each other without turning into a distributed monolith?

That’s where in-process messaging, contracts, and intent-driven communication come in.

But once you’ve done that work, something else becomes painfully obvious.

You open a module and… it’s still chaos.

Not architectural chaos. Organised chaos. The kind that looks tidy at first glance but slows you down every time you touch it. Folders called Controllers, Services, Repositories. Logic smeared across three or four layers. You change a feature and end up hopping between files like you’re playing whack-a-mole.

This is where most modular monoliths stall.

They have boundaries, but inside those boundaries they’re still layered.

And layered architecture optimises for one thing only, explaining code to a diagram. It does not optimise for change.

The Real Unit of Change Is a Feature

Here’s the core idea behind vertical slice architecture, and it’s deceptively simple:

Code should be organised around things that change together.

Not around technical concerns. Not around frameworks. Around features.

When a product owner asks for a change, they don’t ask you to “update the service layer and adjust the repository abstraction”. They ask you to change behaviour. That behaviour should live in one place.

When I’m deep in a system late at night, after the house has finally gone quiet, I don’t want to reconstruct a feature in my head from five folders. I want to open one place and see the whole story.

Vertical slices give you that.

What Layered Architecture Looks Like in Practice

Most of us have built this:

Users/
  Controllers/
    UsersController.cs
  Services/
    UserService.cs
  Repositories/
    UserRepository.cs
  Models/
    User.cs

On paper, it looks reasonable. Responsibilities are separated. Everything has a place.

In reality, a single endpoint touches all of it.

Change the behaviour of “Create User” and you’re editing:

• A controller

• A service

• A repository

• Possibly a validator

• Possibly a mapper

The logic is scattered. The mental overhead is high. And the risk of breaking something unrelated creeps up over time.

What a Vertical Slice Actually Is

A vertical slice is everything needed for one use-case, grouped together.

Not just the handler. Not just the endpoint. Everything.

Here’s the same Users module, sliced vertically:

Users/
  CreateUser/
    CreateUserEndpoint.cs
    CreateUserHandler.cs
    CreateUserCommand.cs
    CreateUserValidator.cs
  GetUser/
    GetUserEndpoint.cs
    GetUserHandler.cs
    GetUserQuery.cs
  UsersModule.cs

Each folder answers a single question:

“How does this feature work?”

You don’t jump around. You don’t search across the solution. You stay inside the slice.

Vertical Slices Inside a Module (Not Instead of Modules)

Vertical slice architecture does not replace modular architecture. It lives inside it.

The hierarchy looks like this:

Modules define ownership and boundaries.
Slices define behaviour and flow.

If you skip modules and go straight to slices, you end up with feature soup. If you skip slices and stick to layers, you end up with tightly coupled sludge.

You need both.

One Request, One Handler, One Path

A rule I follow almost religiously:

One request = one handler = one execution path

No shared “UserService” with 17 methods. No god-objects. No orchestration hidden behind abstractions.

Here’s a simple example.

The request

public sealed record CreateUserCommand(
    string Email,
    string DisplayName
);

The handler

internal sealed class CreateUserHandler(
    IUserDbContext db,
    IClock clock
)
{
    public async Task<Result<Guid>> Handle(
        CreateUserCommand command,
        CancellationToken ct)
    {
        var user = new User(
            Guid.NewGuid(),
            command.Email,
            command.DisplayName,
            clock.UtcNow);

        db.Users.Add(user);
        await db.SaveChangesAsync(ct);

        return Result.Success(user.Id);
    }
}

No layers. No indirection. The behaviour is right there.

If validation fails, it fails here. If persistence changes, it changes here. If rules evolve, this is the file you open.

“But Won’t This Duplicate Code?”

Yes.

Sometimes.

And that’s fine.

Layered architecture optimises for reuse. Vertical slices optimise for clarity and change.

If two features genuinely share logic, extract it. But don’t pre-emptively abstract “just in case”. That’s how shared services become dumping grounds.

I’ve learned to trust duplication far more than premature reuse. Duplication is obvious. Coupling is subtle.

Validation Lives With the Slice

Another mistake I see a lot is “centralised validation”.

It sounds sensible until you realise validation rules are part of the behaviour, not infrastructure.

internal sealed class CreateUserValidator
    : AbstractValidator<CreateUserCommand>
{
    public CreateUserValidator()
    {
        RuleFor(x => x.Email)
            .NotEmpty()
            .EmailAddress();

        RuleFor(x => x.DisplayName)
            .NotEmpty()
            .MaximumLength(100);
    }
}

This validator belongs with CreateUser. Not in a shared folder. Not in a generic pipeline that hides rules from the feature they apply to.

When a rule changes, you want to see it next to the behaviour it affects.

Endpoints Are Just Adapters

In a vertical slice, endpoints become very boring. That’s a good thing.

internal sealed class CreateUserEndpoint : IEndpoint
{
    public void MapEndpoint(IEndpointRouteBuilder app)
    {
        app.MapPost("/users", async (
            CreateUserCommand command,
            CreateUserHandler handler,
            CancellationToken stopToken) =>
        {
            var result = await handler.Handle(command, stopToken);
            return result.Match(
                dto => Results.Created($"/users/{id}", dto),
                r => Results.BadRequest(r));
        });
    }
}

No logic. No branching. No decisions. The endpoint adapts HTTP to your slice and gets out of the way.

Why This is important as the System Grows

This is the part people consistently underestimate. Vertical slices do not feel revolutionary when the system is small and everyone still remembers how everything fits together. Early on, almost any structure feels workable. The payoff only becomes obvious months later, when the codebase has grown, the team has changed, and the original mental model has faded. As the system evolves, clean boundaries let you delete features without collateral damage. You can remove an entire slice and be confident that you have not silently broken unrelated behaviour elsewhere. Refactoring becomes safer because changes stay contained. You are working inside a known boundary instead of tiptoeing through a shared codebase, hoping nothing unexpected snaps. That same structure also keeps future options open. If a slice starts to demand independent scaling, ownership, or deployment, it is already shaped in a way that can be extracted into a service. You are not forced into that move, but you are not blocked by your architecture either. The decision becomes a trade-off, not a rescue mission.

On a more human level, this structure matters when you are under pressure, switching context between work and home life, or simply mentally tired. Clear slices reduce friction. You do not have to re-learn the entire system every time you touch it. You can step into one area, make a change, and step back out again with confidence.

That is the real long-term value. Future-you gets fewer surprises, fewer late-night debugging sessions, and a system that continues to make sense even after the original decisions are no longer fresh in your head.

Vertical Slices and Testing

Testing looks very different once you commit to vertical slices, because the slice itself becomes the unit under test. You stop testing vague layers like “the service layer” or “the application layer” and start testing concrete behaviours. Instead of asking whether the system works in general, you ask whether CreateUser does exactly what it claims to do. Handler tests focus on behaviour. Given a valid request, does the handler produce the correct outcome and side effects? Validator tests are narrower and stricter. They exist to prove that the rules are enforced consistently, regardless of where the request comes from. Endpoint tests sit one level higher again, validating that routing, binding, authorisation, and wiring are correct.

The important part is what you do not need. There are no sprawling test fixtures that try to boot half the system. There are no magic mocks that exist only to satisfy an overly broad abstraction. Each test has a clear purpose and a tight scope, mirroring the structure of the slice itself. Because each slice stands on its own, the test suite stays readable as it grows. When a test fails, you know exactly where to look and why it matters. The structure of the code and the structure of the tests reinforce each other, which is exactly what you want in a system that is expected to evolve over time.

A Quiet Benefit: Easier Onboarding

There is a quieter benefit to vertical slices that I did not fully appreciate at first, and that is onboarding. When someone new joins the team, the system does a surprising amount of the explaining for you. Instead of walking them through layers, conventions, and unwritten rules, you can point to a single feature and say, “everything for that behaviour is here”.

That simple framing removes the need for an architectural lecture on day one. A new developer does not have to build a mental map of the entire codebase before they can be productive. They can open one slice, see the endpoint, the handler, the validation, and the tests, and understand how the system works by following a real, concrete example. Thats more important than most people like to admit. Onboarding is where hidden complexity shows up fast, and it is also where architectural decisions either pay dividends or create friction. Vertical slices reduce that friction by making the structure of the system obvious, discoverable, and grounded in real behaviour rather than abstract rules.

Bringing It Back to Reality

I don’t build systems like this because it’s fashionable. I build them this way because I want to stay productive over the long haul. I have so many side projects going at any given time that it needs to be easy for me to switch contexts quickly and be able to get moving without learning the layout again after a break working on it.

Modules give you safety.
Vertical slices give you speed.

You need both.

Up Next in the Series

Now that we’ve:

1. Enforced boundaries between modules

2. Structured behaviour inside modules

The next hard problem shows up immediately:

How do you isolate data per module without breaking consistency?

That’s where multiple DbContexts, transactions, and reality collide.

That’s what we’ll tackle next.

I’ve been building .NET systems for a long time now, and I’ve learned this the hard way, architecture that relies on discipline alone will eventually fail. Not because people are careless, but because people are busy. They’re under pressure. They’re trying to ship. And sometimes, that one little shortcut feels harmless.

This first post in the series is about how to stop relying on discipline and start enforcing boundaries. Not with dogma. Not with over-engineering. But with practical techniques that work in real .NET codebases.

Why “Modular” Usually Isn’t

Most so-called modular monoliths fail for one simple reason, nothing actually stops one module reaching into another.

At first, everything feels fine. The codebase is small. You remember where things live. You even review every PR yourself. But time passes. New features land. New developers join. Context fades.

Then you see it:

var user = _userRepository.GetById(userId);

That line doesn’t live in the Users module.

It lives in Billing.

Now Billing knows about Users’ persistence model. It knows how Users stores data. And worse, it now depends on Users being present, initialised, and shaped exactly the way Billing expects. The boundary is gone.

At that point, you don’t really have modules. You have folders with opinions.

What a Real Boundary Actually Means

A real boundary is not a folder, a namespace, or a naming convention. It is something you physically cannot cross by accident. If a developer can casually reach across it with a reference, an import, or a “quick” helper method, then the boundary is already broken. Real boundaries create resistance. They force you to slow down and make a conscious decision when you want to interact with another part of the system. The compiler should be your first line of defence. When boundaries are real, the compiler actively helps you stay honest by refusing to compile code that reaches into places it has no business touching. You are not relying on discipline, code reviews, or comments to enforce the architecture. The rules are encoded in the structure of the system itself, and violations show up immediately, not six months later during a refactor.

When a boundary is violated, it should be obvious and slightly painful. You should feel friction when you try to bypass it. If breaking a boundary feels easy, invisible, or harmless, then it was never a real boundary to begin with. At that point, it is purely decorative architecture, something that looks good on a diagram but collapses under day-to-day development pressure.

In practice, this means a module must own its data completely. No other part of the system should be able to reach directly into its tables, collections, or persistence models. The module also controls how others talk to it, exposing only explicit entry points that reflect real use cases rather than internal implementation details.

Most importantly, a module must hide its internals entirely. This is the part people most often get wrong. They allow “just this one” internal type to leak out, or they expose a repository or entity because it feels convenient. Every one of those shortcuts weakens the boundary. Once internals leak, the module stops being a module and starts becoming a shared code bucket with aspirations of structure.

The Shape of a Proper Modular Monolith

Before we get into enforcement, it’s worth being clear about the target shape.

Here’s the mental model I use.

Each module:

• Exposes a public surface

• Keeps everything else private

• Communicates through contracts, not internals

No module reaches directly into another module’s database, repositories, or EF models. If it needs something, it asks.

Assemblies Are Your First Line of Defence

If you take only one thing from this post, let it be this:

Folders do not enforce boundaries. Assemblies do.

The moment you put everything in a single project, you’ve already lost most of your leverage. Yes, you can be disciplined. Yes, you can rely on conventions. But the compiler cannot help you anymore.

A solid baseline is one assembly per module.

/src
  /Modules
    /Users
      Users.Application
      Users.Domain
      Users.Infrastructure
    /Billing
      Billing.Application
      Billing.Domain
      Billing.Infrastructure
  /Api

Already, you’ve gained something important, explicit references. Billing does not magically see Users unless you tell it to.

But we’re not done yet.

Internal by Default, Public by Exception

One of the most underused features in .NET is the internal keyword. Most people default to public and never look back. That’s a mistake.

Inside a module, almost everything should be internal.

Entities. Repositories. EF configurations. Handlers. Services.

Public types should be rare and deliberate.

For example, in the Users module:

namespace Users.Application.Contracts;

public interface IUserLookup
{
    Task<UserSummary> GetUserAsync(Guid userId);
}

That interface is public. It’s the door into the module.

This, on the other hand, is not:

internal sealed class UserRepository : IUserRepository
{
    // implementation
}

Billing never sees this. Cannot see this. And that’s exactly the point.

Friend Assemblies: A Controlled Escape Hatch

“But what about tests?”

This always comes up.

You don’t make everything public just so tests can poke at it. You use friend assemblies.

In your module’s AssemblyInfo:

[assembly: InternalsVisibleTo("Users.Tests")]

Now your tests can see internals, but no other module can. This keeps the production boundary intact while still allowing deep testing.

Forcing Access Through Contracts

Forcing access through contracts is what turns an architectural idea into something that actually holds up under pressure. A boundary is only real if there is a single, sanctioned way in. The moment there are multiple paths, helper shortcuts, or back doors, the boundary starts to erode. People will always take the path of least resistance, especially when deadlines are tight. In a modular monolith, that single entry point is almost always a contract. Other modules do not reach in and grab what they want. They ask for something to be done. That shift, from data access to intention, is subtle but crucial. It forces you to think in terms of behaviour and outcomes rather than structures and tables.

Interfaces are one common way of expressing that contract. They define what a module is willing to do, not how it does it. Request and response models push this idea further by making interactions explicit and self-contained. Instead of passing around internal types, you exchange messages that represent a specific use case.

In-process messaging takes the same principle and applies it even more strictly. Rather than direct calls, modules communicate by sending requests or publishing events inside the same process. The module decides how to handle them, and everyone else is deliberately kept at arm’s length. The result is a system where access is intentional, boundaries are enforced by design, and accidental coupling becomes much harder to introduce.

Here’s a simple pattern that works well.

Users exposes a contract

public sealed record GetUserQuery(Guid UserId);

public sealed record UserResult(Guid Id, string Email);

public interface IUserQueries
{
    Task<UserResult?> GetAsync(GetUserQuery query);
}

Users implements it internally

internal sealed class UserQueries : IUserQueries
{
    public async Task<UserResult?> GetAsync(GetUserQuery query)
    {
        // EF Core, Dapper, whatever
    }
}

Billing depends only on the contract

public sealed class InvoiceService
{
    private readonly IUserQueries _users;

    public InvoiceService(IUserQueries users)
    {
        _users = users;
    }
}

Billing does not know how Users works. It cannot reach into it even if someone wants to.

Blocking “Just This Once” Shortcuts

The most dangerous phrase in software architecture is:

“It’s just this once.”

This is where enforcement comes in.

Assembly Reference Rules

Assembly reference rules are where modular boundaries stop being theoretical and start being enforceable. If a module exposes a single contracts assembly, that is the only thing other modules are allowed to reference. In this case, Billing should reference Users.Application.Contracts and nothing else. That rule should be obvious, and non-negotiable. The moment someone tries to add a reference to Users.Infrastructure or Users.Domain, something should break. Ideally the build fails. At the very least, alarms should go off loudly enough that the violation cannot slip through unnoticed. If those references are possible and nothing complains, then the boundary exists only by convention, and conventions are fragile under real-world pressure.

You enforce these rules first through solution structure. Projects are laid out so that the intended references are obvious and the forbidden ones feel unnatural. On top of that, you add explicit project reference rules, making it mechanically impossible to depend on the wrong assemblies without deliberately bypassing the design.

Then, you back it all up with architecture tests. These tests don’t care about business logic or behaviour. Their job is simply to assert that the dependency graph stays within the lines you have drawn. When someone crosses a boundary, the test fails, the build goes red, and the conversation happens immediately, not months later when the coupling has already spread.

Architecture Tests That Actually Help

Here’s a simple example using NetArchTest.

[Test]
public void Billing_Should_Not_Depend_On_Users_Infrastructure()
{
    var result = Types.InAssembly(typeof(BillingRoot).Assembly)
        .ShouldNot()
        .HaveDependencyOn("Users.Infrastructure")
        .GetResult();

    result.IsSuccessful.Should().BeTrue();
}

This test doesn’t care about business logic. It cares about structure. When it fails, it fails loudly, and early.

That’s the kind of test that saves you months later.

Database Boundaries Are Non-Negotiable

If one module can read another module’s tables, you do not have a modular monolith. You have a shared database with delusions of grandeur.

Each module owns its schema.

The Cost of Boundaries (And Why It’s Worth Paying)

Let’s be honest. This approach has a cost.

You will:

• Write more interfaces

• Pass more data explicitly

• Feel friction early on

But here’s what you get in return:

• Predictable change

• Safer refactoring

• Modules you can actually extract later

I’ve seen systems where adding a feature meant touching 14 projects because everything was intertwined. I’ve also seen systems where a change stayed neatly inside one module. The difference was never intelligence. It was boundaries.

The Human Side of This

I’ll end on something less technical.

I do a lot of my thinking late at night. after everyone else is asleep. Sometimes after reading a bedtime story to my daughter and coming back with a cup of tea, knowing I’ve got an hour to make sense of a problem before sleep wins.

In those moments, the last thing I want is a system that fights me. I want code that tells me when I’m about to do something stupid. I want boundaries that protect future-me, not just present-me.

That’s what enforcement gives you.

What’s Next in the Series

Now that boundaries are enforced, the next question becomes:

How do you design inside a module?

That’s where vertical slices come in, and that’s where we’ll go next.

The trouble starts when that same mental model is carried into a distributed system.

As systems grow, you split responsibilities into services, introduce messaging, and isolate data stores. At that point, the guarantees that made UOW safe begin to dissolve. Rather than abandoning the pattern, many people try to stretch it across service boundaries, coordinating commits or wrapping multiple operations in a higher-level “distributed” transaction.

This is where the abstraction breaks down.

Martin Fowler puts it bluntly when discussing microservices and transactions:

“You can’t maintain ACID transactions across microservices.”

That single sentence quietly invalidates the idea of a distributed UOW. UOW is an ACID-coordinating pattern. Its entire purpose is to provide atomic commit and rollback across a set of changes. If ACID cannot span services, then neither can UOW. This is not a tooling limitation or a framework gap. It is a consequence of running code across processes, machines, and networks that fail independently. Once a UOW crosses a distributed boundary, the assumptions it relies on no longer hold, even if the code still looks correct.

From that point on, the question is no longer “How do we implement distributed Unit of Work safely?”
The real question becomes “What patterns replace it when atomicity is no longer possible?”

The Illusion of Distributed Atomicity

Distributed UOW usually shows up disguised as coordination. One service acts as a conductor, asking each participant whether it is ready to commit. This is the foundation of two-phase commit, and it looks convincing on paper. In practice, it creates a system that is tightly coupled, slow under load, and fragile under failure. The coordinator can crash after some participants have committed and others have not. A participant can commit successfully but fail before acknowledging. A retry can arrive after a partial commit and cause duplicate work. None of these scenarios are edge cases. They are normal behaviour in a real distributed environment.

The deeper issue is that the network itself is unreliable. Messages can be delayed, reordered, duplicated, or lost. UOW assumes a level of synchrony and trust that the network cannot provide.

When services are forced to coordinate a single commit, they must all be alive and responsive at the same time. A slow or unhealthy service does not just fail its own work, it blocks everyone else. Locks are held longer. Throughput drops. Timeouts increase. Recovery becomes manual. In theory, the system is “more correct”. In reality, it is down more often.

This is why many distributed systems that start with coordinated transactions eventually remove them under production pressure. The cost shows up in incidents, not in code reviews.

The Shift in Thinking

The failure of distributed UOW forces a change in mindset. Instead of asking, “How do I make everything commit together?”, the question becomes, “How do I make each step reliable on its own?” This is the pivot from atomicity to durability. Modern distributed systems accept that work happens in stages. Each stage commits locally. Communication between stages is durable. Failures are expected, retried, and compensated for rather than magically rolled back. Once you accept this, the replacement patterns start to make sense.

The first rule is simple: transactions stop at the service boundary.

Inside a service, you still use UOW. Entity Framework’s DbContext remains a perfectly valid abstraction. You load data, apply changes, and commit once. That part does not change. What changes is the expectation that this transaction somehow covers the rest of the system. It doesnt, and it never will.

Anything that crosses the boundary is treated as asynchronous and unreliable by default.

Outbox as the New Commit Boundary

The transactional outbox pattern exists precisely to bridge the gap between local atomicity and distributed communication.

Instead of updating the database and publishing a message as two separate actions, both are recorded inside the same local transaction. The database change represents what happened. The outbox record represents what needs to be communicated. Once the transaction commits, the system is in a consistent state. Even if the process crashes immediately afterwards, the intent to publish the message is safely stored.

Later, a background process reads the outbox and delivers messages until they succeed.

This approach does not pretend that messaging is atomic with database writes. It acknowledges that messaging is eventually reliable and builds around that reality.

Once you introduce retries, you introduce duplicates. In a distributed system, messages can be delivered more than once. Any design that assumes otherwise will eventually corrupt data. The replacement for “exactly once” delivery is idempotent handling. Every message is treated as something that may already have been processed. The handler checks, records, and moves on. This makes retries safe. It allows consumers to crash and restart. It allows operators to replay messages from a dead-letter queue without fear. Most importantly, it removes the psychological need for a distributed UOW. You no longer rely on perfect coordination to avoid duplicates, because duplicates are harmless by design.

Sagas Replace Distributed Rollback

UOW relies on rollback as its safety net. If something fails, everything is undone. In distributed systems, rollback is replaced by compensation. A saga is a sequence of local transactions, each with a defined compensating action. Instead of pretending the work never happened, the system acknowledges that it did happen and applies an explicit reversal. Charging a card can be compensated with a refund. Reserving inventory can be compensated by releasing it. Creating a shipment can be compensated by cancelling it.

This is more work than rollback, but it is also more honest. The system reflects reality rather than hiding it behind abstractions.

In simple cases, services can react to events without central coordination. In more complex workflows, orchestration becomes valuable. A process manager or workflow engine tracks which step has completed, which is pending, and which compensations are required. State transitions are persisted. Timeouts are handled deliberately. Retries are visible. This structure replaces the false simplicity of distributed UOW with explicit control. You can see where a process is stuck. You can reason about partial completion. You can resume or compensate without guesswork.

That visibility is impossible when everything is hidden behind a single commit call.

Why the Pattern Is So Tempting

Distributed UOW is tempting because it feels familiar. It allows developers to pretend they are still working in a monolith, just stretched across the network. But the complexity does not disappear. It accumulates in places that are harder to debug: locks, timeouts, partial commits, and recovery scripts run at three in the morning!

The patterns that replace it feel more complex at first because they surface reality instead of hiding it. Over time, they reduce surprises, incidents, and data corruption.

None of this means UOW is obsolete.

It remains a great pattern inside a service boundary. It remains the right abstraction for coordinating changes within a single database. The mistake is letting it leak beyond that boundary.

Once the boundary is crossed, the rules change.

The failure of UOW in distributed systems is not a tooling issue or a framework limitation. It is a mismatch between assumptions and reality. Distributed systems do not offer global atomicity. They offer unreliable communication, partial failure, and eventual delivery. Designs that embrace those properties succeed. Designs that fight them collapse under load.

If you stop trying to make a distributed system behave like a local one, the architecture becomes clearer, the failure modes become manageable, and the system becomes something you can actually operate in production.

You also dont get as many call outs at 3:00 am when all the over night processes collapse!

As a C# engineer, that immediately raised a practical question. If Rust is becoming the new systems language inside Microsoft, and if more of the runtime, networking stack, and platform tooling will eventually depend on it, then learning Rust now feels like a safe bet rather than a speculative one.

The question is how to approach it.

There is no shortage of Rust tutorials, books, and courses. Many of them are excellent. Most of them also assume you are either new to programming or moving from another low-level language. If you already spend your day working in high-level C#, a lot of that material feels like friction. You end up re-learning concepts you already understand, just with different syntax.

I saw the same advice from experienced developers who had already crossed this bridge. If you already have strong experience and a high level, the fastest way to make Rust click is not to start with toy examples. Its to build something low-level immediately, where Rust’s constraints actually matter.

I decided to build a project.

The idea

The goal was simple on the surface.

Build a small networking system where a sender sends messages, a receiver processes them, and a separate process visualises what is happening internally. Nothing clever. No abstractions for the sake of it.

Under the surface, the goal was more specific.

I wanted to force myself to deal with explicit framing and byte handling, async IO in a real socket loop, and a clearer separation between transport, parsing, and application logic. Ownership and borrowing showed up too, mainly around buffers and task boundaries, though I haven’t pushed into the deeper lifetime-heavy patterns yet.

At the same time, I wanted to keep one foot in familiar territory. I didnt want to abandon C# entirely while learning Rust. So I split the system across two machines.

I had access to a Mac and a PC on the same network so I used that to my advantage.

The Rust receiver runs on the Mac.
The C# sender and telemetry visualiser run on the PC.

That split turned out to be more useful than I expected.

High-level architecture

The system consists of three very small programs.

A C# sender
A Rust receiver
A C# telemetry visualiser

The sender connects directly to the receiver over TCP and sends framed JSON messages. Each message has a length prefix so the receiver can read full frames cleanly.

The receiver reads the raw bytes, parses the JSON, and emits telemetry events as the message conceptually moves through layers. These layers are not pretending to be the real OS network stack. They are a teaching tool. They make invisible steps visible.

The receiver sends telemetry over UDP to the visualiser. UDP keeps the telemetry path simple and non-blocking.

The visualiser listens on a UDP port and renders a live terminal UI showing messages moving through application, transport, network, and link lanes.

There is no router process. No segmentation. No configurable packet sizes. I deliberately removed all of that. The goal here is learning Rust, not building a networking framework.

Why Rust on the receiver side

Putting the receiver in Rust was a deliberate choice.

In C#, reading from a network stream is trivial. You can build something that works in minutes, but it hides a lot of detail. Buffers are managed for you. Memory ownership is implicit. Errors often surface late.

In Rust, you cannot avoid those details.

Reading a length prefix forces you to think about endianness.
Reading into a buffer forces you to manage capacity and resizing.
Passing data between functions forces you to think about ownership.

None of this feels academic when you are dealing with a real socket.

use anyhow::Context;
use bytes::BytesMut;
use chrono::Utc;
use serde::{Deserialize, Serialize};
use std::{env, net::SocketAddr, sync::Arc};
use tokio::{
    io::AsyncReadExt,
    net::{TcpListener, TcpStream, UdpSocket},
    time::{sleep, Duration},
};

#[derive(Debug, Deserialize)]
struct Packet {
    packet_id: u64,
    created_at_ms: u64,
    payload: String,
}

#[derive(Debug, Serialize)]
struct TelemetryEvent<'a> {
    ts_ms: i64,
    source: &'a str, // "receiver"
    packet_id: u64,
    layer: &'a str, // "link" | "network" | "transport" | "application"
    kind: &'a str,  // "enter" | "deliver" | "error"
    note: &'a str,
    size: usize,
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let data_addr = env::var("DATA_BIND").unwrap_or_else(|_| "0.0.0.0:9000".to_string());
    let telemetry_target =
        env::var("TELEMETRY_TARGET").unwrap_or_else(|_| "127.0.0.1:9100".to_string());

    println!("Receiver listening on {}", data_addr);
    println!("Telemetry target: {}", telemetry_target);

    let listener = TcpListener::bind(&data_addr).await?;

    let udp = Arc::new(UdpSocket::bind("0.0.0.0:0").await?);
    let telemetry_addr: SocketAddr = telemetry_target
        .parse()
        .context("TELEMETRY_TARGET must be like 192.168.0.10:9100")?;

    loop {
        let (socket, peer) = listener.accept().await?;
        println!("Accepted connection from {}", peer);

        let udp = udp.clone();
        tokio::spawn(async move {
            if let Err(e) = handle_connection(socket, peer, udp, telemetry_addr).await {
                eprintln!("Connection error ({}): {:?}", peer, e);
            }
        });
    }
}

async fn handle_connection(
    mut socket: TcpStream,
    peer: SocketAddr,
    udp: Arc<UdpSocket>,
    telemetry_addr: SocketAddr,
) -> anyhow::Result<()> {
    loop {
        // Read u32 big-endian length prefix
        let mut len_buf = [0u8; 4];
        if socket.read_exact(&mut len_buf).await.is_err() {
            println!("Connection closed by {}", peer);
            return Ok(());
        }

        let frame_len = u32::from_be_bytes(len_buf) as usize;

        // Read frame bytes
        let mut buf = BytesMut::with_capacity(frame_len);
        buf.resize(frame_len, 0);
        socket.read_exact(&mut buf).await?;

        // Parse JSON
        let packet: Packet = match serde_json::from_slice(&buf) {
            Ok(p) => p,
            Err(_) => {
                let _ = send_event(
                    &udp,
                    telemetry_addr,
                    TelemetryEvent {
                        ts_ms: Utc::now().timestamp_millis(),
                        source: "receiver",
                        packet_id: 0,
                        layer: "transport",
                        kind: "error",
                        note: "json_parse_failed",
                        size: frame_len,
                    },
                )
                    .await;
                continue;
            }
        };

        // Emit layer events (simple, deterministic)
        send_event(
            &udp,
            telemetry_addr,
            TelemetryEvent {
                ts_ms: Utc::now().timestamp_millis(),
                source: "receiver",
                packet_id: packet.packet_id,
                layer: "link",
                kind: "enter",
                note: "frame_received",
                size: frame_len,
            },
        )
            .await?;
        sleep(Duration::from_millis(layer_delay_ms())).await;

        send_event(
            &udp,
            telemetry_addr,
            TelemetryEvent {
                ts_ms: Utc::now().timestamp_millis(),
                source: "receiver",
                packet_id: packet.packet_id,
                layer: "network",
                kind: "enter",
                note: "packet_parsed",
                size: frame_len,
            },
        )
            .await?;
        sleep(Duration::from_millis(layer_delay_ms())).await;

        send_event(
            &udp,
            telemetry_addr,
            TelemetryEvent {
                ts_ms: Utc::now().timestamp_millis(),
                source: "receiver",
                packet_id: packet.packet_id,
                layer: "transport",
                kind: "deliver",
                note: "delivered_to_app",
                size: frame_len,
            },
        )
            .await?;
        sleep(Duration::from_millis(layer_delay_ms())).await;

        send_event(
            &udp,
            telemetry_addr,
            TelemetryEvent {
                ts_ms: Utc::now().timestamp_millis(),
                source: "receiver",
                packet_id: packet.packet_id,
                layer: "application",
                kind: "deliver",
                note: "payload_ready",
                size: frame_len,
            },
        )
            .await?;
        sleep(Duration::from_millis(layer_delay_ms())).await;

        // Console breakdown
        println!("---");
        println!("From:            {}", peer);
        println!("Frame length:    {} bytes", frame_len);
        println!("packet_id:       {}", packet.packet_id);
        println!("created_at_ms:   {}", packet.created_at_ms);
        println!("payload:         {}", packet.payload);

        // Raw preview (first 96 bytes)
        let preview_len = buf.len().min(96);
        let preview = &buf[..preview_len];
        println!("raw preview ({}): {:02X?}", preview_len, preview);
    }
}

async fn send_event(udp: &UdpSocket, target: SocketAddr, ev: TelemetryEvent<'_>) -> anyhow::Result<()> {
    let bytes = serde_json::to_vec(&ev)?;
    udp.send_to(&bytes, target).await?;
    Ok(())
}

fn layer_delay_ms() -> u64 {
    env::var("LAYER_DELAY_MS")
        .ok()
        .and_then(|v| v.parse().ok())
        .unwrap_or(120)
}

The receiver reads four bytes for the frame length, then reads exactly that many bytes into a buffer. It then attempts to deserialise JSON from that buffer. If deserialisation fails, it emits an error event and moves on.

Once the packet is parsed, the receiver emits telemetry events as it conceptually moves through layers. Each event includes a timestamp, packet id, layer name, event kind, and size.

Finally, the receiver prints a detailed breakdown to the console. Peer address. Frame size. Packet id. Timestamp. Payload contents. A raw hex preview of the first bytes. Nothing hidden.

This is where Rust started to click.

The compiler forces you to be explicit. Once the code builds, you know the data flow is correct. There is very little ambiguity.

Why keep the sender and visualiser in C

I kept the sender in C# for two reasons.

First, speed. I didnt want to spend time building input handling and framing logic from scratch while still learning Rust. In C#, that part is muscle memory.

Second, comparison. Writing the sender in C# makes the differences obvious. You can see how much the runtime gives you for free, and what Rust expects you to manage yourself.

The telemetry visualiser also stays in C#. Spectre.Console makes it easy to build a clean terminal UI quickly. More importantly, it keeps the visualisation code separate from the Rust learning exercise.

Rust does one job. The visualiser does another.

That separation matters.

Running across two machines

Running the Rust receiver on a Mac and the C# processes on a PC adds one more layer of realism.

This is not a loopback demo. Real sockets are involved. Real IP addresses. Real failure modes. You immediately run into practical issues. Address binding. Firewalls. Ports already in use. Processes exiting unexpectedly. All of that is part of systems work, and Rust does not shield you from it. It also reinforces a useful mental model. Rust is not replacing C# for application development. It is complementing it. The two languages sit at different levels of the stack, and they interact over simple, explicit protocols. That feels very close to how modern platforms are actually built.

What this project taught me

The biggest takeaway is that Rust makes more sense when you stop trying to learn it in isolation.

If you come from C#, you already understand concurrency, async workflows, and data modelling. Rust is not trying to re-teach those concepts. It is forcing you to be explicit about things the runtime normally hides. Ownership is not abstract when you are passing buffers around. Lifetimes are not theoretical when a socket read depends on them. Error handling feels stricter, but also more honest. Building a real, low-level project immediately surfaces the value Rust brings. You stop fighting the language and start seeing why it exists.

Where next?

Im deliberately stopping this project here. It works. Its understandable. It does exactly what I need it to do. If I extend it later, it will be with a clear purpose. Maybe adding a router process. Maybe experimenting with reordering or loss. Maybe rebuilding the receiver in a different async model.

For now, this was enough. I got what I wanted from it.

It sparked my interest enough to think of a different more involved bigger project to take on next time.

If you are a C# engineer thinking about learning Rust, my advice is simple. Do not start with the beginner examples. Pick a problem where memory, and data flow matter. Build something small but real, get your hands dirty & have fun.

Most workflow engines start from the wrong place.

They begin with diagrams, XML, modelling tools, and an assumption that your business logic wants to be expressed as a flowchart. In practice, most production systems already have workflows. They just live in code, databases, queues, APIs, and human decisions. The missing piece isnt orchestration logic. Its visibility.

All you might want is a reliable way to answer simple questions. What step is this case on? How long did it spend there? Who touched it? What changed? Why did it fall from the happy path? Those questions are about tracking, not modelling.

Below, we’ll think about how we could build a stripped-down workflow tracker in .NET. No BPMN. No visual modeller. No engine deciding what happens next. The application decides that. Our system records what happened, when it happened, and why. Then it makes that data easy to query.

The result looks a lot like the most valuable parts of Camunda, but without the weight.

The core idea

A workflow is not a graph. It is a sequence of events over time.

Every time a piece of work moves forward, something observable happens. A step starts. A step completes. A decision is made. Data changes. Someone intervenes. If we capture those events in a consistent way, we can reconstruct the workflow after the fact with surprising power.

The tracker does not control execution. It listens.

This single constraint simplifies everything. There is no retry logic here. No compensation. No tokens moving through gateways. Your application owns those concerns already. The tracker’s only responsibility is to record transitions and state changes with strong guarantees.

Defining a workflow instance

We start by defining what a workflow instance means in our system.

An instance represents one unit of work moving through a process. That might be a loan application, an insurance submission, a customer onboarding case, or a background job. The tracker does not care.

An instance has a stable identifier, a workflow type, and some high-level metadata. It is created once and never deleted.

In C#, that looks like a simple aggregate.

public sealed class WorkflowInstance
{
    public long Id { get; init; }
    public string WorkflowType { get; init; }
    public string ExternalReference { get; init; }
    public DateTimeOffset CreatedAt { get; init; }
}

The ExternalReference is critical. This is how the rest of your system relates back to the workflow. It might be a ProgramId, SubmissionId, OrderId, or something similar.

Modelling steps as facts, not definitions

Traditional engines define steps up front. We dont.

Instead, every time the application reaches a meaningful point, it emits a step event. A step is identified by a string key that has meaning to the domain.

Examples might be submission_received, assessed, or review_started.

The tracker does not validate these names. It records them.

A step event captures when a step started, when it completed, who or what performed it, and whether it ended successfully.

public sealed class WorkflowStepEvent
{
    public long Id { get; init; }
    public long WorkflowInstanceId { get; init; }
    public string StepKey { get; init; }
    public StepStatus Status { get; init; }
    public DateTimeOffset Timestamp { get; init; }
    public string? Actor { get; init; }
}

The status might be Started, Completed, Failed, or Cancelled. That is enough to reconstruct timelines and durations later.

This model deliberately allows repeated steps. If a case goes back to manual review three times, you will see three distinct events. That turns out to be extremely valuable for analytics.

Capturing decisions and data changes

Steps alone are not enough. Many workflows branch based on data. We want to record those decisions without baking logic into the tracker.

Whenever the application makes a decision that affects flow, it emits a decision event.

public sealed class WorkflowDecisionEvent
{
    public long Id { get; init; }
    public long WorkflowInstanceId { get; init; }
    public string DecisionKey { get; init; }
    public string Outcome { get; init; }
    public DateTimeOffset Timestamp { get; init; }
}

This might represent something like risk_outcome = refer or eligibility = declined. The tracker does not care how the decision was made. It just records the fact that it happened.

Optionally, we can also capture variable snapshots. These are key-value pairs recorded at points of interest.

public sealed class WorkflowVariableSnapshot
{
    public long WorkflowInstanceId { get; init; }
    public string Name { get; init; }
    public string Value { get; init; }
    public DateTimeOffset RecordedAt { get; init; }
}

This is not a full variable store. It is an audit trail. You record what mattered, when it mattered.

Writing events safely

The most important technical requirement is durability. If your application says a step completed, the tracker must not lose that fact.

The simplest approach is an append-only relational schema. Inserts only. No updates except for correcting mistakes explicitly.

Every call into the tracker is a single database transaction that inserts one or more events. There is no orchestration state to lock or mutate. In practice, this scales extremely well.

You can expose the tracker via a small internal API.

public interface IWorkflowTracker
{
    Task RecordStepAsync(
        long instanceId,
        string stepKey,
        StepStatus status,
        string? actor,
        CancellationToken stopToken);

    Task RecordDecisionAsync(
        long instanceId,
        string decisionKey,
        string outcome,
        CancellationToken stopToken);
}

The application calls this at natural boundaries. After a handler completes. When a background job finishes. When a user clicks approve.

This keeps the integration friction low.

Reconstructing the workflow timeline

Once events are stored, the interesting part begins.

To reconstruct the current state of a workflow, you query all events for an instance ordered by timestamp. From that stream, you derive projections.

You can compute the current step by finding the latest Started event without a corresponding Completed or Failed event. You can calculate durations by pairing Started and Completed timestamps. You can detect loops by counting repeated step keys.

This logic lives in read models, not the write path.

public sealed class WorkflowTimeline
{
    public IReadOnlyList<StepExecution> Steps { get; init; }
    public IReadOnlyList<DecisionRecord> Decisions { get; init; }
    public TimeSpan TotalElapsed { get; init; }
}

Because everything is immutable, rebuilding projections is deterministic and safe. You can even change projection logic later and re-run it over historical data.

Analytics

This is where the tracker earns its keep, you can now answer questions like:

How long does each step take on average?
Where do cases get stuck?
How often do we bounce back to manual review?
Which decisions correlate with long cycle times?
How many workflows are active right now and where are they sitting?

These are simple SQL queries over event tables. No engine internals. No proprietary formats.

Because step keys are just strings, teams can evolve workflows without migrations. New steps appear naturally in analytics as they are used.

Heat Mapper

One of the main selling points of Camunda is the the view that highlights where instances spend time (hot spots) and how often paths are taken.

You could build a very similar UI even without BPMN. You just need to choose a visual shape for your workflow, then paint it with metrics.

The simplest approach is to render your workflow as a directed graph where each step key is a node and each observed transition between steps is an edge. You don’t need a modeller. You infer the graph from real executions: for each workflow instance, sort step events by time and emit transitions like stepA -> stepB. Aggregate those transitions across all instances and you get a live map of how the process actually runs.

Once you have nodes and edges, you compute heat metrics:

• Node heat (time): average or p95 time spent in that step. You get this by pairing Started and Completed timestamps per step execution.

• Node heat (volume): how many instances entered the step in a time window.

• Edge heat (frequency): how often a transition happens, optionally as a percentage of all outgoing transitions from the source step.

• Stuck heat: count of instances currently “in” a step (Started with no Completed/Failed yet), and how long they’ve been there.

From a UI point of view, you can build it in React with a graph library like React Flow, Cytoscape.js, or D3. You lay out nodes (either auto-layout or a simple left-to-right rank layout), draw edges, and then colour nodes/edges based on the selected metric. Add a time window filter (last 24h, 7d, 30d), environment filter, and a toggle between average and p95, and you’ve got something that feels very close to Camunda’s operational heatmaps.

A practical structure that works well is:

• Overview page: “Workflow map” with heat applied (time or frequency).

• Step drilldown: click a node to see distributions (p50/p95), failure rate, top decision outcomes, and a list of slowest instances.

• Instance timeline: click an instance to see the ordered event stream with durations, actors, and variable snapshots at key points.

The key limitation is that you won’t automatically get a nice “business diagram” unless you define one. But that’s often a feature, not a bug. Your map becomes an honest picture of runtime behaviour. If you want it to look more like a designed process, you can optionally let teams maintain a lightweight “layout config” (node positions, grouping into lanes, friendly labels) without turning it into BPMN.

Comparing this to a full workflow engine

This approach deliberately gives up control in exchange for clarity.

You cannot model flows visually here. You cannot press a button to advance a token. That is intentional. Those features are expensive to operate and rarely reflect how systems actually behave. What you gain is observability, auditability, and freedom. Your application logic stays in code where it belongs. The tracker becomes a shared language across teams. In many places, this covers eighty percent of the value people think they need BPMN for.

When this approach is not enough

There are real cases where orchestration engines make sense. Long-running sagas with retries and compensation. Complex asynchronous dependencies across systems. Human task scheduling with SLAs and escalations.

The key insight is that you do not need to start there.

A lightweight tracker often becomes the foundation even when a full engine is later introduced. It provides ground truth. It tells you what your workflows actually look like, not what the diagram says they should look like.

If you strip workflow down to its essence, it is just time, decisions, and movement. By recording those facts cleanly, you unlock powerful insight without forcing your system into a modelling straightjacket.This kind of tracker is fast to build, cheap to run, and easy to explain.