# Your Distributed Lock Expired. Your Code Didn't.

A distributed lock can expire while the code that acquired it is still running. Another instance takes ownership, finishes its work and saves the result. Then the original instance resumes and overwrites it. The lock service can behave exactly as designed throughout that sequence. It grants a lease, lets it expire and grants another one. The failure happens because the application treats acquiring a lease as permission to keep writing for as long as its method takes to finish. In a .NET service, that assumption can hide behind a very reassuring `await using` block. The lifetime of the local object is visible in the code. The lifetime of the remote permission is somewhere else.

To protect shared state, the destination needs enough information to reject a worker whose ownership has been superseded. Fencing tokens provide that information. The useful part is understanding where they are issued, where they are checked and precisely what that check guarantees.

## The worker that came back

Imagine a reporting service running across several containers. Each instance can rebuild a customer's report, but we want only one instance publishing that report at a time. We add a lease around the operation. This example uses an application defined lease abstraction, it isnt the API of a particular locking package.

```csharp
public sealed class ReportRebuilder(
    IDistributedLeaseProvider leases,
    IReportBuilder builder,
    IReportRepository reports)
{
    public async Task RebuildAsync(
        Guid reportId,
        CancellationToken stopToken)
    {
        await using var lease = await leases.AcquireAsync(
            $"report:{reportId}",
            TimeSpan.FromSeconds(30),
            stopToken);

        var report = await builder.BuildAsync(reportId, stopToken);

        await reports.ReplaceAsync(reportId, report, stopToken);
    }
}
```

Instance A acquires the lease and starts building. It then stops making progress. For a reproducible example, suspend the process in a debugger. We dont need to assume that a particular garbage collection pause will last thirty seconds. After the lease expires, instance B acquires ownership, builds its report and publishes it. When A resumes, its next instruction is still there. Its local lease object still exists. Unless something actively prevents the write, A publishes too.

![](https://cdn.hashnode.com/uploads/covers/67c36038c69a4b7143c5fc49/b3a164ab-7d9b-43bd-b464-6e3d7be5e9b3.png align="center")

The important failure is a former owner continuing after ownership changes. A delayed request can cause the same problem even when the worker itself is no longer running. Martin Kleppmann's [analysis of distributed locking](https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html) explains why pauses and delayed writes must be considered when protecting a resource with a lease.

## Renewal and cancellation

Renewing the lease is useful. If report generation takes a variable amount of time, renewal helps the current worker retain ownership while it makes progress. However, the renewal loop runs on infrastructure that can also stop making progress. A suspended process cannot reliably renew its lease or react to losing it. Redis's [distributed locking documentation](https://redis.io/docs/latest/develop/clients/patterns/distributed-locks/) explicitly recommends fencing tokens and warns against assuming that a lease remains held while its original process is alive. Increasing the expiry moves the failure further away and increases recovery time after a crash. It doesnt give the destination any way to recognise a former owner. Cancellation still belongs in the implementation. When renewal fails, cancel ongoing computation and stop issuing new operations. In .NET, though, [cancellation is cooperative](https://learn.microsoft.com/en-us/dotnet/standard/threading/cancellation-in-managed-threads). Requesting it cannot retract a write that another system has already accepted.

Even a check immediately before publishing leaves a gap:

```csharp
stopToken.ThrowIfCancellationRequested();

await reports.ReplaceAsync(reportId, report, stopToken);
```

Ownership can change after that check. The report database needs to evaluate the worker's authority as part of accepting the mutation.

## Give ownership a generation

A fencing token is an ordered ownership generation. One acquisition receives `41`; the next receives `42`. A worker includes its generation when accessing the protected resource, which enforces the ordering. In the common high water mark design, the destination remembers the largest generation it has accepted. After accepting `42`, it rejects `41`. Kleppmann describes this resource side check in the [fencing section of his article](https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html#making-the-lock-safe-with-fencing).

A random owner identifier serves a different purpose: it identifies a particular acquisition, for example when deciding whether a release is allowed. It doesnt establish which acquisition came later. There is a subtle limit to the high water mark approach. If B acquires generation `42` but the destination still only knows about `41`, an arriving write carrying `41` may still be accepted. Acquiring a newer generation elsewhere does not instantly update every protected resource. For our report publisher, we can make the handover stronger. The report already lives in SQL Server, so we'll keep its current ownership generation in the same row. Acquiring ownership changes that row immediately. Publication must match the generation currently stored there. This is an application specific design for publishing into one database. It avoids a separate lock service for this example and gives us a single place to order acquisition and publication.

## Keep ownership beside the report

The row contains both coordination state and the published result. It is created when the report is provisioned, before workers attempt to acquire it.

```sql
CREATE TABLE dbo.ReportPublication
(
    ReportId uniqueidentifier NOT NULL
        CONSTRAINT PK_ReportPublication PRIMARY KEY,

    Generation bigint NOT NULL
        CONSTRAINT DF_ReportPublication_Generation DEFAULT (0),

    LeaseOwner uniqueidentifier NULL,
    LeaseUntilUtc datetime2(7) NULL,

    PublishedGeneration bigint NOT NULL
        CONSTRAINT DF_ReportPublication_PublishedGeneration DEFAULT (0),

    Payload nvarchar(max) NULL,
    PublishedAtUtc datetime2(7) NULL
);
```

`Generation` records the current ownership epoch. `PublishedGeneration` records which epoch supplied the visible result. They differ while a newly acquired worker is still building, and they can remain different if that worker fails. The previous report stays available during a rebuild. Taking ownership doesn't clear the payload. Acquisition increments the generation and installs an owner in a single conditional update:

```sql
UPDATE dbo.ReportPublication
SET Generation = Generation + 1,
    LeaseOwner = @owner,
    LeaseUntilUtc = DATEADD(second, @leaseSeconds, SYSUTCDATETIME())
OUTPUT inserted.Generation
WHERE ReportId = @reportId
  AND (
      LeaseUntilUtc IS NULL
      OR LeaseUntilUtc <= SYSUTCDATETIME()
  );
```

One returned value means the acquisition succeeded. No returned row means the report is missing or a lease currently prevents acquisition. A SQL exception is an operational failure and must not be converted into "lease busy". The generation increment belongs to the ownership transition. Allocating a number in the worker before acquiring ownership would allow allocation order and acquisition order to diverge. Generations only need to be ordered within the protected report. Report X and report Y can both be at generation `42`. The counter must persist across releases, so releasing ownership must not delete this row or reset its generation.

## Publish with a conditional write

Publication carries the exact acquisition identity back to the database:

```sql
UPDATE dbo.ReportPublication
SET Payload = @payload,
    PublishedGeneration = @generation,
    PublishedAtUtc = SYSUTCDATETIME()
WHERE ReportId = @reportId
  AND Generation = @generation
  AND LeaseOwner = @owner
  AND LeaseUntilUtc > SYSUTCDATETIME()
  AND PublishedGeneration < @generation;
```

The generation and owner conditions reject superseded workers. The expiry condition also refuses publication when the lease has already expired at the database's admission check. The final condition allows one publication per generation, preventing two competing results from the same acquisition from both being accepted. SQL Server evaluates the condition as part of the update. We dont load the row into application memory, check its generation and later issue an unconditional save. Microsoft's [UPDATE reference](https://learn.microsoft.com/en-us/sql/t-sql/queries/update-transact-sql) documents the statement's filtering and locking behaviour. Both acquisition and publication modify the same existing row. If B's acquisition wins that ordering, A's publication no longer matches. If A's publication wins, it is ordered before B's acquisition. There is no successful publication by A after a committed handover to B. The expiry predicate is an admission check, not a promise that the transaction commits before a particular wall clock instant. We rely on the row's transactional ordering to protect the handover. Publication should remain a short database operation.

![](https://cdn.hashnode.com/uploads/covers/67c36038c69a4b7143c5fc49/1c6029ee-3aa6-4c87-9f28-4bed5a7236d1.png align="center")

Notice that B doesnt need to have published anything before A is rejected. Its successful acquisition has already installed the fence.

## Calling it from .NET

The following adapter targets SQL Server using `Microsoft.Data.SqlClient`. It assumes the table above, provisioned rows and no triggers that change affected row reporting. It deliberately uses short, autocommitted statements; report generation happens outside a database transaction.

```csharp
using System.Data;
using Microsoft.Data.SqlClient;

public sealed record ReportLease(
    Guid ReportId,
    Guid Owner,
    long Generation);

public sealed class ReportPublicationStore(string connectionString)
{
    public async Task<ReportLease?> TryAcquireAsync(
        Guid reportId,
        int leaseSeconds,
        CancellationToken stopToken)
    {
        ArgumentOutOfRangeException.ThrowIfNegativeOrZero(leaseSeconds);
        var owner = Guid.NewGuid();

        await using var connection = new SqlConnection(connectionString);
        await connection.OpenAsync(stopToken);
        await using var command = connection.CreateCommand();

        command.CommandText = """
            UPDATE dbo.ReportPublication
            SET Generation = Generation + 1,
                LeaseOwner = @owner,
                LeaseUntilUtc = DATEADD(second, @leaseSeconds, SYSUTCDATETIME())
            OUTPUT inserted.Generation
            WHERE ReportId = @reportId
              AND (LeaseUntilUtc IS NULL OR LeaseUntilUtc <= SYSUTCDATETIME());
            """;

        command.Parameters.Add("@reportId", SqlDbType.UniqueIdentifier).Value = reportId;
        command.Parameters.Add("@owner", SqlDbType.UniqueIdentifier).Value = owner;
        command.Parameters.Add("@leaseSeconds", SqlDbType.Int).Value = leaseSeconds;

        var result = await command.ExecuteScalarAsync(stopToken);

        return result is long generation
            ? new ReportLease(reportId, owner, generation)
            : null;
    }

    public async Task<bool> TryPublishAsync(
        ReportLease lease,
        string payload,
        CancellationToken stopToken)
    {
        await using var connection = new SqlConnection(connectionString);
        await connection.OpenAsync(stopToken);
        await using var command = connection.CreateCommand();

        command.CommandText = """
            UPDATE dbo.ReportPublication
            SET Payload = @payload,
                PublishedGeneration = @generation,
                PublishedAtUtc = SYSUTCDATETIME()
            WHERE ReportId = @reportId
              AND Generation = @generation
              AND LeaseOwner = @owner
              AND LeaseUntilUtc > SYSUTCDATETIME()
              AND PublishedGeneration < @generation;
            """;

        AddLeaseParameters(command, lease);
        command.Parameters.Add("@payload", SqlDbType.NVarChar, -1).Value = payload;

        return await command.ExecuteNonQueryAsync(stopToken) == 1;
    }

    public async Task ReleaseAsync(
        ReportLease lease,
        CancellationToken stopToken)
    {
        await using var connection = new SqlConnection(connectionString);
        await connection.OpenAsync(stopToken);
        await using var command = connection.CreateCommand();

        command.CommandText = """
            UPDATE dbo.ReportPublication
            SET LeaseOwner = NULL,
                LeaseUntilUtc = NULL
            WHERE ReportId = @reportId
              AND Generation = @generation
              AND LeaseOwner = @owner;
            """;

        AddLeaseParameters(command, lease);
        await command.ExecuteNonQueryAsync(stopToken);
    }

    private static void AddLeaseParameters(
        SqlCommand command,
        ReportLease lease)
    {
        command.Parameters.Add("@reportId", SqlDbType.UniqueIdentifier).Value = lease.ReportId;
        command.Parameters.Add("@owner", SqlDbType.UniqueIdentifier).Value = lease.Owner;
        command.Parameters.Add("@generation", SqlDbType.BigInt).Value = lease.Generation;
    }
}
```

Release carries the generation and owner too. If A eventually reaches its cleanup code after B has acquired the report, A's release affects zero rows. It cannot clear B's ownership. The orchestration can now distinguish successful publication from a refused write. `IReportBuilder` below is application specific and returns the report as a string.

```csharp
public sealed class FencedReportRebuilder(
    ReportPublicationStore publications,
    IReportBuilder builder,
    ILogger<FencedReportRebuilder> logger)
{
    public async Task RebuildAsync(
        Guid reportId,
        CancellationToken stopToken)
    {
        var lease = await publications.TryAcquireAsync(
            reportId, leaseSeconds: 30, stopToken: stopToken);

        if (lease is null)
            return;

        try
        {
            var payload = await builder.BuildAsync(reportId, stopToken);

            var published = await publications.TryPublishAsync(
                lease, payload, stopToken);
            if (!published)
            {
                logger.LogWarning(
                    "Publication refused for report {ReportId}, generation {Generation}",
                    reportId, lease.Generation);
            }
        }
        finally
        {
            using var cleanup = new CancellationTokenSource(TimeSpan.FromSeconds(5));

            try
            {
                await publications.ReleaseAsync(lease, cleanup.Token);
            }
            catch (Exception exception)
            {
                logger.LogWarning(
                    exception,
                    "Release failed for report {ReportId}; lease will expire",
                    reportId);
            }
        }
    }
}
```

Cleanup gets a separate, bounded cancellation token because the request token may already be cancelled. The catch belongs specifically to best effort release, acquisition and publication errors still propagate. Failing to release temporarily delays another worker, while expiry permits eventual recovery. This example intentionally has no renewal loop. A build that exceeds the lease loses its opportunity to publish. If the workload needs renewal, renew only when the report ID, owner and generation still match and the lease remains unexpired. A late renewal must not revive a superseded acquisition.

## What a refused write means

A zero row publication is a refusal, but it doesn't identify one particular cause. Ownership may have changed, the lease may have expired, or that generation may already have published. The handler therefore logs "publication refused", rather than asserting that another worker definitely took over. There is a separate ambiguity when SQL Server commits a publication but the connection fails before the client receives confirmation. Retrying the same publication then affects zero rows because `PublishedGeneration` already matches. The write predicate prevents a second publication; it doesnt provide a complete acknowledgement recovery protocol.

If the caller needs a durable answer, persist a publication operation ID and its outcome in the same transaction as the payload, then resolve an uncertain result by that ID. Retain the outcome independently if later report versions must not erase the evidence. A timeout alone cannot tell the caller whether publication happened. After losing ownership, discard the old build. Reacquiring a new lease and attaching its generation to previously computed output would authorise old work under a new ownership epoch. Build again under the new acquisition, or explicitly validate the source revision before publishing. Even then, ownership order and source data freshness are separate concerns. A worker can own the latest generation while reading an older replica. If the report must reflect a particular source version, carry and validate that version as part of the publication contract.

## Test the handover

The useful integration test controls the order of operations. It doesn't rely on hoping that two tasks collide at the right time. Create a fresh report row and let A acquire it. Hold A immediately before publication using a test barrier. In the isolated test database, set A's expiry into the past, then let B acquire the row. Before allowing B to publish, release A's barrier and attempt publication with A's original lease. A must affect zero rows. B must then publish successfully. Query the database and assert that its payload and `PublishedGeneration` belong to B. This ordering specifically catches implementations that only reject old generations after a newer publication. Repeat the schedule with A attempting release after B acquires. B's owner and generation must remain unchanged. Also exercise publication after expiry with no successor, and a duplicate publication using the same lease. Both must be refused by this design.

Use a separate SQL connection for each worker. An in-memory repository cannot verify SQL Server's concurrency behaviour. Add a genuine overlapping publication acquisition test against the deployments isolation settings, and fault injection coverage for connection loss around commit. A sequential unit test of a number comparison cannot establish the database protocol's behaviour. These examples are illustrative application code, not a supplied, executed integration test project. The important assertions are the accepted database mutations and durable state, rather than which worker happened to log an error.

## The destination defines the protection

This fence protects the report row. It doesnt automatically protect a blob, an email or an external API call performed during `BuildAsync`. Suppose the builder overwrites a shared blob and the database subsequently refuses publication. The blob has already changed. For a report stored outside SQL, one option is to upload an immutable artefact under a unique key, then use the fenced database update to publish its pointer. Readers must follow that pointer; they must not discover and consume unpublished artefacts independently. Failed workers leave unreferenced objects that can be cleaned up later.

Where the destination provides concurrency controls, enforce them there. [Azure Blob Storage supports lease IDs and conditional ETag operations](https://learn.microsoft.com/en-us/azure/storage/blobs/concurrency-manage). Those mechanisms protect operations on the relevant blob; acquiring a lease on a separate coordination blob does not automatically protect every other object your application writes. For a third party side effect, the provider must expose semantics that address the failure you need to prevent. An idempotency key can suppress repeated execution of one logical operation. It cannot, by itself, decide that a different operation belongs to an obsolete owner. Likewise, recording a fence in SQL and then calling an external API leaves a gap unless the receiving system participates in the protocol.

## Keep the generation durable

The SQL design assumes a single authoritative database history. Its generation and payload changes must survive the failures the application claims to tolerate. An asynchronous failover that loses acknowledged transactions can roll that history back, a restored backup can do the same. Operational recovery therefore belongs in the design. Before a restored copy accepts work, quiesce old workers and connections and prevent requests from the previous database history reaching it. If that cannot be guaranteed, introduce a recovery epoch that is itself durably issued and enforced. Resetting a counter while old requests remain possible destroys the ordering on which fencing depends.

The adapter also assumes trusted workers use the prescribed writes. A generation is ordering metadata, not an authentication credential. For stronger enforcement, expose acquisition and publication through stored procedures and grant the service permission to execute those procedures without granting arbitrary updates to the underlying table. Every publication path must preserve the check, including repair scripts and administrative jobs. An unconditional update in a second code path can bypass the entire protocol. When reviewing a distributed lock, follow the work all the way to its final mutation. Suspend the old owner, let another instance take over, then resume the original request. The decisive evidence is the destination rejecting that request. A lease object that still exists in memory tells you very little about whether the worker is still entitled to write.