# How to Investigate a Growing Container Without Guessing

A .NET container uses 600 MB after startup, 850 MB by lunchtime and 1.2 GB by the end of the day. Someone calls it a memory leak and asks for a heap dump. The graph has established that memory use is growing but It hasnt established what is growing. The application could be retaining managed objects, the GC could have committed space it expects to use again, a native library could be allocating outside the managed heap, or file backed pages could be charged to the container. A heap dump is useful for some of those cases and almost irrelevant to others.

I start by lining up three views of memory over the same time window - the container's charge, the process's memory and the .NET runtime's heap. The differences between them usually tell me what to inspect next.

## Confirm what the graph measures

"Container memory" is not one universal number. In Kubernetes, `kubectl top` uses the resource metrics pipeline and reports a working set view. The Kubernetes documentation notes that this can include some cached file backed memory. A dashboard showing cgroup usage, container working set or process resident memory can therefore produce different numbers for the same workload. Check the metric name before interpreting its slope. Record the container memory limit, whether a restart occurred, and whether the graph is per container, per pod or summed across replicas. A rising sum across replicas after a scale out says something quite different from one process growing for eight hours. On a Linux system using cgroup v2, these files give a direct view from inside the relevant container:

```bash
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.events
cat /sys/fs/cgroup/memory.stat
```

`memory.current` is the current charge to the cgroup and its descendants. `memory.max` is its configured maximum, or `max` when no maximum is set there. In `memory.events`, changes to `high`, `max`, `oom` and `oom_kill` help establish whether growth is turning into reclaim pressure or termination. Compare these values at several points; one snapshot cannot establish a trend. The path above assumes the container exposes its own cgroup v2 root at `/sys/fs/cgroup`. Check the environment if the files are elsewhere or if it uses cgroup v1. `memory.stat` helps explain the charge. The `anon` field reflects anonymous memory, `file` includes file backed cache and also tmpfs or shared-memory usage, so a rising `file` value is not automatically harmless cache. `slab` accounts for kernel data structures. These are categories of cgroup usage, not a way to add up managed objects. Do not assume the cgroup contains only the .NET process: sidecars are normally separate containers, but helper processes launched inside this container can contribute to its charge.

## Compare the process with the GC

Next, observe the running .NET process. On .NET 10, `dotnet-counters` can show the process working set alongside the managed heap and GC committed memory. Attach from an environment where the diagnostic tool can see the target process and its diagnostic socket:

```bash
dotnet-counters ps

dotnet-counters monitor -p 1 --counters 'System.Runtime[dotnet.process.memory.working_set,dotnet.gc.last_collection.heap.size,dotnet.gc.last_collection.memory.committed_size]'
```

Replace `1` with the process ID reported in your container. On Linux, the tool and target also need compatible access to the diagnostic socket and temporary directory; running a command on your laptop will not attach to an unrelated process inside a pod. The command shows a live view. For a proper trend, collect counters to a file or use the same runtime metrics in your telemetry system.

These three values measure different things. `dotnet.process.memory.working_set` is physical memory mapped into the process. `dotnet.gc.last_collection.heap.size` describes the managed heap at the most recent GC and includes fragmentation. `dotnet.gc.last_collection.memory.committed_size` includes memory committed by the GC beyond the space occupied by existing objects. The last two update in relation to collections; they are not a continuous live count of all application memory.

Do not subtract the GC heap number from the container number and label the difference "native leak". Process memory and cgroup memory are accounted differently. Shared and file-backed mappings complicate the comparison, and a container may include more than one process. Use the gap as a clue to investigate, not as a precise measurement of unmanaged allocations. A steadily increasing *total allocated* counter means the application is doing work and creating managed objects. It does not by itself mean those objects remain alive. Look at the heap after collections, the rate of collections, fragmentation and the relationship to workload. A heap that rises during a traffic peak and levels off under a stable workload is a different pattern from one that keeps growing across repeated comparable periods.

## Follow the branch that the numbers support

If the managed heap continues to grow after collections during comparable workload, inspect retained objects. `dotnet-gcdump` can provide a managed heap view suitable for comparing object counts and sizes. A full `dotnet-dump` supports deeper inspection, including `dumpheap -stat` and `gcroot` to see why an instance remains reachable.

```bash
dotnet-dump collect -p 1 -o /diagnostics/app.dmp

dotnet-dump analyze /diagnostics/app.dmp
> dumpheap -stat
> gcroot <object-address>
```

The commands are a starting point, not something to run casually on a pod already close to its limit. Dump collection can add significant memory and disk pressure, and a dump may contain secrets or personal data from the process. Plan a writable location, adequate headroom and controlled access. If the process is too close to an out of memory kill, reproduce the pattern with more headroom before collecting a large dump. If the process working set grows while the managed heap remains roughly stable, look outside live managed objects. Native libraries, decompression, image processing, TLS, mapped files, thread stacks and unmanaged buffers are possible contributors. The GC's *committed* metric helps narrow the gap - a stable live heap with rising GC committed memory points somewhere different from stable GC committed memory with rising process residency. Inspect process mappings or native allocation traces as the next step; another managed heap dump is unlikely to explain a clearly non managed trend.

If the container charge grows but process working set does not, inspect `memory.stat` and other processes in the cgroup. File-backed memory, tmpfs and kernel accounting can explain a gap. File cache may be reclaimable under pressure, but `file` also includes tmpfs and shared-memory usage, and not every page counted in a working set is immediately reclaimable. Test the hypothesis with `anon`, `file`, `shmem`, reclaim and memory event trends. Do not dismiss an impending memory limit just because one category is labelled "cache". If memory rises in steps, compare those steps with workload events. A batch import, a particular file type or a scheduled report can be more revealing than an average request rate. Group the measurements by application version and instance. A single long lived instance growing while fresh instances start low is a different lead from every instance settling at the same higher level after a deployment.

## Check whether the container is actually under pressure

A high memory graph can be operationally important even before an OOM kill. In cgroup v2, increasing `memory.events` values for `high` or `max` indicate reclaim or limit pressure. Look at response latency and throughput alongside these counters. If `oom_kill` increments, correlate it with the pod's termination reason and restart time rather than assuming every exception near that time caused the restart. A managed out-of-memory exception and a container killed at its memory limit are also different observations. A .NET process can run out of room for a particular allocation; a cgroup limit can cause the kernel to kill a process. Either may occur without a neat, final application log entry. Preserve the container and runtime measurements from before the restart if you want to understand which path was taken.

Do not start by forcing a full GC whenever the graph rises. A forced collection changes the thing you are observing and can impose a pause on the application. It may temporarily lower memory without explaining whether the underlying pattern is healthy. Likewise, raising the container limit can provide diagnostic headroom or restore service, but it does not identify the cause.

## Keep the investigation reproducible

I would save a small timeline: container usage and limit, `anon`, `file` and `shmem`, memory events, process working set, GC heap after collections, GC committed memory, request load and deployment markers. The values need the same timestamps and instance identity. Five aligned samples are usually more useful than fifty unrelated screenshots. That timeline gives you a testable hypothesis. If the managed heap tracks the container, compare managed snapshots and find the retaining roots. If GC committed memory accounts for much of the difference, inspect GC behaviour and workload before declaring a leak. If process memory grows independently of GC, investigate native allocations and mappings. If cgroup memory rises independently of the process, inspect cache, tmpfs and other processes.

The aim is to arrive at a statement you can verify: "These request objects remain rooted after each batch", "The process's native memory grows with image conversion", or "File backed pages rise during imports while managed memory stays flat". Once you can say which component is growing, the fix becomes an engineering task rather than a guess based on a single graph.

[memory leak investigation guide](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/debug-memory-leak)

[runtime metrics reference](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/built-in-metrics-runtime)

[container diagnostics guidance](https://learn.microsoft.com/en-us/dotnet/core/diagnostics/diagnostics-in-containers)

[Linux cgroup v2 documentation](https://docs.kernel.org/admin-guide/cgroup-v2.html)

[Kubernetes resource metrics documentation](https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/)