How to Investigate a Growing Container Without Guessing
A .NET container uses 600 MB after startup, 850 MB by lunchtime and 1.2 GB by the end of the day. Someone calls it a memory leak and asks for a heap dump. The graph has established that memory use is growing but It hasnt established what is growing. The application could be retaining managed objects, the GC could have committed space it expects to use again, a native library could be allocating outside the managed heap, or file backed pages could be charged to the container. A heap dump is useful for some of those cases and almost irrelevant to others.
I start by lining up three views of memory over the same time window - the container's charge, the process's memory and the .NET runtime's heap. The differences between them usually tell me what to inspect next.
Confirm what the graph measures
"Container memory" is not one universal number. In Kubernetes, kubectl top uses the resource metrics pipeline and reports a working set view. The Kubernetes documentation notes that this can include some cached file backed memory. A dashboard showing cgroup usage, container working set or process resident memory can therefore produce different numbers for the same workload. Check the metric name before interpreting its slope. Record the container memory limit, whether a restart occurred, and whether the graph is per container, per pod or summed across replicas. A rising sum across replicas after a scale out says something quite different from one process growing for eight hours. On a Linux system using cgroup v2, these files give a direct view from inside the relevant container:
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.events
cat /sys/fs/cgroup/memory.stat
memory.current is the current charge to the cgroup and its descendants. memory.max is its configured maximum, or max when no maximum is set there. In memory.events, changes to high, max, oom and oom_kill help establish whether growth is turning into reclaim pressure or termination. Compare these values at several points; one snapshot cannot establish a trend. The path above assumes the container exposes its own cgroup v2 root at /sys/fs/cgroup. Check the environment if the files are elsewhere or if it uses cgroup v1. memory.stat helps explain the charge. The anon field reflects anonymous memory, file includes file backed cache and also tmpfs or shared-memory usage, so a rising file value is not automatically harmless cache. slab accounts for kernel data structures. These are categories of cgroup usage, not a way to add up managed objects. Do not assume the cgroup contains only the .NET process: sidecars are normally separate containers, but helper processes launched inside this container can contribute to its charge.
Compare the process with the GC
Next, observe the running .NET process. On .NET 10, dotnet-counters can show the process working set alongside the managed heap and GC committed memory. Attach from an environment where the diagnostic tool can see the target process and its diagnostic socket:
dotnet-counters ps
dotnet-counters monitor -p 1 --counters 'System.Runtime[dotnet.process.memory.working_set,dotnet.gc.last_collection.heap.size,dotnet.gc.last_collection.memory.committed_size]'
Replace 1 with the process ID reported in your container. On Linux, the tool and target also need compatible access to the diagnostic socket and temporary directory; running a command on your laptop will not attach to an unrelated process inside a pod. The command shows a live view. For a proper trend, collect counters to a file or use the same runtime metrics in your telemetry system.
These three values measure different things. dotnet.process.memory.working_set is physical memory mapped into the process. dotnet.gc.last_collection.heap.size describes the managed heap at the most recent GC and includes fragmentation. dotnet.gc.last_collection.memory.committed_size includes memory committed by the GC beyond the space occupied by existing objects. The last two update in relation to collections; they are not a continuous live count of all application memory.
Do not subtract the GC heap number from the container number and label the difference "native leak". Process memory and cgroup memory are accounted differently. Shared and file-backed mappings complicate the comparison, and a container may include more than one process. Use the gap as a clue to investigate, not as a precise measurement of unmanaged allocations. A steadily increasing total allocated counter means the application is doing work and creating managed objects. It does not by itself mean those objects remain alive. Look at the heap after collections, the rate of collections, fragmentation and the relationship to workload. A heap that rises during a traffic peak and levels off under a stable workload is a different pattern from one that keeps growing across repeated comparable periods.
Follow the branch that the numbers support
If the managed heap continues to grow after collections during comparable workload, inspect retained objects. dotnet-gcdump can provide a managed heap view suitable for comparing object counts and sizes. A full dotnet-dump supports deeper inspection, including dumpheap -stat and gcroot to see why an instance remains reachable.
dotnet-dump collect -p 1 -o /diagnostics/app.dmp
dotnet-dump analyze /diagnostics/app.dmp
> dumpheap -stat
> gcroot <object-address>
The commands are a starting point, not something to run casually on a pod already close to its limit. Dump collection can add significant memory and disk pressure, and a dump may contain secrets or personal data from the process. Plan a writable location, adequate headroom and controlled access. If the process is too close to an out of memory kill, reproduce the pattern with more headroom before collecting a large dump. If the process working set grows while the managed heap remains roughly stable, look outside live managed objects. Native libraries, decompression, image processing, TLS, mapped files, thread stacks and unmanaged buffers are possible contributors. The GC's committed metric helps narrow the gap - a stable live heap with rising GC committed memory points somewhere different from stable GC committed memory with rising process residency. Inspect process mappings or native allocation traces as the next step; another managed heap dump is unlikely to explain a clearly non managed trend.
If the container charge grows but process working set does not, inspect memory.stat and other processes in the cgroup. File-backed memory, tmpfs and kernel accounting can explain a gap. File cache may be reclaimable under pressure, but file also includes tmpfs and shared-memory usage, and not every page counted in a working set is immediately reclaimable. Test the hypothesis with anon, file, shmem, reclaim and memory event trends. Do not dismiss an impending memory limit just because one category is labelled "cache". If memory rises in steps, compare those steps with workload events. A batch import, a particular file type or a scheduled report can be more revealing than an average request rate. Group the measurements by application version and instance. A single long lived instance growing while fresh instances start low is a different lead from every instance settling at the same higher level after a deployment.
Check whether the container is actually under pressure
A high memory graph can be operationally important even before an OOM kill. In cgroup v2, increasing memory.events values for high or max indicate reclaim or limit pressure. Look at response latency and throughput alongside these counters. If oom_kill increments, correlate it with the pod's termination reason and restart time rather than assuming every exception near that time caused the restart. A managed out-of-memory exception and a container killed at its memory limit are also different observations. A .NET process can run out of room for a particular allocation; a cgroup limit can cause the kernel to kill a process. Either may occur without a neat, final application log entry. Preserve the container and runtime measurements from before the restart if you want to understand which path was taken.
Do not start by forcing a full GC whenever the graph rises. A forced collection changes the thing you are observing and can impose a pause on the application. It may temporarily lower memory without explaining whether the underlying pattern is healthy. Likewise, raising the container limit can provide diagnostic headroom or restore service, but it does not identify the cause.
Keep the investigation reproducible
I would save a small timeline: container usage and limit, anon, file and shmem, memory events, process working set, GC heap after collections, GC committed memory, request load and deployment markers. The values need the same timestamps and instance identity. Five aligned samples are usually more useful than fifty unrelated screenshots. That timeline gives you a testable hypothesis. If the managed heap tracks the container, compare managed snapshots and find the retaining roots. If GC committed memory accounts for much of the difference, inspect GC behaviour and workload before declaring a leak. If process memory grows independently of GC, investigate native allocations and mappings. If cgroup memory rises independently of the process, inspect cache, tmpfs and other processes.
The aim is to arrive at a statement you can verify: "These request objects remain rooted after each batch", "The process's native memory grows with image conversion", or "File backed pages rise during imports while managed memory stays flat". Once you can say which component is growing, the fix becomes an engineering task rather than a guess based on a single graph.
memory leak investigation guide
