❔ OOM well below GCHeapLimit (Could not do a full GC) since .NET7
The GCHeapLimitPercent is set to 75%, which equates to about 3gb in the container - working well in netcore3.1 for a long time.
We've updated to .NET7 and getting some OOM, but our collected heap dump is only showing around 300mb allocated on the GC heaps (looking in dotnet dump analyze dumpheap -stat / eeheaps / analyzeoom).
We catch these dumps with a ASP middleware catching OOM exceptions which calls into Environment.FailFast to let createdump capture a dump (we're using COMPlus_DbgEnableMiniDump etc).
Has anything changed in GC since .NET7 that could affect this? I assume I'm going to have to start looking into unmanaged memory but I can't get the dumps open in dotMemory or VS memory debugger for some reason.
9 Replies
are you using this? https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#heap-limit-percent
Garbage collector config settings - .NET
Learn about run-time settings for configuring how the garbage collector manages memory for .NET Core apps.
Sorry - COMPlus_GCHeapHardLimitPercent is set to 75.
It is documented as a hex value, but I remember from testing (which was more than a year ago) we found that it was actually expecting decimal.
(and I realize that 75 is actually the default value 🙂 )
that could have been fixed 🤔 not sure what it would interpret 75 as if it was now expecting hex... 117? but surely it would expect a leading 0x 🤔
Looks like nothing has happened here. I will mark this as stale and this post will be archived until there is new activity.
I've now removed COMPlus_GCHeapHardLimitPercent, OOM still happening. Dotmemory showing 200mb allocated.
Any advice on native memory debugging? Or is it possible (see above) that the heap data is lost?
Was this issue resolved? If so, run
/close - otherwise I will mark this as stale and this post will be archived until there is new activity.Anyone know how to distinguish an OutOfMemoryException raised by the GC from any other?
GitHub
Multiple 'System.OutOfMemoryException' errors in .NET 7 · Issue #78...
I'm seeing an issue very similar to this one when running a memory-heavy app on a linux container with memory limit >128GB RAM. The app started throwing random OutOfMemoryException i...
wow
thanks
but looks like we're throwing at 200mb
the heap limit is 3gb (kubernetes cgroups memory limit)
the node has 16gb
worth a try though
Was this issue resolved? If so, run
/close - otherwise I will mark this as stale and this post will be archived until there is new activity.