dotnet / aspnetcore Public
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
.NET 6 Memory Leak/Issue #38722
Comments
Out of curiosity, is the memory released if you force a GC? I believe dotMemory has an option to do that. |
@pranavkm The app is running in release/production mode and dotMemory is not available on the server. Although it is not advised, would running System.GC.Collect() (https://docs.microsoft.com/en-us/dotnet/api/system.gc.collect?view=net-6.0#System_GC_Collect) have the same effect you want? |
Is the leak coming from those |
@davidfowl I believe so... The main byte[] that is taking up 5gigs is pinned via "OverlappedData" from FileSystemWatcher. See screenshot. I'm kind of new to dotMemory, so if tell me how to navigate to what you're looking for I can do so (I can also zoom/team/share screen - whatever is easiest for your team). |
That memory is pinned so it's not gonna go away with a GC. We need to figure out who is creating those. The best way to go about this would be to capture an allocation trace in production. It'll show you the stack that is causing these allocations. https://docs.microsoft.com/en-us/dotnet/core/diagnostics/debug-memory-leak (though this is just the dump). Where is this file system watcher rooted? |
@davidfowl I read your first link. I've put PerfView on the machine. If I'm understanding this correctly I should be able to just run
On the production server to capture the allocation trace? (I ran a quick sample and it generates a PerfViewGCCollectOnly.etl.zip file.) I don't need to target the process ID (w3wp.exe) or anything? In regards to the dump, is the dotnet-dump preferred over the normal windows memory dump? |
No it collects the entire machine. If you use dotnet trace it'll require the pid
Yes, you're trying to figure out where the allocations are coming from.
Either is fine. |
Unfortunately, the 12 hour PerfView collection I had didn't go well. When I opened it up in PerfView, it was missing the "“GC Heap Alloc Ignore Free (Coarse Sampling)” section. I don't know if it was because of the length of recording or file size. When I sample for say 30 minutes, it's available. I did notice that after 7 hours, the size stayed at "6,300MB" exactly which was strange, but I let it keep running. The process grew from 2gigs to about 11.5gigs over that 12 hours span during the night (theoretically our lower traffic time). I've started a trace again this time using PerfView64 just in case and the /FocusProcess= argument when collecting to focus on the process. I'll report back in a few hours. I didn't want to do an app pool reset yet, so I've started the recording while the current process is sitting at 12gigs |
Started the perfview monitoring at ~9:30AM EST when the process w3wp.exe was already at 12G (it had grown from 2G to 12G overnight). I recorded for 2.5 hours with the following command.
Breakdown dotnet-dump (high level, I have more screenshots if you need) Thoughts
|
Describe the bug
We're noticing a memory issue running a typical ASP.NET Core 6 website behind IIS on a Windows Server. After a few days the worker process (w3wp.exe) memory consumption grows from 2 gigs up to 25 gigs. Performing an IIS Stop / Start "fixes" the issue in the intermediate. I believe the issue is related to a FileWatcher that is not freeing up memory. We've had the issue since .NET 5 and noticed that someone else had a very similar issue (but not necessarily the same) found here #31125 and here #31219. It was supposedly fixed in .NET 6 which is why we upgraded, but we are still having the issue.
Here are some screenshots of dotMemory on the memory data dump when the production server got to 26gigs.
Top level snapshot.
High level inspection page.
Drill down to the Byte[] array section (Similar Retention Section).
Drill down to the Byte[].
Drill down to the OverlappedData section (Instances).
Drill down to an individual OverlappedData.
To Reproduce
I have yet to been able to reproduce this on any of our team's development boxes. This only occurs on production which is making it hard to pinpoint.
Exceptions (if any)
None
Further technical details
dotnet --info
:dotnet --info Output
Including screenshots of the IIS config so we are on the same page / just in case there is something misconfigured that we are unaware of.
Discussion / Side Notes
CreateDefaultBuilder
(https://github.com/dotnet/aspnetcore/blob/main/src/DefaultBuilder/src/WebHost.cs). Similar to issues #31125 and #31219 where they cleared the sources and everything was fine.Code Snippets
Web.config
Program.cs
Startup.cs (only including this so you can see how to utilize a background hosted service)
MySingletonThatAddsToBackground.cs
ScopedWorker.cs
Questions
The text was updated successfully, but these errors were encountered: