❔ F#: General Memory Issue
I am kind of unsure how to describe this but the memory managing of my F# program is abysmal and I don't even know where to start as many things learned in C# don't exactly carry over.
Instead of posting 300 code snippets I will just provide you with the GitHub repo.
https://github.com/Jona4Play/F-Crawler
If you can muster up the time I'd appreciate any advice
9 Replies
I believe you've made the repository private
so we can't see it
What do you mean when you say the memory management is poor? I don't see any issues.
I think they mean the 15 gigabytes of allocated memory shown in the diagnostics panel
Memory is not released when I thought it will be. Obviously I'm missing something.
https://github.com/Jona4Play/F-Crawler Now it's public
GitHub
GitHub - Jona4Play/F-Crawler
Contribute to Jona4Play/F-Crawler development by creating an account on GitHub.
I'm using the Parallel.map module to distribute the tasks and thought that the HTMLDocument will get collected as soon as it is has executed all methods that depend on it and the new Data Structure holding the information (it doesnt hold the HTML Document) has been created
More importantly- I don't get why there are so many HTML Parsing instances. Is this just a quirk of the library? If so I will switch to HTML Agility pack or something else, or am I using it contextually wrong? (Esp as only 70ish of the result data structure has been achieved)
Oh, hahaha! Didn't notice at all 😅.
In response to the 15GB 😋.
I'm not sure about the library, but it doesn't look like there are many parsing instances, but just instances of HtmlToken declared in the parser Parsing module. So the memory consumption looks to be coming from after parsing, the structured HTML data is heavy on memory.
Which site are you parsing? If it's public.
Hmm. The HtmlDocument should lose all references right after you're done extracting the data from it. Unless it landed in Gen2 or LOH. Can you see which GC generation the data lives in?
Yeah I could try to but I'm not too familiar with the Visual Studio Debug Tooling. In the meantime I wrote a similar C# solution and it doesnt have this memory issue. I will try to change the lib used to HtmlAgilityPack and try again
Was this issue resolved? If so, run
/close - otherwise I will mark this as stale and this post will be archived until there is new activity.