Hangfire out of memory exception crashing our app services. Anyone got any advice?
So, last night I noticed our website was slow as hell, and when I checked my logs our server seemed to be restarting over and over, like every 5 minutes or so. This was for both prod and preprod (hosted on azure, app services). Sure enough, CPU was spiking to 100%. I checked crash logs which notified me of multiple crashes, and then I checked the Crashing Thread Callstack
"90 crashes due to (0xE0434352 - CLR Exception)"
CLR Exception System.OutOfMemoryException
Anyone got any clue how I can even trouble shoot this?
3 Replies
No. Cause none of us has the insight to your code.
Taking a wild guess; you're trying to load too much data from the database to some
object, like a DataTable - things has limits, it's never a good idea to SELECT * FROM xy;, which can be millions of values.
And can take GBs of memory. Also, the limit of DataTable is 16,777,216 rows.So, hangfire serializes methods and persists them to the db, polls the db and then a worker deserializes a method argument and invokes the method. Either reading from the db or deserialization fails due to the size of the data, that much seems evident from the image above. But how do I even begin to troubleshoot it? Because it doesnt make sense to me, the db has no mroe jobs left
@SpReeD
I'm not much familiar with Hangfire, never used it in prod.
So I can't tell, judging by the callstack, what exactly fails - is it the client<->server hangfire job retrieval or is it the job itself; taking another wild guess, it's the latter.
Next question that comes into my mind, how does the job look like? What kind of data? How big is it? Do you use caching? like NoSQL redis? Is the serialized data too much for a single NoSQL document? Does hangfire support NoSQL document chunks? - In other terms, look at your jobs code.