Too many requests on pool, website is crashing
Hello everyone!
Once in a while we experience crash(?) on our websites. By websites i mean we have 3 different websites based on same project but all for different regions. They use same database and server. Project, unfortunately, is old and developed like asp.net mvc project but it looks more like an aspx project. Using stored procedures in Views. I know it sounds and looks bad but i'm just a software developer and we don't have enough manpower or other resources to re-develop or fix things like that.
Anyways, by crash, i meant that requests from users can't full fill and application pool swells. Which ends up more requests floating in and so on. We have no idea what causes this. We have a devops engineer which offers to switch to linux. Our only solution so far is stop the websites, let the requests die on their own and restart websites.
Open to all suggestions, except switching to linux obviously 😛
6 Replies
When a site stops responding to requests, you could say it is 'hanging' instead of crashing; its the more common word for that.
When I've seen this in production environments, its usually caused by a resource on the server that is being used up on each request. It can be database connections, outbound TCP ports, or in rarer cases it can be caused by excessive garbage generation (leading to many G2 collections in a row).
Usually resource contention plus timeouts that are too long leads to the server remaining hung.
You might want to check the following:
That your timeouts are not longer than usual. If you have your timeouts set to more than 5 minutes (for example), that can cause havok on system stability as soon as other resources run low or run out.
That you are always properly disposing of database connections. Failure to do so can cause your application code to wait forever on opening a database connection that will never be returned to the pool by your other application code.
If you are accessing other external REST APIs from your server, make sure you are not creating an HttpClient for every request. This will lead to your app running out of available outbound ports to make requests with.
Your first step is to figure out what part of code is hanging. Do you have production logging that can hint to you where your code is hanging?
Do all of your sites hang at the same time, or is it usually one at a time?
Thank you for your advice! I'll check the code about things you said.
We use elmah but it's not detailed and doesn't give me anything, not even an URL.
Websites behaviours are similar most of the time. Our 3rd region almost have no active users so it's slightly better. Most populated region is, main page either takes too long or our host gives an error that you can't reach the website your internet is fine but this website is down bla bla..
Elmah failed message;
Websites behaviours are similar most of the time.Does that mean the sites hang at the same time? Or that they both hang, but at different times?
They(3) all hang at the same time, but their response time is not the same. For example;
All go down at 18:00
First Website(most populated one): Responds to my constant page refreshes after 10 minutes
Second Website: After 7 minutes
Third Website(next to no users): Responds after 6 minutes
And they are gone once again and so on, we call our devops staff, they stop all websites from running. We wait for all requests on the pool to die, then restart the websites. And voila, requests are coming and going instantly, response times are normal etc etc.
So i found a lot of un-disposed database connections today and most of them are frequently used. I'll try to optimize the code more with unnecesarry callings and usings.
If they all go down at the same time, then its likely an issue with a resource they all share. If its only the database that they share, then it sounds like a good place to investigate.
You might want to have someone look at the transactions open while the sites are locking up
Its possible that you are running into some type of transactional deadlock or running out of DB resources. Its really hard to make suggestions as an outsider without any understanding of your system, but I hope you figure it out 🙂
Our dev team also manages db stuff(not maintenance tho, transactions and tables). I tried running some transaction and db response time was fine.
Thanks for all of your advices mate! Even word 'hanging' and check for db connections helped me a lot. Have a great day!