Performance of web applications hosted in IIS can be affected with by the worker process reaching or utilizing the CPU resources to maximum and shows 100% which caused the subsequent requests to pile up due to lack of resources. I planned to write this article as I too had similar problems on one of the projects on the production machine which hindered the users and we got to hear a lot of complaints of the application being down and slow.

The IIS worker process  utilizing a lot of CPU resources can be because of various reason, we will investigate some of the top possibilities and how to troubleshoot the same.

Some of the common reasons which might cause the IIS worker process(w3pwp.exe) to utilize the maximum CPU resources w3pwp.exe are :

1. Requests getting blocked or stuck in the asp.net pipeline
2. High Error rates in the web application
3. High Web Traffic
4. Garbage collection
5. Inefficient .net code
6. Outdated or Issues with the application dependencies

I will cover only certain important points in above scenarios and dig deeper on point 1. To identify and analyse it

Requests getting blocked or stuck in the asp.net pipeline

First thing to do here is to look at the requests which are currently running and investigate whether they are stuck or hanged. This might give a brief idea on which requests are basically taking long time or getting stuck. There might be requests which are queued and not directly part of the problem or cause for alarm.

Identification

There are many steps in lifecycle of ASP.NET request. The basic steps include authentication, authorisation, evaluation of the http request, finishing the request. If facing any performance problems , the first place to start is identifying the specific Http module.

Open IIS management console, you can view the running worker process and select the IIS application pool in question which is causing high CPU and view current running requests

Select the worker processes from the IIS, you can see the currently running IIS worker processes.

Double click on the application pool name, it will take you to view all the currently executing requests. You can see that each request is in different parts of the ASP.NET pipeline and currently executing different HTTP modules.

The above snapshot gives you the below information for the currently executing requests.

• URL : the complete URL that is being executed
• Time: the total amount of time in milliseconds, the web request has been executing.
• Client : The IP address of the user that initiated the request
• State : the stage of the IIS pipeline that the request is currently running in.
• Module: the ASP.Net module that is currently executing.

Analysis

To dig deeper the reason the actual issue what can be done is to obtain a memory dump from the w3wp.exe process like below:

Then use tools like debugdiag by Microsoft or visual studio or  to analyse the memory dump and narrow down the threads which is causing the issue for the request to get stuck. Let’s see how we can understand the same.

Hit Start analysis which will generate a report like a sample below which contains the top threads by CPU time and the details if any errors are occurring in the threads which is causing them to get stuck.

To open the same file in visual studio as below and run analysis on the threads will give you much more information for you to identify and act upon.

Click on Run Diagnostic Analysis on the top right of the above image to get the details further identify the root cause.

Possible Solutions

The exception on threads will give a clear indication on which part of the code is causing the thread to break and request to get stuck. Some pointers which I have come across.

1. In the above case we had a scenario, concurrency dictionary modification pitfalls
2. Check and set the application app pool to reset on a timely manner to clear out.
3. Another possible scenario is to increase RequestQueueLimitPerSession.

High Error rates in the web application

            Another probably reason for the high CPU usage by the worker process is the high number of application errors which you can be unaware of, some of which can be masked by the generic message which users get or a common message. Identification of the errors using error monitoring or application performance management tool, we can ensure for these. Other places or more common place to look for these is the IIS logs generated which contains the details of the request, response from it, any issues with the request, network related failures etc. This should be the ideal place to start your investigation.

Also, you can look at the Performance Monitor in windows to analyse it with the counters as below:

• NET CLR Exceptions -> # of Exceptions Thrown / sec: Check this to see if a lot of exceptions are being thrown by your application. It is possible for your application to have a lot of hidden errors that can cause big performance problems.  Any exceptions at all are bad, but some are unavoidable.
• W3SVC_W3WP -> % 500 HTTP Response Sent:  Any requests with a 500-status code are internal server errors. Make sure this percentage is very low. It should be 0-1%.

High Web Traffic

Sometimes one of the simple explanations for the high CPU usage by worker process could be the increase in web traffic.

You can use the IIS log files to see the number of requests pooled for the particular day, You can use tools like IIS log browser to analyse the requests which gives out major details from timestamp, URL, status, failure reason, client IP etc. and also gives you reports to work out of to identify most hit URL ,most failed URL etc.

Garbage Collection

Microsoft .NET utilizes garbage collection to manage the allocation and release of memory.

Depending on what your application does, the allocation and clean-up of application memory can cause a lot of garbage collection activity. For example, the usage of lots of large string variables on the large object heap causes garbage collection problems.

To measure if garbage collection could be causing problems for your application, check this Windows Performance Counter:

.NET CLR Memory -> % Time in GC: This counter reflects what % of the time your application spends doing garbage collection. If this number spikes a lot above 5-10%, you want to investigate memory usage further.

Garbage collection also has two modes. You may need to enable server mode, which is not the default.

Inefficient .NET Code

Inefficient .net code needs to be looked at if no other things are panning out. You can use profilers like .NET Profiler or ANTS performance profiler to capture functions being called and analyse them further. But profilers do and an overhead on the CPU usage so if you are already on 85%+ it would be difficult to run a profiler which might cause it to go to 100% and make the application unusable.

Outdated or Issues with the application dependencies

Having third party dependencies leads to a version being used which is old and vulnerable, Identify the dependencies on the application and update them regularly to be up to date from security, loopholes, memory leaks, or any other issues which might occur due to these dependencies which doesn’t logged clearly.

Conclusion

Hopefully after reading this article, you have a starting point to analyse your performance problem and improving it and not at critical stage where your application is going for a toss because of the 100% CPU usage. I hope this brief article has given you a perspective guided you to take the necessary steps to tackle the problem.

About the Author

Darshan Raviprakash is a senior full stack developer currently @Version1 with around 9 years of experience in leading design, development, deployment of enterprise level application concentrated on technologies such as .NET, Azure, Devops, Angular, SQL. Along with Microsoft technologies he has a wide area of knowledge python, JD Edwards, Data Science. He enjoys working on exploring new technologies and bring in constant improvement to projects on various aspects. He has worked on performance optimisations on applications, process oriented agile approach to projects.

LinkedIn: https://www.linkedin.com/in/darshan-raviprakash-6b254147