In my previous post I documented a bug in SharePoint where a memory leak can greatly impact that amount of memory you SharePoint WFE will use during the normal processing of web request. The SharePoint PG just recently released the 2010 April CU for WSS 3.0 and MOSS 2007. It requires you have Service Pack 2 before installing these cumulative updates but I can tell you that is something you want to invest in.

WSS 3.0
Main KB: http://support.microsoft.com/kb/981043
Download Link: Download April 2010 Server-Package for WSS (981043)

MOSS 2007
Main KB: http://support.microsoft.com/kb/982741
Download Link: Download April 2010 Server-Package for MOSS 2007 (981042)

If you dig into the description of what has been fixed in the CUs you will come across this KB which has this little blurb regarding the Sasquatch.

A memory leak may cause the application pool to recycle the AppDomain because many instances of the SPHttpApplication reach memory limits.

Recently Mark Arend and I were helping out a co-worker setup Session State on SharePoint 2010 and he pointed out something very important; while I was providing the means to enable the “State Service” I was not providing the method to turn on ASP.Net session state. At first I was not clear on the distinction however once I read his email for the 3rd time after my second cup of coffee I finally got it..while I was using New-SPStateServiceDatabase to setup the State Service DB he was suggesting to use Enable-SPSessionStateService. Yea it looks pretty clear now that these are two very different commands however in my stupor after working on User Profile Sync issues over the past couple of days these were equal in my feeble mind.

So what’s the difference? The State Service is used by InfoPath Forms Services (IPFS), the Chart Web Part, and when the user is not running Silverlight Visio Services will use this too. The ASP.Net session state is used within aspx pages, controls, Web Parts, etc to store user/session scoped state such as when you make a call to Page.Session.Add(“Key”, new object()); As developers, we have access to the Asp.Net Session state only while the State Service is reserved for the Office Server applications mentioned previously.

How to Enable SharePoint’s State Service

Out of the box (OOTB) you may find you receive an error stating Session State is not enabled. This error is referring to the SharePoint State Service which needs to be enabled. If you use the Configuration Wizard you will see this choice present along with numerous other services however if you choose to not enable it through the Configuration Wizard you will need to use PowerShell. For some reason this service does not appear as on option on the “New” ribbon button within the Service Application. In my previous blog post I provided a PowerShell script which setup the State Service. While there are quite a few verbs to this commandlet you only really need the “New” to install/setup this type of session state.

[New|Remove|Get|Set|Mount|Dismount|Initialize|Suspend|Resume]-SPStateServiceDatabase

Here is a sample command which installs the State Service:

New-SPStateServiceDatabase -Name "StateServiceDB" -DatabaseServer "sp2010dev.contoso.com" | New-SPStateServiceApplication -Name "State Service" | New-SPStateServiceApplicationProxy -Name "State Service Proxy" -DefaultProxyGroupl

How to Enable ASP.Net Session State

The only way to install ASP.Net Session State is through PowerShell using *-SPSessionStateService. [Enable | Disable | Get | Set]-SPSessionStateService

Using this example I was able to install the ASP.Net Session State on my development farm. Note you must pass a database name which does not include any spaces. If you happen to do this you will receive an error and you may notice additional SQL Server Agent Jobs (which I will discuss later) which are created each time you try to pass off a DB name with a space (jeez you think I actually did this or what).

Enable-SPSessionStateService -DatabaseName "ASPNet_Session_State" -DatabaseServer "sp2010dev.contoso.com" -SessionTimeout 30

Once enabled each web.config file is modified by adding the System.Web.SessionState.SessionStateModule to the list of Http modules. In addition a new sessionState element is added similar to the following:

<sessionState mode="SQLServer" timeout="30" allowCustomSqlDatabase="true" sqlConnectionString="Data Source=sp2010dev.contoso.com;Initial Catalog=ASPNet_Session_State;Integrated Security=True;Enlist=False;Connect Timeout=15" />

Notice I said “each” web.config file is modified, that is, all root web.config files for all Web Applications are modified, for example see the following image. When new Web Applications are added later they too will receive this web.config modification when provisioned.

Now this does not mean ASP.Net Session state is available on any web application out the gate. An administrator would have to enable session state at each web application by locating the “pages” element such as this: <pages enableSessionState="false" and changing the enableSessionState to true. After making this change Session state is now available to be used.

More Information
When using Visual Studio 2010 to create SharePoint projects you can create aspx pages using designers such as the Application Page designer. These designers create pages with a <%@ Page tag which does not supply an EnableSessionState attribute, the default for this attribute is “true” so for almost all cases you have just enabled session state for all your pages by making the change above. SharePoint 2010 uses the OOTB ASP.Net Session State provider and Session State HttpModule. This means State is binary de-serialized once at the beginning of a page request and serialized back once the request is complete. This happens on any page where 1)session state has been enabled on the farm, 2) enableSessionState is true on the web application, 3) there is a valid session state cookie being sent from the client, 4) the Page does not include an EnableSessionState attribute or the attribute is set to true.

If you decide to use session state it is prudent to ensure a) its enabled (Page.Session != null) and b) you explicitly set enableSessionState to “false” where you don’t need session state or “ReadOnly” where you only need a read only copy of the session state were you do not plan to write back to (note you can do Session.Add(..) or Session[key]=”” within a page with ReadOnly set, the changes just will not be persisted). When “true” or not present two round trips are made to the Session State DB for each page request. Setting this value to “false” eliminates these round trips while setting it to “ReadOnly” means we only fetch the session state from the DB on page request and throw it away when the request completes meaning you can cut the number of round trips in half.

Scale is important here as there is only one Session State DB per farm to be used for all Web Applications, so investigate other means of storing state before enabling this feature. Session State is track via a cookie sent back to the browser when session state is stored for the client, this occurs for each web application which uses session state. So if you have my.contoso.com and intranet.contoso.com which both happen to use session state, both are on the same farm, they will use different session state cookies and different session state objects stored in the same session DB. Since session is stored and shared within a Web Application operations performed on the session state such as Session.Clear(), Session.Abandon(), etc can potentially impact other applications, pages, or sites which use session state. Keep this in mind if you have developers complaining about losing session state – its probably not SQL or ASP.net’s fault, its probably someone trying to be a good citizen and clearing out the session state.

Session Expiry is handled by a SQL Server Agent Job which is installed when you enable the service. This job runs once a minute cleaning up any expired sessions. It’s important your SQL Server Agent service is running. This was not always the case for MOSS 2007 and an important change here; otherwise your ASP.Net session state will not be cleaned up meaning your database will grow, and grow, and…you get it.

Background

I have been working with SharePoint for a few years now and have run into many nasty high memory or Out of Memory (OOM) issues. To date many of SharePoint’s memory problems have been discussed as problems with developers not using the Dispose pattern properly when using the SharePoint OM. And while I have found this to be the case I always had a feeling that there was something more, another larger leak which could not be explained simply by not calling Dispose. Sure not calling dispose will keep the SPRequest COM object (which is fairly large) around longer than necessary however ref-counting will clean these up eventually but even with using the correct coding patterns and practices the w3wp processes still seem larger than necessary; even on my development machine when I run stress against an out of the box installation. So there must be something else going on here; and so started my search for Big Foot.

Big Foot Makes an Appearance

Recently I was contacted by a friend who works on the IIS team to help troubleshoot a memory issue with SharePoint. This is a pretty common request but what really piqued my interest was when he told me he had a 9 GB memory dump. For those not familiar with memory dumps the size represents the size of the memory being used by the process so for this w3wp process it was using 9 GB of memory and to make matters worse the process had only been up about 56 min. This seemed like the best opportunity to find a nasty and elusive memory leak!

Cracking open the memory dump I found something very interesting, the process had upward of 24,600+ ASP.global_asax objects. So why is this important? The global_asax represents the file based HttpApplication which is used during request processing. You can think of these guys as the tour guide to the user’s request which guides the request through ASP.net’s request processing pipeline. The HttpApplicationFactory keeps a free list of HttpApplication instances and as new requests reach the system one is pulledf from the free list and assigned to the request to guide the request through the pipeline or in the case when none are available will instantiate a new HttpApplication. Once complete the HttpApplication is either returned to the free list or is discarded depending on the number of HttpApplication(s) already in the free list. In addition the HttpApplicationFactory will also attempt to trim unused or unneeded HttpApplications from the free list every 30 seconds. For each pass which a free HttpApplication exists within the free list a single HttpApplication will be disposed.

So now that we understand HttpApplication lifetime why is it that this process memory dump  have so many? Looking at the HttpApplicationFactory I found that only about 82 HttpApplications were in the free list and combined with the number which were processing requests at the time of the memory dump we can only account for less than 90. This means the other 24,000+ were either being leaked or have yet to be collected by the GC. Since I did not believe that the lack of GC to be the problem I started to look for what was keeping these objects rooted such that the GC could not collect them. I soon came across this object:

0:000> !do 00000002435a8290 

Name: System.UnhandledExceptionEventHandler 

MethodTable: 0000064278467860 

EEClass: 0000064278005670 

Size: 64(0x40) bytes 

GC Generation: 2 

(C:\WINNT\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll) 

Fields: 

              MT            Field           Offset                 Type VT             Attr            Value Name 

0000064278435ed8  40000ff        8        System.Object  0 instance 00000002435a8290 _target 

0000064278434990  4000100       10 ...ection.MethodBase  0 instance 0000000000000000 _methodBase 

0000064278438ff0  4000101       18        System.IntPtr  1 instance 6882685329544 _methodPtr 

0000064278438ff0  4000102       20        System.IntPtr  1 instance 6882550903584 _methodPtrAux 

0000064278435ed8  400010c       28        System.Object  0 instance 0000000193373b88 _invocationList 

0000064278438ff0  400010d       30        System.IntPtr  1 instance 24686 _invocationCount

This object is the event handler for an AppDomain’s UnhandledExceptionEventHandler. Note the _invocationCount member, its value is 24,686. What this means is that over 24,000 objects have registered for notification when the AppDomain for which this object exists exits abnormally. Digging further I found the SPHttpApplication as the type which the _target was referencing. This object derives from HttpApplication and is itself derived from ASP.Net’s generated global_asax class. In SPHttpApplication’s Init() method it hooks the AppDomain’s UnhandledExceptionEventHandler. While the callback for this event logs what may be important information to the ULS log when an unexpected process exit it does not need to do this 24,000+ times.

As most readers will already know static objects live for the life of the AppDomain so an AppDomain’s lifetime is equal to that of static objects within the .Net Framework. As such when an object registers for event notification from an object which is either static (or in this case the AppDomain) the registering object roots itself for the lifetime of the AppDomain. The GC cannot collect this object until either the AppDomain restarts or the object unregisters itself from the event handler. Unfortunately SPHttpApplication does not unregister itself from the AppDomain’s UnhandledExceptionEventHandler handler so each instance will live as long as the AppDomain. So even after the HttpApplicationFactory releases the SPHttpApplication the object will continue to remain in memory until the AppDomain restarts.

So we have a leak, however it gets worse..the ASP.Net runtime creates a new HttpContext for each request which is processed. Its this object which holds references to the HttpRequest, HttpResponse, and most importantly the handler which will service the request. For the cases where this handler is a page the entire control hierarchy, member variables, etc is referenced. Since the HttpApplication indirectly has a reference to the HttpContext the amount of memory being referenced can be huge. Take a look at these objects which I found in the memory dump, note that for each HttpApplication which is created an HttpContext, HttpResponse, etc are too created.

So what’s the bottom line? – For each HttpApplication which is leaked a potential huge number of child references are too rooted and therefore leaked. While the table above would seem to indicate this is a small leak the numbers above do not reflect the size of all child objects which have been rooted.

So What is The Fix?

As of today we have just begun the process of having this issue addressed by the SharePoint 2007 servicing team so we do not have a fix at the time this blog was published. Once the fix is available I strongly advise everyone to install it and if implemented remove the workaround I provide below.

Workaround Please

For those that do not want to wait for a fix I have a workaround however it will require a bit of coding on your part but its really not that difficult. What we want to do here is ensure we only let the SPHttpApplication register only once for the AppDomain’s UnhandledExceptionEventHandler. Since the SPHttpApplication does this from within its Init() virtual method what we can do is ensure this gets called only once so as to only root one HttpApplication instance.

1. Within VS2008 create a new class library and add a reference to the Microsoft.SharePoint library. Ensure you also strong name the assembly because we want to place this assembly in the GAC.
2. Add a new class and paste the code below into this file.

   using System;
   using Microsoft.SharePoint.ApplicationRuntime;
   
   namespace ToddBlog.Sample
   {
       public class CustomHttpApplication : SPHttpApplication
       {
           private static bool s_initialized = false;
   
           public CustomHttpApplication()
           {
           }
   
           public override void Init()
           {
               if (!s_initialized)
               {
                   lock (typeof(CustomHttpApplication))
                   {
                       if (!s_initialized)
                       {
                           base.Init();
                           s_initialized = true;
                       }
                   }
               }
           }
       }
   }

3. Compile the project and deploy the resulting assembly into the GAC on every SharePoint WFE server within your farm.
4. For each web application edit the global.asax file and change the assembly reference to point to your custom assembly created in step #1 and change the Inherits attribute to reference the class you created in Step #2.

<%@ Assembly Name="[name of your assembly"]%>
<% Application Language="C#" Inherits="ToddBlog.Sample.CustomHttpApplication" %>

Results

After having the customer install the fix I have described here the process which would normally grow to 10 GB an hour and recycle never went above 2.5 GB! And even after taking all servers out of the load balancer and running the entire farm off one WFE the memory never went above 3.3 GB!!

Who’s Impacted

This issue is applicable to MOSS 2007 and WSS 3.0. SharePoint 2010 is not effected by this issue. The impact to your application depends on many factors such as concurrency, traffic and the amount of uptime. For most decent size installations however this could greatly reduce the amount of memory needed by the IIS Application pool process.

Feedback

At the time of this post I have only worked with one customer for which this fix was huge. I would love to hear from others which implement this fix to hear how much impact it had in other environments, so please leave a comment with your results.