NullReferenceException in PipelineModuleStepContainer

One shiny day a complex but pretty reliable application failed with the following exception report:

So I had to spend some days and nights searching for the reason.

Internet is not really good - it mostly reports issues related to registering in Application_Start. That was not the case, because application had been running for some time in production.

(Practially I should have made my mind when I saw those pages - because application in "global.asax.cs" is only a special kind of module).


Ok, so I made some research. It included debugging during the failure, dump file analysis from up to down and reverse-engineering of HttpApplication. I'll skip all those details and proceed to results.

1. Due to integrated mode, IIS manages pipeline. Runtime just gets notifications from IIS. They contain information which module/handler pair should be used for event handling.
2. During first-time app initialization, Runtime registers event handlers in IIS, specifically calling webengine4.MgdRegisterEventSubscription.
3. Registration is in form [moduleName, eventType, moduleIndex], where moduleIndex is the index of PipelineModuleStepContainer instance in HttpApplication._moduleContainers array.
4. Runtime maintains pool of HttpApplication instances (HttpApplicationFactory._freeList). In case if all app instances are busy processing requests, it creates new instance. It is called “normal” instance.
5. During normal instance initialization, event handlers populate HttpApplication._moduleContainers, which will be used for events handling.
6. When handling request, HttpApplication relies on data integrity and doesn’t check absence of handler. It basically makes this._moduleContainers[moduleIndex]._moduleSteps[eventIndex], which triggers NullReferenceException.

At the diagram above IIS contains registration for BeginRequest and AuthenticateRequest handler with module index = 6. At the same time HttpApplication doesn't contain such handlers. So we have a bang.


This gives us common rule of thumb for such class of errors:

NullReference exception occurs, if a module subscribed for event handling during first-time initialization; and for some reasons did not it during initialization of another HttpApplication instance. 

In my particular research NewRelic's module caused issues.