Pause Kinect Camera - Possible error in SDK reguarding event handler

I'm in the process of converting my Microsoft SDK Beta code to the Microsoft SDK Official Release that was released February 2012.

I added a generic PauseKinect() to pause the Kinect. My pause will really only remove the event handler that updated the image

Pros:

No Reinitialization (30+ second wait time)

Cons:

Kinect still processing images

Pause Method (Color Only)

internal void PauseColorImage(bool isPaused)
{
    if (isPaused)
    {
        _Kinect.ColorFrameReady -= ColorFrameReadyEventHandler;
        //_Kinect.ColorStream.Disable();
    }

    else
    {
        _Kinect.ColorFrameReady += ColorFrameReadyEventHandler;
        //_Kinect.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
    }
}

PROBLEM:

Even though I'm removing the event why is it still getting triggered?

NOTE:

Also when I pause the color image I'm also pausing the depth and skeleton in their object.

SIDE NOTE:

If I uncomment my code it works fine, but then it'll take forever to reinitialize which is not what I want to do.

MS in Reflector

public void AddHandler(EventHandler<T> originalHandler)
{
    if (originalHandler != null)
    {
        this._actualHandlers.Add(new ContextHandlerPair<T, T>(originalHandler, SynchronizationContext.Current));
    }
}

public void RemoveHandler(EventHandler<T> originalHandler)
{
    SynchronizationContext current = SynchronizationContext.Current;
    ContextHandlerPair<T, T> item = null;
    foreach (ContextHandlerPair<T, T> pair2 in this._actualHandlers)
    {
        EventHandler<T> handler = pair2.Handler;
        SynchronizationContext context = pair2.Context;
        if ((current == context) && (handler == originalHandler))
        {
            item = pair2;
            break;
        }
    }
    if (item != null)
    {
        this._actualHandlers.Remove(item);
    }
}


public void Invoke(object sender, T e)
{
    if (this.HasHandlers)
    {
        ContextHandlerPair<T, T>[] array = new ContextHandlerPair<T, T>[this._actualHandlers.Count];
        this._actualHandlers.CopyTo(array);
        foreach (ContextHandlerPair<T, T> pair in array)
        {
            EventHandler<T> handler = pair.Handler;
            SynchronizationContext context = pair.Context;
            if (context == null)
            {
                handler(sender, e);
            }
            else if (this._method == ContextSynchronizationMethod<T>.Post)
            {
                context.Post(new SendOrPostCallback(this.SendOrPostDelegate), new ContextEventHandlerArgsWrapper<T, T>(handler, sender, e));
            }
            else if (this._method == ContextSynchronizationMethod<T>.Send)
            {
               context.Send(new SendOrPostCallback(this.SendOrPostDelegate), new ContextEventHandlerArgsWrapper<T, T>(handler, sender, e));
            }
        }
    }
}