How to customize the selected text colors of an NSTextField / NSTextView in an inactive state

Question

I'm using an NSTextField and customizing the fieldEditor using the setupFieldEditorAttributes: method. This allows me to set custom foreground and background colors for the selected text, which is important because my textField has a black background and white text. Generally, this works fine. However, my settings seem to be overridden when I deactivate the application and the window is no longer key. The fieldEditor NSTextView remains there, but drawing changes to a white text color and light gray selection color (the defaults). Does anyone have suggestions for how I can customize this drawing?

Perceivable

Answer 1

You can override [NSWindow willReturnFieldEditor:toObject:] and return there custom NSTextView with changed selection color.

Alfred

Answer 2

Inspired by the answer to this question, the solution is to create an override of the NSLayoutManager that customizes the way in which the highlighting is performed based on the first responder state of the NSText view that owns it.

If the text view associated with this custom layout manager is the first responder, then it draws the selection using the color provided by macOS. If the text view is not the first responder, it uses the text view's background color as the selection color unless a custom color is provided via the setCustomInactiveColor method.

// ---------------------------------------------------------------------------
//  IZLayoutManager CLASS
// ---------------------------------------------------------------------------
// Override NSLayoutManager to change how the currently selected text is
// highlighted when the owning NSTextView is not the first responder.

@interface IZLayoutManager : NSLayoutManager
{
}
-(instancetype)initWithOwningTextView:(NSTextView*)inOwningTextView;
@property (nullable, assign, nonatomic) NSTextView* owningTextView;
@property (nullable, strong, nonatomic) NSColor* customInactiveColor;
@end

@implementation IZLayoutManager

- (instancetype)initWithOwningTextView:(NSTextView*)inOwningTextView
{
    self = [super init];
    
    if (self)  {
        self.owningTextView = inOwningTextView;
    }
    
    return self;
}

- (void) dealloc
{
    // my project is non-ARC; so we maually release any custom color
    // we received; in non-ARC projects this is probably not necessary
    if (self.customInactiveColor != NULL) {
        [self.customInactiveColor release];
        self.customInactiveColor = NULL;
    }
    [super dealloc];
}

// see extensive description of fillBackgroundRectArray in NSLayoutManager.h
// TL;DR: if you change the background color here, you must restore it before
// returning from this call

- (void) fillBackgroundRectArray:(const NSRect *)rectArray count:(NSUInteger)rectCount forCharacterRange:(NSRange)charRange color:(NSColor *)color
{
    BOOL needToReestoreColor = NO;
    
    if (self.owningTextView != NULL && [[self.owningTextView window] firstResponder] != self.owningTextView) {
        if (self.customInactiveColor != NULL) {
            [self.customInactiveColor setFill];
        } else {
            [[self.owningTextView backgroundColor] setFill];
        }
        needToReestoreColor = true;
    }
    
    [super fillBackgroundRectArray:rectArray count:rectCount forCharacterRange:charRange color:color];
    
    if (needToReestoreColor) {
        [color setFill];
    }
}
@end

Then, after you've allocated the NSTextView, you need to do this:

NSTextView* myTextView = ... // get a reference to your text view

// allocate our custom layout manager
IZLayoutManager* layoutManager = [[[IZLayoutManager alloc] initWithOwningTextView:self] autorelease];

// if you want to use a color other than the background for
// the selected text, uncomment the following line and
// supply your desired color

// [layoutManager setCustomInactiveColor:[NSColor redColor]];

[[myTextView textContainer] replaceLayoutManager:layoutManager];

Harts