Determining if a Unicode character is visible?

Question

I am writing a text editor which has an option to display a bullet in place of any invisible Unicode character. Unfortunately there appears to be no easy way to determine whether a Unicode character is invisible.

I need to find a text file containing every Unicode character in order that I can look through for invisible characters. Would anyone know where I can find such a file?

EDIT: I am writing this app in Cocoa for Mac OS X.

Langouste

By "invisible" do you mean a glyph that isn't available in the selected font? Or something else, like characters that are part of a composite? – Teador

I mean characters that do not appear on the screen. I want to replace them with bullets so that users can tell that they are there. – Langouste

I added another answer which may help answer the other question... if not, let me know. – Teador

Answer 1

Oh, I see... actual invisble characters ;) This FAQ will probably be useful:

http://www.unicode.org/faq/unsup_char.html

It lists the current invisible codepoints and has other information that you might find helpful.

EDIT: Added some Cocoa-specific information

Since you're using Cocoa, you can get the unicode character set for control characters and compare against that:

NSCharacterSet* controlChars = [NSCharacterSet controlCharacterSet];

You might also want to take a look at the FAQ link I posted above and add any characters that you think you may need based on the information there to the character set returned by controlCharacterSet.

EDIT: Added an example of creating a Unicode string from a Unicode character

unichar theChar = 0x000D;
NSString* thestring = [NSStirng stringWithCharacters:&theChar length:1];

Teador

Not all invisible characters are control characters. I think you would consider the zero-width characters invisible, but they aren't control characters. It also doesn't include the Unicode LINE SEPARATOR and PARAGRAPH SEPARATOR characters. – Grams

@Peter, Right, which is why I posted the FAQ first and suggested that the appropriate characters be added to the controlCharacterSet. – Teador

Answer 2

Let me know if this code helps at all:

-(NSString*)stringByReplacingControlCharacters:(NSString*)originalString
{
    NSUInteger length = [originalString length];
    unichar *strAsUnichar = (unichar*)malloc(length*sizeof(unichar));
    NSCharacterSet* controlChars = [NSCharacterSet controlCharacterSet];
    unichar bullet = 0x2022;

    [originalString getCharacters:strAsUnichar];
    for( NSUInteger i = 0; i < length; i++ ) {
        if( [controlChars characterIsMember:strAsUnichar[i]] )
            strAsUnichar[i] = bullet;
    }

    NSString* newString = [NSString stringWithCharacters:strAsUnichar length:length];
    free(strAsUnichar);

    return newString;
}

Important caveats:

This probably isn't the most efficient way of doing this, so you will have to decide how you want to optimize after you get it working. This only works with characters on the BMP, support for composted characters would have to be added if you have such a requirement. This does no error checking at all.

Teador

I appreciate your posting that code. It may well come in handy. However, the problem is that I am quite certain that the controlCharacterSet is only a small subset of all invisible characters. – Langouste

You might be interested in the code I am currently using, which can be found here: #300586 – Langouste

It's actually not a small subset, but it's definitely not all the invisible characters. You can look at the Unicode site for the full list of various characters, but some invisible characters you probably dont want to turn into bullets, like joiners and such. – Teador

Answer 3

A good place to start is the Unicode Consortium itself which provides a large body of data, some of which would be what you're looking for.

I'm also in the process of producing a DLL which you give a string and it gives back the UCNs of each character. But don't hold your breath.

Ineffectual

Answer 4

The current official Unicode version is 5.1.0, and text files describing all of the code points in that can be found at http://www.unicode.org/standard/versions/components-latest.html

Fern

Answer 5

For Java, java.lang.Character.getType. For C, u_charType() or u_isgraph().

Esmerolda

Answer 6

you might find this code to be of interest: http://gavingrover.blogspot.com/2008/11/unicode-for-grerlvy.html

Repletion

Answer 7

Its an impossible task, Unicode supports even Klingon, so it's not going to work. However most text editors use the standard ANSI invisible characters. And if your Unicode library is good, it will support finding equivalent characters and/or categories, you can use these two features to do it as well as any editor out there

Edit: Yes I was being silly about Klingon support, but that doesn't make it not true... of course Klingon is not supported by the Consortium, however there is a movement for Klingon in the Unicode's "Private Use Area" defined for Klingon alphabet (U+F8D0 - U+F8FF). Link here for those interested :)

Note: Wonder what editor Klingon programmers use...

Toscano

The actual Unicode standard does not include fictional scripts - perhaps the Consortium will add them someday, but for now they have far more to worry about. But the parts of Unicode that are mapped are very well-defined, so there is a comprehensive list of invisible characters. – Oculo

I think Robert was being facetious about support for Klingon. I realize that there are too many characters to make this approach feasible so I am looking for alternatives. – Langouste

Unicode doesn't support Klingon because everyone who writes Klingon does so w/ ASCII. The fancy Klingon characters aren't used in practice. But if I'm not mistaken, other imaginary scripts are supported. – Chem

Yeah like elvin (I don't remember the proper name for the script tho) – Teador