someString.IndexOf(someString) returns 1 instead of 0 under .NET 4

Question

We have recently upgraded all our projects from .NET 3.5 to .NET 4. I have come across a rather strange issue with respect to string.IndexOf().

My code obviously does something slightly different, but in the process of investigating the issue, I found that calling IndexOf() on a string with itself returned 1 instead of 0. In other words:

string text = "\xAD\x2D";          // problem happens with "­-dely N.China", too;
int index = text.IndexOf(text);    // see update note below.

Gave me an index of 1, instead of 0. A couple of things to note about this problem:

• The problems seems related to these hyphens (the first character is the Unicode soft hyphen, the second is a regular hyphen).

• I have double checked, this does not happen in .NET 3.5 but does in .NET 4.

• Changing the IndexOf() to do an ordinal compare fixes the issue, so for some reason that first character is ignored with the default IndexOf.

Does anyone know why this happens?

EDIT

Sorry guys, made a bit of a stuff up on the original post and got the hidden dash in there twice. I have updated the string, this should return index of 1 instead of 2, as long as you paste it in the correct editor.

Update:

Changed the original problem string to one where every actual character is clearly visible (using escaping). This simplifies the question a bit.

Shoup

Mono 2.10.8.1 (compatible to .net 4) returns 0... – Nahshu

What is your local default region/culture settings? – Yseulta

@Yseulta Program is in en-GB, but since I am comparing the string with itself, would it make a difference? – Shoup

Answer 1

Your string exists of two characters: a soft hyphen (Unicode code point 173) and a hyphen (Unicode code point 45).

Wiki: According to the Unicode standard, a soft hyphen is not displayed if the line is not broken at that point.

When using "\xAD\x2D".IndexOf("\xAD\x2D") in .NET 4, it seems to ignore that you're looking for the soft hyphen, returning a starting index of 1 (the index of \x2D). In .NET 3.5, this returns 0.

More fun, if you run this code (so when only looking for the soft hyphen):

string text = "\xAD\x2D";
string shy = "\xAD";
int i1 = text.IndexOf(shy);

then i1 becomes 0, regardless of the .NET version used. The result of text.IndexOf(text); varies indeed, which at a glance looks like a bug to me.

As far as I can track back through the framework, older .NET versions use an InternalCall to IndexOfString() (I can't figure out to which API call that goes), while from .NET 4 a QCall to InternalFindNLSStringEx() is made, which in turn calls FindNLSStringEx().

The issue (I really can't figure out if this is intended behaviour) indeed occurs when calling FindNLSStringEx:

LPCWSTR lpStringSource = L"\xAD\x2D";
LPCWSTR lpStringValue = L"\xAD";

int length;

int i = FindNLSStringEx(
    LOCALE_NAME_SYSTEM_DEFAULT,
    FIND_FROMSTART,
    lpStringSource,
    -1,
    lpStringValue,
    -1,
    &length,
    NULL,
    NULL,
    1);

Console::WriteLine(i);

i = FindNLSStringEx(
    LOCALE_NAME_SYSTEM_DEFAULT,
    FIND_FROMSTART,
    lpStringSource,
    -1,
    lpStringSource,
    -1,
    &length,
    NULL,
    NULL,
    1);

Console::WriteLine(i);

Console::ReadLine();

Prints 0 and then 1. Note that length, an out parameter indicating the length of the found string, is 0 after the first call and 1 after the second; the soft hyphen is counted as having a length of 0.

The workaround is to use text.IndexOf(text, StringComparison.OrdinalIgnoreCase);, as you've noted. This makes a QCall to InternalCompareStringOrdinalIgnoreCase() which in turn calls FindStringOrdinal(), which returns 0 for both cases.

Jessamyn

@Tigran when I paste your code directly into Visual Studio (i.e. a Unicode-aware editor), .NET 4's String.IndexOf() returns 2. – Jessamyn

I think OP's use VS too, sounds strange that he esplicitly declares a fact that string consists of char(173) + char(45) + reminder. It's true, if you copy/paste it shows more characters, but if you generate a string following declaration of OP, it returns 1. Si I think that the string presented on page here is not that one OP's asking for, and it's better to generate a string folloqing his suggessions. – Eastman

Sorry about that, please check the newly edited version, that should return 1 as stated. – Shoup

But as to your answer ... are you telling me that IndexOf ignores the dash because it will not be displayed? – Shoup

@Shoup yes, and that looks like a bug to me. When comparing strings, it should not ignore control characters. – Jessamyn

Ok, I am going to take this as unintended behavior until shown otherwise. Thanks for all the effort, CodeCaster. – Shoup

Answer 2

It seems be a bug in .NET4, and new changes reverted in .NET4 Beta1 to previous version same as .NET 2.0/3.0/3.5.

What's New in the BCL in .NET 4.0 CTP (MSDN blogs):

String Security Changes in .NET 4

The default partial matching overloads on System.String (StartsWith, EndsWith, IndexOf, and LastIndexOf) have been changed to be culture-agnostic (ordinal) by default.

This change affected the behavior of the String.IndexOf method by changing them to perform an ordinal (byte-for-byte) comparison by default an will be changed to use CultureInfo.InvariantCulture instead of CultureInfo.CurrentCulture.

UPDATE for .NET 4 Beta 1

In order to maintain high compatibility between .NET 4 and previous releases, we have decided to revert this change. The behavior of String's default partial matching overloads and String and Char's ToUpper and ToLower methods now behave the same as they did in .NET 2.0/3.0/3.5. The change back to the original behavior is present in .NET 4 Beta 1.

---

To fix this, change the string comparison method to an overload that accepts the System.StringComparison enumeration as a parameter, and specify either Ordinal or OrdinalIgnoreCase.

// string contains 'unicode dash' \x2D
string text = "\xAD\x2D"; 

// woks in .NET 2.0/3.0/3.5 and .NET 4 Beta 1 and later
// but seems be buggy in .NET 4 because of 'culture-sensitive' comparison        
int index = text.IndexOf(text); 

// fixed version
index = text.IndexOf(text, StringComparison.Ordinal); 

Chock

Please add a short summary of the post you link to directly into the answer. It is always great to refer to a source, but any SO answer should carry meaning without having to follow external links. – Helve

So from what you've quoted they didn't change anything and the breaking changes-link is outdated. – Cotten

Thanks for adding this detailed info to the thread. Very helpful. – Inamorata

Answer 3

From the documentation (my emphasis):

This method performs a word (case-sensitive and culture-sensitive) search using the current culture.

Ie. some distinct code-points will be treated as equal.

What happens if you use an overload that takes a StringComparison value and pass StringComparison.Ordinal to avoid the cultural dependencies?

Sleigh

As stated in the post, changing to Ordinal does fix the issues, but I do not understand why. Since you are comparing the string with itself, it should have the same culture settings so it should not matter that it is doing a case-sensitive and culture-sensitive search? – Shoup

@Shoup A culturally dependent comparison will change the result of many comparisons. Eg. a case-insensitive comparison of I and i in a Turkish locale is unequal. Outside that culture's core character set characters are often ignored. (in English this includes accents). This is a complex area that I only understand a little of. A trawl through Michael Kaplan's blog might be the easiest route. A few years ago he explained how the process works in Win32 (which .NET uses). – Sleigh