Unicode Regex; Invalid XML characters

Question

The list of valid XML characters is well known, as defined by the spec it's:

#x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]

My question is whether or not it's possible to make a PCRE regular expression for this (or its inverse) without actually hard-coding the codepoints, by using Unicode general categories. An inverse might be something like [\p{Cc}\p{Cs}\p{Cn}], except that improperly covers linefeeds and tabs and misses some other invalid characters.

Selfconscious

Answer 1

I know this isn't exactly an answer to your question, but it's helpful to have it here:

Regular Expression to match valid XML Characters:

[\u0009\u000a\u000d\u0020-\uD7FF\uE000-\uFFFD]

So to remove invalid chars from XML, you'd do something like

// filters control characters but allows only properly-formed surrogate sequences
private static Regex _invalidXMLChars = new Regex(
    @"(?<![\uD800-\uDBFF])[\uDC00-\uDFFF]|[\uD800-\uDBFF](?![\uDC00-\uDFFF])|[\x00-\x08\x0B\x0C\x0E-\x1F\x7F-\x9F\uFEFF\uFFFE\uFFFF]",
    RegexOptions.Compiled);

/// <summary>
/// removes any unusual unicode characters that can't be encoded into XML
/// </summary>
public static string RemoveInvalidXMLChars(string text)
{
    if (string.IsNullOrEmpty(text)) return "";
    return _invalidXMLChars.Replace(text, "");
}

I had our resident regex / XML genius, he of the 4,400+ upvoted post, check this, and he signed off on it.

Partible

Instead of text.IsNullOrEmpty() I think you need string.IsNullOrEmpty(text) – Mccartney

I would recommend not to strip invalid characters, but rather replace them with the replacement character � (FFFD). Stripping invalid characters makes debugging harder (problems become invisible) and in some cases it can lead to security holes. – Theologize

Thank you, this just saved me big time with a huge issue I was having – Gogol

@Jeff Atwood: Will it help me in this case? #29431854. Will I not lose some data if I remove those elements from my XML? – Opalescent

@MattEnright FFFE and FFFF are byte order markers and throw errors if you send the XML stream over to Java. – Grevera

Answer 2

I tried this in java and it works:

private String filterContent(String content) {
    return content.replaceAll("[^\\u0009\\u000a\\u000d\\u0020-\\uD7FF\\uE000-\\uFFFD]", "");
}

Thank you Jeff.

Carbone

In my case (c#) \\u000a\\u000d are valid characters – Moltke

Answer 3

For systems that internally stores the codepoints in UTF-16, it is common to use surrogate pairs (xD800-xDFFF) for codepoints above 0xFFFF and in those systems you must verify if you really can use for example \u12345 or must specify that as a surrogate pair. (I just found out that in C# you can use \u1234 (16 bit) and \U00001234 (32-bit))

According to Microsoft "the W3C recommendation does not allow surrogate characters inside element or attribute names." While searching W3s website I found C079 and C078 that might be of interest.

Nameplate

While this is a useful implementation tip, it doesn't really answer my question. Let's assume for arguments sake that the implementation has first-rate support of non-BMP characters, so surrogate characters are not needed at all. – Selfconscious

Answer 4

The above solutions didn't work for me if the hex code was present in the xml. e.g.

<element>&#x8;</element>

The following code would break:

string xmlFormat = "<element>{0}</element>";
string invalid = " &#x8;";
string xml = string.Format(xmlFormat, invalid);
xml = Regex.Replace(xml, @"[\x01-\x08\x0B\x0C\x0E\x0F\u0000-\u0008\u000B\u000C\u000E-\u001F]", "");
XDocument.Parse(xml);

It returns:

XmlException: '', hexadecimal value 0x08, is an invalid character. Line 1, position 14.

The following is the improved regex and fixed the problem mentioned above:

&#x([0-8BCEFbcef]|1[0-9A-Fa-f]);|[\x01-\x08\x0B\x0C\x0E\x0F\u0000-\u0008\u000B\u000C\u000E-\u001F]

Here is a unit test for the first 300 unicode characters and verifies that only invalid characters are removed:

[Fact]
        public void validate_that_RemoveInvalidData_only_remove_all_invalid_data()
        {
            string xmlFormat = "<element>{0}</element>";
            string[] allAscii = (Enumerable.Range('\x1', 300).Select(x => ((char)x).ToString()).ToArray());
            string[] allAsciiInHexCode = (Enumerable.Range('\x1', 300).Select(x => "&#x" + (x).ToString("X") + ";").ToArray());
            string[] allAsciiInHexCodeLoweCase = (Enumerable.Range('\x1', 300).Select(x => "&#x" + (x).ToString("x") + ";").ToArray());

            bool hasParserError = false;
            IXmlSanitizer sanitizer = new XmlSanitizer();

            foreach (var test in allAscii.Concat(allAsciiInHexCode).Concat(allAsciiInHexCodeLoweCase))
            {
                bool shouldBeRemoved = false;
                string xml = string.Format(xmlFormat, test);
                try
                {
                    XDocument.Parse(xml);
                    shouldBeRemoved = false;
                }
                catch (Exception e)
                {
                    if (test != "<" && test != "&") //these char are taken care of automatically by my convertor so don't need to test. You might need to add these.
                    {
                        shouldBeRemoved = true;
                    }
                }
                int xmlCurrentLength = xml.Length;
                int xmlLengthAfterSanitize = Regex.Replace(xml, @"&#x([0-8BCEF]|1[0-9A-F]);|[\u0000-\u0008\u000B\u000C\u000E-\u001F]", "").Length;
                if ((shouldBeRemoved && xmlCurrentLength == xmlLengthAfterSanitize) //it wasn't properly Removed
                    ||(!shouldBeRemoved && xmlCurrentLength != xmlLengthAfterSanitize)) //it was removed but shouldn't have been
                {
                    hasParserError = true;
                    Console.WriteLine(test + xml);
                }
            }
            Assert.Equal(false, hasParserError);
        }

Photoflood

Answer 5

Another way to remove incorrect XML chars in C# with using XmlConvert.IsXmlChar Method (Available since .NET Framework 4.0)

public static string RemoveInvalidXmlChars(string content)
{
   return new string(content.Where(ch => System.Xml.XmlConvert.IsXmlChar(ch)).ToArray());
}

or you may check that all characters are XML-valid.

public static bool CheckValidXmlChars(string content)
{
   return content.All(ch => System.Xml.XmlConvert.IsXmlChar(ch));
}

.Net Fiddle - https://dotnetfiddle.net/v1TNus

For example, the vertical tab symbol (\v) is not valid for XML, it is valid UTF-8, but not valid XML 1.0, and even many libraries (including libxml2) miss it and silently output invalid XML.

Congratulate

Answer 6

In PHP the regex would look like the following way:

protected function isStringValid($string)
{
    $regex = '/[^\x{9}\x{a}\x{d}\x{20}-\x{D7FF}\x{E000}-\x{FFFD}\x{10000}-\x{10FFFF}]+/u';

    return (preg_match($regex, $string, $matches) === 0);
}

This would handle all 3 ranges from the xml specification:

#x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]

Jobey

Curly braces are quantifiers, not matchers - this seems like invalid regex to me. – Poco