Why Does XDocument Reject XML with Invalid Hex Entities While XmlDocument Processes Them?

The difference here is that LINQ to XML strictly enforces the Character Range constraint of the Extensible Markup Language (XML) 1.0 (Fourth Edition), while the older XmlDocument apparently does not:

Character Range

XMLCopied
[2]       Char       ::=      #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]  /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */

As you can see, #x1E is not in the above range, meaning an XML document that contains this character is not, strictly speaking, well-formed according to the version of the XML standard supported by .NET.[1].

If you don't want this, you could create versions of XDocument.Parse() and XDocument.ToString() that set XmlReaderSettings.CheckCharacters and XmlWriterSettings.CheckCharacters to false:

CSHARPCopied
public static class XDocumentExtensions
{
    static readonly XmlReaderSettings noCheckedCharacterParseSettings = new() { CheckCharacters = false, };
    static readonly XmlReaderSettings checkedCharacterParseSettings = new() { CheckCharacters = true, };

    public static XDocument Parse(string xml, bool checkCharacters) =>
        Parse(xml, checkCharacters ? checkedCharacterParseSettings : noCheckedCharacterParseSettings);
    
    public static XDocument Parse(string xml, XmlReaderSettings settings)
    {
        using var reader = new StringReader(xml);
        using var xmlReader = XmlReader.Create(reader, settings);
        return XDocument.Load(xmlReader);
    }

    static readonly XmlWriterSettings noCheckedCharacterToStringSettings = new() { CheckCharacters = false, Indent = true, OmitXmlDeclaration = true, };
    static readonly XmlWriterSettings checkedCharacterToStringSettings = new() { CheckCharacters = true, Indent = true, OmitXmlDeclaration = true, };
    
    public static string ToString(this XNode node, bool checkCharacters) =>
        node.ToString(checkCharacters ? checkedCharacterToStringSettings : noCheckedCharacterToStringSettings);
    
    public static string ToString(this XNode node, XmlWriterSettings settings)
    {
        using var writer = new StringWriter();
        using (var xmlWriter = XmlWriter.Create(writer, settings))
        {
            node.WriteTo(xmlWriter);
        }
        return writer.ToString();
    }
}

Then modify your code as follows:

CSHARPCopied
XDocument xdoc = new XDocument(
    new XElement("greeting", message)
);
Console.WriteLine(xdoc.ToString(checkCharacters : false));

And

CSHARPCopied
XDocument xDoc = XDocumentExtensions.Parse(xmlWithEscapedHexEntity, checkCharacters : false);
Console.WriteLine(xDoc.ToString(checkCharacters : false));

And you will be able to parse and format XML that is malformed purely due to containing invalid XML characters without any exceptions being thrown. Demo fiddle here.

That being said, I don't really recommend doing this, as the XML you generate will not be accepted by any receiving system that requires strict conformance with the XML standard. If you would prefer to remove invalid characters from your XML text, see:

• How do you remove invalid hexadecimal characters from an XML-based data source prior to constructing an XmlReader or XPathDocument that uses the data?
• Remove all hexadecimal characters before loading string into XML Document Object?

Now, as for why this difference exists? Hard to say for sure, but a check of the reference source shows that XmlDocument uses XmlTextReader to parse its XML. Both these types are very old (dating to .NET 1.1), and XmlTextReader was deprecated in .NET 2.0:

Starting with the .NET Framework 2.0, we recommend that you use the XmlReader class instead.

My guess is that Microsoft simply didn't implement character range checking in the initial .NET 1.1 XmlTextReader and XmlTextWriter implementations, then later did so when they introduced XmlReader and XmlWriter in .NET 2, and LINQ to XML in .NET 3.5. And, while a fair amount of guidance about early .NET versions was lost when MSDN links were retired, I did find the MSDN page Creating XML Readers, archived in 2013 but no longer actively available, that alludes to incomplete conformance checking by XmlTextReader:

By using the Create method and the XmlReaderSettings class you get the following benefits:

• Take full advantage of all the new features added to the XmlReader class in the .NET Framework 2.0 release. There are certain features, such as better conformance checking and compliance to the XML 1.0 recommendation, that are available only on XmlReader objects created by the Create method.

So it may be that guidance from Microsoft that character conformance checking was not fully implemented in the .NET 1.1 existed 10 or 15 years ago, but has since been lost.

---

Update

If you need a Parse() function that strips invalid XML characters whether embedded directly in the XML text or hex-encoded as character entities, you could use the following extension methods:

CSHARPCopied
public static class XDocumentExtensions
{
    static readonly XmlReaderSettings noCheckedCharacterParseSettings = new() { CheckCharacters = false, };
    
    public static XDocument ParseAndRemoveInvalidXmlCharacters(string xml, char? fallback = null)
    {
        ArgumentNullException.ThrowIfNull(xml);

        // From testing it seems that CheckCharacters=false only allows invalid character entities whose value falls outside the range from the standard
        // https://www.w3.org/TR/2006/REC-xml-20060816/#NT-Char
        //    [2]       Char       ::=      #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]  /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */
        // Invalid characters directly embedded in the character stream must be stripped out manually.
        using var reader = new StringReader(xml.RemoveInvalidXmlCharacters(fallback));
        using var xmlReader = XmlReader.Create(reader, noCheckedCharacterParseSettings);
        return XDocument.Load(xmlReader).RemoveInvalidXmlCharacters(fallback);
    }
    
    public static TXObject RemoveInvalidXmlCharacters<TXObject>(this TXObject node, char? fallback = null) where TXObject : XObject
    {
        switch (node)
        {
            case XText text:
                text.Value = text.Value.RemoveInvalidXmlCharacters(fallback);   
                break;
            case XAttribute attribute:
                attribute.Value = attribute.Value.RemoveInvalidXmlCharacters(fallback);
                break;
            case XComment comment:
                comment.Value = comment.Value.RemoveInvalidXmlCharacters();
                break;
            case XDocument doc:
                doc.Root?.RemoveInvalidXmlCharacters();
                break;
            case XElement element:
                foreach (var attr in element.Attributes())
                    attr.RemoveInvalidXmlCharacters();
                foreach (var child in element.Nodes())
                    child.RemoveInvalidXmlCharacters();
                break;
            case XContainer container: // XDocument
                foreach (var child in container.Nodes())
                    child.RemoveInvalidXmlCharacters();
                break;
            // Not done: XDocumentType, XProcessingInstruction
        }
        return node;
    }
    
    public static string RemoveInvalidXmlCharacters(this string xmlText, char? fallback = null)
    {
        ArgumentNullException.ThrowIfNull(xmlText);
    
        StringBuilder? sb = null;
    
        for (int i = 0; i < xmlText.Length; i++)
        {
            if (XmlConvert.IsXmlChar(xmlText[i]))
            {
                if (sb != null)
                    sb.Append(xmlText[i]);
            }
            else if (i < xmlText.Length - 1 && XmlConvert.IsXmlSurrogatePair(xmlText[i+1], xmlText[i])) // Yes this order is correct.
            {
                if (sb != null)
                    sb.Append(xmlText, i, 2);
                i++;
            }
            else
            {
                if (sb == null)
                {
                    sb = new();
                    sb.Append(xmlText, 0, i);
                }
                if (fallback != null)
                    sb.Append(fallback.Value);
            }
        }
        
        return sb?.ToString() ?? xmlText;
    }
}

And then do:

CSHARPCopied
var xdoc = new XDocument(
    new XElement("greeting", message)
).RemoveInvalidXmlCharacters();

Or

CSHARPCopied
var xDoc = XDocumentExtensions.ParseAndRemoveInvalidXmlCharacters(xmlWithInvalidCharacters);

Demo fiddle #2 here.

---

[1] While .NET officially only supports the XML 1.0 (4th Edition) standard, the 5th Edition has a similar constraint:

XMLCopied
[2]     Char       ::=      #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]  /* any Unicode character, excluding the surrogate blocks, FFFE, and FFFF. */

Now, as noted by Michael Kay in comments, escape characters such as  are allowed by XML 1.1, however .NET never implemented support for this XML version.