Why does XmlSerializer cause a deadlock in two-way Named Pipe communication?

One problem you are having is that, by default, the XmlReader used by XmlSerializer will try to validate that the input stream is a well-formed XML document that has exactly one root element. And to do that, it must try to read past the current object in your PipeStream. But since nothing has yet been written to the stream by your pipe writer, this hangs.

You have a couple of options to avoid this. Firstly, you could use message framing with length prefixing to delimit each XML object in the pipe stream. See e.g.:

• How to create TCP message framing for the stream

With this approach, each read and write would read no more than the amount specified by the length prefix, copying the received bytes into some MemoryStream for subsequent deserialization.

Or, since you are using PipeTransmissionMode.Message, you could copy full incoming message into some MemoryStream, then deserialize that instead.

This approach could be used with any serialization format such as JSON, not just XML.

Alternatively, you could create an XmlReader with ConformanceLevel.Fragment and use ReadSubtree() to read just the next fragment. First create the following extension methods:

CSHARPCopied
public static partial class XmlExtensions
{
    public static T? DeserializeFragment<T>(this XmlSerializer? serializer, Stream stream)
    {
        XmlReaderSettings settings = new()
        {
            ConformanceLevel = ConformanceLevel.Fragment,
            CloseInput = false,
        };
        using var reader = XmlReader.Create(stream, settings);
        return serializer.DeserializeFragment<T>(reader);
    }
    
    public static T? DeserializeFragment<T>(this XmlSerializer? serializer, XmlReader reader)
    {
        serializer ??= new XmlSerializer(typeof(T));
        reader.MoveToContent();
        using var subReader = reader.ReadSubtree();
        return (T?)serializer.Deserialize(subReader);
    }
}

Then use them like so:

CSHARPCopied
var reply = serializer.DeserializeFragment<Common.Message>(pipe);

Notes:

• While DeserializeFragment<T>(this XmlSerializer? serializer, Stream stream) will only attempt to read as far as the end of the current fragment, there is a chance that additional content might be read from the incoming stream due to buffering. Since your pipe stream only contains a single object, this should not be a problem, but if it is you would need to use a single XmlWriter along with the DeserializeFragment<T>(this XmlSerializer? serializer, XmlReader reader) overload.
• I notice you are using the XmlSerializer(Type, Type[]) constructor. If you use this constructor (or any other constructor other than new XmlSerializer(Type) or new XmlSerializer(Type, String)) you must statically cache and reuse the serializer as explained in the documentation. See also Memory Leak using StreamReader and XmlSerializer.

Mockup fiddle demonstrating the problem here and the fix here.