C#•2y ago

Protobuf deserialization not working after upgrade to .net 6

Hello, I am trying to deserialize a buffer (byte array) into a model. I upgraded my solution from core 3.1 to 6 and it no longer works. I am receiving the error

Error: ProtoBuf.ProtoException: Invalid wire-type (Varint); this usually means you have over-written a file without truncating or setting the length; see https://stackoverflow.com/q/2152978/23354

Error: ProtoBuf.ProtoException: Invalid wire-type (Varint); this usually means you have over-written a file without truncating or setting the length; see https://stackoverflow.com/q/2152978/23354

My code is :

        public WeldSeamImageRun ConvertBuffer(byte[] buffer)
        {
            return DecodeProtoBuf<WeldSeamImageRun>(buffer);
        }

        private T DecodeProtoBuf<T>(byte[] buffer)
        {
            var trimmedBuffer = TrimEnd(buffer);

            var stream = new MemoryStream(trimmedBuffer);

            stream.Position = 0;

            stream.SetLength(trimmedBuffer.Length);

            var telegram = Serializer.DeserializeWithLengthPrefix<T>(stream, PrefixStyle.Base128, 1);

            return telegram;
        }

    [ProtoContract]
    public class WeldSeamImageRun
    {
        [ProtoMember(1)] public List<WeldSeamImage> Images { get; set; }
        [ProtoMember(2)] public string EntryCoil { get; set; }
        [ProtoMember(3)] public List<EntryCoilSection> Sections { get; set; }
    }

        public WeldSeamImageRun ConvertBuffer(byte[] buffer)
        {
            return DecodeProtoBuf<WeldSeamImageRun>(buffer);
        }

        private T DecodeProtoBuf<T>(byte[] buffer)
        {
            var trimmedBuffer = TrimEnd(buffer);

            var stream = new MemoryStream(trimmedBuffer);

            stream.Position = 0;

            stream.SetLength(trimmedBuffer.Length);

            var telegram = Serializer.DeserializeWithLengthPrefix<T>(stream, PrefixStyle.Base128, 1);

            return telegram;
        }

    [ProtoContract]
    public class WeldSeamImageRun
    {
        [ProtoMember(1)] public List<WeldSeamImage> Images { get; set; }
        [ProtoMember(2)] public string EntryCoil { get; set; }
        [ProtoMember(3)] public List<EntryCoilSection> Sections { get; set; }
    }

Thanks for any help in advance.

12 Replies

canton7•2y ago

You haven't shown it, but I'll put money on this issue: https://learn.microsoft.com/en-us/dotnet/core/compatibility/core-libraries/6.0/partial-byte-reads-in-streams

Breaking change: Partial and zero-byte reads in DeflateStream, GZip...

Learn about the .NET 6 breaking change in core .NET libraries where DeflateStream, GZipStream, and CryptoStream handle partial and zero-byte reads differently.

canton7•2y ago

So you're reading from a DeflateStream, GZipStream, or CryptoStream, and you're not properly reading until Read returns 0? That's the main stream-based change in .net 6 which catches people out. Code which is breaking was technically incorrect and broken before, but this .net 6 change exposed those bugs I'm also slightly nervous about what TrimEnd is doing and why it's neede But regardless, I think the issue is with how you get buffer, in code you haven't shown

Determinism•2y ago

There are a ton of extra 0s at the end, which I'm trimming But I wasn't doing that before I upgraded to .net 6, I just figured that may be an issue

canton7•2y ago

There are a ton of extra 0s at the end, which I'm trimming

The fact that you need to do that is a symptom of something else that's wrong earlier... Please share the code where you read from the socket/whatever. I'm 99.8% sure it's wrong

Determinism•2y ago

public static byte[] EncodeProto<T>(T msg)
        {
            System.IO.MemoryStream stream = new System.IO.MemoryStream(2000);
            ProtoBuf.Serializer.Serialize(stream, msg);
            return stream.ToArray();
        }

public static byte[] EncodeProto<T>(T msg)
        {
            System.IO.MemoryStream stream = new System.IO.MemoryStream(2000);
            ProtoBuf.Serializer.Serialize(stream, msg);
            return stream.ToArray();
        }

This is the code for serialization The buffer I'm getting is far bigger than 2000 though So I'm assuming that's the issue, but I don't get why it was working on 3.1`

canton7•2y ago

I think that's fine Where is buffer read from when deserializing? Do you read it over a network? From a file? Something else?

Determinism•2y ago

It's coming from a database

canton7•2y ago

How do you read it from the database? Do you get it as a byte[], or as a stream which you then turn into a byte[]? (You're making this a lot harder than it needs to be btw)

Determinism•2y ago

Sorry lol, well I just read it from the database as a byte[] Then unzip it with

        public byte[] UnZipBuffer(byte[] input)
        {
            var zip = new System.IO.Compression.ZipArchive(new MemoryStream(input));
            var entry = zip.Entries.FirstOrDefault();
            if (entry is null) return null;

            var output = new byte[entry.Length];
            entry.Open().Read(output, 0, output.Length);
            return output;
        }

        public byte[] UnZipBuffer(byte[] input)
        {
            var zip = new System.IO.Compression.ZipArchive(new MemoryStream(input));
            var entry = zip.Entries.FirstOrDefault();
            if (entry is null) return null;

            var output = new byte[entry.Length];
            entry.Open().Read(output, 0, output.Length);
            return output;
        }

canton7•2y ago

There we go! I told you You're reading the Stream returned by entry.Open() incorrectly, and falling into the bug described in my link above Stream.Read isn't guaranteed to return all of the data in one call You need to keep calling Read until it returns 0

using (var stream = entry.Open())
{
    int totalRead = 0;
    while (totalRead < output.Length)
    {
        int bytesRead = stream.Read(output, totalRead, output.Length - totalRead);
        if (bytesRead == 0)
        {
            break;
        }
        totalRead += bytesRead;
    }
}

using (var stream = entry.Open())
{
    int totalRead = 0;
    while (totalRead < output.Length)
    {
        int bytesRead = stream.Read(output, totalRead, output.Length - totalRead);
        if (bytesRead == 0)
        {
            break;
        }
        totalRead += bytesRead;
    }
}

Or so.

Determinism•2y ago

That was the issue, I appreciate it

canton7•2y ago

👍

Gaming

Programming

Protobuf deserialization not working after upgrade to .net 6