C#•3y ago

Converting ReadOnlySequence of bytes to object

I have to convert a ReadOnlySequence<byte> to an object, the way I'm doing it currently is like so:

    public static DataModel ConvertBytes(ReadOnlySequence<byte> b)
    {
        var msg = new DataModel();

        int location = 8;

        msg.FFOn = BitConverter.ToInt32(b, location) != 0; location += 4;
        msg.FFSel = BitConverter.ToInt32(b, location) != 0; location += 4;
        msg.On = BitConverter.ToInt32(b, location) != 0; location += 4;
        msg.Sel = BitConverter.ToInt32(b, location) != 0; location += 4;
        msg.On = BitConverter.ToInt32(b, location) != 0; location += 4;
        msg.Sel = BitConverter.ToInt32(b, location) != 0; location += 4;

        // ...
        
        location += 76 * 4;

        msg.ZoneShapeTarget01 = BitConverter.ToSingle(b, location); location += 4;
        msg.ZoneShapeTarget02 = BitConverter.ToSingle(b, location); location += 4;
    }

    public static DataModel ConvertBytes(ReadOnlySequence<byte> b)
    {
        var msg = new DataModel();

        int location = 8;

        msg.FFOn = BitConverter.ToInt32(b, location) != 0; location += 4;
        msg.FFSel = BitConverter.ToInt32(b, location) != 0; location += 4;
        msg.On = BitConverter.ToInt32(b, location) != 0; location += 4;
        msg.Sel = BitConverter.ToInt32(b, location) != 0; location += 4;
        msg.On = BitConverter.ToInt32(b, location) != 0; location += 4;
        msg.Sel = BitConverter.ToInt32(b, location) != 0; location += 4;

        // ...
        
        location += 76 * 4;

        msg.ZoneShapeTarget01 = BitConverter.ToSingle(b, location); location += 4;
        msg.ZoneShapeTarget02 = BitConverter.ToSingle(b, location); location += 4;
    }

Is there a better way to handle this? BitConverter doesn't support ReadOnlySequence, so I would have to convert it to a byte array which is not ideal. I also have tons of properties on this model so it is a ton of code. Thanks in advance!

29 Replies

mtreit•3y ago

You want to use SequenceReader<byte> instead of BitConverter

var sequence = new ReadOnlySequence<byte>(new byte[] { 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x3 });
var reader = new SequenceReader<byte>(sequence);

while (true)
{
    if (reader.TryReadBigEndian(out int num))
    {
        Console.WriteLine(num);
        continue;
    }

    break;
}

var sequence = new ReadOnlySequence<byte>(new byte[] { 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x3 });
var reader = new SequenceReader<byte>(sequence);

while (true)
{
    if (reader.TryReadBigEndian(out int num))
    {
        Console.WriteLine(num);
        continue;
    }

    break;
}

Here's a small example of reading two integers from a ReadOnlySequence<byte> without unnecessary allocations.

DeterminismOP•3y ago

If I have a string type, how would I read that out? Do I just need to keep looping to read all of the properties I'm just a bit confused on how to parse the ReadOnlySequence<byte> into the model itself

mtreit•3y ago

Well, hand de-serializing a raw byte sequence is not exactly a simple beginners exercise...why are you having to do that in the first place, as opposed to using an off-the-shelf serializer? Strings can be tricky because you have to also encode the length somewhere unless it's a fixed length.

var sequence = new ReadOnlySequence<byte>(new byte[] { 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x3, 0x68, 0x69 });
var reader = new SequenceReader<byte>(sequence);

int read = 0;

while (true)
{
    if (reader.TryReadBigEndian(out int num))
    {
        read += 4;
        Console.WriteLine(num);
        continue;
    }

    if (read == 8)
    {
        // Get the string at the end
        var str = Encoding.UTF8.GetString(sequence.Slice(8));
        Console.WriteLine(str);
    }

    break;
}

var sequence = new ReadOnlySequence<byte>(new byte[] { 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x3, 0x68, 0x69 });
var reader = new SequenceReader<byte>(sequence);

int read = 0;

while (true)
{
    if (reader.TryReadBigEndian(out int num))
    {
        read += 4;
        Console.WriteLine(num);
        continue;
    }

    if (read == 8)
    {
        // Get the string at the end
        var str = Encoding.UTF8.GetString(sequence.Slice(8));
        Console.WriteLine(str);
    }

    break;
}

Assuming we have the easy task of just having a single string at the end, here is an example of reading a string from the byte sequence.

DeterminismOP•3y ago

There is a single string, but I also have float and booleans as well Do you know of a good off the shelf serializer I can use? I've tried MarshalAs but I can't skip bytes that I don't need

[StructLayout(LayoutKind.Sequential, Pack = 1, CharSet = CharSet.Ansi)]
public struct StandControllerDataModel
{
    //public DateTime TimeStamp;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 8)]
    public string? Filler3;
    public bool AutoBendingFFOn;

[StructLayout(LayoutKind.Sequential, Pack = 1, CharSet = CharSet.Ansi)]
public struct StandControllerDataModel
{
    //public DateTime TimeStamp;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 8)]
    public string? Filler3;
    public bool AutoBendingFFOn;

This for example I have to use a "filler" string for the first 8 bytes, because I want to skip them

mtreit•3y ago

Are you passing this to C code or something? By off the shelf serializer I meant something like System.Text.Json

DeterminismOP•3y ago

No I'm just trying to serialize that ReadOnlySequence<byte> into that StandControllerDataModel

mtreit•3y ago

I was wondering why you are using a ReadOnlySequence<byte> in the first place.

DeterminismOP•3y ago

Well I'm receiving it from tcp

DeterminismOP•3y ago

https://hastebin.com/gusuwokihi.csharp

Hastebin: Send and Save Text or Code Snippets for Free | Toptal®

Hastebin is a free web-based pastebin service for storing and sharing text and code snippets with anyone. Get started now.

DeterminismOP•3y ago

Using pipe.Reader.ReadAsync which gives me that ReadOnlySequence<byte>

mtreit•3y ago

I guess I'm wondering why on the side where the data is originating you don't just serialize to something simple like json - now you have a stream of text you can send over the wire (using your tcp implementation) and then on the other side you just deserialize that stream back into your original object. Instead of writing your own completely custom binary serialization mechanism. Also, did you consider just using a standard implementation like gRPC rather than rolling your own?

DeterminismOP•3y ago

Serialize the byte array to json? How would I do that I guess I'm just trying to figure out how I can serialize that byte array into a model using any method possible

mtreit•3y ago

No, I mean instead of a byte array Who creates the byte array? Have that code create json instead Then convert that to bytes to send over the wire Then it's just simple text. That you can trivially deserialize without writing a bunch of custom code.

DeterminismOP•3y ago

I don't have control over the device that is sending the messages

mtreit•3y ago

That's highly unfortunate

DeterminismOP•3y ago

Yes, so in my situation, what do you recommend I do

mtreit•3y ago

Use SequenceReader and Encoding to read the data as I illustrated. If you know all of the fields and they have fixed sizes it shouldn't be too bad. The SequenceReader advances on each read so you don't have to do all that manual offset arithmetic.

DeterminismOP•3y ago

Ok, does that have a way to read in bools and floats or should I code out a way to figure out the type Only have strings, floats, bools, and ints

mtreit•3y ago

I'm pretty sure it supports all of the primitive types Er For bool you probably need to read it as a byte and then convert it to bool But that's trivial You might need to do something a little different for float

DeterminismOP•3y ago

I'll figure something out, thanks for the help

jcotton42•3y ago

@mtreit isn't that part of the stuff introduced for Pipelines?

mtreit•3y ago

This is how you can read a float:

    if (reader.TryReadBigEndian(out int tmp))
    {
        var f = BitConverter.Int32BitsToSingle(tmp);
        Console.WriteLine(f);
    }

    if (reader.TryReadBigEndian(out int tmp))
    {
        var f = BitConverter.Int32BitsToSingle(tmp);
        Console.WriteLine(f);
    }

jcotton42•3y ago

I really need to use those at some point, it looks so neat

mtreit•3y ago

I'm not exactly sure I also want to try out pipelines

jcotton42•3y ago

yeah, the PipeReader returns a buffer as a ReadOnlySequence

mtreit•3y ago

We had someone take some code I wrote and re-write it to use pipelines and it completely regressed performance and I made him revert it. However, I believe he subsequently figured out he had a mistake in how he implemented it. I haven't circled back on it though. To be honest I had never seen a ReadOnlySequence until @Determinism asked this question.

jcotton42•3y ago

yeah, Pipelines isn't necessarily faster it just makes faster, correct, code easier to write

jcotton42•3y ago

https://docs.microsoft.com/en-us/dotnet/api/system.buffers.sequencereader-1.tryadvancetoany?view=net-6.0#system-buffers-sequencereader-1-tryadvancetoany(system-readonlyspan((-0))-system-boolean) 👀

SequenceReader.TryAdvanceToAny(ReadOnlySpan, Boolean) Method (Syste...

Searches for any of a number of specified delimiters and optionally advances past the first one to be found.

jcotton42•3y ago

we should move to #chat

Gaming

Programming

Converting ReadOnlySequence of bytes to object