C#•3y ago

❔ Runtime Endianness for Reading Structured Binary Data

Is there a standard way to deal with reading structured binary data in from a stream where the endianness is known only at runtime? In other words, it would be great if there was something with the same API as the

BinaryReader

BinaryReader

but where the endianness for the reader could be set.

I'm pretty new to C# so I didn't know if there was a idiomatic way to do this. In go I have used the

binary.Read

binary.Read

(https://pkg.go.dev/encoding/binary#Read) API to allow for reading in primitive types while specifying the endianness. Similarly, I have used the Java

ByteBuffer

ByteBuffer

type (https://docs.oracle.com/javase/8/docs/api/java/nio/ByteBuffer.html) to accomplish this on the JVM.

From my research, I have discovered (correct me if I'm wrong) that everything in dotnet is little endian. I have also found the

BinaryPrimatives

BinaryPrimatives

class which offers different methods for reading/writing binary primitives with different endiannesses. However, using this would require conditional statements wherever a read is performed. I wasn't sure if there was an extant class that provides a sleeker API for this. I know I could wrap it myself, but didn't want to reinvent the wheel if it was just something I missed

ero•8/9/23, 8:43 PM

Nothing built-in

ero•8/9/23, 8:44 PM

I'm not sure whether there's a NuGet for it, but writing it yourself shouldn't be too difficult

RJOP•8/9/23, 8:46 PM

Ah, gotcha. Okay, well thanks for the info! Figured I'd ask before writing code haha

cap5lut•8/9/23, 8:51 PM

(https://learn.microsoft.com/en-us/dotnet/api/system.bitconverter.islittleendian?view=net-7.0) is a runtime constant, so basically the if-check of something like

will be reduced to the respective method call

ero•8/9/23, 8:53 PM

oh interesting, i would have solved it with an interface and 2 implementations

ero•8/9/23, 8:54 PM

actually yeah i don't think that's what they want lol

RJOP•8/9/23, 8:55 PM

The endianness of the stream I'm reading is not necessarily that of the host system. It's encoded in a file at a specific location, I read that to determine the endianness of the file, then have to adjust the loads accordingly.

Eero oh interesting, i would have solved it with an interface and 2 implementations

cap5lut•8/9/23, 8:56 PM

same here tbh, its less bloat in regards of writing both variants - tho i usually use SGs for such work

RRJ The endianness of the stream I'm reading is not necessarily that of the host sys...

cap5lut•8/9/23, 8:57 PM

well, then the 2 classes + 1 interface approach would be the best

RJOP•8/9/23, 9:00 PM

Thanks, y'all!

cap5lut•8/9/23, 9:01 PM

are u forced to use a specifc way for the binary (de-)serialization? if not there are quite some libs out there to not re-invent the wheel

RJOP•8/9/23, 9:05 PM

There's no restriction on the method I used to go from binary on disk -> object, so any libraries that could help would be appreciated. I'm new to the C# ecosystem, so not familiar with the usual suspects when it comes to this

cap5lut•8/9/23, 9:06 PM

MessagePack is a quite known one: https://github.com/MessagePack-CSharp/MessagePack-CSharp

GitHub

GitHub - MessagePack-CSharp/MessagePack-CSharp: Extremely Fast Mess...

Extremely Fast MessagePack Serializer for C#(.NET, .NET Core, Unity, Xamarin). / msgpack.org[C#] - GitHub - MessagePack-CSharp/MessagePack-CSharp: Extremely Fast MessagePack Serializer for C#(.NET,...

cap5lut•8/9/23, 9:07 PM

there is also google's protobuf

cap5lut•8/9/23, 9:08 PM

https://github.com/Cysharp/MemoryPack is binary stuff too

GitHub

GitHub - Cysharp/MemoryPack: Zero encoding extreme performance bina...

Zero encoding extreme performance binary serializer for C# and Unity. - GitHub - Cysharp/MemoryPack: Zero encoding extreme performance binary serializer for C# and Unity.

cap5lut•8/9/23, 9:10 PM

dunno which will fit ur needs, maybe none of these, but it gives ya starting point to look around for options or to decide to go for something self-written

cap5lut•8/9/23, 9:11 PM

its also a matter what ur use case is. no need to use something high performant but complex or something selfwritten if its in a cold path

RJOP•8/9/23, 9:19 PM

Awesome, thank you so much for that! I will look through these and see how I can fit it into my goals

cap5lut•8/9/23, 9:29 PM

one last hint: if u need high performance, use source generator based solutions

ero•8/9/23, 9:34 PM

i don't see how source gen would help here

ero•8/9/23, 9:34 PM

they said they only know the endianness at runtime

ero•8/9/23, 9:34 PM

so sg doesn't really make sense as a solution

cap5lut•8/9/23, 9:35 PM

that was apart from the endianness thingy, and u can let these libs generate both versions, for big and little endianness iirc

Accord•8/10/23, 9:49 PM

Was this issue resolved? If so, run

/close

/close

otherwise I will mark this as stale and this post will be archived until there is new activity.