C#•16mo ago

❔ Unsafe Reading different types of structs from a byte array

I'm currently using the Unsafe class to help me read (and also write) different types of structs into a byte array. This is my read function so far:

public static unsafe T ReadStruct<T>(byte[] array, int offset, int size) where T : unmanaged {
    T value = default;
    Unsafe.CopyBlock(ref *(byte*) &value, ref array[offset], (uint) size);
    return value;
}

public static unsafe T ReadStruct<T>(byte[] array, int offset, int size) where T : unmanaged {
    T value = default;
    Unsafe.CopyBlock(ref *(byte*) &value, ref array[offset], (uint) size);
    return value;
}

It works for me, even if the byte array contains something like 3 ints, a byte, and a long after than (21 byte elements). But I've also read about possible alignment issues which I'm not very familiar with... will my code run into problems at some point?

56 Replies

ero•16mo ago

What problem does this solve? Where does the array come from? Are offset and size truly necessary? To me this kinda just feels like Unsafe.ReadUnaligned<T>(ref array[offset]) Or ReadAligned, whichever fits better

br4kejetOP•16mo ago

I'm trying to pack structs into a single byte array but it kinda needs to be high performance I was just worried that at some point those Unsafe functions weren't going to function correctly though I don't know much about how the cpblk instruction works apart from what it's supposed to do

Aaron•16mo ago

I would really recommend just using the MemoryMarshal functions for this, if you can

MODiX•16mo ago

❯ Method: System.Runtime.InteropServices.MemoryMarshal.Read<T>(ReadOnlySpan<Byte>) Reads a structure of type T out of a read-only span of bytes.

React with ❌ to remove this embed.

ero•16mo ago

But why are you packing them into a byte array What are you doing with it

br4kejetOP•16mo ago

So that I can employ a kind of double-buffering thing 2 threads requiring access to the same data, so I wanna use double buffering instead of accessing a single field Then at some point, sync both threads and write the data from the primary thread into the secondary thread's array Saves having to create proxy classes containing the exact same data Like this?

public static unsafe T ReadStructV2<T>(byte[] array, int offset, int size) where T : unmanaged {
    return MemoryMarshal.Read<T>(new ReadOnlySpan<byte>(array, offset, size));
}

public static unsafe T ReadStructV2<T>(byte[] array, int offset, int size) where T : unmanaged {
    return MemoryMarshal.Read<T>(new ReadOnlySpan<byte>(array, offset, size));
}

Aaron•16mo ago

that works, though you don't really need the size

br4kejetOP•16mo ago

oh true

Aaron•16mo ago

you can just slice the span [offset..]

GrabYourPitchforks•16mo ago

Why does this require use of byte[] instead of T[]?

br4kejetOP•16mo ago

Because the array can contain different types of structs I was thinking of just using entirely pointers and using the HAlloc functions but that seems like a cardinal sin...

Aaron•16mo ago

(this is also not great, to be clear, I just wanted to make sure that if you do use this, you use MemoryMarshal to do it, since it does the correct thing)

GrabYourPitchforks•16mo ago

I mean, how many different struct types are we talking about here? You could always have a T[] and a U[] and a V[], for instance. It would be safer and almost certainly faster.

br4kejetOP•16mo ago

int, float, double, long, Vector2, Vector3, Matrix4x4, enums, and probably more

Aaron•16mo ago

how exactly do you know the layout of this array at any given time

br4kejetOP•16mo ago

It gets calculated in the static constructor I kinda replicated WPF's property system idea with it GetValue, SetValue, where the generic type is unmanaged The MemoryMarshal class seems to work nicely, i'll just stick with that

GrabYourPitchforks•16mo ago

You should really strive to do this without MemoryMarshal or Unsafe if possible. These are advanced scenarios that could bite you if you're not careful. If your static constructor knows the layout for any given type, wouldn't that be the correct place to set up a CustomStruct[]?

br4kejetOP•16mo ago

What kind of custom struct?

ero•16mo ago

Is this just for cross-thread communication? Or did I misunderstand

br4kejetOP•16mo ago

Pretty much yeah

GrabYourPitchforks•16mo ago

MemoryMarshal for cross thread comms is fine. That's one of the reasons it exists. But you're still in charge of offset bookkeeping, etc., which introduced potential for bugs. Hence why I am asking if there is any other way (with less potential for introducing bugs) that you can do this. Maybe the answer is no, but it's worth investigating.

ero•16mo ago

Do you have a usage example?

br4kejetOP•16mo ago

I know that unreal engine does what i'm trying to do but their code base is so massive that I can't really figure out exactly how they're doing it

br4kejetOP•16mo ago

Yeah sure, this is how I get and set values

br4kejetOP•16mo ago

Then this is how I could define the stuff

br4kejetOP•16mo ago

ObjectC and ObjectB's byte array would be 17 bytes, whereas ObjectAs would be 9

ero•16mo ago

I'm so confused

GrabYourPitchforks•16mo ago

Why not just use fields? And have R3Property or whatever get / set the fields directly.

br4kejetOP•16mo ago

Idek I coulda probably just used a virtual method called "WriteLiveData" or something like that But that also has a performance hit for hierarchial objects like a scene graph I thought it would be more performant to store a global list of updates that can be published that would automatically write the data from one byte array to another, and that list gets filled when you actually call the setter methods Haven't implemented that part yet but that's my idea

GrabYourPitchforks•16mo ago

That just seems like a really complicated way to say "I don't have an object with fields, I have a byte array."

br4kejetOP•16mo ago

If i did it with just fields I would have to do so much manual work to implement double buffering

GrabYourPitchforks•16mo ago

What kind of work do you expect to have to do?

br4kejetOP•16mo ago

If i had a player object and I wanted to set their position, i'd have to update the 'live' data, and then enqueue some sort of command (when the 2 threads are synced) that copies the 'live' data to the cached data And then the 2 threads can continue executing normally in parallel And that command I'm pretty sure would have to be an object class meaning there would be a ton of GC pressure if there's 10000s of updates per second Whereas with the system I made, I can just use a list of structs containing the object reference and property

GrabYourPitchforks•16mo ago

So copy the fields instead of copying the byte array contents. This can be done as a single line of code.

br4kejetOP•16mo ago

Yeah but like... Where do you think I should do that if I did go the fields way?

GrabYourPitchforks•16mo ago

Wherever you're currently calling Array.Copy, I guess ?

br4kejetOP•16mo ago

I'd set the app into a 'synchronize' state where main thread is ready to transfer data to the other thread (render thread in my case), and that thread is just sitting idle And now that's the perfect time to transfer the 'live' data to the 'cached' data

GrabYourPitchforks•16mo ago

Or interlocked exchange the live & cached objects.

br4kejetOP•16mo ago

If I went the fields way, It would have to be a recursive function for something like a scene object I can't modify any of the data that the render thread may use while it's currently doing things Otherwise I would just wrap every property getter and setter in a lock block My way just seems like the most convenient... unless your way is much easier but I just don't understand what you're trying to say lol

GrabYourPitchforks•16mo ago

It kinda sounds like you already have to write manual recursive logic to deal with offsets being modified when you have scene objects. Otherwise multiple nested complex objects will all try to write to the same index in the array, since the index is stored in the R3Property itself.

br4kejetOP•16mo ago

All of the byte offsets are calculated automatically, so I never have to deal with that I can just register properties and use them immediately And when I implement the global list of update tasks, I can make the set functions add a command to that list that would automatically move the data (that I just set) to the cached data, at the appropriate time All i'd need to do at that point is implement that thread synchronization and then iterate that global list

ero•16mo ago

I'm so confused can you not just send an actual struct over? Or have a reference to a class or struct in the threads and write there? Or use Interlocked.Exchange as mentioned?

br4kejetOP•16mo ago

Where would I do that though?

ero•16mo ago

Like why can you not just send a

struct Player
{
  Vector3 Position;
}

struct Player
{
  Vector3 Position;
}

br4kejetOP•16mo ago

Thats what I have but it would be a class not struct, then I would define the Vector3 Position property in there

ero•16mo ago

So it isn't what you have

br4kejetOP•16mo ago

I mean it's the same kind of layout Sort of...

MODiX•16mo ago

ero

REPL Result: Failure

record struct Foo(
  int Bar);

Foo foo = default;
bool waitForSync = false;

var t = Task.Run(() =>
{
  while (true)
  {
    if (!waitForSync)
      continue;

    if (foo is { Bar: 42 })
    {
      Consone.WriteLine("synced!");
      break;
    }
  }
});

Task.Run(() =>
{
  foo = new(42);
  waitForSync = true;
});

await t;

record struct Foo(
  int Bar);

Foo foo = default;
bool waitForSync = false;

var t = Task.Run(() =>
{
  while (true)
  {
    if (!waitForSync)
      continue;

    if (foo is { Bar: 42 })
    {
      Consone.WriteLine("synced!");
      break;
    }
  }
});

Task.Run(() =>
{
  foo = new(42);
  waitForSync = true;
});

await t;

Exception: CompilationErrorException

- The name 'Consone' does not exist in the current context

- The name 'Consone' does not exist in the current context

Compile: 823.393ms | Execution: 0.000ms | React with ❌ to remove this embed.

ero•16mo ago

I mean whatever you get the idea Why would this not be valid

br4kejetOP•16mo ago

It would be valid, but I'd have to copy foo for every single object in the application that requires double buffered data

br4kejetOP•16mo ago

But if I use my code, then I have a function like this

br4kejetOP•16mo ago

Which is called by this code

br4kejetOP•16mo ago

And that ProcessUpdates function would get called when the 2 threads are in sync PushUpdate can get called at any time I just feel like the likely overhead with properties is so much less than having a sort of recursive 'TransferValue' method, that would transfer all of the data for every object in the app that requires the double buffering

br4kejetOP•16mo ago

Example usage

br4kejetOP•16mo ago

SetValue accesses the live data, same with GetValue, but ReadValue accesses the cached data

Accord•16mo ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

Gaming

Programming

❔ Unsafe Reading different types of structs from a byte array