C#•14mo ago

❔ Mapping common primitive types with protobuf & grpc

I have a bunch of generated classes from proto that contain semi-pritmitive data types, for example Vector3. In every project I have to define (copy-paste) mapping code from the generated proto Vector3 to the proto Vector3. Are there any best practices doing this kind of stuff? I found auto mapping libraries but those still require me to define a mapping and I'd be repeating the mapping definitions in every project where I use the generated files.

37 Replies

JakenVeina•14mo ago

so, you're using a proto file as your source of truth?

AkroMentosOP•14mo ago

no I believe the System.Numerics.Vector3 is the best example I have. imagine the following proto definition:

message Vector3 {
  float X = 1;
  float Y = 2;
  float Z = 3;
}
message EventWithPosition {
   Vector3 Position = 1;
   string something = 2;
   // ...
}

message Vector3 {
  float X = 1;
  float Y = 2;
  float Z = 3;
}
message EventWithPosition {
   Vector3 Position = 1;
   string something = 2;
   // ...
}

At some point in the code after receiving this proto message you'd want to tranform the proto Vector3 into a System.Numerics.Vector3 to use it for calculations / call libraries, etc. In every project that I use the above proto file I have to copy-paste a mapping function that given a Namespace.Of.Just.Generated.Proto.Vector3 into a System.Numerics.Vector3.

JakenVeina•14mo ago

what's generating these models?

AkroMentosOP•14mo ago

I use Grpc.Tools

JakenVeina•14mo ago

which generates from what?

AkroMentosOP•14mo ago

proto ah you meant source of truth for the code generation... I edited OP because for some reason my brain wrote that I have generated classes from grpc instead of "i have generated classes from proto"

JakenVeina•14mo ago

is Grpc.Tools a source generator? or just a scaffolding tool?

AkroMentosOP•14mo ago

it's generating code compile time so I believe it's using source generators, or something similar but no idea how they choose to implement codegen

JakenVeina•14mo ago

well, that's what I was getting at I dunno, unless they specifically have hooks for allowing you to define your own type mappings, you're probably out-of-luck

AkroMentosOP•14mo ago

Even if it did I wouldn't want to define any project/tool specific thing in the proto file as these are shared between completely different teams using different tech stack

JakenVeina•14mo ago

right, I'm talking about Grpc.Tools

AkroMentosOP•14mo ago

so what you're suggesting is that if there was a way it'd be best if instead of the generated Vector3, the code generator could already put System.Numerics.Vector3 in there. just imagingin how that could possible work, at that point the generated code couldn't possibly call the proto write on that vector3 because obviously System.Numerics.Vector3 is not a proto class. There'd need to exist a mapping still somewhere 😐

JakenVeina•14mo ago

right it's entirely possible, but I have no idea if Grpc.Tools supports anything like that you'll have to read their docs or, more likely, as it looks like they don't really HAVE any docs, you'll have to dive through source code

AkroMentosOP•14mo ago

that's fine, can you please give me the tool whith which you're familiar where it's possible? I can take inspiration from that (and yes grpctools has basically no docs..)

JakenVeina•14mo ago

it's... not something you can take inspiration from Grpc.Tools is either extensible, or it isn't

AkroMentosOP•14mo ago

I mean.. it's open source I can fork / rewrite do anything, I can also switch over to what you're using if it has the featureset that I need... I'm not married to grpc.tools...

JakenVeina•14mo ago

what do you mean "what I'm using"? I'm not "using" anything

AkroMentosOP•14mo ago

sorry, I thought you had something specific in mind when you said

it's entirely possible, but I have no idea if Grpc.Tools supports anything like that

it came across that you are also using protobuf and know of a library or tool that supports what I need or have solved this issue otherwise

JakenVeina•14mo ago

I've used protobuf a bit before, but the other way around C# models as the source of truth, with the option to generate proto files for other languages, if I want and I'm vaguely familiar with a few source gen libraries

AkroMentosOP•14mo ago

hm, interesting approach, unfortunately I get the proto files so I have to use that as source

JakenVeina•14mo ago

right you could always write your own source gen wraffSmug

AkroMentosOP•14mo ago

I do that already for other stuff, and I would do it here as well. My main issue is that I have no idea how a solution could look like at the end of the day that mapping has to be defined somewhere

JakenVeina•14mo ago

right so, how does the source gen lib pick up on the proto files to gen for?

AkroMentosOP•14mo ago

<Protobuf Include="../../proto/common/*.proto" ProtoRoot="../../proto" />

<Protobuf Include="../../proto/common/*.proto" ProtoRoot="../../proto" />

then it's either some msbuild step of source generators that do the magic

JakenVeina•14mo ago

sure so, there could be a config file that you include in a similar manner or just config that you specify directly in the MSBuild the config itself ought to be fairly simple, right? you ultimately just need to say "for type X in protobuf, translate it to System.Numerics.Vector3" and the source gen would need to do a little but of analysis to find that system type and make sure it's compatible and emit whatever translation code might be necessary or, like, emit some kind of TypeConverter<T> class for each mapping whatever the appropriate code would be if you were writing it yourself

AkroMentosOP•14mo ago

ooo that's clever, like generate the converters as well

JakenVeina•14mo ago

if that's what it would take

AkroMentosOP•14mo ago

protobuf generates partial classes so I can also generate implicit conversions "into" the proto classes

JakenVeina•14mo ago

like proxy properties? well, that sounds like a reasonable-enough solution maybe not ideal

AkroMentosOP•14mo ago

that or just simply

 public static implicit operator Generated.Protobuf.Vector3(System.Numerics.Vector3 val)

 public static implicit operator Generated.Protobuf.Vector3(System.Numerics.Vector3 val)

so you'd still see the generated type in the protobuf but that can be implicitly converted to the desired domain type

JakenVeina•14mo ago

public class GeneratedMessageType
{
    public GeneratedVectorType Vector { get; set; }

    public Vector3 PrimitiveVector
    {
        get => Vector.Convert();
        set => Vector = value.ConvertBack();
    }
}

public class GeneratedMessageType
{
    public GeneratedVectorType Vector { get; set; }

    public Vector3 PrimitiveVector
    {
        get => Vector.Convert();
        set => Vector = value.ConvertBack();
    }
}

yeah I'd probably be a little more inclined to use an explicit property than an implicit cast those conversions can probably get expensive pretty quickly, so I'd rather have them be very visible

AkroMentosOP•14mo ago

good point, but will have to look out for name collisions

JakenVeina•14mo ago

it still seems like a common-enough scenario that Grpc.Tools ought to be extensible for you to put your own mappings in probably worth asking on their github page

AkroMentosOP•14mo ago

it doesn't have to be, I have a code generator for other purposes where I already do my own .proto -> .desc -> c# code, I can just hook into that and generate these mappings from a config file as well

JakenVeina•14mo ago

and if they don't support it, you could always contribute it yourself

AkroMentosOP•14mo ago

that way I'm still not married to grpc.tools

Accord•14mo ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

Gaming

Programming

❔ Mapping common primitive types with protobuf & grpc