C#•2y ago

✅ Looking for assistance in Deserializing Avro Binary stream

where the schema comes from the API and I dont have a static type

57 Replies

Unknown User•2y ago

Message Not Public

Yea so basically salesforce went full stupid And they EMBED an arvo payload inside a Protoc payload lol gRPC event works fine Struggling with the Avro decoding Let me dump some of my attempts here if you have a second to help

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

Yea you're exactly right - the salesforce API DOES provide the schema I just don't know how to use Arvo in C#

c#
    public static dynamic DeserializeAvro(Schema schema, byte[] payload)
    {
        // Read the payload using Apache Avro.
        using (var memoryStream = new MemoryStream(payload))
        {
            var binaryDecoder = new BinaryDecoder(memoryStream);
            var reader = new GenericDatumReader<GenericRecord>(schema,schema);
            var record = reader.Read(null, binaryDecoder);

            // Map the payload to a dynamic object.
            var payloadDecoder = new BinaryDecoder(memoryStream);
            var payloadRecord = reader.Read(null, payloadDecoder);
            return payloadRecord;
        }
    }

c#
    public static dynamic DeserializeAvro(Schema schema, byte[] payload)
    {
        // Read the payload using Apache Avro.
        using (var memoryStream = new MemoryStream(payload))
        {
            var binaryDecoder = new BinaryDecoder(memoryStream);
            var reader = new GenericDatumReader<GenericRecord>(schema,schema);
            var record = reader.Read(null, binaryDecoder);

            // Map the payload to a dynamic object.
            var payloadDecoder = new BinaryDecoder(memoryStream);
            var payloadRecord = reader.Read(null, payloadDecoder);
            return payloadRecord;
        }
    }

I have this If you have any working code to dig up I would be super duper appreciative @tebeco

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

Didn't know

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

Nah 100% I get it Thanks so much anyway

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

RIP

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

Gahh Salesforce WHY WOULD THEY DO THIS

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

Yea I'm passed that point I have the schema in hand And i have the payload in hand (Binary serialized) And all i need is an example using Apache.Avro Or any Avro library To combine the two and get the data I need out of the payload in a decoded format Within .NET C# 🙂 I have everything i need except good programming skills lol

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

Lol this assumes that my Schema is bsaed on List<User> As in I know before run time

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

I want to take the schema that salesforce gives And use that Which i know is possible I just don't know how

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

I've tried all of these

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

This is likely to not work

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

So the Unary Channels i'm subbing too can change at any time their schema lol Its the nature of salesforce itself Hence why they provide a Schema as a gRPC call

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

Yea don't get me wrong - they've implemented this horribly

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

Hmmmm

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

Hmmmmmm Ok

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

I think you've put me on the right track

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

I will assume the schema is not allowed to change

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

the goal is to allow users to subscribe to any object in salesforce

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

that emits an event on their event bus

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

Wow

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

It is indeed This is frustratingly easy to do in Python lol They have a java example too And even easier in Javascript

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

Pray and spray is completely acceptable in this case

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

If it fails the user will be told and they have to try again Alright so using dynamic which I am extremely aware is unsafe Can this be done and do you have a recommended library Do i even need Avro? I just want to decode the binary payload

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

I don't care what it actually is at all The user is completely responsible for making sure the fields exist in the payload

remmy_clarke_jrOP•2y ago

https://ambitious.systems/avro-binary-format

Avro Binary Format · Ambitious Systems

Avro specifies two serialization encodings: binary and JSON. Binary encoding is the default and used by most applications as it is smaller…

remmy_clarke_jrOP•2y ago

This looks like what I want

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

Hey Because of you I got it working I don't really know how to thank you enough 😦 public static string DecodeBinaryPayload(string schemaJson, byte[] payload) { var topicSchema = JsonToSchema(schemaJson); using (var stream = new MemoryStream(payload)) { var writer = new BinaryDecoder(stream); var datum_reader = new Avro.Generic.GenericDatumReader<dynamic>(topicSchema, topicSchema); var result = datum_reader.Read(stream, writer); return ""; } } This does everything i need

c#
        public static string DecodeBinaryPayload(string schemaJson, byte[] payload)
        {
            var topicSchema = JsonToSchema(schemaJson);

            using (var stream = new MemoryStream(payload))
            {

                var writer = new BinaryDecoder(stream);

                var datum_reader = new Avro.Generic.GenericDatumReader<dynamic>(topicSchema, topicSchema);

                var result = datum_reader.Read(stream, writer);

                return "";
            }
        }

c#
        public static string DecodeBinaryPayload(string schemaJson, byte[] payload)
        {
            var topicSchema = JsonToSchema(schemaJson);

            using (var stream = new MemoryStream(payload))
            {

                var writer = new BinaryDecoder(stream);

                var datum_reader = new Avro.Generic.GenericDatumReader<dynamic>(topicSchema, topicSchema);

                var result = datum_reader.Read(stream, writer);

                return "";
            }
        }

If anyone needs this in the future

Unknown User•2y ago

Message Not Public

remmy_clarke_jrOP•2y ago

AvroConvert.Avro2Json(payload)
'AvroConvert.Avro2Json(payload)' threw an exception of type 'SolTechnology.Avro.Infrastructure.Exceptions.MissingSchemaException'
    Data: {System.Collections.ListDictionaryInternal}
    HResult: -2146233088
    HelpLink: null
    InnerException: null
    Message: "Provide valid schema for the Avro data"
    Source: "SolTechnology.Avro"
    StackTrace: "   at SolTechnology.Avro.Features.AvroToJson.Decoder.Decode(Stream stream, TypeSchema schema)\r\n   at SolTechnology.Avro.AvroConvert.Avro2Json(Byte[] avro)"
    TargetSite: {System.Object Decode(System.IO.Stream, SolTechnology.Avro.AvroObjectServices.Schemas.Abstract.TypeSchema)}
AvroConvert.Avro2Json(payload, schemaJson)
"{\"ChangeEventHeader\":{\"entityName\":\"Account\",\"recordIds\":[\"001Dn00000HIYAzIAP\"],\"changeType\":\"UPDATE\",\"changeOrigin\":\"com/salesforce/api/soap/57.0;client=SfdcInternalAPI/\",\"transactionKey\":\"0001b494-c858-89b2-60c5-7419cfdceb28\",\"sequenceNumber\":1,\"commitTimestamp\":1676904343000,\"commitNumber\":194782163731,\"commitUser\":\"005Dn000004RijMIAS\",\"nulledFields\":[],\"diffFields\":[],\"changedFields\":[\"0x400002\"]},\"Name\":\"ABAEE222222wwwwwww\",\"Type\":null,\"ParentId\":null,\"BillingAddress\":null,\"ShippingAddress\":null,\"Phone\":null,\"Fax\":null,\"AccountNumber\":null,\"Website\":null,\"Sic\":null,\"Industry\":null,\"AnnualRevenue\":null,\"NumberOfEmployees\":null,\"Ownership\":null,\"TickerSymbol\":null,\"Description\":null,\"Rating\":null,\"Site\":null,\"OwnerId\":null,\"CreatedDate\":null,\"CreatedById\":null,\"LastModifiedDate\":1676904343000,\"LastModifiedById\":null,\"Jigsaw\":null,\"JigsawCompanyId\":null,\"CleanStatus\":null,\"AccountSource\":null,\"DunsNumber\":null,
\"Tradestyle\":null,\"NaicsCode\":null,\"NaicsDesc\":null,\"YearStarted\":null,\"SicDesc\":null,\"DandbCompanyId\":null,\"OperatingHoursId\":null,\"CustomerPriority__c\":null,\"SLA__c\":null,\"Active__c\":null,\"NumberofLocations__c\":null,\"UpsellOpportunity__c\":null,\"SLASerialNumber__c\":null,\"SLAExpirationDate__c\":null,\"Test_Email__c\":null}"

AvroConvert.Avro2Json(payload)
'AvroConvert.Avro2Json(payload)' threw an exception of type 'SolTechnology.Avro.Infrastructure.Exceptions.MissingSchemaException'
    Data: {System.Collections.ListDictionaryInternal}
    HResult: -2146233088
    HelpLink: null
    InnerException: null
    Message: "Provide valid schema for the Avro data"
    Source: "SolTechnology.Avro"
    StackTrace: "   at SolTechnology.Avro.Features.AvroToJson.Decoder.Decode(Stream stream, TypeSchema schema)\r\n   at SolTechnology.Avro.AvroConvert.Avro2Json(Byte[] avro)"
    TargetSite: {System.Object Decode(System.IO.Stream, SolTechnology.Avro.AvroObjectServices.Schemas.Abstract.TypeSchema)}
AvroConvert.Avro2Json(payload, schemaJson)
"{\"ChangeEventHeader\":{\"entityName\":\"Account\",\"recordIds\":[\"001Dn00000HIYAzIAP\"],\"changeType\":\"UPDATE\",\"changeOrigin\":\"com/salesforce/api/soap/57.0;client=SfdcInternalAPI/\",\"transactionKey\":\"0001b494-c858-89b2-60c5-7419cfdceb28\",\"sequenceNumber\":1,\"commitTimestamp\":1676904343000,\"commitNumber\":194782163731,\"commitUser\":\"005Dn000004RijMIAS\",\"nulledFields\":[],\"diffFields\":[],\"changedFields\":[\"0x400002\"]},\"Name\":\"ABAEE222222wwwwwww\",\"Type\":null,\"ParentId\":null,\"BillingAddress\":null,\"ShippingAddress\":null,\"Phone\":null,\"Fax\":null,\"AccountNumber\":null,\"Website\":null,\"Sic\":null,\"Industry\":null,\"AnnualRevenue\":null,\"NumberOfEmployees\":null,\"Ownership\":null,\"TickerSymbol\":null,\"Description\":null,\"Rating\":null,\"Site\":null,\"OwnerId\":null,\"CreatedDate\":null,\"CreatedById\":null,\"LastModifiedDate\":1676904343000,\"LastModifiedById\":null,\"Jigsaw\":null,\"JigsawCompanyId\":null,\"CleanStatus\":null,\"AccountSource\":null,\"DunsNumber\":null,
\"Tradestyle\":null,\"NaicsCode\":null,\"NaicsDesc\":null,\"YearStarted\":null,\"SicDesc\":null,\"DandbCompanyId\":null,\"OperatingHoursId\":null,\"CustomerPriority__c\":null,\"SLA__c\":null,\"Active__c\":null,\"NumberofLocations__c\":null,\"UpsellOpportunity__c\":null,\"SLASerialNumber__c\":null,\"SLAExpirationDate__c\":null,\"Test_Email__c\":null}"

EVEN MORE BETTERR I cant tell you how happy i am now 🙂 ❤️ ❤️

Accord•2y ago

Was this issue resolved? If so, run /close - otherwise I will mark this as stale and this post will be archived until there is new activity.

Gaming

Programming

✅ Looking for assistance in Deserializing Avro Binary stream