Dynamically generating C# classes for deserializing data
I'm building a little service for reading large Parquet datasets into 'row' object collections for LINQ querying. So far I have been manually creating my 'row' classes, like one dataset is for junior school maths word problems, and I deserialize the Parquet rows into a collection of
WordProblem objects.
Now I want to generalise the service and have it read a ParquetSchema from the dataset file, generate a class on the fly and deserialize into that. Second best, or maybe a better design, would be to statically generate class files. I'm using C#12 and .NET 8 with ParquetDotNet.
Claude has suggested some CLI tools and packages for static generation, and these libs I have no idea about, hence this question.
- Language Ext - This library allows defining data types based on a schema at runtime without pre-generated classes using LINQ projections.
- Siros - Provides dynamic and typed projection of data sources like Parquet into .NET objects.
- Misc.Extensions - A lightweight library with dynamic schema projection capabilities.
Then there are Roslyn code generators, which I should have learnt about yonks ago but haven't. Which is the current best practice way of doing this?11 Replies
generate a class on the flythis isnt a thing you can do as such you can generate it at compile time, with a source generator
By on-the-fly, I mean between when I run my program (and it is jit compiled) and when it reads the dataset
but at runtime, your only option would be to generate source as text, compile it, load that assembly at runtime, access the type via reflection... it would be a nightmare
yeah so at runtime then?
It wouldn't be that bad a nightmare
¯\_(ツ)_/¯
But what about e.g.
Language Ext, which gives C# functional features and code-generation, https://github.com/louthy/language-ext/wiki/Code-generation?GitHub
Code generation
C# functional language extensions - a base class library for functional programming - louthy/language-ext
these are all at compile time
I'll accept compile time solutions then, it seems like runtime is the province of ES/TS/Python, and not C# at all yet. Unless this is a use case for
dynamic? 😂I mean, it will be god awful to work with
if you truly want this to be runtime, your best bet is proably to do something along the lines of
JObject
but that removes all benefits of using a strongly typed language(you can dynamically generate types at runtime without producing actual text source but it's gross and painful)
if you know what your data looks like ahead of time compile time code generation would be a better option
Maybe that's a cue for a dynamic language like TypeScript then. Or Python, but I've grown to love ES the last year or two, so much easier.
I would like to be able to recompile a text based module with a standard interface that my service uses for deserialized schema based classes. Where should I be looking next?