✅ System.Text.Json serialization problem
hello, I am using System.Text.Json.
I have a struct that has 2 properties: a bool IsDefined and a T value. IsDefined gets set to true when value gets set. This is a container for properties that don't necessarily need to be set, but if they are set, they will get included in the serialization.
The problem is that in the Write method, STJ automatically writes out the property name, which results in this Json if only the name property is set:
{"name":"lol","parent_id":"position":}
.
The STJ write method:
How can I get around this?121 Replies
this sadly doesn't work for my use case because it works like this:
- value set
- value is a normal value --> serialize
- value null / default --> serialize
- value not set --> don't serialize
I want to check whether .IsDefined is true. how would I be able to do that?
https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/converters-how-to?pivots=dotnet-7-0 you could write a own jsonconverter for that class and use it with the system.text.json jsonserializer, then you could handle yourself how you want it to behave
I have that right now
but if it gets into the Write method the property name will automatically be written out.
if only name is set and the other ones arent, they will still get into the Write method and will automatically write out the parent_id: for example which i dont want. can i change this behaviour somehow?
do you also tell your jsonserializer to use that converter?
yes
Shouldn't you be just running the code in the default case if
value.IsDefined
?
Idk my brain hurts from trying to figure it out 😆no because the api wants the ulong and long values to be in string format for whatever reason
but thats not the problem
No yeah, you are right it is not the problem
the problem is that the property name gets automatically written out and im not sure how to prevent that
Isn't
Optional<T>
supported by default? Have you tried that?this sadly doesn't work for my use case because it works like this:
- value set
- value is a normal value --> serialize
- value null / default --> serialize
- value not set --> don't serialize
So nullable isnt usable here
Okay, so apparently this is not supported by
System.Text.Json
. In .NET 6 they added json ignore conditions but there is only predefined conditions (null and default value) you can't write your own custom condition as the creation of new properties seems to be handled internally...So im out of luck? They didnt add customizable conditions in .net 7 or something?
Don't think so I think it may be planned for .NET 8 I'll continue to do some more digging
Thank you very much for your efforts, I really appreciate it! I couldnt find anything about it when i searched for it. Should I open an issue or something there?
There are a few issues about it, but I lost them somewhere :/ It is already known trying to figure out if it is planned or the idea is abandoned now
I hope its not abandoned
GitHub
Developers can customize the JSON serialization contracts of their ...
Background and Motivation Today, System.Text.Json provides two primary mechanisms for customizing serialization on the type level: Customizing the contract via attribute annotations. This is genera...
The issue is closed and marked as resolved?
"unfortunately this scenario is out of scope in this release although we've had long discussion on supporting this - we've decided it's better to support less in this release than supporting everything at once in 2 releases (having said that I'd really wish we could add support for that but at the same time this feature is already large as is and our team has limited resources)."
I really hope I'm not miss understanding this, but I do think this issue is for that xD
I got the same idea as well
These JSON contract resolvers
The next release will be in .net 8?
Honestly, I'm still confused if this would even get more work done in .Net 8
Here is a link for planned things to it but Its so obfuscated I find it hard to read XD
GitHub
Planned System.Text.Json contract customization improvements for .N...
Gathering some of the contract customization features we're planning on working for .NET 8. NB this list is subject to change. Object initialization #73219 #78556 #71944 Contract configurat...
What am i supposed to do now? Im about 90% through with my migration from newtonsoft to STJ. I dont want to throw everything away now because 1 thing isnt supported
:/ I wouldn't know, there must be some way to circumvent this though
I'll try a few things
It seems like that has something to do with this issue
Please let me know if you find something, I tried a lot of things and they didnt work
STJ exposes a CanConvert method but thats useless for my usecase since it only allows type checking and not value checking
How did you register the converter if I may ask?
I mean I got something working but the issue is I need to specify all the concrete types I need to use the converter on 🤔
I'll figure it out
Above the property add the attribute [JsonConverter(typeof(Converter)]
Eggh, its ugly though
¯\_(ツ)_/¯
You figured something out?
Yes, whether to use
JsonIgnoreCondition.WhenWritingDefault
option based on the IsDefined propertyWell that wouldnt be what i need. The library is for discord and discord lets you send values that are null or default values to reset a field
So this would destroy that functionality
I thought about that as well
No, basically it works when IsDefined is true
Let me double check real quick I may be getting ahead of myself 😄
How
Okay double check
The only case where it shouldnt serialize a property is when you havent touched it
I'm using
Optional<T>
which basically just has a constructor for value
and if you use that constructor it sets HasValue to true
I may be just dumb and doing something wrongIm not at my pc right now
But i tested with unset values, set values to default and set values to null. worked fine.
The issue is that I have to pass the concrete type
for each type i use in application
Wait so setting a property to default/null still serializes it?
And only doesnt serialize when the property is actually unset?
Yes, the property is added to the json when the value is set to default/null or any value in that matter
And the property is not added to the json when the property is not set
That would be amazing if it actually worked
- value set
- value is a normal value --> serialize
- value null / default --> serialize
- value not set --> don't serialize
fullfills this
Wait ill get on my pc
But I gotta find a way to remove having to register all the types, its bugging me
no way
You see, you have to register a converter for literally all types used inside a
Optional<>
it actually works
i didnt think the default attribute worked because the description says it ignores the property when writing null
which is insanely misleading
Well that is actually not what you want but this is a hacky way to activate it when you need it
yeah ill have to add the attribute to every property
Because you do want null values to be written
Nah man I'll figure this out
Just wait 😆
Might take me 4 days
xD
you dont have to brother, thank you so much I would have legit never thought about trying it because of the description
I feel so happy right now
you are a good guy
btw i agree with ur about me
The issue is I'd probably have to use reflection and I hate having to use reflection in my
who are you
A programmer with 0 work experience that struggles to improve himself due to the fact that he has no discipline 😆
how old are you
23 I believe, I don't count my birthdays
so you have a job in a different field?
No, I just don't have the discipline to focus on programming for long periods to actually finish projects and have material to show to employers
Like I have been programming as a hobby for many years but I advance so little because I spend so little time coding during the week.
But I just don't have the discipline to sit down and code even when I know that, this is what I need to do to improve myself and reach the goals that I clearly want.
well its good that you see your problem. What I always do is not care about the discipline but just sit down and actually do something, the first part of getting to do something is always the hardest.
I wish you luck and maybe you should try that as well
I just end up staring at the computer screen blankly and eventually give up to the demons and start
You gotta fight that man
@Sh1be feast your eyes on this garbage
reflection
🤮
a factory???
doesnt look very simple and fun
This is how you should do it to to achieve a generic converter based on the msdn docs on how to write a custom converter. There is probably performance issues with it though as with everything else using reflection. I also need to reuse the options instead of creating a new instance all the time...
so its not really worth it huh?
The options instance contains cache of metadata needed for serialization, since I'm now re-using a single options instance for all
Optional<>
types I think it should be good. It is important to note that the cache size will grow with the amount of different(distinct) types you are using when serializing. For example Optional<int>
is one, Optional<string>
is two, etc, having multiple of one type shouldn't add to the cache size. Make sure you not re-creating too many JsonSerializerOptions
as the serializer undergoes a warm up phase during the first serialization of each type.
There are probably more ways of optimizing it but honestly I wouldn't know how to.
Here is a quick benchmark on my shitty machine. As you can see it takes ~2.7 seconds
with Optional vs ~1.1 seconds
without Optional when serializing 100000 entities with same data. Now with the x2.36
slow down you can decide if you find that worth it or not
I should benchmark with Optionals but without the factory but I'm too tired rn, will do it first thing tomorrow cause I'm curiousYou have to look into a resolver and not a converter
.NET Blog
Announcing .NET 7 Preview 6
.NET 7 Preview 6 is now available with improvements to type converters, JSON contract customization, System.Formats.Tar API updates, constraints to .NET template authoring, and performance enhancements in the CodeGen area.
good morning, I fell asleep
https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/ignore-properties?pivots=dotnet-7-0 in the docs it shows different functionality
How to ignore properties with System.Text.Json
Learn how to ignore properties when serializing with System.Text.Json in .NET.
is this a bug?
the Optional wrapper was just as a bypass for the problem. If this attribute works though, the whole wrapper can be thrown away and the default converter can be used
oh i didnt think those would be available already, but if this magic attribute works the way i need it to, then i dont need it
weird that the example msdn talks about is exactly what isnt supposed to be happening
Ah damn! How the hell could I miss this...
how did i miss that as well
Its freaking difficult to figure out though...
Hold on, how are you going to know whether the value is undefined without the optional wrapper?
the JsonIgnore condition somehow takes care of it. try it
i have no idea how or why
what... ok I don't think what you are trying to do now and what you had before in Newtonsoft.Json is the same x)
it produces the desired result though
- value set
- value is a normal value --> serialize
- value null / default --> serialize
- value not set --> don't serialize
Newtonsoft.Json's JsonIgnore always ignores the property by default though? 🤔
Do you have a contract resolver or something for Newtonsoft.Json?
Newtonsoft has these ShouldSerialize methods
Well how'd you do those?
thats what was used before
https://www.newtonsoft.com/json/help/html/conditionalproperties.htm
Yes, I get you but what was your logic inside those methods to tell if the value has been undefined or not?
thats where the Ignorable / Optional wrapper was used
Ah, yes. okay that makes sense
but now that the attribute somehow tracks whether the value has actually been set, the wrapper isnt needed anymore
no idea how it does that
Thats why I asked this cuz you said the optional wrapper can be thrown away, which in reality it can't be as you can't then tell if the value is undefined or not
Wait what the attribute tracks if it has been set?
Where'd you read that?
yeah i tried it without yesterday and it worked
ill fix some errors rn and then show you
I think you are wrong on that assumption 🤔
ugh big projects are annoying
you change one thing and everything breaks
Thats why you got source control
@Pendramon okay turns out it didnt work
not sure why it worked yesterday
im stupid, 2hrs of work for nothing lol
I think it worked for null/default value probably and you just didn't check the case of it not being set
yeah thats the test, doesnt work without the Ignorable
lesson learned
@Pendramon@Pendramongo and finish a project
No, I'm busy getting distracted on discord
Finish being busy with that and start being busy with coding
I wasted 2 hours now u gotta put 2 hours to good use
I wasted 4 hours tryna cast
Optional<of any type>
to Optional<object>
cuz I'm a Lmao
Don't think its possible x)
Try harder
Surprised you put in the 4 hours effort into it though
I get bored after working on the same project for some time, starting new projects that I'll never finish is fun
Lmao
I hope we will finish this project
Else many hours will have been for nothing
I mean technically we achieved our goal
But if its performant enough is another
Its certainly faster than newtonsoft
The barebones implementation i have
The one you have is slower ofc
Why you say that? did you benchmark it?
ok was bout to say I'd be surprised
No i didnt benchmark it
But newtonsoft is still insanely slower
I read a benchmark once
It is slower indeed
Just benchmarked it and the factory I wrote above actually improves the speed by 0.27% instead of the no factory version
But implementing conditional serialization could end up being even faster 🤔 IF it actually can be used for this use case
this contract resolver could be worth a look
perhaps that makes it faster
and clearer, as right now the WhenWritingDefault attribute is kind of a magic box
When I think about it I actually don't think it will be faster
WhenWritingDefault
is basically what we needed the whole time
The only thing that we just needed to do is check if it HasValue
before serializing
Because every unassigned variable is set to its defaultYeah but the question arises why does it still pass if its set to be null/default by the user
That goes against its description and example
Ohhh I see where you are confused
Well crap... found a bug 😄
hm?
I think it never even worked in the first place x)
I don't think I broke anything but I can't reproduce the correct results
you should start a new project
/// <summary>
/// If the value is the default, the property is ignored during serialization.
/// This is applied to both reference and value-type properties and fields.
/// </summary>
WhenWritingDefault = 2,
now this is a better explanation
I understand why this works now
But exactly that is the problem
It does exactly that. hence why it DOESN'T work
- value set
- value is a normal value --> serialize
- value null / default --> serialize
- value not set --> don't serialize
Does not full fill this ^
My tests were just bad...
it works with my wrapper class
because of the IsDefined property
that gets set to true when the value is being set
so it isnt in default state anymore even if the value is default
The only way I'm able to construct an instance of this Optional class is when I pass a value inside the constructor. Hence should work
Are you sure you are not just testing it bad like I was? x)
Try adding a default value and see if it is written
To a non optional property
look here
This is now more like Required instead of Optional
oh i dont have a non optional property
sorry
Was this issue resolved? If so, run
/close
- otherwise I will mark this as stale and this post will be archived until there is new activity.