❔ [Unity-ish/Newtonsoft] Efficient ways to deserialize a large amount of json data.
Update: This problem appears to be a non-issue in builds.
I am going to figure out what the heck the editor is doing here vs builds. But not right now because work.
Problem: It seems that anytime I attempt to deserialize data using the Newtonsoft tools, I'll incur a pretty nasty spike on the CPU and hitch even in an empty project. The API/Endpoints I'm stuck with do not have any data-chunking afaik, and also routinely return data in one large array of objects. In production, I'll be expected to handle up to ~15000 objects from an endpoint.
Code can be viewed here https://pastebin.com/h6qycPmv
Json can be pulled directly from this endpoint: https://jsonplaceholder.typicode.com/comments
Methods 1 :
Coroutine (Picture Labeled A)
Unity Coroutine, reaching out to grab the data, then deserializing it to a defined class:
Note: With this, I was noticing a pretty gnarly spike. Figured I wasn't doing something correctly or was simply abusing deserialization and not using it correctly. I started researching other ways of how to consume JSON data from an endpoint.
Methods 2:
Two variations (pictures B and C). Picture B is the same JSON as Picture A, while Picture C is a trimmed down version.
Goal here was to deserialize the json one element at a time using a Filestream. I had a hunch to add a manual delay in there to add a gap between each deserialization. What's most curious to me is that even with this, I'm incurring that same general cpu hit. Which leads me to believe I'm implementing this stuff totally incorrectly. <-- Likely as hell. Is there not some relatively efficient way to deserialize a mass of incoming objects? Or at least process json from an endpoint without hitching? Edit: also aware of some bad practices here to include the async void stuff, but trying to triage 🙌
Goal here was to deserialize the json one element at a time using a Filestream. I had a hunch to add a manual delay in there to add a gap between each deserialization. What's most curious to me is that even with this, I'm incurring that same general cpu hit. Which leads me to believe I'm implementing this stuff totally incorrectly. <-- Likely as hell. Is there not some relatively efficient way to deserialize a mass of incoming objects? Or at least process json from an endpoint without hitching? Edit: also aware of some bad practices here to include the async void stuff, but trying to triage 🙌
Pastebin
[Serializable]public class RootCommentData{ public int postId; ...
Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
25 Replies
Coroutines run on main thread, Tasks also run on main thread in Unity by default.
Since deserialization is CPU-bound work both won't help in this case.
Indeed. I am aware tasks != ez-threading, suppose I'm using the pattern and testing other approaches to more match what I'm seeing other folks do.
In this case, I was considering options on how to reduce the huge burden on the main thread by spreading the deserializtion out over time. And even then I don't know if that's a production solution, but feels like it's a step in the right direction?
Recently they added some utils methods to run on background thread https://docs.unity3d.com/2023.1/Documentation/ScriptReference/Awaitable.BackgroundThreadAsync.html
Or look into UniTask if your version doesn't support it
Funny enough, I'll incur the same hit with UniTask as the newtonsoft stuff is still going to be main thread bound.
How so? Are you doing
await UniTask.SwitchToThreadPool(); properly?
Newtonsoft won't be thread aware🤔 need to recode some stuff I nuked. And apologies for thinkin I was bound there. I've been on stack overflow firehosing my brain with things I just kinda take for granted.
Regardless, appreciate the quick response. Will try UniTask again and see what I find.
Good luck!
Simply wrapping the serialization part of the code or the function itself still makes the cpu chug pretty hard (with or without the delay)
Thanks again, I probably need to just research some more and read up on the docs of all the stuff I'm using
You can also switch context to a threadpool thread without UniTask, but using UniTask is probably making your life way easier
Async await tasks are very different in unity, then from what you expect in a standard .NET application.
I might be missing something obvious, but newtonsoft.json is slow, and System.Text.Json is much much better
well you are dealing with unity here
ah thought so
$0.20 cents per install and can't even use System.Text.Json
Hahaha, indeed
System.Text.Json is almost non existant there atm
which is sad, but unity also still runs an old mono version
and only now with unity 2021+ you can have netstandard2.1 features
Unfortunately we're kind of bound right now with something already out ~version 2021. Game's already released so any major updates to get the new stuff would be a little more lift.
before you were locked to NET Framework 4.X
I am also finding that using Unity's built in conversion utility is wildly faster than Newtonsoft. No shade to the creators.
Honestly, unitask, which you might already have, is the best option
Mhm
never fiddled with it, since it lacks prettymuch all features of other serialisers
but if it works for your case, and is even faster, good to know lol
Yeah, I need to rope it into the spaghetti I've created up there but figure I'd give it a shot. And once again, big ups to everyone who's helped in this thread. Would like to explore other engines and custom stuff in the future, but currently I've gotta support my leads 🙂
mhm. our company is sitting on a lot of UGC which is built for unity only, i know the pain... PS: godot is a nice engine :p
heheh I'll be 👀. plus it'd be really cool to get involved in the open-source community.
Was this issue resolved? If so, run
/close - otherwise I will mark this as stale and this post will be archived until there is new activity.