Load GLB gradually in Unity
I tried searching but can’t find it. There was a discussion at one point about the ability to have a glb load in unity with the no quick load feature. Will that ever be an option?
84 Replies
Do you "load it slowly" similar to when you load a sketch in Open Brush itself?
Yes . Where it loads in the order the strokes were painted
Can you code at all? I can explain how to do it, but I don't have a ready-to-go script at the moment
Yes I can code c# some
Yes some c#
Cool. By default the exported glb files have timestamp info stored in uv2.
From: https://app.gitbook.com/o/-Mikdl01-6Jga6QTk0rK/s/-Mikdwm98p33AibfiSqN/user-guide/the-open-brush-config-file
Timestamps are a vec3:
x,y = the earliest/latest timestamp in the stroke which contains that vertex.
z = the timestamp for that vertex.
GitBook
GitBook
so InverseLerp(uv2.x, uv2.y, uv2.z) would give you a value between 0 and 1 for each vertex.
And you can identify individual strokes (pretty much) if you group by uv2.x
Awesome thanks!
Hi, was this ever figured out?
I kinda developed a work around, but it wasn't what I had wanted.
Depends what you mean by "figured out". The timestamp data is there, the shaders are easy to modify, I've seen working implementations...
So... I guess, yes? The plugin scripting branch even allows this directly inside Open Brush itself.
I was asking if there was a script developed for this. As for the shaders, I've never had to modify one and wouldn't know where to begin. If there was a tutorial that would be cool. But I guess that would be too much of an ask.
Where did you see the working implementations? I've made some workarounds a while back, but it never quite worked the way I had hope it would.
Is this related to using an Open Brush sketch exported to a separate Unity project? i.e. a glb in conjunction with https://github.com/icosa-foundation/open-brush-toolkit ?
GitHub
GitHub - icosa-foundation/open-brush-toolkit: Scripts and assets th...
Scripts and assets that help you use Open Brush and old Tilt Brush data in your creative projects. - icosa-foundation/open-brush-toolkit
Yes. The Glb was created in Open Brush and loaded into the Unity project. Also, in the Unity Project the toolkit has been imported as well.
Ah - I see the same conversation is also happening in the main channel (outside this thread)
@eerop - jump in this thread for future replies - better if we're all in the same place
I'm talking to Eerop over there as well.
I've seen working implementations...so have you seen a script made by someone that does this? like on github or something @Palaeolak I'll see if I can make a script using the tactic Andy suggested above, here: https://discord.com/channels/783806589991780412/1194152555534618644/1194286056737091625 what was the work around that you made? And why was it not what you wanted
Well, the workaround was I created two separate scripts. The first essentially split a gameobject into a series of smaller objects, i.e. one parent with lots of smaller children objects. The the second, when attached to the parent object, would essentially show the children one by one. I've attached the scripts to this message if you want to trial them.
The issue is, even though it worked, it's very cpu intensive, and it still doesn't give the 'drawn/painted' feel I was hoping for.
thanks for these!
No worries. They were the best I could come up with, but fell short in what I had hoped for.
@Palaeolak making some progress. I'll provide you with a proper script today once it's finished 🔥
Hey, thanks for the update, it's much appreciated. How are you approaching it, are you making use of the time stamps to give the 'stroke by stroke' painting effect? From the vid it look like you're principally working with the alpha cutoff.
Yep, I use the time stamps to choose if a vertex should be visible or not. e.g if the timestamp for a specific vertex is (5000,6000,5500). it means that the stroke started being drawn at 5000 milliseconds into the sketch, and the stroke ended at 6000ms into the sketch, and this vertex was drawn at 5500 ms into the sketch.
the problem with the above tho is that its not working with the original strokes; they are grouped on export. The effect is there but there's no way the strokes couldve been drawn in that order.
So the original strokes need to be acquired in order to animate it properly. And that can be done with timestamps as well: the vertices that are in the same stroke have the same starting timestamp
This is some clever work my friend, thank you for your efforts today. I guess the object could be split into smaller objects based off the timestamps info. Kinda like what I did on the script "SeparateFacesEditor" that I sent over to you. But maybe that would be to memory intensive. Have you thought of a better idea?
didn't manage to get it finished today, I'm not sure if my approach is correct. I'll continue tomorrow though.
There are 2 approaches I was considering: (I went with 1st one)
- (1) each brush Shader has a time parameter (e.g
SketchTime), and then use the timestamp and this time value in the Shader to determine the opacity. Only the timestamp.z value is used, so it's like a global time. This would also mean we don't have to separate it into the original strokes. Have a script that updates the time value in an Update() and uses material.SetFloat("_SketchTime",time) to update the time for each brush material.
- (2) each brush Shader has a Opacity or similar value to determine which vertices should be displayed. Then also have a script that has access to the Mesh of each stroke. This script has an Update() that uses the timestamps data in mesh.uv2 and a time value like Time.time to update the material's Opacity property.
The problem with (2) is that, isn't it really slow to be using material.SetFloat() in an Update() by iterating the vertices? What do you think @andybak?wait nvm, (2) isn't even possible I think. It's not possible to
material.SetFloat individual verticesI woke up and saw this today. What you've done in the video here looks amazing. It actually looks pretty much finished to me, but you're saying that there's more you want to do, what are you thinking?
Seriously, I cannot express my gratitude enough for what you've done. If you need help with anything yourself let me know.
Here's a version you can test and play with. It's not the final version. I still need to fix some bugs and optimize the memory usage. It works for small sketches and meshes, but for me it the editor was hanging when i tried to separate large meshes into strokes. I can also tailor it more to your specific use case, e.g if it's for a game or an art installation
Instructions:
1. import the sdk as a .unitypackage
2. import an Open Brush sketch as a gltf/glb into the Unity project
3. add the gltf/glb into the Unity hierarchy and unpack the prefab completely
4. add the "SketchPlayback" prefab into the Unity hierarchy
5. You need to make sure the meshes have strokeID data before they can be separated:
- select any number of meshes, and go to
Tools/AddStrokeIds
- Note: if you select a lot of meshes at once, this will be slow. I recommend doing this in parts, e.g for 10 meshes at a time.
6. Then, select any number of meshes (ie the Open Brush meshes, as they have timestamp data), and go to Tilt Brush/Labs/Mesh To Strokes.
- Note: if you select a lot of meshes at once, this will be slow. This should be done in parts as well.
This is going to create a new gameobject for each stroke, and they are parented under an empty gameobject that has a "(separated)" suffix.
7. Move these gameobjects to be parented under the "SketchPlayback" prefab
8. Press the "Play" button and make sure the "Play Animation" property of SketchPlayback is true.
https://youtu.be/sJetGM9AHL4the relevant scripts are:
- Assets/TiltBrush/Scripts/SketchPlayback.cs (this contains the logic for revealing the strokes one at a time)
- Assets/TiltBrush/Scripts/Editor/EditorUtils.cs (this contains the logic for the
Tilt Brush/Labs/Mesh To Strokes tool)
- Assets/PreprocessUVs/Editor/AddStrokeIds.cs (this contains the logic for the Tools/AddStrokeIds tool)
If you have any questions about the implementation let me knowHey man, this looks super exciting, thank you so much. I'm doing this for a game/art installation as part of a doctorate in art and archaeology, and I began using gaming as a way to communicate my thesis. In a nutshell, I explore nomadism within the Palaeolithic era, so I utilise the walking sim genre quite a bit. For the longest of time I've been wanting to make a game where the world materialises around the player as they walk about to help explain my research. So what you've done will help with this enormously.
I've quickly loaded it into a test project, I'll properly play about with it tomorrow when I'll have a full day to explore. I'm sure I'll have questions. Once again, thank you. This is some of the gaming stuff I've made: https://palaeolak.itch.io/ most of it's done in open brush.
itch.io
Palaeolak
did you see my DM?
that's awesome 🔥
have you worked with the animation system in Unity? one idea i got was that animationclips could be created for revealing a sketch. Might be more convenient to work with them.
I've done some work with animation in Unity, but not a great deal. What makes you think it would be more convenient? I've always thought the animation system in Unity was quite cumbersome.
I'm a big fan of the timeline for this kind of thing. It bypasses the weird state machine stuff in the Unity Animator (which you don't need for linear animations)
It's probably the simplest way to "make some value change over time" without messing around with tween libraries, coroutines etc.
I usually have one timeline per scene. I've rarely done something that needed anything more complex (multiple timelines/directors etc)
So a single root object called "Timeline" with a Director component on it is usually enough.
It would integrate nicely into Unity's workflows, because
Animator is how animations are usually implemented in Unity. But this particular animation isn't very complex, so I think other approaches like coroutines, tweening, timeline, etc can work just as well.
btw, what platforms are the games you make for?
i noticed that at least The Garden had a Windows download, but what about e.g mobile?Just for windows atm, never done anything for mobile or mac.
And this would be waaaaaaaaay beyond my capabilities at the moment. I do however understand what you're saying about it essentially being this "on/off" type thing which would lend itself well to tweening. I just wouldn't know how to implement it.
ok no worries! So how much programming experience do you have?
It sounds like the logic that the game needs, is that when the player is close enough to an object (sketch from Open Brush), the object must reveal itself.
And maybe also have multiple objects able to reveal themselves at the same time, like e.g 10 trees when the player walks into a forest.
as you said:
For the longest of time I've been wanting to make a game where the world materialises around the player as they walk about to help explain my research.
notee that Animator and Animation are two different components.
i get them mixed up but I think Animator is the complex one with behaviour trees. You probably don't need that.
Well, I can work things out from the knowledge that I have and come up with some simple scripts on that basis. The issue is I don't have in depth knowledge of this subject. So the smarter solutions often allude me.
Absolutely correct. And also dissapear gradually when the play walks away.
It's been so long since I looked at each, I can't remember which one is which.
yeah Animation is the more basic one: https://docs.unity3d.com/6000.0/Documentation/ScriptReference/Animation.html
There was a really nice low-code/no-code XR framework for Unity. You would set up triggers in the editor and link them to actions. I can't remember the name of it now.
This also looks interesting but it's still in development: https://unitydust.com/
Dust for Unity - makes gamedev easier and faster
Dust for Unity
Dust is 80+ simple but powerful components to Unity. Add animations, sequence of actions, fields, catch events, and generate unique scenes - all with Dust components. Easy to use. No coding required!
Interesting, I'll give it a look.
remembered the one i was trying to remember: https://www.mindport.co/vr-builder
VR Builder: Free Unity plug-in for VR development I MindPort
With VR Builder you can create better immersive VR experiences faster in Unity and accelerate the rollout of your VR development. Download now from the Unity Asset Store!
You are a goddam genius, mate. Your code was ultimately so efficient that it didn't need steps 6-7. The assigning of stroke IDs wasn't too memory intensive, but the mesh to stroke steps were taking forever. I tried to play about with it, but couldn't get it to work with complicated objects as it would just take too long. So, I tried to be cheeky and didn't bother with those steps... and it worked amazing!
Just check the vid, the object I used is somewhat complexed, so this initial test is very very promising.
Nice!
yep,
Mesh to Strokes isn't 100% necessary. It's possible to work with the strokes that are there at the time of import. (ie these are the strokes that have been grouped by Brush type on export) For some sketches, this style can look good.
I optimized the Mesh to Strokes to use Unity's Jobs system today. It should be a lot faster now, so you can now also experiment with how it looks when the sketch is drawn in the original order it was drawn in Open Brush. Let me know which one you prefer and if there's any issues!
btw: Make sure to re-run AddStrokeIds for the strokes, as it got a little update as well.
Here's the new version:
https://drive.google.com/file/d/1V5e1EMa2o1Dji-Xdin564TvoNjii8AQ7/view?usp=sharingbtw, do you have any games or artwork or videos in mind that use this similar concept? i've definitely seen this materializing effect somewhere. One thing that came to my mind is this animation from the last harry potter film: https://youtu.be/nD5UWd_4kyA?si=I_t0XiYPqUtHZc7v&t=39
Harry Potter
YouTube
The Tale of the Three Brothers | Harry Potter and the Deathly Hallo...
Care for a bedtime story? Discover where the Elder Wand, the Resurrection Stone, and the Cloak of Invisibility came from as Hermione tells The Tale of the Three Brothers, a popular wizarding world children's story that explains the origins of the Deathly Hallows.
The Harry Potter: Complete 8-Film Collection is available on Blu-ray and Digital n...
I'd forgotten about the animation, it's such a stunning little piece within the larger film. But yes, I have some examples: https://www.youtube.com/watch?v=dY46_hkY_-I this is a short animation made in the app Quill called 'Dear Angelica', there's also a game called 'Shape of The World' which utilises this effect as well https://www.youtube.com/watch?v=SL15GaUK7a8.
Meta Quest
YouTube
Dear Angelica | Oculus Go
From Emmy Award winning Oculus Story Studio comes Dear Angelica, a journey through the magical and dreamlike ways we remember our loved ones. Entirely painted by hand inside of VR, Dear Angelica plays out in a series of memories that unfold around you. An immersive, illustrative short story starring Geena Davis and Mae Whitman.
Shape of the World
YouTube
Shape of the World - Launch Trailer
Steam: https://store.steampowered.com/app/642610/Shape_of_the_World/
Available June 5 on PC and PlayStation 4 and June 6 on Switch and Xbox One. A psychedelic, relaxing and interactive escape to get yourself pleasantly lost.
Shape of the World is a serene first-person explorer unfolding in a world that grows around you. Journey through a psych...
I'll check this out tomorrow morning, I'll set it up in a new project to check out the differences it makes. I'll send some tests back once I have reasonably swish looking mockup ready.
i just discovered a memory bug in the code, but it's now fixed: https://github.com/eeropomell/open-brush-toolkit/releases/tag/jobs-2
download the fixed version from there
also, run it for max 3 meshes at a time for now. Unfortunately there's no limits for how much memory it can allocate, so if the tool is used for a lot of meshes at once, it'll just allocate as much memory as is needed and cause Unity and possibly your whole PC to crash
btw, how much RAM do you have?
Cheers man, I shall have to check out tomorrow. I had an office day today so I didn't get chance to try out the code, but tomorrow I shall do it for sure.
I've got a pretty good desktop. It's not out of this world in terms of its spec, but it can handle a lot. I think its 32gb
ok! What about CPU? I have 16gb ram and Intel(R) Core(TM) i5-9600K CPU @ 3.70GHz. Just asking so I can think about the performance of the tool from your perspective.
on windows 10 you can check it in task manager > performance > CPU
Intel Core i5 10400F 4.30 GHz 16GB RAM 500GB SSD RTX 3060
It's been about 2 years since I checked up on my specs. 32gb was a little off.
added a couple fixes and improvements to the Mesh To Strokes today. There was a pretty bad issue with a lot of unnecessary mesh data being duplicated into the new meshes generated by "Mesh to strokes" and it was causing the scene file to become insanely large.
make sure to use this version now:
https://github.com/eeropomell/open-brush-toolkit/releases/tag/jobs-3
Also: make sure to install the "Burst" package from the Unity Package Manager.
there's also at least 2 brushes not working properly right now: electricity and smoke. When a mesh using either of those brushes is split into strokes, the scale gets messed up.
I'm going to load that into Unity later this evening. I still haven't looked at V2, so I'll just go for this one (V3). Thank you for this, man. I'll have something to show you asap.
Yeah don't try V2 there's a bug that V3 fixes 😄 Also no rush! 🙏
Hey man, I've just got around to installing v3 into a fresh project. Do I need to install anything along with it? Except for Burst.
I got a few console errors I can't clear, and I'm not getting the editor updates (such as 'addstrokeids' and 'meshtostrokes' options along the toolbar).
I'm probably missing something obvious.
what are the console errors saying? Is it something with newtonsoft.json?
There mostly related to Reaktion
could you copy paste an example here?
These are the errors.
ok thanks 👍 1 sec, i'll try to install it in a fresh project
No problem man, if you have a sample project that you worked on can you pop it onto google drives and share? I'm likely setting it up wrong.
This is what I was meant by the editor not updating, these options do not appear.
If you've got red console errors then it will break everything. So wait until that's fixed before checking other stuff.
the problem seems to be that the .asmdef files aren't in the .unitypackage. 1 sec I'll share the updated .unitypackage
also, make sure to install "Mathematics" package. I forgot to mention it. It's needed for Burst
try this one: https://drive.google.com/file/d/19wz40NIgxJy8g9D8Hhq0bDEBEqeSg3V5/view?usp=sharing
if you go to the Assets/ folder in the project and delete "TiltBrush", "ThirdParty", and "Editor" you should be able to install the new .unitypackage straight into the same project without having to make a new fresh project
these warnings are okay:
That seems to have updated the editor and removed the build errors. Thank you. I'll play around with it now and see what I can get going.
It's so obvious I'm the type who would delete the project and start again, haha. So this is good to know.
also if you get errors about newtonsoft.json or something called
Library/PackageCache/[email protected]/Runtime/Newtonsoft.Json.dll e.g not being found, that's expected
i would've done that too in the past, but I just tried it and it worked 😄do the instructions here still apply to your packages? https://github.com/icosa-foundation/open-brush-toolkit/releases/tag/v24.0.0
GitHub
Release Unity SDK: v24.0.0 · icosa-foundation/open-brush-toolkit
Important
Download and install open-brush-toolkit-UnitySDK-v24.0.0.unitypackage from the link below.
More information in our docs: https://docs.openbrush.app/user-guide/open-brush-unity-sdk
The min...
I ran into an issue yesterday where after installing I think Burst, it also installed
com.unity.nuget.newtonsoft-json Which is Unity's own newtonsoft.json package. It can't co-exist with the one that is used by the SDK: https://github.com/ianmacgillivray/Json-NET-for-Unity
- I tried using only com.unity.nuget.newtonsoft-json by deleting the ThirdParty/Json-NET-for-Unity, but it broke the glbimporter.
Then I tried deleting Library/PackageCache/[email protected]/Runtime/Newtonsoft.Json.dll, and the project continued workingyou need to delete the manifest entry - packages can get reinstalled automatically
I think we need to give the advice specific to unity versions
if i recall - it was complex and related to minor unity versions. so instead of getting into that my docs just said:
try this and if you get this error try our packagebut that doesn't handle the situation where people install something after getting it working and it breaks again ideally - we'd just use the unity newtonsoft - but i think that is locked to specific unity versions? i can't remember for sure
yeah I think in the end we should use the unity newtonsoft, as unity is maintaining it. But at this very moment I wanna first make sure the feature works well for one guy's use case, @Palaeolak, before making a PR that can be merged into main
It's much appreciated, thank you
no problem! it's great learning experience for me as well
if you have any questions about the logic or need help with setting it up let me know. I think it could be helpful if it's a part of the tool.
I definitely think it should be part of the tool. It's such an integral part of the Open Brush Experience, and it's an incredible animation asset. Last night I was able to get it to work so fluidly with some pretty large complicated objects, and I was playing around with some different ways to implement it. Tomorrow I should have close to a full day to play around, plus the weekend is looking free to experiment. I'll post some videos here tomorrow so you can see it in action.
sounds good!
Do you say "(pretty much)" because of the precision of the timestamps, e.g 2 strokes may have the same timestamp.x or timestamp.y?
when did i say that?
at the top of this thread
I'm doing a writeup of how I implemented the sketch reveal effect for the SDK:
https://eerop-6a7f4.web.app/post/Sketch%20Reveal%20effect%20for%20Open%20Brush's%20Unity%20SDK (current version is a draft)
looks like this rn: