DASH Content browser for large projects (with huge ammounts of folders with assets)
Hey Dash Team, I'm happy working with Dash, but I was never able tu use the Dash content browser with my big large projects so here we go again 😁
I noticed, in small projects the content browser will compute just fine (because I have 5 folders with 200 assets each). But in large projects, when I try to scan 1 folder, it will somehow try to compute or analize all the folders (lets say 1000 folders with around 200 assets each) first, like, Dash thinks I want to compute the entire 1000 folders, leading to crashes and bootlenecks.
Maybe, there should be a way to let the content browser analize and compute 1 folder actually instead go to the 1000 folders and the 200 assets in each folder, and then, scan the folder I want, making the compute process stable, reliable and fast.
I think Dash gets all the folders and assets in each compute of a project, instead the folder we want, that why the compute process is actually slow.
My only workaround was to split my huge project in very small projects, but takes a lot of time and doesn't feel right.
15 Replies
Example, when I try to scan a pack of rocks, the debug tool (enabled with python) shows dash is scanning the folders of the entire project, not only the folder with rocks.
Hey!
Do you mean that it tries to compute everything (all folders under the Content folder) even if you select a subfolder an press compute inside of it?
Or do you mean by "scanning" that when you open Dash CB and when it loads all your folders, it is that process that is problematic?🤔
Yes, hmm, not compute exactly (in Dash terms) but an initial process (inside "starting process") that analyzes or process all the folders and the assets when I want to compute a single folder, (or a sub-folder with, example, only 1 asset), is like Dash is making a registry of all the assets, the entire content folder (the 1000 folders with subfolders and assets) in the "starting process" Dash seems is reading the full content folder and subfolders before compute the folder and the asset I want.
Opening Dash CB is fine, the issue is when I try to compute a folder inside the content folder or a huge project.
As I said, it works perfectly fine in small and medium projects, because dash process all the folders, but the folders are a few, meanwhile in really big projects, Dash CB will likely panic due the big amount of assets. That's what I think since I have Dash 6 months now.
ahaa, got it, thanks for the details! Sounds like a bug but no idea really.
the guy behind this system is currently asleep as he is based in Vietnam, but he can get back to you tomorrow regarding this🙌
cc @Costantino
In the meantime, are you able to record this happening? And having the UE output log open thoughout your video? This will certainly be Costantinos first question
and also, try what Costantino suggested here?
https://discord.com/channels/826908885024636989/1299868857602674699/1301138054282481675
Yes, I sent months ago a video and logs of this issue, but wasn't resolved because I had no idea what was happening, but now since I'm having more experience, I see what is happening
ahaa sorry, let me try to find that convo
Sounds good, thank you, this issue has been part of Dash since the the content browser was present 😅 If splitting my folders in small projects works, I know there is a solution to make dash think each folder is his own project, maybe, making the compute process even way faster
aha found it and now I remember the convo as well, you me and Costantino had a DM group chat https://discord.com/channels/@me/1244560058440744981/1257168177402613801
Soo, the change we did with the OpenAI tagging did no difference I take it?
No, because the issue happens before the tagging, Dash reads all the content folder, every folder and asset, before compute the selected folder
I know this because scan_data.json has all the assets of the project, not the folder I want, but the full list of assets, folder and subfolders (667331 lines of Json) , so I think, dash, works with the entire content folder, instead the folder selected with the single asset.
got it got it🙌
Let's see what @Costantino says tomorrow 🙌
Sounds good, thanks Daniel.
Hello @Jhosep Chevarria, although the computation of assets only happens in chunks for the selected folders, a stamp of the whole project is currently taken.
With huge projects, that's obviously a limit like you have experienced.
We have some optimizations going on on multiple areas of the content library, with the project library being one of them.
We will likely roll them out with the 1.9 release, so for now I can only suggest to wait for the next update.
Yes, I see the computation works perfectly, but the process before it, is what it makes the issue I have, but that's great news, thanks Constantine, can't wait for 1.9
It make perfect sense, we will make sure a faster and smarter project library makes it for 1.9. I wanted to push for some changes on 1.8.5, but the fab integration really took most of our resources on the content library side.
I really appreciate it, with a faster project library, dash will be superior and amazing.