Efficient reactive rebuild of large derived object
A question about how best to use the reactive primitives for (re)building derived state. I don't think a simple
createMemo will do the job well here.
I have a store with a large number of 'blocks' (just objects), stored as a large object with id/block pairs, e.g. store.blocks[someID].name
I want to build a grouped and sorted "index" of the blocks, e.g an array of 'sections' that groups the blocks by type e.g.
My issue is that a naive implementation using createMemo would end up rebuilding the entire thing every time a change is made to a .type or a .name, or when a block is created/deleted.
I am thinking I can probably do something with multiple dependent memos, but before getting into it I thought I'd ask what others have done. Seems like a not uncommon situation.
Thanks!17 Replies
You might look into using
mapArray or seeing how it's implemented. This way you could skip recreating signals/memos for unchanged items.
It's the original state a store or some immutable input?
If you can derive the new state in an immutable way, you could use reconcile to apply it into a store. Or just do it manually in one go using produce. But that's nor necessarily derived, just updated in an effect.
I've aslo made this experiment for creating a derived state from immutable source: https://primitives.solidjs.community/package/immutable#createimmutable
But maybe there should be something that can work with a fine-grain sourceI've come across this situation before, and realized the the
createMemo in combination with <For ...> was doing a better job than I initially thought.
My use case was just to sort a big array of objects that ends up being rendered in a table. When the signal/store changes, the memo update will trigger to re-run the <For> loop. I was initially assuming this would re-create all the DOM nodes because it's outside of a fine grained store.
However the sorted array still has the same instances than before (for the elements that didn't change), therefore the <For> component will still do the job of not modifying the unmodified elements.
It's likely that your use case is more complex, but I wanted to share the experience that createMemo/<For> is not bad for sorted data from a store.@jc_m yeah I was aware that <For> has that behaviour, good to remember though - maybe this is part of the solution
Btw, I'm also very interested to know if there are better ways to deal with derived data from store.
@thetarnav in my case the original state is a Solid store. Will take a look at your link
I'll post what I end up with, if it seems at all generally interesting
@thetarnav from your createImmutable: "The source can be any signal that is updated in an immutable fashion."
I'm not 100% sure what that means - is it a signal where all the values over time are themselves immutable?
The point is that a change is detected by comparing references/ids
If you want to change a nested property you have to destructure all parent objects all the way to the root
It works just like
reconcile but it's based on nested memos instead of signals
You can treat it as a memo that will produce a store instead of a single signalThat's super cool : ) Since my original data is already a store I don't think this fits my need. Or am I missing something?
the source is, but what the derivation does?
Creates a new data structure - just plain objects/arrays
so in a sense it's immutable
you are returning a new array from the memo each time
you can reuse objects from the previous run if they're unchanged
but in the end you have a single point of change that you have to listen to
you always need to call that memo if you want to listen to it
even if you only care for a single nested property
so you need to nest memos / signals to split the changes
What I'd like to do is update the derived data in a minimal way when the store changes. I think createImmutable is more useful for reacting to changes in the derived data, not building the derived data in the first place?
ok, so you do not care necessarily about producing a fine-grainly updated result
just about reducing the work you have to do by knowing what exactly changed
so yeah that's a different problem
I haven't done anything like that before I think
I think you're right that looking at the implementation of mapArray is a good way to start
in a sense mapArray is similar to createImmutable (I'm even using keyArray for reconciling arrays) as it reacts to a single source, and reconciles the input to figure out what's changed.
It's not setting up any nested computations to be able to skip the reconciliation
It's kinda to hard to do it from inside out
For example you set up a computation listening for the item as index
0
and it reruns
which means that 0 changed
do you have enough information to apply those changes to the derived state without redoing the reconciliation top-down?
What if the item was just reordered. You have no information about if the other computations get triggered or notThe part that seems relevant is that it takes a computation - the map function - that would happen array.length times for every source change, if we used a simple createMemo + Array.map - and only calls it for newly added elements.
But I guess map is a special case because we know each output element only depends on the corresponding input element, so it doesn't have the problem you just mentioned
What we might be missing is a way to listen to changes made to a store from the root
Like "item {} was added to position 9", "
type changed from to at path items[6].type" etc.
Which could be used in such derivationsYeah that's exactly what I'm thinking. Am looking at mapArray now and I see it has to do full scans of the array to figure out what happened.
At least it would be cool to be able to listen for keys added/removed to objects. That can be implemented in a similar way to mapArray I guess, but I wonder if there's a more efficient way more deeply integrated.
@jc_m I remember I said I'd follow up - it turned out for me the best place to do this was application specific. There's a place in my code that every store change is routed through, and I have more information e.g. object added/deleted/changed. Solid's reactivity only tracks "something changed" so this info is lost.