Avoiding rerenders using `setState + reconcile`

Hi, Currently I have a webworker producing data slices which I want to render. It looks a bit like this:

// limit frontend updates to max 100 per second or min 1 per second
const timer;
const lastUpdate = Date.now();
worker.onmessage = ({data}) => {
  clearTimeout(timer);
  const now = Date.now();
  const update = () => {
    lastUpdate = now;
    setState('events', reconcile(data.evts));
  };
  if (now > lastUpdate + 1000) {
    update();
  } else {
    setTimeout(update, 10);
  }
};

// limit frontend updates to max 100 per second or min 1 per second
const timer;
const lastUpdate = Date.now();
worker.onmessage = ({data}) => {
  clearTimeout(timer);
  const now = Date.now();
  const update = () => {
    lastUpdate = now;
    setState('events', reconcile(data.evts));
  };
  if (now > lastUpdate + 1000) {
    update();
  } else {
    setTimeout(update, 10);
  }
};

The problem is, when I have a <For each={state.events}> somewhere, the whole loop keeps rerendering, even if only a few elements have been prepended (they always get prepended). I thought of trying to specify the key in reconcile (the data itself doesn't change, just position in the array) but it is 2 levels deep. The events array is of type:

type EventDataFoo = {timestamp: Date, some_other_data: any};
type EventDataBar = {timestamp: Date, some_data: any};
type Events = ({kind: 'foo', data: EventDataFoo} | {kind: 'bar', data: EventDataBar})[]

type EventDataFoo = {timestamp: Date, some_other_data: any};
type EventDataBar = {timestamp: Date, some_data: any};
type Events = ({kind: 'foo', data: EventDataFoo} | {kind: 'bar', data: EventDataBar})[]

the key would be data.timestamp which is guaranteed to be unique, even over different data kinds. reconcile expects the key to be a string tho which I can only provide for 1 level. Am I misunderstanding something completely or is this really hard to achieve?

5 Replies

bigmistqke•2y ago

mm, strange that the elements re-mount with reconcile. u could try <Index/> instead? But mb also not ideal since you are prepending. It won't re-render the components but it will update all the dom-nodes. mb you could only send the new events from the worker and do something like setState('events', produce(events => events.unshift(new_data)))? or use <Index/> and not prepend but flip the order with flex-direction?

reconcile expects the key to be a string tho which I can only provide for 1 level.

wdym with this?

belstOP•2y ago

reconciles second parameter takes an object of type { merge: bool, key: string } so I can't give it a key function if I do that I can't do 10ms wait anymore, because I would lose events. and if I rerender everytime I get an event, the browser would start to struggle that's the problem I am currently trying to fix right now, after like 3000 events it gets unusable slow (we had all the processing done in the mainthread aswell tho) append + flex-direction might work. need to check how good it works with cutting off the beginning tho, since we only show a slice of the data (data can be up to a few million events) the webworker has a list of all the events, I postMessage the number of events in (signal that gets updated with an intersection observer) and it produces back the last N events. (so I can get more data by scrolling). by default it's only 35 events displayed tho, so when new events come in, the list gets shifted js needs SharedArrayBuffers for more complex data types 🙂 it would all be so nice if I could just have all the objects in a buffer and just get start and end index

bigmistqke•2y ago

did not know that! thanks for the info!

(data can be up to a few million events)

o waw.

belstOP•2y ago

biggest we've seen so far was like 4 million events. the data processing pipeline also cuts off at around 9 million (Buffer max size reached) so there is a hard limit, it is just very high lol

bigmistqke•2y ago

by default it's only 35 events displayed tho, so when new events come in, the list gets shifted

Ok then I think <Index/> would be a good candidate actually. So if I understand correctly, every 10ms there is new data coming in and being presented, with the 35 newest being displayed on the screen? If you would scroll down you can see earlier events? Because if that's the case, it actually would make sense if all the components re-render, since <For/> is reference-based. So if it's constantly new references being shown, it will constantly mount new components. While <Index/> is index-based, so if you wouldn't scroll, it would just mount those 35 components and then update the props. If you would scroll down a bit you would mount more components, but there is probably only so much list-elements you can show anyway so maybe virtualisation can help there. Also the reconciliation-process has overhead too, to match the immutable data to its references, and with millions of elements this can become significantly expensive. With <Index/> you wouldn't need to worry about references and you can skip reconcile all together. very interesting challenge! js has SharedArrayBuffer right? it's probably a good idea honestly, cut out the serialisation/deserialisation cost. maybe use https://github.com/google/flatbuffers or https://github.com/GoogleChromeLabs/buffer-backed-object ? I have never used them myself. https://github.com/Bnaya/objectbuffer is also pretty sick project.

Gaming

Programming

Avoiding rerenders using `setState + reconcile`