C#•4mo ago

Chunking both key and value from dictionary

Hi all, I have a very specific problem of needing to chunk both key AND value of a dictionary (could be any data type). Scenario: I need to export and import a bunch of time series data from one tenant to a different one. The case is that I fetch say 500 Keys, where each key can have a range from 0..? count of values. The time series database service only allow bathc inserts of 100 keys, with 20,000 values per key in each insert. I need to chunk the request prior to creating the request to maintain the aforementioned rules. How would you do this efficiently?

3 Replies

cap5lut•4mo ago

@viceroypenguin | 🦋🐧 got a good solution for that in the advanced channel https://discord.com/channels/143867839282020352/663803973119115264/1271467822853259386

MODiX•4mo ago

viceroypenguin

if they want to reduce it down...

sequence
  .GroupBy(x => x.Key)
  .SelectMany(x => x
    .Chunk(ChunkSize)
    .Select(c => (
      Key: x.Key,
      Chunk: c
    )
  );

sequence
  .GroupBy(x => x.Key)
  .SelectMany(x => x
    .Chunk(ChunkSize)
    .Select(c => (
      Key: x.Key,
      Chunk: c
    )
  );

Quoted by

<@233139962277658624> from #advanced (click here)

React with ❌ to remove this embed.

canton7•4mo ago

Note that that chunks my number of key+value pairs, and doesn't respect the "max 500 keys, each with max 20,000 values per key" requirement

Gaming

Programming

Chunking both key and value from dictionary