Sapphire - Imagine a framework•2mo ago

Serializing string-store result inside another schema

I'm trying to re-work customId serialization in my framework, and my plan is to use string-store for this. One of the features in my framework is that you can make customIds that refer to commands, kind of like this:

class MyCommand extends Command {
  public async execute(i: ChatInputInteraction, meta: Metadata) {
    const bot = meta.getBot();
    const customId = bot.getCommandManager().getCustomIdSerializer().serialize(this);
    // reply with button or something that uses the customId
  }
  
  public executeComponent(i: ComponentInteraction) {
    i.reply('Hello!')
  }
}

class MyCommand extends Command {
  public async execute(i: ChatInputInteraction, meta: Metadata) {
    const bot = meta.getBot();
    const customId = bot.getCommandManager().getCustomIdSerializer().serialize(this);
    // reply with button or something that uses the customId
  }
  
  public executeComponent(i: ComponentInteraction) {
    i.reply('Hello!')
  }
}

The serializer and the router in the command manager then work together to properly route interactions to the #executeComponent() method. My problem comes from how to implement string-store's serialization for this. My main idea was to do something like this (on #execute()):

// data is passed as "extra" data, that the serializer doesn't care about what it actually is, it only knows it's a string
const data = store.serialize(/** data */);
const customId = bot.getCommandManager().getCustomIdSerializer().serialize(this, data);

// data is passed as "extra" data, that the serializer doesn't care about what it actually is, it only knows it's a string
const data = store.serialize(/** data */);
const customId = bot.getCommandManager().getCustomIdSerializer().serialize(this, data);

#executeComponent() would then accept an extra, optional parameter, which would be the user-provided data as a string, resolved from the customId. The way this would work is by making the customIdSerializer have a schema with a store like this:

new Schema(Id.Something).string('extraData').../** other data needed to actually resolve the command */;

new Schema(Id.Something).string('extraData').../** other data needed to actually resolve the command */;

The advantage of this is that the user isn't forced to use string-store, they can just pass anything and they'll receive it back. But then if they do use string-store, the user would be serializing the string, and then the framework would be serializing it again, and I wonder if that could cause issues (from my poor testing it doesn't look like it but I'm pretty much nobody compared to the actual maintainers, maybe there's edge cases). Maybe I'm overthinking it and there's a better way?

Solution:

I don't think it's a good idea to let users write or paste raw string-store data, for starters, it produces strings created by abusing the underlying UTF16 standard, which can lead to very confusing characters... and invisible ones, among which NULL (\0), which is often the delimiter character at the end of a string in C (see strlen()), which... Windows, Linux, and perhaps MacOS as well rely on. That being said, if you convert the binary to base64, then you'll be able to share it in plain text on Discord more nicely. serialize returns an instance of UnalignedUint16Array after all, with some code you can convert it to base64, this should work:

const data = store.serialize(/** data */);
const base64 = Buffer.from(data.toArray().buffer).toString('base64');

const data = store.serialize(/** data */);
const base64 = Buffer.from(data.toArray().buffer).toString('base64');

...

Jump to solution

22 Replies

Favna•2mo ago

@kyra

AmgeloOP•2mo ago

bump

kyra•2mo ago

Sorry for the delay, I have been burnt out lately

Solution

kyra•2mo ago

const data = store.serialize(/** data */);
const base64 = Buffer.from(data.toArray().buffer).toString('base64');

const data = store.serialize(/** data */);
const base64 = Buffer.from(data.toArray().buffer).toString('base64');

As for serializing a string-store result inside another schema, it's completely doable as long as you remember to do data.toString(), because it doesn't return a raw string. 👀

AmgeloOP•2mo ago

ah, yeah I forgot the .toString() call, that's the equivalent to your base64 code right?

kyra•2mo ago

In the codeblock? You should use base64 if it's going to be exposed in plain-text to the user on Discord, a browser, or similar In custom_id? Just data.toString()

AmgeloOP•2mo ago

oh it wouldn't, or well that'd be up to the user I don't see many use cases for that since the system only works for customIds the only thing it'd do is pass back the data the user shared

kyra•2mo ago

I mean, the system is quite generic but... yeah it's mostly for abusing Discord's UTF16

AmgeloOP•2mo ago

ah sorry I meant my system the one for "referring" to commands via customIds

kyra•2mo ago

Oh yeah, sure, just make sure to remember that users may forge the string or modify it in a way it becomes invalid

AmgeloOP•2mo ago

another concern I was having was whether having an entire store for just one schema would be efficient, or if there was a way I could use the only schema directly a solution I was thinking was to make a global store inside a variable since there are other similar systems that also require customId serialization with extra data but: 1. ew, mutable global 2. I don't think it adheres single responsability, one store would be "managing" multiple unrelated systems

kyra•2mo ago

That's more on your implementation than on the library :p

AmgeloOP•2mo ago

I think that'd be more on the framework user I'm not sure if I can/should do much about that what about the first question? anything I could do about that?

kyra•2mo ago

You can just make a store-less schema too, although you'll need a bit of code, the heavy lifting is done on Schema's side. SchemaStore#{de}serialize is just a wrapper of Schema's methods. So as long as you replicate some of the code, you can achieve the same:

AmgeloOP•2mo ago

I think I'll just do that if that code would need to change in my side after an update then it'd be a string-store breaking change right? though I pin all my dependencies but still

kyra•2mo ago

The protocol is something that must stay the same as otherwise different versions of string-store would throw. And Schema's API is subject to semver, so if it changed in a way it'd require a change in SchemaStore, it would be a semver-major change That being said, we rarely ever change our libraries, specially the newest ones, so I don't expect to see a semver-major in the foreseeable future

AmgeloOP•2mo ago

let's just hope discord doesn't change the char limit to size limit lol this is really helpful

kyra•2mo ago

That'll happen the day Discord changes the API from Python to something else The most likely alternative being Rust... which uses UTF32 (Which would double our size limit XD) Either way, 100 or 200 bytes, the unaligned key-less data can still store a lot of data :p

AmgeloOP•2mo ago

@sbear :bcat_stare: true alright I think that's all of the questions I had thanks!

kyra•2mo ago

You're welcome!

kyra•2mo ago

And sorry for the delay 😅

AmgeloOP•2mo ago

np, I understand the burn out

Gaming

Programming

Serializing string-store result inside another schema

Did you find this page helpful?