arktype•8mo ago

Export type to JSON and share it

Hello, I have two TS programs that should cooperate (A B), B import types and validates data, and A creates and "saves" the types. A rough idea

// Program A
import { type } from 'arktype' 

const a = type({
  name: 'string'
})

const b = type({
  userData: a,
  random: 'number'
})

// Somehow export the type as JSON?
fs.writeFileSync("schema.json", JSON.stringify(b))

// Program A
import { type } from 'arktype' 

const a = type({
  name: 'string'
})

const b = type({
  userData: a,
  random: 'number'
})

// Somehow export the type as JSON?
fs.writeFileSync("schema.json", JSON.stringify(b))

// Program B
import { type } from 'arktype' 

const definition = JSON.stringify(fs.readFileSync("schema.json").toString())

const validator = type(definition)

const { data, problems } = validator({
  userData: {
    name: "Jacob"
  },
  random: 5
})

// Program B
import { type } from 'arktype' 

const definition = JSON.stringify(fs.readFileSync("schema.json").toString())

const validator = type(definition)

const { data, problems } = validator({
  userData: {
    name: "Jacob"
  },
  random: 5
})

Is this possible with the current API?

76 Replies

ssalbdivad•8mo ago

Hi! This is possible to some extent today and is definitely something we're interested in One of the primary goals of AT is to offer a serializable definition syntax by default, so that would be the easiest way to achieve this If you want references to other types to be serializable, your best bet is to use a scope: https://arktype.io/docs/next/scopes/ Then you could just write the whole scope definition to your JSON file, read it back in and instantiate it anywhere One unfortunate limitation of TS is that it is currently unable to import JSON as const (please see/upvote this issue https://github.com/microsoft/TypeScript/issues/32063) So you will need to infer the type separately

JameEnderOP•8mo ago

There is the slight inconvinience of providing the entry point with scopes, instead of just having the default entry with the type

ssalbdivad•8mo ago

Yeah with the next release there would be a way to serialize arbitrary types using their underlying representation, but that isn't available yet. There are also edge cases to consider, e.g.a nything with morphs/narrows or other non-serializable data will not work

JameEnderOP•8mo ago

So I would have to do like

{
  entryPoint: 'b',
  schema: {
    a: {
      userData: a,
      random: 'number'
    },
    b: {
      userData: a,
      random: 'number'
    }
  }
}

{
  entryPoint: 'b',
  schema: {
    a: {
      userData: a,
      random: 'number'
    },
    b: {
      userData: a,
      random: 'number'
    }
  }
}

Yeah that's a good point, you can only serialize mostly "simple at compile time" values, not functions and other things

ssalbdivad•8mo ago

It would just be like this, there's not much overhead:

export const types = scope({
    a: {
        name: "string"
    },
    b: {
        userData: "a",
        random: "number"
    }
}).compile()

export const types = scope({
    a: {
        name: "string"
    },
    b: {
        userData: "a",
        random: "number"
    }
}).compile()

And especially if you're going to have more types in the future it is very convenient to be able to use your own aliases in expressions like a|number|string

JameEnderOP•8mo ago

I can't import / export, those two apps can't talk through anything else than JSON values :D

ssalbdivad•8mo ago

Well naturally inferring the types then would be impossible (although generally during the dev process there is a way to share types even if you don't share runtime code) But if you just want the validation dynamically it would still work

JameEnderOP•8mo ago

Is this in your view the best way to share it then, or do you think some other way would be more appropriate?

ssalbdivad•8mo ago

If you need to dynamically create a set of types then yes, that would be the best way to share it. In most circumstances I'd say having a schemas package that could be depended on both by the frontend and backend would be easier to work with especially if you need type safety

JameEnderOP•8mo ago

Then the code could be something like

const validator = scope(json['schema']).compile()[json['entryPoint']]

const validator = scope(json['schema']).compile()[json['entryPoint']]

Okay, thank you :D Yeah I would love to share the types with imports, but as they are two different repos that each dont see into each other, it's sadly impossible

ssalbdivad•8mo ago

Yeah something like this. Of course it depends on how complex the types are, but obviously the easiest thing to do when possible for a trivial example like this would just be to define it as a single type i.e.

const b = type({
  userData: {
      name: 'string'
  },
  random: 'number'
})

const b = type({
  userData: {
      name: 'string'
  },
  random: 'number'
})

JameEnderOP•8mo ago

Yeah that would indeed be the best, is there no API to somehow embed a into b in the initial Program A example?

ssalbdivad•8mo ago

In a simple case like that you don't need a custom API

JameEnderOP•8mo ago

Right, but the second I want to add non primitive or "non objects", like arrays with arrayOf, I can't simple export the initial schema as JSON

ssalbdivad•8mo ago

Everything can be expressed syntactically

JameEnderOP•8mo ago

const b = type({
  userData: {
      name: 'string'
  },
  random: arrayOf('number')
})

const b = type({
  userData: {
      name: 'string'
  },
  random: arrayOf('number')
})

ssalbdivad•8mo ago

Just use TS syntax:

const b = type({
  userData: {
      name: 'string'
  },
  random: 'number[]'
})

const b = type({
  userData: {
      name: 'string'
  },
  random: 'number[]'
})

JameEnderOP•8mo ago

That breaks down when I want an array of objects :D

ssalbdivad•8mo ago

If you need an array of an object, there's a feature called tuple expressions to allow that syntax to work:

const b = type({
  userData: [{
      name: 'string'
  }, '[]'],
  random: 'number'
})

const b = type({
  userData: [{
      name: 'string'
  }, '[]'],
  random: 'number'
})

Obviously if you're nesting a huge amount of them it gets a little unwieldy which is why scopes are so convenient

JameEnderOP•8mo ago

Okay, that is exactly what I needed to have it JSONable. It's quite inconvinient so I think I'll sacrifice the scopes entryPoint unconvience for better DX. Thank you a lot!

ssalbdivad•8mo ago

Actually one last note, if you're willing to try something new and don't need these features https://discord.com/channels/957797212103016458/1029933931090423828/1221863916364369920, maybe go for 2.0.0-dev.5

Discord

Discord - A New Way to Chat with Friends & Communities

Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

ssalbdivad•8mo ago

There is a .json property on Type that will serialize it. There's some other caveats and a few changed APIs (e.g. helper methods gone, now chained like .and, .or, .array() etc.) But maybe worth using that now if you're interested in serialization I can give you an overview of how that would work

JameEnderOP•8mo ago

How do I install the dev version with npm? :D

ssalbdivad•8mo ago

Sorry I wrote it backwards You can use 2.0.0-dev.5 or the dev tag

JameEnderOP•8mo ago

Okay that works, how can I deserialize the .json after?

ssalbdivad•8mo ago

You can use something like type("schema", t.json) Note that will probably change a bit in the next dev build, but nothing meaningful- just that API 😛 It would instead likely be something like schema(t.json)

JameEnderOP•8mo ago

This gives me a ParseError

ssalbdivad•8mo ago

Can you show me The code Oh I see I guess in that version there is no top-level expression for schema. So the API will definitely change 😛 But that does work at runtime

JameEnderOP•8mo ago

const a = type({
  age: 'number'
})

const b = type({
  ages: a.array()
})

const c = type('schema', b.json)

const a = type({
  age: 'number'
})

const b = type({
  ages: a.array()
})

const c = type('schema', b.json)

ssalbdivad•8mo ago

ssalbdivad•8mo ago

Yeah seems like you have to stick it in a tuple to satisfy the types there, my bad. Like I said though, probably will just be schema next release to do that (hopefully today 🙏)

JameEnderOP•8mo ago

It doesn't work in a tuple either, I added the [] and I'm still getting ParseError :D

ssalbdivad•8mo ago

So that's a runtime error you're referring to? Weird

JameEnderOP•8mo ago

const a = type({
  age: 'number'
})

const b = type({
  ages: a.array()
})

const c = type(['schema', b.json as any])

const a = type({
  age: 'number'
})

const b = type({
  ages: a.array()
})

const c = type(['schema', b.json as any])

Yes a runtime error, ParseError: Key 0 is not valid on proto schema

ssalbdivad•8mo ago

I see that 😭 Sorry, I haven't done a lot of testing on this dynamic serialization use case yet. I guess I didn't handle the unpacking the type reference correctly when dealing with JSON? You can always use the scope/tuple definition for now (and are still probably better off using 2.0 if you can), and then you can switch to this once it's stable Oh yeah I think I see the problem

JameEnderOP•8mo ago

It works without the .array() tho

ssalbdivad•8mo ago

Yeah it's because in the general case I expect instanceof to reference a constructor but for builtins like Array I use a string. I must have forgotten to handle that in some case I will fix it for this release and let you know

JameEnderOP•8mo ago

Thanks a lot!

ssalbdivad•8mo ago

No problem, thanks for guinea pigging it sorry it doesn't work yet 🥹

Randall•7mo ago

interesting!

Unknown User•7mo ago

Message Not Public

ssalbdivad•7mo ago

Definitely, I'm still working on the docs for 2.0 as part of the larger release as the top priority!

JameEnderOP•7mo ago

Hello, I'm coming back to this, is there a way to create a new type out of otherType.json, or is that still WIP?

TizzySaurus•7mo ago

otherType.json can be passed into schema from @arktype/schema to re-create otherType if that's what you mean

ssalbdivad•7mo ago

Yeah but if you just copy it 1:1 it will reuse the cached instance You can add a description or something like otherType.describe("foo") What is the use case for needing an exact copy of an existing type?

JameEnderOP•7mo ago

Two independent pieces of code, one is used as a "test" driver, one is the actual working code. The working code, on its run, exports a JSON of the schema, so the tester can later verify the outputs, without affecting the performance or throwing the responsibility onto the working code.

ssalbdivad•7mo ago

Oh you mean you want to serialize it across processes?

JameEnderOP•7mo ago

Yes

ssalbdivad•7mo ago

Yeah you can use .json for that and obviously it won't reuse a cache Keep in mind though if you include anything non-serializable like a custom validator that will not work

JameEnderOP•7mo ago

Will a field that is a type work? Something like this

import { type } from 'arktype'
import { schema } from '@arktype/schema'

const detailsSchema = type({
    age: 'number'
})

const userSchema = type({
    name: 'string',
    details: detailsSchema
})

const duplicated = schema(userSchema.json)

import { type } from 'arktype'
import { schema } from '@arktype/schema'

const detailsSchema = type({
    age: 'number'
})

const userSchema = type({
    name: 'string',
    details: detailsSchema
})

const duplicated = schema(userSchema.json)

ssalbdivad•7mo ago

Yeah that's fine

JameEnderOP•7mo ago

And how do I do the schema thingy? It's complaining about the userSchema.json It seems to work just fine, but typescript is complaining

Argument of type 'Json' is not assignable to parameter of type 'AliasSchema<string> | MorphSchema | UnitSchema<unknown> | { readonly description?: string | undefined; domain?: DomainSchema | undefined; ... 16 more ...; undeclared?: UndeclaredKeyBehavior | undefined; } | ProtoSchema<...> | DomainSchema<...> | UnionSchema<...>'.

Argument of type 'Json' is not assignable to parameter of type 'AliasSchema<string> | MorphSchema | UnitSchema<unknown> | { readonly description?: string | undefined; domain?: DomainSchema | undefined; ... 16 more ...; undeclared?: UndeclaredKeyBehavior | undefined; } | ProtoSchema<...> | DomainSchema<...> | UnionSchema<...>'.

ssalbdivad•7mo ago

Ahh because it's trying to validate it. There's something called rawRoot you can use I will probably rename that to rawSchema in next release. I guess I can also change the type of .json to be a RootSchema for root types For now you can use rawRoot or just cast

JameEnderOP•7mo ago

rawRoot has the same problem

ssalbdivad•7mo ago

Oh yeah it would still look for a schema of some sort huh Just cast it for now I can type json as RootSchema in the next release

JameEnderOP•7mo ago

Omg even .array() works and all This is great! I'm very grateful for all the work youve done

ssalbdivad•7mo ago

It should be very robust because the type system uses that .json to make comparisons As long as you don't have any non-serializable values you'll be fine

JameEnderOP•7mo ago

What does "anything serialized" mean?

ssalbdivad•7mo ago

Sorry bad description

JameEnderOP•7mo ago

Non serializable

ssalbdivad•7mo ago

symbols, functions, === to one of those

JameEnderOP•7mo ago

So something that doesn't work with JSON.stringify()?

ssalbdivad•7mo ago

or === to an object stuff like that

JameEnderOP•7mo ago

Got you

ssalbdivad•7mo ago

Yeah mostly So obviously that would include .narrow and .pipe calls as a major one. But you can actually extract out just the input portion of a type you pipe with .in At some point the way it will work will be it will store named references to non-serializable values that you have to explicitly pass in when deserializing For now I just create those references on my own but they can't persist across processes obviously Hmm I guess given that for now .json returns string references for non-serializable values means saying it is a valid root schema by default would be kind of a lie.

JameEnderOP•7mo ago

Yeah that makes sense And it would have to stay that way, as functions just cant be serialized because of potential external dependencies inside the fn body

ssalbdivad•7mo ago

Yeah. You may need to cast for now I will figure out a more robust solution eventually though

JameEnderOP•7mo ago

That's fine by me, maybe exporting the schema fn as something more verbose would be good for newcomers? Something like typeFromJson or similar

ssalbdivad•7mo ago

As something that could clearly fail maybe

JameEnderOP•7mo ago

But that could be misleading, as you could just parse a json of {"field": "number"} and it would not work as you could expect But at that moment, I'd assume you would just do JSON.parse and use type, idk Right, also I love the return error as value with type, I miss it a lot when I switch from Rust to TS

ssalbdivad•7mo ago

export type TryParseSchema = <castTo = unknown>(
    def: unknown,
    opts?: NodeParseOptions
) => ParseError | Root<castTo, {}>

export const tryParseSchema: TryParseSchema = (def, opts) => {
    try {
        return schema(def as RootSchema, opts) as never
    } catch (e) {
        if (e instanceof ParseError) return e
        throw e
    }
}

export type TryParseSchema = <castTo = unknown>(
    def: unknown,
    opts?: NodeParseOptions
) => ParseError | Root<castTo, {}>

export const tryParseSchema: TryParseSchema = (def, opts) => {
    try {
        return schema(def as RootSchema, opts) as never
    } catch (e) {
        if (e instanceof ParseError) return e
        throw e
    }
}

This?

JameEnderOP•7mo ago

That looks great I'm not sure about the "try" part, as that atleast to me implies that I should use a try catch block if the parsing fails, which isn't the case, because it returns the error as value without throwing But on that note I don't know what the best naming practices in JS space are, so thats more of a feeling than anything

ssalbdivad•7mo ago

parseUnknownSchema? maybeParseSchema parseAsSchema

JameEnderOP•2w ago

maybeParseSchema sounds silly, but gets the message across the best imo Hi, I'd like to ask if you plan on supporting recursive types -> JSON, as it would be a huge improvement for me and my team :) (this is probably the best place I figured to ask lol)

ssalbdivad•2w ago

Yes! We have this tracked here: https://github.com/arktypeio/arktype/issues/1087 My plan was to revisit this sometime after stable v2. The JSON schema side of it may be conducive to external contribution, but it would likely involve a bit of work with type nodes as well. Thanks for your sponsorship as well btw 😊

GitHub

Support conversion of cyclic types to JSON Schema · Issue #1087 · a...

This would allow types like the following to be converted to JSON schema via $ref: import { scope } from "arktype"; const types = scope({ a: { b: "b" }, b: { a: "a" } ...

JameEnderOP•2w ago

No worries, i value the work you're doing and it's the least i can do

Gaming

Programming

Export type to JSON and share it