arktype•4w ago

Validating Large JSON Schemas without high time/memory costs

I'm building a low-code tool for generating user interfaces. The UI structure is defined using JSON. The schema for the JSON structure is quite large. I have been using Zod for validation, but it is very slow—taking minutes to validate a single UI definition. I've tried switching to ArkType but it consumes too much memory, often causing the browser to crash. When it does not crash (and on the server where I can set a high memory limit) the call to type.module takes approximately 15 seconds, and then validation mere milliseconds. Given these performance issues, how can I efficiently validate large JSON structures against a schema without excessive time consumption or memory usage? Are there optimizations, alternative libraries, or hybrid approaches that could help? Would love to hear any insights or best practices! Here is a small section of the ArkType schema to give you an idea of how it works. As you can see, it is inherently cyclic (take a look at component).

const componentModule = type.module({
  "#accordionItem": {
    "title?": "component",
    "content?": "component",
  },
  accordion: {
    "...": componentBaseSchema,
    type: "'accordion'",
    items: "accordionItem[]",
  },

  button: {
    "...": componentBaseSchema.and(mouseEvents),
    type: "'button'",
    variant:
      "'default' | 'secondary' | 'destructive' | 'outline' | 'ghost' | 'link'",
    size: "'default' | 'sm' | 'lg' | 'icon' | 'compact'",
    "href?": "string",
    label: "component",
  },

  "#tabsItem": {
    id: "string",
    title: "component",
    content: "component",
  },
  tabs: {
    "...": componentBaseSchema,
    type: "'tabs'",
    "currentTabVariable?": "string",
    "onTabChange?": eventHandlersSchema,
    items: "tabsItem[]",
  },

  "#singleComponent":
    "string | accordion | button | tabs",
  component: "singleComponent | singleComponent[]",
});
const componentSchema = componentModule.component;

const componentModule = type.module({
  "#accordionItem": {
    "title?": "component",
    "content?": "component",
  },
  accordion: {
    "...": componentBaseSchema,
    type: "'accordion'",
    items: "accordionItem[]",
  },

  button: {
    "...": componentBaseSchema.and(mouseEvents),
    type: "'button'",
    variant:
      "'default' | 'secondary' | 'destructive' | 'outline' | 'ghost' | 'link'",
    size: "'default' | 'sm' | 'lg' | 'icon' | 'compact'",
    "href?": "string",
    label: "component",
  },

  "#tabsItem": {
    id: "string",
    title: "component",
    content: "component",
  },
  tabs: {
    "...": componentBaseSchema,
    type: "'tabs'",
    "currentTabVariable?": "string",
    "onTabChange?": eventHandlersSchema,
    items: "tabsItem[]",
  },

  "#singleComponent":
    "string | accordion | button | tabs",
  component: "singleComponent | singleComponent[]",
});
const componentSchema = componentModule.component;

4 Replies

ssalbdivad•4w ago

Wow it must be a pretty crazy module to take 15 seconds for initialization or minutes for validation in Zod... Initialization could likely be optimized internally as less focus has gone into e.g. organizing our internal implementation to be monomorphic than optimizing validation performance as you can perhaps see. However, for something that extreme on the frontend, precompilation will likely be your best option to remove that overhead and bundle size altogether: https://github.com/arktypeio/arktype/issues/810

GitHub

Precompilation · Issue #810 · arktypeio/arktype

This would allow a set of scopes/types to be precompiled to avoid JIT compilation at runtime. This requires additional investigation around possible targets (ts vs js build output) and tooling (TS-...

StadlyOP•4w ago

I find it very surprising that it takes so long. I got those results with just 10 more type aliases in addition to the accordion, button, and tabs from the example. Precompilation sounds great, but I guess that's not just around the corner? Do you have any other ideas? Is it the cyclicity that causes performance issues, or can it be something else? Would it help to split it up in some way? For example type everything that should be component as unknown to avoid the cyclic nature of the schema, and then create some custom parser that recursively goes through and validates every component?

ssalbdivad•4w ago

It's likely large unions that are O(n^2) by nature that are the most expensive That sounds like something is going very wrong. We have internal benchmarks for scopes with 500 cyclic types and they initialize in ms You can try switching to jitless mode and see if that helps but it would slow down validation significantly

StadlyOP•4w ago

Thanks. I've created two issues. Hopefully there is something to about them 🙂 https://github.com/arktypeio/arktype/issues/1371 https://github.com/arktypeio/arktype/issues/1370

GitHub

Memory usage with large schemas · Issue #1371 · arktypeio/arktype

I have a large schema (more or less the same as in #1370, but with a large optional style schema in most of the type aliases): import { type } from "arktype"; const spacingScale = [ "...

GitHub

TypeScript issues with large recursive schema · Issue #1370 · arkty...

I have a rather large schema, but I would not call it extreme in any way: import { type } from "arktype"; const componentModule = type.module( { "#accordionItem": { "title?...

Gaming

Programming

Validating Large JSON Schemas without high time/memory costs

Did you find this page helpful?