arktype•4w ago

Key validation on arbitrary number of properties.

I'm working on validating an object where: - Some property keys are known and expected. - Other extra/unknown keys are allowed only if they follow a specific pattern. Pattern for extra keys: Keys should match the format:
metafield.<namespace>.<key>
Where: - namespace and key can contain alphanumeric characters, underscores, or hyphens. Here's the regex I'm using:

const regex = /^metafield\.[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+$/;

const regex = /^metafield\.[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+$/;

Example payload:

{
    "knownkey": "value",
    "metafield.test.key": "value",
    "metafield.test.key2": "value",
    "metafield.test.key3": "value",
    "metafield.test.key4": "value"
}

{
    "knownkey": "value",
    "metafield.test.key": "value",
    "metafield.test.key2": "value",
    "metafield.test.key3": "value",
    "metafield.test.key4": "value"
}

Question: How can I define a Arktype type that allows: - Some known, explicitly typed keys. - And any number of extra keys that match the regex pattern above?

18 Replies

Maxiviper117OP•4w ago

My attempt: but issue is if known keys cause error then all invalid metafield invalidation errors excluded from .summery unless they are the only errors:

import { type } from "arktype";

// Define the Arktype schema
const myType = type({
    // Known properties with specific types
    knownkey: "string",
    email: "string.email",
}).narrow((data, ctx) => {
    // Regex for extra keys
    const regex = /^metafield\.[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+$/;

    // List of known keys
    const knownKeys = new Set(["knownkey", "email"]);

    // Iterate over all keys in the data object
    for (let key of Object.keys(data)) {
        // If the key is not one of the known keys, validate it against the regex
        if (!knownKeys.has(key)) {
            if (!regex.test(key)) {
                
                ctx.reject({
                    expected: "key must match metafield.<namespace>.<key>",
                    actual: key,
                    path: [key],
                });
            }
        }
    }
    return true; 
});

import { type } from "arktype";

// Define the Arktype schema
const myType = type({
    // Known properties with specific types
    knownkey: "string",
    email: "string.email",
}).narrow((data, ctx) => {
    // Regex for extra keys
    const regex = /^metafield\.[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+$/;

    // List of known keys
    const knownKeys = new Set(["knownkey", "email"]);

    // Iterate over all keys in the data object
    for (let key of Object.keys(data)) {
        // If the key is not one of the known keys, validate it against the regex
        if (!knownKeys.has(key)) {
            if (!regex.test(key)) {
                
                ctx.reject({
                    expected: "key must match metafield.<namespace>.<key>",
                    actual: key,
                    path: [key],
                });
            }
        }
    }
    return true; 
});

This data would cause this error

// Test case with invalid data
const invalidData = {
    knownkey: "some value",
    email: "[email protected]",
    "metafie2ld.test.key": "value1",
    "invalid.key": "value2", // Invalid key pattern
};

// Validate the data
const invalidResult = myType(invalidData);

console.log("\nInvalid data result:");
if (invalidResult instanceof type.errors) {
    console.error(invalidResult.summary);
} else {
    console.log("Validation passed:", invalidResult);
}

// Test case with invalid data
const invalidData = {
    knownkey: "some value",
    email: "[email protected]",
    "metafie2ld.test.key": "value1",
    "invalid.key": "value2", // Invalid key pattern
};

// Validate the data
const invalidResult = myType(invalidData);

console.log("\nInvalid data result:");
if (invalidResult instanceof type.errors) {
    console.error(invalidResult.summary);
} else {
    console.log("Validation passed:", invalidResult);
}

Invalid data result:
value at ["metafie2ld.test.key"] must be key must match metafield.<namespace>.<key> (was metafie2ld.test.key)
value at ["invalid.key"] must be key must match metafield.<namespace>.<key> (was invalid.key)

Invalid data result:
value at ["metafie2ld.test.key"] must be key must match metafield.<namespace>.<key> (was metafie2ld.test.key)
value at ["invalid.key"] must be key must match metafield.<namespace>.<key> (was invalid.key)

and this data would cause this error (missing metafield errors)

// Test case with invalid data
const invalidData = {
    knownkey: "some value",
    // email: "[email protected]",
    "metafie2ld.test.key": "value1",
    "invalid.key": "value2", // Invalid key pattern
};

// Validate the data
const invalidResult = myType(invalidData);

console.log("\nInvalid data result:");
if (invalidResult instanceof type.errors) {
    console.error(invalidResult.summary);
} else {
    console.log("Validation passed:", invalidResult);
}

// Test case with invalid data
const invalidData = {
    knownkey: "some value",
    // email: "[email protected]",
    "metafie2ld.test.key": "value1",
    "invalid.key": "value2", // Invalid key pattern
};

// Validate the data
const invalidResult = myType(invalidData);

console.log("\nInvalid data result:");
if (invalidResult instanceof type.errors) {
    console.error(invalidResult.summary);
} else {
    console.log("Validation passed:", invalidResult);
}

Invalid data result:
email must be a string (was missing)

Invalid data result:
email must be a string (was missing)

What am I doing wrong or missing ?

ssalbdivad•4w ago

Well I have some good news for you- you don't have to work so hard 😅

const myType = type({
    "+": "reject",
    knownkey: "string",
    email: "string.email",
    "[/^metafield\.[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+$/]": "unknown"
})

const myType = type({
    "+": "reject",
    knownkey: "string",
    email: "string.email",
    "[/^metafield\.[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+$/]": "unknown"
})

Maxiviper117OP•4w ago

Nice but then what about custom error message, otherwise it shows that all extra invalid keys must be removed.

Maxiviper117OP•4w ago

Maxiviper117OP•4w ago

Seems like the issue is that when there's a type invalidation from type({}), it takes precedence and overrides any ArkError added via ctx.reject in a narrow() expression. So even if narrow() adds its own errors, they get ignored unless the known keys type check passes in type({}). But the issue is that I might want to collect all errors to send back in a response—so the user sees everything that failed validation at once, rather than fixing one thing, resubmitting, hitting the next error, and repeating that cycle.

ssalbdivad•4w ago

I mean I'd say that's a pretty good error message but I get wanting to include the correct format. Probably easiest to just transform the messages you want after validation like:

const myType = type({
    "+": "reject",
    knownkey: "string",
    email: "string.email",
    "[/^metafield\.[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+$/]": "unknown"
})

const out = myType({ knownkey: 5, email: "[email protected]", blah: true })

if (out instanceof type.errors) {
    const messages = out.map(e => {
        const invalidKey = /^value at \[(.*)\] must be removed$/.exec(e.message)
        if (invalidKey?.[0]) return `${invalidKey[0]}`
        return e.message
    })
    console.log(messages.join("\n"))
}

const myType = type({
    "+": "reject",
    knownkey: "string",
    email: "string.email",
    "[/^metafield\.[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+$/]": "unknown"
})

const out = myType({ knownkey: 5, email: "[email protected]", blah: true })

if (out instanceof type.errors) {
    const messages = out.map(e => {
        const invalidKey = /^value at \[(.*)\] must be removed$/.exec(e.message)
        if (invalidKey?.[0]) return `${invalidKey[0]}`
        return e.message
    })
    console.log(messages.join("\n"))
}

In terms of .narrow, it has to run after the base checks have succeeded otherwise you would be able to have an inferred input type to narrow from. If you want to give those errors regardless, I would separate it into two types. The first just checks type("object").narrow(...your custom key validation logic...) and the second can check known keys.

Maxiviper117OP•4w ago

I figured that If I wanted this type of operation for this narrow use case, I would probably have to split it up. But if you are interested in a comparison to Zod, someone came up with this in Zod:

import { z } from "zod";

const metaFieldRegex = new RegExp("^metafield.[a-zA-Z0-9_-]+.[a-zA-Z0-9_-]+$");

type ObjectWithMetaFields = {
    [K: `metafield.${string}.${string}`]: string;
};

const schema = z.intersection(
    z.object({
        knownKey: z.string(),
        email: z.string().email(),
    }),
    z
        .record(z.string())
        .superRefine((obj, ctx): obj is ObjectWithMetaFields => {
            delete obj["knownKey"]; // remove knownKey from the object to validate only meta fields

            const keys = Object.keys(obj);

            // Collect all keys that are not in the format "metafield.<namespace>.<key>"
            const invalidKeys = keys.filter(
                (x) => metaFieldRegex.test(x) === false
            );

            if (invalidKeys.length > 0) {
                ctx.addIssue({
                    fatal: true,
                    code: "unrecognized_keys",
                    keys: invalidKeys,
                    message: `If you meant to use a metafield, it should be in the format "metafield.<namespace>.<key>."`,
                });
            }

            return z.NEVER;
        })
);


const example = {
    knownKey1: "valid",
    email: "exampletest.com",
    "metafield.1": "errors_on_bad_meta_key",
    not_meta_field: "errors_on_disallowed_key",
    "metafield.test.test": "valid_key",
};

try {
    const result = schema.parse(example);
} catch (error) {
    if (error instanceof z.ZodError) {
        console.log(error.errors); // This will show the validation errors
    } else {
        console.error("Unexpected error:", error);
    }
}

import { z } from "zod";

const metaFieldRegex = new RegExp("^metafield.[a-zA-Z0-9_-]+.[a-zA-Z0-9_-]+$");

type ObjectWithMetaFields = {
    [K: `metafield.${string}.${string}`]: string;
};

const schema = z.intersection(
    z.object({
        knownKey: z.string(),
        email: z.string().email(),
    }),
    z
        .record(z.string())
        .superRefine((obj, ctx): obj is ObjectWithMetaFields => {
            delete obj["knownKey"]; // remove knownKey from the object to validate only meta fields

            const keys = Object.keys(obj);

            // Collect all keys that are not in the format "metafield.<namespace>.<key>"
            const invalidKeys = keys.filter(
                (x) => metaFieldRegex.test(x) === false
            );

            if (invalidKeys.length > 0) {
                ctx.addIssue({
                    fatal: true,
                    code: "unrecognized_keys",
                    keys: invalidKeys,
                    message: `If you meant to use a metafield, it should be in the format "metafield.<namespace>.<key>."`,
                });
            }

            return z.NEVER;
        })
);


const example = {
    knownKey1: "valid",
    email: "exampletest.com",
    "metafield.1": "errors_on_bad_meta_key",
    not_meta_field: "errors_on_disallowed_key",
    "metafield.test.test": "valid_key",
};

try {
    const result = schema.parse(example);
} catch (error) {
    if (error instanceof z.ZodError) {
        console.log(error.errors); // This will show the validation errors
    } else {
        console.error("Unexpected error:", error);
    }
}

Error shown as :

[
  {
    code: "invalid_type",
    expected: "string",
    received: "undefined",
    path: [ "knownKey" ],
    message: "Required",
  }, {
    validation: "email",
    code: "invalid_string",
    message: "Invalid email",
    path: [ "email" ],
  }, {
    fatal: true,
    code: "unrecognized_keys",
    keys: [ "knownKey1", "email", "metafield.1", "not_meta_field" ],
    message: "If you meant to use a metafield, it should be in the format \"metafield.<namespace>.<key>.\"",
    path: [],
  }
]

[
  {
    code: "invalid_type",
    expected: "string",
    received: "undefined",
    path: [ "knownKey" ],
    message: "Required",
  }, {
    validation: "email",
    code: "invalid_string",
    message: "Invalid email",
    path: [ "email" ],
  }, {
    fatal: true,
    code: "unrecognized_keys",
    keys: [ "knownKey1", "email", "metafield.1", "not_meta_field" ],
    message: "If you meant to use a metafield, it should be in the format \"metafield.<namespace>.<key>.\"",
    path: [],
  }
]

In this case with this Zod type I would be able to collect all errors at once

ssalbdivad•4w ago

I mean if that is super important just define the known keys type, then call it and add errors from the narrow

Maxiviper117OP•4w ago

Problem I mention earlier is if validation failed on known keys then the narrow errors were ignored.

ssalbdivad•4w ago

I really feel like using the index signature is the best way though because you're not having to manually control narrowing flow

Maxiviper117OP•4w ago

Yes but then I have no control on the error messages it produces. And would have to do manual processing afterwards

ssalbdivad•4w ago

const known = type({
    knownKey: "string",
    email: "string.email"
})

const metaFieldRegex = new RegExp("^metafield.[a-zA-Z0-9_-]+.[a-zA-Z0-9_-]+$")

const schema = type.object.narrow((obj, ctx) => {
    const knownResult = known(obj)

    if (knownResult instanceof type.errors) ctx.errors.merge(knownResult)

    const undeclaredKeys = Object.keys(obj).filter(
        k => !known.keyof().allows(k) && !metaFieldRegex.test(k)
    )

    if (undeclaredKeys.length)
        ctx.error({
            data: undeclaredKeys,
            message: `Keys ${undeclaredKeys} are not allowed. If you meant to use a metafield, it should be in the format "metafield.<namespace>.<key>."`
        })

    return ctx.hasError()
})

const known = type({
    knownKey: "string",
    email: "string.email"
})

const metaFieldRegex = new RegExp("^metafield.[a-zA-Z0-9_-]+.[a-zA-Z0-9_-]+$")

const schema = type.object.narrow((obj, ctx) => {
    const knownResult = known(obj)

    if (knownResult instanceof type.errors) ctx.errors.merge(knownResult)

    const undeclaredKeys = Object.keys(obj).filter(
        k => !known.keyof().allows(k) && !metaFieldRegex.test(k)
    )

    if (undeclaredKeys.length)
        ctx.error({
            data: undeclaredKeys,
            message: `Keys ${undeclaredKeys} are not allowed. If you meant to use a metafield, it should be in the format "metafield.<namespace>.<key>."`
        })

    return ctx.hasError()
})

Maxiviper117OP•4w ago

small error in the regex this is the corrected one:

const regex = /^metafield\.[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+$/;

const regex = /^metafield\.[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+$/;

ssalbdivad•4w ago

Okay lol I just copied yours not exactly the point...

Maxiviper117OP•4w ago

yes just made the correction in case you were testing it your side and seeing nothing your last example works well to get all errors at once, appreciate the time taken.

ssalbdivad•4w ago

Glad it was helpful!

Maxiviper117OP•4w ago

The whole use case for this was validating data that would be used to create a Shopify customer via API, where customers on a clients account could have any number of additional metafields that hold extra information apart from the standard. And those metafields need to have that format when recevied to be able to easily parse them to send them correctly to each clients Shopify account to create a customer with unique metafields.

ssalbdivad•4w ago

I also want a dedicated code for undeclared key errors so you could leverage that directly in the future https://github.com/arktypeio/arktype/issues/1403

GitHub

Add dedicated error code for undeclared key · Issue #1403 · arkty...

This would allow customization for undeclared key errors. Currently, we are just using a general "predicate" error: ctx.errorFromNodeContext({ code: "predicate", expected: "...

Gaming

Programming

Key validation on arbitrary number of properties.

Did you find this page helpful?