Theo's Typesafe Cult•2y ago

Using JSZip with Zod and tRPC

I am making a uploadRouter for my website where a user can upload 5-10 images. I want to be able to zip those image and then upload it to a r2 endpoint. I never really worked with zipping and image schemas in Zod and tRPC so im not sure on what to do. Here is what I have so far

import { router, publicProcedure, protectedProcedure } from "../trpc";
import { z } from "zod";
import JSZip from "jszip";

const MAX_FILE_SIZE = 5000000;
const ACCEPTED_IMAGE_TYPES = ["image/jpeg", "image/jpg", "image/png", "image/webp"];

const imageSchema = z.object({
    image: z
        .any()
        .refine((file) => file?.size <= MAX_FILE_SIZE, 'Maximum file size is 50MB')
        .refine((file) => ACCEPTED_IMAGE_TYPES.includes(file?.type), 'Unsupported file type')
})

export const uploadRouter = router({
    upload: protectedProcedure
        .input(imageSchema.array().min(5).max(10))
        .mutation(({ ctx, input }) => {

            const zip = new JSZip();
            const remoteZips = input.map((image) => {
                // ERROR HERE
                zip.file(image.name, image)
            });
            
        })
})

import { router, publicProcedure, protectedProcedure } from "../trpc";
import { z } from "zod";
import JSZip from "jszip";

const MAX_FILE_SIZE = 5000000;
const ACCEPTED_IMAGE_TYPES = ["image/jpeg", "image/jpg", "image/png", "image/webp"];

const imageSchema = z.object({
    image: z
        .any()
        .refine((file) => file?.size <= MAX_FILE_SIZE, 'Maximum file size is 50MB')
        .refine((file) => ACCEPTED_IMAGE_TYPES.includes(file?.type), 'Unsupported file type')
})

export const uploadRouter = router({
    upload: protectedProcedure
        .input(imageSchema.array().min(5).max(10))
        .mutation(({ ctx, input }) => {

            const zip = new JSZip();
            const remoteZips = input.map((image) => {
                // ERROR HERE
                zip.file(image.name, image)
            });
            
        })
})

This is the error im getting

No overload matches this call.
  Overload 1 of 4, '(path: string, data: null, options?: (JSZipFileOptions & { dir: true; }) | undefined): JSZip', gave the following error.
    Argument of type '{ image?: any; }' is not assignable to parameter of type 'null'.
  Overload 2 of 4, '(path: string, data: string | number[] | Uint8Array | ArrayBuffer | Blob | ReadableStream | Promise<string | number[] | Uint8Array | ArrayBuffer | Blob | ReadableStream>, options?: JSZipFileOptions | undefined): JSZip', gave the following error.
    Argument of type '{ image?: any; }' is not assignable to parameter of type 'string | number[] | Uint8Array | ArrayBuffer | Blob | ReadableStream | Promise<string | number[] |

No overload matches this call.
  Overload 1 of 4, '(path: string, data: null, options?: (JSZipFileOptions & { dir: true; }) | undefined): JSZip', gave the following error.
    Argument of type '{ image?: any; }' is not assignable to parameter of type 'null'.
  Overload 2 of 4, '(path: string, data: string | number[] | Uint8Array | ArrayBuffer | Blob | ReadableStream | Promise<string | number[] | Uint8Array | ArrayBuffer | Blob | ReadableStream>, options?: JSZipFileOptions | undefined): JSZip', gave the following error.
    Argument of type '{ image?: any; }' is not assignable to parameter of type 'string | number[] | Uint8Array | ArrayBuffer | Blob | ReadableStream | Promise<string | number[] |

59 Replies

Grey•2y ago

const zip = new JSZip();

input.forEach((file) => {
        const data = fs.readFileSync(file.path);
        zip.file(file.originalname, data);
      });

const content = await zip.generateAsync({ type: 'nodebuffer' });
return content;

const zip = new JSZip();

input.forEach((file) => {
        const data = fs.readFileSync(file.path);
        zip.file(file.originalname, data);
      });

const content = await zip.generateAsync({ type: 'nodebuffer' });
return content;

try this you're using it wrong

elpupperOP•2y ago

hmm my schema is most def wrong

Grey•2y ago

also, maybe this should be a better zod schema

z.array(z.object({
  fieldname: z.string().refine((value) => value === 'images', {
    message: 'Invalid fieldname',
  }),
  originalname: z.string().min(3),
  encoding: z.string().optional(),
  mimetype: z.string().refine((value) => value.startsWith('image/'), {
    message: 'Invalid file type',
  }),
  destination: z.string().optional(),
  filename: z.string().optional(),
  path: z.string().optional(),
  size: z.number().positive(),
}));

z.array(z.object({
  fieldname: z.string().refine((value) => value === 'images', {
    message: 'Invalid fieldname',
  }),
  originalname: z.string().min(3),
  encoding: z.string().optional(),
  mimetype: z.string().refine((value) => value.startsWith('image/'), {
    message: 'Invalid file type',
  }),
  destination: z.string().optional(),
  filename: z.string().optional(),
  path: z.string().optional(),
  size: z.number().positive(),
}));

elpupperOP•2y ago

yep

Grey•2y ago

you're using 1 object it should be an array

elpupperOP•2y ago

upload: protectedProcedure
        .input(imageSchema.array().min(5).max(10))

upload: protectedProcedure
        .input(imageSchema.array().min(5).max(10))

Grey•2y ago

hmm didn't saw that

elpupperOP•2y ago

its not making any sense to me what u sent nor does mine either cause it was just ripped from a blog and even the blog didnt make sense basically what i need is to get the images that the user uploaded so for example i select 5 to 10 images on the website click upload then it goes to this trpc endpoint and zips them

Grey•2y ago

ya know....... you can just.... use a nextjs raw http endpoint to handle file uploads

elpupperOP•2y ago

hmm that is true

Grey•2y ago

I've always used those for doing file stuff as the raw request object is available

elpupperOP•2y ago

ok im gonna do that is there a guide or documentation u can link me to?

Grey•2y ago

hold up I have a code snippet I wrote a few days ago hmm this is what I wrote, it's saving one image a time though you'll have to read the formidable docs to handle first parsing multiple files, then saving them. Or hold up a minute wait don't use that it'll get messier that way

elpupperOP•2y ago

alr i just want a way to get the images and check if they are actually images and check if they are smaller than 50 mb thats it

Grey•2y ago

I have something along those lines, let me remember where I saw it *proompting some code into existance here

import { NextApiRequest, NextApiResponse } from 'next';
import multer from 'multer';
import JSZip from 'jszip';
import fs from 'fs';

const upload = multer({ dest: 'uploads/' });

// Disable built-in body parsing
export const config = {
  api: { bodyParser: false },
};

export default upload.array('images', 5, async (req: NextApiRequest, res: NextApiResponse) => {
  try {
    const zip = new JSZip();

    req.files.forEach((file: Express.Multer.File) => {
      const data = fs.readFileSync(file.path);
      zip.file(file.originalname, data);
    });

    const zipContent = await zip.generateAsync({ type: 'nodebuffer' });

    // Save the zip file
    const zipFilePath = 'path/to/save/zip.zip';
    fs.writeFileSync(zipFilePath, zipContent);

    res.status(200).json({ message: 'Zip file saved successfully' });
  } catch (error) {
    console.error('Error saving zip file:', error);
    res.status(500).json({ message: 'Failed to save zip file' });
  } finally {
    // Cleanup the uploaded files
    req.files.forEach((file: Express.Multer.File) => {
      fs.unlinkSync(file.path);
    });
  }
});

import { NextApiRequest, NextApiResponse } from 'next';
import multer from 'multer';
import JSZip from 'jszip';
import fs from 'fs';

const upload = multer({ dest: 'uploads/' });

// Disable built-in body parsing
export const config = {
  api: { bodyParser: false },
};

export default upload.array('images', 5, async (req: NextApiRequest, res: NextApiResponse) => {
  try {
    const zip = new JSZip();

    req.files.forEach((file: Express.Multer.File) => {
      const data = fs.readFileSync(file.path);
      zip.file(file.originalname, data);
    });

    const zipContent = await zip.generateAsync({ type: 'nodebuffer' });

    // Save the zip file
    const zipFilePath = 'path/to/save/zip.zip';
    fs.writeFileSync(zipFilePath, zipContent);

    res.status(200).json({ message: 'Zip file saved successfully' });
  } catch (error) {
    console.error('Error saving zip file:', error);
    res.status(500).json({ message: 'Failed to save zip file' });
  } finally {
    // Cleanup the uploaded files
    req.files.forEach((file: Express.Multer.File) => {
      fs.unlinkSync(file.path);
    });
  }
});

here an example of using multer to handle files I used this once before so I remember it works cleanly with files. mind you the snippet above is from ChatGPT so use at your own discretion

elpupperOP•2y ago

are u using chatgpt lol

Grey•2y ago

haha just did as an example, I don't have a code snippet for multer

elpupperOP•2y ago

whats multer?

Grey•2y ago

https://github.com/expressjs/multer#readme

GitHub

GitHub - expressjs/multer: Node.js middleware for handling `multipa...

Node.js middleware for handling multipart/form-data. - GitHub - expressjs/multer: Node.js middleware for handling multipart/form-data.

elpupperOP•2y ago

also i want this to be protected thats why trpc made sense

Grey•2y ago

put in some pre-checks like in the above code snippet, before the try block, put in your

const session = await redis.get(req.cookies.get("SID")); 
if (!isSessionValid(session)) {
  return Response("Invalid Request", {status: 401})
}

const session = await redis.get(req.cookies.get("SID")); 
if (!isSessionValid(session)) {
  return Response("Invalid Request", {status: 401})
}

or whatever session validation you have

elpupperOP•2y ago

quick update im fully committed to using trpc and to making it work so currently i have

// create.tsx
const [images, setImages] = useState<File[]>([]);

// create.tsx
const [images, setImages] = useState<File[]>([]);

Grey•2y ago

👍 There's a way, good luck

elpupperOP•2y ago

i just did it 😎

Grey•2y ago

GGs, did you inject the http req body into the context? or did you do it in another way?

elpupperOP•2y ago

first i made my own type

type Image = {
    name: string;
    type: string;
    size: number;
    base64: string;
}

type Image = {
    name: string;
    type: string;
    size: number;
    base64: string;
}

Grey•2y ago

mmhm

elpupperOP•2y ago

then made a function to get the base64 (client side)

    const getBase64 = (file: File) => {
        return new Promise((resolve) => {
            const reader = new FileReader();
            reader.readAsDataURL(file);
            reader.onload = () => resolve(reader.result);
        });
    }

    const getBase64 = (file: File) => {
        return new Promise((resolve) => {
            const reader = new FileReader();
            reader.readAsDataURL(file);
            reader.onload = () => resolve(reader.result);
        });
    }

Grey•2y ago

okay and

elpupperOP•2y ago

then in the file input i got

                                    <input
                                        id="dropzone-file"
                                        type="file"
                                        multiple
                                        onChange={(e) => {
                                            // we need to convert the files to base64
                                            const files = e.target.files;
                                            if (files) {
                                                const filesArray = Array.from(files);
                                                const promises = filesArray.map(async (file) => {
                                                    const base64 = await getBase64(file);
                                                    return {
                                                        name: file.name,
                                                        type: file.type,
                                                        size: file.size,
                                                        base64: base64!.toString(),
                                                    };
                                                });

                                                Promise.all(promises).then((images_) => {
                                                    setImages((prevImages) => images_);
                                                });
                                            }
                                        }}
                                        accept="image/png, image/jpeg, image/jpg"
                                        className="hidden" />

                                    <input
                                        id="dropzone-file"
                                        type="file"
                                        multiple
                                        onChange={(e) => {
                                            // we need to convert the files to base64
                                            const files = e.target.files;
                                            if (files) {
                                                const filesArray = Array.from(files);
                                                const promises = filesArray.map(async (file) => {
                                                    const base64 = await getBase64(file);
                                                    return {
                                                        name: file.name,
                                                        type: file.type,
                                                        size: file.size,
                                                        base64: base64!.toString(),
                                                    };
                                                });

                                                Promise.all(promises).then((images_) => {
                                                    setImages((prevImages) => images_);
                                                });
                                            }
                                        }}
                                        accept="image/png, image/jpeg, image/jpg"
                                        className="hidden" />

pretty much we just convert it to image type

Grey•2y ago

I see

elpupperOP•2y ago

then the upload.ts router is almost the same

Grey•2y ago

so you're abusing a base64 string of an image to uploading it normally to trpc

elpupperOP•2y ago

import { router, publicProcedure, protectedProcedure } from "../trpc";
import { z } from "zod";
import JSZip from "jszip";
import { promises as fsPromises } from 'fs';

const { writeFile } = fsPromises;

const MAX_FILE_SIZE = 5000000;
const ACCEPTED_IMAGE_TYPES = ["image/jpeg", "image/jpg", "image/png", "image/webp"];

const uploadSchema = z.object({
    imageFiles: z
        .array(z
            .any()
            .refine((file) => ACCEPTED_IMAGE_TYPES.includes(file.type), {
                message: "Invalid file type",
            })
            .refine((file) => file.size <= MAX_FILE_SIZE, {
                message: "File is too large",
            })
        )
        .min(5)
        .max(10),
});

import { router, publicProcedure, protectedProcedure } from "../trpc";
import { z } from "zod";
import JSZip from "jszip";
import { promises as fsPromises } from 'fs';

const { writeFile } = fsPromises;

const MAX_FILE_SIZE = 5000000;
const ACCEPTED_IMAGE_TYPES = ["image/jpeg", "image/jpg", "image/png", "image/webp"];

const uploadSchema = z.object({
    imageFiles: z
        .array(z
            .any()
            .refine((file) => ACCEPTED_IMAGE_TYPES.includes(file.type), {
                message: "Invalid file type",
            })
            .refine((file) => file.size <= MAX_FILE_SIZE, {
                message: "File is too large",
            })
        )
        .min(5)
        .max(10),
});

export const uploadRouter = router({
    create: protectedProcedure
        .input(uploadSchema)
        .mutation(async ({ ctx, input }) => {
            console.log("Creating zip file");
            const { imageFiles } = input;
            const zip = new JSZip();

            console.log("Adding files to zip");
            imageFiles.forEach((file, index) => {
                console.log(`Adding file ${index + 1} of ${imageFiles.length}`);
                console.log(`File name: ${file.name}`);
                file.base64 = file.base64.replace(/^data:image\/(png|jpeg|jpg|webp);base64,/, "");
                file.data = Buffer.from(file.base64, "base64");
                zip.file(file.name, file.data);
            });

            const zipFile = await zip.generateAsync({ type: "nodebuffer" });

            console.log("Saving zip file");

            await writeFile("images.zip", zipFile);

            return { success: true };
        })
})

export const uploadRouter = router({
    create: protectedProcedure
        .input(uploadSchema)
        .mutation(async ({ ctx, input }) => {
            console.log("Creating zip file");
            const { imageFiles } = input;
            const zip = new JSZip();

            console.log("Adding files to zip");
            imageFiles.forEach((file, index) => {
                console.log(`Adding file ${index + 1} of ${imageFiles.length}`);
                console.log(`File name: ${file.name}`);
                file.base64 = file.base64.replace(/^data:image\/(png|jpeg|jpg|webp);base64,/, "");
                file.data = Buffer.from(file.base64, "base64");
                zip.file(file.name, file.data);
            });

            const zipFile = await zip.generateAsync({ type: "nodebuffer" });

            console.log("Saving zip file");

            await writeFile("images.zip", zipFile);

            return { success: true };
        })
})

there it is

Grey•2y ago

nice workaround haha, I might have thought of that on my 2nd-3rd try if I was as adamant as you in making trpc work with it

elpupperOP•2y ago

thats all the code i also had to disable the 1mb limit u should probably increase it someone did warn me that there was a 5mb limit in prod so i will try that now

Grey•2y ago

which is not a good thing to do generally (payload bombing to slow the server down/^ cost)

elpupperOP•2y ago

hmm

Grey•2y ago

as long as you keep a limit, it's probably alright

elpupperOP•2y ago

but this is with images so i think its fine

Grey•2y ago

there is already WAFs on serverless

elpupperOP•2y ago

i have the limit set in zod to 50mb per image

Grey•2y ago

it is

elpupperOP•2y ago

so it is fine

Grey•2y ago

hmm but..... are your users going to be uploading 8k raw images? even those don't touch 50mb

elpupperOP•2y ago

nah your right

Grey•2y ago

you should reduce it to around 10mb per image

elpupperOP•2y ago

i should prolly leave it around 10mb yep

Grey•2y ago

👋 well may not be the best solution, but definitely a good one ( I did that too the other day with a custom captcha service I made in go )

elpupperOP•2y ago

yep i hope there is a better solution out there with trpc but this is what i came up with and i couldnt find any online resources

Grey•2y ago

there is, it's just that it's focused on web standard file uploading, where as you're seeing it with the eye of t3 stack's abstracted layer; which I'm sure does not have any references online which is why I recommended a raw next http endpoint in the first place

elpupperOP•2y ago

in the end i think it would be the same

Grey•2y ago

it actually might be a bit slower

elpupperOP•2y ago

how so

Grey•2y ago

but yeah if it's serverless, you're not measuring the server resource costs costs = CPU/RAM usage, which is negligible for a small userbase, but noticable on the 10k+ user base

elpupperOP•2y ago

awe whatever our product has like a 10000% margin

Grey•2y ago

eh trpc setting up it's context to process and all, plus you using b64 to encode and decode 5 images, the cpu does it's work encoding/decoding comes with a cost, just that it's not noticable on the smaller scale. The reason for there being url-form-encoded is to optimize image transfers over http image= file transfers anyhow that's me being over-conscious of the costs 👋 well whatever fits your need is a good solution

elpupperOP•2y ago

hope someone stumbles upon this and finds it useful and if they have a better solution they should tag me

Grey•2y ago

maybe I'll do it IF I ever decide to handle file uploads in this interesting manner I know the better solution...... just that I'm a bit too sleepy to spin up a fresh trpc server to provide it The one I mentioned before: updating your ctx||input variable to include the incoming files from the request into the context, which can then be accessed like ctx.prisma or whatever though in the case of the ctx method, you'll have to do zod validation yourself 🤔 which I like 3-5 more LOC

Gaming

Programming

Using JSZip with Zod and tRPC