Wasp-lang•3mo ago

How to implement file upload middleware for large video files in a Wasp app?

...trying to implement a video upload feature in our Wasp app that can handle large files (>50MB) using chunked uploads. We need to configure middleware for file uploads, but we're encountering issues with setting up express-fileupload middleware in Wasp. What's the correct way to implement custom middleware for file uploads in a Wasp app, particularly for handling large files with multipart uploads to an S3-compatible storage (Wasabi)?

10 Replies

kapa.ai•3mo ago

Hi @Conspyre. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave: ---------------------- To implement file upload middleware in a Wasp app, you can use the multer library, which is a middleware for handling multipart/form-data, primarily used for uploading files. Here is an example of how you can set up multer for file uploads:

import { MiddlewareConfigFn } from '@wasp/middleware';
import multer from 'multer';

export const apiMiddleware: MiddlewareConfigFn = (config) => {
  const upload = multer();

  config.set('multer', upload.single('file'));
}

import { MiddlewareConfigFn } from '@wasp/middleware';
import multer from 'multer';

export const apiMiddleware: MiddlewareConfigFn = (config) => {
  const upload = multer();

  config.set('multer', upload.single('file'));
}

(Source: Github Issue (closed): Api can't handle FormData > Comments Part 1 (https://github.com/wasp-lang/wasp/issues/1683)) For handling large files and storing them in an S3-compatible storage, you can use multer-s3 or multer-storage-s3 libraries which are multer storage engines for AWS S3. These libraries allow you to stream files directly to S3 which can be beneficial for handling large files. However, please note that the Wasp framework currently does not provide out-of-the-box support for chunked uploads. You would need to implement this functionality yourself, possibly by using a library that supports chunked uploads or by manually implementing the chunked upload process in your middleware. Also, remember to configure CORS (Cross-Origin Resource Sharing) for your APIs to ensure they can be used on the client side. You can do this by defining custom middleware for your APIs in the Wasp file. Here is an example:

apiNamespace fooBar {  
 middlewareConfigFn: import { fooBarNamespaceMiddlewareFn } from "@src/apis",  
 path: "/foo"  
}

apiNamespace fooBar {  
 middlewareConfigFn: import { fooBarNamespaceMiddlewareFn } from "@src/apis",  
 path: "/foo"  
}

(Source: Docs > Advanced > Custom HTTP API Endpoints > Using the API > Using the API from the Client > Making Sure CORS Works (https://wasp-lang.dev/docs/advanced/apis#making-sure-cors-works)) For more information about configuring middleware in Wasp, you can refer to the Middleware Configuration (https://wasp-lang.dev/docs/advanced/middleware-config) section in the Wasp documentation. ---------------------- Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: ---------------------- Sorry @Conspyre, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist: Perhaps try again in 15mins?

ConspyreOP•3mo ago

@kapa.ai trying something like this: apiNamespace api { middlewareConfigFn: import { apiMiddleware } from "@src/server/uploadMiddleware.js", path: "/api" } (omitting the config of s3client/multer), looks like the salient point is: export const apiMiddleware = (config) => { config.set('multer', uploadMiddleware); } But when we start the server, we crash like so: [ Server!] /Users/walt/IdeaProjects/quillsense/app/.wasp/out/server/src/middleware/globalMiddleware.ts:38 [ Server!] return Array.from(modifiedMiddlewareConfig.values()) [ Server!] ^ [ Server!] [ Server!] [ Server!] TypeError: Cannot read properties of undefined (reading 'values') [ Server!] at globalMiddlewareConfigForExpress (/Users/walt/IdeaProjects/quillsense/app/.wasp/out/server/src/middleware/globalMiddleware.ts:38:46) [ Server!] at <anonymous> (/Users/walt/IdeaProjects/quillsense/app/.wasp/out/server/src/routes/apis/index.ts:24:20) [ Server!] at ModuleJob.run (node:internal/modules/esm/module_job:194:25) I cant make heads or tails of what's undefined here.

kapa.ai•3mo ago

Let me check the knowledge base for you. One moment! :books: ---------------------- The error message you're seeing suggests that modifiedMiddlewareConfig is undefined at the time when its values() method is being called. This is likely because the config object that you're trying to modify in your apiMiddleware function is not being correctly initialized or passed in. Here's the relevant part of your code:

export const apiMiddleware = (config) => {
    config.set('multer', uploadMiddleware);
}

export const apiMiddleware = (config) => {
    config.set('multer', uploadMiddleware);
}

In this code, config is expected to be an instance of a MiddlewareConfig object. If config is undefined, then calling config.set() will result in a TypeError. To debug this issue, you could add some logging to your apiMiddleware function to check the value of config:

export const apiMiddleware = (config) => {
    console.log('config:', config);
    config.set('multer', uploadMiddleware);
}

export const apiMiddleware = (config) => {
    console.log('config:', config);
    config.set('multer', uploadMiddleware);
}

If config is undefined, then the issue is likely with how apiMiddleware is being called or how the config object is being passed in. You might need to check the part of your code where apiMiddleware is being used. If you're still having trouble, it might be helpful to refer to the Wasp middleware configuration documentation (https://wasp-lang.dev/docs/advanced/middleware-config) or ask for help in the Wasp Discord community (https://discord.com/channels/686873244791210014/1217681705511686194). ---------------------- Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: ---------------------- Sorry @Conspyre, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist: Perhaps try again in 15mins?

ConspyreOP•3mo ago

@kapa.ai we have a viable config object, however the assumption that it has an element "values" is incorrect. config: Map(6) { [ Server ] 'helmet' => [Function: helmetMiddleware], [ Server ] 'cors' => [Function: corsMiddleware], [ Server ] 'logger' => [Function: logger], [ Server ] 'express.json' => [Function: jsonParser], [ Server ] 'express.urlencoded' => [Function: urlencodedParser], [ Server ] 'cookieParser' => [Function: cookieParser] [ Server ] }

kapa.ai•3mo ago

Let me check the knowledge base for you. One moment! :books: ---------------------- The config object in your middleware function is a JavaScript Map object. The values() method is a built-in method of the Map object, which returns a new iterator object that contains the values for each element in the Map object in insertion order. The error message you're seeing suggests that modifiedMiddlewareConfig is undefined at the time when its values() method is being called. This is likely because the config object that you're trying to modify in your apiMiddleware function is not being correctly initialized or passed in. Here's the relevant part of your code:

export const apiMiddleware = (config) => {{
    console.log('config:', config);
    config.set('multer', uploadMiddleware);
}}

export const apiMiddleware = (config) => {{
    console.log('config:', config);
    config.set('multer', uploadMiddleware);
}}

export const apiMiddleware = (config) => {{
    console.log('config:', config);
    config.set('multer', uploadMiddleware);
}}

export const apiMiddleware = (config) => {{
    console.log('config:', config);
    config.set('multer', uploadMiddleware);
}}

miho•3mo ago

You have to return the config object after you are done with editing it 🙂

import { type MiddlewareConfigFn } from 'wasp/server'

export const apiMiddleware: MiddlewareConfigFn = (config) => {{
    console.log('config:', config);
    config.set('multer', uploadMiddleware);
    return config;
}}

import { type MiddlewareConfigFn } from 'wasp/server'

export const apiMiddleware: MiddlewareConfigFn = (config) => {{
    console.log('config:', config);
    config.set('multer', uploadMiddleware);
    return config;
}}

I've added MiddlewareConfigFn to get the type safety. Check the relevant docs: https://wasp-lang.dev/docs/advanced/middleware-config#2-customize-api-specific-middleware

Configuring Middleware | Wasp

Wasp comes with a minimal set of useful Express middleware in every application. While this is good for most users, we realize some may wish to add, modify, or remove some of these choices both globally, or on a per-api/path basis.

ConspyreOP•3mo ago

I have this semi-functional. I can upload to my wasabi bucket just fine. I don't need middleware to achieve it. The Prod setup is to use FUSE to mount the filesystem so we don't have to load GB of video into RAM. This works great. Two thumbs up. I'm trying to fake it on a dev setup, and because there appears not to be a viable FUSE for Mac OS, trying to jury rig it to upload to the local file system. I've tried numerous dead ends.. Any clues how to do something so hacky @kapa.ai?

kapa.ai•3mo ago

Let me check the knowledge base for you. One moment! :books: ---------------------- If you're trying to simulate the behavior of a mounted filesystem on your local development setup, you might consider using a local directory as a stand-in for the S3 bucket. You can configure your upload logic to write files to this local directory when running in a development environment. Here's a rough idea of how you might set this up:

import multer from 'multer';
import path from 'path';
import fs from 'fs';

// Set up multer
const storage = multer.diskStorage({
  destination: function (req, file, cb) {
    // Replace 'local_uploads' with the path to your local directory
    cb(null, 'local_uploads')
  },
  filename: function (req, file, cb) {
    cb(null, file.fieldname + '-' + Date.now())
  }
})

const upload = multer({ storage: storage })

// Use the upload middleware in your route
app.post('/upload', upload.single('file'), (req, res) => {
  // Your upload logic here
});

import multer from 'multer';
import path from 'path';
import fs from 'fs';

// Set up multer
const storage = multer.diskStorage({
  destination: function (req, file, cb) {
    // Replace 'local_uploads' with the path to your local directory
    cb(null, 'local_uploads')
  },
  filename: function (req, file, cb) {
    cb(null, file.fieldname + '-' + Date.now())
  }
})

const upload = multer({ storage: storage })

// Use the upload middleware in your route
app.post('/upload', upload.single('file'), (req, res) => {
  // Your upload logic here
});

In this example, multer.diskStorage is used to configure multer to save uploaded files to a local directory. You can replace 'local_uploads' with the path to the directory you want to use. Please note that this is a simplified example and might not cover all your use cases. You'll likely need to adjust this code to fit your application's needs. Also, remember that storing large amounts of data on your local machine can quickly use up your available storage space. You'll want to periodically clean up this directory to prevent it from filling up. If you're looking for a more robust solution for simulating S3 on your local machine, you might consider using a tool like localstack (https://github.com/localstack/localstack), which provides a fully functional local AWS cloud stack, or minio (https://min.io/), which is a high performance, distributed object storage server. These tools can give you a more accurate representation of an S3 environment for local development. ---------------------- Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: ---------------------- Sorry @Conspyre, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist: Perhaps try again in 15mins?

miho•3mo ago

I'm trying to fake it on a dev setup, and because there appears not to be a viable FUSE for Mac OS, trying to jury rig it to upload to the local file system. I've tried numerous dead ends.. Any clues how to do something so hacky @kapa.ai?

I'm not sure what your exact question is, how to use FUSE on Mac OS?

kapa.ai•3mo ago

Let me check the knowledge base for you. One moment! :books: ---------------------- I'm sorry for the confusion, but based on the provided knowledge sources, I don't have specific information on how to use FUSE on MacOS or how to simulate it for local file uploads in a Wasp app. However, if you're trying to simulate file uploads in a local development environment, you might consider using a local directory as a stand-in for the S3 bucket. You can configure your upload logic to write files to this local directory when running in a development environment. If you're looking for a more robust solution for simulating S3 on your local machine, you might consider using a tool like localstack (https://github.com/localstack/localstack), which provides a fully functional local AWS cloud stack, or minio (https://min.io/), which is a high performance, distributed object storage server. These tools can give you a more accurate representation of an S3 environment for local development. For more specific guidance, I would recommend reaching out to the Wasp community or the maintainers of the Wasp project directly. They might be able to provide more detailed advice based on their knowledge of the framework and its capabilities. ---------------------- Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: ---------------------- Sorry @miho, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist: Perhaps try again in 15mins?

Gaming

Programming

How to implement file upload middleware for large video files in a Wasp app?