Alokai•2y ago

Server Middleware - Express.JS body size limit

Hello, We are currently using VSF 2.5 (which as per package-lock.json uses express 4.17.1) to integrate with our product. In doing so I encountered an issue with the body size limit for a post request used to add a product of one of our customers to a database that should hold that information. The error I am receiving when I try to send the JSON body to our API tells me the following: Headers - General Staus Code: 413 RuntimeError Headers - Request Headers Content-Length: 115309 Judging by the fact that the message is "request entity too large" and the expressjs documentation telling me, that the standard body size limit for express.json() is "100kb", it leads me to believe that the issue is obviously the content size of about 115kb here. The question/-s I find myself with now is, where do I tell VSF to use for example "200kb" as the limit for its Server Middleware? Or do I need to do that with an extension? Reading the documentation I found this article in the docs: https://docs.vuestorefront.io/v2/architecture/server-middleware.html This mentions in its "Configuration" section the "integration interface". So checking that out, its "extensions" seem the way to handle that. To be more precise the optional "extendApp" function seems to be what I should use for that? Since it accepts an "app: Express" and a "configuration". I checked the code in my node_modules folder which leads me to the createServer.ts file in "@vue-storefront/middleware/src" But after all this I am still not sure, how to increase the limit of the express.json() for it. The thing to do in expressjs would be "app.use(express.json({ limit: "200kb" })). Though where do I set this so it uses it? I hope somebody can give a lead into the right direction or tell me if I am on the wrong track.

Server Middleware | Vue Storefront 2

Vue Storefront 2 documentation

22 Replies

rohrig•2y ago

Hi @Mario Schrattenecker 👋 , did setting app.use(express.json({limit: '200kb'})); just before app.listen no have an effect?

Mario SchratteneckerOP•2y ago

Hi @.rohrig , I tried the following: I went to change my middleware.config.js file in the project root and added an extension, that had only the properties "name" and "extendApp". The "extendApp" function had the following form

(params) => { params.app.use(express.json({ limit: "200kb" })) }

(params) => { params.app.use(express.json({ limit: "200kb" })) }

and still it did not change the fact I received the same error when testing. Debugging that showed me that indeed it seemed to consume this configuration. And I am a bit lost which app.listen you are refering to? If you e refering to the "Seperating.." section I wanted to avaoid doing that and stick as close to the standard as possible.

rohrig•2y ago

I'm referring to the app.listen in the middleware.js file

Mario SchratteneckerOP•2y ago

Thanks, I guess I will need to give it a shoot to seperate the server middleware from nuxt.

rohrig•2y ago

I'm not sure I follow. . . What do you mean seperate the server middleware from nuxt ? It's already a separate express.js app, right?

Mario SchratteneckerOP•2y ago

I did not have a middleware.js file, only a middleware.config.js file in the project. The .config contains this (replace some of those with specific names of course)

module.exports = {
  integrations: {
    <integrationName>: {
      location: '<location>',
      configuration: {
        api: {
          authHost: <authHostUrl>,
          client_id: clientId,
          client_secret: clientSecret,
          uri: `${backendHost}`
        },
        cookies: {
          authCookieName: 'vsf-eos-auth'
        }
      }
    }
  }
};

module.exports = {
  integrations: {
    <integrationName>: {
      location: '<location>',
      configuration: {
        api: {
          authHost: <authHostUrl>,
          client_id: clientId,
          client_secret: clientSecret,
          uri: `${backendHost}`
        },
        cookies: {
          authCookieName: 'vsf-eos-auth'
        }
      }
    }
  }
};

From what I can gather, the documentation tells me, that if I create a middleware.js file it will enable me to run the server as a seperate process and configure that. This, was not in the project up until now.

rohrig•2y ago

okay, I'm not familiar with that setup. I'm curious is there a "start:middleware" in your package.json,? and if so, what does the it call?

Mario SchratteneckerOP•2y ago

The package.json contains only one start

"scripts": {
    ...,
    "start": "nuxt start",
    ...
}

"scripts": {
    ...,
    "start": "nuxt start",
    ...
}

I somehow get the feeling that my predecessor who setup the project base, made some interresting choices here if you do not recognize that setup?

rohrig•2y ago

I couldn't say without more info. The middleware must be hosted somewhere. try searching app.listen() or just listen() or maybe you have it documented somewhere? It didn't occur to me to ask before. Are you an enterprise customer?

Mario SchratteneckerOP•2y ago

We are not an enterprise customer, no. Why are you asking? I checked for the .listen() already in the "api-client" package. What that package provides though is the index.server.ts file. As far as I know this file provides the client (in our case axios) for the communication with an external API. That file exports the following:

const { createApiClient } = apiClientFactory({
  onCreate,
  api: CombinedAPIEndpoints,
  extensions: [tokenExtension]
});

const { createApiClient } = apiClientFactory({
  onCreate,
  api: CombinedAPIEndpoints,
  extensions: [tokenExtension]
});

rohrig•2y ago

I thought perhaps the middleware might be hosted by us.

Mario SchratteneckerOP•2y ago

No, we do that ourselves since we have multiple instances of our product running on different servers and each could have a connection to their respective VSF instance.

rohrig•2y ago

ah, okay. So the middleware is hosted separately, it appears So, it seems to me, that the way to check if app.use(express.json({limit: '200kb'})); will work is to find where the middleware is hosted and add it there.

Mario SchratteneckerOP•2y ago

Okay, I went and checked some things again. Checking the nuxt config file, it seems the middleware is run as an extension to the nuxt connect server. Would you suggest to host the middleware as a seperate process than?

rohrig•2y ago

I'm lazy, so I suggest doing whatever works, and is the least amount of work for you 😄

Mario SchratteneckerOP•2y ago

Okay, okay^^ I will check my options and just go with what ultimately works. Thank you for your time.

rohrig•2y ago

np, please let me know what works for you.

Mario SchratteneckerOP•2y ago

I will post here what finally worked once I figured it out.

Mario SchratteneckerOP•2y ago

@here Returning here, since I still did not find a working solution yet. I checked several things and found the following: It seems like the issue is connected to the "createServer" function from @vue-storefront/middleware package(?). Looking at the source code inside there, the expressjs it creates uses

app.use(express.json())

app.use(express.json())

so it uses the defaults. Manually changing that line inside the package to use the mentioned

app.use(express.json({ limit: "1mb" }))

app.use(express.json({ limit: "1mb" }))

lets me pass the json which exceeds the 100kb default limit. I tried to have the extension I am already using in the custom integration, extend the ExpressJS via the "extendApp" function and have the "app" passed to that run the

app.use(express.json({ limit: "1mb" }))

app.use(express.json({ limit: "1mb" }))

, still to no avail. The middleware is definitely run as an extension of the underlying Nuxt Server. Inside the nuxt.config.js it uses the following settings:

server: {
  port: 3000,
  host: '0.0.0.0'
}

server: {
  port: 3000,
  host: '0.0.0.0'
}

modules: [
  '@vue-storefront/middleware/nuxt'
]

modules: [
  '@vue-storefront/middleware/nuxt'
]

Inside the modules array are more entries, but I did not include those here. Apparently I can not pass any seperate options to the ExpressJS` parser for json when using the vue-storefront/middleware standard to run it? Which I would find strange since the documentation states that the "extendApp" function receives an "app" parameter that is an "Express" (https://docs.vuestorefront.io/v2/reference/api/core.apiclientextension.extendapp.html) I am sorry to bother you again with it but I am lost currently.

Vue Storefront 2

Vue Storefront 2 documentation

Mario SchratteneckerOP•2y ago

Running out of time, I did forgo the attempt to try and change the body size limit of the ExpressJS that results from using the @vue-storefront/middleware/nuxt. I have not found a possibility to change what I needed to, since the first appearance in the stack of the ExpressJS Server of the json parser could not be changed and remained with a limit of 100kb. Only option would have been to host our own express, seperated from the nuxt js connect. I went with seperating the data into smaller chunks and sending it seperately to our server to avoid to big an amount on data. @.rohrig I think this can be closed since I went with a workaround that is good enougth.

rohrig•2y ago

Thank you for keep us up to date 😄

FabianClemenz•2y ago

@Mario Schrattenecker with the help from @skirianov we got it working 🙂 1. in the theme folder create a dir serverMiddleware in that dir create a file body-parser.js with following content:

import bodyParser from 'body-parser';
import express from 'express';

const app = express();

console.log('CUSTOM - updating middleware');

app.use(bodyParser.json({ limit: '1mb' }));

console.log('CUSTOM - middleware updated');

export default app;

import bodyParser from 'body-parser';
import express from 'express';

const app = express();

console.log('CUSTOM - updating middleware');

app.use(bodyParser.json({ limit: '1mb' }));

console.log('CUSTOM - middleware updated');

export default app;

2. in nuxt.config.js add following:

  serverMiddleware: [
    '~/serverMiddleware/body-parser.js'
  ],

  serverMiddleware: [
    '~/serverMiddleware/body-parser.js'
  ],

now you should see the CUSTOM log statements when starting the server 🙂

Gaming

Programming

Server Middleware - Express.JS body size limit