Railway•3mo ago

Add caching to Node Dockerfile

I want to ask for feedback on my Dockerfile. This is the first time I've set up caching. This Dockerfile works. My first deploy of this file was 143 secs, the second 124 secs. But that's still really slow IMO for that second try. Have I set this up properly for Node and PNPM as package manager? Is there anything else I can do to speed these builds up? I also noticed it's required to hardcode your session-id. I've now bound this to the service-id of my main production environment in Railway. What are the implication of this when a different environment is created in Railway for your project, for example when you create a new PR on Github? Thanks!

# Use Node.js as the base image
FROM node:20-alpine

# Install pnpm
RUN npm install -g pnpm

# Set working directory
WORKDIR /app

# Install dotenvx
RUN curl -sfS https://dotenvx.sh/install.sh | sh

# Copy package.json and pnpm-lock.yaml
COPY package.json pnpm-lock.yaml ./

# Install dependencies with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.local/share/pnpm/store,target=/root/.local/share/pnpm/store \
    pnpm install --frozen-lockfile

# Copy the rest of the application code
COPY . .

# Specify the environment variables needed at build time
ARG DOTENV_PRIVATE_KEY_PRODUCTION

# Build the application with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.cache/pnpm,target=/root/.cache/pnpm \
    pnpm run build

# Expose the port the app runs on
EXPOSE 3000

# Start the application
CMD ["pnpm", "start"]

# Use Node.js as the base image
FROM node:20-alpine

# Install pnpm
RUN npm install -g pnpm

# Set working directory
WORKDIR /app

# Install dotenvx
RUN curl -sfS https://dotenvx.sh/install.sh | sh

# Copy package.json and pnpm-lock.yaml
COPY package.json pnpm-lock.yaml ./

# Install dependencies with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.local/share/pnpm/store,target=/root/.local/share/pnpm/store \
    pnpm install --frozen-lockfile

# Copy the rest of the application code
COPY . .

# Specify the environment variables needed at build time
ARG DOTENV_PRIVATE_KEY_PRODUCTION

# Build the application with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.cache/pnpm,target=/root/.cache/pnpm \
    pnpm run build

# Expose the port the app runs on
EXPOSE 3000

# Start the application
CMD ["pnpm", "start"]

37 Replies

Percy•3mo ago

Project ID: e87cdb12-ca40-472a-b18b-fc6f19c34f3f,e87cdb12-ca40-472a-b18b-fc6f19c34f3f

Brody•3mo ago

what takes the most time to run?

BudiOP•3mo ago

I believe you'd shared a tool somewhere with which I could extract build logs. Would that be helpful? Building takes ~80 secs Here I filtered the logs for done:

#12 80.01   ✔ done

#12 DONE 80.7s

#13 exporting layers 15.2s done

#13 exporting manifest sha256:7a3c7f748177c383ddcb646668c4feda059f1169a216e894fd1002ea0f51934c 0.3s done

#13 exporting config sha256:7b82807699583bf53ac0eb6662ad7c2becd54cbb7e53ac67cbd19706e62f3a35 done

#13 exporting attestation manifest sha256:6dcc37b278a65409922832067c6f3c92afec59e903491135fa55874c085501ea done

#13 exporting manifest list sha256:4faeed252081451d247c1a960c7e55a06ccba801bac34cdf1dd2f093d8f353e5 done

#14 DONE 0.0s

#13 pushing layers 15.7s done

#13 pushing manifest for us-west1.registry.rlwy.net/e87cdb12-ca40-472a-b18b-fc6f19c34f3f:b7081d8b-891b-480a-a771-b6c31bd57b36@sha256:4faeed252081451d247c1a960c7e55a06ccba801bac34cdf1dd2f093d8f353e5 1.3s done

#13 DONE 32.5s

#12 80.01   ✔ done

#12 DONE 80.7s

#13 exporting layers 15.2s done

#13 exporting manifest sha256:7a3c7f748177c383ddcb646668c4feda059f1169a216e894fd1002ea0f51934c 0.3s done

#13 exporting config sha256:7b82807699583bf53ac0eb6662ad7c2becd54cbb7e53ac67cbd19706e62f3a35 done

#13 exporting attestation manifest sha256:6dcc37b278a65409922832067c6f3c92afec59e903491135fa55874c085501ea done

#13 exporting manifest list sha256:4faeed252081451d247c1a960c7e55a06ccba801bac34cdf1dd2f093d8f353e5 done

#14 DONE 0.0s

#13 pushing layers 15.7s done

#13 pushing manifest for us-west1.registry.rlwy.net/e87cdb12-ca40-472a-b18b-fc6f19c34f3f:b7081d8b-891b-480a-a771-b6c31bd57b36@sha256:4faeed252081451d247c1a960c7e55a06ccba801bac34cdf1dd2f093d8f353e5 1.3s done

#13 DONE 32.5s

so it's the building, exporting and pushing of layers, and pushing the manifest which take the most time.

Brody•3mo ago

https://bookmarklets.up.railway.app/log-downloader/

BudiOP•3mo ago

Can't get the bookmark to work with Arc browser and injecting the JS via console doesn't work either.

Brody•3mo ago

ctrl / cmd + k -> logs

BudiOP•3mo ago

Does that work?

BudiOP•3mo ago

build_logs_sveltecar...

Brody•3mo ago

since its not the full logs, nope doesnt work okay that one works

BudiOP•3mo ago

Sweet 👏🏼

Brody•3mo ago

do you happen to know how big this image is?

BudiOP•3mo ago

How can I find out?

Brody•3mo ago

by building locally

BudiOP•3mo ago

Once I do that, where can I find the image? Does it get created in my project folder? Sorry I'm new to Docker. How do I build the Docker file locally using Railway's cache? Do I have to create a new Dockerfile to build locally without the caching statements?

Brody•3mo ago

you dont build it with railway's cache, nor is that something you can do, you simply build the image, you shouldnt need to change anything

BudiOP•3mo ago

OK I will try. I had to make a bunch of changes to the Dockerfile to get it working locally, but got the local creation of the image done. The size of the image is 1.16GB. Here's the updated Dockerfile:

# Build stage
FROM node:20-alpine AS builder

# Install pnpm
RUN npm install -g pnpm

# Set working directory
WORKDIR /app

# Install dotenvx
RUN curl -sfS https://dotenvx.sh/install.sh | sh

# Copy package.json and pnpm-lock.yaml
COPY package.json pnpm-lock.yaml ./

# Install dependencies with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.local/share/pnpm/store,target=/root/.local/share/pnpm/store \
    pnpm install --frozen-lockfile

# Copy the rest of the application code
COPY . .

# Specify the environment variables needed at build time
ARG DOTENV_PRIVATE_KEY_PRODUCTION

# Set Node options for increased memory
ENV NODE_OPTIONS="--max-old-space-size=4096"

# Build the application with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.cache/pnpm,target=/root/.cache/pnpm \
    NODE_ENV=production pnpm run build

# Production stage
FROM node:20-alpine

# Install pnpm in the production stage
RUN npm install -g pnpm

# Set working directory
WORKDIR /app

# Copy the build output and package.json from the builder stage
COPY --from=builder /app/build ./build
COPY --from=builder /app/package.json ./package.json
COPY --from=builder /app/pnpm-lock.yaml ./pnpm-lock.yaml

# Install only production dependencies
RUN pnpm install --prod

# Expose the port the app runs on
EXPOSE 3000

# Start the application
CMD ["pnpm", "start"]

# Build stage
FROM node:20-alpine AS builder

# Install pnpm
RUN npm install -g pnpm

# Set working directory
WORKDIR /app

# Install dotenvx
RUN curl -sfS https://dotenvx.sh/install.sh | sh

# Copy package.json and pnpm-lock.yaml
COPY package.json pnpm-lock.yaml ./

# Install dependencies with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.local/share/pnpm/store,target=/root/.local/share/pnpm/store \
    pnpm install --frozen-lockfile

# Copy the rest of the application code
COPY . .

# Specify the environment variables needed at build time
ARG DOTENV_PRIVATE_KEY_PRODUCTION

# Set Node options for increased memory
ENV NODE_OPTIONS="--max-old-space-size=4096"

# Build the application with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.cache/pnpm,target=/root/.cache/pnpm \
    NODE_ENV=production pnpm run build

# Production stage
FROM node:20-alpine

# Install pnpm in the production stage
RUN npm install -g pnpm

# Set working directory
WORKDIR /app

# Copy the build output and package.json from the builder stage
COPY --from=builder /app/build ./build
COPY --from=builder /app/package.json ./package.json
COPY --from=builder /app/pnpm-lock.yaml ./pnpm-lock.yaml

# Install only production dependencies
RUN pnpm install --prod

# Expose the port the app runs on
EXPOSE 3000

# Start the application
CMD ["pnpm", "start"]

What do you think of the new Dockerfile? And are there any ways you think I can speed up builds?

Medim•3mo ago

#🛂｜readme 5) don't ping team or conductors 1.16gb is still big but totally doable imo

BudiOP•3mo ago

Oh sorry! I'll remove the tag. Done 🙂 Do you think there are any further optimisations I could make?

Medim•3mo ago

🤔 I don't think so, I never had to optimize the size of my images so I can't really say

Brody•3mo ago

you can definitely get it smaller than 1.16GB. it's also possible your mount targets are incorrect, aka not the actual path pnpm stores the cache at, so nothing actually gets cached

BudiOP•3mo ago

How can I reduce the size of the image? And how would I know where to find the correct mount targets?

BudiOP•3mo ago

I referenced this https://github.com/railwayapp/nixpacks/blob/main/src/providers/node/mod.rs thinking this was the right example for node and found pnpm cache mentioned as such: const PNPM_CACHE_DIR: &str = "/root/.local/share/pnpm/store/v3"

GitHub

nixpacks/src/providers/node/mod.rs at main · railwayapp/nixpacks

App source + Nix packages + Docker = Image. Contribute to railwayapp/nixpacks development by creating an account on GitHub.

BudiOP•3mo ago

Not sure if I derived my mounts correctly from that.

jeremy•3mo ago

these articles (part 1 & 2) were pretty good walkthrough on how to reduce image sizes https://lengrand.fr/reducing-dockers-image-size-while-creating-an-offline-version-of-carbon-now-sh/ mine went from ~900MB to ~350MB after optimization

BudiOP•3mo ago

Nice I incorporated some of the tips. Does the Slim package work with Railway too? Asking since it seems CLI driven so not sure if you could incorporate that into the build file.

Julien's DevRel corner

Reducing our Carbon Docker image size further!

Second post of the series, we keep diving into more ways to reduce our Docker image size. Let's see how we manage to reach a factor 10!

Brody•3mo ago

yes no issues with the -slim node image variants

BudiOP•3mo ago

How would I incorporate that into the creation of my image? Can I run docker-slim in the Dockerfile somehow? Any hints or example repos I could take a look at, perhaps?

Brody•3mo ago

oh sorry i thought you wanted to use slim images, no railway does not support running docker slim

BudiOP•3mo ago

I thought so -- thanks for clarifying. I was about to halve the size of the image though by pruning the node and pnpm modules in the build phase, and then explicitly copying the node_modules in the production step instead of re-installing the dependencies there. 🎉 Any feedback on this? How do I know if I'm using the right mount targets?

Brody•3mo ago

im sure you arent the first to do this on the vast internet, and it wouldn't be specific to railway

BudiOP•3mo ago

Hint taken. I will do more research. Also I had this remaining question:

I also noticed it's required to hardcode your session-id. I've now bound this to the session-id of my main production environment in Railway. What are the implication of this when a different environment is created in Railway for your project, for example when you create a new PR on Github?

Brody•3mo ago

you mean service id and in that case, it will fail

BudiOP•3mo ago

But so far, when I've been testing these builds and put my new code in a new PR, a new Railway environment was started which deployed just fine with the Dockerfile that had my service-id included. I don't understand the limitations of this properly. Is the idea that I hardcode the service-id of each Railway production environment that I'll be deploying the image in?

Brody•3mo ago

again, service id, not session id

BudiOP•3mo ago

Sorry yes, typo. But my question still stands.

Brody•3mo ago

yes you do need to hardcore that service id for every service you deploy to

BudiOP•3mo ago

I see. Understood.

Gaming

Programming

Add caching to Node Dockerfile