R
Railwayβ€’3mo ago
Budi

Add caching to Node Dockerfile

I want to ask for feedback on my Dockerfile. This is the first time I've set up caching. This Dockerfile works. My first deploy of this file was 143 secs, the second 124 secs. But that's still really slow IMO for that second try. Have I set this up properly for Node and PNPM as package manager? Is there anything else I can do to speed these builds up? I also noticed it's required to hardcode your session-id. I've now bound this to the service-id of my main production environment in Railway. What are the implication of this when a different environment is created in Railway for your project, for example when you create a new PR on Github? Thanks!
# Use Node.js as the base image
FROM node:20-alpine

# Install pnpm
RUN npm install -g pnpm

# Set working directory
WORKDIR /app

# Install dotenvx
RUN curl -sfS https://dotenvx.sh/install.sh | sh

# Copy package.json and pnpm-lock.yaml
COPY package.json pnpm-lock.yaml ./

# Install dependencies with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.local/share/pnpm/store,target=/root/.local/share/pnpm/store \
pnpm install --frozen-lockfile

# Copy the rest of the application code
COPY . .

# Specify the environment variables needed at build time
ARG DOTENV_PRIVATE_KEY_PRODUCTION

# Build the application with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.cache/pnpm,target=/root/.cache/pnpm \
pnpm run build

# Expose the port the app runs on
EXPOSE 3000

# Start the application
CMD ["pnpm", "start"]
# Use Node.js as the base image
FROM node:20-alpine

# Install pnpm
RUN npm install -g pnpm

# Set working directory
WORKDIR /app

# Install dotenvx
RUN curl -sfS https://dotenvx.sh/install.sh | sh

# Copy package.json and pnpm-lock.yaml
COPY package.json pnpm-lock.yaml ./

# Install dependencies with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.local/share/pnpm/store,target=/root/.local/share/pnpm/store \
pnpm install --frozen-lockfile

# Copy the rest of the application code
COPY . .

# Specify the environment variables needed at build time
ARG DOTENV_PRIVATE_KEY_PRODUCTION

# Build the application with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.cache/pnpm,target=/root/.cache/pnpm \
pnpm run build

# Expose the port the app runs on
EXPOSE 3000

# Start the application
CMD ["pnpm", "start"]
37 Replies
Percy
Percyβ€’3mo ago
Project ID: e87cdb12-ca40-472a-b18b-fc6f19c34f3f,e87cdb12-ca40-472a-b18b-fc6f19c34f3f
Brody
Brodyβ€’3mo ago
what takes the most time to run?
Budi
Budiβ€’3mo ago
I believe you'd shared a tool somewhere with which I could extract build logs. Would that be helpful? Building takes ~80 secs Here I filtered the logs for done:
#12 80.01 βœ” done

#12 DONE 80.7s

#13 exporting layers 15.2s done

#13 exporting manifest sha256:7a3c7f748177c383ddcb646668c4feda059f1169a216e894fd1002ea0f51934c 0.3s done

#13 exporting config sha256:7b82807699583bf53ac0eb6662ad7c2becd54cbb7e53ac67cbd19706e62f3a35 done

#13 exporting attestation manifest sha256:6dcc37b278a65409922832067c6f3c92afec59e903491135fa55874c085501ea done

#13 exporting manifest list sha256:4faeed252081451d247c1a960c7e55a06ccba801bac34cdf1dd2f093d8f353e5 done

#14 DONE 0.0s

#13 pushing layers 15.7s done

#13 pushing manifest for us-west1.registry.rlwy.net/e87cdb12-ca40-472a-b18b-fc6f19c34f3f:b7081d8b-891b-480a-a771-b6c31bd57b36@sha256:4faeed252081451d247c1a960c7e55a06ccba801bac34cdf1dd2f093d8f353e5 1.3s done

#13 DONE 32.5s
#12 80.01 βœ” done

#12 DONE 80.7s

#13 exporting layers 15.2s done

#13 exporting manifest sha256:7a3c7f748177c383ddcb646668c4feda059f1169a216e894fd1002ea0f51934c 0.3s done

#13 exporting config sha256:7b82807699583bf53ac0eb6662ad7c2becd54cbb7e53ac67cbd19706e62f3a35 done

#13 exporting attestation manifest sha256:6dcc37b278a65409922832067c6f3c92afec59e903491135fa55874c085501ea done

#13 exporting manifest list sha256:4faeed252081451d247c1a960c7e55a06ccba801bac34cdf1dd2f093d8f353e5 done

#14 DONE 0.0s

#13 pushing layers 15.7s done

#13 pushing manifest for us-west1.registry.rlwy.net/e87cdb12-ca40-472a-b18b-fc6f19c34f3f:b7081d8b-891b-480a-a771-b6c31bd57b36@sha256:4faeed252081451d247c1a960c7e55a06ccba801bac34cdf1dd2f093d8f353e5 1.3s done

#13 DONE 32.5s
so it's the building, exporting and pushing of layers, and pushing the manifest which take the most time.
Budi
Budiβ€’3mo ago
Can't get the bookmark to work with Arc browser and injecting the JS via console doesn't work either.
Brody
Brodyβ€’3mo ago
ctrl / cmd + k -> logs
Budi
Budiβ€’3mo ago
Does that work?
Brody
Brodyβ€’3mo ago
since its not the full logs, nope doesnt work okay that one works
Budi
Budiβ€’3mo ago
Sweet πŸ‘πŸΌ
Brody
Brodyβ€’3mo ago
do you happen to know how big this image is?
Budi
Budiβ€’3mo ago
How can I find out?
Brody
Brodyβ€’3mo ago
by building locally
Budi
Budiβ€’3mo ago
Once I do that, where can I find the image? Does it get created in my project folder? Sorry I'm new to Docker. How do I build the Docker file locally using Railway's cache? Do I have to create a new Dockerfile to build locally without the caching statements?
Brody
Brodyβ€’3mo ago
you dont build it with railway's cache, nor is that something you can do, you simply build the image, you shouldnt need to change anything
Budi
Budiβ€’2mo ago
OK I will try. I had to make a bunch of changes to the Dockerfile to get it working locally, but got the local creation of the image done. The size of the image is 1.16GB. Here's the updated Dockerfile:
# Build stage
FROM node:20-alpine AS builder

# Install pnpm
RUN npm install -g pnpm

# Set working directory
WORKDIR /app

# Install dotenvx
RUN curl -sfS https://dotenvx.sh/install.sh | sh

# Copy package.json and pnpm-lock.yaml
COPY package.json pnpm-lock.yaml ./

# Install dependencies with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.local/share/pnpm/store,target=/root/.local/share/pnpm/store \
pnpm install --frozen-lockfile

# Copy the rest of the application code
COPY . .

# Specify the environment variables needed at build time
ARG DOTENV_PRIVATE_KEY_PRODUCTION

# Set Node options for increased memory
ENV NODE_OPTIONS="--max-old-space-size=4096"

# Build the application with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.cache/pnpm,target=/root/.cache/pnpm \
NODE_ENV=production pnpm run build

# Production stage
FROM node:20-alpine

# Install pnpm in the production stage
RUN npm install -g pnpm

# Set working directory
WORKDIR /app

# Copy the build output and package.json from the builder stage
COPY --from=builder /app/build ./build
COPY --from=builder /app/package.json ./package.json
COPY --from=builder /app/pnpm-lock.yaml ./pnpm-lock.yaml

# Install only production dependencies
RUN pnpm install --prod

# Expose the port the app runs on
EXPOSE 3000

# Start the application
CMD ["pnpm", "start"]
# Build stage
FROM node:20-alpine AS builder

# Install pnpm
RUN npm install -g pnpm

# Set working directory
WORKDIR /app

# Install dotenvx
RUN curl -sfS https://dotenvx.sh/install.sh | sh

# Copy package.json and pnpm-lock.yaml
COPY package.json pnpm-lock.yaml ./

# Install dependencies with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.local/share/pnpm/store,target=/root/.local/share/pnpm/store \
pnpm install --frozen-lockfile

# Copy the rest of the application code
COPY . .

# Specify the environment variables needed at build time
ARG DOTENV_PRIVATE_KEY_PRODUCTION

# Set Node options for increased memory
ENV NODE_OPTIONS="--max-old-space-size=4096"

# Build the application with caching
RUN --mount=type=cache,id=s/e87cdb12-ca40-472a-b18b-fc6f19c34f3f-/root/.cache/pnpm,target=/root/.cache/pnpm \
NODE_ENV=production pnpm run build

# Production stage
FROM node:20-alpine

# Install pnpm in the production stage
RUN npm install -g pnpm

# Set working directory
WORKDIR /app

# Copy the build output and package.json from the builder stage
COPY --from=builder /app/build ./build
COPY --from=builder /app/package.json ./package.json
COPY --from=builder /app/pnpm-lock.yaml ./pnpm-lock.yaml

# Install only production dependencies
RUN pnpm install --prod

# Expose the port the app runs on
EXPOSE 3000

# Start the application
CMD ["pnpm", "start"]
What do you think of the new Dockerfile? And are there any ways you think I can speed up builds?
Medim
Medimβ€’2mo ago
#πŸ›‚ο½œreadme 5) don't ping team or conductors 1.16gb is still big but totally doable imo
Budi
Budiβ€’2mo ago
Oh sorry! I'll remove the tag. Done πŸ™‚ Do you think there are any further optimisations I could make?
Medim
Medimβ€’2mo ago
πŸ€” I don't think so, I never had to optimize the size of my images so I can't really say
Brody
Brodyβ€’2mo ago
you can definitely get it smaller than 1.16GB. it's also possible your mount targets are incorrect, aka not the actual path pnpm stores the cache at, so nothing actually gets cached
Budi
Budiβ€’2mo ago
How can I reduce the size of the image? And how would I know where to find the correct mount targets?
Budi
Budiβ€’2mo ago
I referenced this https://github.com/railwayapp/nixpacks/blob/main/src/providers/node/mod.rs thinking this was the right example for node and found pnpm cache mentioned as such: const PNPM_CACHE_DIR: &str = "/root/.local/share/pnpm/store/v3"
GitHub
nixpacks/src/providers/node/mod.rs at main Β· railwayapp/nixpacks
App source + Nix packages + Docker = Image. Contribute to railwayapp/nixpacks development by creating an account on GitHub.
Budi
Budiβ€’2mo ago
Not sure if I derived my mounts correctly from that.
jeremy
jeremyβ€’2mo ago
these articles (part 1 & 2) were pretty good walkthrough on how to reduce image sizes https://lengrand.fr/reducing-dockers-image-size-while-creating-an-offline-version-of-carbon-now-sh/ mine went from ~900MB to ~350MB after optimization
Budi
Budiβ€’2mo ago
Nice I incorporated some of the tips. Does the Slim package work with Railway too? Asking since it seems CLI driven so not sure if you could incorporate that into the build file.
Julien's DevRel corner
Reducing our Carbon Docker image size further!
Second post of the series, we keep diving into more ways to reduce our Docker image size. Let's see how we manage to reach a factor 10!
Brody
Brodyβ€’2mo ago
yes no issues with the -slim node image variants
Budi
Budiβ€’2mo ago
How would I incorporate that into the creation of my image? Can I run docker-slim in the Dockerfile somehow? Any hints or example repos I could take a look at, perhaps?
Brody
Brodyβ€’2mo ago
oh sorry i thought you wanted to use slim images, no railway does not support running docker slim
Budi
Budiβ€’2mo ago
I thought so -- thanks for clarifying. I was about to halve the size of the image though by pruning the node and pnpm modules in the build phase, and then explicitly copying the node_modules in the production step instead of re-installing the dependencies there. πŸŽ‰ Any feedback on this? How do I know if I'm using the right mount targets?
Brody
Brodyβ€’2mo ago
im sure you arent the first to do this on the vast internet, and it wouldn't be specific to railway
Budi
Budiβ€’2mo ago
Hint taken. I will do more research. Also I had this remaining question:
I also noticed it's required to hardcode your session-id. I've now bound this to the session-id of my main production environment in Railway. What are the implication of this when a different environment is created in Railway for your project, for example when you create a new PR on Github?
Brody
Brodyβ€’2mo ago
you mean service id and in that case, it will fail
Budi
Budiβ€’2mo ago
But so far, when I've been testing these builds and put my new code in a new PR, a new Railway environment was started which deployed just fine with the Dockerfile that had my service-id included. I don't understand the limitations of this properly. Is the idea that I hardcode the service-id of each Railway production environment that I'll be deploying the image in?
Brody
Brodyβ€’2mo ago
again, service id, not session id
Budi
Budiβ€’2mo ago
Sorry yes, typo. But my question still stands.
Brody
Brodyβ€’2mo ago
yes you do need to hardcore that service id for every service you deploy to
Budi
Budiβ€’2mo ago
I see. Understood.
Want results from more Discord servers?
Add your server