Zerops•6mo ago

Monorepos

How would i go about setting up zerops.yml in a monorepo, which has multiple packages that can be built? I'm using Astro and nodejs server, the frontend lies in apps/frontend dir and I've added all the pnpm commands to the root package.json. Is this the correct way?

zerops:
  - setup: daliborhondotdev
    # ==== how to build your application ====
    build:
      # what technologies should the build
      # container be based on (can be an array)
      base: nodejs@22

      # what commands to use to build your app
      buildCommands:
        - pnpm --filter @daliborhon.dev/frontend install
        - pnpm run build:frontend

      # select which files / folders to deploy
      # after the build succesfully finished
      deployFiles:
        - apps/frontend/dist
        - apps/frontend/package.json
        - apps/frontend/node_modules

      # *optional*: which files / folders
      # to cache for the next build run
      # cache:
      #   - node_modules
      #   - package-lock.json

    # ==== how to run your application ====
    run:
      # what technologies should the runtime
      # container be based on (can be array)
      base: nodejs@22

      # what ports your app listens on
      # and whether it supports http traffic
      ports:
        - port: 3000
          httpSupport: true
      envVariables:
        PORT: 3000
        HOST: 0.0.0.0

      # how to start your application
      start: pnpm start:frontend

zerops:
  - setup: daliborhondotdev
    # ==== how to build your application ====
    build:
      # what technologies should the build
      # container be based on (can be an array)
      base: nodejs@22

      # what commands to use to build your app
      buildCommands:
        - pnpm --filter @daliborhon.dev/frontend install
        - pnpm run build:frontend

      # select which files / folders to deploy
      # after the build succesfully finished
      deployFiles:
        - apps/frontend/dist
        - apps/frontend/package.json
        - apps/frontend/node_modules

      # *optional*: which files / folders
      # to cache for the next build run
      # cache:
      #   - node_modules
      #   - package-lock.json

    # ==== how to run your application ====
    run:
      # what technologies should the runtime
      # container be based on (can be array)
      base: nodejs@22

      # what ports your app listens on
      # and whether it supports http traffic
      ports:
        - port: 3000
          httpSupport: true
      envVariables:
        PORT: 3000
        HOST: 0.0.0.0

      # how to start your application
      start: pnpm start:frontend

43 Replies

Aleš•6mo ago

each app is supposed to end up in a different service, right?

DallyOP•6mo ago

For starter i just want to try the frontend app

Aleš•6mo ago

with zerops.yml at the root of the monorepo, you'd simply define two setups

zerops:
  - setup: nodejsapp
    build:
    run:

  - setup: astroapp
    build:
    run:

zerops:
  - setup: nodejsapp
    build:
    run:

  - setup: astroapp
    build:
    run:

DallyOP•6mo ago

What is the package manager for Alpine linux 😅 got in nvm

Aleš•6mo ago

apk you can btw use ubuntu if you want

zerops:
  - setup: nodejsapp
    build:
      os: ubuntu
      base: nodejs@22
    run:      
      os: ubuntu
      base: nodejs@22

zerops:
  - setup: nodejsapp
    build:
      os: ubuntu
      base: nodejs@22
    run:      
      os: ubuntu
      base: nodejs@22

also you can define multiple base technologies: in build as an array in base run has to have one primary base, but we have a handy tool called zsc to add additional technologies

zerops:
  - setup: nodejsapp
    build:
      os: ubuntu
      base: 
        - nodejs@22
        - [email protected]
    run:      
      os: ubuntu
      base: nodejs@22
      prepareCommands:
        - zsc install [email protected]

zerops:
  - setup: nodejsapp
    build:
      os: ubuntu
      base: 
        - nodejs@22
        - [email protected]
    run:      
      os: ubuntu
      base: nodejs@22
      prepareCommands:
        - zsc install [email protected]

also note that if you want to permanently add some dependency to build or run image, you'd add it in prepareCommands

zerops:
  - setup: nodejsapp
    build:
      os: ubuntu
      base: nodejs@22
      prepareCommands:
        - apt-get install foo
        - apt-get install bar
    run:      
      os: ubuntu
      base: nodejs@22
      prepareCommands:
        - apt-get install foo
        - apt-get install bar

zerops:
  - setup: nodejsapp
    build:
      os: ubuntu
      base: nodejs@22
      prepareCommands:
        - apt-get install foo
        - apt-get install bar
    run:      
      os: ubuntu
      base: nodejs@22
      prepareCommands:
        - apt-get install foo
        - apt-get install bar

this will create a new build and runtime image that will be cached and used for all subsequent pipelines (unlike if you were to do it in buildCommands, where it would run each time)

Michal Saloň•6mo ago

One thing I would like to add, specifically for monorepos, is that you can use ~ in deployFiles to deploy files from specified folder to root (/var/www/). Basically if you specify apps/frontend/dist it would be deployed into /var/www/apps/frontend/dist But if you specify it as apps/frontend/~/dist the part before ~ is stripped so the file is deployed to /var/www/dist

zerops:
  - setup: daliborhondotdev
      ...
      deployFiles:
        - apps/frontend/~/dist
        - apps/frontend/~/package.json
        - apps/frontend/~/node_modules
      ...

zerops:
  - setup: daliborhondotdev
      ...
      deployFiles:
        - apps/frontend/~/dist
        - apps/frontend/~/package.json
        - apps/frontend/~/node_modules
      ...

Aleš•6mo ago

yea it depends, if the pnpm start:frontend script is defined in the root package.json (which would have to be included in deployFiles)

deployFiles:
  - apps/frontend/dist
  - apps/frontend/package.json
  - apps/frontend/node_modules
  - package.json

deployFiles:
  - apps/frontend/dist
  - apps/frontend/package.json
  - apps/frontend/node_modules
  - package.json

it usually expects that those files are nested (so internally it runs node apps/frontend/dist/server.js or whatever).. essentially the question is: how would you start the production build if you ran it locally

DallyOP•6mo ago

Allright i got firther, but i'm having a problem with env vars. I had a variable defined GH_AUTH_TOKEN in the GUI, which got overwritten when I was messing with them. I added it to the zerops.yml and ran build. After that i removed it from zerops.yml and ray build again. Now I cannot add it anymore as secret because of an error. The variable is nowhere to be seen. The build is still failing, but i'm figuring it out.

Aleš•6mo ago

@Jan Saidl ? @Dally while @Jan Saidl is figuring it out (I believe it's a known bug we are already working on fixing).. is that GH_AUTH_TOKEN needed in build, or in runtime?

DallyOP•6mo ago

Only build time

Aleš•6mo ago

since it's a sensitive kind of env, you are right to put it in secret variables, but secret variables are by default not available to build containers, only to runtime containers, what you can do to use secret envs, and in build time just reference them

zerops:
  - setup: nodejsapp
    build:
      base: nodejs@22
      envVariables:
        GH_AUTH_TOKEN: ${RUNTIME_GH_AUTH_TOKEN}

zerops:
  - setup: nodejsapp
    build:
      base: nodejs@22
      envVariables:
        GH_AUTH_TOKEN: ${RUNTIME_GH_AUTH_TOKEN}

this way the actual value is still securely saved and zerops.yml doesn't contain the actual value (which could be dangerous even if your repo is private) only a reference

Jan Saidl•6mo ago

I have removed the env variable GH_AUTH_TOKEN from your service.

DallyOP•6mo ago

I got again further, now back to the monorepo. The start command fails with (see screenshot) i guess this is due to how pnpm handles modules? I changed the yml to:

            deployFiles:
              - apps/frontend/~/dist
              - apps/frontend/~/package.json
              - apps/frontend/~/node_modules
...

start: pnpm start

            deployFiles:
              - apps/frontend/~/dist
              - apps/frontend/~/package.json
              - apps/frontend/~/node_modules
...

start: pnpm start

Aleš•6mo ago

what monorepo is it using? is it something built into pnpm?

DallyOP•6mo ago

I'm trying it here currently: https://github.com/dallyh/daliborhon.dev

Aleš•6mo ago

ah, turbo, ok let me try to set it up

DallyOP•6mo ago

Turbo is used only for cache I'm using pnpm for all the monorepo stuff like workspaces etc

Aleš•6mo ago

gimme couple of minutes, I'll test it

DallyOP•6mo ago

However the error in question is that some package that Astro uses for React cannot be resolved, that may be because pnpm by default simlinks packages and does not actually install them into node_modules

Aleš•6mo ago

doesn't want to build locally :/

DallyOP•6mo ago

Yeah you need a specific setup for that unfortunately 😦

Aleš•6mo ago

ok, anyway it symlinks to the node_modules in root so what I'd try is

deployFiles:
  - apps/frontend/dist
  - apps/frontend/package.json
  - apps/frontend/node_modules
  - node_modules
  - package.json

deployFiles:
  - apps/frontend/dist
  - apps/frontend/package.json
  - apps/frontend/node_modules
  - node_modules
  - package.json

pnpm run start:frontend in the root package.json with

"start:frontend": "node apps/frontend/dist/server/entry.mjs",

"start:frontend": "node apps/frontend/dist/server/entry.mjs",

or whatever the actual start script should be if it doesn't work I'll try to set up a minimal example

DallyOP•6mo ago

It fails with the same error. What is the service start command retry count set to? Does it run indefinitely?

Aleš•6mo ago

It fails with the same error.

ok let me try to setup pnpm monorepo with a single astro app inside

What is the service start command retry count set to? Does it run indefinitely?

@Michal Saloň

Michal Saloň•6mo ago

I'll check just to be sure, but I think it runs untill the deploy process itself times out (1 hour). If you set up a readinessCheck, that one has a timeout of 5 minutes.

DallyOP•6mo ago

You could try this: https://github.com/dallyh/daliborhon.dev/tree/dev-zerops It is buildable without additional steps, that is build:frontend and start:frontend from root

Aleš•6mo ago

lol your code actually seems to work @Dally, gimme a sec to confirm

DallyOP•6mo ago

It works locally yeah 😄

Aleš•6mo ago

no, even on Zerops

Aleš•6mo ago

the only thing I changed was start, so it runs pnpm run start:frontend (+ formatting from 4 spaces to 2, but it shouldn't matter).. trying one more fresh deploy to confirm

DallyOP•6mo ago

Oh damn i see

Aleš•6mo ago

yeah https://github.com/fxck/daliborhon.dev/blob/dev-zerops/zerops.yml (I left it on the default port 4321 as well)

Aleš•6mo ago

HOST env was right tho

DallyOP•6mo ago

:facepalm: damn

Aleš•6mo ago

yep, here's the final one

DallyOP•6mo ago

It works, thanks for all the help. I got one last issue and that was disk space 😅

DallyOP•6mo ago

If i can have a suggestion for monorepos - currently every push to git means a build. That is if my monorepo hosts multiple apps and they are set up as services, then change in one triggers build for both. Again I'm using cloudflare, and they have something called build watch paths. That is they check if files changed in the specified path, and then only trigger an automatic build. It is really another useful feature when it comes to managing monorepos

Aleš•6mo ago

Thoughts @Backend ? The way we deal with this currently is to use tags/releases where you can put a regular expression to differentiate between releases of different packages. So you'd make release v4.0.0-app and it would only trigger deploy for app, v4.0.0-api would only trigger deploy for api.. but for trigger on push to branch, we'd probably have to implement something like those watch paths alternatively, you could set up GitHub Actions which can then utilize our CLI to trigger the deploy and essentially implement something akin to watch paths yourself.. which is not too user friendly, but it's a possibility

Jan Saidl•6mo ago

In case of watch paths: Even if we know what needs to be changed, we still have to download the code into the build container, which we then discard in case the required changes have not been made.

Aleš•6mo ago

I wonder how they do it.. https://developers.cloudflare.com/pages/configuration/build-watch-paths/

Cloudflare Docs

Build watch paths · Cloudflare Pages docs

When you connect a git repository to Pages, by default a change to any file in the repository will trigger a Pages build. You can configure Pages to …

Aleš•6mo ago

https://stackoverflow.com/a/26962188/301596 perhaps using comparison API so they don't have to download the code

Stack Overflow

GitHub API - how to compare 2 commits

It's possible to get list of changed files between two commits. Something like that comparison between two commits in web version but using GitHub Api.

DallyOP•6mo ago

Yeah wanted to say that Github's API allows to get commit details, which has file paths in it

DallyOP•6mo ago

example

Gaming

Programming

Monorepos