v0.20.x+ has broken docker builds
Hi team!
Seeing the following on container start:
This is when attempting to version bump from v0.12.2 ->v0.20.2
42 Replies
cc @xSwampxFox
Morning @charles, do you mind taking a look at this when you have a moment?
Hi @Skyfox675, how are you setting up v0.20?
Thanks for the reply! Here's my upgrade process I've been following since v0.10.0, we run Twenty on a on-prem K3s cluster:
1. Set twenty-server replicas to 0, wait for container to stop
2. Set twenty-db replicas to 0, wait for container to stop
3. Bump twenty-db image to new version
4. Bump twenty-server image to new version
5. Set twenty-db replicas to 1, wait for successful start
6. Set twenty-server replicas to 1, wait for successful start
Last bump I've tried is v0.12.2 -> v0.20.0 and I get the above error. Previously migrations have been automatic when the server container starts
All env vars, db connection string, and deployment settings are the same, only the docker image tag is being modified
OK! It's seems related to: https://github.com/twentyhq/twenty/issues/5976
GitHub
Migrations get stuck due to a confirmation prompt · Issue #5976 · t...
Here, the script runs npx ts-node. On newer npm CLIs, this requires a --yes flag, otherwise the following prompt is generated: Running database setup and migrations... Need to install the following...
However, you don't need to run the migration automatically at update (I would recommend to do it manually in a second step)
To avoid this behavior, ssh into your pod and create file /app/docker-data/db_status (touch /app/docker-data/db_status)
this will skip the auto-migration process
It's there packages/twenty-docker/twenty/entrypoint.sh
I'm still going to take a look at the issue as this might be tied to a newer npx version
Is this file removed after each successful migration or is the presence of the file the trigger to disable all auto migrations moving forward?
Going to try this shortly! Thank you!
@charles Just noticed the file path
/app/docker-data/ is this the path to persist data for the server?
I've been using /app/.local-storage/ maybe I missed this changeTrue, I forgot about it, we have updated the docker-compose here: https://github.com/twentyhq/twenty/blob/main/packages/twenty-docker/docker-compose.yml
GitHub
twenty/packages/twenty-docker/docker-compose.yml at main · twentyhq...
Building a modern alternative to Salesforce, powered by the community. - twentyhq/twenty
so you can update yours to also have a volume for this docker-data folder
That worked!! Thank you!
@xSwampxFox let's update the Terraform and K8s manifests to reflect this too so it doesn't catch anyone by suprise
Hey @charles how could I deploy it using traefik into my docker swarm?
what labels do I have to set?
I have to set it just into server? the traefik labels?
about the envs:
Can I set
server url: api.twenty.mydomain.com
Front_url: twenty.mydomain.com?
@Skyfox675 hey buddy can you help me?
I'm not very familiar with the network and intra-cluster service routing on Docker Swarm, but Terraform and K8s examples were just merged for both the Server and Postgres DB containers, as well as their services, persistant volumes and persistant volume claims!
https://github.com/twentyhq/twenty/tree/main/packages/twenty-docker/k8s
Keep in mind that this uses Cert-Manager to handle Let's Encrypt dynamic TLS certificate requests, and the Ingress-NGINX Controller for K8s.
One key point that I needed to get this working was to ensure that the "X-Forwarded-For" header existed and this required a custom configuration snippet in the Ingress-NGINX Controller.
GitHub
twenty/packages/twenty-docker/k8s at main · twentyhq/twenty
Building a modern alternative to Salesforce, powered by the community. - twentyhq/twenty
so i use traefik into my swarm
For K8s the flow of traffic in my case is:
1. Ingress from firewall/internet (port 80/443)
2. Sent to 80/443 to the LAN/Host IP of Ingress-NGINX Controller
3. Server Name Identification (SNI) parses the inbound requested Host Header
4. Traffic hits the virtualHost and follows the above Ingress Spec
5. Ingress routes traffic to "twentycrm-server" service via port name "http-tcp"
6. Service twentycrm routes traffic to any available twentycrm-server Pods via 3000
I use container lables for routing of "app=twentycrm-server"
Oh thanks, in my case is a bit different
I'm not sure how much of this will apply to Docker Swarm or Traefik but you can take what you need assuming that Traefik can do ProxyPass and Host header in-flight modification.
I rip Traefik out of all of my clusters and strictly use NGINX based controllers because it's the only one I've found to meet all of my use cases with minimal headaches
I see, thank you again! I use traefik successufuly! but this twenty stack is giviming headache! I think i will go for odoo that stacks easy works.
many stacks i easily deploy into my swarm
but this one is an excepcion
And no one cares about help me here
i tried to talk with a lot of people here no one wanna help
@Skyfox675
SERVER_URL=http://localhost:3000
FRONT_BASE_URL=http://twenty.mysite.com:-http://localhost:3000
its right?
Did you look at the link I posed above?
TwentyCRM's env var namees and value formating is the same regardless of the container orchestration engine you choose to use
yes but, my way to deploy is using the docker-compose into portainer in a easy way
That's fine, you're asking are your variable names and values correct, I posted examples in the link above:
https://github.com/twentyhq/twenty/blob/12c68fd77fd7bca26f662e4193c2b7f35700d92b/packages/twenty-docker/k8s/manifests/deployment-server.yaml#L29-L61
GitHub
twenty/packages/twenty-docker/k8s/manifests/deployment-server.yaml ...
Building a modern alternative to Salesforce, powered by the community. - twentyhq/twenty
I'm using this here
https://github.com/twentyhq/twenty/blob/main/packages/twenty-docker/docker-compose.yml
GitHub
twenty/packages/twenty-docker/docker-compose.yml at main · twentyhq...
Building a modern alternative to Salesforce, powered by the community. - twentyhq/twenty
You asked this...
Your answer is here: https://github.com/twentyhq/twenty/blob/12c68fd77fd7bca26f662e4193c2b7f35700d92b/packages/twenty-docker/k8s/manifests/deployment-server.yaml#L29-L61
GitHub
twenty/packages/twenty-docker/k8s/manifests/deployment-server.yaml ...
Building a modern alternative to Salesforce, powered by the community. - twentyhq/twenty
yes but I can not see front_base_url there
The vars I linked are the only ones required to run the application
I see
so i wiil cut ir off
and try it out again
Maybe this was changed recently...
You can try removing SERVER_URL and adding FRONT_BASE_URL and BACKEND_SERVER_URL and setting the value of each to
https://domain.name:443Thanks a lot @Skyfox675 for the help! Is there any more help needed @lucas lima, we haven't change the env var recently so they should still be up-to-date
@charles I still wondering how to deploy a twenty stack at portainer. I'm working on it more than 1 week 🥲
In general I can get docker-compose.yaml from github and deploy it easily
I have no experience with portainer but we are hosting Twenty on K8 in production
where are you stuck?
hetzner
docker swarm
I mean what issue are you facing?
the best achieve were the 3 services running with ubuntu complete status but error 404 when i access the twenty.mysite.com
I used 1 click deploy locally and worked into my wsl
but into my swarm not
okay, can you ssh into your 3 containers and curl on localhost to check what you have from there
to see if it's a network / dns issue or if it's tied to Twenty
ok I will deploy again to check it. thank you but I understand you because you use k8 and its different way to deploy
root@gestao:~/twenty# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a21c412c9b96 twentycrm/twenty:v0.21.0 "/app/entrypoint.sh …" 37 minutes ago Up 37 minutes twenty-worker-1
b60256cab556 twentycrm/twenty:v0.21.0 "/app/entrypoint.sh …" 37 minutes ago Up 37 minutes (healthy) 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp twenty-server-1
f963dcbe9bfa twentycrm/twenty-postgres:v0.21.0 "/opt/bitnami/script…" 37 minutes ago Up 37 minutes (healthy) 5432/tcp twenty-db-1
a5c3ec47bfd4 cloudflare/cloudflared:latest "cloudflared --no-au…" 37 minutes ago Up 37 minutes cloudflared
root@gestao:~/twenty#
I suspect that the issue is a networking issue and is not tied to the docker-compose direclty
this looks good
let's ssh into twenty-server and curl localhost:3000
you should get a valid html
into container right?
root@gestao:~# docker exec -it b60256cab556 /bin/sh
/app/packages/twenty-server $ curl http://localhost:3000
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" href="/icons/android/android-launchericon-48-48.png" />
<link rel="apple-touch-icon" href="/icons/ios/192.png" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="theme-color" content="#000000" />
<meta name="description" content="A modern open-source CRM" />
<meta
property="og:image"
content="https://raw.githubusercontent.com/twentyhq/twenty/main/docs/static/img/social-card.png"
/>
<meta property="og:description" content="A modern open-source CRM" />
<meta property="og:title" content="Twenty" />
<meta name="twitter:card" content="summary_large_image" />
<meta
name="twitter:image"
content="https://raw.githubusercontent.com/twentyhq/twenty/main/docs/static/img/social-card.png"
/>
<meta name="twitter:description" content="A modern open-source CRM" />
<meta name="twitter:title" content="Twenty" />
<link
rel="stylesheet"
href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600&display=swap"
/>
<title>Twenty</title>
<script src="/env-config.js"></script>
<script type="module" crossorigin src="/assets/index-lx_9acOo.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-5cRSZ-Mg.css">
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
</body>
</html>
@charles
/app/packages/twenty-server # curl https://localhost:3000
curl: (35) OpenSSL/3.1.2: error:0A00010B:SSL routines::wrong version number
/app/packages/twenty-server # curl http://localhost:3000
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" href="/icons/android/android-launchericon-48-48.png" />
<link rel="apple-touch-icon" href="/icons/ios/192.png" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="theme-color" content="#000000" />
<meta name="description" content="A modern open-source CRM" />
<meta
property="og:image"
content="https://raw.githubusercontent.com/twentyhq/twenty/main/docs/static/img/social-card.png"
/>
<meta property="og:description" content="A modern open-source CRM" />
<meta property="og:title" content="Twenty" />
<meta name="twitter:card" content="summary_large_image" />
<meta
name="twitter:image"
content="https://raw.githubusercontent.com/twentyhq/twenty/main/docs/static/img/social-card.png"
/>
<meta name="twitter:description" content="A modern open-source CRM" />
<meta name="twitter:title" content="Twenty" />
<link
rel="stylesheet"
href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600&display=swap"
/>
<title>Twenty</title>
<script src="/env-config.js"></script>
<script type="module" crossorigin src="/assets/index-JGIdMgsI.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-5cRSZ-Mg.css">
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
</body>
</html>
/app/packages/twenty-server #
SERVER_URL: http://api.twenty.mysite.com
FRONT_BASE_URL: http://twenty.mysite.com
labels:
- traefik.enable=true
- traefik.http.routers.twenty-server-api.rule=Host(
api.twenty.mysite.com)
- traefik.http.routers.twenty-server-api.entrypoints=websecure
- traefik.http.routers.twenty-server-api.tls.certresolver=letsencryptresolver
- traefik.http.services.twenty-server-api.loadbalancer.server.port=3000
- traefik.http.routers.twenty-server.rule=Host(twenty.mysite.com)
- traefik.http.routers.twenty-server.entrypoints=websecure
- traefik.http.routers.twenty-server.tls.certresolver=letsencryptresolver
- traefik.http.services.twenty-server.loadbalancer.server.port=3000ok so Twenty seems to be running correctly. The issue seems to be with the way you route your traffic to the container. I have no experience with traefic unfortunately
those are right:
SERVER_URL: http://api.twenty.mysite.com/
FRONT_BASE_URL: http://twenty.mysite.com/
you could make them https
but I don't think it's your issue, these env variables are used by the application to run properly. Here you 404 is happening before hitting the application
yeah
this is not the issue
I'm sorry @lucas lima, I have no experience with traefic and I don't have the bandwidth to look into it. If it's Twenty related, I can help debug quickly but if it's not I can't. They are too many cloud providers with their own way of doing things