theodor Posts - Answer Overflow

theodor

Explore posts from servers

•Created by theodor on 2/26/2024 in #✋｜help

Railway incident ?? Ghost deployment without our interaction

This is incredibly serious. We just found out that railway created a deployment out of nothing out of thin air, taking our site down, using a very old random commit.

25 replies

RRailway

•Created by theodor on 2/2/2024 in #✋｜help

Railway Networking issues

Hi! This is a bit broad as we're still debugging this, but we've been noticing that certain webhook events don't make it to our servers. This is really bad, since we don't really get an error when that happens (since they don't make it to the server!i ) We can also see that some requests fail because the TLS fails, e.g.

request to https://api.stability.ai/v1/generation/stable-diffusion-xl-1024-v1-0/text-to-image failed, reason: Client network socket disconnected before secure TLS connection was established

From our research, this means something may be still wrong with the proxy. Any ideas what we can do? Should we look into cloudflare? And yes, we do sleep 3s before we start the services...

36 replies

RRailway

•Created by theodor on 1/30/2024 in #✋｜help

Setting shared env var doesn't wrok

Tonight it looks like setting shared env vars doesn't work. Setting and redeploying seems to work, but then on startup the env var doesn't exist...

11 replies

RRailway

•Created by theodor on 12/13/2023 in #✋｜help

Issue accessing env vars

7 replies

RRailway

•Created by theodor on 9/12/2023 in #✋｜help

nginx revere proxy 503 error

Hi! I wanted to set up a reverse NGINX proxy with mixpanel. I used the dockerfile provided by them:

FROM nginx
COPY nginx.conf /etc/nginx/nginx.conf

FROM nginx
COPY nginx.conf /etc/nginx/nginx.conf

And this configuration:

events {}
http {
    server {
        listen 80 default backlog=16384;
        listen [::]:80 default backlog=16384;

        location /lib.min.js {
            proxy_set_header X-Real-IP $http_x_forwarded_for;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host $server_name;
            proxy_pass https://cdn.mxpnl.com/libs/mixpanel-2-latest.min.js;
        }

        location /lib.js {
            proxy_set_header X-Real-IP $http_x_forwarded_for;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host $server_name;
            proxy_pass https://cdn.mxpnl.com/libs/mixpanel-2-latest.js;
        }

        location /decide {
            proxy_set_header Host decide.mixpanel.com;
            proxy_set_header X-Real-IP $http_x_forwarded_for;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host $server_name;
            proxy_pass https://decide.mixpanel.com/decide;
        }

        location / {
            proxy_set_header Host api.mixpanel.com;
            proxy_set_header X-Real-IP $http_x_forwarded_for;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host $server_name;
            proxy_pass https://api.mixpanel.com/;
        }
    }
}

events {}
http {
    server {
        listen 80 default backlog=16384;
        listen [::]:80 default backlog=16384;

        location /lib.min.js {
            proxy_set_header X-Real-IP $http_x_forwarded_for;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host $server_name;
            proxy_pass https://cdn.mxpnl.com/libs/mixpanel-2-latest.min.js;
        }

        location /lib.js {
            proxy_set_header X-Real-IP $http_x_forwarded_for;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host $server_name;
            proxy_pass https://cdn.mxpnl.com/libs/mixpanel-2-latest.js;
        }

        location /decide {
            proxy_set_header Host decide.mixpanel.com;
            proxy_set_header X-Real-IP $http_x_forwarded_for;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host $server_name;
            proxy_pass https://decide.mixpanel.com/decide;
        }

        location / {
            proxy_set_header Host api.mixpanel.com;
            proxy_set_header X-Real-IP $http_x_forwarded_for;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host $server_name;
            proxy_pass https://api.mixpanel.com/;
        }
    }
}

I also enabled a custom domain which is verified (reverse-proxy.rizzgpt.app). However, I'm getting a 503 when I try to load it. Any idea why?

15 replies

RRailway

•Created by theodor on 9/4/2023 in #✋｜help

Restore old logs?

Hi! Is there a way to restore old log feature while you figure out the kinks in the new one? The new one is very slow and makes it hard to figure things out during an emergency. I'd rather not use it for now.

20 replies

RRailway

•Created by theodor on 7/21/2023 in #✋｜help

Issues setting up datadog with docker containers

Hi - I'm trying to use private networking to set up datadog. I have two docker files that start normally: server and datadog. Both seem to run fine. I enabled private networking for the project already. 1. What are the environment variables I can use to refer to the hostnames of the services? I had to hard code them in. 2. It doesn't seem like the services are able to talk to one another. I don't see any errors, but I onyl get silence. Is there something off about the way I defined the hostnames below? 3. When I run railway run echo $DD_SERVICE or any similar command to check the environment variables of the service, I don't get anything. Is this ok? Here are the dockerfiles:

# Start from the official Node.js 18 image
FROM node:18

# Define environment variables
ENV DD_ENV=prod \
    DD_LOGS_INJECTION=true \
    DD_SERVICE=cami-server \
    DD_AGENT_HOST='datadog.railway.internal'

# Create app directory
WORKDIR /usr/src/app

# Copy package.json and package-lock.json
COPY package*.json ./

# Install app dependencies
RUN yarn install --frozen-lockfile

# Copy app source code
COPY . .

# Build the app
RUN yarn run build

# Start the app
CMD [ "yarn", "run", "start" ]

# Start from the official Node.js 18 image
FROM node:18

# Define environment variables
ENV DD_ENV=prod \
    DD_LOGS_INJECTION=true \
    DD_SERVICE=cami-server \
    DD_AGENT_HOST='datadog.railway.internal'

# Create app directory
WORKDIR /usr/src/app

# Copy package.json and package-lock.json
COPY package*.json ./

# Install app dependencies
RUN yarn install --frozen-lockfile

# Copy app source code
COPY . .

# Build the app
RUN yarn run build

# Start the app
CMD [ "yarn", "run", "start" ]

Datadog:

# Start from the official Datadog agent image
FROM datadog/agent:latest

# Copy your Datadog configuration to the correct location
COPY datadog.yaml /etc/datadog-agent/datadog.yaml

# Set the hostname and port
ENV DD_HOSTNAME='datadog.railway.internal' \
    DD_LOGS_ENABLED=true \
    DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true \
    DD_BIND_HOST=::

# Start the Datadog agent
CMD ["/init"]

# Start from the official Datadog agent image
FROM datadog/agent:latest

# Copy your Datadog configuration to the correct location
COPY datadog.yaml /etc/datadog-agent/datadog.yaml

# Set the hostname and port
ENV DD_HOSTNAME='datadog.railway.internal' \
    DD_LOGS_ENABLED=true \
    DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true \
    DD_BIND_HOST=::

# Start the Datadog agent
CMD ["/init"]

81 replies

RRailway

•Created by theodor on 7/2/2023 in #✋｜help

redis connection fails

Hi! Having some issues with redis. I'm getting

Error: connect ECONNREFUSED 34.82.163.149:7679
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1300:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '34.82.163.149',
port: 7679
}

Error: connect ECONNREFUSED 34.82.163.149:7679
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1300:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '34.82.163.149',
port: 7679
}

I already tried restarting Redis or restarting/redeploying the project. Any suggestions? everything was working fine previoously

40 replies

RRailway

•Created by theodor on 5/30/2023 in #✋｜help

urgent: horizontal scaling limited at 5 replicas

Hi! We're on the teams plan. We're trying to scale as fast as we can, but the scaling is limited at 5 replicas. is there anything we can do to get past that?

127 replies

RRailway

•Created by theodor on 5/29/2023 in #✋｜help

"horizontal-scaler" missing on railway

Hi! We're being hit hard by a lot of traffic and our server is having trouble scaling. I wanted to use the 'horizontal-scaler' option described in the docs, but it's missing. Anyone know why?

29 replies

RRailway

•Created by theodor on 3/13/2023 in #✋｜help

Environment Variables missing from project

The environment variables seem to be missing from the project. Loading the view also takes much longer than it used to. I noticed this after modifying one of them in Vercel , but I'm not sure if that's related.

8 replies

Gaming

Programming