Railway•11mo ago

Connection issues (fixed) and slow deploys (not fixed)

I've been having problems for the last 30 minutes, I tried to restart the DB and flush the data now I tried spinning up a new DB with de facto connection code I use in all my other projects with the new DB and I'm still having problems. Project: https://railway.app/project/bccd1ccf-de59-4a09-89aa-9e9a4f75a3cc/service/eb00af5c-4703-4644-a9e0-4bff24fa92d7?id=5ac95b9d-b0bc-48b8-bcd4-43820c1f236e

INFO[0000] Connecting to Redis...

panic: dial tcp: lookup monorail.proxy.rlwy.net: i/o timeout

goroutine 1 [running]:

main.main()

/app/main.go:275 +0x6dc

INFO[0000] Connecting to Redis...

panic: dial tcp: lookup monorail.proxy.rlwy.net: i/o timeout

goroutine 1 [running]:

main.main()

/app/main.go:275 +0x6dc

Railway

404 - Page not found

Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.

97 Replies

Percy•11mo ago

Project ID: bccd1ccf-de59-4a09-89aa-9e9a4f75a3cc,eb00af5c-4703-4644-a9e0-4bff24fa92d7,5ac95b9d-b0bc-48b8-bcd4-43820c1f236e

Brody•11mo ago

can you connect to it locally?

cybershizoOP•11mo ago

Yes it seems to work more often if I run my app locally Deploying seems to be consistently broken I would use the internal identifier for the URL but then I can't reuse it for locally running my app

Brody•11mo ago

but it doesnt work at all when you deploy to railway?

cybershizoOP•11mo ago

Yes I keep seeing the timeout issue I posted I am connecting to a new Redis instance, clean and confirmed the connection string is right I'll try again soon

Brody•11mo ago

let me know how that goes, if its still failing i will ping in somebody

cybershizoOP•11mo ago

Kk @Brody No luck

cybershizoOP•11mo ago

cybershizoOP•11mo ago

It works locally, times out on Railway I don't have any conditional code

cybershizoOP•11mo ago

This https://railway.app/project/bccd1ccf-de59-4a09-89aa-9e9a4f75a3cc/service/eb00af5c-4703-4644-a9e0-4bff24fa92d7/settings is trying to consume this resource https://railway.app/project/bccd1ccf-de59-4a09-89aa-9e9a4f75a3cc/service/272f7332-ac63-41b5-b0c9-90872d441ee8

Railway

404 - Page not found

Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.

Railway

404 - Page not found

Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.

cybershizoOP•11mo ago

It's just a junk Redis DB I created a few hours ago, nothing special about it

Brody•11mo ago

@matt - potential network outage, im not too sure, needs investigating

cybershizoOP•11mo ago

@Brody I'll try with the internal URL see if that helps

Brody•11mo ago

good thinking, though you might just get a different error

cybershizoOP•11mo ago

@Brody Like that?

Brody•11mo ago

youd need to specify the port and credentials, but yes side note, thats is some crazy font you have there, i like small font myself, but how do you read that without getting a head ache?

cybershizoOP•11mo ago

Hahahaha

cybershizoOP•11mo ago

It looks like this, I usually use the Mac zoom in feature But because of AI I have so much code I just need the font to be small so I can manage all of it Before AI my font size was like 25% bigger

Brody•11mo ago

thats the opposite of monospace lol

cybershizoOP•11mo ago

@Brody Not luck with internal URL when deploying to Railway There's no sensitive data there

Brody•11mo ago

can you do a dial on the redis host and port?

cybershizoOP•11mo ago

What do you mean? Is there a command you want me to run?

Brody•11mo ago

in your go code, net.dial i think it is? just as a debugging step

cybershizoOP•11mo ago

Sure sec OK I think I fixed it I believe it's panicing on this if err := rdb.Ping(context.Background()).Err(); err != nil { panic(err) } If I remove that everything seems to work normally

cybershizoOP•11mo ago

It's interacting with the Redis DB not just connecting to it The weird thing is I am using the same version of Go redis in other projects with ping and it works fine

Brody•11mo ago

what redis client and what version are you using I'll try to replicate this when I have time

cybershizoOP•11mo ago

I tried to replicate it just now lmao and I can't

cybershizoOP•11mo ago

package main

import (
    "context"

    "github.com/redis/go-redis/v9"
    "github.com/sirupsen/logrus"
)

const REDIS_URL = "redis://default:[email protected]:41452"

func main() {
    // Connect to Redis
    logrus.Info("Connecting to Redis...")
    url, err := redis.ParseURL(REDIS_URL)
    if err != nil {
        panic(err)
    }
    rdb := redis.NewClient(url)
    defer rdb.Close()
    logrus.Info("Connected to Redis")

    logrus.Info("Pinging Redis...")
    if err := rdb.Ping(context.Background()).Err(); err != nil {
        panic(err)
    }
    logrus.Info("Pinged Redis")
}

package main

import (
    "context"

    "github.com/redis/go-redis/v9"
    "github.com/sirupsen/logrus"
)

const REDIS_URL = "redis://default:[email protected]:41452"

func main() {
    // Connect to Redis
    logrus.Info("Connecting to Redis...")
    url, err := redis.ParseURL(REDIS_URL)
    if err != nil {
        panic(err)
    }
    rdb := redis.NewClient(url)
    defer rdb.Close()
    logrus.Info("Connected to Redis")

    logrus.Info("Pinging Redis...")
    if err := rdb.Ping(context.Background()).Err(); err != nil {
        panic(err)
    }
    logrus.Info("Pinged Redis")
}

cybershizoOP•11mo ago

Works as a standalone deploy

cybershizoOP•11mo ago

So in the simple example everything works In th original deploy I am still having problems There's something really weird going on

Brody•11mo ago

is your working test deploying into the same service?

cybershizoOP•11mo ago

Same project different service I am wondering if Redis like blacklisted my original service or something One guess is the failure policy (e.g. restarting 10 times quickly) caused Redis (possibly) to blacklist the IP or something, far out guess

Brody•11mo ago

nah redis isnt going to do that

cybershizoOP•11mo ago

I'm having no issues here doing the same thing https://railway.app/project/bccd1ccf-de59-4a09-89aa-9e9a4f75a3cc/service/05f5b6b8-854c-477a-9afb-42c284aaf1dd?id=4126273a-58c2-452e-a414-fee36eae5004

Railway

404 - Page not found

Railway is an infrastructure platform where you can provision infrastructure, develop with that infrastructure locally, and then deploy to the cloud.

Brody•11mo ago

just use the new service then lol btw these links dont do anything for me, i dont work for railway

cybershizoOP•11mo ago

What? Lmao What do you do this for then?

Brody•11mo ago

to help people

cybershizoOP•11mo ago

I see

cybershizoOP•11mo ago

Well yeah the service is somehow the problem

cybershizoOP•11mo ago

Connection logic is identical TCP is fine btw I tried the dialing thing

Brody•11mo ago

yeah thats why i tagged in matt, there could be a network issue with a single host in their fleet

cybershizoOP•11mo ago

This worked fine If I change the URL of the service it still doesn't work I've also tried restarting and redeploying it The service is somehow bugged

Brody•11mo ago

but the new service you made, works just fine?

cybershizoOP•11mo ago

Yeah I tried to create a minimal repro with the same connection logic in a new service, works fine I am now deploying a new service with exactly the same code to concur it's a service problem

Brody•11mo ago

keep me posted

cybershizoOP•11mo ago

Something weird is going on That is a brand new service

Brody•11mo ago

can you try adding a time.Sleep(3 * time.Second) before you attempt to connect to the database

cybershizoOP•11mo ago

Yeah before pinging it? That's a good idea

Brody•11mo ago

before pinging or connecting whatever comes first

cybershizoOP•11mo ago

Lmao yeah that fixed it Ughhhhh

Brody•11mo ago

awesome

cybershizoOP•11mo ago

https://tenor.com/view/pedro-pascal-cry-laughing-meme-gif-27654463

Tenor

cybershizoOP•11mo ago

Great thanks lmao

Brody•11mo ago

no problem, the io timeout threw me for a loop, it's usually a DNS lookup failures

cybershizoOP•11mo ago

It's a weird one, makes sense why it worked locally now Because the round trip was long enough But when it's side by side in Railway it probably takes a millisecond or less I'll just remember to always pause a second before pinging Redis

Brody•11mo ago

nope, it takes around 2 seconds

cybershizoOP•11mo ago

2 seconds for what?

Brody•11mo ago

for networking to start working properly

cybershizoOP•11mo ago

Ohhhhhhh

Brody•11mo ago

but it's usually a dns error, not an io error

cybershizoOP•11mo ago

I see OK wild Thanks so much I was losing my sanity

Brody•11mo ago

it will not always be required, railway is hands on keyboard to eliminate this network initialisation delay to what I heard will be instantaneous

cybershizoOP•11mo ago

Do you know if there's anyway to make deploys faster? It takes like 30 seconds to a minute just to see the build logs then another minute for the app to finally deploy It's the least productive aspect when I develop

Brody•11mo ago

I have not experienced that myself with my go apps

cybershizoOP•11mo ago

Contrast to Netlify, I just type make deploy and 5 seconds later it's live I am deploying from the CLI

Brody•11mo ago

I do too

cybershizoOP•11mo ago

So railway up --service=... --detach Which plan are you on?

Brody•11mo ago

hobby

cybershizoOP•11mo ago

Me too How long does it take you to deploy a hello world?

Brody•11mo ago

I do my railway up and I can see build logs right after the upload is done

cybershizoOP•11mo ago

I did a test this is like 60-90 seconds to print hello world

cybershizoOP•11mo ago

Worst aspect of Railway imo Am I doing something wrong? It lose my focus it takes so long

Brody•11mo ago

okay well now there an issue

cybershizoOP•11mo ago

That's why debugging the Redis thing took so long Every attempt took like 2-4 minutes to validate

Brody•11mo ago

yeah no this is a clear issue

cybershizoOP•11mo ago

Who do we tag?

Brody•11mo ago

well since i was given permission to tag him for platform issues - @Cooper - build and deploy logs slow to come in

cybershizoOP•11mo ago

To summarize for Jake, first Brody helped me diagnose a network issue. I was calling Ping on a Redis client too quicky (immediately after connecting) but it turns out I needed to wait some 2 seconds for networking to work -- once I added a delay everything started working, I didn't even realize this was an issue and has never come up for me previously) But the whole time I've been having incredibly slow deploys, as showcased in the video attached above. It takes a minute if not more to see the build logs (same delay in terminal and on web), so I'm used to waiting maybe 1-2 minutes on average for every single deploy What's your CLI version? zaydek@Zaydeks-MacBook-Pro experimental-hello-world % railway --version railwayapp 3.4.0

Brody•11mo ago

i can replicate the slow logs, i did a test deploy of a hello-world http server and the server became active before any build or deploy logs came through

cybershizoOP•11mo ago

I tried upgrading the CLI to 3.5.2 but same issue

==> Upgrading railway
  3.4.0 -> 3.5.2 

==> Pouring railway--3.5.2.arm64_ventura.bottle.tar.gz
==> Caveats
zsh completions have been installed to:
  /opt/homebrew/share/zsh/site-functions
==> Summary
🍺  /opt/homebrew/Cellar/railway/3.5.2: 10 files, 11.9MB
==> Running `brew cleanup railway`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
Removing: /opt/homebrew/Cellar/railway/3.4.0... (10 files, 11.5MB)
zaydek@Zaydeks-MacBook-Pro experimental-hello-world % railway up --service=experimental-hello-world
  Indexed                                                                                                                                                                                             
  Compressed [====================] 100%                                                                                                                                                              
  Uploaded                                                                                                                                                                                              Build Logs: https://railway.app/project/b3fb6b03-d46d-4394-9114-1b14a6bd7a73/service/5762a74d-a4db-4bd6-bb77-eb5b7080f254?id=6137da01-5570-48e6-9e51-941a5b120137&

==> Upgrading railway
  3.4.0 -> 3.5.2 

==> Pouring railway--3.5.2.arm64_ventura.bottle.tar.gz
==> Caveats
zsh completions have been installed to:
  /opt/homebrew/share/zsh/site-functions
==> Summary
🍺  /opt/homebrew/Cellar/railway/3.5.2: 10 files, 11.9MB
==> Running `brew cleanup railway`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
Removing: /opt/homebrew/Cellar/railway/3.4.0... (10 files, 11.5MB)
zaydek@Zaydeks-MacBook-Pro experimental-hello-world % railway up --service=experimental-hello-world
  Indexed                                                                                                                                                                                             
  Compressed [====================] 100%                                                                                                                                                              
  Uploaded                                                                                                                                                                                              Build Logs: https://railway.app/project/b3fb6b03-d46d-4394-9114-1b14a6bd7a73/service/5762a74d-a4db-4bd6-bb77-eb5b7080f254?id=6137da01-5570-48e6-9e51-941a5b120137&

Brody•11mo ago

no worries, if I can reproduce it then it's not an issue with how you are doing things

Brody•11mo ago

i can still reproduce this today, baffled as to why no one else has reported it yet

Ray•11mo ago

Could you link me to your project for this please?

Ray•11mo ago

https://status.railway.app/clr6l0hql353667bfn5vsnpqyam

Build/deploy logs may not be showing up for some users - Incident d...

Build/deploy logs may not be showing up for some users - Incident details - Railway Status

Brody•11mo ago

https://railway.app/project/e3ff3e1f-e02f-45d8-ab75-7a7544c22636/service/4dee6202-f128-449a-bb9c-62e9f70cfc14

Ray•11mo ago

This incident has been resolved, apologies for any inconvenience caused! https://status.railway.app/clr6l0hql353667bfn5vsnpqyam

Gaming

Programming

Connection issues (fixed) and slow deploys (not fixed)