Urgent: Two production services down due to database connection issue
@Percy@Parker could I get some urgent assistance
Solution:Jump to solution
Hey @Albert @DrSmrtnik @NoBypass - We have moved all domains over to the new edge proxy, including the domains attached to the surrealdb databases.
66 Replies
Project ID:
N/A
My project service is unable to connect to SurrealDB, it appears to be a
wss (websockets secure) issue with railway
Another person in SurrealDB discord server is having this issue as well. I have made no changes to my configuration and my database is down for two different servicesNew reply sent from Help Station thread:
This issue occurs also in Surrealist. When using WSS , the connection breaks but HTTPS works fine. This was not previously the case, both worked correctly before.You're seeing this because this thread has been automatically linked to the Help Station thread.
hey Albert, can you explain the setup for me. You have SurrealDB running at https://railway.app/project/a72c350e-c977-4440-95f8-1fc4d1c5fdd6.
You are connecting to it with
surrealdb-production-fae7.up.railway.app from clients correct ?Yes. When using
wss protocol, it fails to connect
https works fine
This was previously functional with no issues. Another user is experiencing the same issue with railway and running on a different database versionAre you available to check if it's working in a few minutes ?
Yes
ok
try now
Working! @Mig
What was the issue
I will have to troubleshoot it. We are moving everyone over to a new edge network and I just put your application back on the legacy network.
I still have one service down, oddly
(same project)
what's the issue exactly ?
Exact same connection error, I'll give it a minute to see if it might be some sort of pre-existing connection issue. I'll trigger a re-deploy for the project now
I've just changed DNS settings so existing connections will continue to talk to the previous network
we have migrated 400k applications to our new network and this is the first issue I've seen :/
Websockets can be a bitch
I suspect many other SurrealDB users are experiencing the same issue
I will try to repro right now
Okay, so my other service is online now after a redeploy
Approximately when did this migration take place?
the migration process has been ongoing for the past 2 weeks.
slowly moving apps over
ok, it was using connection to previous network
Do you know roughly when our project might've been migrated?
Just would like to know how much downtime there might've been.
it must've been a few days since I started having this issue on one service before the other one started showing issues (probably the existing connection)
I tried checking but I don't have enough data to know for sure.
I might be able to check 1 more place
@Mig is it normal to have different IPs for the host in the DNS? I'm assuming they're just edge routing or something?
https://dnschecker.org/#A/surrealdb-production-fae7.up.railway.app
DNS Checker
DNS Checker - DNS Check Propagation Tool
Check DNS Propagation worldwide. DNS Checker provides name server propagation check instantly. Changed nameservers so do a DNS lookup and check if DNS and nameservers have propagated.
the IPs resolve to different regional load balancers
so geodns is picking the closest LB
ok thanks just checking
I can confirm the date
just will take time as I grep the log file
If you'd like to reproduce my setup,
I was running
surrealdb/surrealdb:v1.4.2 as the docker image
/surreal start --auth --allow-guests --allow-funcs --deny-funcs "http" --allow-scripting --deny-net --no-banner as the custom start command
Region: US West (Oregon)
Runtime: Legacy but also tried V2 with same issueThe other person having the issue was on
1.5.2
To reiterate, this issue only occurred with wss connection within Surrealist GUI
Not with https
ok, your app was migrated excatly at 2024-08-26T02:32:51Z
utc
so about 48 hours
Yeah that sounds about right
close to 24 hours
I know websockets work
I'm sus about wss:// requiring TLS but our proxy doesn't do TLS to your app.
https:// worked though so maybe not that
ty for this!
@Mig are you saying data isn't encrypted during transit?
it is encrypted all the way to your app, just not using HTTPS.
internet <-- HTTPS --> Edge Network < -- WireGuard encrypted tunnel -- > Your App
did you configure tls certs on your app ?
hmm you might be right,
surreal start has the following option
if so the edge network could be talking over HTTPS (and also wireguard)
most people don't have TLS terminating at their application
most is read as, I've never seen it
Well, I think TLS is enabled for https most definitely, but what about Websockets? How does websockets interact with proxies?
TLS would work for ws the same since that happens before the application (websocket/http) right after tcp
Anyhow, thank you for resolving the issue as fast as you could. I would investigate this as I imagine other users besides the one I spoke with is having this issue
yup, i'm working to reproduce right now
sorry you had this issue.
No problem - these things happen, fortunately I'm not running a critical service
When you are able, I would love an update on what occurred, whether you were able to reproduce/fix the issue with the new network
that drop down is in the surreal ui right that I should be able to visit just from the same service I am deploying on railway
I don't believe so
Surrealist
Powerful graphical SurrealDB query playground and database explorer, available for web and desktop ⚡
You can probably use the web GUI if its easier
otherwise downloading the application
oh I see. the service is just the DB and the screenshot is a frontend to it
That's right yeah
Let me know if you have any questions regarding connection setup, or whether you need me to test with my project once a fix is deployed
I used
Root authentication method via the credentials generated by railway, any namespace and database should work for the connection
The URL should just be the host, and wss for the protocolNew reply sent from Help Station thread:
hello, i have the same issue in my project can you please help meYou're seeing this because this thread has been automatically linked to the Help Station thread. New reply sent from Help Station thread:
@bachargit, what is your domain and I can take a look.You're seeing this because this thread has been automatically linked to the Help Station thread.
New reply sent from Help Station thread:
hi @20K-ultra, my surrealdb database domain name is https://accountify-db.up.railway.appYou're seeing this because this thread has been automatically linked to the Help Station thread.
SurrealDB
SurrealDB | Surrealist
SurrealDB is the ultimate database for tomorrow's serverless, jamstack, single-page, and traditional applications.
New reply sent from Help Station thread:
Please try again now.You're seeing this because this thread has been automatically linked to the Help Station thread. New reply sent from Help Station thread:
okYou're seeing this because this thread has been automatically linked to the Help Station thread. New reply sent from Help Station thread:
yeah its workingYou're seeing this because this thread has been automatically linked to the Help Station thread. New reply sent from Help Station thread:
thank youYou're seeing this because this thread has been automatically linked to the Help Station thread.
Hello.
I cannot connect to my deployed surrealdb instance on railway:
wss://surrealdb-production-c358.up.railway.app/rpc, even though it was working a couple of days ago.
Here is my projectID: 320c6ccd-0556-4d5f-9f87-05d51b1043eddo you still have the issue if you use https:// protocol in the url instead of wss:// ?
I am aware of this surrealdb issue and wss://. Working to reproduce and fix.
i see no issues with the
https:// protocol, only the wss://Hello,
I've been experiencing the same issue. Any way to resolve this?
My ProjectID:
09be9421-29f2-4e16-b051-751c7fb2b16aMake surrealdb client use https:// protocol instead of wss://. The connection will upgrade to websocket anyways.
the library which i use specifically wants a url in the format of wss:// or ws:// though. I tried for a bit but i don't think there's a good way around this...
how was this resolved for Albert and bachargit? is there something else I can try?
The issue is with the edge network being used by the application. We are moving on from our old proxy but I can put your app on it while we try to resolve this issue for surrealdb
yeah if that could at least temporarily resolve the issue i'd appreciate if you could do that
wss:// should work for you in a few seconds.
yeah it works, thank you for the help
Hey @Mig, can you do that for my app as well?
I started needing the wss:// protocol now
.
please do not ping team members, we are currently on a day off
this domain is now on the legacy proxy, should be updated soon
Solution
Hey @Albert @DrSmrtnik @NoBypass - We have moved all domains over to the new edge proxy, including the domains attached to the surrealdb databases.
awesome @Brody thanks for the update
sorry i should have been more clear, we have not patched away whatever surrealdb is doing incorrectly with websocket connections
It appears to be working fine for me @Brody ?
well you are on the v2 proxy, so thats awsome, are you using wss or https?
wss I believe but I could double check soon
actually it may be
https but I am not too sure
I guess I'll find out when I migrate my new update using wss