Slow to connect replica
I am using Uptime Kuma to detect down services. Our game server has a few replicas. And for some reason it seems like one of the replicas are 'slow' to connect to. Using Chrome developer tools I can see that it will randomly take a very long time (1s+) to connect to the replica. I am pinging the health-check endpoint which returns how long the check took. My server responds with <10-20ms, but the request does not resolve for over 1s+.
It only randomly happens. I can keep pinging the health-check end point and I get a lot of very quick responses. But every 4-5 attempts hits the 'slow' replica. (I assume).
Project ID: 4c3b4b0e-006a-407e-90c7-9c3031cd622f
Service: 2077a108-22c6-4367-951d-41afafbfd450
Health-check endpoint is {service url}/health-check
Maybe this will resolve on its own soon, I will keep an eye on it. For now I have set the timeout to 5s so we aren't being flodded with alerts.
23 Replies
Project ID:
4c3b4b0e-006a-407e-90c7-9c3031cd622f,2077a108-22c6-4367-951d-41afafbfd450Are you using the edge proxy?
whats the edge proxy
a new home grown proxy that will replace railways current envoy proxy, it's meant as a faster (to say the least) replacement
and it's a simple toggle switch away
i dont know anything about it
i can try it out?
absolutely
please do
this happened right after i pushed some changes and forced a new build, which i know would suggest its probably something i did.. but..! the pattern doesnt make sense. at all. :3HC_think:
maybe i can log out the instance id on health check to see if its the same instnace that is responding slowly
that would be a very good idea, do you know the env var for the replica id?
yeah i think so
any link to info about this edge proxy thing?
thanks
all seems to be working now, i did enable edge proxy but i also redeployed, well, everything
so yeah uh we are back to normal
:3HC_Shrug:
awsome, let me know if you see any more issues
seems like the new prooxy kills socket io
so ive turned of the edge proxy beta on our socket io service
i wonder if thats worth reporting..?
websockets should work just fine
are you getting any specfic error?
websockets do indeed work
try with a couple of replicas?
i suspect the handshake/upgrade is failing
are you on the legacy or v2 runtime?
V2
i cant reproduce any websocket errors with the v2 or the legacy runtime, or the edge proxy, please share your errors
I'll investigate and open a new thread if needed!
Thanks
sounds good