Is high p99 latency from railway overhead is expected?
I have a very simple http service (doesn't do any IO and it immediately returns a cached value from memory). It usually takes <15ms (through private network) but I occasionally see 100 or even 200+ ms.
Somewhat related to this, I also see 502 time to time.. I'm curious if anyone has a similar experience. I don't see any sign from my logs that it is an application error. I am wondering if these hiccups are from Railway.
39 Replies
Project ID:
e9b4f99b-4d10-4dae-9884-a2be0a1637ebe9b4f99b-4d10-4dae-9884-a2be0a1637eb
the uptime kuma template didnt support testing services on the private network, are you sure youre testing the private network?
I'm not using the public template. I'm pretty sure I'm using the private network (
http://foobar.railway.internal:PORT/...).
I'm actually testing both public and private endpoints and I see that latency is noticeably lower on private endpoint.
My question is more around these unexpected spikes 🥲those are from the internal dns server
with this test i see spikes of up to 160ms
with just tcp, the biggest spike was 7ms
interesting.. so internal DNS lookup could be pretty slow
yep, but as for the 502, thats not railway, there is no gateway involved in the private network, its a wireguard tunnel
I see. I will look more into what's causing 502.
For DNS lookup latency I guess there is no way around it?
use a caching dns resolver?
though i dont know how well that would work, pretty sure the service would get a new ipv6 ip on every deployment
right 😦
ill ping char in next time i see him online, maybe he will have some ideas
thanks for looking into it!
for my info, how did you get these numbers?
uptime kuma, same as you